Home > Events > CLIP Colloquium: Joao Sedoc (Johns Hopkins)
S M T W T F S
 
 
 
 
 
1
 
2
 
3
 
4
 
5
 
6
 
7
 
8
 
9
 
10
 
11
 
12
 
13
 
14
 
15
 
16
 
17
 
18
 
19
 
20
 
21
 
22
 
23
 
24
 
25
 
26
 
27
 
28
 
29
 
30
 

CLIP Colloquium: Joao Sedoc (Johns Hopkins)

Time: 
Wednesday, December 11, 2019 - 11:00 AM to 12:00 PM
Location: 
5105 Brendan Iribe Center

 

Evaluating Conversational Agents

Abstract: There has been a renewed focus on dialog systems, including non-task driven conversational agents. Dialog is a challenging problem since it spans multiple conversational turns. To further complicate the problem, there are many possible valid utterances that may be semantically different. This makes automatic evaluation difficult, which is why the current best practice for analyzing and comparing dialog systems is the use of human judgments. This talk focuses on evaluation, presenting a theoretical framework for the systematic evaluation of open-domain conversational agents, including the usage of Item Response Theory (Lord, 1968) for efficient chatbot evaluation and evaluation set creation.  We introduce ChatEval (https://chateval.org) a unified framework for human evaluation of chatbots that augments existing tools and provides a web-based hub for researchers to share and compare their dialog systems.

Joao Sedoc (Assistant Research Professor, Department of Computer Science)