Blog

Learn About Our Meetup

5000+ Members

MEETUPS

LEARN, CONNECT, SHARE

Join our meetup, learn, connect, share, and get to know your Toronto AI community. 

JOB POSTINGS

INDEED POSTINGS

Browse through the latest deep learning, ai, machine learning postings from Indeed for the GTA.

CONTACT

CONNECT WITH US

Are you looking to sponsor space, be a speaker, or volunteer, feel free to give us a shout.

[D] Need tool for Human evaluation of Generative Dialogue Models

[D] Need tool for Human evaluation of Generative Dialogue Models

Hi, does anyone know of any tools / softwares for manual evaluation of generative dialogue models? I looked into a few annotation tools and also psiturk ( https://github.com/NYUCCL/psiTurk ), but none of them natively supports such a task.Psiturk also seems to have a bit of learning curve which I am not very eager to get into, at the moment. The type of annotation required is similar to A/B testing. The user/annotator will be given a dialogue history and 2/3 sample responses generated by different models. He/She will need to tell which of the responses is better in terms of coherence and grammatical correctness.

Following is example of such an annotation tool. (From https://arxiv.org/pdf/1605.06069.pdf , this source code isn’t public.)

https://i.redd.it/0ij86ijxtkk31.png

submitted by /u/bytestorm95
[link] [comments]

Next Meetup

 

Days
:
Hours
:
Minutes
:
Seconds

 

Plug yourself into AI and don't miss a beat

 


Toronto AI is a social and collaborative hub to unite AI innovators of Toronto and surrounding areas. We explore AI technologies in digital art and music, healthcare, marketing, fintech, vr, robotics and more. Toronto AI was founded by Dave MacDonald and Patrick O'Mara.