Dialogue systems and conversational agents – including chatbots, personal assistants and voice control interfaces – are becoming increasingly widespread in our daily lives. NIPS is sponsoring an open competition to create a chatbot that can hold an intelligent conversation with a human partner.
|Participate as a human evaluator volunteer|
Warning! Some dialogs cound contain words or language that considered profane, vulgar, or offensive by some readers. The dialogs are partially automatically generated and partially generated by anonymous volunteers, so the organizers of ConvAI are not responsible for dialogs contents. It you have spotted a serious issue, please contact organizers about it by email firstname.lastname@example.org.
Recent advances in machine learning have sparked a renewed interest for dialogue systems in the research community. In addition to the growing real-world applications, the ability to converse is closely related to the overall goal of AI. This NIPS Live Competition aims to unify the community around the challenging task: building systems capable of intelligent conversations. Teams are expected to submit dialogue systems able to carry out intelligent and natural conversations about specific news articles with humans. At the final stage of the competition participants, as well as volunteers, will be randomly matched with a bot or a human to chat and evaluate answers of a peer. We expect the competition to have two major outcomes: (1) a measure of quality of state-of-the-art dialogue systems, and (2) an open-source dataset collected from evaluated dialogues.
|April, 2017||Registration is open. Registered teams are allowed to submit applications and solutions for the Qualification Round.|
|25th of June, 2017||Qualification Round is closed.|
|2nd of July, 2017||Notification of acceptance. Teams for the Human Evaluation Round are selected.|
|24th-30th of July, 2017||Human Evaluation Round. One week NLP summer school - DeepHack Turing. Teams, school participants and volunteers evaluate solutions of qualified teams. Evaluation is done remotely via a messaging platform.|
|1st of September, 2017||1st Dataset is published. Data collected at the Human Evaluation Round is published. Dataset is integrated with the ParlAI framework for the efficient training of models. Teams tune their solutions on the 1st Dataset.|
|12th of November, 2017||Submission of conversational agents is closed. Teams submit final solutions for the NIPS Live Competition.|
|20th of November - 3rd of December, 2017||Pre-NIPS Human Evaluation. Teams and volunteers start to evaluate solutions of teams. Evaluation is done remotely via a messaging platform.|
|4th-9th of December, 2017||Conversational Intelligence Live Competition at NIPS. Teams, conference participants and volunteers continue to evaluate solutions of teams. Announcement of winners.|
The competition consists of four rounds.
Both human evaluators and dialogue agents complete the same task.
The submitted system could be excluded from competition temporarily or completely, if there were issues with system used language. We encourage the participants to be guided by common sense, choosing the constraints on system vocabulary.
If there was a spotted issue the participating system is disqualified from competition temporarily, until the issue is resolved. If issue is not resolved until the competition’s end, the system will be disqualified completely from the competition.
The submitted system will be provided with virtual machine with these constraints:
If submitted system would have any issues on running on provided machine, the team would be contacted by organizers to resolve the issue.
Dataset collected during competition will be distributed under Apache 2.0 license.
Participants have to make their systems available open source following the end of the competition, as the goal of the challenge is to enable the community to make sustained progress towards the creation of conversational intelligence. The data used by teams to pretrain models should be made open access or available on request for academic purposes.
10,000 USD for the winning dialogue agent.
We also invite teams to use ParlAI in their solutions and submit applications for ParlAI request for proposals to $10,000 to $20,000 for academic research.
Mikhail Burtsev, Valentin Malykh, MIPT, Moscow
Ryan Lowe, McGill University, Montreal
Iulian Serban, Yoshua Bengio, University of Montreal, Montreal
Alexander Rudnicky, Alan W. Black, Shrimai Prabhumoye, Carnegie Mellon University, Pittsburgh
Please contact email@example.com for information on sponsorship.
Please send your requests to firstname.lastname@example.org