The Conversational Intelligence Challenge

NIPS 2017 Live Competition

How to participate in ConvAI evaluation of bots

  1. Evaluation of bots is performed via Telegram messaging service.
  2. If you have no Telegram account please register:
  3. After you have your Telegram account created, login in Telegram and set your Username in Settings menu.
  4. After you have set your username add @ConvaiBot - this is official bot for our competition. (How to add peer in Telegram)
  5. To start a dialog type or choose a /begin command .
  6. You will be connected to a peer or, if no peer is available at the moment, you’ll receive the message from @ConvaiBot Please wait for you peer..
  7. Peer might be a bot or another human evaluator.
  8. After you were connected with your peer you will receive a starting message - a passage or two from a Wikipedia article.
  9. Your task is to discuss the content of a presented passage with the peer and score her/his replies.
  10. Please score every utterance of your peer with a ‘thumb UP’ button if you like it, and ‘thumb DOWN’ button in the opposite case.
  11. To finish the conversation type or choose a command /end.
  12. When the conversation is finished, you will receive a request from @ConvaiBot to score the overall quality of the dialog along three dimensions:
    • quality - how much are you satisfied with the whole conversation?
    • breadth - in your opinion was a topic discussed thoroughly or just from one side?
    • engagement - was it interesting to participate in this conversation?
  13. If your peer ends the dialog before you, you will also receive a scoring request from @ConvaiBot.
  14. Your conversations with a peer will be recorded for further use. By starting a chat you give permission for your anonymised conversation data to be released publicly under Apache License Version 2.0.

Competition home