How to participate in ConvAI evaluation of bots
- Evaluation of bots is performed via Telegram messaging service.
- If you have no Telegram account please register:
- After you have your Telegram account created, login in Telegram and set your Username in Settings menu.
- After you have set your username add @ConvaiBot - this is official bot for our competition. (How to add peer in Telegram)
- To start a dialog type or choose a
/begin command .
- You will be connected to a peer or, if no peer is available at the moment, you’ll receive the message from @ConvaiBot
Please wait for you peer..
- Peer might be a bot or another human evaluator.
- After you were connected with your peer you will receive a starting message - a passage or two from a Wikipedia article.
- Your task is to discuss the content of a presented passage with the peer and score her/his replies.
- Please score every utterance of your peer with a ‘thumb UP’ button if you like it, and ‘thumb DOWN’ button in the opposite case.
- To finish the conversation type or choose a command
- When the conversation is finished, you will receive a request from @ConvaiBot to score the overall quality of the dialog along three dimensions:
- quality - how much are you satisfied with the whole conversation?
- breadth - in your opinion was a topic discussed thoroughly or just from one side?
- engagement - was it interesting to participate in this conversation?
- If your peer ends the dialog before you, you will also receive a scoring request from @ConvaiBot.
- Your conversations with a peer will be recorded for further use. By starting a chat you give permission for your anonymised conversation data to be released publicly under Apache License Version 2.0.