ConvAI3: Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)

SCAI workshop @ EMNLP 2020

How to participate in ConvAI evaluation of bots

  1. Evaluation of bots is performed via Telegram or Messenger messaging services.
  2. To evaluate using Messenger go to and start a chat, using your Facebook account.
  3. To evaluate using Telegram, you’ll need Telegram account. If you have no Telegram account please register:
  4. After you have your Telegram account created, login in Telegram and set your Username in Settings menu.
  5. After you have set your username add @ConvaiBot - this is official bot for our competition. (How to add peer in Telegram.)
  6. To start a dialog type or choose a /begin command .
  7. You will be connected to a peer or, if no peer is available at the moment, you’ll receive the message from @ConvaiBot Please wait for you peer.
  8. Peer might be a bot or another human evaluator.
  9. Warning! Since your peer is either human, or bot she/it could use words or language that considered profane, vulgar, or offensive by some readers. The organizers of ConvAI are not responsible for dialogs contents. It you have spotted a serious issue, please contact organizers about it by email
  10. After you were connected with your peer you will receive a starting message - a passage or two from a Wikipedia article.
  11. Your task is to discuss the content of a presented passage with the peer and score her/his replies.
  12. Please score every utterance of your peer with a ‘thumb UP’ button if you like it, and ‘thumb DOWN’ button in the opposite case.
  13. To finish the conversation type or choose a command /end.
  14. When the conversation is finished, you will receive a request from @ConvaiBot to score the overall quality of the dialog along three dimensions:
    • quality - how much are you satisfied with the whole conversation?
    • breadth - in your opinion was a topic discussed thoroughly or just from one side?
    • engagement - was it interesting to participate in this conversation?
  15. If your peer ends the dialog before you, you will also receive a scoring request from @ConvaiBot.
  16. Your conversations with a peer will be recorded for further use. By starting a chat you give permission to record your conversation and to release your anonymised conversation data publicly under Apache License Version 2.0.

Competition home