ConvAI3: Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)

SCAI workshop @ EMNLP 2020

How to participate in ConvAI evaluation of bots

  1. Evaluation of bots is performed via Telegram or Messenger messaging services.
  2. To evaluate using Messenger go to https://www.messenger.com/t/convai.io and start a chat, using your Facebook account.
  3. To evaluate using Telegram, you’ll need Telegram account. If you have no Telegram account please register:
  4. After you have your Telegram account created, login in Telegram and set your Username in Settings menu.
  5. After you have set your username add @ConvaiBot - this is official bot for our competition. (How to add peer in Telegram.)
  6. To start a dialog type or choose a /begin command .
  7. You will be connected to a peer or, if no peer is available at the moment, you’ll receive the message from @ConvaiBot Please wait for you peer.
  8. Peer might be a bot or another human evaluator.
  9. Warning! Since your peer is either human, or bot she/it could use words or language that considered profane, vulgar, or offensive by some readers. The organizers of ConvAI are not responsible for dialogs contents. It you have spotted a serious issue, please contact organizers about it by email info@convai.io.
  10. After you were connected with your peer you will receive a starting message - a passage or two from a Wikipedia article.
  11. Your task is to discuss the content of a presented passage with the peer and score her/his replies.
  12. Please score every utterance of your peer with a ‘thumb UP’ button if you like it, and ‘thumb DOWN’ button in the opposite case.
  13. To finish the conversation type or choose a command /end.
  14. When the conversation is finished, you will receive a request from @ConvaiBot to score the overall quality of the dialog along three dimensions:
    • quality - how much are you satisfied with the whole conversation?
    • breadth - in your opinion was a topic discussed thoroughly or just from one side?
    • engagement - was it interesting to participate in this conversation?
  15. If your peer ends the dialog before you, you will also receive a scoring request from @ConvaiBot.
  16. Your conversations with a peer will be recorded for further use. By starting a chat you give permission to record your conversation and to release your anonymised conversation data publicly under Apache License Version 2.0.

Competition home