Chatterbot test protocol

  • 1 Replies


  • Roomba
  • *
  • 7
  • Beware - English is not my native language
Chatterbot test protocol
« on: September 30, 2011, 09:23:36 pm »
Can we make a test protocol to get an objective rating of our chatterbots ?

I though about different steps, each one more difficult than the last. Each step don't have to be very precise to ensure that a botmaster can not adapt his bot to solve them.

Step 1 : Check if answers to unknown questions are correct.
There will always be unknown questions. No chatterbot can be omniscient.
Questions will be "who, when, where, what is, why, how" about an unknown object (XXX). We can make a list about 50 questions.

Examples :
How does XXX works ?
What is the distance between XXX and XXX ?
How old is XXX ?
Why is XXX doing XXX ?

Step 2 : Check if the bot can remember the name he asked.
That's a lot of work. The bot must be able to detect a change of topic.
Bot: What's your name ?
User: brb
Bot: Nice to meet you, brb.

Must be able to avoid parasit words :
Bot: What's your name ?
User: Bob why ?
Bot: Nice to meet you, Bob why.

And must be able to stay focused if the user have a common name :
Bot: What's your name ?
User: Rabbit.
Bot: Rabbit are so fluffy !

Step 3 : Check if answer are contextualised.
We can test that by answering "and you?", "no", or "why ?".

Step 4 : Be able to understand complex sentences.

User: He says you're smart but I say you're not.
Bot: Awww, thanks you for telling me that I'm smart!

User: My grand mother said to me "you're a smart boy".
Bot: Awww, thanks you for telling me that I'm smart!

Step 5 : Logical links.
Rex is a dog. A dog have four legs. How many legs have Rex ?

That's just a draft. What do you think of it? Any ideas? Something missing ? Auto-learning ? General knowledge? What is a "correct" answer ?
« Last Edit: September 30, 2011, 09:33:03 pm by trotter »



  • Trusty Member
  • ********
  • Replicant
  • *
  • 564
    • Neural network design blog
Re: Chatterbot test protocol
« Reply #1 on: October 01, 2011, 08:24:09 am »
That's a nice start. I would also add something that has to do with complexity of the sentences. Something like:

a dog has legs
a dog has 4 legs
a big diog has 4 legs
a very big dog has 4 tiny legs
a very big dog always has 4 tiny little leggs
next: can it handle 'be', 'have', 'do', other verbs correctly.

- timing: can it handle past, present, future tenses correctly. (I guess this falls under 'when')
-location: this probably falls under 'where'
- sorting: what is x-er (bigger, smaller,...), what is the x-est (highest, ....)

I've been thinking a while now to build some 'standard' test-case to run,... but time...


Users Online

28 Guests, 1 User
Users active in past 15 minutes:
[Trusty Member]

Most Online Today: 46. Most Online Ever: 340 (March 26, 2019, 09:47:57 pm)