Chatterbot test protocol

  • 1 Replies
  • 1558 Views
*

trotter

  • Roomba
  • *
  • 7
  • Beware - English is not my native language
    • chatterbots.fr
Chatterbot test protocol
« on: September 30, 2011, 09:23:36 pm »
Can we make a test protocol to get an objective rating of our chatterbots ?

I though about different steps, each one more difficult than the last. Each step don't have to be very precise to ensure that a botmaster can not adapt his bot to solve them.

Step 1 : Check if answers to unknown questions are correct.
There will always be unknown questions. No chatterbot can be omniscient.
Questions will be "who, when, where, what is, why, how" about an unknown object (XXX). We can make a list about 50 questions.

Examples :
How does XXX works ?
What is the distance between XXX and XXX ?
How old is XXX ?
Why is XXX doing XXX ?

Step 2 : Check if the bot can remember the name he asked.
That's a lot of work. The bot must be able to detect a change of topic.
Bot: What's your name ?
User: brb
Bot: Nice to meet you, brb.

Must be able to avoid parasit words :
Bot: What's your name ?
User: Bob why ?
Bot: Nice to meet you, Bob why.

And must be able to stay focused if the user have a common name :
Bot: What's your name ?
User: Rabbit.
Bot: Rabbit are so fluffy !

Step 3 : Check if answer are contextualised.
We can test that by answering "and you?", "no", or "why ?".

Step 4 : Be able to understand complex sentences.

User: He says you're smart but I say you're not.
Bot: Awww, thanks you for telling me that I'm smart!

User: My grand mother said to me "you're a smart boy".
Bot: Awww, thanks you for telling me that I'm smart!


Step 5 : Logical links.
Rex is a dog. A dog have four legs. How many legs have Rex ?

------
That's just a draft. What do you think of it? Any ideas? Something missing ? Auto-learning ? General knowledge? What is a "correct" answer ?
« Last Edit: September 30, 2011, 09:33:03 pm by trotter »

*

Bragi

  • Trusty Member
  • ********
  • Replicant
  • *
  • 564
    • Neural network design blog
Re: Chatterbot test protocol
« Reply #1 on: October 01, 2011, 08:24:09 am »
That's a nice start. I would also add something that has to do with complexity of the sentences. Something like:

a dog has legs
a dog has 4 legs
a big diog has 4 legs
a very big dog has 4 tiny legs
a very big dog always has 4 tiny little leggs
....
next: can it handle 'be', 'have', 'do', other verbs correctly.

- timing: can it handle past, present, future tenses correctly. (I guess this falls under 'when')
-location: this probably falls under 'where'
- sorting: what is x-er (bigger, smaller,...), what is the x-est (highest, ....)
-

I've been thinking a while now to build some 'standard' test-case to run,... but time...

 


All is lie or is it cake?
by LOCKSUIT (Future of AI)
September 21, 2018, 09:33:41 pm
Airbender
by Zero (General Project Discussion)
September 21, 2018, 11:01:16 am
Good language for metaprogramming?
by Zero (General Chat)
September 21, 2018, 11:00:26 am
AI interfacing with equipment mainteance work
by Art (General AI Discussion)
September 20, 2018, 02:19:51 pm
XKCD Comic : Curve-Fitting
by Tyler (XKCD Comic)
September 20, 2018, 12:00:26 pm
Floatbot - chatbot platform
by Zero (General Chatbots and Software)
September 20, 2018, 08:36:04 am
Finding work in a better way
by Schmemly (General Chat)
September 19, 2018, 05:13:03 pm
We can't prove or disprove some sentences
by ranch vermin (General Chat)
September 18, 2018, 10:52:56 pm

Users Online

31 Guests, 0 Users

Most Online Today: 43. Most Online Ever: 208 (August 27, 2008, 09:36:30 am)

Articles