Chatterbot test protocol

  • 1 Replies
  • 2417 Views
*

trotter

  • Roomba
  • *
  • 7
  • Beware - English is not my native language
    • chatterbots.fr
Chatterbot test protocol
« on: September 30, 2011, 09:23:36 pm »
Can we make a test protocol to get an objective rating of our chatterbots ?

I though about different steps, each one more difficult than the last. Each step don't have to be very precise to ensure that a botmaster can not adapt his bot to solve them.

Step 1 : Check if answers to unknown questions are correct.
There will always be unknown questions. No chatterbot can be omniscient.
Questions will be "who, when, where, what is, why, how" about an unknown object (XXX). We can make a list about 50 questions.

Examples :
How does XXX works ?
What is the distance between XXX and XXX ?
How old is XXX ?
Why is XXX doing XXX ?

Step 2 : Check if the bot can remember the name he asked.
That's a lot of work. The bot must be able to detect a change of topic.
Bot: What's your name ?
User: brb
Bot: Nice to meet you, brb.

Must be able to avoid parasit words :
Bot: What's your name ?
User: Bob why ?
Bot: Nice to meet you, Bob why.

And must be able to stay focused if the user have a common name :
Bot: What's your name ?
User: Rabbit.
Bot: Rabbit are so fluffy !

Step 3 : Check if answer are contextualised.
We can test that by answering "and you?", "no", or "why ?".

Step 4 : Be able to understand complex sentences.

User: He says you're smart but I say you're not.
Bot: Awww, thanks you for telling me that I'm smart!

User: My grand mother said to me "you're a smart boy".
Bot: Awww, thanks you for telling me that I'm smart!


Step 5 : Logical links.
Rex is a dog. A dog have four legs. How many legs have Rex ?

------
That's just a draft. What do you think of it? Any ideas? Something missing ? Auto-learning ? General knowledge? What is a "correct" answer ?
« Last Edit: September 30, 2011, 09:33:03 pm by trotter »

*

Bragi

  • Trusty Member
  • ********
  • Replicant
  • *
  • 564
    • Neural network design blog
Re: Chatterbot test protocol
« Reply #1 on: October 01, 2011, 08:24:09 am »
That's a nice start. I would also add something that has to do with complexity of the sentences. Something like:

a dog has legs
a dog has 4 legs
a big diog has 4 legs
a very big dog has 4 tiny legs
a very big dog always has 4 tiny little leggs
....
next: can it handle 'be', 'have', 'do', other verbs correctly.

- timing: can it handle past, present, future tenses correctly. (I guess this falls under 'when')
-location: this probably falls under 'where'
- sorting: what is x-er (bigger, smaller,...), what is the x-est (highest, ....)
-

I've been thinking a while now to build some 'standard' test-case to run,... but time...

 


OpenAI Speech-to-Speech Reasoning Demo
by ivan.moony (AI News )
March 28, 2024, 01:31:53 pm
Say good-bye to GPUs...
by MikeB (AI News )
March 23, 2024, 09:23:52 am
Google Bard report
by ivan.moony (AI News )
February 14, 2024, 04:42:23 pm
Elon Musk's xAI Grok Chatbot
by MikeB (AI News )
December 11, 2023, 06:26:33 am
Nvidia Hype
by 8pla.net (AI News )
December 06, 2023, 10:04:52 pm
How will the OpenAI CEO being Fired affect ChatGPT?
by 8pla.net (AI News )
December 06, 2023, 09:54:25 pm
Independent AI sovereignties
by WriterOfMinds (AI News )
November 08, 2023, 04:51:21 am
LLaMA2 Meta's chatbot released
by 8pla.net (AI News )
October 18, 2023, 11:41:21 pm

Users Online

340 Guests, 0 Users

Most Online Today: 372. Most Online Ever: 2369 (November 21, 2020, 04:08:13 pm)

Articles