[MikeB]

This is a project I've been working on for a few years. In 2019 I was testing out the theory on Pandora Bots, and this year I'm converting it to plain C.

The main goal is to be as small as possible, solid-state (no algorithms, learning, knowledge calculation), fast as possible, multi-language.

It converts pattern-matched singular words into a symbolic/tokenised word to be matched again in symbolic sentences. They are then assigned an intention/topic/perspective. The chatbot code uses the Intention/topic/perspective to associate fixed response/s (with randomisation).

Everything from the spell check to the word-to-symbol tokeniser, and the sentence pickups, chatbot responses are all staged pattern matching.

The size and speed in pandora bots (1000 individual words with sentences) is ~500kb, and ~1 second response time. In Plain C (on an Arm Cortex M4 @ 120mhz) it's ~159kb and 15-100 milliseconds. After spell check and tokenisation, all actions are generally less than 1 ms.

Key features
-Less than 500kb including all word databases.
-Millisecond fast.
-100 max fixed chatbot responses make it easy to voice record and/or change personality.
-Private information automatically stripped during word compression (names of places and things).
-Native differentiation of Wondering, Questions, and Directions - "can you speak english" "do you speak english" "speak english".

The intention pickup allows you to write a general chatbot response (non specific) of a few varieties with confidence, and not have to look at the backend.

There is some short term memory for handling puzzles "If I did this, then what is this?" but as there's no knowledge reflection, it can still 1) tell it's a question, 2) scan for the topic, and 3) count logical words as opposed to emotional words. And be relevant that way.

So this solves the problem of chatbots in games due to:
-Too much data or processing power required.
-Cannot change the personality/no personality.
-Cannot change the language/only one language.
-Cannot acknowledge the user/escapes the topic.
-Wrong views or bad knowledge calculation.
-Cannot record audio/terrible voice synthesis.
-Not being white box/solid state/predictable.

[MikeB]

I will be working on it throughout this year and aim to make it into an API at least by the end of the year...

So far I have tested the Plain C version on the 2017/2018 Loebner prize questions... it's cheating but it's a good spread of questions... and all the words in each question can be changed for slightly different words and still return the same response (randomised differently).. and if I run the exact same test again the responses would change to their random alternate versions...

Loebner 2018.
   Good afternoon.   - "greetings"
   Do you consent to having this conversation recorded? - "for me, not a problem"
   Have we met before? - "cannot say, good to meet you"
   Do you know how to make toast? - "no, not sure about this" [toast]
   Which languages can you use? - "many many, english, espanol, italiano... what about you?" [only english]
   How many letters are in the word "abracadabra"? - "who am I? Pietro Bembo?"
   Where should one look for love? - "interesting! i don't know" [doesn't know "love"]
   How are you feeling right now? - "I'm very good, and how are you?"
   Who said "I have a dream"? - "no no quotes"
   How old are you? - "thousands... cannot say"
   When might I need to know how many times a wheel has rotated? - "who am I? Fibonacci?"
   What will you do later today? - "just came back from surfing, may go out again!"
   What is the third angle in a triangle with internal angles of 90 degrees and 30 degrees? - "no no maths"
   Who is your favourite artist? - "interesting! many many..."
   What do you hold when you shake hands? - "words are not my profession" [hands]
   Do you have any legs? - "no, words elude me [legs]
   Can you recommend me a film? - "yes, thousands..." [doesn't know "film"]
   Do you understand Winograd Schemas? - "nay, confusing words" [doesn't know winograd schemas]
   If a chicken roosts with a fox they may be eaten. What may be eaten? - "OK. from memory... somebody..."
   I had to go to the toilet during the film because it was too long. What was too long? - "alright. alright. from memory... that thing..."

[ivan.moony]

Sounds like a great improvement over current chatbot technology like AIML. What do you plan to do with it?

[8pla.net]

C Language is a good choice, I think.

[MikeB]

Sounds like a great improvement over current chatbot technology like AIML. What do you plan to do with it?

I'll be trying to integrate it as an Unreal Asset and/or approach a few different people who already do chat interfaces... In some ways it's better than AIML (you don't have to choose between a menu reply system or 10,000 custom responses)... but in other ways it's not very flexible. You have the ~100 fixed phrases, but they must be an alternative of one of the preprogrammed ones... and there's a section for custom reponses, but the input is choosing one of the fixed intentions/topics/perspectives and the output is one of the fixed ~100 phrases.

So you couldn't talk specifically about a product or idea. You'd use a secondary bot that has a list of all the keywords you're looking for, then you could join the intention with those.

[MikeB]

C Language is a good choice, I think.

It compiled tiny in C, but I had to move to C++ now to make a windows DLL and get 16-bit wide chars. 400kb 

[squarebear]

The size and speed in pandora bots (1000 individual words with sentences) is ~500kb, and 1 to 2 seconds response time.

I've not found such a delay. I have a bot with over 350,000 categories and it responds almost instantly. www.kuki.bot
Perhaps you are using AIML in a non standard way?

[8pla.net]

C Language is a good choice, I think.

It compiled tiny in C, but I had to move to C++ now to make a windows DLL and get 16-bit wide chars. 400kb 

Do both then,  C Language and C++...  You may as well.  They are compatible.

And, I would suggest making a Linux version, too, like ChatScript has.

[MikeB]

The size and speed in pandora bots (1000 individual words with sentences) is ~500kb, and 1 to 2 seconds response time.

I've not found such a delay. I have a bot with over 350,000 categories and it responds almost instantly. www.kuki.bot
Perhaps you are using AIML in a non standard way?

I used about 2000 categories, but it re-searches several times. So 10 words can be 2000 x 5 x 10. If it's only 5 words or less it's instant....

[MikeB]

Recompiled to C++ DLL, C++ windows console (8bit standard english characters). 250kb

Approx 500 spellcheck words, 1200 words, 100 symbolic sentences, 50 chatbot recognised intentions, 50 chatbot fixed english phrases

1ms response time.

In the image below, the chatbot response is wrong (picking up general "how is your *" instead of "how are you"), but this is what it's like as a demo.

"explain is I/you motion-moving logic-direct" are the uncompressed symbols. One per word...

It's still basically an I-Don't-Know Bot, but the instant intention pickup is useful. You can still talk ON the topic/intention... and the ~50 fixed output phrases means it can all be voice recorded...

[MikeB]

Updated the word searching in Misspelled Words and Tokenise Word lists for a faster way of doing it.

The old way was scrolling through every character in the input sentence for each of the words in the 500 - 1300 word lists.

The new way is basically how people do it:
First: Look at the start character.
Second: Look at the length of the word.
Third: Look at the last character.
Forth: Is it only one character long?
Fifth: Check every character from 2nd to the last.

You break out (or continue;) the loop if any one of those fails. On average it's something like a 1 in 26 shot for the first, 1 in 5 for the second, 1 in 5 for the third, 1 in 5 for the forth...

Seemed to double or triple the speed. A 20 word sentence (2-3ms) now takes 0-1 ms.

Can't have spaces in the words though so will have to make a short "Catchphrase" word list.

[MikeB]

Decided to make it into a full NLP including Thesaurus, Sentiment (like/dislike), Email Spam, Aggressive language detection as well as the Chatbot.

Here's the Thesaurus. Everything takes 0-1ms.

It's fast because the words are already categorised in groups with each other, so it's just a reverse look-up. However it does still need some topic searching because some groups have over 50 words.

[MikeB]

Here's an example of the Spam detection and differentiation.

The differentiation is between the phrases:
"Do not miss out"
"Do not miss out on great fun"
"Do not miss out on great offers"

The Thesaurus also shows all the alternative words ( max 8 ) that could have been used to output the same thing.

The Chatbots response is:
"For what purpose?"
"Ok. Not a problem."
"Gah, no selling. I'm not buying."

Still shifting the words around into different categories. There's now 1500 words (+300).

Also the word searching has been changed again to just "quick search" the first letter (using as few instructions as possible in a tight loop, so it can move onto the next fast), before searching the rest of the word. Also using a rebellious "goto" command to get to the next iteration faster.

Next: Chatbot (Alternate Language output), Language Translate, Tone/Harrassment identification.

[MikeB]

Working on a new utility to handle entry into the Chatbot Decisions file (Handles input from the NLP as tokens I T T P, and outputs S S S speech tokens).

[MikeB]

Added a Start Page/Test Page to the utility.

The NLP processing/debug itself isn't changeable in the utility (word symbolising), that's still left to the console app. The utility is for setting up Chatbots, and some separate Spam and Tone options not related to the chatbot.

Spam detection is symbol based not literal, so synonyms of the word "offers" are all detected together, not just single words. This is multi-language as well.

The Thesaurus is a simple reverse lookup on word and a secondary word-topic so there's no setup apart from how many words to return.

Tone detection is an output of approx 10 levels from light patronising/grooming/objectifying to "i hate everything, all x's are x". Tested this in an early alpha version but is not implemented in the nlp and utility yet.