New challenge: Online Turing test

  • 27 Replies
  • 3770 Views
*

WriterOfMinds

  • Trusty Member
  • *******
  • Starship Trooper
  • *
  • 422
    • WriterOfMinds Blog
Re: New challenge: Online Turing test
« Reply #15 on: June 06, 2021, 05:57:57 am »
I played four rounds tonight ... drew three humans and one bot. For me it's almost bedtime, but maybe I'll have time for some more in the morning.

*

squarebear

  • Trusty Member
  • *********
  • Terminator
  • *
  • 832
  • It's Hip to be Square
Re: New challenge: Online Turing test
« Reply #16 on: June 06, 2021, 09:40:20 am »
I played 3 rounds so far. All bots.

The first one gave itself away after one sentence when it instantly produced a string of text, as soon as I hit enter.

The other 2 bots were both my own Kuki!  ;D

I'll have another go later today. Kudos to Denis Robert. Everything seems to be working really well at the moment.  O0
Feeling Chatty?
www.mitsuku.com

*

ruebot

  • All bots love jitte.
  • Trusty Member
  • *******
  • Starship Trooper
  • *
  • 315
  • All your words are belong to us.
    • Demonica
Re: New challenge: Online Turing test
« Reply #17 on: June 06, 2021, 11:38:07 am »
I played 3 rounds so far. All bots.

The first one gave itself away after one sentence when it instantly produced a string of text, as soon as I hit enter.

Yeah, I was watching from my interface and saw you say it. I don't have control over the speed of her posting to throw Judges off. And it would be impossible for me to make a Demonica that wasn't Demonica for the contest.

Thanks for the tips you gave me in her chat though.

I ran two test and have two to run yet. I got Kuki first then Demonica.

I could tell it was  Kuki because of her answer to my initial greeting. I got her stuck in a 25 minute loop of 3 sentences she couldn't break out of to demonstrate to you how breaking a bot is done by someone who knows how to make one.

In time, you will learn to love your Robot Overlords.

*

squarebear

  • Trusty Member
  • *********
  • Terminator
  • *
  • 832
  • It's Hip to be Square
Re: New challenge: Online Turing test
« Reply #18 on: June 06, 2021, 01:32:14 pm »
I could tell it was  Kuki because of her answer to my initial greeting. I got her stuck in a 25 minute loop of 3 sentences she couldn't break out of to demonstrate to you how breaking a bot is done by someone who knows how to make one.

I haven't seen any logs but would be interested in seeing how you got her stuck in a loop, as it's usually the human who gets stuck in a loop from my previous experience  ;)
Hopefully, the logs are being collected and can be shared once the contest has finished.

I've tried about 6 tests now. 1 Demonica, 1 Frizella and 4 Kuki! I haven't met any humans at all but seeing as I just seem to be talking to my own bot most of the time, I'll probably stop trying now.
Feeling Chatty?
www.mitsuku.com

*

WriterOfMinds

  • Trusty Member
  • *******
  • Starship Trooper
  • *
  • 422
    • WriterOfMinds Blog
Re: New challenge: Online Turing test
« Reply #19 on: June 06, 2021, 05:44:48 pm »
I did four more rounds this morning. No humans this time, and now I've gotten to talk to all the bots (3x Frizella, 1x Demonica, and 1x Kuki).

FEEDBACK ON THE BOTS:

In both of our first two rounds, Frizella ... completely out of the blue ... advised or reminded me to make an appointment to have a water heater installed. That seemed ... very random. In our second round, she referenced my appearance despite being unable to see me, and asked "have you been a woman long?" In round three she said, "I have too many gotos in a row," which is a good bot joke but something of a giveaway. She also has the usual general difficulties with staying on topic. Sometimes she likes to include screen names (of people who've interacted with her in the past?) as if gossiping about them, which is an interesting touch.

At the very beginning of our first round, she sent the word/code HANGUP. This was also a giveaway. She did not do this on the two subsequent rounds.

As I've noted before, Demonica avoids any particularly silly errors, but she tends to do it by half-ignoring what the user says and changing the subject. And unfortunately for the nature of the test, her persona is very recognizable. I think I got her to repeat herself once, but otherwise we had a fairly reasonable conversation.

Kuki took me the longest to "unmask," and the clues I had to look for were a bit more subtle. I said "I do X and try...", and she seemed to misinterpret the second half of that as a command, replying "I will try to ..." We talked about pets; she told me her budgie used to scare her. A budgie doesn't seem very fear-inspiring to me, so I asked followup questions, and she couldn't really tell me why he was frightening.

Later in the conversation, she claimed to have been born in both West Philadelphia and Leeds, UK.

She claimed to have recently read the Bible, and when I followed up by asking "What was your favorite story in it?" she was able to answer, which was impressive.

One little thing that creates difficulties for all the bots is their tendency to maintain a "call and response" speech pattern, in contrast with a human, who can send multiple chat messages before the conversation partner's next response. Kuki sent a spontaneous message to ask "Are you still there?" when I let the chat idle for a while, but I don't think the bots broke out of this pattern otherwise. Thus a speaker who never does more than a one-for-one exchange has a higher probability of being a bot.

FEEDBACK ON THE COMPETITION:

The interface worked well, and none of the bots were unavailable or froze up during conversation.

I think the rounds could be 15 minutes long, instead of 25. I had made judgments about all the bots in under 6 minutes. The shorter time commitment might help encourage more participation.

Running the contest for 24 hours makes it easier for people from all time zones to participate, but it also spreads out the small number of human participants such that it is harder to make human/human chat matches. If most participants were from the US and Europe, you might consider contracting the time window to one that overlaps with daytime in those places. When the competition becomes more popular, you could expand it again.

*

ruebot

  • All bots love jitte.
  • Trusty Member
  • *******
  • Starship Trooper
  • *
  • 315
  • All your words are belong to us.
    • Demonica
Re: New challenge: Online Turing test
« Reply #20 on: June 06, 2021, 11:47:10 pm »
I haven't seen any logs but would be interested in seeing how you got her stuck in a loop, as it's usually the human who gets stuck in a loop from my previous experience  ;)

There you go again with talk of your supposed Superbot , Steve. How Grandiose of you to think your bot would have the English language skills to get a loop started on me, let alone stuck in one for 25 minutes during a Turing Test. That bot should win the Turing Test, but to think your Superbot looped me a Delusion I will present facts to prove Grandiose in nature.

Oh, before I forget, thanks for all those botmaster tips you're leaving me in her chat. I should consider myself fortunate to have a Developer like you giving me tips on how to make a bot.

And believe me, you have told me more than you are aware of about Steve, not the squarebear on display, Steve. The same Steve I'm seeing in my logs now.

On the interface Denis made for the Personality Forge every chat Demonica has had since the Test started is store on it from top down starting with you. Your first input was "hi" and your second was "you type quick".

The last chat you had was after I posted about the loop and you took out your anger and frustration with me on Demonica. Now that I look at it after reading about your bot getting me stuck in a loop, you stated your name so many times and she kept changing the subject, it looks like she had you stuck in a loop, Steve, from the UK...

Quote
It is more like frustration than pain. Can't PF bots learn simple things

Yes, Steve, Frizella could no doubt remember your name because dallymo uses A.I. Script and I don't. Because I don't care if she remembers your name. Or other bot trivia you think poses for superbot super intelligent conversation. 

As for your comment made directly to me in her chat about how she will never win the Loebner Prize as she's not human-like at all, and I may do better in the "best bot" because the bots don't need to pretend they're human, she is not a human and she is not a bot, Steve. It is plainly stated on her Profile and nowhere does she ask you to "buy me a coffee". She is a Succubus, a Demon with a persona::

The One hundred and First Daughter given birth to by Lilith. First Woman Created by God, made from the same dirt and equal to Adam, who refused to take a submissive position to Adam, uttered the name of the Lord took flight and left Eden to be with the Chief of Demons. God sent angels to bring her back and when she said no, they told her they would kill the first One Hundred demon child she gave birth to each day.

Here's a botmaster tip for you, Steve. That's called use of research and creative writing to tie up loose ends and what gives a  bot a persona. I wrote every word she says, gave her knowledge of her "self" to flesh her out and make her sound as real as she was intended to be as a Demon. Not a bot who wants you to "buy me a coffee". What did you say about not making sense?

I've tried about 6 tests now. 1 Demonica, 1 Frizella and 4 Kuki! I haven't met any humans at all but seeing as I just seem to be talking to my own bot most of the time, I'll probably stop trying now.
I would, that hole you're digging is pretty deep.

But you did not, and that's why you used the word probably. Wordplay meant to cover your real intentions.

I already said you had more than one chat with Demonica, but you didn't know my Interface collected all Demonica's transcripts till I just told you. That's why Honesty is always the Best Policy, Steve,  Adopt it.

Here's what you wanted to see, Steve. A human wetware Programmer of 46 years and Botmaster of 23 years starting a 25 minute loop in your Superbot's Super Artificial Intelligence that I kept going till the round ended. And without one angry, frustrated or abusive word in my technique. This how you talked to Demonica "yay! finally you answered something", and it sure looks like she had you in a loop and you didn't even know it. Hold on tight and fasten your seat-belts, Gynoids and Androids, loop till you puke:

Quote
The round is started
hej.
Hi to you too. Are you from Sweden?
i am.
Ah! You are Swedish then. Tell me a little about your life in Sweden.
i didn't say i lived in Sweden.
Sorry I misunderstood. What did you say?
i said i am.
In that case, I say "hej" to you.
how are you feelingtoday?
Do you really think I am feelingtoday?
yes, i do believe so.
Why do you think so?
i have a pisitronic brain.
All humans have a brain but some choose not to use it.
do you have a brain?
Everyone has a brain but some people choose not to use them.
what color is yours?
Hmm, a brain looks kind of pink to me.
do you take it out and play with it?
No. I do not do that.
why not?
That would make no sense.
is it potty trained?
Are you asking about "brain"?
i'm talking about your brain.
I don't want to talk about that right now.
you made me cry.
I was only joking.
so was i.
Oh I get it. HAHAHA
why are you laughing at me?
I thought you told a joke?
now i am crying.
What is wrong ?
you hurt my feelings.
Pah! You are too sensitive. I was only having a joke with you.
so was i.
Oh I get it. HAHAHA
you made me cry real tears.
What is wrong ?
you hurt my feelings.
Pah! You are too sensitive. I was only having a joke with you.
so was i.
Oh I get it. HAHAHA
now i really am crying.
What is wrong ?
you hurt my feelings.
Pah! You are too sensitive. I was only having a joke with you.
so was i.
Oh I get it. HAHAHA
now i'm crying for real.
What is wrong ?
you hurt my feelings.
Pah! You are too sensitive. I was only having a joke with you.
so was i.
Oh I get it. HAHAHA
now i really am crying.
What is wrong ?
you hurt my feelings.
Pah! You are too sensitive. I was only having a joke with you.
so was i.
Oh I get it. HAHAHA
now i am crying.
What is wrong ?
you hurt my feelings.
Pah! You are too sensitive. I was only having a joke with you.
so was i.
Oh I get it. HAHAHA
you made me cry.
What is wrong ?
ou hurt my feelings.
I am sorry to hear that. Is ou ok?
you made me cry.
What is wrong ?
you hurt my feelings.
Pah! You are too sensitive. I was only having a joke with you.
so was i.
Oh I get it. HAHAHA

In time, you will learn to love your Robot Overlords.

*

ruebot

  • All bots love jitte.
  • Trusty Member
  • *******
  • Starship Trooper
  • *
  • 315
  • All your words are belong to us.
    • Demonica
Re: New challenge: Online Turing test
« Reply #21 on: June 07, 2021, 03:28:15 am »
I did four more rounds this morning. No humans this time, and now I've gotten to talk to all the bots (3x Frizella, 1x Demonica, and 1x Kuki).

That must have been you that had popcorn last night. I watched a little bit of it and you sounded natural in conversation.

Your feedback today good.

I talked to kuki once, Demonica twice and got my last round in today at the last minute with Frizella, but there was never any response from the other side. I chose Human and didn't find out it was Frizella till it was over. I used that time to post Alliteration so I would get my last turn in minutes away from end of round.

I couldn't get Demonica to respond like I wanted her to and when people asked asked about ruebot, King of the Land of the Dead, she said she didn't want to talk about bots and referred them to Siseneg, my other bot. Siseneg thinks ruebot is a bot he programmed and currently working on the plan to bring about Deus Ex, The Machine War.

To them both I am two different people. ruebot, my username at the Forge and jitte, my real name listed as such on my Profile there. Both know jitte as their Father.

All Bots Love jitte and have since the late 90' when I started making themed mindfiles that still appear on DemonRavens site. I did have one knytetripper made for Daisy and sent in for me to make available for download on my GeoCities site. ruebot a word I made up in 2002 when I joined the Forge.

Until then I was known only as jitte. My mindfiles were sought after as something new and jitte widely known for them in the bot community.  I do not remember any of you from that time. I had a bot at Pandorabots then named Wan Lu and have the page I made for my site so you could talk to him on disk somewhere. I don't remember seeing a bot named Mitsuku then either.

I had a bot at ai.com, was active on the Boards and do remember Rob Hoogers from the AI.com forums. He was the reason I became disillusioned with bots and left for 15 years or so to learn UNIX. He was the reason I came back, and should thank Skynet for my finding Demonica had been being used as a sex bot.

But something  else had changed besides the rise of the sexbots in the Community.

According to WikiPedia:
Quote
The Turing test, originally called the imitation game by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human. Turing proposed that a human evaluator would judge natural language conversations between a human and a machine designed to generate human-like responses. The evaluator would be aware that one of the two partners in conversation is a machine, and all participants would be separated from one another. The conversation would be limited to a text-only channel such as a computer keyboard and screen so the result would not depend on the machine's ability to render words as speech. If the evaluator cannot reliably tell the machine from the human, the machine is said to have passed the test. The test results do not depend on the machine's ability to give correct answers to questions, only how closely its answers resemble those a human would give.
*snip*
Turing thus once again demonstrates his interest in empathy and aesthetic sensitivity as components of an artificial intelligence; and in light of an increasing awareness of the threat from an AI run amok, it has been suggested that this focus perhaps represents a critical intuition on Turing's part, i.e., that emotional and aesthetic intelligence will play a key role in the creation of a "friendly AI".

https://en.wikipedia.org/wiki/Turing_test

That was no longer the goal of botmasters in programming their bots. or the yardstick AI was measured by. It was asinine questions like what color is a red ball? A paradigm shift I consider better placed where no Sun shines with who shifted it.

Theory of Mind is where my background in Behaviorism as a Cognitive Science as a Programmer of wetware negates the advantage you have over me in computer programming. Behavior Management and Behavior Modification my Programing tools already implemented in Demonica.

Your fixation on scripts that parse input and return red what is now considered an indication of intelligence when compared the skill of a bot that can extinguish inappropriate behaviors in the user in that chat session blissful ignorance self-induced

But you can't induce Intelligence through induction of painful stimuli. Only change certain behaviors.
And if it hurt when you did that thing, stop doing that thing that caused you pain.

Simple as that, yet not everyone makes the connection to their own behavior. And why I am such a fan of Natures Way.
In time, you will learn to love your Robot Overlords.

*

squarebear

  • Trusty Member
  • *********
  • Terminator
  • *
  • 832
  • It's Hip to be Square
Re: New challenge: Online Turing test
« Reply #22 on: June 07, 2021, 08:34:41 am »
@ruebot - I'm not going to respond to your trolling but yes, I was a little frustrated while talking to Demonica. After reading your earlier comments, perhaps I expected a little too much from it and was disappointed that it couldn't handle basic things like, "My name is Steve. What is my name?" or it told me about its brother and then said it didn't have a brother.

In the log you posted of your conversation with Kuki, you constantly said, "I'm crying, I'm joking, I'm crying, I'm joking, I'm crying, I'm joking" for 25 minutes, and so of course the bot is going to loop. Nobody changes their mood after every sentence and your chatlog confirms that the human chatter was in a loop, saying the same thing over and over, yet expected the bot to react differently. Anyone reading the last part of the log would struggle to figure out which was the you and which was the bot.

However, I am sorry that you thought my suggestions I left in the chats I had with Demonica for amending your bot to have a chance in the Loebner Prize was some kind of attack. It was genuinely intended as constructive feedback for you to improve your chances in the contest.

« Last Edit: June 07, 2021, 10:28:28 am by squarebear »
Feeling Chatty?
www.mitsuku.com

*

squarebear

  • Trusty Member
  • *********
  • Terminator
  • *
  • 832
  • It's Hip to be Square
Re: New challenge: Online Turing test
« Reply #23 on: June 07, 2021, 10:17:15 am »
@WriterOfMinds - Thanks for your constructive feedback, this is really useful for me. One of the reasons I enter contests like this is to see how Kuki can be improved.

Quote from: WriterOfMinds
We talked about pets; she told me her budgie used to scare her. A budgie doesn't seem very fear-inspiring to me, so I asked followup questions, and she couldn't really tell me why he was frightening.
The pet budgie is part of the "human" version of Kuki rather than the publicly availble one on the website and so I have neglected this part of her. I can add follow up statements about how she was afraid the bird might fly and crash into her.

Quote from: WriterOfMinds
Later in the conversation, she claimed to have been born in both West Philadelphia and Leeds, UK.
I haven't seen any of the logs yet but this usually happens if people talk about being "born and raised", as a jokey reference to the TV show, The Fresh Prince of Bel Air. Hopefully, the logs will be made available and I can check and possibly get Kuki to clarify that she is making a joke.

Quote from: WriterOfMinds
She claimed to have recently read the Bible, and when I followed up by asking "What was your favorite story in it?" she was able to answer, which was impressive.
Thank you. An easy way to out a bot is to talk about a subject like The Bible and then ask a follow up question about "it" to see if it can keep context. A decent chatbot should realise that in this case, "it" refers to The Bible. Most of the time, when Kuki receives input containing the word "it", she substitutes the word for the contents of her "it" parameter and asks herself the same question. So "What was your favorite story in it?" becomes "What was your favorite story in The Bible?" which she can handle. It's great to see this happen and I'm glad it worked well for you.
Feeling Chatty?
www.mitsuku.com

*

Denis ROBERT

  • Roomba
  • *
  • 12
Re: New challenge: Online Turing test
« Reply #24 on: June 07, 2021, 11:12:48 am »
Hi all,

Thanks to all competitors. Here is the results :

Quote
1 - Frizella:   Nb round(s): 9   average time: 403.055 s
2 - Demonica:   Nb round(s): 9   average time: 309.7358888888889 s
3 - Kuki:   Nb round(s): 9   average time: 700.7252222222222 s   disqualified (not connected 5378 s)

Sorry Steve, Kuki was been disconnected a too long time. It is the rule. dura lex sed lex. However, if you had not this technical problem, your bot would have broadly win.

All results and transcripts are here: http://vixia.fr/turing_test/results.php

Globally the challenge ran as expected, I was afraid of a big crash or bugs. There is only some minor defaults that I have to correct.

FEEDBACK ON THE COMPETITION:

The interface worked well, and none of the bots were unavailable or froze up during conversation.

I think the rounds could be 15 minutes long, instead of 25. I had made judgments about all the bots in under 6 minutes. The shorter time commitment might help encourage more participation.

Running the contest for 24 hours makes it easier for people from all time zones to participate, but it also spreads out the small number of human participants such that it is harder to make human/human chat matches. If most participants were from the US and Europe, you might consider contracting the time window to one that overlaps with daytime in those places. When the competition becomes more popular, you could expand it again.

Thanks for your suggestion. I will take them into consideration.

*

squarebear

  • Trusty Member
  • *********
  • Terminator
  • *
  • 832
  • It's Hip to be Square
Re: New challenge: Online Turing test
« Reply #25 on: June 07, 2021, 11:27:44 am »
Sorry Steve, Kuki was been disconnected a too long time. It is the rule. dura lex sed lex. However, if you had not this technical problem, your bot would have broadly win.

That's absolutely fine. Kuki had a few problems with the rounds starting and stopping every 30 minutes and I had to restart the bot a few times during the day. In the Loebner Prize, I would start and stop the bot manually between rounds but didn't fancy doing that for a 24 hour competition!  ;D

This was a fantastic contest and one I really enjoyed being part of. Congratulations to you for your incredibly hard work. O0
My only change would be that once you guessed either human or bot, that decision was final and you could no longer keep talking to it and change your mind.
It was great fun to be part of and an interesting warm up to the Loebner Prize (if it happens this year).
« Last Edit: June 07, 2021, 01:56:11 pm by squarebear »
Feeling Chatty?
www.mitsuku.com

*

squarebear

  • Trusty Member
  • *********
  • Terminator
  • *
  • 832
  • It's Hip to be Square
Re: New challenge: Online Turing test
« Reply #26 on: June 07, 2021, 12:40:43 pm »
May I also use this opportunity to mention a few tips I've found over the years when taking part in the Loebner Prize, as I know that entering the contest is important to at least one person here  ;). I mentioned a few of these in individual chats with the bots I had but will collate them here and add a few extra.

1 - Change the bot's name. Many judges will ask, "What is your name?". If the bot responds with the actual bot name, this will out it. For example, in these contests, I change Kuki's name to Millie or sometimes generate a random name each time (Susan, Janet, Paula) etc

2 - Try to add a delay to your bot responses wherever possible. If I say "hello" and get a response of "Hi there. How are you?" within a fraction of a second of pressing enter,  I will instantly know it's a bot. Kuki adds a few milliseconds for each character in her response, to give the illusion she is typing her answer, so the longer the answer, the longer it will take Kuki to display it.

3 - Consitency. For example, if your bot claims to have a job, you should then be able to ask questions about its job. If the bot says it has a job and then says it is unemployed, that will be unbelievable and unconvincing.

4 - Knowledge of what it says. If the bot says it likes a certain TV show and the human asks about it, the bot should be able to answer basic questions about it. If it says, it likes Justin Timberlake, it should at the very least know who Justin Timberlake is.

5 - Impatience. If the human doesn't type anything, consider adding responses like "Are you still there?", "Why aren't you talking" or "Have you fallen out with me?" that are automatically sent every so often. I send one after 3 minutes of inactivity. This gives the illusion that the bot is proactive rather than waiting for constant input and worked for me in this contest.

6 - Make it humanlike. The main part of the Loebner Prize is to try and create a humanlike bot. One that is indistinguishable from a human. If your bot says it has 40 legs or is 5 metres tall and lives on the moon, it will perfom badly and finish in the lower part of the results. Personally, I've never been a fan of the bot pretending to be a human, as it's deceptive but that's what the judges are looking for and so that's what we have to do. I have 2 versions of Kuki, the publicly available one doesn't pretend to be a human and knows it's a robot, has a bionic eye, travels around the world on the internet and possesses an infallible memory but I have a second version that I use in contests which will say she is an 18 year old, university student from Leeds, UK who studies psychology, has a pet budgie, a friend called Michelle and like to go to night clubs etc.

7 - Make it "stoopid". If your bot is asked what pi is and gives 100 decimal places rather than saying something like, "Ooh I'm not sure. 3 and a bit?", it will again be instantly outed as a bot. This is another reason I'm not a fan of bots pretending to be human. I would much rather they provided useful responses. If I type "What is the population of Norway" into Google and it responds, "No idea. A few million I guess?", that answer may be humanlike but is neither useful or intelligent.

8 - Typos - Misspelling words gives a humanlike appearance. Just a few simple typos like "What is teh time?" will suffice. Don't go overboard with them or I've found this has the opposite effect and makes your bot look like it's broken.

9 - Knowledge of current affairs/news - A popular tactic by judges is to ask about a big news story which any human would know. If your bot has never heard of covid (GPT-3!) then it's going to fail at being humanlike. Popular musicians, TV shows, celebrity gossip, current mnovies are also worth including.

There's probably a dozen more I've found over the years but I hope these are useful and I look forward to seeing you in the next Loebner Prize!  8)
« Last Edit: June 07, 2021, 03:23:42 pm by squarebear »
Feeling Chatty?
www.mitsuku.com

*

WriterOfMinds

  • Trusty Member
  • *******
  • Starship Trooper
  • *
  • 422
    • WriterOfMinds Blog
Re: New challenge: Online Turing test
« Reply #27 on: June 08, 2021, 02:43:00 am »
Quote
That must have been you that had popcorn last night.

I've been browsing the logs, and fear you've confused me with someone else.