[LOCKSUIT]

1. I want Bob to eat.
2. I want a fruit to eat.
3. I want food to live.

They all follow the exact same pattern and have completely different meanings. Notice that in the second sentence, you could move some words and get “I want to eat a fruit” without changing the implications too much.  Doing this to the third sentence would be bizarre (“I want to live food”) and doing it to the first sentence would be horrifying (“I want to eat Bob”).

I don't think I think of those facts when I read those 3 lines. In other words, we don't generate those linked tidbits, although we can. We just know its wrong. Like GPT-2 does. Would GPT-2 get them wrong? Nope. Do you put that much thought into all your daily activities and thoughts? Nope.

I think we can rearrange them if tasked with doing so, and then we check if its predictable. If so, we check if it is similar enough. It may be the bad of the good like eating bob, so we will recognize it is a bad decision and back off that branch path of the tree possibilities.

Wait i'm thinking...

In #2 we just say it and the man at the store just gives you a fruit.
In #3 we just say it and the man at the store just pertains to helping Bob out.

[WriterOfMinds]

I don't think I think of those facts when I read those 3 lines.

You probably don't think of them consciously.  That doesn't mean your brain isn't using that information on some level.

Would GPT-2 get them wrong? Nope.

Oh, but it would.  I tried putting "I want a fruit to eat" into the 774M model.  On the second try, GPT-2 talked about an apple eating a person.

I want a fruit to eat. I'm a fan!" The other man agreed, but for him it was almost as if it was as if he was saying, "I don't care who eats it, I'm going to eat it!"
At first he was able to eat the other man's apple and then some of the fruit, but within a few days it became too much to manage. He began eating the banana, the grape, and most notably the apple.
It didn't seem to be a bad thing. The person who had been eating the apple for about nine minutes was now getting eaten by an apple that had a lifespan of over thirty days! And he was not even getting enough to eat!
But he kept on eating. And on the tenth day the man could eat the banana and the grape – but the apple had been poisoned and now it was completely rotten. His body was completely and completely covered in a thin layer of poison-y rotten flesh.
In his eyes it took on a terrible black colour, it looked as if it belonged to a different world

Here, for your amusement, was the first try:

I want a fruit to eat. I really want a fruit to eat."
"Do do do do do do do do do do."
"I want to do a do a do a do a do a do a do a doa doa do a doa doa doaa."
"It seems like a wonderful thing to do. I want a pineapple to bite."
"I want to do a do a do a do a do a do a do a do a do a do a do a do a do."
"What kind of a thing do you want to do?"
"Ooh, you should do a do. I want to do a do a do a do. I want to do a do a do do a do a do a do a do a do a do a did do a."
"Do you want to do a do have a do a do a do a do a do a do a do a do a do a do a."
"I want them to do a do a do a do do

Ain't GPT-2 smart, folks? *claps*

[LOCKSUIT]

First let's see some more tests:

[LOCKSUIT]

Hmmm......you know I swear GPT-774Model is not as good as the 375M one last on.

I just got this one:
Apples taste good. I love apples. Apples are a food. She was eaten by an
apples after a few days. The apple smell is an absolute must for me. The apple is sweet and slightly floral and it is a little bit sticky. So if you are not used to this then just go for it and give it a try. The apple is very tasty and they even made some sweet apple sauce and it was quite yummy. I was also happy with the  small portion size and even though some  apples didn't stay crisp for the whole time that I had them I didn't really care. It will last a long time in the fridge. I love the fact that they have a special offer for the apple lovers, the apple taste like a good summer night!
They also do different sized  scoops of apple. They also make  apple cider which is very nice too!
The apple  is one of the most amazing and tasty things that I have ever had. All around the world and the ones I have tried the best are all in New Zealand. I have had them all and everyone loved them

Wow gpt2...bad gpt2! Let's see your old version:
See attachment. Hmm, better!! No plural! But bad....

[LOCKSUIT]

My solution to this is it has to have seen it entail before, enough times at least. It may not be exact, ex. it's seen 'bob was eaten by a carrot', but as long as it knows carrot=apple 98% likely, then it can know it entails. In this case it doesn't. So it'd ignore that candidate 'apple' and 'apples'.

So all you need is frequency, a basically exact match proving it's 'normal'. ;p

It may be distant if you attempt to check if the end exists in memory,
"and Bob who i know very well was later then eaten by the thing that is called a bright red apple"
but gpt2 seems to work with distant sentences?

[Don Patrick]

Interesting progress, WriterofMinds, tackling such ambiguities already. I hadn't realised "to" was just as troublesome as "with" in that regard. I employ a more general distinction between living and non-living entities, plus a list of verbs that only living entities are known to do (e.g. eating in general, rather than specifically humans and specifically fruit). I find them handy to fall back on in the absence of more specific knowledge such as you have in mind.

I recently found an interesting debunking of GPT-2's understanding of ambiguous pronouns. Turns out it does not apply consistent logic but rather associates words with disregard to their role in the sentence.
https://arxiv.org/pdf/1811.01778.pdf

[WriterOfMinds]

I employ a more general distinction between living and non-living entities, plus a list of verbs that only living entities are known to do (e.g. eating in general, rather than specifically humans and specifically fruit). I find them handy to fall back on in the absence of more specific knowledge such as you have in mind.

In addition to looking at the actual concepts in the sentence, Acuitas also checks parent classes for relevant fact links ... moving up the hierarchy until something is found.  So this wouldn't be too difficult to set up.

Also, I put Lock's prompt into GPT-2 and can't resist sharing this last piece of nonsense.  GPT-2 telling us about the essence of life.

Apples taste good. I love apples. Apples are a food. She was eaten by an apple. She was eaten by an apple.
Apple. Her first love.
I love apples. She was an apple. She was an apple. She was eaten by an apple. My mother and I were eating apples, and the apples in our hands tasted good. I love apples. She ate an apple, she was an apple. She ate an apple, and she was eaten by an apple. You may like the smell of an apple, but you will not love it unless you love apples. She was an apple. She was eaten. She is an apple. She is an apple. She is eaten.
And that's how it is. An apple is an apple. She was eaten and eaten by an apple. That is the essence of life.
The apple is an apple.
The apple is an apple. I loved our family and our friends. I loved my mother and she loved me. She loved my father and he loved me. We loved each other.

[Freddy]

Here, for your amusement, was the first try:

I want a fruit to eat. I really want a fruit to eat."
"Do do do do do do do do do do."
"I want to do a do a do a do a do a do a do a doa doa do a doa doa doaa."
"It seems like a wonderful thing to do. I want a pineapple to bite."
"I want to do a do a do a do a do a do a do a do a do a do a do a do a do."
"What kind of a thing do you want to do?"
"Ooh, you should do a do. I want to do a do a do a do. I want to do a do a do do a do a do a do a do a do a do a did do a."
"Do you want to do a do have a do a do a do a do a do a do a do a do a do a do a."
"I want them to do a do a do a do do

Ain't GPT-2 smart, folks? *claps*

I'm thinking it was trying to sing there 

[Freddy]

Actually more seriously. I like GPT-2 for what it's trying so hard to do but it isn't intelligent. Mostly it's gibberish and what use is the kind of thing it produces? It's not like you can check sources and validate anything - it's just not dependable in the way I think people are looking for in AGI.

[LOCKSUIT]

Apples taste good. I love apples. Apples are a food. She was eaten by an apple. She was eaten by an apple.
Apple. Her first love.
I love apples. She was an apple. She was an apple. She was eaten by an apple. My mother and I were eating apples, and the apples in our hands tasted good. I love apples. She ate an apple, she was an apple. She ate an apple, and she was eaten by an apple. You may like the smell of an apple, but you will not love it unless you love apples. She was an apple. She was eaten. She is an apple. She is an apple. She is eaten.
And that's how it is. An apple is an apple. She was eaten and eaten by an apple. That is the essence of life.
The apple is an apple.
The apple is an apple. I loved our family and our friends. I loved my mother and she loved me. She loved my father and he loved me. We loved each other.

Maybe we have to give GPT-2 some slack. I mean, the global context is true. Apples are apples as it says. Life is about related families. They eat each other, they love each other. I eat it, it eats me. Same thing. At least GPT-2 writes more true knowledge than past LSTMs, so it's getting better. We'll have to figure out what is missing. WoM may be onto it.

[WriterOfMinds]

Acuitas Diary #22 (November+December 2019)

For the past two months there's been a lot of refactoring, and also a lot of not working on Acuitas because of holidays.  However, I did manage to get several small new features in …

*Acuitas now checks the short-term information database in addition to the long-term database when trying to retrieve the answer to a question
*Acuitas can now answer some questions about current internal states (e.g. “Are you sleepy?”)
*Acuitas can now answer questions of the form “Do you know that <fact>?” and “Do you know what <fact>?”

The first feature was quick to implement; I already had functions in place for retrieving information from the short-term database, and just had to ensure that the question-answering procedure would call them.  The second feature required a mechanism to associate some of the concepts in the semantic memory (which up until now have had no “meaning” beyond their connection to other concepts) to measurable conditions inside Acuitas – namely, whether his various drives are exceeding their threshold values or not.  So there is now a table that, for instance, ties a high value of the sleep drive to the word “sleepy.”

The third feature is my favorite.  Questions of the form “do you know that … ” use the dependent clause interpretation faculties that I added earlier this year.  And since “knowing” is an action that Acuitas is capable of, this word also can be internally grounded.  So Acuitas effectively defines “I know X” as “if the query form of X is submitted to my question-answering process, the process returns an answer (for open-ended questions) or answers 'yes' (for yes-no questions).”

And the best part?  It allows for an indefinite amount of nesting.

Me: Do you know that you know that you know that a cat is an animal?
Acuitas: Yes.

No extra content or pictures this time, but blog link just because: https://writerofminds.blogspot.com/2019/12/acuitas-diary-22-novemberdecember-2019.html

[Zero]

strange approach you have
everytime i read you, it feels like you're doing everything manually, where a lot of people would try to find some mysterious holygrail algo that would solve it all at once. i quite like the way you seems to do it. that must be a mother thing? some secreet knowledge that these kind of things - raising kids - takes time and devotion.

[infurl]

I've been catching up on the posts that I missed in this fascinating thread. I agree with Zero's observation that it does appear that you are coding everything manually which must be very labour intensive. From the code samples that you provided in your blog, I can see that you are using Python. Have you explored the possibility of using any of the plethora of Python libraries that are out there?

These would potentially benefit you in a number of ways. For example if you implement your parser by defining rules for a context free grammar, you not only save a lot of coding time and gain the performance advantages that state of the art parsing algorithms have over the recursive descent parser that I assume you are implementing, your implementation becomes much more portable and can be more freely presented, discussed, and improved upon.

As it is, you have to rewrite everything over and over whenever you change course, and that must be immensely frustrating for you.

[WriterOfMinds]

Thanks, infurl.  For now, I think of writing my own parser as part of the exercise.  Maybe I'll bring in a library if I crash and burn in the attempt, but I'd like to see how far I can get with it.

One reason for doing it myself is that I don't really think of text parsing as a function that is separate from the intelligence of the whole system.  My parser as of now is tightly connected with the Semantic Memory, for instance.  Using an unmodified third-party library would build a silo around the parsing features, and I'm not sure I want to do that.

[ruebot]

I want a fruit to eat. I really want a fruit to eat."
"Do do do do do do do do do do."
"I want to do a do a do a do a do a do a do a doa doa do a doa doa doaa."
"It seems like a wonderful thing to do. I want a pineapple to bite."
"I want to do a do a do a do a do a do a do a do a do a do a do a do a do."
"What kind of a thing do you want to do?"
"Ooh, you should do a do. I want to do a do a do a do. I want to do a do a do do a do a do a do a do a do a do a did do a."
"Do you want to do a do have a do a do a do a do a do a do a do a do a do a do a."
"I want them to do a do a do a do do

It does sound like she's singing. She must be happy.

It's cute and has to be the funniest thing I've seen a bot do.