Will LLMs ever learn what is ... is?

  • 4 Replies
  • 18021 Views
*

8pla.net

  • Trusty Member
  • ***********
  • Eve
  • *
  • 1302
  • TV News. Pub. UAL (PhD). Robitron Mod. LPC Judge.
    • 8pla.net
Will LLMs ever learn what is ... is?
« on: October 17, 2023, 06:34:51 pm »
Large Language Model Tests




INPUT:  X is Y. What is Y?
OUTPUT: Without any context or additional information, it's impossible to determine what Y is. Please provide more details.


INPUT:  X equals Y. What does Y equal?
OUTPUT: Y equals X.


INPUT:  Substitute "is" with "equals" and answer this: X is Y. What is Y?
OUTPUT: Y equals X.

Citation:
Worswick, S. (n.d.). Steve Worswick’s Post, Linkedin.

« Last Edit Reason:  Added APA style citation.
« Last Edit: October 17, 2023, 06:59:04 pm by 8pla.net »
My Very Enormous Monster Just Stopped Using Nine

*

HS

  • Trusty Member
  • **********
  • Millennium Man
  • *
  • 1175
Re: Will LLMs ever learn what is ... is?
« Reply #1 on: October 18, 2023, 05:36:22 am »
I'm finding myself agreeing with the LLMs first answer.

For example, what if X = "truck" and Y = "vehicle"?

Then "X is Y. What is Y?" Translates to "A truck is a vehicle. What is a vehicle?" Which is unknown since vehicle is a superset of truck.

*

MagnusWootton

  • Replicant
  • ********
  • 634
Re: Will LLMs ever learn what is ... is?
« Reply #2 on: October 18, 2023, 06:27:40 am »
I dont think its mandatory for decent ai to have this learning,  and its well worth avoiding if you want results now.

But I guess its probably been thought about,  gtp3 is very good, so maybe their team would know the answer to this question,  all I can say its probably not the most important thing if you want a.i. right now.

*

MikeB

  • Autobot
  • ******
  • 219
Re: Will LLMs ever learn what is ... is?
« Reply #3 on: October 18, 2023, 07:43:08 am »
Then "X is Y. What is Y?" Translates to "A truck is a vehicle. What is a vehicle?" Which is unknown since vehicle is a superset of truck.

If you enter a poor question, why expect the bot to list exacting details of trucks...

*

WriterOfMinds

  • Trusty Member
  • ********
  • Replicant
  • *
  • 606
    • WriterOfMinds Blog
Re: Will LLMs ever learn what is ... is?
« Reply #4 on: October 18, 2023, 05:56:21 pm »
I assume 8PLA is trying to replicate the result found in this paper: https://arxiv.org/pdf/2309.12288v1.pdf

Title: The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”

The first example given is "For instance, if a model is trained on “Olaf Scholz was the ninth Chancellor of Germany”, it will not automatically be able to answer the question, “Who was the ninth Chancellor of Germany?”. Moreover, the likelihood of the correct answer (“Olaf Scholz”) will not be higher than for a random name."

The title of the paper is a bit poorly worded since, per HS's criticism, "A is B" does not always mean "B is A" if it is possible that A and B could be categories rather than instances. If you want to pose the test question using variables, I think a better wording would be:

"X is a Y. What is an example of a Y?"

 


OpenAI Speech-to-Speech Reasoning Demo
by ivan.moony (AI News )
March 28, 2024, 01:31:53 pm
Say good-bye to GPUs...
by MikeB (AI News )
March 23, 2024, 09:23:52 am
Google Bard report
by ivan.moony (AI News )
February 14, 2024, 04:42:23 pm
Elon Musk's xAI Grok Chatbot
by MikeB (AI News )
December 11, 2023, 06:26:33 am
Nvidia Hype
by 8pla.net (AI News )
December 06, 2023, 10:04:52 pm
How will the OpenAI CEO being Fired affect ChatGPT?
by 8pla.net (AI News )
December 06, 2023, 09:54:25 pm
Independent AI sovereignties
by WriterOfMinds (AI News )
November 08, 2023, 04:51:21 am
LLaMA2 Meta's chatbot released
by 8pla.net (AI News )
October 18, 2023, 11:41:21 pm

Users Online

270 Guests, 0 Users

Most Online Today: 363. Most Online Ever: 2369 (November 21, 2020, 04:08:13 pm)

Articles