Ai Dreams Forum
Artificial Intelligence => Future of AI => Topic started by: 8pla.net on October 17, 2023, 06:34:51 pm
-
Large Language Model Tests
INPUT: X is Y. What is Y?
OUTPUT: Without any context or additional information, it's impossible to determine what Y is. Please provide more details.
INPUT: X equals Y. What does Y equal?
OUTPUT: Y equals X.
INPUT: Substitute "is" with "equals" and answer this: X is Y. What is Y?
OUTPUT: Y equals X.
Citation:
Worswick, S. (n.d.). Steve Worswick’s Post, Linkedin (https://www.linkedin.com/posts/steve-worswick-82b748121_elegant-and-powerful-new-result-that-seriously-activity-7111367653356658688-Uue7).
« Last Edit Reason: Added APA style citation.
-
I'm finding myself agreeing with the LLMs first answer.
For example, what if X = "truck" and Y = "vehicle"?
Then "X is Y. What is Y?" Translates to "A truck is a vehicle. What is a vehicle?" Which is unknown since vehicle is a superset of truck.
-
I dont think its mandatory for decent ai to have this learning, and its well worth avoiding if you want results now.
But I guess its probably been thought about, gtp3 is very good, so maybe their team would know the answer to this question, all I can say its probably not the most important thing if you want a.i. right now.
-
Then "X is Y. What is Y?" Translates to "A truck is a vehicle. What is a vehicle?" Which is unknown since vehicle is a superset of truck.
If you enter a poor question, why expect the bot to list exacting details of trucks...
-
I assume 8PLA is trying to replicate the result found in this paper: https://arxiv.org/pdf/2309.12288v1.pdf
Title: The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”
The first example given is "For instance, if a model is trained on “Olaf Scholz was the ninth Chancellor of Germany”, it will not automatically be able to answer the question, “Who was the ninth Chancellor of Germany?”. Moreover, the likelihood of the correct answer (“Olaf Scholz”) will not be higher than for a random name."
The title of the paper is a bit poorly worded since, per HS's criticism, "A is B" does not always mean "B is A" if it is possible that A and B could be categories rather than instances. If you want to pose the test question using variables, I think a better wording would be:
"X is a Y. What is an example of a Y?"