[LOCKSUIT]

If you constrict the data by setting a persona, you get more on topic responses, but it gets repetitive

Sorta like me? Or korrelan?

On the other hand if you loosen the restrictions and give it access to more data, you also increase the amount of garbage and generic responses that the program will drag in.

Sorta like HS? Sorry had to choose something.

Trust me, more data = more accuracy, I've been working with this stuff, I know... You also get more data using the same data if the AI can extract/mine data smarter/better.

[Don Patrick]

Haha . I'm glad you have some self-awareness.

As a rule, I never trust people whose best argument is "trust me" . I'm not 100% familiar with the terminology, but I do believe that both "overfitting" and "underfitting" are common issues with neural networks. It has also been proven by earlier research that the benefit of adding more data to statistical models decreases exponentially, which is why we now need ridiculously large numbers for minor improvements.

[krayvonk]

That overfitting thing is a little unintuitive... It makes more sense more data the merrier,  I remember looking at some graphs and it actually didnt mean it responded worse its just there is a nice way to peg a net off with a little data and it interpolates well,  but after that the more goes in the better results you get, but u just dont get as much out of it.  (Little nets work well for the amount of data was inserted.)

I could be wrong tho,   but makes more sense the more the better, like Lock says, 2 mer.

[LOCKSUIT]

Hmm, you say there's exponentially less improvement the more data you add, and I said the opposite. If we look at just the last letter of our text story to predict the next letter to get probabilities, and add more data to the algorithm, it gets exponentially less improvement, yes. And, looking at just 1 letter context works good, but if you look at the last 2 letters, it's exponentially less useful the father back you look as well.

Now, GPT-2 looks at the last 512 words using better technology, however the same rule holds that exponentially less improvement is made from looking farther back. However, something very far back could tell us the letter a million letters in the future is a q - and you might trust this person. If I'm working on a cure for cancer and have a 50GB text dataset of all my thoughts that is all about cancer, I can look farther back and consider it all because it's all related. You simply access other, but, related memories, when predicting the next letter.

So how far can you look? A lot, all of it that's related, and all data is. How much data should you add? A lot. While adding more samples increases the model distribution accuracy exponentially less, there's a huge amount of features/ structures in life that all relate, and therefore in actuality you get exponentially more relationships captured the more data you have. It first rises as a exponential curve, and then slopes back to flat, it is an S curve. So you do get a lot of free data to mine, and it does eventually halt once you learn the universe.

[LOCKSUIT]

See images below. On left is bytes trained on. Ignore right. My predictor algorithm VS Green, amazingly the charts are nearly the same! It appears the predictor gets betters the more data I feed it, but the amount it improves by is shrinking, albeit exponentially. But like said the curve should increase exponentially the other way instead - if the algorithm utilizes combinatinoal relationships.

wait a minute.......the 3rd image....it is exponentially increasing.... In image 3 I was curious, I first got the same curve as above to make sure my settings were same, then made the last numbers on right side exponentially smaller (compression size) on purpose by hand, and the curve bends down more! Therefore images 1 and 2 are good news...

[LOCKSUIT]

I tested the best compressor on Earth as well (lol), hmm, all 3 predictors get the same plot! Further, it fluctuates, if I plot 320,000 bytes input the red line goes Up! Then 640,000 bytes it goes down again too, like a crooked fishpole all bent in various ways. Overall it's basically a 45 degree angle ya. Overall the double the input size the same compression but just a bit better than last. Is that small slice growing or not? I'll check tomorrow below:

1250, 790
2500, 1563
5000, 3482
10000, 8354
20000, 16242
40000, 30595
80000, 57032
160000, 100927
320000, 206331
640000, 390287

Perhaps the plot is same for all 3 because they do as better with the same amount of data. Which makes 0 sense [if] it has word2vec capability!