[LOCKSUIT]

got a hash table working! I coded it myself and uses someone's simple generator:

input = 'walking down the street and saw some birds flying north'
HT = [None] * 50
for n in range(len(input) - 2):
  key = input[n:n+2]
  hashsum = 0
  for idx, c in enumerate(key):
      hashsum += (idx + len(key)) ** ord(c)
      hashsum = hashsum % 50
  if HT[hashsum] is None:
    HT[hashsum] = [input[n:n+3]]
  else:
    HT[hashsum].append(input[n:n+3])
print(HT)

[LOCKSUIT]

I had got holed matches working some days ago and might, along with time delayed matches, get my compression score down from 20.3MB to anywhere somewhere around 19.2 to 17MB. It isn't that hard for me to implement this but first I have to make my datastructure much faster/ memory efficient to handle the extra searches. I'm thinking I might use only for 1-3 context lengths the hash table, as tree is pretty fast for longer branches as there is no branch splits while stores the contexts as one in a branch unlike a hash table. But I'm talking to someone on this matter.

[MikeB]

Hi Locksuit, I'm still working on a C++ conversion. It's 99% done. I also have a list of notes for your python version for possible speed improvements (centered around string checking removal). Do you have an email..?

The best overall text compressors seem to be in the range... (WinRAR 3.60b3)
30-60 seconds for 100MB (22MB. <256MB memory)
5-10 minutes for 1GB (220MB. <512MB memory)

If it could reach that you could claim best overall imo... all the other compressors take too long, or use too much resources, or just straight out use a GPU to process it

I think the speed will be within that for the C++ version.

[LOCKSUIT]

That is amazing, wow, thank you for doing that. I will private message you my email.

[infurl]

It would be safer to use private messages to exchange email addresses. Posting email addresses publicly is ill-advised. MikeB please confirm when you have read this and I'll obfuscate the email address.

[LOCKSUIT]

big code update > https://encode.su/threads/3595-Star-Engine-AI-data-compressor?p=69902#post69902

[LOCKSUIT]

I tried cmix's pre-processor with my program - like how other top programs use pre-processors. Decompression matched. I got 19,477,251 bytes for the enwik8 file.

[LOCKSUIT]

how to think and learn, good read if have time to read it:
https://louis030195.medium.com/what-you-do-not-learn-at-school-how-to-learn-d6809922cac

[LOCKSUIT]

These dogs are so intelligent, you can just see it in this video below, they trained it:

https://www.youtube.com/watch?v=RJ9g4kmaXxg

https://www.youtube.com/watch?v=sq8yBfl5Fek

[infurl]

Dogs are smart, no question. I remember once telling a colleague how my dog would pick up an old shoe and use it to knock on the door of the house when he wanted someone to come out and play with him. My colleague said that didn't prove the dog was smart because he probably saw someone knocking on the door. He shut up when I pointed out that being able to learn such a skill by watching someone else do it was pretty smart too. 

[LOCKSUIT]

My lab would almost every time he finished using the outside bathroom pen would go up the steps and scratch the door, then again in ~20 seconds if no answer. If it failed ~2-3 times, he'd bark every maybe 5 or 10 seconds. No one taught him this or bothered such thought on this either, ever. On entering he'd clean his feet by licking each top of paw lol, mom would clean his hands so maybe this was from that. In the winter he'd kick back his back legs in the snow before going up the steps lol! He'd chase a laser beam around the kitchen, and learnt after a dozen times that saying laser beam meant soon would be seeing a red dot, so he'd look for it suddenly on his legs as soon as I said it. And "who wants to go for a walk?" would make him very excited each time, hours panting and unsettled waiting until got said walk, he only ignored it if finished walk and neuron has been fired so to speak. The only thing he didn't eat was olives and salary, he'd eat Kleenex. We caught him a few times laying on his back sleeping.

[MagnusWootton]

Thats great.   Your dog is lucky that he has you guys there to appreciate him,  not all dogs are that lucky...

[Korrelan]

Doggo's are just cool... 

[LOCKSUIT]

So my adventure on me versus openAI.com continues:

So far my code I made from scratch (except the little pre-processor, though I know how it works mostly) got a score of 19,033,243 bytes losslessly compressed for 100,000,000 bytes fed in (enwik8.txt). So I'm 4MB away now from where I "should" be. Still more to come. This is only the beginning of everything.

The text completion results yous seen I had improved as you can see in post #221 below in the link below, but these are from my 20MB score, because the code was older and wasn't running the pre-processor during completion of text, so that's a big difference.

https://encode.su/threads/3594-CM-design-discussion/page6

(project page)
https://encode.su/threads/3595-Star-Engine-AI-data-compressor

Not listed in my how it works on that link is that I am half set up the usage of hole, and delay matching of context, and delayed prediction (ahead of time predicts it). And it works on word level due to the preprocessor, so it can recognize "walked very fast to the" as "walked fast in the >>> new store they made" and predict ahead of time "new [store]" so we get "walked very fast to the store". Many matches cultivate their predictions to get a new set of predictions in form of probabilities then sent to a evaluation function.

What's left to do after this is translation for recency boosting and matching, and mirror ghosting (seeing a partial match and seeing the 2 items not matched are similar so so should my items even though different topic words), and pattern of delay and hole errors, and weighting tricks to tie it all together to get clearer predictions, and a slew of other little thingies.

Next time someone says they don't know what GPT is learning, or how the code works, tell them to look at the code harder and look at the dataset harder, there is just several common patterns in it you can find and allows predictions, it all starts with exact matches too.

[MagnusWootton]

Its good you sound really dedicated to it,  still exciting for ya?

Ive got an optimization for SIMILARITY matching,  but I havent put it together yet to proove it.
I can use to optimize this whole screen multi tracker,    I can do lots of funny things with it I bet, but all I do is use to convert 2d to 3d.
But the similarity matching is also really good for robust classification.