[LOCKSUIT]

got a hash table working! I coded it myself and uses someone's simple generator:

input = 'walking down the street and saw some birds flying north'
HT = [None] * 50
for n in range(len(input) - 2):
  key = input[n:n+2]
  hashsum = 0
  for idx, c in enumerate(key):
      hashsum += (idx + len(key)) ** ord(c)
      hashsum = hashsum % 50
  if HT[hashsum] is None:
    HT[hashsum] = [input[n:n+3]]
  else:
    HT[hashsum].append(input[n:n+3])
print(HT)

[LOCKSUIT]

I had got holed matches working some days ago and might, along with time delayed matches, get my compression score down from 20.3MB to anywhere somewhere around 19.2 to 17MB. It isn't that hard for me to implement this but first I have to make my datastructure much faster/ memory efficient to handle the extra searches. I'm thinking I might use only for 1-3 context lengths the hash table, as tree is pretty fast for longer branches as there is no branch splits while stores the contexts as one in a branch unlike a hash table. But I'm talking to someone on this matter.

[MikeB]

Hi Locksuit, I'm still working on a C++ conversion. It's 99% done. I also have a list of notes for your python version for possible speed improvements (centered around string checking removal). Do you have an email..?

The best overall text compressors seem to be in the range... (WinRAR 3.60b3)
30-60 seconds for 100MB (22MB. <256MB memory)
5-10 minutes for 1GB (220MB. <512MB memory)

If it could reach that you could claim best overall imo... all the other compressors take too long, or use too much resources, or just straight out use a GPU to process it

I think the speed will be within that for the C++ version.

[LOCKSUIT]

That is amazing, wow, thank you for doing that. I will private message you my email.

[infurl]

It would be safer to use private messages to exchange email addresses. Posting email addresses publicly is ill-advised. MikeB please confirm when you have read this and I'll obfuscate the email address.

[LOCKSUIT]

big code update > https://encode.su/threads/3595-Star-Engine-AI-data-compressor?p=69902#post69902

[LOCKSUIT]

I tried cmix's pre-processor with my program - like how other top programs use pre-processors. Decompression matched. I got 19,477,251 bytes for the enwik8 file.