Possibly a breakthrough programming language?

  • 96 Replies
  • 3434 Views
*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • ******************
  • Hal 4000
  • *
  • 4097
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Possibly a breakthrough programming language?
« Reply #90 on: April 01, 2020, 12:57:41 AM »
New: Same 10,000,000 bytes losslessly compressed to 2,305,386 bytes. Same code.
Emergent

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • ******************
  • Hal 4000
  • *
  • 4097
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Possibly a breakthrough programming language?
« Reply #91 on: April 04, 2020, 04:50:29 PM »
I have a new result for my text compressor (well, predictor).

1,000,000
253,800 - Green
249,682 - Mine

10,000,000
2,331,508 - Green
2,303,936 - Mine
Emergent

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • ******************
  • Hal 4000
  • *
  • 4097
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Possibly a breakthrough programming language?
« Reply #92 on: April 06, 2020, 03:15:17 AM »
How my code works:

My algorithm has a 17 letter long window step along the input file 1 letter at a time, updating a tree as it sees new data. The tree's branches are 17 nodes long, and updates node counts if passes any node. For each step the window takes, the algorithm searches the tree for 17 different searches each a letter longer. The children leafs (the final letter of a searched branch) are the predictions with counts seen so far in the file. Layer 1 nodes are children too and need no match. The tree is storing the frequency of all 1/2/3.../17 letters seen so far. The children are what allows you to predict/compress the next letter accurately. These 17 sets of predictions must be mixed because while the longest set is more accurate - we have less statistics, sometimes only 2 counts. We start with the longest found. Ex. 14 letter match in the tree. The 14th set of predictions may say it seen come next a=44, b=33, f=25, w=7. I sum a set's counts up to get a total of (in this case) 109, then I divide each count by the total to get %s that all add up to 1% ex. 0.404% 0.35%.... Now for all these predicted %s, we still have 13 sets to mix and must remove some % from them each. So what I do is I check the total counts of the set against a Wanted Roof ex. 109<>300 (maybe we don't even need to mix lower sets if we got enough stats), and so I cut each % of each prediction by about 1/3rd then in this case. And in this case we still desire 66% more stats. For the next set, if say we have 200<>300, I take away 2/3rds from the 66% - meaning we still desire 22%, not 66% - 2/3rds = 0%! I take away the % got OF the % still desired. A little bit of lower sets always leak in therefore, which is better because we can never be sure even if surpass Roof by lots. Besides, it gave better results. But Roof is decided by how many predicted symbols are in the set (total unique symbols being predicted), so if i have 2 then Roof may be 8 counts wanted. Also, while the Roof is based on how many different symbols are seen in the set, we get a slightly different Roof if we are on the ex. 5th set, i.e. if we have 4 letters in the set #14 then Roof is ex. 33, but if it is set #5 then Roof is ex. 26. Also, based on the Roof's size, a curve's bend is modified. This curve gives small/large total counts in a set an even smaller/larger total (but it isn't used in the Arithmetic Coding, it's only used for deciding how much % this set gets in our mixer). This is meant to be a exponential activation. Finally a global weight is given to each set ex. the 14th set is always given 0.7% of the weight it was going to get lol. I hardcoded the numbers for now but the code isn't grossly large of course. If they were adaptive and were based on the data then the compression would be even better. I just noticed I do exit the mixing before reach lower sets if the Roof is ever surpassed, I'll have to test if this is useful. The Arithmetic Coder takes the combined sets i.e. the prediction %s are combined a, b, c + a, b, c + a, b, c ..... = a, b, c (and now all the predictions add up to 1% i.e. a, b, c = 1%), and the AC then takes a high and low bound 1-0 and takes the middle between the high and low, and starts misusing each % of the set, until matches the final letter in the window (same process whether compress or decompress). So say we stop once reach b in our set ex. a, *b*, c, we are in the float precision now of ex. 0.45-0.22. WE take middle again (0.23) and start misusing (once the window on the file takes another step. The encoding decimal keeps getting more precise, storing the whole file. To work in 16 byte float we need to carry away locked digits, meaning if the high and low are both now 0.457594-0.458988, we store '45' and get now 0.7594-0.8988, and we are going to be taking the middle of these 2 to make the decimal more precise then. This long decimal is then stored as a binary bin number ex. 6456453634636=10100011100111010011. I didn't implement the removing same counts from lower sets that are just from the higher set, because it hurt compression, i.e. if there is 9 counts total in set 3 and 99 total in set 2, 9 of the counts in set 2 are the same observations and 'should' not help us reach Roof. I'll look into it more. Lastly, escape letters, my first set we mix is a dummy set that has super small weight and has every possible letter, in case we need to encode/decode one and hasn't yet seen it in the file, hence requires a small room in the AC high low bounds. I also hardcoded each probability in this dummy set, common letters get more weight. Compression/decompression takes 2 hours and 16 minutes for 10MB, but Python is slower. Ram is fairly big because I didn't implement the pruning.
Emergent

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • ******************
  • Hal 4000
  • *
  • 4097
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Possibly a breakthrough programming language?
« Reply #93 on: April 06, 2020, 03:31:03 AM »
The algorithm searches before saves. And it only saves 1 line (17 letters long), not 17 saves.
Emergent

*

infurl

  • Trusty Member
  • *********
  • Terminator
  • *
  • 835
  • Humans will disappoint you.
    • Home Page
Re: Possibly a breakthrough programming language?
« Reply #94 on: April 06, 2020, 03:33:08 AM »
Have you considered using multiple passes on the data so your algorithm doesn't have to guess which sequences to compress first?

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • ******************
  • Hal 4000
  • *
  • 4097
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Possibly a breakthrough programming language?
« Reply #95 on: April 06, 2020, 04:12:59 AM »
Usually it's not easy, the compressed data has much more noise (less patterns).
Emergent

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • ******************
  • Hal 4000
  • *
  • 4097
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Possibly a breakthrough programming language?
« Reply #96 on: April 20, 2020, 10:49:58 PM »
Since I wrote it up in full and clear and u may need it, here is actually how to make big money and hire AI/art/etc professionals from home. All AI domains are on there, it is pro-AI like heaven.
---------------------------------------------------------------------------------------------
Use Blockly to program in Python overnight, you don't need to learn Python. And Blockly has a language setting for blocks. On Upwork you can get easy 500USD for a single programming job.
- - - - -
Yes programming is really easy and I can do it now, and there is manyyyyy jobs on Upwork and all you need to do is make a good profile up cus like 1,000 apply to Upwork each day and you will be asked questions about Python in a one time interview. Find the easier but larger project that the client can't even understand (hence the large payment is common gesture) and it is just easy. Once you get the first job or two done, your profile will get boosted and more invites will be sent to you. You can invite too. Stay online as much as can and the clients will see you Online and choose you. I do that :) lol. I used to be a client but now I can code.

Upwork has all forms of AI domains, it's so funny lol. And, music, video design, blog posting, etc, any online job. So it's highly useful for making cash and hiring professional artists.

Just don't go off the Upwork cus they don't like that :). And you get a good review if nice and cooperative and on time if they need that, so make sure they give u good reviews! There is a way to legally leave Upwork with clients now btw!

Emergent

 


Users Online

16 Guests, 1 User
Users active in past 15 minutes:
ivan.moony
[Trusty Member]

Most Online Today: 27. Most Online Ever: 340 (March 26, 2019, 09:47:57 PM)

Articles