Ai Dreams Forum
Member's Experiments & Projects => AI Programming => Topic started by: Ben.F.Rayfield on March 02, 2015, 01:37:08 am
-
Immutable Sparse Wave Trees (WaveTree)
Realtime bigdata tool for bit strings up to 2^63 based on AVL forest
https://sourceforge.net/projects/wavetree version 0.2.0 is a 84 kB jar file containing its own source code.
Opensource GNU LGPL 2+
Realtime bigdata tool at the bit level based on immutable AVL forest which can be run in memory or, in future versions, as a merkle forest like a blockchain. Main object is a sparse bit string (Bits) that efficiently scales up to 2^63 bits normally compressed as forest has duplicated substrings. Bits objects support reading bit, byte, short, int, or long (Java primitives) at any bit index in 64 bit range. Example: instead of building a class to hold a header and then data, represent all of that as Bits, subranges of them, and ints for sizes of its parts. Expansion ability for other kinds of compression, since Bits is a Java interface. Main functions on bits are substring, concat, number of 0 or 1 bits, and number of bits (size). All those operations can be done millions of times per second regardless of size because the AVL forest reuses existing branches recursively. Theres a scalar (originally for copy/pasting subranges of sounds) and a bit Java package. Sparse n dimensional matrix.
AVL tree balancing avoids deep and slow forest
Bits substring, concat, and count 1 bits in any subrange or combination costs only log time and memory (millions of times per second on average computer)
Versioning on N dimensional matrix object (Multidim) since its only a view of Bits object. I've tested this on 10000 images from MNIST OCR data.
Scalar and Bit versions - Originally was scalar for copy/paste subranges of sound. Same operations work for bit strings
Can store sounds that are years long since its sparse. Same works for bit strings up to 2^63.
-----
I too often get lost in excessive abstraction, but I'm getting back to keeping it real. This is 1 of my tools that I only came to understand the need for after years of research. It will be at the core of my game, AI, and science network, along with my new kind of mindmap and statistical tools like boltzmann machines and bayesian networks. All those things will be represented using this foundation of bits.
-
Nice to see you back after your last visit. Hope you've been well and weathering our wonder weather.
I'm not far from NG near Springfield....
Any sample images of your forest?
-
You can see wavetree used in http://sf.net/projects/physicsmata (http://sf.net/projects/physicsmata) version 2.0.0 where its used for a 3d array of bits for some of the mnist ocr dataset.
-
63 bits is not very much. my video tracker uses 1024 bits already, and sift uses 1024 BYTES!!!
-
You're talking about an encryption system, like Netflix uses to prevent you from directly copying the video bits. No video files exist more than 2^63 bits or even close to it. You could fit a whole Human brain in 2^63 bits. If it was to hold the whole internet I'd go with 2^128 just to have lots of space left over for expansion over many years, but 1024 is way overkill, and you'll pay for that in hardware only being optimized for 64 bits at a time.
-
https://www.youtube.com/watch?v=uNS1QvDzCVw (https://www.youtube.com/watch?v=uNS1QvDzCVw)