[Zero]

Hey guys, what's up?

So, I realized that for a long time I've been looking for a single middleware that would be able to host the many shapes intelligence can take, when in fact there can't be such a thing, unless you're ok to end up doing intractable things. I was wrong, this is not what I need. I need a common blackboard, rather than a common middleware. Like a central database, all in-memory. Except different parts of the program would be interested by different aspects of the content, which calls for a NoSQL-like solution.

NoSQL goes like: you set up views, and whenever you throw data at it, the db updates views to keep em ready for you. Instead of running 1 query over 100K entries (at query-time), you rather check 1 datum against 100 views (at update-time). Ok it's more complicated but you get the point. Very fast.

I first looked at PouchDB's Github's Issue section. Then I decided to roll my own.

I did a first version, all small. Then I made another version, even smaller and a lot more powerful, by extending Javascript's Map base class. It's the very core of what makes NoSQL cool (to me, at least), once you drop all the fancy. Of course I don't have revs, but I don't need it in my use case. Here:

Code

// Sensitive Maps

class SMap extends Map {

    sensors = {};

    sense(id) {

        return this.sensors[id].members;
    }
    
    removeSensor(id) {
    
        delete this.sensors[id];
    }
    
    sensor(id, validator) {
    
        this.sensors[id] = { validator, members: new Set() };
    
        this.forEach((value, key) => {
            if (validator(key, value))
                this.sensors[id].members.add(key);
        });
    }

    set(...args) {

        for (let s in this.sensors)
            if (this.sensors[s].validator(args[0], args[1]))
                this.sensors[s].members.add(args[0]);
            else
                this.sensors[s].members.delete(args[0]);

        return super.set(...args);
    }

    delete(...args) {

        for (let s in this.sensors)
            this.sensors[s].members.delete(args[0]);

        return super.delete(...args);
    }
}

You'd use it like this:

Code: text

let smap = new SMap();

smap.set('i1', 'foo');

smap.sensor('strings', (k,v) => typeof v == 'string');

smap.set('i2', 'bar');
smap.set('i3', 4);

console.log(smap.sense('strings'));
// => Set(2) { 'i1', 'i2' }

The initial need was to assemble 2 of my previous experiments: the consnet, and the more recent belief-based programming language. Now with Smaps, I think I can assemble pretty much anything, it's like a super-tiny glue.
 

The speed doesn't depend on the size of the db, it only depends on the number of changes made to it. The implementation could be faster I guess.

[Zero]

This one should be a bit faster (avoiding a few useless sensing).

Code

// Sensitive Maps

class SMap extends Map {

    sensors = {};
    updated = new Set();
    deleted = new Set();

    constructor(content) {

        super();

        if (content)
            for (let [k, v] of content) this.set(k, v);
    }

    sense(id) {

        if (this.updated.size || this.deleted.size) {
            
            for (let s in this.sensors) {

                for (let key of this.updated)
                    if (this.sensors[s].validator(key, this.get(key)))
                        this.sensors[s].members.add(key);
                    else
                        this.sensors[s].members.delete(key);

                for (let key of this.deleted)
                    this.sensors[s].members.delete(key);
            }
            this.updated.clear();
            this.deleted.clear();
        }
        return this.sensors[id].members;
    }

    removeSensor(id) {

        delete this.sensors[id];
    }

    sensor(id, validator) {

        this.sensors[id] = { validator, members: new Set() };

        this.forEach((value, key) => {
            if (validator(key, value))
                this.sensors[id].members.add(key);
        });
    }

    set(...args) {

        this.updated.add(args[0]);
        this.deleted.delete(args[0]);

        return super.set(...args);
    }

    delete(...args) {

        this.deleted.add(args[0]);
        this.updated.delete(args[0]);

        return super.delete(...args);
    }
}

Edit: faster again.
Edit: added a constructor so we can clone the dbase.

[MagnusWootton]

I'm not the best at symbolic logic,  but there would be accelleration structures you could use to speed up getting the solution to a query.

INPUT -> "do i turn on the tap - given this many parametres"  OUTPUT -> yes/no.

Handling the specific question is the easy model,  but generalizing it to all questions is the unexplored topic I guess.


INPUT -> the situation of the robot,  OUTPUT -> the activity its going to perform.

thats the harder one.

[Zero]

Smap is more low-level than that. I'm using an inverted index technique. The "sensors" are like constant queries that you set at start up. Then, when you change the content of the dbase, the solutions of queries are automatically updated to reflect the change, so when you actually need a solution, it is already computed and ready to use.

To speed it up, the last version will wait for a call to sense() before applying updates, so several changes to the same document don't trigger unnecessary sensors update.

I don't know whether or not my explanations are clearly understandable?

[MagnusWootton]

The "sensors" are like constant queries that you set at start up. Then, when you change the content of the dbase, the solutions of queries are automatically updated to reflect the change, so when you actually need a solution, it is already computed and ready to use.

That sounds like an awesome way to do it!     I understand a little more now.

So I could use a little more help to understand your idea,   what would these queries be and what would the data be in the database?  Whats a simple application of it?

In my system I''m in need of the form of query that would tell a reinforcement learning agent whether its being morally acceptable or not,   I figure a logic symbolic type system is what I require, to send a score or even just a boolean signal for a perceptron/neural network to bind to the spacial environmental report going in.

[Zero]

I see.
I'll make a more general version of it, so it can be useful in your use-case. Hold on.

[Zero]

Here is the new version.

Code

// Sensitive Maps

class SMap extends Map {

    sensors = {};

    constructor(content) {
        super();
        if (content)
            for (let [k, v] of content) this.set(k, v);
    }

    sense(id) {
        return this.sensors[id].solution;
    }

    removeSensor(id) {
        delete this.sensors[id];
    }

    sensor(id, initial, reducer) {
        this.sensors[id] = { reducer, solution: initial };
        this.forEach((value, key) => {
            this.sensors[id].solution =
                this.sensors[id].reducer(
                    this.sensors[id].solution,
                    key,
                    value
                )
        });
    }

    touch(key, previous) {
        for (let s in this.sensors)
            this.sensors[s].solution =
                this.sensors[s].reducer(
                    this.sensors[s].solution,
                    key,
                    this.get(key),
                    previous
                );
    }

    set(...args) {
        let previous = this.get(args[0]);
        let result = super.set(...args);
        this.touch(args[0], previous);
        return result;
    }

    delete(...args) {
        let previous = this.get(args[0]);
        let result = super.delete(...args);
        this.touch(args[0], previous);
        return result;
    }
}

Example usage.

Code: text

let smap = new SMap();

smap.set('i1', "foo");

smap.set('i2', "bar");

smap.sensor("string counter", 0, function(solution, key, newValue, oldValue) {

    if (typeof oldValue != "string" && typeof newValue == "string") ++solution;
    if (typeof oldValue == "string" && typeof newValue != "string") --solution;

    return solution;
})

smap.sensor("string selector", new Set(), function(solution, key, newValue, oldValue) {

    if (typeof newValue == "string")
        solution.add(key);
    else
        solution.delete(key);
    
    return solution;
});

smap.set('i3', "baz");

console.log("counter:", smap.sense("string counter"));

smap.set('i3', 4);

console.log("counter:", smap.sense("string counter"));
console.log("selector:", smap.sense("string selector"));

// counter: 3
// counter: 2
// selector: Set(2) { 'i1', 'i2' }

Edit: corrected a wrong behavior (when overwriting an existing sensor).
Edit: another bug died today.
Edit: added oldValue argument.
Edit: yet another bug.

[Zero]

The general approach is to put all your data in the same SMap. The sensors let you produce custom solutions, depending on your needs. The only limitation, for now, is that the reducer functions of the sensors should not be modified during usage. Same inputs should return the same output.

The queries are reducer functions. They take as input the current solution of the sensor, the key and the new value which have been modified, and must return the (updated) new solution of the sensor.

[MagnusWootton]

Thankyou, but that's a little too general example for me to get the gist of what's going here really. 
But the work looks very good, the constantly active queries seems like a great idea, looks like you've got alot on your mind, I hope u get some concrete success with this architecture, and even if it gets outdated with something else u come up with. (Definitely didn't get it first time me, that's fer sure.) it should teach you a damn shitload about things that's going to help your next endeavor. win win.

[Zero]

In your system, do you (at some point) need to iterate over a big database contained in your agent?