[LOCKSUIT]

But more importantly, does your program prune away uncommon features like "then were" in "they ate and then were all happy"?

[spydaz]

the numbers refer to the node level(Depth) ... if i expand those nodes you will see the "STOPCHAR" this is for the end of sentence marker.... therefore from the root node to the "StopChar" node is a complete sentence. or ( "SUFFIX" all other prefixes are contain within its own suffix.....)

[spydaz]

But more importantly, does your program prune away uncommon features like "then were" in "they ate and then were all happy"?

If a node STopChar is at level 1/2 then obviously its just a term which has been mention in some string...
But if a node has 2/3 node then it must contain some structural data... if a branch has many sub branches then that branches data all must be structurally related... And so forth
as with the cat node containing all the different actions the cat took ... and dog the dog took... the single nodes had no sub-nodes so they must be some subject which is being referenced by an object (note the single item deteminer(the) Contains dog and Cat nodes) (noun phrase detection)

[LOCKSUIT]

Can it connect the top node "the cat was" to one far below by a link...? Like this:

Else it can't re-use them to make 1 bigger thing

[spydaz]

But more importantly, does your program prune away uncommon features like "then were" in "they ate and then were all happy"?

If a node STopChar is at level 1/2 then obviously its just a term which has been mention in some string...
But if a node has 2/3 node then it must contain some structural data... if a branch has many sub branches then that branches data all must be structurally related... And so forth
as with the cat node containing all the different actions the cat took ... and dog the dog took... the single nodes had no sub-nodes so they must be some subject which is being referenced by an object (note the single item deteminer(the) Contains dog and Cat nodes) (noun phrase detection)

All actions under the ALL determiner would group under all... this is why its useful for "Merging sentences" so now a paragraph of text can have defined structure and meaningful ngrams not brute force ngram creation... remember
we are building techniques which AI can utilise for unsupervised learning. (without training) ..... But also maybe can correct itself (clean bad data) (Success / Failure (reinforcement learning) from self built historical data....     <<<<<<<<<<< See the intelligent algorithm.. collects(unsupervised) / (success / failures (probabilistic / Statistical methods)/ Bayesian methods ( cleans) / refines collection / (learns) <<<<<Self supervised learning? Is that algorithm aware maybe of its own errors and failures and attempts to correct itself thereby learning and improving itself based on experience collected and anyalized (thinking)....Deciding .... Hmm very intelligent for a simple Process? maybe?

[LOCKSUIT]

Yeah I learnt that yesterday, that nodes with deeper structure means it isn't stuff like "were then".....it can't combine it to make "were then is like"! but wait what if it makes "were then all happy"? Oh oh, then it thinks the lower smaller snippet "then were" is a good thing!...

I mean if it was "were then the gold" in "they ate but were then the gold winner and they didn't want to be" then it'd link together 2 bad uncommon pieces forever...and can it go past 2/3 nodes and be a nuisance? "were then the gold winner and"....yes?....

[spydaz]

Can it connect the top node "the cat was" to one far below by a link...? Like this:

Else it can't re-use them to make 1 bigger thing

now when designing an Algorithm you have to set your RULE for the algorithm ;... when presented by your case data it is tested for MATCHES based on your STRATEGY then a selected action can be taken (old time game strategy)

The Inference engine

if that what your rules define as a "Legal" move then yes you can .... Construct a New branch by adding the Sub node to the other sub node at the desired point nodes can always be copied or moved.... if that what your rules denote then you can write a function to perform your action

[LOCKSUIT]

oh just a fix on my pic of your pic:

[LOCKSUIT]

But does *your* program combine 2 distant snippets like the red line I drew shows??

[spydaz]

But does *your* program combine 2 distant snippets like the red line I drew shows??

All seeing all doing remember!

Don't forget to read the posts.... diagrams cant help you .... right now ....

[LOCKSUIT]

So what do you do about if your program reads:

"We were then all tired and sat in the old lobby at the old school and then were all too cold to move."

and saves

"lobby at the old school and"

like this in your format (see image):

[LOCKSUIT]

(see above)

in that image, it does have more than 2/3 nodes, a structure, yet, it is a bad take from a sentence, it'll never be re-used, and yes it got re-used by the word parts that followed but, that's all it'll ever get! It'll ass tons of clutter and false real world knowledge....slow searching...etc!! For a AGI..

[LOCKSUIT]

oh and, is your tree saving them like on the left or the right?

I'd say the one on the left is really bad but hmm wait, thinking...if it scans many sentences and learns structure like that, um, it may learn "lobby at the", "at the old", "the old school" in its lifetime, and, and, no this is horrible clutter...

[LOCKSUIT]

in that image, it does have more than 2/3 nodes, a structure, yet, it is a bad take from a sentence, it'll never be re-used, and yes it got re-used by the word parts that followed but, that's all it'll ever get! It'll ass tons of clutter and false real world knowledge....slow searching...etc!! For a AGI..

Also, That's also bad for re-use, again, it'll never be reusing those features...

- because it's doing the left side example, not the one on the right i show.

[spydaz]

So what do you do about if your program reads:

"We were then all tired and sat in the old lobby at the old school and then were all too cold to move."

and saves

"lobby at the old school and"

like this in your format (see image):

There are a few valid shapes in there but the sentence is an irregular sentence and would need pre-processing such as changing the "WE" to a valid noun or object/subject as well as the and being replaced by full stops... a i say "bad data in bad data out" there is no real data in that sentence only snippets... perhaps pre-processing would give better results....even this irregular sentence has "missing data" so its contents wold not be pre-processed without the related sentences from the paragraph.... this is why a lexer / tokenizer / rule-set is required/// to root out the bad data and block data collection... intelligence.... smart code!