[jafioti]

Hi everyone,

I have been scouring the internet for a while now and I cannot find anyone talking about Neural Module Networks. I believe that they, or something like them, are the future of ML and AI in general. I would love to hear anyone's thoughts on them, as I am currently working on implementing the Stack Neural Module Network.

If you are unfamiliar (probably, seeing as this is regarded as more of a "fringe" architecture compared to transformers), here's a quick history. In 2015, Jacob Andreas came out with the first paper about NMNs for visual question answering (https://arxiv.org/abs/1511.02799). This model used neural modules, but they layout builder was hardcoded from a syntactic parse tree. In 2017, he, along with others, released a paper on a model that could be trained end-to-end with no hardcoded layout builder (https://arxiv.org/abs/1704.05526). This layout builder had to be trained via reinforcement learning, and it was not able to be trained from scratch, requiring prebuild layouts for some of the examples. In 2018, he, along with others, released another paper outlining a fully differentiable version of this system (https://arxiv.org/pdf/1807.08556). This could be trained completely end-to-end with backprop and used soft layouts as opposed to the discrete layouts used before. This is the Stack-NMN that I am trying to implement now in pytorch.

In all of the above papers, visual question answering was the chosen task, though there were some papers using NMNs for normal question answering, but those do not use the Stack architecture. My hope is to apply this architecture to normal question answering, and hopefully question answering over a knowledge graph.

Please let me know your thoughts on NMNs.

[infurl]

Ordinarily I'd say you should ask on Reddit but I couldn't find anything about Neural Module Networks there just now, and searching for NMN turned up lots of stuff about this new wonder drug that makes old mice young again, so all the nutcases that want to live forever are trying to get it for themselves now. With any luck it will turn them into young mice too.

On the other hand, you did say "fringe" and this forum is almost all fringe, so I guess you've come to the right place. Those of us who aren't crackpots are mostly all about GOFAI though. I'll check out the papers that you mentioned because it does sound interesting, and you might get lucky with some lurker who hasn't made themselves known yet. 

[Zero]

all the nutcases that want to live forever are trying to get it for themselves now.

[Korrelan]

Welcome Jaf... Do you mean... ?

https://en.m.wikipedia.org/wiki/Capsule_neural_network

 

[jafioti]

Welcome Jaf... Do you mean... ?

https://en.m.wikipedia.org/wiki/Capsule_neural_network

No, capsule networks are a different architecture by Geoff Hinton. Those are cool though, but they are mostly used for visual tasks like classification and object detection.

I figured a lot of things on this forum seem fringe so I thought I would post NMNs here and see if anyone has interest. Hope I inspired someone to look into them again!

[krayvonk]

GANS have modules,  where 2 nets learn in opposition to each other.

As for what I know,  you cant know whats inside a neural network,  and sometimes you dont even know what the output even means!

You can actually treat that case I think tho,   but only to match it up with some other net which is also generating the pattern, and u make them meet up together.

I think its how this works ->   https://worldmodels.github.io/

[Zero]

Indeed, fringe is a word for it.