Do any neural network training schemes vary the nonlinear activation function?

  • 2 Replies
  • 641 Views
*

elpidiovaldez5

  • Roomba
  • *
  • 13
I have just read about ‘looks linear’ initialisation https://arxiv.org/pdf/1702.08591.pdf.  This offers the possibility to train deep networks simply by initialising their weights properly (i.e. so that non-linearities cancel out, and the network initially 'looks linear') . It made me wonder if anybody has tried training a deep network with an activation function that starts linearly, and gradually becomes nonlinear, as training proceeds. Without non-linearity a deep network is equivalent to a single linear layer, no matter how many layers it contains. Thus at the outset of training a deep network is essentially 1 layer deep, and very easy to train (gradients do not decay or explode at all).  As training proceeds the non-linearity could be increased gradually. I would like to imagine this as the network effectively, slowly 'expanding' to a greater number of layers.  Of couse the weights would need to be adjusted to track the changing network.  It would be nice if the process could be controlled so that the network expands to the number of layers necessary to solve the problem and then stops.

Has anyone heard of anything like this ?  Opinions ?

*

keghn

  • Trusty Member
  • *********
  • Terminator
  • *
  • 772
A visual proof that neural nets can compute any function:   

http://neuralnetworksanddeeplearning.com/chap4.html



Which Activation Function Should I Use? : 


*

keghn

  • Trusty Member
  • *********
  • Terminator
  • *
  • 772
 When Neural Networks are used as transmission  lines to filters , then yes. Brain does not really use multiplexing like computers.
but they can have many many parallel path turned on at the same time to a filter, which is a little Neural network.
Multiplexing: Doing more with less. // Technology: 

   


 


The last invention.
by LOCKSUIT (General Project Discussion)
February 17, 2018, 11:51:20 pm
Supervised AGI
by keghn (General AI Discussion)
February 16, 2018, 08:24:24 pm
Strange learning curves
by Kaeldric (AI Programming)
February 16, 2018, 06:28:14 pm
XKCD Comic : Unification
by Freddy (XKCD Comic)
February 16, 2018, 05:00:26 pm
I want to crack Neural Networks
by keghn (General AI Discussion)
February 15, 2018, 11:48:11 pm
ORWL
by korrelan (General Hardware Talk)
February 15, 2018, 11:22:51 am
Fastest C or C++ math library
by korrelan (AI Programming)
February 14, 2018, 05:08:02 pm
XKCD Comic : Robots
by Tyler (XKCD Comic)
February 14, 2018, 12:01:50 pm

Users Online

44 Guests, 0 Users

Most Online Today: 48. Most Online Ever: 208 (August 27, 2008, 09:36:30 am)

Articles