Do any neural network training schemes vary the nonlinear activation function?

  • 2 Replies


  • Roomba
  • *
  • 9
I have just read about ‘looks linear’ initialisation  This offers the possibility to train deep networks simply by initialising their weights properly (i.e. so that non-linearities cancel out, and the network initially 'looks linear') . It made me wonder if anybody has tried training a deep network with an activation function that starts linearly, and gradually becomes nonlinear, as training proceeds. Without non-linearity a deep network is equivalent to a single linear layer, no matter how many layers it contains. Thus at the outset of training a deep network is essentially 1 layer deep, and very easy to train (gradients do not decay or explode at all).  As training proceeds the non-linearity could be increased gradually. I would like to imagine this as the network effectively, slowly 'expanding' to a greater number of layers.  Of couse the weights would need to be adjusted to track the changing network.  It would be nice if the process could be controlled so that the network expands to the number of layers necessary to solve the problem and then stops.

Has anyone heard of anything like this ?  Opinions ?



  • Trusty Member
  • ********
  • Replicant
  • *
  • 550
A visual proof that neural nets can compute any function:

Which Activation Function Should I Use? : 



  • Trusty Member
  • ********
  • Replicant
  • *
  • 550
 When Neural Networks are used as transmission  lines to filters , then yes. Brain does not really use multiplexing like computers.
but they can have many many parallel path turned on at the same time to a filter, which is a little Neural network.
Multiplexing: Doing more with less. // Technology: 



Boltzmann machine .... amazing!!!
by ranch vermin (General AI Discussion)
Today at 10:32:00 am
AI safety
by Zero (General AI Discussion)
Today at 09:43:38 am
Grats to SquareBear
by Art (General Chatbots and Software)
September 24, 2017, 06:47:12 pm
XKCD Comic : Thread
by Tyler (XKCD Comic)
September 23, 2017, 12:00:05 pm
outline from gadient mask
by ivan.moony (General AI Discussion)
September 23, 2017, 10:45:22 am
by Memnon (New Users Please Post Here)
September 23, 2017, 10:02:32 am
AWESOME program that can find similar matches and exact matches (images) !
by LOCKSUIT (General Chat)
September 23, 2017, 07:53:27 am
Map of Computer Science
by keghn (General AI Discussion)
September 23, 2017, 01:34:02 am

Users Online

22 Guests, 1 User
Users active in past 15 minutes:
[Trusty Member]

Most Online Today: 30. Most Online Ever: 208 (August 27, 2008, 09:36:30 am)