Towards GPT-4

  • 6 Replies
  • 2770 Views
*

infurl

  • Administrator
  • *********
  • Terminator
  • *
  • 991
  • Humans will disappoint you.
    • Home Page
Towards GPT-4
« on: August 08, 2020, 12:03:03 am »
https://www.youtube.com/watch?v=kpiY_LemaTc

Lex Fridman just released an interesting short video where he estimates the cost of training a hypothetical GPT-4 model based on current trends. At ten times the size of GPT-3 it would cost about ten times as much to train at today's prices but in ten years it would cost about the same.
« Last Edit: August 08, 2020, 12:26:32 am by infurl »

*

frankinstien

  • Starship Trooper
  • *******
  • 314
    • Knowledgeable Machines
Re: Towards GPT-4
« Reply #1 on: August 08, 2020, 12:31:52 am »
The human brain only has 86 billion neurons, but each neuron can have up to 10,000 to 100,000 connections! The definition of a synapse is a junction between two nerve cells, consisting of a minute gap across which impulses pass by diffusion of a neurotransmitter. So, the dendrite connections to axons are what add to the complexity of biological systems. Code wise symbolically representing 86 billion neurons is not a problem with today's technology. Where 228 TOPS are possible when in INT8 mode for the RTX 2080 TI, which could simulate 2650 connections per neurode for each of the 86 billion neurons simulated. But when citing the human brain's connectivity, those 100 trillion synapses include the encoding of raw data like audio and video which we don't have to encode in a neural network since we can store data digitally much more efficiently than biological systems! Now it's a matter of function not data that a neural network must be responsible for. So I think we're either at the point to make a true AGI or at least very close, it's matter of figuring out adept solutions with digital or silcone approaches to the type of processing biology has evolved to do, IMHO.
« Last Edit: August 08, 2020, 01:01:24 am by frankinstien »

*

infurl

  • Administrator
  • *********
  • Terminator
  • *
  • 991
  • Humans will disappoint you.
    • Home Page
Re: Towards GPT-4
« Reply #2 on: August 08, 2020, 01:28:42 am »
Each neuron is comparable to a complete computer with processing and storage taking place on a molecular level rather than just a unit of storage. That would make the human brain more like a supercomputer with almost a hundred billion nodes. The most powerful supercomputer in the world currently has fewer than two hundred thousand nodes, so we have a way to go yet. Then there is the question of efficiency; that supercomputer needs a lot more than 25 watts to run.

*

frankinstien

  • Starship Trooper
  • *******
  • 314
    • Knowledgeable Machines
Re: Towards GPT-4
« Reply #3 on: August 08, 2020, 02:11:12 am »
Each neuron is comparable to a complete computer with processing and storage taking place on a molecular level rather than just a unit of storage.

Ah...that's a bit of stretch...Why? A neuron's output, meaning its integration of inputs to produce an output maxes out with a spike train of 300 HZ and that's cycles per second. Where the effective information packet of a neuron is within milliseconds, so only a narrow code range of the spike train is effectively meaningful. Now don't confuse the cellular operations of a neuron which manages metabolism, transmitter transport, membrane pores, etc as the computational process of the neuron. I often analogize neurons to that old board game "Mouse Trap". Where the game has all kinds of contraptions and processes to do one thing and that is drop the cage over the mouse. So too is the neuron's infrastructure complex but simply resolves to a neuron's single spike or a spike train...

*

frankinstien

  • Starship Trooper
  • *******
  • 314
    • Knowledgeable Machines
Re: Towards GPT-4
« Reply #4 on: August 08, 2020, 02:47:59 am »
In comparison, a neurons signal spectrum is from 0 to 300 HZ:



Now look at a NTSC signal:


The bandwidth or information carrying capabilities of a neuron are very low, at best 300HZ compared to an NTSC signal that uses amplitude modulation for basic video(B&W), frequency modulation for the audio and a combination of phase modulation and amplitude modulation for color. So that's three types of information coding super-imposed on a single carrier wave.

So, neurons are no where near the information encoding capabilities of even old NTSC video! The bandwidth in neurology just isn't there to justify calling each neuron a computer onto itself...

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • ******************
  • Hal 4000
  • *
  • 4323
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Towards GPT-4
« Reply #5 on: August 08, 2020, 03:05:41 am »
:) The brain comes pre-built at birth with "extra" synapses/ connections. The brain is constantly losing neurons every day. As the brain fills up its storage space it is never using all of its storage... Just extra head-room. Also, the brain has a ton of connections to and from many neurons just so that it can pick and choose which to use/ strengthen. The brain doesn't depend on all of them.
Emergent

*

yotamarker

  • Trusty Member
  • *********
  • Terminator
  • *
  • 822
    • battle programming
Re: Towards GPT-4
« Reply #6 on: September 03, 2020, 06:44:46 am »
TLDR :
GPT is an AI by elon musks openAI
it uses prediction and lots of parameter to predict verbatim.
making it sound coherent weather it spits out truths or fantasy.

 


humor
by yotamarker (General AI Discussion)
Today at 08:24:13 am
Machine learning for fun and profit.
by infurl (AI Programming)
Today at 12:53:45 am
List of MMO Games for 2020
by infurl (General Chat)
September 22, 2020, 12:20:47 am
Friday Funny
by LOCKSUIT (General Chat)
September 21, 2020, 12:46:55 am
what is the end game for AI ?
by frankinstien (General AI Discussion)
September 20, 2020, 11:34:13 pm
Releasing full AGI/evolution research
by LOCKSUIT (General Project Discussion)
September 20, 2020, 09:14:07 pm
A.I script writer
by yotamarker (General AI Discussion)
September 20, 2020, 02:59:52 pm
map class : nodes vs array grid
by yotamarker (General AI Discussion)
September 20, 2020, 02:00:19 pm
New model beats GPT3
by infurl (AI News )
Today at 12:51:13 am
Robotic vacuum cleaner news.
by infurl (Robotics News)
September 22, 2020, 12:29:53 am
GPT-f SOTA AMTP
by infurl (AI News )
September 13, 2020, 12:56:47 am
Battle dogs!
by LOCKSUIT (Robotics News)
September 12, 2020, 04:27:46 pm
Artificial Intelligence Easily Beats Human Fighter Pilot in DARPA Trial
by LOCKSUIT (AI News )
September 12, 2020, 04:25:46 pm
Important memristor breakthrough.
by infurl (AI News )
September 07, 2020, 12:57:06 am
AllenAct for research in embodied AI
by infurl (AI News )
September 01, 2020, 01:27:35 am

Users Online

130 Guests, 0 Users

Most Online Today: 140. Most Online Ever: 528 (August 03, 2020, 06:16:11 am)

Articles