Neuromorphic computing: An Overview

This is a different blog post than my normal ones. There will most likely be no code (okay there is no code...) and therefore no repository on this one. I'm currently preparing a seminar talk for this topic and find it quite interesting but also relatively complicated. This post is trying to give you an idea about what neuromorphic computing is. Why does it exist? Will it change the world of computing? There will be a some literature references in this post and if you enjoy such non coding posts and hope I'll enjoy writing it :D then I'll consider writing more of these kind. (Writing it was quite nice ;) )

Okay let's begin. What is neuromorphic computing? The idea is basically to make a brain model to understand the human brain better. We wanted to have AI and then we thought okay we know one intelligent creature on this world (whatever intelligent really means) and that is us humans but this idea didn't work out that great mostly because someone found a different way to create something similar which is called deep neural networks. They are easy to train using backpropagation but they have nothing to do with how our brains work. It's similar to airplanes in comparison with birds. They work well but they aren't really biologically inspired. We aren't only interested in creating an AI we are also interested in a better understanding of our own body and especially brain. Which means it is still quite interesting to get back to neuromorphic computing and give it another shot.

Brain vs Deep neural network

I wrote a bit about neural networks on my blog. The general idea is that we have some layers with some neurons and they are all connected to the next layer (fully connected network). The first layer connects to the first hidden layer and the last hidden layer connects to the output layer. The connections have some weights and to obtain the output we go from input layer to output layer and do some matrix matrix multiplications. In general our neural network model is extremely simple and organized and everything flows in one direction. Source: Wikipedia: Artificial neural network

In comparison the brain: We have a bunch of neurons "floating" around. Those are connected to each other in a more random like fashion and there is no direction (it is still a directed graph, but they aren't organized in layers). Additionally artificial neural networks are working with floating point numbers i.e if the input is 0.3 and the weight is 0.2 we get 0.06 as a result input to the next neuron add it up with all the other inputs and then send it over to the next layer. The brain has a different technique. Neurons have more or less two states: They either fire or they chill. That is binary and our computer model uses those complicated floating point numbers. Why??? Good question! The point is that our brain works using time, which is continuous. If the next neuron fires shortly after the current one the weight is strengthen and if the next one fires if the current doesn't the strength shrinks. Now you might think: Okay that sounds pretty simple why are we not using it?

Why not simulate?

It's not that clear how to train it and if we simulate it with our normal CPU or GPU hardware it's rather slow for a lot of neurons and connections. That is the case because we normally have maybe 8 cores in the cpu and some thousand in the GPU but there are billions of neurons in our brain. And? Oh yeah so to simulate a so called spiking neural network we need to know when each of them is firing check whether a potential in a neuron is above the firing threshold and then fire but the inputs into one neuron aren't happening all at the same time like in our artificial version. They all work on their own schedule. Basically a simulation would work like this: We have a loop and check each neuron whether it fired after the last time we checked and get the first time of all those neurons. Do a firing step and start all over again. That makes it extremely hard to simulate this in real time and even real time is quite slow as we are this weird creature on earth for whom everything is too slow. We first walked some kilometers per day and then we used horses and cars and trains and then airplanes. Just to travel faster to a different location. We use E-Mail instead of waiting until we visit our friend next year to talk and then E-Mail was too slow so we all use some instant chat system. Okay sorry... What I wanted to point out is that brains evolved for millions of years but okay we might find out how our brains work and jump straight to a human newborn brain. Then we still have to wait some years until we can use that kind of system to make something useful. Therefore we want to have a artificial human like brain on steroids which can learn everything much faster and our first guess normally is: Well that can't be too hard as the brain "wires" have a speed ranging from 1-100m/s which isn't close to what computers can achieve. All in all the point is that current computer hardware isn't the best for this task. Another important ingredient is power consumption. Your laptop consumes about 30 W and a desktop computer maybe about 200 W. Your brain uses 20 W and a super computer simulating it would consume maybe 10 MW. Slower and a much much bigger power consumption... I think it's clear that we can and want to improve here.

To do that we should first understand how the brain works. Well nobody knows...

“If the human brain were so simple that we could understand it, we would be so simple that we couldn’t”

• Emerson M. Pugh (he mentioned that his father said this)

If you want to know more about that quote: QuoteInvestigator and yes I realized again that the internet is a weird place.

Nevertheless we can still have a look at a very basic understanding of the brain and one goal of the neuromorphic research is to understand the brain better.

According to Jeff W. Lichtman a Professor of Molecular and Cellular Biology we know about: 0.004735% of the human brain. Okay he didn't say it like that. He mentioned that he thinks we walked about 3 inches on the one mile long path of fully understanding the human brain in this YouTube clip and I just converted it. If you are living in the metric system or just want to have a different idea as it is hard to walk 3 inches... If you have a nice walk from New York to Miami (okay it's actually a swim) we would have worked only a single Manhattan city block so far which can be a nice little start of a walk. We still have the rough swim in front of us. Maybe we walked a little further by now as he mentioned this in 2014...

Okay I got a bit distracted here. Let's have a look at our city block understanding of the brain first before we dive into the hardware stuff.

Structure of a simple neuron

A neuron in the brain has three parts: dendrites, soma and axon whereas the dendrites are the input connections, the soma is the processing unit and the axon is the output. Well what is a synapse then? It's the connection between the axon of one neuron and the dendrite of the next. (I refer here to axo-dendritic synapses). One neuron in the brain can have around 15,000 connections to other neurons both short and longer connections to different parts of the brain.

Source: [Illustration by: Quasar Jarosz (Wikimedia)](http://commons.wikimedia.org/wiki/File:Neuron_Hand-tuned.svg) License: [CC BY-SA 3.0](http://creativecommons.org/licenses/by/3.0/)
I've added the legend inside the image.

Btw maybe you still remember that I mentioned a range for the speed of connections between 1-100m/s. This is due to the myelin sheath some connections are missing it and then the speed is 1m/s and with this sheath it is 100m/s which is the explanation of some delayed pain when we hit something involuntarily with our big toe. The slow huge ouch is due to nerve connections without this specially myelin sheath. Not really sure why we don't use the fast ones everywhere but maybe it's expensive like Cat 7 vs Cat 6 Lan cables? I'm really not a biologist as it isn't important for our topic here and I can't read about everything I leave this as a task for you to find out and put it in the comments. Edit: I found the following explanation on biology stackexchange: It's not by design (cause evolution) and takes space. Answer by Andy McKenzie

Encoding information

Let's talk a bit more about encoding information. The basics again: A neuron gets some input and if the input sums up over a threshold we fire and send to all our outgoing connections that we are excited. This firing has a length and an amplitude but these are not used encode any information. The information is encoded in how often they fire and how long the chill phases are in between. After each firing there is a exhaustion phase (that's just my naming here) which is determined by the so called absolute refractory period. In this phase no firing can occur and we need this to encode information in the number of firings as we otherwise wouldn't be able to determine this.

No code in this post yet so at least have some maths stuff... and also some more definitions. There is a presynaptic and a postsynaptic neuron so the presynaptic neuron fires in the direction of the postsynaptic one. Additionally we can have positive and negative weights as in deep neural networks in biology terms the positive ones are called excitatory and the negative ones inhibitory. We have a postsynaptic potential (PSP) which can be excitatory or inhibitory and is referred to as EPSP and IPSP. If nothing happens a neuron has a resting potential \(u_{\text{rest}}\) (the chill state). In general we call \(u_i(t)\) the potential of neuron \(i\) at time \(t\). Now assume the presynaptic neuron fires at \(t=0\) then the postsynaptic neuron \(i\) has a PSP of \(\epsilon_{i j}\):

\[ u_{i}(t)-u_{\mathrm{rest}}=: \epsilon_{i j}(t) \]

Let us assume we have two input signals and one of them is firing at point \(t=0\):

and if we receive four input spikes shortly one after the other we get

and then the neuron tells everyone that it is sooo excited. Of course four inputs isn't always enough it depends on the timing.

This is how it looks like if the last input is just a little bit too late to get us excited.

For simplicity here \(u_{\mathrm{rest}} = 0\)

The function for this potential is based on a physical model with a capacitor C and a resistor R in parallel. We have a current \(I(t)\):

\[ I(t)=\frac{u(t)-u_{\mathrm{rest}}}{R}+C \frac{\mathrm{d} u}{\mathrm{d} t} \]

and \(u\) is the voltage. Now we define \(\tau = RC\) and obtain

\[ \tau_{m} \frac{\mathrm{d} u}{\mathrm{d} t}=-\left[u(t)-u_{\mathrm{rest}}\right]+R I(t) \]

If we assume we have a voltage \(u = u_{\mathrm{rest}}+\Delta u\) at \(t=0\) and we set \(I(t) = 0\) for \(t>0\) we get:

\[ u(t)-u_{\text { rest }}=\Delta u \exp \left(-\frac{t-t_{0}}{\tau_{m}}\right) \]

so it decays to its resting value.

Additionally we have a time before where we have a constant current \(I_0\) for the time \(0 < t < \Delta\). In this time the following holds true:

\[ u(t)=u_{\mathrm{rest}}+R I_{0}\left[1-\exp \left(-\frac{t}{\tau_{m}}\right)\right] \]

If you want to read more about the equations please read: Neuronal Dynamics Chapter 1.3.

This gives us the curve seen above.

Additionally we reset our potential if we fired so that a constant current might produce a so called spike train:

Please don't pay too much attention at my plots as here we reset the current directly instead of resetting it over time (fast) but I assume that my xkcd style plots show that it's just a sketch. Additionally the neuron should fall in its chill state for some time.

It should be mentioned that this is only a rough model and we (humans) know that it has some limitations one of them is that our neuron model doesn't have a memory of previous spikes normally the gaps between spikes would increase until a steady state is reached. If you want to know more about the limits and some other models please continue reading Neuronal Dynamics Chapter 1.4.

Hardware

I'll write about HICANN in this post and maybe TrueNorth from IBM in a follow up if there is interest.

The HICANN chip is developed at the university of Heidelberg where I study computer science atm. In this paper A wafer-scale neuromorphic hardware system for large-scale neural modeling it's mentioned that they implemented some form of adaptation as well but changing the parameters can lead to the simple LIF model described above (Leaky integrate and fire). If you're interested in more detail about how the model looks like then please check that out. Here I think it's sufficient to get the basics of the model and an overview of the hardware implementation.

They call it AdExp which is an exponential integrate and fire model but as mentioned this shouldn't distract us.

I'm neither a neuroscientist nor a hardware specialist but my goal here is to bring you maybe to level 4 on neuromorphic computing with this post where Tim Urban tries to reach level 6 in his WaitButWhy posts which take hours to read and well it's his job so I'm okay with level 4 here. Rough level system from here:

10 - Is world’s leading expert on the idea

6 - Can answer any layman’s question and forms intelligent opinions on the idea

5 - Knows about the idea, and can discern inaccurate statements about the idea

4 - Knows about the idea, and can explain what’s been learned in one’s own words

3 - Heard of the idea, and recites what others have said about it

I explained roughly why our normal hardware isn't perfect for this task and has no background in any biological system. Maybe you heard about von Neumann architecture which is the architecture of your hardware you're reading this on and the projects for neuromorphic hardware is non von Neumann architecture. I know very elegant...

The von Neumann architecture consists of input and output devices and then a separation of processing unit and memory which doesn't exist in the brain in that form. There is no dedicated space in your brain which stores all the memory. Your memory is stored in the connections of the neurons. The other main point is that our brain is a highly distributed system instead of having complicated but few cores we have simple but many of them.

We want to be able to achieve fast transfers between neurons and the neurons should be able to have a variable system of connections so we don't want to have a fixed structure where we determine before creating the chip that neuron 1 is connected to 2-1000 and neuron 2 is connected to the even numbers between 4 and 2000 or stuff like that.

In the HICANN system there is the following structure: The neuron has two input connections, a spiking output to neighboring neurons and to the network as well as a capacitor we mentioned before to integrate, a leaky unit and some special parts for their more complicated model.

Two neurons are connected via synapses but they aren't physically connected (predefined) which wouldn't make much sense as we want to program the chip. Instead each synapse has a bit code (here 4 bits) which is used to activate the synapse for communication. If this bit code is transmitted the synapse knows that it has to do some work. Work means it uses a different short bit code (here 4 bits) which defines its weight and converts this into a current which is the input for the postsynaptic neuron.

As mentioned in the brain section we have excitatory and inhibitory inputs which increase and decrease the potential of a neuron respectively which is the reason why the neuron here has two input connections. A synapse can be deactivated if the weight is set to 0.

They mention that they connected many of this so called analog network cores together and achieved to get 40,000,000 synapses and 180,000 neurons on one wafer which can be connected to bigger units. Unfortunately it isn't really mentioned what they have done with it in this paper.

For a comparison the human brain has about 100,000,000,000 neurons which is more than 500,000 times that much and each neuron can have up to 15,000 connections. In this hardware model one neuron can have up to 14,000 connections via synapses which is close to our brain.

Unfortunately I wasn't able to find out how much power this chip consumes but I'll add it here after I find it out. They mention that it reduces the power consumption by several orders of magnitude which isn't a very precise statement. Their speed up factor is about 1,000 which means that a year can be simulated in about 9 hours.

If we have a model to train this network it is at least a step in the right direction. Maybe I'll have a look at using it for learning some tasks and find out what they are using it for.

Some links you might find interesting:

• YouTube: Event based simulation of spiking neural networks in Julia

• YouTube: Neuromorphic Computing, AI Chips Emulating the Brain

A bit off topic but really interesting and at least the first part is a more detailed and funny introduction to the brain. The idea there is more to read and write from and into the brain instead of building a new one. WaitButWhy: Neuralink

What's next: Well I have to write this down more formally into a paper and do a presentation. Done ;) I would like to have a deeper look into Spiking neural networks and maybe simulate something in Julia. (Currently working on it) Would be nice to train it MNIST or something. Maybe having a look at PyNN which is framework for building neuronal network models and very interesting would be to get some actual hardware like Spikey.

Updates: I finished my presentation and handed in my paper which was accepted by other students in a peer review. You can download the paper here. After watching the live stream of NeuraLink on YouTube I published a small recap post.

Another update: I finished the post about neuromorphic hardware.

I'll keep you updated if there are any news on this on Twitter OpenSourcES as well as my more personal one: Twitter Wikunia_de or if you enjoy Reddit you can subscribe to my subreddit.

Thanks for reading!

If you enjoy the blog in general please consider a donation via Patreon. You can read my posts earlier than everyone else and keep this blog running.

Want to be updated? Consider subscribing on Patreon for free