Biologically-Inspired Neuromorphic Computing

Abstract

Advances in integrated circuitry from the 1950s to the present day have enabled a revolution in technology across the world. However, fundamental limits of circuitry make further improvements through historically successful methods increasingly challenging. It is becoming clear that to address new challenges and applications, new methods of computation will be required. One promising field is neuromorphic engineering, a broad field which applies biologically inspired principles to create alternative computational architectures and methods. We address why neuromorphic engineering is one of the most promising fields within emerging computational technology, elaborating on its common principles and models, and discussing its current state and future challenges.

Why neuromorphic?

The evolution of semiconductor and computing technologies between the 1950s and 2000s is a remarkable and continual progress that revolutionized how information is processed, stored, and dispersed across the world. This was mainly accomplished by the ability to scale the elements of circuits to smaller and smaller sizes, increasing the number of possible components, while decreasing their power consumption and material cost per device. ¹

During this time, personal computers generally relied on a single central processing unit (CPU). Over time, this central processor was simply made faster, with clock speeds jumping orders of magnitude, from megahertz to gigahertz. As a result, the exact same algorithm running on a few years newer hardware, without any modifications, significantly outperform the older system (see Figure 1). This golden era is referred to as the period of “Dennard scaling,” in which shrinking down circuits made them both faster and more efficient.^2,3 Several factors led to these improvements: smaller dimensions greatly reduced parasitic resistances/capacitances present in circuitry and allowed lower voltages to be used.

Figure 1.

Historical trends in microprocessor scaling. The end of Dennard scaling in the mid-2000s can be observed, as frequency levels ceases off and the number of logical cores (processor components which can carry out independent execution threads) begins to increase.

However, by the mid-2000s, it was becoming clear that Dennard scaling was coming to an end. Due to physical limitations of the transistor, such as how much voltage it requires to turn on and off (subthreshold swing) and how much current leaks through its gate oxide, the same gains were no longer made by shrinking the transistor further. Since then, transistors have continued to shrink, but processor clock speeds have remained essentially constant from 2 to 5 GHz. Instead, engineers attempted to give users more performance by adding multiple cores to a processor. While useful in certain situations, multi-core processors can be challenging to create programs for and do not give the same universal performance gains of, for instance, an increased clock speed. As a result, the end of Dennard scaling indicated that despite historical trends, future enhancement of computing capability can not simply increase boundlessly.

Since the mid-2000s, transistor scaling has become ever more aggressive to get smaller, allowing a continual growth in the number of components in an integrated circuit, which has been described as “Moore’s Law.” In this, Gordon Moore (a co-founder of Intel) predicted that the number of transistors in an integrated circuit can double every two years. Though Moore’s Law has continued, processor architectures had to evolve in new directions by adding more cores and prioritizing efficiency due to the end of Dennard scaling. However, even Moore’s Law is being confronted by fundamental physical and economic limits.

State-of-the-art 5 nm transistors can be easily measured in terms of length of the silicon lattice constant (0.54 nm), and while it is true that, as Feynman stated, there is “plenty of room at the bottom,” a more pressing question is whether building there is economically viable. While smaller transistors allow for more circuits to exist in less space, the tools needed to accomplish this task become more complex and precise. As the price per transistor scaled exponentially downward, the cost of facilities and tools to build them scaled exponentially upward. The process of developing and purchasing next-generation tools like extreme ultraviolet lithography has become an extremely intensive and costly task. ⁵

Perhaps due to this, in 2018, GlobalFoundries (one of the largest semiconductor manufacturers in the world) made an announcement that it was completely ceasing development of its 7 nm node and focusing on its existing 10 nm technologies. ⁶ This left only three semiconductor behemoths which are still attempting to create smaller circuits on an industrial scale: Intel, Samsung, and the Taiwan Semiconductor Manufacturing Company (TSMC). At some point, even the ability of these giants to balance the cost, profits, and risk of building smaller may reach a standstill, and scaling may reach an end.

With barriers to advancing conventional computing clearly becoming visible, one must begin to consider alternative approaches if future computers are still to become more capable and efficient.

Currently, there are several major approaches which attempt to address this challenge, and it is likely that all of them have sufficient potential to carve out a niche in the future of computing technologies. “Heterogeneous integration” is a broad term, which generally implies that future circuits will go beyond silicon and its alloys and integrate devices based on many novel materials and effects. ⁷ Another approach is “quantum computing,” which leverages physical phenomena such as superposition and entanglement to crack open certain classes of previously intractable problems. ⁸ Yet another approach is “integrated photonics,” which augments circuits with light-based computation, using the speed and behavior of photons to compute in an entirely different realm.^9–11

However, an intuitive source of inspiration for the next frontier of computing technology exists when one considers the realm of biology: the brain. While one can imagine what quantum or optical computers could achieve, the proof of what a computer that mimics the brain could achieve is clearly visible.

Artificial computing architectures which mimic the brain are generally referred to as “neuromorphic” and include a diverse array of systems. ¹² All artificial neural networks (ANNs) are arguably neuromorphic systems, but often neuromorphic is intended to imply that an architecture carries out its computations in a fully spiking manner (the distinction between spiking and non-spiking ANNs is clarified later).

But while it is one thing to realize the potential of what is possible by observing what exists in the world around us, it is quite different to be able to understand, much less to engineer and re-create, these phenomena. Many fields of study have encountered this frustration, from aviation to nuclear fusion, and neuromorphic engineering is no exception. However, neurology is a well-established field that contains a wealth of knowledge which can be applied to the design of practical neuromorphic technologies.

Neurons: the building blocks of the brain

In order to create a neuromorphic system which can emulate the capabilities of the brain with artificial hardware, it is clear that first we must have an understanding of how the brain and more generally, nervous systems operate and process information.

The neuron is a specialized type of cell that has long been recognized as the fundamental building block of the brain (Figure 2). ¹⁴ Neurons display unique electrical excitability characteristics, which allows them to process and exchange signals on a vast scale and a relatively quick rate. To understand how this is done, a closer look at how a biological neuron operates is necessary.

Figure 2.

Illustration of a single Golgi-stained Purkinje cell, created by Ramon y Cajal (public domain). ¹³ Neurons can have complex morphologies, allowing them to connect with thousands of other neurons. a-d label features noted by Cajal in his original work which are not discussed here.

Normally, neurons maintain an electric potential of approximately –70 mV across the bilipid layer which separates their interior from the outside (Figure 3). This potential is referred to as the resting potential, and it is maintained by particular structures in the membrane called “ion pumps” that use energy for repelling 3 sodium (Na⁺) ions outside the cell and passing 2 potassium (K⁺) ions inside the cell. This creates a charge difference or potential across the membrane. However, other structures known as ion channels also exist in the membrane. These channels can be either open or closed depending upon the local electric field (voltage-gated) or binding with a specific chemical (ligand-gated). When open, ion channels selectively allow certain ions (most commonly Na⁺, K⁺, or Ca²⁺) to passively diffuse through the neuron’s cellular membrane (Figure 3).

Figure 3.

Illustration of the membrane of a neuron, highlighting the ion pumps and channels that allow the membrane to dynamically react to electric potentials.

Through the interplay of ion pumps and channels, the membrane potential of a neuron can go through dramatic changes when the cell is electrically stimulated. If an electrode is used to disturb the potential above a certain threshold, voltage-gated Na⁺ channels in the cell’s wall open, allowing these ions to diffuse into the cell’s interior. This causes a very quick increase in the cell’s local potential. However, the Na⁺ channels are only active under lower voltage and thus close as the potential continues to rise. In contrast, the K⁺ channels begin to open at these higher voltages, and K⁺ ions concentrated in the cell begin to diffuse outward, returning the cell’s potential to slightly below its resting value. This successive, quick rise and fall in voltage creates the eponymous voltage spike which is the basis of electrical communication between neurons (Figure 4). ¹⁵

Figure 4.

Generation of the “spike” due to an increase from the neuron’s resting-potential past the threshold to create an action potential.

Spikes are naturally created in neurons through a variety of means. Often, a spike begins when neurotransmitters emitted by synapses of other neurons bind to receiving sites on a branch of the neuron known as the dendrite (Figure 5). The changes in potential across the dendrites sum in the neuron’s main body, and if they exceed a threshold, they can cause the neuron to generate an action potential.

When an action potential arises in a neuron, it quickly propagates both back toward the dendrites where messages are received and down the long, thin axon toward other neurons to communicate with (Figure 5). After an action potential arises, the neuron enters a short refractory period (undershoot): a time in which it cannot generate another action potential. This creates an important limit on the maximum rate at which it can fire.

Figure 5.

Structure of a generic neuron, with different regions and the flow of information labeled.

However, if an action potential is not created, excess change in the neuron gradually fades away, and the neuron restores its resting potential without creating a spike. This rate at which an excitement fades is often referred to as a leakage current.

Synapses: how neurons connect

While a single neuron can provide some utility, the advanced computational abilities of neurons only emerge when they can interact with one another. As previously mentioned, the dendrites of a neuron contain sites where messages can be received by the neuron. Axon terminals from other neurons approach spines on these dendrites incredibly closely, with the synaptic cleft formed between the two spanning approximately 20 nm. ¹⁵ If it fires, the pre-synaptic (or “sending”) side of the synapse releases particular chemicals termed neurotransmitters, which can diffuse across the synaptic cleft and bind to receptors on the post-synaptic (“receiving”) neuron. Two of the most common neurotransmitters are glutamate and γ-aminobutyric acid (GABA), which bind to specialized ligand-gated ion channels existing in prevalence on dendrites. As a result, of the binding, those channels open and allow the passage of ions through the membrane of the receiving neuron (Figure 6).

Figure 6.

Mechanism of synaptic transmission. A firing pre-synaptic neuron (blue) releases neurotransmitters, which diffuse across the synaptic cleft and can bind to ligand-gated ion channels on the post-synaptic neuron (green). The number of bound ligand-gated channels then influences the potential within the post-synaptic neuron.

Excitatory neurotransmitters like glutamate can depolarize the neuron (making it more likely to fire), and inhibitory neurotransmitters such as GABA hyperpolarize the membrane (making it less likely to fire). The type and amount of a released neurotransmitter, number of ion channels on a post-synaptic dendritic spine, and even the diffusive process of neurotransmitter transport all combine to create a synapse’s effective weight. ¹⁵ The sign and magnitude of a weight summarizes the meaning and significance a post-synaptic neuron assigns to a pre-synaptic neuron’s firing. This weight is effected within a remarkably small area, and with the existence of numerous dendrites and axon terminals, this allows each neuron to have thousands of synapses, for communication with other neurons on a massively parallel scale. ¹⁷

However, synaptic weights are far from static values; many of the learning abilities of the brain are thought to stem from the remarkable plasticity of synapses. A synapse’s weight can be changed through many factors, including the construction or destruction of ion channels and change in dendrite morphology.^18–20 A long-term goal in neuroscience is to determine the underlying mechanisms which regulate how this process occurs, as to better understand how these abilities arise and how diseases can affect them.

One of the most well-established and studied theories behind the regulation of synaptic weights is spike-timing dependent plasticity (STDP).^21,22 In STDP, the weights of incoming synapses which were active in a short period of time before a neuron fires are increased, and the weights of incoming synapses which were active (synapses which receive a spike and influence the neuron’s charge) in a period shortly after a neuron fires are decreased. In this way, messages that are frequently received by a neuron directly prior to its firing become prioritized, and other, less-relevant messages are disregarded. This principle has been experimentally observed in biological neurons, ²¹ used as a source of inspiration for unsupervised learning techniques in machine learning,^23,24 and implemented in a multitude of different neuromorphic architectures (see Figure 7). ²⁵

Figure 7.

Illustration of a stereotypical STDP weight-change curve. STDP predicts that an incoming synapse which is active (one which receives a spike) just before the neuron fires will gain synaptic weight. In contrast, an incoming synapse which is active just after a neuron spikes will eventually be ignored.

However, the extent of STDP’s role within the brain remains a hotly debated topic, and it is just one of several different proposed theories behind learning. Other principles of synaptic behavior are also emerging, most prominently the manner in which dendrites perform their own computations to non-linearly sum incoming signals. ¹⁷ These, and other topics, are major areas of investigation which are likely to strongly influence future neuromorphic computing as they develop.^26–29

The roles and behaviors of neuronal networks

Having established the building blocks of neuronal networks, we can begin to investigate how they work together to perform more advanced functions. For instance, reflexes are examples of simple neuronal circuits that exist in the peripheral nervous system. Specialized neurons in the skin can be activated by heat and send messages further down a limb to the spine or brain. If sufficient messages are received by an integrating neuron, it will send out its own spikes to muscles which reflexively retract a limb from the danger. ¹⁵

Although the processing carried out in this example is extremely simple, the ability to process and react (even on an extremely basic level) is a huge utility to any creature attempting to survive in a changing and potentially hostile environment. As a result, almost every known animal on Earth contains neurons, and the sheer universality of these cells demonstrates their utility. ³⁰

More complex behaviors and representations of information in the brain have been uncovered, and this has begun to shed light on the sheer variety of ways in which neurons can collectively express and process information. One of these discoveries was the existence of “grid cells” neurons in the endorhinal cortex which show the remarkable behavior of becoming more active when an animal’s location was located close to a point on a regular grid defined inside the space which it was navigating. This discovery began to provide a more solid understanding of how neurons can represent and track position within a space^31,32 and has led to new abilities within artificial systems navigating simulated spaces. ³³

One key feature of neurons and synapses is that they are fundamentally non-deterministic. This is a result of many features of neurons’ operation: synapses release variable amounts of neurotransmitters, which randomly diffuse across the synaptic cleft. Neurons’ membranes contain a finite number of ion channels, which are discretely (and sometimes semi-randomly) activated, and this can lead to significant variability in the neuron’s spike timing. These and other features contribute toward a growing picture of the neuron as a stochastic computational element.^34,35

Work inspired by this view has shown that through collective activity of stochastic neurons, a network can produce collective activity which represents samples from a distribution.^36,37 This more general ability provides a basis to discern how neural networks can carry out powerful statistical inferences which we take for granted, such as recognizing which pairs of shapes “go together” in a classic experiment. ³⁸

But while advances are being made in understanding how separate pieces of the brain represent and process information, the remarkable way in which these systems integrate to form a whole remains out of our current understanding. In other words, while we can begin to understand how these individual components (tracking place, recognizing visual information) operate, the knowledge of how they communicate and influence one another to allow what we visually observe to influence our internal sense of place remains a question to be answered. Fully mapping the neural connections and precisely simulating even a small section of a human or mammalian brain is an enormous undertaking, which remains at the forefront of current science.^39–42

Artificial and spiking neural networks

Neuromorphic engineering attempts to re-create the abilities of biological neural networks using artificial hardware. This hardware may be designed for a variety of purposes, including “deep learning” tasks, or development of biologically realistic models of neuronal networks to gain insights into the brain’s functionality.

Before proceeding further, it is important to clarify how the neuromorphic architectures discussed here differ from the more well-known “artificial” neural networks (ANNs), which have seen an enormous resurgence during the 2010s. While very closely related and inspired by the nervous system, these two fields carry important distinctions. ANNs are networks in which information is passed and represented through real numbers, and not spikes. In general, an ANN provides an information processing structure which takes a number of real-valued inputs (representing data such as an image), and propagates this information forward through a number of synapses and neurons to compute a feature (such as a predicted image classification). In one common ANN architecture, a fully connected feed-forward network, each neuron in a hidden layer sums from all neurons in the previous layer, and outputs to all neurons in the next layer. It does not receive or send information through spikes but receives and passes real numbers representing “activation,” which are summed by other neurons and passed into an activation function (commonly sigmoid, or hyperbolic sine/tangent) to keep values within a desired domain (see Figure 8). ⁴³

Figure 8.

Differences between the more common artificial neural networks (ANNs) (a) and spiking neural networks (SNNs) (b). ANNs operate numerically, and do not necessarily change over time. SNNs only pass messages via spikes, and are fundamentally temporal in nature.

This mode of operation has proven wildly successful, and an explosion of applications has been made using these ANN architectures. ⁴⁴ However, their fundamental mode of communication remains strikingly different from true, biological neural networks that apply spikes. This carries penalties particularly in the realms of implementation and operating costs. Each neuron is always computing and summing and must always pass on its value, even if that value is “zero”. Each message must also be represented using a floating point (usually of 10 or more bits), ⁴⁵ which leads to large memory and computational requirements. In contrast, a spike is a single, instantaneous, 1-bit message, which is only passed when needed. This leads to fundamental differences in favor for spiking neural networks (SNNs) over the ANNs.

All messages passed between neurons in SNNs are encoded via series of near-instantaneous spikes through time. Although they may have continuously valued synapses and internal potentials, the use of spikes for communication fundamentally separates SNNs from ANNs. In addition, SNNs process spikes over a temporal domain and vary with respect to time, which is not necessarily the case for ANNs. Here, we define neuromorphic architectures as referring to dedicated hardware for executing SNNs.

Artificial neurons and SNNs

Having discovered the potential of what biological networks of neurons can accomplish, many researchers have attempted to artificially recreate neuron-like elements which capture some aspects of their biological forebears’ capability. One of the simplest and most pervasive descriptions of a neuron is the integrate-and-fire (IF) model (Figure 9).^46,47

Figure 9.

The potential of various integrate-and-fire models over time. The same spike train is incident on all four neurons which are all variations of the integrate-and-fire (IF) model (a). These modifications include noise (b), leakage (c), and both leakage and noise (d).

In this model, the neuron is controlled by one parameter: its internal voltage. It begins at the resting potential, which changes as the neuron accumulates the charge created by incoming spikes, with each being weighted by a synapse. If at any time, its voltage exceeds an internal threshold, the neuron fires, sending out a spike. Then, its potential is reset, and after a refractory period, it resumes accumulating charge.

In addition, a leakage current is often included in the IF model, as leakage can play a critical role in allowing neurons to have a time-varying response. ⁴⁸ this modification means that a voltage above the resting potential will eventually decay back to its resting level. The rate at which this occurs is defined by the leakage current, and this modification is termed the leaky-integrate-and-fire (LIF) model.

The LIF model of a neuron has many direct electronic implementations available, ⁴⁹ and many simulations and dedicated hardware implementations of it have been created. ²⁵ As with real neurons, LIF neurons begin to become useful when used to build more complex networks or SNNs.

Another modification to the IF model considered here is the introduction of noise. As previously mentioned, biological neurons are non-deterministic and demonstrate many different types of stochastic behavior.^50–53 This fact has significant impacts on the design of neuromorphic architectures and SNNs. But to simply approximate the existence of noise in a neuron’s potential, the IF model can be modified to include a stochastic variable which causes voltage to fluctuate through time.

Finally, the noisy and leaky IF models can be composed to create a noisy/leaky-integrate-and-fire (NLIF) model, in which the voltage fluctuates, but always experiences a pressure to return to the resting potential.

While these simple neuron models are utilized as the basis for many artificial neurons, many other models such as the Hodgkin-Huxley, Izhikevich, FitzHugh-Nagumo, and multi-compartment exist. ²⁵ Many of these models attempt to capture a more biologically realistic representation of a neuron’s action into mathematical form.

State-of-the-art in neuromorphic architectures

While any standard computer can be used to simulate an SNN, obtaining the potential advantages of SNN-based computation is fundamentally dependent on their execution on specialized neuromorphic hardware. In part, this rests on the fundamentally parallel nature of neuronal computation; each neuron simultaneously carries out the continual process of calculating its own internal state. If represented by a series of differential equation, a standard von-Neumann computer must sequentially and iteratively calculate the new state of each neuron in the network. This leads to much greater computation times on non-neuromorphic hardware, particularly as the scale of a spiking network grows.

To facilitate efficient and powerful research into SNNs, many neuromorphic architectures have been proposed and developed by a variety of academic and industrial workers. Two academic neuromorphic architectures are SpiNNaker ⁵⁴ and BrainScaleS, ⁵⁵ which aim to provide platforms for simulating SNNs with a high degree of biological realism and are supported under the European Union’s Human Brain Project. SpiNNaker is a digital and highly parallelized architecture, which allows independent processors to solve arbitrary neuron models and pass messages which represent spikes. BrainScaleS is a mixed analog-digital system, which directly uses the electrical dynamics of transistors to create a neuron model, and stores all neuron parameters in local floating-gate transistors.

Experimental neuromorphic architectures have also been created by companies; IBM’s TrueNorth ⁵⁶ and Intel’s Loihi ⁵⁷ are two examples. These two digital architectures focus mainly on exploring the practical applications of SNNs and use modifications of LIF neurons and synapses to prioritize computational and spatial efficiency. One of the main focuses of these commercial prototypes is to explore how ANNs can be adapted into spiking architectures, potentially allowing ANNs to be run and trained on these spiking platforms rather than the more expensive graphics/tensor processing units (GPU/TPU).

The design choices made for each architecture create limitations and challenges for potential end users of the system. For instance, BrainScaleS uses analog components which have variability, and therefore simulations of a “perfect” system can diverge from results run on actual hardware. TrueNorth can achieve great efficiency but has strict limits on the connectivity of each neuron. These and other restrictions can have important implications for utilizing a given system.

Challenges in neuromorphic engineering

While neuromorphic engineering has the ambitious goal to recreating the computing capabilities of nervous systems in hardware, it is still an emerging field. It has few commercial products and applications, particularly when compared with much more prevalent ANNs/DNNs employed by many researchers and business sectors.^25,58 This clearly indicates that the field has many challenges which have yet to be addressed.

One of the foremost issues is that current neuromorphic systems are created from an incomplete picture of how large-scale nervous systems operate. Critical issues such as how information is represented within these networks, how networks organize, and how they carry out processing and learning still make open questions for further research.^59–63 A better understanding of any of these topics could revolutionize neuromorphic engineering.

Another challenge in neuromorphic engineering is to create hardware that can efficiently recreate the already-known properties and abilities of neurons and synapses. One of the key challenges is designing an artificial synapse which is compact, reliable, and efficient enough to enable the huge parallelism that exists between neurons. ⁶⁴

Finally, one of the key issues in neuromorphic architectures and novel computing architectures in general is the difficulty of programming them to do useful tasks. Von Neumann architectures share a common language of procedural programs, which sequentially carry out well-defined mathematical tasks. However, in ANNs and SNNs, programs are most commonly expressed as weighted graphs which represent the synaptic connections between neurons. The meaning of these connections and the overall operation of the graph are often opaque and inscrutable, and programming of these structures must be carried out via an optimizer. Advances in a common platform for expressing, optimizing, and executing neuromorphic programs are another key obstacle in the field.

Conclusion

Neuromorphic architectures aim to provide a dedicated platform for computing spiking neural networks in a biologically-realistic manner, with the goal of exploring and possibly recreating the extraordinary abilities of biological neural systems. This motivation has led to a number of institutions, both academic and industrial, creating neuromorphic architectures. However, the challenge of programming these novel architectures and finding applications well-suited to them is one of the main obstacles preventing their more widespread adoption; these systems must compete in today’s already-complexifying landscape of heterogeneous computation (including GPUs and FPGAs). We expect that advances in programming and a greater understanding of the biological and mathematical underpinnings of spiking networks may be necessary before neuromorphic architectures can fully apply their advantages and gain mainstream adoption.