Computing is not about modelling or training AI models. It’s about executing AI models. How do we run AI models? In this lesson, we focus on why computing matters when designing an AI system’s architecture.
Let me explain it with an example. An architect develops an AI solution for an autonomous vehicle that must run on the embedded controllers in the electric car. The embedded controller runs on power from the car’s battery. If the embedded controllers get superheated, obviously they draw more power from the battery, resulting in a reduction in the range of the vehicle, which customers never prefer to experience. In addition, if the embedded controllers need additional cooling, you need to design cooling circuits. This is also common in designing the deployment modules of most of today’s AI solutions. This is where computing comes in.
We all know that Neural networks are developed, inspired by our brains’ neurons. Our brains’ neurons get trained second, solve complex problems every day, yet still don’t consume much power. Our most advanced AI models still haven’t reached anything close to the human brain. Brains run trillions of operations per second, yet consume just 20 watts of energy. In contrast, the supercomputers – which are not yet capable of competing with human brains from all perspectives – consume megawatts of electricity. Modern data centres consume electricity to train models and use huge amounts of water for cooling.
This is where researchers proposed neuromorphic computing, which mimics the human brain. The marvel of the human brain is that, while doing these computations, not all parts of the brain are equally active. The brain selectively activates parts of itself to do a task. For example, when we solve a mathematical problem, some parts of the brain become more active. In contrast, when we do creative work, other parts of the brain become more active.
In general, current processors follow either Princeton or Von Neumann architecture. As a result, one large cluster of processors runs continuously and consumes energy. Hence, neuromorphic computing uses multiple processors, unlike the single large processor in most computers today. The one big processor is divided into an array of small processors. Each small processor has its own memory, clock, and power. At any given time, one required processor activates; the others remain in sleep mode. This computational technique is called Neuromorphic Computing.
If I have a big Neural Network and I excite it with some inputs, not all the activation functions in the network work together. Some activate, and some don’t. Now, with a neuromorphic computing-enabled processor, we cannot flash a conventional neural network to the processor. We need to convert it to an SNN (Spiking Neural Network) and then flash it to the Neuromorphic Processor.
How to design an SNN? There are multiple methods for designing and training an SNN. I will explain it with an example. One common approach is to develop a CNN (Convolutional Neural Network) first. First, train a conventional CNN with ReLU activations and then replace each artificial neuron with a spiking neuron whose firing rate approximates the original activation. The ReLU activation depends on firing rates, meaning how many spikes occur in a time frame. The simplest explanation of this process is given below.
- Consider a CNN that has a layer: a_i = ReLU(z_i)
- Replace each ReLU neuron with a spiking neuron.
- During inference, simulate the network for TT time steps.
- When the membrane voltage crosses a threshold, the spiking neuron fires.
- Calculate the total number of firings over a time period T.
- f(RELU) is approximated by the weighted firings over the time period T, known as the firing rate r. Hence, ReLU(z)∝r.
Thus, as an AI Generalist, when designing or reviewing an AI solution’s architecture, you must consider use-case-based computing mechanisms to deliver value to the product’s end users. In the next lesson, we will learn about another trending computational method: Quantum Computing.
Leave a Reply