How can we be more than just an AI application developer? How can we develop customised systems or prove the correctness of AI systems? Mathematics gives assurance that an AI System is reliable and explains why it behaves as it does.
I will explain this with an example. I assume the readers of this course are familiar with the basics of AI (if you are not, let me know in the comments) and Python scripting. Whoever has basic knowledge of AI knows about backpropagation and gradient descent.
Let’s do some hands-on now.
Hands-on Assignment
Step 1: Go to Google Colab and create a new Notebook. Insert a Code block and just print “Hello World!” in Python.
Step 2: Next, you can ask Copilot/ChatGPT/any other tool you use to generate the script with the following prompt:
Generate a Python script for neural network training using scikit-learn and a standard dataset. I want to experiment by changing the learning parameter. So the user sets the list of desired learning parameters. Finally, generate a plot showing the learning trajectory for each parameter for comparison. I want to compare, for each learning parameter, how fast and how well the model converges.
Step 3: After generating the complete script, copy and paste it into the code block. Before that, install libraries with the command:
!pip install ....
Step 4: Change the “learning rate” from 0.001 to 1 and see how many iterations the model takes to converge and what its residual error is. You should see something like the following image. Later, set the learning rate to 10 and comment on what you observe.
If you refer to course materials, books, or forums, you will commonly hear that the lower the learning rate, the better the fit, but a lower learning rate takes much longer to reach the converged solution. So they suggest setting a value that is neither too low nor too high. They suggest doing “trial and error” to find the best value for you. If a developer does trial and error, intuitively the developer is doing something called metaheuristic optimisation. You are sure that you found a better solution, but you are unsure that you found the best solution.
A mathematical way to interpret the above simulations
Now let us see if we can understand this mathematically. How can mathematics assure us that we did the right thing?
Let us do some recap. You know that gradient descent is the basic algorithm for training a Neural Network. It uses a cost function (derived from the Neural Network using Calculus), and gradient descent follows the contours of the cost function to find the best solution (the optimal solution). As we all know, the gradient descent algorithm has a tuning parameter called the learning rate. The accuracy of optimisation depends heavily on this learning rate, along with other parameters. When we train a Neural Network, we generally set the learning rate very low to achieve a highly accurate, converged solution/optimisation. On the other hand, if we want to train Neural Networks quickly, we keep the learning rate a little higher. Thus, the developer sets the training parameter intuitively in most cases, and the Neural Network’s states (parameters) are updated according to the following optimisation equation.
(here, x is the state)
Here is the beauty of mathematics from Control Engineering that describes system dynamics. In this case, one can treat the gradient descent algorithm’s steps (every new state) as a dynamical system. Mathematically, if the dynamical system is stable, the system will converge to a solution. When this dynamical system is analysed with Lyapunov’s theory (a well-known mathematical theory to analyse the stability of non-linear systems), the upper limit of the learning rate is found, with which the dynamical system can asymptotically converge:
“r” is the learning rate
“L” is the smoothness constant, which says:
1. The gradient of the objective function cannot change faster than a factor of “L”.
2. “L” is like a “maximum curvature” of the objective function we are trying to minimise.
3. The larger “L” is, the more curved the objective function is, and the smaller the learning rate must be.
Intuitively, if there are two different Neural Networks, or if the datasets of the same Neural Network are different, their gradients will differ in each case, and so will the learning rate. Thus, every AI Generalist must understand the AI system mathematically. Once you think beyond libraries and trust aggregated validation results, you can develop a scalable and robust AI system.
In the next chapter, I will explain another mathematical concept step by step from a well-known transformer model.
Learning Materials
The mathematics above may be challenging to grasp on first reading for those outside the Control Engineering domain. However, readers can read standard books on linear algebra and probability and gradually grasp the mathematics behind AI Models. Two books I would like to recommend for this:
- Linear Algebra and Its Applications by Gilbert Strang
- Probability, Statistics and Random Processes by T Veerarajan
Leave a Reply