AI models are black boxes, so how do we ensure their reliability? How do we design AI systems differently compared with others?
When I built my first AI model in 2015, the most well-known library, TensorFlow, was not open source. Google made TensorFlow open source on November 19, 2015. Since then, with the advent of open-source libraries and silicon chips, their use has increased dramatically. Now most AI enthusiasts can create AI models with the help of popular libraries.
With this democratisation, we can now develop applications more easily. These applications used to take years to develop. Nowadays, we can even use SDLC (Software Development Life-Cycle) AI agents to develop applications. This is a great achievement in this technological era. However, this creates some challenges too. How can we be different in this AI age? If machines generate the code (jokingly, another set of code), what should we contribute?
I hear the above questions in my close coffee conversations. In my opinion, we still have immense scope as humans in the field of AI. In the upcoming lessons, I will structurally explain each of these areas from my experience and empower you to scale your knowledge (this is my sole intention for this project and why I started this).
Explainability: Many AI systems fail because no one thinks about their explainability. Take an example of a ball you kicked. If someone asks you to find the speed of the ball, you can apply Newton’s Laws and write some equations. Now you find that the calculations are wrong. To find the mistake, you check Newton’s equations and find the mistake you made in writing the equation. You fix it, and it works with full confidence. Now, if someone writes the same equation with the help of a Neural Network and it does not work, how do we understand what the mistake is inside the Neural Network? The biggest challenge with these AI models is that they are just networks or graphs, which makes them a black box. How do we explain black-box models? Why did this system make this decision?
Observability: How to consistently monitor AI models? Can we see how the AI system is behaving? Here, the concept of HITL (Human-in-the-Loop) comes into play in Agentic AI. HITL is a critical layer in Agentic AI. So the question is: how can we consistently monitor an AI system? This is extremely important in safety-critical applications.
Reliability: Can I depend on the AI system? What ensures that my AI system is reliable? Simple validation results never suffice to prove an AI system’s reliability. When we develop systems (engineering, business or anything else) with mathematics, we know the equations, we know the equations are well proven, and we know their working range. Let me give you an example from Control Theory. In Control Theory, experts mathematically prove that beyond this range, the system becomes uncontrollable, produces unbounded output, and becomes unstable. This is a classic concept used in every engineering field, from automotive to rocket engineering. Can we do so in AI-based systems? How do I know that the AI system is correctly trained? How do I know the data range was appropriate? All these questions lead to the reliability of the AI models.
In the upcoming lessons, I will cover all the above concepts “mathematically”, including more best practices. At the end of this course (I’d rather call it a project), the reader must be able to understand how, mathematically and structurally, we can analyse AI systems to make them successful, scalable AI products. In the next chapter, we will learn how mathematics is used in AI through a common example we all use every day as AI enthusiasts.
Leave a Reply