What is, and isn't, AI?
Generative AI is an absolute triumph of marketing. I feel like, in most people's heads, all AI is generative AI. And not only that, it seems that for some people, anything done by a computer is AI (and therefore anything done by a computer is generative AI).
In this article, I will attempt to explain how not everything a computer does is AI, and not all AI is generative AI, in the hope that people can better understand the technology around us. I believe that AI is a good thing for humanity, but it must be done right.
Introduction
As a first step, I will separate computer algorithms into two types: deterministic and probabilistic.
Deterministic algorithms
In a determinstic algorithm, the output is just a function of the inputs. There is no randomness. Given the same starting condition and the same inputs, a deterministic algorithm will always produce the same output.
As an example, I'm going to talk about the climate control system in a car.
In this system, we start in the top left. The car measures the temperature in the cabin. Then, it compares it to the set temperature - that is, what the driver has set the desired temperature to be. Depending on how different the set and measured temperatures are, the car adjusts the blowers. Then it goes back round and starts again.
We can actually write some pseudocodeFake code that sort of looks like a program but isn't any particular language. It just shows you how the program would be written, step by step. here, to show you what this might look like as a computer program.
MeasuredTemperature = readTempSensor();
SetTemperature = readTempDial();
Error = SetTemperature - MeasuredTemperature();
Output = SetTemperature + (Gain * Error);
I'll go through line by line.
Gain = 2;
This stores the number '2' and names it 'Gain'. That means anywhere you want to use that number, you write Gain instead of 2. This is useful because if you later wanted to change gain to be 3, you only have to update it in one place instead of however many you wrote 2 in.
MeasuredTemperature = readTempSensor();
Going from right to left. Putting the () brackets after the words means it's a function. That is, it goes away and does something. When it comes back with a number, it gives that number to MeasuredTemperature. Just like Gain, any time you want to use that number, you write MeasuredTemperature instead of a number. Then, when the value of MeasuredTemperature changes, everything will be using the updated number without having to go read the temperature sensor every time!
SetTemperature = readTempDial();
Another function. This has a different name so you can see it is going to do something else. The output of this one is stored as SetTemperature. This is pseudocode, so I don't have to actually write how readTempDial(); works; you can guess what it does from the name, and in pseudocode the "how" isn't important.
Error = SetTemperature - MeasuredTemperature();
Now we take those two numbers we stored, SetTemperature and MeasuredTemperature, and we do one minus the other. The result of that calculation is stored as Error, so we can use it later.
Output = SetTemperature + (Gain * Error);
One last bit of maths. The output is the set temperature plus the gain times the error. What? Well, if the set temperature is 20° but the cabin is at 19°, the error will be: Error = 20 - 19 = 1.
Then, the output would be Output = 20 + (2 * 1) = 22. So it would set the blowers to blow 22° air - a bit warmer than it needs to be so that it can get to the right temperature sooner.
And if the temperature is too high? If it was measured as 21°, the error would be -1 (20 - 21 = -1). The output would be 20 + (2 * -1), which is 18°.
There's no randomness built into this algorithm. Given the same set temperature and the same measured temperature, the output will always be the same.
Probabilistic algorithms
For a probabilistic algorithm, the output is not determined (but it can be influenced) by the inputs. A probabilistic algorithm may produce different outputs even when it is given the same starting conditions.
This one is a bit harder to find a real-world example for, so this is a bit more thinky. Let's say you need an algorithm to find the lowest point on this graph.
Humans are actually incredibly good at this sort of thing. It's obvious where the low bit is, isn't it? Unfortunately, computers are much much less good at this.
One way for a computer to find it would be to look at every single point individually. Once it has looked at the whole graph, it'll report back with where the lowest number was. This could be extremely slow if you had a really big graph.
To speed it up, you could start by picking a couple of random points on the line. Which one of those is the lowest? Focus your search around that one — do another couple of random points near there. It's reasonable to assume the lowest point is near to a low point, not near to a high point. Do that a couple of times and when the 'best' you find stops getting any better, you can be happy you found the minimum.
Compare these two graphs. Look how many points were tested: An exhaustive search needed to look at about 40 points. The smarter search only looked at 11—it would only take a quarter of the time to find the answer.
However, this process is random. There's probability involved. The locations of the four starting points is random, and the probability (likelihood) of it picking a point is the same as it picking any other point. We can run the same experiment on the same line twice and get two different answers because things have been left to chance.
The starting point is random and so the output is sort of random, but the algorithm is steered by things so it isn't completely random.
Pros and cons of deterministic and probabilistic algorithms
Sometimes a deterministic algorithm is a good idea. With the climate control example, you want your heater to act in the same way every time. A probabilistic algorithm has a very small chance (but it's still a chance!) of turning on the air con when you get in your car in the middle of winter and its -5 outside. It also takes longer to run than a simple algorithm that just measures the temperature, calculates the error and adjusts the heater.
Sometimes a probabilistic algorithm is a better idea. This tends to be when you will accept an answer that is good enough, even if it is not perfect. In the example of searching for the lowest point on the graph, the probabilistic version runs much much faster, even though it takes more code to program it.
How do computers think?
(This section is a slight oversimplification, but not as much of an oversimplification as you might think).
Computers are, if you look closely enough at them, incredibly stupid. In the end, basically every program you can think of comes down to a bit of simple maths and some comparisons. Add and multiply some numbers together and if the result is bigger than a threshold, do something, otherwise do something else.
Zoom in enough and a computer is made out of transistors. We're going to ignore that level and zoom out just a little bit. Those transistors are arranged into things called logic gates.
Logic, because they run on Boolean logic (that's a whole separate article. For now I'll link to wikipedia)
Gate, because that's the old name for an electronics switch.
Imagine a corridor full of engineers. At the end of the corridor is two doors. Engineers with beards go to the left door, engineers without go to the right. (It's a roughly 50:50 split.) Someone assesses each engineer's beardiness and moves a gate side to side to corral the engineers through the correct door.
The person operating the gate is using logic. We can write some pseudocodeLooks like code but isn't a particular programming language—just says how the program should roughly work for this task.
if (engineer.beard = true)
openLeftDoor();
else if (engineer.beard = false)
openRightDoor();
end if
goto start
So a logic gate is a switch that switches between one thing and another—in this case, they switch between the output being 1 and the output being 0—based on some mathematical logic.
There are a couple of different types of logic gate—AND, OR, XOR and NOT are the main ones. For example, this is an AND gate. The output is 1 if both the inputs are 1 (ie., if input one AND input two are 1). Otherwise, the output is 0. There is no room for questioning—if both inputs are 1 the output IS 1.
This is a truth table, which comes up a lot in Boolean algebra and logic. The first two columns list all the possible combinations of inputs you can have. The third column says what the output would be for each one of those combinations.
| x1 | x2 | y |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 0 |
| 1 | 0 | 0 |
| 1 | 1 | 1 |
While one logic gate is a very simple thing, put enough of them together and you can do complicated things. For example, this circuit adds two numbers together, as long as the numbers are between 0 and 3.
That's pretty limited functionality, let's be honest, but the circuit has already got a bit complicated. A computer can add together much much bigger numbers, so imagine how complex the layout must be inside them!
However, no matter how many of these logic gates you stick together, it's all still binary. All the way through the circuit, from the start to the end, the answer is always 1 or 0. YES or NO. On or off. There's no variation, no "eeeeeghhhhh sort of".
If we want to do something clever though, we need to have options other than yes or no. We need degrees of maybe.
Fuzzy logic
The proper name for "degrees of maybe" is "fuzzy logic"! 0 and 1 are all hard and defined, but fuzzy logic makes use of all the grey, fluffy maybe in between. Fuzzy logic is the basis of machine learning, computational intelligence and AI.
The circuits used to run fuzzy logic aren't the simple logic gates we just showed. They're called neural nets, and each calculation happens in a neuron—names stolen from biology and squishy brains.
A neural net has inputs and outputs, and a "hidden layer" in between. The hidden layer is where all the magic happens. In that layer, the inputs are weighted and added together and mixed around before they get to the output. Here's an example of a neural network that takes four sandwich ingredients and suggests four more.
The inputs (your sandwich ingredients) are at the top. The outputs (the neural net's suggested additions) are at the bottom. The hidden layer in between does some unknowable mixing together of those ingredients to take your ingredients and come up with four more.
In the first example we've started with a ham and lettuce sandwich. It's recommended we add some butter, cheese, tomato and mayo. Simple, classic, I like it.
In the second example we've gone for a PBJ sandwichI'm British. I had to do research for this bit.. The suggestions are mostly sweet things, but apparently a slice of cheese might be a good addition too.
In the third example, the input only has one slice of bread. The neural net has suggested we add another. SmartThe computer couldn't have come up with bread on its own though. Whoever made this neural net will have had to think in advance that 'more bread' could be a good idea..
What's going on in the hidden layer then? In the first one, it sees ham and lettuce and two slices of bread, which means the outputs of cheese and tomato are getting very strong maybes. The honey and banana outputs are getting strongly downvoted! In the second, pb and j have made the cheese output get approved, but the mayo output is not popular now. In the third, the lack of bread on the input side means the output side is heavily weighted towards more bread. Perhaps it would be more inclined to recommend an open sandwich if we hadn't also included a dollop of Branston—you need another slice of bread to keep that in place.
The upvotes and downvotes can be considered to be weights. A big weight means this thing is important and should be considered. A small weight makes it less important, but it might still be enough to tip a decision.
We can represent the AND gate from earlier as a neural net (ok, actually just one neuron), but now the inputs and outputs are just 0 or 1, not the infinite scope of sandwich ingredients.
Each neuron is modelled as a simple maths equation:
Output is 1 if y >greater than 0
Output is 0 if y ≤less than or equal to 0
In that equation, x1 and x2 are the two inputs. W1 and W2 are the weights for those inputs respectively. b is the bias - like a fixed offset.
If we put the right weights into that equation, we can make it act like an AND gate. The correct weights for an and gate are: both weights are 1 and the bias is -1.
Lets try a few different cases here to demonstrate it.
Case 1: Both inputs are 0.
W1 x1 + W2
x2 + b = y
(1)(0) + (1)(0) + (-1) = -1
The rule said if the answer is less than 0, the output is zero. For an AND gate, the output is 0 if both inputs are 0, so this is correct!
Case 2: One input is 1 and the other is 0.
W1 x1 + W2
x2 + b = y
(1)(0) + (1)(1) + (-1) = 0
The rule said if the answer is less than or equal to 0, the output is zero. For an AND gate, the output is 0 even if one input is 1, so this is correct!
W1 x1 + W2
x2 + b = y
(1)(1) + (1)(1) + (-1) = 1
The rule said if the answer is greater than zero, the output is one. One is greater than zero. For an AND gate, the output is 1 when both inputs are 1, so this is correct!
| x1 | x2 | W1 | W2 | b | W1x1 | W2x2 | y | Expected output | Actual output |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 1 | 1 | -1 | 0 | 0 | -1 | 0 | 0 |
| 0 | 1 | 0 | 1 | 0 | 0 | 0 | |||
| 1 | 0 | 1 | 0 | 0 | 0 | 0 | |||
| 1 | 1 | 1 | 1 | 1 | 1 | 1 |
So you can see that for all of the rows, the expected and actual outputs are the same. However, with all our weights being 1 (and the bias is -1, which is still a sort of 1), it's still a bit black and white and not very fuzzy.
Lets stop this thing working as an AND gate. What if we set W1 to 0.6, W2 to 0.4 and b to -0.2.
| x1 | x2 | W1 | W2 | b | W1x1 | W2x2 | y | Output |
|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0.6 | 0.4 | -0.2 | 0 | 0 | -0.2 | 0 |
| 0 | 1 | 0 | 0.4 | 0.2 | 1 | |||
| 1 | 0 | 0.6 | 0 | 0.4 | 1 | |||
| 1 | 1 | 0.6 | 0.4 | 0.8 | 1 |
Notice that the inputs, x1 and x2, are still only able to be 1 or 0. The output also gets "de-fuzzified"This is the actual term so it's still 1 or 0 by the end. What if we had stuck another one of these neurons before this one? We connect the y outputs of the first neurons, before they get de-fuzzified, to the x inputs. Then the inputs could be -0.2, 0.2, 0.4 or 0.8. Now we've got a whole range of different things the output could be!
| a | b | c | d | y1 | y2 | y | Output |
|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | -0.2 | -0.2 | -0.4 | 0 |
| 0 | 0 | 0 | 1 | -0.2 | 0.2 | -0.24 | 0 |
| 0 | 0 | 1 | 0 | -0.2 | 0.4 | -0.16 | 0 |
| 0 | 0 | 1 | 1 | -0.2 | 0.8 | 0 | 0 |
| 0 | 1 | 0 | 0 | 0.2 | -0.2 | -0.16 | 0 |
| 0 | 1 | 0 | 1 | 0.2 | 0.2 | 0 | 0 |
| 0 | 1 | 1 | 0 | 0.2 | 0.4 | 0.08 | 1 |
| 0 | 1 | 1 | 1 | 0.2 | 0.8 | 0.24 | 1 |
| 1 | 0 | 0 | 0 | 0.4 | -0.2 | -0.04 | 0 |
| 1 | 0 | 0 | 1 | 0.4 | 0.2 | 0.12 | 1 |
| 1 | 0 | 1 | 0 | 0.4 | 0.4 | 0.2 | 1 |
| 1 | 0 | 1 | 1 | 0.4 | 0.8 | 0.36 | 1 |
| 1 | 1 | 0 | 0 | 0.8 | -0.2 | 0.2 | 1 |
| 1 | 1 | 0 | 1 | 0.8 | 0.2 | 0.36 | 1 |
| 1 | 1 | 1 | 0 | 0.8 | 0.4 | 0.44 | 1 |
| 1 | 1 | 1 | 1 | 0.8 | 0.8 | 0.6 | 1 |
Where's the randomness, though? I said this type of algorithm is probabilistic, but in this situation we still have a clear divide: some outputs are above zero, and some are below.
The output of your neural net could use a solid threshold like the example I gave: if y >Greater than 0, output = 1; if y ≤Less than or equal to 0, output = 0. Or you could have a random number generator choose. Perhaps if your value of y is ≥Greater than or equal to 1, the output is 1; if y ≤less than or equal to 0, output = 0. But if 0 >Greater than y >Greater than 1, generate a random number between 0 and 1. If the number is less than y, y = 1 (so a higher value of y is more likely to turn into a 1).
Another interesting thing here is that the inputs can be negative now. In some situations, the second neuron had -0.2 on one of its inputs. It makes a 1 even less likely by working against the other positive numbers. Or, to put it another way, if you made a ham and peanut butter sandwich, the ham works against the peanut butter and makes banana a less likely suggestion than if you just had peanut butter on its own.
In my example of the climate controller, I had a number called Gain and I set it to 2. That "gain" number is like the weights in the neurons. If I was an engineer designing the climate control in a car, I'd have to sit there and try different numbers until my control system got the car to the right temperature quickly without upsetting the humans by blasting them with 50° air or something. It takes a bit of tweaking to get each number right.
In this example, we now have six weights, and three biases too. I set a lot of them to be the same thing in my example but in reality it wouldn't be like that. An engineer designing this would have to twiddle each number in turn to get it all right. And then maybe when you'd got the weights roughly right, you'd have to go and tweak them all again because some of the new numbers will have made the older numbers wrong again. What a pain.
This is the point where neural nets get really exciting and interesting. We don't set those numbers ourselves. We let a computer come up with them for us.
How do you configure a neural net?
Neural nets are ideal when you have an idea of what output you want, but you don't quite know how to achieve it.
As an example, I have guinea pigs. If you give them a pile of hay, they will dig into it and very carefully pull out the exact piece of hay they wanted. I want to be able to give them their favourite bits of hay, but I am a mere human and I can't tell what makes a piece of hay good or bad. This is the perfect job for a neural net: I know my output (sort out the good hay) but I don't know how to get there (what makes it a good piece of hay).
My first step is to define what my neural net is going to look like. I'll give it a whole bunch of inputs: what colour is the hay? Is it stalky or soft? Short or long? Wide or narrow? I give it all the data I think it needs.
I also need an output. In this case it's just a yes or no: good hay or bad?
In between the inputs and outputs, we have the hidden layer. I can set up how many neurons I want, but I don't assign them weights and biases just yet. I'm going to assume this is a really complicated neural net with many layers and interconnections because guinea pigs are really smart creatures and so there must be a lot of thought being put into choosing the best hay. There may be some sarcasm here.
Now we have our rough structure. How do we configure the weights? We train the net!
To train it, you need training data. For my example, I would get my guinea pigs to sort the hay into good and bad piles (without eating all the good bits). I'd give my neural net some random starting numbers for weights and biases as a starting point, and then I'd let it classify all the bits of hay.
Then, we'd do it again, with different random starting numbers. And again and again.
Lets say we did it ten times. We can then look at all the bits of hay it sorted and give it points every time it got one right, and take away points every time it got one wrong.
We started with random numbers, so it's probably going to be pretty random if it gets a piece of hay right or not. But also, some of them will have randomly done a bit better than the others. Now we're going to start breeding! (my guinea pigs are neutered but my neural nets are not).
Just like breeding real animals, you can take two parents, mix up their genes a bit and get some offspring. Also like real life, the weak ones will die off without making offspring (sad face) And also, just like in real life, sometimes you get random mutations and changes. So these are all things that happen between generations when you're training a neural net.
In our first generation, lets say versions 5 and 7 did surprisingly well. 2 and 3 did really badly. The rest were in the middle. So we'll do some breeding between the good ones, maybe try to mix 5 and 7 with the each other? And the others? But also keep a copy of them, for reference. All of these mixes also have the chance of spontaneous mutations—small changes that don't come from either parent—too. The exact way they're all mixed is up to the engineer who is making the neural net, but at the end of it we've ended up with another generation of possible weights.
Then, you run the test again for each one. Hopefully they've got better than the previous generation! But maybe not. Then you do the breeding again. And test it again. Mix them again. Test some more. And so on and on...
Eventually, if everything goes to plan, you'll end up with a set of weights that works. It classifies all the good hay as good and all the bad hay as bad. Hurrah! Now you can take your hay classifier and use it to sort new hay—hay that wasn't part of the original training data—and it should get that right too.
This stage, where you have a trained neural net and you're using it to do its job, is referred to as inference.
(I am not going to discuss bias, overfitting, any of those problems here.)
Remember earlier I said that neural nets have inputs, outputs and hidden layers? Notice that all the weights and biases go in the hidden layer. The trainer, and the user, of the neural net don't really care what the hidden layer looks like and how it all connects up. It's almost certainly very complicated and not worth the time to understand, as long as your network works. It may be that you can't even find out what it looks like, depending on how it was made.
How does an LLM work?
An LLM is a Large Language Model, which is generative AI. It is an absolutely vast neural net, effectively, but runs on the same principles.
It has been trained on as much human-created text as the companies could find. This was put into the neural net, and the LLM would produce some more text as an output. Someone then looked at the text and said "well that's absolute garbage" or similar. The human marked the bits that looked like text and gave it a good score for that, and gave it a bad score for the gibberish. Then they did it again and again and again and again, until you got something that can produce strings of text that appear intelligent.
What's the difference between generative AI and 'real' AI?
There isn't one, really. It's all neural nets when you dig into it. The difference is that generative AI was trained on stolen data by people working in appalling conditions and now is destroying the planet and people's livelihoods, just to make a few obscenely rich people even more obscenely rich.
The thing to be aware of is that the vast majority of the time the news reports on something using AI to solve a problem, it probably isn't generative AI. It will be a custom-trained model to solve their particular problem.
Think about it: my hay-sorting neural net has been trained on lots of hay. I couldn't use the same neural net, with all the same weights, to suggest sandwich fillings; it would be complaining there aren't enough stalky bits in your cheese toastie. It also doesn't understand that it is sorting hay; it just sees a string of numbers (that represent a green bit that's quite wide, but it doesn't know that) and the weights say that is a good string of numbers, even though it has less good strings of numbers too (that represent their housemate not trying to eat itIf your housemate is eating it, that makes it the best piece of hay and you must have it RIGHT NOW.).
In the same way, an LLM does not understand human language. It is a huge network that takes strings of letters and spits out another string of letters that are likely to be related. You can try to use an LLM to classify the best carrot in the veg drawer, but it will just throw words related to orange root vegetables at you. You're much better off making a custom neural net that's designed for the job—and it won't fund fascism either.
References
Neural representation of AND, OR, NOT, XOR and XNOR logic gates
I also highly recommend you read You look like a thing and I love you by Janelle Shane.