Unit 7 · Machine Learning and Artificial Intelligence
7-2 Neural Networks and Deep Learning
How can a machine tell a dog from a cat, or turn speech into text? The answer, loosely inspired by the brain, is a neural network: layers of simple units called neurons, each passing signals to the next. A neuron just multiplies its inputs by weights and adds them up — nothing complex on its own — but stack enough of them across enough layers and the network learns, by adjusting those weights on data, to recognize patterns no human could write rules for. Making the stack deep — deep learning — lets the network build up features from simple edges up to whole objects.
Guiding question
How is a neural network built and trained, and what does making it “deep” add?
Learning outcome. Describe the structure of a neural network and explain how a single neuron processes its inputs, and explain what deep learning is and why stacking many hidden layers brought a leap in AI performance.
1 · Nodes and layers
A neural network has a structure in which many nodes — also called neurons — are connected. Nodes are arranged together in layers, and the nodes of adjacent layers are connected to each other.
2 · Three kinds of layer
| Type of layer | Role |
|---|---|
| Input layer | Receives data from outside |
| Hidden layer (middle layer) | Processes the input information and extracts features |
| Output layer | Outputs the result of processing as an answer |
3 · And the data flows one way
Data enters at the input layer, is processed as it passes through the hidden layers, and reaches the output layer. In a network that judges whether the animal in a photograph is a dog or a cat, the input layer receives the pixel information, the hidden layers extract features such as shape and pattern, and the output layer outputs the judgment “dog” or “cat”.
The number of hidden layers and the number of nodes in each are designed according to the difficulty of the problem. The more complex the problem, the more layers and nodes are needed.
1 · The whole mechanism, in one sentence
Each neuron receives multiple inputs, multiplies them by weights and sums them, then passes the sum through an activation function, which lets the network learn relationships that are not simply proportional, before passing the result to the next layer.
2 · What a weight is
A weight is a numerical value that represents how much each input is emphasised. Weights can be negative as well as positive, and the larger a weight is in size, the more strongly that input influences the neuron’s result.
Do the multiplication yourself
One neuron, deciding whether an email looks like spam. Switch each input on or off, and drag its weight — how much that input is emphasised.
0.9 + 0.6 + 0.8 − 1.2 = 1.1
The sum 1.1 goes through the activation function, which squashes any number into the range 0 to 1. Out comes 0.75 — the neuron’s answer, passed on to the next layer. Above 0.5, so this neuron is leaning towards “spam”.
And a weight can be negative. “You replied to it before” pulls the total down — it is evidence against spam. The book says weights can be negative as well as positive, and this is what that buys you: a neuron can weigh evidence both ways.
3 · Training
Training is the process of gradually adjusting the weights between neurons using a large amount of data.
| Stage | What happens |
|---|---|
| Before training | The weights are set to arbitrary values, and correct predictions cannot be made |
| During training | Data is given, and the weights are gradually adjusted based on the difference between the prediction and the correct answer |
| After training | With the adjusted weights, correct predictions can be made even on new data |
Watch the weights be adjusted, and nothing else
Four emails whose answers are known. The weights begin at arbitrary values — nobody chose them for being good — so the predictions start out wrong.
| Correct answer | What it predicts now | |
|---|---|---|
| [1, 1, 1, 0] | 1 | 0.43 |
| [0, 0, 0, 1] | 0 | 0.62 |
| [1, 0, 1, 0] | 1 | 0.35 |
| [0, 1, 0, 1] | 0 | 0.69 |
Average error after 0 rounds of training: 0.4026
Before training. The weights are arbitrary values and correct predictions cannot be made — look at the last column against the one before it.
4 · Simple parts, complex whole
As training progresses, a combination of weights that allows the network as a whole to produce correct answers is found. Even though the function of each individual neuron is simple, complex judgments become possible by combining many neurons.
Before any of the words, watch the arithmetic happen. One number at a time, so you can see exactly which input caused which change.
Step 1 of 7 — What arrives — the inputs
Four clues about one email. Each is answered 1 for yes or 0 for no. Here all four are yes, so every input is 1. This is the only thing a neuron ever receives: a list of numbers. It never sees the email.
Two things are worth taking from this. First, the running total fell when the fourth clue arrived — that is what a negative weight is, and it is why the book says weights can be negative as well as positive. Second, the activation curve flattens at both ends: a total of 1.1 gives 0.75, but a total of 5 gives only 0.99. It can get more confident, never certain.
The inputs (1, 1, 1, 1). Answers to four yes/no questions about one email. 1 means yes, 0 means no. This is all a neuron ever receives — it never sees the email itself.
The weights (0.9, 0.6, 0.8, −1.2). How much each clue is emphasised. Nobody chose these; training produced them. The fourth is negative, which is why the total fell when it arrived.
The total (1.1). All four clues collapsed into one number — the only place where they get to argue with each other.
The output (0.75). The total after squashing, read as “about 75% confident”. Above 0.5, so this neuron leans towards “spam”.
After the neuron adds everything up it has one number, and that number could be anything — −40, 0.3, 900. That is awkward for two reasons, and the activation function fixes both.
First, it squashes the total into a tidy range (here, between 0 and 1) so it can be read as how confident the neuron is. Second — and this is the one the book means by “not simply proportional” — it puts a bend in the maths. Without a bend, stacking layers would be pointless: adding and multiplying, then adding and multiplying again, is still just one big add-and-multiply. A hundred layers would do no more than one. The activation function is what makes depth worth having.
The book’s wording, which is what an exam wants: …passes the sum through an activation function, which lets the network learn relationships that are not simply proportional. (p.70)
Where the comparison breaks down: A dimmer is set by a person; an activation function is a fixed piece of maths chosen by the engineer and never changed by training. The things that do change during training are the weights — as the book’s table on this page says.
🧮 One neuron, worked by hand — the four inputs shown in the slider above
- 1
Write down the four inputs. All four clues are present here, so: 1, 1, 1, 1.
Why this step: A neuron cannot see an email, a photograph or a word — only numbers. Whatever the real thing is, it arrives at the neuron already turned into a list of numbers. This is the same first step as the spam filter in lesson 1-2.
- 2
Multiply each input by its weight: 1 × 0.9, 1 × 0.6, 1 × 0.8, 1 × (−1.2)
Why this step: The weight is “how much this input is emphasised” — the book’s own words on page 70. Multiplying is what turns “this input is present” into “this input counts this much”. An input of 0 contributes nothing no matter how large its weight, which is why turning a switch off in the demonstration above makes that whole term vanish.
Most people slip here: The fourth weight is negative. A negative weight does not mean the input is unimportant — it means the input is evidence pointing the OTHER way, and a large negative weight is a strong argument against.
- 3
Add the four results: 0.9 + 0.6 + 0.8 − 1.2 = 1.1
Why this step: Adding collapses four separate pieces of evidence into one total, and it is the only place where the inputs get to argue against each other. The −1.2 cancels most of what the first two contributed.
Most people slip here: Add the signed values. A very common slip is to add 1.2 instead of subtracting it, which here would give 3.5 instead of 1.1 — and 3.5 would come out as almost total certainty.
- 4
Pass the total through the activation function: 1 ÷ (1 + e^(−1.1)) = 0.75
Why this step: This is the step everyone skips, and it is the one the exam asks about. The raw total of 1.1 is on no particular scale — is 1.1 a lot? The activation function squashes any total into 0…1 so it can be read as a confidence. It also bends the maths, which is what allows the network to learn relationships that are not simply proportional, and therefore what makes stacking layers worthwhile at all.
Most people slip here: This is not rounding and not a percentage conversion. Feed in a much bigger total, say 5, and the answer creeps to 0.99 without ever passing 1 — the squashing gets harder the further out you go.
- 5
Read the answer: 0.75, which is above the halfway mark of 0.5.
Why this step: The neuron’s output is a strength, not a decision. Turning 0.75 into “yes” needs a threshold, and that line is chosen by a person — exactly as it was for the spam filter in lesson 1-2.
That is worth holding onto, because it is the honest answer to two exam favourites: why an AI needs large-scale data (there are that many numbers to pin down), and why its decisions can be hard to explain (the reason for any answer is spread across all of them at once).
Figures published by the developers of these systems; the exact counts differ between versions.
1 · What “deep” means
Deep learning is a technology that uses neural networks with many hidden layers stacked deeply. “Deep” refers to having many hidden layers — and nothing else.
2 · Features built up step by step
When the hidden layers are made deeper, features of data can be built up step by step and captured. In an image-recognition network, features are built from shallow to deep layers.
Climb the layers one at a time
Deep learning stacks many hidden layers. Here is what each depth learns to notice, in an image-recognition network — and each one is built out of the one before it.
The input image goes in. Before pressing: what do you think the first layer of the network is able to notice about it?
3 · What that makes possible
Through this hierarchical building of features, deep learning can recognize complex patterns with high accuracy.
4 · And what had to arrive first
Deep learning became practical as a result of large amounts of data and high computational power becoming available — in particular specialized processors well suited to AI computation, such as GPUs and TPUs. Before that, it was technically difficult to advance learning sufficiently even when layers were made deeper.
1 · Accuracy that had not been reachable
With deep learning, accuracy close to or surpassing humans has become achievable in many tasks that were previously considered difficult for computers.
2 · Where the progress has been greatest
| Field | Examples |
|---|---|
| Image recognition | Face recognition systems, diagnostic support for medical images, object detection for self-driving cars |
| Speech recognition | Voice operation of smart speakers, transcription from speech, speech translation |
| Natural language processing | Machine translation, summarization of text, answering questions |
3 · The change that mattered most
Before deep learning, features had to be designed by humans one by one and given to the computer. With deep learning, features can be learned automatically from data, allowing the limitations of human design to be exceeded.
As a result of accumulating these capabilities, the foundation has been laid for AI that newly creates text and images — which is where the next lesson begins.
How the real paper is built
The ministry sets half multiple choice and half written answer in every subject. The two halves are not the same skill. In the multiple-choice half you have to spot the wrong options quickly — and the wrong options are built out of the mistakes students really make. In the written half nobody gives you the answer to recognise; you have to produce it, in your own words, with a reason.
Practise both. Being good at one half and weak at the other costs you exactly half the paper.
- node (neuron)
- — a single processing unit.
- layer
- — a group of neurons at one stage.
- input layer
- — receives data from outside.
- hidden layer
- — processes information and extracts features.
- output layer
- — outputs the result as an answer.
- weight
- — how much an input is emphasised; can be negative.
- activation function
- — lets the network learn non-proportional relationships.
- training
- — adjusting weights from data.
- deep learning
- — neural networks with many hidden layers.
- training data
- — the labelled examples a network learns from.
- generalize
- — work correctly on new, unseen data.
Worked Example
(1) Mark each statement with ○ if it is true or × if it is false.
A A neural network has three types of layers: input layer, hidden layer, and output layer.
B The input layer is the layer that outputs the results of processing as an answer to the outside.
C Deep learning is a technology that uses a neural network with only one hidden layer.
D Training a neural network is the process of adjusting the weights between neurons using a large amount of data.
(2) Match each description with a term: A Hidden layer, B Weight, C Input layer.
a In a neural network, the layer that receives data from outside
b In a neural network, a numerical value that represents how much each input is emphasised
c In a neural network, the layer that processes input information and extracts features
1 · Answer the following questions.
2(1) · Which most appropriately describes the layers of a neural network?
Mark ○ for the correct description, and × for the ones that swap the roles.
The input layer has the role of outputting processing results to the outside.
The hidden layer has the role of receiving data from outside.
The output layer has the role of processing input information and extracting features.
Data enters at the input layer, is processed as it passes through the hidden layers, and reaches the output layer in this flow.
2(2) · Which most appropriately describes the training of a neural network?
Mark ○ for the correct description, and × for the others.
From before training begins, the weights between neurons are set to correct values.
As training progresses, a combination of weights that allows the network as a whole to produce correct answers is found.
Training is performed by increasing the number of neurons.
Once training is complete, the network becomes unable to handle new data.
2(3) · Which is NOT an appropriate characteristic of deep learning?
Mark ○ for the real characteristics, and × for the one that is not.
It has a structure with many hidden layers stacked deeply.
It can build up features of data step by step.
The deeper the hidden layers, the more complex features can be captured.
Even if the number of hidden layers is increased, the difficulty of problems that can be handled does not change.
2(4) · Put these in the order features are extracted, from shallow layers to deep layers.
Match each description with the choice that fits it.
Simple features such as lines and changes in colour
Shapes of parts such as eyes, nose and ears
Structure of the whole face or object
Think as an Engineer — Investigate & Decide
A team wants an app that reads handwritten digits 0–9 from scanned school registers and turns them into typed numbers. You advise on building it with a deep learning network.
1 · Design the layers.
Say what the input layer would receive, what the output layer would produce, and in one line why hidden layers — rather than a single step — are needed here.2 · Explain the depth.
Using the idea of hierarchical features, describe what shallow, middle and deep layers might each capture for a handwritten digit — a chain the lesson shows for faces but does not spell out for digits, so you must map it yourself.3 · Judge the cost and risk.
Name the two things this network needs to become practical, and name one way the training data could be too narrow, making the app fail on some students’ handwriting. In pairs, compare designs and agree on the one dataset change that would most improve fairness.
In a New Context
A voice assistant turns spoken commands into text and often works well even for words it was not explicitly programmed with. Name which field of AI progress this belongs to, and explain — using layers and training — how it can handle speech nobody wrote rules for.
Lesson Question — Answered
A neural network is built from layers of connected neurons: an input layer that receives data, hidden layers that process it and extract features, and an output layer that gives the answer, with data flowing input → hidden → output.
Each neuron is simple — it multiplies its inputs by weights, sums them, and passes the sum through an activation function — and the network is trained by gradually adjusting those weights on large amounts of data. Before training, arbitrary weights give wrong answers; after training, tuned weights give correct ones even on new data.
Making the network deep — many hidden layers — lets features be built hierarchically, from simple edges through parts to whole objects, so complex patterns are recognized with high accuracy. Once large data and powerful processors like GPUs made it practical, deep learning let features be learned automatically instead of designed by hand, driving leaps in image recognition, speech recognition and natural language processing.
2 · Read the passage and fill in the blanks (a)–(h).
A neural network has a structure in which many processing units called ( a ) are connected. These are arranged together in ( b ), and are divided into three types: ( c ), which receives data from outside; ( d ), which processes information and extracts features; and ( e ), which outputs the result as an answer. Each ( a ) multiplies its inputs by ( f ), sums them, passes the sum through an activation function, and passes the result to the next layer. Training gradually adjusts ( f ) using large amounts of data. The technology that uses neural networks with many hidden layers stacked deeply is called ( g ), and its characteristic is that it can build up features of data ( h ) and capture them.
3(1) · Which most appropriately describes how a neuron works?
Mark ○ for the correct description, and × for the others.
Each neuron passes one input as it is to the next layer.
Each neuron multiplies its inputs by weights, sums them, passes the sum through an activation function, and passes the result on.
Each neuron averages the input values and passes the result to the next layer.
Each neuron passes only the largest input to the next layer.
3(2) · Which most appropriately describes the background to deep learning becoming practical?
Mark ○ for the correct statement, and × for the ones that get the history backwards.
Computation became simpler by reducing the number of hidden layers to one.
Large amounts of data and high computational power became available.
Connections between neurons were eliminated, simplifying processing.
A method in which humans set weights one by one by hand was established.
3(3) · Which most appropriately describes why deep learning brought a leap in AI?
Mark ○ for the correct statement, and × for the ones that claim too much.
Features could be learned automatically from data without humans designing them one by one.
Computers became able to think with the same mechanism as the human brain.
Computers became able to fully understand the meaning of words.
Computers became able to make completely accurate judgments on any problem.
Key takeaway
A neural network is layers of neurons (input → hidden → output). Each neuron weights and sums its inputs, and training tunes those weights on data. Deep learning stacks many hidden layers to build features hierarchically — edges → parts → wholes — enabled by big data and GPUs.
One of you drag a weight all the way to −2 while the other watches the running total. Then say, in your own words, what a negative weight means about that piece of evidence. If you can say it without using the word “negative”, you have understood it.
Sign In to Keep Going
You've reached the end of the free preview. Sign in — and upgrade to Plus or Pro — to keep going with Programming and Artificial Intelligence.
Full Access
Unlock all 30 lessons