Home / Summaries / COMP 3711 / network-accuracy-neural

Learning Goal - Using Neural Nets to Recognize Handwritten Digits

Q: Describe the architecture of the three-layer network used for MNIST digit classification.

- Input layer : 784 neurons — one per pixel in a 28 x 28 grayscale image, with pixel intensity ∈ [0,1] - Hidden layer : n neurons (experiments: 30 or 100) - Output layer : 10 neurons — one per digit (0–9); the predicted digit is the one with the highest activation

Q: How are weights and biases initialised in the network, and why?

Weights and biases are initialised randomly using Gaussian distributions (mean 0, standard deviation 1). Random initialisation is necessary because: Initialising everything to 0 would cause all neurons to learn identically (symmetry breaking problem) Random values give each neuron a unique starting point, allowing the network to learn diverse features

Q: What accuracy does the 3-layer network achieve on MNIST, and how does it compare to baselines?

Random guessing Accuracy: ~10% 'Darkness' heuristic Accuracy: ~22.25% Neural Network (30 hidden neurons, SGD) Accuracy: ~95% Neural Network (100 hidden neurons, SGD) Accuracy: ~96.59% Support Vector Machine (SVM, reference) Accuracy: Competitive with neural networks The neural network dramatically outperforms naive baselines and is competitive with SVMs — with only ~74 lines of Python.

Q: What is a deep neural network and why does depth matter?

A deep neural network has two or more hidden layers . Depth matters because it allows the network to build a hierarchy of abstractions : Early layers : detect simple features (edges, pixel patterns) Middle layers : combine simple features into parts (loops, strokes) Later layers : combine parts into complex concepts (full digit shapes, faces) Nielsen compares this to modular programming: depth gives abstraction, allowing complex problems to be solved by composing simpler learned sub-problems.

Q: Explain to a non-technical friend why a sigmoid neuron can learn but a perceptron cannot.

'Imagine you're adjusting a thermostat. A perceptron is like a light switch — the tiniest turn either fully turns the heating on or fully off, and the slightest wobble flips the whole house. You can't make careful small adjustments. A sigmoid neuron is like a proper dimmer dial — small turns give small changes in warmth. You can gradually increase or decrease and home in on exactly the right temperature. That's why sigmoid neurons can learn: you can nudge them toward the right answer without the whole network going haywire.'

Q: Why does the 'ball rolling downhill' analogy break down, and what does the real gradient descent algorithm actually do at each step?

A real ball has momentum — it can coast past the bottom of a valley, roll partway up the other side, and oscillate. The gradient descent algorithm has no memory of previous steps — at each step it simply computes the current gradient and moves directly opposite to it. There is no momentum, no coasting. It is purely a local, greedy 'go downhill now' rule, repeated until convergence. (Note: momentum-based gradient descent is a later variant that deliberately adds momentum back in to help escape shallow local minima.)

Q: A classmate says '4 output neurons are more efficient than 10 — why use 10?' How do you respond?

Efficiency of representation is not the goal — learnability is. With 4 binary output neurons, each neuron encodes a bit of the binary digit representation. But bits have no natural correspondence to visual components (loops, curves, strokes). The hidden layer has no clear target to learn toward. With 10 output neurons, neuron i simply needs to learn 'is this digit i ?' — and the hidden layer can learn shape components that directly support that binary question. Empirically, 10 neurons leads to better accuracy, even though it uses more neurons. More neurons, but a more learnable problem.

Q: You train a neural network on MNIST and it reaches 99% training accuracy but only 60% test accuracy. Using concepts from this chapter and earlier chapters, what is happening and what would you try?

This is classic overfitting — the network memorised the training examples instead of learning general digit features. Diagnosis and fixes: Network too large for the data → Reduce hidden neurons No regularisation → Add L2/L1 weight penalty Too many epochs → Early stopping using a validation set Mini-batch too small → Use larger mini-batches for more stable gradient estimates Learning rate too high → Reduce η ; the model may be overshooting the generalising minimum Cross-validate on a held-out validation set (never the test set) to select hyperparameters.

10 important questions on Learning Goal - Using Neural Nets to Recognize Handwritten Digits

Describe the architecture of the three-layer network used for MNIST digit classification.

- Input layer: 784 neurons — one per pixel in a 28 x 28 grayscale image, with pixel intensity ∈ [0,1]
- Hidden layer: n neurons (experiments: 30 or 100)
- Output layer: 10 neurons — one per digit (0–9); the predicted digit is the one with the highest activation

Why use 10 output neurons instead of 4 (which would be enough in binary)?

4 neurons (2^4 = 16 > 10) are mathematically sufficient, but empirically worse. The heuristic argument:

- The hidden layer likely learns to detect component shapes of digits (loops, strokes, curves)
- With 10 outputs, neuron 0 just needs to weigh evidence that those shapes form a "0" — a natural, learnable decomposition
- With 4 outputs, each neuron would encode a bit of the binary representation, but there is no clear reason why component shapes would correlate with a specific bit — the learning signal is harder to interpret

In practice, 10 neurons just works better.

How are weights and biases initialised in the network, and why?

Weights and biases are initialised randomly using Gaussian distributions (mean 0, standard deviation 1). Random initialisation is necessary because:

Initialising everything to 0 would cause all neurons to learn identically (symmetry breaking problem)
Random values give each neuron a unique starting point, allowing the network to learn diverse features

What accuracy does the 3-layer network achieve on MNIST, and how does it compare to baselines?

Random guessing

Accuracy: ~10%

"Darkness" heuristic

Accuracy: ~22.25%

Neural Network (30 hidden neurons, SGD)

Accuracy: ~95%

Neural Network (100 hidden neurons, SGD)

Accuracy: ~96.59%

Support Vector Machine (SVM, reference)

Accuracy: Competitive with neural networks

The neural network dramatically outperforms naive baselines and is competitive with SVMs — with only ~74 lines of Python.

What is a deep neural network and why does depth matter?

A deep neural network has two or more hidden layers. Depth matters because it allows the network to build a hierarchy of abstractions:

Early layers: detect simple features (edges, pixel patterns)
Middle layers: combine simple features into parts (loops, strokes)
Later layers: combine parts into complex concepts (full digit shapes, faces)

Nielsen compares this to modular programming: depth gives abstraction, allowing complex problems to be solved by composing simpler learned sub-problems.

What does Nielsen mean when he says the weights and biases of a trained network are "mysterious"?

While the network performs well, the individual weight values have no clear human-interpretable meaning — you can't look at a weight and understand why it has that value or what the neuron is detecting. The network's knowledge is distributed across thousands of parameters in a way that is opaque to the programmer. This lack of interpretability is a fundamental challenge of neural networks.

Explain to a non-technical friend why a sigmoid neuron can learn but a perceptron cannot.

"Imagine you're adjusting a thermostat. A perceptron is like a light switch — the tiniest turn either fully turns the heating on or fully off, and the slightest wobble flips the whole house. You can't make careful small adjustments.

A sigmoid neuron is like a proper dimmer dial — small turns give small changes in warmth. You can gradually increase or decrease and home in on exactly the right temperature. That's why sigmoid neurons can learn: you can nudge them toward the right answer without the whole network going haywire."

Why does the "ball rolling downhill" analogy break down, and what does the real gradient descent algorithm actually do at each step?

A real ball has momentum — it can coast past the bottom of a valley, roll partway up the other side, and oscillate. The gradient descent algorithm has no memory of previous steps — at each step it simply computes the current gradient and moves directly opposite to it. There is no momentum, no coasting. It is purely a local, greedy "go downhill now" rule, repeated until convergence.

(Note: momentum-based gradient descent is a later variant that deliberately adds momentum back in to help escape shallow local minima.)

A classmate says "4 output neurons are more efficient than 10 — why use 10?" How do you respond?

Efficiency of representation is not the goal — learnability is. With 4 binary output neurons, each neuron encodes a bit of the binary digit representation. But bits have no natural correspondence to visual components (loops, curves, strokes). The hidden layer has no clear target to learn toward.

With 10 output neurons, neuron i simply needs to learn "is this digit i?" — and the hidden layer can learn shape components that directly support that binary question. Empirically, 10 neurons leads to better accuracy, even though it uses more neurons. More neurons, but a more learnable problem.

You train a neural network on MNIST and it reaches 99% training accuracy but only 60% test accuracy. Using concepts from this chapter and earlier chapters, what is happening and what would you try?

This is classic overfitting — the network memorised the training examples instead of learning general digit features. Diagnosis and fixes:

Network too large for the data → Reduce hidden neurons
No regularisation → Add L2/L1 weight penalty
Too many epochs → Early stopping using a validation set
Mini-batch too small → Use larger mini-batches for more stable gradient estimates
Learning rate too high → Reduce η; the model may be overshooting the generalising minimum

Cross-validate on a held-out validation set (never the test set) to select hyperparameters.

The question on the page originate from the summary of the following study material:

COMP 3711

View summary

A unique study and practice tool
Never study anything twice again
Get the grades you hope for
100% sure, 100% understanding

Remember faster, study better. Scientifically proven.

Topics that are related to Learning Goal - Using Neural Nets to Recognize Handwritten Digits

Learning Goal - Gradient Descent and Learning

Learning Goal - Using Neural Nets to Recognize Handwritten Digits

10 important questions on Learning Goal - Using Neural Nets to Recognize Handwritten Digits

Describe the architecture of the three-layer network used for MNIST digit classification.

Why use 10 output neurons instead of 4 (which would be enough in binary)?

How are weights and biases initialised in the network, and why?

What accuracy does the 3-layer network achieve on MNIST, and how does it compare to baselines?

What is a deep neural network and why does depth matter?

What does Nielsen mean when he says the weights and biases of a trained network are "mysterious"?

Explain to a non-technical friend why a sigmoid neuron can learn but a perceptron cannot.

Why does the "ball rolling downhill" analogy break down, and what does the real gradient descent algorithm actually do at each step?

A classmate says "4 output neurons are more efficient than 10 — why use 10?" How do you respond?

You train a neural network on MNIST and it reaches 99% training accuracy but only 60% test accuracy. Using concepts from this chapter and earlier chapters, what is happening and what would you try?

Topics that are related to Learning Goal - Using Neural Nets to Recognize Handwritten Digits

Learning Goal - Gradient Descent and Learning

Summaries related to Classification - Impurity Measures

COMP 3711

Global politics

Essentials of international relations

Behavioral genetics

Management and organisational behaviour

Follow Up Engels idioom 4/5 H

International Business

Marketing fundamentals

Projectmanagement, A practical Approach-Engl…

Basic Management Accounting for the Hospital…

International business

Organization theory and design