Home / Summaries / COMP 3711 / training-gradient-direction

Learning Goal - Gradient Descent and Learning

Q: What is the learning rate η and what happens at the extremes?

η controls the step size in gradient descent: - Too large : overshoots the minimum — the ball 'jumps over the valley' and may diverge - Too small : converges correctly but very slowly — requires many more iterations - Just right : efficient convergence to the minimum Choosing η is a key hyperparameter decision in training.

Q: What is an epoch in neural network training?

One epoch = one complete pass through all training inputs. In SGD, training examples are divided into mini-batches; an epoch is completed once every mini-batch has been used once. Networks are typically trained for many epochs.

5 important questions on Learning Goal - Gradient Descent and Learning

What is the quadratic cost function (MSE)? Write the formula and explain each term.

C(w, b) = (1 / (2n)) ∑_x || (y(x) − a)² ||

- n = total number of training examples
- x = a training input
- y(x) = the desired (correct) output for input x
- a = the actual network output for input x
- The 1/2 is a convenience to make the derivative cleaner

The cost is the average squared difference between desired and actual outputs — minimising it means the network's outputs are close to the correct answers.

Explain gradient descent conceptually using Nielsen's "ball in a valley" analogy.

Imagine the cost function as a hilly landscape with a deep valley at the minimum. Gradient descent simulates a ball rolling downhill:
1. Compute the gradient (steepest uphill direction) at the current position
2. Move in the opposite direction (downhill) by a small step

Key distinction from a real ball: no momentum — the algorithm always moves directly downhill at each step, never coasting across a slope or going uphill.

Update rule: v→v′=v−η∇C, where η is the learning rate.

What is the learning rate η and what happens at the extremes?

η controls the step size in gradient descent:
- Too large: overshoots the minimum — the ball "jumps over the valley" and may diverge
- Too small: converges correctly but very slowly — requires many more iterations
- Just right: efficient convergence to the minimum

Choosing η is a key hyperparameter decision in training.

What is Stochastic Gradient Descent (SGD) and why does it work?

Instead of computing the gradient over the full training set (expensive), SGD estimates it using a small randomly chosen mini-batch of m examples:

∇C ≈ 1/m ∑^m_(j=1) ∇C_(xj)

Why it works: Nielsen's analogy — like political polling. A small random sample estimates the full population's opinion well enough. The estimate isn't perfect, but it gives a good enough direction to reduce the cost. Averaged over many steps, SGD converges to (approximately) the same result as full gradient descent, but much faster.

What is an epoch in neural network training?

One epoch = one complete pass through all training inputs.
In SGD, training examples are divided into mini-batches; an epoch is completed once every mini-batch has been used once.
Networks are typically trained for many epochs.

The question on the page originate from the summary of the following study material:

COMP 3711

View summary