Learning Goal - Notation
3 important questions on Learning Goal - Notation
What does w^l_{jk} denote? Why is the index order j then k — not k then j?
w^l_(jk) = the weight connecting the k-th neuron in layer l−1 to the j-th neuron in layer l.
j before k (destination before source) is counterintuitive but intentional: it means the weight matrix w^l can be written so that the activation equation is clean:
a^l = σ(w^l · a^(l−1) + b^l)
If we used k for output and j for input, we'd need a transpose every time, making the notation much messier.
j before k (destination before source) is counterintuitive but intentional: it means the weight matrix w^l can be written so that the activation equation is clean:
a^l = σ(w^l · a^(l−1) + b^l)
If we used k for output and j for input, we'd need a transpose every time, making the notation much messier.
Define z^l (the weighted input). What is its role?
z^l ≡ w^l · a^(l−1) + b^l
z^l is the pre-activation of layer l — the weighted sum of inputs plus bias, before the sigmoid is applied. The full activation is then a^l = σ(z^l).
Separating z^l from a^l is critical for backpropagation: the error δ^l is defined in terms of z^l, and σ'(z^l) appears naturally in the equations.
z^l is the pre-activation of layer l — the weighted sum of inputs plus bias, before the sigmoid is applied. The full activation is then a^l = σ(z^l).
Separating z^l from a^l is critical for backpropagation: the error δ^l is defined in terms of z^l, and σ'(z^l) appears naturally in the equations.
What is the Hadamard product (s ⊙ t)?
The Hadamard product is elementwise multiplication of two vectors of the same size:
(s ⊙ t)ⱼ = sⱼ · tⱼ
It is used in backpropagation to scale each component of a vector independently — e.g., multiplying each neuron's cost gradient by that neuron's σ' value. Using standard matrix multiplication instead would require large, wasteful diagonal matrices.
(s ⊙ t)ⱼ = sⱼ · tⱼ
It is used in backpropagation to scale each component of a vector independently — e.g., multiplying each neuron's cost gradient by that neuron's σ' value. Using standard matrix multiplication instead would require large, wasteful diagonal matrices.
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
