Learning Goal - Extension to Other Activation Functions
3 important questions on Learning Goal - Extension to Other Activation Functions
What property must an activation function have for the visual proof to work?
The activation function s(z) must:
- Be well-defined as z → +∞ and z → −∞
- Have different limits at the two extremes (e.g., 0 at −∞ and 1 at +∞ for sigmoid)
Why are linear neurons not universal? Why does depth not help?
A linear neuron computes output = z = w·x + b — a linear function of its inputs. The composition of linear functions is still linear (any chain of matrix multiplications and additions produces a single linear map). No matter how many layers you stack, a network of linear neurons can only compute linear functions — it can never approximate a curve, a bump, or any non-linear function.
Depth without non-linearity adds no expressive power.
Depth without non-linearity adds no expressive power.
Are ReLU neurons universal? Can the sigmoid-based visual proof apply to them?
ReLU neurons are universal — but the sigmoid visual proof does not directly apply to ReLU. ReLU (max(0, z)) does not approach two different constant limits as z → ±∞ (it diverges for z > 0), so the "large weight = step function" argument breaks down.
Alternative proofs exist for ReLU universality using different constructions (e.g., piecewise linear functions). The result is the same — universality holds — but the geometric argument differs from Nielsen's sigmoid-based approach.
Alternative proofs exist for ReLU universality using different constructions (e.g., piecewise linear functions). The result is the same — universality holds — but the geometric argument differs from Nielsen's sigmoid-based approach.
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
