Learning Goal - The Universality Theorem
3 important questions on Learning Goal - The Universality Theorem
State the universality theorem precisely.
A neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function to any desired accuracy ε > 0.
Formally: for any continuous f(x) and any ε > 0, there exists a network g(x) such that:
|g(x) − f(x)| < ε for all inputs x
Formally: for any continuous f(x) and any ε > 0, there exists a network g(x) such that:
|g(x) − f(x)| < ε for all inputs x
What does the universality theorem NOT say? Give three limits.
1. Not exact: the network always approximates — it never computes f exactly. Precision improves with more hidden neurons.
2. No construction recipe: the theorem guarantees such a network exists, but gives no algorithm to find the right weights. Learning must still be done by training.
3. Not for discontinuous functions: the theorem applies to continuous functions. Functions with hard jumps (discontinuities) may not be approximatable, though a continuous approximation is usually good enough in practice.
2. No construction recipe: the theorem guarantees such a network exists, but gives no algorithm to find the right weights. Learning must still be done by training.
3. Not for discontinuous functions: the theorem applies to continuous functions. Functions with hard jumps (discontinuities) may not be approximatable, though a continuous approximation is usually good enough in practice.
Why does the universality theorem not make deep networks unnecessary?
A single hidden layer can approximate any function, but it acts like a brute-force lookup table — requiring a potentially enormous number of neurons for complex functions. This is computationally impractical.
Deep networks are practically superior because they learn hierarchies of knowledge: early layers detect simple features (edges, pixel patterns), later layers combine these into abstract concepts (shapes, objects). This is far more efficient than trying to learn an entire complex mapping in one flat layer.
Deep networks are practically superior because they learn hierarchies of knowledge: early layers detect simple features (edges, pixel patterns), later layers combine these into abstract concepts (shapes, objects). This is far more efficient than trying to learn an entire complex mapping in one flat layer.
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
