Home / Summaries / COMP 3711 / neuron-bump-failure

Learning Goal - A Visual Proof That Neural Nets Can Compute Any Function

Q: How do you build a bump function from two hidden neurons? Be precise about the output weights.

Pair two step-neuron hidden units: Neuron 1 : step position s₁, output weight +h — 'turns the bump on' at x = s₁ Neuron 2 : step position s₂ > s₁, output weight −h — 'turns the bump off' at x = s₂ Weighted sum of their outputs: x < s₁: Neuron 1 = 0, Neuron 2 = 0, Sum = 0 s₁ < x < s₂: Neuron 1 = +h, Neuron 2 = 0, Sum = h x > s₂: Neuron 1 = +h, Neuron 2 = −h, Sum = 0 Result: a bump of height h between s₁ and s₂, zero everywhere else.

Q: Why does the bump construction require exactly 2k hidden neurons to represent k bumps — not k?

Each bump requires two neurons working as a pair: One to 'turn on' at the left edge of the bump (step at s_start, weight +h) One to 'turn off' at the right edge (step at s_end, weight −h) A single neuron can only create a step — it stays at h forever once triggered. The cancellation from the second neuron is what creates the finite-width bump. Without it, every activation would spread across the entire remaining input range. So k bumps → 2k hidden neurons.

4 important questions on Learning Goal - A Visual Proof That Neural Nets Can Compute Any Function

How do you build a bump function from two hidden neurons? Be precise about the output weights.

Pair two step-neuron hidden units:

Neuron 1: step position s₁, output weight +h — "turns the bump on" at x = s₁
Neuron 2: step position s₂ > s₁, output weight −h — "turns the bump off" at x = s₂

Weighted sum of their outputs:

x < s₁: Neuron 1 = 0, Neuron 2 = 0, Sum = 0
s₁ < x < s₂: Neuron 1 = +h, Neuron 2 = 0, Sum = h
x > s₂: Neuron 1 = +h, Neuron 2 = −h, Sum = 0

Result: a bump of height h between s₁ and s₂, zero everywhere else.

How does the bump function construction act as a "functional lookup table"?

By gluing together many such neuron pairs (each pair = one bump), the network can create bumps of different heights at different locations along the x-axis. Each bump covers a small interval and takes the height of f(x) in that interval. Together, the bumps tile the entire input range — producing a piecewise-constant approximation to any target function f(x).

More pairs = narrower bumps = finer resolution = better approximation.

What is a "window of failure" and how is it addressed?

Because real neurons only approximate step functions (the sigmoid is never perfectly vertical), there is a narrow transition region where the neuron's output is between 0 and 1 instead of a clean jump. This region is the window of failure — the approximation is inaccurate here.

Fix: use multiple overlapping approximations shifted slightly from one another. Their individual errors in the transition windows are at different locations and average out, giving a much smoother and more accurate combined fit.

Why does the bump construction require exactly 2k hidden neurons to represent k bumps — not k?

Each bump requires two neurons working as a pair:

One to "turn on" at the left edge of the bump (step at s_start, weight +h)
One to "turn off" at the right edge (step at s_end, weight −h)

A single neuron can only create a step — it stays at h forever once triggered. The cancellation from the second neuron is what creates the finite-width bump. Without it, every activation would spread across the entire remaining input range. So k bumps → 2k hidden neurons.

The question on the page originate from the summary of the following study material:

COMP 3711

View summary

A unique study and practice tool
Never study anything twice again
Get the grades you hope for
100% sure, 100% understanding

Remember faster, study better. Scientifically proven.

Topics that are related to Learning Goal - A Visual Proof That Neural Nets Can Compute Any Function

Learning Goal - A Visual Proof That Neural Nets Can Compute Any Function

4 important questions on Learning Goal - A Visual Proof That Neural Nets Can Compute Any Function

How do you build a bump function from two hidden neurons? Be precise about the output weights.

How does the bump function construction act as a "functional lookup table"?

What is a "window of failure" and how is it addressed?

Why does the bump construction require exactly 2k hidden neurons to represent k bumps — not k?

Topics that are related to Learning Goal - A Visual Proof That Neural Nets Can Compute Any Function

Learning Goal - The Universality Theorem

Learning Goal - Building Blocks: Sigmoids as Step Functions

Learning Goal - The Inverse Sigmoid Trick

Learning Goal - Extension to Multiple Inputs

Learning Goal - Extension to Other Activation Functions

COMP 3711

Global politics

Essentials of international relations

Behavioral genetics

Management and organisational behaviour

Follow Up Engels idioom 4/5 H

International Business

Marketing fundamentals

Projectmanagement, A practical Approach-Engl…

Basic Management Accounting for the Hospital…

International business

Organization theory and design

Learning Goal - A Visual Proof That Neural Nets Can Compute Any Function

4 important questions on Learning Goal - A Visual Proof That Neural Nets Can Compute Any Function

How do you build a bump function from two hidden neurons? Be precise about the output weights.

How does the bump function construction act as a "functional lookup table"?

What is a "window of failure" and how is it addressed?

Why does the bump construction require exactly 2k hidden neurons to represent k bumps — not k?

Topics that are related to Learning Goal - A Visual Proof That Neural Nets Can Compute Any Function

Learning Goal - The Universality Theorem

Learning Goal - Building Blocks: Sigmoids as Step Functions

Learning Goal - The Inverse Sigmoid Trick

Learning Goal - Extension to Multiple Inputs

Learning Goal - Extension to Other Activation Functions

Summaries related to Classification - Impurity Measures

COMP 3711

Global politics

Essentials of international relations

Behavioral genetics

Management and organisational behaviour

Follow Up Engels idioom 4/5 H

International Business

Marketing fundamentals

Projectmanagement, A practical Approach-Engl…

Basic Management Accounting for the Hospital…

International business

Organization theory and design