Learning Goal - The Inverse Sigmoid Trick
3 important questions on Learning Goal - The Inverse Sigmoid Trick
Why can't the hidden layer just output f(x) directly? What goes wrong?
The output neuron applies another sigmoid: its output is σ(Σⱼ wⱼaⱼ + b). If the hidden layer produces f(x), the final output is σ(f(x)) — distorted by the output sigmoid. This is not f(x).
What is the inverse sigmoid trick and how does it fix the problem?
Instead of designing the hidden layer to output f(x), design it to output σ⁻¹(f(x)) — the inverse sigmoid of the target.
Then the output neuron computes:
σ(σ⁻¹(f(x))) = f(x)
The two sigmoids cancel, and the network's final output is exactly f(x). The bump heights h in the hidden layer are set to σ⁻¹ of the desired output values rather than the values themselves.
Then the output neuron computes:
σ(σ⁻¹(f(x))) = f(x)
The two sigmoids cancel, and the network's final output is exactly f(x). The bump heights h in the hidden layer are set to σ⁻¹ of the desired output values rather than the values themselves.
What constraint does the inverse sigmoid trick place on f(x)?
The inverse sigmoid σ⁻¹ is only defined for values strictly between 0 and 1. Therefore, the inverse sigmoid trick requires the target function f(x) to take values in the open interval (0, 1). For functions with outputs outside this range, a different output neuron (e.g., linear) or a rescaling approach is needed.
The question on the page originate from the summary of the following study material:
- A unique study and practice tool
- Never study anything twice again
- Get the grades you hope for
- 100% sure, 100% understanding
Remember faster, study better. Scientifically proven.
