Summary: Comp 3711

Read the summary and the most important questions on COMP 3711

• 2 Neural Networks and Deep Learning

• 2.1.1 The Problem and the Approach

• Why is handwriting recognition hard to solve with explicit rules, and what does the neural network approach do differently?

  Human visual pattern recognition involves countless exceptions and edge cases that are nearly impossible to enumerate as rules. Instead of hard-coding rules, neural networks learn rules automatically from training examples — the more examples, the more accurate the inferred rules. The programmer never writes a single explicit "if this pixel pattern, then this digit" rule.

• Why is handwriting recognition a good prototype problem for introducing deep learning?

  It is challenging enough to require real techniques (trivial approaches fail) but not so hard that it demands immense computational power or data. It allows the introduction of fundamental ideas — perceptrons, sigmoid neurons, gradient descent — in a concrete, tractable setting.

• 2.1.2 Perceptrons

• What is a perceptron? Describe its inputs, weights, and output rule.

  A perceptron is an artificial neuron (developed 1950s–60s) that:
  

  • Takes several binary inputs x₁, x₂, …
    
  • Associates each input with a real-valued weight wⱼ (expressing its importance)
    
  • Produces a binary output (0 or 1):
    

  
  output = 0 if the weighted sum (w₁x₁ + w₂x₂ + …) is ≤ threshold
  output = 1 if the weighted sum (w₁x₁ + w₂x₂ + …) is > threshold

• What is the bias b of a perceptron, and why does Nielsen introduce it?

  Bias b ≡ −threshold. 
  Nielsen introduces it to simplify the notation: instead of writing a threshold on the right-hand side, the output rule becomes:
  
  output = 0 if w·x + b ≤ 0
  output = 1 if w·x + b > 0
  
  Conceptually, bias measures how easy it is for the neuron to fire:
  

  • Large positive b → fires very easily
    
  • Large negative b → fires rarely
    

• What logic gates can a perceptron implement? Why does NAND matter?

  A perceptron can implement AND, OR, and NAND gates by choosing appropriate weights and biases.
  
  NAND matters because NAND is universal for computation — any other logical function (NOT, OR, XOR, full adders, etc.) can be built entirely from NAND gates. Therefore, networks of perceptrons can compute any logical function — they are computationally universal.

• Why is NAND called "universal"? Sketch how you'd build NOT from NAND.

  NAND is universal because every other Boolean function can be expressed as a combination of NAND gates alone.
  NOT from NAND: feed the same input into both inputs of a NAND gate:
  NAND(x, x) = NOT(x AND x) = NOT x
  From NOT and NAND you can build AND, OR, and therefore any logic circuit.

• 2.1.4 Architecture of Neural Networks

• What are the three types of layers in a feedforward neural network?

  1. Input layer — receives the raw features (e.g., pixel values)
  2. Hidden layer(s) — intermediate layers, neither input nor output; where learned representations are built
  3. Output layer — produces the final prediction
  
  Layers that are neither the first nor last are "hidden" because their activations are not directly observed.

• What is a multilayer perceptron (MLP) and why is the name misleading?

  MLP is the historical name for multi-layer neural networks. It is misleading because modern MLPs almost always use sigmoid (or other smooth) neurons — not perceptrons. The name stuck from historical convention even though the underlying neuron type changed.

• What is the difference between feedforward and recurrent networks?

  • Feedforward Networks
  • Information flows in one direction only (no loops)
    

  • Stateless — the same input always produces the same output
  • Considered less similar to how the brain works
    
  • More widely used and better understood today

  • Recurrent Networks
  • Allow loops — outputs can be fed back as inputs

  • Have memory — outputs depend on previous states
  • More similar to how the brain works
  • Less widely used and generally harder to train

  
  
  

• 2.1.5 Classifying Handwritten Digits

  This is a preview. There are 2 more flashcards available for chapter 2.1.5
  Show more cards here

• Why is the digit recognition problem split into segmentation and classification?

  An image may contain multiple digits (e.g., a phone number).
  Segmentation first breaks the image into individual digit images.
  Classification then identifies each isolated digit.
  
  Chapter 1 focuses only on classification — taking a single already-isolated digit image and identifying which digit it is.