Neural Networks

Trying to recreate a brain from scratch

Lukas Schmoigl

lukas.schmoigl@wifo.ac.at

Data Scientist at WIFO

A Simple Network

What a neural network is

MNIST Digits

Recognise a handwritten digit.

• Trivial for you, and you have no idea how you do it
• No sensible set of if rules exists
• This is the whole reason the field looks the way it does

Data: MNIST — 60,000 labelled training images, 10,000 for testing.

From Image to Data

after 3Blue1Brown, But what is a neural network?

Which is really: 784 numbers in, 10 numbers out

• Input: one number per pixel, \(0\) (black) to \(1\) (white) → 784 inputs

• Output: one number per digit, \(0\) to \(1\) → 10 outputs

• The answer is whichever output neuron is brightest

• Everything in between is our design choice. The classic choice: two hidden layers of 16 neurons each

The network

784 → 16 → 16 → 10, as in 3Blue1Brown chapter 1

A neuron is just a thing that holds a number

• That number is its activation, between 0 and 1
• Input neurons: the pixel brightness. Output neurons: how strongly the network believes in that digit

• A neuron is better thought of as a function: it takes every activation in the previous layer and spits out one number

• The whole network is then just one big function: 784 numbers in, 10 numbers out

Watch it run

\[ \mathbf{a}^{(1)} = \sigma\!\left(W_0\,\mathbf{a}^{(0)} + \mathbf{b}_0\right) \]

forward propagation — activations in one layer determine the next

Intuition behind layers

The hope is that layers correspond to levels of abstraction:

A 9 is a loop up top plus a line down the right; an 8 is two loops.

Keep this hope in mind. We will come back and check whether it is true.

after 3Blue1Brown, BreakUpMacroPatterns / BreakUpMicroPatterns

One neuron, in full

\[ a = \sigma\!\left(w_1 a_1 + w_2 a_2 + \cdots + w_n a_n + b\right) \]

after 3Blue1Brown, IntroduceWeights / IncludeBias

The one formula that matters

\[ a = \sigma\!\left(w_1 a_1 + w_2 a_2 + \cdots + w_n a_n + b\right) \]

• \(a_i\) — activations of the previous layer (given)
• \(w_i\) — weights: what pattern this neuron is looking for
• \(b\) — bias: how high the weighted sum must be before the neuron fires
• \(\sigma\) — activation function: squishes the result into a usable range

The weights and biases are the only things we get to change. Everything we call “learning” is choosing numbers for them.

Weights are a picture

there are 784 weights into each hidden neuron — one per pixel — so you can draw them

Squishing: the activation function

\[\sigma(x) = \frac{1}{1 + e^{-x}}\]

\[\mathrm{ReLU}(x) = \max(0,\, x) \vphantom{\frac{1}{1 + e^{-x}}}\]

after 3Blue1Brown, IntroduceSigmoid / IntroduceReLU

Why any non-linearity at all?

Without \(\sigma\), a layer computes \(Wa + b\). Stack two of them: \[ W_2(W_1 a + b_1) + b_2 = (W_2 W_1)\, a + (W_2 b_1 + b_2) \]

Still linear. A hundred layers without an activation function collapse into a single linear model — you have re-invented OLS, expensively.

• Sigmoid: the historical choice, from the biological “fires / does not fire” analogy
• ReLU: what is actually used — trains far faster, no vanishing gradient

All of it, compactly

Collect the weights of a layer into a matrix \(W\), the activations and biases into vectors: \[ \mathbf{a}^{(l+1)} = \sigma\!\left(W_l\, \mathbf{a}^{(l)} + \mathbf{b}_l\right) \]

This is the whole forward pass. It is why neural network code is short and why GPUs are involved: a layer is one matrix multiplication.

Counting parameters for 784 → 16 → 16 → 10:

\[ \underbrace{784\cdot 16 + 16 \cdot 16 + 16 \cdot 10}_{\text{12,960 weights}} \;+\; \underbrace{16 + 16 + 10}_{\text{42 biases}} \;=\; \mathbf{13{,}002} \]

Chapter 1

Gradient Descent

How the 13,002 numbers get chosen

What “learning” means

Start with random weights and biases. The network will be terrible.

We need to tell it how terrible, with a single number.

For one training image, the cost is \[ C_0 = \sum_{j=0}^{n_L - 1} \left(a^{(L)}_j - y_j\right)^2 \] where \(y_j\) is what we wanted: \(1\) for the correct digit, \(0\) for the other nine.

• Small when the output is confident and right
• Large when the output is confidently wrong, or just mush

The cost of the whole network

\[ C = \frac{1}{n}\sum_{k=0}^{n-1} C_k \] the average cost over all \(n\) training examples.

Now look at what kind of object \(C\) is:

• Input: 13,002 weights and biases
• Output: one number, the cost
• Parameters: all 60,000 training examples

Learning = minimising \(C\). Which is now an ordinary, if enormous, calculus problem.

Minimising a function of one variable

after 3Blue1Brown, SingleVariableCostFunction

Rolling downhill

Solving \(\frac{dC}{dw} = 0\) is hopeless for a function this complicated.

So do the stupid thing instead: start anywhere, look at the slope, take a step downhill, repeat. \[ w \leftarrow w - \eta \, \frac{dC}{dw} \]

• The slope is negative → step right. Positive → step left
• Steps shrink automatically as the slope flattens → you converge instead of overshooting
• \(\eta\) is the learning rate: too small and you wait forever, too large and you bounce out of the valley

Two variables, and the honest picture

after 3Blue1Brown, TwoVariableInputSpace / LocalVsGlobal

Gradient descent

For a function of many variables, the gradient \(\nabla C\) is the vector of all partial derivatives. It points in the direction of steepest increase.

So \(-\nabla C\) points downhill. The algorithm is one line:

1. Compute \(\nabla C\)
2. Take a small step in the direction of \(-\nabla C\)
3. Repeat

In 13,002 dimensions you cannot picture the surface — but “compute the gradient, step against it” needs no picture.

The gradient is the answer sheet

after 3Blue1Brown, ShowFullCostFunctionGradient

Reading the gradient

\(-\nabla C\) has 13,002 components, one per parameter. Each tells you two things:

• the sign — nudge this weight up, or down
• the size — how much this weight matters right now

A component of \(+0.82\) against one of \(+0.03\) says: changing the first weight buys you 27 times more than changing the second.

Two things worth noticing:

• Nothing here guarantees a global minimum. Different random initialisations land in different valleys
• Continuous activations are what make this work at all — with a hard step function there is no slope to follow

Chapter 2

Does it work? Yes. Does it do what we hoped? No.

This network gets ~96% of unseen digits right. Tuned a little, ~98%.

So the second layer must be detecting loops and edges, as we hoped?

Look at the weights again: mostly noise with a loose pattern in the middle. It found some structure in the training set that happens to work. Not our structure.

Worse: feed it random noise and it will confidently name a digit. It has not learned to recognise digits — it has learned to sort the 60,000 images it was shown.

This gap between “high test accuracy” and “learned the concept” is the single most important caveat when you report results from a black box.

Backpropagation

Where the gradient actually comes from

The problem

Gradient descent needs \(\nabla C\): 13,002 partial derivatives, recomputed at every single step.

Backpropagation is the algorithm that computes them all in one backward sweep, at roughly the cost of one forward pass.

The intuition first, the calculus after.

What one training example wants

after 3Blue1Brown, WalkThroughTwoExample

Three ways to make an output neuron brighter

Take \(a^{(L)} = \sigma\!\left(w_1 a_1 + \cdots + w_n a_n + b\right)\) and ask how to increase it:

1. Increase the bias \(b\)
2. Increase the weights \(w_i\) — with most effect on the \(w_i\) whose \(a_i\) is already large
3. Change the previous activations \(a_i\) — up where \(w_i > 0\), down where \(w_i < 0\)

Only 1 and 2 are ours to change. Number 3 is a request, passed back to the previous layer.

Point 2 is Hebbian theory in miniature: neurons that fire together, wire together. The biggest weight increases go to the connections that were already most active.

Propagating backwards

The neuron for “2” wants the previous layer to change. So do the other nine output neurons — and they disagree.

Add up all ten requests. That gives a desired nudge for every neuron in layer \(L-1\).

Now recurse: apply the same three-way logic one layer back. And again. All the way to the input.

Do this for every training example, average the results, and you have \(-\nabla C\) — one vote per example, per parameter.

Being honest about the cost

That means a full forward and backward pass over all 60,000 images for one step of gradient descent. And you need thousands of steps.

So: shuffle the data, chop it into mini-batches of, say, 100, and take a step using each batch.

Each step is now a bad estimate of the true gradient — but you get 600 of them for the price of one. This is stochastic gradient descent.

Careful vs. fast

after 3Blue1Brown, OrganizeDataIntoMiniBatches

Chapter 3

In Practice

Knobs, and an example

Training vocabulary

• Iteration — one weight update, from one mini-batch
• Batch size — examples per update
• Epoch — one full pass over the training data

60,000 examples with batch size 100 → 600 iterations per epoch.

• Too few epochs → underfitting
• Too many → overfitting: the training cost keeps falling while test performance gets worse

Watch the test cost, not the training cost. Stop when it turns around.

Always scale your inputs

Weights start small and random. A feature measured in the thousands and one measured in \([0,1]\) do not get a fair hearing.

Min-max, to \([0,1]\): \[ x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}} \]

Or standardise, to mean \(0\) and standard deviation \(1\): \[ x' = \frac{x - \mu}{\sigma} \]

Fit the scaling on the training data only, then apply it to the test data. Otherwise you have leaked information.

A regression example in R

Predict median house values, neuralnet against glm:

library(neuralnet); library(MASS)
set.seed(3456)

# 1. scale everything into [0, 1] -- neural nets need it, OLS does not care
mins <- apply(Boston, 2, min); maxs <- apply(Boston, 2, max)
scaled <- as.data.frame(scale(Boston, center = mins, scale = maxs - mins))

# 2. split
train_i <- sample(1:nrow(Boston), round(0.75 * nrow(Boston)))
train <- scaled[train_i, ]; test <- scaled[-train_i, ]

# 3. two hidden layers, 5 and 3 neurons; linear output for regression
f <- as.formula(paste("medv ~", paste(setdiff(names(Boston), "medv"),
                                      collapse = " + ")))
net <- neuralnet(f, data = train, hidden = c(5, 3), linear.output = TRUE)

Compare it against OLS

# same data, same scaling, so RMSE is comparable
pred_net <- predict(net, test[, setdiff(names(test), "medv")])
pred_lm  <- predict(glm(medv ~ ., data = train), test)

c(net = caret::RMSE(pred_net, test$medv),
  lm  = caret::RMSE(pred_lm,  test$medv))

• The net usually wins here, by a little
• Rerun it with a different seed and the margin moves. Report that

• 13 predictors and 506 rows is not where neural networks shine. Their edge appears when the input is high-dimensional and the structure is genuinely non-linear — images, text, audio

Full script: res/nn-code.R

Wrapping Up

What to take away

1. A network is one big function, built from \(a = \sigma(Wa + b)\) repeated
2. Learning = minimising a cost function over its 13,002 parameters
3. Gradient descent does the minimising; backpropagation supplies the gradient
4. Backpropagation is the chain rule, applied backwards, and nothing more

5. The activation function is what stops the whole thing collapsing into OLS
6. Scaling is not optional

And what to be suspicious of

• It is a black box. The hidden layers did not learn the concepts we hoped for, even at 98% accuracy
• It is not causal. A good predictor of \(y\) is not an estimate of the effect of anything

• No global optimum is guaranteed. Different seeds, different valleys
• No rulebook for layer count, layer size, learning rate, batch size
• It is expensive, in compute and in labelled data

Fit one, then ask whether OLS would have done as well. Often it will.

Assignment

• Unemployment prediction - your turn
  1. Take the large synthetic unemployment data, explore it and split it into training and test data
  2. Train a model for target_low: at least 3 months of unemployment in the next 6 months
  3. Train a model for target_high: at least 6 months of unemployment in the next 2 years
  4. Judge both in the test data with the criteria we covered, such as the confusion matrix, ROC- and PR-curves, and say which target was harder to predict and why

OR

• Kaggle competition - your turn
  1. Pick an open competition whose data and task you find interesting
  2. Build a model and submit at least one entry to the leaderboard
  3. Report your score and the approaches you tried along the way
  4. Reflect on what held your model back and what you would try next

Either way the model is yours to pick: a neural network, or anything else that does the job.

Due date: January 8th ### Sources and further reading {.smaller}

The visual material on these slides

• 3Blue1Brown, Neural Networks series (2017) — videos · manim source
• Figures redrawn from that source: src/nn-figures.py

Books and papers

• Nielsen, M. (2015). Neural Networks and Deep Learning. free online — the book the 3b1b series follows
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
• Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386–408.
• McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5(4), 115–133.
• Ciaburro, G., & Venkateswaran, B. (2017). Neural Networks with R. Packt.