Pretrain+CPT · S1 · Pretraining

S1 · Open

S1 · Pretraining

Live board · one idea per frame · yellow card = what to say

Today's story We start with an untrained model that emits nonsense, then builds a number that says how wrong each next-token guess is. We walk one toy sentence end to end: shift → logits → probabilities → loss → train loop → decoding knobs.

You should leave able to… • Build inputs/targets from one sentence • Hand-compute −log p for one target token • Contrast what enters position t in train vs generate • Read a train/val curve for overfitting • Explain temperature without changing weights

S1 · 1 Untrained

1 · Why bother training?

Generate first · see unstructured text · then introduce loss

Prompt Rain falls

Untrained continuation Rain falls quilt zebra the the mat sat Near-uniform next-token probs → little structure.

What we need A scalar: how wrong was the next-token guess? That scalar is the loss. Then gradients can update weights.

Say next Say: 'Before any training, logits are nearly random. Softmax spreads probability thin. We need −log p on the true next token so the optimizer has a direction.'

S1 · 2 Setup

2 · Pin the running example

Keep this frame in mind for the next four slides

TOY VOCAB (5 words only so every vector fits on screen) id token 0 the 1 cat 2 sat 3 on 4 mat SENTENCE: the cat sat on mat IDS: [0, 1, 2, 3, 4]

Self-supervised The next word in the text IS the label. Nobody wrote a separate answer key. That is pretraining for a causal LM: next-token prediction on ordinary text.

Say next Point at id 1 = cat. Say: 'When the model sees the, we want high probability on cat. We will score that with −log p.'

S1 · 3 Shift

3 · Inputs and targets are the same sequence, shifted

Book Ch2 data loader · reused in Ch5.1.2

tokens

0 the

1 cat

2 sat

3 on

4 mat

input

0 the

1 cat

2 sat

3 on

target

1 cat

2 sat

3 on

4 mat

Say next Trace column 0: input the → target cat. Column 1: the cat → sat. Say: 'targets = inputs shifted left by one. That is the whole label.'

S1 · 4 Logits

4 · One position → five raw scores (logits)

Context = the · target should be cat (id 1) · invented numbers before training

Position 0 only (what comes after 'the'?) token id logit the 0 0.40 ← currently highest cat 1 0.10 ← true target (too low) sat 2 0.30 on 3 0.05 mat 4 0.15 Logit = raw score from the LM head. Not a probability yet. Can be any real number.

Teaching point Before training, the model may prefer repeating the. Training must raise the logit for cat relative to the others.

Say next Say: 'Five scores, one per vocab row. Softmax turns them into a lottery. Cross-entropy only cares about the probability on the target row (cat).'

S1 · 5 Loss

5 · Softmax, then loss = −log p_target

Book Ch5.1.2 · do the arithmetic out loud

Softmax: exp(z_i) p_i = ───────────────── Σ_j exp(z_j) Rough p (from those logits): the 0.26 · cat 0.19 · sat 0.23 · on 0.18 · mat 0.19 Target = cat → p_target ≈ 0.19 L = − log(0.19) ≈ 1.66 (high = bad) If later p_cat = 0.80: L = − log(0.80) ≈ 0.22 (low = good) Average L over positions, then over the batch.

PyTorch note cross_entropy(logits, targets) expects RAW logits. It does log-softmax inside. Do not softmax first, then call it.

Say next Compute with them: 'p=0.19 → loss about 1.66. Raise p to 0.8 → loss about 0.22. Lower is better. That is the whole learning signal.'

S1 · 6 Train≠Gen

6 · Training ≠ generation (same model, different inputs)

This contrast prevents half the confusion in Ch5

TRAINING

Position t gets the TRUE token from the corpus. Compare prediction to the TRUE next token. Early mistakes do NOT enter the history. Every position is a clean score.

GENERATION

Position t gets the token the model JUST PRODUCED. No target. Sample or argmax instead. A bad early sample stays in the history and pollutes later steps.

Say next Say: 'Training always feeds the real prefix. Generation feeds its own outputs back in. That is why decoding settings matter after weights are fixed.'

S1 · 7 Curves

7 · Train loss vs validation loss

Same formula · held-out text · read BOTH curves

step train_loss val_loss 0 4.10 4.12 near random 100 2.40 2.55 both falling ✓ 500 1.10 1.45 still OK 1000 0.40 1.80 train↓ val↑ ← overfit

What to say at step 1000 Training loss kept falling. Validation loss rose. The model is fitting train-set quirks, not structure that transfers. That is overfitting.

Say next Say: 'Never celebrate train loss alone. If val rises while train falls, stop or regularize. Ch5 plots both for this reason.'

S1 · 8 Loop

8 · One training step

Walk the six boxes left to right · then say 'repeat for every batch'

1

Batch

shifted inputs + targets

2

Forward

logits = model(x)

3

Loss

cross- entropy

4

Backward

gradients

5

Step

AdamW update

6

Zero

clear grads

Say next Say: 'Each batch raises p(true next) a little. One epoch = one pass over the training text. After enough steps: model.eval() and generate.'

S1 · 9 Decode

9 · Decoding knobs · weights stay fixed

After training, given 'the cat sat on' suppose p(mat)=0.70

Greedy Always take argmax. Here: mat. Deterministic. Can get repetitive on real models.

Temperature T p = softmax(z / T) T < 1 → sharper (safer) T = 1 → unchanged T > 1 → flatter (wilder) Does NOT retrain anything.

Top-k Keep only the k largest logits. Mask the rest to −∞. Then softmax + sample. k=2 → only mat and (say) sat can be chosen.

Say next Say: 'Temperature and top-k change how we SAMPLE. Training still used plain cross-entropy on the full vocabulary. Chat UIs expose these as inference settings.'

S1 · 10 Close

10 · S1 in one pass (before CPT)

Point down the row · then run the exit ticket

Shift targets = tokens[1:]

Loss L = −log p(true next)

Train true prefixes only

Generate own outputs back in

Val watch both curves

Decode T / top-k after weights

S1 · Glossary

Glossary · say the term, then the plain line

Keep this frame visible while teaching

Pretraining next-token training on unlabeled text until you have a base model.

Input / target Same token ids; targets shifted one step left

Logit Raw score per vocab row before softmax.

Softmax Turns logits into probabilities that sum to 1.

Cross-entropy L = −log p of the true next token.

Overfitting Train loss falls while validation loss rises

Temperature Rescale logits before softmax at generation time.

Top-k Keep only k highest logits, then sample

S1 · Check

Exit ticket · say aloud

If these three land, move on

Prompt 1 From [the, cat, sat, on, mat], write inputs and targets. What is the target after 'the'?

Prompt 2 If p(cat)=0.19, is −log p high or low? What changes if p(cat)=0.80?

Prompt 3 In one sentence: what enters position t during training vs during generation?