AI3 Loss curves, epochs and overfitting · Lab

1 The idea

Every step adjusts the numbers to lower the error on the training set, but what you want is a good result on data the model has never seen. So you keep some data aside to check (validation). If the training error falls while validation rises, it has started to overfit: it is memorising noise.

Capacity is how much the model can bend; here it is the polynomial degree. Degree 1 is a straight line that can’t follow the vignette (underfit). Degree 10 goes through almost every training point but wiggles between them (overfit).

RMSE = √ mean((model(x) − y)²) · train ↓ + validation ↑ = overfit

2 Hands-on

Task: pick a capacity and train so the validation error ends below 0.075

The dashed white line marks the best validation step: that is the checkpoint worth keeping

SkyVFX Lab · AI3interactive · 1001

3 Deep dive

The words you need

A step is one update of the numbers. An epoch is one full pass over the dataset. A batch is how many examples go in together in one step.

So total steps = epochs × dataset size ÷ batch size. For example 2,000 epochs, 8 image pairs and a batch of 4 give 4,000 steps.

Term	Meaning
step	One update of the weights
epoch	One full pass over the data
batch size	Examples per step
checkpoint	The weights saved at one step

Reading the two curves

Training error almost always falls, because the model is tuned for exactly that data. Validation error stands in for unseen data. At first both fall together; once the model starts memorising noise, validation stops falling or starts to rise.

In the lesson, degrees 2–7 reach a validation error around 0.07, while degree 10 gets a tiny training error (about 0.006, below the 0.05 noise) and a validation error around 0.19. That is what overfitting looks like.

CopyCat is meant to fit one shot

In general ML, overfitting is something to avoid. CopyCat is usually meant to be good at one shot only, and that is a strength, because the frames of one shot are very alike.

What you still guard against is failing on the “new data” that matters: the other frames of the same shot that aren’t in the dataset. Always keep some aside to check.

Hold back 2–3 frames of the same shot for checking; keep them out of the dataset

Fixing underfit and overfit

Underfit: results stay soft or miss the target even on the example frames; train longer or use a larger model. Overfit: the example frames look great but others go wrong; add more varied frames or use an earlier checkpoint.

1Examples still poor = underfit
2Train longer / bigger model
3Examples good, others wrong = overfit
4Add varied frames / earlier checkpoint

4 In real work

In Nuke

While CopyCat trains, read the loss graph together with the contact sheets it writes to its output folder.
Contact sheets show random crops: input, the model’s result and Ground Truth side by side. Use them to check quality and that the pairs line up.
Run Inference with the .cat from different checkpoints on your held-back frames and pick the best by eye.
Use a Merge set to difference to compare results with hand-made frames and see where they differ.

Common mistakes → the fix

✗ Trusting the loss graph alone (see AI2)✓ A low loss can mean it memorised the examples; look at frames it didn’t train on.
✗ Training as long as possible and always using the last result✓ An earlier checkpoint can be better; compare a few on held-back frames.
✗ Holding nothing back: every hand-fixed frame goes into the dataset✓ Keep 2–3 aside so you have a measure the model has never seen.

Before you send the shot

There are held-back check frames outside the dataset
Loss graph read together with contact sheets
At least 2 checkpoints compared on held-back frames
Held-back frames pass by eye at full size

5 Check yourself

One question to check yourself

Training error keeps falling, but validation starts rising after step 500. What does it mean?

What you learned

🔒 Answer the question above and this card opens