1 The idea
A model can be trusted only across the range of values it has seen. Outside that range it guesses, and it can guess badly. The training loss only tells you how well it fits the examples, not whether it works on the whole shot.
The grade here is a curve (contrast), not a straight line, and the model is a tiny neural net (3 → 8 → 3). Train on the ground and rock first, look at the error map in the sky, then add patches and train again.
2 Hands-on
Task: pick patches and train until the whole-frame error is below 0.012
Changing patches restarts training from zero, just like changing the dataset
3 Deep dive
A model is only trustworthy where it has seen data
Between the values in the dataset a model usually fills the gaps well (interpolation). Beyond the edge of the dataset it has to guess outward (extrapolation), and that can go badly wrong. In the lesson the sand and rocks are warm and dark, with blue below about 0.18, while the cloudy sky’s blue reaches about 0.27, so training on the ground alone breaks the sky.
| What the dataset should cover | Example in a shot |
|---|---|
| Brightness range | Frames in shade and frames in sun |
| Colour and light | Lights changing colour, flashing light |
| Pose and angle | Facing camera, profile, looking down |
| Size in frame | Far from and close to camera |
| Motion blur and focus | Fast moves, soft-focus moments |
How to pick the frames
Scrub the whole shot first and look for the frames that differ most, not the easiest ones. Consecutive frames usually look almost identical and add little new information.
Start with a few spread-out frames, train, and look at the whole shot. Wherever it breaks, hand-fix that frame, add it to the dataset, and repeat until it passes. Keep 2–3 hand-fixed frames out of the dataset as check frames (see AI3).
- 1Scrub the shot
- 2Pick the most different frames
- 3Train
- 4Inference the whole shot
- 5Broken frame → add to dataset
The frames the model fails on are your next dataset
Low loss, broken picture
In the lesson, training on the ground and rock drives the loss below 10⁻⁶, yet the whole-frame error is still about 0.017, about 1.4× the goal, and about 0.038 in the sky. The loss is measured on the examples only, so always look at results on frames outside the dataset.
4 In real work
In Nuke
- Scrub the shot in the Viewer and note the frames with the most different light, colour, pose or size.
- Use FrameHold to pull each chosen frame out for hand-fixing, then gather them with AppendClip on both the Input and Ground Truth sides.
- Train CopyCat, run Inference over the whole shot and hunt for broken frames.
- Fix the broken frames, add them to the AppendClips and train again (leave your 2–3 check frames out of the AppendClips).
Common mistakes → the fix
- ✗ Picking consecutive frames at the start because they are easy✓ Spread the frames across the shot and pick the most different ones.
- ✗ Delivering because the loss is low✓ Run Inference on the whole shot and check every section, especially those not in the dataset.
- ✗ Training longer to fix frames that aren’t in the dataset✓ Add frames like those to the dataset instead; training longer adds no new information.
Before you send the shot
- The dataset covers the brightest and darkest parts of the shot
- There are frames from the start, middle and end of the shot
- Different poses, angles and motion blur are represented
- Inference results checked on frames outside the dataset
5 Check yourself
The loss is very low, yet some frames of the shot break. What do you suspect first?
🔒 Answer the question above and this card opens
1. An actor walks from shade into full sun. You hand-fix the first 6 frames, all in shade. What happens?
2. Which frames should you add to the dataset in the next round?
Where the numbers come from: A 3 → 8 → 3 (tanh) neural net trained with real Adam on 16×16 patches, sampled every 2nd pixel (64 pixel pairs per patch). The whole-frame error is measured every 2nd pixel against the hidden grade (contrast around 0.18 with power 1.4 and a warm tint). In real work you have no “true grade” for the whole frame; you judge by eye and against hand-made frames.