1 The idea
Repetitive per-frame fixes such as marker removal, logo removal or beauty work are where CopyCat shines. You fix a few frames by hand as Ground Truth; the model learns from small crops that “when the surroundings look like this, the centre should be this”, then runs on every frame.
Here the model looks at 5×5 pixels at a time, while the surface slides and brightens through the shot. Train from one frame, then from frames spread across the shot, and watch the error row on frames the model never saw.
2 Hands-on
Task: leave less than 25% of the marker on unseen frames, with damage elsewhere under 0.012
Even when it passes, the error under the marker isn’t zero: the model fills in something plausible, not what was really hidden
3 Deep dive
How a patch model works
The model doesn’t look at the whole frame at once. It looks at a small area and answers what the pixel there should be. In the lesson that is a 5×5 window. CopyCat looks at a much larger area through many layers, but the idea is the same: the surroundings decide the centre.
Because it only sees a small area, the model works anywhere in the frame; wherever the marker moves, no tracking is needed. But if the surroundings don’t carry enough information, it can decide wrong.
- 1Crop around a pixel
- 2Model
- 3New centre value
- 4Repeat for every pixel, every frame
Under the marker is a plausible guess
The texture hidden by the marker isn’t in the image, so the model invents something that fits the surroundings. In the lesson the marker never drops to 0%, because the model can’t know what was really underneath. Real work is the same, so always check by eye.
Judge the fixed area in playback, not just on a still
Which jobs suit it
It suits fixes where the problem looks the same through the shot and has to be done on every frame. It doesn’t suit fixes that happen on a few frames only, or that differ so much there is no pattern to learn.
| Job | Good fit? |
|---|---|
| Face tracking markers all through a shot | Yes: repetitive, same look |
| A logo on a moving shirt | Often, if examples cover the folds |
| Skin beauty work | Often; pick frames across light and angles |
| Removing a passer-by over 10 frames | Rarely: short, and needs new background built |
4 In real work
In Nuke
- Pick frames spread across the shot, isolate each with FrameHold and fix it with RotoPaint to delivery quality.
- Gather the original plate frames with AppendClip into CopyCat’s Input, and the fixed frames, in the same order, into its Ground Truth.
- Train, then load the .cat into Inference on the full plate.
- Limit the result to the area being fixed with a Keymix (or a Merge with a Roto mask) so nothing else in the frame changes.
Common mistakes → the fix
- ✗ Hand-fixing just one frame, or consecutive frames (see AI2)✓ Pick frames that differ in light, position and size, spread across the shot.
- ✗ Using the Inference result over the whole frame as is✓ The model may nudge other areas slightly; mask the result to the fixed area.
- ✗ Expecting the real detail to come back under the marker✓ It only guesses to match the surroundings. If it must be true detail, use data from other frames (a clean plate) or hand paint.
Before you send the shot
- Ground Truth frames spread across the shot, delivery quality
- The result is masked to the fixed area
- In playback, the fixed area doesn’t flicker or swim
- Grain in the fixed area matches the rest of the plate
5 Check yourself
Which frames should you fix by hand as Ground Truth?
🔒 Answer the question above and this card opens
1. Why doesn’t a patch model need the marker tracked?
2. The marker is gone, but the fixed area looks oddly still in playback. Likely cause?
Where the numbers come from: A 25 → 12 → 1 neural net trained with real Adam on 5×5 windows (mean removed) from 8 procedural 64×64 frames we generate. The 25% and 0.012 limits are ours. Real CopyCat uses a much deeper network on larger crops (set by its crop size) and trains far longer.