1 The idea
If something in the plate is closer than the CG, the CG must be cut away there (a holdout), or it floats in front of what should hide it. Roto the near object and stencil it out of the CG before the over.
Here an ellipse shape stands in for a post in front of the robot. Pick the operation that removes the robot where the ellipse is.
2 Hands-on in the mini compositor
Task: set the "Merge holdout" operation so the robot is removed inside the shape
B = robot, A = shape: you want B outside A
3 Deep dive
You have to state the depth order
Merge over doesn’t know what is closer to the camera; it always puts A on top of B. If the plate has a post in front of the robot, over paints the robot across the post and the shot looks wrong at once.
The plate has no alpha for the post, so you build the post’s matte yourself and cut the robot away there before the over. Once the hole is cut, the post that is already in the plate shows through.
- 1Roto the post
- 2Stencil it out of the robot
- 3The cut robot over the plate
- 4The plate’s post reads in front
Where the matte can come from
Anything that says where the near object is: a roto of the post, a key if it stands in front of a coloured screen, depth if both plate and CG have it, or a matte rendered from 3D when the near object is CG too.
In keying, “holdout matte” usually means a different job: a rough roto that forces areas that must be solid to an alpha of 1 (combined into the alpha with max), while a garbage matte cuts away what you don’t want (stencil). Both tell the comp what is where.
The holdout’s edge
The post matte’s edge must be as soft as the post’s edge in the plate, and if the camera or post moves it needs the same motion blur. Same rule as the feather lesson.
A roto smaller than the post leaves a thin line of robot over the post’s edge. One that is larger bites too much out of the robot, and you see background through a gap that shouldn’t be there.
| Symptom | Cause |
|---|---|
| Thin line of robot on the post edge | Roto too small or too hard |
| Robot bitten, gap beside the post | Roto too large |
| Post edge chatters in playback | Roto not tracked, or no motion blur |
Cut everything that is behind the post
Our robot carries its shadow in alpha, so the stencil also cuts the shadow where the post covers it. That is right when the shadow is behind the post, like the robot.
Every CG layer behind the post needs the same holdout: robot, shadow, dust, glow. Miss one and that layer floats in front of the post.
Cut the CG before the over; don’t patch plate back on top afterwards
4 In real work
In Nuke
- Roto of the post → Merge (stencil) with A = roto, B = robot.
- The cut robot → Merge (over) onto the plate.
- Branch the roto with Dots to every CG layer behind the post.
- Moving camera: a Tracker on the post drives the roto; turn on the Roto’s motion blur to match the plate.
Common mistakes → the fix
- ✗ Putting the robot over the plate first, then patching the post back on top✓ With exactly the same plate and matte the maths comes out the same, but the patch breaks as soon as anything comp-wide (grain, defocus, lens effects) sits between the over and the patch, and every new CG layer needs patching again. Stencil the robot before the over so the holdout travels with the CG.
- ✗ Using the robot’s own alpha as the holdout matte✓ The robot alpha only describes the robot. The holdout must be a matte of the near object.
- ✗ Cutting the robot but forgetting the shadow, dust or glow layers✓ Every layer behind the post uses the same holdout.
- ✗ A hard-edged post roto in a panning shot✓ Feather it to the real edge and turn on motion blur to match the plate.
Before you send the shot
- No CG layer floats in front of the post
- The post edge is as soft and as blurred as in the plate
- No thin robot line on the post edge, no gap beside it
- Full playback: the holdout follows the post all the way
5 Check yourself
Where does a holdout matte come from in real work?
🔒 Answer the question above and this card opens
1. Why does cutting a hole in the robot make the post appear in front?
2. After the holdout, a thin line of robot sits over the post’s edge. Most likely cause?
Where the numbers come from: Computed in your browser. Saturation, Multiply, Clamp, Invert, Keymix, Copy and Grade match Nuke 17.0v2 (checked). Blur, Defocus, Glow, Sharpen, ChromaticAberration, Vignette, LensDistort, Grain, Erode, Toe, the Keyer and Despill are teaching versions, not identical to Nuke’s nodes. Transform here is also a teaching version: bilinear filtering, no concatenation, cropped at the frame edge. Images are our own Blender renders (the green screen is fully code-built; the desert scene uses Gaea 2.3 hills, a Poly Haven CC0 HDRI sky, Poly Haven CC0 rocks and grass, Quixel sand and cliff textures and a Mixamo rig for the robot, with Quixel and Mixamo used in the renders only).