1 The idea
Besides over there are merges that cut with alpha: in = A where B is, out = A where B isn’t, mask = B where A is, stencil = B where A isn’t, atop = A over B only where B has something.
They make holdouts, split images apart or limit effects. Switch the operation and look.
2 Hands-on in the mini compositor
3 Deep dive
All five in one table
Capital A and B are each input’s colour; lowercase a and b are their alphas. The examples use this lab: A = robot, B = ellipse shape.
| op | Formula | What it keeps |
|---|---|---|
| in | A·b | Robot inside the ellipse |
| out | A·(1 − b) | Robot outside the ellipse |
| mask | B·a | Ellipse only where the robot is |
| stencil | B·(1 − a) | Ellipse with a robot-shaped hole |
| atop | A·b + B·(1 − a) | Robot over the ellipse, only inside it |
Who does the cutting
in and out keep A and use B’s alpha as the scissors. mask and stencil keep B and use A’s alpha as the scissors.
So they are mirror pairs: “robot in ellipse” gives the same pixels as “ellipse masks robot” with the inputs swapped, and out pairs with stencil the same way. Pick the one that lets B stay on the main comp stream without swapping pipes.
Read in, out, atop as “A ... B”; mask and stencil are what A does to B
Real numbers from Nuke
These were measured in Nuke 17 with A = 0.3 (alpha 0.5, premultiplied) and B = 0.4 (alpha 0.8). Work them out and compare with the lab. Notice atop’s alpha equals b: B’s outline doesn’t change.
| op | rgb | alpha |
|---|---|---|
| in | 0.24 | 0.4 |
| out | 0.06 | 0.1 |
| mask | 0.2 | 0.4 |
| stencil | 0.2 | 0.4 |
| atop | 0.44 | 0.8 |
Where each one earns its place
stencil cuts a hole for something in front (a holdout) or removes junk with a garbage matte. in limits an effect to an object, such as dust only on the robot. atop lays a texture or extra light onto the robot without spilling outside its outline.
Over itself is a stencil followed by a plus: A + B·(1 − a). Understand that one line and most dark-edge and glowing-edge problems explain themselves.
4 In real work
In Nuke
- Merge: set operation to in, out, mask, stencil or atop. A is the input labelled A; B is the main stream.
- Use Dots to bring the roto or matte in from the side so the B stream runs straight down.
- When you only need the alpha, not the colour, use Copy (alpha → alpha) or a ChannelMerge on the alpha channel.
- Check the alpha channel as well as the rgb.
Common mistakes → the fix
- ✗ Using in when you meant to keep B✓ in keeps A. To keep B by A’s alpha, use mask.
- ✗ Swapping A and B to make an op work, so the main comp ends up on A✓ Pick the mirror op instead (in ↔ mask, out ↔ stencil) and keep B as the main stream.
- ✗ Using a matte whose data is in rgb, not alpha✓ These ops read alpha. Move the data into alpha first with a Shuffle or Copy.
Before you send the shot
- For each Merge you know whether it keeps A or B, and whose alpha cuts
- B is still the main comp stream
- The matte really is in the alpha channel
- The result’s alpha has been checked
5 Check yourself
You want only the part of the robot outside the ellipse (A = robot). Which op?
🔒 Answer the question above and this card opens
1. A = robot, B = ellipse. You want the ellipse with a robot-shaped hole. Which op?
2. A = 0.3 (alpha 0.5), B = 0.4 (alpha 0.8). What alpha does atop give?
Where the numbers come from: Computed in your browser. Saturation, Multiply, Clamp, Invert, Keymix, Copy and Grade match Nuke 17.0v2 (checked). Blur, Defocus, Glow, Sharpen, ChromaticAberration, Vignette, LensDistort, Grain, Erode, Toe, the Keyer and Despill are teaching versions, not identical to Nuke’s nodes. Transform here is also a teaching version: bilinear filtering, no concatenation, cropped at the frame edge. Images are our own Blender renders (the green screen is fully code-built; the desert scene uses Gaea 2.3 hills, a Poly Haven CC0 HDRI sky, Poly Haven CC0 rocks and grass, Quixel sand and cliff textures and a Mixamo rig for the robot, with Quixel and Mixamo used in the renders only).