Home · Lab · Keying Lab

K5 Fixing edges with alpha gamma

The first edge fix after despill

Keying Lab · ~5 min · Free

1 The idea

Before eroding, try a gamma on the alpha just before the Merge (a Grade with channels = alpha). Below 1 thins the edge so more background shows; above 1 thickens it. It keeps far more detail than an erode.

In this lab the Grade sits before the Premult, so the edge moves towards the background colour: a brighter background makes it brighter, a darker one darker. Move the Grade after the Premult, right before the Over, and a lower alpha can only make the edge brighter or leave it. Watch the robot’s arm edges.

2 Hands-on in the mini compositor

SkyVFX mini compositor · K5640×360 · float · 1001

3 Deep dive

Gamma only touches the edge

Out = pow(a, 1/g). 0 and 1 always stay where they are, so the solid inside and the empty background don’t change. Only the values in between, the edges, hair and thin lines, move. g below 1 lowers them; g above 1 raises them.

alphag = 0.5g = 0.8g = 1g = 1.25g = 2
0.10.010.0560.10.1580.316
0.40.160.3180.40.480.632
0.50.250.420.50.5740.707
0.90.810.8770.90.9190.949

Small moves, 0.8–1.25, already show

Against erode on the antennas

The green-screen robot’s antenna wires are so thin they only cover part of a pixel. Say a pixel on the wire reads an alpha of about 0.4. An erode of −1 takes the whole line to 0, while a gamma of 0.8 lowers it to 0.318: thinner, but still all there.

This is why alpha gamma always comes before erode: it changes how heavy the edge is without changing the shape.

Where you put it changes the result

Before the Premult (as in the lab): the edge colour is multiplied by the new alpha, so the edge slides towards the background colour. A brighter background makes it brighter, a darker one darker.

After the Premult, right before the Over: the edge RGB is fixed. Over = A + B × (1 − a), so lowering a only adds more background, and the edge always gets brighter or stays the same. It is a quick fix for a dark rim around the robot against a bright sky. A gamma on the B side does nothing for this edge.

Positiong < 1 makes the edge
Before PremultMove towards the background colour
After Premult, before OverBrighter or the same

Halo fix order in a real shot

Grey or dark edges mostly come from the edge colour, not the alpha, so start with despill. Then an alpha gamma limited by a roto to the problem area. Only if that isn’t enough, edge extend, and erode last of all.

Watch motion blur and hair: a low gamma fades them faster than a normal edge. Check the whole shot, not one frame.

  1. 1Despill
  2. 2Alpha gamma (with roto)
  3. 3Edge extend
  4. 4Erode

4 In real work

In Nuke

  1. Grade: channels = alpha, adjust gamma, placed just before the Merge (over)
  2. Plug a Roto into the Grade’s mask input so only the problem edge is hit
  3. Check with the Viewer gain and gamma both up and down, alongside the a channel
  4. If that isn’t enough: EdgeExtend, then FilterErode as the last step

Common mistakes → the fix

  • ✗ Putting the gamma on the background’s (B) alpha to fix the edge✓ The gamma belongs on A’s alpha, which decides how much background shows through
  • ✗ Gamma on the whole robot until hair and motion blur fade✓ Limit it with a roto and check the whole shot
  • ✗ Eroding before trying an alpha gamma✓ despill → alpha gamma → edge extend → erode
  • ✗ Not knowing whether the Grade is before or after the Premult, then wondering why the edge behaves oddly✓ Before Premult the edge moves towards the background; after it, a gamma below 1 can only make the edge brighter or leave it. Pick the one that fits the problem

Before you send the shot

  • No grey or dark line on the edge with the Viewer gain up
  • Thin lines (antennas, hair) survive the gamma
  • The alpha Grade is limited by a roto if the problem is local
  • Motion blur doesn’t fade oddly anywhere in the shot
  • Order: despill → alpha gamma → edge extend → erode

5 Check yourself

One question to check yourself

What does an alpha gamma below 1 do to the edge?

What you learned

🔒 Answer the question above and this card opens

More questions

1. An antenna’s alpha is 0.5. With gamma 0.5, what does it become?

2. A gamma of 0.8 on the robot’s alpha after the Premult, right before the Over. What happens to the edge?

Where the numbers come from: Computed in your browser. Saturation, Multiply, Clamp, Invert, Keymix, Copy and Grade match Nuke 17.0v2 (checked). Blur, Defocus, Glow, Sharpen, ChromaticAberration, Vignette, LensDistort, Grain, Erode, Toe, the Keyer and Despill are teaching versions, not identical to Nuke’s nodes. Transform here is also a teaching version: bilinear filtering, no concatenation, cropped at the frame edge. Images are our own Blender renders (the green screen is fully code-built; the desert scene uses Gaea 2.3 hills, a Poly Haven CC0 HDRI sky, Poly Haven CC0 rocks and grass, Quixel sand and cliff textures and a Mixamo rig for the robot, with Quixel and Mixamo used in the renders only).