Put a Remotion animation on a green screen (tracked screen insert)

A tracked screen insert fills a coloured screen in your footage — typically a phone or laptop showing a plain green screen while someone holds or uses it — with an animation, so that the animation moves, turns and tilts with the device as if it were really on the screen. Rounded corners, the camera notch and a hand passing in front of the screen are kept.

Yes, ReelBolt can do this, and the animation is a Remotion scene that ReelBolt renders itself. There is no separate "render" step: the motion-graphics AI (MotionGraphicsPlanner, or the director of a GraphicsRoom panel) writes the Remotion scene, renders it, and names it in its answer. The VideoCompile step then pins it into the tracked screen.

Put one of your own videos on the screen (no AI needed)

When you already have the clip that should play on the screen — a promo, a screen recording, an image — you don't need any AI step at all. Upload both videos to the project and build a two-step workflow:

1. VideoAnalyze with the green-screen footage as its source and Detect tracked screen-insert regions switched on. Pick only the green-screen footage here — not the clip you want on the screen. Every source of this step is footage for the edit; a clip listed here ends up in the finished video as a scene of its own.
2. VideoCompile with Keep every source whole (no edit) on, Enable tracked screen inserts on, and your clip picked under Insert content (your file).

In JSON, step 2 looks like this:

{
  "version": 1,
  "decision": {"from": "Previous"},
  "analysisStepOrder": 1,
  "keepWholeSources": true,
  "enableInserts": true,
  "insertContentProjectFileId": "PUT-YOUR-CLIP-FILE-ID-HERE",
  "insertFit": "Auto"
}

• keepWholeSources keeps the footage uncut, so no editor agent is needed (decision is then ignored).
• insertContentProjectFileId is the clip shown on the screen. It must be a video or image in this project.
• insertFit decides what happens when the clip's shape differs from the screen's: "Auto" (default) keeps your clip's shape and adds black bars when it is very different (a landscape video on an upright phone), "Cover" crops to fill the screen, "Stretch" distorts it to fit.

The tracker tunes itself to each video: it tries several settings on your footage and keeps the one whose outline sits exactly on the screen's edge from frame to frame. Open the result in the Editor to see the tracked screen drawn over the video, its confidence, whether it was cut to the screen's exact shape ("matted"), and which settings the tracker chose.

Different videos on different screens

One edit can show different files on different screens, for example a phone recording on the phones and a desktop recording on the laptop. In the compile step, under Screen inserts, click Add content for other screens and, for each extra file, say which screens it is for:

• Only on screens in clip number: the clip's place in the first step (1 = the first clip).
• Only on screen: one screen's id (such as r2), shown in the first step's result.
• Leave both empty to use the file on every screen not covered otherwise.

A screen that matches nothing extra shows the file picked under Insert content (your file). In JSON this is "insertContents": [{"projectFileId": "…", "sourceIndex": 1, "fit": "Contain"}] (sourceIndex counts from 0). Each extra file can have its own fit.

When your screen recording fades to black

Screen recordings often go black for a moment between screens. On a filmed phone that looks as if the phone switched off. ReelBolt finds these black moments in your file and, by default, keeps showing the last picture until the recording continues, so the screen never goes dark and the timing of the recording stays the same. Under When your video fades to black you can instead Skip the black part (the recording gets a little shorter) or Leave it as it is ("insertContentDips": "Freeze", "Cut" or "Keep"). The compile step's result lists the black moments it found under inserts.appliedInserts[].contentDips. Videos made by the AI for the screen, and pictures, are never changed.

How a tracked screen insert works, in one picture

A tracked screen insert takes four steps, each with one job:

1. VideoAnalyze finds the coloured screen and follows its four corners through every frame ("detectInsertRegions": true). Each screen it follows is offered to the AI as a short id: r0, r1, ... with plain hints (how confident, how big, moving or still, its shape, flat or curved).
2. VideoStoryEditor (an Agent step) decides which parts of the footage to keep. The compile step cannot run without this decision. To keep the whole clip, say so in the request text.
3. MotionGraphicsPlanner (an Agent step) — or a GraphicsRoom step — chooses which offered screen to fill, writes and renders a Remotion scene shaped like that screen, and returns it in its answer's inserts list: regionId (for example r0), renderedAssetStorageKey (the saved render, filled in by the agent itself) and reason.
4. VideoCompile reads that answer through its graphicsPlan setting and, with "enableInserts": true and "mode": "Reencode", warps the scene onto the moving screen frame by frame and cuts it to the screen's exact shape.

The field that says what fills the screen is the plan's inserts[].renderedAssetStorageKey, produced by step 3 — a scene it rendered, or, when your request names one of your own videos, that file's id. To skip the AI entirely, see Put one of your own videos on the screen.

Prerequisites for a tracked screen insert

Before building a tracked screen insert workflow, you need:

• Footage with a plain, evenly coloured screen, uploaded to the project. Green works best; blue and magenta are also supported. The screen should be at least a small part of the frame and visible for at least 1 second. A real phone or laptop showing a full-screen green image is ideal.
• A working chat AI provider (for the editor and the motion-graphics AI). The motion-graphics AI writes and renders code, so a capable model helps.
• The Remotion sandbox service running (it is part of the standard ReelBolt installation; the promo templates use it too).
• Optional: a transcription provider, if the footage has speech the edit should follow.

Nothing in this recipe costs money beyond your AI provider's normal usage.

Build the tracked screen insert workflow

The quickest way is to start from the Footage edit with titles and callouts template (video-derush-edit-graphics), which already has the four steps in the right order plus a review step, and change three settings. You can also ask the assistant: "Set up a tracked screen insert workflow for my green-screen phone video".

The full workflow, step by step:

#	Label	Step type	Agent	Context
1	Analyze source video	VideoAnalyze	VideoTransform	—
2	Decide which spans to keep	Agent	VideoStoryEditor	PreviousStepOnly
3	Plan graphics and screen inserts	Agent	MotionGraphicsPlanner	FullWorkflow
4	Compile edited video with insert	VideoCompile	VideoTransform	—
5 (optional)	Review edit quality	ReviewLoop	VideoReviewAgent	FullWorkflow, back to step 2, minScore 8, maxIterations 3

Set steps 2, 3 and 5 to "cacheMode": "Never" so each run makes fresh decisions (step 3 is never reused anyway, because it renders).

Step 1 settings (videoAnalyzeConfigJson) — pick your uploaded video as the source:

{
  "version": 1,
  "source": {"kind": "ProjectFile", "projectFileId": "PUT-THE-FILE-ID-HERE"},
  "detectInsertRegions": true,
  "insertRegionColor": "green",
  "maxInsertPlatesPerFrame": 1,
  "measureInsertCurvature": true,
  "solveInsertChromaKey": true
}

• detectInsertRegions is the switch that matters; it is off by default.
• insertRegionColor — "green" (default), "blue" or "magenta".
• maxInsertPlatesPerFrame — raise to 2 or more when two screens appear at the same time.
• measureInsertCurvature and solveInsertChromaKey are already on by default; they make the corners precise, handle curved screens, and keep rounded corners, notches and hands in front.
• Add "emitOverlayPlacements": true only if you also want titles and lower-thirds.

Step 4 settings (videoCompileConfigJson):

{
  "version": 1,
  "decision": {"from": "Step", "stepOrder": 2},
  "analysisStepOrder": 1,
  "mode": "Reencode",
  "graphicsPlan": {"from": "Step", "stepOrder": 3},
  "enableInserts": true,
  "insertSurface": "Auto",
  "enableInsertMatte": true,
  "transitionPolicy": "Off"
}

• graphicsPlan must point at the planner (or graphics room) step. Inserts are read from that same plan; without it nothing is inserted.
• enableInserts turns the feature on. enableGraphics is not needed for inserts; add "enableGraphics": true only if you also want the planner's titles and lower-thirds.
• mode must be "Reencode" (the default). "StreamCopy" makes the step fail with INSERTS_REQUIRE_REENCODE.
• insertSurface — "Auto" (default) chooses a flat or curved warp from what was measured; "Planar" always flat; "Mesh" always curved (for testing only).
• enableInsertMatte (default true) cuts the scene to the screen's exact shape.
• "transitionPolicy": "Off" keeps plain cuts. The template uses "Auto", which can add dissolves between cuts; an insert is trimmed away from any dissolve, so with "Auto" the insert can start late or disappear near a cut.

If you start from the template: in step 1 add "detectInsertRegions": true (and keep "emitOverlayPlacements": true only if you want titles too); in step 4 add "enableInserts": true, optionally remove "enableGraphics": true, and consider "transitionPolicy": "Off". Its step 4 already has "graphicsPlan": {"from": "Step", "stepOrder": 3}.

Tell the AI what to show on the tracked screen

The motion-graphics AI decides what the screen shows, so tell it. When you click Run, type a brief such as:

Keep the whole clip. In the phone screen (r0), show an animated mockup of our app's home screen: dark background, our logo fading in at the top, then three cards sliding up. Brand colour #6C5CE7. No titles or lower-thirds.

Good requests:

• Say keep the whole clip (or what to cut). This instruction is for the editor in step 2.
• Describe the content of the screen: what appears, colours, text, motion.
• Say whether you also want titles or lower-thirds.
• Mention the screen by its role ("the phone screen") if there may be several.

What the AI does with it: it writes a small Remotion scene in the sandbox, shaped to the screen's measured proportions (for example a tall portrait scene for a phone held upright), renders it as a normal solid video, and puts the result in its answer's inserts list. It can also read your project files, so a brief uploaded to the project can be referred to by name.

Run the tracked screen insert workflow and check the result

Run the workflow and follow it on the run page. What to check, step by step:

• Step 1 — the progress messages say "Tracking insert regions" and, per screen found, something like "Insert region r0: flat, silhouette key ... (agreement 92%)". In the step result, the meta.insertTracking.regions count should be at least 1 and meta.offeredInsertRegionIdCount should be the same number. The view.insertRegions list shows each screen's conf (high/medium/low), size, motion, aspect and surface (Flat/Curved).
• Step 2 — the editor's keep list should cover the part of the clip where the screen is visible.
• Step 3 — the result's inserts list should contain an entry with regionId and a non-empty renderedAssetStorageKey. The step's Output Video player shows the scene the agent rendered. This step is slow: writing, checking and rendering code can take many minutes.
• Step 4 — the result's inserts section reports appliedInsertCount and, for each insert, whether it was flat or curved (surface, mesh size) and matted. Anything skipped is listed in droppedInserts with a reason. Watch the Output Video.

Why a tracked screen insert was not applied

When the screen still shows plain green in the final video, the VideoCompile step's result says why. The reasons, in plain words:

Reason shown	What it means	What to do
"No GraphicsPlan configured"	Step 4 has no graphicsPlan and no insertContentProjectFileId.	Set "graphicsPlan": {"from": "Step", "stepOrder": 3}, or pick your own file as insert content.
content_not_in_project / content_not_video_or_image	The chosen insert content is not a video or image of this project.	Pick a video or image uploaded to this project.
empty inserts, no dropped entries	The planner returned no inserts (no screen offered, or it chose none).	Check step 1 found a screen; ask for the insert explicitly in the request.
unknown_region_id	The planner named a screen id that step 1 did not offer in this run.	Check step 1's offeredInsertRegionIdCount; see the next section about large views.
confidence_below_threshold	The screen was tracked with low confidence (below minInsertConfidence, default 0.5).	Use footage with a more even, better-lit screen, or lower minInsertConfidence carefully.
invalid_asset_storage_key	The planner's scene is missing, invented, or from another run.	Run again; the scene must be rendered in the same run.
asset_download_or_probe_failed	The rendered scene could not be read.	Run again.
cut_away	The part of the clip with the screen was cut out by the edit.	Tell the editor to keep that part.
window_consumed_by_transition_overlap	A dissolve covered the whole insert.	Use "transitionPolicy": "Off".
max_inserts_exceeded	More inserts than maxInserts (default 3).	Raise maxInserts.
duplicate_region_id	The same screen was chosen twice.	Only the first is used; nothing to do.
unavailable: true	The video tools on the server lack a required feature.	Ask your administrator.

Pitfalls with tracked screen inserts

Common reasons a tracked screen insert goes wrong:

• Step 1's source is empty. Templates leave the file unselected; the run fails with SOURCE_UNRESOLVED until you pick the video.
• The screen list was left out because the description got too big. Step 1 keeps the description it hands to the AI under maxOutputChars (24000 characters). On long or busy footage it may leave out the whole insertRegions list to fit; then step 3 cannot see any screen. Check that meta.offeredInsertRegionIdCount is above 0. If it is 0 while meta.insertTracking.regions is above 0, raise maxOutputChars (for example to 40000) or set "visualDetail": "None" in step 1.
• The editor cut the screen out. Ask it to keep the whole clip, or the part with the screen.
• No brief. Without a brief in the run's confirmation box, the planner guesses what to show (or shows nothing). Describe the content.
• Uneven or reflective screens. Strong glare, a very dark green, or green objects elsewhere in the shot lower tracking confidence. A brighter, evenly lit screen tracks best.
• Colour grade and matte together. The screen's exact colour is measured on the original footage, but a colour grade (enableColorGrade) is applied before the insert. A strong grade can shift the screen's colour so the cut-out shape is less exact. Leave colour grading off for insert shots, or use a subtle look.
• One insert per stretch of screen. If the edit cuts the screen's appearance into two separate pieces, the insert appears only in the first piece.
• What is not handled: relighting the scene to match the room, motion blur on fast movement, and screens that are not a plain colour (a screen showing real content cannot be tracked).

Variations of the tracked screen insert workflow

Ways to adapt a tracked screen insert workflow:

• Panel of artists instead of one planner. Replace step 3 with a GraphicsRoom step: {"version": 1, "view": {"from": "Step", "stepOrder": 1}}. Its InsertArtist seat argues about which screens to fill and what to show; the director renders the scenes in its final call. Step 4's graphicsPlan then points at the room step. Point the room's view at step 1, not at Previous.
• Titles too. Add "emitOverlayPlacements": true to step 1 and "enableGraphics": true to step 4. The same plan carries both titles (overlays) and screen inserts (inserts).
• Two screens at once. Set "maxInsertPlatesPerFrame": 2 in step 1; each screen gets its own id (r0, r1) and its own scene.
• Blue or magenta screen. Set "insertRegionColor": "blue" or "magenta".
• Several clips. VideoAnalyze can analyse several videos together (sources list); inserts work when the edit combines clips.

Checking green-screen tracking only (no insert)

To see whether ReelBolt can track your screen before building the whole workflow, make a workflow with just the VideoAnalyze step from above and run it. The step uses no chat AI; speech-to-text still runs if a transcription provider exists (add "transcription": "Off" to skip it). On the step's result:

• Progress messages list each screen found, whether it reads flat or curved, and how well its exact colour was found ("agreement" percentage).
• meta.insertTracking shows whether tracking ran (applied), whether it hit a problem (degraded), and how many screens were found (regions).
• view.insertRegions shows each screen's id, confidence, size, motion, shape (aspect) and surface.
• The full corner-by-corner tracking is in the step's detailed record (data view).

To see a composite without any AI, add the two-step VideoCompile from Put one of your own videos on the screen with any uploaded clip as the insert content. For troubleshooting, insertSurface "Planar" or "Mesh" on the compile step forces the flat or curved warp, and "enableInsertMatte": false turns off the exact-shape cut-out.