Render real clips in a nightly end-to-end check#115
Merged
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
nmbrthirteen
force-pushed
the
audit-7-onboarding
branch
from
July 13, 2026 11:53
8be3b91 to
943c64c
Compare
nmbrthirteen
force-pushed
the
audit-8-nightly
branch
from
July 13, 2026 11:53
574efa6 to
e18b096
Compare
nmbrthirteen
force-pushed
the
audit-7-onboarding
branch
from
July 13, 2026 12:11
943c64c to
b4a5de0
Compare
nmbrthirteen
force-pushed
the
audit-8-nightly
branch
from
July 13, 2026 12:11
e18b096 to
ff34dd8
Compare
nmbrthirteen
force-pushed
the
audit-7-onboarding
branch
from
July 13, 2026 12:17
b4a5de0 to
75c107b
Compare
nmbrthirteen
force-pushed
the
audit-8-nightly
branch
from
July 13, 2026 12:18
ff34dd8 to
ed80269
Compare
CI only ran unit tests, so nothing in the pipeline ever produced a clip. A change could break rendering for every user and still go green: a recent TypeScript upgrade would have done exactly that. The harness builds a synthetic source, drives the real pipeline, and asserts the artifact: streams, duration against the requested range, and framing. It runs per caption style on all three platforms.
nmbrthirteen
force-pushed
the
audit-7-onboarding
branch
from
July 13, 2026 13:52
75c107b to
0d796bb
Compare
nmbrthirteen
force-pushed
the
audit-8-nightly
branch
from
July 13, 2026 13:52
ed80269 to
0282f6c
Compare
Every assertion in the e2e check ran on the container, not the picture: a path came back, the file was over 20KB, it had both streams, the duration was right, the frame was taller than it was wide. Deleting the words so the caption stage never runs, and Remotion is never invoked, printed seven `ok` lines and exited 0. The one regression the check exists to catch was the one it could not see. Render the clip twice now, with words and without, and diff the frames. Captions sit in the bottom half in all four styles, so the caption band must differ from the caption-free render on every sampled frame while the top of the frame stays put. Measured coverage of that band is 1.4-11.9% across the styles against a 0.4% floor, and 0.0% when the words are removed. Also: crop with the product's default `speaker` strategy so YuNet and the reframing path are exercised instead of the center-crop shortcut, cap the job at 45 minutes so a hung headless Chrome cannot sit on a runner for six hours, and upload the rendered clips and sampled frames when the job goes red.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI only ran unit tests, so nothing in the pipeline ever produced a clip. A
change could break rendering for every user and still go green: a recent
TypeScript upgrade would have done exactly that.
The harness builds a synthetic source, drives the real pipeline, and asserts
the artifact: streams, duration against the requested range, and framing. It
runs per caption style on all three platforms.
Part of the product audit. Stacked on
audit-7-onboarding, so review only this PR's diff and merge in order.