Chalkline
Lesson coverage · Evaluation design · Human-in-the-loop
The system read the lesson correctly, then made a video with none of it in.
ChalklineMy automated quality check approved it. So did a human, in seventy seconds.
- 0 of 7
- lesson items the system found, then left out of the finished video(Measured)
- 11 of 12
- beats of the video spent retelling the poem, none on the exercises(Measured)
- 70 seconds
- the human approval that let it through(Measured)
- Status
- In use at the school. Pro bono.
- My role
- Every architecture, evaluation, product and budget decision. I wrote none of the code by hand.
The six decisions
- 01
I benchmarked the agent against a video I had made by hand, not against a rubric.
My first attempt at an agentic video system failed on visual inconsistency. So I built the next one manually, one shot at a time, and made it the answer key: 8 shots, each split into three timestamped blocks, each block carrying six named fields and four mandatory audio tracks. The live system was producing 28 to 31 flat shots with none of that structure.
Result: a structural gap, not a polish gap(Measured). A rubric would have scored it fine.
Hand-built golden set
8 shots
10 seconds each(Measured), built by hand one shot at a time
3 timestamped blocks per shot
6 named fields per block
4 mandatory audio sub-tracks
What the live system was producing
28 to 31 shots
flat micro-shots per video(Measured)
No internal timestamped beats, no audio direction, no continuity bridges, and generic style language.
- 02
I audited one run end to end, and it had thrown the lesson away.
The lesson plan named five sub-topics. Stage 1 found and correctly tagged all seven of the lesson's exercises: the fill-in-the-blank pairs, prepositions of place, the ascending-order sets, the spelling correction, the crossword, the listening activity, the cultural extension. Then the planner spent all 300 seconds of runtime retelling the poem.
Result: 7 of 7 items understood, 0 of 7 delivered(Measured). Comprehension was never the problem.
- 03
Both my quality gates passed it, one automated, one human.
The critic ran a full redraft cycle, two calls to it and two back to the writer, then approved the same plan again. The human gate cleared the script in 1m10s(Measured). Neither was holding a list of what the lesson contained, so neither could see what was missing.
Result: I had been treating "the human approved it" as proof of correctness. It proves nothing without a reference to check against.
- 04
I banned silent dropping.
Every activity in the lesson brief must now either be assigned to a beat or explicitly marked with a reason for being skipped. Prepositions of place was a named concept with zero beats and the critic approved the plan anyway, so the critic must now fail exactly that case.
Result: the failure moves from invisible to logged. Silence is indistinguishable from correctness until somebody audits by hand, and in this project's history that happened once.
- 05
I split coverage from craft, and gave each its own critic.
A critic named "quality" had quietly narrowed to mean "narrative quality," and about the narrative it was right every time. I stopped widening it and specified a second single-purpose critic that checks one thing: did every concept get taught. Two independent passes, never one blended score.
Result: a run can now fail for the reason it actually failed. A blended score had been hiding the only axis that mattered.
- 06
I cut the one capability I could not govern, and never opened a code editor.
Lip-sync came out of the spec entirely, 13 stages down to 12. Narration runs as external voiceover instead: cheaper, removes a whole class of drift between clips, and hands me control of script and pronunciation. Spend is checked against a run-level cap before calls go out, not after.
Result: $0.58 against a $5.00 cap on the audited run(Measured). Stages 1 to 6 only. Video generation spend was never captured, so I am not quoting a per-video figure.
What I changed because of it
| Before the audit | After | |
|---|---|---|
| Coverage check | none. The critic scored narrative quality | a dedicated critic that fails any concept with zero beats |
| Dropping an activity | permitted, and silent | assigned to a beat, or marked with a reason |
| What the human sees at the script gate | the script | the script beside the list of what the lesson contained |
There is no after-measurement on this page. I found the failure, specified the fix and made it the project's top priority. A fix is not a result, and I am not going to present it as one.
What this does not claim
- One lesson, one audit. Everything here comes from a single end-to-end review. No held-out set, no repeat runs, no regression suite. Anyone quoting a pass rate off this should be able to show you the denominator, and I cannot.
- No before and after. No baseline exists for what a lesson video cost or took at this school beforehand, so there is no saving to quote and I have not invented one.
- The coverage fix is a specification. It was the top priority six weeks after the audit and I have not published a measurement of it working.
- The school and its founder are unnamed by agreement.
The long version · about 15 minutes
The twelve stages, the golden set methodology, the full audit, the seven human gates and what each one blocks, the failure analysis and the delivery governance. Where I would want an evaluation team to spend their time.
Read the long version
Next case study
Mapping Studio
Then I tested the quality metric I had built to prove it worked. It could settle two arguments out of eighty-one, so this ships as a review console rather than an automated pipeline.
Read it