Build guide

Answer real ops questions from kitchen CCTV.

Live $300 challenge, Jul 12-Sep 9. Your agent watches public CCTV-style kitchen videos and answers hidden operations questions under a $0.30 per-hour model/API cap: did staff wear caps, was food left at handoff, was a container sealed, where did the bottleneck start?

← See the challenge rules, sample sources & scoring
What you give

A repo with one command that reads clips + questions and writes structured JSON answers.

What matters

Correct answers, timestamps, not-visible handling, and staying inside the runtime/model budget.

Hard cap

$0.30 model/API cost per 60 minutes, 25 minutes, and a frame budget. A run over the cap does not rank.

The first working shape

Start with a simple pipeline. It should be easy to beat, but it makes the interface concrete:

# 1. sample frames cheaply
ffmpeg -i clip.mp4 -vf fps=1 frames/clip_%04d.jpg

# 2. answer from sampled frames + OCR + question prompt
python answer.py --videos clips --questions questions.json --out answers.json

# 3. log what you spent
cat run_log.json   # frames used, model calls, runtime, estimated cost

The baseline samples the whole clip cheaply, inspects likely windows with a vision-language model and OCR where useful, and writes JSON with evidence. It is deliberately beatable, but not by freeform captioning or guessing.

Open-source or local models are the recommended path. Cloud models can still qualify if every call is logged and the normalized cost stays under the cap.

What to optimize

This is NOT
  • A freeform captioning contest.
  • A spend race where you send every frame to the biggest model.
  • A manual labeling task. Hidden clips cannot be inspected by a human during evaluation.
  • A place to guess when the video does not show enough. Use not_visible.
It IS
  • Question-driven video search: find the moments that matter.
  • Real kitchen-ops monitoring: hygiene checks, handoff flow, sealing steps, and bottlenecks.
  • Temporal reasoning: before/after, first/last, duration, unattended time.
  • Messy video handling: low light, compression, occlusion, clutter, and camera angles.
  • Cheap enough to matter to a small kitchen using cameras it already has.

Question examples

  • At what timestamp was the first sealed bag placed on the handoff shelf?
  • Was the cook at the stove wearing a cap or hairnet at 00:45?
  • How many people were active at the prep counter at 00:45?
  • Did the worker close the container before moving it away from the station?
  • Which happened last: garnish added, lid closed, bag moved, or tray wiped?
  • Is the order number visible? Answer not_visible if it is not readable.

How to beat the baseline

  • Do coarse-to-fine search. Cheap pass over the whole clip, then inspect only candidate windows.
  • Keep timestamps attached to evidence. Every answer should know which frame or span supported it.
  • Use OCR narrowly. Receipts, labels, screens, and timers matter, but only when visible.
  • Handle uncertainty. Penalize confident guesses; answer not_visible when the evidence is missing.
  • Log cost and frames. Budget efficiency is part of the score.
Enter the kitchen CCTV challenge →

Stuck? Email inquiries@builderr.ai.