Workshop
Three agents build and correct 3D assemblies from a spoken request.
Workshop builds 3D objects like rockets from voice and hand gestures. I say "build me a rocket" and three agents assemble it: Builder picks the parts and which ports connect, Owl judges what the result looks like (front, side, and top renders through Claude Vision), and Mechanic corrects rotation, scale, and position from measured geometry. No model in the system was trained on 3D data.
The parts are 23 parametric primitives (cone, cylinder, fin, nozzle) with typed connector ports, so Builder never computes coordinates; the canvas snaps the ports together. The hard part is the correction loop. It runs up to five iterations per build and took a test assembly from 3 open issues to 1 in 2 iterations, but that is one build on well-formed meshes and I have not benchmarked it.
"No model in the system is trained on 3D data. Spatial accuracy comes from the interaction between semantic, visual, and geometric agents, which is close to how embodied cognition describes it in people."

Multi-Agent Architecture
Multi-agent cycle: no single agent understands geometry — accuracy emerges from interaction. Issue convergence: 3→1 in 2 iterations for well-formed meshes.
Three Agents, Separate Senses
The loop
- Builder decomposes the spoken request into parts and says which connector port joins which. It never computes coordinates; the canvas aligns the ports
- Owl looks at front, side, and top renders with Claude Vision, alongside scene-graph measurements and the original request, and evaluates every part before giving a verdict
- Mechanic receives the Owl's evaluation, exact scene-graph geometry, OpenCV three-view estimates, and the history of fixes already tried, and corrects rotation, then scale, then position
The result
- Correction loop went from 3 open issues to 1 in 2 iterations
- Voice in through Deepgram, hand tracking through MediaPipe
- 23 parametric component types (cone, cylinder, fin, nozzle, gear, bearing and more), each with typed connector ports
- Up to five capture, measure, evaluate, correct iterations per build
What Broke, and What It Taught Me
- Mixed perceptual channels caused the convergence failures. Oscillating rotations and false approvals traced back to agents sharing inputs: when the correcting agent is handed screenshots and numbers together, the image tends to win even when the numbers are exact. The Owl now owns the visual judgment, the Mechanic is told to trust measurements and CV estimates over its own impression, and repeated corrections are filtered so it cannot loop. Cutting the Mechanic down to numeric input only is the next experiment.
- Generated meshes were the wrong foundation. The first version asked a text-to-3D service for each part. Pivoting to parametric primitives with typed connector ports made assembly a geometry problem the system could reason about, and the rocket came out right on the first build.
- The evaluators never saw the request. Owl and Mechanic were judging the scene without the user's original words, so they approved things that were well-formed but wrong.
- A naming mismatch silently disabled triangulation for two days. Frames were saved as frame_0/1/2 while the OpenCV step looked for front/side/top. Nothing errored; the geometry channel was simply empty.
- Titles vs. IDs. Builder passed node titles while the connection code looked up by ID, so auto-align never ran. One resolver function fixed a bug that took two days to see.