As an early look at the learning-based side, we let a general-purpose vision-language model, Claude, decide where the robot goes. The model never drives the legs. It is given a task in plain English and works through a small set of tools: look (the camera image plus a map of the area around the robot, with lettered places it can go), go to one of those places, turn, take the stairs, remember a sign on the map, and done. Each move is a single target that the map shows to be free and reachable, and the vendor’s navigation stack walks the robot there with its own obstacle avoidance.
Reading Signs
A separate text-recognition (OCR) model reads the camera twice a second the whole time, much faster than the language model could. Anything it reads is pinned to the map as a landmark. A move can be told to stop as soon as a given word is read, and when it ends the model gets a digest of what was read on the way. That digest is how the model decides whether it has reached the goal; it can only finish by quoting the goal’s sign.
A First Run
In the run below it was asked to find Skirkanich Hall on the same floor, with a hint: “go straight to the dead end, take a right, then go straight and take a left”. It reached the hall in four moves, about 34 m in under three minutes, and finished when it read the hall’s hanging sign.
Claude guiding the X30 to Skirkanich Hall from a plain-English hint (sped up). Left: the map with the route walked so far and the model's reasoning for each move. Right: the robot's camera. The run ends when it reads the "SKIRKANICH HALL" sign.
How It Runs
laptop robot
Claude ──tools──> MCP server ──TCP 30000──────────────────> robot_server (.106): pose, moves, cancel
│ map and floors ──ssh──────────────> active map: grids, mapping path
│ camera ──RTSP─────────────────────> front camera (.105)
└─ OCR watcher: reads the camera at 2 Hz the whole time
Foxglove view (ws://localhost:8765), every step saved with a summary page
| Tool | Does |
|---|---|
look() |
The camera image, a map 24 m across centered on the robot with lettered places to go, landmarks and nearby stairs, and a summary |
go(target, stop_for, via) |
Walk to a lettered place (up to 10 m) or up to a landmark. stop_for ends the move as soon as given words are read |
turn(degrees) |
Turn in place; everything swept past is read |
take_stairs(direction) |
One flight up or down, using the same stair waypoints as our multi-floor routes |
remember(kind, text, image_x) |
Pin something read in the image to the map, traced along its bearing to the first wall |
done(answer, evidence) |
Finish; the evidence must quote the goal’s sign |
Guardrails. A move is refused unless the robot is localized, not e-stopped or fallen, and above 20% battery. A move goes at most about 10 m at low speed, with the vendor’s obstacle avoidance on the whole way. A run gets at most 40 moves, and --confirm asks a person to approve each one. Someone stays at the remote with the soft e-stop.
Offline rehearsal. A simulated robot_server moves an imaginary robot on a copy of the map while the camera replays a recorded walk, so runs can be practiced with no robot.
Next
- Our own controller underneath. Run the model’s moves on
nav3dand pure pursuit instead of the vendor’s planner, withterrain_gaitchoosing the gaits. - Full VLM control. Move from a model that picks targets to vision-language(-action) models that take over more of the driving, and compare them against the classical controllers.