Skip to content

Scene Representation

Purpose

Build an incrementally maintained scene representation from VLMInput to give the VLM structured, accumulated knowledge of the environment across ticks — replacing the flat text rationale chain with a proper graph of what has been seen, from where, and what remains unexplored.

The design follows SysNav: Multi-Level Systematic Cooperation Enables Real-World, Cross-Embodiment Object Navigation (arXiv 2603.06914) and adapts its three-level hierarchical graph to the sensor suite and constraints of the CMU VLN Challenge.

Original SysNav design

SysNav builds a three-level graph ℛ updated incrementally as the robot navigates an indoor building:

Level Node type Key attributes
High Room category label, room mask, best representative RGB image
Mid Viewpoint position, coverage region (observed voxels), panoramic image
Low Object label, confidence, 3D point cloud, bounding box, representative image

Edge types: room–room (doorway connectivity), room–viewpoint (affiliation), room–object (containment), viewpoint–object (visibility).

Sensor inputs: 360° panoramic RGB + LiDAR point cloud at each step. Object detection uses YOLOv8x + SAM2.1; room boundaries are identified via vertical planar surface fitting on the LiDAR point cloud (wall detection). Viewpoints are created when new coverage exceeds a threshold ε relative to the accumulated coverage union.

VLM query modes:

  • Room-query: context contains uncovered room attributes + trajectory. VLM picks the next room to enter.
  • Early-stop: context contains current room's objects + task goal. VLM decides whether to terminate local exploration.

The representative room image is central to VLM reasoning — it gives the model a wide-angle memory of the room to reason about object layout, even when the robot is currently looking at a different part of the space.

Adaptations for the CMU VLN Challenge

What changes and why

Room level — one room instead of many. The challenge environment is a single bounded scene (not a multi-room building), so room-to-room routing and wall-detection-based segmentation are not needed. However, the Room node's core value — storing the best representative image of the scene — is retained. The single RoomNode acts as the global scene memory: it holds the best-coverage panoramic frame seen so far, updated whenever the robot reaches a new viewpoint.

Viewpoint level — pose-driven instead of coverage-driven. SysNav creates viewpoints by comparing the current observation's voxel coverage against the accumulated union and adding a node when the new area exceeds threshold ε. This requires a 3D occupancy map. In our stack, the only readily available spatial signal is the robot's pose from /state_estimation. Viewpoint nodes are therefore created when the robot moves more than viewpoint_radius metres from every existing node — a simpler proxy for "the robot is now seeing a meaningfully different slice of the scene." The threshold (default 2.0 m) is tunable.

Object level — add_object() is the entry point; multiple producers can call it. SysNav runs YOLOv8x + SAM2.1 on every frame to maintain the object graph autonomously. We use the same model family but as a sidecar (xiao-hei/perception, see Perception Sidecar) — YOLOv8x-World v2 + SAM 2.1 Hiera Tiny behind a FastAPI service that the upcoming PerceptionResponder calls each tick. The add_object() method is the single insertion point; the perception path calls it from real detections, with no scene-graph code changes required.

Object colour — sampled from the detection mask. Each detection also carries an appearance colour: the responder takes the median RGB of the camera pixels inside the mask (median, like the point-lifter, to resist specular highlights and mask-edge bleed) and labels it with the nearest basic colour ("red", "brown", …). This lets the graph answer colour-qualified references ("the red samovar") without a second model. Stored as color_rgb / color_name on ObjectObservation and refreshed in lock-step with position when a higher-confidence observation merges in.

Object detection — perception sidecar (YOLO-World v2 + SAM 2.1). The challenge sensor suite (360° equirectangular RGB + LiDAR) is well-suited to an open-vocabulary detector: YOLO-World accepts class names at inference time, so the same model handles task-specific question vocabulary (e.g. "the red samovar") and generic scene-prior labels. The sidecar unwraps the equirect frame into 4 perspective faces, batches YOLO across them, runs SAM 2.1 per bbox, and reprojects masks back into equirect coordinates — the responder consumes a single equirect mask per detection and projects it through the registered LiDAR scan to lift to 3D.

Wall detection — replaced by scene bounds. SysNav fits vertical planes on the LiDAR point cloud to segment rooms and define coverage boundaries. With only one room, room segmentation is unnecessary. Coverage boundaries are instead derived for free from the registered scan: a numpy min/max over the XY extent of the point cloud gives an approximate scene bounding box (scene_bounds) stored on the RoomNode. This tells the VLM how large the search area is — enough to reason about whether full coverage has been achieved — without any model inference.

Edges — storage and entry points now, reasoning logic later. Full graph edge reasoning is deferred, but the storage fields and entry point hooks are wired now so the interface does not need to change later. Two edge types are prepared:

  • Room → Viewpoint (affiliation): RoomNode.viewpoint_tick_ids: list[int] — appended in _maybe_add_viewpoint(). Trivial with one room; present for API completeness.
  • Viewpoint → Object (visibility, stored as a reverse edge on the object): ObjectObservation.observing_viewpoint_ids: list[int] — appended in add_object() with the current viewpoint id (the latest viewpoint added at-or-before the observing tick). Entries are guaranteed real ViewpointNode.tick_id values, so the field doubles as the reverse graph edge. To query "which objects were visible from viewpoint N": filter objects where N is in observing_viewpoint_ids. Logic can later be enriched with LiDAR ray-casting.

Object → Object edges were removed

An ObjectObservation.spatial_relations field once stored Object→Object edges, auto-populated by derive_near_relations(threshold). It was removed in TASK 19: nothing consumed the edges. The live scene_gemini responder never derived them, the offline batch sends a compact object list rather than the relation graph, and the perception responder answers by label match and pose distance. Proximity is inferred from object coordinates, which every consumer already has.

Final data structure

SceneRepresentation
│
├── room: RoomNode
│     label: str                              # "scene" or question-derived
│     best_image: ImageFrame | None           # best-coverage frame
│     best_image_tick_id: int | None
│     best_image_position: Vector3 | None
│     scene_bounds: (Vector3, Vector3) | None # (min_xyz, max_xyz) from registered scan
│     viewpoint_tick_ids: list[int]           # ── edge: Room → Viewpoint (affiliation)
│
├── viewpoints: list[ViewpointNode]
│     tick_id: int
│     position: Vector3                       # map frame
│     yaw: float                              # heading in radians
│
├── objects: list[ObjectObservation]
│     object_id: int                          # stable, monotonic, never reused
│     label: str
│     position: Vector3                       # map frame, estimated
│     confidence: float
│     color_rgb: (int, int, int) | None       # median RGB (0-255) of masked pixels
│     color_name: str | None                  # nearest basic-colour label ("red", …)
│     bbox_min / bbox_max: Vector3 | None     # from LiDAR, optional
│     first_tick_id / last_tick_id: int
│     observing_viewpoint_ids: list[int]      # ── edge: Viewpoint → Object (visibility, reverse)
      relation: str                           # e.g. "left_of", "near", "on_top_of"

Public API

rep = SceneRepresentation(
    viewpoint_radius=2.0,          # m — min travel before new viewpoint node
    merge_radius=1.5,              # m — max distance to merge same-label objects
)

rep.update(snapshot: VLMInput)                                       # call every tick before inference
rep.add_object(obs: ObjectObservation)                               # call after inference if object identified

Design choices and trade-offs

Decision Choice Alternative Reason
Number of rooms 1 fixed Multi-room with wall detection Challenge is single-scene; wall detection needs occupancy map we don't have
Viewpoint novelty Distance threshold on pose Voxel coverage delta (SysNav) No 3D map available; pose is cheap and always present
Best image selection Updated at every new viewpoint Updated only if new position is farther from prior best Simpler; new viewpoint = novel coverage by definition
Object population YOLOv8x-World v2 + SAM 2.1 sidecar via add_object() YOLOv8x + SAM 2.1 in-process (SysNav-style) Sidecar isolates the GPU stack from the ROS image and lets us swap detector/segmenter independently.
Wall detection Replaced by scene_bounds (numpy min/max of registered scan) RANSAC plane fitting One room only; terrain map already encodes obstacles; bounds are free from the scan
Graph edges Storage + entry points now; reasoning logic deferred Fully wired now Storage fields and hooks added to avoid later interface changes; reasoning filled in once object population matures
Engine interface Unchanged (single image) Multi-image (room best + current) Engine change deferred; best_image stored on RoomNode ready to be used

Test plan

1. ViewpointNode creation

  • First update() with a pose always creates one viewpoint.
  • Second update() with the robot < viewpoint_radius away → no new viewpoint added.
  • Second update() with the robot > viewpoint_radius away → new viewpoint added.
  • Stored position and yaw match the pose input.
  • update() with pose=None → no viewpoint added, no crash.

2. RoomNode best image

  • Initialises with best_image=None.
  • Updated with the current frame when a new viewpoint is created.
  • NOT updated when the robot hasn't moved enough (no new viewpoint node).
  • best_image_tick_id and best_image_position track the correct tick and position.

3. ObjectObservation merging

  • Adding to empty list → creates new node.
  • Same label, within merge_radius, higher confidence → updates position and confidence in place.
  • Same label, within merge_radius, lower confidence → existing node unchanged.
  • Same label, outside merge_radius → creates a second node (different spatial instance).
  • Different label, within merge_radius → creates a new node (different object type).

4. Edge storage

Room → Viewpoint:

  • room.viewpoint_tick_ids is empty on init.
  • Appended with the correct tick_id each time a new viewpoint is created.
  • Not appended when robot hasn't moved enough.

Viewpoint → Object (observing_viewpoint_ids):

  • Empty on a newly created object when no viewpoint exists yet.
  • Current viewpoint id appended when an object is first added (or merged with higher confidence) and at least one viewpoint exists.
  • Dedup against the last entry — multiple observations within the same viewpoint don't inflate the list.
  • Not appended when a lower-confidence observation is rejected.

5. Yaw extraction from quaternion

  • Identity quaternion (w=1, x=y=z=0) → yaw = 0.
  • Pure Z rotation 90° → yaw ≈ π/2.
  • Pure Z rotation -90° → yaw ≈ -π/2.

Integration plan (next steps)

  1. Phase 3 (this task): implement SceneRepresentation and tests.
  2. Responder wiring: SceneGeminiResponder calls rep.update(snapshot) while exploring and serialises the populated representation into the Gemini request via serialize_for_gemini.
  3. Object population: after parsing ObjectReferenceResponse or extracting entity mentions from the numerical rationale, call rep.add_object().
  4. Engine multi-image: pass scene.room.best_image alongside the current frame once the engine interface is extended to accept multiple images.