messages¶
xiao_hei_vln.messages
¶
Pydantic data classes that define the VLM I/O contract.
Header
¶
Bases: BaseModel
Mirror of std_msgs/Header (stamp + frame_id).
Stamp
¶
Bases: BaseModel
ROS time stamp, split into seconds and nanoseconds.
VLMInput
¶
Bases: BaseModel
Bundle of latest sensor values at the moment of the VLM tick.
Any of the optional fields may be None if the corresponding topic
has not yet produced a message at snapshot time (cold start).
is_ready
property
¶
True iff the inputs sufficient for any kind of inference are present.
NumericalResponse
¶
Bases: BaseModel
Answer to a how many ... question, published on /numerical_response.
ObjectReferenceResponse
¶
Bases: BaseModel
Selected-object bounding box, published on /selected_object_marker.
center/size are in the map frame; heading is the yaw (rad)
of the object's local frame around +Z.
Waypoint
¶
Bases: BaseModel
A single 2D waypoint in the map frame.
WaypointPathResponse
¶
Bases: BaseModel
Sequence of waypoints, published one-by-one on /way_point_with_heading.
ImageFrame
¶
Bases: BaseModel
sensor_msgs/Image consumed from /camera/image (frame_id="camera").
Raw bytes are stored as-is; conversion to an ndarray is left to the consumer to avoid forcing a numpy dependency on every read.
LidarScan
¶
Bases: BaseModel
sensor_msgs/PointCloud2 from /registered_scan or /sensor_scan.
Points are flattened into an (N, 4) ndarray of (x, y, z, intensity).
/sensor_scan has no intensity field on the wire, so the adapter
fills the trailing column with zeros.
OdomPose
¶
Bases: BaseModel
nav_msgs/Odometry from /state_estimation (map → sensor).
TerrainMap
¶
Bases: BaseModel
sensor_msgs/PointCloud2 from /terrain_map (5 m) or /terrain_map_ext (20 m).
Points are (x, y, z, cost) — the intensity PointField encodes the
traversability cost emitted by the terrain analysis module.
parse_vlm_output(data)
¶
Validate arbitrary input (dict, JSON-decoded payload, ...) as a VLMOutput.
classify_question(text)
¶
Heuristic classifier for the three challenge categories.
"How many ..." / "Count ..." → numerical; "Find ..." or a bare "The ..." noun phrase → object reference; anything else → instruction following. ("Count the number of chairs with pillows on them." is an official numerical item, so a leading "Count" routes to numerical.)
The leading-"The" case matters: 3 of the 30 official object_reference questions drop the imperative and read "The red pillow closest to the sushi." / "The blue chair that is closest to ...". The official instruction_following questions never start with "the" — they all begin with an action verb (Go / First / Take) — so routing a leading "the" to object_reference is unambiguous and avoids misclassifying those items.
Common types¶
xiao_hei_vln.messages.common
¶
Inputs¶
xiao_hei_vln.messages.inputs
¶
VLMInput — the snapshot the VLM consumes on each tick.
VLMInput
¶
Bases: BaseModel
Bundle of latest sensor values at the moment of the VLM tick.
Any of the optional fields may be None if the corresponding topic
has not yet produced a message at snapshot time (cold start).
is_ready
property
¶
True iff the inputs sufficient for any kind of inference are present.
Outputs¶
xiao_hei_vln.messages.outputs
¶
VLMOutput — discriminated union of the three valid response shapes.
NumericalResponse
¶
Bases: BaseModel
Answer to a how many ... question, published on /numerical_response.
ObjectReferenceResponse
¶
Bases: BaseModel
Selected-object bounding box, published on /selected_object_marker.
center/size are in the map frame; heading is the yaw (rad)
of the object's local frame around +Z.
Waypoint
¶
Bases: BaseModel
A single 2D waypoint in the map frame.
WaypointPathResponse
¶
Bases: BaseModel
Sequence of waypoints, published one-by-one on /way_point_with_heading.
parse_vlm_output(data)
¶
Validate arbitrary input (dict, JSON-decoded payload, ...) as a VLMOutput.
Sensors¶
xiao_hei_vln.messages.sensors
¶
Sensor messages exposed to the VLM by the challenge platform.
Field rates and conventions mirror what we observed from the official
simulator (see docs/task1_phase1_measurements.md). The contract is
plain Python: anything that accepts a numpy array can be tested without
ROS.
ImageFrame
¶
Bases: BaseModel
sensor_msgs/Image consumed from /camera/image (frame_id="camera").
Raw bytes are stored as-is; conversion to an ndarray is left to the consumer to avoid forcing a numpy dependency on every read.
LidarScan
¶
Bases: BaseModel
sensor_msgs/PointCloud2 from /registered_scan or /sensor_scan.
Points are flattened into an (N, 4) ndarray of (x, y, z, intensity).
/sensor_scan has no intensity field on the wire, so the adapter
fills the trailing column with zeros.
TerrainMap
¶
Bases: BaseModel
sensor_msgs/PointCloud2 from /terrain_map (5 m) or /terrain_map_ext (20 m).
Points are (x, y, z, cost) — the intensity PointField encodes the
traversability cost emitted by the terrain analysis module.
OdomPose
¶
Bases: BaseModel
nav_msgs/Odometry from /state_estimation (map → sensor).
Question¶
xiao_hei_vln.messages.question
¶
Inbound /challenge_question (std_msgs/String) and its classification.
The challenge has three question categories; the response topic is
determined by which category the question falls into. We start with the
same keyword heuristic as the reference dummyVLM.cpp and leave the
classifier swappable for a learned model later.
classify_question(text)
¶
Heuristic classifier for the three challenge categories.
"How many ..." / "Count ..." → numerical; "Find ..." or a bare "The ..." noun phrase → object reference; anything else → instruction following. ("Count the number of chairs with pillows on them." is an official numerical item, so a leading "Count" routes to numerical.)
The leading-"The" case matters: 3 of the 30 official object_reference questions drop the imperative and read "The red pillow closest to the sushi." / "The blue chair that is closest to ...". The official instruction_following questions never start with "the" — they all begin with an action verb (Go / First / Take) — so routing a leading "the" to object_reference is unambiguous and avoids misclassifying those items.