evaluator¶

`xiao_hei_vln.evaluator` ¶

Offline evaluation pipeline for CMU VLN Challenge 2026.

Supports Task 1 (Numerical) and Task 2 (Object Reference) metrics. Task 3 (Instruction-Following) requires the closed-source official evaluator.

Run evaluation on a list of EvalSamples and produce an EvalReport.

One evaluation example: a question paired with ground truth and prediction.

Shared data types for the evaluation pipeline.

One evaluation example: a question paired with ground truth and prediction.

Evaluator: groups EvalSamples by type and dispatches to metric computers.

Run evaluation on a list of EvalSamples and produce an EvalReport.

EvalReport: aggregates metrics across all question types and formats output.