Open-OmniCoach: Evidence-Enhanced Trajectory Learning for Omni Video Agents

Demos

Follow the evidence.

Six recorded cases. Watch the search, the correction, and the evidence come together.

Overview

Connect. Correct. Credit.

Multi-span and reflection data for SFT, then evidence-guided step credit for RL.

Mechanism	Prior limitation	Our approach
Connect	Single-span retrieval The agent retrieves one span per tool call.	Multi-span retrieval The agent retrieves and compares multiple audio-visual spans in one call.
Correct	Success-only supervision The agent learns from successful rollouts alone.	Recovery supervision The agent learns from teacher corrections to its own failed rollouts.
Credit	Trajectory-level credit The RL objective assigns one advantage to every step in a rollout.	Evidence-guided step credit SEPO, our RL objective, rescales each step’s advantage using evidence gain.

Dataset

OmniCoach-240K

Evidence-grounded trajectories that teach both search and self-correction.

Trajectories: 248.2K
QA & grounding tasks: 95.7K
Average video duration: 8.87 min

How the data is built

Figure 2. OmniCoach-240K data generation pipeline. (a) Hierarchical evidence QA generation: fine-grained audio-visual scripts are linked to a global event timeline, supporting skill-guided MCQ and TG generation followed by quality-aware filtering. (b) Multi-span trajectory generation: using Stage-1 and other-source QA samples, a teacher generates trajectories in the Dynamic Omni-Evidence Harness, which supports joint evidence acquisition across multiple time spans and modalities. (c) Reflection trajectory generation: a verifier diagnoses errors in student rollouts. At confirmed recovery points, a teacher preserves the erroneous draft, adds an evidence-grounded correction and a revised action, and continues the interaction. Both teacher and reflection trajectories are retained only after passing task-level outcome checks. Full resolution (PNG)

Stage 1: Hierarchical Evidence QA Generation

95.7K tasks: 55.3K MCQ and 40.4K TG.

Fine-grained Evidence Generation. Segment-level audio-visual scripts record people, sound sources, and objects with a global entity table. Audiovisual Script Refinement then re-examines each segment to fix sound-source attribution, boundary drift, and missing facts.
Event and Evidence Matching. Scripts are merged into a concise global event timeline. Each event keeps links to its source intervals, so task construction can pick events top-down and expand local evidence bottom-up.
Skill-guided QA Generation. 18 MCQ reasoning skills and one TG skill guide the model to compose events and confounding distractors while recording task-level evidence annotations 𝒜_x.
Quality-aware Filtering. Text-only guessable MCQs are dropped. A verifier answers each task from 𝒜_x alone and from the full video; the answers must agree. TG requires a one-to-one match with a tIoU above threshold.

Stage 2: Multi-Span Trajectory Generation

A teacher model (Gemini 3.6) interacts with the Dynamic Omni-Evidence Harness. Each trajectory starts with the task, tool specs, timestamped frames covering the whole video at 0.5 FPS (up to 256 frames at 224² pixels), and the full audio track. The teacher may issue up to 32 tool calls. Coarse frames are sampled at 1 FPS, detail frames at 2 FPS; the default longest side is 384 px, with resolution=high up to 1280 px for up to 5 s.

The key extension is get_multi_span, which requests 2–8 ordered intervals in one call, each returning frames, audio, or both. This lets the model compare candidate spans side by side instead of accumulating errors across sequential calls. Trajectories pass an outcome check: correct answer for MCQ, or set-level tIoU plus minimum coverage for TG.

See multi-span retrieval and harness-limit adaptation in the Salon S10–S12 explorer, and 4-span visual search in the Istanbul S13–S14 explorer / Istanbul video demo.

Dynamic Omni-Evidence Harness tool interface and per-call limits
Tool	Intervals	Returned evidence	Per-call limit
`get_frames`	1	Sampled frames	At most 256 frames
`get_audio`	1	Continuous audio	At most 256 s
`get_clip`	1	Frames + audio	At most 60 s
`get_multi_span`	2–8 ordered	Frames, audio, or both per interval	At most 256 frames and 128 s audio total

Stage 3: Reflection Trajectory Generation

17K reflection trajectories from failed student rollouts.

Trajectory Collection. Eight rollouts per query are sampled from the Stage-2 SFT student; failed trajectories are selected for diagnosis.
AV-Decoupled Stepwise Verification. A verifier examines key steps in temporal order, checking audio, visual, cross-modal, and agentic support, and stops at the first confirmed error.
Same-Turn Recovery. The student's erroneous draft is preserved, an evidence-grounded correction and revised tool call are added in the same turn, and the trajectory continues to a verified final answer.

Inspect a real student visual hallucination and teacher high-resolution recovery in the Porsche S2 explorer / Porsche video demo.

Data composition & comparison

**Figure 3. QA data flow behind OmniCoach-240K.** Band widths encode video hours between the first two columns and QA counts afterward. The brackets mark the source videos, Stage-1 (Hierarchical Evidence QA Generation), and the QA samples used for Stage-2 and Stage-3 trajectory generation; other-source QA samples join in the third column. Filtered out: source videos without any retained QA sample. Agentic SFT QA: QA samples covered by at least one retained Stage-2 teacher or Stage-3 reflection trajectory. Full resolution (PNG)

Comparison with existing video and omni-video training datasets

Dataset	Mod.	Diverse QA				Dynamic Evidence Acquisition		Error Recovery
Dataset	Mod.	#QA	Formats	Avg. Duration	Evidence Ann.	#Traj.	Multi-span	Error Recovery
LongVALE	AV	N.R.	TG, FF	3.9 min	Time + text	--	--	×
OmniVideo-100K	AV	100K	MCQ, FF	103 s	Time + text	--	--	×
LongVT	V	N.R.	TG, FF	N.R.	Time	34.6K	×	×
Video-o3	V	N.R.	MCQ, FF	N.R.	Time	173K	×	✓
OmniAgent	AV	N.R.	TG, MCQ, FF	N.R.	N.R.	58K	×	✓
Open OmniCoach (Ours)	AV	95.7K	TG, MCQ, FF	8.87 min	Time + modality	248.2K	✓	✓

TG: temporal grounding; MCQ: multiple-choice QA; FF: free-form QA/generation; N.R.: not reported. For Ours, average duration is the mean duration of unique videos covered by QA data.

Method

SEPO: credit useful steps.

Adjust each step’s update by the evidence it adds, while preserving the trajectory advantage’s sign.

Step-level advantage

\widetilde A_{i,k}=A_i+\lambda\lvert A_i\rvert z_{i,k}

z_{i,k}=\operatorname{clip}\!\left(\frac{\Delta U_{i,k}-\mu_i}{\sigma_i+\delta},-1,1\right)

Evidence gains are standardized within each rollout, then clipped to [−1, 1].

0\leq\lambda<1

The advantage keeps its sign.

Above-average evidence gain

Positive advantageStronger reinforcement

Negative advantageWeaker suppression

Illustration with λ = 0.5 and normalized gain z = 1.

01
Evidence utility
Score the best subset of retrieved spans against the task’s annotated evidence.
Evidence coverageTemporal localization
02
Marginal gain
Measure improvement over the best evidence found so far. Repeated evidence adds no new gain.
$\Delta U_{i,k}=U_{i,k}-U_{i,k-1}$
03
Step advantage
Use the normalized gain to adjust the strength of each step’s token updates.

Evidence scoring & optimization

Task-specific evidence utility. The scorer matches the modalities and temporal intervals of retrieved spans to the task’s evidence annotations. For QA, it combines coverage and locality: $q_m=\alpha c_m+(1-\alpha)\ell_m$ , with α = 0.5 by default. For temporal grounding, it uses the TimeLens2 grounding reward scaled to [0, 1]. Scores are combined across annotated modalities.

Historical best, then incremental gain. A bounded subset search selects the best-matching retrieved evidence. The utility keeps the historical best score, so redundant or irrelevant retrieval does not lower evidence already acquired. The next step’s gain is the improvement over that best score.

Within-rollout normalization. μ and σ are the mean and standard deviation of the step gains; δ is a positive stability term. An above-average gain gives z > 0. A positive raw gain alone does not guarantee a positive adjustment. The group-relative trajectory advantage comes from outcome and format rewards.

Token updates. Each generated token receives its step’s adjusted advantage in the GSPO-token objective. With λ = 0, SEPO reduces to the GSPO update. Evidence annotations guide training; the agent does not receive them at inference.

Full training recipe

Multi-span SFT
Train the base model on teacher-generated multi-span trajectories. Use this first SFT model to collect student rollouts and build reflection trajectories from its failures.
Mixed-data SFT
Start again from the same base checkpoint and train on both multi-span and reflection trajectories. This model initializes reinforcement learning.
Evidence-guided RL
Optimize the interaction policy with SEPO, combining trajectory-level outcome feedback with the evidence contribution of individual steps.

Results

Compare Open OmniAgent with its base model across audio-visual and video benchmarks.

Open OmniAgent-30B-A3B vs. Qwen3-Omni-30B-A3B

av benchmarks, 30B-A3B, compared with the base model
Benchmark	Base model	Open OmniAgent	Gain
WorldSense	54.0	54.8	+0.8
Daily-Omni	71.9	75.0	+3.2
JointAVBench	63.6	64.9	+1.3
OmniVideoBench	38.4	47.9	+9.5
LVOmniBench	35.8	48.0	+12.2
Average	52.7	58.1	+5.4

Accuracy (%) and percentage-point gains over the base model.

Open OmniAgent-7B vs. Qwen2.5-Omni-7B

av benchmarks, 7B, compared with the base model
Benchmark	Base model	Open OmniAgent	Gain
WorldSense	45.4	48.9	+3.5
Daily-Omni	62.1	70.2	+8.1
JointAVBench	56.5	59.3	+2.8
OmniVideoBench	29.3	40.4	+11.1
LVOmniBench	32.0	37.8	+5.8
Average	45.1	51.3	+6.3

Accuracy (%) and percentage-point gains over the base model.

Open OmniAgent-30B-A3B vs. Qwen3-Omni-30B-A3B

video benchmarks, 30B-A3B, compared with the base model
Benchmark	Base model	Open OmniAgent	Gain
Video-MME	70.5	75.9	+5.4
Video-MME v2 · Acc.	34.1	44.4	+10.3
Video-MME v2 · Rating	17.1	25.5	+8.4
LongVideoBench	66.9	67.4	+0.5
LVBench	50.2	63.5	+13.3
VideoZeroBench · L3	—	12.0	—

Accuracy (%) except for the official Video-MME v2 Rating. VideoZeroBench uses Level-3 accuracy. Video-MME v2 baselines use the leaderboard’s with-subtitle/audio setting. Gains use the same scale as each metric; — means unavailable.

Open OmniAgent-7B vs. Qwen2.5-Omni-7B

video benchmarks, 7B, compared with the base model
Benchmark	Base model	Open OmniAgent	Gain
Video-MME	64.3	67.9	+3.6
Video-MME v2 · Acc.	—	36.5	—
Video-MME v2 · Rating	—	18.4	—
LongVideoBench	53.8	57.1	+3.3
LVBench	43.0	51.7	+8.7
VideoZeroBench · L3	—	6.4	—

Accuracy (%) except for the official Video-MME v2 Rating. VideoZeroBench uses Level-3 accuracy. Video-MME v2 baselines use the leaderboard’s with-subtitle/audio setting. Gains use the same scale as each metric; — means unavailable.

What each training stage contributes

Qwen3-Omni-30B-A3B-Instruct · accuracy (%). Both SFT recipes start from the base checkpoint.

Ablations on Qwen3-Omni-30B-A3B-Instruct, accuracy (%)
Training recipe	Video-MME	Video-MME v2	OmniVideoBench	LVOmniBench
Base model	70.5	34.1	38.4	35.8
Multi-span SFT	72.9	40.7	44.7	46.8
Multi-span + reflection SFT	74.4	42.6	46.3	47.5
Reflection SFT → GSPO	75.4	43.3	46.9	47.7
Reflection SFT → SEPO	75.9	44.4	47.9	48.0

GSPO and SEPO are alternative RL runs from the same reflection SFT checkpoint.

Full benchmark comparison

Reported scores for proprietary, open-source generalist, and agentic models. Blue rows mark Open OmniAgent. — means a result is unavailable.

Full av comparison reported in the manuscript
Method	Size	WorldSense	Daily-Omni	JointAVBench	OmniVideoBench	LVOmniBench	Average
Proprietary Models
GPT-4o	—	42.6	56.5	45.0	—	—	—
Gemini-2.0-Flash	—	—	67.8	—	41.5	42.9	—
Gemini-2.5-Pro	—	65.1	80.6	65.3	58.9	—	—
Gemini-3.0-Pro	—	66.4	81.1	—	61.8	65.8	—
Open-source Generalist Models
video-SALMONN 2+	7B	50.9	71.8	46.7	36.4	32.7	47.7
OmniVinci	9B	48.2	66.5	—	32.1	—	—
Qwen2.5-VL	7B	35.5	40.7	47.7	29.8	—	—
Qwen2.5-Omni	7B	45.4	62.1	56.5	29.3	32.0	45.1
Qwen3-Omni-Instruct	30B-A3B	54.0	71.9	63.6	38.4	35.8	52.7
Open-source Agentic Models
OmniAgent	7B	47.2	64.8	—	37.1	—	—
Open OmniAgent (Ours)	7B	48.9	70.2	59.3	40.4	37.8	51.3
Open OmniAgent (Ours)	30B-A3B	54.8	75.0	64.9	47.9	48.0	58.1

Full video comparison reported in the manuscript
Method	Size	Video-MME	Video-MME v2 · Acc.	Video-MME v2 · Rating	LongVideoBench	LVBench	VideoZeroBench · L3
Proprietary Models
GPT-4o	—	71.9	—	—	66.7	48.9	—
Gemini-1.5-Pro	—	75.0	—	—	64.0	33.1	—
Gemini-2.5-Pro	—	—	—	—	76.8	67.4	16.4
Gemini-3.0-Pro	—	88.6	66.1	49.4	75.9	77.0	17.0
Open-source Generalist Models
Qwen2.5-Omni	7B	64.3	—	—	53.8	43.0	—
Qwen3-Omni-Instruct	30B-A3B	70.5	34.1	17.1	66.9	50.2	—
Qwen3-VL-Instruct	8B	71.4	35.7	18.2	66.4	58.0	8.2
Qwen3-VL-Instruct	235B-A22B	79.2	43.3	25.0	—	67.7	9.6
Qwen3.5	397B-A17B	83.7	42.1	24.5	—	75.5	10.6
Open-source Agentic Models
OmniAgent	7B	67.8	—	—	—	50.5	—
Open OmniAgent-7B	7B	67.9	36.5	18.4	57.1	51.7	6.4
Open OmniAgent-30B	30B-A3B	75.9	44.4	25.5	67.4	63.5	12.0

A closer look

Explore the full trajectories.

Step-by-step tool calls, returned audio-visual evidence, and model reasoning.

Paper & resources

Paper, code, OmniCoach-240K, and model checkpoints will be released after anonymous review.

Watch demos Explore trajectories

Open-OmniCoach Evidence-Enhanced Trajectory Learning for Omni Video Agents

Follow the evidence.

Connect. Correct. Credit.

OmniCoach-240K

Stage 1: Hierarchical Evidence QA Generation

Stage 2: Multi-Span Trajectory Generation

Stage 3: Reflection Trajectory Generation

Comparison with existing video and omni-video training datasets

SEPO: credit useful steps.

Evidence utility

Marginal gain

Step advantage

Multi-span SFT

Mixed-data SFT

Evidence-guided RL

Results

What each training stage contributes

Explore the full trajectories.

Paper & resources

Open-OmniCoach Evidence-Enhanced Trajectory Learning
for Omni Video Agents