Interpretable video generation evaluation
Diagnostic evaluation of spatio-temporal instruction following, from atomic prompt constraints to causal failure propagation.
The evaluation framework
VGIF-Score evaluates whether a generated video completes the requested structure and whether the result satisfies prompt-conditioned perceptual criteria.
Prompts are decomposed into an ST-DAG of entities, attributes, locations, actions, states, and causal relations. A node receives credit only when its QA answer and dependency expression both pass.
An instruction-conditioned AutoRubric scores four dimensions from 1 to 5: cinematography, visual purity, motion smoothness, and physics adherence.
VGIF-Bench
The benchmark spans eight macro categories and 38 micro categories, with deep graphs and multi-parent dependencies designed to expose instruction-following failures.
Benchmark landscape
Camera-ready Table 1 places VGIF-Bench against ten video-generation benchmarks. Select any point to compare prompt complexity, graph structure, and diagnostic coverage.
Objective and subjective assessment, explicit ST-DAG structure, and failure diagnosis.
| Benchmark | Objective | Subjective | ST-DAG | Diagnosis |
|---|
W, U, and Dep. denote average words, estimated atomic units, and estimated dependencies. VGIF-Bench additionally provides explicit ST-DAGs averaging 16.4 nodes and 17.7 edges.
Why VGIF-Score
VGIF-Score turns a single opaque judgment into a traceable account of what appeared, what depended on it, and how well the result was presented.
?
One overall score cannot reveal which instruction was missed.
ST-DAG converts the prompt into typed, atomic constraints.
A missed trigger blocks dependent events instead of hiding the cause.
Completion and perceptual quality meet at the sample level.
Interactive diagnosis
Choose a domain, compare two distinct model outputs, inspect the atomic ST-DAG decomposition, and follow each missed constraint into its downstream judgment.
Loading the prompt and diagnostic annotations...
Select a node to inspect its evaluation question.
Instruction-conditioned AutoRubric
Each dimension is generated from this prompt. Switch dimensions to inspect the criteria, score anchors, and the selected model's diagnostic judgment.
Loading rubric...
Eight curated VGIF-Bench prompts and 16 generated videos across eight model families. This diagnostic subset is published for visualization; bulk per-sample benchmark results remain excluded.
Paper results
Explore the aggregate scores reported in Table 2. No per-sample evaluation outputs are included.
| Rank | Group |
|---|
All values are the aggregate percentages printed in the finalized paper. Click a column label to sort.
Domain analysis
Switch between the camera-ready macro-domain table and a finer diagnostic view aggregated over the 38 micro domains.
Eight macro domains from the camera-ready paper.
Diagnostic findings
The graph structure reveals failure modes hidden by a single holistic quality score.
Even the strongest commercial models score below five on causal instruction following.
Average QA accuracy falls from 67.9% in the first fifth to 10.1% in the final fifth.
Beyond depth four, accuracy falls below three percent for every evaluated model.
Failure localization
Kling-V3 completes all 12 QA items in the paper case study. CogVideoX-1.5 preserves the scene setup but misses q9, causing dependency-aware failures at downstream q10 and q12.
Human validation
Use VGIF-Score
Reproduce the evaluation protocol with the public implementation and load VGIF-Bench directly from Hugging Face.
@misc{xu2026vgifscoreinterpretablediagnosticevaluation,
title={VGIF-Score: Interpretable and Diagnostic Evaluation of Spatio-Temporal Instruction Following in Video Generation},
author={Songyu Xu and Xin Wang and Qiang Chen and Xinran Wang and Muxi Diao and Yuxuan Zhang and Kongming Liang and Rui Lin and Zhanyu Ma},
year={2026},
eprint={2607.13527},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2607.13527},
}