Interpretable video generation evaluation
Diagnostic evaluation of spatio-temporal instruction following, from atomic prompt constraints to causal failure propagation.
The evaluation framework
VGIF-Score evaluates whether a generated video completes the requested structure and whether the result satisfies prompt-conditioned perceptual criteria.
Prompts are decomposed into an ST-DAG of entities, attributes, locations, actions, states, and causal relations. A node receives credit only when its QA answer and dependency expression both pass.
An instruction-conditioned AutoRubric scores four dimensions from 1 to 5: cinematography, visual purity, motion smoothness, and physics adherence.
VGIF-Bench
The benchmark spans eight macro categories and 38 micro categories, with deep graphs and multi-parent dependencies designed to expose instruction-following failures.
Interactive diagnosis
Compare two generated videos from the paper case study, inspect the atomic ST-DAG decomposition, and see how one missed trigger changes every downstream judgment.
Loading the prompt and diagnostic annotations...
Select a node to inspect its evaluation question.
Instruction-conditioned AutoRubric
Each dimension is generated from this prompt. Switch dimensions to inspect the criteria, score anchors, and the selected model's diagnostic judgment.
Loading rubric...
Selected qualitative case from the camera-ready paper. Only these two generated videos and their diagnostic annotations are published here.
Paper results
Explore the aggregate scores reported in Table 2. No per-sample evaluation outputs are included.
| Rank | Group |
|---|
All values are the aggregate percentages printed in the finalized paper. Click a column label to sort.
Domain analysis
Switch between the camera-ready macro-domain table and a finer diagnostic view aggregated over the 38 micro domains.
Eight macro domains from the camera-ready paper.
Diagnostic findings
The graph structure reveals failure modes hidden by a single holistic quality score.
Even the strongest commercial models score below five on causal instruction following.
Average QA accuracy falls from 67.9% in the first fifth to 10.1% in the final fifth.
Beyond depth four, accuracy falls below three percent for every evaluated model.
Failure localization
Kling-V3 completes all 12 QA items in the paper case study. CogVideoX-1.5 preserves the scene setup but misses q9, causing dependency-aware failures at downstream q10 and q12.
Human validation
Use VGIF-Score
Reproduce the evaluation protocol with the public implementation and load VGIF-Bench directly from Hugging Face.
@misc{xu2026vgifscoreinterpretablediagnosticevaluation,
title={VGIF-Score: Interpretable and Diagnostic Evaluation of Spatio-Temporal Instruction Following in Video Generation},
author={Songyu Xu and Xin Wang and Qiang Chen and Xinran Wang and Muxi Diao and Yuxuan Zhang and Kongming Liang and Rui Lin and Zhanyu Ma},
year={2026},
eprint={2607.13527},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2607.13527},
}