Dynamics-Aware
Sparse Attentionfor Efficient Autoregressive
Video Diffusion

Yuanyu He^1,2* · Zhuokun Chen^3* · Yefei He^1† · Zhiwei Tang² · Jiasheng Tang² · Yinghao Yu² · Jianfei Cai³ · Bohan Zhuang^1‡

¹ Zhejiang University · ² Alibaba Group · ³ Monash University

Paper Code

MemoryFrom the memory perspective, we retain dynamic history and refresh stale tokens.

ComputeFrom the computation perspective, we reuse stable regions and denoise only changed ones.

TrainingFor training, we further protect motion by upweighting high-change regions during distillation.

Motivation

Change is concentrated.

Most of the scene stays stable while attention concentrates on active content.

BOTTLENECK 01

Memory grows with every frame.

A 480P, 60-second rollout can use about 300GB of KV cache, versus 56GB for the model weights.

BOTTLENECK 02

Attention is not uniformly useful.

Stable backgrounds remain similar across frames. Only a small set of regions carries new visual information.

FIGURE 1 · CORE OBSERVATION

Most of a frame can be reused

Across camera panning and subject motion, the background changes slowly. High temporal cosine similarity marks tokens that can be reused. Concentrated attention identifies tokens that matter for the next step.

Spatiotemporal dynamics across camera panning and subject motion. — Real scenes · temporal attention · cosine similarity with the preceding frame.

Stable regionsreuse the history

Changed regionsallocate compute

Method

One observation.
Three mechanisms.

DASA turns the observation into a unified design for memory, computation, and training.

01 · MEMORY

Keep dynamic history, refresh the rest

Historical KV tokens are pruned by layer-wise cosine similarity. The last frame of the previous chunk is kept as an anchor, and a static token is refreshed when its age reaches τ.

DASA storage and computation framework. — Layer-wise retention + static-token refresh.

02 · COMPUTATION

Skip redundant denoising

Local w × w alignment compares the current chunk to the reference frame. Tokens with similarity ≥ γ exit early and reuse aligned features. Only dynamic regions continue through denoising.

DASA threshold ablations. — Selected operating point: γ = 0.92, τ = 4.

03 · DISTILLATION

Do not train away motion

Static reuse can create a motion-lagging shortcut. DASA weights the DMD loss by W = 1 + α · max(0, 1 − sim), amplifying errors in high-motion regions.

Dynamics-guided distillation diagram. — Motion-aware weighting preserves natural dynamics.

Experiments

Faster generation,
competitive fidelity.

Evaluation uses VBench Extended: 946 prompts, five samples per prompt, and 16 quality dimensions. Throughput is measured on H100 GPUs.

1.45×30-second video

1.73×720P LongLive

2.16×1080P LongLive

27.58 FPSwith adaptation

TABLE 1 · SHORT VIDEO GENERATION

5-second generation on VBench

Higher is better ↑

Backbone	Method	FPS ↑	Speedup ↑	Quality ↑	Semantic ↑	Total ↑
Self Forcing 1.3B	Baseline	18.42	1.00×	84.97	81.06	84.19
	TeaCache	21.69	1.18×	84.22	80.47	83.47
	Dummy Forcing	25.19	1.37×	84.47	81.08	83.79
	DASA	26.21	1.42×	84.57	80.79	83.81
	DASA + adaptation	27.58	1.50×	84.64	81.32	83.98
LongLive 1.3B	Baseline	18.30	1.00×	83.77	80.73	83.16
	TeaCache	21.82	1.19×	83.10	80.73	82.63
	Dummy Forcing	25.10	1.37×	83.16	80.99	82.73
	DASA	26.50	1.45×	83.18	80.87	82.72
	DASA + adaptation	27.91	1.53×	83.78	80.35	83.09

DASA with adaptation adds dynamics-guided distillation to the inference-time method.

Long-video and high-resolution results · Table 2

Setting	Backbone	Baseline FPS	DASA FPS	Speedup	VBench Total
30 seconds	Self Forcing	18.42	26.21	1.42×	83.37 → 83.20
30 seconds	LongLive	18.30	26.50	1.45×	83.05 → 82.83
720P	Self Forcing	5.92	10.10	1.71×	83.68 → 83.71
720P	LongLive	5.98	10.37	1.73×	83.31 → 83.01
1080P	Self Forcing	1.35	2.81	2.08×	83.18 → 82.79
1080P	LongLive	1.36	2.94	2.16×	81.78 → 81.63