A new video-generation system is designed to put camera movement, object movement and depth control behind one kind of input: tracks in three-dimensional space. The proposed interface treats those controls as different uses of a single 3D-track signal.
The system also has a faster streaming version. In tests, that version generated 20.6 frames per second and recorded an end-point error, or EPE, of 5.48. EPE measures how closely the generated motion follows the requested tracks. The reported score was within 4% of the reference teacher model.
One control language for camera and objects
The work is a preprint built around a computational video-generation method. After filtering, its training collection retained approximately 0.4 million high-definition clips.
Inside the model, a motion pathway projects 3D tracks to normalized image coordinates, encodes scalar depth in a separate branch and concatenates the resulting features with noisy latents before DiT patch embedding. In ordinary terms, the design gives the generator separate signals for a point's image position and its depth before combining them.
For its joint-control test, the researchers used 30 in-the-wild videos from the DAVIS validation set. The methods received the same first frame and text prompt where applicable, while each system used its own native way of accepting motion instructions.
The fast version uses more memory
On the DAVIS benchmark, the full teacher model reported the lowest EPE at 5.29 and the lowest LPIPS at 0.404, while recording the highest SSIM at 0.479. LPIPS is a measure of perceptual difference, so lower is generally better; SSIM measures structural similarity, so higher is generally better. These were descriptive benchmark comparisons, not results from a statistical test.
A matched-backbone comparison also reported better motion and perceptual scores with 3D tracks than with 2D tracks: EPE was roughly 33% lower and LPIPS was 5% lower. The result is specific to the reported comparison between the two track representations.
The streaming version was distilled from the bidirectional teacher, reducing inference from 50 steps to four while keeping memory independent of video length. In a separate efficiency test on 20 validation videos, the student used a 21-frame window and four-frame chunks. Its reported latency was 0.756 seconds, compared with 34.92 seconds for the full-attention teacher, and its throughput was 20.6 frames per second versus 0.84 frames per second.
The speed improvement did not mean lower memory use in that comparison. The student used 31.8 gigabytes of memory, while the teacher used 23.9 gigabytes. The reported table gives spread values for the efficiency measures, but no confidence intervals or significance tests were provided.
The gains have clear limits
Across three sequences of up to 350 frames, per-frame SSIM started above 0.90, declined gradually and stayed above 0.75. The paper reports that appearance and motion remained temporally coherent over those sequences, although the metric declined as the videos grew longer.
An ablation also points to the value of preserving track identity. On 20 clean validation videos with 81 frames, removing the track-ID embedding was associated with EPE rising from 0.985 to 2.455 and LPIPS rising from 0.200 to 0.283. In another comparison, single-stage full-resolution training produced EPE of 3.217 versus 0.985 for the complete training curriculum, while SSIM was 0.613 versus 0.661.
The paper reports visible failure cases as well. Small subjects or objects can become progressively blurred, and object permanence can break down, with examples including unexplained damage to a cake and a piping bag appearing later in a sequence.
Taken together, the tests describe a trade-off. The matched comparison associates 3D tracks with lower EPE and LPIPS, and the streaming student is far faster than the teacher, but the student still has a teacher-relative EPE gap and the system can lose detail or object consistency in generated sequences.
The reported findings therefore describe performance under the stated benchmarks and model settings, rather than establishing physically or causally correct video in general. The long-video results show gradual metric decline, and the failure cases show why motion scores alone do not settle questions about object permanence.
The acknowledgment reports partial support from the National Natural Science Foundation of China and the Beijing Nova program. The document remains a preprint.
Paper data and sources
Original title: 4DStreamCtrl: Interactive Video Generation with Online 4D Control
Authors: Shiqian Li, Chenguo Lin, Zhiguang Liu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text