Scaling Video Prediction with Spatio-Temporal Patches

Abstract

The article presents a new architecture for video data processing, the Vision Byte Latent Transformer (V-BLT), which adapts the principles of successful byte-level language models to the visual modality. Unlike standard approaches that use fixed-size patching, which are computationally inefficient due to the uniform resource allocation regardless of visual content complexity, V-BLT operates directly on the video byte stream. This allows for avoiding information loss associated with prior tokenization and enhances processing flexibility. The key contributions include the concept of spatiotemporal latent patches, the implementation of N-dimensional Rotary Positional Embeddings to preserve data coherence in the flattened byte stream, and a multi-level transformer architecture for hierarchical processing. To validate the hypothesis and test the model, a new synthetic dataset with rotating 2D and 3D shapes was developed for a controlled evaluation of the model’s spatiotemporal reasoning capabilities. It is experimentally demonstrated that V-BLT effectively predicts future frames, achieving high scores on MSE, SSIM, and PSNR metrics comparing to ViViT and UNet3D with better computational efficiency. The developed architecture according to the design has the ability to generate per-pixel entropy maps that visualize prediction uncertainty and correlate with dynamically complex regions of the scene. This paves the way for the implementation of dynamic, content-dependent, on-the-fly allocation of computational resources, representing a promising direction for creating more efficient and scalable foundation models for video analytics.