Adobe Research Division Solves Long-Term Memory Problem in Video Generation

Adobe's research division has announced that it has overcome the 'long-term memory' problem that has long plagued video generation AI. By combining state space models (SSM) with local attention mechanisms, the approach addresses the issue of videos "forgetting" earlier content when generating longer videos.

A fundamental challenge for video generation AI has been that as video length increases, it becomes harder to maintain consistency with past frames. General Transformer-based models require dramatically increased computational power to handle relationships between distant frames. As a result, longer videos tend to suffer from problems where scene settings or character appearances change unexpectedly partway through.

The current research employs a design combining SSM (state space models) with local attention. SSM is a mechanism well-suited to efficiently handling dependencies between distant frames, serving as the 'memory' function across the entire video. Local attention, conversely, functions to maintain fine-grained consistency between adjacent frames—for instance, the smoothness of motion or the continuity of local visual content. By combining these two approaches, the architecture achieves both macro-level consistency and micro-level naturalness.

The training methodology also incorporates thoughtful design, employing two strategies: 'Diffusion Forcing' and 'frame-local attention.' Diffusion Forcing is a technique related to training diffusion models that generate video frames through step-by-step denoising, and is known to help models learn temporal context more effectively. By combining these approaches, training was conducted to maintain content consistency even in long-form video generation.

The significance of this achievement is substantial from the perspective of practical video generation technology. Current video generation AI can handle short clips but has struggled to apply to longer video content like films or television dramas. If this approach becomes practical, it could expand the possibilities for AI to generate long-form videos with consistency, potentially introducing new options to video production workflows.

Adobe is a company that develops creative tools including the video editing software Premiere Pro and the image editing software Photoshop, and has been actively pursuing AI integration into its products. While how this research will be applied at the product level remains unclear at this time, it deserves attention as a technology that could enhance video generation quality and consistency in relation to future product development.

Video generation AI, as a field, lags behind text and image generation in terms of technological maturity. The addition of the temporal axis as a new dimension means that beyond simple quality, 'narrative consistency' becomes a critical concern. Adobe Research's approach to addressing this challenge head-on represents one methodology that could influence broader industry technological trends.