Adobe Research Division Solves Long-Term Memory Problem in Video Generation

Adobe's research division announced that it has overcome the 'long-term memory' problem, which has been a long-standing challenge in video generation AI. Through a combination of state-space models (SSM) and local attention mechanisms, it is addressing the issue of 'forgetting' earlier content when generating longer videos.

The fundamental challenge that video generation AI has faced is that as video length increases, it becomes difficult to maintain consistency with past frames. Typical Transformer-based models experience a sharp increase in computational complexity when handling relationships between distant frames. Consequently, longer videos tend to suffer from problems such as changes in scene settings or character appearances in the middle of generation.

In this research, a design combining SSM (state-space models) and local attention was adopted. SSM is a mechanism well-suited to efficiently handling dependencies between distant frames and serves as the 'memory' function throughout the video. Meanwhile, local attention functions to maintain fine-grained alignment between adjacent frames, such as smooth motion and local visual continuity. By combining these two approaches, the resulting structure simultaneously ensures macro-level consistency and micro-level naturalness.

The learning methodology also incorporates innovations, with two strategies adopted: 'Diffusion Forcing' and 'frame local attention'. Diffusion Forcing is a technique related to training diffusion models that progressively generate video frames through noise removal, and is said to help models learn temporal context appropriately. By combining these approaches, training is conducted to maintain content consistency even in long-form video generation.

The significance of this achievement is substantial from the perspective of video generation technology's practical applicability. Current video generation AI can handle short clips but has struggled to apply to long-form video content like movies or dramas. If this approach is commercialized, it could expand the possibility of AI generating long-form videos with consistency, potentially bringing new options to video production workflows.

Adobe is a company that develops creative tools including the video editing software Premiere Pro and the image editing software Photoshop, and has been actively pursuing AI integration into its own products. While it remains unclear how this research achievement will be utilized at the product level, as a technology to enhance video generation quality and consistency, it is noteworthy in relation to future product developments.

Video generation AI as a whole is a field where technological maturity has lagged behind text and image generation. With the addition of a temporal dimension, not just simple quality but 'narrative coherence' becomes a critical concern. Adobe Research's initiative represents one approach that directly addresses this challenge and may have the potential to influence industry-wide technological trends.