Runway: How a Bug Became a Feature in AI Development

Runway, an AI video generation startup, publicly revealed the challenges it faced during model development and how it solved them. At VB Transform 2026, Ryan Phillips, head of enterprise products at Runway, took the stage to explain in detail how the company builds, evaluates, and releases AI models. Phillips noted that the insights shared contain lessons that can be applied even by companies that do not develop their own models.

Runway is an AI company engaged in research and development of a general-purpose "world model." The company's 'Runway Characters' is a video model that enables real-time two-way interaction with AI-generated avatars, designed to minimize latency in video generation. According to Phillips, five years ago, even creating blurry text or low-frame-rate video required creators to spend hundreds of hours manually stitching together individual frames. Today, Runway's model can generate interactive video on the spot.

The development process encountered unexpected obstacles. During real-time video generation, an AI avatar would drift away from the center of the screen—a bug that the Runway team spent weeks trying to solve with technical fixes. However, they ultimately decided to implement it as a front-end workaround rather than a back-end patch, converting the bug into a feature. This decision to turn a bug into a "feature" exemplifies the trade-off in AI development between pursuing perfect fixes and settling on practical solutions.

Phillips also presented concrete methods for the model evaluation process. Creating a high-quality evaluation set requires coordination across multiple departments—product, design, research, and sales—to establish a unified definition of what "good generation" means across the entire organization. Runway conducts repeated internal workshops where team members discuss generated samples together and share failure patterns down to the smallest details.

In designing evaluation sets, Phillips emphasized the need to address not only common use cases but also extreme edge cases. As an example, the team uses a non-human character called "Tooth" with no nose and unusual tooth shape to verify that the model behaves stably even with cases that exceed typical human facial structure. Additionally, even subtle artifacts—such as background nets distorting while the character's face remains intact—are strictly flagged as "failures."

Notably, despite this cutting-edge development, the evaluation management tool is simply an Excel spreadsheet. The team records test results daily and classifies outputs as either "minor failures" or "major failures." The pass rate that serves as the release criterion is set in advance, and the model is shipped once that threshold is achieved. Phillips explicitly stated, "Once we hit the criteria, we release. That's it," underscoring the importance of establishing objective, non-arbitrary release standards.

The series of examples demonstrates the importance of "practical judgment" over "perfectionism" in AI development. Bug conversion, cross-organizational quality definition, and rigorous use of simple evaluation tools are not techniques only large enterprises can practice—they are approaches applicable to development teams of any size. As generative AI-driven product development becomes more widespread, how to standardize the evaluation and release decision process is becoming an industry-wide challenge.