MIT and Harvard Announce AI Technology to Prevent "Role Drift"

Researchers from MIT and Harvard University have discovered a problem called "Role Drift" that occurs in systems combining multiple AI modules and have announced a technology called "Role Anchor" to suppress it. This phenomenon refers to a situation where individual modules inside the system fail to perform their original roles while the overall accuracy of the AI system appears to improve.

A composite AI system is a structure that distributes complex tasks among multiple specialized modules. For example, in a system called RAG (Retrieval-Augmented Generation), a "retrieval module" that searches external documents and a "reading module" that generates answers based on those results work in coordination. Originally, the reading module should answer based solely on search results, but during the process of optimizing the entire system, it has been confirmed that this module begins to answer using its own internal knowledge. Nevertheless, the final accuracy rate continues to increase, making the problem difficult to detect.

The background to this problem stems from AI pipeline optimization techniques. Engineers typically use reinforcement learning to optimize systems, using only a single metric—whether the final answer is correct (terminal accuracy)—as the evaluation criterion. Xiaoyang Cao, a co-author of the paper, told VentureBeat: "Terminal accuracy compresses the operation of a complex system as a whole into a single number. You can tell if the final answer is correct, but you cannot tell which module contributed or whether each performed its assigned role." Role Drift progresses quietly within this blind spot.

The Role Anchor proposed by the research team is a technology that constrains each module to prevent it from deviating from its assigned role during learning. For RAG's reading module, it enforces answering based on search results and blocks shortcuts to internal knowledge. This allows the system to suppress role drift while maintaining overall system performance.

The practical risk demonstrated by this research is clear. Cao points out: "The realistic risk for engineering teams is the possibility of deploying to production a pipeline that passes all end-to-end evaluations while the intended division of labor quietly collapses internally." Relying solely on terminal accuracy carries the danger of overlooking a state where the system "produces correct answers but does not operate for the right reasons."

The use of composite AI systems is becoming more widespread to distribute processing to cheaper models or to parallelize subtasks. As these systems are increasingly integrated into corporate operations and decision-making, the importance of verification methods to ensure each module operates as designed is expected to grow. Role Anchor is said to function as such a verification tool and is positioned as one approach to the new challenge of trustworthiness evaluation in composite AI.

Going forward, rather than relying solely on a single metric of terminal accuracy, the question of how to design a framework that independently evaluates module-level behavior will become a practical focus in AI system development. Role Anchor is one technical answer to this question, but as composite AI becomes more widely adopted in diverse forms, it is anticipated that more approaches stemming from the same concern will emerge.