NVIDIA Develops Cross-Model KV Cache Transfer Technology for AI Models

NVIDIA researchers have developed a technology capable of significantly reducing cost and latency in systems combining multiple AI models. Known as 'Cross-Model KV Cache Transfer,' this technique enables conversational history data held by one model to be converted into a format usable by another model and seamlessly inherited. Experiments confirmed that for compatible model combinations, 2.7 to 25 times faster processing can be achieved compared to conventional recomputation methods, while maintaining accuracy of up to 98% of standalone model performance.

To understand the problem this technology aims to solve, one must know how LLMs (Large Language Models) process conversations. When an LLM receives input, it first performs a 'prefill' operation, reading all prior conversation content and storing it in a memory region called KV Cache (Key-Value Cache). In the subsequent 'decode' phase, it generates the next tokens (word chunks) sequentially while referencing this cache. The presence of the cache eliminates the need to recalculate past conversation content from scratch each time, thereby optimizing processing efficiency.

Problems arise when attempting to switch models mid-conversation. For example, in AI agent systems, design patterns exist where lightweight processing is delegated to smaller models, then handed off to larger models when complex reasoning becomes necessary. However, because different models have different internal structures, KV Cache data formats lack compatibility. Consequently, every time a switch occurs, the receiving model must recalculate all prior conversation from scratch, leading to increased computational cost and response latency. The longer the conversation and the larger the models, the more this recomputation cost grows.

The technique developed by NVIDIA solves this problem through 'linear transformation'—a relatively simple mathematical operation. By performing mapping to convert the sending model's KV Cache into the format expected by the receiving model, memory data can be inherited without re-executing the prefill operation. Another distinctive feature of this technique is that it does not employ computationally expensive mechanisms such as deep learning models.

The significance of this technology extends beyond mere processing speed improvement. Multi-model workflows combining multiple LLMs are positioned as promising approaches for achieving cost efficiency and capability optimization. However, the recomputation cost incurred by each model switch has served as a bottleneck, becoming a barrier to practical implementation. Cross-Model KV Cache Transfer has the potential to lower this barrier and provide design options for enterprise-grade AI systems handling long-term tasks.

Conversely, experimental results represent only values for 'compatible model combinations,' and whether similar effects can be achieved across all model pairs remains unclear at present. Transfer accuracy between models with significant architectural differences and behavior in actual production environments remain areas requiring future validation. Key focus points going forward include which models and frameworks will support this technology and under what conditions it tends to be most effective.