The Next Challenge for AI Agents: Designing "Memory"

In AI agent development, the practice of increasing the volume of text processed (token count) became a temporary evaluation metric, but there is growing resistance to this approach. During the first half of 2026, competition over token consumption was observed, but criticism has intensified that this measures only activity volume rather than actual results. Behind this debate, a more fundamental design challenge has emerged.

The core of the problem lies in the finite nature of the "context window." A context window refers to the frame of information that an AI can reference at once, and there is a limit to how much information can be packed into it. According to insights based on conversations with over 100 customers conducted across 15 cities and 6 countries, many organizations are arriving at the same conclusion. The challenge is not "fitting more information in" but "determining what should be included."

The concept emerging as an answer is "memory." However, this use of the term differs from how "memory" typically refers to the context window itself. It refers to memory as an independent system that exists persistently outside the model and can be searched and referenced as needed. Rather than information that disappears when a session ends, it means a structure where information is accumulated and managed in a form that can be reused in subsequent processing.

A properly designed memory system for agents is understood to require three functions. First, it should store results generated by AI in past sessions so that inferences made at considerable cost can be leveraged in future interactions. Second, it should apply "role-based access control" to control who and which teams can access stored memories, enabling safe information sharing within the enterprise. Third, it should enable searching past information by "meaning"—that is, it should feature "semantic search" that retrieves necessary information based on the similarity of sentence intent and meaning rather than exact keyword matching.

This third point is deeply connected to database history. Over the past 60 years, database technology has evolved specializing in storing and searching structured data with precise conditions. Meanwhile, the information that AI agents handle is unstructured text output by generative AI, requiring mechanisms to search by semantic proximity. This represents a different design philosophy from traditional database technology, and whether both can be handled within the same platform significantly influences the difficulty of implementation.

Teams that are currently implementing this successfully are reported to be completing semantic search, access control, and generated content storage within a single data platform rather than stitching together separate systems. AI agent development has only about 18 months of history, and compared to the 60 years of database technology, it can be said to be at the beginning of the learning curve. A standard configuration like the "LAMP Stack" in web development does not yet exist in agent development, and the current state is that each team is continually reconsidering architecture from scratch.

The design of a memory system can be viewed as the foundation for AI agents to transition from being tools that perform single processing tasks to entities that work while maintaining context continuously. There is a shift underway from a thinking of "quantity" focused on fitting information into the context window to discussion of "quality and design" regarding what to remember when and how to retrieve it. Agent development is at a stage where the focus is transitioning in this direction.