Building Production-Ready AI Agents with Knowledge Graphs

AI engineer Cassie Shum has positioned knowledge graphs as a core foundation for AI agent systems and unveiled design patterns capable of withstanding operational demands in production environments. The presentation advocates for more robust architectures that transcend simple RAG (Retrieval-Augmented Generation).

RAG is a technique in which AI retrieves and references external information when generating answers, and has been widely adopted in recent generative AI systems. However, when attempting to integrate it into actual business systems, challenges often arise such as inability to accurately grasp information relevance and difficulty in tracing the basis of processing. Building on these limitations, Shum explained that leveraging knowledge graphs can address these issues. A knowledge graph is a form of database that structures relationships between information in a web-like manner, and is characterized by its ability to maintain context and relevance that simple keyword searches cannot capture.

Shum presented four architecture patterns that have been operationalized for production environments. First is 'context bundling,' a technique that aggregates related information and passes it to AI, ensuring no necessary context is missed in processing. Second is 'decision provenance,' a mechanism that records and traces the path of how AI made decisions based on which information. Third is an approach called 'code as truth,' referring to the explicit management of system behavior definitions as code. Fourth is 'agent observability,' a design philosophy that enables external monitoring of how AI agents operate internally.

Furthermore, Shum introduced an engineering harness built on top of the knowledge graph. This harness serves three functions: optimization of feedback loops, optimization of token usage, and maintenance of system-wide reliability. A token is the unit of text that AI processes, and since increased usage leads to higher costs and processing time, its optimization becomes a critical operational concern.

The growing attention to these efforts is driven by the acceleration of efforts to deploy AI agents in actual business environments. Beyond accuracy evaluation at the research stage, operational requirements such as continuous stable operation and explainability of decision-making are now being questioned, and the necessity of incorporating these elements from the design phase has become paramount.

A crucial perspective presented by Shum's work is the shift from 'buildable' to 'safely operable at scale' for AI agents. In particular, patterns such as decision provenance and agent observability carry significant meaning from the standpoint of AI decision accountability. By establishing mature technology like knowledge graphs as a foundation, the direction of structurally mitigating the uncertainty of generative AI can be viewed as a promising approach for future production AI system design.

It can be said that the field has entered a phase where the design quality of the overall system is questioned with the same importance as the selection of high-accuracy models in AI agent operationalization. The combination of four patterns and the engineering harness provides a framework that engineers and development organizations can readily reference as agnostic design principles independent of specific models. Going forward, attention should be paid to in which actual operational environments these design patterns are validated and what level of reliability they can guarantee.