Kiji Inspector: Visualizing AI Agent Decision-Making

Hannes Hapke, an engineer at Dataiku, has published a method to reveal why AI agents select specific tools by analyzing them from the inside. A concrete approach to analyzing AI decision-making processes using the company's open-source tool 'Kiji Inspector' has been shared.

An AI agent is a system that autonomously selects actions toward a given goal and uses external tools or information sources as needed. While the practical application of such agent-type AI is progressing in recent years, the problem has been noted that the basis for decisions—"why was that tool chosen?"—is difficult for humans to see. This is a challenge that the entire industry faces when evaluating the trustworthiness and safety of AI.

Kiji Inspector focuses on 'decision tokens,' data generated at the moment when an AI makes a decision. By directly applying SAE (sparse autoencoders—compression and reconstruction models that extract only important features from input data) to this data, it reveals which internal features influence the decision. Additionally, a technique called 'causal ablation' is also employed. This is a method to verify causality by observing how output changes when a specific internal process in the AI is intentionally disabled.

From a technical implementation perspective, the approach patches the vLLM engine (an inference framework for running large language models at high speed) to enable external acquisition of activation states across each layer of the AI. Using this mechanism, one can track layer-by-layer what happens at each stage of decision-making. Furthermore, explanations are provided on how to incorporate it into monitoring workflows in production environments, with content designed with actual system deployment in mind.

This announcement is particularly significant because of its difference from conventional 'explainable AI' methods. The widely used 'chain of thought' approach—where AI explains its reasoning in words—cannot be certain that the explanation accurately reflects actual internal processing. Kiji Inspector takes a more rigorous approach by directly analyzing the numerical states within the model rather than relying on such surface-level explanations.

The tool is designed primarily for software architects and senior engineering leaders. For those responsible for deploying and operating AI agents in production environments, ensuring transparency in decision-making is an essential element in maintaining system reliability. Its release as open-source is also a factor that can promote adoption across a wide range of development environments.

As the scope of AI agent applications expands, demands for auditing decision processes and accountability are expected to increase. If techniques like Kiji Inspector that directly analyze internal states become widespread, engineers will be able to understand AI behavior more rigorously, leading to earlier problem detection and improved quality management accuracy. Going forward, attention will focus on whether similar approaches spread to other tools and frameworks.