AI Agents: Growing Corporate Concerns Over Evaluation Reliability

Half of enterprises have experienced situations where AI agents or large language models (LLMs) that passed internal evaluations caused user-facing failures in actual operations. According to the VB Pulse survey conducted in June 2026 (157 respondents from companies with 100 or more employees), one in four of these companies has experienced similar failures multiple times. It should be noted that this survey is a self-selected sample rather than a probability sample, so results should be interpreted as trends.

Despite these ongoing failures, enterprises are not slowing down the pace of automation. In the survey, 66% of respondents answered that they either 'already permit some deployments to production environments without human review' or are 'building such systems within the next 12 months.' However, only 5% of companies said they 'completely trust' automated evaluation tools, revealing a significant gap between expanding autonomy and trust in evaluations.

Why does this gap emerge? The background lies in the behavioral characteristics of AI agents. Traditional software testing confirms whether a specific input produces an expected output, but AI agents select procedures themselves, invoke external tools, and operate while referencing data, meaning results can differ with each execution. Even if individual decisions appear reasonable, a sequence of actions can lead to incorrect final results. For example, sends can occur without approval or confidential information can leak.

The most commonly cited reason for not trusting automated evaluations was 'large gaps between evaluation results and actual operational outcomes' at 29%, followed by 'bias or inconsistency' at 21%, 'difficulty explaining results' at 18%, and 'concerns about data breaches or privacy' at 17%. This ranking indicates that companies are concerned not with the speed or cost of evaluation, but with the core issue that 'scores cannot predict actual behavior in real-world settings.'

The U.S. National Institute of Standards and Technology (NIST) also points to similar challenges in its guidance on generative AI. It states that measurements taken in controlled environments may not apply directly to actual deployment environments where prompts, users, context, and operating conditions differ, and it calls for establishing field testing, post-deployment monitoring, and incident reporting processes.

Success in a single test only demonstrates that an agent 'can perform a task,' not that it 'can reliably perform it.' This is a challenge unique to agent AI. Currently, enterprises tend to release agents first and build governance and evaluation frameworks afterward, with control layer enhancements such as identity management, cost management, and orchestration following later.

The future focus can be seen as shifting from agent capability itself to 'ensuring reliability.' The reality that only 5% of companies can trust automated evaluations provides a foundation for growing demand for evaluation, monitoring, and governance-related tools. As agent AI adoption accelerates, building trustworthy evaluation infrastructure is likely to emerge as an industry-wide imperative.