Bringing AI Agents to Production: Scalable Architecture with Ray

A presentation was made public that systematically presents the challenges faced when transitioning AI agents from prototype stages to production environments and the solutions to address them. The presenters, Deepak Mohanakumar Chandramohuli and Bhumik Vinodkumar Thakkar, share infrastructure design insights leveraging the distributed computing framework Ray.

In AI agent development, it is common practice to first build a small prototype that runs in a local environment. While prototypes using Python's lightweight web framework FastAPI are easy to create, delivering a service to many users in a stable manner requires a completely different design approach. In production environments, handling large volumes of requests while tolerating failures and enabling automatic scaling up and down of system capacity becomes essential.

The presentation explained the topology for building a RayCluster on Kubernetes—a platform that automatically manages applications in the cloud—specifically addressing how to position and coordinate server clusters. Concrete insights were also provided on handling stateful memory processing, which allows AI agents to retain conversation context and state, as well as implementing autoscaling mechanisms that automatically adjust system capacity in response to fluctuations in access volume. Additionally, methods for designing a robust control plane that governs the entire system were discussed.

Furthermore, the presentation covered the installation of guardrails to prevent AI from executing incorrect operations, ensuring availability through multi-region deployments spanning multiple geographic areas, and building LLM-as-a-judge evaluation pipelines where another LLM (large language model) automatically assesses the quality of AI agent outputs. LLM-as-a-judge is a mechanism where a separate AI model plays the evaluator role instead of having humans check each output individually, representing an emerging approach to automate quality assurance in large-scale operations.

The initiatives presented in this talk address a common industry-wide challenge: the transition from the stage of "building something that works" to "operating stably," a shift the entire AI development community is facing. A particularly notable aspect is the framework-agnostic design approach, which avoids dependence on specific AI frameworks and reflects a design philosophy that prevents the need for rebuilding foundational systems when organizations update or change tools in the future.

As AI agent operationalization accelerates across industries, the accumulation of design knowledge bridging the gap between prototypes and production operations represents significant value for the entire developer community. Going forward, whether operational techniques such as multi-region deployments and automated evaluation pipelines become standard elements in enterprise AI agent adoption is considered an important focal point.