Building an agent isn't just calling an LLM in a loop: it's managing conversational state, tools, memory, and, increasingly, other agents. AI Foundry Agent Service manages that complexity so the team can focus on business logic.
AI Foundry Agent Service (formerly Azure AI Agent Service) takes the conceptual model of OpenAI's Assistants API -- persistent threads, asynchronous runs, model-invokable tools -- and turns it into a managed service within Azure, with the key difference that state, files, and data connections live inside the customer's security perimeter, not in generic third-party storage.
An agent is defined once (model, instructions, tools) and then reused across threads, each representing an independent conversation with its own history. Sending a message triggers a run: the service decides whether the model responds directly or invokes one or more tools, executes those calls, and continues the cycle until producing a final response. This model cleanly separates the agent's definition from its conversation instances, which simplifies scaling the same agent to thousands of concurrent users.
The service includes ready-to-use tools: code interpreter (runs Python in an isolated sandbox for calculations or data analysis), file search (automatic indexing and retrieval over uploaded files), Bing grounding (answers grounded in current web search results), the Azure AI Search tool for RAG over your own indexes, and support for OpenAPI and Azure Functions tools that let you invoke any internal API documented with a standard specification, without writing integration code by hand.
Connected Agents lets a primary agent delegate subtasks to specialized agents -- one for document search, another for financial calculations, another for drafting -- registering them as tools of the orchestrator agent. The service manages routing the conversation between agents, sparing the team from having to implement a multi-agent orchestration state machine by hand.
There are two provisioning modes. Basic setup uses Microsoft-managed resources for thread and file storage, ideal for rapid prototyping. Standard (bring-your-own) setup connects the agent to the customer's own Storage account, Azure AI Search service, and Cosmos DB account, inside their VNet, which is mandatory for regulated workloads where conversation data can't reside on shared infrastructure managed solely by Microsoft.
For scenarios that go beyond what Connected Agents natively solves -- complex approval flows, agents running on different runtimes, heavy business logic in the orchestrator -- Semantic Kernel Agent Framework and AutoGen integrate directly with agents created in AI Foundry, letting you treat a service agent as just another participant in a multi-agent orchestration built in code.
Every run is traced with OpenTelemetry, visible in Application Insights: tokens consumed, tools invoked, latency per step. This is essential for catching agents stuck in tool-calling loops (a common and costly production problem) and for attributing token cost by agent and by customer when the same service serves multiple lines of business.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel