Azure AI Foundry is Microsoft's answer to a real problem: building with generative AI means coordinating models, data, evaluation, security, and deployment in a single lifecycle, not in isolated tools.
In November 2024, at Ignite, Microsoft consolidated everything that used to live scattered across Azure AI Studio, Azure Machine Learning, and individual cognitive services under a single product: Azure AI Foundry. It's not a cosmetic rebrand. The real change is one of mental model: instead of thinking in terms of separate "cognitive services" (Language, Vision, Speech, OpenAI), AI Foundry organizes everything around projects that consume a shared model catalog, agent tooling, and shared evaluation pipelines.
For an engineering team, this translates into less operational friction: a single identity control plane (Microsoft Entra ID), a single consolidated billing model, and a single observability surface via Application Insights and Azure Monitor.
The top-level organizational unit is the hub, an Azure resource that centralizes compute, network configuration, and connections to dependent services: Azure AI Search for RAG, Azure Key Vault for secrets, Application Insights for traces, and storage accounts for artifacts. One or more projects live inside a hub -- the actual workspace where a team deploys models, builds agents, and runs evaluations.
This separation matters at enterprises: it lets a central platform (governed by the architecture team, for example) define network policies and connections at the hub level, while each product team operates autonomously within its project without being able to bypass those boundaries.
AI Foundry exposes a model catalog that includes Azure OpenAI Service models (GPT-4o, GPT-4.1, the o1/o3 series, text-embedding-3 embeddings), Meta models (Llama 3.x and 4), Mistral, Cohere, DeepSeek, xAI Grok, and Microsoft's own Phi family optimized for edge and low-cost inference. Each model deploys either as Model as a Service (a serverless endpoint billed per token) or as managed compute (a dedicated GPU instance you manage and pay for by the hour).
The choice between the two isn't trivial: serverless is simpler but less predictable in cost at scale; managed compute gives fine-grained control over latency and network isolation but requires managing GPU quota.
The web portal is useful for rapid prototyping, but the production workflow runs through the `azure-ai-projects` SDK, which unifies agent creation, model invocation, and evaluation under a single client.
AI Foundry integrates predefined evaluators (groundedness, relevance, coherence, fluency, similarity) and lets you run them both in development and in production over real traces captured via OpenTelemetry. This closes a historical gap: evaluating a RAG pipeline used to require building your own tooling with loose notebooks; now it's part of the project's lifecycle, with results versioned alongside the model and prompt that generated them.
Every project inherits Azure AI Content Safety controls, Prompt Shields, and the hub's network policies. You can require Private Link for all inference traffic, apply Customer-Managed Keys for encryption at rest, and use Azure Policy to block creating connections to unapproved models or regions. This is critical in regulated sectors where the security team needs to audit which models are being used and from which network, without relying on each development team to report it manually.
If your organization already operates on Azure, has Microsoft Entra ID as its corporate identity, and needs more than one model or more than one agent in production, AI Foundry significantly reduces integration cost: one access control, one billing system, one observability layer. If the use case is a single chat endpoint with no need for agents or continuous evaluation, calling Azure OpenAI Service directly remains valid and simpler. The practical rule: AI Foundry wins when there's more than one model, more than one team, or more than one agent in play.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel