Cloud Services for Generative AI: Azure Offerings

A practical map of Azure's generative AI services: Microsoft Foundry, Azure OpenAI, deployment types, Foundry IQ, Agent Service, guardrails and cost.

Generative AI

Azure’s generative AI offering is easier to understand once you know that almost all of it now lives under one name: Microsoft Foundry. Models, retrieval, agents, safety filters and evaluation are features of a Foundry resource, and Azure OpenAI, the service most people know, is now one part of it. Around Foundry sit Azure Machine Learning and GPU virtual machines for teams that train or host their own models, and finished assistants such as GitHub Copilot and Microsoft 365 Copilot that you buy per user.

This article covers each of those layers as it stands today, explains the parts that trip people up (deployment types, data handling and the many renames), and shows a minimal call to a model. The companion articles cover AWS and Google Cloud, and Comparing Cloud Platforms for Generative AI puts the three side by side.

How the pieces fit together

flowchart TB
    Users["Users and applications"] --> Apps["Assistants: GitHub Copilot, Microsoft 365 Copilot, Azure Copilot"]
    Users --> Foundry["Microsoft Foundry"]
    Foundry --> Models["Foundry Models: OpenAI, Microsoft, Anthropic, open models"]
    Foundry --> IQ["Foundry IQ and Azure AI Search"]
    Foundry --> Agents["Foundry Agent Service"]
    Foundry --> Guard["Guardrails: Azure AI Content Safety"]
    Foundry --> Tools["Foundry Tools: Speech, Vision, Language, Document Intelligence"]
    Users --> Build["Azure Machine Learning and GPU VMs"]
    Foundry -.-> Entra["Microsoft Entra ID, private endpoints, Azure Monitor"]
    Build -.-> Entra

Most projects start and stay in the middle: you deploy a model in Foundry, call it from your code, and add retrieval and safety filters as the application grows. You only drop to Azure Machine Learning or raw GPU virtual machines when you need to train a model, fine-tune in ways Foundry doesn’t offer, or serve weights that aren’t in the catalog.

Everything is tied to the rest of Azure in the same way. Identities come from Microsoft Entra ID, traffic can stay on your virtual network through private endpoints, and logs go to Azure Monitor. That integration is the main reason teams already on Azure use it for AI rather than calling a model vendor directly.

A note on names

Azure’s AI products have been renamed often, and older tutorials, including an earlier version of this article, use names that no longer exist:

Older nameCurrent name
Azure AI Studio, then Azure AI FoundryMicrosoft Foundry
Azure OpenAI ServiceAzure OpenAI in Foundry Models (still widely called Azure OpenAI)
Cognitive Services, then Azure AI servicesFoundry Tools
Azure Cognitive SearchAzure AI Search
Azure Active DirectoryMicrosoft Entra ID

Microsoft also runs two versions of the Foundry portal. The current one works with Foundry projects that sit directly under a Foundry resource. The classic portal still serves older hub-based projects, which are built on Azure Machine Learning. New features arrive on Foundry resources and projects, so start new work there.

Microsoft Foundry and its models

A Foundry resource is an Azure resource (its kind is AIServices) that holds model deployments, connections and projects. A project is where a team keeps its agents, evaluations and files, and it inherits networking and access settings from the resource. You can have several projects per resource to separate teams or applications.

Which models it hosts

The Foundry catalog has thousands of models, but they fall into two groups that matter for contracts and support. Models sold directly by Azure are hosted by Microsoft, billed on your Azure invoice and covered by Microsoft’s product terms. Models from partners and the community are listed in the same catalog, and some are billed through Azure Marketplace with the provider’s own terms.

FamilyExamplesNotes
OpenAIGPT-5.x models, gpt-oss, image and audio modelsThe widest OpenAI selection outside OpenAI itself
MicrosoftPhi small language models, MAI image and voice modelsPhi models are also used on devices through Foundry Local
AnthropicClaude modelsGenerally available on Azure since mid-2026
Open and partner modelsMeta Llama, Mistral, DeepSeek, xAI GrokAvailability differs by region and deployment type

Two features help when you don’t want to pick one model and keep it forever. Model router is a deployment that looks at each request and sends it to a model from a pool, set to favour quality, cost or a balance of the two. And because every model sits behind the same Foundry endpoint, switching models is a change to a deployment name rather than to your code.

Model versions are retired on a published schedule. Check the Foundry model retirement page before you build on a specific version, and keep the deployment name in configuration so moving to a newer one is a one-line change.

Deployment types

You never call a catalog model directly. You create a deployment of it, give that deployment a name, and choose a deployment type. The type decides where your prompts are processed and how you pay.

Deployment typeWhere data is processedBillingFits
Global StandardAny Azure region where the model is deployedPer tokenThe default for most workloads; highest quotas and earliest access to new models
Data Zone StandardOnly within a Microsoft data zone, such as the US or the EUPer tokenData residency rules that allow a zone but not the whole world
Standard (regional)The region of your resourcePer tokenStrict single-region requirements, where the model is offered
Provisioned (global, data zone or regional)As above, per scopeReserved throughput unitsSteady, high-volume traffic with latency targets
Batch (global or data zone)As above, per scopeDiscounted per token, results within 24 hoursLarge jobs that can wait: classification, summarising archives, evaluations

The trade-off is between reach and residency. Global deployments get the most capacity because Azure can route your request to any region with free GPUs, but your prompt may be processed outside your geography. Data at rest stays in your resource’s geography either way. If compliance rules forbid processing outside a region or zone, choose the narrower type and accept that some models arrive there later, or not at all.

Calling a model

Azure OpenAI now exposes a v1 API that works with OpenAI’s standard client libraries: you point the client at your resource’s /openai/v1/ path and no longer pass an api-version. Authenticate with Entra ID instead of an API key. Locally that uses your az login session; in Azure it uses a managed identity, so no secret lives in your code.

import os

from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import OpenAI

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://ai.azure.com/.default"
)

client = OpenAI(
    base_url="https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/",
    api_key=token_provider,  # called before each request to get a fresh token
)

response = client.responses.create(
    model=os.environ["DEPLOYMENT_NAME"],  # your deployment name, not the catalog model ID
    input="Explain Azure private endpoints in two sentences.",
)
print(response.output_text)

Two details cause most first-day errors. The model parameter takes the name you gave the deployment, which may differ from the model’s catalog name. And the identity running the code needs a role on the resource that allows inference, such as Cognitive Services OpenAI User; being an Owner of the subscription is not enough on its own, because Owner covers management operations rather than data-plane calls. A 404 usually means the base URL doesn’t end in /openai/v1/; a 401 usually means a missing role or the wrong token scope.

Your call to Azure OpenAI fails because the model was not found, even though the model is in the catalog. What is the most likely cause?

  • Azure OpenAI only accepts API keys, not Entra ID tokens
  • The model parameter must be your deployment name, not the catalog model name
  • The v1 API requires an api-version query parameter

Correct! You call a deployment you created, by the name you gave it, so the model parameter has to match that name.

Grounding answers in your own data

A model only knows what it was trained on. To answer questions about your policies, products or tickets, you retrieve relevant passages from your own documents and include them in the prompt. That pattern is retrieval-augmented generation (RAG), and on Azure it is built on Azure AI Search.

Azure AI Search indexes your content, stores vector embeddings next to the text, and runs hybrid queries that combine keyword and vector matching, with an optional semantic ranker to reorder the results. You can build the whole pipeline yourself: chunk documents, embed them with an embedding model deployed in Foundry, write them to an index, and query it before each model call.

Foundry IQ is the managed layer on top. You define a knowledge base that points at sources such as Blob Storage, OneLake or existing search indexes, and agents query it through one endpoint. It uses agentic retrieval: the model breaks a complex question into several searches, runs them, and merges the results with references back to the source. Foundry IQ knowledge bases became generally available in June 2026, though some connectors are still in preview, so check the source you need. It still needs an Azure AI Search resource underneath, which is where much of its cost comes from.

sequenceDiagram
    participant U as User
    participant App as App or agent
    participant CS as Content Safety
    participant KB as Foundry IQ
    participant S as Azure AI Search
    participant M as Model deployment
    U->>App: Question
    App->>CS: Screen prompt
    App->>KB: Retrieve
    KB->>S: One or more searches
    S-->>KB: Matching chunks
    KB-->>App: Merged results with references
    App->>M: Instructions, chunks, question
    M-->>App: Draft answer
    App->>CS: Screen output
    App-->>U: Answer with citations

Two design points matter more than the choice of tool. First, retrieval should respect who is asking: if the source documents have permissions, the search results should too, or the assistant becomes a way to read files a user can’t open. Second, keep retrieval close to the data. If your documents live in SharePoint and OneLake, Azure’s retrieval services are a natural fit; if they live in another cloud, copying them across adds egress charges and a second copy to secure.

Safety filters and data handling

Every model deployment in Foundry has guardrails applied, built on Azure AI Content Safety. A default policy is on unless you change it, and you can create custom policies per deployment or agent.

ControlWhat it does
Harm filtersScore prompts and responses for hate, sexual content, violence and self-harm, with an adjustable threshold for each
Prompt ShieldsDetect direct jailbreak attempts from users and indirect attacks hidden in documents or emails the model reads
Groundedness detectionFlags responses that aren’t supported by the source material you supplied
Protected material detectionCatches known copyrighted text and public code appearing in outputs
BlocklistsBlock terms specific to your business

These filters reduce risk; they don’t remove it. Indirect prompt injection is the hard case: an agent that reads a web page or an email can be told by that content to do something you never intended. Filters catch some of it, and the rest is handled by limiting what the agent’s identity is allowed to do.

On data handling, Microsoft’s documentation states that prompts and completions sent to models sold directly by Azure are not used to train, retrain or improve the base models, and that fine-tuning data is not used to train foundation models without your permission. Some features do store data in your resource: the stateful Responses API, agent threads and stored completions keep message history so a conversation can continue. Read the data, privacy and security page for the models you use, and pay particular attention to how global deployments process prompts and to abuse monitoring retention.

Agents with Foundry Agent Service

An agent is a model that can call tools in a loop: search a knowledge base, call an API, run code, then decide what to do next. Foundry Agent Service is Azure’s managed runtime for them. It handles conversation state, tool calls, identity and tracing, and it can host agents written with Microsoft Agent Framework or other open-source frameworks, not only agents defined in the Foundry portal. Hosted agents became generally available in 2026.

Built-in tools include Foundry IQ for retrieval, web grounding, code execution, Azure Functions, OpenAPI endpoints and MCP servers. Each agent runs with an Entra identity, so you grant it access to resources with the same role assignments you use for applications.

The feature that sets Azure apart is distribution. A Foundry agent can be published into Microsoft 365 Copilot and Teams, where it is governed by the same Entra and Microsoft 365 admin controls as other agents in the tenant. If your users spend their day in Outlook and Teams, that saves you from building and securing a separate chat front end.

flowchart LR
    User["User in Teams or an app"] --> Agent["Foundry agent"]
    Agent --> Model["Model deployment"]
    Agent --> Know["Foundry IQ knowledge base"]
    Agent --> API["Your APIs and MCP servers"]
    Agent --> Code["Code execution"]
    Agent -.->|"Entra identity, least privilege"| API
    Agent -.-> Trace["Tracing and evaluation"]

The risk that comes with agents is that they act. Give each agent the narrowest role that works, keep write and delete actions behind a human confirmation step, and evaluate the agent on realistic tasks before release using Foundry’s evaluation and tracing tools rather than a handful of manual chats.

Your support agent reads customer emails and can update orders. What most limits the damage from an instruction hidden in an email?

  • A narrowly scoped identity for the agent, with confirmation before changes
  • Using the largest available model
  • Turning off the default guardrails so the agent sees the full email

Correct! Prompt Shields help, but least privilege and human confirmation limit what a misled agent can actually do.

Beyond Foundry: training, hosting and devices

Azure Machine Learning

Azure Machine Learning is the platform for building and training your own models: managed compute clusters, experiment tracking, pipelines, a model registry and managed online endpoints for serving. Use it when you train a model from your own data, run a fine-tuning recipe Foundry doesn’t offer, or host open weights with custom inference code. It is more work than a Foundry deployment, because you manage the compute, the scaling and the patching of your serving image.

GPU virtual machines

For large training runs and self-managed inference, Azure offers ND-series virtual machines with NVIDIA GPUs, up to the ND GB300 v6 series built on Blackwell Ultra GPUs in rack-scale NVLink systems. These are the same kind of machines Azure uses to serve OpenAI’s models. Capacity for the newest series is limited and usually needs a quota request or a reservation, so plan well ahead. The Azure VM article covers the basics of creating one.

Foundry Tools and Foundry Local

Foundry Tools, formerly Azure AI services, are task-specific APIs: Azure Speech for transcription and text to speech, Azure Vision, Azure Language, Translator and Document Intelligence for extracting fields from forms and invoices. They are often cheaper and more predictable than asking a general model to do the same narrow task, and they plug into agents as tools.

Foundry Local runs selected models, such as Phi, on the user’s own computer, using its CPU, GPU or NPU. It became generally available in April 2026 and suits applications that must work offline or keep data on the device.

flowchart TD
    Q{"Is the model in the Foundry catalog?"} -->|Yes| D["Deploy it in Foundry"]
    Q -->|No| W{"Do you train or fine-tune it yourself?"}
    W -->|Yes| ML["Azure Machine Learning"]
    W -->|"No, just serve it"| H{"Need full control of the stack?"}
    H -->|No| ML
    H -->|Yes| VM["GPU VMs or AKS"]
    D --> Dev{"Must it run offline on devices?"}
    Dev -->|Yes| Local["Foundry Local with a small model"]
    Dev -->|No| Done["Call the Foundry deployment"]

Assistants you buy per user

Microsoft also sells finished assistants, priced per user rather than per token. For many organisations these are the first generative AI services they pay for.

AssistantWhat it does
GitHub CopilotCode completion, chat and coding agents in editors and on GitHub
Microsoft 365 CopilotAssistant in Word, Excel, Outlook and Teams, grounded in the user’s Microsoft 365 data
Copilot StudioLow-code builder for agents that run in Microsoft 365 Copilot and Teams
Azure CopilotAssistant in the Azure portal, with agents for troubleshooting, deployment, cost optimization and resiliency

The line between these and Foundry is who builds the experience. If a finished assistant does the job, buying it is faster than building one. Build in Foundry when the workflow is specific to your business, needs your own data and tools, or has to run inside your own application.

What it costs

Token prices change often and differ by model, version, region and deployment type, so this article doesn’t print them. Use the official Azure OpenAI pricing page, the pricing shown for each model in the Foundry catalog, and the Azure pricing calculator.

What you can plan for is the shape of the bill:

Cost itemHow it is chargedHow to control it
Model tokensPer input and output token, with cached input billed at a lower rate on supported modelsSmaller models for simple tasks, model router, shorter prompts, prompt caching
Batch jobsDiscounted per tokenMove anything that can wait 24 hours to a batch deployment
Provisioned throughputReserved throughput units, hourly or on a monthly or yearly reservationBuy only for steady load you have measured on pay-per-token first
Azure AI SearchPer search unit for the service tier, plus the semantic rankerRight-size the tier; it often costs as much as the model in RAG apps
Agents and toolsModel tokens plus charges for tools such as code execution and web groundingCap tool calls per run and log them
Monitoring and storageLog ingestion in Azure Monitor, storage for files and threadsSample traces in production and set retention

New Azure accounts get free credits, covered in the Azure free tier article, but generative AI usage burns through them quickly. Set a budget with alerts on the subscription before you deploy anything public, and rate-limit your endpoint per user, for example with Azure API Management in front of Foundry, so one script can’t spend a month’s budget in an afternoon.

You need to summarise two million archived support tickets by the end of the week. Which deployment type fits best?

  • Provisioned throughput reserved for a year
  • Global Standard, calling the model in a loop
  • A Global or Data Zone Batch deployment

Correct! Batch deployments are discounted and return results within 24 hours, which suits a large job with a deadline days away.

Getting started

A sensible first project takes an afternoon:

  1. Create a Foundry resource and a project in the Foundry portal, in a region that offers the models you want.
  2. Deploy a small current model as Global Standard, or Data Zone Standard if residency matters.
  3. Assign your user, and later your application’s managed identity, an inference role on the resource. Don’t use API keys for anything beyond a first test.
  4. Try prompts in the portal playground, then call the deployment from code as shown above.
  5. Add a knowledge base over a small set of documents, and run an evaluation on 20 to 50 real questions before showing it to anyone.
  6. Set a budget alert, and review the guardrail policy on the deployment.

If you prefer the CLI, the Foundry resource and a deployment can be created with az cognitiveservices account create --kind AIServices and az cognitiveservices account deployment create. Introduction to Microsoft Azure covers the Azure basics, and Building Your First Generative AI Model is a hands-on start on the modelling side. For the different routes to OpenAI’s models, see Exploring OpenAI’s GPT Models.

Further reading

Key takeaways

  • Microsoft Foundry is the home of Azure’s generative AI: model deployments, Foundry IQ retrieval, Foundry Agent Service and guardrails all live in a Foundry resource. Azure AI Studio, Azure AI Foundry and Azure AI services are old names.
  • Foundry hosts OpenAI’s models alongside Microsoft’s Phi and MAI models, Anthropic’s Claude and open models, all behind one endpoint and Entra ID authentication.
  • The deployment type decides where prompts are processed and how you pay: Global Standard for reach, Data Zone or regional for residency, Provisioned for steady load, Batch for work that can wait.
  • Call deployments with the standard OpenAI client and the v1 API, authenticate with Entra ID rather than keys, and keep the deployment name in configuration.
  • Agents can be published straight into Microsoft 365 Copilot and Teams, which is Azure’s strongest advantage if your users already work there.
  • Budget for Azure AI Search, agent tools and logging as well as tokens, and check current prices on the official pricing pages.

Part 27 of 30 · Introduction to Generative AI

  1. 01Introduction to Generative AI: What Is It?
  2. 02The History of Generative AI: From Concept to Reality
  3. 03Key Definitions and Concepts in Generative AI
  4. 04Pioneers of Generative AI: Meet the Inventors
  5. 05Understanding Neural Networks: The Backbone of AI
  6. 06Deep Learning Basics: Layers and Learning Rates
  7. 07Generative vs. Discriminative Models: What's the Difference?
  8. 08Overview of Generative Models: VAEs, GANs, and More
  9. 09Introduction to Variational Autoencoders (VAEs): From Basics to Advanced Applications
  10. 10Introduction to Generative Adversarial Networks (GANs)
  11. 11The Role of Transformers in Generative AI
  12. 12Applications of Generative AI in Everyday Life
  13. 13Ethical Considerations in Generative AI: Navigating the New Frontier
  14. 14Generative AI in Art and Music: The Creative Revolution
  15. 15Case Study: AI-Generated Artwork
  16. 16Generative AI in Text Generation
  17. 17Exploring OpenAI's GPT Models: Word Up! Bot
  18. 18Real-World Use Cases: Deepfakes and Their Impact
  19. 19Generative AI in Image and Video Synthesis
  20. 20Introduction to Reinforcement Learning in Generative AI
  21. 21The Future of Generative AI: Trends to Watch
  22. 22Generative AI in Healthcare: Opportunities and Challenges
  23. 23Generative AI in Finance: Risk Modeling and Beyond
  24. 24Generative AI in Marketing and Personalization
  25. 25Cloud Services for Generative AI: AWS Offerings
  26. 26Cloud Services for Generative AI: GCP Offerings
  27. 27Cloud Services for Generative AI: Azure Offerings
  28. 28Comparing Cloud Platforms for Generative AI
  29. 29Recap and Q&A: Introduction to Generative AI
  30. 30Building Your First Generative AI Model: A Practical Guide
Ready to check your understanding?Take the Word Up! Bot quiz
esc

Type to search all articles · ↑↓ to move · ↵ to open