Azure OpenAI Service Hardening

Enforce Entra ID (managed identity) authentication and disable local API key authentication via disableLocalAuth: true. API keys are long-lived credentials susceptible to leakage in code repositories, CI pipelines, and application logs. Disabling them forces all callers to present an Entra ID token — enabling full caller attribution in audit logs, per-identity token rate-limiting, and Conditional Access enforcement. See azure-iam-06 — managed identity for the prerequisite managed-identity setup.

Remediation — Azure CLI

# Azure CLI 2.x
# Audit: find Azure OpenAI resources with local auth enabled
az cognitiveservices account list \
  --query "[?kind=='OpenAI' && properties.disableLocalAuth!=true].{name:name, rg:resourceGroup}" \
  --output table

# Remediate: disable local API key authentication
az cognitiveservices account update \
  --name "${AOAI_ACCOUNT}" \
  --resource-group "${RG}" \
  --custom-domain "${AOAI_ACCOUNT}" \
  --api-properties '{"DisableLocalAuth": true}'

Remediation — Terraform

# Terraform AzureRM provider ~> 3.0
resource "azurerm_cognitive_account" "aoai" {
  name                = var.account_name
  location            = var.location
  resource_group_name = var.resource_group_name
  kind                = "OpenAI"
  sku_name            = "S0"
  local_auth_enabled  = false

  identity {
    type = "SystemAssigned"
  }
}

Remediation — Bicep

targetScope = 'resourceGroup'

@description('Azure OpenAI account name.')
param accountName string

param location string = resourceGroup().location

resource aoai 'Microsoft.CognitiveServices/accounts@2024-10-01' = {
  name: accountName
  location: location
  kind: 'OpenAI'
  sku: { name: 'S0' }
  identity: { type: 'SystemAssigned' }
  properties: {
    customSubDomainName: accountName
    disableLocalAuth: true            // Entra ID only — disables shared keys
    publicNetworkAccess: 'Disabled'
    networkAcls: { defaultAction: 'Deny' }
  }
}

Remediation — Pulumi (TypeScript)

import * as pulumi from "@pulumi/pulumi";
import * as cs from "@pulumi/azure-native/cognitiveservices";
import * as resources from "@pulumi/azure-native/resources";

const rg = new resources.ResourceGroup("aoai-rg");

new cs.Account("aoai", {
  resourceGroupName: rg.name,
  kind: "OpenAI",
  sku: { name: "S0" },
  identity: { type: cs.ResourceIdentityType.SystemAssigned },
  properties: {
    customSubDomainName: "aoai-hardened",
    disableLocalAuth: true,            // Entra ID only — disables shared keys
    publicNetworkAccess: "Disabled",
    networkAcls: { defaultAction: "Deny" },
  },
});

Compliance mapping

AzureActivity
          | where ResourceId has "Microsoft.CognitiveServices/accounts"
          | where OperationNameValue in ("Microsoft.CognitiveServices/accounts/write", "Microsoft.CognitiveServices/accounts/regenerateKey/action")
          | extend body = tostring(parse_json(Properties).requestbody)
          | where body has "\"disableLocalAuth\":false" or OperationNameValue endswith "/regenerateKey/action"
          | project TimeGenerated, Caller, ResourceId, OperationNameValue, body
          | order by TimeGenerated desc
          | take 200

Equivalent on: AWS — IAM least privilege · GCP — SA scoping · OCI — IAM least privilege

Enable Azure AI Content Safety Prompt Shields for both direct user-prompt injection (jailbreak attempts) and document/RAG indirect injection (malicious instructions embedded in retrieved documents). Prompt Shields is a SEPARATE service from the Azure OpenAI content filter — it is an API endpoint in Azure AI Content Safety that detects injection attacks before the prompt reaches the model. Configure for both userPromptAnalysis (direct injection) and documentsAnalysis (indirect/RAG injection). Prompt Shields went GA in 2024.

Important architectural distinction: Prompt Shields (this control) and the Azure OpenAI content filter (azure-genai-02) are architecturally distinct: Prompt Shields detects injection attacks at the input layer; content filters moderate harm categories at the output layer. Both are required. Using only the content filter does not protect against prompt injection; using only Prompt Shields does not moderate harmful output.

Audit — Azure CLI

# Azure CLI 2.x
# Check current Content Safety configuration on the Azure OpenAI resource
az cognitiveservices account show \
  --name "${AOAI_ACCOUNT}" \
  --resource-group "${RG}" \
  --query "properties.contentSafetyConfig"

Configuration — Azure AI Content Safety REST API

{
  "userPromptAnalysis": {
    "disable": false
  },
  "documentsAnalysis": {
    "disable": false
  }
}

Submit via POST https://{endpoint}/contentsafety/text:shieldPrompt?api-version=2024-09-01 with your prompt and retrieved documents as the request body. Integrate Prompt Shields as a pre-flight check in your application before forwarding to the Azure OpenAI inference endpoint.

Remediation — Bicep

targetScope = 'resourceGroup'

@description('Azure OpenAI account name.')
param accountName string

@description('RAI policy with Prompt Shields jailbreak + indirect-attack detection enabled.')
resource raiPolicy 'Microsoft.CognitiveServices/accounts/raiPolicies@2024-10-01' = {
  name: '${accountName}/prompt-shields-enforced'
  properties: {
    mode: 'Blocking'
    contentFilters: [
      { name: 'jailbreak',           blocking: true, enabled: true, source: 'Prompt' }
      { name: 'indirect_attack',     blocking: true, enabled: true, source: 'Prompt' }
      { name: 'protected_material_text', blocking: true, enabled: true, source: 'Completion' }
      { name: 'protected_material_code', blocking: true, enabled: true, source: 'Completion' }
    ]
  }
}

Remediation — Pulumi (TypeScript)

import * as pulumi from "@pulumi/pulumi";
import * as cs from "@pulumi/azure-native/cognitiveservices";

new cs.RaiPolicy("prompt-shields-enforced", {
  resourceGroupName: "<rg>",
  accountName: "<aoai-account>",
  raiPolicyName: "prompt-shields-enforced",
  properties: {
    mode: "Blocking",
    contentFilters: [
      { name: "jailbreak",               blocking: true, enabled: true, source: "Prompt" },
      { name: "indirect_attack",         blocking: true, enabled: true, source: "Prompt" },
      { name: "protected_material_text", blocking: true, enabled: true, source: "Completion" },
      { name: "protected_material_code", blocking: true, enabled: true, source: "Completion" },
    ],
  },
});

Compliance mapping

AzureDiagnostics
          | where ResourceProvider == "MICROSOFT.COGNITIVESERVICES" and Category == "RequestResponse"
          | extend filterJB = tostring(parse_json(properties_s).contentFilterResults.jailbreak.filtered)
          | extend filterIA = tostring(parse_json(properties_s).contentFilterResults.indirect_attack.filtered)
          | where filterJB == "false" or filterIA == "false"
          | project TimeGenerated, Resource, identity_claim_appid_g, filterJB, filterIA, properties_s
          | order by TimeGenerated desc
          | take 200

Equivalent on: AWS — Guardrails prompt attack · GCP — safety filters (partial) · OCI — PI component

Configure the Azure OpenAI content filter (RAI policy) with non-default severity thresholds for the four harm categories — Hate, Sexual, Self-harm, and Violence — applied to both prompt input and completion output. The default content filter is not an acceptable sole control; it must be explicitly configured at recommended or stricter thresholds and attached to each model deployment. RAI policy threshold configuration requires the Azure OpenAI REST API; az cognitiveservices account update does not support RAI policy threshold configuration (known CLI limitation — use REST API or Terraform azurerm_cognitive_account_rai_policy).

Audit — Azure CLI

# Azure CLI 2.x
# Locate the resource custom domain (needed for REST API calls)
az cognitiveservices account show \
  --name "${AOAI_ACCOUNT}" \
  --resource-group "${RG}" \
  --query "properties.customSubDomainName"

# Then retrieve current RAI policies via REST API:
# GET https://{endpoint}/openai/rai/policies?api-version=2024-10-01

Configuration — Azure OpenAI REST API (RAI policy)

{
  "name": "recommended-baseline",
  "type": "UserManaged",
  "contentFilters": [
    {"name": "Hate",      "blocking": true, "enabled": true, "allowedContentLevel": "Low",    "source": "Prompt"},
    {"name": "Hate",      "blocking": true, "enabled": true, "allowedContentLevel": "Low",    "source": "Completion"},
    {"name": "Sexual",    "blocking": true, "enabled": true, "allowedContentLevel": "Low",    "source": "Prompt"},
    {"name": "Sexual",    "blocking": true, "enabled": true, "allowedContentLevel": "Low",    "source": "Completion"},
    {"name": "SelfHarm",  "blocking": true, "enabled": true, "allowedContentLevel": "Low",    "source": "Prompt"},
    {"name": "SelfHarm",  "blocking": true, "enabled": true, "allowedContentLevel": "Low",    "source": "Completion"},
    {"name": "Violence",  "blocking": true, "enabled": true, "allowedContentLevel": "Medium", "source": "Prompt"},
    {"name": "Violence",  "blocking": true, "enabled": true, "allowedContentLevel": "Medium", "source": "Completion"}
  ]
}

Submit via POST /openai/rai/policies/{policyName}?api-version=2024-10-01. Then attach the policy to each deployment: PATCH /openai/deployments/{deploymentName}?api-version=2024-10-01 with body {"rai_policy_name": "recommended-baseline"}.

Remediation — Terraform

# Terraform AzureRM provider ~> 3.0
# azurerm_cognitive_account_rai_policy — verify argument completeness
# for all harm categories against Terraform Registry at deployment time
resource "azurerm_cognitive_account_rai_policy" "baseline" {
  name                 = "recommended-baseline"
  cognitive_account_id = azurerm_cognitive_account.aoai.id
  mode                 = "Blocking"

  content_filter {
    name               = "Hate"
    filter_enabled     = true
    blocking_enabled   = true
    severity_threshold = "Low"
    source             = "Prompt"
  }
  content_filter {
    name               = "Hate"
    filter_enabled     = true
    blocking_enabled   = true
    severity_threshold = "Low"
    source             = "Completion"
  }
  content_filter {
    name               = "Violence"
    filter_enabled     = true
    blocking_enabled   = true
    severity_threshold = "Medium"
    source             = "Prompt"
  }
  content_filter {
    name               = "Violence"
    filter_enabled     = true
    blocking_enabled   = true
    severity_threshold = "Medium"
    source             = "Completion"
  }
}

Remediation — Bicep

targetScope = 'resourceGroup'

@description('Azure OpenAI account hosting the deployment.')
param accountName string
@description('Deployment name.')
param deploymentName string
@description('Content-filter policy name (must exist on the account).')
param contentFilterPolicy string = 'Microsoft.DefaultV2'

resource aoai 'Microsoft.CognitiveServices/accounts@2024-10-01' existing = {
  name: accountName
}

resource deployment 'Microsoft.CognitiveServices/accounts/deployments@2024-10-01' = {
  parent: aoai
  name: deploymentName
  sku: { name: 'Standard', capacity: 10 }
  properties: {
    model: { format: 'OpenAI', name: 'gpt-4o', version: '2024-11-20' }
    raiPolicyName: contentFilterPolicy
  }
}

Compliance mapping

AzureActivity
          | where OperationNameValue startswith "Microsoft.CognitiveServices/accounts/raiPolicies/"
          | extend body = tostring(parse_json(Properties).requestbody)
          | where OperationNameValue endswith "/delete" or body has "\"allowedContentLevel\":\"high\"" or body has "\"allowedContentLevel\":\"medium\""
          | project TimeGenerated, Caller, ResourceId, OperationNameValue, body
          | order by TimeGenerated desc
          | take 200

Equivalent on: AWS — content filter · GCP — safety filters · OCI — content moderation

Deploy a Private Endpoint via Azure Private Link for the Azure OpenAI resource and disable public network access. Without this control, inference traffic (including prompts and completions) traverses the public internet. See azure-net-04 — private endpoint pattern for the general pattern; this control applies it specifically to the Azure OpenAI Cognitive Services resource.

Remediation — Azure CLI

# Azure CLI 2.x
# Disable public network access on the Azure OpenAI resource
az cognitiveservices account update \
  --name "${AOAI_ACCOUNT}" \
  --resource-group "${RG}" \
  --custom-domain "${AOAI_ACCOUNT}" \
  --public-network-access Disabled

# Verify: confirm public access is disabled
az cognitiveservices account show \
  --name "${AOAI_ACCOUNT}" \
  --resource-group "${RG}" \
  --query "properties.publicNetworkAccess"

Remediation — Terraform

# Terraform AzureRM provider ~> 3.0
resource "azurerm_cognitive_account" "aoai_private" {
  name                          = var.account_name
  location                      = var.location
  resource_group_name           = var.resource_group_name
  kind                          = "OpenAI"
  sku_name                      = "S0"
  public_network_access_enabled = false

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_private_endpoint" "aoai_pe" {
  name                = "${var.account_name}-pe"
  location            = var.location
  resource_group_name = var.resource_group_name
  subnet_id           = var.subnet_id

  private_service_connection {
    name                           = "${var.account_name}-psc"
    private_connection_resource_id = azurerm_cognitive_account.aoai_private.id
    subresource_names              = ["account"]
    is_manual_connection           = false
  }
}

Remediation — Bicep

targetScope = 'resourceGroup'

@description('Azure OpenAI account resource ID to wrap with a private endpoint.')
param aoaiResourceId string
@description('Subnet resource ID hosting the private endpoint NIC.')
param subnetId string

param location string = resourceGroup().location

resource pe 'Microsoft.Network/privateEndpoints@2024-03-01' = {
  name: 'pe-aoai'
  location: location
  properties: {
    subnet: { id: subnetId }
    privateLinkServiceConnections: [
      {
        name: 'aoai-link'
        properties: {
          privateLinkServiceId: aoaiResourceId
          groupIds: ['account']
        }
      }
    ]
  }
}

Compliance mapping

AzureActivity
          | where ResourceId has "Microsoft.CognitiveServices/accounts"
          | where OperationNameValue == "Microsoft.CognitiveServices/accounts/write"
          | extend body = tostring(parse_json(Properties).requestbody)
          | where body has "\"publicNetworkAccess\":\"Enabled\""
          | project TimeGenerated, Caller, ResourceId, body
          | order by TimeGenerated desc
          | take 100

Equivalent on: AWS — VPC endpoint · GCP — VPC-SC · OCI — private endpoint

Assign the Cognitive Services OpenAI User role per-resource (not at subscription or resource-group scope) for application service identities. Use Cognitive Services OpenAI Contributor only for deployment management operations. Never assign the generic Contributor role at the resource scope for data-plane access — it grants management-plane rights that far exceed what inference workloads require. See azure-iam-03 — Privileged Identity Management for the general RBAC least-privilege pattern.

Audit — Azure CLI

# Azure CLI 2.x
# Find overly broad Cognitive Services OpenAI User assignments at subscription scope
az role assignment list \
  --scope "/subscriptions/${SUBSCRIPTION_ID}" \
  --query "[?roleDefinitionName=='Cognitive Services OpenAI User']" \
  --output table

# Check correct per-resource assignments
az role assignment list \
  --scope "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RG}/providers/Microsoft.CognitiveServices/accounts/${AOAI_ACCOUNT}" \
  --query "[?roleDefinitionName=='Cognitive Services OpenAI User']" \
  --output table

Remediation — Terraform

# Terraform AzureRM provider ~> 3.0
# Assign Cognitive Services OpenAI User at the specific resource scope only
resource "azurerm_role_assignment" "aoai_user" {
  scope                = azurerm_cognitive_account.aoai.id
  role_definition_name = "Cognitive Services OpenAI User"
  principal_id         = var.app_service_principal_id
}

Remediation — Bicep

targetScope = 'resourceGroup'

@description('Azure OpenAI account resource ID.')
param aoaiResourceId string
@description('Application principal that should call the model.')
param appPrincipalId string

// Cognitive Services OpenAI User (data-plane, no key management)
var openAiUserRoleId = '5e0bd9bd-7b93-4f28-af87-19fc36ad61bd'

resource assign 'Microsoft.Authorization/roleAssignments@2024-04-01' = {
  name: guid(aoaiResourceId, appPrincipalId, openAiUserRoleId)
  scope: resourceGroup()
  properties: {
    principalId: appPrincipalId
    principalType: 'ServicePrincipal'
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', openAiUserRoleId)
  }
}

Compliance mapping

AzureActivity
          | where OperationNameValue == "Microsoft.Authorization/roleAssignments/write"
          | where ResourceId has "Microsoft.CognitiveServices/accounts"
          | extend body = tostring(parse_json(Properties).requestbody)
          | where body has "Cognitive Services Contributor" or body has "Owner" or body has "OpenAI Contributor"
          | project TimeGenerated, Caller, ResourceId, body
          | order by TimeGenerated desc
          | take 200

Equivalent on: AWS — IAM least privilege · GCP — SA scoping · OCI — IAM least privilege

Configure Diagnostic Settings to forward Azure OpenAI resource logs to a Log Analytics workspace. Enable the Audit and RequestResponse log categories. Resource logs are not enabled by default — explicit configuration is required. Apply PII redaction before log storage; do not store raw unredacted prompts (anti-feature #1 in General GenAI — Common Misconfigurations). See Azure Logging for the general diagnostic settings pattern.

Remediation — Azure CLI

# Azure CLI 2.x — enable diagnostic logs to Log Analytics
az monitor diagnostic-settings create \
  --name "aoai-diag" \
  --resource "${AOAI_RESOURCE_ID}" \
  --workspace "${LOG_ANALYTICS_WORKSPACE_ID}" \
  --logs '[{"category":"Audit","enabled":true},{"category":"RequestResponse","enabled":true}]'

Remediation — Terraform

# Terraform AzureRM provider ~> 3.0
resource "azurerm_monitor_diagnostic_setting" "aoai_diag" {
  name                       = "aoai-diagnostics"
  target_resource_id         = azurerm_cognitive_account.aoai.id
  log_analytics_workspace_id = var.log_analytics_workspace_id

  enabled_log {
    category = "Audit"
  }

  enabled_log {
    category = "RequestResponse"
  }
}

Remediation — Bicep

targetScope = 'resourceGroup'

@description('Azure OpenAI account resource ID.')
param aoaiResourceId string
@description('Log Analytics workspace resource ID.')
param workspaceId string

resource diag 'Microsoft.Insights/diagnosticSettings@2024-01-01-preview' = {
  name: 'aoai-prompt-audit'
  scope: az.resourceId('Microsoft.CognitiveServices/accounts', last(split(aoaiResourceId, '/')))
  properties: {
    workspaceId: workspaceId
    logs: [
      { categoryGroup: 'audit',     enabled: true }
      { categoryGroup: 'allLogs',   enabled: true }
    ]
    metrics: [
      { category: 'AllMetrics', enabled: true }
    ]
  }
}

Compliance mapping

AzureActivity
          | where ResourceId has "Microsoft.CognitiveServices/accounts"
          | where OperationNameValue startswith "Microsoft.Insights/diagnosticSettings/"
          | extend body = tostring(parse_json(Properties).requestbody)
          | project TimeGenerated, Caller, ResourceId, OperationNameValue, body
          | order by TimeGenerated desc
          | take 200

Equivalent on: AWS — invocation logging · GCP — audit logs · OCI — audit logging

Keep Azure OpenAI abuse monitoring enabled. Default-enabled is the secure baseline; the secure action is to not disable it. Microsoft's abuse monitoring applies automated classifiers and human review to flagged prompts and completions; disabling human review degrades detection accuracy for novel attack patterns and coordinated abuse campaigns. Applying for the Limited Access exemption to disable human review without formal risk acceptance and documented compensating controls is anti-feature #5 from General GenAI — Common Misconfigurations.

Audit — Azure CLI

# Azure CLI 2.x
# Confirm abuse monitoring status — default is enabled
# There is no CLI command to "enable" abuse monitoring; it is the default state.
# This command confirms it has not been modified to a non-default configuration.
az cognitiveservices account show \
  --name "${AOAI_ACCOUNT}" \
  --resource-group "${RG}" \
  --query "properties.abuseMonitoring"

Note: If properties.abuseMonitoring returns null or is absent from the output, abuse monitoring is at its default-enabled state. A non-null value indicating human review is disabled is the misconfiguration to remediate.

Remediation — Bicep

targetScope = 'resourceGroup'

@description('Azure OpenAI account name. Abuse monitoring is ON by default; do NOT request the modified-abuse-monitoring exception unless legally required.')
param accountName string

resource aoai 'Microsoft.CognitiveServices/accounts@2024-10-01' existing = {
  name: accountName
}

// No template knob disables abuse monitoring — its absence here is the control.
// Gate policy: alert if Microsoft.CognitiveServices/accounts/properties.userOwnedStorage is set
// (a signal that a modified-abuse-monitoring exception was filed without security review).
output abuseMonitoringEnforced bool = true

Compliance mapping

AzureActivity
          | where ResourceId has "Microsoft.CognitiveServices/accounts"
          | where OperationNameValue == "Microsoft.CognitiveServices/accounts/write"
          | extend body = tostring(parse_json(Properties).requestbody)
          | where body has "abuseMonitoring" or body has "modifiedAbuseMonitoring"
          | project TimeGenerated, Caller, ResourceId, body
          | order by TimeGenerated desc
          | take 200

Enable Customer-Managed Key (CMK) encryption via Azure Key Vault for the Azure OpenAI resource. CMK gives control over the encryption lifecycle — rotation, revocation, and independent audit of key usage — that Microsoft-managed keys do not provide. CMK is not the default; it requires explicit Key Vault reference configuration. Note regional and SKU availability constraints: verify availability for your deployment region and SKU tier at deployment time, as CMK support is not uniform across all Azure regions.

Remediation — Azure CLI

# Azure CLI 2.x
# Step 1: Ensure the resource has a system-assigned identity (required for Key Vault access)
az cognitiveservices account identity assign \
  --name "${AOAI_ACCOUNT}" \
  --resource-group "${RG}"

# Step 2: Grant Key Vault access to the managed identity
# (Key Vault must have soft-delete and purge protection enabled)
az keyvault set-policy \
  --name "${KEY_VAULT_NAME}" \
  --object-id "${AOAI_IDENTITY_PRINCIPAL_ID}" \
  --key-permissions get wrapKey unwrapKey

Remediation — Terraform

# Terraform AzureRM provider ~> 3.0
resource "azurerm_cognitive_account" "aoai_cmk" {
  name                = var.account_name
  location            = var.location
  resource_group_name = var.resource_group_name
  kind                = "OpenAI"
  sku_name            = "S0"

  identity {
    type = "SystemAssigned"
  }

  customer_managed_key {
    key_vault_key_id = var.key_vault_key_id
  }
}

Remediation — Bicep

targetScope = 'resourceGroup'

@description('Azure OpenAI account name.')
param accountName string
@description('Versionless CMK URI (with key vault).')
param keyVaultUri string
@description('Key name in vault.')
param keyName string
@description('User-assigned managed identity granted Key Vault Crypto Service Encryption User.')
param identityResourceId string

param location string = resourceGroup().location

resource aoai 'Microsoft.CognitiveServices/accounts@2024-10-01' = {
  name: accountName
  location: location
  kind: 'OpenAI'
  sku: { name: 'S0' }
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: { '${identityResourceId}': {} }
  }
  properties: {
    customSubDomainName: accountName
    disableLocalAuth: true
    publicNetworkAccess: 'Disabled'
    encryption: {
      keySource: 'Microsoft.KeyVault'
      keyVaultProperties: {
        keyVaultUri: keyVaultUri
        keyName: keyName
        identityClientId: reference(identityResourceId, '2023-01-31').clientId
      }
    }
  }
}

Compliance mapping

AzureActivity
          | where OperationNameValue in ("Microsoft.CognitiveServices/accounts/write", "Microsoft.KeyVault/vaults/keys/delete")
          | extend body = tostring(parse_json(Properties).requestbody)
          | where body has "\"keySource\":\"Microsoft.CognitiveServices\"" or OperationNameValue endswith "keys/delete"
          | project TimeGenerated, Caller, ResourceId, OperationNameValue, body
          | order by TimeGenerated desc
          | take 100

Equivalent on: GCP — CMEK · OCI — Vault encryption

Configure per-deployment TPM (tokens-per-minute) and RPM (requests-per-minute) quota limits in Azure OpenAI to bound resource consumption and mitigate model-DoS (LLM10:2025). For multi-tenant or high-volume applications, enforce token consumption limits at the Azure API Management layer using the azure-openai-token-limit policy (also known as llm-token-limit). Quota limits prevent a single deployment or caller from exhausting capacity that is shared across all users and applications on the Azure OpenAI resource.

Audit — Azure CLI

# Azure CLI 2.x
# Check current quota and rate limit settings for a deployment
az cognitiveservices account deployment show \
  --name "${AOAI_ACCOUNT}" \
  --resource-group "${RG}" \
  --deployment-name "${DEPLOYMENT}" \
  --query "{tpm:properties.rateLimits[?key=='token'].count | [0], rpm:properties.rateLimits[?key=='request'].count | [0]}"

# Update deployment quota (requires REST API for full TPM/RPM configuration)
# GET /openai/deployments/{deploymentName}?api-version=2024-10-01
# PATCH with {"sku": {"capacity": N}} where N = desired TPM / 1000

Azure APIM Token-Limit Policy

# Azure API Management — azure-openai-token-limit policy
# Applies token consumption limit per calling consumer
<azure-openai-token-limit
  tokens-per-minute="10000"
  tokens-consumed-variable-name="tokensConsumed"
  estimate-prompt-tokens="false"
  failed-condition="limit-exceeded">
  <set-header name="x-ms-ai-model" exists-action="override">
    <value>@(context.Request.Headers.GetValueOrDefault("x-ms-ai-model", "unknown"))</value>
  </set-header>
</azure-openai-token-limit>

Remediation — Bicep

targetScope = 'resourceGroup'

@description('Azure OpenAI account hosting the deployment.')
param accountName string
@description('Deployment name.')
param deploymentName string
@description('Tokens-per-minute capacity (1000s; SKU capacity).')
param tpmCapacity int = 10

resource aoai 'Microsoft.CognitiveServices/accounts@2024-10-01' existing = {
  name: accountName
}

resource deployment 'Microsoft.CognitiveServices/accounts/deployments@2024-10-01' = {
  parent: aoai
  name: deploymentName
  sku: { name: 'Standard', capacity: tpmCapacity }
  properties: {
    model: { format: 'OpenAI', name: 'gpt-4o', version: '2024-11-20' }
    raiPolicyName: 'Microsoft.DefaultV2'
  }
}

Compliance mapping

AzureDiagnostics
          | where ResourceProvider == "MICROSOFT.COGNITIVESERVICES" and Category == "RequestResponse"
          | extend appId = tostring(identity_claim_appid_g)
          | summarize tokens=sum(toint(properties_s.responseMetadata.totalTokens)) by bin(TimeGenerated, 5m), appId, Resource
          | join kind=leftouter (
              AzureDiagnostics
              | where Category == "RequestResponse"
              | extend appId2 = tostring(identity_claim_appid_g)
              | summarize p99=percentile(toint(properties_s.responseMetadata.totalTokens), 99) by appId2
          ) on $left.appId == $right.appId2
          | where tokens > p99 * 3
          | order by tokens desc
          | take 200