Part 3 of 13: Governing Azure AI Foundry at Enterprise Scale

Published by: Azure User Group Nepal
Series: Enterprise AI Governance, Security & Infrastructure with Azure AI Foundry

Introduction

You've learned the architecture (Part 1) and security controls (Part 2). Now comes the governance question: Who decides what?

In a traditional IT environment, governance is often centralized: the IT team makes all decisions. But Azure AI Foundry governance is distributed. The Hub Admin makes some decisions, the Project Owner makes others, the Data Team makes others. Without clear governance, you get chaos: conflicting decisions, compliance violations, security breaches.

Your retail company has multiple business units (HR, IT, Operations, Finance). Each wants to build AI applications. Each has different compliance requirements. Each has different security requirements. Your governance framework must enable each business unit to move fast while ensuring compliance and security.

This post explains how to build a governance framework for Azure AI Foundry.

What you'll learn in this post:

• The three pillars of AI governance
• How to define governance policies
• How to implement governance controls
• How to measure governance effectiveness
• How to scale governance across the enterprise

Prerequisites: Parts 1-2 (Architecture and Security)

Complexity Level: Medium

The Three Pillars of AI Governance

Governance in Azure AI Foundry rests on three pillars:

Pillar 1: Decision Rights

Decision rights define who can make what decisions.

Key decisions in Azure AI Foundry:

In your retail scenario:

• Enterprise Architecture decides to create a Hub for the retail company
• Hub Admin (IT team) approves Project creation requests
• Project Owner (HR lead) adds team members to the chatbot Project
• Data Team creates Connections to Azure OpenAI and HR systems
• Project Owner requests production deployment
• Security Review approves production deployment
• Security Team rotates encryption keys quarterly
• Compliance Team ensures audit retention meets GDPR requirements

Why this matters:

• Ensures decisions are made by people with appropriate authority
• Prevents unauthorized decisions
• Enables accountability
• Enables compliance

Pillar 2: Policies

Policies define the rules that govern Azure AI Foundry.

Key policies in Azure AI Foundry:

In your retail scenario:

• Data Residency Policy: EU employee data must be stored in EU region
• Encryption Policy: All employee data must be encrypted with customer-managed keys
• Network Policy: All connections to external systems must use Private Endpoints
• RBAC Policy: Only Hub Admin can create Projects
• Audit Policy: All activity must be logged for 90 days
• Compliance Policy: All Projects must pass security review before production deployment

Why this matters:

• Ensures consistent governance across all Projects
• Enables compliance with regulations
• Reduces security risk
• Simplifies decision-making

Pillar 3: Controls

Controls are the technical mechanisms that enforce policies.

Key controls in Azure AI Foundry:

In your retail scenario:

• Network Control: Private Endpoints enforce network isolation
• Identity Control: RBAC enforces access control (only Hub Admin can create Projects)
• Encryption Control: Key Vault enforces encryption (all data encrypted with CMK)
• Audit Control: Log Analytics enforces audit logging (all activity logged)
• Compliance Control: Azure Policy enforces compliance (all Projects must be in EU region)
• Cost Control: Cost Management enforces budget limits (Project budget cannot exceed $10,000/month)

Why this matters:

• Ensures policies are actually enforced
• Prevents policy violations
• Reduces manual enforcement effort
• Enables automated compliance

Terraform Implementation Approach

To implement governance controls, you'll use Terraform to create:

# Governance Control 1: Azure Policy for Data Residency
resource "azurerm_policy_definition" "data_residency" {
  name         = "enforce-data-residency"
  display_name = "Enforce Data Residency"
  
  policy_rule = jsonencode({
    if = {
      field  = "location"
      notIn  = ["eastus", "westus"]
    }
    then = {
      effect = "deny"
    }
  })
}

# Governance Control 2: RBAC for Hub Admin
resource "azurerm_role_assignment" "hub_admin" {
  scope              = azurerm_machine_learning_workspace.hub.id
  role_definition_name = "Owner"
  principal_id       = azurerm_user_assigned_identity.hub_admin.principal_id
}

# Governance Control 3: Key Vault for Encryption
resource "azurerm_key_vault_key" "hub_key" {
  name            = "hub-encryption-key"
  key_vault_id    = azurerm_key_vault.hub_kv.id
  key_type        = "RSA"
  key_size        = 4096
  
  key_opts = [
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey",
  ]
}

# Governance Control 4: Log Analytics for Audit
resource "azurerm_monitor_diagnostic_setting" "hub_audit" {
  name               = "hub-audit-logs"
  target_resource_id = azurerm_machine_learning_workspace.hub.id
  
  log_analytics_workspace_id = azurerm_log_analytics_workspace.hub_logs.id
  
  log {
    category = "AmlComputeClusterEvent"
    enabled  = true
  }
}

What this does:

• Creates an Azure Policy to enforce data residency
• Creates RBAC role assignment for Hub Admin
• Creates encryption key in Key Vault
• Creates diagnostic setting to log audit events

For complete Terraform code with all parameters, see:

• Terraform Azure Provider - Policy Definition
• Terraform Azure Provider - Role Assignment
• Terraform Azure Provider - Key Vault Key
• Terraform Azure Provider - Monitor Diagnostic Setting

Compliance & Governance Implications

This governance framework enables compliance:

GDPR Compliance

• Decision Rights: Compliance Team decides data residency policy
• Policies: Data Residency Policy enforces EU data stays in EU
• Controls: Azure Policy enforces data residency at deployment time

PCI-DSS Compliance

• Decision Rights: Security Team decides encryption policy
• Policies: Encryption Policy enforces CMK encryption
• Controls: Key Vault enforces encryption key management

SOC 2 Type II Compliance

• Decision Rights: Compliance Team decides audit policy
• Policies: Audit Policy enforces activity logging
• Controls: Log Analytics enforces audit log retention

Operational Considerations

Governance Maturity Levels

Governance maturity progresses through levels:

Your retail company should target Level 3-4 for production workloads.

Governance Metrics

Track governance effectiveness:

• Policy Compliance Rate: % of resources compliant with policies
• Decision Cycle Time: Time from request to decision
• Audit Coverage: % of activity logged and auditable
• Compliance Violations: Number of policy violations detected

Conclusion & Next Steps

You now understand the three pillars of governance:

• Decision Rights: Who can make what decisions
• Policies: What are the rules
• Controls: How are policies enforced

This governance framework enables compliance, reduces risk, and simplifies decision-making.

In Part 4, we'll dive deeper into operating model: how to organize teams and define roles.

Next steps:

1. Define your decision rights matrix (who decides what)
2. Define your governance policies (data residency, encryption, audit)
3. Implement governance controls (Azure Policy, RBAC, Key Vault)
4. Measure governance effectiveness (compliance rate, decision cycle time)
5. Read Part 4 to understand operating model

Relevant Azure documentation:

• Azure Governance
• Azure Policy
• Azure RBAC
• Azure Audit Logs

Connect & Questions

Want to discuss Azure AI Foundry governance, share feedback, or ask questions?

Reach out on X (Twitter) @sakaldeep

Or connect with me on LinkedIn: https://www.linkedin.com/in/sakaldeep/

I look forward to connecting with fellow cloud professionals and learners.

Published by: Azure User Group Nepal
Series: Enterprise AI Governance, Security & Infrastructure with Azure AI Foundry
Part: 3 of 13