Tracking AI Agent Tool Usage in APIM

You can't manage what you can't see.

When you expose a dozen backend tools to an army of AI agents, you need to know exactly who is calling what. If a rogue agent gets stuck in a loop and hammers your incident management API, you need to spot it instantly.

Azure API Management gives you the telemetry. You just have to extract it from the traffic stream.

Extracting Caller Identity

Before you can emit a metric, you need to know who the caller is.

You start by validating the Entra JWT. Crack the token open at the gateway edge and extract the caller's context—specifically the Object ID (OID) or Application ID. Store these as context variables.

The Inbound Policy Pipeline

Order of operations is everything here.

You cannot emit a metric for an unauthorized call. You cannot rate-limit a user whose token you haven't validated. Your inbound policy pipeline must follow a strict, logical sequence. Validate. Throttle. Record.

<include-fragment fragment-id="validate-entra-token" />
<include-fragment fragment-id="rate-limit-per-subscription" />
<include-fragment fragment-id="quota-per-subscription" />
<include-fragment fragment-id="emit-tool-call-metric" />
 

Dashboarding in Azure Monitor

Once you configure the emit-tool-call-metric fragment, the data flows directly into Azure Monitor.

You now have a custom metric dimensioned by caller ID and tool name. You can build dashboards to track cross-tenant usage, set up alerts for sudden spikes in specific function calls, and definitively prove the ROI of your internal MCP servers.

See the code here https://github.com/jackweldonweb/apim-mcp-terraform, check license for usage information.