Blog

AI Governance

AI Security

AI agent governance: control access, actions, and autonomy

AI agent governance: control access, actions, and autonomy

Govern AI agents with clear deployment classes, action limits, oversight, audit evidence, and review triggers before expanding access.

Hands Off, Eyes On.
Hands Off, Eyes On.

A customer-service agent drafts a refund. Later, the team lets it issue refunds below $100 without waiting for approval. The model may be unchanged, but the agent now has authority to move money. Its governance needs to change with that authority.

AI agent governance defines what an agent may access and do, who is accountable, and what evidence shows that it stayed within those limits. The review has to cover the application, tools, identities and workflow around the model.

Start with the operating decision: which actions can proceed automatically, which require approval, and which must be denied? Then make those boundaries enforceable in the systems that perform the work.

For security and GRC teams, that means classifying the deployment, testing the limits, and keeping the approval current as tools and permissions change. The following is a practical governance approach, not a claim that one autonomy scale fits every organization.

What makes AI agent governance different?

An agent can turn an output into a change in another system. A mistaken summary may mislead a reviewer; a mistaken tool call may issue a refund, alter a record or send information outside the organization. Both matter, but they require different evidence and controls.

The May 2026 joint guidance on careful adoption of agentic AI recommends beginning with defined, lower-risk tasks and expanding scope through tested controls and ongoing assessment. An initial successful demonstration is not evidence that all later workflows are safe.

Accountability also needs to survive delegation. When an agent acts for an employee or calls another agent, the record should preserve who initiated the work, which identity authorized each action, and who owns the service. A shared credential without that context makes review harder.

Classify the deployment before assigning autonomy

Distinguish an employee assistant, an internal workflow and a customer-facing production service. Then assess data sensitivity, action permissions and who could be affected. Stanford’s AI Security Framework similarly separates deployment category from risk tier. It is institutional guidance rather than a policy or mandate.

Autonomy is a second dimension. A private assistant with broad document access may create more disclosure risk than a public bot limited to published help pages. Use the following rubric to describe how actions are authorized, while applying identity, logging and revocation controls wherever sensitive access exists:

Mode

What the agent does

Additional review focus

  1. Assist

Reads permitted data and suggests an answer; a person acts.

Data scope, output quality and permitted destinations.

  1. Prepare

Drafts a specific action for review.

Approval shows the target, data and consequence.

  1. Act within limits

Executes defined actions inside enforced boundaries.

System-enforced thresholds, aggregate limits and alerts.

  1. Act across workflows

Coordinates multiple actions without step-by-step approval.

Delegation, combined permissions, interruption and recovery.

Revisit the classification when data, audience, tools or authority changes. A new integration may create a disclosure path without changing the model or the agent’s stated purpose.

Keep the deployment owner, approved workflow and permission boundary in the same record. Review that record through the access and change-management processes your organization already operates.

What does this look like in practice?

For the refund agent, enforce the $100 boundary at the payment interface. Decide whether it applies per refund, per customer or across total activity. Otherwise, several individually permitted refunds could exceed the intended exposure. Check the approved recipient and amount again when execution occurs.

A prompt asking the agent to stay under budget is not the control. The system that moves the money should reject an unauthorized request, even if the agent believes it is helping the customer.

Five controls to establish before granting access

The implementation will differ by deployment, but these five questions should have concrete answers before an agent reaches sensitive data or operating systems.

1. A named human owner

Name the person accountable for the agent’s permitted use and the team that operates it. Specify who can approve a scope change, respond to an incident and accept remaining risk. A team name alone should not leave those decisions unassigned.

2. Attributable identity and limited authority

Internal services generally need a dedicated workload identity. An individual assistant may act through the user’s identity, with explicit delegated scope. In either case, record the initiating user where applicable, the agent and run, and the authorization used. Make credentials revocable and prefer short-lived access where supported.

3. Approval gates for high-impact actions

Choose approval points by impact and reversibility. The reviewer should see the actual target, data and consequence. Bind the approval to that operation so a changed recipient, amount or permission requires a new decision. Do not let the agent decide for itself when approval can be skipped.

4. Evidence of what happened

Keep tool requests, permission decisions, approvals and observed outcomes, with source references where needed. A model explanation can help a reviewer, but research on reasoning faithfulness shows why it should not be treated as an independent account of the decision. Protect the logs and avoid retaining sensitive prompt content without a clear need.

5. A tested stop and recovery procedure

Test how to pause execution and revoke access while work is active. Include queued actions and delegated agents. Establish which completed actions can be reversed, which need a compensating action, and who performs recovery. The required response time should match the potential consequence.

How does AI agent governance fit frameworks you already use?

Existing governance disciplines provide a home for these controls. Use them to organize evidence, while checking that their scope includes the actual agent deployment:

Framework or discipline

What it contributes

Evidence to retain

NIST AI Risk Management Framework

A voluntary structure for governing, mapping, measuring and managing AI risk.

Use-case record, ownership, tests and risk-treatment decisions.

ISO/IEC 42001

Requirements for an AI management system.

Applicable policies, assessments, reviews and corrective actions.

Identity and access management

Authority to reach data and systems.

Identity, delegation, permission tests and revocation evidence.

Change management

Review of changes to an approved deployment.

Baseline, changed scope, test results and approval.

ISO describes certification as assurance about an AI management system. It does not replace applicable laws or prove that every action of a particular agent is correct. Retain deployment-specific tests alongside management-system evidence.

What should you check before an AI agent goes live?

Before launch, ask the owner to demonstrate the following in the intended environment:

  • The inventory identifies the deployment, owner, data, users and permitted actions.

  • The operating mode and limits are approved for this workflow.

  • Identity and permission tests reject access outside the authorized scope.

  • High-impact approvals remain bound to the action actually executed.

  • Logs link the initiating authority, request, decision and observed outcome.

  • Tests cover legitimate work, hostile input, denied actions, failures and retries.

  • Interruption, credential revocation and recovery have been exercised.

For agents inside a supplier product, request the same evidence through your AI vendor assessment process. Where the agent connects through MCP, the MCP security review examines the host, server and combined permissions in more detail.

Measure task success and control failures separately

Anthropic’s guidance on agent evaluations distinguishes the transcript from the final state of the environment. Apply that distinction to the refund example: the agent saying “refunded” is not the same as the right payment being issued once. Check the source system and keep the trace that explains the sequence of actions.

Track unauthorized attempts, duplicate actions, approval overrides, incomplete work and time to revoke access alongside successful tasks. Review meaningful failures with the owner. Repeat the affected tests after changes to the model, tools, permissions or workflow; a dated test result should not become a permanent safety claim.

Expand authority only when the evidence supports it

A successful pilot can justify a specific next step: another data source, a higher limit or a new action. Document that step and its acceptance conditions. If failures expose a weak boundary, reduce scope while the team fixes and retests it.

Start with the agents that can move money, disclose confidential information or change access. For each one, establish a clear answer to four questions: what may it do, what stops it from doing more, who owns the decision, and what evidence can another person verify?

Agent controls belong within the broader AI program. Our enterprise AI governance guide covers shared ownership, intake, risk decisions and monitoring across agents, internal models and purchased AI.

Keep that record with the system’s risk assessment and review it whenever the deployment changes.

Related articles

See autonomous GRC in action.

Get a walkthrough of how Zania reviews vendors, evidence, and AI systems for your team.