AI governance: a practical enterprise guide

AI governance becomes an enterprise problem when decisions made in one team create consequences for another. A forecasting model changes what operations buys. A customer-facing assistant gives an answer the business must stand behind. An employee connects a writing tool to a shared drive. Each may be a reasonable use of AI, but someone needs to decide what it is allowed to do, how its performance will be judged, and when to intervene.

What is enterprise AI governance?

Enterprise AI governance is the system of policies, responsibilities and operating processes an organization uses to direct the development, acquisition and use of AI throughout its lifecycle. It covers internally built models, AI in purchased software, employee use of generative AI, AI features delivered to customers, and agents that take actions.

The program has to connect business objectives with the consequences of using AI. Security and privacy matter, alongside reliability, effects on people, legal obligations, cost and the ability to recover when a system fails. A secure system can still make unsuitable recommendations. A technically accurate model can be used for a decision it was never designed to support.

To establish an AI governance program, define its scope and decision rights, inventory AI uses, assess their risks, and test the controls required for deployment. To operate it, monitor results, review material changes, respond to incidents and retire unsuitable systems. The sections below explain the owners and evidence needed at each stage.

Define the program scope and AI policy

The executive sponsor should establish what the program covers, what it is meant to achieve, and who has authority to make decisions. Write this in a short charter before choosing an assessment questionnaire. Include business units, products, internal systems and employee use. If some environments are outside the initial rollout, record those gaps and who will bring them into scope.

Set objectives that allow leadership to judge the program. These might include enabling approved employee tools, making consequential AI uses traceable to accountable owners, reducing time spent waiting for reviews, and identifying systems that fail to deliver their expected benefit. The program needs resources for those activities, including testing and monitoring after launch.

Define the organization’s boundaries through concrete decisions. Where may AI advise a person? Where may it act without prior approval? Which uses require specialist review, and which will the business not permit? A general commitment to responsible AI leaves these questions unanswered.

The AI policy should tell people which tools and data they may use, when to register a use, how to request an exception, and where to report a problem. Give employees an approved way to complete common tasks. Test training with realistic work: can an employee spot confidential material before uploading it, can an engineer investigate a failed access test, and can an operator stop a faulty workflow? Course completion alone does not answer those questions.

Who owns AI governance?

A central team can maintain the policy, common assessment method, inventory and reporting. Business units need to own the uses they introduce and operate. This division lets the enterprise set consistent expectations without asking a central committee to understand every operational decision.

The program owner may sit in risk, technology, data, legal or another executive function. Its placement matters less than its mandate and access to decision-makers. The CISO owns security expertise and relevant controls; that role does not automatically make the CISO accountable for a forecasting model’s usefulness or a product’s effect on customers.

Document escalation and suspension authority alongside approval authority. If a model begins producing unsuitable recommendations, the operator should know who can pause its use and how the business will continue. A council can settle a disputed decision; it should not be the only route for responding to an active failure.

Build an enterprise AI inventory

An enterprise AI inventory needs to represent what the organization builds and does. Start with existing model registries, application and product inventories, development platforms, procurement records, identity systems and team interviews. Each source provides part of the picture. A purchased license establishes access to a tool; it does not establish how employees use it.

Keep the system and its uses connected. One model may serve several products, and one use may depend on several models, datasets and external services. A shared model or provider record avoids duplicating technical facts. A use record describes the business purpose, users, affected people, deployment, data, outputs and permitted actions.

At minimum, an active use should have a business owner, technical owner, lifecycle stage, risk assessment status and current operating decision. Link to the configuration and evidence where they live. Record unknown ownership or dependencies as open work instead of filling the register with guesses.

Include employee tools and AI features enabled inside existing applications. A new feature may never pass through a new purchase. Check application settings and interview the teams using it. Our shadow AI discovery guide examines discovery sources and their limits. For internally developed systems, connect registration to development and release processes so the inventory stays current as work progresses.

Review employee tools, embedded AI and internal systems

Different entry points need different checks. In Scaling AI: The AI Governance and Security Playbook for Executives, Sol Rashidi and Steve Klementowski distinguish public tools, AI embedded in software, and internally assembled systems. Apply that distinction when assigning review work:

• Employee tools: define approved accounts, allowed uploads and connections, output review, and a route for reporting accidental disclosure.

• AI inside existing software: inspect feature activation, inherited permissions, data processing and retention. Reassess when the feature or its configuration changes.

• Internally built systems: assign ownership across datasets, models, retrieval, APIs and tools. Test the integrated application, including any external services it depends on.

Assess AI risk and set review requirements

Assess the proposed use in its actual setting. Who could be affected? What happens if the output is wrong, unavailable or misused? What data does it depend on? How much authority does the system have, and can a person detect and reverse a mistake? For decisions affecting people, examine fairness, explainability and how someone can challenge an outcome. Include the expected benefit and the alternatives, including a simpler system or an existing manual process.

Risk categories should lead to different decisions and evidence requirements. Drafting public copy with an approved tool may fit an established pattern. An internal forecast that informs purchasing needs validation against the operational consequences of error. A system used in employment decisions needs examination of its effects on people and applicable obligations. The same model can therefore require different oversight in different uses.

Use standard patterns for repeatable, bounded activities. Specify the allowed data, configurations and human review, then let teams proceed within those conditions. Route material departures to the relevant reviewers. An internal risk tier helps organize this process; it does not settle legal classification or remove a specialist’s blocking concern.

Set review routes and approval authority

Keep risk severity separate from the deployment setting. An internal assistant with access to payroll records can carry more risk than a public tool limited to approved product information. Assess data sensitivity, permitted actions, autonomy and exposure together; an “internal” label is not a reason to skip review.

For an enterprise program, make each route usable. A standard route needs published eligibility conditions, required evidence and an accountable owner. A specialist route needs a named reviewer and a way to resolve findings. A refusal needs a reason and a clear account of what would have to change. Teams should be able to establish which route applies without negotiating a new process for every use.

Record review findings and operating authorization separately. Security review may be complete while data-use permission or a product release decision remains outstanding. Name the person who can authorize operation once the required conditions are met, and identify findings that their delegated authority does not allow them to waive.

Map AI security risks to controls and evidence

AI security risk mapping connects a system’s data and capabilities to a specific failure, its consequences, and the controls needed to reduce the risk. NIST’s AI RMF Map function starts with the system’s context and potential impacts. A useful enterprise record makes that context concrete enough for security, engineering and the business owner to examine together.

Trace data, permissions and actions

Draw the data path before scoring the risk: source systems, training or retrieval data, model, tools, outputs and logs. Mark where information is copied, transformed or sent across a department, customer or external-service boundary. Include retrieval indexes and caches; a source document’s restrictions need to survive those copies. Record four things:

• Data: what the system can read, retain or disclose, including retrieved documents, prompts, logs and secrets.

• Authority: whose identity it uses, which records it can change, and which tools or other agents it can invoke.

• Exposure: who can supply inputs or receive outputs, including through documents and connected services.

• Autonomy and recovery: which actions need approval, how far an error can spread, and whether the action can be stopped or reversed.

For a retrieval-augmented assistant, test whether a user can retrieve a document after their access is removed, including through an older index or cached answer. Check who may add or replace source material and how those changes are validated. OWASP’s guidance on vector and embedding weaknesses calls for permission-aware stores, source validation and retrieval logging. Turn those controls into tests against your own data path.

Connect failure scenarios to controls and evidence

Then write the failure scenario. “Prompt injection” names an attack technique. “Instructions hidden in a supplier document cause the assistant to send confidential contract terms outside the company” identifies an entry point, an action and a consequence. The following examples show how to connect that reasoning to controls and evidence; they are illustrative, not a complete control set.

For a worked approach to defining scenarios and choosing controls, see AI risk assessment beyond the heat map.

AIUC-1 makes this mapping more specific. Its C001 requirement calls for a risk taxonomy grounded in system capabilities and deployment context. For the purchase-order scenario above, relevant requirements include B006 on unauthorized agent actions and D003 on unsafe tool calls. Link the scenario to the implemented access restrictions, tool authorization and approval workflow, then attach the configuration and test results. These are candidate mappings to assess; adding requirement IDs to a risk register does not demonstrate that the controls work.

Assess residual risk and record the decision

Assess likelihood and impact using the organization’s risk method, recording assumptions and uncertainty. Keep the exposure before treatment separate from the residual risk supported by tested controls. A planned control has no operating evidence yet; an unknown permission or data path remains an open finding. Avoid reducing a serious scenario to a reassuring average score.

Give each material scenario a risk owner, control owner, evidence link, review date and decision: proceed within limits, remediate, restrict or stop. The authorized risk owner decides whether the remaining risk is acceptable within policy. Reopen the assessment when data access, tools, audience or autonomy changes. Framework mappings can then show which requirements those controls support.

Keep this security assessment alongside the wider governance assessment. Access controls cannot establish whether a forecast is useful, an employment decision is fair, or a use of personal data is appropriate. Those questions need their own evidence and accountable reviewers.

Govern AI development and deployment

For an internally built system, settle data use before training or integration. Record who owns each source, why it is suitable, whether the proposed use is permitted, and how long copies may be kept. Distinguish data used to train or fine-tune a model from data supplied at inference time. They have different update and removal paths. Use curated or synthetic data for early experiments where suitable, then evaluate the intended production data separately.

Example: govern a demand-forecasting model

Consider an illustrative demand-forecasting system built with machine learning. The operations owner wants fewer stockouts without excessive inventory. Before development, the team agrees which purchasing decisions the forecast will inform, what baseline it must improve on, and how planners will handle unavailable or unreliable predictions. Data owners establish whether the inputs are suitable and permitted for this purpose.

Evaluation then needs to reflect the task. Test on periods held out from model development, account for information that would not have been available at prediction time, and examine performance across relevant products and regions. An acceptable aggregate error can conceal persistent under-forecasting in one part of the business. Agree which failures would prevent deployment before examining the final results.

The release decision should identify the version evaluated, deployment scope, unresolved limitations, required controls and accountable approver. A limited rollout might keep planners in control while the team measures operational results. “Human review” needs to specify what the planner sees, what they check, and what action they can take.

Make deployment readiness observable. A record might allow a forecasting pilot for two regions, with daily planner review and a fallback when inputs are missing. Before activation, the release owner should verify those limits in the deployed configuration and attach the results. If the system is already purchasing automatically, it does not match that decision, however complete its assessment appears.

Set responsibilities for purchased AI

Acquired AI needs an explicit division of responsibility. Establish which controls the provider operates and which the enterprise must configure, test and monitor. A provider’s assurance report cannot establish that your users have appropriate permissions or that its output is fit for your process. Record data handling, retention and model-training terms for the service and account type actually used. The AI vendor risk assessment guide covers that supplier-specific work.

Evaluate AI systems and human oversight

A model score is only one part of the evidence. Test the data pipeline, application, permissions, integrations and workflow that turn an output into a business result. For a customer-facing assistant, that includes the information retrieved, the answer shown, escalation to a person, and the consequences of an incorrect answer.

Select evaluations for the risk being assessed. Quality tests examine whether outputs are fit for the task. Access tests establish whether users can retrieve only permitted data. Where outcomes affect people, examine relevant differences in performance and treatment. For an agent, test whether permissions and action limits hold when instructions or tool responses are unexpected.

For action-taking systems, write tests around prohibited outcomes. Attempt to send an external message before authorization, write outside the permitted records, or continue after access is revoked. Record the observed result and the control that enforced the limit. A prompt asking the model to behave is insufficient evidence that an integration will reject an unauthorized action.

Give reviewers evidence they can challenge: the test population, method, results, limitations, configuration and unresolved failures. Higher-consequence uses warrant stronger independent challenge. Reusing a provider’s benchmark does not establish performance on the enterprise’s data or in its workflow.

Test human oversight as part of the system. Measure whether people notice consequential errors, have enough context to intervene, and can do so before an action takes effect. If checking the output takes as long as doing the original work, that result belongs in the business case. Anthropic’s discussion of trustworthy agents describes the tension between useful oversight and approval prompts that people learn to ignore.

Reassess changes to data and agent permissions

The current operating decision should describe the conditions under which the system may run. Revisit it when those conditions change: a different population, a new purpose, retraining, a new data source, a model replacement, or expanded permissions. The depth of review should follow the significance of the change. A failure or near miss can also reveal that the original assessment missed something, even when no release has occurred.

For the forecasting model, expansion to a market with different demand patterns can invalidate assumptions behind the original evaluation. For an assistant, access to a shared drive or permission to send messages changes what it can expose or do. The illustration below shows how those capabilities change one workflow. It identifies questions for review; it does not calculate a risk tier or grant approval.

Reading documents and sending messages can create an outbound data path when enabled together. Test whether instructions embedded in a document can cause restricted material to be sent to an external recipient. Check the enforced destination and data restrictions, including what happens when the model requests a forbidden action. Our MCP security review guide covers connector permissions, configuration and the evidence needed to review them.

Record each connector’s owner, permissions and the AI application using it. Review the combined configuration: approval for one application does not establish that another uses the connector safely. Apply the same scrutiny to reusable instruction packages, including their source, contents, access and updates. A model version alone cannot describe a system whose behavior depends on changing tools and instructions.

Keep a versioned decision history so operators can distinguish what is currently permitted from a proposal under review. An exception should identify its owner, rationale, compensating measures and expiry. If a required control fails, narrow or suspend the affected operation rather than allowing an expired exception to become the default.

Choose an identity model that makes actions attributable and permissions revocable. Document whether an agent acts through a service identity or delegated user access, which permissions it receives, and how those permissions are removed. Test that authorization is enforced by the connected system. The AI agent governance guide examines these implementation choices in more detail.

Monitor AI systems and respond to incidents

Define monitoring while planning deployment. The owner needs to know what signals will indicate a problem, who receives them, and what response is expected. Depending on the use, signals may include quality degradation, changes in input data, uneven outcomes, unauthorized access, unexpected actions, complaints, cost or loss of business value.

Set thresholds in the context of the task. The forecasting model might require investigation when errors rise for a particular product group, followed by a fallback to the prior planning method if the output becomes unsuitable. An assistant exposing restricted information may need immediate containment. These situations should not wait for the same meeting or follow an identical response.

Investigate incidents and test recovery

Connect AI incidents to existing operational, security and complaint-handling processes. For a restricted document appearing in an assistant’s answer, trace the request to its user, retrieved sources, access decision, model and connector versions, and any downstream action. Contain the affected data path, assess who received the information, and retest the failed control before resuming. Protect investigation records with access and retention limits; collecting every raw prompt can create another sensitive data store.

Exercise the stop procedure before relying on it. Verify which actions it prevents, whether scheduled or queued work can continue, and who can restore service. Preserve enough evidence to investigate the failure without retaining sensitive inputs unnecessarily.

Retirement is also a governance decision. Establish what replaces the system, remove integrations and credentials, stop scheduled work, and determine what data and decision records must be retained or deleted. Marking a use inactive in the inventory should follow completion of those operational steps.

Measure AI governance performance and business value

Report business value and control performance together. For a forecasting model, track stockouts and excess inventory against the agreed baseline alongside error rates and overrides. For an assistant, measure completed work, correction time and cost per accepted result alongside access failures and incidents. Give each measure an owner, a defined population, a reporting period and a threshold that prompts a decision.

Use denominators and explain exclusions. “40 of 50 known active uses have a current decision” makes a coverage gap visible. For an illustrative task, reducing drafting from 20 minutes to 5 saves little if checking and correcting the draft takes another 12. Measure the full workflow before claiming a productivity gain. Review speed, adoption and model accuracy are useful signals, but none establishes value or acceptable risk on its own.

A practical operating rhythm is a regular triage of new uses, changes and overdue actions, with immediate escalation for urgent issues. Leadership reviews the portfolio at a frequency appropriate to its scale and pace of change. Independent assurance can examine whether the process works in practice, including uses that were refused, paused or retired.

Which AI governance frameworks should you use?

NIST AI RMF and ISO/IEC 42001

NIST’s AI Risk Management Framework provides voluntary guidance for managing AI risks across design, development, use and evaluation. ISO/IEC 42001 specifies requirements for establishing and continually improving an organizational AI management system. Both are broader than supplier assessment.

AIUC-1 for agent requirements and assurance

AIUC-1 adds agent-specific requirements and evidence guidance across data and privacy, security, safety, reliability, accountability, and societal risks. It gives teams a concrete basis for selecting controls and testing agent behavior. Use it alongside the enterprise management system and lifecycle risk process described above.

Keep the scope explicit. AIUC-1’s scoping guidance covers agentic systems and distinguishes developers from deployers. Record the agents, configurations, standard version and applicable requirements being assessed. A certificate for a defined set of agents does not establish the adequacy of governance across every enterprise AI use.

Apply frameworks to your operating process

Stanford AISF is an example of an institution-specific security framework. Use such examples to challenge your own review process; their local risk categories, approved-provider lists and decision rights do not transfer automatically to another enterprise.

Use the relevant requirements and guidance to examine the program’s scope, responsibilities and evidence. Map them to the activities teams actually perform: data decisions, evaluations, deployment approvals, monitoring, corrective action and management review. Applicable legal obligations need their own analysis. A framework mapping or management-system certificate does not establish that every AI use is appropriate.

Start with an AI governance pilot

Begin with a manageable cross-section of enterprise uses: an internally built model, an AI feature in a business application, an employee tool, and a customer-facing or action-taking system if the organization has one. This exposes differences in ownership and evidence that a supplier-only pilot would miss.

First establish the sponsor, scope, decision rights and intake route. For each pilot, produce a use record, data-flow map, scenario-based risk assessment, test results and an operating decision with named owners. Exercise a permission change, a failed control and a retirement. Review what worked with the people doing the work, fix unclear responsibilities, and then expand coverage.

The resulting program should let a business owner explain why a use is worthwhile, let a reviewer challenge whether it is appropriate, and let an operator find the current limits and act when they fail. Software should support those responsibilities and preserve their evidence. If you are also evaluating AI-enabled software for broader GRC work, our GRC software evaluation guide offers a separate purchasing checklist.