Why Embedded AI Agents Demand New Enterprise AI Governance
The next enterprise control failure may be disguised as a routine software update. A release can activate a new AI feature inside an established platform, connect it to data and workflows already in use, and allow it to influence decisions before the organization has fully assessed its inherited authority. What appears to be incremental product functionality can alter the operating risk of the entire application.
AI now enters the enterprise through purpose-built models, enterprise AI platforms, SaaS platforms, productivity suites, security tools, customer systems, and workflow applications. Increasingly, these capabilities retrieve information, select tools, initiate transactions, and coordinate work across connected systems, allowing familiar software providers to introduce a new class of operational exposure.
Success with enterprise AI strategy will increasingly depend on how confidently organizations can extend trust from software that informs to software that intervenes. For AI-enabled software vendors, that shift elevates product governance from a back-office requirement to a condition of enterprise adoption. Vendors that build these capabilities early will be better positioned as enterprise demand for AI-enabled software grows. AI vendor risk is fundamentally a delegated-authority problem. That authority should be bounded through permission, observable through proof, interruptible through pause, and continuously reassessed as software changes.
Classifying Enterprise AI Risk for Embedded Agents
In 2025, 20% of EU enterprises used AI technologies and adoption reached 55% among large enterprises, according to Eurostat. AI can enter through applications enterprises already use, including embedded capabilities introduced through product updates and new feature rollouts.
Governing AI agents embedded in enterprise software now requires enterprise infrastructure controls. That shift appears in standards activity: in February 2026, NIST launched an AI Agent Standards Initiative focused on interoperable and secure AI agents, including identity, authorization, protocol development, security evaluations, and implementation guidance.
Executives therefore need a more discriminating AI risk assessment to classify exposure. Six factors determine the operating significance of an embedded agent: how independently it can act, the authority it has to access or change systems and data, the reach of its connections, the visibility of its actions, the frequency with which its behavior can change, and the reversibility of its outcomes. A meeting assistant that drafts a summary warrants a different risk tier from a CRM agent that amends customer records, a coding agent that commits code, a finance agent that approves invoices, or an identity agent that disables accounts.
Model accuracy matters, but the same error can produce radically different consequences depending on what the system has permission to do. AI risk management should therefore center on its delegated authority.
Key Takeaway
Classify AI-enabled software by the authority it can exercise, the systems it can reach, and the reversibility of its actions, rather than by model type or the presence of an AI label.
Explore Our Consulting Services
P&C Global delivers end-to-end consulting services with accountable outcomes
Why Conventional Vendor Reviews Break at Runtime
The governance gap is already visible. A recent poll of 3,400 digital-trust professionals found that while 90% believed employees were using AI in their organizations, only 38% reported having a comprehensive AI policy. One-quarter reported no active policy, according to ISACA.
When AI arrives as a feature within software the enterprise already uses, it may not receive the scrutiny applied to a net-new platform or vendor relationship. The underlying application may already have cleared procurement, security, privacy, and legal reviews, while the embedded AI capability introduces new permissions, data uses, decision support, or workflow actions. AI governance must therefore reassess established software whenever AI materially changes what that software can do.
Traditional software reviews address financial viability, cybersecurity, privacy, availability, data location, and liability. Those controls remain essential, but they do not answer the questions that determine agent risk: Can the system distinguish the agent’s identity from the user’s? Does the agent inherit the user’s full authority? Which model, retrieval source, plugin, tool server, or external API contributed to an action? Can the enterprise detect when that operating boundary changes?
“Model-update risk” captures only part of the exposure. Agent behavior can change when a foundation model is replaced, but also when a system prompt, tool definition, retrieval corpus, memory store, safety policy, integration, or upstream service changes. A vendor may not retrain a model and still materially alter what the product can access, infer, or execute. A customer-service platform, for example, could retain its model but add a connector to CRM, order-management, and billing systems—moving its agent from drafting a response to changing a subscription, issuing a credit, or updating a customer record. Enterprise AI governance must therefore extend from point-in-time approval to continuous assurance across the product lifecycle. Continuous assurance means reassessing risk when material changes occur in models, tools, permissions, data sources, integrations, or upstream dependencies, rather than relying only on annual review.
The NIST Generative AI Profile reinforces this approach by identifying value-chain integration as a distinct risk and recommending that organizations address embedded AI during procurement, inventory third parties with content access, continuously monitor third-party systems, reassess material changes, and establish fallbacks for critical services.
Enterprise customers should require advance notice of material changes, precise upstream-dependency disclosures, rights to evaluate relevant controls, and the ability to disable or roll back capabilities. AI-enabled software vendors should build these requirements into product design, documentation, and customer support processes. A one-time contracting questionnaire cannot provide sufficient assurance for software whose capabilities, connections, and behavior can change after initial approval.
Key Takeaway
Replace one-time vendor approval with continuous governance, supported by ongoing assurance across deployment, material changes, runtime performance, incidents, and retirement.
The AI Governance Controls Enterprise Customers Will Require
Most enterprise identity architectures were designed to govern people and conventional applications, not AI agents that can act autonomously across connected systems. A Cloud Security Alliance study commissioned by Strata Identity found that only 21% maintained a real-time agent registry, while 28% could reliably trace activity to a human or system across all environments. The study also identified continued reliance on static credentials and shared accounts—access patterns that blur the boundary between a human, an application, and an agent.
Many organizations therefore cannot reliably identify every agent or attribute its actions to a responsible person or system, limiting their ability to enforce least privilege, investigate incidents, demonstrate compliance, and rapidly revoke access. AI agents require an identity model that establishes who or what is acting, on whose behalf, with what authority, and how each action can be verified.
NIST’s 2026 concept work on software and AI agent identity and authorization centers on authentication, delegation, policy-based authorization, non-repudiation, and data provenance. The initiative remains under development, underscoring that human-centric identity controls do not yet fully address the requirements of autonomous software.
That gap calls for a practical control plane built around three requirements: permission, proof, and pause.
Permission gives every agent a distinct identity and limits its authority to a defined purpose. Access should reflect task, data sensitivity, transaction value, environment, time, and delegating principal. High-impact actions require stronger authorization than low-risk work; credentials should expire and permissions should narrow when the task ends. For example, an agent retrieving a policy document may receive time-bound, read-only access to an approved knowledge base. An agent that can issue a customer credit or alter supplier banking details should require distinct authorization, tighter transaction limits, and human approval before execution.
Proof preserves enough evidence to reconstruct an event: the initiating principal, agent and model versions, instructions, data sources, tool calls, policy decisions, approvals, actions, and outcomes. Logs must resist tampering while observing privacy and retention requirements.
Pause contains a questionable action through shutdown, credential revocation, transaction limits, circuit breakers, rollback, quarantine, and tested fallback. Human oversight should focus on material or irreversible decisions. Approval everywhere creates fatigue; approval nowhere creates uncontrolled authority.
Microsoft Copilot Studio, for example, automatically assigns a Microsoft Entra Agent ID to each new agent and records administrative, maker, and user interactions in audit logs. ServiceNow applies access controls, user identities, role masking, and data permissions to AI agents. Its AI Control Tower can also require AI-steward approval before an AI asset is deployed.
These developments signal a clear market direction. The unresolved challenge is integrating identity, permissions, evidence, and intervention controls into a consistent enterprise operating model that spans vendors, systems, and business workflows.
Key Takeaway
Give every agent a distinct identity, bounded authority, traceable delegation, and a tested path to containment.
Explore Our AI Consulting Services
P&C Global’s AI consulting services help enterprises move from pilot-stage initiatives to scalable AI operating models.
The AI Vendor Playbook: Make Trust a Product Capability
For enterprise customers, trust in AI-enabled software begins with the product itself. Identity and access controls, evidence, change control, recovery, and exit capabilities must be engineered into the platform rather than bolted on through contracts or late-stage assurance. Vendors that productize these capabilities can reduce security exceptions and give enterprises greater confidence to deploy AI in consequential workflows.
The commercial imperative already extends well beyond technical compliance. Among EU enterprises that considered but did not adopt AI in 2025, 53% cited uncertain legal consequences and 49% cited data-protection or privacy concerns, according to Eurostat. Although these figures do not measure vendor selection directly, they demonstrate how unresolved trust requirements can constrain adoption. Product assurance has become part of market access.
Standards activity reinforces this direction. In January 2026, ETSI published EN 304 223, establishing 13 baseline AI cybersecurity principles across the lifecycle from design through end of life. ISO/IEC 42001 sets requirements for an organizational AI management system, while ISO/IEC 23894 integrates AI-specific risk into enterprise risk management.
A practical vendor response begins with bounded authority. Customers should be able to see and configure which data, tools, and transactions an agent can access and apply approval rules based on consequence. Routine product updates should never silently expand that authority
Vendors must also make evidence portable. A reusable customer assurance package should cover intended uses and limitations, architecture and dependencies, data-use and retention terms, control mappings, evaluation results, change history, logging capabilities, and incident procedures. Model cards can contribute, but customers buy systems, not isolated models. Assurance must cover the complete product context.
Customers also need enough observability to determine which agent performed an action, under whose authority, with which product version, and against which policy—without requiring vendors to expose trade secrets or internal reasoning.
Upstream dependencies require the same discipline. In 2026, CISA and G7 partners published minimum elements for an AI software bill of materials, extending software component transparency to the models, datasets, and other relationships used to build AI systems. Vendors should know which critical components support their products, how changes are assessed, and what happens when an upstream provider becomes unavailable, insecure, noncompliant, or commercially unviable.
Finally, vendors need to engineer recovery and exit. Customers require tested ways to stop an agent, revoke access, preserve evidence, restore operations, delete retained data, and confirm decommissioning. Residual credentials, abandoned integrations, and retained memory can otherwise become lasting exposure.
Key Takeaway
Build assurance into product architecture and customer evidence so trust accelerates adoption instead of becoming a late-stage sales obstacle.
Shared AI Governance Accountability Across the Value Chain
AI-enabled software creates shared accountability. Vendors and enterprise customers control different parts of an agent’s operating environment and must explicitly define their responsibilities before consequential workflows go live.
Enterprise customers are best positioned to own use-case selection, local configuration, business-process controls, workforce readiness, data access, and the authority granted within their environments. AI vendors are best positioned to own secure product design, accurate capability disclosures, upstream-component governance, version control, vulnerability handling, and timely incident communication.
Regulation reinforces this allocation of responsibility. The EU AI Act assigns distinct duties to providers, deployers, importers, distributors, product manufacturers, and parties that substantially modify high-risk systems. Under Article 25, a party that makes a substantial modification to a high-risk system, or changes an AI system’s intended purpose so that it becomes high risk, may assume provider responsibilities. For qualifying high-risk systems, providers and deployers must retain automatically generated logs under their control for at least six months.
Some responsibilities must be jointly operated: acceptance tests, material-change thresholds, approval requirements, log availability, incident escalation, fallback, and post-update revalidation. Information-sharing must protect intellectual property, system security, and other customers’ privacy.
Contracts remain important, but they should document an operating model rather than substitute for one. An effective AI addendum should address permitted use, data use and retention, upstream providers, material changes, control evidence, incidents, continuity, liability, data return, and decommissioning. If the architecture cannot enforce the agreement, the clause provides limited protection.
Key Takeaway
Allocate responsibility explicitly across vendor design, customer configuration, and the controls both parties must operate together.
The Executive Scorecard for Agent Assurance
Many organizations still lack the visibility and response readiness needed to govern AI-enabled systems with confidence. In ISACA’s 2026 research, 56% of respondents did not know how long it would take to halt an AI system after a security incident, while 39% did not know whether a documented shutdown or override process existed. Only 38% expressed confidence in their board’s understanding of and action on AI risk.
AI governance is a leadership issue as much as a technical one. Enterprise executives should require a concise scorecard that shows whether delegated authority remains controlled throughout deployment and operation:
- Inventory coverage: Percentage of AI-enabled applications and agents with a named owner, defined purpose, risk tier, and current dependency record.
- Identity and authorization: Percentage of agents with unique identities, time-bounded credentials, scoped permissions, and traceable delegation.
- Action control: Percentage of material or irreversible actions subject to policy enforcement, transaction limits, or meaningful human authorization.
- Traceability: Percentage of agent actions that can be reconstructed across systems, including the initiating authority, tools used, policy decisions, and outcome.
- Change assurance: Percentage of material model, tool, permission, and data-source changes that receive notice, testing, approval, and rollback preparation before production use.
- Containment and recovery: Time required to identify an agent, revoke its authority, stop propagation, preserve evidence, and restore the affected process.
- Supplier resilience: Percentage of critical agent services with mapped upstream dependencies, incident commitments, tested fallback, and an executable exit plan.
For most coverage measures, the target should converge toward 100%. As controls mature, executive reporting should shift toward exceptions: what remains outside the standard, the risk created, who owns remediation, and when it will be resolved. Vendors should apply the same discipline to customer-facing capabilities, tracking measures such as change-notice compliance, privileged-action exceptions, remediation timelines, incident containment, and successful decommissioning.
Key Takeaway
Measure whether delegated authority is known, bounded, traceable, and recoverable—not the number of AI policies, pilots, or principles the organization has produced.
Trust Must Be Engineered Before Autonomy Scales
The enterprise software market is approaching a dividing line between AI capabilities that customers can activate but cannot fully govern and products designed to make increasing autonomy observable, bounded, and controllable. As agents enter more consequential workflows, that distinction will become commercially and operationally material.
Enterprise customers must govern the authority each system receives. Vendors must make that authority visible, enforceable, traceable, and interruptible. The next standard of enterprise trust will be determined not simply by what AI can do, but by whether organizations can reliably control what it is allowed to do.
