2026/08/20Who Is Accountable When AI Starts Taking Action?
Artificial Intelligence and Machine Learning | Appar Technologies Co., Ltd. | 2026/08/20
AI Agents are evolving from being assistants that answer questions to becoming "digital actors" capable of querying data, operating systems, and executing processes. As AI begins to truly modify customer data, create orders, and initiate workflows, the issues businesses face shift from "how accurate are the answers" to "what is it authorized to do, who allows it, and who is responsible if it makes a mistake." Recent governance frameworks by Microsoft, NIST, and OWASP all point to the same thing: for AI Agents to truly integrate into core business processes, identity, permissions, decision boundaries, and audit capabilities must be established together.
Within Microsoft, the governance of AI agents is no longer just an occasional security review by the IT department. As employees begin using agents to perform multi-step tasks, connect enterprise data, and automatically trigger processes, Microsoft Digital must establish a management system that can operate long-term, continuously tracking which agents are operating within the company, who created them, what data they can use, and what actions they have taken. Microsoft builds this governance on four foundations: security, governance, management, and observability, and begins treating agents as a formally managed enterprise workload, not just a simple chat tool.
This change reflects the biggest difference between AI Agents and past generative AI. Previously, when employees asked AI to write a letter, organize a report, or analyze a document, even if AI made a mistake, there was usually a person at the final step to confirm. However, agents can operate continuously, choose tools based on information, complete a series of operations across systems, and even trigger subsequent tasks without step-by-step instructions from a human. As AI transitions from "providing answers" to "taking action," the nature of risk changes accordingly.
Therefore, in 2026, Microsoft Entra officially launched Agent ID specifically for AI Agents, allowing agents to have an independent identity different from human accounts and traditional applications. Microsoft's documentation states that these agent identities can be created, authorized, restricted, managed throughout their lifecycle, and their verification and activities can leave audit records. This means businesses are beginning to accept a previously non-existent premise: "identity" within an organization no longer only includes employees, partners, and applications, but may also include hundreds or even thousands of autonomous AI agents.
This is also why the management of AI Agents cannot simply follow the practices of chatbots. If a customer service agent is only responsible for suggesting responses, an error might be an incorrect piece of text; if the same agent can modify orders, initiate refunds, and create compensation plans, an error could directly result in financial loss. In 2026, Salesforce also emphasized when discussing agent governance that as AI moves from answering questions to executing transactions, governance mechanisms must upgrade from "guardrails" at the prompt level to policies and data controls that truly constrain actions.
As more and more businesses allow agents into customer service, finance, procurement, human resources, and information systems, governance will not just be an ancillary task of AI projects but will gradually become part of daily operations. The real question is no longer "whether to trust AI," but how to precisely define: what tasks can be safely entrusted to it, what tasks should only allow it to make suggestions, what situations must be handed back to humans, and how the entire process can be tracked.
AI Agents Begin Taking on Real Work
Microsoft's own agent development history is an example of this transformation. Microsoft Digital states that internal agents have been used to automate multi-step tasks, integrate different systems, and simplify work that originally relied on human coordination. As the number and autonomy of agents increase, Microsoft found that the previously dispersed control methods across different management interfaces were no longer sufficient, so they began using a centralized management approach to understand the creators, users, accessible data, and operational status of agents.
These capabilities may seem like technical management issues, but they are closely related to corporate responsibility systems. Suppose a procurement agent finds a certain raw material below safe inventory levels, automatically inquires with suppliers, and creates a purchase order; this series of actions may fully comply with business processes, but the company still needs to know under whose authorization this agent operates, what amount it can procure, which suppliers can be used, and beyond which threshold it must be handed over for human review. As long as one of these boundaries is not clearly defined, AI's autonomous capabilities could become a gray area of responsibility.
This is also why the NIST AI Risk Management Framework has always emphasized that "governance" must run through the AI lifecycle. NIST's generative AI risk management documents do not focus management on the model-building stage but require organizations to continuously assess, monitor, and manage AI risks in real-world environments, adjusting control methods according to usage scenarios, risk tolerance, and legal requirements. For AI Agents, this continuous governance is especially important because once the same model gains different tools and permissions, the actual impact it can have is completely different.
In other words, companies cannot just set a phrase for agents like "do not perform unauthorized operations" and then expect the language model to comply on its own. When discussing the architecture of authorization-based agents, Microsoft clearly states that in regulated or high-security environments, permissions should be truly enforced by identity and authorization systems, not just relying on prompts to tell the model what it cannot do. This is an important governance principle: security boundaries must exist outside the model.
Redefining Responsibility
To ensure AI Agents continue to create value, companies must first redefine responsibility. In the past, AI projects were often led by IT departments, data teams, or innovation departments, with business units only making requests; however, when agents are actually performing customer service, finance, procurement, or business tasks, the business unit responsible for that work cannot view the agent's results as solely the responsibility of the IT department.
For example, whether a customer service agent can issue a refund, under what circumstances they need to escalate to a supervisor, and what kind of responses meet the company's service standards are not decisions that information engineers can make alone. The technical team can establish permissions, integrate systems, and keep records, but the customer service unit itself is the one that truly understands which decisions are reasonable and which actions might harm customer relationships. Whoever's business process the AI executes must participate in defining its behavioral boundaries.
Microsoft, in its governance framework, also adopts a cross-departmental approach rather than being led by a single IT department. Microsoft Digital recommends that companies establish cross-functional AI governance or centers of excellence, allowing IT, security, legal, data governance, and business units to jointly define principles. Then, based on the agent's scope of use, data sources, actions that can be taken, and risk levels, different levels of management methods are established. This governance is not to make all agents follow the exact same rules but to subject higher-risk agents to stricter controls.
Defining Four Governance Boundaries
By synthesizing Microsoft's corporate practices, NIST's AI risk management framework, and OWASP's research on Agentic AI attack surfaces, we can organize the core AI agent governance issues into four boundaries. These four boundaries are not a list of technical product features but are four sets of questions that a company must be able to answer before allowing an agent to access formal systems.
- Identity Boundary: Who exactly is it?
Every agent performing official work should have an identifiable identity, responsible person, and lifecycle. Companies must know who created the agent, whom it represents, and whether it is still valid. They cannot use a shared account of an employee or an unknown credential source for a long time. - Permission Boundary: What can it see and use?
An agent should not obtain full system permissions just because it "might need it." An agent responsible for querying inventory does not need to modify it; an agent drafting quotes does not necessarily need permission to officially submit them. The principle of least privilege becomes even more important in the age of agents. - Decision Boundary: How far can it autonomously go?
Some tasks can be fully automated, some can only make suggestions, and some must stop and wait for human approval when specific amounts, risks, or conditions arise. Designing where "humans should re-enter the process" is key to safely expanding AI agents. - Audit Boundary: Can events be reconstructed afterward?
Companies must be able to answer what task an agent accepted at what time, what data it used, what tools it called, what changes it made, and whether it was ever approved by a human. Without complete records, it's hard to establish a true accountability system.
These four boundaries are quite similar to the basic logic companies use when managing real employees. When employees join a company, they receive an account, are assigned job permissions, are subject to authorization level restrictions, and their accounts are deactivated upon leaving. Important operations also leave records. The difference is that agents can be massively replicated, continuously operate, and perform far more operations in a short time than humans, so companies need to make these traditional governance principles more automated and real-time.
Capabilities of Mature AI Agent Governance
We believe that a governance system that can truly bring agents into a company's formal environment, like past enterprises establishing information security management, identity management, and DevOps systems, must balance control and operational efficiency. Simply prohibiting all actions renders the agent valueless; relying entirely on the model's judgment exposes the company to unacceptable risks. A mature system needs at least six capabilities:
- Identifiable Agent Identity: Each agent has its own identification, owner, and responsible unit, and can establish a clear relationship with the user or service that created it. Microsoft Entra Agent ID has formally incorporated this type of identity management into the corporate identity system.
- Granular Access Control: Permissions should be determined by user, agent, tool, and data scope, not just by a single shared key.
- Risk Grading: Agents that only query public knowledge should not undergo the same review process as those that can modify enterprise resource planning system data. Microsoft's internal governance also adopts different levels of governance based on the agent's development tools, sharing scope, data sources, and action capabilities.
- Human-Machine Handover Mechanism: When agents encounter high amounts, low confidence, sensitive data, or rule exceptions, the system must know when to stop and hand over to a human, rather than trying indefinitely.
- Complete Observability: Beyond whether the system executed successfully, it should also be able to see agent usage, tool calls, errors, abnormal behavior, human escalation ratios, and actual business outcomes.
- Lifecycle Management: Agents should not exist permanently after creation. When the responsible person leaves, the project ends, tools are replaced, or they remain unused for a long time, there should be mechanisms for decommissioning, re-evaluation, or revoking permissions.
These capabilities also illustrate that AI agent governance is not just "an additional check by the information security department." It involves the company's information architecture, data governance, departmental responsibilities, and operational processes. In OWASP's threat model for Agentic AI, reasoning, memory, tools, identity, human oversight, and multi-agent interaction are all listed as attack surfaces because the risk of agents is no longer concentrated in the model itself but exists in the interaction between the entire system.
Therefore, in the future, companies evaluating whether agents are well-managed cannot only look at "no security incidents occurred." Truly mature governance should also answer another question: Do these restrictions allow low-risk agents to go online faster? If every agent has to go through several months of review, governance itself becomes a bottleneck for innovation. A good system should make risks clearer, allowing companies to automate more confidently.
How to Establish AI Agent Governance
When companies begin to establish governance for Agents, the most common mistake is attempting to create a comprehensive AI policy that covers all future scenarios at once. The technology of Agents evolves too rapidly for a one-time document to address all issues. Microsoft, through its own governance experience, also emphasizes that governance frameworks must be continuously reviewed, as models, tools, Agent capabilities, and corporate adoption methods will change.
A more practical approach is to start with an Agent inventory. Companies first need to know which Agents are currently in use, who created them, what processes they serve, what data they use, and whether they can take action. Only by making Agents visible can they be further managed. Microsoft Agent 365 and AWS Agent Registry have recently highlighted 'centralized discovery and inventory of Agents' as a core capability in enterprise Agent management, reflecting the rapid emergence of this issue.
The second step is to classify Agents by risk, not by model. An Agent using the most advanced large language model but only querying public company documents may pose less risk than an Agent using a smaller model but having the authority to modify payroll data. Governance should be determined by data sensitivity, action permissions, irreversibility, the number of people affected, and financial risk, rather than asking whether it uses GPT, Claude, or Gemini.
The third step is to place real permission control outside the model. Agents can be responsible for understanding needs and planning the next steps, but whether 'it has the authority to call this tool' should be determined by identity, authorization, and corporate policy. This way, even if prompts are attacked or the model makes an error, there remains a second layer of truly effective boundaries. Recommendations from Microsoft, OWASP, and NIST are gradually aligning towards this approach of 'model responsible for reasoning, external control responsible for restriction.'
Finally, governance systems must leave data that can be learned from. Which Agents are frequently interrupted by people? Which tools are most prone to failure? Which processes almost never require human intervention? Which users consistently request operations beyond their permissions? This information not only helps to address issues but also aids companies in gradually readjusting the division of labor between humans and Agents. Without this feedback, governance is just static rules; with this data, governance becomes operational management.
This also means that future AI Agent governance should not occur just once before a project goes live. It is more like cybersecurity monitoring, site reliability engineering, or DevOps, a continuous cycle of observation, adjustment, and improvement. Model updates, workflow changes, and increased data sources can all alter an Agent's original risk, so governance must evolve with the product itself.
AI Governance is Becoming a New Essential Corporate Capability
As Agents begin to enter core enterprise systems, platforms and systems specifically for managing Agents are rapidly taking shape. Microsoft has already incorporated Agent identity and lifecycle into Entra, Amazon Web Services has established Agent Registry and AgentCore Gateway, and OWASP has released a security risk classification specifically for Agentic AI. These actions indicate that the industry is moving from 'how to build Agents' to the next question: 'After building a hundred Agents, how does a company manage them?'
Therefore, now is the time for companies to establish a basic governance framework. Waiting until there are hundreds of Agents in the company to start inventorying will certainly be more costly than designing identity, permissions, records, and lifecycle from the start. Especially as Agents begin to have real execution capabilities, governance cannot wait until the first incident occurs to be established.
Technology itself will not decide responsibility for companies. Models can reason, Agents can execute, systems can automate, but ultimately it must be the company that decides who has the authority to assign what work to AI and how far AI can go. When AI only answers questions, we manage the answers; when AI starts to truly perform tasks, companies must begin to manage its power.
