As AI agents begin handling more steps in business workflows, companies face a familiar governance question in a new form: who is accountable when automated systems act?
From answering to acting
For years, the most visible business use of artificial intelligence was generating an answer: a summary of a report, a draft email or a proposed line of code. The next stage is more operational. AI-enabled systems can be connected to business tools, retrieve information and move between steps of a workflow. An assistant that once suggested how to resolve an invoice discrepancy might now be asked to investigate it, draft a supplier response and prepare an adjustment for approval. The difference is not simply more capable software. It changes where business decisions are made and where control must be exercised.
The governance gap
An automated workflow can be useful even when no single model output is especially impressive. Its value may come from reducing repetitive handovers and making information available earlier. Yet each connection to a business system creates a potential source of error. An agent may draw on outdated records, misunderstand a request, follow misleading instructions embedded in retrieved content or carry an initial error into subsequent actions. The operational consequence depends on what the system is allowed to do. A mistaken summary is inconvenient; an incorrect payment instruction, customer notification or access change can be harder to reverse.
What established guidance contributes
The US National Institute of Standards and Technology offers a useful starting point. Its voluntary AI Risk Management Framework encourages organisations to govern, map, measure and manage AI-related risks across the lifecycle of a system [1]. NIST's generative-AI profile expands on risks particular to these systems, including issues associated with content generation, evaluation and oversight [2]. Neither document is a certification that a particular AI agent is safe. Instead, the frameworks help management ask whether a use case has a defined purpose, understood limitations and a way to detect failures before they become material.
Designing permission around consequence
Companies should distinguish between what an AI system can recommend, what it can prepare and what it can execute. For a low-risk task, producing a first draft for staff review may be enough. For access permissions, payment details, regulatory communications or changes to customer accounts, narrower permissions and explicit approval checkpoints may be justified. Controls can include restricting system credentials, separating testing from production data, logging actions, checking outputs against authoritative records and establishing a reliable way to suspend an automated process. Human review is valuable when it is meaningful: a person needs enough context and time to challenge the result, not just a button marked approve.
Measure failures, not only speed
A business case built entirely on minutes saved will miss important costs. Evaluation should also measure escalation rates, correction effort, inappropriate actions, customer complaints and the proportion of cases that still require expert intervention. Small pilots can expose the difference between a compelling demonstration and a dependable operation. If an AI tool performs well on routine cases but fails on unusual exceptions, the organisation needs to know how frequently those exceptions arise and how quickly staff can identify them.
Accountability cannot be automated away
The governance challenge is therefore less about deciding whether AI agents will become useful and more about deciding what authority they should receive. Boards and executives remain responsible for the processes they oversee, even when a vendor or model performs part of the work. Procurement, technology, legal, compliance and operations teams may each understand one part of an AI deployment; effective oversight requires those perspectives to meet before software is given real-world permissions. The winning design may not be the agent that acts most independently, but the one that makes consequential actions visible, contestable and reversible wherever possible.
References
[1] NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), January 2023
https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10
[2] NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024
https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence