When an AI agent should require human approval

An AI agent should require human approval when the proposed action has a material effect that the system cannot safely bound, reverse or independently authorize.

That rule is more useful than approving every tool call. A read-only lookup and a £5,000 refund do not deserve the same control.

The problem

A prototype often gives the agent one identity and a set of tools. If the model selects a tool and produces valid arguments, the tool runs. The model has effectively become both planner and authorizer.

That is fragile. A hallucination, ambiguous request or prompt injection can produce a well-formed action that the user never intended. OWASP describes excessive agency in terms of excessive functionality, permissions and autonomy. Human approval can reduce autonomy for high-impact actions, but it does not replace narrow permissions or downstream authorization.

Classify the effect before choosing the control

For each action, write down five properties:

1. Reversibility: can the effect be undone completely?
2. Impact: what is the financial, security, legal or customer impact?
3. Scope: can the action cross a user or tenant boundary?
4. Detectability: will the team know quickly if the action was wrong?
5. Ambiguity: can the user’s intent and the tool arguments be established deterministically?

Then choose one of four controls.

Control	Use it when
Automatic	The action is low impact, bounded and independently authorized
Policy-gated	A deterministic rule can fully describe the safe boundary
Human approval	The action is material or intent remains ambiguous
Prohibited	The impact is unacceptable or the system cannot enforce the boundary

“Human approval” is not the default answer. Unnecessary prompts slow the workflow and create approval fatigue.

Approval must bind to an exact action

An approval should contain the tool name, normalized arguments, affected resource, expected effect, policy version and expiry.

For example:

Refund £240 to order 8472 for duplicate delivery. This changes the payment record and will notify the customer.

If the amount or order changes, the approval no longer applies. “Let the agent continue” is too vague because the approver cannot see the consequence.

Amazon Bedrock Agents can request confirmation for specific action-group functions. Its OpenAPI schema supports x-requireConfirmation, and function schemas support requireConfirmation. That is a useful mechanism for pausing execution. The application still needs its own business rules around identity, tenant, thresholds and permitted arguments.

Keep policy outside the model

The model can propose:

{
  "tool": "refund_order",
  "arguments": { "order_id": "8472", "amount": 240 },
  "reason": "Duplicate delivery"
}

Application code should then:

1. validate the schema;
2. retrieve the order from the authoritative system;
3. confirm user and tenant scope;
4. calculate the permitted refund independently;
5. evaluate the approval rule;
6. execute with an idempotency key and scoped identity;
7. record the proposal, decision and outcome.

The model’s reason can help a human understand the proposal. It should not decide whether the refund is allowed.

What usually goes wrong

The tool is too broad. A general database or HTTP tool makes it difficult to define a safe boundary. Replace it with task-specific operations.

The agent uses a shared privileged identity. The downstream system cannot tell which user or tenant authorized the action. Carry user context and enforce scope at execution.

Approval happens before arguments are final. The agent can change the action after the human accepts it. Bind approval to a digest of the final action.

Retries repeat the side effect. An ambiguous timeout causes a second refund or message. Use idempotency and reconcile the original attempt before retrying.

Every action requires approval. Reviewers become a rubber stamp. Keep harmless reads automatic and reserve interruption for consequential decisions.

When approval is not enough

Do not expose an action if the executor cannot limit permissions, enforce tenant scope, validate arguments or prevent duplicate effects. A confirmation dialog cannot repair a missing authorization boundary.

Some actions should remain outside the agent workflow: deleting an account, changing security policy or moving a large amount of money may need an established operational process with separation of duties.

Recommended approach

Start the production release with read-only tools and draft outputs. Add one mutating action at a time. For each addition, require a written action classification, policy tests, adversarial tests, idempotency behaviour, audit fields and rollback or reconciliation path.

The goal is not maximum autonomy. It is enough autonomy to create value inside a boundary the team can explain, test and operate.