Every company building internal AI agents is building a new kind of employee account. It can read across departments, it works at machine speed, and unless someone designs otherwise, it has no manager, no badge and no record of what it did. That is the unaccountable insider: not a character, but a set of credentials with more reach than anyone reviewed.
The controls that prevent it are well understood one at a time. Security teams talk about least privilege, data teams about retrieval, platform teams about where the model runs. The gap is that nobody connects them, and an agent is only as accountable as the weakest of the three. This piece puts them in one design path, for the CTOs and operations leaders who have to sign off on software that acts.
What Is the Short Answer?
Treat every agent as an insider account from the first day, and build the controls in the order the data flows. The model runs on infrastructure you control. Retrieval checks the source system's permissions for the requester on every query.
The agent acts under its own delegated identity, with a named owner, least privilege and a human approval on anything irreversible. Every action writes an audit record, and every task returns its result to the person who assigned it.
Miss any one of those and the others stop protecting you. A private model with a flattened index still leaks documents internally. Perfect retrieval with a shared service account still leaves no one accountable for an action. An agent with every control but no return path is a notification feed, and nobody reviews a feed.
Why Is an Internal AI Agent an Insider Risk?
Because it has what an insider has: legitimate access, standing credentials and knowledge of how the business runs. The difference is that it acts on instructions it receives in text, including text inside the documents it reads, and it can do in a minute what a person would take a week to do.
The breach data shows where this goes wrong. IBM's 2025 Cost of a Data Breach report found 13% of organizations reported breaches of AI models or applications, and 97% of those had no AI access controls in place (IBM, 30 July 2025). The 2026 edition found roughly one in five organizations reported an AI-related breach, 92% of those lacked proper AI access controls, and only 40% of organizations use access controls on AI models and data at all (IBM X-Force, 29 July 2026).
The same report names the agent problem directly. Agents create new identities that reach applications, APIs and data stores, often with elevated privileges, and IBM found fewer than half of organizations are actively securing non-human identities. That is the unaccountable insider in one sentence: a login nobody inventoried, doing work nobody reviews.
There is still time to get this right. Menlo Ventures' December 2025 survey found only 16% of enterprise deployments qualify as true agents, systems where a model plans and executes actions, observes feedback and adapts (Menlo Ventures, 2025). Most companies are designing their first agents now, which is the cheapest moment to design accountability in.
What Are the Layers of an Accountable Agent?
Three layers, in the order a task passes through them, plus the loop that brings the result back. Each answers a different question, and none can answer the others'.
| Layer | The question it answers | What fails without it |
|---|---|---|
| 01. Private model | Where does the context go? | Documents, prompts and agent instructions leave your boundary |
| 02. Permission-aware retrieval | What may this requester see? | The agent becomes a way around your file permissions |
| 03. Delegated identity | What may this agent do, for whom, and who answers for it? | Actions no one can attribute, scope or revoke |
| The closed loop | Did the work get done, and does the requester know? | A one-way feed that reports activity, not results |
Not sure which of your AI tools already hold standing access to company data? Five questions, under a minute, and you will know where to look first.
Find Your AI Ops Score →Why Start With a Private Model?
Because an agent accumulates the most sensitive context your company has. To act, it needs your documents, your processes written precisely enough for software to follow, and a running log of what it did. We set out why that context is the real intellectual property, and the four ways to deploy a model, in should you run your own LLM on-prem.
For agents the argument is stronger than for chat. A chat assistant sees what a person pastes into it. An agent reaches into systems on its own schedule, so the question of where its context is processed has to be settled before it is given any access. A model in your own cloud tenant or on your own hardware settles it.
It settles only that question, though. The IBM figures above are about access, not location: the breached organizations lacked controls over who could reach their AI systems. The model's location decides where data goes. It says nothing about who, or what, is allowed to reach it.
Can RAG Leak Documents to Employees Who Shouldn't See Them?
Yes, and it is usually a design decision, not a bug. Retrieval-augmented generation (RAG) finds the passages most relevant to a question and hands them to the model. If the index behind it was built by copying documents out of their source systems, it has usually lost the access lists that governed them. Relevance is then the only filter, and a salary review is highly relevant to a question about compensation.
OWASP's 2025 Top 10 for LLM applications covers this under vector and embedding weaknesses (LLM08), warning that misaligned access controls can expose embeddings containing sensitive information and that shared vector stores risk context leakage between users. Its mitigation is "permission-aware vector and embedding stores" (OWASP LLM08:2025). Its entry on sensitive information disclosure (LLM02) adds the underlying rule: grant access only to data necessary for the specific user or process (OWASP LLM02:2025).
The design that holds is a permission check at query time, and it has three parts:
- Every chunk keeps its source. The index stores which document, in which system, each passage came from. That id is what makes a check possible.
- Every query carries the requester. Retrieval runs as the person the task is for, not as a service account that can see everything.
- The check asks the source, now. Before a passage reaches the prompt, the system confirms the requester can open that document in the source system today. Permissions copied at index time go stale the day someone changes roles.
Researchers are converging on the same principle. A September 2025 position paper demonstrates data exfiltration through gaps in RAG and fine-tuning, and argues that any content used in retrieval or generation must be explicitly authorized for all users involved in the interaction, enforced deterministically rather than left to the model (Bhatt et al., arXiv:2509.14608). That last point matters: an instruction in a prompt is not an access control.
A useful test for any internal AI search or agent: ask it, as a new junior employee, for the one document in your company that person should never see. If the check is at query time, it stops at the permission check. If the index was flattened, the answer tells you so.
Should Each AI Agent Have Its Own Identity?
Yes, always. An agent running on a shared service account, or on the login of the engineer who built it, cannot be told apart in your logs, cannot be revoked without breaking something else, and cannot be scoped to its job. Its own identity is what makes every other control enforceable.
A delegated identity has four properties:
Its own credentials
A non-human identity issued to that agent alone, with secrets you can rotate and revoke without touching anyone else's access.
One named human owner
A specific employee accountable for what the agent does, who reviews its permissions and its record. Not a team alias. When the owner changes roles, ownership moves explicitly or the agent stops.
Acting on behalf of the requester
When the agent works for someone, it carries that person's authorization, and the effective permission is the narrower of the two. OWASP's excessive agency guidance calls for actions to be executed on downstream systems "in the context of that specific user" (OWASP LLM06:2025).
Least privilege for one job
Access to the systems, records and actions its defined job needs, and nothing it might need later. A new job gets a new review, not a wider grant.
The third property is the one most often missed. An agent with broad standing access that serves anyone who asks is the flattened index again, one layer up: it lets a person do through the agent what they could not do themselves.
What Permissions Should an Internal AI Agent Have?
Fewer than feels convenient, and graded by what an action can undo. OWASP identifies three root causes of excessive agency: excessive functionality, excessive permissions and excessive autonomy (OWASP LLM06:2025). A permission ladder addresses all three by assigning every action type a ceiling.
Applied to common action types, the ladder gives a matrix like this. Your own will differ by system and by risk appetite. The shape should not.
| Action type | Read | Draft | Act with approval | Act autonomously |
|---|---|---|---|---|
| Search and summarise internal documents | Yes, as the requester | Yes | n/a | n/a |
| Log a call note, update a status field, create an internal task | Yes | Yes | Yes | Yes, reversible and logged |
| Email or message a client or supplier | Yes | Yes | Yes, ceiling | No |
| Issue a quote, change pricing or contract terms | Yes | Yes | Yes, ceiling | No |
| Pay an invoice or move money | Yes | Yes | Yes, ceiling | No |
| Delete or overwrite records | Yes | Yes, as a proposal | Yes, ceiling | No |
| Grant access, change permissions, alter its own scope | Yes | No | No | No. A person does this directly |
The last row is deliberate. An agent that can widen its own permissions, or anyone else's, has no ceiling at all, so that action stays entirely with people. NIST's Generative AI Profile lists human-AI configuration among the 12 risks it identifies, including automation bias and over-reliance (NIST AI 600-1, July 2024). An approval gate only works if the approver reads what they approve, so keep the number of approvals low enough that people still do.
What Has to Be True Before an Agent Takes an Irreversible Action?
An irreversible action is anything you cannot take back by editing a record: a message that has been read, money that has moved, a deleted file, a signed document, an access grant already used. Before software does any of those, five conditions should hold, and the system should check them rather than assume them.
- It is inside the agent's defined job. Not merely possible with its credentials.
- Both parties are permitted. The requester and the agent can each perform this action in the source system, checked at the time of the action.
- A named person has seen exactly what will happen. The recipient, the amount, the record, the wording. Approving a summary is not approving the action.
- The approval is recorded. Who approved, when, and what they were shown, written to the audit record before the action runs.
- The outcome is confirmed by the system of record. The action is complete when the target system says it is, not when the model says it is.
OWASP's guidance is to "require a human to approve high-impact actions before they are taken" (OWASP LLM06:2025). The five conditions are what makes that approval mean something.
How Do You Audit What an AI Agent Did?
With one record per action, written at the moment of the action, into a log the agent's own identity cannot edit or delete. The test of an audit record is simple: six months later, could someone who was not there reconstruct what happened, on whose behalf, and whether it was allowed?
| Field | What it records | Why it is there |
|---|---|---|
| Action id and timestamp | A unique id and the time the action ran | Ordering, and a handle for investigation |
| Agent identity | The agent's own non-human identity and version | Separates agent actions from human ones |
| Owner | The named person accountable for the agent | Someone answers for every action |
| Requester and task | Who assigned the work, and the originating request | Ties the action to a purpose and a person |
| Context retrieved | Source document ids, and the result of each permission check | Shows what the output was based on, and that it was allowed |
| Model and instructions | The model version and instruction version in use | Explains behaviour changes over time |
| System and action | The target system, the operation and its parameters | What actually changed, and where |
| Approval | Approver, time and what they were shown, or "not required" with the rung | Evidence the gate held |
| Outcome | The response from the system of record, success or failure | Stops software reporting work it did not do |
| Result returned to | Where the result went, and when | Proves the loop closed |
The outcome field deserves the most care. A model's summary of its own work is a claim. The target system's response is evidence. When the two disagree, the record should hold the evidence, and the discrepancy should be flagged to the owner.
Records built this way are also what an auditor will ask for, which is why we build SOC 2 by architecture from the first agent rather than reconstructing evidence later.
How Does the Result Get Back to Whoever Asked?
Through the same task it started from. Delegation is a loop: a person assigns a task, the agent accepts it under its own identity, does the work within its permissions, and returns the result to that person with what it did and what it touched. The person can then accept it, correct it or send it back.
Much of what is sold as an agent today is the other thing: a one-way stream of alerts and summaries. It reports that something happened and leaves a person to act. That is a dashboard. It adds to the reader's workload, and because nothing is ever handed back, no result is ever owned.
Closing the loop also closes the accountability gap. When every task has a requester, an owner and a returned result, the audit record is no longer a log someone might read. It is the receipt for a piece of work, attached to the person who asked for it.
This is how we operate our own internal agents, on our own self-hosted model: each runs under its own identity with a named owner, reads as the person it is working for, waits for a person before anything irreversible, and returns its result to whoever assigned the task. It is the architecture we design for clients in our AI business automation work, and it follows our method in order.
Map your entire company first, so every agent's job and every handoff is known. Control the data next: where it lives, who touches it and how it moves, settled before any agent is given access. Then Leverage tasks across the org with company-specific co-pilots scoped to the map, and only then build the full AI automation strategy on top.
For the model decision underneath all of this, including which workloads belong on a model you control and which can stay on an enterprise API, see our comparison of on-prem, private cloud and API deployment. For why the order matters, see why AI projects fail when they start with technology.
Where Is the Evidence Thin?
A page making a case should name its weak points, so here are ours.
| The claim | Why it is softer than it looks |
|---|---|
| Breach figures | IBM's percentages describe organizations that reported an AI-related breach, a minority of respondents. They show where breaches happen, not how likely one is for you. |
| How many agents exist | "Agent" has no agreed definition. Menlo's 16% uses a strict one: a system that plans, acts, observes and adapts. Looser definitions produce much larger numbers. |
| Permission-aware retrieval at scale | Checking source permissions on every query costs latency, and not every source system exposes permissions cleanly. The principle is settled; the engineering depends on your systems. |
| The permission matrix | It is our design pattern, not a published standard. OWASP and NIST support the principles behind it, not these exact rows. |
The pattern to take from that table: the direction is well supported, and the details are yours to set. Decide the ceilings per action type with the people who own those systems, and write them down before the first agent gets credentials.
Frequently Asked Questions
The fewest that let it do one defined job, checked on every action rather than granted once. Think in four rungs: read, draft, act with approval, and act autonomously. Most agents should read and draft as the person who assigned the task, act only with a named person's approval on anything that is sent, paid, deleted or signed, and act alone only on reversible, bounded work such as logging a note or creating an internal task. OWASP lists excessive functionality, excessive permissions and excessive autonomy as the three root causes of excessive agency in LLM applications.
Write one audit record per action, at the moment of the action, into a log the agent cannot edit. The record names the agent's identity, its human owner, the person the task was done for, the originating task, what was retrieved and whether each permission check passed, the system and action called, any approval with its approver and time, and the outcome as reported by the system of record rather than by the model's own summary. That last field is what stops software from reporting work as done when it was not.
Yes, and the usual cause is architectural. When documents are copied into a shared index and their access lists are dropped, retrieval ranks by relevance alone, so a restricted file can surface in anyone's answer. OWASP's 2025 guidance on vector and embedding weaknesses warns that misaligned access controls can expose embeddings and recommends permission-aware vector stores. The fix is to keep each chunk's source id and check the source system's current permissions for the requester on every query, so a file the person could not open never reaches the prompt.
Yes. An agent that borrows a shared service account or an employee's login cannot be told apart in the logs, cannot be revoked on its own and cannot be scoped to its job. Give each agent its own non-human identity, its own credentials and one named human owner, and have it act on behalf of the requester with the narrower of the two sets of permissions. IBM's 2026 breach report found that fewer than half of organizations are actively securing non-human identities.
Five things. The action is within the agent's defined job, the requester and the agent are both permitted to do it in the source system, a named person has seen exactly what will happen and approved it, the approval is recorded with who and when, and the outcome is confirmed by the system of record afterwards. Sending to a client, moving money, deleting records, signing and changing anyone's access all qualify. OWASP recommends human approval for high-impact actions before they are taken.
Because an agent accumulates the most sensitive context a company has: its documents, its processes written precisely enough for software to follow, and a log of what it did. On a model you control, that context stays inside your boundary and under your access controls. A private model is necessary but not sufficient. IBM's 2026 report found only 40% of organizations use access controls on AI models and data, so the model's location settles where data goes, not who can reach it.
Delegation closes a loop. A person assigns a task, the agent accepts it under its own identity, does the work, and the result returns to that person with what was done and what it touched, where it can be reviewed or corrected. A notification feed only flows one way: it tells people something happened and leaves them to act. A feed can be useful, but it is a dashboard. It does not reduce anyone's workload, and it gives no one a result to hold accountable.