RAG Is Not the Decision Layer Your Agent Needs

RAG can retrieve evidence. It cannot decide what the company currently believes, which source wins, or which action is allowed. That is the part many agent architectures hide inside a prompt. Imagine a GTM agent asked: should we email the CFO at Acme today? A good retrieval system can find the latest call transcript, the CRM opportunity, a Slack thread, the pricing note, and the account owner’s comment. The model can summarize all of it. It can sound careful. It can even cite the right snippets. But the real decision is not “which snippets are relevant?” The real decision is: what is the accepted account state, what changed most recently, which source has authority, what promises are open, what approvals are required, and what next action is valid? That is not retrieval. That is operating state.

What retrieval is supposed to do

Retrieval-augmented generation is a strong pattern when the model needs information that is private, large, or changing. Instead of asking the model to answer from parametric memory, the system searches a corpus and passes relevant chunks into the model. That is useful. It is often necessary. Policies, docs, transcripts, sales notes, product information, support articles, and research archives all become more useful when an agent can search them instead of guessing. Graph-based retrieval pushes this further. Microsoft’s GraphRAG docs describe GraphRAG as a structured, hierarchical approach to retrieval that extracts a knowledge graph from text, builds community summaries, and uses those structures for question answering over private data. That is a real improvement over naive semantic search. But retrieval, even graph-shaped retrieval, mostly answers this question: What evidence should the model inspect? A production company agent needs another question answered first: What state does the company currently accept as true, and what can safely happen from here?

The missing object is accepted state

Accepted state is the company’s current governed belief about an object. It is not the raw CRM row. It is not the latest Slack message. It is not the transcript. It is not the model’s best summary. Accepted state is compiled from evidence, freshness, source priority, business rules, and sometimes human approval. For an account, accepted state might include:

AccountContext
- account: Acme
- current_stage: procurement
- stage_source: call transcript, June 9
- crm_stage: evaluation, stale
- open_commitments: security questionnaire by Friday
- active_risk: DPA review
- owner: Maya
- blocked_actions: no pricing claims without legal approval
- allowed_next_actions: draft follow-up for owner review
- evidence: call note, CRM record, Slack decision, product policy

That object is different from a retrieved bundle of text. It tells the agent which version of the company to reason over. It also gives the agent an action boundary. The agent is not merely reading about Acme. It is operating on the accepted state of Acme.

Why chunks are not enough

The easiest way to see the failure is to create a source conflict.

• The CRM says the opportunity is in evaluation.
• The latest call says procurement is already involved.
• Slack says the champion left.
• A spreadsheet says the account is temporarily do-not-contact.
• A sales playbook says the next step after evaluation is a mutual action plan. A retrieval system may pull all five artifacts. The model may notice the conflict and write a cautious answer. But unless the system has a decision layer, the agent still has to improvise the thing a good operator would know: which source wins, what field should change, who owns the next move, and whether outbound is blocked. This is where demos lie. In a demo, a model that sees five snippets and produces a polished recommendation looks intelligent. In production, that same recommendation can be wrong because the system never represented the company’s actual state. The problem was not that retrieval failed. Retrieval did its job. The missing layer was the governed model of the company.

RAG, GraphRAG, and ontology are different layers

A useful way to separate the layers:

• RAG retrieves relevant evidence.
• GraphRAG improves retrieval with entities, relationships, hierarchy, and summaries.
• A company ontology defines the operating objects, relationships, source rules, and action boundaries the company wants agents to use. Those layers can work together. They should. A company ontology does not replace retrieval. It gives retrieval a job. The ontology says what kind of thing the agent is working on: account, person, opportunity, meeting, commitment, risk, ticket, approval, owner, source, action. The graph says how those things relate. Retrieval brings in supporting evidence. The decision layer turns evidence into accepted state and allowed action. The trace records what the agent saw, decided, and did, so a human can correct it later. That is the difference between an agent that searches company data and an agent that can operate inside a company. Ortelian’s shorter notes on access not being a map and what an ontology is make the simple version of this argument. This piece is the implementation version: retrieval is necessary, but it is not the decision layer.

What the architecture should look like

A retrieval-first agent often works like this:

1. Receive a task.
2. Search docs, CRM, Slack, calls, and tickets.
3. Put the top chunks in the context window.
4. Ask the model to answer or act. A company-state agent adds a layer before action:
5. Resolve the object: account, person, ticket, promise, workflow, risk.
6. Load the relevant ontology slice: what this object is and how it relates to others.
7. Query current accepted state from the graph or state layer.
8. Retrieve evidence for fields that need support or dispute resolution.
9. Apply source priority and business rules.
10. Assemble a context packet with state, evidence, rules, allowed actions, and approval gates.
11. Let the agent reason and act through governed tools.
12. Record the trace and corrections. The key move is that evidence and state are not the same thing. Evidence says, “this call contains a promise.” Accepted state says, “this promise exists, belongs to this account, is owned by this person, is due Friday, and blocks this next action until resolved.” An agent can cite evidence. It should act on accepted state.

What to model first

Do not model the whole company on day one. Pick a workflow where wrong context causes visible pain. For GTM, the first useful slice is often account follow-up:

• Account
• Person
• Opportunity
• Meeting
• Commitment
• Risk
• Owner
• Next action
• Approval rule
• Source Then define relationships:
• Meeting is about Account.
• Meeting produced Commitment.
• Commitment has Owner.
• Risk blocks Opportunity.
• Source supports Fact.
• Approval Rule governs Action.
• Next Action depends on Commitment. Then expose agent-facing functions:
• get_account_context(account)
• list_open_commitments(account)
• resolve_source_conflict(object, field)
• propose_next_action(account)
• request_approval(action) Those functions are not dashboard filters. They are the agent’s interface to the company’s operating model.

The practical test

Before adding more retrieval, ask seven questions:

1. What company object is the agent operating on?
2. Which relationships must it understand before acting?
3. Which source wins when systems disagree?
4. Which facts are accepted state versus raw evidence?
5. Which actions are allowed, blocked, or approval-gated?
6. What evidence should be cited back to the human?
7. How does a correction update the state for the next run? If the system cannot answer those questions, better search will only make the agent better at finding ambiguity. Retrieval gives the agent evidence. The decision layer tells it what the evidence means.