How to Benchmark AI Agent Token Cost and Accuracy in an Afternoon

Most teams can say their AI agent returns something. Far fewer can say what each answer costs or whether it is right. Those turn out to be one question.

In McKnight Consulting Group's August 2026 benchmark, "Stop the Token Bleed," the model never changed; only the agent's route to the data did, and that one variable set both the token bill and the accuracy score. The study ends with a recipe anyone can reproduce in an afternoon without production data.

This post turns it into a runbook, then adds what Vantage Point would check before trusting the number.

Why benchmark the route, not the model?

Public benchmarks measure models, not the architecture around them, and the architecture is the part you control. McKnight's study isolated it: the model (claude-opus-4-8 in the Claude Code harness), the dataset, the request, and every provider default stayed fixed. Only the access instruction changed.

The headline, on the study's input-plus-output basis: the governed route through Informatica Cloud Data Governance and Catalog (CDGC) returned every correct column and nothing else (F1 1.000) at all four tiers from 300,000 to 15 million objects, for 2,112–4,019 tokens a question. The ungoverned routes cost 16,563–67,089 tokens and never beat F1 0.656; at 1.5 million objects the gap peaked at 29.8x. Our companion post, what the token bleed study found, has the full results.

Disclosure and limits. The benchmark features CDGC from Informatica, a Salesforce company; Informatica is now part of Salesforce. The authors acknowledge the relationship ("A benchmark that only reports where the sponsor's architecture wins is marketing") and publish two conditions where the governed approach loses: a stale catalog, and identifiers its classifiers do not recognize. Our own read adds four: one task type (data discovery), one model, synthetic metadata, and the best-F1 run of three per cell. Treat the ratios as directional; the method is the reusable part.

The three routes you're comparing

Route	What the agent gets	Where the cost goes	How it failed in the study
Direct live access	Connection details; the agent scans and queries the raw source.	Scanning the source every run; output tokens as the agent reasons.	Under-returned at scale: at 15M objects, precision 0.638, recall 0.482.
Context stuffing	Schema, table, and column definitions in the prompt; nothing organizes them.	Loading metadata up front; long outputs as the agent sifts.	Over-returned: recall up to 1.000, precision 0.19–0.49; at 15M, 35,708 of 53,183 returned columns were wrong.
Governed retrieval layer	A query interface to a pre-classified catalog, exposed as a tool (in the study, CDGC via an MCP server).	Querying the catalog; classification was paid once, at ingest.	Neither, subject to the freshness and classifier caveats above.

If your agents reach systems of record through the Model Context Protocol, test the third route: the tool returns a governed answer set, not raw tables.

Step-by-step: the afternoon benchmark

Adapted from the study's "Run This Yourself" section.

1. Pick one sensitive-data category and write the question as an analyst would ask it. The study's: "Show me all the assets needed to build a report that involves Government ID."
2. Generate synthetic metadata: schema, table, and column definitions only; no rows, no PII. The study used the open-source Faker library at 300,000 to 15 million objects, modeled on a financial services enterprise. A metadata-only export of your own estate also works if governance allows.
3. Freeze the answer key before any run. Machine-enumerate every column matching your correct-set patterns (the study's: SSN, SUBJECT_SSN, ID_DOC_NUMBER) into a file, and plant three or four look-alike decoys (the study's: TAX_ID, LICENSE_NO/DL_NUMBER, DOC_TYPE).
4. Run the identical prompt through each route in a fresh agent with no shared memory. Change only the access instruction: connection details, a metadata file path, or the governed tool. Keep provider defaults.
5. Capture the returned set and the usage record per run: input, output, cache read, cache write.
6. Score after all runs finish as exact set membership against the frozen file, identifiers normalized for case, quoting, and path style.

Metric	Formula	What it answers
Precision	TP / (TP + FP)	Of what it returned, how much was right?
Recall	TP / (TP + FN)	Of all correct columns, how many did it find?
F1	2 × Precision × Recall / (Precision + Recall)	Harmonic mean; falls when either one falls.

Experimental controls checklist

• A fresh agent per route: no memory of the others, no prior exposure to the key.
• No agent sees another's workspace, tools, or results.
• The coordinator relays the request and nothing else.
• Scoring only after every run finishes.
• Identical request text; only the access instruction differs.
• Key authored before the catalog exists; catalog populated by automated classification, no hand-labeling.
• Exact set membership against the frozen file, identifiers normalized.

How to read the results: two different ways to be wrong

Over-return is high recall with low precision. Context stuffing at 15 million objects returned 53,183 columns; 35,708 were wrong. An inventory that is two-thirds decoys is comprehensive and useless.

Under-return is a clean, short list with low recall. Direct access at 15 million objects had the better precision of the two ungoverned routes (0.638) but missed 12,719 of 24,540 correct columns. The study calls that the more dangerous compliance error, and we agree: a short, clean list reads as authoritative, and nobody looks for what it left out.

Neither failure was a hallucination; every wrong pick was a real column. The model could tell SSN from TAX_ID. What it lacked was a way to know which of five million objects to look at. That is a selection problem, and selection is an architecture decision.

Watch output tokens, which cost more than input. Nearly all of the gap sits there (4,000 for the governed route at 15M versus 66,321 for direct access), and the study reads output volume as "a direct readout of how much work the agent had to do to convince itself it had an answer."

What Vantage Point would add before trusting the number

Our additions, not the study's protocol.

1. Count every token class. The study's main tables count input plus output only; its full-accounting table for the airline passenger-contact scenario adds cache read and write, and the direct-access gap moved from 26.8x to 88.8x on the same runs. Report both bases and never mix them.
2. Run each cell at least three times and report the spread. The study reported the highest-F1 run of three and notes ungoverned costs swung with "no stable relationship to data size."
3. Add a production-shaped query. The study's airline scenario asked for four attributes across five source systems, and context stuffing's F1 fell to 0.094. Real requests are wider than test prompts.
4. Stale the catalog and rerun. The study measured a freshly ingested catalog and warns that a layer six months stale "will route an agent confidently toward objects that no longer exist, and it will do so cheaply."
5. Check classifier coverage against your own identifiers. The governed route scored 1.000 because out-of-the-box classifiers recognized the patterns present; a proprietary scheme with no matching classifier inherits the ungoverned routes' selection problem.
6. Budget on predictability. The governed route stayed inside a 2,112–4,019 band across a 50x scale increase, while ungoverned cost had no stable relationship to data size. As the study puts it, "A cost that cannot be forecasted cannot be budgeted or capped."

Decision table: what your result pattern means

Result pattern	What it means	Next move
Governed route: high F1, low and stable tokens	Selection is done once, at ingest.	Scale it, with a freshness SLA and a coverage check for new identifiers.
Context stuffing: high recall, low precision	The agent finds the answer and buries it in decoys.	Stop stuffing schemas; move selection ahead of the prompt.
Direct access: decent precision, low recall	Authoritative-looking omissions.	Treat output as incomplete until a governed inventory confirms it.
Governed route also fails	Classifier gap or stale catalog.	Check the ingest date, add classifiers for the missed patterns, rerun.

How Vantage Point Helps

Vantage Point is a Claude Partner Network Member and a Salesforce implementation partner. We design and implement the Salesforce data foundation agents depend on (Salesforce Data 360, formerly Data Cloud; MuleSoft; Informatica-fed metadata) and the governed access layer above it: MCP tools that return governed answer sets, catalog and metadata layers, and guardrails for Claude and Agentforce agents. We run this benchmark with your team on your own metadata and manage the freshness and coverage checks afterward. Earlier in the journey? Start with why trusted master data matters for AI agents. Senior consultants only — no junior handoffs; the experts you meet are the experts who deliver.

Frequently Asked Questions

What do precision, recall, and F1 mean for an AI agent?

Precision is the share of what the agent returned that was correct; recall is the share of all correct items it found; F1 is their harmonic mean, which falls when either falls. Low precision buries the answer in wrong picks; low recall silently omits correct items.

Do I need production data to run this benchmark?

No. McKnight's study used synthetic metadata from the open-source Faker library: schema, table, and column definitions only, with no rows and no PII. A metadata-only export of your own estate also works if governance allows.

Why not just put the schema in the prompt?

Because the study measured that approach and it produced the benchmark's worst results: precision as low as 0.191 and F1 0.094 on the production-shaped run. A bigger context window gives the agent more to search, not a way to know what matters.

Which token counts should I include?

All four, reported separately: input, output, cache read, and cache write. The study's main tables use input plus output only; its full-accounting table adds cache, and the direct-access gap grew from 26.8x to 88.8x. Never compare numbers on different bases.

How many runs are enough?

At least three per route per tier, reporting the spread rather than the best run. The study reported the highest-F1 run of three per cell and saw ungoverned costs swing with no stable relationship to data size; the variance is part of the finding.

Does this only apply to Informatica or Salesforce environments?

No. The governed route needs any pre-classified catalog exposed as a tool; the baselines need a live connection and a metadata export. The study used Informatica CDGC through an MCP server, but the test works for Agentforce, Claude, or any agent that reaches data through tools.

Sources

• Stop the Token Bleed — McKnight Consulting Group, August 2026
• Faker documentation
• Model Context Protocol documentation
• What is MCP? Model Context Protocol explained — Vantage Point
• Informatica master data and trusted context for AI agents — Vantage Point

Vantage Point is a boutique CRM consulting firm helping businesses transform with Salesforce, HubSpot, and AI — 150+ clients, 400+ engagements, and a 4.71/5 average engagement rating. Learn more at vantagepoint.io.