Quick Answer
OpenAI's October 6, 2026 research post reports a computer-use study run with Ironclad, the AI contracting platform: GPT-6 Astra averaged 55.0% of the available rubric criteria across 11 realistic contracting tasks, at an estimated 19.2 minutes per attempt, versus 41.6% and 37.0 minutes for GPT-5.6 Sol. The gain is real — a 32% higher average score at 48% lower estimated time — and OpenAI's own conclusion is that human oversight remains essential. For CCOs and COOs at banks, insurers, and RIAs scoping 2027 agent pilots, the lesson is where controls live: in the system the agent works inside — validation rules, approval steps, locked fields, audit trail — not in a bet on the next model.
Key Takeaways (TL;DR)
- What happened: OpenAI published results from a research collaboration with Ironclad — 11 contracting tasks, each scored against 8–50 criteria, run inside hosted Ironclad environments.
- The headline number is a score, not a completion rate: GPT-6 Astra averaged 55.0% of available criteria per task, GPT-5.6 Sol 41.6% — so on average, roughly 45% of the rubric criteria were still missed.
- Speed roughly doubled: estimated time per attempt fell from 37.0 to 19.2 minutes. OpenAI notes these are simulated estimates, not measured customer time savings.
- For regulated firms: a half-right agent is not half as valuable. Every missed criterion is an exception a person has to catch, so controls belong in the system, not in model optimism.
- Bottom line: scope agents inside a governed CRM with approval steps, locked language, and an audit trail — don't hand an agent a browser and hope.
Contract work is where AI agents meet their hardest test: multi-step processes, business rules that cannot bend, and a record that has to be right when an examiner asks. That is why OpenAI's new study with Ironclad deserves a careful read from compliance and operations leaders — not because the numbers are dazzling, but because they are unusually honest.
On October 6, OpenAI published Advancing computer use with Ironclad, the first results from a research partnership with Ironclad, a leader in AI contracting. The study tested how well frontier models operate real contract lifecycle management (CLM) software, end to end, against detailed rubrics. This post covers what was tested, what the results actually say, and how regulated firms should read them before scoping an agent pilot.
What did OpenAI and Ironclad actually test?
Ironclad employees and people who use Ironclad at OpenAI helped researchers identify 11 tasks across legal, commercial, and procurement work — setting up nondisclosure agreements, creating procurement approval processes, and updating a reusable legal clause so it reflects the jurisdiction a requester selects. OpenAI estimates each task would take an experienced user about 30 to 40 minutes.
These are not chat prompts. OpenAI's own example is a legal operations team configuring a software-purchasing process: Finance approves purchases above a threshold, Security reviews certain requests, Legal reviews nonstandard terms — and the person setting it up has to turn that short list into an intake form, document templates, approval rules, and a record of the final agreement. An agent doing the same work must keep every requirement in view as it moves through the software. As OpenAI puts it, getting individual steps right is not enough: the finished process must work across the situations it was designed to handle.
Each task was scored against 8 to 50 criteria, depending on complexity, so researchers could see exactly which requirements a model met and where it fell short. The models practiced in hosted Ironclad environments and were trained with reinforcement learning on synthetic tasks built from contracts publicly available in the SEC's EDGAR database, filtered to remove personal information. OpenAI states that no OpenAI customer data, no OpenAI internal contracts, and no nonpublic Ironclad customer data were used for training or evaluation.
What did the study find?
OpenAI compared GPT-6 Astra and GPT-5.6 Sol at the settings where each scored highest — Max reasoning for Astra, High reasoning for Sol:
| Metric | GPT-5.6 Sol (High reasoning) | GPT-6 Astra (Max reasoning) |
|---|---|---|
| Mean rubric score across 11 tasks | 41.6% | 55.0% |
| Estimated average time per attempt | 37.0 minutes | 19.2 minutes |
That is a 32% higher average score at 48% lower estimated time per attempt — a genuine one-generation jump, and OpenAI says an internal development model already reaches 63.7% on the same tasks. On one demonstration task, Astra met about 94% of the criteria in an estimated 20 minutes; Sol met about 85% in an estimated 32. Two honest caveats from OpenAI's footnotes: the results cover these 11 research tasks, not all Ironclad workflows, and the times are simulated estimates based on assumed processing and generation speeds — not measured customer time savings.
Why does a 55% average score change the math for regulated workflows?
Read the headline carefully. GPT-6 Astra did not complete 55% of the tasks; it averaged 55.0% of the available rubric criteria. On the average task, roughly 45 of every 100 graded requirements were missed — a skipped approval threshold, a clause updated for the wrong jurisdiction, a record field left wrong. The rubric is what makes this visible: each missed criterion is a specific, named failure.
In an unregulated workflow, a missed criterion might be a typo someone fixes in review. In a regulated one, it is an exception with consequences: an approval that never happened, nonstandard terms that reached a counterparty unreviewed, a record that will not survive an examination. A half-right agent is not worth half as much as a right one — it can be worth less than zero, because someone still has to review every output, now hunting for failures they did not make themselves. Until average scores approach the high nineties, human review of every agent output is not a phase; it is the operating model.
Where should the controls live?
Not in the model, and not in the prompt. The durable controls live in the system the agent works inside, where they apply identically to every model version, every vendor, and every future jump in benchmark scores:
| Control | What it does | What it catches |
|---|---|---|
| Validation rules | Reject records that violate business rules (thresholds, required fields, allowed values) | A finance approval skipped above the spending limit |
| Approval steps | Route work to named humans before it can proceed; the agent cannot bypass the queue | Nonstandard terms reaching execution without Legal review |
| Locked fields and templates | Compliance-approved language and locked clauses the agent can use but cannot edit | A jurisdiction clause drifted from approved wording |
| Audit trail | Immutable record of who or what changed which field, when, and why | The examiner question: show me exactly what the agent did |
| Scoped permissions | The agent sees and touches only the objects its role requires | Curiosity-driven access to unrelated client data |
Notice what these controls share: none of them depends on the agent behaving well. That is the design goal. Model scores will keep improving — this study shows a 13.4-point jump in one generation — and controls that live in the system get more valuable, not less, as agents get faster.
Why scope agents inside a governed CRM instead of handing them a browser?
The OpenAI–Ironclad setup is instructive here: the models worked inside a hosted instance of a governed platform, with defined tasks and rubric-checkable outcomes — not loose on the open web. The same principle applies inside your firm. An agent handed a browser has maximum surface area and minimum constraint. An agent scoped inside a governed CRM like Salesforce or HubSpot works within objects, fields, flows, permissions, and logging that already encode your business rules:
| Dimension | Agent with a browser | Agent scoped in a governed CRM |
|---|---|---|
| Reach | Anything it can navigate to | Only the objects and actions its role permits |
| Guardrails | Instructions in a prompt | Validation rules, flows, and approvals it cannot route around |
| Evidence | Screen recordings and self-reports | Field history and audit logs an examiner can verify |
| Failure mode | Silent, novel, unbounded | Rejected at the rule, queued for human review |
What should compliance and operations leaders do before a 2027 pilot?
- Pick rubric-scorable work. Choose pilot tasks where you can write down the 8–50 criteria that define "correct" — if you cannot score it, you cannot supervise it.
- Baseline the human process. OpenAI's benchmark was 30–40 minutes of experienced-user time per task; know your own baseline before claiming agent value.
- Build the system controls first. Validation rules, approval routing, locked templates, and audit trails should exist before the agent arrives, not after its first exception.
- Plan for 100% review initially. Staff an exception queue and track which criteria agents miss; that list becomes your control roadmap.
- Write the data-use answer now. OpenAI could state plainly that no customer data was used. Your vendors should be able to say the same, in contract language.
- Let model gains lower the exception rate, not the control bar. Re-evaluate quarterly, but retire a control only when your own rubric scores justify it.
How Vantage Point Helps
Vantage Point is an OpenAI Partner Network member and an implementation partner for Salesforce and HubSpot, so we sit on both sides of this question: what agents can do, and what must be true in your systems before you let them. We design the governed workflows agents operate inside — validation rules, approval routing, compliance-locked templates, and audit trails in Salesforce and HubSpot — through our compliance and security solutions practice. One digital asset investment firm we work with runs SEC/FINRA-governed sequence templates with compliance-locked language, so every automated touch stays inside approved wording by construction. Senior consultants only — no junior handoffs; the experts you meet are the experts who deliver.
Scoping a 2027 Agent Pilot?
The firms that get agent pilots right start with the controls, then choose the model. Vantage Point's senior consultants can help you design the governed workflows — validation rules, approvals, locked language, audit trail — that make any agent safer to run. Talk to our team about an agent-readiness assessment for your CRM.
Frequently Asked Questions
What is OpenAI's Ironclad computer-use study?
A research collaboration, published October 6, 2026, in which OpenAI trained and evaluated its models on 11 realistic contracting tasks inside hosted Ironclad environments. The tasks — spanning legal, commercial, and procurement work such as NDA setup and procurement approvals — were identified with Ironclad employees and Ironclad users at OpenAI, estimated at 30 to 40 minutes each for an experienced user, and scored against 8 to 50 criteria per task.
Is the 55% figure a task completion rate?
No. The 55.0% is GPT-6 Astra's mean rubric score: the average share of available criteria it met across the 11 tasks. OpenAI did not report a pass/fail completion rate. The precise reading is that the best model satisfied, on average, about 55 of every 100 graded requirements — and missed the other 45.
How much better is GPT-6 Astra than GPT-5.6 Sol on these tasks?
On OpenAI's research evaluation, GPT-6 Astra (Max reasoning) averaged a 55.0% rubric score at an estimated 19.2 minutes per attempt, versus 41.6% and 37.0 minutes for GPT-5.6 Sol (High reasoning) — a 32% higher average score at 48% lower estimated time. OpenAI adds that an internal development model already reaches 63.7%, so further gains are expected in future models.
Does this mean AI agents can run contracting workflows unsupervised?
No. OpenAI's own post says the work underscores why human oversight still matters and why a full contracting platform remains essential. Even the best published result leaves about 45% of rubric criteria unmet on the average task, and in regulated workflows each missed criterion is an exception a person must catch.
What controls should regulated firms require before an agent pilot?
Controls that live in the system rather than the prompt: validation rules that reject non-compliant records, approval steps the agent cannot bypass, locked fields and templates with compliance-approved language, scoped permissions that limit what the agent can touch, and an immutable audit trail of every change. These controls apply identically to every model version and vendor.
Should we wait for the next model before piloting agents?
No — the control question does not change with the model. This study showed a 13.4-point score jump in one generation, and better models will shrink the exception rate, but the same validation rules, approvals, locked language, and audit trails will still be what make agent work defensible to examiners. Pilot now inside governed systems and let model gains reduce exception volume over time.
Did OpenAI train on customer contracts for this study?
OpenAI states it did not. The synthetic training tasks were built from contracts publicly available in the SEC's EDGAR database after filters designed to remove personal information, and OpenAI says no OpenAI customer data, no OpenAI internal contracts, and no nonpublic Ironclad customer data or contracts were used for training or evaluation.
Sources
- Advancing computer use with Ironclad — OpenAI research post, October 6, 2026
- GPT-6 Astra: A new generation of intelligence — OpenAI
- Ironclad — AI contracting platform
Vantage Point is a boutique CRM consulting firm helping businesses transform with Salesforce, HubSpot, and AI. Learn more at vantagepoint.io.
