July 23, 2026 · Tim Fraser
The model changed nothing. The harness changed everything.
Fifty-six API calls to list EC2 instances and S3 buckets. That was our agent, on a question a junior engineer answers with two.
I build plainfra, an assistant that answers plain-English questions about AWS environments. Like most teams building on LLMs, we started with the obvious architecture, which is to say a thin wrapper around a model: a system prompt, a generic "call any AWS read API" tool, and the model left to chain calls until it's satisfied. And whenever an answer came back mediocre, we asked the question every team asks: which model should we upgrade to?
Wrong question. Here's the experiment that showed it.
Seven questions covering inventory, cost, security and IAM, blind-graded by an independent strong model, with one question carrying embedded ground truth as a hallucination trap. Three configurations.
Our production wrapper with our production model scored 3.29 out of 5.
The same wrapper with a model roughly 4x cheaper per run scored 3.29.
Then we put that same cheap model inside a real harness: 4.43. Tool calls across the seven questions dropped from 193 to 27. Each run cost a twelfth of production. Better on every axis we measured, and the model never changed.
Stated plainly, because it's the whole article: in a controlled benchmark, changing the model changed almost nothing, and changing the harness changed almost everything.
Seven questions is a small bench, run once per configuration, so treat the numbers as an indication. Every secondary metric moved the same direction.
Same seven questions, same grader. Quality and tool correctness were blind-graded 1 to 5 by an independent model on live AWS accounts.
- Thin wrapper, production model (generic API tool, model plans every call): quality 3.29, tool correctness 3.57, 94 tool calls, median answer time 38.7s, $0.170 per run.
- Thin wrapper, budget model (same wrapper, roughly 4x cheaper per run): quality 3.29, tool correctness 3.14, 193 tool calls, median answer time 26.9s, $0.043 per run.
- Real harness, same budget model (curated tools, router, verification): quality 4.43, tool correctness 4.71, 27 tool calls, median answer time 16.5s, $0.014 per run.
The last two configurations share the identical model.
An agent is a model plus everything around it: the tools, the context, the routing, the verification. That everything-around-it is the harness, and our benchmark says it's where nearly all the leverage lives. Which means the industry's default reflex, blame the model and wait for the next one, has most of us optimising the wrong variable. The same model went from mediocre to excellent and we never touched a weight.
Three changes did most of the work.
The tools stopped being generic. A model holding one all-purpose API tool has to plan: which service, which call, which region, what next. Planning is where cheap models drown; that's the 56-call question. We replaced the escape hatch with tools that each answer a whole question in one shot: get_inventory, check_iam_posture, check_encryption. And this produced the most underrated finding of the whole exercise: with curated tools, the budget model scored at least as well as a premium model in our follow-up bench. When the harness does the planning, models become interchangeable, and you get to pick on price. Our roughly 4x cheaper option won the default slot on merit at the time; the default has since changed.
Routing got simpler, not smarter. We assumed complex questions needed a heavier agent, so we trialled a planning framework with todo lists and subagents. On our workload it was pure tax: planning fired unpredictably, added calls and seconds, improved nothing. Most of our questions resolve in one to six tool calls; that machinery is built for hours-long tasks. What shipped instead is a deterministic router, no model call at all, that assigns each question a lane: a small tool budget for lookups, a bigger budget plus a "cover every part of the question" nudge for diagnostics. Escalation means more budget and a verification pass, not a different agent.
Discovery moved out of the conversation. Most "what do I have" questions don't need live API calls; they need a fresh map. A scheduled scan, deterministic code with no LLM in it, condenses each environment into a compact snapshot that rides along in the prompt. It costs pennies, because most AWS read APIs are free (Cost Explorer, at about a cent a request, is the notable exception) and nothing bills by the token. Structural questions now take zero tool calls, and when something needs live confirmation, the snapshot says exactly where to look: one targeted call instead of an expedition.
Two scars worth passing on.
First: your benchmark is often measuring your integration bugs, not the model. A budget model scored 2.28 in an early bench and looked unusable. It wasn't. Our wrapper couldn't parse that model's tool-argument format, so 68 of its 114 tool calls silently returned nothing, and the model truthfully reported it couldn't get the data. We hit the same bug family three times in one week: our handler dropping a reasoning model's thinking instead of passing it back on the next request, a days-old framework failing the identical way, and that poisoned benchmark. Every one looked like "this model is bad." All three were "our harness is bad." Audit the plumbing before you audit the model.
Second: a model without a verification policy will invent numbers with total confidence. Our trap question asked about brand-new VPC flow logs where the true answer is "zero bytes, because this VPC has no network interfaces." The harness with verify-before-answer rules (every figure must trace to a tool result; an empty result means investigate why; failures get admitted) found the root cause and projected roughly $0. Configurations without those rules produced confident, fabricated dollar figures. The same discipline extends to the edges: when the tool budget runs out, synthesise a best-effort answer from what was gathered rather than dead-ending, and never stream a raw provider error to a customer. In an ops product, "I couldn't reach your account" beats a plausible guess every time.
None of this was intuition. Intuition kept saying "better model." It took deterministic evals that assert which tools were called, blind grading, a trap question, and one variable per run to see where the leverage actually was.
In our benchmarks, a competent model in a real harness beat a frontier model in a thin wrapper. So the next time an agent disappoints you, don't ask which model to upgrade to. Ask whether what you've built around the one you have is a harness, or just a wrapper.
PS: back in February I argued that AI isn't $40-a-month SaaS, it's infrastructure, and the economics only work if frontier models are reserved for the work that actually needs them. This bench turned out to be the harness-level evidence for that argument.