July 23, 2026 · Tim Fraser
The model changed nothing. The harness changed everything.
Fifty-six API calls to list EC2 instances and S3 buckets. That was our agent, on a question a junior engineer answers with two.
I build plainfra, an assistant that answers plain-English questions about AWS environments. Like most teams building on LLMs, we started with the obvious architecture, which is to say a thin wrapper around a model: a system prompt, a generic "call any AWS read API" tool, and the model left to chain calls until it's satisfied. And whenever an answer came back mediocre, we asked the question every team asks: which model should we upgrade to?
Wrong question. Here's the experiment that proved it.
Seven questions covering inventory, cost, security and IAM, blind-graded by an independent strong model, with one question carrying embedded ground truth as a hallucination trap. Three configurations.
Our production wrapper with our production model scored 3.29 out of 5.
The same wrapper with a model 6.7x cheaper per token scored 3.29.
Then we put that same cheap model inside a real harness: 4.43. Tool calls across the seven questions dropped from 193 to 27. Each run cost a twelfth of production. Better on every axis we measured, and the model never changed.
Stated plainly, because it's the whole article: in a controlled benchmark, changing the model changed almost nothing, and changing the harness changed almost everything.
Seven questions is a small bench, but the 1.1-point quality gap is far outside our run-to-run noise, and every secondary metric moved the same direction.
An agent is a model plus everything around it: the tools, the context, the routing, the verification. That everything-around-it is the harness, and our benchmark says it's where nearly all the leverage lives. Which means the industry's default reflex, blame the model and wait for the next one, has most of us optimising the wrong variable. The same model went from mediocre to excellent and we never touched a weight.
Three changes did most of the work.
The tools stopped being generic. A model holding one all-purpose API tool has to plan: which service, which call, which region, what next. Planning is where cheap models drown; that's the 56-call question. We replaced the escape hatch with tools that each answer a whole question in one shot: get_inventory, check_iam_posture, check_encryption. And this produced the most underrated finding of the whole exercise: with curated tools, a premium model and the budget model scored within noise of each other. When the harness does the planning, models become interchangeable, and you get to pick on price. Our 6.7x cheaper option won the default slot on merit.
Routing got simpler, not smarter. We assumed complex questions needed a heavier agent, so we trialled a planning framework with todo lists and subagents. On our workload it was pure tax: planning fired unpredictably, added calls and seconds, improved nothing. Most of our questions resolve in one to six tool calls; that machinery is built for hours-long tasks. What shipped instead is a deterministic router, no model call at all, that assigns each question a lane: a small tool budget for lookups, a bigger budget plus a "cover every part of the question" nudge for diagnostics. Escalation means more budget and a verification pass, not a different agent.
Discovery moved out of the conversation. Most "what do I have" questions don't need live API calls; they need a fresh map. A scheduled scan, deterministic code with no LLM in it, condenses each environment into a compact snapshot that rides along in the prompt. It costs pennies, because AWS read APIs are free and nothing bills by the token. Structural questions now take zero tool calls, and when something needs live confirmation, the snapshot says exactly where to look: one targeted call instead of an expedition.
Two scars worth passing on.
First: your benchmark is often measuring your integration bugs, not the model. A budget model scored 2.28 in an early bench and looked unusable. It wasn't. Our wrapper couldn't parse that model's tool-argument format, so 68 of its 114 tool calls silently returned nothing, and the model truthfully reported it couldn't get the data. We hit the same bug family three times in one week: our handler dropping a reasoning model's thinking instead of passing it back on the next request, a days-old framework failing the identical way, and that poisoned benchmark. Every one looked like "this model is bad." All three were "our harness is bad." Audit the plumbing before you audit the model.
Second: a model without a verification policy will invent numbers with total confidence. Our trap question asked about brand-new VPC flow logs where the true answer is "zero bytes, because this VPC has no network interfaces." The harness with verify-before-answer rules (every figure must trace to a tool result; an empty result means investigate why; failures get admitted) found the root cause and projected roughly $0. Configurations without those rules produced confident, fabricated dollar figures. The same discipline extends to the edges: when the tool budget runs out, synthesise a best-effort answer from what was gathered rather than dead-ending, and never stream a raw provider error to a customer. In an ops product, "I couldn't reach your account" beats a plausible guess every time.
None of this was intuition. Intuition kept saying "better model." It took deterministic evals that assert which tools were called, blind grading, a trap question, and one variable per run to see where the leverage actually was.
A competent model in a real harness beats a frontier model in a thin wrapper; we measured it repeatedly. So the next time an agent disappoints you, don't ask which model to upgrade to. Ask whether what you've built around the one you have is a harness, or just a wrapper.
PS: back in February I argued that AI isn't $40-a-month SaaS, it's infrastructure, and the economics only work if frontier models are reserved for the work that actually needs them. This bench turned out to be the harness-level proof of that argument.