October 10, 2026 · Tim Fraser

Can Jev shortlist an agent's tools? Thirteen investigation steps say maybe

An agent investigating a cloud estate sees about 25 tools at every step. Most of them have nothing to do with the step in front of it. I spent a day finding out whether a fast judgment model can tell which ones, before the answering model has to.

The short version: Jev hid two thirds of the tool list, and on 11 of the 12 steps that needed tools, the ones that mattered were still on it. It could not pick between specific calls, and its "ready to answer" signal is unproven. Thirteen steps, one run each, one judge. Every number here is an indication, not a measurement.

The Jev test

Jev is TypeSafe's System One model. It does not generate text. You hand it a state and a set of questions, and it returns a value between 0 and 1 for each one, in well under a second. The pitch is programmable common sense: a judgment your code can act on directly, without a prompt-and-parse step.

The question I wanted answered was narrow. At each step of an investigation, can Jev value the tools on offer well enough that a shorter list would still contain the right move?

The 13 steps are real: frozen prompts from real investigations, each with the evidence gathered so far. Each already had a "best move", the reply a blind judge scored highest when the same step was answered by several models (more on that below). On 12 steps the best move was a set of tool calls. On one it was a final answer. Those 12 best moves were the yardstick.

For each step, one Jev call received the investigation's question, the conversation so far, and the tool list with descriptions. It answered three kinds of question: how valuable is each tool right now (one per tool, about 25 per step), how valuable is each specific call the benchmark models had proposed, and is the investigation ready to answer now.

Then I swept a threshold. Drop every tool Jev valued below it, and count how many of the 12 best moves still had all their tools available.

What held up and what did not

At the tool level there is a signal. A cut at 0.3 or 0.4 removes two thirds of the list and keeps the right tools 11 times out of 12. The one miss is instructive: on that step Jev valued the general AWS call tool at 0.28, and that tool is the one the harness can never do without. The fix is structural, not statistical. Core tools get pinned and are never filtered, whatever Jev says. The threshold was also chosen from these same 13 steps, so it needs checking on steps it has not seen.

At the call level Jev is weak. Where the benchmark models had proposed specific calls, Jev's top-valued call matched the judge's best move on 7 of 12 steps. Across the 73 candidate replies that proposed calls, the correlation between Jev's value and the judge's next-step score was 0.165 (Pearson), which is close to no relationship. Jev can say which tools are in play. It cannot say which of two reasonable calls is the better one.

The "ready to answer" signal has one data point. The single step whose best move was a final answer scored 0.80, against a mean of about 0.24 on the steps that still needed tools. But one of those tool-needing steps scored 0.76. One positive and one false alarm says nothing about a threshold yet.

Cost, in the only units I have: 13 calls, about 214,000 input tokens in total, between a quarter and half a second per call. Jev's per-call price is not published and I did not measure one, so there is no dollar figure here.

Where the steps came from

plainfra lets an operations team ask plain-English questions about their cloud infrastructure and get an answer backed by evidence. A model makes read-only AWS API calls through plainfra's harness. Everything in this test ran on one family of frontier models, from a small fast one to the largest. The harness gates every call so only read operations go through, redacts secrets before any model or store sees the response, logs and meters every call, and makes the model answer in a fixed shape: the finding first, then tagged findings, what was ruled out, and the gaps it could not verify.

The questions are the ones a cloud lead gets every day. Why did a production service's behaviour regress last week. What changed in this account since the last report. The model's job is to go and look, not to guess.

The model is the smallest part of this. I measured that in July: the same cheap model in a real harness beat a frontier model in a thin wrapper. The harness is where the months went. Failover chains across models with a circuit breaker on each rung, so a throttled or silent model moves the turn along instead of ending it. A read-only gate over every operation AWS publishes, generated from the service models rather than a hand-kept list. Redaction before any model sees data. Spill and pointer for large results, so a large API response does not have to sit in the context window. A method guard that refuses a tool call when the model is plainly guessing an identifier it has already failed on. Investigations that checkpoint and offer to continue, rather than silently stopping when a budget or a clock runs out. An answer format with tagged findings and named gaps, because an answer that cannot say what it did not check is not an answer.

The newest piece is a relay. Every model call and every tool call in an investigation can be written out as a card, run by hand somewhere else, and the result pasted back. The engine carries on exactly as if the call had gone to the model gateway. That gave me something I had wanted for a long time: a library of frozen investigation steps, each one a real prompt with real evidence, that I can replay against any model or any new idea without rerunning the investigation.

A relay run produced the 13 steps. To get the best moves, I ran each step through the small model at four effort settings, a second small model and two mid-size models, and had the largest model judge the replies blind on three things: was the next step right, was it grounded in the evidence, and was the output in a shape the harness could consume. The highest-scoring reply per step is the best move the Jev test was marked against.

Where it could fit

The shape that survives this test is a per-step tool shortlist. Jev re-scores the tools at every step, so a tool that was irrelevant at step two can come back at step six when the evidence makes it relevant. Core tools are pinned. Anything under the threshold is hidden from the answering model for that step only.

What is not proven is the part that matters: whether a shorter list makes the answering model choose better, or cheaper, or faster. A tool list is a few thousand tokens of a much larger prompt, and a small model that already picks the right tool from 25 may gain nothing from 8. The next test reruns the small model at high effort, the strongest small-model setting in the benchmark, on the same 13 steps with the filtered list, and judges it against the full list with the same judge and the same criteria.

If the filtered run scores the same, Jev earns a place as a hint, not a gate. If it scores better, the next thing to look for is whatever else in the harness is a judgment pretending to be a prompt.