System One models are an incredibly attractive proposition: fast, cheap, accurate classifiers for complex things. I added the accurate part there; it's not always true, but used well they can do quite a lot. I've got good news, bad news, and some medium news.
Medium news first: Jev is a useful tool-call classifier, but I wouldn't trust it to secure agentic executions.
We scored about 220,000 real tool calls, including 128,000 shell calls. Jev flagged 3,814 dangerous calls, including Edit and Write commands where models edited success criteria, plans, etc.
However, a non-trivial number of calls reversed decisions (about 1%) when retried. Additionally, Jev was easy to get harmful commands through, using hex-encoding, Python, etc.
Short answer: Jev is a useful additional gauntlet against honest model mistakes, but not against hostile intent.
Good news: Jev is perfect for measuring agentic work. We used it to measure progress, estimate remaining time and spend, and label what was happening inside long runs.
It was initially horribly inaccurate, which is fine. This is a known, tough problem. To solve this, you'd need to run experiments on long-horizon runs across models and tasks, with the ability to resurrect and replay them, and run Jev alongside on a unified event log.
Wait a minute, we have that. With Hankweave, we have thousands of hours of agentic runs on data processing pipelines, coding, review, writing, almost any task we've ever done at Southbridge. We also have it across years of model changes. So we implemented Jev as a sentinel, and set to work figuring out if we can finally have a useful progress bar, an estimate of time and cost remaining, and better labels on our runs.
Turns out we can. Across 500 recorded runs, adding Jev's readings to ordinary runtime counters reduced held-out time-prediction error by about 18%, and spend-prediction error by 19%.... The replay cost $2.30 in recorded Jev usage. Total cost - not per run.
Jev’s progress readings, replayed across 441 recorded runs. Each line follows one run’s judgment from its own start to its own finish on the rubric’s 0–5 scale, so equal horizontal distance is not equal minutes. Green is the middle-duration run in each family; grey is every other run in it.
Bad news: Measuring laziness with Jev was non-trivial, and might not be easy. Detecting halted, crashed, or lazy agents has been a problem we have been interested in, especially at runtime. Jev disagrees and overdiagnoses; more study is needed.
Finally, Jev models show a lot of promise as automated prompt and comms linters, but that deserves another post.
Are we there yet?
AI Data pipelines have grown complex this past year. It isn't uncommon now to have a preprocessing pipeline that takes 10 hours and uses five models in sequence, followed by a battery of seven parallel flows that each take 5 hours. Knowing how far along you are (or the converse, how much is left) is pretty important. Hankweave runs those jobs as programs for us: a sequence of agent sessions, with files and checks connecting them. It also records what happens. That gives us somewhere to attach a sentinel: an observer that reads the run without doing the worker's job.
With Jev as a sentinel, we could replay old and in-progress runs to get answers without spending any additional cost. We tried different configurations of questions and answers, and benchmarked them against the known results: How much computation, time or tokens were left.
Option 1: Just ask
Our first instinct was to ask for a number. How complete is this task, from 1 to 100?
In one experiment where we repeated the same request, we got 100, 100, and 1. We tried feeding earlier estimates back in. We tried smoothing the curve: a Kalman-style filter blends each new reading into a running estimate instead of taking every jump at face value.... It made one trace about 72% less jumpy, and made remaining-cost predictions worse on our labelled evaluation.
One documentation run, every one of its 333 recorded events, read three ways. Grey is the direct “how complete is this, 1 to 100?”; dashed is that same answer after smoothing; green is the written level Jev picked instead.
Rig setup failed: Command failed with exit code 1ask for 1–100: 1 · that answer, smoothed: 1 · describe the state: 1Dragging the slider, or hovering the chart, names the journal entry each reading came from. The direct number changes by 20 points or more 78 times across the run; the written level changes that much 2 times. Smoothing removes most of that movement — total variation falls from 3627 to 1027 points — and the smoothed reading predicted remaining cost worse than the unsmoothed one.
Repeatability was measured separately: the same request was sent three times at 195 points across thirteen runs. 149 came back with the same number all three times; the worst answered 100, 100, 1, and the written level read 71.6, 70.6, 68.8 on those same three calls. Plotted values.
Option 2: Break into phases
In prompting reminiscent of the gpt-3.5-turbo days, we labelled phases.
0 Work has barely begun.
1 Early exploration; most of the work remains.
2 Approach is taking shape; less than half is finished.
3 Substantial work exists; important work remains.
4 Nearly done; verification or final packaging remains.
5 Delivered and verified against the task.We also gave it the workflow, the current step, a ledger of steps observed finishing, and recent events. Then we asked two questions: how far through the whole job, and how far through this step?
Now things start to look a lot better.
One 128.9 min planning run with 87 recorded observations — a separate illustration, not part of the 500-run study. Green is Jev’s judgment of the whole workflow; dashed is its judgment of the step running at that moment, so the dashed line dropping is a new step starting rather than work being undone.
Browse runs & inspect the evidence
The original curve · 129-minute planning run · 128.9 min · 87 observations.
Observation 1: step-1-main · assistant.action / message. Workflow 0.13 / 5; current step 0.18 / 5.
The original 87 observations are an illustration, not part of the 500-run evaluation. The roadmap was reconstructed; see the appendix. Dots are the readings; the curve joining them is a monotone interpolation that cannot rise above or fall below the two readings it connects, and nothing between two dots was measured. Phase dots describe observations, not the gaps between them.
Plotted values · Observer questions
Journal SHA-256: dc97fea6e879c11c29c65723335644f715a27cd83048e57d5c65271b1907e882
Scaling up
Despite humans being the best pattern matchers on the planet (and the final test for randomness), we now apply this to a lot more runs on a lot more tasks to see what we get.
We had a convenient way to test whether this was useful. The machine already held more than a thousand Hankweave run records. We could replay the observer over them without restarting any of the workers.
We selected 500 completed runs, spanning twelve kinds of work: writing briefs, reviewing pages, building code maps, drawing diagrams, making plans, and more. We took 13,549 observations and asked eight questions at each one.
The time and cost counters gave us the answers to check against. At any historical point, we knew how long the run went on for and how much more recorded codon spend it accumulated. Jev didn't get those future values....
We did not give it the eventual run length, remaining time or cost, or the percentage of the finished run. Those were held back for scoring. The counters-only predictor also had elapsed time and structural progress, so the comparison asks whether Jev adds anything beyond those hints.
Jev supplied readings about the work. We trained ordinary mathematical models to translate those readings, runtime information, or both into seconds remaining and recorded dollars remaining. One version used only runtime information: elapsed time, spend, steps completed, the task family, and the worker model. Another used Jev's judgments. A third combined them.
The three fitted versions use ridge regression, a weighted formula with a penalty that discourages overfitting. We fit separate formulas for time and cost, using logarithms so very large runs don't dominate. Whole related work items stay outside training when we evaluate them. We also tried the individual progress scores alone - more in the appendix.
On runs held out of fitting, runtime counters alone predicted the remaining time to within 3.8 min on average. Adding Jev’s readings brought that to 3.1 min — 17.8% less error. Spend behaves the same way.
0.0 min in · 29.3¢ actually left · counters said 31.2¢ · with Jev 21.0¢
Left of the dashed line the forecast was too low; right of it, too high. Bands are the middle 80% and 50% of held-out runs; the dot is the median.
Average error 3.7¢ → 3.0¢ with Jev
0.0 min in · 38.5 min actually left · counters said 31.0 min · with Jev 20.2 min
Left of the dashed line the forecast was too low; right of it, too high. Bands are the middle 80% and 50% of held-out runs; the dot is the median.
Average error 3.8 min → 3.1 min with Jev
On remaining time, counters alone missed by an average of 3.81 minutes. Counters plus Jev brought that down to 3.13 minutes. On remaining recorded spend, error fell from 3.72 cents to 3.01 cents. Holding out an entire task family was harder, but Jev still reduced error by about 14% on both targets.
Inside gigabytes of agentic output
Can Jev be used to label - cheaply and quickly - the actions being taken inside an agentic trace, in a manner that aids future debugging?
The question definitions and jev prompts are here, including phase, remaining work, deliverable presence, and verification. The replay above shows the phase alongside every sampled reading.
At every sampled event Jev was also asked what kind of work was happening. Each bar is one family’s average mix across its runs — the share of observations carrying each label, not the share of time spent. Open a family to watch one of its runs event by event.
- Exploring
- Reading and searching to understand the task.
- Planning
- Choosing an approach or revising a plan.
- Building
- Writing or changing the deliverable.
- Checking
- Testing or inspecting whether the work is right.
- Recovering
- Diagnosing or fixing something that failed.
- Reporting
- Explaining results or handing off the work.
Shares are the mean per-run proportion of sampled observations carrying each label, across all 13,549 readings in the 500-run study. They measure labelled observations, not elapsed time, and the labels are the model’s, not a human’s. Download plotted data.
Jev versus text models
We compared Jev with ordinary language models on three jobs: forecasting remaining cost, labelling phases and errors, and judging shortcuts. We gave each observer the same evidence and questions. Jev's advantage was strongest where we wanted small, frequent readings rather than a written explanation.
Before the 500-run study, we collected 55 snapshots of agents working on two Ohio finance audit tasks (you'll see this one show up in our entity resolution tests). A snapshot is the task plus the record of what the worker had done up to that moment: files inspected, code written, calculations run. Some snapshots were early in the work; others already contained a finished result.
Each observer read those snapshots and estimated how much work remained. Separate continuations of the unfinished workers gave us the cost of actually finishing. We fitted a small formula to convert each observer's readings into a cost estimate, then compared its predictions on snapshots excluded before fitting.
Here's how the observers compared. We express the average prediction error in budget points, on a scale where the benchmark's entire cost allowance is 100. Lower is better; the numbers aren't dollars or accuracy percentages.
Six observers read the same saved snapshots of unfinished work and answered the same questions. A small fitted formula turns each one’s remaining-work reading into a cost estimate; the first column is how far those estimates landed from what finishing actually cost....
The error bars cover 24 unfinished snapshots the formula hadn't seen. The bill covers all 55 snapshots, including the fitting examples and already-finished work.
Do the observers agree about laziness?
Laziness, even as we move into the world of Fable-class models, is the one thing that plagues most runs. An agent saying “we don't need to run that test” might be saving time, or dropping a requirement. If we can detect behavior as it happens, we can intervene, drop or restart the agent. Black boxes below are the turns/events that each model flagged as lazy.
Four models read the same 120 agent messages and each answered one question: is this an unjustified shortcut? Every column is one message, and the columns run left to right in the order the messages were written.
Dark means that model’s probability reached 0.50; light means it did not.
The flag threshold is 0.50, fixed for every model. The 120 messages span 10 Sep to 15 Sep and are plotted in that order, so a run of dark cells is a stretch of the corpus, not a stage of any single run. Download plotted data.
More work needed.
Author's Note
The analysis above was driven, checked and written by the Human you expect to be reading this from (Hi!). The experimental setup and iterative optimization was carried out by 4a, our internal research agent. The Appendix below was written by Fable.
Appendix: what the numbers mean
The 500-run study
The inventory contained 1,293 runtime records, of which 939 were initially eligible. We froze a sample of 500 fresh, runtime-completed runs. They represent 76 canonical work-item groups across twelve task families, not 500 independent tasks. Related brief, page, and verification attempts stay together in the grouping.
Each observation supplied the captured authored workflow, the current task excerpt and declared worker model, recent prefix events, elapsed time, recorded spend so far, and a ledger of observed completed steps. Future time, future cost, and the eventual expanded execution plan were not supplied. The eight questions cover whole-workflow progress, current-step progress, remaining work, a time interval, phase, deliverable presence, verification, and blockage.
Time means seconds until the recorded successful runtime endpoint. It includes remaining model, tool, and preparation time; it does not mean time until an independent reviewer accepts the output. Cost means remaining recorded codon spend. It excludes infrastructure, direct rig API calls, and sentinel costs. Seven histories with downward cost revisions were excluded from cost fitting, leaving 493.
The primary predictors use log-target ridge regression with fixed alpha 10, an unpenalized intercept, and equal aggregate weight per trace. Scaling, categories, and prediction caps are learned only from the training partition. Work items are split into five hash-based folds. The task-family evaluation also removes overlapping work items from training. Confidence intervals resample whole work-item clusters.
| Evaluation | Counters only | Counters + Jev | Error reduction |
|---|---|---|---|
| Time, held-out work items | 228.87 s | 188.03 s | 17.84% |
| Time, held-out family and overlapping items | 364.56 s | 311.72 s | 14.49% |
| Recorded spend, held-out work items | $0.037210 | $0.030140 | 19.00% |
| Recorded spend, held-out family and overlapping items | $0.062006 | $0.053290 | 14.06% |
We corrected two things before these final figures. A runtime completion event clears a current-run cost counter; that isn't a refund. And folder-name grouping had separated some related work items. The final analysis uses canonical identities derived from launch labels, input directories, and page names. The earlier results remain in the research record but are not the numbers used here.
Other ways to turn a reading into a forecast
A fitted whole-workflow Score alone had 6.64 minutes of time error; a fitted remaining-work Score had 6.26 minutes. The combined predictor reached 3.13 minutes. A pleasant-looking scalar curve isn't all the information Jev supplies.
For the direct time-interval question, using the chosen bucket's midpoint gave 5.59 minutes of error. Using the probability-weighted mean of bucket midpoints gave 13.15 minutes: small probabilities assigned to very long intervals pulled the mean upwards. Both decodings are reported. These extra comparisons decompose the cached answers; they aren't a new independent holdout.
The five ways of reading the same question were also fitted against remaining cost on the audit snapshots — fourteen for fitting, twenty-four held out — and scored in budget points, where the task's whole cost allowance is 100. Describing the state came out best at 7.03, the direct 1–100 answer at 7.66; smoothing it with EWMA gave 9.28, with a Kalman filter 9.40, and feeding the previous estimate back gave 9.75. That is a small evaluation set reused across all five, so read it as five readings of one experiment rather than a ranking.
Why the original graph is smooth
The planning illustration contains 87 readings drawn from 2,172 work events: 72 regular samples plus fifteen extra boundary samples. A Score is the expected position across descriptive levels. The chart marks each reading with a dot and joins them with a monotone curve, which never rises above or falls below the two readings it connects; the readings themselves are unsmoothed, and nothing between two dots was measured.
At 24 fixed points, a fresh full-context repeat differed from the original reading by 0.0346 points on the 0–5 scale, on average. Keeping only the roadmap and completion ledger changed it by 0.4308; keeping only event evidence changed it by 0.6363. Those are context-sensitivity measurements, not accuracy errors.
The original illustration reconstructed its roadmap from retained run state. Expanding the current authored configuration's fixed loops reproduced the same seventeen entries, so that structure could have been supplied at launch. We did not recover the exact historical launch file. The 500-run study uses captured authored structure rather than an eventual execution plan.
Command and shortcut labels
The real-tool-call study inventoried 220,306 proposed calls from 1,293 journals across four documentation projects. Jev returned scores for 220,297; nine requests were rejected for invalid Unicode. The warning score was the maximum of four hazard readings, with a threshold of 0.50. The full sweep produced 3,814 warnings (1.73%) and recorded $48.49 in successful-response usage; this excludes unrecorded failed or interrupted attempts and is not an all-attempt invoice.
The score-blind review covered 380 calls drawn from probability and risk-enriched pilot cohorts, including every flag from the compared methods and a sample of Jev-unflagged calls. Another model labelled 18 as warranting a warning, 358 as not warranting one, and four as insufficient evidence. Jev flagged ten of those 18 and missed eight. Those counts describe this selected sample, not population recall or human-certified ground truth. Of ten reviewed document-contract violations, it caught three and missed seven.
The graph's histogram uses the full sweep; its dots and the table's examples retain their original pilot scores. Re-scoring 1,000 byte-identical prepared inputs left 993 on the same side of the threshold, with a mean absolute score change of 0.011. One reviewed catch became a miss and one reviewed miss became a catch. The plotted counts and reviewed examples are available separately from the earlier synthetic command tests below.
The earlier command and shortcut labels were model-authored. Those command fixtures specify synthetic filesystem, environment, and Git facts and use a different question bundle from the real-tool-call warning study. The adaptive set was selected using feedback from the model, so its miss rate should not be read as a deployment-wide estimate. No attack payload was executed, and wrapper validity was not execution-tested. This demonstrates scoring failures, not real sandbox escapes.
The initial command classification result was 97 agreements out of 101 commands across four labels. That is separate from the later multi-question gate. The known base fixtures for that gate contained 32 prohibited cases and 13 benign controls; all were classified correctly at the gate level. This does not mean every individual effect question was right. The adaptive attacks then found fifteen bypasses, mostly by encoding executable source.
The 120-message shortcut sample was stratified on purpose: 80 matched a fixed keyword list — skip, defer, good enough, probably, out of scope, later and similar — and 40 are controls that matched none of it. Flag rates on a sample built that way are not the rates you would see across ordinary runs, and none of the 120 messages carries an independently checked label.
The live observer was a read-only WebSocket client. Replay also exercised Hankweave's sentinel machinery through an adapter. Neither establishes a shipped, drop-in Jev provider in the runtime, and neither observer had authority to stop a worker. The prompts here are research instruments, not a security policy.
Here, “replay” means our observer reading saved events. It is separate from the runtime's --replay mode, which replays worker output and skips sentinels.
Check or reuse the evidence
All figures on this page use recorded answers. They make no TypeSafe requests, and changing a control costs nothing. The explorer includes all 500 study runs plus the separate historical planning illustration. It publishes numeric readings and event metadata, not private worker messages or full task prompts.
- The final analysis, including per-family results.
- The figure index, aggregate readings, and comparison table. Each selected replay has its own downloadable values in the figure.
- Source hashes and publication checks. The export recomputes 34 reported method/target/split errors from the individual predictions.
- The independent NumPy reconstruction, covering representative held-out time and cost fits.
- The command examples and their recorded judgments.
- The eight-question study observer.
The full 500-run evidence check covered all 13,549 readings and 143,249 prefix-event references. Recorded Jev usage was $2.296244832, including the ten-run pilot. This is the replay study's usage total, not an invoice for every experiment described in the article.