Just-in-Time Ontologies: What Cheap Intelligence Means for Data

This is the written version of my talk for the online track of AI Engineer New York 2026. You can watch it in full below. A few of the visuals from the video are inline, where they help.

Transformation was expensive

I really love this article from Hannes at DuckDB, where he asks if we lost a decade chasing distributed architectures that presumed compute and storage were slow and expensive....

Play

Capability, in every era, defines architecture and tooling, and the tooling lags behind, consistently. We first made compute cheaper, then storage, then we gave it to everybody, then we made it faster, and every time, architectures undergo this massive change..........

All of our data infrastructure today is built on the idea that transformation is expensive. What if that wasn't true anymore?

I believe we've solved the final bottleneck in transformation: general intelligence, or judgement. And we've made it accessible on tap, all along the Pareto frontier....

Which puts us right on the verge of a paradigm shift.

In this talk, we're going to cover why single ontologies always fail, the three roles models play in pipelines, intersecting model priors for better results, and Pareto surfing: how today's models can tune a workload for cost, speed and accuracy all at the same time.

Here's a quick example.

When Jev launched, we already had an entity resolution pipeline running: working out which of Ohio's campaign finance records belong to the same person, company or committee. It was projected to cost about $10,000. Our systems took the new model, worked out where it fit, tested over a dozen approaches, and integrated it.

Ten days after our first call to Jev, we had final ground truth for every donor case in Ohio, for under $800. That is the power of Pareto surfing, and just-in-time transformations. So let's get into it.

There is no superschema

Ontologies are the enterprise version of mind maps.

It's a tarpit that sucks us all in. Including me: I talked about a single universal schema when I founded Southbridge, two years ago.

The popular conception (and it's not wrong, it's just a small part of the picture) is that migrations and ontologies fail because of a lack of buy-in. That they're abandoned halfway through. That organizations and people never switch over.

And all of that is true. A hundred percent.

But buy-in isn't the primary reason. If you could migrate all of your information into a single superschema in the blink of an eye, you would still have to contend with the fact that there is no superschema.

There are actually four schemas.

The first one belongs to the source, wherever the information comes from. Let's take temperature as an example. Say I ask you to write down the temperature.

The source schema is whatever you have access to. Your phone, which gives you your region's temperature. A sensor on your desk, less accurate, but more local. Or your watch. Even your patience in following through on my request is part of the source schema.

The second one is the storage schema. There's only so much you can write down. You can't write down the firmware of the device you're using. You can't even write the time down to the second, because you couldn't finish writing inside that second. This is the schema imposed on you by storage.

The third one belongs to the decision. Why do I need you to write the temperature down?

Maybe I'm figuring out what makes you comfortable. Then I need temperatures from you, spread through the day, and I need to make sure it's you doing the measuring.

Maybe I'm building weather migration maps. Then I don't care who does the measuring, but I care a lot about your exact location, the time, the device you used, and whether it's changed.

Or maybe I'm deciding where to install sensitive equipment, and you yourself become a variable I need to cancel out, because you're not precise enough.

There's a fourth one, which we'll get to when we talk about priors. It's a recent addition.

The true problem is that you will never know all four at the same time, except at the moment the decision is being made.

In almost every enterprise, most of the data being recorded has little idea what it will be used for. Take the customer, the first box in every ontology. To billing, a customer is whoever pays. To support, it's whoever calls. To compliance, it's whoever we're on the hook for.

The only time you know is when the decision is known. Should we refund this order? Can we extend this account more credit? Suddenly, what you need to know, how, and to what degree, becomes extremely clear.

Play

This is why I was wrong. There is no universal schema. There is no concept of a customer that's divorced from the question you want to answer. Pulling these schemas together is the process of transformation, of making information actually useful: taking the context it was recorded in, and merging it with the context of the decision.

And that takes discernment. Doing that merge when the decision space is known, instead of ahead of time, is what I mean by a just-in-time ontology.

We'll come back to discernment with a story from my last job, where the only real option was to make sure certain numbers did not ever exist.

Who we are

But before that, a little bit about me and us. We're obsessives who embed intelligence into pipelines: pipelines that scale, that run over and over again to take work truly off our hands, and that mature in cost, performance and speed on their own.

Our data work breaks down into three regimes.

The first is write-once, run-once. If you program, think of these as monster-scale, immediately invoked functions. They're built for something single-use, like joining a dataset, mining information, or answering one specific question, and then they're torn down.

The second is write-many, run-once, where lots of smaller and larger workloads are built for one large common goal. The closest thing you might know is a large migration.

Years of records, moved out of systems nobody fully remembers: many workloads built for that one move, each checked against the others, because it only happens once.

The third, and the most common, is write-many, run-many. These are repeatable pipelines, built and maintained over time as we add more information in. At Southbridge, we've moved so much work off the main thread and into repeatable systems, from benchmark design and tracking, to making subtitles for movies we want to watch.

Almost everything we do in the modern world is a transformation of data, or it's bottlenecked by how hard it is to transform data.

In all of these cases, our job is simple: make sure people can rely on the results.

In two years we've gone from figuring out how to make early models reliable on healthcare datasets, to building hankweave, our open-source harness for data work, to getting models to work together at terabyte scale.

Today we have systems that can self-optimize. Give them the problem, a way to fail safely, and a place to work, and they find the shape of the solution, pick the right models, run the pilots, scale up, and Pareto-surf their way to the best option they can find.

When not to produce a number

At my last job, the question we heard most often was: can we make this decision data-led?

Sometimes that was the right question, when we had thousands of vessels on the platform. The problem is that we also asked it when we only had fifty users.

If you know statistics, you'll know that A/B testing fifty users is a bad idea. Not because you won't get a result. You will. It's just not enough data to reach a conclusion.

So the only real option was not to produce the number at all. Because the moment a number exists, it argues for a result.

Today it's worse. Interactive agents will always give you an answer, and they'll give it to you quickly. You ask for a transformation, "load this for me", and you get a confident: "yes, sir."

This means mistakes add up quickly. A moment in chat can be a prod nightmare, waiting to be discovered.

The primary reason this happens is that intelligence, as it relates to data work today, occupies three separate roles that coding agents mash into one chat window.

Let's pull them apart. It'll help a lot when we get to decision models, because they fit into exactly one of the three.

Builders, workers, hillclimbers

The first is building and designing the pipeline: breaking down the problem, writing the early specs, authorizing spend.

The second, and this is often confused with the first and given to the same models, is working inside the pipeline: actually doing the work. Reading the PDFs, writing to databases, checking results.

Before we get to the third one, I want to make it clear that these two roles pull in very different directions.

The first wants slower models, with heavier, thicker tokens. Models that can think more, walk around a problem and see the angles, digest documentation, understand how pipelines get built and predict how something will do before it even starts. Opus or up.

The models you want inside pipelines are the opposite. You want them cheaper, faster, specialized, good at following instructions: repeatable workers, with knowable defects.

We see the same in nature. A leafcutter ant colony runs on four sizes of worker, and each size does different work....

As you look at any system you design, review or hear about, look for what slots in where in this layer cake. It makes a big difference.

Because we had this distinction, bringing in Jev, and the whole class of decision models, was easy. They're workers: built for quick contact with information, deployed at the bottom end of large workloads.

Oh, right. The third category.

This is something we call the hillclimbers, or surfers: models that can take a first run, and self-optimize. Given the right knowledge, instructions, and an incredibly stable surface to stand on, they can try new things, vary budgets to limit cost exposure, debug quickly, take notes, and surf their way, not just to the right solution, but to the optimal one.

We call it Pareto surfing, and we'll cover it at the end. First, let's talk about our experience with Jev.

Jev: inside the arrow

Jev launched on a Tuesday, when we were deep into an entity resolution project: working out which donors in Ohio's campaign finance records are the same person, company, or committee.

At this point a lot of people have seen this chart. We talked about the results before, and adapted some of the ongoing work to the Ohio campaign finance dataset, so you can download it for yourself, prompts and all.

What we didn't talk about is everything behind that one arrow, and what happened when we ran the same idea on the whole state.

Here are all of the architectures we tried. We started with three designs on a small development set: a cheap reasoning model on its own; Jev first, with a reasoning model for anything it wasn't sure about; and a reasoning model proposing answers, with Jev checking them.

On two hundred held-out families, the cheap reasoning model on its own did as well as Fable, for about two percent of the cost. Jev-first got the core answers right on all two hundred, but it missed nine household links, because it had confidently said no to every one of them.

So we rewrote two of the questions, without touching the models or the thresholds. The household question now said what counts as enough evidence, and Jev went from "no" on all thirteen development questions, to "yes" on all thirteen.

As we scaled it statewide, we ran over a dozen rounds of qualification. Each round fixed a definition: what a committee is, what "filer" means, when two people at one address count as a household.

Every one of Ohio's 203,317 cases now has a label that says what its answer rests on, and how far it was checked. This is the hierarchy we discovered for this specific problem: rules in code, then Jev, DeepSeek Flash, Luna, Sol, and Fable at the top.

Here's what we learned about putting decision models into pipelines:

1. Say what counts as enough evidence, not only what fails. Naming it took households from 0/13 to 13/13 "yes".
2. Ask a second, differently worded question; review when either answer is unsure. A question about the evidence flags confident mistakes.
3. Batch the work. 4 cases per request: 190–192 of 200. One at a time: 134.
4. Thresholds are policy, not truth. Of 352 pairs, 0.7 kept 275 with none wrong; 0.9 kept 153.
5. Two different models agreeing beats one model twice. Two Jev runs agreed on 3 wrong; Jev + Luna on none.
6. Review breaks things too. A fallback reviewer overruled a right answer, twice.
7. Choose cases for review by doubt, not a fixed percentage. A 2.5% random sample caught 0 of 8 disagreements.
8. Your answer key is a pipeline too; audit it. One audit of our key moved 78 labels.
9. Compose in code, and keep every attempt. Code merges; no merge crosses a type or uncertainty fence.
10. Reserve money before every call; do the biggest cases first. One $5.85 call, counted twice, froze the ledger by design.

Here's where the money went. The whole statewide job came to $795, against an $800 budget that was set ahead of time.

Checking cost about four times as much as doing: $545 on second opinions, settling disagreements, and the audit, against $128 for first answers.

Before Jev, we expected this work to cost about $10,000. And now that the pipeline is built and verified, we expect the audit budget for the next state to be close to zero.

The most amazing thing is that our agent, 4a, running on Astra and then Opus, found most of these approaches incrementally, on its own: by building the system, running it, debugging it, and trying new things.

It ran for ten days and tried over a dozen architectures. Its hanks ran more than four thousand times, alongside ninety thousand calls to Jev, and the models inside those runs went through roughly six billion tokens.

Once it was done, we wrote what we learned back into our library of agentic patterns. This work leaned on existing ones, and we're adding new, tested ones, like the checking ladder.

This is the value of taking work off the main thread, for humans, and for agents. Because every one of these iterations was version-controlled as a hank, they were easy to revert, understand, debug and critique.

So, can models do this natively now? Not quite. They're missing four things, which we'll go over. And there's something that doesn't go away as models get bigger and more capable: the model's prior.

Priors: the fourth schema

Just like humans, models bring their own priors, their beliefs about what should be in the world, to every task. This is the fourth schema I mentioned at the start: the one the model brings with it.

Bigger models, in a strange way, are sometimes worse at this. They hold on to their beliefs and argue for them even more vehemently..........

Here are a few we ran into.

One pipeline looked for "in kind" spelled with an underscore. The file spelled it with a capital I and a hyphen. So the model counted 15,658 in-kind contributions as ordinary money, and didn't double-check. The loan repayments disappeared the same way.

Another tried four date formats, when the file used a fifth. 99.4 percent of the transaction dates came out empty, and every row count and dollar total still matched, so the agent moved on.

Several runs noticed that loan amounts were negative, decided the sign was redundant, and stored the absolute value. A net of $784,699 became $5.3 million of debt in the end, validated by multiple models.

This is quickly becoming the largest class of error I see across companies and data work, as models take over.

Sometimes you can solve it just by not vibing: by checking results, and applying human intuition. What you're doing when you do this is intersecting your priors with the model's.

In that first example, any human would catch the missing in-kind contributions, or think to check: it's how we found it. But humans are expensive. They're slow, they pause timelines, they're even more nondeterministic than models, and they hate interruptions.

It turns out you can also get really far with cross-model prior intersection. On our two hundred held-out families, Jev and Luna agreed on 194, and none of those shared answers was wrong. Every error sat in the six families where they disagreed.

Models from different families, from different pedigrees, tend to disagree more and have different priors than models in the same group. The exception (I'm not going to say who) being models that are distilled outside the same company.

You can shrink this surface quite a bit, and the errors along with it, down to the much smaller, but persistent, class of problems that apply to LLMs as a species....

In one benchmark, four different model families produced the same total, to the cent, because they all shared a sign convention the answer key didn't.

Once you get that far, the only way forward is to list them, look out for them, and apply specific solutions. Here are ten useful checks:

1. Look before you transform. List each column's real values and formats before writing code for them.
2. Compare before and after loading. Null rates, category counts and totals should match the raw file.
3. Recompute every number you report. Check each count in the docs against the data, in code.
4. Prove a lossy step can be undone. If a step throws information away, keep the original beside it.
5. A passing test isn't the finish. For each green check, sample one question it can't answer.
6. Save budget for the end. Leave room for the write-up and a self-check. Watch for retry loops.
7. Take access away, don't just ask. Models will explain a shortcut and still take it. Remove the path.
8. Make every check fail once. Plant a known bug. A check counts only after it catches it.
9. Agree on conventions first. Settle signs, grain and dedup rules before calling a gap an error.
10. Run it more than once. On complex data tasks one run is noise: pass/fail flips a third of the time.

What's missing, and Pareto surfing

So, going back: what are the four things we're missing, to be able to self-optimize?

Number one is a stable deployment surface. We use hankweave for this: a harness that takes the fast-moving surface of agent runtimes like Claude Code and Pi, and exposes a consistent, well-tested log and API interface, with checkpointing, rollback and budget enforcement built in. Something you can build on.

Other things fit this description, but for headless data work, hankweave is still the oldest, most stable, and most open. If you'd like to try it, here's the skill:

Number two is some way to bound your experiments. You can't test unless you can control cost, time, and access as resources.

The third is the know-how and the methodology. We found that this new class of models can take everything we'd learned from two years of deployments, agentic patterns, debugging techniques, prompting, and apply it to complex problems. With the right abstraction layer, you can express it very token-efficiently, and the models can implement it.

The fourth, and the most important, is a failure surface. Archimedes said: give me a place to stand, and I will move the earth.

In our case, we haven't met a problem or constraint we couldn't iterate our way to solving, given a stable stack to build on, and a failure surface to iterate against.

This last one has been the hardest to put into words, and into tooling and instructions that help models get there. The current best models are still only barely good at this. Sparks of intelligence, so to speak....

The whole process involves going through six stages.

First, we spec the job and split it. For Jev, that meant splitting 883,012 rows into 203,317 small cases, and each case into typed questions: what kind of donor is this, is it one donor, are these two the same.

Then we decide the failure surface. Here, we ran good models on a small slice of the data, with some of it checked by hand, to set a baseline we can fail against. In other cases, it might be properties of the data we want to preserve, or automated checks, but we always want to define a failure surface before we start.

Now comes the fun part: we run minimal, bounded pilots to test architectures.

Once something starts working, we move checks into rigs: code that runs between bounded work segments we call codons. Codons are reused, looped, independently validated, and placed in sequence.

Finally, working pipelines are deployed to production, where they can slowly mature.

Play

That middle part is where the surfing happens. We find an architecture that gives a solid result, and use it as a baseline. Then we swap models in some parts for faster, cheaper options, compare against known intermediates, and tighten our requirements for success, moving closer and closer to the ideal. And you just watch the frontier move.

Being able to break a complex problem into pieces means each piece is free to move to its own best position, as Adam Smith would say.

In a system like this, ontologies exist as intermediates between intelligence and result, which makes them easier to reuse. That's the other half of what I mean by a just-in-time ontology: it's built when the job needs it, kept with its receipts, and reused when the next job needs the same thing.

Closing: an apology

Once you have them, you can make these things cheaper, and you can run them in parallel, across datasets, to scale up.

I close with an apology, that the answer is: there is no ontology. Not a universal one, at least. There never was one.......

The good news is that this makes every ontology and schema valid, as long as you can label and preserve the context around it.

If we can harness the cheap intelligence that's starting to percolate through the world around us, we're rapidly approaching a future where any information, in any format, is made as useful as it can be. At Southbridge, we're working as hard as we can to make that future a reality.

Thank you!

Links from the talk:

• The Jev entity resolution article, and the study files, prompts and all
• hankweave docs
• hankweave runtime on GitHub