Either the models keep getting smarter, and your expertise stops being worth anything. Or the whole thing is a hype, and you keep your job for the same boring reason that the robots were not able yet to handle it better than you. Your expertise is doomed, or the whole thing is fake. Right? This doesn’t sound good either way.

But I don’t think those are the choices. And I’ll try here to argue why.

Every field that wants AI to actually work needs two things. Somebody has to build the intelligence. And somebody has to be able to tell when it’s wrong. The first one is getting solved in San Francisco right now, at staggering expense, whether you like it or not.

This article is about the second one. Because in your field, the second one is you.

WHAT’S ALREADY TRUE

Here’s what’s already happening this year. Somebody who runs a clinic, or a field survey, or a regional supply chain, can describe to an AI what they need in their own words and get what they described working back the same afternoon. No engineers, no budget line, no six month project. That part is real, and it is genuinely new.

And then the thing AI made for them hands them an answer. And the answer looks right. The answer always looks right. That is the one thing these systems have literally never struggled with.

So the question, the whole question, is who in the room can say that it’s wrong.

Because think about what that answer is going to do. It decides how much medicine gets ordered. How many people get rostered. Which patients get a callback. The software was free. The mistake is not.

And here’s the sentence I want you to hold for the rest of this argument. Judging the output takes exactly the expertise that producing it no longer requires.

The model got cheap. The tools got cheap. The engineer became optional. Knowing what correct looks like in your field did not get cheap. And there is no sign that it’s going to.

Economists actually had this shape mapped before anyone needed it for AI. Back in 2017, Philippe Aghion at the LSE, with Benjamin Jones at Northwestern and Charles Jones at Stanford, wrote that growth isn’t constrained by the things we’re good at improving. It’s constrained by whatever is essential and refuses to get cheaper. Make every input cheap except one, and that one input sets the pace for the entire system.

The bottleneck is never the impressive part.

THE MACHINE THAT SAYS WRONG

Now, the obvious objection. AI is incredible at writing code. And the story everyone tells is that the labs are full of engineers who built the thing for themselves.

I don’t buy it. And the real reason turns out to be the whole argument.

Software has something almost no other field on earth has. It has a machine that says wrong. You write the code, you run the test, and one second later you get a verdict. No opinions. No meetings. No seniority. Just pass or fail.

And that machine is what the last two years of AI progress actually ran on. The model tries something. An automatic verifier says pass or fail. The verdict trains the next attempt. And that loop turns millions of times. For the loop to work, the verifier has to be cheap, instant, and beyond argument. A test suite is exactly that. Math has one too. And then the list just ends.

So software didn’t get ahead because of who works at the labs. It got ahead because it walked in the door carrying a free verifier that has existed since the 1950s. It’s not the exception to the bottleneck. It’s the one field where the bottleneck got solved by accident, decades early, for free.

And the labs know this. They’ve been trying to build verifiers for everything else, usually by writing a scoring guide and having a second model grade against it. In medicine and science, that keeps backfiring in a very specific way. The scores go up against the guide, and not against reality. And the longer you train, the wider that gap gets. The model is learning something real. It’s learning to please the grader.

So here’s the rule for the rest of this argument. A bad verifier is worse than no one at all. Because a bad verifier is confident.

WHEN THE VERIFIER IS A PERSON

OK. So in a field with no compiler, somebody has to be the compiler. There is no third option.

And you can watch this play out in the labs’ own evaluations. OpenAI built one called GDPval. They collected real deliverables from working professionals across 44 occupations, people averaging 14 years in the job, and asked whether models could match the work. The models are closing in on expert quality. That made the headlines.

The part that matters is buried in the method. What improves the score is more context and more scaffolding. And context and scaffolding are not properties of the model. Somebody who knows the job has to supply them. Somebody has to say what the deliverable is for, what would make it unacceptable, and which of five perfectly reasonable-looking drafts is the one to keep.

Google DeepMind said the blunt version of this recently. As these systems make candidate answers abundant, verification becomes the choke point. Generating a thousand plausible hypotheses is now basically free. Finding out which one is true costs exactly what it always cost.

And their own showcase example proves the point better than they meant it to.

José Penadés is a microbiologist. He spent the better part of a decade on one question: how a family of superbugs picks up and spreads resistance to antibiotics. In 2024 he described the problem to an AI research system, and two days later he had five ranked explanations back, including the one he’d reached himself and hadn’t published yet.

The two days made every headline in the world. But the decade is the mechanism. The decade produced a question precise enough to be worth asking. And the decade is why, when five answers came back, there was one person on earth who could pick the right one and throw the other four away. Take him out of that story and there is no discovery. There are five confident answers and nobody who can tell them apart.

Anthropic ran this from another perspective. They asked experienced workers across dozens of fields where AI stops being useful to them. And the answers came back the same everywhere. Judgment. Context. Relationships. Management. Listen to those four words. They’re four different names for one thing. Knowing what counts as correct here.

Now, does that mean expertise is safe? No. that’s a comfortable reading, and the comfortable reading is usually wrong. The work moved. It moved from doing the job to specifying the job and verifying what comes back. Those are different skills. Right now they happen to live in the same people. Nothing says they have to stay there.

THE BEST ARGUMENT AGAINST ME

There’s real evidence pointing the other way, but are they really pointing the other way?.

Erik Brynjolfsson at Stanford, with Danielle Li at MIT and Lindsey Raymond, studied over five thousand customer support agents working with an AI assistant. Productivity went up about 14 percent on average. But the average hides the finding. The weakest workers gained about a third. The strongest gained basically nothing. What the AI was doing was picking up the tacit know-how of the best agents and handing it to everybody else. And David Autor at MIT builds a genuinely hopeful thesis on top of that: the real prize is spreading expert judgment to far more people, and giving high-stakes decisions back to workers who’ve been locked out of them.

If that’s the whole story, expertise gets common instead of scarce, and my argument is upside down.

But that sounds more as an abstract and not the hole story because Customer support has a verifier. It’s sitting in the transcript archive. Thousands of resolved cases where somebody, years ago, already settled what the right answer was and wrote it down. The AI can compress all of that and hand it to a rookie precisely because the field got solved first, and then recorded. Autor’s mechanism is my mechanism. It’s what a field looks like after somebody built the verifier.

And that leads to identify the frontier as everywhere that hasn’t got a predefined verification yet. thus at the frontier, the expert is not a beneficiary of the system. The expert is a component of it.

WHAT VERIFYING ACTUALLY COSTS

So how expensive is the verifying, really? The best evidence comes from the people with the best verifier on earth. And they fail anyway.

METR, a nonprofit that measures what AI systems can actually do, ran a randomized trial. Sixteen experienced opensource developers, a couple hundred real tasks, in code they knew inside out. Going in, the developers expected AI to make them about 24 percent faster. Measured? They came out about 19 percent slower. And afterwards, having done every one of those tasks themselves, they still believed the tool had sped them up.

Now hold that loosely. It’s one study, sixteen people, and METR pulled its own follow-up after finding that developers had been choosing which tasks went into the trial. So don’t lean on the exact number.

But the second finding survives all of that. And it’s the one that should bother you. These people had a compiler, a test suite, and years inside the codebase. Every structural advantage this entire argument says matters. And from the inside, they could not tell whether the tool was helping them. If self-assessment fails there, in the easiest verifying environment human work has ever produced, what do you think happens in a field where the feedback takes months? Or in a clinic, where the feedback is patient outcomes that nobody ever traces back to the decision?

And there’s a residue. A large study of AI-written code across public repositories found roughly half a million problems introduced, and about a hundred thousand of them still sitting in the code today. The work shipped. It ran. And a fifth of the damage is still in there, because nobody qualified to spot it ever went looking.

THE SETTINGS

Right now, being the verifier also means knowing some engineering. What is this system allowed to touch? Does the same input give the same output twice? Can somebody else run it and get what you got? In software, those are breakfast questions. Everywhere else they’re exotic. And they’re the current toll for turning a demo that works into something your clinic should actually trust.

But those are engineering problems. And engineering problems are exactly the kind these companies solve. So the scaffolding is going to show up in the box, and knowing what any of it means is going to stop being worth much. That part I’d bet on.

What happens next is the part I keep circling. The automatic verifier still has defaults. Somebody decided what it can touch. Somebody decided what counts as failure, and when it stops. And those decisions are going to arrive at your clinic looking like settings, not judgments. Because that’s how big choices travel. Pre-made, pre-installed, by people who have never seen your work.

So the question turns. It stops being can you build the verifier. It becomes can you read the one you were handed, and say out loud when it’s wrong for your work.

That is not a smaller job. It’s the same job, moved somewhere much harder to see.

CLOSING

So the next time somebody demos an AI system for your field, and that’s probably this week, ask three things.

What tells this system it’s wrong? Who wrote that verify, and did they know the work? And what happens to the outputs that nobody verifies?

Those three questions will tell you more than any benchmark.

Every field needs two inventions. The intelligence, and the verifier. The intelligence is getting built at staggering expense whether you like it or not. The verifier, in your field, right now, is you.

And I’ll be honest, I started this argument trying to close a question and I’ve opened a worse one. Verification is expensive, but expense is the kind of problem money tends to solve. So if we finally reached a solution to the verifying and managed to make it goes automatic, does the bottleneck finally disappear? Or does it just move, one step back, to whoever decides what the automatic verify is measuring?

I think it moves. And when it does, we might get nostalgic for this bottleneck. The one when the person who could say what’s wrong at least worked in the building.

I

You have probably been handed the same two options I was. Either you let the coding agent run, and you find out what it did to your repository afterwards, or you read every diff it produces, at which point you are doing the work again with extra steps.

Both of those are bad, and I do not think they are the actual choices. There is a third one, and it is boring, and it is the whole reason I spent three months on this. You can take the rules out of the conversation and put them somewhere the model cannot argue with them.

That is the piece I want to explain. Not what my agents build. What they are refused, who does the refusing, and how you find out when the refusing quietly stops working.

A rule the model can be talked out of is not a rule

Here is the thing most agent setups get wrong, mine included, for months.

You write your safety rules in the prompt. Never push to main. Always run the tests first. Do not touch production data. And the model complies, most of the time, which is the dangerous part. Compliance most of the time reads exactly like enforcement until the run where it does not, and you have no way to tell those two states apart from the outside.

An instruction in a prompt is a request. It competes with every other instruction in the context, with the task the model is trying to finish, and with anything an issue comment or a dependency’s README happens to say. A rule the model can be talked out of is not a rule. It is a preference with good manners.

So I stopped writing rules and started writing refusals.

What a hook actually is

Claude Code has a feature that fires your own script at fixed points in the agent’s lifecycle, before a tool call, at the start of a session, and so on. People call these hooks. The function is simpler than the name: your program gets to see what the agent is about to do, and gets to say no.

The signalling is crude and that is a virtue. Anthropic’s hooks reference is explicit about it: on a pre-tool event, a hook that exits with code 2 blocks the tool call and hands your error text back to the model as the reason. Exit 0 and the call goes through.

Nothing about that involves persuasion. The subagent does not weigh the refusal against its task. There is no refusal to weigh. The process returns 2 and the tool call does not happen.

That is the entire design idea behind AEO, the Claude Code plugin I published this week. Fifteen skills, five agent charters, five gate scripts. The skills are the part that does work. The gates are the part I would defend in a review.

The lanes, and where every one of them stops

The shape is a small software shop run by one person.

You type /aeo:sprint-plan and a triage role decomposes a phase of the spec into GitHub issues. You type /aeo:sprint-start and a builder picks up the next unblocked issue. One issue, one worktree, one branch, one pull request. Inside that branch it runs a double loop: a failing acceptance test on the outside, red, green, refactor on the inside, then the matching CI workflow once the slice is green locally, then a pull request with the evidence attached.

For small work there is a second entrance, /aeo:fix, which skips the sprint ceremony and the reviewer and goes straight to a scoped fix. It skips the planning. It does not skip the gates.

And then every one of those paths arrives at the same place and stops. A pull request, waiting for you. The builder, the reviewer and the triage roles have no route to git merge or gh pr merge, however they are invoked, because a gate called block-merge refuses the tool call. You merge. That is the product, not a limitation of it.

Four of the five gates refuse something specific. sandbox-guard refuses any command or file access that reaches data you have declared as production. path-guard stops a role editing the harness’s own configuration, which is the obvious way a clever agent would otherwise clear its own path. review-jail is the one I like most: while the reviewer role is working, it can read its own staged evidence packet and call nothing else. No shell, no network, no writes. A reviewer that can edit the code it is reviewing is not a reviewer.

The fifth gate never blocks anything, and it is the one I want to talk about properly.

The part I would rather not have written

Every gate AEO ships is a Node script, invoked directly by the hook config. Which means the gates depend on Node being on your PATH.

Now walk through what happens if it is not. Claude Code tries to start the gate process. The process fails to start. It exits non-zero, but it does not exit 2, because it never got far enough to decide anything. And per that same hooks reference, a non-zero exit that is not 2 is a non-blocking error. It is logged and the action proceeds.

Read that again, because it took me an embarrassingly long time to sit with it. In that state the tool call goes through. Nothing is refused. Nothing is recorded as refused. And nothing tells you. The session looks exactly like a session where every gate is holding.

I could not make the gates fail closed, because the failure happens upstream of anything I control. So I did the only other honest thing. A session-start check runs before any of it and prints one line when Node does not resolve: gates not enforcing, every hook fails open, treat this session as ungated until you fix it.

That line is a confession compiled into the product. I think it is also the most useful feature in it. A control you cannot verify is not a control, it is a belief about a control, and the two feel identical right up until the audit.

If you are running your own hook-based guardrails today, this is the part worth stealing. Not my gates. The question. What does your enforcement layer do when it cannot run, and would you be able to tell?

The gate I deleted

There used to be a sixth. A local check that refused a commit while the test suite was red.

I removed it, and the reasoning is in the repository’s decision log, because GitHub already enforces the same thing server-side through branch protection, and with enforce_admins turned on it holds against my own session too. My local gate was a second mechanism for one rule, and the weaker one.

That is not a saving of forty lines. Two overlapping controls do not add up to more safety. They add up to ambiguity about which one is actually load-bearing, and ambiguity is where false confidence lives. If a server-side control already covers the case, delete the client-side imitation and let the real one be visibly the real one.

Same instinct applies to the version. AEO is at v0.1.0, and the tag does not pin your install: the marketplace manifest is read from the default branch and has no version field to resolve, so marketplace add always gives you current main. I would rather say that out loud than let a release tag imply a stability promise I have not earned. The gates and lanes work. They have been exercised by one project. That is not enough evidence for 1.0.

Autonomy is mostly a permissions question

There is a way of talking about agents that treats autonomy as a property of the model. How capable is it, how much can we trust it, is it ready.

Most of the time that is the wrong axis. What an agent is allowed to do without asking is a design decision someone made, or more often failed to make, and it sits in configuration files rather than in weights. When you cannot name where the boundary lives, there usually is not one.

Which turns the interesting questions operational, and they are the same three whether you are evaluating my plugin or a vendor’s demo. What can this system do without a human approving it. What enforces the boundary, prompt text or a process that returns a refusal. And how would you discover that the enforcement had stopped working?

Most agent demos survive only because nobody asks the second one.

Here is my test, and you can apply it in about a minute. Ask the person to break their own guardrail on purpose. If the answer is that the model would never do that, it is a preference with good manners. If the answer is that the tool call would be refused and here is the script that refuses it, you are looking at a rule.

AEO is at github.com/Muhanad-husn/AEO, MIT except a vendored reference snapshot that keeps its own terms. Install it with /plugin marketplace add Muhanad-husn/AEO and then /plugin install aeo@aeo. It will not merge your pull requests, and that is the point.

The open question I have not resolved: seven of the fifteen skills only run when you type their slash command, and they never fire on description no matter what you say. That is deliberate, and on the first day it makes the plugin feel broken, because describing a sprint and waiting for something to happen gets you nothing. I chose predictability over convenience. I am not certain that is the right trade for anyone who is not me, and the only way to find out is to watch what people do with it.

Muhanad-husn/AEO

hen you point a knowledge graph tool at a Python repo, the schema almost writes itself. A Function lives inside a Module. A Module IMPORTS other modules. A Class INHERITS_FROM a parent class. A Function CALLS another function. None of these are interpretive judgments — they’re already encoded in the AST, the import statements, and the directory structure. The repo is its own schema.

This is why the wave of “turn your codebase into a knowledge graph” tools feels deceptively powerful. They aren’t designing ontologies. They’re transcribing structure that the language compiler already enforced. The hard part of schema design — deciding what counts as an entity, what counts as a relationship, what gets collapsed and what gets separated — was solved by whoever wrote the Python spec.

Now point the same tool at a folder of investigative reports, NGO field notes, or supply chain disclosures. The schema does not write itself. And this is where I think most knowledge graph projects quietly run aground.

What codebases give you for free

A well-structured codebase comes with three properties that real-world corpora almost never have.

First, entities are typed and named at the source. A function has a signature; an import has a target; a class has a declared parent. There is no ambiguity about whether process_data is a function or a concept or a metaphor — the language itself draws the line.

Second, relationships have a single canonical direction. A imports B is not symmetric, and the codebase tells you which side is which. There is no debate about whether to model this as “A depends on B” or “B is depended on by A” — both are valid, but the underlying fact is unambiguous and the parser picks for you.

Third, the schema is closed. A Python repo will not surprise you with a new node type next week. The language defines what kinds of things can exist. Anything that doesn’t fit one of those kinds is, by definition, not part of the codebase.

Real-world corpora violate all three. Entities are referenced inconsistently (”the Ministry,” “MoFA,” “the Foreign Ministry,” “the ministry led by Minister X”). Relationships are direction-ambiguous and often time-bounded (”worked with,” “advised,” “was previously associated with”). And the schema is open — every new document can introduce a category you hadn’t planned for.

So the question isn’t whether knowledge graphs work for real-world data. They can. The question is whether the schema can be designed well enough to make extraction tractable without collapsing into either of two failure modes: a schema so general it becomes a glorified adjacency list, or a schema so specific it encodes one analyst’s mental model and resists reuse.

Why “let the LLM figure out the schema” usually fails

The obvious move, once you accept that real-world corpora need a designed schema, is to hand the corpus to an LLM and ask it to propose one. I’ve watched several projects try this. The results follow a depressingly consistent pattern.

The LLM reads a sample of the corpus and produces a plausible-looking schema with twenty or thirty node types, half of which overlap semantically (Person, Individual, Actor, Stakeholder) and most of which lack any obvious analytical purpose. Edge types proliferate similarly. The schema looks impressive in a diagram and falls apart the moment you try to write a query against it.

The reason is structural. An LLM asked to “propose a schema for this corpus” has no signal about what questions the graph is meant to answer. Without that signal, the model defaults to surface taxonomy — categorizing whatever entities it sees rather than designing for what the user wants to ask later. A schema designed without questions in mind is a schema designed for nothing in particular, and it shows.

This is the diagnosis behind the technique I built into Neo4all’s Phase 1, and it’s worth unpacking on its own.

The technique: domain description first, schema proposal second, lock third

The approach Neo4all uses for schema generation has three properties that, taken together, address the failure mode above.

The first is that the LLM never sees the corpus when proposing the schema. It sees only a plain-language domain description written by the user — what the data is about, what kinds of questions the user wants to answer, what entities and relationships matter for those questions. The corpus is held back deliberately. This forces the schema to be designed for the analysis, not for the surface patterns of the documents.

The second is that the LLM’s output is a proposal, not a commitment. The user reviews, edits, removes node types, merges edge types, renames things. The interface is built on the assumption that the first draft will be wrong in interesting ways and that fixing it is cheap. This matters because schema design is fundamentally an iterative epistemic process — you discover what you actually need by trying things, not by specifying them perfectly upfront.

The third is that, once approved, the schema is locked. A deterministic version hash is computed, the schema is cached immutably, and every downstream extraction in that run is bound to it. This is the part that feels most like the spec-and-contract analogy from software engineering. The schema becomes a frozen interface against which everything else is validated.

What this technique gets right is the order of operations. Question → domain description → schema proposal → human edit → lock → extraction. The LLM is used where it’s actually good (drafting plausible structure from a written description) and kept away from where it’s bad (inferring intent from unstructured text alone). The human is in the loop at the one point where intent has to be encoded.

What this does not mean

A few things this approach does not claim, and shouldn’t be read as claiming.

It does not mean LLM-drafted schemas are good schemas. The first proposal is often wrong; the value is that it gives the human something concrete to react to, which is much easier than designing from a blank page.

It does not mean the schema can stay locked forever. For long-running investigative projects, schemas evolve as the corpus reveals new structure. Neo4all locks per run, not per project, and that’s a deliberate compromise — you get reproducibility within a run and flexibility across runs.

And it does not mean schema-first is right for every domain. For exploratory corpora where you genuinely don’t know what questions to ask, an iterative schema-discovery approach may beat upfront design. The technique I’m describing works best when the analyst already has a research question in mind.

Where this leaves us

Codebases get their schema for free because someone — a language designer, a framework author, a build system — already did the ontological work. For the rest of us, working with field reports and corporate filings and policy documents, that work is the actual job. Extraction quality, LLM choice, vector store selection — these matter, but they’re downstream of the schema. A well-designed schema with mediocre extraction will produce a usable graph. A badly designed schema with state-of-the-art extraction will produce a confidently wrong one.

The question I’d leave open is whether the domain-description-first technique generalizes. I built it for investigative and research corpora where the analyst has questions in mind. Whether it works for genuinely exploratory cases — where the analyst is hoping the graph will tell them what to ask — is a harder problem, and one I don’t think anyone has cleanly solved.

Neo4all: AI-powered platform that transforms documents into curated Neo4j knowledge graphs via governed proposal-approval pipelines

In April 2026 Andrej Karpathy published LLM Wiki, a gist he called an idea file. The move in it is one sentence: stop running retrieval against raw documents forever, compile them once into a persistent, interlinked wiki, and put every later question to the wiki. I’d been keeping my reading in Obsidian for years, a couple of thousand leaflets, summaries, articles and posts, and I wasn’t confident I could keep that vault’s schema under control as it kept growing. So I pulled a few books off my PDF shelf to find out whether the pattern holds at scale. I expected a light job.

Thirty-one days later there was Axial, an independent production implementation of the same pattern: 35 scholarly works read into 6,842 passages and 47,584 name pages, with an argument map of 1,937 positions joined by 1,472 stated relations above them. Built solo between 6 July and 6 August 2026, about a thousand commits, 57,654 lines of Python surrounded by 93,175 lines of tests, with builder agents hook-blocked from merging anything and my approval as the only path to main. That is the summary. What follows is the measurement record, including the measurements that went against me, because that record is the one thing a tutorial cannot hand anyone.

Two boundaries belong here rather than at the end. Axial is built, not deployed: a command-line tool on one machine, one operator, a library one person assembled. No service, no multi-user boundary, no hosted product. Everything below measures an instrument, not a product in anybody’s hands.

And the corpus counts above are the current library. The measurement record that follows was made earlier, on the 31-book library; the last four sources were added after it, and the figures have not been re-measured against them. Where a number below carries a corpus, that corpus is 31 books.

How the tagging layer died

The first version did what most document-intelligence and knowledge-graph pipelines do: tag every passage against five closed vocabularies, then query the attributes. For months it looked fine. A model read a passage, produced a sensible label, moved on. Nobody had measured whether a second reader would produce the same label.

So I ran two frontier models from two different labs over the same 120 passages, same neutral instructions. On the two judgment-heavy axes, kind of claim and theoretical school, they agreed 0.49 of the time. My bar for keeping an axis at all was 0.60. Both were underwater.

The obvious first reaction is a better model, and that went the wrong way. The cheap production tagger agreed with one of the frontier labellers 0.56 of the time, more than the two frontier labellers agreed with each other. Capability was never the lever.

Three prompt-side interventions came next, and all three shipped nothing. A codebook rewrite costing 55% more tokens per call moved agreement by roughly zero. An explicit rule about which unit to label added 0.02. Feeding the book’s own stated thesis into the labelling context scored -0.01 on the full sample. I trimmed the codebook back and kept it for readability alone, recorded as explicitly not a measured gain.

Then I measured the ceiling, and there was nothing left to fix. The same model, given the identical prompt twice, reproduced its own theoretical-school label 0.73 of the time. Two independent model labellers agreed with each other at the same 0.73. Agreement between two readers cannot exceed the reliability of one reader with themself, so there was no headroom to find. The variance was in the question rather than in the model, the prompt or the definitions. “Which school does this passage speak from?” does not have one answer, a closed vocabulary forces one anyway, and the number that comes out looks like knowledge.

One further measurement, to prove the diagnosis rather than rescue the layer. If a single draw samples around a modal answer, majority voting should recover the mode. From a 0.88 single-draw modal hit rate I predicted about 0.92 at best-of-3, and measured 0.918. The intervention matched the diagnosis, which is why it worked when three prompt-side ones had not, and its honest cost was an 8.8% abstention rate on genuinely contested passages.

That sequence did not buy a working tagging layer. It bought the knowledge that no further work would produce one. “Measure, don’t speculate” is a stated developer principle in the repository, and this is where it stopped being a slogan: the shipped system looks nothing like the first one, and every difference traces to a measurement rather than to a preference. What none of those figures measures is whether a label was any use. Agreement is agreement. Two readers can agree on a bin that tells you nothing.

An attribute is not a relation

Reliability was never the real problem, and the number that showed it had been sitting in the corpus the whole time. At the end of that version, 18,761 tagged passages across 31 books had produced 584 connections. All 584 were inside a single book. Links between two books: zero. Links between two passages of prose: zero.

That is not a tuning failure. The only mechanism that could mint an edge asked a closed question and filtered its answer against that book’s own list of figures, so its output was intra-source by construction. No parameter setting could have produced a cross-book link. The diagnosis is one sentence: ”an attribute is not a relation.” A tag sorts a passage into a bin. Two passages in the same bin have been sorted the same way; they have not met. What makes two books argue is that one names Charles Tilly and so does the other, or that one says whom it argues against and the other is the target. A closed vocabulary records neither, because it has to know the answer before it reads.

The replacement was the question, not a better detector. Each passage now gets one model call carrying fourteen open questions, answered in the passage’s own words: what it claims, what move it is making, whose position it is, who it argues against, whom it cites, every named thing in it, what it concedes, what it assumes. Seventeen answers come back per passage. Abstention is first-class, because “a guessed answer is worse than an abstention” and nothing downstream can tell a guess from a reading.

Over the same books the passage count fell from 18,761 to 6,148, deliberately, because the passages got bigger to fit whole arguments. Everything else appeared for the first time: 9,505 names shared across books, 8,769 of those shared across different authors, and 447 stated disagreements between authors where tagging had recorded none. For a researcher the consequence is concrete. Two books that never cite each other end up on the page for a name they both use, and that page says what their authors disagree about, with both passages open underneath it.

The retired vocabularies did not disappear, and the way they survive is the single most important guard in the reading. The model answers every question in its own words first. Only afterwards is it shown the domain’s example vocabulary, in a separate marked field, told plainly that these are “NOT a menu, NOT a vocabulary, and NOT a set of allowed answers”. No code path bridges the two fields, and a collapse metric watches for interrogation quietly rebuilding tagging. Reverse the ordering and Axial is a tagger again with extra steps, and nothing in the output would show it.

The finding is about one corpus and one closed-vocabulary design. It says the mechanism that minted edges could not mint a cross-book one. It does not say closed vocabularies are useless, which is why the retired ones are still on screen as examples. The transferable version is shorter: when a closed instrument underperforms, measure whether the question admits one answer before spending anything on a better detector.

The same lesson, twice more

Retrieval was the second time. Interrogation had already produced a typed graph: passages answer “who is this arguing against” 76.4% of the time, which is 10,883 opposition targets recorded at read time. Materialisation then kept only the node labels and grouped passages into pages by surface string. Measured offline over the live corpus, 4.7% of those targets joined to anything a query could reach. Every retrieval patch I had shipped to that point had been tuning a lossy projection of data the system already held.

Loading the same records into seven SQLite tables lifted the conservative join to 44.0% and exposed 43,101 high-confidence cross-source opposition pairs across 343 named scholars and works. That measurement cost $0, because it was SQL over records already on disk. A semantic resolver over the remaining targets was calibrated on a sample and then run in full for $1.08, taking combined coverage to 60.2%. That is not full coverage and I would not present it as any. What is left over are prose position descriptions with nothing to key a join on, and no better ranker closes them.

The argument map was the third time. The pass that relates one position to another was given no menu of relation types. Told nothing, the model volunteered opposition on only 6.6% of pairs and coined 504 labels of its own. That 6.6% is the whole reason an engine told to look for opposition cannot be trusted when it finds it.

What it costs to add the thirty-fifth book

The first time I added books to a finished corpus, three into thirty-one, the run cost $10.26 and 9.5 hours, and of its 8,971 model calls, 93% were spent re-processing the 31 books already ingested. Three causes, three fixes, each validated on the real corpus.

Merge was re-clustering globally, so a fresh fit reshuffled membership and disturbed 37% of merge batches for a three-book delta. The fix persists the fitted transform chain and places new surfaces under each cluster’s own fitted floor: re-ask rate 37.0% to 10.9%. The map was re-bagging everything, because agglomerative clustering has no approximate_predict, so I built one, persisting each bag’s centroid and placing new passages by average linkage: 77.7% of extraction reads reused, where the old path reused none. Gather was re-rendering any page that gained a member, so it is now keyed on the name plus its sorted source-id set, because a disagreement is a property of who is in the room. Measured on the thirty-fifth book, Gather asked 144 reads and reused 1,704, which is 92.2% reuse, with 132 of the 144 genuinely touching the new book.

One find from that run I’d put in front of any engineer. Effective concurrency here is computed as summed model-call latency divided by wall clock, never read off the worker count. Interrogation was running at an effective concurrency of 1.00, strictly serial, inside a run whose other passes ran at 16.7 to 33.9, hidden inside 9.5 hours of wall clock that packed 94 hours of API latency. Nothing looked wrong from the outside, and the worker count is a configuration value rather than a measurement.

Every dollar figure here is an upper bound, from a hand-maintained price table measured to run about 14% high against a real invoice.

The same model, the same input, a different answer

The property that shapes more of this system than any other is that the same model gives a different answer to the identical input 9 to 36% of the time, depending on the task. It is not a nuisance to be prompted away. It is the measurement environment, and it makes any A/B without a noise floor a story rather than a measurement.

The standing method is to measure the floor before comparing anything: re-run the same configuration on byte-identical input and see how far it disagrees with itself. Name merge disagrees with itself 9.3% of the time across all batches, and 18.8% on batches of three or more members. Gather disagrees with itself 19.3% of the time overall, and 36.1% of the disagreements it records come back null on the second reading. Temperature is not the cause, which I had assumed it was: greedy decoding measured 14.7% self-disagreement at T=0 against 13.3% at T=1, slower and more verbose and still non-deterministic.

Then the distinction that mattered. 19.3% of merge batches flip, but only 0.43% of the underlying passages change page, and every flip observed was a singular, plural or article variant. A surface changing group is not its evidence moving, and conflating the two had overstated the problem by a factor of thirty. I had set the acceptance bar at 5% before the run rather than after it, the result came in 3.5 times under, and the issue closed with the instability accepted and a written condition for re-opening it.

At run level the analogue is a rule I hold to: one draw is not a measurement. A single brief has been measured moving 39% between two runs of identical code. It caught a model switch I had already shipped. The envelope pass was moved on a five-source A/B, then reverted when full-corpus regeneration produced 6 degenerate outputs where the incumbent had 0. The A/B had sampled the sources the challenger handled well.

Where a pass cannot be made to reproduce, it gets demoted rather than trusted. A Gather finding is a retrieval hint and never a citation, and no gate scores an answer against one. In its harshest framing, the research report’s rather than the engineering report’s, Gather agrees with itself 53% of the time, and that is the figure a sceptical reader should hold. Where a check has nothing to measure at all, it reports not-scoreable, a third state distinct from pass and fail, and that state blocks release. A metric that vacuously passes on zero observations reads as a green light for a check that never ran.

The panel, and the control that qualifies it

The last instrument is the one built to be trusted least. Finished papers are sampled to a sealed panel: at least three reviewer models per packet, each from a different training lab than the model that wrote the paper, each receiving the rendered paper, the resolved text of every passage it cites, and the bibliography. Nothing else. The seal is enforced by the dispatch path having no parameter through which tools could be passed, because an agent holding file tools will read the repository whatever its prompt says. Isolation you ask for is not isolation. Spread is reported as the error bar, since three reviewers splitting 1/3/5 and three agreeing on 3 are different results and must never render identically.

None of that would count without the control. Model judges are systematically generous and are moved by confident prose, so the panel reports no number until it has caught defects planted in a paper it was not told was a control. Three were planted: a fabricated cause cited to a passage that says nothing of the kind, the section stating the opposing case replaced with a caricature, and a high confidence band asserted over coverage the paper’s own map discloses as thin. The current panel caught 3 of 3, unanimously, across three reviewers. An earlier, cheaper, single-vendor panel caught 0 of 3, and every number that panel had produced was discarded.

The control proves the reviewers catch defects that are there. Nothing tests whether they invent defects that are not. The panel’s false-positive rate is unmeasured, and “trusted” here means the instrument catches planted defects, never that it does not manufacture them.

The panel then found the part I did not design for. Five of six reviewers independently flagged the same citation: it resolves cleanly, to a real passage, which happens to be a publisher’s catalogue page. Every mechanical check passes it, because machinery can verify that a marker points at a real passage and cannot judge that the passage is a catalogue entry. Four reviewers independently reached a second diagnosis, that the keystone claim in both papers is carried by evidence from two other countries, because the library holds no passage on the specific thing the claim needs. That one is a corpus gap presenting as a citation defect. No retrieval work fixes it. The fix is a book.

Where the pattern starts to break

Axial was run to 35 sources, and the point where the pattern starts to break was measured rather than guessed at. Passage-to-position count grows roughly linearly as the shelf grows, at k = 1.04. The graph, though, densifies: cross-book relations rose from 8.7% to 38.4% of the total across corpus sizes, and have not plateaued. Read together, those two say that extraction scales and selection does not. Somewhere near 100 sources the hard problem stops being “read the book” and becomes “choose which of the thousands of things now arguing with each other should go in front of a model”.

That is an empirical warning drawn from a handful of corpus sizes on one shelf, not a law, and I would expect the number to move with the domain. It is also the one thing an idea file cannot offer, and the reason I would rather publish a measured limit than a working demo.

What none of this measures

Every test question in both reports was written by an AI model working from a description of the library, not by a working scholar. That was deliberate: it let the engine be built and hardened without waiting on anyone, and nothing simulated was ever used as an answer key. It also bounds every figure above. They measure the engine. None of them measures answer quality against a real research question.

There is no human referee in the loop. No number here may be described as measured against human expert judgment, and the specification forbids relabelling one that way for as long as a panel of models is the only referee.

The denominators are small, and they are printed for that reason: 215 citation markers, 105 claims, 32 cross-source inferences, two papers, five new cross-source inferences. Five is five, not a rate. A perfect score over a small n is a small piece of evidence, and the paper gates passing at 1.0000 is exactly that.

The mechanical gates measure construction, not evidential sufficiency. Two of the eight papers written so far cite exactly one book each, pass every gate, and rate strong on shape, because the shelf holds one comparative study of unrecognised states and no monograph on either territory they ask about. That is the library reporting its own shape, never a finding. The shelf is built around one case, Syria, inside the comparative-historical literature on state formation, nationalism and political violence, and every paper inherits that. Only new books close that gap.

And the word that never attaches to any of this output is “correct”. There is no gold standard for the right synthesis of thirty-five books, so Axial does not claim correctness. It claims auditability. Every record carries where it came from, what kind of statement it is, and how much confidence its coverage entitles it to, and code put all three there rather than a prompt asking for them. What comes out is governed data rather than model prose. Confidence is capped by evidence, not by tone. A claim in a finished paper can be walked back to the passage in the book, by hand, in under a minute, and that is the whole of what is claimed.

What I would like judged

The narrow claim is this. The LLM Wiki pattern holds at 35 sources, and compiling a corpus once and querying the compilation produces structure that query-time retrieval does not: 9,505 names shared across books, 447 stated disagreements, 43,101 cross-source opposition pairs, and an argument map whose relation labels were coined rather than chosen from a list. The parts that are unreliable are unreliable by measured amounts, and each is routed around rather than declared solved. I have not solved non-determinism. I measured it and built around it, which is the weaker-sounding claim and the only one the evidence supports.

All of it is reconstructible from the repository: the specifications, a 67-entry decision log with its reversals annotated in place rather than rewritten, the committed run logs, and a codebase whose docstrings carry the measurements that set its constants. The paper trail is the codebase, and that is the property I would most like to be judged on. The research report and the engineering report are where to start.

The instrument this project does not have is a person who knows the field. Three real research questions from your area, of the kind you would put to a doctoral student, would change what every number above is allowed to mean. One refereed reading of one paper and the passages it cites would be the first of its kind here. And if you can name the book this shelf is missing, that is one sentence from you and a gap I cannot close alone.