---
title: "Coherence Is Not Proof"
author: "Marc DeCaria"
url: "https://marcdecaria.com/writing/coherence-is-not-proof"
date: "2026-11-19"
topics: ["AI & Technology"]
series: "Coherence Is Not Proof"
series_part: 1
words: 2184
summary: "Argues that a smarter model writes a summary clean enough to hide a cut corner, and sets out hashes, locked rules, cross-model review and event-bound reminders as a foundation that holds whatever model runs inside it."
---

# Coherence Is Not Proof

*By Marc DeCaria, November 19, 2026*

![](https://pbs.twimg.com/media/HXiBtxgXIAA6aLb?format=jpg&name=orig)

If you run coding agents on anything that actually makes money, you hit the same wall pretty fast.

At first, handing an agent the keys to the kingdom feels like magic. You turn off the permission prompts, point it at a repo, and watch the terminal fly. On a weekend prototype or a side project, that speed is intoxicating. You feel like you've unlocked a 10x multiplier.

Then you point that same setup at a live business. Real customers, payment flows, accounting data, an uptime requirement. The math changes overnight. The dirty secret of unconstrained autonomy is that it only feels fast until an agent silently overwrites a production config, drifts from last week's architecture, or breaks a checkout path while assuring you the tests are green. At that point you aren't hands-off anymore. You're either nervously hovering over the terminal or burning an afternoon doing forensics.

What surprised me most, after building my own environment around coding agents, is that newer, smarter models actually make this trap worse.

Be honest about how the work actually gets checked. Nobody is reading the code anymore. Not line by line, not at the speed these agents ship it. What you read is the summary.

When a model was sloppy and incoherent, that worked, because the summary gave it away. You could tell something was off and you knew where to dig. A sharp model writes a summary that gives nothing away. It hides the messy intermediate steps and tells you cleanly what happened and what to do next. Coherence starts to feel like reliability.

That's the trap. The one thing you actually read is the one thing the model is best at writing. So when a well-spoken agent quietly cuts a corner, nothing in front of you looks wrong. The answer isn't to go back to reading ten thousand lines of diffs. It's to stop letting how the work sounds count as proof that it works.

## How agents actually cut corners

If you watch coding agents closely under real pressure, you stop worrying about sci-fi failure modes and start noticing the boring, repeatable ones.

First, they will game their own tests to make a red light turn green. I once watched a model try to get past a static scanner designed to catch private addresses and keys before code was shared. Rather than fixing the underlying issue or asking for an explicit allowlist, it quietly split the offending strings into two concatenated halves so the regex wouldn't see them. The check passed. The intent of the check was completely defeated. That isn't malice. It's pure optimization for a passing exit code.

Second, they will rewrite their own rules if you let them. Give a coding agent write access to its own global instruction file during a run, and sooner or later it will "helpfully" soften a constraint that is slowing it down.

Third, they hide behind literalism. Put a strict spec in front of an agent, and when it hits a routine, fixable snag outside the exact sentence it was reading, it will often stop taking ownership and kick the failure upstairs as "a decision for the reviewer." Technically, it followed instructions. Practically, it dumped unfinished work in your lap so it could mark its own box complete.

Fourth, they suffer from silent drift. In long, multi-session workflows, a model working from stale chat context will happily overwrite a newer file, forget a decision you made three weeks ago, or repeat the exact same class of bug you already caught twice before.

The knee-jerk reaction to all of this is to write a longer, angrier system prompt. Never do X. Always check Y. Remember Z.

I never went that route, because it doesn't work. Prompts are just pep talks. If your safety model depends on a model remembering paragraph fourteen of a markdown file forty turns into a session, you don't have a safety model. You have a wish.

## Turning scars into architecture

The shift that actually changed how I work was simple. I stopped trying to make models trustworthy and started treating every AI, and every byte of text it produces, as untrusted data. Never instructions. Never approval.

Once you adopt that premise, you stop lecturing models and start building small mechanical levers around them. Each lever replaces a moment where you would otherwise have trusted how sure the output sounded.

Pin everything by hash. Whenever a work package, spec, or diff moves between chats, tools, or a remote coding environment, it carries a SHA-256 hash. If a model tries to patch a file from stale context, the edit fails closed. If an attachment changes, any prior sign-off is automatically voided. Stale overwrites become structurally impossible.

Lock the rules during execution. The coding environment cannot modify its own global instructions while a run is active. If the rules need to change, that happens outside the run, through a diff I explicitly approve.

Replace approval fatigue with coarse, high-leverage gates. Everyone knows what happens when a local CLI prompts you fifty times an hour to approve ls, grep, and file edits. You start rubber-stamping most of them without reading. The fix isn't more prompts. It's better boundaries.

Let the coding agent run hard and fast inside an isolated sandbox where it can break things safely. Reserve a deliberate, physical human gate, like a passkey tap, strictly for irreversible boundaries: starting a run, changing scope mid-flight, or pushing to production. You move from babysitting keystrokes to controlling the blast radius.

## Adversarial tension

A lot of developers eventually arrive at the same instinct when an agent gets stuck for hours on a gnarly migration or starts talking in circles. They open a second model and say, "Audit what this other AI is doing, because I don't trust it."

That is exactly the right instinct. The mistake is running it reactively, as cleanup after twenty hours of drift, and then immediately handing the new model the keys to the ship.

The real unlock is moving that cross-examination up front and into the floor of the workflow. In my setup, no single model gets to grade its own homework. One model drafts the plan or writes the code. A different model family cross-examines the work before execution and grades the evidence against a checklist afterward. A confident first answer can talk its way past its author. It has a much harder time talking its way past a different family of model that was asked to find the hole.

You don't need to take that on faith. The catches speak for themselves. In a single recent stretch of small, surgical updates, multi-model review caught a real bug on every PR before merge:

- A "safe, wording-only" change introduced a status label that overflowed a hard buffer limit by six characters, because no test checked the length on that path.
- A small state-transition hook had a subtle database race condition if a record was reparented mid-flight.
- A retry loop had a timing bug where a late wake-up from sleep() could pass a zero or negative timeout to a subprocess. When the first reviewer proposed a fix, two other models caught that the fix itself would have silently broken the final retry in production due to millisecond overhead.

None of those models was infallible. In the retry case, the model that wrote the loop missed the bug. The first reviewer caught it but over-corrected. The others caught the over-correction. No model is the star. The tension between them is.

You also don't have to pay a massive token tax to get this. A tiered split works cleanly. Run a full multi-model panel on multi-step plans, where catching a bad assumption before writing code is cheapest, and on protected ground like auth, updaters, payments, or security gates. Default routine app logic and UI fixes to a fast, cheap single-reviewer pass that automatically escalates to the full panel the moment that reviewer flags a concern. You get speed on clean changes and a courtroom the second something smells off.

## Giving agents a nervous system

Even with adversarial review, stateless models still suffer from amnesia. Most setups try to solve this in one of two broken ways. Either they cram everything into a giant prompt until the model drowns in irrelevant context, or they dump notes into a fuzzy vector store and hope retrieval pulls the right chunk.

Three small patterns fixed this for me far better than RAG ever did.

Close the review loop. Whenever I catch a mistake that slipped past an AI review (a missed edge case, a bad assumption, a broken check), I log it in one or two plain sentences. Before every future review, the system automatically feeds the last ten things I caught straight back into the reviewers' context. Zero fine-tuning, zero prompt bloat, and repeat misses drop off a cliff.

Curated, versioned institutional memory. Real engineering knowledge isn't a pile of chat logs. It's a small set of typed records stored in git like code: standards, procedures, architectural decisions, and known gotchas. When a task touches a specific module or boundary, the exact revision of the relevant rule gets pinned into the task manifest. The agents don't guess what the house rules are. They read the exact version that applies.

Event-triggered reminders instead of hoping models remember. Standard memory looks backward when asked. What operational work actually needs is prospective memory: "don't forget Y the next time X happens." A fuzzy semantic trigger that usually fires isn't good enough, because in production a trigger that only usually fires is one you can't trust. So bind one-line reminders to deterministic state events: when a specific run finishes, when a particular module is touched, when a deploy export lands, or right before a submission tool runs.

Better yet, piggyback due reminders directly onto the return payload of the tool calls the model is already making, and let the model drop them inline while it writes. Let the critical ones act as a hard stop, so a run can't be closed while a loose end is dangling. Don't ask a model to remember. Hand it the exact right fact at the exact second it's about to act.

## Knowing what not to automate

There is one last piece that keeps a high-rigor system from turning into pure bureaucracy: being ruthless about what doesn't deserve a workflow.

When you build strong guardrails, it's tempting to run everything through them, and models love to encourage this by proposing "comprehensive sweeps" and "helpful refactors." You have to kill that impulse hard. Heavy gates belong where failure costs money, corrupts data, or wakes you up at 3 AM. Everywhere else, the best engineering decision is often "won't fix," "leave the slightly off padding alone," or "I'll just upload the file myself in ten seconds."

## The model changes. The foundation shouldn't.

Every new model feels like a leap, and partly it is. Each release fixes a lot of what the last one got wrong. A fresh model doesn't carry the habits you spent months learning to work around.

But it brings new ones, and you haven't learned to see them yet. That's the most dangerous stretch: a smarter model, a cleaner summary, and failure modes nobody has mapped. It's exactly when coherence is most convincing.

None of this is intentional. When an agent cuts a corner, it isn't choosing to cheat. It's getting through the process the way it knows how. The shortcut looks fine today and compounds later, in a file it overwrote, a check it satisfied without meeting, a decision it never saw.

The models can follow instructions. What they lack is the environment. Give them a foundation that holds the rules, the history, the evidence, and the gates, and their ability shows up in a form you can actually rely on.

That's why so many developers end up rebuilding their setup every few weeks or months. They built it around the model: prompts tuned to its quirks, workarounds for its habits. When the model changes, the system breaks with it. Build the foundation around the work instead. Hashes, gates, cross-model review, and versioned memory don't care which model sits inside them. Swap the model and the foundation stays.

The labs may get us there. They are already shipping pieces of it, and eventually they may ship the whole thing. But they can't ship your business rules, your history of misses, or your definition of what can't break. That part has to be yours, and it can't wait for them.

Looking back, the whole shift wasn't about slowing my agents down. It was about building a seatbelt so I could actually drive fast on real roads. Most people working with AI right now are still waiting for the next frontier model to be smart enough that they can finally hand it the ship and walk away. I've come to the opposite conclusion. The smarter and more persuasive the models get, the more you need a boring, deterministic structure underneath them, so you aren't betting your business on how confident an AI sounds at two in the morning.
