Measure, don’t assume: seven assumptions that turned out wrong while building a small coding agent

What is the project?

The sentence I kept running into while reading about agentic coding was: Agent = Model + Harness. The model is the LLM that takes text in and produces text. The harness is everything around it: which tools are described to the model, how the context is built, where state lives, when the loop stops. The model doesn’t execute anything; it only asks. The code that reads the file, runs the test and returns the result lives in the harness.

Note · The starting point for this experiment was the SWE-agent team’s mini-swe-agent [1], which showed how far a small, readable agent loop can go [2].

I suspected there was a gap between reading that sentence and understanding it, so I decided to write the harness myself. I set a few rules:

• No agent framework. I’ll look at tools like LangGraph or Google ADK once my own gaps show me why they exist.
• No LLM SDK. I talk to the provider over raw HTTP; the only runtime dependency is httpx.
• Model: Gemini Flash-Lite, free tier. My local GPU wasn’t available at the time, and that shouldn’t block the project.
• Target repo: tomlkit, pinned to a fixed commit. Zero runtime dependencies, 1,052 tests, runs in ~1.3 seconds.
• Everything is recorded. Every model call, every tool call and every result is written line by line to a JSONL trace that can be read back losslessly later.

The way I work is part of the experiment too: Claude Code writes the code, I read it and push back; I prepare the design questions and decisions for each step in a separate Claude session. Some of the wrong guesses below are mine, some are the two assistants’. What’s interesting is that all three of us made the same kind of mistake.

This way of working can be seen as an early, manual form of the Project Historian idea I plan to add to the agent later. Decisions are written in a separate session before the code, standing decisions are kept in a table, and each step is handed over to the next through a status document. It isn’t a full implementation. But some of the lessons below show where this setup holds and where it doesn’t.

1“I’ll read the docs and write the provider”

The first job was the class that sends requests to the Gemini API. Before writing it, I sent raw requests and measured what the protocol actually returns. Had I written it from the documentation, I would have started with four silent bugs:

• thoughtsTokenCount isn’t zero on Flash-Lite; it’s absent. Code that reads usage["thoughtsTokenCount"] would have raised a KeyError on the first call.
• Thinking tokens are not included in candidatesTokenCount, but they are included in totalTokenCount. I would have undercounted output tokens by 3–4×, and cost is one of the main metrics in this project.
• finishReason is "STOP" even on a response that contains a tool call. A loop that answers “did the model ask for a tool?” by looking at finishReason would never see a single tool call.
• thoughtSignature. An opaque signature of the model’s internal reasoning, delivered inside the same structure as the tool call. It can’t be read, generated or verified; it can only be sent back verbatim in the next request. Code that converts the response into a clean internal format and rebuilds it later would silently drop this field.

The class written after measuring worked with zero surprises on its first contact with the real API. I learned one more thing on the way: tool calling isn’t a “capability” of a model. It’s a contract opened by the tools block in the request, and the envelope of that contract changes from provider to provider. Gemini returns functionCall, OpenAI tool_calls, Anthropic tool_use. Serve the same open model raw and you parse the tags yourself. The format comes from the serving layer, not the weights.

2“One run will show me the effect of a change”

Once the agent managed to answer a question about a real codebase correctly, I wanted to see how much one sentence in the system prompt affected its behaviour. I removed the sentence and ran the same task again. Same task, nearly the same setup, four runs:

A 2.8× cost spread, and a quarter of the runs with no answer. Had I compared only the baseline with ablation 2, I would have concluded that the sentence increased cost by 2.4×. The only thing this data says is that an n=1 comparison says nothing.

The reason is simple: LLMs aren’t deterministic, and an agent doesn’t absorb that randomness, it amplifies it. A small difference in turn one means an entirely different path by turn ten. I measured it again later on write tasks: on a task where all three runs succeeded, the spread was 2.6×.

The same table held a second trap: “finished” doesn’t mean success. Two runs ended with an answer but found only one of the three correct locations. The cheap runs were cheap because they did less work. An unconditional “tokens per episode” ranking would have put the fully correct answer last.

This table changed the rest of the plan: many runs per arm, graded outcomes (no answer / partial / full), cost conditioned on outcome quality, and bug-fix tasks verified by tests as the main measurement.

3“A decision I’ve written down is a decision implemented”

The data structure representing a model response had retries and retry_wait_ms fields. The decision was clear: how many times each call was retried would be written to the trace. But the code that wrote the trace never wrote those two fields. The tool that read the trace read them with .get(..., 0), so the screen showed “0 retries”.

The absence of a measurement looked like a measurement. What hid the bug was a defensive line written to be “safe”.

This pattern repeated many times. The principle “an unreported value is not zero” was written in one file and violated in another. The code that returned the repo to a clean state after each task wasn’t inside a finally; the first crash left the pinned repo modified. The same code also wiped the agent’s change when a task finished. That was the most informative output of a bug-fix episode, and once it survived only by luck. In the words of its own report: “the guardrail existed; half of it didn’t.”

The fix was simple. I added a column to the table where I keep the project’s standing decisions: “where is this decision enforced?” Any row where you can’t write a test name or a line of code is a half-finished decision.

4“A lesson I’ve recorded is a lesson learned”

In tests, I record real API responses and match them by request hash (these recordings are called cassettes). At almost every step someone predicted “this change will invalidate the recordings”, and it didn’t. The same wrong prediction was made a third time after being corrected twice.

The correction was written down somewhere: in the learning log. But that file wasn’t read when the plan for the next step was written. The record existed; it wasn’t retrieved.

In another case, the rule was in the file read every session. It was retrieved, but only half-applied in the code. That’s a failure at a different gate.

Putting the two side by side produced a taxonomy:

Each gate can fail on its own. That matters, because the second half of the project is adding memory to the agent. A memory experiment that measures only the first gate (“did it save this?”) will report a system that doesn’t work as “working”. The right question is: would a query the agent could reasonably make retrieve this information, and when it did, did the behaviour change?

My practical fix for myself was to add a “known false beliefs” section to the single status document I regenerate at every step. I moved the correction to the place that actually gets read.

5“A good summary can stand in for the source”

In the four-run table above, ablation 2 was marked “partial”. That label was carried through several discussions without being questioned and started to be read as “wrong answer”. At one point I asked, “but isn’t this the same output?” Looking at the answers one by one made it clear: ablation 2 was correct, just narrow. It found the Trivia class, responsible for preserving comment lines, with the right file and line; it missed the Comment class and the related function in the parser.

The label survived because it was cheaper than opening the source. Yet the source was one command away; the tool I wrote to read traces losslessly existed precisely for this.

The lesson I drew: the dangerous summary isn’t the bad one, it’s the good one. A good summary is usually right, so it’s trusted, so it replaces the source. The same incident showed me something else: a human who has to grade 160 episodes consistently is not a metric. That became the third reason to move to test-verified tasks.

It also gave me a prediction for the memory experiment: in the memory arm, a drop in read_file / search_code calls is a warning sign, not efficiency, until it’s matched by a correctness measurement.

6“The agent’s first problem will be bad edits”

I gave the agent write capabilities: edit_file, which replaces exact text in a file; run_tests, which runs the tests and returns the names of failing ones; read-only git commands; and an allowlisted shell. The task type changed too. From tomlkit’s history I pick commits that changed both source and test, revert the source change, keep the test, and tell the agent “make this test pass”.

Before running, I predicted five failure modes for edit_file: whitespace/indentation mismatch, multiple matches, hallucinating text that doesn’t exist, and similar. Then I ran three tasks three times each:

On the easy task the path was the same every time, and sensible: run the tests, get the failing test’s name, find the test, find the source, read it, make one edit, run the tests again, check its own change with git diff.

In the other six episodes, the agent read and searched for 22–25 turns and didn’t make a single edit. None of the five failure modes I predicted could happen, because there were no edits. I was asking the wrong question. The real question wasn’t “is it editing correctly?” but “does it decide to edit at all?” That’s the same mistake as in item 4: designing to measure the next gate without checking whether the gate before it exists.

I tested two cheap explanations with one run each:

• Did it run out of room? I raised the turn limit from 25 to 40. The extra 15 turns became 15 more turns of the same search-read-search-read oscillation. This was the most expensive episode in the project (366,359 tokens), and still zero edits.
• Does it not know its budget? I added a line like “turn 7/25” to the context every turn. It found the buggy function in turn 7 and spent the remaining 18 turns still searching.

Then I read the traces one by one. In four of the six failures, the agent had found and read the buggy function, but didn’t edit it. One had that function on screen at turn 4 and searched for 21 more turns. The remaining two were the same task: the test’s name said “dotted inline table”, the agent stayed in the parser for 25 turns, and the bug was in Container.append.

The difficulty labels I’d given the tasks collapsed as well. I had labelled them by the size of the source change (4, 9 and 24 lines). That held at the easy/not-easy boundary and predicted nothing between medium and hard. A “how many steps from the failing test’s name to the buggy function?” measure also gave the wrong order. My current hypothesis: what separates them is whether the test’s language points to the right module. It’s an observation from three tasks; not a finding, a falsifiable hypothesis.

7“My measuring instrument is reliable”

While the repo was being cleaned between tasks, Claude Code noticed something: Python’s bytecode cache (.pyc) checks whether a source file is up to date using its second-resolution modification time and size in bytes. If a - b becomes a + b within the same second, neither changes, the old bytecode runs, and the agent sees its own correct fix fail. From the outside, that’s indistinguishable from “an agent that can’t fix the bug”.

I measured the mechanism separately:

natural race in a tight loop      → stale bytecode ran in 198 of 200 trials
PYTHONDONTWRITEBYTECODE=1         → STILL stale bytecode
after deleting __pycache__        → correct result

The first fix that comes to mind doesn’t work. PYTHONDONTWRITEBYTECODE=1 only prevents writing new .pyc files; it doesn’t prevent reading the stale one that already exists. The right fix is to delete the __pycache__ directories before every task.

The second half of the story was just as instructive. The first reaction was “then all the numbers we collected are dirty.” Before deleting and rerunning, we looked at the timestamps in the traces: the edits had been made 29, 41 and 201 seconds after the repo reset; the other six episodes had no edits at all. The data was clean. You have to measure before you throw data away, too.

Open question: harness or model?

Two cheap explanations are out. Two candidates remain:

1. The harness never asks for a decision. The loop lets the agent search and read forever. Each individual call is reasonable on its own, and no stage asks it to commit to a hypothesis.
2. The model can’t do this task. Flash-Lite is a small, cheap model.

Both fit the same data, and all eleven episodes run so far used the same model. With one model, the two can’t be told apart. So the next step asks a single question: same task, same harness, a stronger model. Keeping the model layer behind an interface since the start of the project was for exactly this kind of question.

Either answer is useful. If the stronger model doesn’t edit either, I’ll have measured with two models that “without a decision point in the harness, the model doesn’t decide, whichever model it is.” If it does, the experiment’s first decision becomes the choice of model. You can’t measure the effect of memory with a model that can’t solve two thirds of the tasks, because there’s no range left to measure.

There’s an obvious fix too: write “form a hypothesis first, then edit, then test” in the system prompt. It would probably work, and that’s exactly why I’m not doing it for now. Then what I’d be measuring would be the prompt, not the agent. That’s a separate decision and a separate arm of the experiment.

What’s next?

After this step I’ll add memory to the agent, along with a component I call Project Historian. The Historian is a layer that carries the decisions and history of a project across tasks. There is prior work on similar ideas; my contribution isn’t the idea itself but measuring it on a working agent. Then I’ll compare the agent with and without memory on the same tasks.

I ran into the failures in items 3 and 4 while applying the Historian to my own way of working by hand: the decision was written down but not enforced; the lesson was recorded but not retrieved. Those two gates are exactly what the Historian added to the agent will need to measure.

Some of the findings in this note have already become inputs to that experiment. For example, “in this repo, test names point to the wrong place for the bug” is exactly the kind of project knowledge a memory system should carry. The three-gate taxonomy above also determines how that experiment will be evaluated.

What I’ve learned so far

• Reasonable predictions about system behaviour were frequently wrong in this project. I can’t give a rate. The wrong ones were recorded because they cost something; the right ones mostly weren’t written down, so my sample isn’t clean. But the direction is clear: the only reliable filter is running the measurement.
• The agent amplifies the model’s uncertainty. A single-run comparison tells you nothing.
• The most valuable code in the project isn’t the agent, it’s the trace. Almost every lesson above was possible because I could answer “what happened?” with a record rather than a guess.
• Guardrails must be in place before the run, not after. And you need to check that they’re enforced, not just that they exist.
• The hard part of memory isn’t storing, it’s retrieving and applying. I learned that in my own way of working first.

The project is ongoing. When the harness-or-model question is answered, I’ll update the last section of this note.