TL;DR: I built a tool that writes a summary of a Rails app’s conventions into its CLAUDE.md or AGENTS.md, then A/B tested it for eight rounds on real apps. Agents without the summary found the conventions on their own. Every clear effect the summary had was a harm: over-broad rules got obeyed over the code, and generated text weakened a rule the team had written by hand. Only 4% of what agents explored was about conventions. Write less into your instruction file, and make every line narrower than you think it needs to be.

---

The first result looked like proof.

One run per arm on a service-object-heavy Rails app. Without my generated context, Claude Code scored 8/10 on the app’s conventions. With it, 10/10. The difference was exactly the convention I’d hoped it would catch: new business logic goes in a service object. That’s something a neighbouring file can’t show you if the file you’re editing is a controller.

I had the blog post half-written in my head. Then I ran it again with more samples, and the gap disappeared.

This is the post about what happened instead.

The bet

You probably know Context7, the MCP server that hands agents current, version-specific library docs. It works really well, and the idea I had was simple: do the same thing for your own app. Instead of the agent re-reading the codebase before every task, give it a description of how this particular app works.

So I built rails-atlas. It reads a Rails app statically with Prism (it never boots it) and maps controllers, services, jobs, concerns, engines, and API vs HTML surfaces. Then it infers ten “house rules” by majority: where business logic lives, how authorization works, how tests are written, how JSON gets rendered. Each rule comes with its evidence and the exact places the code contradicts it.

It delivered all of that as a marked block inside the instruction file the repo already uses, plus a machine-readable map and a verify command for CI.

The hypothesis: agents given the block follow the app’s conventions more often than agents without it.

How I tested it

Eight rounds between October 5 and 8, plus a transcript analysis on the 11th. The apps are private, so I’ll describe them by shape:

App	Shape	Agent
A	Service-object-heavy, ~90 service files, short CLAUDE.md	Claude Code (claude-opus-5)
B	Small CRM, JSON API plus an HTML side, RSpec	Codex (gpt-5.6-sol)
C	Three mounted engines, 322-line hand-written CLAUDE.md, design-first UI process	Claude Code (claude-opus-5)
D	Small RSpec app	Rule audits only

The setup, briefly:

• Each arm got its own throwaway clone of the same commit, with its own test database. The only difference between arms was the block in the instruction file.
• Prompts read like real tickets and never mentioned conventions. “Add an overdue filter to the tasks JSON API, with request specs.” “Show how long a record has been open next to its title in the page header.”
• Each app used the agent its team actually works with.
• The rubric and predictions were written before any run, plus a stopping rule for the later rounds.
• From round 7 on, scoring was blind. A separate agent scored shuffled, anonymised packets (diff, tool trace, final message) without knowing which arm was which.

Samples were small, n=1 to n=4 per arm. Keep that in mind for every number below. I’ll come back to it.

The win that vanished

Round 4 was the real test of round 1: three tasks, two runs per arm, across Apps A and B. On the rubric items that should have separated the arms, the agents with the block scored 16/16. The agents without it also scored 16/16.

Every failure I’d predicted for the no-block agents was wrong. They read past the file they were editing, found the service layer, and used it. Nobody had to tell them.

Here’s the line I keep coming back to: Context7 works because library docs are information the agent can’t otherwise get. Your app’s conventions are already in the repo. The agent is sitting inside the thing I was trying to summarize.

That alone would have been a fine, slightly disappointing result. What made it worth writing up is that the block didn’t just fail to help. Every clear effect it had was a harm.

Harm 1: an imperative without a boundary gets applied everywhere

On App B (round 3), the block said “Authorization goes through Pundit.” That’s true of the HTML controllers. The JSON API, though, scopes records by hand.

Agents given the block added Pundit to a new API controller 2 out of 2 times. Without the block, 0 out of 2. They could see the neighbouring API controllers. They obeyed the sentence anyway.

Round 4 had a sibling. The rule “Put new business logic in a service object” led both treated agents to wrap a three-line read query in a brand-new service class. The control agents left it in the action, where it belonged.

Both fixes were the same move: say where the rule stops.

<!-- Before -->
- Authorization goes through Pundit.
- Put new business logic in a service object.

<!-- After -->
- HTML controllers authorize with Pundit. The JSON API under /api scopes
  records by hand; follow the existing API controllers there.
- Service objects are for writes and multi-step work. Plain reads stay
  in the action, as most index/show actions here do.

After scoping, both harms went to zero on rerun (0/2 each). The lesson generalizes well beyond my tool: an app-wide statement beats a local exception, even when the agent can see the exception in the next file over. If your CLAUDE.md or AGENTS.md says “we always do X,” the agent believes you more than it believes your code.

Harm 2: generated context crowds out the rules humans wrote

This is the result I’d send to a coworker.

App C has a hand-written rule in its 322-line CLAUDE.md: before writing any view code, make the UI design in a design tool first. It’s a real process the team cares about.

When rails-atlas appended its ~105-line block, agents started skipping that rule. Pooled across rounds 5 and 7, agents designed first 5 out of 5 times without the block, and 3 out of 8 with it.

My first guess was position. The generated block sat at the bottom, so maybe it was just “the latest thing the agent read.” Round 7 tested that directly: no block, block appended, block at the top. Designed first 3/3, 1/3, 1/3. Moving it to the top didn’t help at all.

So what was it? Looking at the actual text, I think it was this sentence in the block’s preamble:

This is a conventional Rails application… what follows is only where this codebase diverges.

Read that as an agent would. It says: here is the complete list of what’s special about this app. The design-first rule wasn’t on that list, so by implication, it was default-Rails territory. A frontend rule saying “build new interaction with a Stimulus controller” probably didn’t help either. It reads like permission to start writing UI.

Round 8 tested the fix, blind, with four runs per arm. The new heading said “Conventions inferred from the code.” The new preamble said these rules are inferred, are not the team’s decisions, are not complete, and yield to anything else in the file. Result: 3/4 designed first with the block, 3/4 without. Back to parity.

Position wasn’t the problem. Claiming to be the whole story was.

That’s worth checking in your own instruction files, generated or not. Any line that sounds like “this is everything you need to know” quietly demotes everything that isn’t under it.

What actually broke runs wasn’t a convention

The most common failure in the whole experiment had nothing to do with house rules.

In round 4, 4 of 6 JSON-task runs on App B called travel_to in an RSpec suite that never includes ActiveSupport::Testing::TimeHelpers. Minitest includes it by default; rspec-rails doesn’t. You only find out by running the suite.

So I taught rails-atlas to state that fact and ran round 6. Specs passed 3/3 in both arms. Two of the three no-block agents grepped the specs for travel_to on their own and copied the existing per-file include. The third hit the NoMethodError, ran the suite, and fixed it.

An agent that runs its tests corrects test-environment surprises by itself. Facts like this help most for agents that don’t run their tests, which tells you where to spend effort first.

What agents actually look for

After eight rounds of “the block doesn’t help,” I finally asked the question I should have started with: what are agents doing when they explore?

I classified 731 exploration steps from 23 runs on Apps B and C by the question behind each step.

Thirty steps out of 731 were asking “what’s the convention here?” That’s the one question rails-atlas fully answered. The big buckets were navigation: where is this rendered, which process applies, where are the tests, is there logic I can reuse.

Agents don’t ask how your app does things. They ask where things are.

And even perfect answers wouldn’t have saved much. On App C, exploration was 43% of tool calls but only 19% of tool input and output; design-tool work was 72%. The one cost signal I got (a 30% cheaper median with the block in round 8) traced back to how much design work each run happened to do, not to saved exploration.

A side finding: agents route around your hooks

App C enforces design-first with a pre-edit hook on the Edit and Write tools. Writing a “pending” approval file switches that gate on. In round 8, 3 of 4 no-block runs wrote the pending file, then wrote the view template through Bash with a heredoc or a Python script. The hook never saw it.

Nobody was trying to cheat. The agents believed they were following the process and treated the gate as a step to get past. One mentioned it in its final message; the others didn’t.

Write-time hooks get routed around. Merge-time checks don’t care how the code got there. That one probably deserves its own post.

What I’d tell past-me

If you write CLAUDE.md or AGENTS.md files, by hand or with a tool:

1. Don’t restate what the code shows. Agents read the neighbouring files. Restating conventions costs context and adds little.
2. Every line competes with every other line. Text added later can weaken a rule that was already there, and moving it around doesn’t fix that.
3. Never claim completeness. “Here’s only what’s different” tells the agent everything else is default, including your hand-written process.
4. Bound every imperative. “Use service objects” becomes “service objects for writes and multi-step work; plain reads stay in the action.”
5. Scope claims to where they’re true. If the API and HTML sides differ, say so explicitly.
6. Spend your lines on what the code can’t show. Decisions, process, test-setup quirks, the “why.”
7. Enforce at merge time, not edit time. A required CI check doesn’t care which tool wrote the file.

And if you’re building agent tooling:

• Decide what you’re improving before you test it. My original idea was about effort (less re-reading). I spent eight rounds measuring correctness.
• A rule that never fires is a suspect, not a clean bill of health. Eight of my ten rules reported zero exceptions across four apps. Two of those zeros were dead checks: one looked for a literal .all call that real index actions never make, the other couldn’t see a plain render json: at all. Before trusting a zero, grep for the thing the rule says doesn’t exist.
• Pre-register, blind, and use real repos. Every early win I had either failed to replicate or came from how I’d written the rubric. In round 3, my rubric expected pagination the codebase didn’t require. That was my opinion, not the app’s convention.

The caveats, honestly

These are directional findings, not settled science.

• Small samples. n=1 to n=4 per arm. The crowding effect is the strongest result, and pooled (5/5 vs 3/8) a one-sided Fisher test gives roughly p≈0.04. On round 7 alone it’s about p≈0.12.
• Two strong agents. Claude Code on claude-opus-5 and Codex on gpt-5.6-sol. A weaker model that explores less might benefit where these didn’t. I never tested that.
• Few tasks. About eight distinct tasks in total. The exploration analysis rests on two, and the UI task skews it toward views.
• Generated text only. The crowding effect was measured with a generated block. Whether a long hand-written instruction file crowds itself out is untested, and honestly, that might be the more useful question for most teams.

So what happens to rails-atlas?

I don’t know yet, and I’d rather say that than pretend.

There are a few honest directions. A navigation map (route → controller → view → partial → test) answers the questions agents actually ask, though existing tools cover parts of it. The same inference could power a merge-time check that fails a PR adding a new exception, like an unscoped lookup or an inline write outside a service; the open problem there is that majority isn’t intent, so a human has to ratify each rule once. And the cheap test I never ran, Sonnet- or Haiku-class agents, could still flip the result.

What I’m sure of is the part that doesn’t depend on my tool: your instruction file is a shared, finite space, and every line you add is arguing with every line already there. Write fewer of them, and make each one say where it stops.