Creating rules that stick

This keeps happening to me. I'm using Claude Code, Cursor, Codex, whatever. The agent writes some code. I look at it and ask, "Hey, did you apply the rule I gave you?"

"You're right to point that out. I did not."

Cool. So why did I write it?

I wanted to understand this a little better. I went down a rabbit hole of rule files, context windows, and verification. Here's what I've learned, and what has helped me get fewer of those "you're right to point this out" moments.

What I mean by rules

Mostly two things:

1. AGENTS.md and CLAUDE.md files with general project instructions.
2. Scoped rules, like Cursor's .cursor/rules/*.mdc files, Claude Code's .claude/rules/*.md files, or files people keep in .agents/rules/.

I'll leave skills aside for now. They're markdown too, and they share some of these problems, but that's another conversation.

Problems with rules

1. Every tool handles them a little differently

In Cursor, a rule can apply to every conversation, load when a file matches a pattern, or wait for you to call it manually. You can also let the agent decide when it needs the rule.

Claude Code and Codex have their own ways of loading rules. Which means different locations, different metadata and different triggers.

In a team, where everyone uses a different coding tool, it’s a bit of a risk. You can share the same repo and still get different behavior from the same instructions. It’s a bit of a Wild West right now, but I believe things will get better.

2. Competing for attention

I like Dex Horthy's way of describing a context window as having a "smart zone" and a "dumb zone." The fuller it gets, the harder a time your agent has.

A rule buried in a long conversation has to compete with everything else in there, like tool results, code, old plans, things you asked for an hour ago and then changed your mind about.

When I first started writing rules, I assumed that getting them into context was enough. Apparently not. The file can be loaded and still do absolutely nothing for your task.

This paper found a U-shaped attention bias in the models it studied. Information at the beginning and end got more attention. Relevant information in the middle was harder to use. So where the rule sits actually matters.

3. Rules fighting each other

In a large context window, rules can conflict.

Here's a simplified example. A frontend rule says:

And a testing rule says:

If the agent reads these two rules, it now has to resolve the problem on its own. Some models ask you, but in my experience, it often just decides for you. It’s eager to finish the task. Which honestly I think is a good design, otherwise I’d imagine it would be pretty hard to get anything done if it stopped at every ambiguity.

Another rule fight can occur when team rules try to override user rules. Cursor and Claude Code offer a rule hierarchy, which sounds amazing, but practically is pretty easy to break. It's still just guidance to a model, and a team member can override it. I wouldn't treat a markdown file instruction as a guarantee.

4. Compaction

When a conversation gets too long, or you run something like /compact, the tool compresses it.

I used to think that compaction is basically a shorter text summary, but found out that it is actually a bit more complicated as this article about Codex shows. Claude Code re-reads the project-root CLAUDE.md after compaction, but nested files and path-scoped rules reload only when it reads files they apply to.

5. Maintenance

One thing I’ve been following with my eyebrows raised was how quickly we decided it is a good idea to keep so many markdown files in the codebase.

Before AI, a project would have one outdated README at best. Now each developer adds 20 markdown files full of instructions to the codebase each month. But the maintenance of these is largely unsolved, which makes all of the problems above even worse. You cannot refactor them the same way you could with code.

Still, I don't think we should throw them all away.

Solving the problem

What you put in the file matters

If you're as online as I am, you probably noticed a wave of advice to delete your CLAUDE.md and AGENTS.md files because they hurt agent performance. I wrote about that discussion before, but since then the research continued, and the results are mixed.

The ETH Zurich study, Evaluating AGENTS.md, found that context files didn't generally improve task success and increased inference costs by over 20% on average. It also found that agents followed the instructions well. In those experiments, doing what the file said wasn't necessarily helping with the task.

Lulla and colleagues looked at 10 repositories and 124 pull requests. They found context files were associated with 28.6% lower median runtime and 16.6% fewer output tokens, with comparable task completion behavior.

And in an Agentic AI Foundation experiment, repeating the runs changed the conclusion you'd draw from a single comparison.

Safe to say we don’t have a universal answer yet.

But from what we know, I think we can gather some signals. I personally lean toward keeping the files, but I'd write them by hand and keep out any extra fluff.

Things like "This project uses React." or "run tests using npx playwright test command" are things that the agent will probably figure out when it reads your package.json.

The AGENTS.md file should answer the question "What does it need to know that it can't easily find there?"

For example your team has a particular way to seed a database. There might be 5 ways to do it, but you chose one for a reason. Maybe that reason isn’t even traceable from the particular project.

Verification as a separate step

I like to think about code generation and code verification as two independent steps.

There’s temptation in thinking that a good enough model can generate everything perfectly, but I think that’s actually false. Better models will not solve this. Imagine a model that is so incredibly perfect that it writes 0 bugs on every task.

But zero bugs is not enough. Consider the difference in scope of a single Linear ticket vs. the scope of integrating that code into a larger system. Just think of your last bug - it was probably something that wasn’t visible on a diff, rather a distantly related service. Looking back, it always makes sense. Looking forward it’s more like "who would have thought?" Verification is arguably a harder problem to solve, because the context needed for verification requires covering a much larger surface area.

But I’m getting side-tracked. Back to rules.

Agents practically never follow instructions 100%. And as we established earlier, loaded in your chat does not mean enforced, which means we need to check back. Asking "Did you apply the rule?" or "Did you use the skill?" is a good start, but it’s tedious and we’re not prompt engineers anymore.

Hooks, loops and factories

Naturally, we want to introduce some sort of automation into the verification process. Claude Code's hooks guide offers some basic examples. While hooks offer a pretty good starting point for verifying agent outputs, they are still subject to context saturation problems. There’s also an issue with not just having the same conversation, but the same model, that will be subject to the same blind spots.

A smarter way to wire this up is through a loop. One agent does the work, another reviews it, and the first fixes what's missing. Repeat until the work meets the goal. Many harnesses now offer /goal mode which is basically this.

Software factories are the same idea, although more complex. More agents, tickets, docs, connected repositories. Sounds nice. But verification is still hard, arguably harder in complex systems like a software factory. Making sure that not just your rules stick, but your quality is kept and standards don’t drift is a challenge.

PR checks

The good news is that verification is part of many teams' workflows already, in the form of PR checks. We generate code, open up a pull request, get feedback, and fix what comes back.

I’ve been using it for my personal projects and it works great for me as well as our clients at Qodo. Sure I may be a bit biased, but especially when it comes to rules, Qodo does a great job checking if the proposed change actually sticks to proposed rules. All of the issues mentioned above are mitigated by an independent set of checks.

And while it’s a good approach, I definitely think there’s a way to do even better. Doing a check at PR might already be a bit too late.

That's why I think an even tighter feedback loop works better. Move the check earlier, closer to the work that needs checking. It's the same idea we've called "shift left" for years. And we can take it a few steps further with agents.

Shift left with the Agentic Toolbox

The first step is to review the code before opening the PR. Qodo's Agentic Toolbox runs the same set of checks you’d get on a PR in earlier stages of development. The agent can get feedback and fix issues while the work is still in the coding session, before I push anything.

But we don't have to wait for the whole feature to be done either. We can build checks into the workflow as the agent writes code. Finish a piece, run the relevant tests, call Qodo to review it, and fix what comes back before continuing. I'd rather catch a rule violation before the agent copies the same pattern into five more places.

Then we can move left again and check the agent’s plan. Say I'm changing a login service, but the dashboard depends on the response format I'm about to change. I want the agent to catch that before writing code. Qodo's codebase context can help it investigate connected repos and previous decisions, then check its proposed approach against those dependencies and our rules.

And one step earlier is the ticket itself. If that login ticket never mentions the dashboard, the agent can follow it perfectly and still break something. We can ask it to use the same toolbox to investigate the affected services and compare what it finds with the requirements. Then we can add the missing compatibility requirement before asking for an implementation plan. Then, based on that analysis, you can load the rules that are relevant for the whole change as it integrates into the system.

Making rules stick

The fact of the matter is that there’s no way we can make rules work 100% for when a coding agent is working on a task. Some improvements can be made, and thanks to the power of models and coding harnesses, those don’t need to be too drastic. But in order to ensure that rules are followed, we need to verify them. A separate, ideally independent verification step needs to be a part of every delivery system.