I’ve noticed something similar with coding agents. Specific rules seem to work better than simply saying “this is for AI.” I wonder if files like AGENTS.md work differently because the tools are designed to read from them first.
Indeed. Another thing worth exploring is how much the position of instructions matter; Claude claims to read docs as a whole unless they're very large, though it might be that it only grabs the first lines of a doc sometimes, in which case there is an "above the fold" effect.
Not sure, but if shared in the same docs as what we would read, it also allows us to skip over it with the assumption that it's a special set of instructions for AI.
Agreed on the first experiment, that's why the post says it hit the ceiling. The others are not saturated. I'd love to find a way of making the first experiment more discriminating.
Documentation is only useful for agents if the source is unavailable, and if you are trying to save on tokens and trust that the docs are accurate.
If none of that is relevant to you, or rather accuracy is your concern, it's much better just to search the source code for your answers, from how to set up the application to configuration questions to troubleshooting. Bonus points if you strip out all the comments programmatically via linting before you even let the agent view the code, to avoid possible context poisoning.
It also redirects the agents from adding enormous sloppy comments in the code - I have a linter that only allows 1 line comments with a reference to a docs folder tag `See BOB-IS-UNCLE.md`
Then you can ruthlessly prune the docs folder on PR review and regular passes when e.g. claude really wants to write a novel about the glorious intricacies of bugs it's solved.
I’ve noticed something similar with coding agents. Specific rules seem to work better than simply saying “this is for AI.” I wonder if files like AGENTS.md work differently because the tools are designed to read from them first.
Indeed. Another thing worth exploring is how much the position of instructions matter; Claude claims to read docs as a whole unless they're very large, though it might be that it only grabs the first lines of a doc sometimes, in which case there is an "above the fold" effect.
Not sure, but if shared in the same docs as what we would read, it also allows us to skip over it with the assumption that it's a special set of instructions for AI.
I think it's easier to serve those bits as Markdown to agents only through content negotiation.
The experiment in TFA is saturated so there's no signal there, unfortunately.
Agreed on the first experiment, that's why the post says it hit the ceiling. The others are not saturated. I'd love to find a way of making the first experiment more discriminating.
[flagged]
Documentation is only useful for agents if the source is unavailable, and if you are trying to save on tokens and trust that the docs are accurate.
If none of that is relevant to you, or rather accuracy is your concern, it's much better just to search the source code for your answers, from how to set up the application to configuration questions to troubleshooting. Bonus points if you strip out all the comments programmatically via linting before you even let the agent view the code, to avoid possible context poisoning.
Code doesn't explain best practices, architectural concerns, and so on. Best combo? Docs + code.
It also redirects the agents from adding enormous sloppy comments in the code - I have a linter that only allows 1 line comments with a reference to a docs folder tag `See BOB-IS-UNCLE.md`
Then you can ruthlessly prune the docs folder on PR review and regular passes when e.g. claude really wants to write a novel about the glorious intricacies of bugs it's solved.