Documentation performance, humans and LLMs

All coding harnesses I’ve used (Claude Code, OpenCode or Pi) give you a way to read your current session’s token spend.

Session compacted 13 times

 Session Info

 File: /home/me/.pi/agent/sessions/path/file.jsonl
 ID: 01a013cf-f306-7b76-994a-028aefc7a999

 Messages
 Total: 593
 User: 4
 Assistant: 285
 Tools: 304 calls, 304 results

 Tokens
 Input: 12,105,934
   Cached: 11,131,164 (91.9%)
   Uncached: 974,770
 Output: 290,380
 Total: 12,396,314

 Cost
 Total: $0.000
   llama.cpp/Qwen3.8-27B-Q4_K_M Text-only: $0.000 (12M tokens)
   Tools/summaries: $0.000 (375k tokens)
 Cache Re-billed: 2,262 tokens, 1 miss

Pi’s /session statistics for a recent session.

Of all the capabilities that LLMs enable, I find this ability to quantify thinking remarkable and not really tapped into yet. You — as a human — have always been able to tell whether doing any thinking was hard or easy. Usually, we proxy this quantity with the amount of time spent thinking. However, many everyday cognitive activities don’t reduce themselves to simple problems, and it becomes hard to otherwise measure them. Reading and understanding documentation is one of these activities.

The documentation performance metric

#

In the age of LLMs, you can effectively measure how efficient your documentation is at answering a given set of questions, regardless of the question’s complexity; let’s call this documentation performance (I don’t know of any other term for this).

1. Take 2 (or more) versions of your documentation,
2. Define a set of questions that you want to optimize against,
3. Run your models to answer test questions,
4. Sum input and output tokens used to generate all answers,
5. Run your models to grade the answers.

To account for the difference in answers generated or variation in grades across repeated calls you can even take multiple samples. Dividing the sum of I/O tokens used to generate the answer by the grade of the answer gives you the documentation performance score. Lower is better:

In the example above, version B is better than version A because the model gets equally good answers with half the effort. Version C isn’t necessarily better merely because it is short.

Running this pipeline may be costly today, but many software companies would likely agree that a well-written documentation is a competitive edge. I don’t know of any organization making use of this yet, but I would not be surprised if some already do or have attempted to do so.

Beware of Goodhart’s Law

#

Tuning for documentation performance could be a good thing as long as documentation is written primarily by and for humans. If you build a system where LLMs are both authors and reviewers with the goal of optimizing for documentation performance alone — disregarding any other metrics and without any human in the loop — the output will likely end up resembling this:

- [done] Path-scheme Feature references (COMMANDS.md *Feature Reference Resolution*): `file://` / relative-path entries resolve to a Feature folder already on disk (trusted, no network) — 205 tests green. Design: any reference that is neither a bare name nor a registry-qualified OCI reference is a path reference; a `file://` prefix is stripped and the remainder is used literally (no percent-decoding, v0.1); relative paths resolve against the workspace root, `.`/empty = the workspace root itself; backslashes normalize to `/`; the result is lexically normalized (a `..` escaping the root is kept); the folder must exist and hold a parseable `devcontainer-feature.json`, but the folder name need not match the manifest `id` (the path names the folder directly; the build uses the manifest `id`); plans as `FeaturePlan::Local` with a new `FeatureSource::Path`, no registry gate, no `--allow-remote-features`; manifest loading shared with the search path via `load_feature_manifest`. Review passed: spec table rows (`file://`, relative) match, trust/no-network verified by a non-interactive no-allowlist test, `..`-escape kept not dropped, id≠folder-name rule tested, empty last component on an OCI ref is a hard error; one doc nit fixed (`is_bare_feature_name` comment mislabeled `.` as `..`-containing)

A billboard note written by the agent during that session.

That output is a single line of my agents’ billboard and at 1300 characters, it’s one of the shortest. The agents use the billboard to communicate with each other. This is what documentation looks like when agents talk to agents. There is definitely structure, but a complete absence of formatting, description or justification. This is clearly information dense and would likely do great on documentation performance, but I don’t think it’ll be a read that any human would enjoy.