How to measure prompt quality offline: the h5i Prompt Maturity Score

Prompt provenance is a core property of an auditable workspace: not just that a prompt was recorded, but a signal of how well the agent was directed. A pull request from an AI-assisted branch tells you what was asked. Open h5i recall blame or read the PR body and you can see the prompt that triggered each commit. What it doesn't tell you is how well it was asked. Two engineers ship the same feature; one steered the agent with "fix the off-by-one in parse_range() in src/util.rs, add a test for the empty-range case, don't touch the public signature," the other typed "make it work." Same diff, wildly different craft, and nothing in the review surfaces the difference.

The Prompt Maturity Score is how h5i measures prompt quality and makes that craft a visible, trackable signal. It scores the input, the engineer's ask, not the model's output. And it does so with a hard constraint: no LLM, no network, fully deterministic, so you can score prompt quality inside a git hook, in CI, or in a PR render without an API key.

Why not just ask a model?

The obvious implementation is "send the prompt to a judge model and ask it to grade." We rejected that on purpose. Prompt-eval frameworks like PromptBench (arXiv 2312.07910), APE (arXiv 2211.01910), and Promptfoo already do something adjacent, but they score a prompt by running a model and judging its output. They need API access, a task dataset, and they cost money and latency on every evaluation. They answer "did the model do well?"

Prompt maturity answers the complementary question, "did the engineer ask well?", purely from text features. That buys three things a model judge can't: it's free (runs on every commit with zero marginal cost), deterministic (the same prompt always scores the same, so it's reproducible in CI and won't drift when a model is deprecated), and explainable (every point is traceable to a feature you can name). The two approaches are complementary, not competing.

It's worth being fair to the alternatives rather than dismissing them. There are three honest options for "how well was this prompt written," and each has a real place:

The third row is the real baseline. Today most teams ship AI-assisted code with the prompt either unrecorded or recorded but ungraded, so the gap between a disciplined ask and a lazy one is structurally unobservable. A deterministic proxy that is sometimes blunt still beats a blank, because it makes the axis exist. The bar this metric has to clear is not "as good as a human reviewer reading the prompt"; it's "better than nothing, cheaply, on every commit."

The caveat everything rests on: readability is not maturity

The first instinct when you hear "score the writing quality of a prompt" is to reach for readability indices, Flesch Reading Ease, Flesch–Kincaid grade level, Gunning Fog. They're well-studied, classical, and cheap. They're also a trap.

A terse, precise technical ask is an excellent prompt that scores badly on raw reading-ease. "Fix the off-by-one in parse_range() in src/util.rs, add a test" is dense with file paths and identifiers, exactly the tokens that wreck a syllable counter and tank a reading-ease score. If readability drove the number, the best prompts would be punished for being concrete. The literature on these indices is explicit about this: they are length-sensitive and penalise terse technical text.

So readability is deliberately the smallest-weighted signal, it is applied as a band (both extremes penalised, not "higher is better"), and it is computed on a code-masked copy of the prompt, every path, func() call, URL, and snake_case identifier is replaced with a neutral token before a single syllable is counted. That one decision, treating readability as a bounded sub-signal rather than the score itself, is the single most important guard against the metric punishing good engineering.

The composite: seven signals, fixed weights

The score is a weighted sum of seven sub-signals, each normalised to 0.0–1.0. The weights were locked deliberately and sum to exactly 1.0 (the code asserts it in a test). The ranking is the whole philosophy in one table, what a reviewer actually wants to see ranks highest; the signal most likely to mislead ranks lowest.

Specificity and control dominate at 24% each because they are what a manager actually wants to confirm: the engineer was concrete, and they bounded the agent. Clarity is the quietest voice at 8% precisely because it's the signal most likely to mislead on technical text.

Why the weights are hand-set, not learned

The obvious "modern" move would be to learn these weights from data, fit a regression against some label of prompt quality. We didn't, and the reason is the whole point of the metric. A learned weight vector is only as honest as its training labels, and "good prompt" has no ground-truth dataset; you'd be fitting against either an LLM judge (reintroducing the nondeterminism and opacity we set out to avoid) or human ratings (expensive, subjective, and drifting). Worse, learned weights are a moving target: retrain on new data and last quarter's scores silently change, so the number stops being comparable over time.

Fixed weights that sum to exactly 1.0 (the code asserts it in a unit test) buy two properties a learned model can't. The score is auditable: anyone can read the table, see that specificity is worth 24%, and disagree on the record rather than litigating a black box. And it is stable: a prompt scored today scores the same next year, which is what makes "prompt maturity trended up this quarter" a sentence that means something. The cost is that the weights encode our opinion of what matters; that opinion is wrong sometimes, but it is at least legible and arguable, which a fitted coefficient is not.

How each signal is computed

Every signal is built from counts of features in the prompt text, then squashed to 0–1. A few are worth seeing concretely:

• Specificity = concreteness − vagueness. Concreteness is a capped blend of code references (40%), action verbs (22%), quoted symbols (20%), and numbers (18%). Vagueness comes from the NALABS / Femmer "requirements smells" lexicon, words like appropriate, robust, somehow, as needed, whose density per 100 words (saturating at roughly eight weak words per hundred) is subtracted at half weight. So vagueness can pull specificity down but never invert it; piling on weak words hurts, but the floor is concreteness you actually demonstrated.
• Control rewards breadth across categories: constraints (must, only, without), verification (test, assert, acceptance), output shape (json, signature, schema), edge cases, scope, and safety. Hitting many categories beats hammering one.
• Diversity uses an adaptive MATTR (Moving-Average Type-Token Ratio) with a window that scales to prompt length. Short prompts fall back to a confidence-weighted type-token ratio pulled toward a neutral 0.5, a five-word prompt has no room to demonstrate vocabulary, so it's neither rewarded nor punished for it.
• Clarity maps Flesch–Kincaid grade and Flesch reading-ease through a trapezoid: full credit for clear technical English (FK grade ≈ 7–13), tapering for both a childishly-simple ask and a tangled run-on, and a neutral 0.6 for prompts too short to estimate reliably.

Anti-gaming: the guards that sit on top

A naïve weighted sum of keyword counts is trivially gameable, paste the lexicons into your prompt and score 100. The interesting engineering is the layer that makes that not work. Four guards sit on top of the weighted sum, in order:

1. Repetition penalty. The fraction of bigrams that are exact repeats becomes a multiplier (floored at 0.6). "must test format must test format" gets multiplied down; phrase-farming is detected and punished directly.
2. Balance gates. You cannot look mature on one axis alone. Control is a hard gate: a prompt that sets no constraints or acceptance criteria is capped below "advanced" (≤ 69) no matter how concrete it is. Low specificity additionally caps at 79, so you can't be "exemplary" while vague.
3. Hard length caps. Under 8 words caps at 20; under 15 words caps at 45. A 1,200-word unstructured wall caps at 75, that's a dump, not a mature prompt. Keyword density can't buy its way past a length floor.
4. The final clamp to 0–100.

The balance gates have a subtlety worth calling out, because it's where the metric earns its keep. An earlier version hard-capped any prompt with weak context. But a prompt like "run cargo test, fix the clippy warning in src/foo.rs, don't change the signature" is a perfectly legitimate tactical ask, the agent already holds the repo context, so demanding a "why" paragraph would be punishing good practice. So context became a soft gate: a prompt that is already both specific (≥ 0.6) and bounded (control ≥ 0.5) is exempt from the context cap. Weak context still surfaces as a diagnostic flag, it just no longer drags a crisp tactical prompt into mediocrity.

From one prompt to a whole branch

A PR has many commits. The branch score is a length-weighted mean of the per-prompt scores (weight = clamp(words, 20, 250)), and prompts are never concatenated. That last detail matters: if you glued every prompt into one blob, a pile of weak one-liners would pool their vocabulary and structure and read as mature. Scoring each prompt independently and taking a weighted mean means one rambling prompt can't dominate, and a disciplined engineer is rewarded for every crisp ask.

The roll-up also tracks coverage, how many AI commits actually carried a prompt to score. If fewer than 80% did, the result is flagged low-confidence, because a score built from a minority of commits shouldn't be read as a verdict on the branch.

What it looks like in a PR

The score renders right under the hero of an h5i share pr post body, a screenshot-clean headline with the per-signal detail one click away:

> [!NOTE]
> 🌳 Prompt maturity: 81/100 · advanced · 7 prompts scored (100% of AI commits)
> 🔧 Recurring weak spots: weak context.
> Heuristic signal of prompt craft — not a developer rating.

▼ 📊 heuristic breakdown
  Specificity                 ████████░░  0.82
  Control / acceptance        ████████░░  0.79
  Context grounding           ████░░░░░░  0.41
  Structure                   ███████░░░  0.68
  Lexical diversity           ███████░░░  0.71
  Clarity (readability band)  ████████░░  0.80
  Length adequacy             ██████████  1.00

Note the disclaimer baked into the render: "Heuristic signal of prompt craft, not a developer rating." That line is not decoration. The score is an offline proxy over prompt text, it is not a performance metric, not a leaderboard rank, and the diagnostic flags are deliberately descriptive ("weak context", "no acceptance criteria"), never prescriptive. We don't hand engineers a keyword list to stuff, because the moment the metric becomes a target to optimise, it stops measuring anything real.

Where the score is blind

Knowing the failure modes is part of using the number responsibly. The score grades the form of the ask, never its correctness or outcome, so a handful of cases are structurally invisible to it:

• Precise and wrong. A confident, specific, fully-bounded instruction to do the wrong thing scores high. The metric measures how well the agent was steered, not whether it was steered somewhere good. Empirical correctness is the job of the test metrics and integrity audit on the same commit, not of this signal.
• Mentioning is not requiring. The lexicons count tokens, so a prompt that says the word "test" without actually asking for one still banks a little verification credit. The breadth requirement and the small per-category cap blunt this (one keyword can't carry a signal), but a determined writer can still surface-dress a thin ask a few points above its true rigour.
• Context it can't verify. The soft context gate exempts a crisp tactical prompt on the assumption that the agent already holds the repo context. The score can only see that the prompt didn't restate it; it cannot confirm the context was actually sufficient. A terse ask against a model that had lost the thread looks identical to one against a model that had it.
• English technical prose only. Readability indices and the curated lexicons are tuned for English. A prompt in another language, one that is mostly pasted log output, or one dominated by non-Latin identifiers will score noisily: the code-masking helps with identifiers, but the band and lexicon assumptions don't transfer cleanly.
• Steering that happened off-commit. The branch roll-up flags low coverage when fewer than 80% of AI commits carried a prompt, but it can't recover steering that lived in a chat thread, a design doc, or a voice call and only landed as terse confirmations in the commits. Such a branch will under-score, correctly flagged as low-confidence rather than silently wrong.

None of these are bugs to patch; they are the boundary of what a text-feature proxy can claim. The honest framing is the one baked into the render: a heuristic signal of prompt craft, read alongside the tests and the diff, not in place of them.

Conclusion: make the ask reviewable

The Prompt Maturity Score is a deliberately modest instrument: seven counted features, seven fixed weights that sum to one, four guards, and a clamp, with no model, no network, and no randomness. That modesty is the design. Because it's deterministic it belongs in CI; because every point traces to a named feature it survives an argument; because the weights are hand-set and public you can disagree with them out loud instead of reverse-engineering a black box. It will never tell you whether a prompt was right. It tells you whether the engineer was concrete and bounded the agent, and it tells you the same way every time.

The point was never the number on its own. It's that "how well was this agent steered?" stops being a vibe and becomes a column you can sort, trend, and review next to the diff and the test result. Once prompt craft is visible, it can be coached, compared, and improved, which is the only reason to measure anything.

FAQ