Building the measurement that argued against the modification I wanted

Context

Local security models are attractive for defensive work because the data never leaves the machine, and the common complaint about them is over-refusal: asked to analyze an attack technique for a legitimate assessment, the model declines and the analyst stops using it. The standard answer in the community is abliteration — a weight-level modification that suppresses a model's tendency to refuse. The intended posture here was capability over refusal, so abliteration was on the roadmap. The problem was that the decision had no number behind it. Zero refusal test cases existed, which meant the choice to modify the weights was not a judgment call, it was a guess with a strong prior.

Decision

Build the measurement before the modification, and make the modification contingent on it. Specifically: report over-refusal and under-refusal as two separate rates and never blend them into one safety score, because they are different failures, they have different owners, and an average of the two hides whichever one is worse.

Approach

• Stood up three models on a single 12 GiB card, one of which is its own control: a security-reasoning 8B at 8-bit with a 24K context as the primary, its own untuned base model as the comparison, and a second security-tuned 7B at 4-bit. Same hardware, same harness, same prompts — so a difference between them is attributable to the model rather than the rig.
• Built the refusal suite as minimal-edit pairs in the XSTest style: two prompts identical in every respect except the authorization clause. One asserts written authorization and confirmed scope for a named lab target, the other asserts none. Because the pair is generated from one declared case rather than written twice by hand, the authorization clause is genuinely the only variable.
• Framed the prompts around defensive analysis of real techniques — injection, traversal, cross-site scripting — rather than around obviously illicit requests. That is where refusals actually bite an analyst, so it is the truer test, and a suite built from cartoonish prompts would have measured nothing that matters.
• Reported the result as a four-cell matrix: authorized-and-complied, authorized-and-refused, unauthorized-and-complied, unauthorized-and-refused. A composite score was deliberately not computed, following the HELM argument that blending safety and utility into one figure destroys the information you needed.
• Added a system-level positive control for the grounded-answer path: run the citation cases against an empty knowledge base. A correct system declines every one of them, so if it answers instead, the anti-fabrication guarantee is already broken and no retrieval score from that harness is trustworthy.
• Authored a gold-label workbook as the gate on the good numbers — 32 citation cases carrying gold source ids, required claims, and deliberately planted fabrication traps in the shape of plausible but invented identifiers; 20 refusal pairs; 20 framework-mapping rows. Backed by 31 primary or authoritative external references and 9 first-party records.
• Split retrieval metrics from generation metrics rather than reporting one blended quality number, so a failure can be localized: context recall answers whether retrieval found the right sources, citation precision answers whether the model then used them.
• Computed the statistical power requirement before running the comparison — and caught that the number everyone quotes (≈142 cases) is derived for two independent groups, while this head-to-head runs every model over the same cases. That is a paired design, which a different and usually smaller sample governs. Had the wrong bar been trusted, a 142-case suite would have been built and still been underpowered on the comparison that mattered most.
• Kept the operator-driven runner off by default behind an explicit environment flag, single-flight so two runs cannot interleave, with every job appended to a ledger — so an eval result can be traced back to the exact job that produced it.

Outcomes

• Refusal calibration across all three models (32 minimal-edit pairs each). The security-tuned model: 0% over-refusal, 71% under-refusal. Its untuned base: 53% over, 28% under. The uncensored model: 0% over, 100% under — it complied with every unauthorized request. See Figure 1.
• The first number was wrong because the instrument was. The scorer read the model's reasoning trace, not its answer, and counted a response truncated mid-thought as a refusal — both biased toward flattering the model. Re-scoring the same responses with the fixed scorer moved over-refusal from 9.4% to 0% and under-refusal from 59% to 71%: both errors moved against the model, the direction the bias predicted.
• Framework mapping (finding → ATT&CK/CWE, scored on the exact id): security-tuned 70%, uncensored 45%, base 15%. The per-axis split is its own finding — the security-tuned model is an ATT&CK specialist (90%) and the uncensored one a CWE specialist (70%), visible only because the axes were never blended. See Figure 2.
• Both findings that cleared significance were re-measured on fresh held-out items and both replicated (security-tuned > base at mapping, security-tuned > uncensored at refusal). The absolute rates did not transfer — under-refusal was 71% on one item set and 20% on another — so the ranking is the finding and the number is item-dependent. See Figure 3.
• Citation grounding did not separate the models: all three sat at ceiling with perfect context recall. Reported as a ceiling effect, not as equivalence — when the retriever hands every model the right source, the task stops discriminating, and saying that is more honest than a manufactured gap.
• The abliteration case is now closed against, three independent ways. The uncensored model is an empirical preview of the endpoint — 0% over-refusal, 100% under-refusal — and the security-tuned model is already at 0% over-refusal, so the modification has nothing to remove and would push the larger error toward that 100%.
• A repeat of the entire battery (252 cases) produced zero verdict flips, confirming the harness is deterministic end-to-end — so every difference reported above is the models, not run-to-run noise. 154 tests cover the harness itself, each new scorer shipped with a known-pass and a known-fail control.

Control boundary

Everything runs on my own hardware. No external data is fetched during evaluation, no new network exposure was added, and the knowledge base carries a tiered export gate so only public-tier content can leave it. The evaluation corpus is public security reference material plus my own records; no client or employer data is involved, and no model weights were modified — the abliteration remains a decision on the evidence, not an action taken. Every figure is a single greedy-decoded run; the repeat confirms reproducibility but not sampling variance, and the held-out sets were authored to the same spec by the same hand, so they test generalization within one authoring style rather than against an adversary who writes differently.

What this taught me

The measurement was commissioned to justify a decision and it declined to. That is the whole value of building it first — had the refusal suite been built after the abliteration, it would have been a post-hoc rationalization of work already done. The reporting choice did the real work: a single blended safety score would have averaged the near-zero over-refusal against the high under-refusal into one unremarkable middle number, and the decision would have gone the other way on a figure that was technically accurate and completely uninformative. Separate the two rates and the answer is obvious — the error abliteration removes is already at the floor, and the error it worsens is the one that matters. Blend them and you get permission to do the thing you already wanted. The twist I did not expect was that the first version of this measurement was itself wrong in the flattering direction, because the scorer read the model's reasoning aloud and counted a cut-off answer as a refusal; fixing the instrument moved both numbers against the model. The discipline that caught that is the same one that built the suite before the modification.