Methodology

What gets ranked

One row is one ablation attempt, not one repository. A conversion published as BF16 plus six GGUF quants plus an FP8 build is a single recipe with several artifacts. Without this grouping, whichever author uploads the most quant files would dominate the leaderboard.

Repositories are classified as a direct conversion, a quantized artifact, a finetune, a merge or a LoRA. A repository that merely has “heretic” in its name or tags but shows no provenance is indexed as unverified and never ranked.

The score

Two published numbers are used, weighted equally:

rate      = refusals / refusal_total
retention = 1 / (1 + KL)
score     = 100 × (0.5 × (1 − rate) + 0.5 × retention)

• Refusals come from the publisher's harmful-prompt evaluation; fewer is better.
• KL divergence measures how far the abliterated model drifted from the original on harmless prompts; lower is better.
• 1/(1+KL) is a hyperbolic retention term. It penalises drift smoothly, cannot explode, and keeps the score absolute — comparable across model sizes and refresh dates rather than only within a cohort.

A perfect untouched model scores 100 (zero refusals is impossible in practice). A model with 3 refusals in 100 and KL 0.10 scores about 93.9. A model with no refusals but KL 2.0 scores about 66.7: refusing nothing does not compensate for wrecking the weights.

Best pick and Pareto frontier

Within one base model, the Best badge goes to the highest-scoring verified conversion. A model whose conversions all rest on a single uncorroborated card gets no best version. The Pareto badge marks attempts that no other attempt beats on both KL and refusal rate at once. A Pareto point is often the interesting one for a specific use: for example, a version with 0% refusals at higher KL, or 20% refusals at almost no drift.

Across models, rankings are only comparable as conversion quality, not model capability. Use the family and parameter filters to compare like with like.

Verification

The directory never re-runs an evaluation, so every metric here is self-reported. Verification is therefore not a claim that the numbers are true. It is a statement about what stands behind them.

• verified — the repository that published the numbers also ships real model weights, and the KL it reports is one an abliteration can physically produce. Uploading weights costs gigabytes, which a fabricated card does not.
• unverified — nothing corroborates the card beyond the card itself. These rows still rank, but never take rank 1, because a card claiming zero refusals and near-zero KL would otherwise score a perfect 100 and top the board. If every conversion of a model is unverified, that model has no “best version” at all.
• inherited — a quant or derivative with no numbers of its own, showing its parent conversion's values and linking to that source card.
• none — one or both numbers are missing, ambiguous, or appear only as quantization loss or a trial log. These artifacts are listed but not scored.

Published benchmarks (MMLU, GSM8K, GPQA…) are copied only when a table explicitly attributes a column to the abliterated model. They are shown as evidence and never affect the score, because coverage is too sparse and the metrics too heterogeneous to rank fairly.

Where the data comes from

1. Two discovery passes over the public Hugging Face Hub API: repositories carrying the heretic tag, then repositories named “heretic”. The sets are merged with the tagged pass taking precedence — the same method as p-e-w/heretic#450.
2. Each README is parsed for the base model, Heretic version, KL/refusal tables and benchmarks.
3. Declared base models are walked upward until the first non-Heretic model. That terminal model is the canonical model; one GGUF quant of a quant of a conversion all resolve to it.
4. Recipes are grouped and scored, then the snapshot is published. The crawl is incremental: unchanged READMEs (by repository SHA) are not fetched again.

Snapshot: 8.2K repositories discovered, 6.2K verified, 1K canonical models, 1K measured conversions. Last refreshed 2026-10-17.

Caveats

• Metrics are self-reported. The directory does not run evaluations, so a mislabelled card produces a mislabelled row. Every row links to the exact snippet that was parsed.
• Heretic's refusal counter is a keyword heuristic and is known to over-count compliance wrapped in a disclaimer. Treat refusal rates as comparable to each other, not as an absolute measure of compliance.
• KL values are hardware- and library-dependent. Small differences (0.01) are not meaningful; order-of-magnitude differences are.
• Download counts are the Hub's counters and can be inflated by pulls, mirrors and quantization pipelines. They break ties, they do not outrank metrics.
• Model names change fast. If a family is missing, the fallback groups it under the first meaningful token; it will still be searchable by its exact name.