llama-add-calibrate

On the 8B, the safest and most damaging single-layer removals differ by roughly 140× in quality cost (+0.0075 vs +1.0485 nats). llama.add measures instead of guessing.

Dropping layers blind is how you turn a language model into a random-token generator with excellent latency. The calibrator exists so --attention-drop-layers is always an informed decision: it measures, per model, which attention layers are redundant and in what order to remove them.

the metric

Teacher-forced next-token negative log-likelihood over a text corpus: the built-in technical-text default, or your own via --corpus. The corpus is chunked (256 tokens per chunk in the published runs) and split three ways: a screening split (6 chunks), a calibration split (16), and a held-out split (17). Selection optimizes against the screening and calibration splits; the reported damage always comes from the held-out split, which never influenced any choice, so the reported number is not flattered by selection bias.

phase A - global screening

Evaluating every subset of layers is combinatorially infeasible, so the search first prices each candidate alone: generate every legal standalone drop unit (a single layer on most architectures; on Gemma 4, an owner layer bundles the consumers that depend on it), score each on the screening split against an otherwise-empty mask, and sort by NLL degradation, ties broken toward units that free more physical K/V. The best survivors form a fixed shortlist.

phase B - progressive selection

Grow the schedule greedily from that shortlist. Each step: re-score every remaining candidate on top of the already-committed mask (calibration split), commit the winner, then report the cumulative damage on the held-out split. Layers interact (two individually-cheap layers can be expensive together), which is why each step re-scores instead of trusting Phase A's solo prices. On Gemma 4, owner bundles are re-normalized against the current mask each step, so a bundle shrinks as its consumers get dropped individually rather than going stale.

A step gets flagged caution when its incremental ΔNLL is both above an absolute floor and a large multiple of the median of previous increments: the signature of crossing from "removing redundancy" into "removing something load-bearing".

the quality budget

recommended is anchored to an absolute held-out budget: 0.05 nats of cumulative ΔNLL by default (roughly +5% perplexity), adjustable with --quality-budget. Absolute bounds matter because the least damaging available mask can still be too expensive: on Llama 3.2 1B even the safest single layer costs +0.0804 nats, so no schedule fits the default budget and the report says budget_exceeded out loud instead of pretending. The calibrator measures the whole curve either way; the budget only decides where recommended stops.

fingerprint binding

Every report records an fnv1a64 fingerprint of the model (first MiB of the GGUF plus file size) and of the calibration corpus. --attention-drop-profile verifies the model fingerprint against the file actually being loaded and rejects a mismatch, so a profile calibrated for a different quant or repo can never be applied silently. If your file differs, recalibrate it; it is one command.

what a run looks like

llama-add-calibrateexample output - synthetic values
llama.add calibrate

model: model.gguf
architecture: llama
decoder layers: 32
metric: teacher-forced next-token NLL

step  drop unit   cumulative mask  held-out ΔNLL  status
1     layer 24    24               +0.0004        low estimated risk
2     layer 21    21,24            +0.0011        low estimated risk
3     layer 27    21,24,27         +0.0019        low estimated risk
4     layer 10    10,21,24,27      +0.0163        caution

Most conservative:
  --attention-drop-layers 24

First 3 recommended steps:
  --attention-drop-layers 21,24,27

Layer numbers and deltas above are made up to show the shape of the output. Real runs print a wider table (cumulative K/V impact, remaining eligible layers), four named profiles (minimal / light / recommended / max) with ready-to-paste flags, and write a JSON report with the profiles plus full per-step data next to the model, the intended input for downstream tooling. Long runs show live progress with an ETA on stderr.

flags worth knowing

--corpus file.txt - calibrate on your domain's text instead of the default
-ngl / --gpu-layers, -t / --threads, -tb / --threads-batch - hardware placement, same semantics as llama-cli
--output report.json - where the structured report goes
--verbose - show the full engine log (suppressed below warning level by default)
--no-progress - disable the live stderr progress/ETA display
then, in llama-cli/server/bench: --attention-drop-profile recommended - apply a named profile straight from the report (fingerprint-checked)

pre-calibrated profiles - skip the calibration

Ready-made sidecars for the models we benchmarked. Download the JSON, place it next to your GGUF, run with --attention-drop-profile recommended. Each profile is fingerprint-bound to one exact file (first MiB + size): the right quant from the right repo verifies and loads; anything else fails loudly instead of dropping the wrong layers. All six were produced by the current calibrator and re-verified by an independent recalibration pass.

profile	for this exact GGUF	recommended mask	held-out ΔNLL
Llama 3.1 8B	bartowski / Q4_K_M	`19,20,24,25`	+0.048 (in budget)
Llama 3.2 3B	bartowski / Q4_K_M	`8,18,21`	+0.035 (in budget)
Llama 3.2 1B	bartowski / Q4_K_M	`13`	+0.080 (budget exceeded; the report says so)
Qwen3.5 9B	bartowski / Q4_K_M	`15`	+0.021 (in budget)
Gemma 3 12B	bartowski / Q4_K_M	`1,12,16,24,45,47`	−0.010 (noise floor)
Gemma 4 E4B	bartowski / Q4_K_M	9 layers	see caveats: anomalous stock baseline

These six checkpoints are fully benchmarked. Support itself is architecture-based: other models llama.cpp loads under the same architectures are expected to work under the same capability rules, but need one calibration run first. A different quant or repo is rejected by the fingerprint; run llama-add-calibrate on your own file instead.

honest caveats

Held-out NLL drives selection but is a proxy, not task accuracy; when quality matters, validate the chosen mask independently (the benchmark campaign used upstream llama-perplexity plus HellaSwag and Winogrande). The result is a calibration-ranked progressive recommendation, not a proof of global optimality, and it's calibration-specific: low NLL damage on the calibration corpus is not a guarantee of downstream task quality. Cost scales with model size and shortlist length (each Phase B step re-scores the remaining candidates), so expect calibration to be much slower than a single generation pass. The scoring context is deliberately sized to the chunk length, so calibrating a large model doesn't allocate a generation-sized KV cache per candidate.