benchmarks

Measured 2026-07-31 on GCP c4a-highcpu-16: 16 × Google Axion (Arm Neoverse-V2), Ubuntu 24.04 arm64, CPU-only, Release build with KleidiAI (-DGGML_CPU_KLEIDIAI=ON) and native Armv9 codegen (-mcpu=neoverse-v2 +dotprod +i8mm +sve), 16 threads, warmup enabled. Repetitions: 5 for pp512/tg128, 3 for depth decode, and a documented single repetition for pp32768 (one repetition costs about 2.5 hours per configuration). Single variable per comparison: the attention-drop profile. Raw data and scripts: bench/ in the repo; full methodology and caveats: docs/llama-add-benchmarks.md.

The baseline is KleidiAI-enabled and compiled for Neoverse-V2; only the attention mask changes between compared runs. Both sides of every comparison are the same Release binary, run with -t 16 and warmup enabled. Benchmarking an unoptimized build (KleidiAI off, generic -march, wrong thread count) and then switching those on can manufacture a multiple-fold "speedup" that reflects nothing but a bad starting point; no such headroom is borrowed here.

headline - Llama 3.1 8B Q4_K_M, recommended profile (4 of 32 layers)

The gain is a function of sequence length. Both curves below are the same 4-layer profile, costing +5.3% perplexity.

KV cache drops by exactly the dropped-layer fraction at every context size: 1024 → 896 MiB at 8k, and 8192 → 7168 MiB at 64k, a full gibibyte. On a cloud server that is real capacity: more memory headroom, longer contexts, or room for concurrent workloads. The tensors are never allocated, not allocated-then-freed. Quality is measured by upstream llama-perplexity on a corpus disjoint from the one that selected the mask, and agrees with the calibrator's own prediction to within 0.004 nats. recommended stops at a 0.05-nat budget by default; --quality-budget moves along the measured curve.

why the gain grows with length

The measured gain rises as sequence length increases: prefill cost is quadratic in length, and during decode every enabled layer re-reads its whole K/V cache for each token. Prefill throughput on the tested Axion instance falls 15× between 512 and 32,768 tokens, and as context grows, attention and KV traffic become a larger share of the measured workload. Neither curve has flattened by 32k. Per-model results at 8k:

tg128 at depth 0, tg64 at depth (5 and 3 repetitions); KV at -c 8192; ΔPPL from upstream llama-perplexity on a disjoint WikiText-2 slice. Masks are the ones a 0.05-nat quality budget selects, a deliberately conservative default. Gemma 4 E4B quality is excluded: its stock baseline perplexity of 58.5 is anomalous in the tested build and reproduces with plain upstream llama.cpp on the same GGUF: an upstream model-support anomaly, not a llama.add result. Its speed and memory numbers stand. Full methodology and caveats in docs/llama-add-benchmarks.md.

this composes - it is not an alternative

llama.add works on a different axis from the usual optimization levers, so it stacks with them instead of competing. Kernel libraries make each operation faster; llama.add removes operations. The profiler shows exactly that: attention samples fall 12.7% while instructions-per-cycle holds at 2.72, so the surviving work runs just as efficiently as before.

does the model still work?

Perplexity measures next-token surprise on a corpus. It is a weak proxy: it cannot tell a model that predicts text slightly worse from one that has stopped reasoning. Both benchmarks below ship inside upstream llama-perplexity. Llama 3.1 8B, baseline vs recommended:

Both moved slightly up, both far inside the confidence intervals (±2.6 and ±2.4 points). The defensible claim is no measurable regression on HellaSwag or Winogrande in the tested configurations, and nothing tighter: the +0.30 and +0.16 deltas are noise, not improvement. Resolving a sub-1-point difference needs the full 10,042-task set, about seven hours across both configs.

vs quantization - the eight-way

Why drop layers instead of changing quant? Same mask (19,20,24,25) applied to every format. tg128 values are interleaved re-measurements (4×5 reps).

Three facts, all true at once:

1 - Against the K-quant ladder, the mask wins the quality-matched fight. Q4_K_M+mask (7.78) vs Q3_K_M (7.71): same quality, +157% at pp512. In this benchmark configuration, the tested sub-4-bit K-quants were slower and lower-quality: they lack the interleaved-GEMM path. Confirmed by control, not just inferred: Q5_K_M (5.3 GB) and Q6_K (6.1 GB) are larger than Q4_K_M yet prefill at 91.2 and 98.7 t/s, while the smaller Q3_K_M crawls at 43.7. Size is not the variable; kernel coverage is.

2 - Q4_0+KleidiAI is genuinely fast at short prompts, and the advantage decays to zero with context. Q4_0 ÷ Q4_K_M: 1.51× at pp512 → 1.08× at pp8192 → 1.03× at tg128 → 0.97× at 16k decode (it loses). The KleidiAI edge is a weight-GEMM edge; as context grows, attention and K/V traffic become a larger share of the measured workload, and no weight format touches them. The measured Q4_0 advantage shrinks as context grows while the measured benefit of the mask increases.

3 - The fastest configuration in every column (32k decode included) is Q4_0 + mask, and it requires llama.add. On top of Q4_0, the mask adds +7.2% pp512, +12.7% pp8192, +13.6% decode@16k, +10.9% decode@32k (9.71 vs 8.76 t/s; measured because a crossover to the K-quant was plausible there; it did not materialize) for +5.6% PPL: composability with Arm's own kernel path measured, not asserted. Buffer routing is confirmed from server logs: Q4_0 uses CPU_KLEIDIAI (3744 MiB), while Q4_K_M uses CPU_REPACK (4402 MiB).

Task benchmarks (HellaSwag 1,000 / full Winogrande, same protocol as above): Q4_K_M 77.4/73.0 :: Q4_K_M+mask 77.7/73.2 :: Q4_0 78.2/73.5 :: Q4_0+mask 78.3/73.0. Four configs spanning +0% to +8.2% perplexity, statistically indistinguishable on tasks. The suspicion that Q4_0 hides capability damage its PPL does not show was tested and is not supported.

KV memory is untouched by quantization: all four formats allocate identical caches (1024 MiB @8k), so only the mask reduces it. The mask transfers across formats at stable cost (+0.052 to +0.059 nats). Q4_0 provenance: QuantFactory (no imatrix); a non-imatrix Q4_K_M control measured PPL 7.348 vs bartowski's 7.383, so the provenance effect is ~0.5% and reversed - negligible. Scope: this is llama.cpp's Arm kernel coverage at this commit, not quantization as a technique.

profiler evidence - Arm Performix

Arm Performix profiled the depth-8192 decode workload on five models, baseline vs recommended: 20 raw captures (code_hotspots and cpu_microarchitecture recipes × two configs × five models), shipped in bench/results/evidence/performix/ with an ID-to-model index. On the 8B, attention-kernel samples fell 5.26M → 4.59M (-12.7%, matching the 4/32 = 12.5% of attention layers removed) and total CPU samples fell 10.8%. In the microarchitecture captures (four models), IPC is unchanged to two decimals: 2.72 → 2.72 on the 8B, 2.52 → 2.52 on Qwen3.5, 2.56 → 2.55 on Gemma 3, 2.39 → 2.39 on E4B. The processor is not doing the same work more efficiently; there is less work to execute.

methodology notes

Benchmark noise and interleaving. Short-context decode deltas are small enough that run ordering matters: one depth-0 pair read -3.8% in one ordering and +2.0% interleaved. Every quotable short A/B figure was therefore re-measured in 4 alternating rounds × 5 repetitions; the headline depth-0 number above (36.13 → 37.01, +2.45%) is the interleaved figure, and the docs mark the older non-interleaved value as superseded.

Reproduction battery. A 12-run spot-check re-measured a sample of published values with fresh runs, across all three model vendors and both workload classes: 12 of 12 within tolerance (3% for prefill, 4% for depth decode, which pays a fresh cache prefill). Raw files: bench/results/evidence/spotcheck/.

Calibration validation. On the 8B, the calibrator's held-out prediction (+0.0479 nats) agrees with independent upstream llama-perplexity (+0.0518) to 0.004 nats. All six models were also recalibrated from scratch with a rebuilt binary mid-campaign: every mask, every held-out ΔNLL to four decimals, and every profile anchor reproduced identically.

Validation history. The validation process caught real defects, fixed before the final data: a missing BOS token in the calibrator's scorer (found by cross-checking against upstream perplexity) and a recommended profile that originally had no absolute quality bound (exposed when the 1B recommended a +2.53-nat schedule; the 0.05-nat budget and budget_exceeded reporting came out of that). Gemma 4 E4B quality results are excluded because its stock baseline perplexity (58.5) is anomalous in the tested build and reproduces with plain upstream llama.cpp on the same GGUF; its throughput and memory results stand.

limitations

• Single-stream (batch-1) decode; server multi-slot throughput was not measured.
• One measured instance type (Google Axion / Neoverse-V2). Absolute numbers will differ on other hardware; the trend should be validated independently.
• The default calibration corpus is ~45 KB of WikiText; domain-specific deployments should recalibrate on representative text.
• Gemma 3's per-step held-out deltas sit at the noise floor of a 17-chunk split; its final mask still verifies independently at +3.5% PPL.
• Gemma 4 E4B quality is excluded (anomalous upstream baseline, see above).

reproduce it yourself

# calibrate, then bench the same model with and without the mask
llama-add-calibrate -m model.gguf
llama-bench -m model.gguf
llama-bench -m model.gguf --attention-drop-layers <mask>

Every figure on this page is a measurement, not an estimate: raw CSVs and server logs are in bench/results/, and bench/results/summary.json is regenerated from the raw files by bench/scripts/parse_results.py, never edited by hand.

The whole campaign is scripted and checkpointed. bench/scripts/run-campaign-v2.sh runs calibrate → throughput → depth → KV → perplexity and resumes if interrupted; stage scripts carry their method notes in their headers: run-depth.sh (decode at KV depth), run-32k.sh (32k prefill/decode), run-kv-contexts.sh (allocator memory), run-perplexity.sh (upstream quality), run-taskeval.sh (HellaSwag/Winogrande), run-quant*.sh (the eight-way study), run-spotcheck.sh (the reproduction battery). bench/README.md maps every published number to the script that produced it and the raw file that stores it.