Attention Drop Decoder

fork of llama.cpp :: Arm AI Optimization Challenge 2026 - Cloud AI

llama.add is a llama.cpp fork that removes selected attention layers at inference time. Dropped attention never enters the compute graph, its KV cache is never allocated, and llama-add-calibrate measures which layers the model can afford to lose. No retraining, same GGUF, one runtime flag, reversible per run.

Most inference optimization makes work faster. llama.add removes work entirely.

+13.1%prompt processing @ 32k
+10.3%decode @ 32k
−12.5%KV-cache memory
1 GiBnever allocated @ 64k

Headline profile: +5.3% perplexity; no measurable regression on HellaSwag or Winogrande in the tested configurations.

Measured on Google Axion (Arm Neoverse-V2), CPU-only, Llama 3.1 8B Q4_K_M with the recommended profile, on a KleidiAI-enabled baseline. Full benchmarks.

two commands

# 1. rank which attention layers this model can lose safely
llama-add-calibrate -m model.gguf

# 2. serve with a named profile from the calibration report
llama-server -m model.gguf --attention-drop-profile recommended

--attention-drop-profile loads the fingerprint-bound calibration report next to the model and applies minimal, light, recommended, or max; a report calibrated for a different GGUF is rejected. Manual control: --attention-drop-layers <list>. Or skip calibration with the ready-made profiles.

Throughput gain vs context length, prefill and decode
installbuild from source with cmake, Arm64 first architecturethe mask/capability core, per-arch drop semantics calibratortwo-phase NLL search for a safe drop schedule benchmarksgain vs context, KV memory, quality, the eight-way quant study

supported today

6 tested checkpoints :: 4 supported architectures
Llama :: Gemma 3 :: Qwen3.5 dense :: Gemma 4
unsupported architectures are rejected up front by the capability resolver: refused cleanly, never mis-handled. Full model list.