architecture
One resolve/validate path, two consumers. The mask lives in src/llama-add/; everything downstream only reads it.
The runtime core is five files, ~16 KB: layer-capabilities.h/.cpp (per-architecture role resolution, architecture-neutral enum), attention-mask.h (context-local immutable mask), mask-validation.h/.cpp (the single validation entry point). Consumers live where the work happens: the four per-architecture graph builders in src/models/ (one mask check each) and the KV allocation filter in llama_model::create_memory(). CLI parsing, profile loading, and the banner are three shared headers under common/.
layer capabilities - layer-capabilities.cppper-architecture: assigns every logical layer exactly one role (full / local-SWA / gated / shared-KV / recurrent / none), so nothing downstream ever asks "what kind of layer is this?" with ad-hoc ifs.
↓
attention mask - attention-mask.hthe drop list becomes one boolean per logical layer, resolved once at context creation, immutable afterwards: graph and cache are both derived from it, once.
↓
mask validation - mask-validation.cppthe only validation entry point. A representable mask can still describe a model that can't run; this is where that fails loudly at startup instead of corrupting generation later.
↓ ↓
graph builders - src/models/*.cppskip the whole attention branch per dropped layer; residual feeds the FFN directly.
KV-cache allocator - llama_model::create_memory()composes the mask with the arch's own filter; dropped layers' K/V is never allocated.
the capability model
| capability | droppable | owns K/V | meaning |
|---|
| full_kv_attention | yes | yes | ordinary full self-attention |
| local_kv_attention | yes | yes | sliding-window (SWA) attention |
| gated_kv_attention | yes | yes | Qwen3.5 gated attention |
| shared_kv_attention | yes | no | attends over another layer's K/V |
| stateful_non_kv | never | - | recurrent / DeltaNet state |
| none | no | - | safe default for unsupported archs |
Callers use two predicates, can_drop_attention(il) and attention_owns_kv_cache(il), never raw capability values. Unsupported architectures resolve every layer to none, which is how the fork refuses cleanly instead of mis-handling.
Llama - the reference
Every layer is full_kv_attention. When dropped (src/models/llama.cpp), the entire branch - pre-norm, Q/K/V projections, RoPE, attention, output projection, residual add - is skipped and the incoming residual stream feeds the FFN directly. Final-layer output-row selection is preserved on both paths. This is the architecture the mask/capability/validation system was designed against; the others adapt it.
Gemma 3 - two kinds of cache
Layers alternate sliding-window and global attention on a fixed pattern (is_swa(il) → local_kv_attention, else full_kv_attention). Both are droppable and drop identically, but they live in separate physical caches, so what you get back differs: a global layer frees full-context K/V, an SWA layer frees only a window's worth. The skip path (src/models/gemma3.cpp) has two Gemma-specific wrinkles: an extra post-attention norm and a per-layer attention scale applied to Q; both are part of the skipped branch, so a dropped layer runs neither.
Qwen3.5 dense - hybrid, with a floor
Most layers are DeltaNet-style recurrent (is_recr(il) → stateful_non_kv: no K/V cache, never droppable); gated full-attention layers are interleaved (gated_kv_attention). The hard invariant, enforced in mask-validation.cpp: at least one gated attention layer stays enabled, because the hybrid memory path needs at least one active attention K/V source; disabling all of them throws at startup instead of building a broken graph. In the builder (src/models/qwen35.cpp) the branch condition is is_recurrent || attention_enabled, with an assert guarding that the skip path can never fire for a recurrent layer, defense in depth on top of the capability system.
Gemma 4 - owners and consumers
Only the first stretch of layers owns physical K/V; every later layer is a shared_kv_attention consumer that computes attention against an earlier owner's cache (one owner for the global pattern, one for SWA; resolved at capability time from the model's native reuse mapping). Two rules follow:
1. An enabled consumer requires its owner enabled. Dropping the owner out from under it is a validation error: the consumer would attend over freed memory.
2. Dropping a consumer frees compute but no K/V (it never owned any); dropping an owner means dropping its remaining consumers with it.
The builder (src/models/gemma4.cpp) keeps two conditions orthogonal: does the attention block run at all (the mask) and does this layer compute K/V or reuse another's (has_kv(il)). The cache filter has a matching carve-out: an enabled consumer stays visible to the cache's native reuse mapping even though it owns no allocation. A consumer with no valid owner is a topology bug and throws, never a silent fallback.
supported models
The six checkpoints below are the models we fully calibrated and benchmarked; they ship with ready-made profiles:
- Meta Llama 3.1 8B Instruct - Q4_K_M (bartowski)
- Llama 3.2 3B Instruct - Q4_K_M (bartowski)
- Llama 3.2 1B Instruct - Q4_K_M (bartowski)
- Qwen3.5 9B - Q4_K_M (bartowski)
- Gemma 3 12B it - Q4_K_M (bartowski)
- Gemma 4 E4B it - Q4_K_M (bartowski)
llama.add support itself is architecture-based rather than checkpoint-specific: other models loaded by llama.cpp under the same supported architecture enums should work under the same capability rules, but have not all been individually tested. Any of them needs one calibration run first. Family by family:
- Llama (
llama) - Llama 2 / 3 / 3.1 / 3.2 / 3.3 text models at every size. The 3.2 Vision models are a different architecture and are not supported.
- Gemma 3 (
gemma3) - the text checkpoints at every size the loader recognizes: 270M, 1B, 4B, 8B, 12B, 27B. Gemma 3n (gemma3n), Gemma 1/2, and Gemma Embedding are different architectures and are not supported.
- Qwen3.5 (
qwen35) - the dense checkpoints only: 0.8B, 2B, 4B, 9B, 27B. The MoE variants are a different architecture (qwen35moe) and are not supported, nor are Qwen3 and the VL variants.
- Gemma 4 (
gemma4) - E2B, E4B, 26B-A4B, and 31B (the sizes the loader recognizes; 26B-A4B's MoE lives inside the same architecture, in the FFN path that attention-dropping never touches). Gemma 4 Assistant (gemma4-assistant) is a different architecture and is not supported.
The quantization study additionally covered the 8B at Q4_0 (QuantFactory), Q3_K_M, Q2_K, Q5_K_M, and Q6_K with the same mask. Architectures outside the four families are refused at startup by the capability resolver.
why Arm
This project is built for the Arm AI Optimization Challenge (Cloud AI track), and every published measurement comes from Google Axion (Arm Neoverse-V2), CPU-only. On the tested Axion workload, single-stream decode is dominated by memory traffic. Enabled attention layers repeatedly consume weight and KV-cache bandwidth during generation; dropping a layer removes its attention compute and KV traffic and reduces the resident KV footprint.
It is also deliberately orthogonal to Arm's own optimizations: llama.add contains no Arm-specific instruction paths; its code is model-architecture aware (Llama/Gemma/Qwen), not ISA aware. KleidiAI int4/int8 microkernels and NEON/i8mm accelerate each surviving layer's matmuls; llama.add reduces how many layers survive. The two are designed to compose: build with -DGGML_CPU_KLEIDIAI=ON and calibrate the same artifact you deploy.
why immutable?
The compute graph and the KV-cache layout are both built once from the mask. A mask that could change mid-flight would leave allocated caches and compiled graphs disagreeing about which layers exist, so there is no per-request or per-session mask, deliberately. Mask semantics are always logical transformer layer indices; if a cache needs a compact internal layout, that mapping stays inside the cache.