"Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM" by Sergii Kozyrev , Davyd Maiboroda
TLDR:
The text highlights that fully quantizing hybrid LLMs to 4-bit NVFP4, including recurrent Gated DeltaNet layers, while pairing softmax attention with linear-attention layers, preserves accuracy across various benchmarks by localizing outliers and leveraging robust delta-rule dynamics. Initially quantizing to 4-bit precision was based on the intuition that errors accumulate in recurrences over long contexts, leading to the development of Minima: NVFP4 W4A4. This methodology, applied to all linear layers including GDN, proved to be efficient across multiple benchmarks, matching BF16 with less memory usage and faster processing time. The study delves into the mechanisms that explain why this quantization method worked well, pointing out factors like localized scaling, gate projection insensitivity, delta-rule recurrence dynamics, and the quantization cost reducing with context. Overall, the approach of quantizing all components and utilizing KV scales offers a practical solution for hybrid LLMs while shedding light on the ease of quantizing the recurrent part.
Read Paper / Blog
huggingface.co
Paper page - Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Join the discussion on this paper page
1September 4, 2026 42
