LatchQ: Latching onto Hot-Cache Query Interactions for KV Cache Quantization

Anonymous authors

Query evidence in the hot cache

The KV cache grows as a language model processes more tokens. Quantization makes it smaller. But the best numerical reconstruction is not always the one that best preserves attention.

Recent keys already spend time in a full-precision hot cache. During that window, subsequent queries attend to them. Those interactions reveal which keys are used and which channels are sensitive to error.

LatchQ turns that observation window into evidence for better clipping. QTrace summarizes the interactions. QClip uses the summary to choose a quantization range, preserving the baseline bit allocation, storage format, and cache schedule.

In the paper’s analysis, clipping choices based on hot-cache observations agree with the all-query oracle in 63% of evaluated decisions. At 2,048 tokens, LatchQ reduces KIVI-2’s NLL gap to FP16 by approximately 64%. [§2.2]

Method

01Observe & accumulate

QTrace

Measure sensitivity to quantization.

For the same query, quantized keys can produce different attention. QTrace estimates this sensitivity from the interactions observed while keys are still in the hot cache.

Attention-weighted query energy accumulates into H_d, a compact statistic for each key and channel.

The QTrace statistic

H_d[h,i,c] = ∑r, t ∈ Eᵢ a^(r)_t,i (q^(r)_t,c)²d

Here a is original attention, q is the observed query, and d is the head dimension. Query heads r share KV head h; Eᵢ contains observed subsequent queries. The statistic is computed once and reused across candidates.

02Evaluate & select

QClip

Select the argmin. Then quantize.

QClip scores 25 clipping candidates using reconstruction error weighted by H_d. It keeps the candidate with the smallest score.

That single choice determines how the key tiles become 2-bit codes, with a shared scale and offset.

The selection objective

arg minℓ, u∑i ∈ B_g w_h,i,c (k_h,i,c − k̂_h,i,c)²

Selection is performed for each KV head, group, and channel. The weights combine QTrace evidence with a small positive floor. Scale and offset are stored in FP16.

What stays the same

Bit allocation
Storage format
Cache schedule

Experiments & results

LongBench results

Interactive charts require JavaScript. At 2.25 bpd, LatchQ scores 53.03 on Llama-3.1-8B-Instruct and 52.88 on Qwen2.5-7B-Instruct; at 3.0 bpd, the scores are 53.51 and 53.75. See Table 1 in the paper for all methods.

View all reported LongBench averages

Table 1 — LongBench macro-average (higher is better)
Method	Llama · 2.25	Llama · 3.0	Qwen · 2.25	Qwen · 3.0
FP16	53.74	53.74	54.40	54.40
KIVI-2	52.23	53.00	52.08	53.34
KVQuant*	50.11	52.64	49.93	52.63
SQuat	51.55	52.61	42.23	47.68
TurboQuant*	49.03	52.23	41.69	52.03
LatchQ	53.03	53.51	52.88	53.75

* At the 2.25 and 3.0 bpd targets, respectively, KVQuant uses 2.3 and 3.3 bpd; TurboQuant uses 3.0/2.75 and 4.0/3.75 K/V bpd. FP16 uses 16 bpd.

Reasoning & code

Reasoning and code generation

On Llama-3.1-8B-Instruct at 2.25 bpd, LatchQ improves GSM8K accuracy from 65.50 to 74.75 over KIVI-2. LiveCodeBench pass@1 reaches 11.38, compared with 11.53 for FP16. [Table 2]

Storage efficiency

KV storage

At 32,768 tokens with a 128-token FP16 hot cache, calculated KV storage is 0.58 GB at 2.25 bpd, matching KIVI-2. This is 14.4% of the FP16 reference. [Table 5]

Abstract

KV cache quantization reduces the memory footprint of large language model (LLM) inference, but minimizing reconstruction error alone may fail to preserve attention behavior. Although future queries are unavailable at quantization time, keys can interact with subsequent queries while residing in the full-precision hot cache. Motivated by this observation, we introduce LatchQ, which uses these observed interactions to guide key clipping while preserving the baseline bit allocation. QTrace accumulates attention-weighted query energy for each key and channel, preserving differences in observed key usage and channel sensitivity. QClip leverages this evidence to select shared clipping ranges by minimizing interaction-weighted reconstruction error. Our method preserves the baseline storage format and cache schedule without extending hot-cache residency. Our analysis reveals 63% agreement between clipping range selections based on hot-cache observations and those based on both observed and future queries. Furthermore, LatchQ achieves the highest average LongBench scores among evaluated KV quantizers at 3.0 and 2.25 bits per dimension (bpd). It also outperforms the baseline quantizer on reasoning and code generation.