Paper Shows Perplexity Metric Hides KV-Cache Quality Losses

A team of four researchers, led by S M Mahbub Murshed, built a deliberately small language model, 8.39M parameters and 33.6 MB on disk, so they could trace exactly what happens inside a compressed key-value cache rather than treat it as a black box. The paper, EdgeLLM-Nano, argues that most existing evaluations of KV-cache compression never actually route inference through the compressed cache under test, which can mask the damage compression does.

Two specific methods come out of the audit looking worse than advertised. A polar-guided key-cache quantizer loses an order of magnitude more fidelity than a plain round-to-nearest baseline, and a consistency-anchored progressive weight pruner performs no better, and sometimes worse, than one-shot magnitude-based pruning, a far simpler technique. Both results run against the usual pitch for these newer, more elaborate compression schemes.

The paper’s sharper point is methodological. Standard perplexity, the token-prediction metric that gives language models their field’s name, assigns an identical 5.461 score to every compression backend the authors tested, even though the backends behave very differently once used for actual generation. To see that difference, the authors introduce Long-Context Decode Fidelity (LCDF), a token-by-token test of teacher-forced log-losses computed through the compressed cache itself rather than around it, paired across seeds so the effect sizes are measurable rather than anecdotal.

A companion test on a Raspberry Pi 5 adds an energy result that cuts against the usual assumption behind compression. Every compression backend the team measured used more energy per generated token than uncompressed FP16 inference, meaning the memory savings did not translate into power savings on that hardware. Only one tested method, KIVI, came close to lossless under the new LCDF test; everything else left room for improvement that perplexity alone would never have surfaced.

The numbers

• 33.6 MB: the full size of the nano-scale testbed model, small enough to trace every backend, seed and power sample end to end.
• 8.39M: the parameter count behind that 33.6 MB footprint.
• 5.461: the perplexity score given to every compression method tested, regardless of how each one actually performed once its cache was used during generation, which is the gap LCDF was built to expose.

For engineers deciding how to compress a model before it ships on constrained hardware, the paper’s practical advice is to check a candidate method against a fidelity test that actually runs through the compressed cache, and to budget for energy rather than assume a smaller cache means a smaller power draw.