Exact Extract — Huawei Training Material, Pages 92, 167 and 168
"High memory usage for long sequences"
"The longer the sequence length (N + m, input + output), the higher the memory usage and the greater the inference resource consumption."
"Repeated computation of a large number of historical tokens results in high TTFT"
"Identify and match historical prefix cache by the unique hash code to eliminate repeated computing via querying and improve the inference throughput by 43%"
The inference architecture also distinguishes the prefill stage from incremental decoding, during which previously generated KV-cache information is reused.
Technical Deep Dive:
The Key-Value (KV) cache is a fundamental optimization technique used during autoregressive Transformer inference.
In Transformer attention, each token is represented by query, key, and value vectors. During autoregressive text generation, the model generates tokens sequentially and repeatedly attends to previously processed tokens.
Without KV caching, previously calculated attention key and value representations would need to be recomputed unnecessarily during subsequent decoding steps.
The KV cache stores these representations so that they can be reused.
Option A is correct. Reusing previously calculated attention states reduces redundant computation and improves inference execution efficiency.
Option D is correct. Avoiding repeated calculations is the principal mechanism by which KV caching accelerates autoregressive inference.
Option B is not a direct function of KV caching. HBM provides high-bandwidth memory for model weights, activations, and cached attention states. KV caching consumes HBM capacity rather than inherently increasing its utilization efficiency.
Option C is not a primary guaranteed function. KV caching can improve useful computational throughput, but its effect on hardware utilization depends on memory bandwidth, concurrency, and the inference scheduler.
For long-sequence workloads, the KV cache can become a significant memory bottleneck.
Huawei addresses this through hierarchical caching, memory compression, and cache offloading in its inference acceleration architecture.
The primary functions are therefore reducing repeated computation and improving inference efficiency.
Submit