Réduire la latence d'inférence
A model's latency is the time between the question and the first word of the answer, then the pace of the words that follow. Nine levers attack it, each on a different point of the decoding path: hardware-specific kernels (`flash attention`, compute written for the card), `page attention` (paging the key-value cache like an OS's virtual memory), model compilation (`TorchCompile`, `TensorRT`, for native execution on the chip), speculative decoding (a small model proposes five words, the big one approves them in one pass), continuous batching (inserting new requests as soon as a slot frees up), async preprocessing (preparing the next input while the model works), request-level caching (same question, answer already stored, served instantly), model warmup (loading it into GPU memory for zero cold start) and quantization (weights moved from 32-bit to 8-bit, same model, faster).
Strengths
- Nine clear levers, each tied to an identifiable bottleneck on the decoding path
- Turns « it is slow » into a measurable decision: which bottleneck, which lever, which gain
- Some levers (quantization, warmup, request cache) are reachable without rewriting the model
Limitations
- Stacking levers blindly adds complexity with no gain if the real bottleneck is elsewhere
- Several levers touch low layers (kernels, compilation) out of reach without control over the inference server
Best for
- Devs serving a model and watching response time drag down the user experience
- Tech leads who need to know which latency lever is worth the effort before committing the team