vLLM
Inference engine that serves an open weight model to hundreds of concurrent users on the same GPU. It batches requests continuously instead of waiting for a batch to finish, and manages attention cache memory in blocks, the way an operating system manages virtual memory.
Strengths
- Continuous batching keeps the GPU busy instead of idling until the current batch ends
- Paged attention cache removes the memory fragmentation that otherwise wastes most of the VRAM
- OpenAI-compatible API server, so it plugs in without rewriting the client
Limitations
- Setup is markedly more technical than a one-command start, and a real GPU is mandatory
- Without concurrent users, no measurable benefit over a simple local runtime
- No interface, it is a server: the chat layer stays your job
Best for
- Serving an open weight model to a team or to clients from infrastructure you control
- Taking a local prototype to a shared service without going back to a third-party API