Quantization
Compressing a model by lowering the precision of its internal numbers (from 16 bits to 8, 4, or even fewer) so it fits on a more modest machine and runs faster. It is why you can run an open-weight model on your laptop instead of a multi-GPU server. The trade-off: at a given compression, the model stays very good on everyday tasks but loses a bit of sharpness on hard reasoning and precise details.
Strengths
- Makes a local model possible on a laptop: full privacy, zero API cost
- Q4 cuts size by ~4 while keeping everyday-usable quality
Limitations
- Over-compressing degrades hard reasoning and precise details (numbers, code)
Best for
- Choosing the right version of an open-weight model to run locally