Quantization

Quantization and efficient LLM inference

FP32/FP16/BF16/FP8/INT8/INT4 formats, GPTQ, AWQ, SmoothQuant, GGUF, NF4, outliers, continuous batching, PagedAttention and speculative decoding.