A 70B-parameter model at full precision doesn't fit on almost any GPU you can get your hands on. Quantization compresses it -- the question is which format makes sense for where you're going to run it.
A parameter at FP16 precision takes up 2 bytes; a 70B-parameter model in FP16 needs roughly 140GB just for the weights, before you even count inference context. Quantizing reduces the numerical precision of the weights (down to 8, 4, or even fewer bits per parameter) so the same model fits in a fraction of that memory, at the cost of a quality hit that in practice tends to be small if the quantization method is good. The three formats that dominate the open-source ecosystem in 2026 -- GGUF, AWQ, and GPTQ -- solve this problem with different approaches, each optimized for different scenarios.
GGUF (the successor to GGML) is the format used by the llama.cpp project, designed from the ground up to run models efficiently on CPU and on consumer GPUs with limited memory, including Apple Silicon (Metal). It packages quantized weights together with model metadata (tokenizer, architecture, parameters) into a single portable file, which simplifies distribution -- you can download a `.gguf` and run it directly without depending on the original training framework.
GGUF's advantage is accessibility: it runs on a laptop with no dedicated GPU. The downside is that its throughput on high-end server GPUs doesn't compete with formats optimized specifically for that hardware.
GPTQ applies post-training quantization (PTQ) using a small calibration dataset to decide which weights are more sensitive to precision loss and adjust the quantization error layer by layer, instead of rounding every weight uniformly. This lets it get down to 4 bits or fewer with less quality degradation than naive rounding, and it's well supported on GPU via optimized kernels (ExLlama, AutoGPTQ), which makes it popular for running large models on a single consumer GPU with limited VRAM (e.g., a 24GB RTX card).
AWQ (Activation-aware Weight Quantization) starts from a different observation: not all weights matter equally, but what determines which ones to protect isn't just the weight's magnitude -- it's how it interacts with the model's actual activations during inference. AWQ identifies that small percentage of "salient" weight channels (the ones that most affect the output given the activation pattern) and protects them with higher precision, quantizing the rest aggressively. The result usually preserves quality better than GPTQ on generation benchmarks, and its design makes it faster to integrate with production serving engines like vLLM and TGI.
To run a model locally on a laptop or a server with no dedicated GPU, GGUF via llama.cpp or Ollama (which uses GGUF under the hood) is the most direct option with the best tooling ecosystem. To maximize quality on a single consumer GPU with limited memory, GPTQ remains a solid, widely supported option. To serve a model in production through vLLM or another inference engine at scale, AWQ usually gives the best balance between preserved quality and throughput, and it's the format most modern serving engines prioritize supporting well.
No quantization format recovers capability the base model didn't have -- they compress what already exists, they don't improve it. And quality loss isn't uniform across tasks: free-form text generation tolerates aggressive quantization reasonably well, while tasks that depend on exact numerical precision (calculations, following very specific formatting instructions) degrade more noticeably at 4 bits or below. It's worth evaluating with your own representative prompts before deciding on a quantization level for production, rather than trusting generic benchmarks published by each format alone.
Carlos Montiel is an enterprise AI solutions architect. He implements LLMs, Agents, RAG, and orchestrators for companies across Guatemala and Latin America. Reach out for a consultation.
Contact Carlos Montiel