First, separate 2 ideas: containers vs. quantization methods Most confusion comes from mixing 2 layers. A container defines how tensors are stored on disk. A quantization method defines how weights are squeezed into fewer bits. Containers: safetensors, GGUF, PyTorch pickle ( .bin / .pt ). Methods: GPTQ, AWQ, bitsandbytes NF4, llama.cpp K-quants and I-quants. Both at once: EXL2 and EXL3 are a method plus a storage layout tied to one inference library. A quick memory rule of thumb Weight memory ≈ parameters × bits-per-weight ÷ 8. Model 16-bit ~4.5 bits per weight 8B ~16 GB ~4.5 GB 70B ~140 GB ~39 GB This is arithmetic, not a vendor benchmark. It covers weights only. The KV cache and runtime overhead add more on top. 1. Full precision: safetensors and PyTorch .bin Unquantized models usually ship as 16-bit weights, in either pytorch_model.bin or model.safetensors . The older .bin / .pt files use Python pickle. Loading a pickle file can execute arbitrary code, which makes untrusted checkpoints a security risk. Safetensors , created at Hugging Face, removes that risk. A file is a small JSON header plus raw tensor buffers, with nothing executable inside. Tensors can be memory-mapped and loaded one at a time without reading the whole file. Safetensors is now listed as a PyTorch Foundation project . Important nuance: most GPTQ, AWQ, EXL2, EXL3, and MLX models are also stored in .safetensors files. The quantization lives in the tensor contents and a config file, not in a new container. 2. GGUF (llama.cpp) What it is GGUF is a binary format for running models with GGML and GGML-based
Source: MarkTechPost