Wednesday, October 14, 2026 · 9:41 AM
Model quantization · about 11 min
Fictional dialogue · primary sources below
i keep seeing the same model in 4-bit, 8-bit, whatever
am i downloading the cheap version? like did they take some of the brain out
you're downloading a version that stores its numbers with fewer bits
the interesting question is how much of the model's behavior survives that rounding
i need the context first. what are these numbers actually doing
start with a neural network. it takes numbers in, runs them through layers, and produces numbers out
inside a layer, inputs get multiplied by learned numbers called weights, then added together
training adjusts those weights so the network gets better at its task
1
so a weight is like how strongly one thing affects another
yes. useful intuition, as long as you don't picture each weight as one stored fact
a fact or skill can depend on patterns spread across lots of weights
and when a model says 8B, that's 8 billion of these adjustable numbers?
roughly, yes. B means billion parameters. weights are the main thing we'll focus on here
quantizing those weights usually preserves the network's shape and parameter count
so an 8B model in 4-bit still has about 8 billion parameters
right. picture a recipe with the same ingredients and steps, but a measuring spoon with fewer markings
you can still cook the recipe. you just can't express every original measurement exactly
ok that sounds dangerous if the recipe needs 0.013 grams of something
exactly. some measurements tolerate rounding. others matter a lot
good quantization tries to spend its limited precision where it helps most
2
what does a bit buy me here
a bit is a binary digit. each extra bit doubles the number of possible codes
4 bits give 16 codes. 8 bits give 256. the codes stand for numerical values
16 possible numbers for a whole model seems absurd
the same 16 codes can mean different values in different groups of weights
think of 16 marks on a measuring spoon whose size you can change between ingredients
so 4-bit doesn't mean every weight has to be a whole number between 0 and 15
right. a code gets interpreted using extra information, often a scale for its group
and some formats use a lookup table of values rather than evenly spaced steps
give me actual numbers before i lose this
toy example: let one code step mean 0.1, with zero represented by code 0
a weight of 0.26 becomes code 3. when used, that code represents approximately 0.3
and 0.24 becomes 0.2? so they end up on opposite sides
yes. that's rounding error. the original numbers land on the nearest available steps
0.26 and 0.31 could both become 0.3. after that, you can't tell them apart
even if i turn the model back into a big float file later
you can store 0.3 with lots of bits, but that doesn't recover whether it used to be 0.26 or 0.31
the lost detail is gone. that's why this is lossy compression
where does that scale come from
the quantizer picks it from the values it needs to represent
a common mapping is q = round(x / scale) + zero_point, then clamp q to the allowed code range
the approximate value is scale × (q − zero_point)
3
zero_point sounds like something i can ignore
it tells you which integer code represents actual zero
with a symmetric signed scheme, that can simply be code 0
an offset helps when the useful range sits mostly on one side of zero
3
i thought models were all floating point tho. FP16, BF16, all that
those are floating-point formats. they store a sign, an exponent, and precision for the value
the exponent lets the spacing between representable numbers change with magnitude
so INT8 and FP8 aren't interchangeable even though both say 8
correct. the bit count tells you the budget, while the format tells you how the budget gets spent
FP16 and BF16 also spend their 16 bits differently. BF16 gives more room to the exponent
↗
more exponent means more range, less room for fine detail?
yes. range and precision are separate concerns
you'll also see low-bit floating-point formats and block-scaled formats. quantization covers more than integers
4
what does any of this save in practice
let's calculate just the raw weight storage for a hypothetical dense 8B model
8 billion weights × 2 bytes each at 16 bits = 16 GB, using decimal GB
at 8 bits it's 8 GB. at 4 bits it's 4 GB, before extra storage
so 4-bit is a quarter of the 16-bit weight storage
for that idealized calculation, yes. the parameter count stayed the same
scales, offsets, higher-precision tensors, and file metadata make the actual file larger
if i have 8 GB free, the 4 GB model fits comfortably then?
you still need working memory. the weight file is only part of the budget
there are temporary tensors, runtime buffers, and the attention cache for your conversation
oh. is that why a model loads fine and then crashes halfway through a huge prompt
that can happen. for many transformers, the KV cache grows as the conversation gets longer
it stores attention keys and values from earlier tokens so generation can reuse them
long contexts and simultaneous requests can make that cache a major memory cost
5
but i already quantized the model
weight precision and cache precision are separate settings
a 4-bit weight file can still use a higher-precision KV cache
some runtimes support cache quantization. it trades another chunk of numerical detail for memory
test it separately. Hugging Face warns it can worsen latency for short contexts when memory is already sufficient
5
wait why would making things smaller make them slower
because compact storage adds work: unpacking codes, applying scales, and converting values
the implementation has to save more work or memory traffic than those conversions cost
i assumed 4-bit meant the GPU does all its math in 4-bit
storage precision, multiply precision, and accumulation precision can differ
some weight-only implementations load compact weights and do the actual arithmetic in a floating-point format
accumulation is the adding part?
yes. matrix multiplication adds up lots of products
those sums often use a wider format to avoid overflow or too much rounding
3
so what's the reason it can still be faster if the math isn't cheaper
moving the weights from memory can be the bottleneck
during single-request decoding, a dense model often reads lots of weights for each new token
fewer bytes per weight can mean less time waiting on memory
6
like the measuring spoons are smaller, but now we're talking about carrying the whole recipe around
yep, different part of the problem. size matters for both capacity and movement
you can win by fitting in fast memory, by moving fewer bytes, or by using faster low-precision arithmetic
and i shouldn't expect 4 times the speed just because it's a quarter the bytes
correct. kernels, hardware, context length, and batch size all affect the result
PyTorch's GPT-fast example even showed an unfused INT8 implementation slower than its FP16 baseline
6
what's a kernel in this context
a small hardware-specific program that performs an operation, like a matrix multiply
a fused kernel can unpack weights and use them without writing a big intermediate tensor back to memory
what about when i paste a long document versus when it's answering
those are different phases: prefill processes the prompt, decoding generates new tokens
speed them separately when testing. the same quantized model can help one phase more than the other
6
i'm still stuck on how rounding billions of weights doesn't wreck everything
errors affect the computation differently. some changes barely move the output, others have a larger effect
GPTQ uses an approximation of that sensitivity to choose weight quantization more carefully
so it checks whether rounding one weight changes what the layer does
that's the basic goal. GPTQ uses approximate second-order information to guide the process
its paper demonstrated accurate 3- and 4-bit weight quantization for the models it tested
7
so the real trick is picking the rounding, not just chopping digits off
yes. the method matters alongside the number of bits
but a paper's small average accuracy loss doesn't guarantee your specific task is unaffected
what can go wrong with just picking one scale for everything
imagine most values sit between −0.5 and 0.5, with one unusual value at 20
a scale wide enough to cover 20 leaves coarse steps around the small values you have lots of
and if you pick the scale for the small values, 20 doesn't fit
then you clip it to the largest representable value. that adds clipping error
you're balancing rounding error inside the range against clipping error outside it
can we just give every weight its own scale
then you'd store so much extra information that the compression benefit could disappear
a common compromise is a separate scale per channel or per small group of weights
smaller groups can represent local ranges better, at the cost of more metadata
2
does a weirdly large number always mean it's important
importance also depends on how the model uses it
AWQ looks at activation statistics to identify weight channels worth protecting from quantization error
activations are the input numbers?
they're the values moving through the network during a particular run
weights are the learned settings. activations change with the input
so activation-aware means it watches the model working
yes. AWQ uses those observations to choose channel scaling before weight quantization
the compensating transformation preserves the higher-precision computation before rounding is introduced
8
then why don't we quantize the activations too
we can. but their values vary with the input, and activation outliers can make it harder
SmoothQuant shifts some of that difficulty from activations to weights using an equivalent rescaling
it targets 8-bit weights and 8-bit activations, often written W8A8
9
oh, W4A16 means 4-bit weights, 16-bit activations?
that's the usual shorthand. it tells you which part got compressed
still check the exact implementation for exceptions and accumulation precision
where do the observations for AWQ or SmoothQuant come from
a calibration dataset: example inputs passed through the trained model to collect statistics
the point is to see the kinds of values the network actually produces
8,9
is it learning new facts from those examples
calibration usually measures behavior and chooses quantization settings
post-training quantization, or PTQ, starts from an already-trained model
so if the calibration samples are all English chat, i'd want to check my code and other-language tasks separately
yes. I'd use representative examples and then evaluate the workload you actually care about
one clean-looking conversation doesn't tell you whether it handles your long documents or strict JSON reliably
how do people train around the rounding instead
quantization-aware training, QAT, simulates quantization effects during training or fine-tuning
the weights can adjust while the model experiences that numerical error
like practicing the recipe using the worse measuring spoons
yes. then the final conversion produces the lower-precision model
PyTorch's QAT workflow has a prepare step to simulate the numerics and a convert step after training
10
so QAT costs training compute, PTQ is the cheaper conversion route
that's a useful distinction. QAT can recover quality, but the result still needs evaluation
and the training process can keep higher-precision state even when the final model is low-bit
where does QLoRA fit? i thought that was also training in 4-bit
QLoRA keeps the base model frozen in a 4-bit representation and trains small LoRA adapters
gradients pass through the base model, while the adapters hold the trainable updates
so it avoids updating all the big original weight matrices
yes. that reduces the memory needed for fine-tuning
the QLoRA paper also introduced NF4, a 4-bit format designed around normally distributed weights
NF4 still has 16 codes, but the values aren't evenly spaced?
right. it's a codebook designed for that distribution
QLoRA also uses double quantization: compressing the quantization constants themselves
11
lol even the measuring spoon labels get compressed
pretty much. though your fine-tuning memory still includes adapters, activations, and optimizer state
ok now explain the filenames. GGUF, Q4, GPTQ, AWQ. i can't tell which words are competing
GGUF is a file format. it packages tensors and metadata for loading a model
it can hold different tensor data types, including quantized ones
so a GGUF file doesn't automatically mean 4-bit
right. you need to inspect its quantization type and metadata
GPTQ and AWQ name quantization methods. a file format and a quantization method answer different questions
12
and two files labeled 4-bit can be meaningfully different
yes. group sizes, scaling choices, calibration, and which tensors stay higher precision can differ
bitsandbytes, for example, lets you configure 4-bit compute dtype separately from the storage choice
13
what would you actually do if you were choosing a local model
I'd start with the task and memory budget, then check what the intended runtime supports
for a manageable comparison, choose 2 precision versions of the same checkpoint
so you're mainly testing the quantization choice. comparing unrelated models changes too many things at once
keep the prompt template, generation settings, and test inputs fixed too
what tests would you use beyond asking it something and seeing if it sounds normal
for code, run the code and its tests. for JSON, parse it and validate the schema
for document questions, check the answers against the source. include cases it can get subtly wrong
so fluent output is a pretty weak test
I'd also measure peak memory, time to first token, and generation speed at the context lengths you'll use
repeat the runs. if results differ, you want to know whether it's numerical changes or sampling randomness
does quantization change which word it picks even when i turn sampling off
it can. the model computes scores for possible next tokens
rounding can move those scores enough to change which token ranks highest
then one changed token changes the rest of the conversation too
yes. autoregressive generation feeds each chosen token into later steps
compare whether the answer works, rather than demanding the exact same wording
14
last thing. bigger 4-bit model or smaller 16-bit model
I'd test both under the same memory and latency limits. parameter count and precision affect different things
the bigger model may bring capabilities the smaller one lacks, while quantization may weaken behavior you need
so no universal winner from the filename
right. my rule would be: use the lowest precision that passes your real tasks and operational limits
keep the higher-precision checkpoint if you may need to make another quantized version later
i think i have it now
same network, coarser numbers. scales make the codes useful. method decides where the errors go
and count the whole memory budget, check the runtime, measure speed, test quality
ok. i'll stop treating 4-bit like a percentage of the brain. thanks
Read Wed, Oct 14 · 9:52 AM