Lesson 8 · Part 1. How language models work · 9 min

Quantization

Goal. Estimate the size of a model at different bit widths, and say why quantization lets a phone run it.

The problem

A model has billions of weights. A weight stored as a 16-bit number takes 2 bytes. A 1-billion-parameter model then takes about 2 GB. A 7-billion-parameter model takes about 14 GB. That is more than most phones have.

The idea

Most weights do not need 16 bits. You can often store them with 4 bits and lose only a little quality. Storing weights with fewer bits is called quantization. It is like saving an image as a JPEG with lower quality: the file is much smaller, and you often cannot see the difference.

A rule of thumb for the memory of the weights:

memory in bytes ≈ number of parameters × bits per weight ÷ 8

For a 1B model at 4 bits: 1,000,000,000 × 4 ÷ 8 = 500,000,000 bytes, about 0.5 GB.

This is only the weights. The app also needs working memory and the KV cache, which grows with the context length. So plan for more than the table shows.

Size table

Model size	16-bit	4-bit	Realistic target
1B	2 GB	0.5 GB	Any recent phone
3B	6 GB	1.5 GB	Phones with 8 GB or more RAM
7B	14 GB	3.5 GB	Flagship phones only, risky

This table shows sizes computed from the rule above, not measured ones. A 70B model, even at 4 bits, needs about 35 GB. It is out of reach for a phone.

Grouped bars of file size in GB at three precisions. The 1.5B model: 3.0 GB at 16-bit, 1.5 GB at 8-bit, 0.75 GB at ideal 4-bit. The 0.5B model: 1.0, 0.5 and 0.25 GB. A solid marker on the 1.5B 4-bit bar shows the measured file, 0.84 GB. — File size is parameters times bytes per parameter. A real 4-bit file is about 4.5 bits per weight, so it is a little larger than the ideal.

What we measured

Real files differ a little from the rule, because some parts are stored with more bits. These are models from our own project.

Model file	Size on disk	Notes
Qwen2.5-1.5B, 8-bit, for Android (LiteRT-LM)	1.6 GB	Ran on a Pixel 6; process memory about 1.2 GB on GPU and about 2.0 GB on CPU
Qwen2.5-1.5B, 4-bit, fused with our adapter (MLX)	839 MB	Ran on an iPhone 17; peak memory 1,049 MB
Qwen2.5-0.5B, 4-bit, fused with our adapter (MLX)	276 MB	Ran on an iPhone 17; peak memory 406 MB

A useful model can be smaller than many app assets. A 4-bit 1.5B model is 839 MB. A 4-bit 0.5B model is 276 MB. Sizes in this course are as du shows them, in binary megabytes. In decimal units the 1.5B file is 880 MB.

Note Size on disk and memory while running are different numbers. The table shows both.

Why it matters

Quantization is why the model fits in the RAM of a phone. The cost is some loss of quality. For 8-bit and 4-bit that loss is small on most tasks.

For our task we measured it once, with the fine-tuned round-2 1.5B on the Mac and 100 phrases. The 16-bit model scored 90% exact (2.9 GB), the 8-bit model 90% (1.5 GB) and the 4-bit model 88% (839 MiB). The 2 points are within noise. 8-bit matched 16-bit, and 4-bit was faster (1.19 s against 2.05 s per phrase). Lesson 18 shows how these files were made.

Check yourself

A 3B model at 4 bits: about how big are its weights?

What is the trade-off of quantization?

The 839 MB model ran on the iPhone 17 with a peak memory of 1,049 MB. What does that show?

Recap

Memory for weights is about parameters × bits ÷ 8. 4-bit is about a quarter of 16-bit.
Measured files: 1.5B 4-bit is 839 MB, 0.5B 4-bit is 276 MB.
Running memory is higher than the file size.
Next lesson: why small models need help, and what the two kinds of help are.