Lesson 4 · Part 1. How language models work · 12 min

Tokens

Goal. Say what a token is, see how text is cut into tokens, and explain why token count matters more than character count.

What a token is

A model does not read letters. It does not read words either. It reads tokens.

A tokenizer cuts text into tokens. A token is often 3 to 4 characters. A short common word can be one token. A long word is split into several pieces.

Each token has a number. The model has a fixed list of every token it knows. This list is the vocabulary. The model never sees your text. It sees numbers, like [1432, 98, 7751].

Note A tokenizer is like the step in your app that turns a String into bytes before a network call. The model works with the bytes, not with the String.

Try it

Pick a sample and look at the colours. Each colour block is one token.

These are real splits from the same tokenizer family as our fine-tuned Qwen models. They were computed on a Mac and saved, so the page works offline.

Now compare. The English sample has 58 characters and uses 17 tokens. The JSON sample has 68 characters and uses 23 tokens. The JSON costs more tokens even though it is not much longer. Symbols like {, " and : often become tokens of their own.

Why tokens matter for an app

Token count, not character count, decides three things.

• Speed. The model reads and writes one token at a time. More tokens means more time.
• Memory. Every token in the context uses RAM while the model runs.
• Cost. Cloud APIs charge per token. On a phone you pay in battery and time instead.

This matters for our project. The input phrases come in five languages. The output is JSON. Both cost more tokens per character than plain English.

Warning Do not estimate size with text.length. Count tokens with the model's own tokenizer, or you will under-count JSON and non-English text.

Do it: count tokens yourself

You need ollama and a model. If you have not installed them, do the setup in lesson 2 first. Then run this on your Mac. It sends a prompt and asks for one token back, so Ollama reports how many tokens your prompt used.

curl -s http://localhost:11434/api/generate -d '{"model":"llama3.2:1b","prompt":"Remote jobs in Berlin that pay at least 60k, no part time.","stream":false,"raw":true,"options":{"num_predict":1}}' | python3 -c "import sys,json; print(json.load(sys.stdin)['prompt_eval_count'], 'tokens')"

The number can differ from the demo above. A different model has a different tokenizer. That is the point: always count with the tokenizer of the model you ship.

Check yourself

Recap

• A token is a small piece of text with a number. Models read tokens, not letters.
• Token count drives speed, memory and cost.
• JSON and non-English text use more tokens per character.
• Next lesson: how the model picks the next token, and the setting called temperature.