Lesson 5 · Part 1. How language models work · 12 min
Sampling and temperature
Goal. Understand how the next token is picked, and choose a temperature for each feature in your app.
From scores to a pick
In lesson 3 the model returned a probability for each token. Something must now pick one. That step is sampling.
The simplest rule is to always take the top token. That is called greedy. A second rule is to roll a weighted dice. A token with probability 0.7 wins 7 times in 10. A token with probability 0.05 still wins sometimes.
Temperature is a setting that reshapes the probabilities before the roll.
- Low temperature (0 to 0.3) makes the top token dominate. Output is focused and repeatable. At 0 the model always takes the top token.
- High temperature (0.8 to 1.2) flattens the probabilities. Less likely tokens get a real chance. Output is more varied and riskier.
Inside the model, each token first gets a raw score called a logit. A formula called softmax turns the scores into probabilities. Temperature divides the scores before softmax. You do not need to do this maths. The demo below does it for you.
Try it
Move the slider. Watch the bars change. Then press "Sample 20 times".
What to look for:
- At temperature 0, one bar takes everything. Sampling 20 times gives the same token 20 times.
- At 1.0, the bars show the model's own probabilities.
- At 2.0, the bars flatten. Sampling now picks many different tokens, including odd ones.
Which temperature for which feature
Pick the temperature per feature, not once per app.
| Feature | Temperature | Reason |
|---|---|---|
| Extract data, produce JSON | 0 or close to 0 | You want the same correct answer every time. |
| Answer a factual question | 0 to 0.3 | Fewer wrong guesses. |
| Brainstorm names or slogans | 0.8 to 1.2 | You want different ideas. |
Our project uses temperature 0 in every measurement. A filter extractor is not creative work.
Do it: feel temperature
Start a chat with a small model.
ollama run llama3.2:1b
Set temperature 0, clear the chat, and ask the same question five times.
/set parameter temperature 0
Write a one-line slogan for a coffee shop.
Use /clear between runs so earlier answers do not change the result. Then repeat five times with temperature 1.2.
/set parameter temperature 1.2
What we measured with llama3.2:1b on a Mac:
| Setting | Result in 5 runs |
|---|---|
| Temperature 0 | All five slogans were the same: "Fuel your day, one cup at a time." |
| Temperature 1.2 | Four different slogans, one of which repeated the temperature-0 slogan; once it ignored "one-line" and returned a list of five options. |
Tip On a phone you can set a random seed. With the same seed, temperature 1.2 gave the same text again in our Android test. If you want variety, change the seed.
Check yourself
You are building a feature that returns JSON from a sentence. Which temperature fits?
You want the same correct answer every time. Variety only adds errors for extraction.
What happens to the probabilities when you raise the temperature?
Higher temperature spreads probability across more tokens. That gives variety and more risk.
Recap
- Sampling picks the next token from the probabilities.
- Temperature reshapes those probabilities: low is focused, high is varied.
- Use about 0 for extraction and JSON. Use 0.8 to 1.2 for brainstorming.
- Next lesson: the context window, which limits how much text the model can see at once.