Lesson 9 · Part 1. How language models work · 12 min

Small models need help

Goal. Say why phone-sized models need help on narrow tasks, compare a 1B and a 3B model yourself, and name the two kinds of help.

Models on phones are small

A cloud model may have hundreds of billions of parameters. A phone model has 0.5 to 3 billion. A small model knows less and follows instructions less reliably. It can write fluent text, but on a narrow task with a strict format it often fails.

Do it: experiment 5, 1B against 3B

We asked two sizes of the same model family, llama3.2:1b and llama3.2:3b, the same questions. Run the larger one the same way as before:

ollama run llama3.2:3b

Ask the three questions from lesson 6, and then these two.

Write a Python one-liner that reverses a string.

Who wrote 'Don Quixote'?

What we measured

Question	1B answer	3B answer
Capital of Australia	Canberra	Canberra
17 * 23	391	391
Three primary colors of light	Red, green, blue	Red, green, blue, plus one extra sentence about the visible spectrum
Python one-liner to reverse a string	Used `reversed` and `join`, but wrapped the code in `input()` and said it would "print" the result, though the code does not print	Used `reversed` and `join` and printed the result of a fixed string
Who wrote Don Quixote	Miguel de Cervantes Saavedra, with a short note on its fame	Miguel de Cervantes, with a longer note including the 1605 first publication

On easy questions the two sizes give the same correct answer. The 3B model writes more detail. On the code question both used the same method, and the 3B answer was cleaner. We looked at five questions only. This is a feel for the difference, not a benchmark.

The lesson is not "bigger is always better". For easy tasks a small model is enough, and it is much cheaper on a phone. Choose the smallest model that does the job. Then use two tools to close the gap: a good prompt, and fine-tuning.

Help 1: prompts

A prompt is the text you send to the model. The system prompt is a block of fixed instructions placed first. It tells the model what job to do and what format to use. The user's text comes after it.

A system prompt for our feature says things like: "You extract job search filters. Return one JSON object with these fields. Use only these values." It may also include a few examples. Giving examples in the prompt is called few-shot prompting. "4-shot" means four examples.

Prompts are fast to try. You change text and run again. Think of them as the UI spec for the model.

But prompts have a limit. In our project, a 1.5B model with a prompt scored 3% exact match with no examples and 30% with four examples. A prompt also costs time on every request. Our system prompt is about 450 tokens, and the model must read it before every answer. On the Pixel 6 that cost 2.2 seconds to the first token on the GPU, and 11 seconds on the CPU.

Help 2: fine-tuning

fine-tuning means training an existing model a little more, on examples of your own task. Each example is an input and the exact output you want. For us: a sentence and the correct JSON filters.

Full training changes all the weights, and that needs a lot of memory. A cheaper way is lora. LoRA freezes the original weights and trains a small extra set of weights next to them. This small set is the adapter. In our project the adapter for the 1.5B model is 20 MB. You can keep it as a separate file, or fuse it into the base model to get one new model file.

You do not teach the model new facts about the world this way. You teach it a habit: always answer in this format, with these rules.

The effect is large. On the same 100 test phrases, the same 1.5B model went from 30% exact match (4 examples in the prompt) to 73% after the first round of fine-tuning. After round 2, with 3,000 more varied training phrases written by AI subagents, it reached 90%.

Note Part 2 explains how to prepare data and train. For now: a prompt tells a model what to do, and fine-tuning makes it a habit.

Check yourself

Why does a long system prompt slow down every answer, even a short one?

What does LoRA train?

In experiment 5, what did the 1B and 3B models show?

Recap

Phone models have 0.5 to 3 billion parameters and often fail strict formats on narrow tasks.
Help 1, prompts: fast to try, but 3% to 30% for our task, and they cost time on every request.
Help 2, fine-tuning with LoRA: a small adapter that took the same model from 30% to 73% (round 1) and 90% (round 2).
Next lesson: evaluation, which is how you know any of this worked.