When you type a message to an LLM, the first thing that happens is your text gets tokenised — broken into chunks called tokens. Here is the critical thing most people get wrong: tokens are not words.

A token is roughly 3–4 characters of text on average. Common short words like "the" or "is" are single tokens. Longer or unusual words get split. "unhappiness" might become ["un", "happi", "ness"]. Numbers are especially fragmented — "1234567" could be 4+ tokens.

The practical implications are significant for anyone building AI tools. Token count drives your cost — every API call bills by tokens in plus tokens out. Non-English text, code, and structured data (JSON, CSV) typically use 20–40% more tokens than equivalent English prose. This affects how you design prompts and what you include in context.

Every LLM has a context window — the maximum number of tokens it can process at once. Think of it as the model's working memory. Everything the model can "see" when generating its next token must fit inside this window.

This includes your system prompt, the entire conversation history, any documents you injected, and the model's own previous output. When the window fills up, older content gets cut off — the model literally cannot see it anymore.

Context window sizes have grown dramatically. GPT-3 had 4K tokens. Claude now supports up to 200K tokens. But more context ≠ better performance. Models tend to pay more attention to the beginning and end of context (the "lost in the middle" problem). Relevant information buried in the middle of a huge context gets attended to less reliably.

Before the transformer processes tokens, each token gets converted into an embedding — a list of numbers (a vector) that represents its meaning. A typical embedding might be 1,536 numbers long. Every token in the model's vocabulary has its own embedding vector.

The remarkable thing is that these vectors capture semantic meaning spatially. Words that mean similar things have vectors that are close together in that high-dimensional space. Words with opposite meanings are far apart.

The transformer was introduced in a 2017 paper called "Attention Is All You Need." Before it, language models processed text sequentially — one word at a time, left to right. The transformer changed everything by allowing the model to process all tokens simultaneously and let each token directly attend to every other token in the sequence.

At its core, a transformer does one thing repeatedly: it takes a sequence of token embeddings and transforms them through many layers. Each layer refines the representation of every token based on its relationship to all other tokens. After enough layers, each token's representation has been enriched by the full context of every other token it appeared with.

Attention answers one question: when predicting the next token, which previous tokens should I pay the most attention to?

In the sentence "The trophy didn't fit in the suitcase because it was too big," what does "it" refer to? Humans immediately know it refers to "trophy." Attention is the mechanism that allows the transformer to resolve this too — it learns to assign high attention weight between "it" and "trophy" in this context.

When we say a model has "7 billion parameters," we mean it has 7 billion numbers — the weights in every attention head, every feed-forward layer, every embedding. These numbers are what the model learned from training data. They encode, in aggregate, an enormous amount of world knowledge, language structure, and reasoning patterns.

Training is the process of adjusting these numbers. You show the model a piece of text, ask it to predict the next token, compare its prediction to the actual next token, measure how wrong it was (the loss), and adjust all parameters slightly in the direction that reduces the loss. Do this billions of times across trillions of tokens, and the parameters converge to values that encode a surprisingly coherent model of language and world knowledge.

There are two distinct phases in creating a modern LLM. Pre-training is where the base model is created. The model sees trillions of tokens of internet text, books, code, and scientific papers. Its only job: predict the next token. After this phase, you have a powerful but raw text-completion engine. It will complete your sentences, but not necessarily in a helpful or safe way.

Fine-tuning is where personality, helpfulness, and safety emerge. The pre-trained model is further trained on curated datasets of good (and bad) responses. This is where Claude's politeness, GPT-4's formatting style, and Llama's different tendencies all originate.

RLHF stands for Reinforcement Learning from Human Feedback. It is the technique that transformed raw language models into assistants you can have a coherent conversation with. Here is how it works:

RLHF is not free. The process of training a model to be safe and helpful sometimes reduces its raw capability in specific areas. This tradeoff is called the alignment tax.

A base model asked to write a persuasive essay will write one — no caveats, no hedging, no "however, the other side argues..." An aligned model hedges, adds disclaimers, and balances perspectives. In many real-world use cases, this is exactly what you want. But in others — like writing marketing copy or generating creative fiction — the hedging gets in the way.

After all the attention and transformer layers run, the model produces a probability distribution over its entire vocabulary — typically 50,000–100,000 tokens. For each possible next token, there is a probability. The token "the" might have 15% probability. "a" might have 8%. "therefore" might have 0.03%.

The model does not just pick the highest-probability token every time. It samples from this distribution. This is where the randomness comes from — and why the same prompt can produce different responses.

Top-K limits sampling to the K most probable tokens. Top-K = 50 means only the 50 highest-probability tokens are considered for sampling, regardless of their probabilities. Top-P (nucleus sampling) is more dynamic — it includes enough of the top tokens to sum to probability P. Top-P = 0.9 includes however many tokens it takes to reach 90% cumulative probability. These two parameters work together to prevent the model from sampling highly improbable tokens.

KV-Cache is a speed optimisation. During inference, the attention mechanism needs to look back at all previous tokens. Recomputing the key-value pairs for every previous token on every new token generation would be extremely slow. The KV-cache stores these computed pairs so they only need to be computed once. This is why generating a long response does not get progressively slower with each token. It is also why API providers can offer "prompt caching" — if your system prompt is the same across many calls, they can cache its KV-pairs and charge you less.

Every prompt operates across four layers simultaneously. Beginners think about layer one. Experts think about all four before writing a single word.

When you tell the model "You are a senior growth marketer with 10 years experience in SaaS," you are not pretending. You are statistically shifting which part of the model's training distribution gets activated. All those marketing books, case studies, and expert interviews in the training data become more relevant.

When you say "think step by step," you are telling the model to write out the reasoning before giving the answer. Each reasoning step becomes tokens in the context window — and those tokens are then available as context for generating the next reasoning step. You are forcing computation to happen in the output itself.

When you need to parse model output programmatically, XML tags are the most robust way. The model has seen enormous amounts of XML in its training data and reliably places content inside specified tags.

A few-shot prompt includes 2–5 examples of ideal input-output pairs before the actual task. Examples do not just show what to produce — they demonstrate the level of detail, vocabulary, tone, structure, and reasoning approach you expect. A good example is worth a paragraph of instructions.

Notice what the examples communicate beyond the obvious: benefit-led opener, conversational but punchy tone, specific number when possible, 3 hashtags at the end. None of this was stated in instructions — the examples showed it.

Negative examples are underused and extremely effective. If there is a specific failure mode your model keeps hitting, showing it an example of what bad looks like (labelled bad) is often faster than writing instructions that try to prevent it.

Every bad output falls into one of five categories. Name the failure mode first — then the fix becomes obvious.

A golden dataset is a set of input-output pairs representing what "correct" looks like for your specific use case. You write these manually, based on your expert judgement. Even 20 examples is enough to start. Once you have it, you can evaluate any prompt change objectively.

Most people learn to call LLM APIs by copying a quickstart example. Production code is different. It handles the failure cases — which in real-world usage are routine events, not edge cases.

The tool description is a prompt. Write it like one — specifically, clearly, with context about when to use it. A vague description leads to incorrect tool selection. A precise description leads to reliable, predictable agent behaviour.

You cannot manually review every output your agent produces. But an LLM can review outputs at scale — evaluating them against your criteria automatically. The key is writing the judge prompt with specific, measurable criteria, not "is this good?"