What the numbers on a model actually mean
A plain-language guide to the words people use when they talk about running AI models: what a token is, how a 180B model can run faster than a 27B one, what quantization trades away, and why replies slow down as a chat gets longer.
New to all of it? Read top to bottom, since each section builds on the one before. Already know the basics? Jump to any topic from the index. Each section has a nerd rating from 1 (anyone can follow it) to 5 (the deep end), most have a demo you can play with, and you can ask questions any time.
Start here: decode a model name
Tokens
Short versionModels don't read letters or words. They read tokens: chunks of text that average about 4 characters in English. Context limits, prices and speeds are all counted in tokens.
Before a model sees your text, a tokenizer chops it into pieces from a fixed vocabulary, usually 100,000 to 250,000 entries. Common words get a single token. Rare or long words get split into several. The space before a word is usually glued onto the front of it, so " cat" and "cat" are different tokens.
- Rule of thumb: 1 token ≈ ¾ of an English word. 1,000 tokens ≈ 750 words ≈ 1.5 pages.
- Code, numbers, and most non-English languages use more tokens for the same amount of meaning.
- This is why models struggle to count the r's in "strawberry". They see a couple of chunks, not ten letters.
Try it: type anything
Parameters
Short versionParameters (also called weights) are the numbers a model learned during training. The "9B" or "27B" in a model's name counts them in billions. It's the quickest guide to how much memory the model needs.
Under the hood, a language model is a long stack of matrix multiplications. The entries in those matrices are the parameters. Training nudges billions of them until the model gets good at predicting the next token. After training they're frozen, and running the model (called inference) means pushing your tokens through them.
Every parameter is a number that has to sit in memory, so the memory math is simple:
| Model size | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 9B | 18 GB | 9 GB | 4.5 GB |
| 27B | 54 GB | 27 GB | 13.5 GB |
| 120B | 240 GB | 120 GB | 60 GB |
| 552B | 1,104 GB | 552 GB | 276 GB |
More parameters usually means a more capable model, but it isn't the whole story. Training data and technique matter a lot, and a recent 9B model can beat a 70B model from two years earlier on many tasks. Big dense models have also become rare: most large new releases are MoE.
Keep one more fact in mind for later: to generate each token, the model has to read (nearly) every parameter from memory once. That single fact explains most of the speed section.
Quantization
Short versionQuantization stores each weight with fewer bits, a bit like saving a photo as a smaller JPEG. A 4-bit model takes about a quarter of the memory of the 16-bit original, and usually loses surprisingly little quality.
Models are trained in 16-bit precision (usually BF16), which can represent tens of thousands of distinct values per weight. Most weights don't need that much detail. Quantization rounds each weight to one of a small set of levels: 8-bit allows 256 levels, 4-bit allows 16, 2-bit allows only 4.
Good schemes split the weights into small blocks (often 32 to 256 weights) and give each block its own scale, so the levels fit the local range of values. Those scales take space too, which is why a "4-bit" file usually works out to 4.5–5 bits per weight.
Names you'll see
- GGUF files (llama.cpp, Ollama, LM Studio):
Q8_0,Q6_K,Q5_K_M,Q4_K_M,Q3_K_M,IQ2_XS… The number is roughly the bits per weight. - Other formats: AWQ, GPTQ, EXL2/EXL3, MLX (Apple), and FP8 / FP4 on newer GPUs.
How much quality do you lose?
- 8-bit: practically indistinguishable from the original.
- 6- and 5-bit: very close. Hard to notice in normal use.
- 4-bit (Q4_K_M): the popular sweet spot. Small but measurable loss.
- 3-bit and below: noticeably worse. Huge models tolerate it better than small ones.
A useful rule: at the same memory budget, a bigger model at 4-bit usually beats a smaller model at 8-bit.
Try it: round the weights
Dense vs Mixture of Experts
Short versionA dense model uses all of its parameters for every token. A Mixture-of-Experts (MoE) model is split into many "experts" and only runs a few of them per token. You get the knowledge of a big model at the speed of a smaller one, but you still need memory for the whole thing.
In an MoE model, part of each layer (the feed-forward block, which holds most of the parameters) is split into many separate experts, anywhere from 8 to several hundred. A small router looks at each token and picks the few experts that should handle it. The rest sit idle for that token. Attention and a few other shared parts always run.
So an MoE model has two sizes that matter:
- Total parameters decide how much memory you need. All experts have to be loaded, because any token might need any expert.
- Active parameters decide how much work each token takes, and so how fast it runs.
| Model | Total | Active per token |
|---|---|---|
| Qwen3.8 27B (dense) | 27B | 27B |
| Qwen3.6-35B-A3B | 35B | 3B |
| gpt-oss-120b | 117B | 5.1B |
| Qwen3.8-Flash-Next | ~180B | 6B |
| DeepSeek V4.1 Flash | 552B | 16B |
Names often spell this out: Qwen3.6-35B-A3B means 35B total, 3B active. Under the hood it has 256 experts and uses 8 of them per token, plus one "shared" expert that always runs.
Totals can get fuzzy. Qwen3.8-Flash-Next has a 125B core, plus a 51B n-gram lookup table and a 4B MTP module, so you'll see it quoted as anything from 125B to 180B. DeepSeek V4.1 Flash similarly carries about 196B parameters of n-gram tables (its Engram module).
Common misconception
The experts aren't "the math expert" and "the poetry expert". Routing happens per token, per layer, and a single sentence bounces between many experts. Whatever specialization they develop is statistical and hard for humans to read.
On quality, an MoE model usually lands somewhere between a dense model of its active size and a dense model of its total size.
Watch it: one token at a time
Dense 27B (Qwen3.8 27B)
MoE 35B-A3B (Qwen3.6-35B-A3B)
Attention
Short versionAttention is how each token looks back at earlier tokens to decide which ones matter for it right now. It's what lets the model work out that "it" refers to "the cat".
For every token, the model computes three vectors:
- Query: what this token is looking for.
- Key: what this token offers, used for matching.
- Value: the information this token passes along if it's picked.
A token compares its query against the keys of every earlier token, turns those scores into percentages, and takes a weighted blend of their values. A model runs dozens of these attention heads in parallel per layer, each tracking different kinds of relationships, across dozens of layers.
In chat models, attention is causal: a token can only look backward, never at tokens that come after it.
The cost: each new token compares itself against every previous token. The longer the context, the more work each step takes. The keys and values from all those earlier tokens are what the next section is about.
Try it: click a word to see where it looks
KV cache & context
Short versionThe KV cache stores the keys and values for every token already in the conversation, so they don't have to be recomputed for each new token. It makes generation practical, but it grows with every token, and at long context it can take as much memory as the model itself.
Without a cache, generating token number 1,001 would mean reprocessing all 1,000 tokens before it, then doing it again for token 1,002. With the cache, each new token computes its own key and value once, adds them to the pile, and reads the pile for attention.
The cache holds one key and one value per token, per layer, per KV head:
- Context window is the maximum number of tokens the model can handle at once: your prompt, the chat history, and its reply. Setting a 128k context in your app reserves room for a cache that big.
- GQA (grouped-query attention) lets several query heads share one KV head. Llama 2 7B (2023) had 32 KV heads; Llama 3.1 8B (2024) has 8, which makes its cache 4× smaller. They're old, but they show the jump clearly in the demo below.
- Hybrid attention only gives some layers full attention. Qwen3.8 27B has 64 layers, but 48 of them use a "linear attention" variant (Gated DeltaNet) that keeps a small fixed-size state instead of a growing cache. Only 16 layers keep a KV cache, so its cache is a fraction of what a 64-layer model would normally need.
- KV cache quantization stores the cache in 8-bit (or 4-bit), halving it or better, at a small quality cost.
- Other tricks: sliding-window attention, and DeepSeek's compressed attention. DeepSeek V4.1 Flash stores its cache in 4-bit and squeezes it to about 890 bytes per token, so even a 1M-token context needs under 1 GB.
This is also why models get slower as the conversation grows: every new token has to read the whole cache, on top of the weights.
Try it: how big does the cache get?
Reasoning tokens
Short versionReasoning models write out a working-out before the answer. Those thinking tokens are generated one at a time like any other token, billed at the output rate, and they fill up context and KV cache. A thinking budget (or "effort" setting) caps how long that working-out is allowed to get.
Some models take a detour before answering. Instead of predicting the reply directly, they first write a private stretch of text: restating the problem, trying an approach, spotting a mistake, backing up. These are reasoning tokens, also called thinking tokens. Almost every major model can do this now; you control it with settings like extended thinking or reasoning effort.
The mechanism is not special. The model runs the same decode loop and produces one token at a time, and each one costs a full pass over the weights, exactly as described in Prefill & decode. Thinking is simply generation that happens before the part you read.
Where it came from
Reasoning models went mainstream with OpenAI's o1 (previewed in September 2024, fully released in December 2024) and DeepSeek-R1 (January 2025). R1 shipped open weights and a paper on how it was trained, so open models picked up the technique within months.
Today basically every frontier model is a thinking model. Early on, labs shipped a separate "-thinking" or "-reasoning" variant next to the plain one. Those suffixed models are being deprecated: thinking is built into the main model, and you turn it up, down or off with a setting.
What they cost you
- Time. At 40 tokens per second, 2,000 hidden thinking tokens is 50 seconds of staring at a spinner before the first visible word. The speed described in Tokens per second applies to thinking tokens too, so a reasoning model feels much slower than a plain one on the same hardware.
- Money, as output. Providers bill reasoning tokens as output tokens, usually even when they are hidden or only summarised. A reply of 300 visible tokens that needed 2,000 thinking tokens is billed as 2,300 output tokens. At $3 per million output tokens that is about $0.007 instead of $0.001.
- Context and KV cache. Thinking tokens are tokens in the sequence, so they land in the KV cache and count against the context window. A model that thinks for 20,000 tokens has spent 20,000 tokens of room before it writes anything, on top of your prompt. See KV cache & context. Some APIs drop earlier turns' reasoning instead of sending it back, which saves room but makes the model redo work.
- Your output limit. Thinking tokens usually count against the maximum output length. Set the limit to 1,000 while the model wants to think for 3,000 and the answer gets cut off or never arrives.
Thinking budgets
Most providers give you a dial for how much thinking is allowed. The names differ, the idea is the same:
- A token cap (max thinking tokens, thinking budget): a hard ceiling on the reasoning phase before the answer starts.
- Effort levels (low, medium, high): the model is steered to think briefly or at length. On some models it also chooses how many tokens to use, so the same prompt can spend far less on low.
- Switching it off: many open models ship a thinking mode you can disable entirely, trading some accuracy on hard problems for much lower latency and cost.
A budget that is too small is worse than none: the model can run out mid-thought and answer from a half-finished plan. If you cap thinking, give the answer room too.
When it pays off
Thinking helps on multi-step work: arithmetic, code, puzzles, careful extraction, anything where a wrong first guess is expensive. It adds little on short factual questions and costs a lot on bulk jobs, where the extra tokens per document multiply across thousands of requests. A useful habit is to leave thinking on for hard requests and switch it off for the routine ones.
Tool calling
Short versionA model can't search the web or run code on its own. Tool calling means it writes a request instead of an answer: a tool name plus arguments, as JSON. Your app runs the tool, pastes the result back into the conversation, and calls the model again to finish the reply.
Everything a model does is still predicting tokens. Tool calling (also called function calling) doesn't change that. It's a convention the model was trained on: when a question needs outside information or an action, it writes a structured call and stops, then waits for the result to show up in its context.
One round trip
- Your app describes the tools. Each one gets a name, a one-line description and a JSON schema for its arguments. The chat template turns these into ordinary text at the top of the prompt.
- The model writes a call. Instead of answering, it emits something like
{"name": "get_weather", "arguments": {"city": "Osaka"}}inside special markers, then ends its turn. - The server parses it. llama.cpp, vLLM or a hosted API spots the markers and hands your app a clean
tool_callslist instead of plain text. - Your app runs the tool. The model is idle here. Your code calls the weather API, runs the query, or reads the file.
- The result goes back in as a tool message, and the whole conversation is sent to the model again. Now it can answer, or call another tool.
An agent is this loop on repeat: call, read the result, decide the next call, until the job is done. The Ask tab on this site works this way: the model calls a search tool to find the right sections of this wiki before it answers.
What it costs
- Prompt tokens on every request. Tool definitions sit in the prompt each time. A few hundred tokens per tool is common, so a big toolbox can add thousands of tokens before the user has typed anything.
- A full model call per step. Each round trip is a new request with its own prefill and decode. Prompt caching helps: the server can reuse the KV cache for the part of the conversation it has already seen.
- Results fill the context. A fetched web page or a long file listing can be tens of thousands of tokens, and it stays in the conversation for every later step.
Running it locally
The model has to be trained for tool calling, and the server has to know that model's exact call format. That format lives in the model's chat template (stored inside the GGUF for llama.cpp). Most recent open models, Qwen and DeepSeek included, support it well. Small or heavily quantized models are more likely to write broken JSON or pick the wrong tool. Servers can prevent the broken JSON with constrained decoding, which only allows tokens that keep the output valid against the schema.
MCP (Model Context Protocol) is a separate, higher-level piece. It's a standard way for apps to offer tools to a model, so one weather or database tool server can plug into many different chat apps and agents. Underneath, the model still sees tool definitions and writes calls exactly as above.
Step through it: one question, one tool call
Activations
Short versionActivations are the in-between numbers the model computes as your tokens pass through each layer. If weights are the recipe, activations are the cooking in progress. When running a model they're small and short-lived. When training one, they're the main reason it needs so much more memory.
Each layer takes a list of numbers per token (its input), multiplies it through the layer's weights, and hands a new list of numbers to the next layer. Those lists are the activations. For a model with a hidden size of 5,120, each token carries 5,120 numbers from layer to layer, and the feed-forward block briefly expands that to a much wider list in the middle.
Three kinds of memory are easy to mix up:
| What | Lives | Grows with |
|---|---|---|
| Weights | The whole time the model is loaded | Model size and quantization |
| KV cache | As long as the conversation | Tokens in the context |
| Activations | One layer at a time, then thrown away | Tokens processed at once |
Why you'll run into them
- Prefill uses more. During prefill, hundreds of tokens go through each layer together, so the scratch space is much bigger than during decode. In llama.cpp this shows up as the "compute buffer", and the
-ub(micro-batch) setting controls how many tokens go through at once: bigger is faster prefill but more memory. - Training keeps them all. To learn, the model has to work backward through every layer, which needs every layer's activations. That's why fine-tuning a model takes several times the memory of running it.
- Quantization names. Labels like
W4A16orW8A8mean "weights in 4-bit, activations in 16-bit" and so on. Activations are harder to squeeze than weights because they have occasional huge outlier values.
Watch it: what stays in memory
VRAM vs unified memory
Short versionA graphics card has its own small, very fast memory (VRAM). Unified-memory machines like Macs share one big, slower pool between CPU and GPU. VRAM wins on speed. Unified memory wins on how big a model you can load.
Discrete GPU + VRAM
VRAM sits right next to the GPU chip and is extremely fast: 1,000–1,800 GB/s on a high-end card. But there isn't much of it: 8–32 GB on consumer cards, 96 GB on workstation cards. Your regular system RAM is slower (about 80–100 GB/s for a typical dual-channel DDR5 desktop) and sits on the other side of the PCIe bus.
When a model doesn't fit, runtimes like llama.cpp split it: some layers on the GPU (-ngl), the rest on the CPU. Speed drops toward the speed of the slow part.
There's a good trick for MoE models: keep attention and shared layers on the GPU, and put only the expert weights in system RAM (llama.cpp's --n-cpu-moe or --override-tensor). Each token only touches a few experts, so not much has to be read from slow memory. This is how people run 100B+ MoE models on one gaming GPU plus a lot of RAM. Models with n-gram tables go a step further: the table can sit on an NVMe SSD.
Unified memory
Apple Silicon Macs, AMD Ryzen AI Max+ ("Strix Halo") machines and NVIDIA's DGX Spark have one shared pool that both CPU and GPU use, with 128–512 GB available and 250–820 GB/s of bandwidth. They can load models no consumer GPU can hold. The tradeoffs: generation is slower than the same model in VRAM, and prefill is noticeably slower because these chips have less raw compute.
On a Mac, the GPU can only use part of the memory by default (roughly two-thirds to three-quarters). You can raise the limit with the iogpu.wired_limit_mb setting.
Rule of thumb
- Small and mid-size dense models: a GPU with enough VRAM is fastest.
- Big MoE models: unified memory, or a GPU plus lots of system RAM with expert offloading.
Try it: what can hold what
Prefill & decode
Short versionEvery request runs in two phases. Prefill reads your whole prompt at once and is fast per token. Decode writes the reply one token at a time, and that's the speed you feel while watching it type.
Prefill (prompt processing)
All the prompt tokens are already known, so the model can process them in parallel with big, efficient matrix multiplications. The limit is raw compute. Benchmarks call this pp (prompt processing) speed, often hundreds to thousands of tokens per second. It sets your time to first token (TTFT): paste a 50-page document and this is the wait before anything appears. CPUs and Macs have less compute than big GPUs, so long prompts hurt most there.
Decode (generation)
Each new token depends on the one before it, so they have to come out one at a time. Every step reads all active weights from memory to do a fairly small amount of math. The limit is memory bandwidth, not compute. Benchmarks call this tg (text generation) speed.
That's why a tool like llama-bench reports two numbers, such as pp512 and tg128, and why a machine can be great at one and mediocre at the other.
Watch it
Tokens per second
Short versionGeneration speed is mostly a memory-bandwidth problem. A rough ceiling is your memory bandwidth divided by the size of the active weights.
During decode, the hardware streams every active weight through the processor once per token. So:
Example: an RTX 4090 moves about 1,000 GB/s. Qwen3.8 27B at Q4_K_M is about 16 GB. The ceiling is about 60 tokens/sec, and real software reaches maybe 60–80% of that.
This one formula explains a lot:
- Quantization speeds things up, because fewer bytes have to be read per token.
- MoE models are fast for their size, because only the active experts get read.
- Long chats slow down, because the KV cache gets read every token too.
- Not fitting in VRAM is brutal. Layers that spill into system RAM run at system-RAM bandwidth (often 10× slower), and the slow part dominates.
For a sense of scale: people read at roughly 5–10 tokens/sec. Around 20+ feels quick for chat. Coding agents and reasoning models, which generate lots of text you never read, benefit from much more. The next section lets you see the difference.
Try it: estimate your speed
Feel the speed
Short version"40 tok/s" is hard to picture. Here's the same answer streaming at different speeds, side by side.
Roughly what each speed is like in practice:
- ~5 tok/s: a big model mostly running from system RAM. About reading speed, so chat is usable but long answers drag.
- ~20 tok/s: a mid-size model on a Mac or a squeezed GPU. Comfortable for chat.
- ~60 tok/s: a dense model that fits in a fast GPU. Faster than you can read.
- ~150 tok/s: a small-active MoE on a good GPU. Code appears in chunks.
- 1,000+ tok/s: specialized inference chips or diffusion models. The answer is basically just there.
Race: same answer, different speeds
Concurrency & batching
Short versionServing several conversations at once is far more efficient than serving one, because the weights are read once per step no matter how many conversations ride along. Each person gets a little slower, the total goes way up, and the limit is usually memory for everyone's KV cache.
Remember that decode is limited by reading weights, not by math. If one conversation needs the GPU to read 16 GB per token, ten conversations can share that same read: the GPU does ten tokens' worth of math on the weights it already pulled in. This is batching.
- Per-user speed drops slowly as you add users, because each one adds a bit of KV cache reading and math.
- Total throughput (all users' tokens per second added up) climbs steeply, until the GPU's compute becomes the limit.
- Memory caps it. Every conversation needs its own KV cache. Long contexts mean fewer users fit.
In practice
- llama.cpp (
llama-server):--parallel N(-np) sets how many conversations ("slots") run at once. Depending on version and flags, the context size you set is either split between slots or shared by them, so raise it when you add slots. - vLLM, SGLang: built for this. Continuous batching slots new requests in as soon as others finish, and paged KV caches avoid wasting memory.
- Batching helps MoE models less: different conversations pick different experts, so more weights get read per step.
- Running agents or batch jobs? Several parallel requests can finish far sooner than the same requests one after another.
Try it: build your own server
Groq, Cerebras & fast chips
Short versionGPUs are held back by how fast they can read weights from memory. Groq and Cerebras get around that by keeping the whole model in memory built right into the chip (SRAM), which is far faster. One user can get 500 to 2,000+ tokens per second. The catch: that memory is tiny, so a single model needs hundreds of chips or several dinner-plate-sized wafers.
Recall the speed formula: decode speed ≈ memory bandwidth ÷ active weights. A GPU keeps its weights in HBM, stacks of memory chips sitting next to the processor. Even a GB300 reads about 8 TB/s.
SRAM is memory built into the processor itself, the kind GPUs use for their small caches. It's dramatically faster, but it takes up a lot of chip area, so there's very little of it. The SRAM approach is to spread the model across enough chips that every weight lives in SRAM, and the bandwidth problem mostly goes away.
Groq: lots of small chips
Groq's LPU has about 230 MB of SRAM per chip and no HBM at all. A big model is split across hundreds of chips wired together, each holding a slice. Its compiler plans every data movement ahead of time, with no caches or guesswork, so timing is predictable and latency is very low.
Cerebras: one giant chip
Cerebras uses an entire silicon wafer as a single chip. The WSE-3 has about 900,000 cores and 44 GB of SRAM on one wafer, with on-chip bandwidth measured in petabytes per second. Bigger models are split across several wafers. In independent benchmarks it's usually the fastest for a single user: around 1,700+ tok/s on gpt-oss-120b.
Others
- SambaNova mixes SRAM, HBM and regular memory in tiers, so one system can hold more models.
- Google TPUs, Amazon Trainium and similar chips use HBM like GPUs. They compete on cost and scale more than on single-user speed.
- Transformer-specific chips (such as Etched's) hard-wire the transformer's math for efficiency, at the cost of flexibility.
The tradeoffs
- Capacity is expensive. Hundreds of chips per model means these providers offer a shorter list of models, and long contexts need room for the KV cache too.
- Speed per user vs cost per token. GPUs serving many users at once (batching) are often cheaper per token. SRAM chips win when one user's speed matters: coding agents, reasoning models, voice assistants that must answer instantly.
- Precision. To fit, models are sometimes served at reduced precision. Check what a provider runs if small quality differences matter to you.
Try it: how many chips to hold a model?
Measured: gpt-oss-120b, output tokens per second for one user
MTP & speculative decoding
Short versionDecode is limited by reading weights, not by math, so checking several guessed tokens costs about the same as generating one. Speculative decoding and multi-token prediction (MTP) use that to get several tokens per pass, without changing what the model would have written.
Speculative decoding
A small, fast draft model guesses the next few tokens. The big model then checks all the guesses in a single pass, the same way it handles a prompt during prefill. It keeps every guess up to the first wrong one, and replaces the wrong one with its own token. Wrong guesses waste a little compute but never change the output: you get exactly what the big model would have produced alone.
Multi-token prediction (MTP)
Instead of a separate draft model, the model is trained with extra prediction heads that guess the tokens 2, 3 or more steps ahead. DeepSeek-V3 popularized this, and many newer open models ship with MTP layers. Qwen3.8-Flash-Next, for example, includes a 4B MTP module. At inference time those heads can act as a built-in draft model. Support varies: some runtimes use the MTP layers, others skip them and run the model normally.
How much faster?
It depends on how often the guesses are right. Predictable text like code, JSON, or rewriting a document you pasted can see 2–3× speedups. Creative writing sees much less. There are also draft-free tricks that reuse text already in the prompt, covered in n-grams.
Watch it: drafts get checked in one pass
N-grams
Short versionAn n-gram is just a run of n tokens in a row. Counting them was how language models worked before neural networks. Today they're back as a free source of guesses for speculative decoding, and as extra memory inside some new models.
A 2-gram (bigram) is two tokens in a row, like "peanut butter". A 3-gram (trigram) is three, like "peanut butter and".
The old way: counting
Early language models just counted. After "peanut butter and", the word "jelly" showed up most often in the training text, so that's the prediction. It works for common phrases and falls apart quickly: most longer runs of words never appear in any training text, so the counts have nothing to say. Neural networks replaced this because they generalize to sentences they've never seen.
Back again: n-gram speculation
When you ask a model to edit code or summarize a document, its answer often repeats chunks of your prompt. N-gram (or "prompt lookup") speculation takes the last few tokens the model wrote, finds the same sequence earlier in the context, and offers whatever came next as a draft. No draft model is needed, so the guesses are free. llama.cpp and vLLM both support this kind of drafting.
Back again: n-gram embeddings
Some new models add a big lookup table keyed on short runs of recent tokens, which gives the model extra stored knowledge for common phrases. The idea is to let a cheap table handle memorizing, so the network itself can spend its effort on reasoning.
- DeepSeek V4.1 Flash uses DeepSeek's Engram module: about 196B parameters of tables, split across two layers. For each token, it hashes the last 2, 3 and 4 tokens, looks up a few rows, and blends them into the model.
- Qwen3.8-Flash-Next has a 51B-parameter n-gram table on top of its 125B core.
Looking up a few rows is almost free compared to running the network, so these tables add a lot of memory but little work per token.
Put the table on your SSD
Here's the useful part for running these models at home. Which rows get fetched depends only on which tokens came before, not on anything the model computes, so the runtime knows what it needs ahead of time. It can fetch the rows early, while the GPU is still busy with the previous layers. That makes it practical to keep the table somewhere much slower and cheaper than VRAM:
- NVMe SSD: the popular choice. SemiAnalysis served DeepSeek V4.1 Flash with its ~189 GiB of Engram tables memory-mapped from a local NVMe drive. That's ~189 GiB that doesn't need to fit in VRAM or RAM at all.
- System RAM: faster, if you have room. SGLang can keep Engram tables in host memory to free GPU memory for the KV cache.
- Tiered: the frequently used rows cached in VRAM or RAM, and the long tail of rare phrases on the SSD.
Compare that with ordinary weights: putting a dense layer or an MoE expert on an SSD is painfully slow, because the model has to read it every time it's needed and can't know in advance which experts it will pick. N-gram tables are the rare case where SSD-speed storage barely costs you speed.
Try it: count n-grams and draft from the prompt
Counting: what usually comes next
Prompt lookup: free draft
mul copies the - from sub, so the model would accept the first guesses and reject that one.Diffusion vs autoregressive
Short versionMost LLMs are autoregressive: they write one token at a time, left to right. Diffusion language models start with a blank draft and fill in many tokens at once over a few rounds of refinement, like an image sharpening out of noise.
Autoregressive (almost every chat model)
Predict a token, add it to the text, predict the next one. Everything earlier in this guide (decode, KV cache, speculative decoding) is about making this loop faster. Its weakness is that it's strictly one after another: a 1,000-token answer needs 1,000 steps, and the model can't go back and fix an early word.
Diffusion
The model starts with a fixed-length block of "masked" placeholders. Each step, it predicts every position at once and locks in the ones it's most confident about. After a handful of steps, the whole block is filled. Because many tokens appear per step, diffusion models can be very fast, and they can revise earlier parts of the text, which helps with things like filling in the middle of code.
Examples: Inception Labs' Mercury, Google's experimental Gemini Diffusion, and open research models such as LLaDA and Dream. Some newer designs mix the two ("block diffusion"): they write left to right in blocks, and fill each block with diffusion.
The tradeoffs
- Quality has trailed the best autoregressive models, though the gap has been shrinking.
- Output length is usually set in advance per block, and tooling like llama.cpp support is far less mature.
- Image generators like Stable Diffusion and Flux use the same idea on pixels. That's where the name comes from.
Watch it: same sentence, two ways
Autoregressive: one token per step
Diffusion: fill everything, sharpen over steps
Decision models
Short versionA decision model doesn't write anything. You give it some text plus a fixed list of possible answers, and it returns a probability for each answer in a single pass. With no tokens to generate, an answer takes tens to hundreds of milliseconds instead of seconds, and it can never come back in the wrong format.
Plenty of what apps ask an LLM isn't really writing. Which team should get this ticket? Is this message a prompt injection? Should the agent use the search tool or the calculator next? The answer is one item from a short list, but a normal LLM still has to decode its reply one token at a time, often as JSON that might not parse.
Decision models became their own category in September 2026, when TypeSafe AI launched Jev. Within a few weeks there were several more, including Cloudflare's open-weight Clef (27B) and Clef-flash (9B), and OpenAI's Decisions API.
How they work
- You write the question and the allowed answers. For example "Which team handles this?" with
billing,returns,techandaccount, or a plain yes/no. - The model reads everything in one pass, like prefill, and gives each allowed answer a score. The scores are turned into probabilities that add up to 100%.
- You get a typed result: the top answer, the probability of every option, and how confident it is. Because it only ever picks from your list, the output is always valid.
Vendors don't all say how their scores are made. Common approaches are reading a language model's probabilities for each option, or putting a small scoring head on top of an existing model. Several of the open ones are post-trained versions of open models like Qwen.
Images and audio too
The input doesn't have to be text. Liquid AI's open-weight d1 models (October 2026) are built for small devices: d1-3B takes text and images, and d1-omni-600M, an early experimental release, takes text plus either an image or audio. The picture or sound goes in as tokens, the same way as in any multimodal model, and the answer still comes out of one pass. That suits questions like "is anything blocking this doorway?" for a camera feed, or "does this caller sound upset?" for a call line, on hardware right next to the camera or microphone. Liquid reports 8 ms per question for d1-3B on an RTX 4090 and 50 ms on a Jetson Orin Nano, and it runs in llama.cpp.
Why the probabilities matter
The useful part isn't only the answer, it's the confidence. If the model says "billing, 93%", you can route the ticket automatically. At "returns, 61%", you send it to a person or to a full LLM instead. That only works if the numbers are calibrated: answers given at 90% should be right about 90% of the time. Vendors train for this, but calibration measured on their data doesn't automatically carry over to yours, so check it on a sample of your own labelled examples.
Limits
- It can only pick from your list. If none of the answers fit, it still picks one. Add an "other" option.
- Always valid isn't always right. A perfectly formatted answer can still be the wrong answer, and confident wrong answers happen.
- No explanation. You get a choice and a number, not reasoning you can audit.
- Small details shift the result. Option order and wording can move the probabilities, and a prompt tuned for one decision model can make another one worse.
- Not automatically better. In Red Hat's independent tests on prompt injection and content safety, decision models were not clearly faster, cheaper or more accurate than an LLM judge, and a small classifier fine-tuned for the job was faster than both.
The idea isn't entirely new. Zero-shot classifiers and rerankers have scored fixed options for years. What's new is building it on top of an LLM, so one model can answer any typed question you write, with no training of your own. A good fit is a fast gate around an LLM: decide cheaply, and only call the big model when something actually needs writing. You can also approximate one locally with any model: ask a multiple-choice question and read the probability of each option's first token from the server's logprobs, instead of letting it write a reply.
Try it: route a support ticket
Images, video & audio
Short versionModels "see" and "hear" by turning images, video frames and sound into tokens the language model can read. That's why one photo can cost hundreds or thousands of tokens of context, and why a video can fill a context window.
Images
A vision encoder cuts the image into small square patches (often 14 or 16 pixels across) and turns each patch into a vector. Neighbouring patches are usually merged (2×2 is common) to save tokens. A small projector then maps those vectors into the same space as the model's text tokens, and the language model reads them like any other tokens. Higher resolution means more patches, which means more tokens and more prefill time.
In llama.cpp, the vision part ships as a separate mmproj file that you load next to the main GGUF.
Video
Video is handled as a series of images, sampled at a low frame rate (often 1–2 frames per second), usually at a lower resolution, and often with neighbouring frames merged. Even so, a few minutes of video can run to tens of thousands of tokens.
Audio
Sound is turned into a spectrogram (a picture of which frequencies are present over time), and an audio encoder, often based on Whisper, turns slices of it into tokens: typically tens of tokens per second of audio. Models that can also speak back (often called "omni" models) add a part that turns output tokens back into sound.
What this means for you
- Images and video eat context and make prefill slow, especially on Macs and CPUs.
- Downscaling an image before sending it can save a lot of time, if fine detail doesn't matter.
- Small text in screenshots needs high resolution to read, which costs more tokens.
Try it: how many tokens is that?
Embedding models
Short versionAn embedding model turns text into one fixed-length list of numbers, a vector that stands for its meaning; similar texts get similar vectors. It never writes an answer: search and RAG use it to find passages before a chat model reads them. Newer ones like EmbeddingGemma 2 also embed images, audio and video into the same space, so a text query can fetch a photo.
An embedding model is a language model with a different job. A chat model takes text in and writes text out, one token at a time. An embedding model takes text in and gives you back a single vector: a list of a few hundred or few thousand numbers that stands for what the text means. There is nothing to read in the reply.
One text in, one vector out
Inside it is the same kind of transformer as a chat model, with the same layers and attention. The difference is the end. There is no loop that samples the next token. The model runs one forward pass, then squeezes the per-token results into one vector: older BERT-style models average all the token vectors (mean pooling), and models built from chat models, like Qwen3-Embedding, take the last token's vector. The result is finally scaled to length 1, so it is a direction rather than a stretch.
The length of that vector is fixed by the model, not by the text. Common sizes are 384, 768, 1,024, 1,536 and 3,072 numbers. Every text you pass in, a single word or a whole page, comes back as exactly that many numbers.
Why similar meaning lands nearby
Embedding models are trained on pairs of texts that should match: a question and the passage that answers it, a title and its body, two phrasings of the same question. The training pulls paired texts towards each other and pushes unrelated texts apart. This is called contrastive training. The popular all-MiniLM-L6-v2 was fine-tuned on over a billion such pairs.
Once the vectors exist, closeness is just cosine similarity: 1 means the same direction, 0 means unrelated. If the vectors are already unit length, the cosine is simply their dot product. In the Qwen3-Embedding example, "What is the capital of China?" scores 0.76 against "The capital of China is Beijing" and 0.14 against a passage about gravity.
Typical sizes and context limits
Embedding models are usually much smaller than chat models, because they only need to compress text, not reason or write. all-MiniLM-L6-v2 has about 23M parameters, EmbeddingGemma 300M, EmbeddingGemma 2 740M, BGE-M3 about 0.6B, and the Qwen3-Embedding series runs from 0.6B to 8B. A small one runs comfortably on a CPU: there is no decode phase, so the cost is one prefill pass per text, and thousands of short chunks are practical without a GPU.
| Model | Dimensions | Max input |
|---|---|---|
| all-MiniLM-L6-v2 | 384 | 256 tokens |
| bge-small-en-v1.5 | 384 | 512 tokens |
| EmbeddingGemma | 768 (cuttable to 512, 256, 128) | 2,048 tokens |
| EmbeddingGemma 2 | 768 (cuttable to 512, 256, 128) | 8,192 tokens, shared with images and audio |
| nomic-embed-text-v1.5 | 768 (cuttable to 512, 256, 128, 64) | 8,192 tokens |
| bge-m3 | 1,024 | 8,192 tokens |
| Qwen3-Embedding-0.6B | 1,024 (settable from 32) | 32,768 tokens |
| text-embedding-3-small (OpenAI) | 1,536 | 8,192 tokens |
| text-embedding-3-large (OpenAI) | 3,072 | 8,192 tokens |
Context limits vary more than you would guess, from 256 tokens to 32k. Text past the limit is usually cut off silently, so check the number before you chunk a document. Several newer models let you shorten the vector, often called Matryoshka representation learning: you keep the first 256 or 512 numbers and re-normalize. OpenAI exposes this as a dimensions setting, and Google reports EmbeddingGemma losing about 1 point of MTEB average going from 768 to 256.
Multimodal embedding models
Text embeddings only help when what you want is text. A multimodal embedding model puts images, audio and video in the same vector space as text, so a text query can retrieve a photo, and one index can hold text chunks and page images side by side.
The first generation, like CLIP and SigLIP, used two towers: one encoder for images and one for text, trained on paired image and caption data so that "a dog on a beach" lands near a photo of one. That is strong at text-to-image search, but each tower only ever sees its own modality, so a page with a chart and a paragraph becomes a picture with no interaction between them.
Newer models do what chat models do with images (images, video & audio): an encoder turns the media into tokens, and the same transformer reads them interleaved with text. Google's EmbeddingGemma 2 is the open example. It is 740M parameters in total: a 270M text backbone, a 170M vision encoder and a 300M audio encoder, all mapping into one 768-number vector. The encoders are optional, so a text-only deployment loads 270M, text plus image is 440M, and the full model is 740M. One input can mix them, for example Waterproof running shoes. <|image|> breathable mesh <|image|> grip test: <|video|>, and out comes a single vector you can compare against a plain text query.
Media eats context at a fixed rate, which sets how much you can embed in one go:
| Input | Tokens used | Fits in 8,192 tokens |
|---|---|---|
| Text | 1 per subword token | the whole window |
| One image | 280 by default, tunable from 70 to 1,120 | about 29 images |
| Video | 140 per sampled frame, 1 frame per second by default | about 58 frames, roughly a minute |
| Audio | 25 per second of 16 kHz mono audio | about 5 minutes |
Cross-modal scores are usually lower, and always differently spread, than text-to-text scores: in Google's own example the video embedding scores 0.688 for "a cat dozing" and 0.440 for "a sports car racing on a highway", so a threshold you tuned on text will not carry over. For visually rich PDFs, where OCR loses the tables and layout, jina-embeddings-v4 is a local alternative: it is built on Qwen2.5-VL-3B, embeds text and images into 2,048 numbers with 32k of input, and has no audio. It ships under a research licence rather than Apache 2.0. Gemini Embedding 2 handles all four modalities, but only through Google's API.
What RAG does with them
- Split your documents into chunks that comfortably fit the model's limit, typically a few hundred tokens each.
- Embed every chunk once and store the vector next to the text. This is what a vector database is for, though a file of vectors with FAISS or SQLite also works. In a multimodal index, store the image or page picture next to its vector too.
- Embed the user's question with the same model, then take the handful of chunks whose vectors are closest.
- Paste those chunks into a chat model's prompt and let it write the answer. They fill its context, so keep the number retrieved small. If a hit is an image, pass the image to a chat model that can read pictures.
Storage adds up: 1,000,000 chunks at 768 dimensions is 768,000,000 numbers, about 3 GB as 32-bit floats and half that in 16-bit. That is why people shorten vectors and why quantization exists for embedding models too.
Things that trip people up
- The same model must embed the documents and the query. Vectors from two different models are not comparable, and switching models means re-embedding everything.
- Some models expect a prefix: nomic wants
search_query:andsearch_document:, EmbeddingGemma wants a task line, and Qwen3-Embedding improves a few percent with an instruction. Skip them and retrieval quietly gets worse. In EmbeddingGemma 2 the prefixes apply to text only: pass images, audio and video without one. - Run EmbeddingGemma 2 in
bfloat16orfloat32, neverfloat16. Its activations exceed float16's range, and instead of an error you get NaN or quietly worse vectors. - Closeness means "about the same thing", not "answers the question". Mixing in keyword search and adding a reranker, which scores the query and each chunk together, is the usual fix; BGE's own card recommends exactly that pipeline.
Abliteration
Short versionAbliteration removes a model's tendency to refuse requests without retraining it. Researchers found that refusing is mostly controlled by a single direction inside the model's activations. Abliteration finds that direction and edits the weights so the model can no longer move along it. Models with "abliterated" in the name have had this done.
The discovery
Inside a model, each token is represented by a long list of numbers (its activations), which you can think of as a point in a space with thousands of dimensions. In 2024, researchers showed that for many chat models, whether the model is about to refuse comes down to one particular direction in that space. Push the activations along it and the model refuses harmless requests. Remove it and the model stops refusing. The name is a mix of "ablation" (removing a part to see what changes) and "obliterate".
How it works, in three steps
- Collect examples. Run the model on two sets of prompts: ones it answers normally and ones it refuses, and record the activations at some layer.
- Find the direction. Average each set. The difference between the two averages points from "answering" toward "refusing". That's the refusal direction.
- Edit the weights. Adjust every weight matrix that writes into the model's main data stream so its output has nothing along that direction (this is called orthogonalizing). From then on, no layer can push the model toward refusing.
No training data or retraining is needed. It's a one-time edit that takes minutes, and the result is an ordinary model file you can quantize and run like any other.
What it doesn't do, and the side effects
- It doesn't add knowledge or skills. The model only stops declining. It's no smarter, and it still gets things wrong.
- Quality often drops a little. The direction isn't perfectly clean, so removing it can also nudge unrelated behavior: slightly lower benchmark scores, more made-up answers, or weaker reasoning. Some releases run a short fine-tune afterwards to recover ("healing"), and newer variants try to edit more gently.
- It's not all or nothing. Refusal isn't purely one direction, so some refusals or lectures can survive, and some models lose useful pushback too, like correcting a false premise.
- The guardrails are gone. The model's built-in safety behavior was part of what got removed, so responsibility for how it's used moves to whoever runs it. Model licenses and acceptable-use terms still apply.
Why would anyone want a model that doesn't refuse?
Mainstream chat models are tuned to be cautious with everyone, so they also turn down plenty of legitimate work. Common reasons people reach for an abliterated or less-filtered model:
- Cybersecurity. Defenders need to understand attacks: writing an exploit to prove a bug is real during an authorized penetration test, picking apart a malware sample, solving capture-the-flag challenges, or writing detection rules for a technique. Cautious models often refuse these because the same knowledge could be used to attack.
- Over-refusal of harmless requests. Questions about medication doses and harm reduction, sexual health, how historical atrocities happened, or a crime novel with a convincing villain can all trip a filter that's matching on keywords rather than intent.
- Content moderation and research. A model that labels hateful or dangerous text has to read it and describe it plainly. Safety researchers also need to see what a model would do without its refusals in order to measure risk.
- Private, local use. On your own hardware there's no provider in the loop, and some adults simply don't want to be lectured or second-guessed by software they run themselves.
The tradeoff is real, though. Abliteration is blunt: it removes the good refusals along with the bad ones, and the same model that helps a pentester will help anyone. Before reaching for it, it's worth trying a model with better-judged refusals, a clear system prompt that explains your context (for example, an authorized security assessment), or a model fine-tuned for your field, such as a security-focused model.
Related: steering
The same trick works for other behaviors. Instead of editing the weights permanently, steering adds or subtracts a direction while the model runs, to make it more formal, more concise or more upbeat, for example. Abliteration is the permanent version applied to one behavior.
Try it: remove the direction
LoRA & QLoRA fine-tuning
Short versionLoRA fine-tuning freezes the original weights and trains a small pair of extra matrices next to them, so you only train a fraction of a percent of the parameters. That cuts the memory for a 7B model from roughly 110 GB (full fine-tuning) to about 16 to 24 GB. QLoRA goes further by keeping the frozen base model in 4-bit, which brings the same job down to roughly 6 to 12 GB.
Fine-tuning means continuing to train a finished model on your own examples so it picks up a style, a format or a narrow skill. The catch is memory. Full fine-tuning updates every parameter, and that means holding far more than the weights themselves: gradients, optimizer state, and for activations, everything each layer computed on the way forward, kept so the model can work backward. That is why fine-tuning needs several times the memory of just running the model.
How LoRA works
LoRA (Low-Rank Adaptation) leaves the original weights frozen and adds a small pair of thin matrices beside each weight matrix it wants to change. Instead of learning a full 4096×4096 update, it learns two matrices of shape 4096×16 and 16×4096. The "rank" (16 here) is the dial: higher rank means more capacity and more memory.
Because the update starts at zero, the model behaves exactly like the original at step one, and the trained adapters can later be merged back into the base weights so inference costs nothing extra. A finished adapter for a 7B model is usually a file of tens of megabytes, not gigabytes.
How much VRAM compared with full fine-tuning
Full fine-tuning with the Adam optimizer needs roughly 16 bytes per parameter: 2 for the BF16 weights, 2 for gradients, 8 for Adam's two running averages and 4 for a full-precision copy. LoRA needs that for the adapter only, but still needs the frozen base weights and the activations.
| 7B model | What sits in memory | Rough VRAM |
|---|---|---|
| Run it, 4-bit | Quantized weights plus KV cache | 4–6 GB |
| Run it, BF16 | Weights plus KV cache | 14–16 GB |
| QLoRA | 4-bit frozen weights, tiny adapter state, activations | 6–12 GB |
| LoRA, BF16 base | 14 GB of weights, tiny adapter state, activations | 16–24 GB |
| Full fine-tune, Adam | Weights, gradients, optimizer, master copy | ~110 GB |
The activation row is the one that moves. It scales with batch size times sequence length, and gradient checkpointing trades speed for memory by recomputing activations instead of storing them. So a 7B QLoRA run can fit on an 8 GB card at short sequence lengths and need 12 GB or more at long ones. Bigger models scale roughly with size: 13B QLoRA lands around 10 to 12 GB, and 70B needs roughly 40 GB, so one 48 GB card or two 24 GB ones.
What QLoRA adds
QLoRA keeps the frozen base model in 4-bit (a scheme called NF4) and adds "double quantization", which also compresses the scaling factors. It unpacks the weights on the fly for each matrix multiply, so training steps are slower than LoRA on a 16-bit base, but the memory saving is large. The original QLoRA work fine-tuned a 65B model on a single 48 GB GPU and reported quality matching 16-bit LoRA on their benchmarks.
What you give up
- Less room to learn. Rank 8 to 64 handles style, format and task behavior well. Teaching a new language or a large body of new facts usually wants a higher rank or full fine-tuning.
- Nothing is added to the base model. Like abliteration, an adapter is a small edit on top of what the model already knows. It can imitate a style it has seen but struggles to teach genuinely new skills.
- Quality still depends on data. A few hundred good examples beat tens of thousands of sloppy ones, and the model will faithfully learn your mistakes.
Common tools for this are PEFT and TRL, Unsloth, Axolotl and LLaMA-Factory; on Apple hardware, MLX has its own LoRA training. All of them take a base model in quantized or BF16 form and produce an adapter file.
Quick glossary
- Abliterated
- A model whose "refusal direction" has been edited out of its weights, so it no longer declines requests. See abliteration.
- Adapter
- The trained LoRA weights on their own, usually a file of tens of megabytes that can be merged into the base model or loaded alongside it.
- Batching
- Running many requests together so they share each read of the weights.
- BF16 / FP16
- 16-bit number formats. Most models are trained and released in BF16, the "full precision" baseline.
- Calibration
- Whether a model's confidence matches reality: answers given at 90% should be right about 90% of the time.
- Chunk
- A slice of a document, short enough to fit the embedding model's context limit, that gets its own vector.
- Context window
- The maximum tokens the model handles at once, prompt and reply together.
- Continuous batching
- Adding and removing requests from a running batch as they start and finish, instead of waiting for the whole batch.
- Cosine similarity
- A score from -1 to 1 for how closely two vectors point in the same direction. 1 means identical direction, 0 means unrelated.
- Decision model
- A model that picks from a fixed list of answers you give it and returns a probability for each, in one pass, instead of writing text. Examples: Jev, Clef and the d1 models from Liquid AI.
- Dimension
- How many numbers are in the vector, for example 384 or 768. Set by the model, not by the length of the text.
- Draft model
- A small, fast model that guesses tokens for a bigger one to check, in speculative decoding.
- Effort level
- A low/medium/high setting that steers how long a model thinks. Higher effort means more tokens, more time and more cost.
- Embedding
- A fixed-length list of numbers that represents the meaning of a piece of text. Similar meanings give similar vectors.
- Engram
- DeepSeek's n-gram memory module: large lookup tables keyed on the last few tokens, used in DeepSeek V4.1 Flash.
- Flash Attention
- A faster, memory-efficient way to compute attention. Same result, less overhead, especially at long context.
- FP8 / FP4
- Low-precision formats with hardware support on newer GPUs. A form of quantization that runs fast natively.
- GGUF
- The single-file model format used by llama.cpp, Ollama and LM Studio. Comes in many quantization levels.
- GQA / MLA
- Attention variants that shrink the KV cache, by sharing KV heads or compressing them.
- Gradient checkpointing
- A trick that throws away activations during the forward pass and recomputes them on the way back, saving memory at the cost of speed.
- Logprobs
- The log-probabilities a server can return for each generated token (and the likely alternatives). Useful for reading how sure a model was.
- LoRA
- Low-Rank Adaptation: fine-tuning that freezes the original weights and trains a small pair of extra matrices instead.
- Matryoshka representation learning
- Training an embedding so that its first 256 or 512 numbers still work on their own, letting you trade a little accuracy for much smaller vectors.
- MCP
- Model Context Protocol. A standard way for apps to offer tools to a model, so one tool server works with many chat apps and agents.
- Micro-batch (
-ub) - How many prompt tokens llama.cpp pushes through at once during prefill. Bigger is faster but uses more memory for activations.
- mmproj
- The separate vision (or audio) encoder file that gives a GGUF model image understanding in llama.cpp.
- Offloading
- Putting some layers on the GPU and the rest in system RAM. Lets bigger models run, but slower.
- Omni model
- A model that takes in text, images and audio, and can often answer with speech.
- PCIe
- The connection between a graphics card and the rest of the computer. Much slower than VRAM.
- QLoRA
- LoRA where the frozen base model is stored in 4-bit, so the same job needs far less VRAM but trains slower.
- Rank (r)
- The width of those extra matrices. Higher rank means more trainable parameters, more capacity and more memory.
- Reasoning model
- A model that writes out "thinking" tokens before answering. Better answers, but many more tokens to generate, so decode speed matters more.
- Reasoning token
- A token the model generates while working out an answer, before the reply you see. Billed as output.
- Reranker
- A model that reads a query and a candidate chunk together and scores how well the chunk answers it, which is more accurate and slower than comparing vectors.
- Slots
- llama-server's name for conversations it can handle at the same time (
--parallel). - Steering
- Adding or subtracting a direction in the activations while a model runs, to change a behavior like tone or verbosity without retraining.
- Thinking budget
- A limit on how many reasoning tokens a model may generate before it must start answering.
- Tool calling
- When a model writes a structured request (tool name plus JSON arguments) instead of an answer. Your app runs the tool and feeds the result back. Also called function calling.
- TTFT
- Time to first token: how long until the reply starts. Mostly set by prefill.
- Unified memory
- One memory pool shared by CPU and GPU, as on Apple Silicon, Strix Halo and DGX Spark. Big but slower than VRAM.
- Vector database
- A store built to hold millions of vectors and quickly return the ones nearest a query vector.
- VRAM
- Memory on the graphics card. Very fast, and usually the limit on what fits.