As hardware for running local AI models continues to move to the forefront, a new set of acronyms, definitions, and terminology is constantly popping up. Understanding these foundational concepts is essential for working with and deploying modern large language models.
This primer covers core terminology related to text-based large language models, chat applications, programming assistants, and agentic bots.
Model Architecture
An LLM is computed from training sets of mostly text data on the scale of terabytes to petabytes, sourced from the internet, books, and transcripts. The model is roughly analogous to a set of highly advanced layers of statistical analysis mapping relationships between tokens, which represent words, parts of words, computer code, punctuation, and other symbols.
The tokens themselves are not part of the model, but they are packaged along with it, and the AI model generates tokens in its outputs.
Understanding Weights
Computing layers within an AI model rely on corresponding weights to represent the strength of connections between sections of tokens. When the model runs, these values establish how tokens relate to each other.
Weights reflect how often specific words or token sequences appear in training data. Importantly, weights cannot change without retraining the model or parts of it.
LLM specifications often list parameters like 4B, 8B+, or 20B, indicating how many parameters or weights the model uses. Consumer-grade models typically measure in billions, while frontier models measure in trillions.
A model having more weights does not automatically make it superior; the weights must be meaningful for the output rather than merely scaling the parameter count.
Open-weight models refer to systems where entities freely publish model weights online, allowing developers to tweak and run them locally provided they have adequate hardware.
Quantization and Precision
Programs use numbers of varying width and precision depending on their purpose. While scientific calculations may require floating-point 64-bit arithmetic, computing memory is a finite resource, especially in VRAM-limited GPUs.
Developers use the smallest data units possible as a tradeoff between processing speed and numerical accuracy.
During training, labs use high precision for weights, but keeping high precision in final versions produces excessively large, slow LLMs. Quantization applies clever rounding during post-training passes to transform highly precise weights.
Frontier models predominantly use relatively high precision for weights, such as BF16, to maintain response quality and reasoning. Open-weight models often provide multiple quantization levels to suit varying hardware capabilities.
Lower precision allows models to run on consumer hardware, trading response accuracy for reduced memory footprint.
Context Window Limits
The context window indicates how long a session can run before performance degrades, measured in tokens and functioning analogously to computer RAM.
The context window serves as the LLM's working space, containing current questions, replies, system prompts, uploaded files, and search results.
To the LLM, all session inputs are reduced to tokens in one large pile, making strict separation between data and code technically challenging.
LLMs re-run statistical programs over all tokens to predict subsequent outputs, making larger context windows beneficial despite higher VRAM costs.
Excessively large windows can trigger the lost-in-middle effect, where models pay attention to the beginning and end of conversations while overlooking details in the center.
KV Cache Optimization
The key-value cache stores data created when a model runs that can be reused between queries in the same session, reducing redundant computational overhead.
Saved data represents relations between tokens for a given session, forming vectors used to determine subsequent token selection.
Because text generation is linear, calculating each new token requires every preceding key-value pair, making cache reuse crucial for performance.
Without a cache, long conversations with hundreds of thousands of tokens would require recalculating key-value pairs for every interaction, significantly slowing response times.
Quantization is also applied to KV cache contents to reduce precision and keep VRAM usage manageable during extended multi-turn conversations.
Mixture-of-Experts (MoE)
Recent model architectures increasingly rely on mixture-of-experts designs to scale capability without scaling active parameter counts proportionally.
An MoE model consists of sub-models trained on the same dataset with randomized initial weights, causing different sub-models to specialize in distinct token patterns.
Experts establish relations between tokens rather than understanding specific semantic topics, meaning high accuracy for certain subjects is an emergent property of token patterns.
MoE models maintain higher processing speeds by routing requests through only a subset of active experts rather than running the entire parameter set for every token.
Router mechanisms select experts on a per-token basis, allowing models to answer complex prompts more accurately and quickly at the cost of higher VRAM usage.
Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation allows chat applications to fetch external data, documents, or search results and feed them directly into the model context window.
Activating RAG instructs the application to find relevant background information to help answer user queries accurately.
Search queries used in RAG are frequently generated by smaller subsidiary models designed to identify optimal information retrieval parameters.
The primary drawback of RAG is high computational intensity, as retrieved documents can consume hundreds of thousands of tokens in addition to the base prompt.
Inference, Prefill and Decode
Inference is the process of using a trained model to generate outputs, split into two primary operational phases: prefill and decode.
Prefill is the initial phase where input history and prompts are tokenized and processed in parallel, populating the key-value cache.
Prefill workload determines time to first token metrics, where longer prompts require more processing time before output generation begins.
Decode follows prefill, generating response tokens sequentially. Because decode requires streaming weights and reading the key-value cache, it is heavily constrained by memory bandwidth.
Speculative decoding uses drafting mechanisms to guess upcoming tokens in parallel, which the main model verifies to accelerate generation speeds.
Multi-token prediction attempts to predict multiple upcoming tokens simultaneously, serving as a built-in drafting mechanism for speculative decoding pipelines.
These foundational concepts provide a comprehensive framework for understanding how hardware and software interact during local and remote AI execution.
