Understanding Local LLMs
Everything you need to know to run AI models on your own hardware
What is a Local LLM?
Running a Large Language Model (LLM) locally means the AI model executes entirely on your own hardware (CPU, GPU, and RAM) rather than relying on a cloud service like ChatGPT or Claude APIs.
Privacy First
No data leaves your device. Perfect for sensitive documents or private conversations without internet.
Zero API Costs
Once you have the hardware, running queries is free. No recurring subscriptions or per-token charges.
Full Control
Choose exact model versions, customize system prompts, and avoid sudden API deprecations.
Hardware Specific
ModelFitCheck helps you figure out which specific models will fit in your system's VRAM and RAM.
Parameters & Model Sizes
Model size is typically measured in "Parameters" (often abbreviated as "B" for Billions). A parameter is a learned weight connecting neurons in the network. More parameters generally mean a smarter, more capable model, but they require proportionally more memory.
Quantization Explained
Quantization reduces the precision of a model's weights to save space, functioning much like image compression. It allows a large, capable model (like a 70B parameter model) to fit into less VRAM with only a slight loss in reasoning detail.
Naming Convention: "Q" = Quantization. Number = bits used per weight. "K" = K-quant algorithm. "M/S/L" = Medium/Small/Large variant.
| Format | Quality Impact | Description |
|---|---|---|
| F16 | Baseline | Original 16-bit float. Massive memory footprint. |
| Q8_0 | Lossless | Indistinguishable from F16. Use if you have spare memory. |
| Q6_K / Q5_K_M | Excellent | Near-lossless. Great for coding and logic tasks. |
| Q4_K_M ⭐ | Very Good | Community default. The sweet spot of speed, size, and quality. |
| Q3_K_M | Noticeable Loss | Use when struggling to fit a larger model into VRAM. |
| Q2_K / IQ2 | Emergency Only | Significant degradation. Use only as a last resort. |
GGUF Format
GGUF (GPT-Generated Unified Format) is the modern standard for sharing local LLMs. It bundles the model weights, metadata, and tokenizer rules into a single file, replacing the older GGML format.
GGUF
Single-file structure. Optimized for CPU/GPU inference via llama.cpp, Ollama, LM Studio.
SafeTensors
Standard for HuggingFace Transformers. Multi-file, requires Python environments like vLLM.
PyTorch (.bin)
Legacy multi-file format. Has security risks (arbitrary code execution). Avoid.
You can find GGUF models on HuggingFace by searching for your desired model name appended with -GGUF.
VRAM & Memory
VRAM (Video RAM) is the dedicated memory on your GPU. Loading a model entirely into VRAM ensures maximum inference speed. If a model doesn't fit, layers can be offloaded to slower system RAM, causing a major performance hit.
Where does the memory go?
- Model Weights: The bulk of the memory usage, determined by parameters and quantization.
- KV Cache: Memory required to remember the conversation context. Scales with context length.
- Token Embedding: Typically kept in system RAM.
- Overhead: Additional buffers for inference operations (usually ~10% safety margin).
Context Length & KV Cache
Context length is how many tokens the AI can "remember" in a single interaction. For the model to remember this context efficiently, it stores Key and Value pairs in memory — this is the KV Cache.
KV Cache Size = layers × numKVHeads × headDim × context_length × 2 (for F16 precision)
Because of this formula, doubling the context length doubles the required KV Cache memory.Practical Tip: Most normal chats and tasks easily fit within 4K to 8K tokens. Only set the context to 32K or higher if you are pasting in huge documents or codebases.
Mixture of Experts (MoE)
MoE architectures (like Mixtral 8x7B) divide the model into several "expert" sub-networks. During inference, a router network directs each token to only a couple of experts.
- The Benefit: A 46B parameter model might run as fast as a 13B model because only part of it activates per token.
- The Catch: All experts still need to be loaded into memory simultaneously, so the VRAM requirement is still extremely high.
Try It: Quantization Explorer
Estimates are approximate. Actual VRAM depends on your inference engine, batch size, and other running processes. KV-cache usage scales with context length — try changing it above to see the impact.