Optimize Your Local LLM Setup
From first install to peak performance — a step-by-step guide
Getting Started in 5 Minutes
Install Ollama
The easiest way to get started. Visit ollama.com to download the one-click installer for your OS.
Pull a model
Open your terminal and run this command to download an 8B parameter model (~4.7 GB). Ollama defaults to the efficient Q4_K_M quantization.
Chat!
Run the model to enter the interactive chat prompt.
Alternative: LM Studio
If you prefer a full graphical interface instead of the terminal, LM Studio is highly recommended. It lets you search for GGUF models, download them, and chat all in one desktop app.
Inference Engines Compared
| Engine | Ease of Use | Speed | GPU Support | API | Best For |
|---|---|---|---|---|---|
| Ollama | ⭐⭐⭐⭐⭐ | Fast | NVIDIA / AMD / Apple | REST | Beginners, quick setup |
| LM Studio | ⭐⭐⭐⭐⭐ | Fast | NVIDIA / AMD / Apple | OpenAI-compat | GUI users |
| llama.cpp | ⭐⭐⭐ | Fastest | All | CLI | Power users, max control |
| text-gen-webui | ⭐⭐⭐ | Good | NVIDIA | Web UI | Advanced features |
| vLLM | ⭐⭐ | Fastest | NVIDIA | OpenAI-compat | Production serving |
Optimization Strategies
1. Choose the Right Quantization
Don't waste VRAM on uncompressed weights. Use Q4_K_M as your default. Step up to Q5_K_M for critical reasoning tasks, or step down to Q3_K_M if you're trying to cram a huge model into limited memory.
2. Match Context to Your Needs
Massive context windows eat your VRAM quickly. Most basic chats only require 4K to 8K context.
| Context | KV Cache (7B) | KV Cache (70B) |
|---|---|---|
| 128K | ~4 GB | ~40 GB |
| 32K | ~1 GB | ~10 GB |
| 8K | ~256 MB | ~2.5 GB |
| 4K | ~128 MB | ~1.25 GB |
3. GPU Layer Offloading
Always offload as many layers to your GPU as possible. Full offload is the fastest. If a model doesn't fully fit, offloading partial layers (e.g., 20 out of 32) will still give a massive speed boost over pure CPU inference. In CLI, use the --n-gpu-layers flag.
4. Flash Attention
If supported by your engine and hardware, enable Flash Attention (e.g. --flash-attn in llama.cpp). It saves ~20-30% of the KV cache memory and runs significantly faster with almost zero drawbacks.
5. Batch Size
A lower batch size means less memory usage per prompt during the ingestion phase. The default of 512 is usually perfect for chat interactions. Only increase it if you are batch processing thousands of requests.
Hardware Upgrade Guide
VRAM is the single most important specification for running local LLMs. Here is what different GPU VRAM tiers typically unlock:
6 GB VRAM
e.g. GTX 1060, RTX 2060
Unlocks: 7B models at Q4.
Good for basic chat, but larger models will spill to system RAM.
8 GB VRAM
e.g. RTX 3060 8GB, RTX 4060
Unlocks: 7B-13B Q4, some Q5.
The current entry-level sweet spot for good performance.
12 GB VRAM
e.g. RTX 3060 12GB, RTX 4070
Unlocks: 13B Q4, 7B Q6/Q8.
Excellent balance for quality reasoning models without breaking the bank.
16 GB VRAM
e.g. RTX 4060 Ti 16GB
Unlocks: 13B Q5, 30B Q3.
Capable of running middle-weight models or very long context windows.
24 GB VRAM
e.g. RTX 3090, 4090
Unlocks: 30B Q4, 70B Q3.
The enthusiast standard. Runs capable 70B models at high speeds.
48+ GB VRAM
e.g. RTX A6000, 2× 24GB
Unlocks: 70B Q5-Q6, flagship models.
Professional/Enterprise tier for cutting-edge local AI without compromise.
Common Pitfalls
My model is running on CPU only
ollama ps in the terminal to see if it lists a GPU underneath the processor column.