Contents

Ollama GPU Setup and Performance Tuning Tutorial: Why Is It So Slow?

The most common complaint about local models is that they are “too slow.” In most cases, the problem is not that your computer is underpowered, but that the model does not fit entirely in GPU memory.

Step 1: Check whether the GPU is being used

Run a model first, then check:

ollama run qwen3:8b "Hello" --verbose
ollama ps

`ollama ps` PROCESSOR	Meaning	Speed
`100% GPU`	Entirely on the GPU	Fastest
`30%/70% CPU/GPU`	The GPU cannot fit it all, so some is in system memory	Much slower
`100% CPU`	GPU is not being used at all	Slowest

The eval rate in --verbose output is the number of tokens generated per second. Use it to compare performance before and after adjustments.

NVIDIA GPU is not being used

Update the driver. An outdated driver is the most common cause.
Check that the system recognizes the GPU:
```
nvidia-smi
```
Check the Ollama logs for GPU-related messages:
- Linux: journalctl -u ollama --no-pager | grep -i -E "gpu|cuda"
- Windows: %LOCALAPPDATA%\Ollama\server.log
- macOS: ~/.ollama/logs/server.log
If your laptop has integrated and discrete GPUs, make sure Ollama is using the discrete GPU (in Windows, you can select the high-performance GPU for an app in “Graphics settings”).
If you have multiple GPUs, use CUDA_VISIBLE_DEVICES=0 to specify which GPUs to use.

AMD GPUs

Ollama supports some AMD GPUs through ROCm. See the official documentation for the list of supported models. For GPUs not on the list, you can sometimes use the HSA_OVERRIDE_GFX_VERSION environment variable to force a similar architecture, but stability is not guaranteed.

Apple Silicon Macs

M-series chips use the GPU automatically. A Mac's “unified memory” is shared by the CPU and GPU, but by default the system lets the GPU use only part of it (about two-thirds to three-quarters). The model and context together must stay within that limit.

What to do when the model does not fit in GPU memory

Listed in order of effectiveness:

1. Switch to a smaller model or quantized version

For example, switch from 14b to 8b, or from q8_0 to q4_K_M. This is usually much faster than running a large model with half of it on the CPU. See Choosing a model and hardware requirements.

2. Shorten the context length

The memory required for context (KV cache) increases with its length. If CONTEXT in ollama ps is large, try lowering OLLAMA_CONTEXT_LENGTH or num_ctx.

3. Quantize the KV cache

OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0

q8_0 roughly halves the memory used by the context, with little impact on quality. q4_0 saves more memory, but has a more noticeable impact on quality. KV cache quantization requires Flash Attention.

4. Close other programs using GPU memory

Browsers, games, and other AI tools all use VRAM. Use nvidia-smi to see how much each program is using.

5. Load only one model at a time

Loading multiple models at the same time makes them compete for memory:

ollama ps
ollama stop model-you-no-longer-need

Or set OLLAMA_MAX_LOADED_MODELS=1.

Other factors that affect speed

Symptom	Cause and solution
The first response is especially slow	The model is loading. Increase `OLLAMA_KEEP_ALIVE` to keep it loaded
Long pause before the response starts	The input (prompt) is long, so the model needs to process it first. This time is mainly affected by GPU compute power
Reasoning models respond slowly	The model is “thinking.” You can turn off thinking mode when deep reasoning is unnecessary
Slower when multiple people use it at once	Requests are queued. See `OLLAMA_NUM_PARALLEL`

Rough speed guidelines

Generation speed is mainly limited by memory bandwidth. The general rules are:

On the same GPU, larger models are slower, roughly in inverse proportion to the model file size.
At over 20 tokens per second, reading feels smooth; below 10 tokens per second, it feels like you're waiting.

If your speed is far below the typical level for comparable hardware, the model is almost certainly spilling over to the CPU. Go back to the first step and check.

Updated October 12, 2026