Run Ollama with Docker: NVIDIA, AMD GPU, and Compose

Running Ollama with Docker on a server makes it easier to manage and upgrade, and you can deploy it alongside other services with Compose. If you’re new to Docker, start with the Docker tutorial.

Docker on macOS cannot use Apple GPUs. On a Mac, install the Ollama app directly.

CPU only

docker run -d --name ollama \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama

• -v ollama:/root/.ollama: Models are stored in a Volume named ollama and won’t be lost when you recreate the container.
• -p 11434:11434: Lets the host connect to the API.

NVIDIA GPU

1. Install the NVIDIA Container Toolkit

Install the NVIDIA driver on the host first, then install the NVIDIA Container Toolkit (Ubuntu is used here as an example; see the NVIDIA documentation for detailed steps):

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

2. Start the container

docker run -d --name ollama --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --restart unless-stopped \
  ollama/ollama

Confirm that the GPU is being used:

docker exec ollama nvidia-smi

AMD GPU

Use the rocm image and pass the devices into the container:

docker run -d --name ollama \
  --device /dev/kfd --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:rocm

Use Ollama in the container

docker exec -it ollama ollama run qwen3:8b
docker exec ollama ollama pull nomic-embed-text
docker exec ollama ollama list

The API works just like a local installation, at http://localhost:11434.

Environment variables

Pass environment variables with -e. Inside the container, OLLAMA_HOST already defaults to 0.0.0.0, so the host can connect once you publish the port with -p:

docker run -d --gpus=all --name ollama \
  -e OLLAMA_CONTEXT_LENGTH=8192 \
  -e OLLAMA_KEEP_ALIVE=30m \
  -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 \
  ollama/ollama

127.0.0.1:11434:11434 allows connections only from the local machine, keeping the API off the network.

Docker Compose: Ollama + Open WebUI

services:
  ollama:
    image: ollama/ollama
    volumes:
      - ollama:/root/.ollama
    environment:
      OLLAMA_KEEP_ALIVE: 30m
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      OLLAMA_BASE_URL: http://ollama:11434
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama:
  open-webui:

docker compose up -d
docker compose exec ollama ollama pull qwen3:8b

Open http://localhost:3000 to use the web interface. Ollama doesn’t expose a port externally; only Open WebUI can reach it over the Compose network. If you don’t have an NVIDIA GPU, remove the deploy section.

See Open WebUI for instructions on using it.

Update

docker pull ollama/ollama
docker rm -f ollama
# Recreate the container with your original docker run command; models are in the Volume, so you don't need to download them again

When using Compose:

docker compose pull
docker compose up -d

Download models in advance

To automatically download a model when the container starts, add this to your deployment script:

docker exec ollama ollama pull qwen3:8b

Or package the model directly into a custom image (the image will be very large, so this usually isn’t recommended).

Updated October 12, 2026