Ollama Embeddings and RAG Tutorial: Answer Questions from Your Documents

Language models don't know your company's policies, product manuals, or personal notes. RAG (Retrieval-Augmented Generation) works by finding passages related to your question in your documents, then giving those passages and the question to the model so it can answer based on the information.

Question ──▶ Find the most relevant document passages ──▶ "Answer based on the following information: ..." ──▶ Model ──▶ Answer

The key technology for finding "relevant passages" is text embeddings.

What are text embeddings?

An embedding model turns a piece of text into a series of numbers (a vector). Text with similar meanings has similar vectors, even when the words are completely different:

• "How do I request time off?"
• "How should I submit a vacation request?"

The vectors for these two sentences are close, which enables "semantic search" that keyword search can't do.

Download embedding models

ollama pull bge-m3            # Multilingual, performs well in Chinese
ollama pull nomic-embed-text  # Lightweight, primarily for English

Embedding models can't be used for chat; they can only generate vectors.

Generate embeddings

curl http://localhost:11434/api/embed -d '{
  "model": "bge-m3",
  "input": ["How do I request time off?", "How should I submit a vacation request?"]
}'

import ollama

result = ollama.embed(model="bge-m3", input=["How do I request time off?", "How should I submit a vacation request?"])
vectors = result.embeddings
print(len(vectors), len(vectors[0]))  # 2 vectors, 1024 dimensions each

input can be a single string or a list of strings. Batch processing is much faster than sending one item at a time.

Calculate similarity

The most commonly used measure is cosine similarity. Its value ranges from -1 to 1, and values closer to 1 indicate greater similarity:

import math


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))


print(cosine(vectors[0], vectors[1]))

Build a local knowledge base Q&A system

Suppose you have a company policy file named handbook.txt. Here's the complete program:

import math
import ollama

EMBED_MODEL = "bge-m3"
CHAT_MODEL = "qwen3:8b"


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))


# 1. Read the file and split it into chunks (split on blank lines)
text = open("handbook.txt", encoding="utf-8").read()
chunks = [c.strip() for c in text.split("\n\n") if c.strip()]

# 2. Generate embeddings for each chunk (do this once and save them for reuse)
chunk_vectors = ollama.embed(model=EMBED_MODEL, input=chunks).embeddings


def ask(question: str, top_k: int = 3) -> str:
    # 3. Find the chunks most similar to the question
    q_vec = ollama.embed(model=EMBED_MODEL, input=question).embeddings[0]
    ranked = sorted(zip(chunks, chunk_vectors), key=lambda p: cosine(q_vec, p[1]), reverse=True)
    context = "\n\n".join(c for c, _ in ranked[:top_k])

    # 4. Give the chunks and question to the model
    response = ollama.chat(
        model=CHAT_MODEL,
        messages=[
            {
                "role": "system",
                "content": "You are the company's internal assistant. Answer in Traditional Chinese using only the provided information."
                "If the information doesn't contain an answer, reply \"The information doesn't contain a relevant answer\". Don't make guesses.",
            },
            {"role": "user", "content": f"Information:\n{context}\n\nQuestion: {question}"},
        ],
        options={"temperature": 0.2},
    )
    return response.message.content


while True:
    q = input("\nQuestion: ").strip()
    if not q:
        break
    print(ask(q))

This is a basic but fully functional RAG system, and all your data stays on your machine.

Ways to improve it

Chunking

Chunks that are too large can include irrelevant content; chunks that are too small can lose context. Common approaches include:

• Use about 300–800 characters per chunk, with 50–100 characters of overlap between adjacent chunks.
• Split by document structure (headings and paragraphs) instead of a fixed character count.
• Add the relevant heading before each chunk, such as "Chapter 3: Time-off policy > Annual leave."

Use a vector database

When you have many chunks, comparing each one with every vector can be slow, and you'll have to recalculate them each time you restart. You can use a vector database to store and search them instead:

Hybrid search

Semantic search isn't good at exact matches (for example, product models or names). Combining keyword search (BM25) with semantic search usually works better.

Watch the context length

When you put multiple chunks in a prompt, their total length can't exceed the model's context length, or the beginning will be truncated. If needed, increase num_ctx; see Environment variables and settings.

Don't want to write code yourself?

Open WebUI has built-in document uploads and knowledge base features. You can upload PDF and Word files directly and have the model answer based on them.

Updated October 12, 2026