Optimizing Local Embedding & Reranker Pipelines: Running BGE-M3 and ColBERT Without Choking RAM

The High Cost of Local Retrieval-Augmented Generation

Building local Retrieval-Augmented Generation (RAG) applications—such as offline documentation search, internal codebase Q&A, and private legal discovery—requires more than just a large language model. A production-grade local pipeline relies on a three-stage architecture:

1. Dense Bi-Encoder Embedder: e.g., bge-m3 or nomic-embed-text (Vector similarity retrieval).
2. Cross-Encoder Reranker: e.g., bge-reranker-large or ColBERT (Precision semantic re-scoring).
3. Generative LLM: e.g., llama-3.1-8b-instruct (Final response synthesis).

When developers load all three models into memory simultaneously using naive Ollama or HuggingFace endpoints, memory usage balloons past 18GB. Add your IDE, browser tabs, and vector database indices, and your machine begins swapping uncontrollably.

The Architecture of Staged Memory Orchestration

In a typical RAG query, the models are never used at the exact same millisecond. The pipeline is fundamentally sequential:

Tuning Ollama for Dynamic Model Eviction

If you run embeddings and generation through Ollama, by default Ollama keeps every loaded model in VRAM for 5 minutes (keep_alive: 5m). When your RAG app queries an embedding model and then calls an LLM, Ollama attempts to hold both models in VRAM at the same time.

You can force Ollama to instantly release memory after embedding generation by passing keep_alive: 0 on your embedding API payload:

import httpx

async def get_embedding(text: str):
    async with httpx.AsyncClient() as client:
        # keep_alive: 0 unloads the embedding weights immediately after inference
        res = await client.post("http://localhost:11434/api/embeddings", json={
            "model": "bge-m3",
            "prompt": text,
            "keep_alive": 0
        })
        return res.json()["embedding"]

Memory-Mapped Embeddings (mmap) vs. Locked RAM

For dedicated embedding microservices, configure your ONNX or llama.cpp runtime with memory mapping (mmap). Unlike VRAM allocations which are wired and unpageable, memory-mapped files allow the operating system kernel to evict cached model pages from physical RAM when memory pressure rises, reloading them transparently on the next read.