When Switching Embeddings Doubles Recall: bge-m3 vs nomic on a Local RAG

At Wazo, we run a fully local RAG. Data sovereignty is not a marketing argument: the knowledge base mixes product documentation, technical docs shared with our partners, open-source and proprietary code, and a large volume of private notes. All of it stays on our servers. Nothing goes to a cloud third party.

That volume forces continuous improvement work. The RAG feeds both pre-sales and support — two use cases where a wrong or slow answer is costly. Recently, an embedding change took our search recall from 0.45 to 0.95 and cut latency by twelve. Measured, not guessed.

Here is how we tested it, what the numbers say, and why we switched.

The problem#

The knowledge base is 100% local (Ollama + Mistral + ChromaDB). It mixes two linguistic worlds:

• the public Wazo Platform documentation is in English;
• product PDFs and field expertise are in French;
• and users ask their questions in French.

nomic-embed-text is an English-only model — its model card tags it English, its technical report (arXiv 2402.01613) describes training on English pairs (StackExchange, Quora, Amazon, web queries). No multilingual pretraining. Using it on a bilingual FR/EN corpus meant running it outside its training domain.

Our workaround: before each search, translate the question FR→EN with Mistral, then run the search in both languages. In practice, this was the only way to bring nomic back into its training language. It worked… at the cost of an extra LLM call per query (~2 s on CPU).

The intuition: a natively bilingual embedding would make this workaround unnecessary. bge-m3 (1024 dimensions, 8192-token context, XLM-RoBERTa backbone, 100+ languages) is available via ollama pull. But there is a tradeoff: the model is much larger. XLM-RoBERTa-large (~560M params) versus a 137M BERT for nomic, meaning ~2× resident RAM (1.14 GB vs 0.57 GB, measured via ollama ps) and ~2.7× slower query embedding (0.16 s vs 0.06 s). What remained to prove was that the recall gain was worth the overhead.

The method: isolate a single variable#

The trap of an embedding comparison is changing ten things at once. So we built a protocol where the only variable is the model.

1. A labeled evaluation set. 20 questions in French, each associated with the source(s) that actually answer it. The split reflects real difficulty:

• 10 questions on the English documentation (the core of the cross-language gap);
• 9 questions on French product PDFs;
• 1 French expertise question (the easy case).

2. Two collections ingested with the same code. A 768-dimension collection (nomic) and a 1024-dimension one (bge-m3) are produced by the same chunking, the same contextual prefix, the same corpus (858 files, 2923 chunks — identical to the chunk). Only the embedding call differs.

3. A harness measuring four configurations:

Metrics: Recall@8 (is the right source in the top 8, our prod TOP_K?), MRR (how high is it?), average position of the right source, and per-query latency.

The results#

Overview#

Where it hurt: breakdown by category (Recall@8)#

The per-category breakdown tells the essential story. bge-m3 scores 1.00 on all English content (where translation capped at 0.60) and 0.89 on French product PDFs, versus 0.22 for nomic with translation — content nomic missed even with its crutch.

What the numbers teach us#

1. bge-m3 crushes the cross-language gap. French question, English answer: 0.60 → 1.00. That is exactly what a bilingual embedding is built for.

2. Translation was a crutch — and it became harmful. Compare A and B: without translation, nomic collapses (0.45 → 0.10). So translation was doing real work with nomic. But with bge-m3, adding it degrades recall (C 0.95 → D 0.90). Why? Translating a French question into English to fetch a French PDF scrambles the signal. The right answer is not “translate better,” it is “stop translating.”

3. The POPC mystery is solved. During a previous PDF cleanup, one question kept failing: “What is POPC?” returned a hallucinated answer, and the right document stayed unfindable. We had diagnosed a retrieval problem, not an extraction one. The measurements confirm it:

End to end, the answer is now grounded in the real document (supervision, transfers, unified directories — the actual PDF content), instead of the invented “Post Office Private Connection.”

4. Faster, as a bonus. Removing translation eliminates a Mistral call: search latency drops from 2.37 s to 0.20 s (~12×). bge-m3 embeds a query in 0.16 s versus 0.06 s for nomic — negligible against the 2 s saved.

5. One residual, stated honestly. One question resists in all configs: a Subscription sheet. Neither nomic nor bge-m3 surfaces it. This is a case for re-ranking or source enrichment, not an embedding failure.

The price#

Nothing is free:

• Full re-ingestion required. A ChromaDB collection’s dimension is fixed: you do not go from 768 to 1024 via a flag. You must wipe and reingest.
• ~2× RAM. bge-m3 occupies 1.14 GB resident versus 0.57 GB for nomic (measured via ollama ps). On the 8 GB VM, to monitor alongside the 4.1 GB of Mistral.
• ~2.8× slower CPU ingestion: 2434 s versus 860 s for the 2923 chunks. No impact on query latency, but to plan for during re-indexing.

The tradeoff is quickly settled: doubling recall and dividing latency by twelve is well worth 0.6 GB of RAM and a one-off re-ingestion.

The decision#

We adopt config C: bge-m3 as the embedding, translation disabled by default. The translation function stays in the code, in case we ever return to a non-bilingual embedding.

Why measure rather than feel#

The question set has become a shared internal deliverable: any future retrieval optimization is measured against it. We no longer “feel” whether a search improves — we quantify it.

That is the most useful part of this work, more than the choice of bge-m3 itself. An embedding that wins today may lose tomorrow on another corpus. The protocol stays. Just rerun the four configurations to know.