I Built an AI Knowledge Bot. Here's the Silent Bug That Was Breaking It.

TLDR: My RAG chatbot kept answering “I don't have information on that” for lessons that were in the index. The cause: the index and the live queries were embedded through two different paths, the hosted Hugging Face model and a quantized ONNX fallback that switched on when an API token was missing. The vectors no longer lined up, so correct matches scored under the 0.55 threshold and were thrown away. The fix: stop the fallback from happening silently, embed everything through one path, lower the threshold and add an LLM-level relevance check.

The Promise I Made

The idea was simple: a chatbot that answers questions from a library of course lessons, so people can ask it instead of digging through hours of video.

The build was a standard RAG setup: split the lesson content into chunks, turn each chunk into an embedding with paraphrase-multilingual-MiniLM-L12-v2 (a multilingual sentence-transformers model on Hugging Face that maps text to a 384-dimension vector), store the vectors in a Pinecone index, and serve the chat app from Vercel.

Before launch I checked the index stats. The vectors were there. The data was real, and on paper everything checked out.

So I shipped it. What I did not know yet was that it was quietly throwing away good answers.

What Users Started Seeing

The symptom was a polite non-answer. Questions about topics the course clearly covered came back with the bot saying it had no information on that topic. Rephrasing the question did not help.

At that point I stopped assuming user error and opened the terminal.

The Investigation

My first move was the obvious one: verify the index actually contains the content.

I pulled the Pinecone index stats. The vector count matched what I had indexed, so the content was physically in the index. This was not a missing-data problem.

So I ran a direct vector query outside the chatbot UI: take the exact question a user had asked, embed it on my machine, and send it straight to Pinecone with no score filter, asking for the top 10 results.

The right lesson came back in the results. But its similarity score sat well below the production threshold.

The MIN_SCORE threshold in production was 0.55.

The answer existed. The bot was finding it. Then it was discarding it because the score was too low to pass the filter.

That explained the symptom, but not why a directly relevant lesson scored so poorly. A multilingual model is supposed to handle an English question against Chinese content. That is the point of paraphrase-multilingual-MiniLM-L12-v2.

I started digging into the embedding pipeline itself.

Three Embedding Paths, One Broken Index

Here is something I did not fully appreciate when I built this system: paraphrase-multilingual-MiniLM-L12-v2 can be run in three different ways in code, and they are not all equivalent.

Path 1: Hugging Face's hosted inference API. You send text with a Bearer token and get back a 384-dimension vector. This is the path the production app used whenever its token was set.

Path 2: SentenceTransformer locally. You load the model in Python with the sentence-transformers library, which downloads the weights and runs inference on your machine. Same weights and same math as the hosted model, so the vectors should match.

Path 3: a quantized ONNX port (ONNX-Q) through fastembed. Qdrant publishes a quantized ONNX port of this exact model for the fastembed library. It is fast and light on memory. It is also not the same arithmetic as paths 1 and 2, so it produces slightly different vectors.

The quick way to confirm this is to embed the same sentence through each path and compare the outputs with cosine similarity.

The hosted API and a local SentenceTransformer run are the same model with the same weights, so they should agree almost exactly.

The quantized ONNX version is where the difference shows up.

Different. Measurably different. Not catastrophically different, since the vectors are not random noise and land in roughly the same region, but different enough that similarity scores between a vector from one path and a query from the other come out lower than they should. Not enough to drop results out of the top 10. More than enough to push borderline matches under a 0.55 threshold.

Now I understood the mechanism. The question was how the two paths ended up mixed.

The answer was embarrassingly simple: a missing HF_TOKEN. The embedding code had fallback logic. If the token was not set, it switched to the quantized ONNX model as a “graceful” alternative. Fast, quiet, and wrong for this use case.

One environment had the token and one did not. Vectors built through one path were being searched with queries embedded through the other. Two slightly different vector spaces, one Pinecone index, a 0.55 threshold, and a knowledge base that quietly buried answers it actually had.

Why Cross-Lingual Retrieval Makes It Worse

There is another layer to this problem that made the bug harder to catch and more damaging in practice.

Even with one consistent embedding path, English questions against Chinese content can score lower than same-language matches. The multilingual model aligns meaning across languages well, but a cross-language pair does not always score as high as a same-language pair of equal relevance. A threshold tuned on English-only tests does not account for that gap.

The lesson that should have answered the question was hit by both problems at once: the mismatch between embedding paths and the normal cross-language gap. Together they pushed its score under the 0.55 threshold, so anyone asking in English never saw it.

If my knowledge base had been entirely in English, the drift would have been smaller and I might have gotten away with the bug for longer. The multilingual content exposed it faster, which is the only silver lining in this whole story.

The Fix (Three Changes)

I did not patch around this. I went after the root cause.

Fix 1: Set HF_TOKEN in every environment, and stop the fallback from being silent.

Adding the token was step one. The fallback now logs a warning that names the missing token instead of switching paths quietly. If I built it again, I would make a missing token a hard error so the script stops before it touches the index. A silent fallback to a different model path does not belong in a production embedding pipeline.

Fix 2: Put indexing and queries on the same embedding path.

Every vector in the index has to come from the same model path your queries use. Mixed vectors cannot be patched one document at a time. If an index was built during the drift, rebuild it with the correct path confirmed at both ends. Once the paths matched, the lessons people had asked about started coming back.

Fix 3: Lower the retrieval threshold and add an LLM-level relevance filter.

Even with the paths fixed, a threshold tuned on same-language tests is too strict for a multilingual knowledge base. I lowered the minimum score so borderline matches were not discarded. Lowering it without compensation passes more noise to the language model, though.

The solution: after vector retrieval, the language model judges each candidate chunk as relevant or not before composing the final answer. That replaces a blunt vector cutoff with a semantic check at the point where it matters. A lower retrieval floor plus an LLM-level filter gives better coverage while keeping off-topic excerpts out of the answer.

What to Check In Your RAG Stack

If you are running a RAG chatbot in production, whether it is on Vercel, Railway, Fly.io, or a self-hosted setup, here are three checks worth running today.

Check 1: Model path consistency test.

Write a short script. Take one sentence and embed it through every code path your system uses: local development, CI and production. Compare every pair with cosine similarity. They should agree almost exactly. If any pair diverges noticeably, you have embedding drift and your index cannot be trusted. Run it as a pre-flight check before every re-index.

Check 2: Cross-lingual threshold calibration.

If your knowledge base contains content in multiple languages, test your similarity threshold explicitly against cross-lingual pairs before setting it in production. Take a known-relevant piece of content in Language A. Query it in Language B. Measure the actual score. If your threshold is cutting those results off, lower the threshold and compensate with an LLM relevance filter downstream. Do not assume your multilingual model scores cross-lingual matches the same as same-language matches, it does not, and calibrating on same-language pairs alone will produce a threshold that is too aggressive for multilingual retrieval.

Check 3: LLM-level relevance filter for noise.

A low retrieval threshold brings in more candidate chunks, which means more noise. The right response to this is not to raise the threshold back up, it is to add semantic filtering after retrieval. Pass your candidate chunks to the language model with a simple prompt asking whether each chunk is relevant to the query. Filter out the failures before composing the final response. This catches semantic near-misses that vector similarity alone cannot reliably distinguish, and it keeps the answer quality high even when the retrieval net is cast wider.

FAQ Section

What is embedding drift in a RAG chatbot?

Embedding drift is a mismatch between the vector representations generated during index creation and those generated at query time. When the embedding model, model version, or model execution path differs between indexing and querying, the resulting vectors occupy different regions of the vector space. Cosine similarity scores between a query vector and an indexed vector will be lower than expected, causing relevant content to fall below retrieval thresholds and disappear from the bot's answers, even though the content is physically present in the index.

How do I know if my RAG system has an embedding consistency problem?

The most reliable test is to take a sentence you know is in your knowledge base, embed it through your production query path, and compare it with cosine similarity against the indexed version of that same sentence. The score should be very close to 1.0. If it is noticeably lower, your index and query paths are producing different vectors. You can also catch it indirectly: if your bot keeps saying it has no information on topics you have clearly covered, and a direct Pinecone query shows those vectors exist with lower-than-expected scores, drift is the likely cause.

Why does a quantized ONNX model produce different vectors than the full precision model?

Quantization stores a model at lower numerical precision. ONNX Runtime's quantization, for example, is 8-bit linear quantization that maps 32-bit floating-point values to 8-bit integers, and ONNX Runtime's own documentation says quantization is not lossless and may affect accuracy. For many tasks the difference is acceptable. In a RAG system that compares a quantized vector with a full-precision query vector, the difference can be enough to push similarity scores under the retrieval threshold.

Does a multilingual model work for English queries against Chinese content?

Yes, and it works well, but a cross-language match can score lower than a same-language match of equal relevance. A threshold calibrated only on English-to-English test pairs can be too strict for a multilingual knowledge base. Test your threshold against real cross-language pairs and set it from measured scores rather than assumptions borrowed from single-language testing.

What is the safest way to prevent silent fallback behavior in an embedding pipeline?

Fail loudly and immediately. Any code path that would fall back to a different model, API, or execution environment when a dependency is missing should throw a hard error with a specific, actionable message, not silently continue with a different implementation. In a data ingestion pipeline, silent fallbacks are the most dangerous failure mode because they produce plausible-looking output: vectors get generated, the index gets populated, no error is logged, and everything appears to work. By the time you notice the problem, you may have an entire production index built on the wrong vector space. Treat a missing embedding API token the same way you would treat a missing database connection: stop immediately and tell the operator exactly what needs to be fixed before the pipeline can run.

Do I need to rebuild my entire Pinecone index after fixing embedding drift?

Yes. If your index was built with a different embedding path than your query environment uses, every vector indexed during the drift is in the wrong vector space relative to your queries. Re-indexing only the documents you think are affected is not enough, because the problem is systematic. Delete the namespace, re-run the full embedding pipeline with the correct credentials and model path confirmed at both ends, verify with a model path consistency test, then restore production. A rebuild is a one-time cost; the consistency check above is cheap and catches the problem before the index is built.

Related: For the full build process, see my RAG system development guide.

If you want a second pair of eyes on a RAG build, read how I work or get in touch.