I Built an AI Knowledge Bot. Here's the Silent Bug That Was Breaking It.

> **TLDR** > - My RAG chatbot was returning "I don't have training on that" for content I *knew* existed — 4,290 vectors confirmed in the index, 3,836 chunks embedded > - Root cause: a missing `HF_TOKEN` in my local `.env` silently switched the embedding pipeline to a quantized ONNX model at index time, while production used the Hugging Face API at query time — two different vector spaces > - A Chinese-language LLC lesson scored only 0.3762 against an English query; my threshold was 0.55, so the bot discarded the correct answer > - Three fixes: add the token, re-embed all 3,836 chunks, lower the minimum score to 0.35 and add an LLM-level relevance filter

---

The Promise I Made

About six months ago I told my online community something I genuinely believed: "Ask the bot anything. Everything I've ever taught is in there."

I meant it. I had spent three weeks building a RAG chatbot to serve as an always-on knowledge base — a searchable brain covering everything from LLC formation for immigrants, to supply chain logistics, to AI automation workflows. The architecture was solid on paper: chunk the source content, generate embeddings using `paraphrase-multilingual-MiniLM-L12-v2` (a 384-dimension multilingual model from Hugging Face), push everything into a Pinecone vector index, wire up a Next.js front end, deploy on Vercel.

By the numbers, it looked complete. I ran verification scripts before going live. 4,290 vectors confirmed in the index. 3,836 content chunks embedded and stored. The data was there. The index was real. The math checked out.

I made the promise. I shipped it. And for several weeks, I had no idea it was lying to people.

---

What Members Started Saying

It started as a trickle of messages. Polite ones, the kind where someone does not want to make you feel bad.

"Hey Frank, I searched for the LLC registration stuff — I remember you covering how to set up a US company as a foreigner. The bot said it didn't have anything on that. Did I miss where it's stored?"

My first instinct was user error. Maybe they typed the question differently than I expected. I told them to rephrase.

Same result. "I don't have information on that topic in my knowledge base."

Then a second member. Then a third, asking about logistics sourcing — a topic I had covered across multiple detailed lessons. Same dead silence from the bot. Same empty response where there should have been a thorough answer.

At that point I stopped making excuses and opened the terminal.

---

The Investigation

My first move was the obvious one: verify the index actually contains the content.

I pulled a direct Pinecone stats call. The numbers came back clean — 4,290 vectors, consistent with what I had indexed. The data was physically present in the index. This was not a missing data problem.

So I ran a direct vector query. Not through the chatbot UI. I wrote a test script that took the exact query one of my members had typed — something about registering a US LLC as a foreign national — converted it to an embedding on my local machine, and fired it straight at Pinecone with no threshold filter, asking for the top 10 results.

The LLC lesson came back. The one titled 如何註冊美國公司 (How to Register a US Company) was in the results. But the similarity score attached to it was **0.3762**.

My `MIN_SCORE` threshold in production was `0.55`.

The answer existed. The bot was finding it. And then it was throwing it away because 0.3762 was too low to pass the filter.

That explained the symptom. It did not explain *why* a directly relevant lesson was scoring so poorly. A multilingual model is supposed to handle English queries against Chinese content. That is the entire selling point of `paraphrase-multilingual-MiniLM-L12-v2`. So why was it scoring at 0.3762 instead of the 0.60+ I would expect?

I started digging into the embedding pipeline itself.

---

Three Embedding Paths, One Broken Index

Here is something I did not fully appreciate when I built this system: the model `paraphrase-multilingual-MiniLM-L12-v2` has three distinct ways you can run it in code, and they are not all equivalent.

**Path 1: Hugging Face Inference API.** You send text to `api-inference.huggingface.co`, pass a Bearer token, and get back a 384-dimension vector. This is what my Vercel production environment was configured to use at query time.

**Path 2: SentenceTransformer local.** You run `SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')` locally in Python, and it downloads the model weights and runs inference on your machine. This is computationally equivalent to the API — same weights, same math.

**Path 3: fastembed with ONNX quantization (ONNX-Q).** This is the fast, low-memory option. The `fastembed` library ships a quantized ONNX version of many popular embedding models. It runs without the full PyTorch stack. It is fast and convenient. And it produces a **different vector space** than paths 1 and 2.

I ran a quick test to confirm my suspicion. I took a sample sentence, embedded it using the HF API, then embedded the same sentence using SentenceTransformer locally, and computed cosine similarity between the two output vectors.

The result: **1.0000**. Identical. Paths 1 and 2 are the same model, the same weights, the same vectors. They produce the same 384-dimension output regardless of whether the computation happens on Hugging Face's servers or on my laptop.

Then I embedded the same sentence using fastembed's ONNX-Q version and compared it against the HF API output.

Different. Measurably different. Not catastrophically different — the vectors are not random noise, they occupy roughly the same region of space — but different enough that cosine similarity scores between an ONNX-Q indexed vector and an HF API query vector are consistently lower than they should be. Not enough to make results disappear from the top-10 list entirely. More than enough to push borderline matches below a 0.55 threshold.

Now I understood the mechanism. The question was: how did my production index end up built with ONNX-Q when my code was supposed to use the HF API?

The answer was embarrassingly simple. My local `.env` file was missing `HF_TOKEN`. No API key present, no HF API call possible. My embedding script had fallback logic: if the token is missing, use fastembed's ONNX-Q implementation as a "graceful" alternative. Fast, silent, and completely wrong for this use case.

I had indexed 3,836 chunks of content using ONNX-Q vectors. My Vercel environment had `HF_TOKEN` properly set as an environment variable. At query time, Vercel generated HF API embeddings. Two different vector spaces, one Pinecone index, a threshold set at 0.55, and my knowledge base was silently burying half its answers every time someone asked a question.

---

Why Cross-Lingual Retrieval Makes It Worse

There is another layer to this problem that made the bug harder to catch and more damaging in practice.

Even with a perfectly consistent embedding path — no drift, no model mismatch, everything correct — English queries against Chinese content naturally score lower than same-language queries. The multilingual model does a genuinely good job of semantic alignment across languages. But "genuinely good" still means roughly a 0.10–0.15 point penalty on cosine similarity compared to matching English-to-English or Chinese-to-Chinese content of equal relevance.

Think about what that means in practice:

• A well-indexed English chunk responding to an English query might score 0.72
• The same conceptual content stored in Chinese, queried in English, might score 0.58–0.62 under ideal conditions with a consistent embedding path
• With embedding drift layered on top, that Chinese chunk drops to 0.37

The LLC lesson 如何註冊美國公司 was getting hit by both problems at once: the vector space mismatch from ONNX-Q indexing, and the inherent cross-lingual similarity penalty. The two effects stacked. The result was a score of 0.3762 — well below my 0.55 threshold, completely invisible to any member asking in English.

If my knowledge base had been entirely in English, the drift would have been smaller and I might have gotten away with the bug for longer. The multilingual content exposed it faster, which is the only silver lining in this whole story.

---

The Fix (Three Changes)

I did not patch around this. I fixed it at the root and rebuilt correctly.

**Fix 1: Add `HF_TOKEN` to the local `.env` file — and make its absence a hard error.**

Not just adding the token to my `.env`. I removed the silent fallback entirely. Any code path that previously fell back to fastembed when `HF_TOKEN` was missing now throws a hard error with a clear message: "HF_TOKEN required. Set it and re-run the indexing pipeline." If you do not have the token, the script exits before touching the index. Silent fallbacks to alternate model paths are a category of bug that should not exist in a production embedding pipeline.

**Fix 2: Re-run the embedding pipeline for all 3,836 chunks.**

I deleted the old Pinecone namespace and re-indexed from scratch using the HF API consistently throughout. The rebuild took approximately 40 minutes. When I ran the same LLC query against the rebuilt index, 如何註冊美國公司 scored **0.5741**. Above the original threshold. The bot surfaced it correctly.

The logistics content that other members had been asking about came back at similar scores, all clearing 0.55. The vector space mismatch was gone. The index was now internally consistent with the query path.

**Fix 3: Lower `MIN_SCORE` to 0.35 and add an LLM-level relevance filter.**

Even after fixing the drift, I recognized that 0.55 was too aggressive for a multilingual knowledge base where the cross-lingual similarity penalty is real and unavoidable. I lowered the minimum score to 0.35 to stop discarding borderline matches. But lowering the threshold without compensation would pass more noise to the language model.

The solution: after vector retrieval, the language model evaluates each candidate chunk and makes a binary relevance judgment — relevant or not — before composing the final answer. This replaces the blunt vector-threshold filter with a smarter semantic judgment at the point where it actually matters. The combination of a lower retrieval floor and an LLM-level filter gives better coverage without increasing hallucination risk.

---

What to Check In Your RAG Stack

If you are running a RAG chatbot in production — whether it is on Vercel, Railway, Fly.io, or a self-hosted setup — here are three checks worth running today.

**Check 1: Model path consistency test.**

Write a five-line script. Take a single sentence. Embed it using every code path your system uses: your local development environment, your CI environment, your production environment. Compute cosine similarity between all pairs. They should all produce a result at or very near 1.0000. If any pair shows significant divergence, you have embedding drift and your index is unreliable. This test takes about 10 minutes to write and should run as a pre-flight check before every re-index operation.

**Check 2: Cross-lingual threshold calibration.**

If your knowledge base contains content in multiple languages, test your similarity threshold explicitly against cross-lingual pairs before setting it in production. Take a known-relevant piece of content in Language A. Query it in Language B. Measure the actual score. If your threshold is cutting those results off, lower the threshold and compensate with an LLM relevance filter downstream. Do not assume your multilingual model scores cross-lingual matches the same as same-language matches — it does not, and calibrating on same-language pairs alone will produce a threshold that is too aggressive for multilingual retrieval.

**Check 3: LLM-level relevance filter for noise.**

A low retrieval threshold brings in more candidate chunks, which means more noise. The right response to this is not to raise the threshold back up — it is to add semantic filtering after retrieval. Pass your candidate chunks to the language model with a simple prompt asking whether each chunk is relevant to the query. Filter out the failures before composing the final response. This catches semantic near-misses that vector similarity alone cannot reliably distinguish, and it keeps the answer quality high even when the retrieval net is cast wider.

---

FAQ Section

**What is embedding drift in a RAG chatbot?**

Embedding drift is a mismatch between the vector representations generated during index creation and those generated at query time. When the embedding model, model version, or model execution path differs between indexing and querying, the resulting vectors occupy different regions of the vector space. Cosine similarity scores between a query vector and an indexed vector will be lower than expected, causing relevant content to fall below retrieval thresholds and disappear from the bot's answers — even though the content is physically present in the index.

**How do I know if my RAG system has an embedding consistency problem?**

The most reliable test is to take a sentence you know is in your knowledge base, embed it using your production query path, and compute cosine similarity directly against the indexed version of that same sentence. The score should be very close to 1.0. If it is 0.85 or lower, your index and query paths are using different vector representations. You can also catch it indirectly: if your bot consistently says it has no information on topics you have explicitly covered, and a direct Pinecone query shows those vectors do exist with lower-than-expected scores, drift is the likely cause.

**Why does a quantized ONNX model produce different vectors than the full precision model?**

Quantization reduces numerical precision in model weights — typically from 32-bit floats to 8-bit integers — to speed up inference and reduce memory usage. The output vectors are computed through a compressed approximation of the original model's arithmetic. For most downstream tasks like classification or clustering, the quality difference is small enough to be acceptable. But in a RAG system where you are computing cosine similarity between an ONNX-Q indexed vector and a full-precision query vector, the difference is measurable and can push similarity scores below the retrieval threshold, making relevant content invisible.

**Does a multilingual model work for English queries against Chinese content?**

Yes, and in practice it works well — but with a consistent score penalty. Cross-lingual retrieval with `paraphrase-multilingual-MiniLM-L12-v2` typically produces cosine similarity scores roughly 0.10–0.15 lower than same-language retrieval for content of equivalent semantic relevance. A threshold calibrated only on English-to-English test pairs will be too aggressive for a multilingual knowledge base. Always test your threshold explicitly against cross-lingual pairs and set it based on measured scores rather than assumptions borrowed from single-language testing.

**What is the safest way to prevent silent fallback behavior in an embedding pipeline?**

Fail loudly and immediately. Any code path that would fall back to a different model, API, or execution environment when a dependency is missing should throw a hard error with a specific, actionable message — not silently continue with a different implementation. In a data ingestion pipeline, silent fallbacks are the most dangerous failure mode because they produce plausible-looking output: vectors get generated, the index gets populated, no error is logged, and everything appears to work. By the time you notice the problem, you may have an entire production index built on the wrong vector space. Treat a missing embedding API token the same way you would treat a missing database connection: stop immediately and tell the operator exactly what needs to be fixed before the pipeline can run.

**Do I need to rebuild my entire Pinecone index after fixing embedding drift?**

Yes. If your index was built with a different embedding path than your query environment uses, every vector in that index is in the wrong vector space relative to your queries. Selective re-indexing of "affected" documents is not sufficient because the problem is systematic — every chunk indexed during the drift period is affected. Delete the namespace, re-run the full embedding pipeline with the correct credentials and model path confirmed at both ends, verify consistency with a model-path consistency test, then restore production. In my case, rebuilding 3,836 chunks took about 40 minutes. That is a one-time cost for catching the bug late; the preventive check I described above takes 10 minutes and catches it before the index is built.