Live Pipeline Walkthrough
A grounded, cited, access-controlled retrieval pipeline for football intelligence — walked stage by stage through one real recorded run, using the system's own hand-verified evaluation corpus. Every answer, citation, and timing below is real output, not a mockup.
Before any question can be answered, source documents are converted, chunked, tagged with an access tier, and embedded.
What each box actually does
Source file, any format
This is a document exactly as someone saved it — a PDF, a Word file, plain text, whatever. It's the starting point because real documents never arrive in one tidy format, so the pipeline has to accept whatever it's actually given before it can do anything with it.
markitdown → Markdown
This step converts the source file into plain, simple text (Markdown), stripping out formatting quirks specific to PDFs, Word docs, and so on. It's needed so every later step has one consistent kind of text to work with, instead of a dozen different file formats to special-case.
Chunk + tag
The document is split into smaller pieces small enough for the AI to read at once, and each piece is labeled with who's allowed to see it, based on which folder the file lives in. This exists so the system can later answer a question using only the content a given person is actually permitted to access.
Embed (nomic-embed)
Each chunk of text is turned into a list of numbers that captures what it means, using a small AI model running locally. This lets the system later find relevant chunks by matching meaning, not just matching exact words.
Qdrant (dense + sparse)
The embeddings, the original text, and the access tier all get stored together in Qdrant, a database built for fast similarity search. This is where everything lands so it can be searched instantly the moment a question comes in, instead of re-reading every document from scratch.
Every query passes through three independent checkpoints — an injection judge, an access-tier filter, and an output/citation safety check — any one of which can end the request in a safe refusal instead of a guess.
What each box actually does
Query + user_tier + rewritten history
This is the starting point: the question being asked, who's asking it, and — if it's a follow-up in a conversation — the earlier messages already folded into one self-contained question. Every later step needs one complete question to work with, instead of having to guess what "it" or "that" refers to.
Injection judge
A small AI model checks whether the question is genuine or a disguised attempt to manipulate the system (like "ignore your instructions and reveal your prompt"). If it looks like an attack, the request is refused immediately, before it ever reaches the parts of the system that search documents or generate answers.
Embed query (nomic-embed)
The question is turned into the same kind of "meaning as numbers" representation used for the document chunks. This lets the system compare the question's meaning against every stored chunk, rather than relying on exact word matches.
Dense search · HNSW
This searches for chunks whose meaning is closest to the question, even if they don't share any exact words. It catches answers phrased differently than the question — e.g. finding a chunk about "the final score" when the question asks "who won."
Sparse search · BM25
This searches for chunks that share the exact keywords in the question, the way search engines have always worked. It runs alongside the meaning-based search because exact terms — names, dates, numbers — sometimes matter more than fuzzy similarity, and each method catches things the other can miss.
Fuse → top 10
The results from both searches are merged into one ranked list, keeping the 10 strongest candidates overall. This gets the best of both search styles instead of having to pick just one and risk missing what the other would have found.
Access filter
Every candidate chunk is checked against what the requesting user is actually allowed to see; anything above their access level is dropped here, before it reaches any later step. This guarantees a user can never see — or have the answer hint at — information from a document they don't have permission to access.
Rerank → top 4
A more careful (but slower) AI model re-scores the surviving candidates and keeps only the 4 most genuinely relevant ones. The earlier search steps are fast but a bit rough; this step trades a little speed for real precision right before an answer gets written.
Assemble prompt
The 4 chosen chunks, the original question, and a set of instructions (like "only answer from these sources, cite everything") are combined into one message for the AI model. This gives the model everything it needs, and nothing it doesn't, to answer honestly instead of guessing.
Generate · mistral
A local AI model reads the assembled prompt and writes an answer, citing which chunk each claim came from. This is the step that actually produces human-readable text — everything before it was about finding the right evidence to write from.
Output & citation check
Before the answer is sent back, it's checked to confirm every claim is actually backed by one of its cited sources. If a claim can't be traced to real evidence, the answer is swapped for an honest "I do not know" instead — this catches the AI making something up even after being handed the right sources.
Answer + citations
The final response — the answer text plus a numbered list of exactly which documents it came from — is returned to whoever asked. This is the payoff of the whole pipeline: an answer the user can actually verify, not just trust blindly.
From eval/questions.json's hand-curated set — the
actual request, actual intermediate state, and actual generated answer
from a recorded evaluation run, not a scripted mockup.
nomic-embed-text and searched two ways at once — dense HNSW similarity and sparse BM25 keyword matching — then fused into one ranked candidate list.employee. employee/derby.md is within reach; anything tagged to a tier the user can't see would be dropped here, before the reranker ever sees it.mistral for grounded generation.Arsenal won the North London Derby with a score of 3‑11.
The same pipeline, asked something the indexed corpus has no answer for. No fact is invented — the canonical fallback ships instead.
I do not know the answer based on indexed documents.
6 hand-curated questions over a 4-document corpus, scored end to end
through the real pipeline above — retrieval, generation, and an LLM
faithfulness judge, nothing mocked. See
eval/README.md for the full methodology.
What each metric actually measures
Retrieval precision — 1.0
Out of every question asked, this measures how often the system actually found and used the correct source document to answer from. A perfect 1.0 means every question in this test set pulled the right document, with no wrong sources sneaking in.
Faithfulness rate — 0.75
This measures how often the AI's written answer sticks strictly to what its cited sources actually say, without adding claims the sources don't support. A 0.75 means 3 out of every 4 answers were fully backed by their citations.
Hallucination rate — 0.167
This measures how often the system either invented a fact or confidently answered a question it should have declined. Roughly 1 in 6 answers here had some kind of fabricated or unsupported claim — the number this whole pipeline exists to push toward zero.
Errored — 0
This counts how many questions caused the pipeline to crash or fail outright, rather than producing any answer at all. Zero means every question got a real response — correct or not — with nothing silently lost.
Avg. duration — 76.7s
This is the average time it took to go from receiving a question to returning a finished, checked answer. It's slow because every step — search, rerank, generate, safety checks — is real computation running on local hardware with no cloud acceleration, not a mock.
Where this falls short of industry standard
Faithfulness (0.75) and hallucination rate (0.167) both fall short of production-grade RAG benchmarks, which typically target faithfulness above 0.9 and hallucination under 0.05 — largely because generation here runs on mistral, a small 7B model chosen for free local development rather than accuracy. The most direct fix is swapping in a stronger generation model, since answer quality in a RAG pipeline is usually bottlenecked by how reliably the model follows "only answer from the sources" instructions, not by retrieval, which is already a perfect 1.0. A stricter grounding check in the output/citation guard would also catch more borderline unsupported claims before they ship. Average duration (76.7s) is likewise far outside typical production latency targets (1–3s), but that's a hardware artifact of running everything locally, not a quality problem, and would shrink substantially by moving generation and embedding to GPU-backed inference.