Grounded Retrieval Engine for an AI-Native LMS

An AI-native LMS wants to generate assignments, tests, answer evaluations, flashcards and videos. Every one of those features needs the same thing: find the right material in a large, messy content library and prove exactly where it came from. Let each build its own retrieval and you get several inconsistent, half-grounded pipelines and no single source of truth for citations. This is a 34-page architecture blueprint, v1.0 dated 16 July 2026, specifying the alternative: one retrieval engine every generation engine calls, returning grounded context plus exact citations plus an honest confidence signal, and never authoring the artifact the user sees. I am the author; this is a design document, not a system I have deployed, and the sections below describe what it specifies rather than what is running.

The setting the design has to survive

The library grows along two axes. Global content is curated by the SaaS team - publisher textbooks, standard references, question banks - and is read-only to every school. School content is uploaded by an individual school: teacher lesson notes, worksheets, past papers, and student-generated material such as scanned handwritten notes. It arrives in every conceivable format, from clean born-digital PDFs to photographs of a whiteboard, slide decks, spreadsheets, diagrams and recorded lectures. The target output is a citation a teacher could check by hand, of the form "this flashcard comes from Chapter 3, page 42, paragraph 3 of the NCERT Biology textbook," or, for a messy source, "this claim is supported by the teacher's note, page 2, at OCR confidence 0.72."

Six goals are ranked in priority order, with never losing provenance first and tenant isolation second, and the non-goals are stated as flatly as the goals. The engine does not generate user-facing artifacts. It does not decide what to pull - which book or chapter or concept is an orchestration call owned by the generation service. And it does not own the learner model. Everything downstream follows from treating retrieval as infrastructure and generation as an application sitting on top of it.

Provenance is a property of the data, not of the generation step

The central argument is that you cannot ask a language model to cite its own sources. The literature review behind it puts untrained RAG citation accuracy at 65-70%, with roughly 57% of generated citations being post-hoc rationalisations where the model answers first and then finds token overlap to justify itself.

So provenance is pushed into the data model. A Block is the immutable citation anchor, extracted once, never rewritten, carrying page number, paragraph index, a bounding box normalised to 0-1 of the page, extraction confidence, and for audio and video a start and end timestamp. A Chunk stores block_refs[] and is forbidden from holding a raw locator. Two things follow, and they are why the discipline pays. You can re-chunk or re-embed with a new model without breaking a single citation, because citations hang off blocks and blocks never move. And a chunk spanning a page break resolves to a page range rather than picking one and lying about it. A citation's confidence is the minimum over its cited blocks, so it is never more confident than its shakiest source.

Resolution is therefore a lookup rather than an inference. The resolver takes a chunk's block references, groups them by document and page, derives a page range and a paragraph range, and picks a primary block - highest scoring under SEARCH, tightest sentence-level entailment under VERIFY. Because boxes are stored normalised against the page's pixel dimensions, drawing the highlight back onto a rendered page image works at any DPI, which is what turns a citation into a "show me the source" overlay instead of a string. Blocks below a confidence threshold are flagged for optional teacher review, excluded from VERIFY entirely and down-weighted in SEARCH, so a bad OCR pass degrades ranking rather than quietly becoming ground truth.

Isolation comes from the token, never from the request

Global content curated by the SaaS team is read-only to every school; a school's own uploads are visible to that school alone. The blueprint cites research showing up to 95% of benign queries can trigger cross-tenant leakage when isolation is left to application code, and puts the filter at the index layer, derived from a signed JWT claim.

The sharper move is separating that hard boundary from soft ones. Crossing tenant scope is a breach; grade and subject are relevance facets, so crossing those is a quality problem. Both ride the same pre-filter machinery at different strengths, which lets grade and subject double as partition keys: a Grade-7 Science query never scans Grade-11 History vectors, so the mechanism keeping results on topic also shrinks the ANN search space.

Modelling the boundary as a scope attribute rather than as two literal buckets is what lets a third tier - a per-teacher class scope - arrive later without a migration. The effective predicate is a conjunction where only the first term is non-negotiable: scope global or tenant matching the caller, and that clause comes from the token and cannot be widened by the request body. Grade allows a policy-bounded fallback of plus or minus one when coverage is thin. Subject is a straight filter. Concept IDs are off unless the caller passes focus_concepts. Underneath, the defence is layered - per-school object storage prefixes graduating to per-tenant buckets and KMS keys for high-risk tenants, short-lived signed URLs for page images so nothing durable is public, and an adversarial isolation test in CI that asserts School A queries never return School B chunks. The security claim is a test, not a paragraph.

Three modes, because one retrieval shape fits nothing

"Flashcards for all of Chapter 3" is a complete ordered fetch over a bounded scope. "The exact line that proves this claim" is a precision problem. The engine exposes FETCH (locator-driven, ranking off), SEARCH (dense plus BM25 fused with Reciprocal Rank Fusion at k=60, then an adaptive cross-encoder rerank) and VERIFY (claim to sentence entailment before a citation counts as supporting). The caller declares intent, the engine decides mechanics.

Assembly is still retrieval. Before returning, the engine dedupes near-duplicates, orders passages by document position, fits them to the requested token budget, attaches citations and prefers the school's copy over the global one where a school has annotated a book it already had. No model authors anything at this step; it only selects and arranges. The rule is stated outright - the engine may use models to find and assemble context, and never to author the artifact the user sees. Its half of the contract ends at passages, citations, a calibrated grounding confidence and a no_good_match boolean, and the generation engines own the prompts, the LLM call and the option of calling VERIFY back on their own output.

Where the 400 milliseconds go

The latency budget targets p95 under about 400 ms and is honest about where it goes: fusion is under 5 ms and citation resolution under 10 ms, while the reranker is 50-150 ms and dominates. So the biggest lever is skipping the reranker when fused scores already separate cleanly. FETCH, which does no search or ranking at all, comes in under about 60 ms. Three caches sit in front of the expensive stages - an exact query-embedding cache, a semantic cache for near-duplicate queries, and a hot-content cache for popular chunks - and the observation that justifies them is that generation workloads repeat their queries heavily, so hit rates are high in exactly the workload this engine serves. The cost model makes the same split: ingestion is a one-time cacheable capital cost per document, retrieval is a recurring per-query cost dominated by the reranker, so you optimise ingestion for correctness and retrieval for the rerank-skip rate.

Getting every format into one Block model

Five extraction paths converge on one canonical Block. A classifier routes on magic bytes, the presence of a text layer and estimated handwriting percentage. Born-digital PDFs go through PyMuPDF in-house and yield page, block, line, bbox and character offsets. Scans, photos and handwriting go to a managed document-AI OCR, which returns a page-block-paragraph-line-token tree with bounding polygons and per-block confidence. Slides map slide number onto page and placeholder order onto paragraph. Figures become a captioned, multimodally embedded chunk with a stored crop, so "the diagram on page 42" is both retrievable and citable. Audio and video go through ASR with timed segments and a null page, citing as "Lecture 5, 12:34-12:58."

Handwriting OCR is the clearest build-versus-buy call in the document, and it goes to buy. A small team is not going to beat managed document AI on messy handwriting quickly, and the managed providers already return the word-level boxes the citation overlay needs. It sits behind an OCRProvider interface so a self-hosted model is a later swap, with a deskew/denoise/contrast pass in front of it to lift baseline confidence on bad scans.

What else the blueprint specifies

• A ten-stage ingestion pipeline as a resumable state machine, with artifacts persisted so a re-chunk or re-embed never re-runs OCR.
• Idempotency keys on every stage - OCR keyed by checksum, embeddings by chunk hash and embed model version - plus priority lanes so a teacher's generate-now upload jumps bulk publisher ingest, and per-tenant concurrency caps.
• Structure-aware chunking at 200-400 tokens, cut on heading and paragraph boundaries from the Block layer rather than on a fixed window.
• Contextual chunk prefixes after Anthropic's Contextual Retrieval, the section breadcrumb doubling as a human-readable citation.
• Three representations per chunk - dense embedding for paraphrase, BM25 for formulae and proper nouns and question IDs, and a multimodal embedding for figures.
• Embedding version discipline, where a model upgrade is a blue/green reindex.
• A migration path from pgvector to Qdrant behind a VectorIndex interface, with the crossover placed by vector count rather than by preference.
• A public API of /ingest, /retrieve, /resolve, /citation/{id} and /feedback, every one of them deriving tenant and scope server-side.
• An evaluation harness tracking recall@k and nDCG alongside citation accuracy, the metric nobody else measures, broken out per grade and per subject rather than in aggregate.
• Per-retrieve tracing recording which indices were hit, the effective tenant filter and whether the rerank actually ran.
• A learner model outside the engine, starting at an exponentially-weighted mastery score per concept and leaving room for BKT or DKT, feeding focus_concepts through a shared concept taxonomy that both content and test questions are tagged against.
• A four-phase roadmap, each phase closed by a concrete acceptance test.

What it is and what it is not

This is a design, not a deployed system, and it says so. The roadmap is the part that keeps it honest: Phase 0 is born-digital PDFs, one tenant, dense-only pgvector, and its acceptance test is to ingest a textbook, run ten hand-written queries and open the page image to confirm each cited page and paragraph truly contains the text. Hybrid search, the reranker and the tenant filter are Phase 1; OCR, ASR and figures are Phase 2; the concept taxonomy and the Qdrant migration are Phase 3. Nothing is claimed to work before the phase that builds it.

What the document demonstrates is the reasoning. There is a prior-art review that says plainly what it takes from each source and what it rejects - the whole Contextual Retrieval recipe in, LLM-authored citations out, namespace-plus-JWT tenancy in, application-code isolation out. The risk table names cross-tenant leakage as the one severe entry and puts an adversarial CI test against it. And the no_good_match contract means a consumer that refuses to answer is making a deliberate choice rather than hitting an accident.