Festival RAG Chatbot

Every festival runs on the same forty questions asked five thousand times: when does registration close, where is the venue, what does the pass cover. This service was the attempt to answer them from the event copy itself rather than from a hand-written FAQ tree, so the answers stay correct when the copy changes.

The FAQ tree is the naive version and it fails in a specific way. Someone writes it once, three weeks before the festival, and then the schedule shifts, a venue changes and a pass tier gets added, and now the tree is a machine for confidently giving wrong answers. Retrieval flips the dependency: the copy is the source, the index is derived, and re-ingesting is cheaper than editing. The cost of that flip is that every question the system answers well is a question whose answer literally appears somewhere in the corpus, which turns out to be the harder half of the problem.

The pipeline

Ingestion takes raw text over HTTP, writes it to a scratch file so LangChain's TextLoader can pick it up, and splits it with RecursiveCharacterTextSplitter at 500 characters with 50 characters of overlap, preferring paragraph breaks, then line breaks, then sentence periods. Chunks are embedded locally by all-MiniLM-L6-v2 through sentence-transformers — no embedding API call, no per-token cost on ingest — and written into a Chroma collection that persists to chroma_db/ on disk.

Query time is deliberately thin. A similarity search pulls the top three chunks, they get concatenated into a context block, and that block is prepended to a system prompt telling the model to answer only from the context, to say it does not know rather than guess, and to cite the source. Generation runs on Groq against mixtral-8x7b-32768 at temperature 0.5, capped at 1024 tokens, non-streaming.

Those two numbers are coupled and worth naming together. Five hundred characters is roughly a paragraph, and three of them is roughly one event's worth of description, which is the right unit for "when does the music night start". It is the wrong unit for "which events are free", because that answer is spread across every chunk in the corpus and top-three retrieval will only ever see three of them. A festival knowledge base has both kinds of question in it, and this design is honest about only serving the first kind.

What's wired up

• POST /ingest takes a text_content string, rebuilds the vector store and reports how many chunks landed in the collection.
• POST /chat takes a query and returns a typed answer, with Pydantic models on both sides of every endpoint.
• Local embeddings, instantiated once at module import so the model loads on startup rather than on the first request.
• Persistent Chroma with a named collection and a get_vectorstore_from_disk helper for reattaching to an existing store without re-ingesting.
• Top-k retrieval through a single get_relevant_documents(vectorstore, query, k=3) helper, so the retrieval knob is one argument in one place rather than scattered through the route.
• A grounding prompt that pins the model to retrieved context and gives it an explicit "I don't know" escape hatch.
• CORS wide open for browser clients, with a comment noting it needs narrowing before anything real.
• Explicit 400 when a chat request arrives before anything has been ingested, instead of an empty-retrieval answer.
• Temp-file cleanup in a finally block so a failed ingest does not leave scratch files behind.

Two files, and a clean line between them

The split is the one design decision I would keep. app.py is transport: Pydantic models, CORS, the Groq call, the HTTP error mapping. processor.py is retrieval: the embedding function, chunking, the Chroma handle, the similarity search. Nothing in processor.py imports FastAPI and nothing in app.py knows what a chunk is, which means swapping Chroma for something else or moving from a route to a queue worker touches exactly one file. At two hundred lines that split looks like ceremony; it is the thing that would still be paying off at two thousand.

Unfinished edges

This never got past prototype, and reading it back the reasons are visible in the file. app.py constructs a Groq client that it never imports. ingest_data calls create_vectorstore with one argument when the function signature requires a collection name. The __main__ guard is misspelled with single underscores, so the uvicorn block never fires and the app only runs under an external server command. The vector store also lives in a module-level global, which quietly undoes the point of persisting to disk — a restart loses the handle even though the embeddings are still sitting in chroma_db/. That is exactly what get_vectorstore_from_disk exists to fix, and app.py never imports it either.

There is a fourth failure of the same kind that only shows up on the wrong operating system. The .env names the key in lowercase while app.py reads GROQ_API_KEY, and because Windows treats environment variable names case-insensitively, this starts fine on the machine it was written on and raises at import on any Linux host. requirements.txt has the matching gap: it lists langchain-community but not langchain, while processor.py imports from langchain.text_splitter and langchain.vectorstores.base. Nothing in the file is pinned to a version. The CORS block sets allow_origins=["*"] together with allow_credentials=True, which browsers reject as a combination, so the permissive setting is not even permissive. The model id is a bare string literal with no fallback, which is a problem the first time a hosted model is retired. And the /chat comment says it sends the full chat history when it sends a system prompt and one user turn, so follow-up questions carry no memory of the previous answer.

The corpus in data/ is one 95 MB PDF of event descriptions. Nothing in the dependency list can read a PDF — TextLoader handles plain text, and /ingest takes its content as a JSON string field, so the actual knowledge base cannot enter the pipeline it was collected for without a loader that was never written. Fixing the wiring is an afternoon; deciding what "the festival knowledge base" actually is was the harder question, and the festival stack ended up shipping without a chatbot.