Design a retrieval augmented generation system over our internal documentation. What are the moving parts?

Ingest and chunk the documents with structure aware splitting, embed the chunks, and index them in a vector store alongside metadata for filtering. At query time, embed the question, retrieve candidates, optionally rerank with a cross encoder, then build a prompt with the retrieved passages and citations. Evaluate retrieval and generation separately, and keep the index refreshed as documents change.

Offline, I chunk on structure rather than a fixed character count, so headings and sections stay intact, with a little overlap so a sentence spanning a boundary is not lost. Each chunk gets an embedding plus metadata: source, section, last modified, and access group, because permissions have to be enforced at retrieval time, not by asking the model nicely. That goes into a vector index with a keyword index alongside it, since hybrid search consistently beats pure vector on internal docs full of error codes and product names that embeddings handle badly. Online, I embed the query, pull maybe fifty candidates, rerank with a cross encoder down to five or six, then build the prompt with those passages and instructions to cite them and to say plainly when the context does not cover the question. The part I insist on is separate evaluation. I build a set of real questions with known correct source documents and measure recall at k for retrieval on its own. Most of the time when an answer is wrong, retrieval never surfaced the right chunk, and no amount of prompt work fixes that.