Retrieval-augmented generation shines in demos and stalls when grounding, evaluation, and latency meet real documents. These lessons come from shipping RAG behind authentication with messy PDFs and shifting policies.
Chunking and grounding discipline
Tune chunk boundaries to semantic headings and tables; naive fixed windows fracture tables that finance teams rely on.
Attach citations to spans users can verify. If an answer cannot cite a chunk, return a refusal rather than hallucinating structure.
Evaluation loops that survive updates
Maintain golden questions per vertical with expected citations and banned phrases. Run them on every index rebuild.
Track retrieval precision separately from generation quality—many regressions are recall problems, not prompt problems.
Latency, cost, and caching
Cache embeddings for stable corpora and memoize tool calls when policies allow. Users tolerate thoughtful delays more than inconsistent speeds.
Instrument token usage per tenant for billing forecasts and noisy-customer detection.
