How to Build a RAG Pipeline for Enterprise Search: A Practical Guide

Retrieval-Augmented Generation gets pitched as “add a vector database and your chatbot knows your docs.” In production, the vector database is the easy 20%. The other 80% is chunking strategy, retrieval quality, and knowing when RAG is the wrong tool. I built a RAG-based research copilot that let sales teams query deep company intelligence in natural language, backed by ChromaDB and CosmosDB — cutting company research that used to take hours down to seconds. Here’s what actually mattered.

The pipeline, at a high level

Ingest — pull source documents (CRM notes, call transcripts, web research, internal docs)
Chunk — split into retrievable units
Embed — turn each chunk into a vector
Store — index vectors for similarity search
Retrieve — given a query, pull the most relevant chunks
Generate — pass retrieved context + query to an LLM, get an answer

Steps 3 through 6 are commodity now — every vector DB vendor has a quickstart. Steps 1 and 2 are where quality is actually won or lost.

Chunking is the real architecture decision

Naive fixed-size chunking (split every 500 tokens) is the default in most tutorials and it’s usually wrong for anything beyond a toy demo. A 500-token chunk cut mid-sentence, mid-table, or mid-list destroys the semantic unit the retriever needs to match against. What worked better:

Structure-aware splitting — split on headings, paragraphs, and table boundaries first, then only fall back to token-count splitting within an oversized section
Overlap — 10–15% overlap between adjacent chunks so a fact sitting near a chunk boundary isn’t orphaned
Metadata-rich chunks — every chunk carries its source document, section title, and timestamp as metadata, not just as embedded text. This is what makes filtered retrieval (WHERE source = 'call_transcript' AND date > X) possible, and it’s the difference between a RAG system that can say “I don’t know” versus one that hallucinates from stale data.

Embeddings: don’t overthink the model, overthink the input

Model choice matters less than most people assume once you’re past the very cheapest options — the bigger lever is what text you actually embed. Embedding a raw HTML dump versus a cleaned, structure-preserved version of the same content can be the difference between mediocre and good retrieval. Strip boilerplate, preserve structure signals (headings, lists) as plain text markers, and keep chunks focused on one topic each.

Vector store choice is a scaling decision, not a quality decision

ChromaDB is genuinely good for getting to production fast — it’s simple to run, has a clean API, and for most mid-size corpora (hundreds of thousands of chunks, not tens of millions) performance is not your bottleneck. Where I’ve seen teams over-invest early is picking a heavyweight, ops-intensive vector database before they have enough data or query volume to need it. Start simple, migrate when you have a measured reason to.

Where RAG actually fails in production

Query-document mismatch. A user asks “what did we discuss about pricing last quarter” — that’s a temporal + topical query, and pure semantic similarity search on the raw question often won’t retrieve the right chunks. Query rewriting (turning a conversational question into a more retrieval-friendly query, or extracting explicit filters) fixes most of this.
Too much context, not enough signal. Retrieving the top 20 chunks and stuffing them all into the prompt doesn’t help the model — it dilutes the signal and increases the chance of the model latching onto an irrelevant chunk. Re-ranking retrieved results with a cheaper, faster relevance step before generation consistently improved answer quality more than any embedding model swap did.
No feedback loop. The systems that get better over time log every query, every retrieved chunk set, and every generated answer, and use that to catch retrieval misses. Without that loop, you’re flying blind on whether the system is actually improving.

The honest takeaway

RAG is not “add embeddings, get a chatbot.” It’s an information retrieval system with a language model bolted onto the output stage — and it will only ever be as good as the retrieval half. If you’re evaluating a RAG project, spend your first week on chunking and metadata design, not on comparing LLM providers. That’s where the quality actually comes from.