Case study · Anonymised
Enterprise RAG on Bedrock across 2M+ documents
The problem
More than 2 million internal documents (PDFs, Confluence spaces, Slack exports) sat across systems with no unified search. Engineers spent 30+ minutes finding answers that were already documented, and keyword search missed the meaning of the question.
Constraints
Data had to stay inside the AWS environment. Engineers would not rely on an AI answer unless they could see the exact source. Repeated questions would make model usage expensive without cost controls.
The decision
Build retrieval-augmented generation on Amazon Bedrock with hybrid keyword and vector search in OpenSearch Serverless, a semantic cache in ElastiCache, and source attribution on every answer. Keep ingestion, retrieval and answer generation separable so each can be rolled back on its own.
The decisions that mattered
Source attribution was required for adoption. Engineers would not trust an answer without the exact document and page behind it. Each answer shows the chunks that informed it, and this feature drove most of the adoption.
Chunk size was the biggest lever on retrieval quality. 256-, 512- and 1,024-token chunks were tested. 512 tokens with a 50-token overlap gave the best balance of precision and context.
Hybrid retrieval rather than vectors alone. Vector similarity combined with keyword filtering beat pure vector search on the relevance benchmark.
Options considered
- OpenSearch Serverless or Pinecone. OpenSearch was chosen for AWS-native integration, keeping data inside the VPC, and built-in hybrid keyword and vector search.
- Model choice. Claude on Bedrock was chosen for its longer context window, instruction following and native Bedrock availability.
- Semantic caching or none. An ElastiCache layer returned cached answers for semantically similar questions (cosine similarity above 0.95).
How it was built
Lambda connectors synced each source incrementally. A Step Functions pipeline handled extraction, chunking, Titan embeddings and indexing into OpenSearch Serverless. Answers streamed to a React interface over a WebSocket API.
The useful production signals were retrieval latency, cache hit rate, answers with no source, failed ingestion jobs and user feedback on source relevance.
What went wrong
The semantic cache cut LLM cost but created an invalidation problem: when a source document was updated, every related cache entry had to be flushed. That required building a document-to-cache mapping.
Results
| Measure | Before | After | When |
|---|---|---|---|
| Time to find a documented answer | more than 30 minutes | under 10 seconds | After rollout |
| Search relevance (MRR@10) | 0.31 (keyword search) | 0.78 (hybrid retrieval) | Relevance benchmark |
| Repeated-query LLM spend | no caching | about 60% lower | After the semantic cache shipped |
Figures are as reported by Rahul and recorded in the InfraTales proof register. Measured on the client's systems; not a promise of the same result elsewhere.
When this approach would not fit
A semantic cache helps only when many users ask similar questions about content that changes slowly. When documents change constantly or every query is unique, the cost of invalidation can outweigh the savings. Hybrid retrieval is worth building when users search with exact identifiers as well as natural language. For a small, uniform corpus, a managed knowledge base may be enough.
Verified record
- Engagement
- As Senior AWS Solutions Architect on an enterprise engagement under NDA, Rahul architected a production RAG platform on Amazon Bedrock over more than 2 million internal documents, with hybrid retrieval, semantic caching and source attribution, working with a team of three engineers for five months.
- Results
- Time to find a documented answer fell from more than 30 minutes to under 10 seconds, search relevance (MRR@10) rose from 0.31 with keyword search to 0.78 with hybrid retrieval, and semantic caching cut repeated-query LLM spend by about 60%.