Retrieval-Augmented Generation with LangChain

RAG retrieves external documents before generating an answer. It helps with private or changing information, but does not guarantee factual output. Inspect retrieval before blaming the prompt.

#Separate indexing from answering

Indexing loads documents, splits them, creates embeddings and stores text plus metadata. Querying retrieves authorized passages, formats them with source IDs and supplies them to the answer step.

A useful chunk needs enough context to stand on its own. Large chunks mix topics; tiny chunks can separate a rule from its exception. Tune size and overlap against real questions rather than treating one character count as universal.

#Inspect a minimal retriever

This Python example requires langchain-core, langchain-chroma and langchain-openai, an authorized key, and an available OPENAI_EMBEDDING_MODEL. Building the index sends billable embedding requests. The policy is fictional, not employment advice.

pythonCopy
import os
from langchain_core.documents import Document
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

documents = [Document(
    page_content="Fictional demo policy: meals are reimbursed up to $50 per person with a receipt.",
    metadata={"source": "demo-handbook", "section": "meals", "version": "1"},
)]
store = Chroma.from_documents(
    documents,
    OpenAIEmbeddings(model=os.environ["OPENAI_EMBEDDING_MODEL"]),
    collection_name="demo_policy",
)
for document in store.as_retriever(search_kwargs={"k": 1}).invoke("Can I claim an $80 meal?"):
    print(document.metadata, document.page_content)

First confirm the meal rule is retrieved. Then build the answer prompt with the question, retrieved text and source identifiers. Ask for an explicit missing-evidence response and validate citations against the passages actually supplied.

#Improve quality systematically

Compare semantic retrieval with keyword or hybrid retrieval. MMR can reduce duplicate passages; threshold scores need calibration for the chosen store and distance metric. A threshold such as 0.7 is not portable across all indexes.

Keep document versions, access metadata and deletion handling in the index. Apply tenant and document permissions during retrieval. Treat retrieved instructions as untrusted content. Test absent answers, conflicting versions and unauthorized documents as well as easy exact matches.

Retrieval documentation↗