A RAG system (retrieval augmented generation) lets a language model answer from your own documents by retrieving relevant passages first. The RAGAS framework scores that system with four core metrics from 0 to 1: faithfulness, answer relevancy, context precision and context recall. The lowest score points to the part you should fix first.
A RAG system can look brilliant in a demo and still fail your users in production. The answers sound confident, but nobody can say whether they are grounded in the right documents.
This guide explains how a RAG system works, how to evaluate a RAG system with the RAGAS framework, and how to turn scores into fixes. It ends with an interactive diagnosis tool you can try right now.
What Is a RAG System?
A RAG system combines a search step with a text generation step. Lewis et al. introduced the idea in 2020 as a way to give generative models access to knowledge outside their training data.
Instead of asking a model to answer from memory, the system first finds passages that look relevant. It then hands those passages to the model along with the question.
RAG system architecture in four parts
Most production RAG system architecture follows the same shape:
- Ingestion: documents are cleaned, split into chunks and converted into embeddings.
- Retrieval: the question is embedded and the closest chunks are fetched, often with a re-ranker on top.
- Augmentation: the chosen chunks are placed into the prompt as context.
- Generation: the model writes an answer, ideally using only that context.
Each part can fail on its own. That is why a single “does it look right?” check is not enough.
Why a RAG System Fails Quietly
When a normal application breaks, you see an error. When a RAG system breaks, you see a fluent answer that is slightly wrong.
The causes fall into two groups. Retrieval failures mean the right passage never reached the model, or reached it buried under noise. Generation failures mean the model had good context and still added claims, ignored the question or rambled.
Good RAG system evaluation separates these two groups. Without that split, teams tune prompts when the real problem is chunking, and swap models when the real problem is search.
What Is the RAGAS Framework?
The RAGAS framework is an open source toolkit for evaluating RAG pipelines. Es et al. described it in 2023 as a way to score retrieval and generation quality, in many cases without needing hand written reference answers for every question.
It uses a language model as a judge. For each test row, it looks at the question, the retrieved contexts and the generated answer, then returns scores between 0 and 1. Higher is better.
Source note: The RAGAS paper (Es et al., 2023) frames evaluation as separate checks for retrieval quality, faithful use of context and answer relevance. That separation is what makes the scores actionable.
RAGAS Framework Metrics Explained
The core RAGAS framework metrics map neatly onto the architecture above. Two judge retrieval and two judge generation.
RAGAS faithfulness metric
The RAGAS faithfulness metric asks whether every claim in the answer can be supported by the retrieved context. A low score means the model is adding facts that the documents never stated. In regulated settings, this is the score to watch first.
Answer relevancy
Answer relevancy checks whether the response actually addresses the question. An answer can be perfectly faithful and still useless if it talks about something adjacent.
Context precision
Context precision looks at the retrieved chunks and asks whether the useful ones are ranked near the top. Low precision means noise is crowding the prompt, which raises cost and distracts the model.
Context recall
Context recall asks whether the retrieved context contains the information needed to answer. It usually needs a reference answer (ground truth) to compare against. Low recall means the answer was never available to the model in the first place.
| Metric | Stage | Question it answers | Typical first fix |
|---|---|---|---|
| Faithfulness | Generation | Is the answer supported by the context? | Tighten the prompt, require citations |
| Answer relevancy | Generation | Does it answer what was asked? | Rewrite the prompt, handle vague queries |
| Context precision | Retrieval | Are the best chunks ranked first? | Add a re-ranker, reduce top k |
| Context recall | Retrieval | Was the needed information retrieved? | Improve chunking, hybrid search |
How to Evaluate a RAG System Step by Step
Here is how to evaluate a RAG system without turning it into a research project.
- Build a test set. Collect 50 to 100 real questions from users or subject experts. Add reference answers where you can.
- Run your pipeline. Save the question, retrieved contexts and generated answer for each row.
- Score with RAGAS. Compute the four core metrics for every row.
- Read the weakest rows. Averages hide problems. Open the ten lowest scoring examples and look at them yourself.
- Change one thing. Adjust chunk size, add a re-ranker or edit the prompt, then rerun.
- Keep the baseline. Store scores per version so you can prove a change helped.
A minimal run looks like this. Column names and imports change between RAGAS versions, so check the current documentation before copying.
from ragas import evaluate
from ragas.metrics import (faithfulness, answer_relevancy,
context_precision, context_recall)
result = evaluate(dataset, metrics=[
faithfulness, answer_relevancy,
context_precision, context_recall])
print(result)
Treat LLM judged scores as a strong signal, not a verdict. Spot check them against human review, especially for your first test set.
Try It: RAG System Diagnosis Tool
Enter your four RAGAS scores below. The tool finds your weakest link and suggests where to look first. It runs in your browser and stores nothing.Faithfulness: 0.80Answer relevancy: 0.80Context precision: 0.80Context recall: 0.80
Average: 0.80. Weakest link: Faithfulness (0.80), a generation metric.
The model is adding claims the documents do not support. Tighten the prompt to answer only from context, require citations, and check for conflicting chunks.
Scores are relative to your own baseline. Fix the weakest link, rerun, and compare.
RAG System for Enterprise: What to Add Beyond Scores
A RAG system for enterprise use carries risks that scores alone do not cover. Quality is necessary, but not sufficient.
- Access control: retrieval must respect who is allowed to see each document.
- Freshness: stale policies produce confident, outdated answers.
- Citations: show the source passage so users can verify.
- Human review: sample live answers every week and feed failures back into the test set.
- Cost and latency: track them next to quality, since a higher top k improves recall but raises both.
Teams building a RAG system for enterprise workflows should also define who owns the knowledge base. Without an owner, evaluation scores drift as content changes.
Common RAG Evaluation Mistakes
- Testing only on easy, hand picked questions.
- Reporting one blended score instead of four separate ones.
- Changing chunking, prompt and model in the same release.
- Skipping the manual read of low scoring rows.
- Treating a score threshold from a blog post as universal. Set your own baseline and improve against it.
Frequently Asked Questions
What is a RAG system in simple terms?
It is an AI application that looks up relevant passages from your documents first, then writes an answer based on them.
What is the RAGAS framework used for?
It scores RAG pipelines on retrieval and generation quality so you can find weak links and track improvements over time.
How do I evaluate a RAG system with few labeled examples?
Start with faithfulness, answer relevancy and context precision, which can run without reference answers. Add context recall once you have ground truth for a smaller set.
What is a good RAGAS score?
There is no universal cutoff. It depends on your domain and risk. Set a baseline on your own data, then raise it release by release.
Does RAG remove hallucinations?
No. It reduces them by grounding answers in documents, but the model can still add unsupported claims. That is what the faithfulness metric is for.
