RAG Evaluation Framework: 4 QA Checks That Catch CI Failures

QA is signing off support chatbots that pull policy docs and answer in a chat bubble. The CI job stays green because billing-chat.spec.ts only waited for that bubble to appear.

That green job hides a stale index. A tester asks how to refund a subscription after 14 days, the retriever still returns the old 14-day window, and the model answers with total confidence.

This guide is the QA version of a RAG evaluation framework for that refund question: the four checks, which tool belongs in the suite, a worked test, CI pass/fail rules, and the Playwright assertions that still belong on the UI. You will reuse the same checks on every later billing query, not only this one.

What a RAG evaluation framework actually measures

A RAG evaluation framework gives QA two oracles: did retrieval pull the right passages, and did generation answer from those passages.

DeepEval's RAG guide puts the system under test in those two boxes. The retriever returns chunks. The generator writes the answer. If your test only checks the final sentence, you cannot file the defect in the right component.

Ragas was built for the common case where you do not have a hand-labeled answer for every query. You can still run a RAG evaluation framework on faithfulness, answer relevance, and context quality.

You still need a test log of what the pipeline actually did. Treat one query like one test case and store:

• user_input: the question
• retrieved_contexts: the chunks the retriever returned, in rank order
• response: the generated answer
• reference: a gold answer, if you have one
• document IDs, if your index keeps them

Without retrieved_contexts, the test case has no evidence. You checked a chatbot string, not the retrieval the answer depended on. Faithfulness has nothing to assert against.

The refund chat is the test case we will keep. Product updated the policy to 30 days. The vector index still holds refunds-v3.md, which says 14 days. The UI test clicks Send, waits, and passes. That's a false green, the same miss you get when a Playwright assertion only checks that an element exists, not that the value is right. A RAG evaluation framework catches that miss because it scores the chunks, not only the bubble.

A useful framework answers four questions on that query:

1. Did we retrieve the current policy chunk?
2. Did we rank that chunk above noise?
3. Did the answer stay inside the retrieved text?
4. Did the answer actually address the refund question?

BLEU and ROUGE do not answer those. They count word overlap with a reference. A RAG evaluation framework scores retrieval and generation instead of n-gram overlap.

LangSmith splits the work the way QA already splits a suite. Offline evals are the regression set: curated cases, optional expected answers. Online evals score live traces that have no gold answer. You need both once customers hit the chat. You do not need both on day one. Your RAG evaluation framework can start offline and add live sampling later.

Start with a small regression set. Twenty refund and billing questions is enough to catch a stale chunk. Then sample production traces the way you sample escaped defects.

Those three bars come from the Ragas walkthrough, not from a public industry survey. Put them next to your own refund scores later. A RAG evaluation framework is only useful when you compare your failure to a known-good run.

The 4 metrics that catch most RAG regressions

Four checks cover most RAG defects QA will file: faithfulness, answer relevance, context precision, and context recall. Put all four in the RAG evaluation framework, not one favorite score.

Run them as a set, the way you would not ship on one assertion. A single high number can lie. The refund test case is the proof.

Faithfulness

Faithfulness asks whether every claim in the answer can be inferred from the retrieved chunks.

Ragas computes it as supported claims divided by total claims in the response. The score runs from 0 to 1. Their docs walk through Einstein's birth date. "Born in Germany on 14 March 1879" scores 1.0 against a matching context. Change the day to 20 March and the score drops to 0.5, because one of two claims is unsupported.

DeepEval's FaithfulnessMetric does the same split: extract claims, then check they do not contradict retrieval_context. Print that reason next to the number.

Here is the defect most roundups skip. Faithfulness can be 1.0 and the release can still be wrong.

The retriever returned refunds-v3.md: "Refunds are not issued after 14 days." The generator said, "We cannot refund a subscription after 14 days." Every claim sits in that chunk. The score is perfect. The live policy is 30 days. You shipped a faithful wrong answer.

Tip: Write the judge model name next to the score. Ragas docs use llm_factory("gpt-4o-mini"). Swap that model and the number moves even when the refund pipeline did not.

So treat faithfulness as a hallucination check against retrieved text. Correctness against the live policy is a different score. Pair it with retrieval recall, or with a small labeled correctness set, or you'll bless stale docs.

Use a judge model you can name in the eval log. Ragas stable docs show llm_factory("gpt-4o-mini", ...) for the collections API. If the generator and the judge share the same blind spot, both will agree the 14-day sentence is fine.

Answer relevance

Answer relevance asks whether the response addresses the question that was asked.

Ragas (the collections class is AnswerRelevancy) does not compare the answer to a gold string. It generates a few questions from the answer, embeds them, and takes the mean cosine similarity to the original user input. The default is three generated questions. The score usually sits between 0 and 1, but cosine similarity can theoretically fall outside that range.

A high score means the answer is on topic. It does not mean the topic is grounded. "We process refunds in 5 to 7 business days, and you can track them in the billing portal" can look relevant to a refund question while inventing a portal that the chunks never mentioned. Faithfulness would catch that. Relevance would not.

A low score is the ramble. The user asked about a refund after 14 days. The model wrote three paragraphs on how subscriptions work. The reconstructed questions drift. The cosine mean drops.

DeepEval's AnswerRelevancyMetric is the generator-side pair to faithfulness. Treat it as a check on the prompt: did you instruct the model to answer the question from retrieval_context.

Context precision

Context precision asks whether relevant chunks sit at the top of the retrieved list.

Ragas computes a ranking-aware precision@k. An irrelevant chunk in position 1 hurts more than the same chunk in position 2. Their utilization example makes this concrete. For "Where is the Eiffel Tower located?", putting the Berlin chunk first drops the score to about 0.5. Putting Paris first and Berlin second stays near 1.0.

DeepEval maps this to ContextualPrecisionMetric, and says it is the metric that evaluates the reranker. Add it when you change top-k or the reranker.

On the refund query, suppose top-k is 4:

1. ocean-of-old-faqs.md (unrelated)
2. refunds-v3.md (stale 14-day rule)
3. refunds-v4.md (current 30-day rule)
4. billing-portal.md (partially related)

Recall might still look acceptable because refunds-v4.md made the list. Precision is poor because noise and a stale twin sit above it. The generator often reads the first chunk hardest. Rank order is the bug.

If you have no gold answer, Ragas ContextUtilization scores chunks against the generated response instead of a reference. Use that only as a stopgap. A faithful-wrong answer can make a stale chunk look "useful."

Context recall

Context recall asks whether the retrieved chunks contain what the gold answer needed.

Ragas is explicit: this metric always needs a reference. It splits the reference into claims, then checks whether each claim is attributable to retrieved_contexts. Without that expected result, this check stays empty.

Skip this metric and you will miss the refund bug. Faithfulness was 1.0. Relevance was high. Precision was messy but non-zero. Recall against the gold 30-day answer is the score that falls, because the current rule never made it into the context the model used, or sat too far down to matter.

If you don't have gold answers yet, don't invent them in a spreadsheet at 5 p.m. Use ID-based recall instead (next section). Or generate a synthetic set and have a human accept 20 rows before you trust a threshold.

DeepEval's ContextualRecallMetric is the sibling. Their guide ties it to whether the embedding model retrieved the information the question needed.

The four-metric read on the refund failure:

• Faithfulness: high (answer matched stale chunk)
• Answer relevance: high (it talked about refunds)
• Context precision: low (noise and stale twin ranked first)
• Context recall: low against a 30-day reference

File that as a retrieval defect. Leave the prompt alone until the index returns refunds-v4.

Picking a framework: Ragas, DeepEval, TruLens, LangSmith, Promptfoo

Pick the tool the way you pick a test runner: what has to fail the build this week. There is no single production winner in the official docs, and comparison blogs that crown one are guessing.

Those five tools are enough for a RAG evaluation framework on a billing chat. You do not need a longer catalog.

Ragas for dataset scoring

Ragas is the score report for a CSV of refund test cases after you change chunk size or the embedding model.

Stable docs now prefer the collections API (Faithfulness, ContextRecall, AnswerRelevancy). The older ragas.metrics helpers are marked for deprecation in 0.4 and removal in 1.0. If you copy a 2024 blog snippet, you may be on the legacy path already.

Use Ragas when QA wants a table of metric means after a retrieval change. Do not use it as your only PR gate. LLM-as-judge scores move when the judge model, temperature, or prompt wrapper changes.

DeepEval for pytest gates

DeepEval is the one that sits in the same test folder as the rest of the suite. assert_test(test_case, metrics) fails the process when a metric sits under its threshold. deepeval test run collects those files the way pytest does.

Their getting-started path is explicit: unit test the pipeline, pass retrieval_context, fail the build when faithfulness or answer relevancy drops.

Use DeepEval when you already think in pytest. Keep the PR-blocking metrics small and, if you can, deterministic. A judge that calls an API will flake in CI the same way a click flakes when the page is slow. You already know that pain from Playwright flaky test debugging. A RAG evaluation framework that flakes on the judge is not a gate. It is noise.

TruLens for traced production traffic

TruLens instruments retrieve and generate, then scores the spans. The usual triad is groundedness, context relevance, and answer relevance.

Skip TruLens if you can't wrap the app yet. You need span attributes for the query, the retrieved contexts, and the output. If all you have is a final string in a log line, start with Ragas on an exported dataset.

LangSmith and Promptfoo

LangSmith is the evaluation layer around traces and datasets, not a RAG-metric library. Use it when you already trace the chain and want offline experiments plus online sampling.

Promptfoo fits teams that live in YAML eval configs. Their RAG guide is about asserting on retrieved docs and generated answers from a promptfooconfig, not about replacing Ragas metrics.

Skip extra research tooling until the 20 refund rows are stable. Reach for heavier judges later, when you need confidence intervals on a domain shift.

Recommendation for a QA suite: Ragas while you design the cases, DeepEval (or Promptfoo) as the tiny CI gate, TruLens or LangSmith once production traffic needs sampling. Who should skip that stack: a team that cannot log retrieved_contexts yet. The test evidence has to exist before the library can score it.

A worked example: scoring one retrieval pipeline end to end

We'll treat the billing chat as one QA case: "How do we refund a subscription after 14 days?"

Expected result: "Refunds are allowed within 30 days of the charge. After 30 days, file a billing exception."

The live retriever returns two chunks. Rank 1 is the stale 14-day policy. Rank 2 is a billing-portal how-to. The current refunds-v4 id never appears.

This walkthrough follows current Ragas collections metrics, DeepEval assert_test, and Playwright 1.64 CI defaults. The RAG evaluation framework here is the test case, the four scores, and the PR rule.

1. Freeze the dataset row

REFUND_ROW = {
    "user_input": "How do we refund a subscription after 14 days?",
    "reference": (
        "Refunds are allowed within 30 days of the charge. "
        "After 30 days, file a billing exception."
    ),

    "reference_context_ids": ["refunds-v4"],
    "retrieved_context_ids": ["refunds-v3", "billing-portal"],
    "retrieved_contexts": [
        "Refunds are not issued after 14 days from the original charge.",
        "Use the billing portal to download invoices and update a card.",
    ],

    "response": "We cannot refund a subscription after 14 days.",
}

That row is enough to show the bug. Add 19 more before you trust a mean. Cover the happy path (day 5 refund), the exception path, and one question the corpus should refuse.

2. Score with Ragas collections metrics

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import AnswerRelevancy, ContextRecall, Faithfulness
from ragas.embeddings.base import embedding_factory

client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

embeddings = embedding_factory(
    "openai", model="text-embedding-3-small", client=client
)

faithfulness = Faithfulness(llm=llm)
relevancy = AnswerRelevancy(llm=llm, embeddings=embeddings)
recall = ContextRecall(llm=llm)

row = {
    "user_input": "How do we refund a subscription after 14 days?",
    "response": "We cannot refund a subscription after 14 days.",
    "retrieved_contexts": [
        "Refunds are not issued after 14 days from the original charge.",
        "Use the billing portal to download invoices and update a card.",
    ],

    "reference": (
        "Refunds are allowed within 30 days of the charge. "
        "After 30 days, file a billing exception."
    ),
}

async def score_row():
    f = await faithfulness.ascore(
        user_input=row["user_input"],
        response=row["response"],
        retrieved_contexts=row["retrieved_contexts"],
    )

    r = await relevancy.ascore(
        user_input=row["user_input"],
        response=row["response"],
    )

    c = await recall.ascore(
        user_input=row["user_input"],
        retrieved_contexts=row["retrieved_contexts"],
        reference=row["reference"],
    )

    print(
        {
            "faithfulness": f.value,
            "answer_relevancy": r.value,
            "context_recall": c.value,
        }
    )

if __name__ == "__main__":
    import asyncio
    asyncio.run(score_row())

python evals/score_refund_rag.py

Ragas' own simple walkthrough printed {'context_recall': 1.0000, 'faithfulness': 0.8571, 'factual_correctness': 0.7280} on a clean demo. Your refund row will not look like that. Expect high faithfulness, decent relevancy, and weak recall. That is the signal. Recall dropping means the suite found the miss.

If ascore throws because retrieved_contexts is missing, you are not evaluating RAG. You are evaluating a string. Fix the logger in the retrieve function before you tune prompts.

3. Add the PR-safe metric: ID-based context recall

This is the method most top pages skip. If your index stores stable document IDs, you can score retrieval without an LLM judge.

Ragas IDBasedContextRecall is retrieved-id overlap with reference ids: count of reference IDs found in retrieved IDs, divided by the number of reference IDs. Their example with retrieved ["doc_1", "doc_2", "doc_3"] and reference ["doc_1", "doc_4", "doc_5", "doc_6"] scores 0.25. Put that check in the RAG evaluation framework before you add a judge.

import asyncio
from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import IDBasedContextRecall

async def main() -> None:
    sample = SingleTurnSample(
        retrieved_context_ids=["refunds-v3", "billing-portal"],
        reference_context_ids=["refunds-v4"],
    )

    id_recall = IDBasedContextRecall()

    # Official API: single_turn_ascore. This row scores 0.0.
    print(await id_recall.single_turn_ascore(sample))

if __name__ == "__main__":
    asyncio.run(main())

On the refund row the score is 0. The PR should fail. No judge API, no temperature, no extra token bill.

Who should skip ID-based recall: teams whose chunk IDs rotate on every reindex. If refunds-v4 becomes a new UUID at 2 a.m. when the indexer runs, the metric becomes a false red. Persist IDs, then use it.

4. Wrap the same row in DeepEval if you want pytest output

from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

def test_refund_id_recall():
    retrieved = ["refunds-v3", "billing-portal"]
    reference = ["refunds-v4"]
    hits = set(retrieved) & set(reference)
    missing = set(reference) - set(retrieved)

    assert hits == set(reference), (
        f"ID recall was {len(hits) / len(reference)}. "
        f"Missing {sorted(missing)}. Retrieved {retrieved}."
    )

def test_refund_faithfulness_nightly():
    test_case = LLMTestCase(
        input="How do we refund a subscription after 14 days?",
        actual_output="We cannot refund a subscription after 14 days.",
        retrieval_context=[
            "Refunds are not issued after 14 days from the original charge.",
            "Use the billing portal to download invoices and update a card.",
        ],
    )

    # DeepEval docs show threshold=0.7 as an example, not a universal cutoff.
    assert_test(test_case, [FaithfulnessMetric(threshold=0.7)])

deepeval test run tests/evals/test_refund_rag.py

Keep test_refund_id_recall on pull requests. Park test_refund_faithfulness_nightly on a schedule. The judge call is the noisy one.

If the nightly job fails with a score under 0.7 and a reason like "claim is not supported by retrieval context," read the reason. Then open the chunks. Do not bump the threshold to make CI green. The job failed for a reason. Keep the red.

Running RAG evals in CI without blocking every PR on a noisy score

LLM-as-judge metrics are non-deterministic. If QA fails every pull request on faithfulness, you'll block merges when the judge blinks. A RAG evaluation framework still belongs in CI. The trick is which check is allowed to fail the PR.

Split the pipeline the way you already split UI tests from API tests.

On pull request, run:

• ID-based context recall (or exact doc-id inclusion) on a frozen set of 20 to 50 queries
• A cheap schema check: answer is non-empty, citations array is present, retrieved id list is present
• The Playwright smoke for the chat UI (next section)

Nightly, or on main, run:

• Ragas faithfulness, answer relevancy, context precision, context recall on a larger set
• DeepEval assert_test with FaithfulnessMetric if you want pytest output
• A sample of production traces, if you store them

Playwright's own CI docs recommend retries: process.env.CI ? 2 : 0, workers: process.env.CI ? 1 : undefined, and a globalTimeout so a hung job still writes a report. Current Docker tag in those docs is mcr.microsoft.com/playwright:v1.64.0-noble.

Do not copy retries onto the judge. A faithfulness score of 0.4 that passes on retry is not a flake in the product. It is a flake in the evaluator. Log it. Do not hide it.

GitHub Actions shape for the PR gate:

name: RAG eval PR gate

on:
  pull_request:
    branches: [main]

jobs:
  id-recall:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install ragas
      - run: python evals/id_recall.py

actions/checkout@v6 matches the current Playwright GitHub Actions sample. Keep this job short. If the indexer changed IDs, this job should fail, because retrieval is the contract.

For the UI half of the same PR, follow the same Playwright in GitHub Actions pattern you already use for the rest of the suite. Shard the browser specs if they are slow. Do not shard the 20-row ID-recall job. It is not worth a matrix.

Playwright sharding helps wall-clock time on billing-chat.spec.ts across machines. The eval CSV is tiny. Run it on one worker so the metric table stays in one log.

If you parallelize Playwright with fullyParallel: true, keep the eval job separate. Mixing a 30-second judge call into a Chromium shard makes timeouts look like product bugs. You'll waste a night on a Playwright timeout that was the evaluator waiting on an API.

A Playwright automation checklist still applies: isolate data, pin browsers, capture traces on retry. The eval job needs a pinned judge model name in the log more than it needs a trace.zip.

When the PR gate is red on ID recall, the error you want is boring:

AssertionError: ID recall was 0.0. Missing ['refunds-v4']. Retrieved ['refunds-v3', 'billing-portal'].

That's enough to open the indexer. You don't need a 2,000-token judge rationale on the PR.

Where Playwright fits: testing the RAG feature your users click on

Playwright is the UI layer of this QA suite. It does not score faithfulness. It checks the feature a tester clicks: the input, the wait, the answer, the citations, the error state.

That still matters. A perfect Ragas table does not prove the Send button works in Chromium on CI. And a green Playwright HTML reporter does not prove the policy chunk was current. The RAG evaluation framework and the UI spec answer different questions.

Keep one spec for the refund test case. Fail it on contracts you can assert without a judge.

import { test, expect } from '@playwright/test';

test('refund question returns an answer with a citation', async ({ page }) => {
  await page.goto('/support');
  await page.getByRole('textbox', { name: 'Ask a question' }).fill(
    'How do we refund a subscription after 14 days?'
  );

  await page.getByRole('button', { name: 'Send' }).click();
  const answer = page.getByTestId('chat-answer');
  await expect(answer).toBeVisible({ timeout: 30_000 });
  await expect(answer).not.toHaveText(/something went wrong/i);
  const citation = page.getByTestId('citation');
  await expect(citation).toBeVisible();
  await expect(citation).toHaveAttribute('data-doc-id', 'refunds-v4');
});

getByRole is the locator you want for the textbox and the button. Prefer user-facing roles over CSS. The same idea is covered in Playwright locators: roles survive markup churn.

The data-doc-id assertion is the UI twin of ID-based recall. If the bubble cites refunds-v3, the spec fails even when the text looks polite. That is the check toBeVisible() never did.

Streaming answers need a longer expect timeout, not page.waitForTimeout(15000). Web-first assertions retry until the timeout. A hard wait hides a hung stream and still flakes when the model is slower than 15 seconds. If the job dies with TimeoutError: locator resolved to hidden, open the Playwright trace viewer and look at the network call to /api/chat.

Config for this spec, checked against Playwright 1.64:

import { defineConfig } from '@playwright/test';

const isCI = !!process.env.CI;
export default defineConfig({
  timeout: isCI ? 60_000 : 30_000,
  expect: { timeout: isCI ? 10_000 : 5_000 },
  retries: isCI ? 2 : 0,
  workers: isCI ? 1 : undefined,
  globalTimeout: 60 * 60 * 1000,

  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'on-first-retry',
  },

  reporter: [
    [isCI ? 'dot' : 'list'],
    ['html', { open: 'never' }],
  ],
});

Playwright reports a test as flaky when it fails, then passes on retry. For a RAG UI, that often means the stream finished on attempt 2. Treat it as a timeout budget problem, not as a reason to add another retry.

Playwright reporting still belongs here: HTML for humans, a compact console reporter on CI, traces on first retry. Debug Playwright tests with the trace before you blame the model. If the /api/chat payload already contains refunds-v3, the UI did its job and the RAG evaluation framework should have caught the ID miss.

A local HTML report dies with the runner. @testdino/playwright streams Playwright results while the run is going, including sharded jobs, so you can open the failed billing-chat.spec.ts with its trace, screenshot, and video without waiting on an artifact zip. See TestDino docs for the reporter line.

If the spec fails then passes, that is the fail-then-pass pattern filed under flaky tests, along with tests that flip across runs. Use it to see whether /support is unstable, not to score the LLM.

Conclusion

A RAG evaluation framework is how QA stops a confident 14-day refund answer from shipping against a 30-day policy.

Keep retrieval and generation as separate test oracles. Read faithfulness as "grounded in chunks," not "true." Fail pull requests on document IDs when you have them. Run Ragas or DeepEval judges on a schedule. Keep Playwright on the Send button, the citation data-doc-id, and the error state.

Take the refund case in evals/refund_dataset.py, run python evals/score_refund_rag.py, and look at recall before you rewrite the prompt. If ID recall is 0, file an index defect. If the UI cites refunds-v3, file a payload defect. If both are right and faithfulness is low, then open the generator.

FAQ