Evaluate a RAG pipeline

A gate over a RAG pipeline’s recorded answers: question, answer, retrieved passages and a citation check, with no model needed.

Source: examples/rag. Read the guide for the concepts behind it.

gate.yaml

# A gate over a RAG pipeline's recorded answers: each record carries the
# question (input), the answer (output), the retrieved passages (context) and
# the expected answer (reference). No model, no server and no judge needed:
#   pip install evalsi
#   evalsi run -f gate.yaml        # exit 3: the second answer cites [4], and there are two passages
# Add judge-based metrics (faithfulness, answer-relevance, context-precision,
# context-recall) by listing them below and passing --judge-* options.
apiVersion: evals.si/v1alpha1
kind: EvalRun
metadata: {name: rag-gate}
spec:
  dataset: {path: rag.jsonl}
  evaluators:
    - ref: citation-accuracy
  gates:
    - {metric: citation-accuracy, min: 0.9}

rag.jsonl

{"id": "refund-window", "input": "How long is the refund window?", "output": "Refunds are accepted for 30 days [1].", "context": ["Our policy allows refunds within 30 days of purchase.", "Shipping takes 3-5 business days."], "reference": "30 days"}
{"id": "shipping-time", "input": "How long does shipping take?", "output": "Shipping takes 3-5 business days [4].", "context": ["Our policy allows refunds within 30 days of purchase.", "Shipping takes 3-5 business days."], "reference": "3-5 business days"}