Gate an agent in CI

A runnable agent gate: a stand-in agent, a hidden checker and a pass/fail gate that exits 3, plus a GitHub Actions workflow.

Source: examples/agent-gate. Read the guide for the concepts behind it.

github-actions.yml

# Gate every pull request on an agent run, with no server: the job installs
# Evals.si, runs the spec in a sandbox and fails when a gate fails (exit 3).
# Copy this to .github/workflows/agent-gate.yml and point `-f` at your spec.
name: agent-gate
on:
  pull_request:
    paths: ['agent/**', 'prompts/**', 'evals/**']

permissions:
  contents: read

concurrency:
  group: agent-gate-${{ github.ref }}
  cancel-in-progress: true

jobs:
  gate:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: '3.12'}
      - name: Sandbox prerequisites
        # Ubuntu 24.04 restricts unprivileged user namespaces through AppArmor,
        # which the sandbox needs, unless this is relaxed.
        run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
      - run: pip install evalsi evalsi-harness

      # Exit code 3 when a gate fails, which fails the job.
      - name: Run the agent gate
        # A real agent calls a model: pass its key here, and name it in the
        # spec's cli.env_from so only the agent sees it.
        # env: {AGENT_MODEL_KEY: '${{ secrets.AGENT_MODEL_KEY }}'}
        run: evalsi run -f evals/agent-gate.yaml --output results.json

      - name: Build the report
        if: always()
        run: evalsi report results.json -o report.md

      - name: Show the report in the job summary
        if: always()
        run: cat report.md >> "$GITHUB_STEP_SUMMARY"

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: agent-gate-report
          path: |
            results.json
            report.md

run.yaml

# A pull-request gate for an agent: it must fix every task, or the run exits 3.
# Runs offline and free, with a stand-in agent, so you can see the gate work:
#   pip install evalsi evalsi-harness
#   evalsi run -f run.yaml          # exit 0: the gate passes
# The guide is docs/guides/gate-an-agent-in-ci.md (https://evalsi.dev/docs/guides/gate-an-agent-in-ci/).
apiVersion: evals.si/v1alpha1
kind: EvalRun
metadata: {name: calc-agent-gate}
spec:
  # Your agent goes here. Any command works: it runs inside the sandbox, in the
  # task's working directory, and gets the instruction on stdin.
  target:
    agent:
      cli:
        command: [python3, agent.py]
  dataset: {path: tasks.jsonl}
  harness: {}
  environment:
    files:
      # A stand-in agent so this runs offline, free and the same every time.
      # Replace it with your own agent's command.
      agent.py: |
        import pathlib

        pathlib.Path("calc.py").write_text("""\
        def add(a, b):
            return a + b


        def mul(a, b):
            return a * b
        """)
      # The task's starting state: two bugs.
      calc.py: |
        def add(a, b):
            return a - b


        def mul(a, b):
            return a + b
    setup:
      - git init -q && git add -A && git -c user.email=e@x -c user.name=e commit -qm base
    sandbox: {min_isolation: namespaced, network: deny}
    checker:
      # Written after the agent finishes, so the agent never sees it.
      files:
        test_hidden.py: |
          from calc import add, mul
          assert add(2, 3) == 5 and mul(2, 3) == 6
      command: [python3, test_hidden.py]
  evaluators:
    - ref: task-success
  trials: 1
  gates:
    - {metric: task-success, min: 1.0}

tasks.jsonl

{"id": "add", "input": "calc.py has a bug: add(2, 3) returns -1. Fix it."}
{"id": "mul", "input": "calc.py has a bug: mul(2, 3) returns 5. Fix it."}