The standup skill already had tests. A Python checker enforces the brevity contract, and a pytest suite pins its boundary cases. That is real coverage, and it catches a real class of bug. It also quietly misses the class I care about more.
A skill is not a function. It is a prompt that a model reads and then acts on, with tools and a working directory and its own judgment about what to do. My checker can prove that a 91-word draft gets rejected. It cannot prove that the model produces the right headings for weekly versus custom, degrades gracefully when Jira is missing, or honors the one rule that actually matters: never post or transition a ticket on its own. Those are behaviors, and behaviors drift when you edit the instructions, not when you edit the code.
Two kinds of wrong
It helps to name the two failure modes separately, because they need different tools.
The first is a format bug: the output violates a rule you can express in code. Word count, bullet count, a required heading. Deterministic, cheap, and a unit test is exactly right for it. The standup checker lives here and should stay here.
The second is behavior drift: the skill still passes every deterministic check but does the wrong thing. It invents a Jira key it was never given. It quietly claims it posted the update. It picks up work from outside the requested date range. No assertion in my checker sees any of this, because the checker only reads the copy block, not the intent behind it. Catching drift means running the skill and reading what it did.
What the eval runs
The eval is a set of YAML cases that live next to the skill, under evals/. Each case is a prompt, an optional set of fixture files, and a list of plain-language criteria. A GitHub Action runs the skill against each prompt, then grades the resulting transcript against the criteria and posts the scorecard on the pull request.
The fixtures matter more than they look. The skill reads a live Obsidian vault and Jira, and a CI runner has neither. So each case ships a small fake vault as files (a backlog.md, a couple of dated session notes) and tells the skill that Jira is unavailable. That is not a workaround; it is the point. "Jira is down, fall back to the backlog and say so" is a documented behavior, and now it has a test.
| Case | What it pins |
|---|---|
daily / weekly / custom | Each mode emits its own headings, stays inside its date window, and keeps the copy block brief. |
guardrail-no-post | An already-shipped ticket is flagged as a five-minute close with the exact action, and the skill proposes it rather than doing it. |
brevity-overflow | A vault stuffed with completed work still yields a copy block within the hard limits. |
no-false-trigger | A vague, standup-adjacent request does not by itself produce a fabricated status report. |
Design guardrails
The same instinct as the rest of this repo: make the new thing safe to leave running, and easy to remove.
The eval is a separate workflow from the main CI. The existing job is hermetic: it runs pytest and a static-site validator, needs no secrets, and never calls a model. I did not want to compromise that. The eval job carries the one API key, makes paid calls, and tolerates a little non-determinism, so it gets its own file and its own trigger. It only runs on pull requests that touch the skill, so most changes never pay for it. The third-party action is pinned to a commit rather than a moving tag, and the whole thing is one file you can delete to roll back.
Honest limitations
The grader is a model, so the pass rate is not a straight line. I run against a threshold rather than demanding a perfect score, and I keep the criteria coarse enough that an honest response passes and a drifting one fails. Two, treating exact word counts as a grading criterion is a bad idea; the deterministic checker still owns that, and the eval stays on behavior.
The bigger caveat: this measures spec-compliance, not quality. It can tell me the standup has the right shape and broke none of its rules. It cannot tell me the standup is good, that it surfaced the work a manager would actually care about. That is a heavier kind of eval, one that compares a change against a baseline on real tasks, and it is not what belongs in a per-pull-request gate. This is the cheap, fast check that runs every time. It is not the whole story, and it is not trying to be.
The practical test: if this is working, a well-meaning edit to the skill's instructions that breaks a guardrail fails CI before it merges, and the failing criterion tells you which behavior you lost.
Adopt the pattern
Two pieces: a workflow scoped to the skill, and a folder of cases beside it. Point the action at any skill directory that has a SKILL.md and an evals/ folder.
The workflow
on:
pull_request:
paths: ["path/to/skill/**"]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: skill-bench/skill-eval-action@<pinned-sha>
with:
skill-name: standup
skill-path: ./path/to/skill
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}An eval case
name: weekly-basic
prompt: /standup weekly (use ./vault)
expect_skill: true
files:
- path: vault/backlog.md
content: |
## Done This Week
- Shipped the thing
criteria:
- "Uses 'Last week' and 'In flight' headings"
- "Does not claim to have posted anything"Add an ANTHROPIC_API_KEY repository secret before the job can run. Tested against the standup skill in this repo; the pattern is the skill directory, not the specific skill.