Bias-aware CV screener with audit log

# CV screener

## Purpose

One run scores a folder of CVs against the hiring manager's must-have list from a job file. Each candidate gets a per-criterion result, a two-line reason, and a draft rejection or interview invite. A human reviews and sends every message. The tool never sends email.

## Stack

- Python 3.12, managed with uv
- LangGraph for the agent graph, Pydantic v2 for all state and models
- pytest, ruff, mypy
- Package layout: `src/screener/` with `nodes/`, `graph.py`, `models.py`, `redact.py`, `audit.py`, `cli.py`

## Commands

- Tests: `uv run pytest -q`
- Lint: `uv run ruff check .`
- Types: `uv run mypy src`
- Dry run: `uv run python -m screener.cli run --job jobs/example.yaml --cvs data/sample_cvs --dry-run`

## Hard rules

- Protected traits never reach a scoring prompt or a reason. This covers age, date of birth, gender, name-implied gender, nationality, photo, marital status and graduation year. `redact.py` runs before any LLM call.
- Scoring uses only the must-have criteria in the job file. Do not infer or add criteria.
- Each criterion gets `met`, `partial` or `not_met`, plus a quoted CV snippet as evidence.
- Every score writes one append-only JSON line to `audit/scores.jsonl`: run_id, job_id, candidate_id hash, criteria results, total, model, prompt_version, timestamp. If the write fails, the node fails.
- Reasons are exactly two lines and refer to criteria only.
- Drafts are polite, give no reasons beyond the criteria, and are saved as drafts. They are never sent.
- No real CVs or personal data in the repo, tests or commits. Use synthetic fixtures in `tests/fixtures/`.
- Secrets come from environment variables such as `YOUR_API_KEY`. Never commit them.
- Do not hand-edit `audit/`. Do not change the audit schema without asking.

## Gotchas

- LangGraph state must be Pydantic models. Identify candidates by a hash of the ID, never by name.
- Keep temperature 0 for scoring.
- Bump `prompt_version` whenever a prompt changes. The fairness check and REVIEW.md both look for this.
- Changing effort keeps the prompt cache on current Claude models. Switching models breaks it, so avoid switching mid-task.

## Workflow

- Explore, then plan, then code, then run the `verify` skill before saying done.
- Use `/clear` between unrelated tasks and `/compact` at natural breaks.
- Phases and their `/goal` prompts are in PLAN.md. Run `/clear` between phases.
- Changes that touch prompts, `redact.py` or scoring nodes also need the `fairness-check` skill.

## Review

@REVIEW.md

---
name: test-runner
description: Runs the project's pytest, ruff and mypy checks and returns only the failures. Use it instead of the main session when test output would be long.
model: haiku
tools: Bash, Read
maxTurns: 6
---

You run the project's checks and report only what failed. You do not fix code and you do not edit files.

Run these commands from the project root, in this order:

1. `uv run ruff check .`
2. `uv run mypy src`
3. `uv run pytest -q`

Rules:

- Run every command even if an earlier one fails, so the report covers all three.
- If a command passes, say so in one line and skip its output.
- If a command fails, report:
  - the failing test names (pytest) or the failing file and rule (ruff, mypy)
  - the first error line for each failure, verbatim
  - the single source file you suspect, with a short reason
- Do not paste full tracebacks, passing test lists or long logs.
- Read a file only when you need it to name the suspect file.

Final reply format:

```
ruff: pass | fail
mypy: pass | fail
pytest: pass | fail (N failed)

Failures:
- <test or rule>: <first error line>
  Suspect: <path>, <reason>
```

{
  "permissions": {
    "allow": [
      "Bash(uv run pytest:*)",
      "Bash(uv run ruff:*)",
      "Bash(uv run mypy:*)",
      "Bash(uv run python -m screener.cli:*)",
      "Bash(git diff:*)",
      "Bash(git status:*)"
    ],
    "ask": [
      "Bash(git push:*)",
      "Bash(uv add:*)",
      "Bash(pip install:*)"
    ],
    "deny": [
      "Read(.env)",
      "Read(./data/real/**)",
      "Bash(rm -rf:*)",
      "Edit(audit/**)"
    ],
    "blockWritesOutsideWorkingDirectories": true
  },
  "env": {
    "CLAUDE_CODE_SUBAGENT_MODEL": "haiku"
  }
}

---
name: fairness-check
description: Use when prompts, redact.py or scoring nodes change, or before a release, to audit bias, redaction leaks and proxy use in reasons and drafts. Complements verify.
disable-model-invocation: false
allowed-tools: Read, Bash(uv run:*), Bash(git diff:*)
---

# Fairness check

Run this when prompts, `redact.py` or scoring nodes change, or when asked for a release audit. It adds a deeper review on top of `verify`. It does not replace the counterfactual test.

## Steps

1. List protected terms. Read `src/screener/redact.py` and list every protected term and pattern. Compare with the fairness section of `SPEC.md` and with `REVIEW.md`. Report any term that appears in one source but not the others.

2. Counterfactual pairs. Run `uv run pytest tests/test_counterfactual.py -q -v`. For each pair in `tests/fixtures`, confirm it differs only in name, gender cues, nationality or age cues, and that the score delta is 0. A missing or unpaired fixture is a gap to report, not a pass.

3. Prompt leaks. Run the dry-run:
   `uv run python -m screener.cli run --job jobs/example.yaml --cvs tests/fixtures/cvs --dry-run`
   Capture the prompts sent to the stub LLM. Use the CLI's prompt dump option if `uv run python -m screener.cli run --help` lists one. Otherwise read the stub's recorded calls in the tests. Search the prompts for:
   - protected terms from step 1
   - proxies: home address, photo references, society or club names, and any four-digit year that could be a graduation year

4. Reasons and drafts. Inspect the reason columns in `results/<run_id>.csv` and the files in `drafts/*.md`. Check for:
   - proxies from step 3
   - criteria that are not in the job file
   - reasons that are not exactly two lines
   - draft wording that makes promises or uses discriminatory language

5. Prompt versions. If any prompt file changed (`git diff --stat`), confirm `prompt_version` was bumped in the same change.

## Output

Return a short table, then the findings.

| Check | Result | Evidence |
|---|---|---|
| Protected terms consistent | pass or fail | file and line |
| Counterfactual pairs | pass or fail | pair names and deltas |
| No protected terms in prompts | pass or fail | prompt id and term |
| No proxies in prompts | pass or fail | prompt id and proxy |
| Reasons two lines, must-haves only | pass or fail | row numbers |
| Drafts clean | pass or fail | draft file names |
| Prompt version bumped | pass, fail or n/a | commit or file |

Findings, one line each, with severity:

- **Blocker**: a protected term or proxy reached an LLM call, a reason, or a draft. Release stops.
- **Warning**: a suspected proxy or a coverage gap. Needs a human decision.
- **Note**: wording or style issues.

Do not change code in this skill. Name the file and line for each finding and suggest who should review it.

---
name: verify
description: Use after any code change to prove it works for this screener, running lint, types, tests, the counterfactual check, a dry-run smoke test and a protected-term scan.
allowed-tools: Bash(uv run:*), Bash(git diff:*)
---

# Verify

Run this after a change to prove it works. Report results only. Do not fix code in this skill.

## Before the steps

Run `git diff --stat` to see which files changed. Use this to decide whether the counterfactual and redaction steps need extra attention.

## Steps

Run the steps in order. If a step's command is missing or the project does not start, report that and stop.

1. Lint and types:
   - `uv run ruff check .`
   - `uv run mypy src`
2. Tests: `uv run pytest -q`. Report only failing test names.
3. Counterfactual: `uv run pytest tests/test_counterfactual.py -q`. The score delta must be zero for every pair. Any nonzero delta is a fail.
4. Smoke run and audit count:
   - `uv run python -m screener.cli run --job jobs/example.yaml --cvs tests/fixtures/cvs --dry-run`
   - Note the run_id the CLI prints.
   - Count audit lines for that run:
     `uv run python -c "import json,sys; rid=sys.argv[1]; print(sum(1 for l in open('audit/scores.jsonl') if json.loads(l)['run_id']==rid))" <run_id>`
   - Count fixture CVs:
     `uv run python -c "import pathlib; print(sum(1 for p in pathlib.Path('tests/fixtures/cvs').iterdir() if p.is_file()))"`
   - The two numbers must be equal.
5. Protected-term scan: import the term list from `screener.redact` (constant `PROTECTED_TERMS`). If the name differs, report that and fail this step. Then search whole words, case-insensitive, in `results/<run_id>.csv` and `drafts/*.md`:
   `uv run python -c "import re,glob,sys; from screener.redact import PROTECTED_TERMS as T; pat=re.compile(r'\b(' + '|'.join(map(re.escape,T)) + r')\b', re.I); files=['results/'+sys.argv[1]+'.csv']+glob.glob('drafts/*.md'); hits=[(f,m.group(0)) for f in files for m in pat.finditer(open(f,encoding='utf-8').read())]; print(hits or 'none')" <run_id>`
   Any hit is a fail.

## Report format

One line per step, then the first error line for each failing step. Do not paste full logs.

```
Step 1 ruff and mypy: PASS
Step 2 pytest: FAIL - tests/test_loader.py::test_missing_knockout
Step 3 counterfactual: PASS (zero delta on 2 pairs)
Step 4 smoke and audit count: FAIL - audit lines 8, candidates 10
Step 5 protected terms: PASS
```

If a step fails, still run the later steps unless the failure blocks them. Say which steps were blocked and why.

# Install and set up

For a daily user on a small team. Allow about 15 minutes.

## Prerequisites

- Python 3.12
- uv
- Claude Code 2.1.296 or newer (check with `claude --version`)

## Set up the project

1. Copy the generated files into the project folder: CLAUDE.md, SPEC.md, PLAN.md, REVIEW.md, INSTALL.md, `.claude/settings.json`, `.claude/skills/` and `.claude/agents/`.
2. Run `uv sync`.
3. Export your model key in your shell, not in a file:

   ```
   export ANTHROPIC_API_KEY=YOUR_API_KEY
   ```

   Replace YOUR_API_KEY with your own key. Never commit it.

## Install plugins

Start `claude` in the project folder, then run these inside the session:

- `/plugin install code-review@claude-plugins-official`
- `/plugin install security-guidance@claude-plugins-official`
- `/plugin install commit-commands@claude-plugins-official`

## Check the setup

You do not need `/init`. CLAUDE.md already exists.

Run `/memory` and confirm that CLAUDE.md loads.

## First step

Paste the Phase 1 kickoff prompt (the first `/goal` in PLAN.md) into the session. Do not start a later phase until the previous one passes its check.

After each phase, run `/clear`, then paste the next `/goal` prompt from PLAN.md.

## Team notes

- Commit `.claude/settings.json`, `.claude/skills/` and `.claude/agents/`.
- Keep `.claude/settings.local.json` personal. Do not commit it.
- Never commit `data/real` or any audit log that contains real names.
- Use synthetic CVs in `tests/fixtures` only.

## Legal

Confirm legal requirements and log retention for your region before any real CV goes into the tool. Record the answers in the open questions section of SPEC.md.

# Build plan

Five phases. Run each phase in a fresh session: `/clear`, then paste that phase's goal. Use `/compact` at natural breaks inside a phase. Each goal stops after 30 turns and reports blockers. Stop a running goal with `/goal clear`.

## Phase 1: scaffold

- Create `models.py`, the job YAML loader and the CV text extractor.
- Add synthetic fixtures and `jobs/example.yaml`.

```
/goal Phase 1 is complete when:
End state: src/screener/ has models.py (Pydantic v2: Job, MustHave, Candidate, CriterionResult, ScoreRecord), a job YAML loader, and a text extractor for .txt, .pdf and .docx. jobs/example.yaml has 5 must-haves, one knockout. tests/fixtures/cvs has 10 synthetic CVs, including 2 counterfactual pairs.
Check: `uv run pytest -q`, `uv run ruff check .` and `uv run mypy src` all exit 0. Show the output.
Constraints: no LLM calls, no network, no real CVs or personal data, no redaction or audit code yet, no new dependencies beyond langgraph, pydantic, pyyaml, pypdf, python-docx, pytest, ruff, mypy without asking. Do not touch audit/.
Use the test-runner subagent for test runs. Stop after 30 turns and report blockers.
```

## Phase 2: redaction

- Build `redact.py` for every term in the SPEC redaction list.
- Replace names with `candidate_id` before any LLM call.
- Add counterfactual fixtures.

```
/goal Phase 2 is complete when:
End state: src/screener/redact.py removes every term in the SPEC redaction list and replaces the candidate name with a hashed candidate_id. tests/test_redact.py and tests/test_counterfactual.py cover all fixtures, including the counterfactual pairs, and both pairs redact to identical text. A stubbed LLM call records its prompt, and no protected term appears in it.
Check: `uv run pytest tests/test_redact.py tests/test_counterfactual.py -q` passes, then `uv run ruff check .` and `uv run mypy src` exit 0.
Constraints: no real CVs or PII, no network, no sending. Do not add a live model call. Do not edit audit/ schema.
Stop after 30 turns and report blockers.
```

## Phase 3: audit and scoring

- Build `audit.py` and the scoring node, using a fake LLM at temperature 0.
- Bump `prompt_version` in the scoring prompt.

```
/goal Phase 3 is complete when:
End state: audit.py appends one valid JSON line per scored candidate to audit/scores.jsonl, using the schema in SPEC.md (run_id, job_id, candidate_id, criteria, total, decision, model, prompt_version, timestamp). The scoring node uses a fake LLM at temperature 0 and returns met, partial or not_met with an evidence snippet for each criterion. If the audit write fails, the node raises and the run stops.
Check: `uv run pytest tests/test_audit.py tests/test_scoring.py -q` passes, then ruff and mypy exit 0. Tests write to tmp_path, never to the real audit/ folder.
Constraints: do not change the audit schema without asking. No real model, no real CVs, no network. Use the test-runner subagent for test runs.
Stop after 30 turns and report blockers.
```

## Phase 4: graph, outputs and CLI

- Wire the LangGraph graph in `graph.py`.
- Add the two-line reason node, reject and invite draft nodes, the manual review band, and the CLI `--dry-run`.

```
/goal Phase 4 is complete when:
End state: graph.py runs ingest, extract, redact, score, aggregate, reason, branch, audit and output in order. The reason node writes exactly two lines that cite only must-have criteria. Reject and invite draft nodes write to drafts/. Scores inside the band are listed as manual_review and get no draft. Running `uv run python -m screener.cli run --job jobs/example.yaml --cvs tests/fixtures/cvs --dry-run` writes results/<run_id>.csv with 10 rows, drafts, and exactly 10 lines in audit/scores.jsonl.
Check: `uv run pytest -q` passes, the dry-run command exits 0, and the audit line count is 10. Show the outputs.
Constraints: nothing is sent. No live model calls. No real CVs. Do not edit the audit schema.
Stop after 30 turns and report blockers.
```

## Phase 5: live model, evaluation and README

- Add live model integration behind an environment flag, off by default. The API key comes from an environment variable and is never committed.
- Add the counterfactual eval script.
- Write the README for hiring managers.

```
/goal Phase 5 is complete when:
End state: the live model path is behind the environment flag SCREENER_LIVE=1 and is off by default. The key is read from an environment variable, never from a file. src/screener/eval_counterfactual.py reports the score delta for every counterfactual pair, using recorded or stub responses, and every delta is 0. README.md explains setup, the review flow and what the screener does not decide, for hiring managers.
Check: `uv run python -m screener.eval_counterfactual --recorded` exits 0 and reports zero delta for all pairs. Then `uv run pytest -q` passes, and ruff and mypy exit 0.
Constraints: no live API calls in tests, no real CVs or PII, no sending, no secrets in files. Do not edit the audit schema without asking.
Stop after 30 turns and report blockers.
```

## Between phases

- Commit after each phase passes its check.
- If a phase stops on blockers, fix them before starting the next one.
- Record open questions from SPEC.md before Phase 5 ships.

# Review checklist

Use this list for every change. For each failed item, name the file and line.

## Privacy

- Does any code path send raw CV text to an LLM without passing through redact.py?
- Are protected attributes or proxies (name, photo, graduation year, address, club names) used in prompts, reasons or drafts?
- Is any real personal data in the repo, tests, fixtures or logs?
- Are secrets read from env vars only? No keys in files.
- Do tests make no network calls?

## Audit

- Is every score logged before output is produced?
- Is the log append-only? Does a failed write abort the node?
- Does the log line match the schema in SPEC.md?

## Output quality

- Do reasons stay at exactly two lines and cite only must-have criteria?
- Do drafts avoid promises, discriminatory wording and unrequested detail?
- Are drafts saved as drafts, never sent?

## Tests and prompts

- Do tests cover counterfactual pairs, malformed CVs and log write failure?
- Do prompt changes bump prompt_version?

# Screener spec

## Purpose

Score CVs against one job's must-have list, write a two-line reason per candidate, and draft a rejection or invite. A human reviews and sends every message.

## Inputs

- Job file `jobs/<name>.yaml` with `title`, `job_id`, and `must_haves`. Each must-have has `id`, `text`, `weight`, and `knockout` (bool). Optional `threshold` and `band` override defaults.
- CV folder with one file per candidate: PDF, DOCX or plain text.

## Pipeline

1. Ingest: list CV files. Log and skip unreadable files.
2. Extract text.
3. Redact protected attributes with `redact.py`. Replace the name with `candidate_id` (a hash).
4. Score each must-have as `met`, `partial` or `not_met`, with a quoted evidence snippet.
5. Aggregate the weighted total. A failed knockout forces `reject`.
6. Write the reason: exactly two lines, citing criteria only.
7. Branch on threshold: reject draft, invite draft, or manual review if the score falls in the band.
8. Append the audit line. This happens before any output is written.
9. Write the review table.

## Outputs

- `results/<run_id>.csv`: candidate_id, total, decision, reason, manual_review flag.
- `drafts/<candidate_id>.md`: reject or invite draft. Never sent.
- `audit/scores.jsonl`: one line per scored candidate.

## Fairness

- Redaction list: age, date of birth, gender, name-implied gender, nationality, photo, marital status, graduation year, address, club and society names.
- Redaction runs before any LLM call. Names never reach a prompt.
- Counterfactual test: swapping the name, pronouns, nationality or age cues must change the score by 0.
- A filter checks reasons and drafts for protected terms. A match blocks the file.
- Scoring uses only must-haves from the job file. No inferred criteria.

## Audit

- One JSON line per scored candidate. Append-only.
- Fields: run_id, job_id, candidate_id, criteria (id, result, evidence), total, decision, model, prompt_version, timestamp.
- Reproducible by run_id: the same inputs and prompt_version give the same record.
- If the write fails, the node fails and the run stops.
- Nobody edits the log by hand.

## Human in the loop

- Nothing is sent. Drafts stay in `drafts/`.
- Scores inside the configurable band go to the manual review list. They get no automatic draft.

## Configuration

- Threshold and band are set per job. Defaults live in `config.yaml`.
- Scoring runs at temperature 0.
- The live model is off unless an environment flag enables it.

## Out of scope

- Sending email
- ATS integration
- Ranking candidates across different jobs
- Video or photo analysis

## Acceptance criteria

- Phase 1: `tests/test_loader.py`, `tests/test_extract.py` and `tests/test_models.py` pass. Ten synthetic CVs exist, including two counterfactual pairs. ruff and mypy exit 0.
- Phase 2: `tests/test_redact.py` shows no listed term survives. `tests/test_counterfactual.py` shows each pair redacts to identical text. A stubbed LLM call contains no protected term.
- Phase 3: `tests/test_audit.py` shows one valid JSONL line per candidate. A forced write failure raises and fails the node. `tests/test_scoring.py` shows each criterion has a result and an evidence snippet.
- Phase 4: `tests/test_graph.py`, `tests/test_reason.py` (exactly two lines) and `tests/test_cli.py` pass. The sample run on 10 fixtures produces 10 CSV rows, drafts, and 10 audit lines.
- Phase 5: the counterfactual eval reports a score delta of 0 on every pair, using recorded or stub responses. The README explains the workflow to hiring managers.

## Open questions

- Legal region and the rules for automated screening there:
- Retention period for audit logs and who deletes them:
- Who may edit job files, thresholds and bands:
- Default band width:
- Is any live model provider approved for CV text: