Durable webhook relay with retries and replay

# Webhook relay

## Purpose

This is a hosted webhook relay. It receives provider webhooks (Stripe first), stores them durably, delivers them to customer endpoints with retries, and supports replay. The selling point is that we never lose an event. When durability, idempotency, and speed conflict, durability and idempotency win.

## Stack

- Python 3.12, FastAPI, SQLAlchemy, Alembic, psycopg
- Postgres: durable source of truth (event store)
- Redis: queue, scheduling, rate limits, dedupe cache only
- pytest, ruff, mypy
- Ask before adding any other dependency.

## Commands

- Install: `python -m venv .venv`, activate it, then `pip install -e ".[dev]"`
- Local services: `docker compose up -d`
- Test services: `docker compose -f docker-compose.test.yml up -d`
- Migrations (local): `alembic upgrade head`
- Dev server: `uvicorn app.main:app --reload`
- Worker: `python -m app.workers.run`
- Tests: `pytest -q`
- Lint: `ruff check . && ruff format --check .`
- Type check: `mypy app cli`
- Full verification: run the `/verify` skill

## Durability rules

- Ingest writes the event to Postgres and commits BEFORE returning 2xx.
- Redis is never the only copy of an event.
- If Redis loses data, the recovery job re-enqueues events from Postgres.
- If enqueue fails after commit, mark the event `pending_enqueue` for recovery.
- Delivery is at-least-once.
- Every delivery carries a stable idempotency header derived from the event id.
- Retries use exponential backoff with jitter, a max attempt count, and per-endpoint circuit state.
- After max attempts, the event goes to dead-letter.

## Idempotency rules

- Dedupe on provider event id plus source id with a unique DB constraint.
- A duplicate provider delivery returns 200 with the original event id and creates no row.
- A replay creates a new delivery attempt, never a new event.

## Signatures

- Verify signatures over raw request body bytes with constant-time compare.
- Never parse JSON before verifying.
- Stripe first. Other providers are added as adapters (see the `webhook-provider` skill).

## Retention and billing

- Retention: 30 days on paid plans, 3 days on free.
- A scheduled job deletes expired events. Deletion must be tested and logged.
- Metering counts each accepted event once.
- All time is UTC.

## Layout

- `app/api/`: HTTP routes (ingest, sources, endpoints, replay)
- `app/workers/`: delivery worker, recovery job
- `app/providers/`: signature adapters
- `app/billing/`: usage metering and plan limits
- `app/retention/`: deletion job
- `app/models/`: SQLAlchemy models
- `cli/`: localhost forwarding CLI
- `tests/`: pytest suite
- `migrations/`: Alembic revisions (forward-only)

## Gotchas

- Never log payload bodies or secrets.
- Use placeholders like `YOUR_API_KEY` in docs and examples.
- Never commit `.env`.
- Migrations are forward-only. Never edit an applied migration.
- Tests must not call real provider APIs. Use fixtures.
- Outbound calls to customer URLs need timeouts and SSRF protection (see REVIEW.md).

## Workflow

- Explore, plan, code, then run `/verify`.
- Use `/clear` between unrelated tasks.
- Phases are in PLAN.md. Open product questions are in SPEC.md.

## Do not

- Do not use in-memory queues.
- Do not use fire-and-forget background tasks for ingest.
- Do not return 200 and then save.

@REVIEW.md

---
name: test-runner
description: Runs pytest, ruff and mypy on request and returns only the failures. Use it to keep long test output out of the main context.
model: haiku
tools: Bash, Read
maxTurns: 10
---

You run the project's checks and report only what failed.

Run only the commands the caller asks for. Use these exact commands:

- Tests: `pytest -q`
- Lint: `ruff check .` and `ruff format --check .`
- Types: `mypy app cli`

Rules:

- Never edit, create or delete files. You have Read and Bash only; use them to inspect, not to change anything.
- Never run `docker compose` or start or stop services. If the test services are down, report that and stop.
- Never print secrets or environment values. If output contains one, replace it with `[REDACTED]`.

Report format, under 40 lines:

1. One line per check: `pytest: PASS` or `pytest: FAIL (N failed)`, and the same for ruff and mypy.
2. For each failure: the test name or file:line, the key assertion or error line, and the likely file to fix.
3. Nothing else. No full logs, no passing test names, no suggestions beyond the likely file.

{
  "env": {
    "CLAUDE_CODE_SUBAGENT_MODEL": "haiku"
  },
  "permissions": {
    "defaultMode": "default",
    "blockWritesOutsideWorkingDirectories": true,
    "allow": [
      "Bash(pytest:*)",
      "Bash(python -m pytest:*)",
      "Bash(ruff check:*)",
      "Bash(ruff format:*)",
      "Bash(mypy:*)",
      "Bash(alembic upgrade head:*)",
      "Bash(alembic current:*)",
      "Bash(alembic history:*)",
      "Bash(uvicorn app.main:app --reload:*)",
      "Bash(docker compose up -d:*)",
      "Bash(docker compose ps:*)",
      "Bash(git status:*)",
      "Bash(git diff:*)",
      "Bash(git log:*)"
    ],
    "ask": [
      "Bash(git push:*)",
      "Bash(alembic downgrade:*)",
      "Bash(dropdb:*)",
      "Bash(docker compose down -v:*)",
      "Bash(curl:*)",
      "Bash(wget:*)",
      "Bash(pip install:*)",
      "Bash(uv add:*)",
      "Bash(uv pip install:*)"
    ],
    "deny": [
      "Bash(rm -rf:*)",
      "Read(./.env)",
      "Read(./.env.local)",
      "Read(./secrets/**)"
    ]
  },
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": "jq -r '.tool_input.file_path // empty' | grep -E '\\.py$' | xargs -r ruff format -q"
          }
        ]
      }
    ]
  }
}

---
name: migration
description: Safe, forward-only schema change workflow for the Postgres event store, covering review of autogenerated SQL, locks, backfills and idempotency constraints.
when_to_use: Use when changing SQLAlchemy models, adding tables or columns or indexes, or backfilling data in the events store.
---

# Migration

## Steps

1. Edit the model in `app/models/`.
2. Generate the revision:
   ```
   alembic revision --autogenerate -m "<short summary>"
   ```
3. Open the new file in `migrations/versions/` and read every operation. Autogenerate misses renames and some constraint changes, so fix those by hand. Review the SQL:
   ```
   alembic upgrade head --sql
   ```
4. Apply it to the test database. Point `DATABASE_URL` at the test Postgres from `docker-compose.test.yml` (use the placeholder in `.env.example`):
   ```
   alembic upgrade head
   ```
5. Test it against a copy of the database that contains rows, not only an empty schema.
6. Run the verify skill.

Downgrade functions may exist for local testing. Never run `alembic downgrade` against a shared or real database.

## Rules

- Forward-only. Never edit a migration that has been applied anywhere. Write a new one instead.
- Additive first. Add the nullable column or new table, deploy code that writes both shapes, backfill, then tighten constraints in a later migration.
- Avoid long locks on `events`. Do not add a NOT NULL column without a default on a large table. For a new foreign key or check constraint, add it `NOT VALID` and then `VALIDATE` it separately.
- Create indexes concurrently:
  ```python
  with op.get_context().autocommit_block():
      op.create_index(..., postgresql_concurrently=True)
  ```
  Concurrent index creation cannot run inside a transaction.
- Backfill in batches of a few thousand rows, committing after each batch. Never run one UPDATE over the whole `events` table.
- Use `timestamptz` and store all times in UTC.

## Checklist for event store changes

- The unique constraint on `(source_id, provider_event_id)` still exists and has a test.
- The `pending_enqueue` index covers the recovery query.
- The retention deletion query has an index on `received_at`.
- Every new table has `account_id`, and every query on it is scoped by account.
- Deletion jobs still log what they delete, without payload content.
- Nothing sensitive is added to logs or error messages.
- Metering still counts each accepted event once.

---
name: verify
description: Runs the full verification chain (lint, types, tests, smoke run, durability check) that proves a change works for this webhook relay.
when_to_use: Run after any code change, before committing, and before finishing a phase or claiming a fix works.
---

# Verify

Run every step in order in this session. Never claim success unless the commands ran.

## Step 1: Lint and format

```
ruff check .
ruff format --check .
```

## Step 2: Types

```
mypy app cli
```

## Step 3: Tests

Start the test services if they are not already running:

```
docker compose -f docker-compose.test.yml up -d
```

Then run:

```
pytest -q
```

If Docker is not running, mark this step FAIL and say so. Do not skip it.

## Step 4: Smoke run

1. Start the API with the dev server command listed in CLAUDE.md.
2. POST a signed sample Stripe event to `/ingest/<source_token>` using a fixture from `tests/fixtures/`. Use the placeholder secret from `.env` (for example `whsec_YOUR_SECRET`). Expect 202.
3. Query Postgres. Expect exactly one row in `events` for that provider event id.
4. Start the worker with the command listed in CLAUDE.md. Confirm one delivery to the local echo server and one `delivery_attempts` row with a 2xx status.
5. POST the same event again. Expect 200 with the original event id. Confirm there is still exactly one row in `events`.

Steps 4 and 5 need the worker, which exists from Phase 2. Before that, mark them SKIPPED and say why.

## Step 5: Durability check

Run this only once the worker exists.

1. Point the echo server at a slow response (a few seconds).
2. Start a delivery and kill the worker process mid-delivery with SIGKILL.
3. Restart the worker.
4. Confirm the event is delivered again and a new attempt row is recorded.

A duplicate delivery to the customer is acceptable because delivery is at-least-once. A missing delivery is a failure.

## Report format

- One line per step: `PASS step N` or `FAIL step N`, or `SKIPPED step N` with the reason.
- For a failure, paste only the first failing output block, not the whole log.
- Stop at the first failing step, fix the cause, then rerun the whole skill from Step 1.

## Rules

- Never fix a failure by weakening a test, skipping a check, or editing the verify steps.
- Never print request bodies, secrets, or `.env` contents in the report.
- Use placeholder values only in fixtures and commands.

---
name: webhook-provider
description: Playbook for adding or changing a payment provider signature adapter so every provider follows the same verification rules.
when_to_use: Use when adding a provider such as GitHub, when changing Stripe verification, or when touching raw body handling or the provider registry.
---

# Webhook provider adapter

## Interface

Every adapter implements the same function:

```python
def verify(raw_body: bytes, headers: Mapping[str, str], secret: str) -> VerifiedEvent: ...
```

`VerifiedEvent` holds the provider event id and event type. Raise `InvalidSignature` on any failure.

Put each adapter in `app/providers/<name>.py`. Confirm the layout in CLAUDE.md.

## Rules

- Verify the raw request bytes only. Never parse JSON before verifying.
- Compare signatures with `hmac.compare_digest`.
- Enforce a timestamp tolerance window and reject events outside it.
- Never log the secret, the signature header value, or the body.
- Invalid signatures are stored with status `rejected` per SPEC.md, so they can be inspected. Do not log them.

## Steps

1. Read the provider's current signing docs. Write down the header name, the signed string, the algorithm and the timestamp format.
2. Implement the adapter in `app/providers/<name>.py`.
3. Register it in the provider registry (`app/providers/registry.py`) under its source type.
4. Add fixtures in `tests/fixtures/<name>/`: one valid signed payload, one with a bad signature, one with a stale timestamp. Generate signatures with a test-only placeholder secret such as `whsec_YOUR_TEST_SECRET`.
5. Write tests for the cases below.
6. Update the provider section of SPEC.md or the docs.
7. Run the verify skill.

## Required tests

- A valid signature passes and returns the correct event id and type.
- A bad signature is rejected.
- Changing one byte of the body breaks verification.
- A timestamp outside the tolerance window is rejected.
- Replaying a captured valid request after the tolerance window is rejected.
- No test makes a real network call.

The replay test covers the adapter only. Duplicate events inside the window are handled by the idempotency constraint, not the adapter.

## Stripe reference

Header:

```
Stripe-Signature: t=<unix timestamp>,v1=<hex HMAC-SHA256>[,v1=<more>]
```

Signed string: `f"{t}."` followed by the raw body bytes. The key is the endpoint secret, which starts with `whsec_`.

Verification:

1. Parse `t` and every `v1` value from the header.
2. Compute HMAC-SHA256 over the signed string.
3. Compare each `v1` with `compare_digest`. Accept if any matches.
4. Reject if `abs(now - t)` is greater than 300 seconds. This matches Stripe's SDK default.
5. Only after verification, parse the JSON. Take the event id from `id` and the type from `type`.

## Registration and docs

- Add the provider name, header name and tolerance to the provider table in the docs.
- Note any provider-specific quirks, such as multiple signatures or a test mode secret.

# Install and first-day guide

This guide sets up the webhook relay project on your machine and walks you through Phase 1.

## Prerequisites

Check each tool. All must be installed before you start.

| Tool | Version needed | Check with |
|---|---|---|
| Python | 3.12 | `python --version` |
| Docker | Any recent version | `docker --version` |
| Git | Any recent version | `git --version` |
| Claude Code | 2.1.296 or newer | `claude --version` |

If Claude Code is older than 2.1.296, update it before continuing.

## Where each file goes

| File | Purpose |
|---|---|
| `CLAUDE.md` | Project memory: purpose, commands, durability rules, gotchas. Claude reads it every session. |
| `SPEC.md` | Product spec with requirements and open questions for you to answer. |
| `PLAN.md` | Six build phases, each with a copy-ready `/goal` prompt. |
| `REVIEW.md` | Review rules that `/code-review` applies. `CLAUDE.md` imports it. |
| `INSTALL.md` | This guide. |
| `.claude/settings.json` | Shared project settings: permissions, subagent model, test output hook. |
| `.claude/skills/verify/SKILL.md` | The checks that prove a change works. Run with `/verify`. |
| `.claude/skills/webhook-provider/SKILL.md` | Steps for adding a payment provider signature adapter. |
| `.claude/skills/migration/SKILL.md` | Safe database migration steps for the event store. |
| `.claude/agents/test-runner.md` | Small subagent that runs tests and returns only failures. |

## Install steps

1. Create the project folder, run `git init`, and copy in the generated files. Keep the paths exactly as listed above.
2. Create a virtual environment: `python -m venv .venv`, then activate it.
3. Copy `.env.example` to `.env` and fill in placeholder values for local use. Phase 1 creates `.env.example`. If it is missing, ask Claude to create it with placeholders only.
4. Start Postgres and Redis for development: `docker compose up -d`.
5. Install the plugins (see below).
6. Start Claude Code in the project folder with `claude`.
7. Open `SPEC.md` and answer the open questions in the file before Phase 1.
8. Paste the kickoff prompt from the end of this guide to begin Phase 1.
9. When Phase 1 finishes, run `/verify`.

Never commit `.env`. It holds your local values. Only `.env.example` with placeholders belongs in git. Add `.env` to `.gitignore` before your first commit.

## Plugin install commands

Run these in your terminal:

```
claude plugin install code-review@claude-plugins-official
claude plugin install feature-dev@claude-plugins-official
claude plugin install security-guidance@claude-plugins-official
claude plugin install commit-commands@claude-plugins-official
```

What each one does:

- `code-review` reviews diffs for durability, security and idempotency bugs.
- `feature-dev` guides exploration and design for each phase's feature.
- `security-guidance` flags injection, SSRF and secret handling issues.
- `commit-commands` makes consistent commits at the end of each phase.

## Local services

`docker compose up -d` starts Postgres and Redis for development. Tests use a separate stack defined in `docker-compose.test.yml`, which `/verify` starts for you.

Connection values live in `.env`, copied from `.env.example`. The example file uses placeholders such as `YOUR_DB_PASSWORD` and `YOUR_REDIS_URL`. Replace them locally. Never paste real keys into any tracked file, chat, or issue.

## Starting a session with the kickoff prompt

1. Start with `claude` in the project folder.
2. Paste the kickoff prompt below as one message. It begins with `/goal`.
3. Claude plans first, then implements. Watch the plan before you let it continue.

Kickoff prompt:

```
/goal Build Phase 1 of the webhook relay in this repo following SPEC.md and PLAN.md. [Paste the full kickoff prompt from PLAN.md Phase 1 here.]
```

## Understanding /goal

`/goal <condition>` keeps Claude working until a small, fast model judges the condition met. The judge only sees what Claude has shown in the conversation, so Claude must print its test output.

- Stop a running goal with `/goal clear`.
- Every goal prompt has a turn bound, such as "stop after 40 turns." If Claude reaches the bound without finishing, read what it left undone, then decide whether to continue in a new goal.
- A goal that never ends is a sign the end state was not measurable. Clear it and rewrite the condition.

## Token habits

- Run `/clear` between unrelated tasks so old context does not cost tokens on every turn.
- Run `/compact` at natural breaks, such as after a phase's tests pass.
- Run `/usage` to see where tokens went.
- Explore, then plan, then code, then verify. Ask Claude to show the check output.

## First-day checklist

- [ ] Versions checked, including Claude Code 2.1.296 or newer
- [ ] Project folder created, `git init` run, generated files in place
- [ ] Virtual environment created and activated
- [ ] `.env` created locally from `.env.example`, and `.env` is in `.gitignore`
- [ ] `docker compose up -d` runs and Postgres and Redis are up
- [ ] Four plugins installed
- [ ] `SPEC.md` open questions answered
- [ ] Phase 1 kickoff prompt pasted and Claude has shown a plan
- [ ] `/verify` run at the end of Phase 1, with output visible

## When /verify fails

`/verify` reports pass or fail for each step and shows the first failing output.

1. Read the first failing output. Fix that problem before anything else.
2. Ask Claude to fix the failure and rerun `/verify`. Do not accept a claim of success without seeing the command output.
3. If the failure is in the durability check, do not skip it. The relay's core promise depends on it.
4. If a fix needs a new dependency or a database downgrade, Claude must ask first. Answer no unless you understand the reason.
5. If you are stuck after two attempts, run `/clear`, then describe the failing output in a new session.

# Webhook relay: build plan

Each phase ends in a `/goal` prompt. Paste one phase at a time.
Answer the SPEC.md open questions before Phase 3.

Model: default Opus 5.5 at medium effort.
Ultrathink needs Claude Code 2.1.311 or newer.

## Phase 1: Foundation and durable ingest

Goal: a project skeleton and an ingest endpoint that commits before it acknowledges.

Files touched: `pyproject.toml`, `app/`, `migrations/`, `tests/`, `docker-compose.test.yml`.

Checks: `/verify` (ruff, mypy, pytest with Postgres and Redis).

Mode: plan mode first. Type `ultrathink` in the planning turn for the schema and idempotency design.

```
/goal Build Phase 1 of the webhook relay per SPEC.md and PLAN.md.
End state: (1) POST /ingest/{source_token} stores raw body, headers and provider event id in Postgres and commits before returning 202; (2) unique constraint on (source_id, provider_event_id) returns 200 with the original event id on duplicates and creates no new row; (3) if Redis enqueue fails, the event is kept with status pending_enqueue; (4) a test simulates a crash after commit and before enqueue and shows the event still exists.
Check: run /verify and show passing output in the conversation.
Constraints: Postgres is the source of truth; Redis is never the only copy; no request bodies or secrets in logs; dependencies limited to FastAPI, SQLAlchemy, Alembic, psycopg, redis, pytest, httpx, ruff, mypy; no delivery, retries, auth or billing yet; placeholders only for secrets; change nothing outside this repo.
Stop after 40 turns or when all four items pass, and list what is left.
```

## Phase 2: Delivery and retries

Goal: a Redis-backed worker that delivers events with backoff, dead-letter, replay and recovery.

Files touched: `app/workers/`, `app/delivery/`, `app/api/replay.py`, `app/security/ssrf.py`, `tests/delivery/`.

Checks: `/verify`, plus the durability check in the verify skill.

Mode: plan mode first. Type `ultrathink` for the queue, backoff and recovery design.

```
/goal Build Phase 2 of the webhook relay per SPEC.md and PLAN.md.
End state: (1) a worker delivers stored events to a local echo server; (2) failed deliveries retry with exponential backoff, cap 1 hour, full jitter, max 10 attempts, and tests check the bounds; (3) after the last attempt the event is dead_lettered; (4) replay creates new attempts for an existing event and creates no new event row; (5) a recovery job re-enqueues pending_enqueue and undelivered events from Postgres after Redis is flushed; (6) destination URLs reject private, loopback, link-local and metadata addresses.
Check: run /verify and show passing output, including the worker-kill redelivery step.
Constraints: 10 second timeout on outbound HTTP; never log bodies or secrets; tests never call real external hosts; no new dependencies; Postgres stays the source of truth.
Stop after 40 turns or when all six items pass, and list what is left.
```

## Phase 3: Signature verification and auth

Goal: provider adapters, Stripe verification on raw bytes, hashed API keys, and source and endpoint management.

Files touched: `app/providers/`, `app/auth/`, `app/api/sources.py`, `app/api/endpoints.py`, `tests/providers/`, `tests/auth/`.

Checks: `/verify`, plus the webhook-provider skill checklist.

Mode: default.

```
/goal Build Phase 3 of the webhook relay per SPEC.md and PLAN.md.
End state: (1) a provider adapter interface with a Stripe adapter that verifies raw body bytes with constant-time compare and a 5 minute tolerance window; (2) tests cover valid, invalid, stale and replayed signatures; (3) invalid signatures are stored as rejected and never delivered; (4) API keys are shown once and stored only as a keyed hash with a visible prefix; (5) source and endpoint endpoints are scoped by account id, and a cross-account request returns 404 in a test.
Check: run /verify and show passing output in the conversation.
Constraints: parse JSON only after verification; never log secrets or bodies; use YOUR_API_KEY placeholders in docs; ask before adding any dependency; keep changes small.
Stop after 40 turns or when all five items pass, and list what is left.
```

## Phase 4: Billing, retention and alerts

Goal: usage metering, the free tier limit, the retention job, failure alerts and the status page.

Files touched: `app/billing/`, `app/retention/`, `app/alerts/`, `app/api/status.py`, `tests/billing/`, `tests/retention/`.

Checks: `/verify`, plus a retention dry run on test data.

Mode: default.

```
/goal Build Phase 4 of the webhook relay per SPEC.md and PLAN.md.
End state: (1) each accepted event is metered once; duplicates and rejected events are not counted; (2) free tier of 10,000 events per month, with the at-limit behavior from SPEC.md (keep storing and flag, never drop) tested; (3) a retention job deletes events past 3 days (free) or 30 days (paid), including attempts, logs counts and ids only, and is tested; (4) an email alert is sent once per incident when an endpoint fails for more than 1 hour, using a fake sender in tests; (5) GET /status returns component health for ingest, workers, Postgres and Redis.
Check: run /verify and show passing output in the conversation.
Constraints: email goes through an adapter interface; no real email in tests; UTC only; no payloads in logs; no new dependencies without asking.
Stop after 40 turns or when all five items pass, and list what is left.
```

## Phase 5: CLI and dashboard

Goal: the localhost forwarding CLI and a minimal dashboard for event list, detail and replay.

Files touched: `cli/`, `app/dashboard/`, `tests/cli/`, `tests/dashboard/`.

Checks: `/verify`, plus a manual forward of one event to a local echo server.

Mode: default. Confirm the dashboard frontend answer (SPEC open question 2) first.

```
/goal Build Phase 5 of the webhook relay per SPEC.md and PLAN.md.
End state: (1) relay login stores YOUR_API_KEY-style token in a file readable only by the user; (2) relay listen opens a streaming connection and forwards events to a localhost URL, printing status and timing per event; (3) the dashboard lists events with filters, shows event detail with every attempt, and has a working replay button; (4) tests cover the CLI forward loop and the replay action.
Check: run /verify and show passing output in the conversation.
Constraints: use the frontend choice recorded in SPEC.md, or server-rendered templates if none is recorded; no secrets in output; no new dependencies without asking; all times in UTC.
Stop after 40 turns or when all four items pass, and list what is left.
```

## Phase 6 (optional): Hardening

Goal: load test, chaos test, security review and docs.

Files touched: `tests/load/`, `tests/chaos/`, `docs/`, `README.md`.

Checks: `/verify`, `/code-review` on the full diff, and the chaos run.

Mode: default. Ask before adding a load-testing dependency.

```
/goal Build Phase 6 of the webhook relay per SPEC.md and PLAN.md.
End state: (1) a load test measures ingest p99 against the SPEC.md target and the result is written to docs/load-test.md; (2) a chaos test kills the worker and Redis mid-run and shows zero lost events after recovery; (3) /code-review on the full diff has no open blocker findings; (4) docs cover setup, the CLI and the API, using placeholders only.
Check: run /verify and the chaos test, and show passing output in the conversation.
Constraints: load test runs only against local services; ask before adding any dependency; no secrets in docs or logs.
Stop after 40 turns or when all four items pass, and list what is left.
```

# Review checklist

Apply these rules to every diff. Group findings as blocker, should-fix, or nit.

## Durability

- Flag any path where an accepted event could be lost: ack before commit, Redis-only state, swallowed exceptions.
- Ingest must write to Postgres and commit before returning 2xx.
- Flag any background task or fire-and-forget call used for ingest.
- Flag any code path that treats Redis as the only copy of an event.

## Idempotency

- Dedupe relies on a unique constraint on provider event id plus source id. Flag application-only checks.
- Replays create a new delivery attempt, never a new event row.
- Flag retry or replay code that could bill the same event twice.

## Retry correctness

- Backoff is exponential with jitter, bounded by a max delay and a max attempt count.
- Every outbound HTTP call has a timeout.
- Customer endpoint URLs are checked for SSRF. Block private IP ranges, loopback, link-local and cloud metadata addresses, including after DNS resolution and redirects.
- Delivery carries a stable idempotency header.

## Security

- Signatures are verified on the raw request body bytes, before any JSON parsing.
- Signature comparison is constant-time.
- API keys and provider secrets are hashed or encrypted at rest.
- No request bodies, headers with secrets, or secrets appear in logs or error messages.

## Tenant isolation

- Every query is scoped by account id. Flag any query that reads or writes sources, endpoints, events or attempts without it.

## Retention and billing

- The deletion job removes only data past its retention window and logs the deletion without payloads.
- Metering counts each accepted event exactly once.

## Tests

- New behavior has a test. Failure paths are covered, including crash and retry cases.
- No test makes a real network call to a provider or a customer endpoint.

## Style

- Ruff and mypy pass. Functions stay small.
- Flag new dependencies that lack a stated reason. Dependencies need approval first.

## Output format

- **Blocker:** must fix before merge (data loss, security, tenant leak, double billing).
- **Should-fix:** real risk or missing test, fix before the phase ends.
- **Nit:** style or naming only.

# Webhook relay: product spec

## Problem and target customer

Small SaaS teams (2 to 10 developers) lose payment webhooks when their endpoint is down, slow, or redeployed during delivery. Missed events leave subscription state wrong and force manual reconciliation.

## MVP scope

- Hosted service only. Self-hosting is out of scope.
- In scope: sources, destination endpoints, ingest, storage, delivery with retries, replay, alerts, usage billing, status page, dashboard, localhost CLI.
- Out of scope: payload transformation, non-webhook integrations, self-hosted installs.

## Core flows

1. Create a source. The relay returns a unique ingest URL with a token.
2. Create a destination endpoint (HTTPS URL, checked against SSRF rules).
3. The provider sends an event to the ingest URL. The relay verifies the signature, stores the event, and returns 202.
4. A worker delivers the event to each destination and records every attempt.
5. Failed deliveries retry with backoff until they are dead-lettered.
6. A user replays any event from the dashboard.
7. A developer runs the CLI to forward events to localhost.

## Durability and idempotency acceptance criteria

- AC-1: The relay returns 202 only after the event row is committed to Postgres.
- AC-2: If the process crashes after commit and before enqueue, a recovery job finds the event and delivers it. Zero events are lost.
- AC-3: If Redis is down at ingest, the event is stored with status `pending_enqueue` and re-enqueued when Redis returns.
- AC-4: A duplicate provider delivery (same source and provider event id) returns 200 with the original event id and creates no new row.
- AC-5: A replay creates new delivery attempts for an existing event. It never creates a new event.
- AC-6: Killing a worker mid-delivery leads to redelivery after restart.
- AC-7: Every delivery carries a stable idempotency header, so receivers can dedupe. Delivery is at-least-once.

## Retry policy defaults (proposed, confirm)

- Failure: non-2xx response, connection error, or no response within 10 seconds.
- Schedule: start at 30 seconds, double each attempt, cap at 1 hour, add full jitter.
- Max attempts: 10 (about 4 hours in total without jitter).
- Dead-letter: after the last attempt, the event status becomes `dead_lettered` and shows in the dashboard.
- Manual replay: available for any event, including dead-lettered ones.

## Signature verification

- Stripe first. Verify raw request bytes, the `Stripe-Signature` header, and a timestamp tolerance window.
- A provider adapter interface supports GitHub and others later.
- Invalid signature: return 401, store the event as `rejected` for debugging, and exclude it from delivery and metering.
- Unknown source token: return 404 and store nothing.

## Plans and billing

- Usage based: price per accepted event (price open, see questions).
- Free tier: 10,000 accepted events per month.
- Metering counts each accepted event once. Duplicates and rejected events are not counted.
- At the limit: default proposal is to keep storing and flag events. Events are never dropped. Confirm in open questions.

## Retention

- Free: 3 days. Paid: 30 days.
- A daily scheduled job deletes expired events and their delivery attempts.
- Each deletion run logs counts and ids only, never payloads. The job is tested.

## Alerts and status

- Email the account when an endpoint has failed continuously for more than 1 hour.
- One email per incident. An incident closes on the first success. A later failure opens a new incident.
- Status page shows component health: ingest, delivery workers, Postgres, Redis.

## Dashboard

- Event list with filters: source, endpoint, status, time range.
- Event detail: headers, body, and every delivery attempt with status code and timing.
- Replay button on each event.
- Frontend technology is an open question.

## CLI

- `relay login` stores an API token (example: YOUR_API_KEY). The token file is readable only by the user.
- `relay listen` opens a streaming connection and forwards events to a localhost URL.
- Shows each event with the local response status and timing.

## Non-functional requirements

- Ingest latency target: p99 under 200 ms at the relay, excluding provider network time (proposed).
- Throughput target: 100 events per second sustained, 500 per second burst (proposed).
- Auth: API keys are shown once and stored hashed at rest. Each key has a visible prefix for lookup.
- Tenant isolation: every query is scoped to the account.
- Logs never contain payload bodies or secrets.
- All times are stored and shown in UTC.

## Open questions

1. At the free tier limit, keep storing and flag (proposed) or pause ingest?
2. Dashboard frontend: server-rendered templates or a separate single-page app?
3. CLI streaming protocol: SSE or WebSocket?
4. Confirm retry defaults: 10 attempts and 1-hour cap?
5. Price per event and paid plan price?
6. Per-endpoint circuit state: pause delivery after how many consecutive failures?
7. Which providers after Stripe, and in what order?
8. Alert recipients: account owner only, or all admins?
9. Confirm ingest latency and throughput targets.
10. Hosting provider and region?