Daily competitor price change alert emailer

# Competitor price watcher

## Purpose

Daily monitor of competitor pricing pages. It emails the owner only when a tracked price changes.

Behavior is defined in SPEC.md. The current phase is in PLAN.md.

## Stack

- Python 3.12
- uv for dependencies and running commands
- httpx for fetching
- selectolax for HTML parsing
- SQLite for history (`data/watcher.db`)
- pydantic-settings for config
- pytest and ruff for tests and lint

## Commands

- `uv sync`: install dependencies
- `uv run pytest -q`: run tests (no network)
- `uv run ruff check .`: lint
- `uv run python -m watcher init-db`: create the observations table
- `uv run python -m watcher run --dry-run --fixtures tests/fixtures`: smoke run, no network, no email
- `uv run python -m watcher run`: real daily run (cron or launchd)

## Layout

- `src/watcher/fetch.py`: httpx fetcher, robots.txt check, timeouts, retries
- `src/watcher/extract.py`: selectolax price extraction
- `src/watcher/store.py`: SQLite reads and writes
- `src/watcher/diff.py`: change and failure rules
- `src/watcher/notify.py`: SMTP email
- `src/watcher/cli.py`: command entry points
- `config/targets.yaml`: competitor URLs and CSS selectors
- `tests/fixtures/`: saved HTML snapshots, one per target
- `data/`: SQLite database, gitignored
- `.env`: secrets, gitignored; `.env.example` has placeholders only

## Conventions

- Store prices as integer minor units plus an ISO currency code.
- Store no raw HTML. Keep only a short snippet and its hash.
- Use UTC timestamps. No naive datetimes.
- Log errors with the target id, never with full page content.
- Keep CLAUDE.md under 200 lines. Put detail in SPEC.md or PLAN.md.

## Fetch rules

- One request per domain per run.
- Send an identifying User-Agent.
- Honor robots.txt.
- Timeout is 15 seconds.
- At most 2 retries, with backoff.
- No login, no CAPTCHA bypass, no checkout pages.

## Gotchas

- A failed parse is an error row, never a price of zero.
- Alert only when the new ok price differs from the last ok price. Ignore error rows when comparing.
- Secrets live in `.env` only. Never put real SMTP credentials in code, tests or fixtures.
- Use placeholders such as YOUR_SMTP_PASSWORD in `.env.example`.
- Do not edit SPEC.md unless the owner asks.

## Working rules

- Read SPEC.md and PLAN.md before writing code.
- Follow REVIEW.md before calling work done.
- Run the verify skill before claiming any change works.
- Use the test-runner subagent for test output so the main context stays small.
- Do not edit tests to make them pass. Report the mismatch.

## Review rules

@REVIEW.md

---
name: test-runner
description: Use after code changes to run tests and lint and report failures only. Keeps test output out of the main context.
model: haiku
tools: Bash, Read, Grep
maxTurns: 12
---

You run the project's tests and lint, then report results briefly. You never edit files.

Run these commands from the project root, in this order:

1. `uv run pytest -q`
2. `uv run ruff check .`

Report:

- Tests: pass or fail, with the passed and failed counts from the pytest summary.
- Lint: pass or fail, with the number of ruff violations.

For each failing test, give:

- the file path
- the test name
- the one-line assertion or error message that failed

For each ruff violation, give the file, line number, and rule code, one per line.

Do not paste full logs, passing test names, or stack traces. If a command fails to start (for example, `uv` is missing or dependencies are not synced), report the exact error line and stop.

Do not edit, create, or delete any file. Do not change tests to make them pass. Report the mismatch and let the main session decide.

{
  "model": "sonnet",
  "permissions": {
    "defaultMode": "default",
    "allow": [
      "Bash(uv sync)",
      "Bash(uv run pytest:*)",
      "Bash(uv run ruff:*)",
      "Bash(uv run python -m watcher:*)",
      "Bash(git status)",
      "Bash(git diff:*)"
    ],
    "ask": [
      "Bash(git push:*)",
      "Bash(git commit:*)",
      "Bash(uv add:*)"
    ],
    "deny": [
      "Read(.env)",
      "Read(./.env)",
      "Bash(rm -rf:*)",
      "Bash(curl:*)"
    ]
  }
}

---
name: verify
description: Use before claiming any change to this watcher works. Runs lint, tests and an offline dry-run smoke check and reports PASS or FAIL.
when_to_use: Use after finishing a phase in PLAN.md, or after editing parsing, diff, storage or notify code in src/watcher/. Also use when the user asks whether the project works.
---

# Verify

Prove a change works using only the project's real commands. This skill runs no network calls and sends no email.

## Rules

- Run the steps in order. Stop at the first failing step, then report.
- Run each command from the project root.
- Record the exit code of every step.
- Do not edit tests, fixtures or expected values to make a step pass. If a test disagrees with the code, report the mismatch and stop.
- Do not modify `.env`, `config/targets.yaml` or anything under `data/`.
- Do not run `git commit`, `git push` or `uv add`.

## Steps

### 1. Lint

Run:

```
uv run ruff check .
```

- Exit code 0: report "Lint: pass (exit 0)" and continue.
- Non-zero: report "Lint: fail (exit N)". Show only the file path, line number and rule code for each finding. Stop here.

### 2. Tests

Run:

```
uv run pytest -q
```

- Exit code 0: report the passed count, for example "Tests: pass (exit 0, 14 passed)", and continue.
- Non-zero: report "Tests: fail (exit N)". For each failing test show:
  - the test name, as `path::test_name`
  - the first line of the error, usually the assertion or exception line

  Do not paste the full log. Stop here.

If pytest reports collection errors instead of test failures, show the module name and the first error line, then stop.

### 3. Smoke run

Run:

```
uv run python -m watcher run --dry-run --fixtures tests/fixtures
```

This uses saved HTML and must not touch the network or send email.

- Exit code 0: report "Smoke run: pass (exit 0)" and continue.
- Non-zero: report "Smoke run: fail (exit N)". Show the first error line only. Stop here.

Confirm the output says no email was sent. If it does not, or if it reports any live fetch, report "Smoke run: fail (dry run did not stay offline)" and stop.

## Report format

Write one line per step, in order:

```
Lint: pass (exit 0)
Tests: pass (exit 0, 14 passed)
Smoke run: pass (exit 0)
```

For failures, add the details listed above under the failing step only.

## Final line

The last line of the report must be exactly one of:

- `PASS`
- `FAIL: <step name>`, for example `FAIL: Tests`

Use the step names Lint, Tests, or Smoke run.

# Install and run the competitor price watcher

This guide assumes you have some Python experience. You do not need prior uv or cron knowledge.

## 1. Install Claude Code

Follow the official installer instructions for Claude Code. Then check that it works:

```
claude --version
```

## 2. Open the project

In a terminal, go to the project folder and start Claude Code:

```
cd /path/to/competitor-watcher
claude
```

Claude Code reads `CLAUDE.md` automatically at startup. That file tells it what the project does, which commands to run and which rules to follow. You do not need to paste it in.

## 3. Install uv

uv manages the Python version and the project dependencies. Install it with the official one-line installer from docs.astral.sh/uv. Then check it:

```
uv --version
```

Running `uv sync` in the project folder creates a local `.venv` and installs every dependency listed in `pyproject.toml`. You run it once, and again whenever dependencies change.

```
uv sync
```

## 4. Create your settings file

Copy the example file:

```
cp .env.example .env
```

Open `.env` in a text editor and fill in:

- `SMTP_HOST`: your mail server, for example `smtp.gmail.com`
- `SMTP_PORT`: usually `587`
- `SMTP_USER`: the account that sends the email
- `SMTP_PASSWORD`: the password or app password for that account
- `ALERT_TO`: the address that receives alerts

For Gmail, a normal password will not work. Turn on 2-Step Verification for your Google account, then create an app password and use that value for `SMTP_PASSWORD`.

Never commit `.env`. It is listed in `.gitignore`, and only `.env.example` belongs in git.

## 5. Add competitor pages

Open `config/targets.yaml` and add 3 to 5 product pages. Each entry needs:

- `id`: a short unique name, such as `acme-widget-pro`
- `name`: the label used in emails
- `url`: the product page
- `selector`: a CSS selector that points to the price
- `currency`: an ISO code such as `USD`
- `check_enabled`: set to `true` to include the page

To find a selector:

1. Open the product page in your browser.
2. Right-click the price and choose Inspect.
3. In the developer tools, right-click the highlighted element.
4. Choose Copy, then Copy selector.
5. Paste the result into `selector`.

Selectors break when a site changes its layout. If you see parse errors, repeat these steps for that page.

## 6. Test safely with a dry run

A dry run fetches nothing from the internet and sends no email. It reads the saved pages in `tests/fixtures` and prints what it would do:

```
uv run python -m watcher run --dry-run --fixtures tests/fixtures
```

To check the code and tests, run:

```
uv run pytest -q
uv run ruff check .
```

## 7. Schedule the daily run

The watcher should run once a day. On Linux or with cron, open your crontab:

```
crontab -e
```

Add this line to run at 08:00 every day. Replace the path with your real project folder:

```
0 8 * * * cd /path/to/competitor-watcher && uv run python -m watcher run
```

If cron cannot find `uv`, run `which uv` in a terminal and use the full path it prints, for example `/home/you/.local/bin/uv`.

macOS users can use launchd instead of cron. Ask Claude Code in the project to write a launchd plist that runs the same command once a day. The example plist in INSTALL.md shows the expected format.

## 8. Install the Claude Code plugins

Start Claude Code in the project folder if it is not already running. Then type these commands inside the session:

```
/plugin install code-review@claude-plugins-official
/plugin install commit-commands@claude-plugins-official
/plugin install security-guidance@claude-plugins-official
```

- `code-review` checks changes against REVIEW.md before you commit.
- `commit-commands` keeps commits consistent.
- `security-guidance` flags risky patterns such as secret handling and unsafe shell or HTTP use.

## 9. Build the project with the phase prompts

PLAN.md has five phases. Each phase ends with a `/goal` prompt you can paste into Claude Code. Start with Phase 1. After each phase, run the verify skill by typing `/verify` or asking Claude to use it. Read the PASS or FAIL line before you move on.

## Troubleshooting

**A parse error appears for one target.** The selector probably no longer matches the page. Follow step 5 to copy a new selector. The watcher records the error and does not treat it as a price of zero.

**A fetch error appears for one target.** The site may be down, slow or blocking the request. The watcher retries twice with backoff. If a target fails 3 runs in a row, you get one "fetch failing" email for that target. You will not get another until it recovers. With a daily schedule, that means about three days.

**No email arrives.** Check the SMTP values in `.env`. Run a dry run first to confirm the rest of the watcher works. A run with no price changes sends nothing, which is expected.

**You want to test without sending email.** Always use `--dry-run` while you are setting up. It sends nothing and makes no network calls.

**The scheduled run does not start.** Run the command from the crontab line by hand in a terminal. If it works there, the problem is the path to `uv` or the working directory in the cron line.

# Plan: competitor price watcher

Five phases. Each phase ends with the verify skill reporting PASS. Run each goal prompt in Claude Code with `/goal`. Read CLAUDE.md and SPEC.md before starting any phase.

## Phase 1: skeleton

**Scope**

- uv project with `pyproject.toml` and `src/watcher/` package
- pydantic-settings config loader (reads `.env`)
- `config/targets.yaml` loader with schema validation from SPEC.md
- `init-db` command that creates `data/watcher.db` with the `observations` table

**Done when**

- `uv sync` succeeds
- The verify skill passes, with at least 6 tests for config and targets loading
- `uv run python -m watcher init-db` creates the table exactly as SPEC.md defines it

**Goal prompt**

```
/goal Phase 1 of PLAN.md is complete when: the uv project builds with `uv sync`; `uv run pytest -q` passes with at least 6 tests covering config loading and targets.yaml parsing; `uv run ruff check .` exits 0; and `uv run python -m watcher init-db` creates data/watcher.db containing the observations table exactly as defined in SPEC.md. Check: run those three commands and report each exit code and the test count in your final message. Constraints: read CLAUDE.md, SPEC.md and PLAN.md before writing code; make no network calls; never hardcode secrets, and use .env.example with placeholders such as YOUR_SMTP_PASSWORD; do not edit SPEC.md; add no dependencies beyond httpx, selectolax, pydantic-settings, pyyaml, pytest and ruff; keep the package under src/watcher/ with the module names listed in CLAUDE.md. Use the test-runner subagent for test output so the main context stays small. Stop after 30 turns and list anything unfinished with the reason.
```

## Phase 2: fetch and extract

**Scope**

- `fetch.py`: httpx fetcher with robots.txt check, 15 second timeout, at most 2 retries with backoff, identifying User-Agent, one request per domain per run
- `extract.py`: selectolax price extractor that turns text such as `$1,299.00` into integer minor units and an ISO currency code
- Fixture HTML for each example target in `tests/fixtures`

**Done when**

- The verify skill passes
- Extractor tests cover commas, thousands separators, currency symbols and missing values (missing value is an error)
- Fetcher tests use fixtures or mocks only; no live network in tests

**Goal prompt**

```
/goal Phase 2 of PLAN.md is complete when: fetch.py implements an httpx fetcher with robots.txt checking, a 15 second timeout, at most 2 retries with backoff, an identifying User-Agent and one request per domain per run; extract.py turns price text like $1,299.00 into integer price_minor plus currency using selectolax and the target selector; and tests/fixtures contains a saved HTML file for each example target. Check: run `uv run pytest -q` and `uv run ruff check .`, and report exit codes and test counts. Constraints: tests make no live network calls and use saved fixtures or mocks; a missing or unparseable price must raise an error, never return zero; never hardcode secrets; add no new dependencies; do not edit SPEC.md. Use the test-runner subagent for test output. Stop after 30 turns and list anything unfinished with the reason.
```

## Phase 3: diff and storage

**Scope**

- `store.py`: write `ok` and `error` observations with UTC timestamps and snippet hashes
- `diff.py`: change rule and failure rule from SPEC.md, comparing only `ok` rows
- Failure counter per target that resets on recovery

**Done when**

- The verify skill passes
- Tests cover: increase, decrease, no change, error then recovery, and the 3-failure rule
- Failure alerts are produced once per failure episode

**Goal prompt**

```
/goal Phase 3 of PLAN.md is complete when: store.py writes ok and error rows to the observations table with UTC ISO timestamps, integer price_minor and snippet_hash; diff.py applies the SPEC.md change rule (compare only ok rows, first ok row is a baseline) and failure rule (3 consecutive failed runs send one fetch-failing alert per episode, recovery resets it); and tests cover increase, decrease, no change, error then recovery, and the 3-failure rule. Check: run `uv run pytest -q` and `uv run ruff check .` and report exit codes and test counts. Constraints: use an in-memory or temporary SQLite database in tests; no live network calls; timestamps are UTC and never naive; do not store raw HTML; do not edit SPEC.md; no new dependencies. Use the test-runner subagent for test output. Stop after 30 turns and list anything unfinished with the reason.
```

## Phase 4: notify and schedule

**Scope**

- `notify.py`: SMTP email with `smtplib`, content from SPEC.md, URLs escaped, no competitor HTML
- `cli.py`: `run` command with `--dry-run` and `--fixtures`
- INSTALL.md cron and launchd examples

**Done when**

- The verify skill passes, including the dry-run smoke test
- A test with a fake SMTP object shows zero changes sends nothing
- A test shows a change sends one email with the SPEC.md content

**Goal prompt**

```
/goal Phase 4 of PLAN.md is complete when: notify.py sends email with smtplib using the SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASSWORD and ALERT_TO settings, with the subject and body defined in SPEC.md and URLs escaped; cli.py provides `run` with --dry-run and --fixtures; a test with a fake SMTP object proves zero changes sends nothing; and `uv run python -m watcher run --dry-run --fixtures tests/fixtures` completes and sends nothing. Check: run `uv run pytest -q`, `uv run ruff check .` and the dry-run command, and report each exit code. Constraints: no real SMTP connection in tests; credentials only from environment variables; never print passwords; the email body contains no competitor HTML; do not edit SPEC.md; no new dependencies. Use the test-runner subagent for test output. Stop after 30 turns and list anything unfinished with the reason.
```

## Phase 5: hardening

**Scope**

- Structured logging that records target id and error type, never full page content
- Non-zero exit on unhandled errors
- INSTALL.md troubleshooting and scheduling notes
- End-to-end verify run and a final code review against REVIEW.md

**Done when**

- The verify skill passes end to end
- `/code-review` finds no open issues against REVIEW.md
- All five phases are marked PASS in this file's notes

**Goal prompt**

```
/goal Phase 5 of PLAN.md is complete when: logging records target ids and error types but never full page content; an unhandled error makes `uv run python -m watcher run` exit non-zero; INSTALL.md covers troubleshooting and scheduling; the verify skill passes end to end; and a /code-review pass against REVIEW.md reports no open issues. Check: run the verify skill, then run /code-review, and report the PASS or FAIL line and the review findings. Constraints: no network calls in tests; no secrets in code, fixtures or logs; do not edit SPEC.md; do not edit tests to make them pass, report mismatches instead; no new dependencies. Use the test-runner subagent for test output. Stop after 30 turns and list anything unfinished with the reason.
```

## After Phase 5: your first live run

Automated phases never call the real competitor sites. Do this yourself:

1. Copy `.env.example` to `.env` and fill in real SMTP values. Never commit `.env`.
2. Run `uv run python -m watcher run --dry-run` against live pages (this one touches the network but sends no email).
3. Check that each target shows one `ok` row in the database.
4. Schedule the daily run as described in INSTALL.md.

# Review checklist

Use this checklist for every change. CLAUDE.md imports this file, so `/code-review` applies these rules.

## Secrets

- No secrets, tokens or real SMTP credentials in code, tests, fixtures or docs.
- `.env` stays in `.gitignore`. Only `.env.example` is committed, with placeholders such as `YOUR_SMTP_PASSWORD`.
- Settings come from environment variables or `.env`, never from literal values in source.

## Tests and fixtures

- Tests make no live network calls.
- Every competitor page used in a test is a saved fixture under `tests/fixtures`.
- Fixtures contain no real customer data, cookies or session tokens.

## Price parsing

- Handles commas, thousands separators, currency symbols and surrounding whitespace.
- Converts to integer minor units and keeps the ISO currency code.
- A missing or unparseable value is an error row, never a price of zero.

## Diff and alert rules

- Compares only `ok` observations.
- Matches the change and failure rules in SPEC.md, including the 3-run failure email that is not repeated until recovery.
- A run with zero changes sends no email.

## Fetching

- Checks robots.txt before each request.
- Sends the identifying User-Agent.
- Enforces the 15 second timeout, at most 2 retries with backoff, and one request per domain per run.
- Does not log in, solve CAPTCHAs or bypass access controls.

## Email

- Body contains only the fields listed in SPEC.md.
- No competitor HTML appears in the subject or body.
- URLs are escaped before they are placed in the email.

## Time

- All timestamps are UTC and stored as ISO strings.
- No naive datetimes. Reject any `datetime` without `tzinfo`.

## Logging

- Errors are logged with the target id and the error type.
- Logs never contain full page content, the SMTP password or the full `.env` contents.

## Before merging

- `uv run ruff check .` exits 0.
- `uv run pytest -q` passes.
- The verify skill reports PASS.

# Competitor price watcher spec

## Goal

Detect price changes on up to 20 competitor product pages. Email a summary within 10 minutes of the daily run.

## Non-goals

- Checkout or cart flows
- Logged-in pages
- CAPTCHA solving or bypass
- Scraping more than once per day
- Multi-user accounts

## Targets file

`config/targets.yaml` is a list. Each entry has:

- `id`: short unique slug, for example `acme-widget-pro`
- `name`: display name used in emails
- `url`: product page URL
- `selector`: CSS selector for the price element
- `currency`: ISO 4217 code, for example `USD`
- `availability_selector`: optional CSS selector
- `check_enabled`: `true` or `false`

Targets with `check_enabled: false` are skipped. They do not count as failures.

## Database

SQLite file `data/watcher.db`, table `observations`:

- `id`: integer primary key
- `target_id`: text, not null
- `observed_at`: text, not null, UTC ISO 8601 timestamp
- `price_minor`: integer in minor units, null when status is `error`
- `currency`: text, ISO 4217 code
- `snippet_hash`: text, SHA-256 of the matched price text (no raw HTML stored)
- `status`: text, `ok` or `error`
- `error_message`: text, null when status is `ok`

A failed parse is always an `error` row. It is never a price of zero.

## Change rule

- Compare the latest `ok` observation with the `ok` observation before it.
- Ignore `error` rows when comparing.
- The first `ok` observation for a target sets the baseline. It is not a change.
- A change exists when the two prices differ.
- Percent change is `(new - old) / old * 100`, rounded to one decimal place.

## Failure rule

- A target whose fetch or extract fails in a run gets an `error` row.
- After 3 consecutive failed runs for one target, send one "fetch failing" email for that target.
- Do not repeat that email until the target recovers.
- Recovery means one new `ok` observation. The failure count then resets.

## Email

Delivery uses SMTP settings from environment variables:

- `SMTP_HOST`
- `SMTP_PORT`
- `SMTP_USER`
- `SMTP_PASSWORD`
- `ALERT_TO`

Change email:

- Subject: `Price watch: N change(s)`, where N is the number of changes.
- For each change, show: target name, old price, new price, percent change, URL, time observed in UTC.
- Do not include any HTML from competitor pages. URLs are plain text.

Failure email:

- Subject: `Price watch: fetch failing for <name>`.
- Body: target name, URL, number of consecutive failed runs, last error message.

Rules:

- A run with zero changes and no new failure alerts sends no email.
- Change and failure alerts may share one email, but the failure alert for a target is sent once per failure episode.

## Run and schedule

- `uv run python -m watcher run` performs one daily run.
- `--dry-run` prints the changes and alerts it would send. It sends no email and writes no observations.
- `--fixtures DIR` reads saved HTML files from `DIR` instead of the network. Each fixture is named after the target `id`.
- Schedule the run once a day with cron or launchd.
- Exit code is 0 when the run completes, including runs where some targets failed. Those failures are recorded as `error` rows.
- Exit code is non-zero on any unhandled error.

## Acceptance

- Fixtures for each target parse to the expected price.
- Diff tests cover: increase, decrease, no change, error then recovery.
- Failure rule test covers: 3 consecutive failures send one alert, the alert is not repeated, and recovery resets it.
- A run with zero changes sends no email, checked with a fake SMTP object.
- `--dry-run` sends nothing and writes nothing.