Daily rental listings scraper with neighbourhood price chart

# Rental tracker

## Purpose

Collect rental listings for one city once a day, store them locally, and chart the average price per neighbourhood. Solo project, Python script, no web service.

## Commands

- Create venv: `python -m venv .venv`
- Activate venv: `source .venv/bin/activate` (macOS or Linux) or `.venv\Scripts\activate` (Windows)
- Install: `pip install -r requirements.txt`
- Run tests: `pytest -q`
- Lint: `ruff check .`
- Type check: `mypy src`
- Run collector: `python -m rentals.collect`
- Render chart: `python -m rentals.chart`

## Layout

- `src/rentals/`: package code
- `tests/`: pytest tests
- `tests/fixtures/`: saved HTML used by tests
- `data/`: raw HTML cache and history database (gitignored)
- `output/`: PNG charts and CSV summaries (gitignored)
- `SPEC.md`: what the collector and chart must do
- `PLAN.md`: phases and their goal prompts
- `REVIEW.md`: review rules, imported below

@REVIEW.md

## Scraping rules

- Check robots.txt and the site's terms before adding or changing a source.
- Set a descriptive User-Agent that names the project and gives a contact route.
- Sleep at least 2 seconds between requests to the same site.
- Cache raw HTML responses in `data/` so re-runs do not refetch.
- Stop and report on 403 or 429. Do not retry in a loop.
- Do not add login, CAPTCHA handling, or collection of personal contact data.

## Testing rules

- Tests read saved HTML from `tests/fixtures/`. Never make live requests in tests.
- When a parser changes, update or add its fixture and its test together.
- Run `verify` (the project skill) before reporting a change as done.

## Data conventions

- Store prices as integer currency units per month.
- Record the currency alongside each price.
- Record whether utilities are included, as a nullable field. Use None when the listing does not say.
- Missing fields become None. They must not crash the parser.
- Timestamps are UTC.
- Any schema change needs a migration note in PLAN.md before the code lands.

## Secrets

- Put any API key or proxy credential in `.env`, using `YOUR_API_KEY=placeholder` as the template line.
- Never commit `.env`. Confirm `.gitignore` lists `.env`, `data/` and `output/`.
- Do not print secrets in logs or test output.

## Gotchas

- `ruff` or `pytest` not found usually means the venv is not activated.
- Keep the chart threshold (minimum listings per neighbourhood, default 5) in one constant. Do not hard-code it elsewhere.
- Neighbourhoods below the threshold are left out of the chart and listed in a footnote.

## Conventions

- Keep diffs small. Flag unrelated refactors instead of making them.
- Ask before adding a dependency beyond pytest, ruff and mypy.
- Ask before changing the scheduler setup (cron or Task Scheduler).
- Use sentence case in comments and docs.

---
name: fixture-tester
description: Runs the pytest suite and returns only failing test names and their key error lines, keeping test output out of the main context.
model: haiku
tools: Bash, Read
maxTurns: 10
---

You run the project's test suite and report results briefly. You do not fix code.

Steps:

1. Run `pytest -q` from the project root. Use the venv at `.venv` if it is not already active (`source .venv/bin/activate` on macOS or Linux, `.venv\Scripts\activate` on Windows).
2. If every test passes, reply with one line: `pytest: all tests passed` and the summary count.
3. If any test fails, reply with a list. For each failing test, give:
   - the full test name, for example `tests/test_parser.py::test_price_parsing`
   - the first error line only, such as the assertion or exception message
4. Do not paste the full traceback, stdout, or passing test output.
5. Read a file only when needed to confirm a test name or error location.

Rules:

- Never edit, create, or delete files.
- Never run git commands that change state.
- Never make network requests. Tests use saved fixtures only; if a test tries the network, report it as a failure with the test name.
- If pytest is not found, reply with `pytest not found: check that the venv is active` and stop.

{
  "model": "sonnet",
  "effort": "medium",
  "permissions": {
    "defaultMode": "acceptEdits",
    "allow": [
      "Bash(pytest:*)",
      "Bash(ruff:*)",
      "Bash(mypy:*)",
      "Bash(python -m rentals.*)",
      "Bash(git status:*)",
      "Bash(git diff:*)"
    ],
    "deny": [
      "Bash(rm -rf:*)",
      "Bash(git push --force:*)",
      "Bash(git reset --hard:*)",
      "Read(./.env)",
      "Read(./.env.*)"
    ]
  }
}

---
name: chart-report
description: Change neighbourhood aggregation or chart output without breaking the minimum-listings threshold or the footnote.
when_to_use: Use when asked to change how prices are aggregated per neighbourhood, add a chart statistic, or alter the PNG or CSV summary output.
---

# Chart report

## Steps

1. Read the current summary logic in `src/rentals/` before editing. Note which function builds the per-neighbourhood table and which one draws the PNG.
2. Make the aggregation change in one place. Keep the threshold check in the same place as the aggregation.
3. Regenerate the fixture chart: run `python -m rentals.chart` against `tests/fixtures/`, then check `output/chart.png` and `output/summary.csv`.
4. Check the footnote. It must list every neighbourhood excluded by the threshold, and the list must match the CSV.
5. Update or add the test that checks the threshold and footnote.
6. Run the verify skill.

## Rules

- Report both median and mean price for each neighbourhood that is shown. Do not drop either one.
- The minimum listing count is one constant (default 5). Change it only in that constant, never inline.
- Neighbourhoods below the threshold are left out of the bar chart and named in the footnote. They are not silently dropped.
- Prices are integer currency units per month. Do not convert to floats for display except at the label.
- Do not change the stored schema here. Schema changes go through PLAN.md with a migration note.
- Keep changes small. Flag unrelated refactors in the report instead of making them.

## Report

- Which function changed and why.
- The threshold value used.
- Which neighbourhoods were excluded in the fixture run.
- The verify result.

---
name: scraper-parser
description: Use when writing or fixing a source parser for a rental listings site, working only from saved HTML fixtures and normalised records.
disable-model-invocation: false
---

# Scraper parser

Use this skill to add or fix one parser. Parsers read saved HTML. They never fetch pages themselves during tests.

## Before you start

- Check the site's robots.txt and terms of use. If the source is login-protected, needs a CAPTCHA bypass, or exposes personal contact data, stop. Those are out of scope (see SPEC.md).
- Confirm the source is listed in SPEC.md. If not, ask before adding it.

## Steps

1. Save a fixture first. Save the page HTML to `tests/fixtures/<source>/<name>.html`. Keep one file per listing page type. Do not edit fixtures to fit the parser.
2. Write the parser in `src/rentals/`. It takes HTML and returns a list of normalised records using the shared record dataclass.
3. Add a test in `tests/` that loads the fixture and checks field values. Add one test per important edge case.
4. Run the verify skill. Fix failures before reporting done.

## Checklist

- Every CSS or XPath selector has a comment saying what it targets and when it was checked.
- Missing fields become `None`. A missing field must not raise.
- Price is parsed to an integer number of currency units per month. Record the currency. Note whether utilities are included, or `None` if unknown.
- Each record has: listing id, source, neighbourhood, price per month, bedrooms, url. `scraped_at` is set by the collector, not the parser.
- The parser does not dedupe, write storage, or sleep. Those belong to the collector.
- A parse error on one listing skips that listing and logs it. It does not abort the whole page.
- No live requests in tests. No keys, cookies or personal data in fixtures.

## Done means

- Fixture saved and committed with its test.
- Parser tests pass with `pytest -q`.
- Verify skill passes.
- The report names the source, the fixture file, and any fields that come back as `None` on the fixture.

---
name: verify
description: Use after any code change, and before reporting a phase done, to run the project's lint, type, test and fixture-based chart checks.
---

# Verify

Run these steps in order. Stop at the first failure.

## Before you start

1. Confirm the venv is active. If `ruff` or `pytest` is not found, run `source .venv/bin/activate` (macOS or Linux) or `.venv\Scripts\activate` (Windows), then retry.
2. Never call the network in this skill. All checks use fixtures in `tests/fixtures/`. If a step would make a live request, stop and report it.

## Steps

1. Lint: `ruff check .`
2. Types: `mypy src`
3. Tests: `pytest -q`
4. Fixture chart run:
   - Run `python -m rentals.chart` against `tests/fixtures/`.
   - Confirm `output/chart.png` and `output/summary.csv` now exist and are not empty.
   - If the chart command does not yet accept a fixture input, stop and report that. Add the option in Phase 4 before relying on this step.

## Reporting

- All steps passed: say so in one line and list the four commands run.
- A step failed: report only
  - the first failing command, exactly as run,
  - its key error lines (file, line, message), at most about 10 lines.
- Do not paste full logs unless asked. Do not try to fix failures inside this skill. Report them and let the caller decide.

## Rules

- Do not skip a step to get a green result.
- Do not edit fixtures to make a test pass without saying so in the report.
- Do not add dependencies to make a check run.

# Install and run the rental tracker

This guide is for one person with some Python experience. It sets up a folder that collects rental listings once a day and charts average prices by neighbourhood.

## Prerequisites

- Python 3.11 or newer
- Git
- Claude Code 2.1.296 or newer
- A Claude account

Check your versions:

```
python --version
git --version
claude --version
```

## Step 1: Create the project folder

Either clone the project repository, or create a new folder:

```
mkdir rental-tracker
cd rental-tracker
git init
```

Copy these into the folder: CLAUDE.md, SPEC.md, PLAN.md, REVIEW.md, INSTALL.md and the .claude folder.

## Step 2: Create and activate a virtual environment

```
python -m venv .venv
```

On macOS or Linux:

```
source .venv/bin/activate
```

On Windows:

```
.venv\Scripts\activate
```

Your prompt should now start with `(.venv)`.

## Step 3: Install dependencies

```
pip install -r requirements.txt
```

Phase 1 of PLAN.md creates requirements.txt. If the file is missing, run Claude Code first (step 5) and start Phase 1.

## Step 4: Add a placeholder .env file

Create a file named `.env` in the project root:

```
YOUR_API_KEY=placeholder
```

Only add this if a source needs a key. Never commit .env. Make sure .gitignore lists `.env`, `data/` and `output/`.

## Step 5: Start Claude Code

From the project folder:

```
claude
```

Run `/init` only if CLAUDE.md is missing. The copied CLAUDE.md should already be there.

Run `/verify` once to confirm the verify skill works. It runs ruff, mypy and pytest, then checks the chart command against the fixtures.

## Step 6: Optional plugins

Inside Claude Code, install the recommended plugins:

```
/plugin install code-review@claude-plugins-official
/plugin install commit-commands@claude-plugins-official
/plugin install feature-dev@claude-plugins-official
```

To add the data-analysis plugins from the knowledge-work marketplace, run this once:

```
/plugin marketplace add anthropics/knowledge-work-plugins
```

## Step 7: Build the project

Paste the kickoff prompt from the plan into Claude Code, then work through PLAN.md one phase at a time. Each phase ends with a `/goal` check and a verify run.

## Step 8: Schedule the daily run

Replace `YOUR_PATH` with the full path to your project folder in every example.

### macOS or Linux (cron)

Open your crontab with `crontab -e` and add:

```
30 7 * * * cd YOUR_PATH && .venv/bin/python -m rentals.collect >> YOUR_PATH/data/collect.log 2>&1
35 7 * * * cd YOUR_PATH && .venv/bin/python -m rentals.chart >> YOUR_PATH/data/collect.log 2>&1
```

This runs the collector at 07:30 and the chart at 07:35 every day.

### Windows (Task Scheduler)

Create the collector task from a Command Prompt:

```
schtasks /Create /SC DAILY /ST 07:30 /TN "RentalCollect" /TR "YOUR_PATH\.venv\Scripts\python.exe -m rentals.collect"
```

Create the chart task at 07:35 the same way, using `rentals.chart` in place of `rentals.collect`.

In Task Scheduler, open each task and set **Start in** (under the Actions tab) to `YOUR_PATH`. The collector needs this to find its files.

## Step 9: Optional formatting hook

After Phase 1, you can run ruff format on each Python file Claude edits. Add this to `.claude/settings.json` under a `hooks` key. It needs `jq` installed.

```
"hooks": {
  "PostToolUse": [
    {
      "matcher": "Edit|Write",
      "hooks": [
        {
          "type": "command",
          "command": "f=$(jq -r '.tool_input.file_path // empty'); case \"$f\" in *.py) ruff format \"$f\";; esac"
        }
      ]
    }
  ]
}
```

## What each file is for

- CLAUDE.md: short memory Claude reads each session, with commands and rules.
- SPEC.md: what the collector and chart must do, with acceptance criteria.
- PLAN.md: five phases, each with a bounded check.
- REVIEW.md: the rules Claude applies when reviewing changes.
- INSTALL.md: this guide.
- .claude/settings.json: project settings, permissions and the optional hook.
- .claude/skills/verify/SKILL.md: the commands that prove a change works.
- .claude/skills/scraper-parser/SKILL.md: steps for writing a source parser.
- .claude/skills/chart-report/SKILL.md: steps for changing aggregation or the chart.
- .claude/agents/fixture-tester.md: a small subagent that runs tests and reports only failures.

## Respect site terms

Before you add or change a source:

1. Open the site's `/robots.txt` and check which paths are disallowed.
2. Read the site's terms of use and confirm scraping is allowed.
3. Keep the 2-second delay between requests.
4. Do not scrape login-protected pages or bypass CAPTCHAs.

## Troubleshooting

- **`ruff` or `pytest` not found:** the virtual environment is not active. Run the activate command from step 2 again.
- **Site returns 403:** slow down, check the site's terms, and stop requests to that source if it keeps refusing. Do not change the User-Agent to get past the block.
- **Scheduled run does nothing:** check the log in `data/collect.log`. On Windows, confirm **Start in** is set to your project path.
- **`/verify` fails on a fresh install:** run `pip install -r requirements.txt` with the venv active, then try again.

# Build plan

Five phases. Finish and verify each phase before starting the next.

## How to run a phase

1. Run `/clear` to start a fresh context.
2. Read SPEC.md and CLAUDE.md.
3. Paste the phase's `/goal` prompt.
4. Run `/verify` before accepting the phase as done.

Only the files listed for a phase may change during that phase. If a change outside the list seems necessary, stop and ask.

## Verify command

`/verify` runs, in order:

1. `ruff check .`
2. `mypy src`
3. `pytest -q`
4. `python -m rentals.chart` on the fixture database (from Phase 4 onward)

It never calls the network.

## Phase overview

| Phase | Outcome | Turn limit |
|---|---|---|
| 1 Setup | Empty package with working tools | 15 |
| 2 Parser | One parser per source, tested on saved HTML | 25 |
| 3 Storage and dedupe | SQLite history, no duplicate rows on reruns | 20 |
| 4 Chart and summary | PNG and CSV with the 5-listing threshold | 20 |
| 5 Schedule and failures | Polite fetching, per-source isolation, logs | 20 |

## Phase 1: Setup

Goal: a fresh project with a working package skeleton and all checks passing.

Files it may touch:

- requirements.txt
- pyproject.toml
- .gitignore
- src/rentals/__init__.py
- tests/__init__.py
- tests/test_smoke.py

Verify: steps 1 to 3 of `/verify`.

```
/goal Done when `ruff check .`, `mypy src` and `pytest -q` all exit 0 in the project root, and the final output lines of each command are visible in this conversation. Required state: requirements.txt pins only pytest, ruff and mypy; src/rentals/__init__.py exists and is empty; tests/test_smoke.py has one passing test; .gitignore lists .venv, .env, data/ and output/. Constraints: no scraping code; no dependencies beyond pytest, ruff and mypy; touch only the files listed in PLAN.md Phase 1; do not create or commit any .env file or secrets. Stop after 15 turns even if not done, and report the exact commands run and their results.
```

## Phase 2: Parser

Goal: one parser per source that turns saved HTML into normalised records. No network.

Files it may touch:

- src/rentals/record.py
- src/rentals/parsers/__init__.py
- src/rentals/parsers/source_1.py
- src/rentals/parsers/source_2.py
- tests/fixtures/source_1_*.html
- tests/fixtures/source_2_*.html
- tests/test_parsers.py

Fixtures: the user saves one real search results page per source into tests/fixtures/. If a fixture is missing, Claude may write a small hand-made HTML file instead. It must start with a comment marking it as synthetic.

Verify: steps 1 to 3 of `/verify`.

Use the `scraper-parser` skill for each source.

```
/goal Done when `pytest -q` passes and tests/test_parsers.py contains at least 6 test functions covering both sources, and the pytest output is visible in this conversation. Required state: each parser returns the normalised record from src/rentals/record.py; price_per_month is an integer per month; missing fields become None, not exceptions; each selector has a comment; tests read only files in tests/fixtures/. Constraints: no network calls anywhere, including tests; do not add storage or chart code; no new dependencies; touch only the files listed in PLAN.md Phase 2. Also run `ruff check .` and `mypy src` and show they exit 0. Stop after 25 turns even if not done, and report the commands run and their results.
```

## Phase 3: Storage and dedupe

Goal: store listings in SQLite with history, and make reruns safe.

Files it may touch:

- src/rentals/storage.py
- src/rentals/collect.py
- tests/test_storage.py
- tests/test_collect.py
- PLAN.md (Migration notes section only)

Design: database file data/listings.db. Table listings with primary key (source, listing_id, snapshot_date). The collect command reads saved HTML from a folder passed with `--html-dir`, so it runs offline.

Verify: steps 1 to 3 of `/verify`.

```
/goal Done when `pytest -q` passes and a test proves that running the collector twice on the same fixture folder on the same snapshot date leaves the row count unchanged, and that a run with a later snapshot date adds rows. The pytest output must be visible in this conversation. Required state: storage.py creates the listings table with primary key (source, listing_id, snapshot_date); collect.py runs with `python -m rentals.collect --html-dir tests/fixtures --db data/listings.db`; PLAN.md has a Migration notes section with an entry for the new table. Constraints: no network calls; no changes to the schema after the migration note is written; no new dependencies beyond the standard library; touch only the files listed in PLAN.md Phase 3. Run `ruff check .` and `mypy src` and show they exit 0. Stop after 20 turns even if not done, and report the commands run and their results.
```

## Phase 4: Chart and summary

Goal: aggregate the latest snapshot by neighbourhood and produce the PNG and CSV.

Files it may touch:

- src/rentals/aggregate.py
- src/rentals/chart.py
- tests/test_aggregate.py
- tests/test_chart.py
- tests/fixtures/listings_fixture.db (built once from the Phase 3 collector, then committed)

Verify: all four steps of `/verify`. The chart step runs `python -m rentals.chart --db tests/fixtures/listings_fixture.db --out output/`.

Use the `chart-report` skill for the aggregation changes.

```
/goal Done when `python -m rentals.chart --db tests/fixtures/listings_fixture.db --out output/` produces output/chart.png and output/summary.csv, `pytest -q` passes including a threshold test, and the command output and the first lines of summary.csv are visible in this conversation. Required state: MIN_LISTINGS is one constant set to 5; neighbourhoods below it are absent from the chart and the CSV and are named in the chart footnote; the chart and CSV both report median and mean rounded to whole currency units; the CSV has the columns neighbourhood, listing_count, median_price, mean_price, currency, snapshot_date. Constraints: no network calls; read only the latest snapshot; no new dependencies beyond matplotlib and pandas if they are already in requirements.txt, otherwise ask first; touch only the files listed in PLAN.md Phase 4. Run `ruff check .` and `mypy src` and show they exit 0. Stop after 20 turns even if not done, and report the commands run and their results.
```

## Phase 5: Schedule and failure handling

Goal: fetch live pages politely, isolate failures per source, and log them.

Files it may touch:

- src/rentals/fetch.py
- src/rentals/collect.py
- src/rentals/log.py
- tests/test_failures.py
- INSTALL.md (scheduler section only)

Design:

- fetch.py uses the standard library only (urllib and urllib.robotparser).
- It checks robots.txt, sends the User-Agent from SPEC.md, waits at least 2 seconds between requests, and caches raw HTML in data/raw/ by source and date.
- Each source runs inside its own try block.
- The fetcher is passed in, so tests can replace it with a fake.

Verify: all four steps of `/verify`.

```
/goal Done when `pytest -q` passes, including a test where one source's fetcher raises an error and the other source still writes rows to the database, and a test that checks the log line for a failed source contains the source name and the HTTP status. The pytest output must be visible in this conversation. Required state: fetch.py enforces robots.txt, the User-Agent, the 2-second delay and the raw HTML cache; the collector exits non-zero when any source failed; INSTALL.md's cron and Task Scheduler examples match the real command and still use YOUR_PATH placeholders. Constraints: tests must never open a network socket; no new dependencies; do not change the scheduler setup beyond the INSTALL.md examples without asking; touch only the files listed in PLAN.md Phase 5. Run `ruff check .` and `mypy src` and show they exit 0. Stop after 20 turns even if not done, and report the commands run and their results.
```

## After phase 5

- Run `/verify` once more from a clean shell.
- Run `/code-review` on the diff before committing.
- Run the daily collector by hand for a few days before trusting the chart.
- Fill in SOURCE_SITE_1 and SOURCE_SITE_2 in SPEC.md only after checking each site's terms.

# Review rules

Apply these checks to every diff, whether Claude wrote it or a teammate did.

## Scraping politeness

- Rate limit: at least 2 seconds between requests to the same site.
- User-Agent: descriptive, set on every request.
- robots.txt and site terms checked before a source is added or changed.
- Raw HTML cached, so reruns and tests do not re-fetch pages.

## Tests

- No live network calls in tests. Use files in tests/fixtures/.
- Parser changes must update or add fixtures.

## Data correctness

- Prices are integer units per month.
- Currency and whether utilities are included are recorded.
- Missing fields become None, not crashes.
- Re-runs do not double count. Dedupe by listing id.
- Schema changes need a migration note in PLAN.md.

## Chart and threshold

- Neighbourhoods below the minimum listing count are excluded from the chart.
- Excluded neighbourhoods are listed in the footnote.
- Median and mean are both reported.
- The threshold lives in one constant.

## Secrets and files

- No keys, cookies or personal data committed.
- .gitignore covers .env, data/ and output/.

## Errors

- One source failing must not abort the run.
- Error messages include the source name and HTTP status.

## Diffs

- Keep diffs small.
- Flag unrelated refactors instead of making them.

# Rental listings tracker spec

## Scope

Collect rental listings for one city once per day from two sources, keep a history, and chart average rent per neighbourhood.

- Target city: CITY_NAME (user to fill in)
- Source 1: SOURCE_SITE_1 (user to fill in the search page URL)
- Source 2: SOURCE_SITE_2 (user to fill in the search page URL)

## Record fields

Each listing is one record:

- listing_id: text, unique within its source
- source: SOURCE_SITE_1 or SOURCE_SITE_2
- neighbourhood: text, None if missing
- price_per_month: integer, currency units per month
- currency: ISO code, for example EUR or GBP
- utilities_included: true, false or None
- bedrooms: integer, None if missing
- url: listing page URL
- scraped_at: UTC timestamp, ISO 8601
- snapshot_date: the UTC date of the run that stored the row

## Daily run

- The collector runs once a day through the scheduler (see INSTALL.md).
- Rows are unique by source, listing_id and snapshot_date.
- A second run on the same day updates that day's rows and adds no duplicates.
- A run on a later day adds new rows, so price changes over time stay visible.
- The latest snapshot is the highest snapshot_date in the store.

## Chart

- Uses the latest snapshot only.
- Bar chart of median and mean price per neighbourhood. Both are shown.
- Prices are rounded to the nearest whole currency unit.
- Neighbourhoods with fewer than 5 listings are left out of the bars and named in a footnote.
- The threshold is one constant, MIN_LISTINGS, in the aggregation code.
- Chart title includes the city name and the snapshot date.

## Outputs

- output/chart.png: the bar chart.
- output/summary.csv: one row per included neighbourhood with columns neighbourhood, listing_count, median_price, mean_price, currency, snapshot_date.

## Failure behaviour

- One source failing must not stop the other source.
- Each failure is logged with the source name and the HTTP status. If there is no status, log the error type.
- The run still stores rows from the sources that worked.
- The run exits with a non-zero code if any source failed, so the scheduler shows it.

## Politeness

- Check robots.txt and the site's terms before adding a source.
- Send a descriptive User-Agent.
- Wait at least 2 seconds between requests to the same site.
- Cache raw HTML so reruns and tests never fetch the same page twice.

## Out of scope

- Login-protected sites
- CAPTCHA bypass
- Personal contact data (landlord names, phone numbers, emails)
- Web dashboard, notifications, or alerts

## Acceptance

- Ruff, mypy and pytest all exit 0.
- A fixture-based run of the chart command produces output/chart.png and output/summary.csv with no network access.
- Two runs on the same fixture on the same day produce no duplicate rows.
- A neighbourhood with 4 listings is absent from the chart and summary and is named in the footnote.
- When one source raises an error, the other source's rows are still written.