fuzzyworld-tools / commit
Issues: bug reports by email become issue pages; run records and replay links; fw deploy extra files
.gitignore | 6 + PLAN.md | 56 +- README.md | 38 +- bin/fw | 3 +- lib/deploy.mjs | 59 +- server/comments.py | 29 +- server/fw | 21 +- server/gen.py | 4 +- server/install.sh | 2 +- server/issues.py | 896 +++++++++++++++++++++++++++ server/jobs/crontab | 1 + server/jobs/issues-ingest.sh | 9 + server/site/htaccess | 5 + server/site/style.css | 2 +- tests/deploy.test.mjs | 44 ++ tests/fixtures/issues/explodorama-report.eml | 147 +++++ tests/fixtures/issues/q101-record.json | 1 + tests/test_issues.py | 873 ++++++++++++++++++++++++++ 18 files changed, 2151 insertions(+), 45 deletions(-) diff --git a/.gitignore b/.gitignore index ae7c391..50b24da 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,9 @@ __pycache__/ *.pyc .DS_Store .playwright-mcp/ + +# credentials: never commit +imap.json +*-imap.json +.env +.env.* diff --git a/PLAN.md b/PLAN.md index 0d6c2f8..a08602e 100644 --- a/PLAN.md +++ b/PLAN.md @@ -62,7 +62,7 @@ The order goes from cheap to expensive, and each step reuses what the one before - heatmap: a commit-activity SVG per repo and site-wide 4. **Wiki**: render a repo's `docs/` (or `wiki/`) folder with a sidebar. Import the GitLab wikis in `~/code/gitlab-export/*/wikis.json`. Static rendering only; editing comes with pull requests. -5. **Issues**: see [Issues](#issues). The first feature that reads email; builds the mailbox plumbing. +5. **Issues** (built 2026-11-03): see [Issues](#issues). The first feature that reads email; builds the mailbox plumbing. 5b. **Blog comments** by email (see [Comments](#comments)): right after Issues, because it reuses the mailbox plumbing. 6. **Pull requests**: see [Pull requests](#pull-requests). Last, because it reuses the issues mailbox plumbing and needs the CI runner (approved CI runs on contributed code). @@ -308,19 +308,20 @@ because it runs on a machine in my home (part of Pull requests). ## Issues -- Reports arrive at `bugs-k7q4+<repo>@fuzzyworld.net` (already live). +- Reports arrive at `bugs-k7q4+<repo>@fuzzyworld.net` (only the address was live before 2026-11-03; the ingest and + the pages were built that day, see the task list under Report replays). - A cron job reads the mailbox (IMAP, Python standard library) and renders each thread as static pages. - Reporters' email addresses are never shown. - **Configurable per repo**, the same way as visibility: `git config fuzzyworld.issues public|reviewed|private`, set with `fw issues <repo> <mode>`: - - `public`: every report is published as it arrives + - `public`: a report whose sender passed email authentication is published as it arrives (an unauthenticated one waits for approval) - `reviewed`: a report is published only after I mark it public - - `private`: nothing is published; reports only reach my mailbox + - `private`: nothing is published; reports are still stored in `~/fw/issues` (and listed by `fw issues list`) - anything other than exactly `public` or `reviewed` counts as `private`, so nothing is published by accident. A private repo's issues are always private, whatever this says. - Import the GitLab issues and notes in `~/code/gitlab-export/*/issues.json`. -### Report replays (Q104; planned 2026-11-03, nothing built here yet) +### Report replays (Q104; planned 2026-11-03, built 2026-11-03 in this repo) Idea from the Explodorama bug mails: a report can carry a **machine-readable payload** (for the game: the run record, a few KB of JSON: seed, rules version and actions), and the issue page can then show *what the reporter @@ -334,11 +335,13 @@ saw* as a looping animation. The game side (the replay page, its limits, how old - **Store it beside the issue.** For a repo whose issue mode allows publishing, the cron writes the payload as a static file next to the issue page (`…/issues/<n>/record.json`), under the same visibility rule as the report itself: `public` as it arrives, `reviewed` only after I mark it public, `private` never. The payload - is validated first (size ≤ 64 KB, parses as JSON, expected keys); anything else stays only in my mailbox. + is validated first (size ≤ 64 KB, parses as finite JSON, an integer `seed` and `simVersion`, a list `actions`) and only the keys + `simVersion, seed, startLevel, startCharge, options, actions` are kept; a record with an address-like string is refused. + Anything else stays only in my mailbox. Reporter emails are still never shown; the device string stays out of the page by default. - **Link a viewer, don't build one.** An app announces its viewer in `package.json`, for example `"fuzzyworld": { "replay": "replay.html?report={record}" }`; with it, an issue page that has a payload shows - "Watch what happened", a link (or an iframe on the same origin) to that page of the app, with `{record}` set + "Watch what happened", a plain link to that page of the app, with `{record}` set to the stored file. Without the key, nothing changes. This keeps the core free of any game knowledge (🔌 plugin-shaped: an app opts in). - **CSP.** The viewer is the app's own script on the app's own origin, so the site-wide `script-src 'self'` @@ -351,18 +354,51 @@ saw* as a looping animation. The game side (the replay page, its limits, how old served locally with the dev page's `&rules=` mode, which can use any commit from git), and the buttons that mark a report public, close it, or turn it into a regression fixture in the app's repo (`test/fixtures/`). The panel is a thin local web page over `fw` commands; it holds no state of its own. + **Decided 2026-11-03: local-only, never hosted on DreamHost.** Every panel action is privileged (publish, + merge, deploy), and a login form on shared hosting would make the most valuable target on the site; the + local panel's security is that it is unreachable, and its authority is the ssh key already held. It also + keeps "DreamHost only serves files". Cost: no triage from a phone (reports still arrive as email, and the + replay link works anywhere because it is a static page). Possible later add, not planned: one narrow + endpoint behind a per-report signed link in the report mail ("mark this report public"), with no login + and no general admin surface. + +**This repo's task list (2026-11-03; Pavel: "add to the plan, start implementing").** Items 1-4 were built +on 2026-11-03 (`server/issues.py`, `lib/deploy.mjs`); the first line of each says so. Not yet committed or +deployed is deliberately not recorded here. In dependency order: + +1. **Issues** (built 2026-11-03; Next 5): cron ingest of the bugs mailbox over IMAP (reuse the `comments.py` / + `comments-ingest.sh` plumbing), one static page per thread under `…/<repo>/issues/<n>/`, + `fw issues <repo> public|reviewed|private` (anything else = private), reporter addresses never shown, and + a "mark public" action for `reviewed` mode. +2. **Record extraction** (built 2026-11-03): decode the quoted-printable mail, find the `Run record: {…}` line, validate + (≤ 64 KB, valid JSON, expected keys), write `record.json` beside the issue under the issue's visibility + rule; device string stays out of the page. Test with a real report (Explodorama's + `test/fixtures/q101-bomb-pass.json` and its `.eml`). +3. **App opt-in** (built 2026-11-03): a `replay` key in the `"fuzzyworld"` block of `package.json` + (`"replay": "replay.html?report={record}"`); the issue-page generator reads it and shows "Watch what + happened". No key, no change. +4. **`fw deploy` carries extra files, kept forever** (built 2026-11-03): a per-app glob list + (`"extra": ["sim/*.js"]`) published with the deploy; extra files are protected from `--delete` through an rsync + filter (uploaded and updated, never removed), so a later deploy never deletes old rules versions. + +No CSP change (the viewer is the app's own script) and no server code on DreamHost. The admin panel (Q102, +local-only) comes after all four. **Order:** (1) the game's production replay page and limits (its PLAN §20.2 steps 1-2), by hand, with a stored record; (2) the cron extracting and publishing `record.json` and the `replay` key; (3) per-rules-version sim files in `fw deploy`; (4) the panel. Each step is useful alone. -**Open:** whether the viewer opens in the same page (iframe) or its own; a `reviewed` report's payload being -published at the same moment as its text; how long old `sim/` files are kept; compact records in the game's -`mailto:` link (game PLAN §20.2 step 4) before or after step 2. +**Decided 2026-11-03 (Pavel):** the viewer opens as its own page (a plain link, no iframe); a `reviewed` +report's record is published at the same moment as its text; old `sim/` files are kept forever; compact records +in the game's `mailto:` link (game PLAN §20.2 step 4) come after the cron (step 2); the published page defaults +to the plain recorded run, with comparison as an option (game PLAN §20.2 step 5). Nothing open here. ## Pull requests - Contributions arrive at `contrib-68uq+<repo>@fuzzyworld.net` as a branch URL or patches (already live). + **Forward-only today** (2026-11-03): like `bugs-k7q4`, it must be recreated as a fully hosted mailbox before a + cron job can read it over IMAP, and so that the receiving server's `Authentication-Results` header is trusted + (a forwarded copy carries the destination's header instead). Give it its own account, never the comments one. - A cron job fetches each one into `refs/contrib/<n>` in the bare repo, never onto `main`. - It renders a review page: the diff, CI results (after I approve the run), and the discussion from email replies. - **Transcripts.** A submitter can attach the AI conversation behind the change, so the reviewer sees exactly diff --git a/README.md b/README.md index eb53309..54bbb26 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,9 @@ Add to the app repo's `package.json`: "checks": ["typecheck", "test"], // scripts that must pass (default: whichever of typecheck/lint/test exist) "build": "make public", // default: `<pm> run build` "out": "dist", // default: dist, else public, else build - "keep": ["my-game.apk"] // files on the server a deploy must not delete or replace (e.g. a download built elsewhere) + "keep": ["my-game.apk"], // files on the server a deploy must not delete or replace (e.g. a download built elsewhere) + "extra": ["sim/*.js"], // build-output files every later deploy keeps: uploaded and updated, never deleted (e.g. old rules versions) + "replay": "replay.html?report={record}" // a page of this app that plays a bug report's run record; issue pages link to it (see Issues) }, "devDependencies": { "fuzzyworld-tools": "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0" } ``` @@ -71,7 +73,7 @@ Then `pnpm run deploy` / `npm run deploy`. **`fw deploy`**: 2. installs with the lockfile's package manager (`pnpm install --frozen-lockfile`, `npm ci`, `yarn install --frozen-lockfile`) 3. runs `prebuild`, then the `checks`, then the build. Any failure stops the deploy. 4. checks the output: every same-site URL in `index.html`, the web manifest's `id`/`scope`/`start_url`, and every `serviceWorker.register(...)` script and scope must stay inside `/<app>/`. That keeps apps on the shared origin from capturing each other, and makes each one install separately on Android. Any problem stops the deploy. -5. `rsync --delete` to `apps.fuzzyworld.net/<app>/` (except `keep` paths), writes `.fuzzyworld.json` (commit, time; not web-readable), and refreshes the apps index and the repo's code page +5. `rsync --delete` to `apps.fuzzyworld.net/<app>/` (except `keep` paths; `extra` files are protected from `--delete` by an rsync filter, so they are uploaded and updated but never removed), writes `.fuzzyworld.json` (commit, time, and the `replay` template; not web-readable), and refreshes the apps index and the repo's code page 6. prints the URL `--dry-run` does everything except upload. @@ -90,6 +92,8 @@ On the laptop, `bin/fw` runs `deploy` locally and forwards repo commands to Drea | `fw repoint [DIR] [--https] [--dry-run]` | for each repo in DIR whose `origin` is on gitlab.com: rename it `gitlab`, add `origin` → `dh:git/<repo>.git` (or read-only `https://code.fuzzyworld.net/<repo>.git` with `--https`, public repos only), fetch, and retarget branch tracking. Changes only remotes; checks that uncommitted work, HEAD and stashes are unchanged | | `fw usage [--all\|--now]` | Claude Code token use per session from `~/.claude/projects` (runs locally): messages, current context size, cache-read, output (cache-read and output include subagents). `--now` prints one line for the newest session, to decide when to put it down | | `fw comments list\|show\|approve\|reject\|remove\|untrust\|publish\|export\|ingest\|imap-check` | moderate blog comments received by email (see [Blog](#blog)); runs on the server | +| `fw issues <repo> public\|reviewed\|private` | set who sees the repo's bug-report issue pages (`git config fuzzyworld.issues`; default private) and republish them | +| `fw issues list\|show\|approve\|hide\|close\|reopen\|publish\|publish-all\|ingest\|imap-check` | moderate bug-report issues received by email (see [Issues](#issues)); runs on the server | | `fw list` | every repo, its visibility and last commit | | `fw new <name> [--public] [--description TEXT]` | create `~/git/<name>.git` on DreamHost; **private unless `--public`** | | `fw public <name>` | publish the source on code.fuzzyworld.net right away | @@ -112,7 +116,7 @@ Each bare repo stores its own setting: `git config fuzzyworld.visibility public| **Anything other than exactly `public` is private**, including a missing value, so a repo is never published by accident. A private repo has no pages, no clone URL and no index entry. Pushing to it publishes nothing. -Other settings in the repo config: `fuzzyworld.description`, `fuzzyworld.ci` (`off`: pushes queue no CI jobs), plus `fuzzyworld.gitlab-id` / `fuzzyworld.gitlab-path` for repos migrated from GitLab. +Other settings in the repo config: `fuzzyworld.issues` (`public`, `reviewed` or anything else = private: who sees bug-report issue pages, see [Issues](#issues); a private repo never shows issues), `fuzzyworld.description`, `fuzzyworld.ci` (`off`: pushes queue no CI jobs), plus `fuzzyworld.gitlab-id` / `fuzzyworld.gitlab-path` for repos migrated from GitLab. ## Layout on DreamHost @@ -130,6 +134,7 @@ Other settings in the repo config: `fuzzyworld.description`, `fuzzyworld.ci` (`o | `~/fw/pylib/` | Python-Markdown 3.7 (README rendering) and Pygments 2.21.0 (syntax highlighting), `pip install --target`; `install.sh` adds both (the landing page needs Markdown) | | `~/fw/queue/pending/`, `running/` | CI jobs, one JSON file each (see [CI](#ci)); `running/<job>.beat` is the runner's heartbeat, `declined/` the runners that gave a job back | | `~/fw/runners/<runner>` | one file per runner: its tags, and (by its time) when it last checked in | +| `~/fw/issues/` | bug-report issues: stored reports, `secret`, `imap.json`, locks (see [Issues](#issues)); outside every web root | | `~/fw/ci/<name>/<sha>/` | CI results uploaded by runners: `pipeline.json` and one log per job | | `~/apps.fuzzyworld.net/_shots/<app>/` | screenshots of each app, uploaded by runners (`shots.json` lists them) | | `~/apps.fuzzyworld.net/vscode/` | VS Code in the browser (`vscode-web/install.sh`; not listed on the apps index) | @@ -148,6 +153,7 @@ Every repo uses the shared hook via `core.hooksPath=~/fw/tools/server/hooks`. `p | `/<name>/blame/<path>.html` | the commit that last changed each line (text files ≤ 256 KB) | | `/<name>/raw/<path>` | raw copies of images only (png, jpg, gif, webp, ico, avif, bmp, svg; ≤ 2 MB) | | `/<name>/log/` | the branch's commits (up to 2000) | +| `/<name>/issues/`, `/<name>/issues/<n>/` | bug-report issues and one issue's page (only when the repo's `fuzzyworld.issues` mode and the issue's state allow it; see [Issues](#issues)) | | `/<name>/commit/<sha>.html` | one commit with its diff (truncated at 300 KB); a page for every commit on every branch, up to 2000 per repo | | `/<name>/refs/` | branches and tags | | `/<name>/usage/` | Claude Code token usage per commit and per session, read from `Usage:` trailers in commit messages (only for repos that have them; see below) | @@ -199,6 +205,8 @@ Jobs live in `server/jobs/`. `install.sh` writes `server/jobs/crontab` into the |---|---|---|---| | `north-star-cleanup.sh` | Tuesdays 06:00 UTC (was the GitLab schedule) | updates `~/fw/jobs/north-star` to `main`, then runs `scripts/clean-old-firebase.sh --days=30` as a preview and then `--apply` | `~/fw/logs/cron-north-star-cleanup.log` | +| `issues-ingest.sh` | hourly | `issues.py ingest`: turns new mail in the bugs mailbox into issues and republishes the touched repos (quiet unless something happened) | `~/fw/logs/cron-issues-ingest.log` | + Run one by hand: `ssh dh '~/fw/tools/server/jobs/north-star-cleanup.sh --now --preview-only'` (or without `--preview-only` for the real thing). Secrets are in `~/.config/<project>.env` on DreamHost (mode 600, never in a repo or the web root): `north-star.env` holds `FIREBASE_TOKEN`. @@ -291,6 +299,30 @@ pictures go to `apps.fuzzyworld.net/_shots/<app>/` and appear on the apps index Without runner screenshots, the images in a public repo's `screenshots/` folder are shown instead. +## Issues + +Bug reports arrive by email at `bugs-k7q4+<repo>@fuzzyworld.net` (the address is on every repo's summary page; the repo name is part of it, and a repo that does not exist under `~/git` sends the mail to the bad folder). `server/issues.py` (`fw issues ...`, Python standard library only) turns each report into an issue and, when allowed, a static page. It reuses the comments plumbing in `comments.py` (IMAP or Maildir source, `Authentication-Results` check, locks, atomic JSON files). + +**Modes.** `fw issues <repo> public|reviewed|private` sets `git config fuzzyworld.issues` in the bare repo and republishes. Anything other than exactly `public` or `reviewed` is `private`, and a repo that is itself private (`fuzzyworld.visibility`) never shows issues, whatever its mode. + +| Mode | An issue's page appears | +|---|---| +| `private` | never; reports are stored and listed by `fw issues list` only | +| `reviewed` | after `fw issues approve <repo> <n>` | +| `public` | as soon as its first message passes email authentication (unauthenticated mail is stored with `authenticated: false` and waits for `approve`) | + +In `public` and `reviewed` mode each sender may send 5 messages per 24 hours (the rest goes to the bad folder). + +`fw issues hide <repo> <n>` takes an issue off the site in any mode and keeps it off until `approve`. `close` / `reopen` change the state shown on the page. A reply (its `In-Reply-To` or `References` matches an earlier message of an issue of the same repo) is added to that issue. `approve` marks the messages the issue has at that moment; a message that arrives later is shown only in `public` mode and only if its sender passed authentication, or after another `approve`. An unauthenticated reply is never shown automatically. A mail that repeats a stored Message-ID or a stored byte-for-byte copy is ignored. + +**Pages.** `/<repo>/issues/` lists the visible issues, `/<repo>/issues/<n>/` shows one. Every publish rebuilds the whole `issues/` folder, so hiding removes the page; when the folder appears or disappears the summary page is rebuilt so its "Issues" link follows. The reporter is always shown as "Reporter". **Privacy rules:** no address, name, Message-ID or sender hash is written to a page; every address-shaped string in the title and text is replaced by `[address hidden]`; lines starting `Device:` or `Run record:` are left out, and so are the lines that wrap a long `Device:` value (following lines are dropped while they are indented or look like a user agent: `token/version`, parentheses or semicolons; a blank line, a template label such as `Seed ...`, or plain prose ends the block, so the reporter's own words are kept); format=flowed text is re-joined first (an empty line always ends a paragraph) and the text is NFKC-normalized before any filtering (so fullwidth `Ｄevice:` or `＠` count); the text is cut at the start of a quoted reply so quoted names never publish: an attribution line in any language (a colon line with an address, or an `On`/`Am`/`Le`/`El`/`Op`... line with a date that says wrote/schrieb/a écrit/schreef/escribió; any such line directly followed by a quoted `>` line), a wrapped attribution (up to three lines ending in a colon, directly followed by a quoted line, or ending in a separate `wrote:` line), a `-----Original Message-----` line, or a block of at least three `Label: value` lines that is a mail header (`From:`/`Von:`/`De :`..., with an address or three header labels; an Outlook `____` line before it is dropped); a colon line followed only by quoted lines at the end of the text is cut too, but never when it would leave nothing, and prose such as `I pressed <Enter> and wrote:` stays; `delsp=yes` is honoured when re-joining flowed text; all text is HTML-escaped inside `<pre class="issue-text">` and no page has a script. + +**Run records.** If the text part holds a line `Run record: {json}` (the Explodorama bug-report mail does), it is read from the full decoded part (the text shown is cut to 4000 characters, the record is not) and checked: at most 64 KB, valid JSON, nesting at most 32 deep, an object with integer `seed` and `simVersion` and a list `actions`, finite numbers only. Only the keys `simVersion`, `seed`, `startLevel`, `startCharge`, `options` and `actions` are kept (all others are dropped), and a record with an address-like string anywhere in it is refused. A run record split over several lines by a mail client is not recognised. The first line that passes wins, so the mail's later "too long for an email link" footer is ignored; if the decoded text has none, the raw mail is decoded as quoted-printable and searched again. The record is stored re-serialized, never as mail text, and only the first record of an issue is kept (a record in an unauthenticated reply is ignored; every refusal is logged in `rejected.log`). A record is as public as the message that carried it: one in a reply is published only once that reply is shown (after `approve` in `reviewed` mode; at once in `public` mode for an authenticated sender). It is published as `/<repo>/issues/<n>/record.json` at the same moment as the issue page (`approve` publishes both in `reviewed` mode; `hide` removes both). `server/site/htaccess` lets `https://apps.fuzzyworld.net` read it (CORS) and sends `no-cache`. + +**Replay link.** If a deployed app of the repo (`apps.fuzzyworld.net/<app>/.fuzzyworld.json` with `repo` equal to the repo and an `index.html`) has a `replay` template (the `"fuzzyworld"` block of its `package.json`, e.g. `"replay": "replay.html?report={record}"`), a visible issue with a record shows "Watch what happened": a plain link to `https://apps.fuzzyworld.net/<app>/<replay>` with `{record}` replaced by the URL-encoded address of `record.json`. The app named like the repo is preferred, so a `-next` copy is not chosen. It is a link, not an iframe (the site's CSP blocks framing). No `replay` key, no link. + +**State** is in `~/fw/issues/` (override with `FW_ISSUES_DIR`), outside every web root: `repos/<repo>/<n>/issue.json` and `record.json`, `secret` (salt for sender hashes; mode 600), `msgids.json` (threading), `keys.json` (a mail is never stored twice), `rejected.log`, the locks. Mail source: `~/fw/issues/imap.json` (mode 600, the same format as the comments one; finished mail goes to `issues-done` / `issues-bad`) or, without it, the Maildir `~/Maildir/.bugs` (`.bugs.done`, `.bugs.bad`). **The bugs mailbox needs its own account or its own `folder`** in `imap.json`: ingest and `imap-check` refuse to run when it has the same host, user and folder as the comments `imap.json` (the two jobs would move each other's mail to their bad folders). Ingest also refuses to run until `authserv_id` is set in `server/comments/sites.json`. The hourly job is `issues-ingest.sh`. + ## Blog A generic static-blog generator: a folder of Markdown files and one JSON config become a site (`server/blog.py`). diff --git a/bin/fw b/bin/fw index 1e24f8a..2b64227 100755 --- a/bin/fw +++ b/bin/fw @@ -14,6 +14,7 @@ # fw blog watch install|uninstall <config> [--print] the launchd agent that runs it every 60 s (ONE machine only) # fw describe <name> TEXT | publish <name> | publish-all | ci <name> [branch] | ci-list | ci-watch # fw comments list|show|approve|reject|remove|untrust|publish|export|ingest|imap-check (blog comments) +# fw issues <repo> public|reviewed|private | list|show|approve|hide|close|reopen|publish|publish-all|ingest|imap-check (bug-report issues) # (run on DreamHost over ssh) set -euo pipefail HOST=${FW_HOST:-dh} @@ -86,7 +87,7 @@ case "${1:-help}" in shift; exec python3 "$HERE/lib/usage.py" "$@" ;; repoint) shift; exec bash "$HERE/lib/repoint.sh" "$@" ;; - list|new|public|private|describe|publish|publish-all|ci|ci-list|ci-watch|comments|help|-h|--help) + list|new|public|private|describe|publish|publish-all|ci|ci-list|ci-watch|comments|issues|help|-h|--help) # printf %q keeps arguments (e.g. a description with spaces) intact through ssh. exec ssh "$HOST" "~/bin/fw $(printf '%q ' "$@")" ;; *) diff --git a/lib/deploy.mjs b/lib/deploy.mjs index f9e455f..c0e43e5 100644 --- a/lib/deploy.mjs +++ b/lib/deploy.mjs @@ -17,11 +17,14 @@ // "build": "make public", // build command (default: `<pm> run build`) // "out": "dist", // output directory (default: dist, else public, else build) // "keep": ["app.apk"] // files already on the server that a deploy must not delete or replace +// "extra": ["sim/*.js"], // build-output files that every later deploy must keep (uploaded, updated, never deleted) +// "replay": "replay.html?report={record}" // viewer for bug-report records: an issue page links here (page of this app, relative) // } import { spawnSync, execFileSync } from 'node:child_process'; import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; +import { fileURLToPath } from 'node:url'; const HOST = process.env.FW_HOST || 'dh'; const APPS_URL = 'https://apps.fuzzyworld.net'; @@ -65,7 +68,31 @@ function parseArgs(argv) { return o; } -function readConfig(dir) { +// A relative path or glob inside the app folder: no leading slash, no "..", no empty segments. +function checkPath(k, key, file) { + if (typeof k !== 'string' || !k || k.startsWith('/') || k.split('/').some((p) => p === '..' || p === '')) { + die(`invalid fuzzyworld.${key} entry ${JSON.stringify(k)} in ${file}`); + } +} + +// The replay template becomes a link on issue pages, so it must stay a page of this app. +export function checkReplay(t, file = 'package.json') { + const head = typeof t === 'string' ? t.split('?')[0] : ''; + if (typeof t !== 'string' || t.length > 200 || !t.includes('{record}') || t.startsWith('/') || t.includes('//') || + t.includes('..') || t.includes('\\') || head.includes(':') || head.includes('%') || /[\s\x00-\x1f\x7f]/.test(head)) { + die(`invalid fuzzyworld.replay ${JSON.stringify(t)} in ${file} (a relative page of the app, containing {record})`); + } +} + +// `keep` paths are excluded: rsync neither uploads nor deletes them. `extra` paths get a "protect" filter: +// they are uploaded and updated as usual, but --delete never removes them (old rules versions stay). +export function rsyncArgs(cfg, out, dest) { + const keep = (cfg.keep || []).flatMap((k) => ['--exclude', `/${k}`]); + const extra = (cfg.extra || []).flatMap((g) => ['--filter', `P /${g}`]); + return ['-rlptz', '--delete', ...keep, ...extra, `${out}/`, dest]; +} + +export function readConfig(dir) { const file = path.join(dir, 'package.json'); if (!fs.existsSync(file)) die(`no package.json in ${dir}`); const pkg = JSON.parse(fs.readFileSync(file, 'utf8')); @@ -73,11 +100,12 @@ function readConfig(dir) { if (!cfg) die(`no "fuzzyworld" section in ${file}`); if (!cfg.app || !NAME_RE.test(cfg.app)) die(`invalid or missing fuzzyworld.app in ${file}`); if (cfg.repo && !NAME_RE.test(cfg.repo)) die(`invalid fuzzyworld.repo in ${file}`); - for (const k of cfg.keep || []) { - if (typeof k !== 'string' || !k || k.startsWith('/') || k.split('/').some((p) => p === '..' || p === '')) { - die(`invalid fuzzyworld.keep entry ${JSON.stringify(k)} in ${file}`); - } + for (const k of cfg.keep || []) checkPath(k, 'keep', file); + for (const g of cfg.extra || []) { + checkPath(g, 'extra', file); + if ((cfg.keep || []).includes(g)) die(`fuzzyworld.extra entry ${JSON.stringify(g)} is also in keep (keep files are never uploaded)`); } + if (cfg.replay !== undefined) checkReplay(cfg.replay, file); return { pkg, cfg }; } @@ -225,19 +253,18 @@ export function deploy(argv) { console.log(' ✓ all URLs, manifest and service worker stay inside', `/${app}/`); fs.writeFileSync(path.join(out, '.fuzzyworld.json'), JSON.stringify({ - app, repo, title: (cfg.title || cfg.app) + (o.as ? ` (${o.as.startsWith(cfg.app + '-') ? o.as.slice(cfg.app.length + 1) : o.as})` : ''), description: cfg.description || '', + app, repo, title: (cfg.title || cfg.app) + (o.as ? ` (${o.as.startsWith(cfg.app + '-') ? o.as.slice(cfg.app.length + 1) : o.as})` : ''), description: cfg.description || '', replay: cfg.replay || '', commit, ref: o.here ? '(working copy)' : o.ref, deployed: new Date().toISOString(), }, null, 2)); if (o.dryRun) { console.log(`\n(dry run) would upload ${out}/ to ${HOST}:apps.fuzzyworld.net/${app}/`); + if (cfg.extra?.length) console.log(`(dry run) kept across deploys: ${cfg.extra.join(', ')}`); return; } step(`uploading to apps.fuzzyworld.net/${app}/`); run('ssh', [HOST, `mkdir -p apps.fuzzyworld.net/${app}`]); - // `keep` paths are excluded, so rsync --delete neither removes nor overwrites them. - const keep = (cfg.keep || []).flatMap((k) => ['--exclude', `/${k}`]); - run('rsync', ['-rlptz', '--delete', ...keep, `${out}/`, `${HOST}:apps.fuzzyworld.net/${app}/`]); + run('rsync', rsyncArgs(cfg, out, `${HOST}:apps.fuzzyworld.net/${app}/`)); run('ssh', [HOST, `~/bin/fw app-deployed ${app} ${repo}`]); console.log(`\n✅ ${APPS_URL}/${app}/ (commit ${commit.slice(0, 12)})`); } finally { @@ -245,10 +272,12 @@ export function deploy(argv) { } } -try { - deploy(process.argv.slice(2)); -} catch (e) { - if (!(e instanceof DeployError)) throw e; - console.error(`\nfw deploy: ${e.message}`); - process.exit(1); +if (process.argv[1] && fs.realpathSync(process.argv[1]) === fileURLToPath(import.meta.url)) { + try { + deploy(process.argv.slice(2)); + } catch (e) { + if (!(e instanceof DeployError)) throw e; + console.error(`\nfw deploy: ${e.message}`); + process.exit(1); + } } diff --git a/server/comments.py b/server/comments.py index 0946b1a..e60b335 100644 --- a/server/comments.py +++ b/server/comments.py @@ -659,8 +659,8 @@ def imap_config_path(): return os.path.expanduser(os.environ.get("FW_COMMENTS_IMAP") or os.path.join(state_dir(), "imap.json")) -def load_imap_config(): - path = imap_config_path() +def load_imap_config(path=None): + path = path or imap_config_path() mode = os.stat(path).st_mode if mode & 0o077: raise CommentError(f"{path} must not be readable by group or others (chmod 600 it)") @@ -689,16 +689,21 @@ IMAP_NAME_RE = re.compile(r"^[A-Za-z0-9._/-]+$") class IMAPSource: - """The comments mailbox on an IMAP server. Finished mail is copied to comments-done / comments-bad.""" - - def __init__(self): + """The comments mailbox on an IMAP server. Finished mail is copied to comments-done / comments-bad. + issues.py reuses this class for the bugs mailbox: its own config file (cfg_path), folder stem, size limit and logger.""" + + def __init__(self, cfg_path=None, stem="comments", max_size=IMAP_MAX_SIZE, log=None): + self.cfg_path = cfg_path + self.stem = stem + self.max_size = max_size + self.log = log or log_rejection self.m = None self.cfg = None self.validity = "" self.caps = set() def available(self): - return os.path.exists(imap_config_path()) + return os.path.exists(self.cfg_path or imap_config_path()) def _redact(self, text): pw = self.cfg["password"] if self.cfg else "" @@ -720,7 +725,7 @@ class IMAPSource: return data def _open(self, readonly=False): - self.cfg = load_imap_config() + self.cfg = load_imap_config(self.cfg_path) try: self.m = imaplib.IMAP4_SSL(self.cfg["host"], 993, ssl_context=ssl.create_default_context(), timeout=IMAP_TIMEOUT) self.m.login(self.cfg["user"], self.cfg["password"]) @@ -787,7 +792,7 @@ class IMAPSource: self.prefix = "" def _folder(self, outcome): - return self.cfg[outcome + "_folder"] or f"{self.prefix}comments-{outcome}" + return self.cfg[outcome + "_folder"] or f"{self.prefix}{self.stem}-{outcome}" def close(self): if self.m is not None: @@ -818,14 +823,14 @@ class IMAPSource: sizes = self._sizes(uids) keys = [] for uid in uids: - if sizes.get(uid, 0) > IMAP_MAX_SIZE: # never downloaded + if sizes.get(uid, 0) > self.max_size: # never downloaded try: self._move(uid, "bad") except (CommentError, OSError, imaplib.IMAP4.error) as e: # leave it, keep going - log_rejection(new_id(), f"cannot move oversize message {uid}: {e}") + self.log(new_id(), f"cannot move oversize message {uid}: {e}") continue - log_rejection(new_id(), f"message over {IMAP_MAX_SIZE // 1024} KB") - print(f"bad: message over {IMAP_MAX_SIZE // 1024} KB") + self.log(new_id(), f"message over {self.max_size // 1024} KB") + print(f"bad: message over {self.max_size // 1024} KB") else: keys.append(f"{self.validity}:{uid}") return keys diff --git a/server/fw b/server/fw index 4a5cdc7..7f0332e 100755 --- a/server/fw +++ b/server/fw @@ -20,6 +20,8 @@ # fw ci-pages <name> rebuild a repo's CI pages and the dashboard (runner-api runs this) # fw ci-dashboard rebuild code.fuzzyworld.net/ci/ # fw comments <cmd> moderate blog comments (list, show, approve, reject, ... ingest, imap-check) +# fw issues <repo> public|reviewed|private who sees a repo's bug-report issue pages (default private) +# fw issues <cmd> moderate issues (list, show, approve, hide, close, reopen, publish, publish-all, ingest, imap-check) # # Visibility is `git config fuzzyworld.visibility` inside each repo. Anything other # than exactly "public" is treated as private. @@ -82,6 +84,8 @@ publish_one() { fi git -C "$pub" symbolic-ref HEAD "$(git -C "$src" symbolic-ref HEAD)" git -C "$pub" update-server-info + # Issue pages (bug reports) live in the repo's folder; rebuilt under the lock we already hold. + python3 "$TOOLS/server/issues.py" publish "$name" --locked || echo "fw: issues publish failed for $name" >&2 "${GEN[@]}" repo "$name" } @@ -215,6 +219,20 @@ cmd_ci_list() { done } +cmd_issues() { + case "${1:-}" in + ingest|imap-check|list|show|approve|hide|close|reopen|publish|publish-all|help|-h|--help|"") + python3 "$TOOLS/server/issues.py" "$@" ;; + *) + # fw issues <repo> <mode> + [ $# -eq 2 ] && [[ "$2" =~ ^(public|reviewed|private)$ ]] || die "usage: fw issues <repo> public|reviewed|private (or fw issues help)" + need_repo "$1" + git -C "$(repo_dir "$1")" config fuzzyworld.issues "$2" + python3 "$TOOLS/server/issues.py" publish "$1" + echo "$1: issues $2" ;; + esac +} + cmd_ci_watch() { local last="" now while true; do @@ -247,6 +265,7 @@ case "${1:-}" in ci-pages) need_repo "${2:-}"; with_lock "$2" "${GEN[@]}" ci "$2" ;; ci-dashboard) "${GEN[@]}" ci-dashboard ;; comments) shift; python3 "$TOOLS/server/comments.py" "$@" ;; - ""|-h|--help|help) sed -n '2,24p' "$0" | sed 's/^# \{0,1\}//' ;; + issues) shift; cmd_issues "$@" ;; + ""|-h|--help|help) sed -n '2,27p' "$0" | sed 's/^# \{0,1\}//' ;; *) die "unknown command: $1 (try: fw help)" ;; esac diff --git a/server/gen.py b/server/gen.py index e6e3789..ce6d106 100755 --- a/server/gen.py +++ b/server/gen.py @@ -1229,7 +1229,9 @@ def build_summary(repo, view, desc, files, ctx, commits, empty, commit_set): links = f'<p class="meta">App: <a href="{app}">{esc(app)}</a></p>' if app else "" contrib, bugs = f"contrib-68uq+{name}@fuzzyworld.net", f"bugs-k7q4+{name}@fuzzyworld.net" links += (f'<p class="meta">Contribute: <a href="mailto:{esc(contrib)}">{esc(contrib)}</a>' - f' (<a href="/contribute/">how</a>) · Report a bug: <a href="mailto:{esc(bugs)}">{esc(bugs)}</a></p>') + f' (<a href="/contribute/">how</a>) · Report a bug: <a href="mailto:{esc(bugs)}">{esc(bugs)}</a>' + + (f' · <a href="{url(name, "issues")}/">Issues</a>' if os.path.isfile(os.path.join(SITE, name, "issues", "index.html")) else "") + + "</p>") readme_html, license_html = "", "" if not empty: blobs = BlobReader(repo) diff --git a/server/install.sh b/server/install.sh index 4b03510..881fe41 100755 --- a/server/install.sh +++ b/server/install.sh @@ -18,7 +18,7 @@ git config --global pack.threads 1 git config --global pack.windowMemory 64m git config --global pack.deltaCacheSize 16m git config --global core.bigFileThreshold 16m -mkdir -p ~/bin ~/fw/state ~/fw/logs ~/fw/queue/pending ~/fw/queue/running ~/fw/ci +mkdir -p ~/bin ~/fw/state ~/fw/logs ~/fw/queue/pending ~/fw/queue/running ~/fw/ci ~/fw/issues chmod +x ~/fw/tools/server/runner-api ~/fw/tools/server/mirror-only # Pygments colors source code on the file pages (gen.py works without it, uncolored). # Python-Markdown renders the README pages and the landing page text (`gen.py landing` needs it). diff --git a/server/issues.py b/server/issues.py new file mode 100644 index 0000000..97ee4be --- /dev/null +++ b/server/issues.py @@ -0,0 +1,896 @@ +#!/usr/bin/env python3 +"""Bug reports by email become issue pages on code.fuzzyworld.net. Python standard library only. + +Usage (on the server: `fw issues <cmd>`; from the laptop: `bin/fw issues <cmd>`): + + list [repo] issues (repo, number, state, whether the page is public, title) + show <repo> <n> one issue in full (no addresses are stored, so none are shown) + approve <repo> <n> mark the issue public (what `reviewed` mode waits for) + hide <repo> <n> take the issue off the site and keep it off (even in `public` mode) + close <repo> <n> / reopen <repo> <n> + publish <repo> rewrite <repo>/issues/ on the site from the stored issues (--locked: the caller holds the repo lock) + publish-all the same for every repo that has issues + ingest read new mail from the bugs mailbox (IMAP if imap.json exists, else Maildir) + imap-check connect to the IMAP mailbox and print what it holds (header names only) + fw issues <repo> public|reviewed|private set the repo's mode (`git config fuzzyworld.issues`; anything else = private) + +Modes. `private` (also the default and any unknown word): issues are stored, never published. `reviewed`: an issue +page appears only after `approve`. `public`: an issue whose first message passed authentication appears at once. +A private repo (fuzzyworld.visibility) never publishes issues, whatever its mode. `hide` beats both modes. + +Mail arrives at bugs-k7q4+<repo>@fuzzyworld.net; <repo> must be a repo that exists under ~/git, else the mail goes to +the bad folder. A reply (In-Reply-To / References that match an earlier Message-ID) is added to its issue. + +State, outside every web root (override the directory with FW_ISSUES_DIR): + ~/fw/issues/imap.json IMAP login (mode 600), same format as comments' imap.json; finished mail goes to + <prefix>issues-done / <prefix>issues-bad. Without it: Maildir ~/Maildir/.bugs + (override with FW_ISSUES_MAILDIR), with .bugs.done / .bugs.bad beside it. + ~/fw/issues/secret random salt for sender hashes (mode 600) + ~/fw/issues/msgids.json sha256(Message-ID) -> [repo, n, message_index], so replies find their issue; never published + ~/fw/issues/keys.json sha256(raw mail) -> [repo, n, message_index], so a mail is never stored twice; an entry + counts only if that message holds the same mkey (crash recovery) + ~/fw/issues/rejected.log "id date reason" per rejected mail (rotated at 1 MB) + ~/fw/issues/lock, ingest.lock held for every change to the store / for a whole ingest run + ~/fw/issues/repos/<repo>/<n>/issue.json {n, repo, title, created, state, approved, hidden, record, record_msg, + messages: [{date, author, text, sender_hash, authenticated, approved, mkey, mid}]}; + numbers count up per repo; record_msg is the message that carried the record + ~/fw/issues/repos/<repo>/<n>/record.json the run record, only when one passed the checks (see below) +Published (when visible): <site>/<repo>/issues/index.html and <site>/<repo>/issues/<n>/index.html, plus +<n>/record.json. The whole issues/ folder is rebuilt on every publish, so a hidden issue disappears. + +Privacy. No address, name, Message-ID or sender hash is ever written to a page: the author is always "Reporter", +every address-shaped string in the title and text is replaced by "[address hidden]", lines starting "Device:" or +"Run record:" are left out of the text, and all text is HTML-escaped inside <pre class="issue-text">. +Authentication is the same as for comments (comments.authenticated, topmost Authentication-Results from the +"authserv_id" in sites.json). Unauthenticated mail is stored with authenticated:false and never shown by `public` +mode alone; it needs `approve`. Each sender may start or continue at most 5 issues per 24 hours in a `public` or `reviewed` repo. + +Run records. A line "Run record: {json}" in the text part is read from the FULL decoded part (before the text is cut +to 4000 characters), because a record is ~9 KB. The first line that passes the checks wins (a later "Run record: +too long for an email link" footer is ignored): at most 64 KB, valid JSON, an object with integer seed and +simVersion and a list "actions". It is stored re-serialized (never as mail text). Only the first record of an issue is +kept. A record from an unauthenticated reply is ignored. record.json is published only while the issue is visible. +Only the keys simVersion, seed, startLevel, startCharge, options, actions are kept, and a record with an address-like +string is refused. `approve` marks the messages present at that time; later ones need another approve (or, in `public` +mode, an authenticated sender). The bugs mailbox must not be the comments mailbox (same host, user and folder is refused). +The issue page links to the app's replay viewer when the app's .fuzzyworld.json has a `replay` template. +""" + +import hashlib +import json +import os +import quopri +import re +import secrets +import shutil +import subprocess +import sys +import time +import unicodedata +from contextlib import contextmanager +from email import message_from_bytes, policy +from email.header import decode_header, make_header +from email.utils import getaddresses +from urllib.parse import quote + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import comments # noqa: E402 +import gen # noqa: E402 + +try: + import fcntl +except ImportError: # not on Windows servers; only the lock is lost + fcntl = None + +Error = comments.CommentError +REPO_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$") +BUGS_RE = re.compile(r"^bugs-k7q4\+([A-Za-z0-9][A-Za-z0-9._-]*)@fuzzyworld\.net$", re.I) +ADDRESS_RE = re.compile(r"[^\s@<>]+@[^\s@<>]+") +RECORD_RE = re.compile(r"^Run record: (\{.*\})\s*$", re.M) +RECORD_MAX = 64 * 1024 +RECORD_DEPTH = 32 +RECORD_KEYS = ("simVersion", "seed", "startLevel", "startCharge", "options", "actions") # every key of the real records; the rest is dropped +MAX_MAIL = 5 * 1024 * 1024 +MAX_TITLE = 120 +SENDER_LIMIT = 5 # issues or replies per sender per SENDER_WINDOW seconds, in `public` and `reviewed` repos +SENDER_WINDOW = 24 * 3600 +MODES = ("public", "reviewed", "private") +HIDDEN = "[address hidden]" + + +# ---- configuration and paths ---- + +def state_dir(): + return os.path.expanduser(os.environ.get("FW_ISSUES_DIR") or "~/fw/issues") + + +def imap_path(): + return os.path.join(state_dir(), "imap.json") + + +def maildir_path(): + return os.path.expanduser(os.environ.get("FW_ISSUES_MAILDIR") or "~/Maildir/.bugs") + + +def check_repo(repo): + if not (isinstance(repo, str) and REPO_RE.match(repo) and not repo.endswith(".git")): + raise Error(f"invalid repo name: {repo!r}") + return repo + + +def check_n(n): + if not (isinstance(n, str) and n.isascii() and n.isdigit() and 0 < int(n) < 10**9): + raise Error(f"invalid issue number: {n!r}") + return int(n) + + +def repo_dir(repo): + return os.path.join(gen.GIT_ROOT, check_repo(repo) + ".git") + + +def issue_dir(repo, n): + return os.path.join(state_dir(), "repos", check_repo(repo), str(n)) + + +def log_rejection(cid, reason): + os.makedirs(state_dir(), exist_ok=True) + reason = comments.clean(reason).replace("\n", " ") + log = os.path.join(state_dir(), "rejected.log") + try: + if os.path.getsize(log) > 1_000_000: + os.replace(log, log + ".1") + except FileNotFoundError: + pass + with open(log, "a") as f: + f.write(f"{cid} {comments.now_iso()} {reason}\n") + + +@contextmanager +def _lock(name): + os.makedirs(state_dir(), exist_ok=True) + with open(os.path.join(state_dir(), name), "w") as f: + if fcntl: + fcntl.flock(f, fcntl.LOCK_EX) + yield + + +def locked(): + return _lock("lock") + + +def ingest_lock(): + """Job-wide: one ingest at a time (a second one waits).""" + return _lock("ingest.lock") + + +@contextmanager +def repo_lock(repo): + """The lock `fw` takes around publishing one repo (STATE/locks/<repo>.lock).""" + d = os.path.join(gen.STATE, "locks") + os.makedirs(d, exist_ok=True) + with open(os.path.join(d, check_repo(repo) + ".lock"), "w") as f: + if fcntl: + fcntl.flock(f, fcntl.LOCK_EX) + yield + + +def secret(): + path = os.path.join(state_dir(), "secret") + try: + with open(path) as f: + value = f.read().strip() + except FileNotFoundError: + value = "" + if not value: + value = secrets.token_hex(32) + comments.write_atomic(path, value + "\n") + return value + + +def sender_hash(address): + return hashlib.sha256((secret() + address.strip().lower()).encode()).hexdigest() + + +def sha(text): + return hashlib.sha256(text.encode() if isinstance(text, str) else text).hexdigest() + + +# ---- store ---- + +def issue_numbers(repo): + d = os.path.join(state_dir(), "repos", check_repo(repo)) + # a folder without issue.json is the leftover of a crash (a record written before the issue): not an issue + return sorted(int(x) for x in os.listdir(d) if x.isdigit() and os.path.isfile(os.path.join(d, x, "issue.json"))) if os.path.isdir(d) else [] + + +def stored_repos(): + d = os.path.join(state_dir(), "repos") + return sorted(x for x in os.listdir(d) if REPO_RE.match(x)) if os.path.isdir(d) else [] + + +def load_issue(repo, n): + issue = comments.read_json(os.path.join(issue_dir(repo, n), "issue.json"), None) + if issue is None: + raise Error(f"no issue {repo}#{n}") + return issue + + +def save_issue(issue): + comments.write_json(os.path.join(issue_dir(issue["repo"], issue["n"]), "issue.json"), issue) + + +def load_issues(repo): + out = [] + for n in issue_numbers(repo): + issue = comments.read_json(os.path.join(issue_dir(repo, n), "issue.json"), None) + if issue: + out.append(issue) + return out + + +def record_path(repo, n): + return os.path.join(issue_dir(repo, n), "record.json") + + +def recent_index(): + """sender hash -> dates of that sender's stored messages (read once per run).""" + index = {} + for repo in stored_repos(): + for issue in load_issues(repo): + for m in issue["messages"]: + if m.get("sender_hash"): + index.setdefault(m["sender_hash"], []).append(m.get("date", "")) + return index + + +# ---- visibility and publishing ---- + +def issue_mode(repo_path): + """public | reviewed | private; anything else (a typo, an empty value) is private.""" + mode = gen.repo_config(repo_path, "fuzzyworld.issues") + return mode if mode in ("public", "reviewed") else "private" + + +def visible(repo_path, issue, mode=None): + mode = mode or issue_mode(repo_path) + if mode == "private" or not gen.is_public(repo_path) or issue.get("hidden"): + return False + first = issue["messages"][0] if issue.get("messages") else {} + return bool(issue.get("approved")) or (mode == "public" and bool(first.get("authenticated"))) + + +def replay_ok(t): + """The same rule as checkReplay in lib/deploy.mjs: a relative page of the app, containing {record}.""" + head = t.split("?")[0] if isinstance(t, str) else "" + return (isinstance(t, str) and not re.search(r"[%\s\x00-\x1f\x7f]", head) and len(t) <= 200 and "{record}" in t and not t.startswith("/") and "//" not in t + and ".." not in t and "\\" not in t and ":" not in head) + + +def replay_link(repo, n): + """An <a> to the repo's app replaying issue n's record, or "" when no deployed app has a `replay` template.""" + found = [] + try: + entries = sorted(os.listdir(gen.APPS)) + except OSError: + return "" + for app in entries: + try: + with open(os.path.join(gen.APPS, app, ".fuzzyworld.json")) as f: + meta = json.load(f) + except (OSError, ValueError): + continue + if (isinstance(meta, dict) and meta.get("repo", app) == repo and replay_ok(meta.get("replay")) + and os.path.isfile(os.path.join(gen.APPS, app, "index.html"))): + found.append((app != repo, len(app), app, meta["replay"])) # the app named like the repo first (not a -next copy) + if not found: + return "" + _, _, app, replay = min(found) + record_url = f"{gen.BASE_URL}/{repo}/issues/{n}/record.json" + href = f"{gen.APPS_URL}/{quote(app)}/" + replay.replace("{record}", quote(record_url, safe="")) + return f'<a href="{gen.esc(href)}" rel="noopener">Watch what happened</a>' + + +def shown_messages(issue, mode): + """Messages `approve` has marked, plus (in public mode only) authenticated ones. A message that arrives after + an approve is not marked, so it needs another approve unless the repo is public and its sender authenticated.""" + return [m for m in issue["messages"] if m.get("approved") or (mode == "public" and m.get("authenticated"))] + + +def record_shown(issue, mode): + """A run record is as public as the message that carried it (issue["record_msg"], 0 for the opening report).""" + if not issue.get("record"): + return False + i = issue.get("record_msg", 0) + return 0 <= i < len(issue["messages"]) and issue["messages"][i] in shown_messages(issue, mode) + + +def issue_page(repo, issue, with_record, mode): + n = issue["n"] + parts = [f'<h1>#{n} {gen.esc(issue["title"])}</h1>'] + meta = f'Opened {gen.esc(issue["created"][:10])} · {gen.esc(issue["state"])}' + if with_record: + link = replay_link(repo, n) + meta += (" · " + link if link else "") + parts.append(f'<p class="meta">{meta}</p>') + for m in shown_messages(issue, mode): + parts.append(f'<p class="meta">{gen.esc(m["author"])} · {gen.esc(m["date"][:10])}</p>' + f'<pre class="issue-text">{gen.esc(m["text"])}</pre>') + crumbs = [f'<a href="{gen.url(repo)}/">{gen.esc(repo)}</a>', f'<a href="{gen.url(repo, "issues")}/">Issues</a>', f"#{n}"] + return gen.page(f"{repo} issue #{n}", "".join(parts), crumbs=crumbs) + + +def index_page(repo, issues): + rows = "".join( + f'<tr><td class="num"><a href="{gen.url(repo, "issues", str(i["n"]))}/">#{i["n"]}</a></td>' + f'<td>{gen.esc(i["title"])}</td><td class="date">{gen.esc(i["created"][:10])}</td><td>{gen.esc(i["state"])}</td></tr>' + for i in sorted(issues, key=lambda i: -i["n"])) + body = f'<h1>{gen.esc(repo)} issues</h1><table><tr><th>#</th><th>Title</th><th>Opened</th><th>State</th></tr>{rows}</table>' + return gen.page(f"{repo} issues", body, crumbs=[f'<a href="{gen.url(repo)}/">{gen.esc(repo)}</a>', "Issues"]) + + +def refresh_summary(repo): + """Rebuild the repo's pages so the summary page shows (or drops) its Issues link.""" + subprocess.run([sys.executable, os.path.join(os.path.dirname(os.path.abspath(__file__)), "gen.py"), "repo", repo], + check=False) + + +def write_site(repo): + """Rebuild <site>/<repo>/issues/ from the store. Returns (whether index.html existed before, whether it does now).""" + path = repo_dir(repo) + with locked(): + issues = load_issues(repo) + site_repo = os.path.join(gen.SITE, repo) + final = os.path.join(site_repo, "issues") + had = os.path.isfile(os.path.join(final, "index.html")) + mode = issue_mode(path) if os.path.isdir(path) else "private" + shown = [i for i in issues if os.path.isdir(path) and visible(path, i, mode)] + if not shown: + shutil.rmtree(final, ignore_errors=True) + return had, False + tmp = os.path.join(site_repo, ".issues-new") + shutil.rmtree(tmp, ignore_errors=True) + for i in shown: + has_record = record_shown(i, mode) and os.path.isfile(record_path(repo, i["n"])) + gen.write(os.path.join(tmp, str(i["n"]), "index.html"), issue_page(repo, i, has_record, mode)) + if has_record: + with open(record_path(repo, i["n"])) as f: + gen.write(os.path.join(tmp, str(i["n"]), "record.json"), f.read()) + gen.write(os.path.join(tmp, "index.html"), index_page(repo, shown)) + shutil.rmtree(final, ignore_errors=True) + os.replace(tmp, final) + return had, True + + +def publish(repo, caller_holds_lock=False): + """Rewrite the repo's issue pages. `fw publish` takes the repo lock itself, so it passes caller_holds_lock + (and runs gen.py repo afterwards); otherwise this takes the lock and refreshes the summary link when needed.""" + check_repo(repo) + if caller_holds_lock: + write_site(repo) + return + with repo_lock(repo): + had, has = write_site(repo) + if had != has and gen.is_public(repo_dir(repo)): + refresh_summary(repo) + + +# ---- email intake ---- + +def scrub(s): + return ADDRESS_RE.sub(HIDDEN, s) + + +def repo_from_mail(msg): + """(repo, None) from the bugs-k7q4+<repo>@fuzzyworld.net address, else (None, reason).""" + values = [str(v) for h in ("To", "Cc", "Delivered-To", "X-Original-To") for v in (msg.get_all(h) or [])] + names = {m.group(1) for _, a in getaddresses(values) if (m := BUGS_RE.match(a.strip()))} + if not names: + return None, "no bugs-k7q4+<repo>@ address in To, Cc, Delivered-To or X-Original-To" + if len(names) > 1: + return None, "mail addressed to more than one repo" + name = names.pop() + if name.endswith(".git") or not os.path.isdir(repo_dir(name)): + return None, f"no such repo {name[:60]!r}" + return name, None + + +def message_ids(value): + return re.findall(r"<[^<>\s]+>", str(value or "")) + + +def thread_of(msg, repo=None): + """(repo, n) of the issue this message replies to (via In-Reply-To / References), else None.""" + ids = message_ids(msg.get("In-Reply-To")) + message_ids(msg.get("References"))[::-1] + known = comments.read_json(os.path.join(state_dir(), "msgids.json"), {}) + for mid in ids: + hit = known.get(sha(mid)) + if hit and (repo is None or hit[0] == repo) and os.path.isfile(os.path.join(issue_dir(hit[0], hit[1]), "issue.json")): + return hit[0], hit[1] + return None + + +def strings_in(value): + """Every string (and dict key) inside a JSON value, iteratively.""" + todo = [value] + while todo: + v = todo.pop() + if isinstance(v, str): + yield v + elif isinstance(v, dict): + todo.extend(v.keys()) + todo.extend(v.values()) + elif isinstance(v, list): + todo.extend(v) + + +def check_record(s): + """(record dict, None) or (None, reason). The checks the replay viewer relies on. Only RECORD_KEYS survive.""" + if len(s.encode("utf-8", "replace")) > RECORD_MAX: + return None, f"run record over {RECORD_MAX // 1024} KB" + + def refuse(name): # NaN and Infinity are not JSON, and the viewer would choke on them + raise ValueError(name) + + def number(text): # 1e400 reads as infinity + x = float(text) + if x != x or x in (float("inf"), float("-inf")): + raise ValueError(text) + return x + try: + rec = json.loads(s, parse_constant=refuse, parse_float=number) + except (ValueError, RecursionError): + return None, "run record is not valid JSON" + if not isinstance(rec, dict): + return None, "run record is not an object" + depth, level = 0, [rec] + while level: # iterative, so a deeply nested record cannot exhaust the stack (here or in the viewer) + depth += 1 + if depth > RECORD_DEPTH: + return None, f"run record nested deeper than {RECORD_DEPTH}" + level = [c for v in level for c in (v.values() if isinstance(v, dict) else v if isinstance(v, list) else ())] + rec = {k: rec[k] for k in RECORD_KEYS if k in rec} + for key in ("seed", "simVersion"): + v = rec.get(key) + if not isinstance(v, int) or isinstance(v, bool) or abs(v) > 2**53: + return None, f"run record needs an integer {key}" + if not isinstance(rec.get("actions"), list): + return None, "run record needs a list of actions" + if any(ADDRESS_RE.search(unicodedata.normalize("NFKC", t)) for t in strings_in(rec)): + return None, "run record holds an address-like string" + return rec, None + + +def find_record(text, raw=b""): + """(record, None), or (None, "") when the text holds no "Run record:" line, or (None, reason) when none passes. + Every matching line is tried and the first valid one wins: the mail repeats the label in a footer.""" + why = "" + sources = [text] + if b"quoted-printable" in raw.lower(): # fallback: decode the whole raw mail (the rule of Explodorama's report-replay.mjs) + sources.append(quopri.decodestring(raw).decode("utf-8", "replace")) + for src in sources: + for m in RECORD_RE.finditer(src.replace("\r\n", "\n")): + rec, why = check_record(m.group(1)) + if rec: + return rec, None + return None, why + + +WROTE_RE = re.compile(r"\b(wrote|schrieb|a écrit|escribió|schreef|skrev|scrisse|написал)\s?[:：]$", re.I) +COLON_RE = re.compile(r"\s?[:：]$") +ORIGINAL_RE = re.compile(r"^\s*[-_=]{2,}\s*(Original Message|Forwarded message|Urspr\w+ Nachricht)\s*[-_=]*\s*$", re.I) +HEADERISH_RE = re.compile(r"^(Sent|Date|To|Cc|Subject|Reply-To):", re.I) +UNDERLINE_RE = re.compile(r"^\s*[_=-]{5,}\s*$") +# the labels of the Explodorama report template (src/main.ts bugDetails) and the like, or any short "Label:" line +KNOWN_LINE_RE = re.compile(r"^(Seed|Version|Errors caught|Run record|Rules|Level|Turn|Modes|What happened)(?=[ :])|^[A-Z][A-Za-z ]{0,20}:") # "Version/17.0" is a user agent, not a label +USER_AGENT_RE = re.compile(r"\S/\S|[();]") # what a wrapped user-agent line looks like; prose rarely has it + + +BARE_VERB_RE = re.compile(r"^((a )?écrit|wrote|schrieb|escribió|schreef|skrev|scrisse|написал)\s?[:：]$", re.I) +VERB_ANY_RE = re.compile(r"\b(wrote|schrieb|écrit|escribió|schreef|skrev|scrisse|написал)\b", re.I) +OPENER_RE = re.compile(r"^(On|Am|Le|El|Op|Il|Den|Em|Dne|W dniu)\b", re.I) +# one "Label: value" line of a mail-header block, in any language ("Von:", "De :", "Envoyé :") +HEADER_LINE_RE = re.compile(r"^([^\W\d_][\w .-]{0,19}?)\s?[:：]\s*\S") +HEADER_LABELS = {"from", "von", "de", "da", "van", "från", "sent", "gesendet", "envoyé", "date", "datum", "verzonden", + "fecha", "enviado", "to", "an", "à", "aan", "para", "cc", "subject", "betreff", "objet", "onderwerp", + "asunto", "reply-to", "antwort an"} +TEMPLATE_LABELS = {"device", "run record", "errors caught", "seed", "version", "rules"} # the game's own report lines + + +def has_address(text): + return bool(ADDRESS_RE.search(unicodedata.normalize("NFKC", text))) + + +def strong_attribution(line, nxt): + """True for a reply's "who wrote this" line in any language: a colon line that holds an address, or that opens like + one (On, Am, Le, ... plus a digit) and says wrote/schrieb/a écrit/..., or says it and is followed by a quoted line; + or an opener/address line wrapped so that only "wrote:" is on the next line. "I pressed <Enter> and wrote:" is prose.""" + t = line.strip() + if line.startswith(">") or not t: + return False + if COLON_RE.search(t): + if has_address(t): + return True + return bool(WROTE_RE.search(t)) and ((OPENER_RE.match(t) and re.search(r"\d", t)) or nxt.startswith(">")) + return bool(BARE_VERB_RE.match(nxt.strip())) and bool(OPENER_RE.match(t) or has_address(t)) + + +def header_run_start(lines, i): + """True when lines[i:] begin a block of >= 3 consecutive "Label: value" lines that is a mail header (it holds an + address, or at least three of its labels are From/To/Subject/Date words in some language).""" + run = [] + for line in lines[i:]: + m = HEADER_LINE_RE.match(line.strip()) + if not m or m.group(1).strip().lower() in TEMPLATE_LABELS or line.startswith(">"): + break + run.append((m.group(1).strip().lower(), line)) + if len(run) < 3: + return False + return any(has_address(l) for _, l in run) or sum(1 for lab, _ in run if lab in HEADER_LABELS) >= 3 + + +def find_cut(lines): + """Index of the first line of the quoted reply, attribution or signature, or None.""" + cuts = [] + n = len(lines) + quoted = lambda l: l.startswith(">") + for q in range(1, n): # a wrapped attribution: up to 3 plain lines ending in a colon, directly followed by a quoted line + if not quoted(lines[q]) or quoted(lines[q - 1]) or not lines[q - 1].strip() or not COLON_RE.search(lines[q - 1].strip()): + continue + lo = q + while lo > max(q - 3, 0) and lines[lo - 1].strip() and not quoted(lines[lo - 1]): + lo -= 1 + for k in range(lo, q): + joined = " ".join(x.strip() for x in lines[k:q]) + if (VERB_ANY_RE.search(joined) or has_address(joined)) and ( + OPENER_RE.match(lines[k].strip()) or has_address(lines[k]) or "<" in lines[k]): + cuts.append(k) + break + for i, line in enumerate(lines): + nxt = lines[i + 1] if i + 1 < n else "" + if line.rstrip() == "--" or ORIGINAL_RE.match(line) or strong_attribution(line, nxt) or header_run_start(lines, i): + cuts.append(i) + break + if (COLON_RE.search(line.strip()) and nxt.startswith(">") and not quoted(line) # a weak match: only when the rest is + and all(not l.strip() or quoted(l) for l in lines[i + 1:]) # quoted, and text came before it + and any(l.strip() and not l.lstrip().startswith((">", "Device:", "Run record:")) for l in lines[:i])): + cuts.append(i) + break + return min(cuts) if cuts else None + + +def unflow(lines, delsp=False): + """RFC 3676 format=flowed: a line ending in a space continues on the next one (quote marks aside). The space is + tested on the physical line, and an empty physical line always ends a paragraph.""" + out, cur = [], None + for raw in lines: + if raw.startswith(">") or raw == "-- ": # quotes and the signature separator are tested on the raw line + if cur is not None: + out.append(cur) + cur = None + out.append(raw) + continue + line = raw[1:] if raw.startswith(" ") else raw # space-stuffing + if line == "": + if cur is not None: + out.append(cur) + cur = None + out.append("") + continue + if cur is not None: + if line.lstrip().startswith(("Device:", "Run record:")): + out.append(cur) # these always start a line of their own + elif cur.lstrip().startswith("Device:") and KNOWN_LINE_RE.match(line): + out.append(cur) # a Device: line never swallows the paragraph after it + else: + line = (cur[:-1] if delsp else cur) + line + cur = None + if raw.endswith(" "): + cur = line + else: + out.append(line) + if cur is not None: + out.append(cur) + return out + + +def report_text(content, flowed=False, delsp=False): + """The reporter's words: quotes, signature, Device: and Run record: lines out, replies cut at their attribution + (any language, wrapped or not) or mail-header block, addresses hidden, cut to 4000. NFKC comes first, so fullwidth + letters and @ count. Device: and the lines that wrap it: following lines are dropped while they look like a user + agent (indented, or containing token/version, parentheses or semicolons); a blank line, a template label ("Seed + ...") or plain prose ends the block, so the reporter's own words after a Device line are kept.""" + lines = unicodedata.normalize("NFKC", comments.clean(content)).split("\n") + if flowed: + lines = unflow(lines, delsp) + cut = find_cut(lines) + kept, in_device = [], False + for line in lines[:cut]: + s = line.lstrip() + if s.startswith("Device:"): + in_device = True + continue + if in_device: + if line.strip() and not KNOWN_LINE_RE.match(s) and (line[0] in " \t" or USER_AGENT_RE.search(line)): + continue + in_device = False + if s.startswith((">", "Run record:")): + continue + kept.append(line) + while cut is not None and kept and (not kept[-1].strip() or UNDERLINE_RE.match(kept[-1])): # Outlook's ____ line + kept.pop() + return scrub("\n".join(kept).strip())[:comments.MAX_TEXT] + + +def parse_report(msg, authserv_id, raw=b""): + """(report dict with the sender "_address", None) or (None, reason). Content problems only.""" + froms = msg.get_all("From") or [] + addrs = [a for a in getaddresses([str(x) for x in froms]) if a[1]] + if len(froms) != 1 or len(addrs) != 1 or "@" not in addrs[0][1]: + return None, "need exactly one From header with one address" + addr = addrs[0][1] + ok = comments.authenticated(msg, addr.rsplit("@", 1)[1].lower(), authserv_id) is None + body = msg.get_body(("plain",)) + content = body.get_content() if body is not None else "" + flowed = body is not None and (body.get_param("format") or "").lower() == "flowed" + delsp = body is not None and (body.get_param("delsp") or "").lower() == "yes" + text = report_text(content, flowed, delsp) + if not text: + return None, "empty text" + subject = str(make_header(decode_header(str(msg.get("Subject", ""))))) + title = " ".join(scrub(unicodedata.normalize("NFKC", comments.clean(subject))).split())[:MAX_TITLE].strip() or "(no subject)" + record, record_why = find_record(content, raw) + return {"title": title, "text": text, "author": "Reporter", "authenticated": ok, "date": comments.now_iso(), + "record": record, "record_why": record_why, "_address": addr}, None + + +def next_number(repo): + nums = issue_numbers(repo) + return (nums[-1] if nums else 0) + 1 + + +def store_report(repo, report, thread, mkey, mid): + """Add the report to its thread or open a new issue. Returns (issue, new?, note).""" + message = {"date": report["date"], "author": report["author"], "text": report["text"], + "sender_hash": report["sender_hash"], "authenticated": report["authenticated"], "approved": False, + "mkey": mkey, "mid": sha(mid) if mid else ""} # mkey and mid: for crash recovery, never published + if thread: + issue, new = load_issue(*thread), False + issue["messages"].append(message) + else: + n = next_number(repo) + issue = {"n": n, "repo": repo, "title": report["title"], "created": report["date"], "state": "open", + "approved": False, "hidden": False, "record": False, "record_msg": 0, "messages": [message]} + new = True + note = "" + index = len(issue["messages"]) - 1 + where = [repo, issue["n"], index] + # Remember the mail first: a crash after this leaves a key without a message, which stored() does not count, + # so the mail is read again (no lost report) and never stored twice (no duplicate issue). + keys = comments.read_json(os.path.join(state_dir(), "keys.json"), {}) + keys[mkey] = where + comments.write_json(os.path.join(state_dir(), "keys.json"), keys) + if mid: + ids = comments.read_json(os.path.join(state_dir(), "msgids.json"), {}) + ids[sha(mid)] = where + comments.write_json(os.path.join(state_dir(), "msgids.json"), ids) + write_record = None + if report["record"]: + if issue["record"]: + note = "second run record ignored" + elif not new and not report["authenticated"]: + note = "run record from an unauthenticated reply ignored" + else: + write_record = json.dumps(report["record"], separators=(",", ":"), allow_nan=False) + issue["record"], issue["record_msg"] = True, index + elif report["record_why"]: + note = report["record_why"] + if write_record: # the file first, so issue.json never claims a record that is missing; issue_numbers ignores a lone folder + comments.write_atomic(record_path(repo, issue["n"]), write_record) + save_issue(issue) + return issue, new, note + + +def stored(where, field, value): + """Whether the message a keys.json / msgids.json entry points at is really in its issue (same mkey / mid).""" + try: + issue = comments.read_json(os.path.join(issue_dir(where[0], where[1]), "issue.json"), None) + return bool(issue) and issue["messages"][where[2]].get(field) == value + except (Error, IndexError, TypeError, KeyError): + return False + + +def ingest_messages(source, authserv_id): + with locked(): + recent = recent_index() + cutoff = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() - SENDER_WINDOW)) + touched = [] + for key in sorted(source.keys(), key=lambda k: [int(x) if x.isdigit() else x for x in re.split(r"(\d+)", k)]): # oldest first, so a reply finds its issue + with locked(): # per message, so `fw issues approve` is not starved by a big backlog + try: + raw = source.read(key) + except KeyError: + continue + except comments.SkipMessage as e: # leave it in the inbox and go on with the others + log_rejection(comments.new_id(), f"skipped {key}: {e}") + print(f"skipped {key}: {e}") + continue + mkey = sha(raw) + keys = comments.read_json(os.path.join(state_dir(), "keys.json"), {}) + if mkey in keys and stored(keys[mkey], "mkey", mkey): # stored already: a crash or overlap left the mail behind + log_rejection(comments.new_id(), "duplicate of a stored report; moved to done") + source.finish(key, raw, "done") + continue + repo = report = thread = mid = None + try: + if len(raw) > MAX_MAIL: + why = f"message over {MAX_MAIL // 1024} KB" + else: + msg = message_from_bytes(raw, policy=policy.default) + repo, why = repo_from_mail(msg) + if repo: + report, why = parse_report(msg, authserv_id, raw) + if report: + thread = thread_of(msg, repo) + mid = (message_ids(msg.get("Message-ID")) or [None])[0] + known = comments.read_json(os.path.join(state_dir(), "msgids.json"), {}) + if mid and sha(mid) in known and stored(known[sha(mid)], "mid", sha(mid)): # the same Message-ID again + report, why = None, "duplicate Message-ID of a stored report" + except (Error, OSError): # state problems abort the run and leave the mail in the inbox + raise + except Exception as e: # a malformed message must not stop the rest + report, why = None, f"unreadable message: {e}" + if report: + report["sender_hash"] = h = sender_hash(report.pop("_address")) + if issue_mode(repo_dir(repo)) != "private" and sum(1 for d in recent.get(h, []) if d >= cutoff) >= SENDER_LIMIT: + report, why = None, f"sender limit reached ({SENDER_LIMIT} per 24 hours)" + if report is None: # rejected mail is not remembered: moved back from bad, it is tried again + cid = comments.new_id() + log_rejection(cid, why) + print(f"bad {cid}: {why}") + source.finish(key, raw, "bad") + continue + issue, new, note = store_report(repo, report, thread, mkey, mid) + if note: + log_rejection(comments.new_id(), f"{repo}#{issue['n']}: {note}") + recent.setdefault(report["sender_hash"], []).append(report["date"]) + print(f"{'new issue' if new else 'reply to'} {repo}#{issue['n']}" + (" with run record" if issue["record"] and new else "")) + if repo not in touched: + touched.append(repo) + source.finish(key, raw, "done") + for repo in touched: + try: + publish(repo) + except (Error, OSError) as e: # the data is stored; `publish` later fixes the pages + print(f"issues: could not publish {repo} ({e}); run `fw issues publish {repo}` later", file=sys.stderr) + + +def message_source(): + if os.path.exists(imap_path()): + return comments.IMAPSource(cfg_path=imap_path(), stem="issues", max_size=MAX_MAIL, log=log_rejection) + return comments.MaildirSource(maildir_path()) + + +def check_mailbox_separate(): + """Refuse to share the comments mailbox: the two jobs would move each other's mail to their bad folders.""" + mine, theirs = imap_path(), comments.imap_config_path() + if not (os.path.exists(mine) and os.path.exists(theirs)): + return + a, b = comments.load_imap_config(mine), comments.load_imap_config(theirs) + if tuple(a[k].casefold() for k in ("host", "user", "folder")) == tuple(b[k].casefold() for k in ("host", "user", "folder")): + raise Error(f"{mine} and {theirs} point at the same mailbox ({a['user']}@{a['host']}, folder {a['folder']}); " + "the bugs mailbox needs its own account or its own \"folder\"") + + +def cmd_ingest(): + source = message_source() + if not source.available(): + return + if isinstance(source, comments.IMAPSource): + check_mailbox_separate() + authserv_id = (comments.load_config().get("authserv_id") or "").strip() + if not authserv_id: + raise Error('authserv_id is empty in sites.json; refusing to ingest (set it to the authserv-id ' + 'your mail server puts at the start of its Authentication-Results header)') + try: + with ingest_lock(): + ingest_messages(source, authserv_id) + finally: + source.close() + + +# ---- moderation ---- + +def cmd_list(repo=None): + for r in ([check_repo(repo)] if repo else stored_repos()): + path = repo_dir(r) + mode = issue_mode(path) if os.path.isdir(path) else "private" + for i in load_issues(r): + flags = ("approved " if i.get("approved") else "") + ("hidden " if i.get("hidden") else "") + ("record " if i.get("record") else "") + shown = "PUBLIC" if os.path.isdir(path) and visible(path, i, mode) else "-" + print(f"{r}#{i['n']} {i['state']:<6} {shown:<6} {mode:<8} {flags}{i['title'][:60]}") + + +def cmd_show(repo, n): + n = check_n(n) + print(json.dumps(load_issue(check_repo(repo), n), indent=2, ensure_ascii=False)) + + +def cmd_change(repo, n, action): + n = check_n(n) + with locked(): + issue = load_issue(check_repo(repo), n) + if action == "approve": + issue["approved"], issue["hidden"] = True, False + for m in issue["messages"]: + m["approved"] = True + elif action == "hide": + issue["approved"], issue["hidden"] = False, True + for m in issue["messages"]: + m["approved"] = False + elif action == "close": + issue["state"] = "closed" + elif action == "reopen": + issue["state"] = "open" + save_issue(issue) + publish(repo) + path = repo_dir(repo) + print(f"{action}: {repo}#{n} ({'public' if os.path.isdir(path) and visible(path, issue) else 'not public'})") + + +def cmd_publish(repo): + publish(repo) + print(f"{repo}: issues published") + + +def cmd_publish_all(): + for r in stored_repos(): + try: + publish(r) + except Error as e: + print(f"issues: {r}: {e}", file=sys.stderr) + print("issues: published") + + +def main(argv): + if not argv or argv[0] in ("-h", "--help", "help"): + print(__doc__) + return 0 + cmd, args = argv[0], argv[1:] + flags = [a for a in args if a.startswith("--")] + args = [a for a in args if not a.startswith("--")] + try: + if cmd == "list" and len(args) <= 1: + cmd_list(args[0] if args else None) + elif cmd == "show" and len(args) == 2: + cmd_show(*args) + elif cmd in ("approve", "hide", "close", "reopen") and len(args) == 2: + cmd_change(args[0], args[1], cmd) + elif cmd == "publish" and len(args) == 1 and set(flags) <= {"--locked"}: + if "--locked" in flags: + publish(args[0], caller_holds_lock=True) + else: + cmd_publish(args[0]) + elif cmd == "publish-all": + cmd_publish_all() + elif cmd == "ingest": + cmd_ingest() + elif cmd == "imap-check": + source = comments.IMAPSource(cfg_path=imap_path(), stem="issues", max_size=MAX_MAIL, log=log_rejection) + if not source.available(): + raise Error(f"no IMAP config at {imap_path()}") + check_mailbox_separate() + try: + source.check() + finally: + source.close() + else: + print("usage: issues.py list|show|approve|hide|close|reopen|publish|publish-all|ingest|imap-check (try --help)", file=sys.stderr) + return 2 + except (Error, OSError) as e: + print(f"issues: {e}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/server/jobs/crontab b/server/jobs/crontab index 7fd77f7..9e105f4 100644 --- a/server/jobs/crontab +++ b/server/jobs/crontab @@ -2,4 +2,5 @@ # Each job script checks its own UTC schedule (if it has one); cron only wakes it hourly. 0 * * * * $HOME/fw/tools/server/jobs/north-star-cleanup.sh >> $HOME/fw/logs/cron-north-star-cleanup.log 2>&1 0 * * * * $HOME/fw/tools/server/jobs/comments-ingest.sh >> $HOME/fw/logs/cron-comments-ingest.log 2>&1 +0 * * * * $HOME/fw/tools/server/jobs/issues-ingest.sh >> $HOME/fw/logs/cron-issues-ingest.log 2>&1 # fw:end diff --git a/server/jobs/issues-ingest.sh b/server/jobs/issues-ingest.sh new file mode 100755 index 0000000..49e4661 --- /dev/null +++ b/server/jobs/issues-ingest.sh @@ -0,0 +1,9 @@ +#!/bin/bash +# Hourly: turn new mail in the bugs mailbox (IMAP, or ~/Maildir/.bugs) into issues (see server/issues.py). +# Quiet unless something happened or failed. +set -euo pipefail +mkdir -p "$HOME/fw/state/locks" +exec 9>"$HOME/fw/state/locks/job-issues-ingest.lock" +flock -n 9 || exit 0 +out=$(python3 "$HOME/fw/tools/server/issues.py" ingest 2>&1) || { echo "== $(date -u '+%F %T UTC') issues ingest failed"; echo "$out"; exit 1; } +[ -z "$out" ] || { echo "== $(date -u '+%F %T UTC') issues ingest"; echo "$out"; } diff --git a/server/site/htaccess b/server/site/htaccess index 2bfef89..f547d28 100644 --- a/server/site/htaccess +++ b/server/site/htaccess @@ -47,6 +47,11 @@ RedirectMatch 404 ^/[^/]+\.git/(config|description|hooks/|logs/|info/exclude|FET <If "%{REQUEST_URI} =~ m#^/[^/]+/(branch/[^/]+/)?files\.json$#"> Header set Cache-Control "no-cache" </If> + # A bug report's run record is read by the replay viewer on apps.fuzzyworld.net. + <If "%{REQUEST_URI} =~ m#^/[^/]+/issues/[0-9]+/record\.json$#"> + Header set Access-Control-Allow-Origin "https://apps.fuzzyworld.net" + Header set Cache-Control "no-cache" + </If> # objects/<sha> is named by its contents, so it never changes. <If "%{REQUEST_URI} =~ m#^/[^/]+/objects/[0-9a-f]{40}$#"> Header set Cache-Control "public, max-age=31536000, immutable" diff --git a/server/site/style.css b/server/site/style.css index 2a60803..eb69758 100644 --- a/server/site/style.css +++ b/server/site/style.css @@ -264,7 +264,7 @@ pre { overflow-wrap: anywhere; } -pre.message { +pre.message, pre.issue-text { white-space: pre-wrap; overflow-wrap: anywhere; } diff --git a/tests/deploy.test.mjs b/tests/deploy.test.mjs new file mode 100644 index 0000000..cff1b10 --- /dev/null +++ b/tests/deploy.test.mjs @@ -0,0 +1,44 @@ +// node --test tests/deploy.test.mjs +import test from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { rsyncArgs, checkReplay, readConfig } from '../lib/deploy.mjs'; + +function configIn(fuzzyworld) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'fw-deploy-test-')); + fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: 'x', fuzzyworld: { app: 'x', ...fuzzyworld } })); + return dir; +} + +test('extra becomes protect filters, keep stays an exclude', () => { + const args = rsyncArgs({ keep: ['app.apk'], extra: ['sim/*.js'] }, 'dist', 'dh:x/'); + assert.deepEqual(args, ['-rlptz', '--delete', '--exclude', '/app.apk', '--filter', 'P /sim/*.js', 'dist/', 'dh:x/']); +}); + +test('no keep or extra: plain rsync', () => { + assert.deepEqual(rsyncArgs({}, 'dist', 'dh:x/'), ['-rlptz', '--delete', 'dist/', 'dh:x/']); +}); + +test('extra entries are validated like keep', () => { + for (const bad of ['/abs', '../up', 'a//b', '', 5]) { + assert.throws(() => readConfig(configIn({ extra: [bad] })), /fuzzyworld\.extra/); + } + assert.ok(readConfig(configIn({ extra: ['sim/*.js', 'a/**/b'] }))); +}); + +test('an entry in both keep and extra is refused', () => { + assert.throws(() => readConfig(configIn({ keep: ['sim/*.js'], extra: ['sim/*.js'] })), /also in keep/); +}); + +test('replay templates', () => { + checkReplay('replay.html?report={record}'); + checkReplay('r/replay.html?x=1&report={record}'); + for (const bad of ['replay.html', '/replay.html?r={record}', '//evil.com/?r={record}', 'https://evil.com/?r={record}', + '../x.html?r={record}', 'javascript:alert(1)//{record}', 'a\\b?r={record}', 'x'.repeat(201) + '{record}', 5, + 'a%2e%2e/x.html?r={record}', 'a%00.html?r={record}', 'a b.html?r={record}', 'a\tb.html?r={record}', 'a\nb.html?r={record}']) { + assert.throws(() => checkReplay(bad), /fuzzyworld\.replay/, String(bad)); + } + assert.throws(() => readConfig(configIn({ replay: 'nope' })), /fuzzyworld\.replay/); +}); diff --git a/tests/fixtures/issues/explodorama-report.eml b/tests/fixtures/issues/explodorama-report.eml new file mode 100644 index 0000000..73eb0e3 --- /dev/null +++ b/tests/fixtures/issues/explodorama-report.eml @@ -0,0 +1,147 @@ +Authentication-Results: mx; dkim=pass header.d=example.org; dmarc=pass header.from=example.org +Date: Tue, 03 Nov 2026 19:51:56 -0800 +From: Reporter <reporter@example.org> +To: bugs-k7q4+explodorama@fuzzyworld.net +Subject: Bug report: version 0.2.0 (rules 43), level 4 +Message-ID: <B3B7666D-1090-4EEF-8F63-2DE90181EF6D@example.org> +MIME-Version: 1.0 +Content-Type: text/plain; + charset=utf-8 +Content-Transfer-Encoding: quoted-printable + +What happened: + +Version 0=2E2=2E0 (rules 43), commit 805e7a9 +Device: TestPhone / Firefox/1.0 +Seed 1464004408, level 4, 66 turns taken, 3 lives, playing; modes: none +Errors caught: none +Run record: {"simVersion":43,"seed":1464004408,"startLevel":1,"startCharge= +":null,"options":{"dev":false,"immortal":false},"actions":[{"id":"move","ta= +rget":{"x":1,"y":2}},{"id":"move","target":{"x":1,"y":3}},{"id":"move","tar= +get":{"x":1,"y":4}},{"id":"bomb","target":{"x":1,"y":5}},{"id":"move","targ= +et":{"x":1,"y":3}},{"id":"move","target":{"x":2,"y":3}},{"id":"wait"},{"id"= +:"wait"},{"id":"move","target":{"x":1,"y":3}},{"id":"move","target":{"x":1,= +"y":4}},{"id":"move","target":{"x":1,"y":5}},{"id":"move","target":{"x":1,"= +y":6}},{"id":"choose","kind":"kick"},{"id":"move","target":{"x":1,"y":5}},{= +"id":"move","target":{"x":2,"y":5}},{"id":"bomb","target":{"x":3,"y":5}},{"= +id":"move","target":{"x":1,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"wait"= +},{"id":"move","target":{"x":2,"y":5}},{"id":"move","target":{"x":3,"y":5}}= +,{"id":"move","target":{"x":4,"y":5}},{"id":"bomb","target":{"x":5,"y":5}},= +{"id":"kick","target":{"x":5,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"wai= +t"},{"id":"bomb","target":{"x":5,"y":5}},{"id":"move","target":{"x":3,"y":5= +}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":4,"= +y":5}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"= +},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"i= +d":"bomb","target":{"x":5,"y":5}},{"id":"move","target":{"x":3,"y":5}},{"id= +":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":4,"y":5}},= +{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":6}},{= +"id":"move","target":{"x":5,"y":7}},{"id":"move","target":{"x":5,"y":8}},{"= +id":"move","target":{"x":5,"y":9}},{"id":"continue"},{"id":"bomb","target":= +{"x":2,"y":1}},{"id":"kick","target":{"x":2,"y":1}},{"id":"wait"},{"id":"wa= +it"},{"id":"wait"},{"id":"bomb","target":{"x":1,"y":2}},{"id":"kick","targe= +t":{"x":1,"y":2}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"bomb","t= +arget":{"x":1,"y":2}},{"id":"kick","target":{"x":1,"y":2}},{"id":"wait"},{"= +id":"wait"},{"id":"wait"},{"id":"bomb","target":{"x":1,"y":2}},{"id":"kick"= +,"target":{"x":1,"y":2}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"w= +ait"},{"id":"move","target":{"x":1,"y":2}},{"id":"move","target":{"x":1,"y"= +:3}},{"id":"move","target":{"x":1,"y":4}},{"id":"move","target":{"x":1,"y":= +5}},{"id":"move","target":{"x":1,"y":6}},{"id":"move","target":{"x":1,"y":7= +}},{"id":"move","target":{"x":2,"y":7}},{"id":"move","target":{"x":3,"y":7}= +},{"id":"move","target":{"x":4,"y":7}},{"id":"move","target":{"x":5,"y":7}}= +,{"id":"move","target":{"x":5,"y":6}},{"id":"move","target":{"x":5,"y":5}},= +{"id":"move","target":{"x":5,"y":4}},{"id":"move","target":{"x":5,"y":3}},{= +"id":"move","target":{"x":6,"y":3}},{"id":"move","target":{"x":7,"y":3}},{"= +id":"bomb","target":{"x":7,"y":4}},{"id":"kick","target":{"x":7,"y":4}},{"i= +d":"wait"},{"id":"wait"},{"id":"move","target":{"x":6,"y":3}},{"id":"move",= +"target":{"x":5,"y":3}},{"id":"bomb","target":{"x":5,"y":4}},{"id":"kick","= +target":{"x":5,"y":4}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wai= +t"},{"id":"move","target":{"x":5,"y":2}},{"id":"wait"},{"id":"move","target= +":{"x":5,"y":3}},{"id":"move","target":{"x":5,"y":4}},{"id":"move","target"= +:{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":6}},{"id":"move","target":= +{"x":5,"y":7}},{"id":"move","target":{"x":4,"y":7}},{"id":"move","target":{= +"x":3,"y":7}},{"id":"move","target":{"x":2,"y":7}},{"id":"move","target":{"= +x":1,"y":7}},{"id":"bomb","target":{"x":1,"y":6}},{"id":"kick","target":{"x= +":1,"y":6}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target"= +:{"x":2,"y":7}},{"id":"move","target":{"x":3,"y":7}},{"id":"move","target":= +{"x":4,"y":7}},{"id":"move","target":{"x":5,"y":7}},{"id":"move","target":{= +"x":6,"y":7}},{"id":"move","target":{"x":7,"y":7}},{"id":"bomb","target":{"= +x":7,"y":6}},{"id":"kick","target":{"x":7,"y":6}},{"id":"wait"},{"id":"move= +","target":{"x":7,"y":6}},{"id":"wait"},{"id":"bomb","target":{"x":7,"y":5}= +},{"id":"kick","target":{"x":7,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"w= +ait"},{"id":"move","target":{"x":7,"y":5}},{"id":"bomb","target":{"x":7,"y"= +:4}},{"id":"move","target":{"x":7,"y":6}},{"id":"wait"},{"id":"wait"},{"id"= +:"wait"},{"id":"move","target":{"x":7,"y":5}},{"id":"bomb","target":{"x":7,= +"y":4}},{"id":"wait"},{"id":"kick","target":{"x":7,"y":4}},{"id":"wait"},{"= +id":"move","target":{"x":7,"y":4}},{"id":"move","target":{"x":7,"y":3}},{"i= +d":"move","target":{"x":7,"y":2}},{"id":"move","target":{"x":7,"y":3}},{"id= +":"move","target":{"x":6,"y":3}},{"id":"move","target":{"x":5,"y":3}},{"id"= +:"move","target":{"x":4,"y":3}},{"id":"choose","kind":"tnt"},{"id":"move","= +target":{"x":5,"y":3}},{"id":"move","target":{"x":5,"y":4}},{"id":"move","t= +arget":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":6}},{"id":"move","ta= +rget":{"x":5,"y":7}},{"id":"move","target":{"x":6,"y":7}},{"id":"move","tar= +get":{"x":7,"y":7}},{"id":"continue"},{"id":"move","target":{"x":1,"y":2}},= +{"id":"bomb","target":{"x":1,"y":3}},{"id":"kick","target":{"x":1,"y":3}},{= +"id":"move","target":{"x":1,"y":1}},{"id":"wait"},{"id":"wait"},{"id":"bomb= +","target":{"x":1,"y":2}},{"id":"kick","target":{"x":1,"y":2}},{"id":"wait"= +},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":1,"y":2}},{"id":"m= +ove","target":{"x":1,"y":3}},{"id":"move","target":{"x":1,"y":4}},{"id":"mo= +ve","target":{"x":1,"y":5}},{"id":"move","target":{"x":1,"y":6}},{"id":"mov= +e","target":{"x":1,"y":7}},{"id":"move","target":{"x":1,"y":8}},{"id":"move= +","target":{"x":1,"y":9}},{"id":"bomb","target":{"x":2,"y":9}},{"id":"kick"= +,"target":{"x":2,"y":9}},{"id":"move","target":{"x":1,"y":8}},{"id":"wait"}= +,{"id":"wait"},{"id":"move","target":{"x":1,"y":9}},{"id":"move","target":{= +"x":2,"y":9}},{"id":"move","target":{"x":3,"y":9}},{"id":"move","target":{"= +x":4,"y":9}},{"id":"sellCharge"},{"id":"move","target":{"x":5,"y":9}},{"id"= +:"move","target":{"x":4,"y":9}},{"id":"move","target":{"x":3,"y":9}},{"id":= +"bomb","target":{"x":3,"y":8}},{"id":"move","target":{"x":2,"y":9}},{"id":"= +wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":3,"y":9}},{"i= +d":"move","target":{"x":3,"y":8}},{"id":"move","target":{"x":3,"y":7}},{"id= +":"move","target":{"x":4,"y":7}},{"id":"move","target":{"x":5,"y":7}},{"id"= +:"bomb","target":{"x":5,"y":6}},{"id":"kick","target":{"x":5,"y":6}},{"id":= +"move","target":{"x":4,"y":7}},{"id":"wait"},{"id":"wait"},{"id":"move","ta= +rget":{"x":5,"y":7}},{"id":"move","target":{"x":5,"y":6}},{"id":"move","tar= +get":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":4}},{"id":"choose","ki= +nd":"diagonal"},{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":= +{"x":5,"y":6}},{"id":"diagonal","target":{"x":6,"y":7}},{"id":"diagonal","t= +arget":{"x":7,"y":8}},{"id":"continue"},{"id":"buy","kind":"bombPass"},{"id= +":"buy","kind":"fusePlus"},{"id":"continue"},{"id":"move","target":{"x":1,"= +y":2}},{"id":"diagonal","target":{"x":2,"y":3}},{"id":"diagonal","target":{= +"x":3,"y":4}},{"id":"move","target":{"x":3,"y":5}},{"id":"move","target":{"= +x":2,"y":5}},{"id":"move","target":{"x":1,"y":5}},{"id":"bomb","target":{"x= +":1,"y":6}},{"id":"move","target":{"x":2,"y":5}},{"id":"wait"},{"id":"wait"= +},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":3,"y":5}},{"id":"m= +ove","target":{"x":4,"y":5}},{"id":"move","target":{"x":5,"y":5}},{"id":"bo= +mb","target":{"x":6,"y":5}},{"id":"move","target":{"x":5,"y":6}},{"id":"mov= +e","target":{"x":5,"y":7}},{"id":"move","target":{"x":6,"y":7}},{"id":"move= +","target":{"x":7,"y":7}},{"id":"move","target":{"x":7,"y":6}},{"id":"move"= +,"target":{"x":7,"y":5}},{"id":"move","target":{"x":8,"y":5}},{"id":"move",= +"target":{"x":7,"y":5}},{"id":"move","target":{"x":6,"y":5}},{"id":"diagona= +l","target":{"x":5,"y":4}},{"id":"diagonal","target":{"x":6,"y":3}},{"id":"= +bomb","target":{"x":7,"y":3}},{"id":"diagonal","target":{"x":5,"y":4}},{"id= +":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"diagonal","targe= +t":{"x":4,"y":3}},{"id":"move","target":{"x":3,"y":3}},{"id":"move","target= +":{"x":2,"y":3}},{"id":"diagonal","target":{"x":1,"y":2}},{"id":"bomb","tar= +get":{"x":1,"y":1}},{"id":"diagonal","target":{"x":2,"y":3}},{"id":"wait"},= +{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"diagonal","target":{"x":1,= +"y":2}},{"id":"diagonal","target":{"x":2,"y":1}},{"id":"move","target":{"x"= +:3,"y":1}},{"id":"move","target":{"x":2,"y":1}},{"id":"diagonal","target":{= +"x":1,"y":2}},{"id":"diagonal","target":{"x":2,"y":3}},{"id":"diagonal","ta= +rget":{"x":3,"y":4}},{"id":"diagonal","target":{"x":4,"y":5}},{"id":"diagon= +al","target":{"x":5,"y":6}},{"id":"diagonal","target":{"x":6,"y":7}},{"id":= +"diagonal","target":{"x":5,"y":6}},{"id":"move","target":{"x":5,"y":5}},{"i= +d":"move","target":{"x":5,"y":4}},{"id":"move","target":{"x":5,"y":3}},{"id= +":"bomb","target":{"x":5,"y":4}},{"id":"kick","target":{"x":5,"y":4}},{"id"= +:"wait"},{"id":"wait"},{"id":"wait"},{"id":"bomb","target":{"x":5,"y":4}},{= +"id":"move","target":{"x":5,"y":4}},{"id":"move","target":{"x":5,"y":5}},{"= +id":"move","target":{"x":5,"y":4}}]} + +Got stuck: fired a bomb before the last enemy died, autopilot walked me in= +to death=2E + +--- +Version 0=2E2=2E0 (rules 43), commit 805e7a9 +Device: TestPhone / Firefox/1.0 +Seed 1464004408, level 4, 66 turns taken, 3 lives, playing; modes: none +Errors caught: none +Run record: too long for an email link=2E The full details are on your cli= +pboard: please paste them here=2E diff --git a/tests/fixtures/issues/q101-record.json b/tests/fixtures/issues/q101-record.json new file mode 100644 index 0000000..1096cb1 --- /dev/null +++ b/tests/fixtures/issues/q101-record.json @@ -0,0 +1 @@ +{"simVersion":43,"seed":1464004408,"startLevel":1,"startCharge":null,"options":{"dev":false,"immortal":false},"actions":[{"id":"move","target":{"x":1,"y":2}},{"id":"move","target":{"x":1,"y":3}},{"id":"move","target":{"x":1,"y":4}},{"id":"bomb","target":{"x":1,"y":5}},{"id":"move","target":{"x":1,"y":3}},{"id":"move","target":{"x":2,"y":3}},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":1,"y":3}},{"id":"move","target":{"x":1,"y":4}},{"id":"move","target":{"x":1,"y":5}},{"id":"move","target":{"x":1,"y":6}},{"id":"choose","kind":"kick"},{"id":"move","target":{"x":1,"y":5}},{"id":"move","target":{"x":2,"y":5}},{"id":"bomb","target":{"x":3,"y":5}},{"id":"move","target":{"x":1,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":2,"y":5}},{"id":"move","target":{"x":3,"y":5}},{"id":"move","target":{"x":4,"y":5}},{"id":"bomb","target":{"x":5,"y":5}},{"id":"kick","target":{"x":5,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"bomb","target":{"x":5,"y":5}},{"id":"move","target":{"x":3,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":4,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"bomb","target":{"x":5,"y":5}},{"id":"move","target":{"x":3,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":4,"y":5}},{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":6}},{"id":"move","target":{"x":5,"y":7}},{"id":"move","target":{"x":5,"y":8}},{"id":"move","target":{"x":5,"y":9}},{"id":"continue"},{"id":"bomb","target":{"x":2,"y":1}},{"id":"kick","target":{"x":2,"y":1}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"bomb","target":{"x":1,"y":2}},{"id":"kick","target":{"x":1,"y":2}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"bomb","target":{"x":1,"y":2}},{"id":"kick","target":{"x":1,"y":2}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"bomb","target":{"x":1,"y":2}},{"id":"kick","target":{"x":1,"y":2}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":1,"y":2}},{"id":"move","target":{"x":1,"y":3}},{"id":"move","target":{"x":1,"y":4}},{"id":"move","target":{"x":1,"y":5}},{"id":"move","target":{"x":1,"y":6}},{"id":"move","target":{"x":1,"y":7}},{"id":"move","target":{"x":2,"y":7}},{"id":"move","target":{"x":3,"y":7}},{"id":"move","target":{"x":4,"y":7}},{"id":"move","target":{"x":5,"y":7}},{"id":"move","target":{"x":5,"y":6}},{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":4}},{"id":"move","target":{"x":5,"y":3}},{"id":"move","target":{"x":6,"y":3}},{"id":"move","target":{"x":7,"y":3}},{"id":"bomb","target":{"x":7,"y":4}},{"id":"kick","target":{"x":7,"y":4}},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":6,"y":3}},{"id":"move","target":{"x":5,"y":3}},{"id":"bomb","target":{"x":5,"y":4}},{"id":"kick","target":{"x":5,"y":4}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":5,"y":2}},{"id":"wait"},{"id":"move","target":{"x":5,"y":3}},{"id":"move","target":{"x":5,"y":4}},{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":6}},{"id":"move","target":{"x":5,"y":7}},{"id":"move","target":{"x":4,"y":7}},{"id":"move","target":{"x":3,"y":7}},{"id":"move","target":{"x":2,"y":7}},{"id":"move","target":{"x":1,"y":7}},{"id":"bomb","target":{"x":1,"y":6}},{"id":"kick","target":{"x":1,"y":6}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":2,"y":7}},{"id":"move","target":{"x":3,"y":7}},{"id":"move","target":{"x":4,"y":7}},{"id":"move","target":{"x":5,"y":7}},{"id":"move","target":{"x":6,"y":7}},{"id":"move","target":{"x":7,"y":7}},{"id":"bomb","target":{"x":7,"y":6}},{"id":"kick","target":{"x":7,"y":6}},{"id":"wait"},{"id":"move","target":{"x":7,"y":6}},{"id":"wait"},{"id":"bomb","target":{"x":7,"y":5}},{"id":"kick","target":{"x":7,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":7,"y":5}},{"id":"bomb","target":{"x":7,"y":4}},{"id":"move","target":{"x":7,"y":6}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":7,"y":5}},{"id":"bomb","target":{"x":7,"y":4}},{"id":"wait"},{"id":"kick","target":{"x":7,"y":4}},{"id":"wait"},{"id":"move","target":{"x":7,"y":4}},{"id":"move","target":{"x":7,"y":3}},{"id":"move","target":{"x":7,"y":2}},{"id":"move","target":{"x":7,"y":3}},{"id":"move","target":{"x":6,"y":3}},{"id":"move","target":{"x":5,"y":3}},{"id":"move","target":{"x":4,"y":3}},{"id":"choose","kind":"tnt"},{"id":"move","target":{"x":5,"y":3}},{"id":"move","target":{"x":5,"y":4}},{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":6}},{"id":"move","target":{"x":5,"y":7}},{"id":"move","target":{"x":6,"y":7}},{"id":"move","target":{"x":7,"y":7}},{"id":"continue"},{"id":"move","target":{"x":1,"y":2}},{"id":"bomb","target":{"x":1,"y":3}},{"id":"kick","target":{"x":1,"y":3}},{"id":"move","target":{"x":1,"y":1}},{"id":"wait"},{"id":"wait"},{"id":"bomb","target":{"x":1,"y":2}},{"id":"kick","target":{"x":1,"y":2}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":1,"y":2}},{"id":"move","target":{"x":1,"y":3}},{"id":"move","target":{"x":1,"y":4}},{"id":"move","target":{"x":1,"y":5}},{"id":"move","target":{"x":1,"y":6}},{"id":"move","target":{"x":1,"y":7}},{"id":"move","target":{"x":1,"y":8}},{"id":"move","target":{"x":1,"y":9}},{"id":"bomb","target":{"x":2,"y":9}},{"id":"kick","target":{"x":2,"y":9}},{"id":"move","target":{"x":1,"y":8}},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":1,"y":9}},{"id":"move","target":{"x":2,"y":9}},{"id":"move","target":{"x":3,"y":9}},{"id":"move","target":{"x":4,"y":9}},{"id":"sellCharge"},{"id":"move","target":{"x":5,"y":9}},{"id":"move","target":{"x":4,"y":9}},{"id":"move","target":{"x":3,"y":9}},{"id":"bomb","target":{"x":3,"y":8}},{"id":"move","target":{"x":2,"y":9}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":3,"y":9}},{"id":"move","target":{"x":3,"y":8}},{"id":"move","target":{"x":3,"y":7}},{"id":"move","target":{"x":4,"y":7}},{"id":"move","target":{"x":5,"y":7}},{"id":"bomb","target":{"x":5,"y":6}},{"id":"kick","target":{"x":5,"y":6}},{"id":"move","target":{"x":4,"y":7}},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":5,"y":7}},{"id":"move","target":{"x":5,"y":6}},{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":4}},{"id":"choose","kind":"diagonal"},{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":6}},{"id":"diagonal","target":{"x":6,"y":7}},{"id":"diagonal","target":{"x":7,"y":8}},{"id":"continue"},{"id":"buy","kind":"bombPass"},{"id":"buy","kind":"fusePlus"},{"id":"continue"},{"id":"move","target":{"x":1,"y":2}},{"id":"diagonal","target":{"x":2,"y":3}},{"id":"diagonal","target":{"x":3,"y":4}},{"id":"move","target":{"x":3,"y":5}},{"id":"move","target":{"x":2,"y":5}},{"id":"move","target":{"x":1,"y":5}},{"id":"bomb","target":{"x":1,"y":6}},{"id":"move","target":{"x":2,"y":5}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"move","target":{"x":3,"y":5}},{"id":"move","target":{"x":4,"y":5}},{"id":"move","target":{"x":5,"y":5}},{"id":"bomb","target":{"x":6,"y":5}},{"id":"move","target":{"x":5,"y":6}},{"id":"move","target":{"x":5,"y":7}},{"id":"move","target":{"x":6,"y":7}},{"id":"move","target":{"x":7,"y":7}},{"id":"move","target":{"x":7,"y":6}},{"id":"move","target":{"x":7,"y":5}},{"id":"move","target":{"x":8,"y":5}},{"id":"move","target":{"x":7,"y":5}},{"id":"move","target":{"x":6,"y":5}},{"id":"diagonal","target":{"x":5,"y":4}},{"id":"diagonal","target":{"x":6,"y":3}},{"id":"bomb","target":{"x":7,"y":3}},{"id":"diagonal","target":{"x":5,"y":4}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"diagonal","target":{"x":4,"y":3}},{"id":"move","target":{"x":3,"y":3}},{"id":"move","target":{"x":2,"y":3}},{"id":"diagonal","target":{"x":1,"y":2}},{"id":"bomb","target":{"x":1,"y":1}},{"id":"diagonal","target":{"x":2,"y":3}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"diagonal","target":{"x":1,"y":2}},{"id":"diagonal","target":{"x":2,"y":1}},{"id":"move","target":{"x":3,"y":1}},{"id":"move","target":{"x":2,"y":1}},{"id":"diagonal","target":{"x":1,"y":2}},{"id":"diagonal","target":{"x":2,"y":3}},{"id":"diagonal","target":{"x":3,"y":4}},{"id":"diagonal","target":{"x":4,"y":5}},{"id":"diagonal","target":{"x":5,"y":6}},{"id":"diagonal","target":{"x":6,"y":7}},{"id":"diagonal","target":{"x":5,"y":6}},{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":4}},{"id":"move","target":{"x":5,"y":3}},{"id":"bomb","target":{"x":5,"y":4}},{"id":"kick","target":{"x":5,"y":4}},{"id":"wait"},{"id":"wait"},{"id":"wait"},{"id":"bomb","target":{"x":5,"y":4}},{"id":"move","target":{"x":5,"y":4}},{"id":"move","target":{"x":5,"y":5}},{"id":"move","target":{"x":5,"y":4}}]} \ No newline at end of file diff --git a/tests/test_issues.py b/tests/test_issues.py new file mode 100644 index 0000000..fc730b1 --- /dev/null +++ b/tests/test_issues.py @@ -0,0 +1,873 @@ +"""Tests for server/issues.py (bug reports by email become issue pages, run records, replay links). + +Run with: python3 -m unittest discover tests +""" + +import contextlib +import importlib.util +import io +import itertools +import json +import mailbox +import os +import shutil +import subprocess +import tempfile +import unittest +from unittest import mock + +ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..") +FIXTURES = os.path.join(ROOT, "tests", "fixtures", "issues") +spec = importlib.util.spec_from_file_location("issues", os.path.join(ROOT, "server", "issues.py")) +issues = importlib.util.module_from_spec(spec) +spec.loader.exec_module(issues) + +AUTH = "mx; dkim=pass header.d=example.org; dmarc=pass header.from=example.org" +_counter = itertools.count(1) + + +def mail(subject="Bug: crash on start", body="It crashed.\n", repo="demo", sender="Rita <reporter@example.org>", + auth=(AUTH,), extra_headers=(), mid=None, to=None): + mid = mid or f"<{next(_counter)}@test>" + lines = [f"Message-ID: {mid}", *extra_headers] + lines += [f"Authentication-Results: {a}" for a in auth] + lines += [f"From: {sender}", f"To: {to or f'bugs-k7q4+{repo}@fuzzyworld.net'}", f"Subject: {subject}", + "Content-Type: text/plain; charset=utf-8", "", body] + return "\n".join(lines).encode() + + +def record(**over): + r = {"simVersion": 43, "seed": 7, "startLevel": 1, "options": {"dev": False}, "actions": [{"id": "wait"}]} + r.update(over) + return json.dumps(r, separators=(",", ":")) + + +def rd(path): + with open(path) as f: + return f.read() + + +def read_tree(root): + for dirpath, _, names in os.walk(root): + for n in names: + p = os.path.join(dirpath, n) + with open(p, "rb") as f: + yield p, f.read().decode("utf-8", "replace") + + +class Base(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmp) + j = lambda *p: os.path.join(self.tmp, *p) + self.git, self.site, self.apps, self.fwstate = j("git"), j("site"), j("apps"), j("fwstate") + self.state = j("issues") + self.maildir = j("Maildir", ".bugs") + with open(j("sites.json"), "w") as f: + json.dump({"authserv_id": "mx"}, f) + env = {"FW_ISSUES_DIR": self.state, "FW_COMMENTS_SITES": j("sites.json"), "FW_ISSUES_MAILDIR": self.maildir} + old = {k: os.environ.get(k) for k in env} + os.environ.update(env) + self.addCleanup(lambda: [os.environ.pop(k) if v is None else os.environ.__setitem__(k, v) for k, v in old.items()]) + for name, value in (("GIT_ROOT", self.git), ("SITE", self.site), ("APPS", self.apps), ("STATE", self.fwstate)): + p = mock.patch.object(issues.gen, name, value) + p.start() + self.addCleanup(p.stop) + self.summaries = [] + p = mock.patch.object(issues, "refresh_summary", lambda repo: self.summaries.append(repo)) + p.start() + self.addCleanup(p.stop) + self.repo("demo", "public", "public") + + def repo(self, name, visibility="public", mode=None): + path = os.path.join(self.git, name + ".git") + subprocess.run(["git", "init", "--bare", "--quiet", path], check=True) + subprocess.run(["git", "-C", path, "config", "fuzzyworld.visibility", visibility], check=True) + if mode is not None: + subprocess.run(["git", "-C", path, "config", "fuzzyworld.issues", mode], check=True) + return path + + def ingest(self, *raws): + os.makedirs(os.path.dirname(self.maildir), exist_ok=True) + box = mailbox.Maildir(self.maildir, create=True) + for raw in raws: + box.add(raw) + out = io.StringIO() + with contextlib.redirect_stdout(out): + issues.cmd_ingest() + return out.getvalue() + + def cli(self, *argv): + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = issues.main(list(argv)) + return code, out.getvalue(), err.getvalue() + + def page(self, repo="demo", *parts): + return os.path.join(self.site, repo, "issues", *parts) + + def count(self, outcome): + path = self.maildir + "." + outcome + return len(mailbox.Maildir(path, create=False).keys()) if os.path.isdir(path) else 0 + + def rejected(self): + try: + return rd(os.path.join(self.state, "rejected.log")) + except FileNotFoundError: + return "" + + +class Modes(Base): + def test_public_publishes_at_once(self): + self.ingest(mail()) + self.assertTrue(os.path.isfile(self.page("demo", "index.html"))) + text = rd(self.page("demo", "1", "index.html")) + self.assertIn("crash on start", text) + self.assertIn("It crashed.", text) + self.assertIn('<pre class="issue-text">', text) + self.assertEqual(self.summaries, ["demo"]) # the index appeared: the summary link is refreshed + + def test_reviewed_waits_for_approve_and_hide_removes(self): + self.repo("rev", "public", "reviewed") + self.ingest(mail(repo="rev")) + self.assertFalse(os.path.exists(self.page("rev"))) + self.assertEqual(self.cli("approve", "rev", "1")[0], 0) + self.assertTrue(os.path.isfile(self.page("rev", "1", "index.html"))) + self.assertEqual(self.cli("hide", "rev", "1")[0], 0) + self.assertFalse(os.path.exists(self.page("rev"))) + + def test_hide_beats_public_mode(self): + self.ingest(mail()) + self.cli("hide", "demo", "1") + self.assertFalse(os.path.exists(self.page("demo"))) + self.ingest(mail(subject="Another")) + self.assertFalse(os.path.exists(self.page("demo", "1"))) + self.assertTrue(os.path.isfile(self.page("demo", "2", "index.html"))) + self.assertNotIn("crash on start", rd(self.page("demo", "index.html"))) + + def test_private_and_unknown_modes_publish_nothing(self): + for name, mode in (("p1", "private"), ("p2", "pubic"), ("p3", ""), ("p4", None)): + self.repo(name, "public", mode) + self.ingest(mail(repo=name)) + self.assertFalse(os.path.exists(self.page(name)), name) + self.assertEqual(issues.issue_mode(os.path.join(self.git, name + ".git")), "private") + self.assertTrue(os.path.isfile(os.path.join(self.state, "repos", name, "1", "issue.json"))) + + def test_private_repo_publishes_nothing_even_in_public_mode(self): + self.repo("secret", "private", "public") + self.ingest(mail(repo="secret")) + self.cli("approve", "secret", "1") + self.assertFalse(os.path.exists(os.path.join(self.site, "secret"))) + + def test_close_and_reopen(self): + self.ingest(mail()) + self.cli("close", "demo", "1") + self.assertIn("closed", rd(self.page("demo", "index.html"))) + self.cli("reopen", "demo", "1") + self.assertNotIn("closed", rd(self.page("demo", "index.html"))) + + def test_unauthenticated_mail_is_held_in_public_mode(self): + self.ingest(mail(auth=("mx; dkim=fail",))) + self.assertFalse(os.path.exists(self.page())) + stored = json.load(open(os.path.join(self.state, "repos", "demo", "1", "issue.json"))) + self.assertFalse(stored["messages"][0]["authenticated"]) + self.cli("approve", "demo", "1") + self.assertTrue(os.path.isfile(self.page("demo", "1", "index.html"))) + + def test_foreign_authentication_header_counts_as_unauthenticated(self): + self.ingest(mail(auth=("evil; dkim=pass header.d=example.org",))) + self.assertFalse(os.path.exists(self.page())) + + def test_unauthenticated_reply_is_not_shown_until_approved(self): + first = mail(mid="<a@test>") + self.ingest(first, mail(mid="<b@test>", body="Spoofed reply.\n", auth=(), extra_headers=("In-Reply-To: <a@test>",))) + self.assertNotIn("Spoofed", rd(self.page("demo", "1", "index.html"))) + self.cli("approve", "demo", "1") + self.assertIn("Spoofed", rd(self.page("demo", "1", "index.html"))) + + def test_sender_limit_in_public_mode(self): + self.ingest(*[mail(subject=f"Bug {i}") for i in range(issues.SENDER_LIMIT + 2)]) + self.assertEqual(len(issues.issue_numbers("demo")), issues.SENDER_LIMIT) + self.assertEqual(self.count("bad"), 2) + + +class Intake(Base): + def test_reply_joins_the_issue(self): + self.ingest(mail(mid="<first@test>"), + mail(mid="<second@test>", subject="Re: Bug", body="More detail.\n", + extra_headers=("In-Reply-To: <first@test>", "References: <first@test>"))) + self.assertEqual(issues.issue_numbers("demo"), [1]) + stored = json.load(open(os.path.join(self.state, "repos", "demo", "1", "issue.json"))) + self.assertEqual(len(stored["messages"]), 2) + text = rd(self.page("demo", "1", "index.html")) + self.assertIn("It crashed.", text) + self.assertIn("More detail.", text) + + def test_reply_via_references_only_and_unknown_parent(self): + self.ingest(mail(mid="<first@test>"), mail(mid="<x@test>", extra_headers=("References: <old@test> <first@test>",)), + mail(mid="<y@test>", extra_headers=("In-Reply-To: <nobody@test>",))) + self.assertEqual(issues.issue_numbers("demo"), [1, 2]) + self.assertEqual(len(issues.load_issue("demo", 1)["messages"]), 2) + + def test_numbers_count_up_per_repo(self): + self.repo("other", "public", "public") + self.ingest(mail(), mail(), mail(repo="other")) + self.assertEqual(issues.issue_numbers("demo"), [1, 2]) + self.assertEqual(issues.issue_numbers("other"), [1]) + + def test_unknown_repo_and_bad_addresses_go_to_bad(self): + self.ingest(mail(repo="nosuchrepo"), mail(repo="-evil"), mail(to="somebody@example.org"), + mail(to="bugs-k7q4+demo@fuzzyworld.net, bugs-k7q4+other@fuzzyworld.net")) + self.assertEqual(self.count("bad"), 4) + self.assertEqual(issues.stored_repos(), []) + self.assertIn("no such repo", self.rejected()) + + def test_address_in_cc_or_x_original_to(self): + self.ingest(mail(to="list@example.org", extra_headers=("X-Original-To: bugs-k7q4+demo@fuzzyworld.net",))) + self.assertEqual(issues.issue_numbers("demo"), [1]) + + def test_repo_with_dot_git_in_the_address_is_refused(self): + self.ingest(mail(repo="demo.git")) + self.assertEqual(self.count("bad"), 1) + + def test_same_mail_twice_is_stored_once(self): + raw = mail() + self.ingest(raw, raw) + self.assertEqual(issues.issue_numbers("demo"), [1]) + self.assertEqual(len(issues.load_issue("demo", 1)["messages"]), 1) + self.assertEqual(self.count("done"), 2) + + def test_oversize_mail_goes_to_bad(self): + with mock.patch.object(issues, "MAX_MAIL", 2000): + self.ingest(mail(body="x" * 5000)) + self.assertEqual(self.count("bad"), 1) + self.assertEqual(issues.stored_repos(), []) + + def test_empty_text_and_two_froms_go_to_bad(self): + self.ingest(mail(body=" \n"), mail(extra_headers=("From: Other <o@example.org>",))) + self.assertEqual(self.count("bad"), 2) + + def test_no_address_reaches_any_file(self): + body = "Mail me at secret.person@example.net or <other@example.com>.\nDevice: Pixel (sam@example.org)\n" + self.ingest(mail(subject="Bug from boss@example.net", body=body, sender="Rita Reporter <reporter@example.org>", + mid="<id-1@example.org>")) + found = 0 + for root in (self.site, self.state): + for path, text in read_tree(root): + found += 1 + for needle in ("@example", "Rita", "id-1"): + self.assertNotIn(needle, text, path) + self.assertGreater(found, 3) + page = rd(self.page("demo", "1", "index.html")) + self.assertIn("[address hidden]", page) + self.assertNotIn("Device:", page) + self.assertNotIn("sender_hash", page) + stored = rd(os.path.join(self.state, "repos", "demo", "1", "issue.json")) + self.assertIn('"author": "Reporter"', stored) + + def test_script_is_escaped(self): + self.ingest(mail(subject="<script>alert(1)</script>", body="<script>alert(2)</script> & <b>\n")) + for p in (self.page("demo", "index.html"), self.page("demo", "1", "index.html")): + text = rd(p) + self.assertNotIn("<script>alert", text) + self.assertIn("<script>alert", text) + self.assertNotIn("<script", rd(self.page("demo", "1", "index.html")).replace("<script", "")) + + def test_sender_hash_uses_its_own_secret(self): + self.ingest(mail()) + self.assertTrue(os.path.isfile(os.path.join(self.state, "secret"))) + self.assertEqual(oct(os.stat(os.path.join(self.state, "secret")).st_mode & 0o777), "0o600") + + def test_publish_takes_the_repo_lock_unless_told(self): + self.ingest(mail()) + self.assertTrue(os.path.isfile(os.path.join(self.fwstate, "locks", "demo.lock"))) + self.assertEqual(self.cli("publish", "demo", "--locked")[0], 0) + + def test_list_show_and_usage(self): + self.ingest(mail()) + code, out, _ = self.cli("list") + self.assertIn("demo#1", out) + self.assertIn("PUBLIC", out) + self.assertIn("crash on start", self.cli("show", "demo", "1")[1]) + self.assertEqual(self.cli("show", "demo", "9")[0], 1) + self.assertEqual(self.cli("show", "../x", "1")[0], 1) + self.assertEqual(self.cli("nonsense")[0], 2) + self.assertEqual(self.cli("publish-all")[0], 0) + + +class Records(Base): + def eml(self): + with open(os.path.join(FIXTURES, "explodorama-report.eml"), "rb") as f: + return f.read() + + def fixture(self): + with open(os.path.join(FIXTURES, "q101-record.json")) as f: + return json.load(f) + + def setUp(self): + super().setUp() + self.repo("explodorama", "public", "reviewed") + + def stored_record(self, repo="explodorama", n=1): + with open(os.path.join(self.state, "repos", repo, str(n), "record.json")) as f: + return json.load(f) + + def test_record_from_the_real_mail_matches_the_fixture(self): + self.ingest(self.eml()) + rec = self.stored_record() + self.assertEqual(rec, self.fixture()) + self.assertEqual(rec["seed"], 1464004408) + self.assertEqual(len(rec["actions"]), 290) + self.assertEqual(len(rec["actions"]), len(self.fixture()["actions"])) + raw = rd(os.path.join(self.state, "repos", "explodorama", "1", "record.json")) + self.assertEqual(raw, json.dumps(self.fixture(), separators=(",", ":"))) # re-serialized, compact + + def test_find_record_skips_the_too_long_footer(self): + text = "x\nRun record: " + record() + "\n\n---\nRun record: too long for an email link. Paste it.\n" + rec, why = issues.find_record(text) + self.assertEqual(rec["seed"], 7) + rec, why = issues.find_record("Run record: too long for an email link.\n") + self.assertEqual((rec, why), (None, "")) + # an invalid line first, a valid one later: the valid one wins + rec, _ = issues.find_record("Run record: {bad}\nRun record: " + record(seed=9) + "\n") + self.assertEqual(rec["seed"], 9) + + def test_page_leaves_out_device_and_record_lines(self): + self.ingest(self.eml()) + stored = rd(os.path.join(self.state, "repos", "explodorama", "1", "issue.json")) + self.cli("approve", "explodorama", "1") + for text in (stored, rd(self.page("explodorama", "1", "index.html"))): + self.assertNotIn("Device:", text) + self.assertNotIn("Firefox", text) + self.assertNotIn("TestPhone", text) + self.assertNotIn("Run record", text) + self.assertIn("autopilot walked me into death", text) # the reporter's words come after the 9 KB record + self.assertIn("Bug report: version 0.2.0 (rules 43), level 4", rd(self.page("explodorama", "index.html"))) + + def test_record_json_appears_only_after_approve_in_reviewed_mode(self): + self.ingest(self.eml()) + self.assertFalse(os.path.exists(self.page("explodorama"))) + self.cli("approve", "explodorama", "1") + with open(self.page("explodorama", "1", "record.json")) as f: + self.assertEqual(json.load(f), self.fixture()) + self.cli("hide", "explodorama", "1") + self.assertFalse(os.path.exists(self.page("explodorama", "1", "record.json"))) + + def test_record_is_published_at_once_in_public_mode(self): + self.repo("pub", "public", "public") + self.ingest(mail(repo="pub", body="It broke.\nRun record: " + record() + "\n")) + self.assertTrue(os.path.isfile(self.page("pub", "1", "record.json"))) + + def test_bad_records_are_refused_but_the_issue_is_kept(self): + too_big = record(note="x" * (issues.RECORD_MAX + 10)) + cases = { + "over 64 KB": too_big, + "invalid JSON": '{"seed":1,"simVersion":1,"actions":[}', + "bool seed": record(seed=True), + "float seed": record(seed=1.5), + "missing actions": '{"seed":1,"simVersion":1}', + "actions not a list": record(actions={}), + "missing simVersion": '{"seed":1,"actions":[]}', + "NaN": '{"seed":1,"simVersion":1,"actions":NaN}', + "deep nesting": '{"seed":1,"simVersion":1,"actions":' + "[" * 30000 + "]" * 30000 + "}", + } + for label, line in cases.items(): + rec, why = issues.find_record("Run record: " + line + "\n") + self.assertIsNone(rec, label) + self.assertTrue(why, label) + self.assertEqual(issues.check_record("[1]")[0], None) + self.repo("bad", "public", "public") + self.ingest(mail(repo="bad", body="Hello.\nRun record: " + record(seed=True) + "\n")) + self.assertEqual(issues.issue_numbers("bad"), [1]) + self.assertFalse(os.path.exists(os.path.join(self.state, "repos", "bad", "1", "record.json"))) + self.assertFalse(issues.load_issue("bad", 1)["record"]) + self.assertIn("integer seed", self.rejected()) + + def test_quoted_printable_raw_fallback(self): + raw = b"Content-Transfer-Encoding: quoted-printable\n\nRun record: " + record().replace(":", "=3A").encode() + b"\n" + rec, why = issues.find_record("nothing useful here", raw) + self.assertEqual(rec["seed"], 7) + self.assertEqual(issues.find_record("nothing useful here", b"plain mail")[0], None) + + def test_only_the_first_record_is_kept(self): + self.repo("pub", "public", "public") + self.ingest(mail(repo="pub", mid="<r1@test>", body="First.\nRun record: " + record(seed=1) + "\n"), + mail(repo="pub", mid="<r2@test>", body="Again.\nRun record: " + record(seed=2) + "\n", + extra_headers=("In-Reply-To: <r1@test>",))) + self.assertEqual(self.stored_record("pub")["seed"], 1) + self.assertIn("second run record ignored", self.rejected()) + + def test_record_from_an_unauthenticated_reply_is_ignored(self): + self.repo("pub", "public", "public") + self.ingest(mail(repo="pub", mid="<r1@test>"), + mail(repo="pub", mid="<r2@test>", auth=(), body="Hi.\nRun record: " + record() + "\n", + extra_headers=("In-Reply-To: <r1@test>",))) + self.assertFalse(issues.load_issue("pub", 1)["record"]) + + +class Review(Base): + """Fixes from the review: per-message approval, text and record privacy, setup checks.""" + + def test_nonfinite_numbers_are_refused(self): + for line in ('{"seed":1,"simVersion":1,"actions":[1e400]}', '{"seed":1,"simVersion":1,"actions":[-1e999]}', + '{"seed":1,"simVersion":1,"actions":[Infinity]}'): + self.assertIsNone(issues.find_record("Run record: " + line + "\n")[0], line) + rec, _ = issues.check_record('{"seed":1,"simVersion":1,"actions":[1.5e10]}') + self.assertEqual(json.dumps(rec, allow_nan=False), '{"simVersion": 1, "seed": 1, "actions": [15000000000.0]}') + + def test_later_reply_after_approve_is_not_public_in_reviewed_mode(self): + self.repo("rev", "public", "reviewed") + self.ingest(mail(repo="rev", mid="<a@test>")) + self.cli("approve", "rev", "1") + self.ingest(mail(repo="rev", mid="<b@test>", body="Sneaky follow-up.\n", extra_headers=("In-Reply-To: <a@test>",))) + self.assertNotIn("Sneaky", rd(self.page("rev", "1", "index.html"))) + self.assertIn("It crashed.", rd(self.page("rev", "1", "index.html"))) + self.cli("approve", "rev", "1") + self.assertIn("Sneaky", rd(self.page("rev", "1", "index.html"))) + + def test_public_mode_shows_later_authenticated_reply_but_never_an_unauthenticated_one(self): + self.ingest(mail(mid="<a@test>")) + self.cli("approve", "demo", "1") + self.ingest(mail(mid="<b@test>", body="Authentic follow-up.\n", extra_headers=("In-Reply-To: <a@test>",)), + mail(mid="<c@test>", body="Forged follow-up.\n", auth=(), extra_headers=("In-Reply-To: <a@test>",))) + text = rd(self.page("demo", "1", "index.html")) + self.assertIn("Authentic follow-up.", text) + self.assertNotIn("Forged", text) + + def test_hide_clears_message_approvals(self): + self.repo("rev", "public", "reviewed") + self.ingest(mail(repo="rev")) + self.cli("approve", "rev", "1") + self.cli("hide", "rev", "1") + self.assertFalse(issues.load_issue("rev", 1)["messages"][0]["approved"]) + + def test_sender_limit_applies_in_reviewed_mode(self): + self.repo("rev", "public", "reviewed") + self.ingest(*[mail(repo="rev", subject=f"Bug {i}") for i in range(issues.SENDER_LIMIT + 2)]) + self.assertEqual(len(issues.issue_numbers("rev")), issues.SENDER_LIMIT) + + def test_flowed_text_is_rejoined(self): + text = "Sorry, this was reported by boss@exa \nmple.net and it hurts. \nNew paragraph.\n" + out = issues.report_text(text, flowed=True) + self.assertIn("[address hidden]", out) + self.assertNotIn("exa", out) + self.assertIn("New paragraph.", out) + self.assertNotIn("boss@exa", issues.report_text(text, flowed=False).replace("[address hidden]", "")) + + def test_device_continuation_line_is_dropped(self): + ua = "Device: Android 378x694; Mozilla/5.0 (Android 17; Mobile; rv:159.0) Gecko/" + out = issues.report_text(f"Hello.\n{ua}\n159.0 Firefox/159.0\nSeed 12, level 4\nBye.\n") + self.assertNotIn("Firefox", out) + self.assertNotIn("Android", out) + self.assertIn("Seed 12, level 4", out) + out = issues.report_text("Device: Pixel\n Firefox/1.0\nafter\n") + self.assertEqual(out, "after") + + def test_quoted_replies_are_cut_at_the_attribution(self): + for cut in ("On Tue, 3 Nov 2026 at 10:00, Jane Roe <jane@example.net> wrote:", + "On Tue, 3 Nov 2026 at 10:00, Jane Roe\nwrote:", + "-----Original Message-----", + "From: Jane Roe\nSent: Tuesday\nTo: Someone"): + out = issues.report_text(f"My answer.\n\n{cut}\nJane Roe says secrets.\n") + self.assertEqual(out, "My answer.", cut) + + def test_fullwidth_at_sign_is_scrubbed(self): + out = issues.report_text("write to boss\uff20example.net now") + self.assertNotIn("boss", out) + self.assertIn("[address hidden]", out) + self.ingest(mail(subject="Mail me\uff1a boss\uff20example.net", body="boss\uff20example.net\n")) + for p in (self.page("demo", "index.html"), self.page("demo", "1", "index.html")): + self.assertNotIn("boss", rd(p)) + + def test_record_keeps_only_the_allowlist_and_refuses_addresses(self): + rec, _ = issues.check_record(record(deviceId="abc", notes="x", startLevel=2, startCharge=None)) + self.assertEqual(sorted(rec), sorted(["simVersion", "seed", "actions", "startLevel", "startCharge", "options"])) + for bad in (record(options={"name": "me@example.net"}), record(actions=[{"id": "say", "text": "a\uff20b.org"}]), + record(options={"x@example.org": 1})): + self.assertIsNone(issues.check_record(bad)[0], bad) + with open(os.path.join(FIXTURES, "q101-record.json")) as f: + real = json.load(f) + self.assertEqual(issues.check_record(json.dumps(real))[0], real) # nothing of the real record is lost + self.ingest(mail(body="Hi.\nRun record: " + record(secretKey="zzz") + "\n")) + self.assertNotIn("secretKey", rd(os.path.join(self.state, "repos", "demo", "1", "record.json"))) + self.assertNotIn("secretKey", rd(self.page("demo", "1", "record.json"))) + + def imap_config(self, path, **over): + cfg = {"host": "imap.example.org", "user": "u", "password": "pw", "folder": "INBOX", **over} + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w") as f: + json.dump(cfg, f) + os.chmod(path, 0o600) + + def test_same_mailbox_as_comments_is_refused(self): + theirs = os.path.join(self.tmp, "comments", "imap.json") + with mock.patch.dict(os.environ, {"FW_COMMENTS_IMAP": theirs}): + self.imap_config(theirs) + self.imap_config(issues.imap_path()) + with self.assertRaisesRegex(issues.Error, "own account or its own"): + issues.cmd_ingest() + self.assertEqual(self.cli("imap-check")[0], 1) + self.imap_config(issues.imap_path(), folder="INBOX.bugs") + issues.check_mailbox_separate() # a different folder is fine + self.imap_config(issues.imap_path(), user="bugs") + issues.check_mailbox_separate() + + def test_check_n_refuses_non_ascii_digits(self): + for bad in ("\u0661", "\uff11", "0", "-1", "1 ", ""): + with self.assertRaises(issues.Error, msg=bad): + issues.check_n(bad) + self.assertEqual(issues.check_n("12"), 12) + self.ingest(mail()) + self.assertEqual(self.cli("show", "demo", "\u0661")[0], 1) + + def test_replay_template_rules(self): + for ok in ("replay.html?report={record}", "r/p.html?x=%20&report={record}"): + self.assertTrue(issues.replay_ok(ok), ok) + for bad in ("a%2e%2e/x.html?r={record}", "a%00.html?r={record}", "a b.html?r={record}", "a\tb.html?r={record}", + "a\nb.html?r={record}", "a\x7f.html?r={record}", "x.html", "/x?r={record}", "javascript:1?{record}"): + self.assertFalse(issues.replay_ok(bad), bad) + + def test_same_message_id_is_stored_once(self): + self.ingest(mail(mid="<same@test>", body="One.\n"), mail(mid="<same@test>", body="Two, different bytes.\n")) + self.assertEqual(issues.issue_numbers("demo"), [1]) + self.assertEqual(len(issues.load_issue("demo", 1)["messages"]), 1) + + def test_a_crash_while_saving_creates_no_duplicate_and_loses_no_report(self): + with mock.patch.object(issues, "save_issue", side_effect=OSError("disk full")): + with self.assertRaises(OSError): + self.ingest(mail(mid="<crash@test>")) + self.assertEqual(self.count("done"), 0) # the mail is still in the inbox + self.assertIn("keys.json", os.listdir(self.state)) # remembered before the issue was saved + self.ingest() # next run: stored once + self.ingest() + self.assertEqual(len(issues.issue_numbers("demo")), 1) + self.assertEqual(len(issues.load_issue("demo", 1)["messages"]), 1) + + +class Review2(Base): + def test_reply_record_needs_approval_in_reviewed_mode(self): + self.repo("rv", "public", "reviewed") + self.ingest(mail(repo="rv", body="First, no record.\n", mid="<a@x>")) + self.cli("approve", "rv", "1") + self.ingest(mail(repo="rv", body="Reply.\nRun record: " + record(options={"note": "unmoderated"}) + "\n", + mid="<b@x>", extra_headers=("In-Reply-To: <a@x>",))) + self.assertTrue(issues.load_issue("rv", 1)["record"]) + self.assertFalse(os.path.exists(self.page("rv", "1", "record.json"))) + self.assertNotIn("Watch what happened", rd(self.page("rv", "1", "index.html"))) + self.cli("approve", "rv", "1") + self.assertTrue(os.path.isfile(self.page("rv", "1", "record.json"))) + + def test_authenticated_reply_record_is_live_in_public_mode(self): + self.ingest(mail(body="No record.\n", mid="<a@x>"), + mail(body="Reply.\nRun record: " + record() + "\n", mid="<b@x>", extra_headers=("In-Reply-To: <a@x>",))) + self.assertTrue(os.path.isfile(self.page("demo", "1", "record.json"))) + + ATTRIBUTIONS = ( + "Am Mo., 3. Nov. 2026 um 10:00 Uhr schrieb Pavel Safronov <p@x.net>:", + "Le lun. 3 nov. 2026 \u00e0 10:00, Pavel Safronov <p@x.net> a \u00e9crit :", + "Op ma 3 nov 2026 om 10:00 schreef Pavel Safronov <p@x.net>:", + "Pavel Safronov <p@x.net> wrote:", + "El lun, 3 nov 2026 a las 10:00, Pavel Safronov escribi\u00f3:", + ) + + def test_attributions_in_any_language_cut_the_text(self): + for cut in self.ATTRIBUTIONS: + out = issues.report_text(f"My answer.\n\n{cut}\n> Pavel Safronov wrote this\nPavel Safronov again\n") + self.assertEqual(out, "My answer.", cut) + out = issues.report_text("My answer.\n________________________________\nFrom: Pavel\nSent: Monday\nTo: me\n\nPavel text") + self.assertEqual(out, "My answer.") + self.assertIn("Steps to reproduce:", issues.report_text("Steps to reproduce:\n1. tap\n")) # a plain colon line stays + + def test_device_block(self): + d65 = "Device: " + "x" * 57 + out = issues.report_text(f"{d65}\nAppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148\nSeed 5, level 2\n") + self.assertEqual(out, "Seed 5, level 2") + d83 = "Device: Android 378x694; Mozilla/5.0 (Android 17; Mobile; rv:159.0) Gecko/159.0 Fir" + self.assertEqual(len(d83), 83) + out = issues.report_text(f"{d83}\n\nThe game froze when I tapped twice on level 3.\n") + self.assertEqual(out, "The game froze when I tapped twice on level 3.") + out = issues.report_text(f"{d83}\nThe game froze when I tapped twice on level 3.\n") # prose right after: kept + self.assertEqual(out, "The game froze when I tapped twice on level 3.") + out = issues.report_text("Device: A\n Firefox 1\n more\nVersion 1.0\n") + self.assertEqual(out, "Version 1.0") + + def test_unflow_keeps_pending_text_before_a_quote(self): + out = issues.report_text("Thanks for the fix \nit works now \n> quoted\n", flowed=True) + self.assertEqual(out, "Thanks for the fix it works now") + + def test_unflow_signature_and_delsp_and_device(self): + out = issues.report_text("Hello there \n-- \nSignature name\n", flowed=True) + self.assertEqual(out, "Hello there") + self.assertEqual(issues.report_text("long wo \nrd end\n", flowed=True, delsp=True), "long word end") + self.assertEqual(issues.report_text("long wo \nrd end\n", flowed=True), "long wo rd end") + out = issues.report_text("Device: Pixel (Mobile; x) \n\nMy own paragraph.\n", flowed=True) + self.assertEqual(out, "My own paragraph.") + out = issues.report_text("Device: Pixel \nSeed 5, level 2\n", flowed=True) + self.assertEqual(out, "Seed 5, level 2") + + def test_flowed_mail_end_to_end_with_delsp(self): + raw = mail(body="Thanks for the fix \nit works now \n> quoted\n").replace( + b"Content-Type: text/plain; charset=utf-8", b"Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes") + self.ingest(raw) + self.assertIn("Thanks for the fixit works now", issues.load_issue("demo", 1)["messages"][0]["text"]) + + def test_crash_recovery_does_not_confuse_a_reused_slot(self): + real = issues.save_issue + calls = [] + + def boom(issue): + calls.append(1) + if len(calls) == 1: + raise OSError("disk full") + real(issue) + a, b = mail(body="Report A\n", mid="<A@x>"), mail(body="Report B\n", mid="<B@x>") + with mock.patch.object(issues, "save_issue", boom): + with self.assertRaises(OSError): + self.ingest(a) + box = mailbox.Maildir(self.maildir, create=False) + k = list(box.keys())[0] + a_raw = box.get_bytes(k) + box.remove(k) + self.ingest(b) # takes issue 1 while A's key still points at it + box.add(a_raw) + self.ingest() + texts = sorted(m["text"] for i in issues.load_issues("demo") for m in i["messages"]) + self.assertEqual(texts, ["Report A", "Report B"]) + + def test_no_numbered_folder_without_an_issue_after_a_crash(self): + real = issues.save_issue + calls = [] + + def boom(issue): + calls.append(1) + if len(calls) == 1: + raise OSError("x") + real(issue) + with mock.patch.object(issues, "save_issue", boom): + with self.assertRaises(OSError): + self.ingest(mail(body="A\nRun record: " + record() + "\n")) + self.assertEqual(issues.issue_numbers("demo"), []) + self.ingest() + self.assertEqual(issues.issue_numbers("demo"), [1]) + self.assertTrue(os.path.isfile(self.page("demo", "1", "record.json"))) + + def test_mailbox_comparison_ignores_case(self): + theirs = os.path.join(self.tmp, "comments", "imap.json") + with mock.patch.dict(os.environ, {"FW_COMMENTS_IMAP": theirs}): + Review.imap_config(self, theirs) + Review.imap_config(self, issues.imap_path(), host="IMAP.Example.ORG", user="U", folder="inbox") + with self.assertRaises(issues.Error): + issues.check_mailbox_separate() + + +class Review3(Base): + def test_real_text_with_colons_and_brackets_is_not_cut(self): + for text in ("I pressed <Space> and\nthen this happened:\nthe bomb vanished", + "The game crashed when I pressed\nthe <Enter> key:\nnothing else\n", + "What happened:\nI pressed <Enter> on the menu:\nnothing", + "He wrote notes and\nthen it broke:\nboom"): + self.assertEqual(issues.report_text(text), text.strip(), text) + + def test_attribution_right_after_a_text_line(self): + text = ("Still happens in 0.3.0\nOn Tue, Nov 3, 2026 at 7:51 PM Code <bugs-k7q4+demo@fuzzyworld.net> wrote:\n> x\n" + "Code again\n") + self.assertEqual(issues.report_text(text), "Still happens in 0.3.0") + + def test_weak_attribution_never_empties_the_text(self): + self.assertEqual(issues.report_text("Steps:\n> quoted by the app"), "Steps:") + self.assertEqual(issues.report_text("Hello\nSteps:\n> quoted"), "Hello") + self.assertEqual(issues.report_text("Pavel <p@x.net> wrote:\n> hi"), "") # a real attribution may leave nothing + + def test_wrapped_user_agent_starting_with_version_slash_is_dropped(self): + text = ("Bug here\nDevice: Mozilla/5.0 (iPhone; CPU iPhone OS 17_0) AppleWebKit/605.1.15\n" + "Version/17.0 Mobile/15E148 Safari/604.1\nSeed 1\n") + self.assertEqual(issues.report_text(text), "Bug here\nSeed 1") + self.assertIsNotNone(issues.KNOWN_LINE_RE.match("Version 0.2.0 (rules 43)")) + self.assertIsNone(issues.KNOWN_LINE_RE.match("Version/17.0 Mobile")) + + def test_flowed_device_line_is_not_joined_onto_the_previous_line(self): + text = "Bug \nDevice: Pixel (Mobile; x)\nSeed 1\n" + self.assertEqual(issues.report_text(text, flowed=True), "Bug \nSeed 1") + self.assertEqual(issues.report_text("Bug \nRun record: {}\nafter\n", flowed=True), "Bug \nafter") + + +class Review4(Base): + R = staticmethod(issues.report_text) + + def test_blank_line_ends_a_flowed_paragraph(self): + self.assertEqual(issues.unflow(["Para one. ", "", "Para two."]), ["Para one. ", "", "Para two."]) + self.assertEqual(self.R("Para one. \n\nPara two.", flowed=True), "Para one. \n\nPara two.") + for text in ("It broke. \n\nOn Mon, 3 Nov 2026, Bob Smith <bob@x.com> wrote:\n> old", + "It broke.\n\nOn Mon, 3 Nov 2026, Bob Smith <bob@x.com> wrote:\n> old"): + self.assertEqual(self.R(text, flowed=True), "It broke.") + + def test_wrapped_attributions_do_not_leak_names(self): + cases = { + "en_wrap_noaddr": "Bug text.\n\nOn Mon, 3 Nov 2026 at 10:00, Bob Smith\nwrote:\n> old", + "en_wrap_mid": "Bug text.\n\nOn Mon, 3 Nov 2026 at 10:00, Bob\nSmith <bob@x.com> wrote:\n> old", + "en_name_nl": "Bug text.\n\nOn Mon, 3 Nov 2026 at 10:00 Bob Smith <\nbob@x.com> wrote:\n> old", + "en_no_blank": "Bug text.\nOn Mon, 3 Nov 2026 at 10:00, Bob Smith <bob@x.com> wrote:\n> old", + "de_wrap": "Bug text.\n\nAm 03.11.2026 um 10:00 schrieb Hans M\u00fcller\n<hans@x.de>:\n> alt", + "de_wrap2": "Bug text.\n\nAm Mo., 3. Nov. 2026 um 10:00 Uhr schrieb Hans M\u00fcller <\nhans@x.de>:\n> alt", + "fr_wrap": "Bug text.\n\nLe lun. 3 nov. 2026 \u00e0 10:00, Jean Dupont <jean@x.fr> a\n\u00e9crit :\n> vieux", + "fr_nbsp": "Bug text.\n\nLe lun. 3 nov. 2026 \u00e0 10:00, Jean Dupont <jean@x.fr> a \u00e9crit\u00a0:\n> vieux", + "nl_wrap": "Bug text.\n\nOp ma 3 nov 2026 om 10:00 schreef Jan Jansen\n<jan@x.nl>:\n> oud", + "es_wrap": "Bug text.\n\nEl lun, 3 nov 2026 a las 10:00, Juan P\u00e9rez (<juan@x.es>)\nescribi\u00f3:\n> viejo", + "gmail_blank_before_quote": "Bug text.\n\nOn 3 Nov 2026, at 10:00, Bob Smith <bob@x.com> wrote:\n\n> old", + "schrieb_am": "Bug text.\n\nBob Smith <bob@x.com> schrieb am Mo., 3. Nov. 2026, 10:00:\n> alt", + "schrieb_am_wrapped": "Bug text.\n\nBob Smith <bob@x.com> schrieb am Mo., 3. Nov. 2026,\n10:00:\n> alt", + "wrote_plain_name": "Bug text.\n\nBob Smith wrote:\n> old", + "flowed": "Bug text. \n\nOn Mon, 3 Nov 2026 at 10:00, Bob Smith \n<bob@x.com> wrote:\n> old", + } + for label, text in cases.items(): + out = self.R(text, flowed=(label == "flowed")) + self.assertEqual(out.strip(), "Bug text.", label) + + def test_non_english_outlook_header_blocks_are_cut(self): + blocks = ( + "________________________________\nVon: Bob Smith <bob@x.com>\nGesendet: Montag, 3. November 2026 10:00\nAn: bugs\nBetreff: Re: x\n\nalt", + "De : Bob Smith <bob@x.com>\nEnvoy\u00e9 : lundi 3 novembre 2026 10:00\n\u00c0 : bugs\nObjet : Re: x\n\nvieux", + "Van: Bob Smith\nVerzonden: maandag 3 november 2026 10:00\nAan: bugs\nOnderwerp: Re: x\n\noud", + "From: Bob Smith\nSent: Monday, November 3, 2026 10:00\nTo: bugs\nSubject: Re: x\n\nold", + ) + for b in blocks: + self.assertEqual(self.R("Bug text.\n\n" + b), "Bug text.", b) + + def test_header_block_rule_leaves_the_games_own_template_and_plain_lists_alone(self): + with open(os.path.join(FIXTURES, "explodorama-report.eml"), "rb") as f: + raw = f.read() + msg = issues.message_from_bytes(raw, policy=issues.policy.default) + text = self.R(msg.get_body(("plain",)).get_content()) + self.assertIn("autopilot walked me into death", text) + self.assertIn("Seed 1464004408", text) + self.assertIn("Errors caught: none", text) + lst = "Expected: the bomb\nActual: nothing <Enter>\nSteps: tap\nNote: twice" + self.assertEqual(self.R(lst), lst) + + def test_bare_verb_in_prose_is_not_an_attribution(self): + for text in ("I pressed <Enter> and wrote:\nnothing happened", "He wrote:\nthe thing broke", + "On the menu screen I typed and wrote:\nnothing"): + self.assertEqual(self.R(text), text) + + def test_weak_match_needs_everything_after_it_to_be_quoted(self): + self.assertEqual(self.R("It crashed.\nError shown:\n> TypeError: x\nThat was it."), + "It crashed.\nError shown:\nThat was it.") + self.assertEqual(self.R("Error shown:\n> TypeError: x\nThat was it."), "Error shown:\nThat was it.") + self.assertEqual(self.R("It crashed.\nError shown:\n> TypeError: x\n\n> more"), "It crashed.") + + def test_nfkc_comes_before_the_device_filter(self): + self.assertEqual(self.R("\uff24evice: Pavel's iPhone\nbroke"), "broke") + self.assertEqual(self.R("\uff32un record: {}\nafter"), "after") + + +class FwScript(Base): + """server/fw publish_one and `fw issues <repo> <mode>` against temp directories (flock is faked: macOS has none).""" + + def setUp(self): + super().setUp() + self.bin = os.path.join(self.tmp, "bin") + os.makedirs(self.bin) + shim = os.path.join(self.bin, "flock") + with open(shim, "w") as f: + f.write("#!/bin/sh\nexit 0\n") + os.chmod(shim, 0o755) + self.env = {**os.environ, "PATH": self.bin + os.pathsep + os.environ["PATH"], "HOME": self.tmp, "FW_GIT_ROOT": self.git, + "FW_CODE_SITE": self.site, "FW_STATE": self.fwstate, "FW_LOGS": os.path.join(self.tmp, "logs"), + "FW_APPS_SITE": self.apps, "FW_TOOLS": ROOT} + os.makedirs(self.site) + + def fw(self, *args): + return subprocess.run(["bash", os.path.join(ROOT, "server", "fw"), *args], env=self.env, capture_output=True, text=True) + + def config(self, repo="demo"): + return subprocess.run(["git", "-C", os.path.join(self.git, repo + ".git"), "config", "--get", "fuzzyworld.issues"], + capture_output=True, text=True).stdout.strip() + + def test_mode_setter_and_publish_one(self): + self.ingest(mail()) + r = self.fw("issues", "demo", "reviewed") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(self.config(), "reviewed") + self.assertFalse(os.path.exists(self.page())) + r = self.fw("issues", "demo", "public") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertTrue(os.path.isfile(self.page("demo", "1", "index.html"))) + shutil.rmtree(self.page()) + r = self.fw("publish", "demo") # publish_one runs `issues.py publish --locked` before gen.py repo + self.assertEqual(r.returncode, 0, r.stderr) + self.assertTrue(os.path.isfile(self.page("demo", "index.html"))) + self.assertIn('/demo/issues/', rd(os.path.join(self.site, "demo", "index.html"))) # the summary links to Issues + r = self.fw("issues", "demo", "private") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertFalse(os.path.exists(self.page())) + + def test_other_words_and_unknown_repos_are_refused(self): + for args in (("issues", "demo", "bogus"), ("issues", "demo", "Public"), ("issues", "nosuch", "public"), ("issues", "demo")): + r = self.fw(*args) + self.assertNotEqual(r.returncode, 0, args) + self.assertEqual(self.config(), "public") # unchanged + + +class Replay(Base): + def app(self, name, repo="demo", replay="replay.html?report={record}", index=True): + d = os.path.join(self.apps, name) + os.makedirs(d) + if index: + with open(os.path.join(d, "index.html"), "w") as f: + f.write("x") + with open(os.path.join(d, ".fuzzyworld.json"), "w") as f: + json.dump({"app": name, "repo": repo, "replay": replay}, f) + + def test_no_link_without_a_replay_key_or_app(self): + self.assertEqual(issues.replay_link("demo", 1), "") + self.app("demo", replay="") + self.assertEqual(issues.replay_link("demo", 1), "") + + def test_link_is_encoded_and_prefers_the_app_named_like_the_repo(self): + self.app("demo-next", replay="next.html?r={record}") + self.app("demo") + link = issues.replay_link("demo", 3) + self.assertIn('href="https://apps.fuzzyworld.net/demo/replay.html?report=' + 'https%3A%2F%2Fcode.fuzzyworld.net%2Fdemo%2Fissues%2F3%2Frecord.json"', link) + self.assertIn('rel="noopener"', link) + self.assertNotIn("next.html", link) + self.assertNotIn("<iframe", link) + + def test_link_needs_an_index_page_and_a_safe_template(self): + self.app("demo", index=False) + self.assertEqual(issues.replay_link("demo", 1), "") + self.app("other", repo="demo", replay="javascript:alert({record})") + self.assertEqual(issues.replay_link("demo", 1), "") + + def test_other_repos_apps_are_ignored(self): + self.app("game", repo="somebody-else") + self.assertEqual(issues.replay_link("demo", 1), "") + + def test_page_shows_the_link_only_with_a_published_record(self): + self.app("demo") + self.ingest(mail(mid="<a@test>"), mail(subject="With record", body="Boom.\nRun record: " + record() + "\n")) + self.assertNotIn("Watch what happened", rd(self.page("demo", "1", "index.html"))) + page = rd(self.page("demo", "2", "index.html")) + self.assertIn("Watch what happened", page) + self.assertIn("replay.html?report=", page) + + def test_summary_link_refresh_follows_the_index(self): + self.ingest(mail()) + self.cli("hide", "demo", "1") + self.assertEqual(self.summaries, ["demo", "demo"]) # appeared, then disappeared + + +class ImapParameters(unittest.TestCase): + def test_defaults_are_the_comments_values_and_issues_can_override(self): + c = issues.comments.IMAPSource() + c.cfg, c.prefix = {"done_folder": "", "bad_folder": ""}, "INBOX." + self.assertEqual(c._folder("bad"), "INBOX.comments-bad") + self.assertEqual(c.max_size, issues.comments.IMAP_MAX_SIZE) + i = issues.comments.IMAPSource(cfg_path="/x/imap.json", stem="issues", max_size=5, log=print) + i.cfg, i.prefix = {"done_folder": "", "bad_folder": ""}, "INBOX." + self.assertEqual(i._folder("done"), "INBOX.issues-done") + self.assertFalse(i.available()) + + +if __name__ == "__main__": + unittest.main()