fuzzyworld-tools / commit
fw admin: a local web panel for bug reports and blog comments; Issues repo tab; --json commands
.gitignore | 1 + PLAN.md | 39 ++++ README.md | 20 +- bin/fw | 4 + lib/admin.py | 589 +++++++++++++++++++++++++++++++++++++++++++++++++ server/comments.py | 36 ++- server/fw | 4 +- server/gen.py | 2 + server/issues.py | 104 +++++++-- tests/test_admin.py | 490 ++++++++++++++++++++++++++++++++++++++++ tests/test_comments.py | 39 ++++ tests/test_issues.py | 105 +++++++++ 12 files changed, 1413 insertions(+), 20 deletions(-) diff --git a/.gitignore b/.gitignore index 50b24da..f7a9651 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ imap.json *-imap.json .env .env.* +.vscode/ diff --git a/PLAN.md b/PLAN.md index a08602e..58cd48d 100644 --- a/PLAN.md +++ b/PLAN.md @@ -199,6 +199,17 @@ he wants; see the tiers in the README, form = T2, email = T1). The form stays in ## Backlog +- **URGENT (Pavel, 2026-11-04): AI usage analysis per repo, on the CI/usage pages.** There is a lot of public chatter + about Claude's usage limits, so each repo should show what its AI use looks like over time. It builds on the + existing per-commit "Usage" tab (`repo_usage` / `usage_costs` in `server/gen.py`, from the `FW-Tokens-*` commit + trailers) and on the planned "Stats and heatmap" step (Next 3). Candidate views, as static SVG charts (no JS): + AI usage across time (output tokens, cached reads, cost per week or per session); AI share of commits and of pull + requests over time (`Co-Authored-By`, with the Provenance caveats); lines of code added and removed over time; + **usage per line of code** (output tokens, and estimated cost, per added line, per week and per commit, with the + median and the hot commits already marked on the Usage tab); and a site-wide roll-up across repos. Open points: + how to price tokens (a per-model price table kept in the repo, stated as an estimate); what a "line" is (added + lines in non-generated, non-vendored files); sessions that span several repos; commits without trailers (the + ChatGPT era) shown as unknown, never as zero. Data stays derived from git; no transcripts are published. - File finder and code search (one shipped script plus a prebuilt index per repo) - code-server on TrueNAS, for editing in the browser (private, over Tailscale) - Provenance stats and transcript publishing (see below) @@ -393,6 +404,34 @@ report's record is published at the same moment as its text; old `sim/` files ar in the game's `mailto:` link (game PLAN §20.2 step 4) come after the cron (step 2); the published page defaults to the plain recorded run, with comparison as an option (game PLAN §20.2 step 5). Nothing open here. +## Admin panel (`fw admin`, Q102; designed 2026-11-04, decisions by Pavel) + +**Version 1 built 2026-11-04** (`lib/admin.py`, `tests/test_admin.py`; server side: `issues list|show --json`, `issues record`, `comments list --json`). Not yet tried against the real server (tested only with a fake runner). Decisions made while building: +- **Login:** a one-time login URL (random key) that sets a session cookie; everything else needs the cookie, plus the per-launch POST token and Host / Origin / Sec-Fetch-Site checks. +- **Watch = dev replay:** a link to the deployed viewer when the record is public, "Download record" always, and "Prepare dev replay" (runs the app's `report-replay` npm script with the record and `--rules <commit>`; needs `npm run dev` running in the app). +- **Recently approved comments** (the 20 newest) with a Remove button are in v1, not later. +- **`FW_CODE_DIR`** (also `--code-dir`, default `~/code`) is configurable: where the apps' working copies live. +- Still not in v1: "make this a regression fixture", pull requests, CI, blog state. + +A **local-only** web page over `fw` commands (see "Report replays": never hosted on DreamHost). `fw admin` starts a +small Python (stdlib) server on `127.0.0.1` and opens the browser; every button runs the same `fw ...` command over +ssh that Pavel would type, so the panel holds no state and has no authority beyond his ssh key. + +- **Decided:** version 1 is **reports and blog comments** together (both are "read it, then approve or reject"); + **no JavaScript at first** (server-rendered pages and `POST` forms, a full reload after each action); the command + is `fw admin`. +- **Safety:** bind to 127.0.0.1 only; check the `Host` header (DNS rebinding); every action is a `POST` carrying a + random token made at launch; a confirm step for destructive actions; never show passwords. +- **Version 1:** (1) one list of reports across repos (repo, number, title, state, visibility, record, age; + filters "waiting for approval" and "has a replay"); (2) a report page with the full stored text, **including + unpublished reports**, the details table and thread, and Approve / Hide / Close / Reopen and the repo's mode + switch; (3) blog comments: pending ones with approve, reject, trust sender; (4) a Watch link per report with a + record (the dev replay page until the game's production viewer exists). +- **Needs from the existing code:** `fw issues list --json`; `fw issues record <repo> <n>` (read an unpublished + record); a machine-readable `fw comments list`. +- **Later, each useful alone:** pull requests (after contrib intake), CI and runners, blog build/deploy and watcher + state (Q100), "make this report a regression fixture" (writes `test/fixtures/` in the app's repo). + ## Pull requests - Contributions arrive at `contrib-68uq+<repo>@fuzzyworld.net` as a branch URL or patches (already live). diff --git a/README.md b/README.md index 2f4c23e..0acf4d0 100644 --- a/README.md +++ b/README.md @@ -91,6 +91,7 @@ On the laptop, `bin/fw` runs `deploy` locally and forwards repo commands to Drea | `fw deploy [--ref B] [--as NAME] [--here] [--dry-run]` | build and publish this repo's app (above). `--as NAME` publishes it at `apps.fuzzyworld.net/NAME/` instead, a second copy of the same repo (e.g. `explodorama-next` built from `main` while `/explodorama/` is deployed from a tag with `--ref`); the build gets `FW_APP=NAME`, which the vite plugin uses for `base`, and the index title gets the suffix | | `fw repoint [DIR] [--https] [--dry-run]` | for each repo in DIR whose `origin` is on gitlab.com: rename it `gitlab`, add `origin` → `dh:git/<repo>.git` (or read-only `https://code.fuzzyworld.net/<repo>.git` with `--https`, public repos only), fetch, and retarget branch tracking. Changes only remotes; checks that uncommitted work, HEAD and stashes are unchanged | | `fw usage [--all\|--now]` | Claude Code token use per session from `~/.claude/projects` (runs locally): messages, current context size, cache-read, output (cache-read and output include subagents). `--now` prints one line for the newest session, to decide when to put it down | +| `fw admin [--port N] [--no-open] [--code-dir DIR]` | a local web page for reports and blog comments, over the commands below (runs locally; see [Admin panel](#admin-panel)) | | `fw comments list\|show\|approve\|reject\|remove\|untrust\|publish\|export\|ingest\|imap-check` | moderate blog comments received by email (see [Blog](#blog)); runs on the server | | `fw issues <repo> public\|reviewed\|private` | set who sees the repo's bug-report issue pages (`git config fuzzyworld.issues`; default private) and republish them | | `fw issues list\|show\|approve\|hide\|close\|reopen\|publish\|publish-all\|ingest\|imap-check` | moderate bug-report issues received by email (see [Issues](#issues)); runs on the server | @@ -321,8 +322,24 @@ In `public` and `reviewed` mode each sender may send 5 messages per 24 hours (th **Replay link.** If a deployed app of the repo (`apps.fuzzyworld.net/<app>/.fuzzyworld.json` with `repo` equal to the repo and an `index.html`) has a `replay` template (the `"fuzzyworld"` block of its `package.json`, e.g. `"replay": "replay.html?report={record}"`), a visible issue with a record shows "Watch what happened": a plain link to `https://apps.fuzzyworld.net/<app>/<replay>` with `{record}` replaced by the URL-encoded address of `record.json`. The app named like the repo is preferred, so a `-next` copy is not chosen. It is a link, not an iframe (the site's CSP blocks framing). No `replay` key, no link. +**Machine-readable commands** (used by [`fw admin`](#admin-panel); the text of unpublished reports IS included, it is a private tool; sender hashes, `mkey` and `mid` never are): `fw issues list [repo] --json` prints every stored issue (`repo`, `n`, `title`, `created`, `updated`, `state`, `approved`, `hidden`, `public`, `record`, `messages`, `shown`, `waiting`) and each repo's `mode` and `public_repo`; `waiting` means a public or reviewed repo's issue that is not hidden and has messages not yet shown. `fw issues show <repo> <n> --json` prints one issue with the full text of every message (label, date, authenticated, approved, shown), the details table rows, prev/next numbers, the public page URL (only when public) and the replay URL (only when the record is shown). `fw issues record <repo> <n>` prints `record.json` as stored, even for an unpublished issue (error `no record for repo#n` when there is none). Plain `list` and `show` are unchanged. + **State** is in `~/fw/issues/` (override with `FW_ISSUES_DIR`), outside every web root: `repos/<repo>/<n>/issue.json` and `record.json`, `secret` (salt for sender hashes; mode 600), `msgids.json` (threading), `keys.json` (a mail is never stored twice), `rejected.log`, the locks. Mail source: `~/fw/issues/imap.json` (mode 600, the same format as the comments one; finished mail goes to `issues-done` / `issues-bad`) or, without it, the Maildir `~/Maildir/.bugs` (`.bugs.done`, `.bugs.bad`). **The bugs mailbox needs its own account or its own `folder`** in `imap.json`: ingest and `imap-check` refuse to run when it has the same host, user and folder as the comments `imap.json` (the two jobs would move each other's mail to their bad folders). Ingest also refuses to run until `authserv_id` is set in `server/comments/sites.json`. The hourly job is `issues-ingest.sh`. +## Admin panel + +`fw admin [--port N] [--no-open] [--code-dir DIR]` (`lib/admin.py`, Python standard library only, runs on the laptop) starts a small web page on `127.0.0.1` for reports (bug-report issues) and blog comments, prints a one-time login URL and opens it in the browser. The pages are server-rendered HTML with plain `POST` forms and **no JavaScript**. Every button runs the matching `~/bin/fw ...` command on the server over ssh, exactly what you would type (`FW_HOST`, default `dh`); the panel keeps no state and never shows a password. It is local-only on purpose: it is never hosted on DreamHost. + +**Security model.** The server binds to `127.0.0.1` only. The login URL carries a random key that works once; it sets a session cookie named `fwadmin-<port>` (`HttpOnly`, `SameSite=Strict`), and every other request without that cookie gets 403. The `Host` header must be exactly `127.0.0.1:<port>` (DNS rebinding), else 421. Every `POST` also needs a random per-launch token in its form (a separate secret from the cookie), an `Origin` (if sent) equal to the page's own, and `Sec-Fetch-Site` (if sent) `same-origin` or `none`; otherwise 403 and no command runs. Responses carry a strict `Content-Security-Policy` (no scripts, forms only to itself, no framing), `nosniff`, no referrer and no caching. Mail text never reaches a command line: only a validated repo name, issue number, comment id and a fixed set of action words, each checked again before ssh is run. Hide, reject, untrust, remove and every mode change ask for confirmation first. Query strings are never logged. + +**Pages.** +- **Reports** (`/reports`): one table across repos (repo, number, title, state, public, mode, record, messages shown/total, age) with the filters Waiting (default when anything waits), Has replay and All. +- **A report** (`/reports/<repo>/<n>`): the details table and every message in full, **including reports that are not published**, with who sent it (Reporter A, B, ...), whether it passed authentication and whether it is shown; buttons Approve, Hide, Close / Reopen; radio buttons for the repo's mode (public, reviewed, private) with Set; previous / next links; a link to the public page when there is one. +- **Watch** (on a report with a run record): a link to the deployed replay viewer when the record is public; always "Download record" (`issues record`); and, when `<code dir>/<repo>/package.json` has a `report-replay` script, "Prepare dev replay": it saves the record to a fresh temporary folder and runs `npm run --silent report-replay -- <record.json>` in `<code dir>/<repo>` (with `--rules <commit>` when the report's details have a commit), 60 s at most. The output is shown, with the `http://localhost:5173/?replay=...` address as a link; it works only while `npm run dev` runs in that repo. The record folder is kept (the dev server needs it): one `fw-admin-<repo>-<n>-...` folder per replay under the system temp dir, mode 700; delete them when you like. +- **Comments** (`/comments`): pending comments in full with the post link, author, date and source, and Approve, Approve + trust (email comments only), Reject and Untrust (trusted senders only); below, "Recently approved" (the 20 newest) with a Remove button each. + +The server handles one request at a time (an idle connection cannot stall it). After each action the page reloads with the command's output (or its error) at the top (these messages are server-wide, not per browser tab). `FW_CODE_DIR` (or `--code-dir`, default `~/code`) is the folder holding the apps' working copies. Needs the server-side `fw` to know `issues list --json`, `issues show --json`, `issues record` and `comments list --json` (deploy the server scripts first). + ## Blog A generic static-blog generator: a folder of Markdown files and one JSON config become a site (`server/blog.py`). @@ -369,7 +386,8 @@ fw blog watch install|uninstall ~/Blog/fw-blog.json [--print] # the launchd ag - **Comments by email:** set `"comments_site": "writing"` (or `"til"`, a key of `server/comments/sites.json`) and `"comments_mailbox": "<address>"` in the config: each post then gets a "Comment by email" link (subject `Comment [writing:/2024/05/01/slug/]`) and a container that `blog.js` fills from `/_comments/<post path>.json`. - Comments are moderated on the server with `fw comments list|show|approve [--trust]|reject|remove|untrust|publish|export|ingest|imap-check`; + Comments are moderated on the server with `fw comments list|show|approve [--trust]|reject|remove|untrust|publish|export|ingest|imap-check` + (`list --json` prints the pending comments in full plus the 20 newest approved ones, with each post's URL and whether the sender is trusted, never a sender hash; `fw admin` uses it); a cron job runs `ingest` hourly. `ingest` reads the comments mailbox: a hosted mailbox over IMAP when `~/fw/comments/imap.json` exists (`{"host", "user", "password", "folder"}`, mode 600, created by hand on the server; TLS with certificate check; processed mail moves to the mailbox folders `comments-done` and `comments-bad`, with the server's folder prefix such as `INBOX.` added when it needs one; `done_folder` and `bad_folder` in the file override the names), otherwise a Maildir folder diff --git a/bin/fw b/bin/fw index 2b64227..83c703a 100755 --- a/bin/fw +++ b/bin/fw @@ -13,6 +13,8 @@ # fw blog watch <config> [--once] [--dry-run] mirror the source into git and auto-deploy edits (see server/blog_watch.py); # fw blog watch install|uninstall <config> [--print] the launchd agent that runs it every 60 s (ONE machine only) # fw describe <name> TEXT | publish <name> | publish-all | ci <name> [branch] | ci-list | ci-watch +# fw admin [--port N] [--no-open] [--code-dir DIR] local web page for reports and blog comments (see lib/admin.py); +# runs the server-side fw commands over ssh (FW_HOST); FW_CODE_DIR = app folders # fw comments list|show|approve|reject|remove|untrust|publish|export|ingest|imap-check (blog comments) # fw issues <repo> public|reviewed|private | list|show|approve|hide|close|reopen|publish|publish-all|ingest|imap-check (bug-report issues) # (run on DreamHost over ssh) @@ -85,6 +87,8 @@ case "${1:-help}" in shift; exec node "$HERE/lib/deploy.mjs" "$@" ;; usage) shift; exec python3 "$HERE/lib/usage.py" "$@" ;; + admin) + shift; exec python3 "$HERE/lib/admin.py" "$@" ;; repoint) shift; exec bash "$HERE/lib/repoint.sh" "$@" ;; list|new|public|private|describe|publish|publish-all|ci|ci-list|ci-watch|comments|issues|help|-h|--help) diff --git a/lib/admin.py b/lib/admin.py new file mode 100644 index 0000000..2c07008 --- /dev/null +++ b/lib/admin.py @@ -0,0 +1,589 @@ +#!/usr/bin/env python3 +"""fw admin: a LOCAL web page for the server's reports (bug-report issues) and blog comments. Python standard library only. + + fw admin [--port N] [--no-open] [--code-dir DIR] + +Starts a small server on 127.0.0.1 (never another address), prints a one-time login URL and opens it. Every page is +server-rendered HTML; every button is a POST form; there is NO JavaScript. Each action runs the matching `~/bin/fw ...` +command on the server over ssh (host from FW_HOST, default `dh`), exactly what you would type. The panel holds no state +of its own and never shows a password. Mail text never goes into a command line: only a validated repo name, issue +number, comment id and a fixed set of action words. + +Access. The login URL carries a random key that works once and then sets a session cookie (HttpOnly, SameSite=Strict). +Every request needs that cookie and Host 127.0.0.1:<port> (DNS rebinding); every POST also needs the random token in its +form, and an Origin / Sec-Fetch-Site that says "same site". Destructive actions (hide, reject, untrust, remove, a mode +change) ask first. + +"Prepare dev replay" needs the app's working copy: <code dir>/<repo>/package.json with a "report-replay" script +(FW_CODE_DIR or --code-dir, default ~/code). It saves the report's run record to a temporary folder and runs +`npm run --silent report-replay -- <record.json> [--rules <commit>]` there. +""" + +import argparse +import hmac +import html +import http.cookies +import json +import os +import re +import secrets +import shlex +import subprocess +import sys +import tempfile +import webbrowser +from datetime import datetime, timezone +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from urllib.parse import parse_qs, quote, urlsplit + +HOST_RE = re.compile(r"^[A-Za-z0-9._-]+\Z") # \Z, not $: "$" also matches before a trailing newline +SAFE_ARG = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*\Z") +FLAG_ARG = re.compile(r"^--(json|trust)\Z") +REPO_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*\Z") # as server/issues.py check_repo +ID_RE = re.compile(r"^\d{8}T\d{6}Z-[0-9a-f]{6}\Z") # as server/comments.py +COMMIT_RE = re.compile(r"^[0-9a-f]{7,40}\Z") +REPLAY_URL_RE = re.compile(r"http://localhost:5173/\?replay=[A-Za-z0-9._~%&=/+:,-]+") +RESERVED_REPOS = {"list", "show", "record", "approve", "hide", "close", "reopen", "publish", "publish-all", "ingest", + "imap-check", "help"} # `fw issues <word>` would run that command instead of setting a mode +MODES = ("public", "reviewed", "private") +REPORT_ACTIONS = ("approve", "hide", "close", "reopen", "replay") +COMMENT_ACTIONS = ("approve", "trust", "reject", "untrust", "remove") +CONFIRM = {"hide", "reject", "untrust", "remove"} # these ask first; so does every mode change +MAX_BODY = 32 * 1024 +COOKIE = "fwadmin" # the cookie is named fwadmin-<port>, so two instances (or another local server) do not clash +HEADERS = { + "Content-Security-Policy": "default-src 'none'; style-src 'self'; form-action 'self'; frame-ancestors 'none'; base-uri 'none'", + "X-Content-Type-Options": "nosniff", + "Referrer-Policy": "no-referrer", + "Cache-Control": "no-store", +} + + +class FwError(Exception): + def __init__(self, message, code=None): + super().__init__(message) + self.code = code + + +# ---- running fw on the server ---- + +class SshRunner: + """Runs `~/bin/fw <args>` over ssh. The arguments are checked first: nothing odd is ever sent.""" + + def __init__(self, host, timeout=60): + if not (HOST_RE.match(host) and not host.startswith("-")): + raise FwError(f"invalid FW_HOST: {host!r}") + self.host, self.timeout = host, timeout + + def run(self, args): + for a in args: + if not (isinstance(a, str) and (SAFE_ARG.match(a) or FLAG_ARG.match(a))): + raise FwError(f"refused unsafe argument {a!r}") + remote = "~/bin/fw " + " ".join(shlex.quote(a) for a in args) + try: + p = subprocess.run(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", self.host, remote], + stdin=subprocess.DEVNULL, capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=self.timeout) + except subprocess.TimeoutExpired: + raise FwError(f"timed out after {self.timeout} s") + except OSError as e: + raise FwError(f"could not run ssh: {e}") + if p.returncode == 255: + raise FwError(f"could not reach {self.host} (try `ssh {self.host} true`)", 255) + if p.returncode != 0: + raise FwError(p.stderr.strip() or f"fw exited with code {p.returncode}", p.returncode) + return p.stdout + + def json(self, args): + out = self.run(args) + try: + data = json.loads(out) + except ValueError: + raise FwError("the server did not answer with JSON (is `fw` on the server up to date?)") + if not isinstance(data, dict): + raise FwError("the server answered with JSON of the wrong shape") + return data + + +# ---- checks on what the browser sends ---- + +def valid_repo(repo): + return bool(REPO_RE.match(repo)) and not repo.endswith(".git") + + +def valid_n(n): + return n.isascii() and n.isdigit() and 0 < int(n) < 10**9 + + +def esc(value): + return html.escape(str(value), quote=True) + + +def https(url): + """Links from the server are shown only when they are https://.""" + return isinstance(url, str) and url.startswith("https://") + + +def ext_link(url, text): + return f'<a href="{esc(url)}" rel="noreferrer noopener">{esc(text)}</a>' if https(url) else esc(text) + + +# ---- look: copied by hand from server/site/style.css (this file ships without server/) ---- + +CSS = """ +:root { color-scheme: light dark; --bg: #fff; --fg: #1f2328; --muted: #656d76; --line: #d0d7de; --soft: #f6f8fa; + --link: #0969da; --ok: #1a7f37; --ok-bg: #dafbe1; --bad: #cf222e; --bad-bg: #ffebe9; --warn: #9a6700; --warn-bg: #fff8c5; } +@media (prefers-color-scheme: dark) { + :root { --bg: #0d1117; --fg: #e6edf3; --muted: #8d96a0; --line: #30363d; --soft: #161b22; --link: #4493f8; + --ok: #3fb950; --ok-bg: #12261e; --bad: #f85149; --bad-bg: #2d1214; --warn: #d29922; --warn-bg: #272115; } +} +body { margin: 0; background: var(--bg); color: var(--fg); font: 16px/1.55 system-ui, sans-serif; } +main { max-width: 64rem; margin: 0 auto; padding: 0 1rem 3rem; } +a { color: var(--link); text-decoration: none; } +a:hover { text-decoration: underline; } +nav.top { background: var(--soft); border-bottom: 1px solid var(--line); padding: .6rem 1rem; } +nav.top a { margin-right: 1rem; color: var(--muted); } +nav.top a[aria-current] { color: var(--fg); font-weight: 600; } +h1 { font-size: 1.5rem; } +h2 { font-size: 1.15rem; margin-top: 2rem; } +table { border-collapse: collapse; width: 100%; } +th, td { text-align: left; padding: .35rem .6rem; border-bottom: 1px solid var(--line); vertical-align: top; } +th { color: var(--muted); font-weight: normal; font-size: .85rem; } +.badge { display: inline-block; padding: .1rem .5rem; background: var(--soft); color: var(--muted); border: 1px solid var(--line); + border-radius: 999px; font-size: .75rem; } +.badge-success { background: var(--ok-bg); color: var(--ok); border-color: var(--ok); } +.badge-error { background: var(--bad-bg); color: var(--bad); border-color: var(--bad); } +.badge-warning { background: var(--warn-bg); color: var(--warn); border-color: var(--warn); } +.meta { color: var(--muted); font-size: .9rem; } +pre.issue-text { white-space: pre-wrap; overflow-wrap: anywhere; max-width: 100%; box-sizing: border-box; overflow-x: hidden; + background: var(--soft); border: 1px solid var(--line); border-radius: 6px; padding: .6rem .8rem; font-size: .9rem; } +table.issue-details { width: auto; max-width: 100%; font-size: .8rem; margin: .5rem 0; } +table.issue-details th { padding-right: 1rem; } +.card { border: 1px solid var(--line); border-radius: 6px; padding: .2rem 1rem 1rem; margin: 1rem 0; } +.actions form, form.inline { display: inline-block; margin: 0 .4rem .4rem 0; } +button { font: inherit; padding: .25rem .8rem; border: 1px solid var(--line); border-radius: 6px; background: var(--soft); + color: var(--fg); cursor: pointer; } +button.danger { color: var(--bad); border-color: var(--bad); } +.flash { border: 1px solid var(--ok); background: var(--ok-bg); border-radius: 6px; padding: .4rem .8rem; margin: 1rem 0; + white-space: pre-wrap; overflow-wrap: anywhere; } +.flash.err { border-color: var(--bad); background: var(--bad-bg); } +.filters a { margin-right: 1rem; } +.filters a[aria-current] { color: var(--fg); font-weight: 600; } +.pager { display: flex; justify-content: space-between; margin-top: 1.5rem; } +label.mode { margin-right: 1rem; } +""" + + +def age(iso): + """"3 d" / "5 h" / "12 min" since an ISO date; "" when it cannot be read.""" + try: + then = datetime.strptime(iso[:19], "%Y-%m-%dT%H:%M:%S").replace(tzinfo=timezone.utc) + except (ValueError, TypeError): + return "" + s = max(0, int((datetime.now(timezone.utc) - then).total_seconds())) + return f"{s // 86400} d" if s >= 86400 else f"{s // 3600} h" if s >= 3600 else f"{s // 60} min" + + +def badge(state): + return f'<span class="badge{" badge-success" if state == "open" else ""}">{esc(state)}</span>' + + +def yes(flag): + return "yes" if flag else "no" + + +# ---- the server ---- + +class AdminServer(ThreadingHTTPServer): + daemon_threads = True + + def __init__(self, runner, port=0, code_dir=None): + super().__init__(("127.0.0.1", port), Handler) + self.runner = runner + self.code_dir = code_dir or os.environ.get("FW_CODE_DIR") or os.path.expanduser("~/code") + self.session = secrets.token_urlsafe(32) # the cookie's value + self.csrf = secrets.token_urlsafe(32) # the token in every form: a different secret from the cookie + self.lock = threading.Lock() # one request is handled at a time; an idle socket cannot stall the others + self.login_key = secrets.token_urlsafe(32) # works once + self.flash = [] # (ok, html) shown on the next page and then dropped + + @property + def port(self): + return self.server_address[1] + + def login_url(self): + return f"http://127.0.0.1:{self.port}/login?k={self.login_key}" + + +def make_server(runner, port=0, code_dir=None): + return AdminServer(runner, port, code_dir) + + +def same(a, b): + return hmac.compare_digest(str(a).encode(), str(b).encode()) + + +class Handler(BaseHTTPRequestHandler): + timeout = 15 # an idle connection gives up after this long (it holds only its own thread) + + # never log query strings (the login key sits in one) + def log_message(self, *args): + pass + + def log_request(self, code="-", size="-"): + sys.stderr.write(f"{self.command} {urlsplit(self.path).path} {code}\n") + + # -- sending -- + + def send(self, status, body="", ctype="text/html; charset=utf-8", headers=()): + data = body.encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", ctype) + self.send_header("Content-Length", str(len(data))) + for k, v in (*HEADERS.items(), *headers): + self.send_header(k, v) + self.end_headers() + if self.command != "HEAD": + self.wfile.write(data) + + def redirect(self, where, headers=()): + self.send(303, "", headers=(("Location", where), *headers)) + + def text(self, status, message): + self.send(status, self.page("fw admin", f"<h1>{esc(message)}</h1>", nav=False)) + + def flash_error(self, e): + self.server.flash.append((False, esc(e))) + + # -- access -- + + def check_host(self): + if self.headers.get("Host") != f"127.0.0.1:{self.server.port}": + self.text(421, "Wrong host") + return False + return True + + def has_session(self): + jar = http.cookies.SimpleCookie() + try: + jar.load(self.headers.get("Cookie", "")) + except http.cookies.CookieError: + return False + morsel = jar.get(f"{COOKIE}-{self.server.port}") + return bool(morsel) and same(morsel.value, self.server.session) + + def form(self): + """The POST body as {name: value}, after the CSRF checks; None when the request was refused.""" + origin, site = self.headers.get("Origin"), self.headers.get("Sec-Fetch-Site") + if origin is not None and origin != f"http://127.0.0.1:{self.server.port}": + self.text(403, "Forbidden") + return None + if site is not None and site not in ("same-origin", "none"): + self.text(403, "Forbidden") + return None + if self.body is None: + self.text(400, "Bad request") + return None + fields = {k: v[0] for k, v in parse_qs(self.body.decode("utf-8", "replace"), keep_blank_values=True).items()} + if not same(fields.get("token", ""), self.server.csrf): + self.text(403, "Forbidden") + return None + return fields + + # -- pages -- + + def page(self, title, body, nav=True, current=""): + mark = " aria-current=page" + links = "".join(f'<a href="/{p}"{mark if current == p else ""}>{t}</a>' + for p, t in (("reports", "Reports"), ("comments", "Comments"))) + flash = "" + if nav and self.server.flash: + flash = "".join(f'<div class="flash{"" if ok else " err"}">{h}</div>' for ok, h in self.server.flash) + self.server.flash = [] + return ('<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><title>fw admin</title>' + '<meta name="viewport" content="width=device-width, initial-scale=1">' + '<link rel="stylesheet" href="/style.css"></head><body>' + f'{f"<nav class=top>{links}</nav>" if nav else ""}<main>{flash}{body}</main></body></html>') + + def button(self, action_path, fields, label, danger=False): + hidden = "".join(f'<input type="hidden" name="{esc(k)}" value="{esc(v)}">' for k, v in fields.items()) + return (f'<form method="post" action="{esc(action_path)}"><input type="hidden" name="token" value="{esc(self.server.csrf)}">' + f'{hidden}<button{" class=danger" if danger else ""}>{esc(label)}</button></form>') + + def confirm_page(self, action_path, fields, question, back): + form = self.button(action_path, {**fields, "confirm": "1"}, "Yes, do it", danger=True) + return self.send(200, self.page("fw admin", f'<h1>{esc(question)}</h1><div class="actions">{form}' + f'<a href="{esc(back)}">Cancel</a></div>')) + + # -- GET -- + + def do_GET(self): + with self.server.lock: + self.handle_get() + + def handle_get(self): + if not self.check_host(): + return + url = urlsplit(self.path) + parts = url.path.split("/") + if url.path == "/login": + return self.login(parse_qs(url.query).get("k", [""])[0]) + if not self.has_session(): + return self.text(403, "Forbidden") + try: + if url.path == "/style.css": + return self.send(200, CSS, "text/css; charset=utf-8") + if url.path == "/": + return self.redirect("/reports") + if url.path == "/reports": + return self.reports(parse_qs(url.query)) + if url.path == "/comments": + return self.comments_page() + if len(parts) in (4, 5) and parts[1] == "reports" and (len(parts) == 4 or parts[4] == "record.json"): + if not (valid_repo(parts[2]) and valid_n(parts[3])): + return self.text(400, "Bad request") + if len(parts) == 4: + return self.report_page(parts[2], parts[3]) + return self.download(parts[2], parts[3]) + except FwError as e: + return self.send(502, self.page("fw admin", f'<div class="flash err">{esc(e)}</div>')) + except (KeyError, TypeError, ValueError, AttributeError): + return self.send(502, self.page("fw admin", '<div class="flash err">The server answered in an unexpected shape.</div>')) + self.text(404, "Not found") + + def login(self, key): + server = self.server + if server.login_key is None or not same(key, server.login_key): + return self.text(403, "Forbidden") + server.login_key = None # single use + self.redirect("/reports", headers=(("Set-Cookie", f"{COOKIE}-{server.port}={server.session}; HttpOnly; SameSite=Strict; Path=/"),)) + + def reports(self, query): + show, repo = query.get("show", [""])[0], query.get("repo", [""])[0] + if show not in ("", "waiting", "replay", "all") or (repo and not valid_repo(repo)): + return self.text(400, "Bad request") + data = self.server.runner.json(["issues", "list", "--json"]) + rows = data["issues"] + repos = data.get("repos", {}) + if repo: + rows = [i for i in rows if i["repo"] == repo] + show = show or ("waiting" if any(i["waiting"] for i in rows) else "all") + shown = [i for i in rows if show == "all" or (show == "waiting" and i["waiting"]) or (show == "replay" and i["record"])] + shown.sort(key=lambda i: (i["updated"], i["repo"], i["n"]), reverse=True) + suffix = f"&repo={quote(repo)}" if repo else "" + filters = "".join(f'<a href="/reports?show={k}{suffix}"{" aria-current=page" if show == k else ""}>{t}</a>' + for k, t in (("waiting", "Waiting"), ("replay", "Has replay"), ("all", "All"))) + body = [f'<h1>Reports{f" in {esc(repo)}" if repo else ""}</h1><p class="filters">{filters}</p>'] + if shown: + trs = [] + for i in shown: + link = f'/reports/{quote(i["repo"])}/{int(i["n"])}' + wait = ' <span class="badge badge-warning">waiting</span>' if i["waiting"] else "" + trs.append(f'<tr><td><a href="/reports?show={show}&repo={quote(i["repo"])}">{esc(i["repo"])}</a></td>' + f'<td><a href="{link}">#{int(i["n"])}</a></td><td><a href="{link}">{esc(i["title"])}</a>{wait}</td>' + f'<td>{badge(i["state"])}</td><td>{yes(i["public"])}</td>' + f'<td>{esc(repos.get(i["repo"], {}).get("mode", ""))}</td><td>{yes(i["record"])}</td>' + f'<td>{int(i["shown"])}/{int(i["messages"])}</td><td>{esc(age(i["created"]))}</td></tr>') + body.append("<table><tr><th>Repo</th><th>#</th><th>Title</th><th>State</th><th>Public</th><th>Mode</th>" + f'<th>Record</th><th>Shown</th><th>Age</th></tr>{"".join(trs)}</table>') + else: + body.append("<p>Nothing here.</p>") + self.send(200, self.page("fw admin", "".join(body), current="reports")) + + def replay_available(self, repo): + try: + with open(os.path.join(self.server.code_dir, repo, "package.json")) as f: + pkg = json.load(f) + return isinstance(pkg, dict) and isinstance(pkg.get("scripts"), dict) and "report-replay" in pkg["scripts"] + except (OSError, ValueError): + return False + + def report_page(self, repo, n): + d = self.server.runner.json(["issues", "show", repo, n, "--json"]) + path = f"/reports/{quote(repo)}/{int(n)}" + meta = [f'{esc(repo)}', f'opened {esc(str(d["created"])[:10])}', f'mode {esc(d["mode"])}', + f'page public: {yes(d["public"])}' + (f' ({ext_link(d["page_url"], "open")})' if d.get("page_url") else "")] + out = [f'<h1>#{int(n)} {esc(d["title"])} {badge(d["state"])}</h1><p class="meta">{" · ".join(meta)}</p>'] + if d.get("details"): + out.append('<table class="issue-details">' + "".join(f"<tr><th>{esc(k)}</th><td>{esc(v)}</td></tr>" for k, v in d["details"]) + "</table>") + for m in d["messages"]: + out.append(f'<div class="card"><p class="meta">{esc(m["label"])} · {esc(m["date"])} · authenticated: {yes(m["authenticated"])}' + f' · {"shown" if m["shown"] else "not shown"}</p><pre class="issue-text">{esc(m["text"])}</pre></div>') + toggle = ("reopen", "Reopen") if d["state"] == "closed" else ("close", "Close") + buttons = [self.button(path, {"action": "approve"}, "Approve"), self.button(path, {"action": "hide"}, "Hide", danger=True), + self.button(path, {"action": toggle[0]}, toggle[1])] + out.append(f'<h2>Actions</h2><div class="actions">{"".join(buttons)}</div>') + radios = "".join(f'<label class="mode"><input type="radio" name="mode" value="{m}"{" checked" if m == d["mode"] else ""}> {m}</label>' + for m in MODES) + out.append(f'<form method="post" action="/repos/{quote(repo)}/mode"><input type="hidden" name="token" value="{esc(self.server.csrf)}">' + f'<input type="hidden" name="n" value="{int(n)}"><p>Who sees {esc(repo)}\'s reports: {radios} <button>Set</button></p></form>') + watch = [] + if d.get("watch_url"): + watch.append(ext_link(d["watch_url"], "Watch what happened")) + if d.get("record"): + watch.append(f'<a href="{path}/record.json">Download record</a>') + if self.replay_available(repo): + watch.append(self.button(path, {"action": "replay"}, "Prepare dev replay")) + out.append(f'<h2>Watch</h2><div class="actions">{" ".join(watch) if watch else "<p>This report has no run record.</p>"}</div>') + nav = (f'<a href="/reports/{quote(repo)}/{int(d["prev"])}">← #{int(d["prev"])}</a>' if d.get("prev") else "<span></span>") + \ + (f'<a href="/reports/{quote(repo)}/{int(d["next"])}">#{int(d["next"])} →</a>' if d.get("next") else "<span></span>") + out.append(f'<p class="pager">{nav}</p>') + self.send(200, self.page("fw admin", "".join(out), current="reports")) + + def download(self, repo, n): + try: + record = self.server.runner.run(["issues", "record", repo, n]) + except FwError as e: + self.flash_error(e) + return self.redirect(f"/reports/{quote(repo)}/{int(n)}") + self.send(200, record, "application/json", headers=(("Content-Disposition", f'attachment; filename="{repo}-{int(n)}-record.json"'),)) + + def comments_page(self): + data = self.server.runner.json(["comments", "list", "--json"]) + out = ["<h1>Comments</h1>"] + if not data["pending"]: + out.append("<p>No pending comments.</p>") + for c in data["pending"]: + buttons = [self.button("/comments", {"id": c["id"], "action": "approve"}, "Approve")] + if c["source"] == "email": + buttons.append(self.button("/comments", {"id": c["id"], "action": "trust"}, "Approve + trust")) + buttons.append(self.button("/comments", {"id": c["id"], "action": "reject"}, "Reject", danger=True)) + if c["trusted"]: + buttons.append(self.button("/comments", {"id": c["id"], "action": "untrust"}, "Untrust", danger=True)) + out.append(f'<div class="card"><p class="meta">{self.post_link(c)} · {esc(c["author"])} · {esc(c["date"])} · {esc(c["source"])}' + f'{" · trusted sender" if c["trusted"] else ""}</p><pre class="issue-text">{esc(c["text"])}</pre>' + f'<div class="actions">{"".join(buttons)}</div></div>') + out.append("<h2>Recently approved</h2>") + if not data["approved"]: + out.append("<p>None.</p>") + for c in data["approved"]: + out.append(f'<div class="card"><p class="meta">{self.post_link(c)} · {esc(c["author"])} · {esc(c["date"])}</p>' + f'<pre class="issue-text">{esc(c["text"])}</pre><div class="actions">' + f'{self.button("/comments", {"id": c["id"], "action": "remove"}, "Remove", danger=True)}</div></div>') + self.send(200, self.page("fw admin", "".join(out), current="comments")) + + @staticmethod + def post_link(c): + label = f'{c["site"]}:{c["path"]}' + return ext_link(c.get("post_url"), label) + + # -- POST -- + + def do_POST(self): + try: + length = int(self.headers.get("Content-Length") or 0) + except ValueError: + length = -1 + self.body = self.rfile.read(length) if 0 <= length <= MAX_BODY else None # read before the lock: a client that stalls here stalls only itself + with self.server.lock: + try: + self.handle_post() + except Exception as e: # a failure of any kind still ends in a page, not a dropped connection + self.server.flash.append((False, esc(f"unexpected error: {type(e).__name__}"))) + self.redirect("/reports") + + def handle_post(self): + if not self.check_host(): + return + if not self.has_session(): + return self.text(403, "Forbidden") + f = self.form() + if f is None: + return + parts = urlsplit(self.path).path.split("/") + runner = self.server.runner + if len(parts) == 4 and parts[1] == "reports": + repo, n, action = parts[2], parts[3], f.get("action", "") + if not (valid_repo(repo) and valid_n(n) and action in REPORT_ACTIONS): + return self.text(400, "Bad request") + back = f"/reports/{quote(repo)}/{int(n)}" + if action in CONFIRM and f.get("confirm") != "1": + return self.confirm_page(back, {"action": action}, f"Hide {repo}#{n}? It leaves the public site and stays hidden.", back) + if action == "replay": + self.replay(repo, n) + else: + self.run_flash(["issues", action, repo, n]) + return self.redirect(back) + if len(parts) == 4 and parts[1] == "repos" and parts[3] == "mode": + repo, mode, n = parts[2], f.get("mode", ""), f.get("n", "") + if not (valid_repo(repo) and repo not in RESERVED_REPOS and mode in MODES and (n == "" or valid_n(n))): + return self.text(400, "Bad request") + back = f"/reports/{quote(repo)}/{int(n)}" if n else "/reports" + if f.get("confirm") != "1": + return self.confirm_page(f"/repos/{quote(repo)}/mode", {"mode": mode, "n": n}, + f"Set the reports of {repo} to {mode}? This republishes its issue pages.", back) + self.run_flash(["issues", repo, mode]) + return self.redirect(back) + if parts == ["", "comments"]: + cid, action = f.get("id", ""), f.get("action", "") + if not (ID_RE.match(cid) and action in COMMENT_ACTIONS): + return self.text(400, "Bad request") + if action in CONFIRM and f.get("confirm") != "1": + return self.confirm_page("/comments", {"id": cid, "action": action}, f"{action.capitalize()} comment {cid}?", "/comments") + args = {"trust": ["comments", "approve", cid, "--trust"]}.get(action, ["comments", action, cid]) + self.run_flash(args) + return self.redirect("/comments") + self.text(404, "Not found") + + def run_flash(self, args): + try: + out = self.server.runner.run(args).strip() + self.server.flash.append((True, esc(out or "done"))) + except FwError as e: + self.flash_error(e) + + def replay(self, repo, n): + """Save the record to a fresh folder and run the app's `report-replay` script on it (see the module docstring).""" + folder = os.path.join(self.server.code_dir, repo) + if not self.replay_available(repo): + return self.flash_error(f"{folder}/package.json has no \"report-replay\" script") + try: + record = self.server.runner.run(["issues", "record", repo, n]) + commit = dict(self.server.runner.json(["issues", "show", repo, n, "--json"]).get("details") or []).get("Commit", "") + path = os.path.join(tempfile.mkdtemp(prefix=f"fw-admin-{repo}-{n}-"), "record.json") + with open(path, "w") as out: + out.write(record) + cmd = ["npm", "run", "--silent", "report-replay", "--", path] + (["--rules", commit] if COMMIT_RE.match(commit) else []) + p = subprocess.run(cmd, cwd=folder, stdin=subprocess.DEVNULL, capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=60) + except FwError as e: + return self.flash_error(e) + except subprocess.TimeoutExpired: + return self.flash_error("the replay script timed out after 60 s") + except (OSError, ValueError, TypeError) as e: + return self.flash_error(f"could not prepare the replay: {e}") + text = (p.stdout + p.stderr).strip() or "done" + shown, last = "", 0 + for m in REPLAY_URL_RE.finditer(text): # the dev page's address becomes a link + shown += esc(text[last:m.start()]) + f'<a href="{esc(m.group(0))}">{esc(m.group(0))}</a>' + last = m.end() + shown += esc(text[last:]) + esc(f"\nneeds `npm run dev` running in {repo}") + self.server.flash.append((p.returncode == 0, shown)) + + +def main(argv): + ap = argparse.ArgumentParser(prog="fw admin", description="A local web page for reports and blog comments.") + ap.add_argument("--port", type=int, default=0, help="port on 127.0.0.1 (default: any free one)") + ap.add_argument("--no-open", action="store_true", help="print the login URL but do not open the browser") + ap.add_argument("--code-dir", help="folder with the apps' working copies (default: FW_CODE_DIR or ~/code)") + args = ap.parse_args(argv) + try: + runner = SshRunner(os.environ.get("FW_HOST") or "dh") + server = make_server(runner, args.port, args.code_dir) + except (FwError, OSError) as e: + print(f"fw admin: {e}", file=sys.stderr) + return 1 + url = server.login_url() + print(f"fw admin: {url}\n(works once; ctrl-c to stop; server host: {runner.host}; app folders: {server.code_dir})") + if not args.no_open: + webbrowser.open(url) + try: + server.serve_forever() + except KeyboardInterrupt: + pass + finally: + server.server_close() + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/server/comments.py b/server/comments.py index e60b335..2ca6465 100644 --- a/server/comments.py +++ b/server/comments.py @@ -3,7 +3,8 @@ Usage (on the server: `fw comments <cmd>`; from the laptop: `bin/fw comments <cmd>`): - list pending comments (id, site, path, author, source, first 160 chars) + list [--json] pending comments (id, site, path, author, source, first 160 chars); --json (for `fw admin`): + {"pending": [full comments + post_url, trusted], "approved": [the 20 newest approved]} show <id> one pending or approved comment in full approve <id>... [--trust] publish pending comments; --trust also trusts the sender (email only) reject <id>... discard pending comments (logged in rejected.log) @@ -375,6 +376,35 @@ def trust(h): write_json(trusted_path(), t + [h]) +APPROVED_LISTED = 20 + + +def post_url(site, path): + origin = (load_config().get(site) or {}).get("origin") or "" + return origin.rstrip("/") + path if origin else "" + + +def cmd_list_json(): + """For `fw admin`: pending comments in full and the newest approved ones. Never a sender hash, only whether it is trusted.""" + trusted = set(read_json(trusted_path(), [])) + meta = read_json(meta_path(), {}) + pending = [] + for cid in pending_ids(): + c = read_json(pending_path(cid), {}) + pending.append({"id": cid, "site": c.get("site"), "path": c.get("path"), "post_url": post_url(c.get("site"), c.get("path") or ""), + "author": c.get("author"), "date": c.get("date"), "source": c.get("source"), "parent": c.get("parent"), + "text": c.get("text", ""), "trusted": bool(c.get("sender_hash")) and c["sender_hash"] in trusted}) + approved = [] + for site in site_names(): + for path, items in load_approved(site).items(): + for c in items: + approved.append({"id": c.get("id"), "site": site, "path": path, "post_url": post_url(site, path), + "author": c.get("author"), "date": c.get("date"), "source": None, "parent": c.get("parent"), + "text": c.get("text", ""), "trusted": meta.get(c.get("id")) in trusted and c.get("id") in meta}) + approved.sort(key=lambda c: (c["date"] or "", c["id"] or ""), reverse=True) + print(json.dumps({"pending": pending, "approved": approved[:APPROVED_LISTED]}, ensure_ascii=False)) + + def cmd_list(): for cid in pending_ids(): c = read_json(pending_path(cid), {}) @@ -974,8 +1004,8 @@ def main(argv): flags = [a for a in args if a.startswith("--")] args = [a for a in args if not a.startswith("--")] try: - if cmd == "list": - cmd_list() + if cmd == "list" and not args and set(flags) <= {"--json"}: + (cmd_list_json if "--json" in flags else cmd_list)() elif cmd == "show" and len(args) == 1: cmd_show(args[0]) elif cmd == "approve" and args and set(flags) <= {"--trust"}: diff --git a/server/fw b/server/fw index 7f0332e..78678f7 100755 --- a/server/fw +++ b/server/fw @@ -21,7 +21,7 @@ # fw ci-dashboard rebuild code.fuzzyworld.net/ci/ # fw comments <cmd> moderate blog comments (list, show, approve, reject, ... ingest, imap-check) # fw issues <repo> public|reviewed|private who sees a repo's bug-report issue pages (default private) -# fw issues <cmd> moderate issues (list, show, approve, hide, close, reopen, publish, publish-all, ingest, imap-check) +# fw issues <cmd> moderate issues (list, show, record, approve, hide, close, reopen, publish, publish-all, ingest, imap-check) # # Visibility is `git config fuzzyworld.visibility` inside each repo. Anything other # than exactly "public" is treated as private. @@ -221,7 +221,7 @@ cmd_ci_list() { cmd_issues() { case "${1:-}" in - ingest|imap-check|list|show|approve|hide|close|reopen|publish|publish-all|help|-h|--help|"") + ingest|imap-check|list|show|record|approve|hide|close|reopen|publish|publish-all|help|-h|--help|"") python3 "$TOOLS/server/issues.py" "$@" ;; *) # fw issues <repo> <mode> diff --git a/server/gen.py b/server/gen.py index ce6d106..fd8f56f 100755 --- a/server/gen.py +++ b/server/gen.py @@ -219,6 +219,8 @@ def page(title, body, crumbs=None, view=None, site="code", refresh=None, section f'<a href="{view.url("tree")}/">Files</a>' f'<a href="{view.url("log")}/">Log</a>' f'<a href="{url(view.name, "refs")}/">Branches & tags</a>' + + (f'<a href="{url(view.name, "issues")}/">Issues</a>' + if os.path.exists(os.path.join(SITE, view.name, "issues", "index.html")) else "") + (f'<a href="{url(view.name, "ci")}/">CI</a>' if ci_pipelines(view.name) else "") + (f'<a href="{url(view.name, "usage")}/">Usage</a>' if repo_usage(view.name)["costs"] else "") + "</nav>") diff --git a/server/issues.py b/server/issues.py index 37ae727..1c14260 100644 --- a/server/issues.py +++ b/server/issues.py @@ -3,8 +3,11 @@ Usage (on the server: `fw issues <cmd>`; from the laptop: `bin/fw issues <cmd>`): - list [repo] issues (repo, number, state, whether the page is public, title) - show <repo> <n> one issue in full (no addresses are stored, so none are shown) + list [repo] [--json] issues (repo, number, state, whether the page is public, title); --json (for `fw admin`): + {"repos": {repo: {mode, public_repo}}, "issues": [...with record, messages, shown, waiting]} + show <repo> <n> [--json] one issue in full (no addresses are stored, so none are shown); --json (for `fw admin`): + the full text of every message, shown or not, labels, details, prev/next, page and watch URLs + record <repo> <n> record.json as stored, even when the issue is not published (error when there is none) approve <repo> <n> mark the issue public (what `reviewed` mode waits for) hide <repo> <n> take the issue off the site and keep it off (even in `public` mode) close <repo> <n> / reopen <repo> <n> @@ -266,8 +269,8 @@ def replay_ok(t): and ".." not in t and "\\" not in t and ":" not in head) -def replay_link(repo, n): - """An <a> to the repo's app replaying issue n's record, or "" when no deployed app has a `replay` template.""" +def replay_href(repo, n): + """The URL of the repo's app replaying issue n's record, or "" when no deployed app has a `replay` template.""" found = [] try: entries = sorted(os.listdir(gen.APPS)) @@ -286,8 +289,13 @@ def replay_link(repo, n): return "" _, _, app, replay = min(found) record_url = f"{gen.BASE_URL}/{repo}/issues/{n}/record.json" - href = f"{gen.APPS_URL}/{quote(app)}/" + replay.replace("{record}", quote(record_url, safe="")) - return f'<a href="{gen.esc(href)}" rel="noopener">Watch what happened</a>' + return f"{gen.APPS_URL}/{quote(app)}/" + replay.replace("{record}", quote(record_url, safe="")) + + +def replay_link(repo, n): + """An <a> to the repo's app replaying issue n's record, or "" when no deployed app has a `replay` template.""" + href = replay_href(repo, n) + return f'<a href="{gen.esc(href)}" rel="noopener">Watch what happened</a>' if href else "" def shown_messages(issue, mode): @@ -314,17 +322,23 @@ def badge(state): return f'<span class="badge{" badge-success" if state == "open" else ""}">{gen.esc(state)}</span>' -def details_table(text): - """A small table of the game's own report lines (strict patterns); "" when none matched.""" +def details_rows(text): + """(label, value) pairs for the game's own report lines (strict patterns); [] when none matched.""" rows = [] v, d = VERSION_RE.search(text), SEED_RE.search(text) if v: rows += list(zip(("Version", "Rules", "Commit"), v.groups())) if d: rows += list(zip(("Seed", "Level", "Turns", "Lives", "Status", "Modes"), d.groups())) + return [(k, val.strip()[:DETAIL_MAX]) for k, val in rows] + + +def details_table(text): + """A small table of the game's own report lines (strict patterns); "" when none matched.""" + rows = details_rows(text) if not rows: return "" - cells = "".join(f"<tr><th>{k}</th><td>{gen.esc(val.strip()[:DETAIL_MAX])}</td></tr>" for k, val in rows) + cells = "".join(f"<tr><th>{k}</th><td>{gen.esc(val)}</td></tr>" for k, val in rows) return f'<table class="issue-details">{cells}</table>' @@ -858,6 +872,33 @@ def cmd_ingest(): # ---- moderation ---- +def mode_of(repo): + """(issue mode, whether the repo exists and is public) for a repo name.""" + path = repo_dir(repo) + if not os.path.isdir(path): + return "private", False + return issue_mode(path), gen.is_public(path) + + +def cmd_list_json(repo=None): + """For `fw admin`: every stored issue with its flags, plus each repo's mode. No sender hash, mkey or mid.""" + repos, out = {}, [] + for r in ([check_repo(repo)] if repo else stored_repos()): + mode, is_pub = mode_of(r) + repos[r] = {"mode": mode, "public_repo": is_pub} + path = repo_dir(r) + for i in load_issues(r): + msgs = i.get("messages", []) + shown = len(shown_messages(i, mode)) + seen = bool(msgs) and is_pub and visible(path, i, mode) + out.append({"repo": r, "n": i["n"], "title": i["title"], "created": i["created"], + "updated": msgs[-1]["date"] if msgs else i["created"], "state": i["state"], + "approved": bool(i.get("approved")), "hidden": bool(i.get("hidden")), "public": seen, + "record": bool(i.get("record")), "messages": len(msgs), "shown": shown, + "waiting": not i.get("hidden") and mode in ("public", "reviewed") and is_pub and shown < len(msgs)}) + print(json.dumps({"repos": repos, "issues": out}, ensure_ascii=False)) + + def cmd_list(repo=None): for r in ([check_repo(repo)] if repo else stored_repos()): path = repo_dir(r) @@ -873,6 +914,39 @@ def cmd_show(repo, n): print(json.dumps(load_issue(check_repo(repo), n), indent=2, ensure_ascii=False)) +def cmd_show_json(repo, n): + """For `fw admin`: one issue with the full text of every message, shown or not. No sender hash, mkey or mid.""" + n = check_n(n) + issue = load_issue(check_repo(repo), n) + mode, is_pub = mode_of(repo) + msgs = issue.get("messages", []) + shown = shown_messages(issue, mode) + public = bool(msgs) and is_pub and visible(repo_dir(repo), issue, mode) + rec_shown = record_shown(issue, mode) + watch = replay_href(repo, n) if rec_shown else "" + numbers = issue_numbers(repo) + out = {"repo": repo, "n": n, "title": issue["title"], "created": issue["created"], "state": issue["state"], + "approved": bool(issue.get("approved")), "hidden": bool(issue.get("hidden")), "mode": mode, "public_repo": is_pub, + "public": public, "page_url": f"{gen.BASE_URL}/{repo}/issues/{n}/" if public else None, + "record": bool(issue.get("record")), "record_shown": rec_shown, "watch_url": watch or None, + "details": [list(r) for r in details_rows(msgs[0]["text"])] if msgs else [], + "prev": max((x for x in numbers if x < n), default=None), "next": min((x for x in numbers if x > n), default=None), + "messages": [{"label": label, "date": m.get("date", ""), "authenticated": bool(m.get("authenticated")), + "approved": bool(m.get("approved")), "shown": any(m is s for s in shown), "text": m.get("text", "")} + for label, m in zip(reporter_labels(msgs), msgs)]} + print(json.dumps(out, ensure_ascii=False)) + + +def cmd_record(repo, n): + """record.json exactly as stored (for `fw admin`: the record of a report that is not published).""" + n = check_n(n) + try: + with open(record_path(check_repo(repo), n)) as f: + sys.stdout.write(f.read()) + except FileNotFoundError: + raise Error(f"no record for {repo}#{n}") + + def cmd_change(repo, n, action): n = check_n(n) with locked(): @@ -917,10 +991,12 @@ def main(argv): flags = [a for a in args if a.startswith("--")] args = [a for a in args if not a.startswith("--")] try: - if cmd == "list" and len(args) <= 1: - cmd_list(args[0] if args else None) - elif cmd == "show" and len(args) == 2: - cmd_show(*args) + if cmd == "list" and len(args) <= 1 and set(flags) <= {"--json"}: + (cmd_list_json if "--json" in flags else cmd_list)(args[0] if args else None) + elif cmd == "show" and len(args) == 2 and set(flags) <= {"--json"}: + (cmd_show_json if "--json" in flags else cmd_show)(*args) + elif cmd == "record" and len(args) == 2 and not flags: + cmd_record(*args) elif cmd in ("approve", "hide", "close", "reopen") and len(args) == 2: cmd_change(args[0], args[1], cmd) elif cmd == "publish" and len(args) == 1 and set(flags) <= {"--locked"}: @@ -942,7 +1018,7 @@ def main(argv): finally: source.close() else: - print("usage: issues.py list|show|approve|hide|close|reopen|publish|publish-all|ingest|imap-check (try --help)", file=sys.stderr) + print("usage: issues.py list|show|record|approve|hide|close|reopen|publish|publish-all|ingest|imap-check (try --help)", file=sys.stderr) return 2 except (Error, OSError) as e: print(f"issues: {e}", file=sys.stderr) diff --git a/tests/test_admin.py b/tests/test_admin.py new file mode 100644 index 0000000..44fab5f --- /dev/null +++ b/tests/test_admin.py @@ -0,0 +1,490 @@ +"""Tests for lib/admin.py (the local `fw admin` web page). + +Run with: python3 -m unittest discover tests +""" + +import http.client +import importlib.util +import json +import os +import re +import shutil +import subprocess +import tempfile +import threading +import unittest +from unittest import mock + +ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..") +spec = importlib.util.spec_from_file_location("admin", os.path.join(ROOT, "lib", "admin.py")) +admin = importlib.util.module_from_spec(spec) +spec.loader.exec_module(admin) + +CID = "20241231T101010Z-abcdef" +EVIL = "<script>alert(1)</script>" + + +def issue_list(**over): + i = {"repo": "demo", "n": 3, "title": "Crash", "created": "2026-11-01T10:00:00Z", "updated": "2026-11-02T10:00:00Z", + "state": "open", "approved": False, "hidden": False, "public": False, "record": True, "messages": 2, "shown": 1, + "waiting": True} + i.update(over) + return {"repos": {"demo": {"mode": "reviewed", "public_repo": True}}, "issues": [i]} + + +def issue_show(**over): + d = {"repo": "demo", "n": 3, "title": "Crash", "created": "2026-11-01T10:00:00Z", "state": "open", "approved": False, + "hidden": False, "mode": "reviewed", "public_repo": True, "public": False, "page_url": None, "record": True, + "record_shown": False, "watch_url": None, "details": [["Version", "1.2"], ["Commit", "abcdef1"]], "prev": 2, "next": None, + "messages": [{"label": "Reporter A", "date": "2026-11-01T10:00:00Z", "authenticated": True, "approved": False, + "shown": False, "text": "It crashed."}]} + d.update(over) + return d + + +def comment_list(**over): + c = {"id": CID, "site": "writing", "path": "/2024/05/01/hello/", "post_url": "https://writing.fuzzyworld.net/2024/05/01/hello/", + "author": "Ann", "date": "2026-11-01T10:00:00Z", "source": "email", "parent": None, "text": "Nice post.", "trusted": True} + c.update(over) + return {"pending": [c], "approved": [{**c, "id": "20241230T101010Z-123456", "source": None, "text": "Old one"}]} + + +class FakeRunner: + host = "fake" + + def __init__(self): + self.calls, self.replies, self.errors = [], {}, {} + + def run(self, args): + self.calls.append(list(args)) + if tuple(args) in self.errors: + raise admin.FwError(self.errors[tuple(args)]) + reply = self.replies.get(tuple(args), "ok") + return reply if isinstance(reply, str) else json.dumps(reply) + + def json(self, args): + return json.loads(self.run(args)) + + +class Base(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmp) + p = mock.patch.object(admin.Handler, "log_request") + p.start() + self.addCleanup(p.stop) + self.fake = FakeRunner() + self.fake.replies[("issues", "list", "--json")] = issue_list() + self.fake.replies[("issues", "show", "demo", "3", "--json")] = issue_show() + self.fake.replies[("comments", "list", "--json")] = comment_list() + self.server = admin.make_server(self.fake, 0, code_dir=self.tmp) + self.port = self.server.port + t = threading.Thread(target=self.server.serve_forever, daemon=True) + t.start() + self.addCleanup(t.join) + self.addCleanup(self.server.server_close) + self.addCleanup(self.server.shutdown) + self.cookie = f"fwadmin-{self.port}={self.server.session}" + + def req(self, method, path, body=None, headers=None, login=True, token=True): + h = {"Host": f"127.0.0.1:{self.port}"} + if login: + h["Cookie"] = self.cookie + if body is not None: + body = dict(body) + if token: + body.setdefault("token", self.server.csrf) + from urllib.parse import urlencode + body = urlencode(body) + h["Content-Type"] = "application/x-www-form-urlencoded" + h.update(headers or {}) + c = http.client.HTTPConnection("127.0.0.1", self.port, timeout=10) + try: + c.request(method, path, body=body, headers=h) + r = c.getresponse() + return r.status, dict(r.getheaders()), r.read().decode() + finally: + c.close() + + def get(self, path, **kw): + return self.req("GET", path, **kw) + + def post(self, path, body, **kw): + return self.req("POST", path, body=body, **kw) + + +class Access(Base): + def test_no_cookie_is_forbidden(self): + self.assertEqual(self.get("/reports", login=False)[0], 403) + self.assertEqual(self.get("/style.css", login=False)[0], 403) + self.assertEqual(self.fake.calls, []) + + def test_login_works_once(self): + key = self.server.login_key + status, headers, _ = self.get(f"/login?k={key}", login=False) + self.assertEqual((status, headers["Location"]), (303, "/reports")) + cookie = headers["Set-Cookie"] + self.assertIn(f"fwadmin-{self.port}={self.server.session}", cookie) + for word in ("HttpOnly", "SameSite=Strict", "Path=/"): + self.assertIn(word, cookie) + self.assertEqual(self.get(f"/login?k={key}", login=False)[0], 403) + self.assertEqual(self.get("/login?k=wrong", login=False)[0], 403) + self.assertEqual(self.get("/reports")[0], 200) + + def test_wrong_login_key_does_not_burn_the_real_one(self): + self.assertEqual(self.get("/login?k=nope", login=False)[0], 403) + self.assertEqual(self.get(f"/login?k={self.server.login_key}", login=False)[0], 303) + + def test_wrong_host_is_421(self): + self.assertEqual(self.get("/reports", headers={"Host": "evil.example:80"})[0], 421) + self.assertEqual(self.get("/reports", headers={"Host": f"localhost:{self.port}"})[0], 421) + self.assertEqual(self.post("/comments", {"id": CID, "action": "approve"}, headers={"Host": "evil.example"})[0], 421) + self.assertEqual(self.fake.calls, []) + + def test_post_without_token_or_with_foreign_origin_is_forbidden(self): + body = {"action": "approve"} + self.assertEqual(self.post("/reports/demo/3", body, token=False)[0], 403) + self.assertEqual(self.post("/reports/demo/3", {**body, "token": "wrong"})[0], 403) + self.assertEqual(self.post("/reports/demo/3", body, headers={"Origin": "http://evil.example"})[0], 403) + self.assertEqual(self.post("/reports/demo/3", body, headers={"Origin": "null"})[0], 403) + self.assertEqual(self.post("/reports/demo/3", body, headers={"Sec-Fetch-Site": "cross-site"})[0], 403) + self.assertEqual(self.post("/reports/demo/3", body, login=False)[0], 403) + self.assertEqual(self.fake.calls, []) + ok = self.post("/reports/demo/3", body, headers={"Origin": f"http://127.0.0.1:{self.port}", "Sec-Fetch-Site": "same-origin"}) + self.assertEqual(ok[0], 303) + + def test_csrf_token_is_not_the_cookie_value(self): + self.assertNotEqual(self.server.csrf, self.server.session) + self.assertIn(self.server.csrf, self.get("/reports/demo/3")[2]) + self.assertNotIn(self.server.session, self.get("/reports/demo/3")[2]) + self.assertEqual(self.post("/reports/demo/3", {"action": "approve", "token": self.server.session})[0], 403) + self.fake.calls.clear() + self.assertEqual(self.get("/reports", headers={"Cookie": f"fwadmin={self.server.session}"})[0], 403) # not the per-port name + self.assertEqual(self.fake.calls, []) + + def test_an_idle_connection_does_not_stall_others(self): + import socket + import time + idle = socket.create_connection(("127.0.0.1", self.port)) + self.addCleanup(idle.close) + half = socket.create_connection(("127.0.0.1", self.port)) # headers promise a body that never comes + self.addCleanup(half.close) + half.sendall(f"POST /comments HTTP/1.0\r\nHost: 127.0.0.1:{self.port}\r\nContent-Length: 50\r\n\r\n".encode()) + start = time.time() + self.assertEqual(self.get("/reports")[0], 200) + self.assertLess(time.time() - start, 2) + + def test_security_headers(self): + _, h, _ = self.get("/reports") + self.assertIn("default-src 'none'", h["Content-Security-Policy"]) + self.assertIn("form-action 'self'", h["Content-Security-Policy"]) + self.assertIn("frame-ancestors 'none'", h["Content-Security-Policy"]) + self.assertEqual((h["X-Content-Type-Options"], h["Referrer-Policy"], h["Cache-Control"]), ("nosniff", "no-referrer", "no-store")) + + def test_unknown_route_is_404_and_no_javascript(self): + self.assertEqual(self.get("/nothing")[0], 404) + body = self.get("/reports")[2] + self.get("/comments")[2] + self.get("/reports/demo/3")[2] + self.assertNotIn("<script", body) + self.assertNotIn("onclick", body) + self.assertIn("<title>fw admin</title>", body) + + def test_style_is_served(self): + status, h, body = self.get("/style.css") + self.assertEqual((status, h["Content-Type"].split(";")[0]), (200, "text/css")) + self.assertIn(".badge", body) + + +class Actions(Base): + def test_approve_runs_the_command_and_redirects(self): + status, h, _ = self.post("/reports/demo/3", {"action": "approve"}) + self.assertEqual((status, h["Location"]), (303, "/reports/demo/3")) + self.assertEqual(self.fake.calls, [["issues", "approve", "demo", "3"]]) + + def test_close_reopen_and_mode(self): + self.post("/reports/demo/3", {"action": "close"}) + self.post("/reports/demo/3", {"action": "reopen"}) + self.assertEqual(self.fake.calls, [["issues", "close", "demo", "3"], ["issues", "reopen", "demo", "3"]]) + + def test_hide_needs_confirm(self): + status, _, body = self.post("/reports/demo/3", {"action": "hide"}) + self.assertEqual(status, 200) + self.assertEqual(self.fake.calls, []) + self.assertIn('name="confirm" value="1"', body) + self.assertIn('name="action" value="hide"', body) + self.assertIn("Cancel", body) + self.assertEqual(self.post("/reports/demo/3", {"action": "hide", "confirm": "1"})[0], 303) + self.assertEqual(self.fake.calls, [["issues", "hide", "demo", "3"]]) + + def test_mode_change_needs_confirm(self): + status, _, body = self.post("/repos/demo/mode", {"mode": "public", "n": "3"}) + self.assertEqual((status, self.fake.calls), (200, [])) + self.assertIn('name="mode" value="public"', body) + status, h, _ = self.post("/repos/demo/mode", {"mode": "public", "n": "3", "confirm": "1"}) + self.assertEqual((status, h["Location"]), (303, "/reports/demo/3")) + self.assertEqual(self.fake.calls, [["issues", "demo", "public"]]) + + def test_invalid_input_is_400_without_a_call(self): + for path, body in (("/reports/a;b/3", {"action": "approve"}), ("/reports/demo/1x", {"action": "approve"}), + ("/reports/demo/0", {"action": "approve"}), ("/reports/demo/3", {"action": "delete"}), + ("/reports/demo/3", {"action": "--json"}), ("/reports/demo.git/3", {"action": "approve"}), + ("/repos/demo/mode", {"mode": "open"}), ("/repos/list/mode", {"mode": "public", "confirm": "1"}), + ("/repos/demo/mode", {"mode": "public", "n": "x", "confirm": "1"}), + ("/comments", {"id": "../etc", "action": "approve"}), ("/comments", {"id": CID + "\n", "action": "approve"}), ("/comments", {"id": CID, "action": "nuke"})): + self.assertEqual(self.post(path, body)[0], 400, (path, body)) + self.assertEqual(self.get("/reports/a;b/3")[0], 400) + self.assertEqual(self.get("/reports/demo/1x")[0], 400) + self.assertEqual(self.get("/reports/demo/1x/record.json")[0], 400) + self.assertEqual(self.get("/reports?show=bogus")[0], 400) + self.assertEqual(self.get("/reports?repo=a;b")[0], 400) + self.assertEqual(self.fake.calls, []) + + def test_comment_actions(self): + self.post("/comments", {"id": CID, "action": "approve"}) + self.post("/comments", {"id": CID, "action": "trust"}) + self.assertEqual(self.fake.calls, [["comments", "approve", CID], ["comments", "approve", CID, "--trust"]]) + + def test_comment_reject_untrust_remove_need_confirm(self): + for action in ("reject", "untrust", "remove"): + self.fake.calls.clear() + status, _, body = self.post("/comments", {"id": CID, "action": action}) + self.assertEqual((status, self.fake.calls), (200, []), action) + self.assertIn('name="confirm" value="1"', body) + self.assertEqual(self.post("/comments", {"id": CID, "action": action, "confirm": "1"})[0], 303) + self.assertEqual(self.fake.calls, [["comments", action, CID]]) + + def test_flash_shows_output_once_and_escapes_errors(self): + self.fake.replies[("issues", "approve", "demo", "3")] = "approve: demo#3 (public)" + self.post("/reports/demo/3", {"action": "approve"}) + page = self.get("/reports/demo/3")[2] + self.assertIn("approve: demo#3 (public)", page) + self.assertNotIn("approve: demo#3 (public)", self.get("/reports/demo/3")[2]) + self.fake.errors[("issues", "close", "demo", "3")] = f"boom {EVIL}" + self.post("/reports/demo/3", {"action": "close"}) + page = self.get("/reports/demo/3")[2] + self.assertIn("boom <script>", page) + self.assertNotIn(EVIL, page) + + def test_unexpected_error_in_a_post_still_redirects_with_a_flash(self): + def boom(args): + raise ValueError("<b>bad</b>") + self.fake.run = boom + status, h, _ = self.post("/reports/demo/3", {"action": "approve"}) + self.assertEqual(status, 303) + self.assertIn("unexpected error: ValueError", self.get("/reports/demo/3")[2]) + self.assertNotIn("<b>bad</b>", self.get("/reports/demo/3")[2]) + + def test_failed_listing_is_shown_not_crashed(self): + self.fake.errors[("issues", "list", "--json")] = "could not reach dh" + status, _, body = self.get("/reports") + self.assertEqual(status, 502) + self.assertIn("could not reach dh", body) + + +class Pages(Base): + def test_reports_list(self): + body = self.get("/reports")[2] + for text in ("Crash", "demo", "reviewed", "1/2", "waiting", 'href="/reports/demo/3"', "Has replay"): + self.assertIn(text, body) + self.assertEqual(self.fake.calls, [["issues", "list", "--json"]]) + + def test_default_filter_is_waiting_else_all_and_filters_apply(self): + self.fake.replies[("issues", "list", "--json")] = issue_list(waiting=False, record=False) + self.assertIn("Crash", self.get("/reports")[2]) # nothing waiting: all + self.assertNotIn("Crash", self.get("/reports?show=waiting")[2]) + self.assertNotIn("Crash", self.get("/reports?show=replay")[2]) + self.assertIn("Crash", self.get("/reports?show=all")[2]) + + def test_report_page(self): + body = self.get("/reports/demo/3")[2] + for text in ("It crashed.", '<pre class="issue-text">', "Reporter A", "authenticated: yes", "not shown", "Commit", "abcdef1", + "Approve", "Hide", "Close", 'name="mode" value="reviewed" checked', "Download record", 'href="/reports/demo/2"'): + self.assertIn(text, body) + self.assertNotIn("Prepare dev replay", body) # no package.json in code_dir + self.assertEqual(self.fake.calls, [["issues", "show", "demo", "3", "--json"]]) + self.fake.replies[("issues", "show", "demo", "3", "--json")] = issue_show(state="closed") + self.assertIn("Reopen", self.get("/reports/demo/3")[2]) + + def test_external_links_only_when_https(self): + self.fake.replies[("issues", "show", "demo", "3", "--json")] = issue_show(public=True, page_url="https://code.fuzzyworld.net/demo/issues/3/", + watch_url="javascript:alert(1)") + body = self.get("/reports/demo/3")[2] + self.assertIn('href="https://code.fuzzyworld.net/demo/issues/3/" rel="noreferrer noopener"', body) + self.assertNotIn("javascript:", body.replace("javascript:alert(1)</", "")) + self.assertNotIn('href="javascript', body) + self.fake.replies[("issues", "show", "demo", "3", "--json")] = issue_show(watch_url="https://apps.fuzzyworld.net/demo/replay.html?r=1&x=2") + body = self.get("/reports/demo/3")[2] + self.assertIn("Watch what happened", body) + self.assertIn("r=1&x=2", body) + + def test_hostile_text_is_escaped_everywhere(self): + self.fake.replies[("issues", "list", "--json")] = issue_list(title=EVIL) + self.fake.replies[("issues", "show", "demo", "3", "--json")] = issue_show( + title=EVIL, details=[[EVIL, EVIL]], messages=[{**issue_show()["messages"][0], "text": EVIL, "label": EVIL}]) + self.fake.replies[("comments", "list", "--json")] = comment_list(text=EVIL, author=EVIL, site=EVIL, post_url="javascript:" + EVIL) + for path in ("/reports", "/reports/demo/3", "/comments"): + body = self.get(path)[2] + self.assertNotIn("<script>", body, path) + self.assertIn("<script>alert(1)</script>", self.get("/reports/demo/3")[2]) + self.assertNotIn('href="javascript', self.get("/comments")[2]) + + def test_comments_page(self): + body = self.get("/comments")[2] + for text in ("Nice post.", "Approve + trust", "Reject", "Untrust", "Recently approved", "Old one", "Remove", + 'href="https://writing.fuzzyworld.net/2024/05/01/hello/" rel="noreferrer noopener"', "20241230T101010Z-123456"): + self.assertIn(text, body) + self.assertEqual(self.fake.calls, [["comments", "list", "--json"]]) + + def test_comments_buttons_depend_on_source_and_trust(self): + self.fake.replies[("comments", "list", "--json")] = comment_list(source="form", trusted=False) + pending = self.get("/comments")[2].split("Recently approved")[0] + self.assertNotIn("Approve + trust", pending) + self.assertNotIn("Untrust", pending) + + def test_download_record(self): + self.fake.replies[("issues", "record", "demo", "3")] = '{"seed":7}' + status, h, body = self.get("/reports/demo/3/record.json") + self.assertEqual((status, body), (200, '{"seed":7}')) + self.assertIn("attachment", h["Content-Disposition"]) + self.fake.errors[("issues", "record", "demo", "3")] = "no record for demo#3" + status, h, _ = self.get("/reports/demo/3/record.json") + self.assertEqual((status, h["Location"]), (303, "/reports/demo/3")) + self.assertIn("no record for demo#3", self.get("/reports/demo/3")[2]) + + +class Replay(Base): + def app(self, scripts=None): + os.makedirs(os.path.join(self.tmp, "demo")) + with open(os.path.join(self.tmp, "demo", "package.json"), "w") as f: + json.dump({"scripts": scripts if scripts is not None else {"report-replay": "node x.mjs"}}, f) + + def test_button_only_with_the_script(self): + self.app({"dev": "vite"}) + self.assertNotIn("Prepare dev replay", self.get("/reports/demo/3")[2]) + shutil.rmtree(os.path.join(self.tmp, "demo")) + self.app() + self.assertIn("Prepare dev replay", self.get("/reports/demo/3")[2]) + + def test_replay_runs_npm_with_the_record_and_rules_commit(self): + self.app() + self.fake.replies[("issues", "record", "demo", "3")] = '{"seed":7}' + done = subprocess.CompletedProcess([], 0, stdout="open http://localhost:5173/?replay=/tmp/x/record.json&rules=abcdef1 now\n", stderr="") + with mock.patch.object(admin.subprocess, "run", return_value=done) as run: + self.assertEqual(self.post("/reports/demo/3", {"action": "replay"})[0], 303) + cmd = run.call_args.args[0] + self.assertEqual(cmd[:6], ["npm", "run", "--silent", "report-replay", "--", cmd[5]]) + self.assertEqual(cmd[6:], ["--rules", "abcdef1"]) + self.assertTrue(cmd[5].endswith("record.json")) + with open(cmd[5]) as f: + self.assertEqual(f.read(), '{"seed":7}') + shutil.rmtree(os.path.dirname(cmd[5])) + self.assertEqual(run.call_args.kwargs["cwd"], os.path.join(self.tmp, "demo")) + self.assertIn("fw-admin-demo-3-", cmd[5]) + self.assertEqual(run.call_args.kwargs["errors"], "replace") + self.assertEqual(run.call_args.kwargs["timeout"], 60) + page = self.get("/reports/demo/3")[2] + self.assertIn('<a href="http://localhost:5173/?replay=/tmp/x/record.json&rules=abcdef1">', page) + self.assertIn("needs `npm run dev` running in demo", page) + + def test_replay_without_commit_or_with_a_bad_one(self): + self.app() + self.fake.replies[("issues", "show", "demo", "3", "--json")] = issue_show(details=[["Commit", "not-hex"]]) + done = subprocess.CompletedProcess([], 0, stdout="ok", stderr="") + with mock.patch.object(admin.subprocess, "run", return_value=done) as run: + self.post("/reports/demo/3", {"action": "replay"}) + shutil.rmtree(os.path.dirname(run.call_args.args[0][5])) + self.assertEqual(len(run.call_args.args[0]), 6) + + def test_replay_without_the_script_is_an_error(self): + with mock.patch.object(admin.subprocess, "run") as run: + self.post("/reports/demo/3", {"action": "replay"}) + run.assert_not_called() + self.assertIn("report-replay", self.get("/reports/demo/3")[2]) + + def test_code_dir_comes_from_the_environment(self): + with mock.patch.dict(os.environ, {"FW_CODE_DIR": "/somewhere/else"}): + s = admin.make_server(self.fake, 0) + self.addCleanup(s.server_close) + self.assertEqual(s.code_dir, "/somewhere/else") + s2 = admin.make_server(self.fake, 0, code_dir="/given") + self.addCleanup(s2.server_close) + self.assertEqual(s2.code_dir, "/given") + + +class Ssh(unittest.TestCase): + def done(self, code=0, out="", err=""): + return subprocess.CompletedProcess([], code, stdout=out, stderr=err) + + def test_exact_argv(self): + with mock.patch.object(admin.subprocess, "run", return_value=self.done(out="hi")) as run: + self.assertEqual(admin.SshRunner("dh").run(["issues", "show", "demo", "3", "--json"]), "hi") + self.assertEqual(run.call_args.args[0], ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", "dh", + "~/bin/fw issues show demo 3 --json"]) + kw = run.call_args.kwargs + self.assertEqual((kw["stdin"], kw["capture_output"], kw["text"], kw["timeout"]), (subprocess.DEVNULL, True, True, 60)) + self.assertEqual((kw["encoding"], kw["errors"]), ("utf-8", "replace")) + + def test_unsafe_arguments_are_refused_before_anything_is_sent(self): + r = admin.SshRunner("dh") + for bad in ("a;b", "$(id)", "-x", "--rm", "a b", "", "x\n", "a/b", "--json ", "`ls`"): + with mock.patch.object(admin.subprocess, "run") as run: + with self.assertRaises(admin.FwError): + r.run(["issues", bad]) + run.assert_not_called() + with mock.patch.object(admin.subprocess, "run", return_value=self.done()) as run: + r.run(["comments", "approve", CID, "--trust"]) + run.assert_called_once() + + def test_host_is_checked(self): + for bad in ("-oProxyCommand=x", "a b", "a;b", "", "x@y"): + with self.assertRaises(admin.FwError): + admin.SshRunner(bad) + admin.SshRunner("my-host.example") + + def test_timeout_unreachable_and_failure(self): + r = admin.SshRunner("dh", timeout=5) + with mock.patch.object(admin.subprocess, "run", side_effect=subprocess.TimeoutExpired("ssh", 5)): + with self.assertRaisesRegex(admin.FwError, "timed out after 5 s"): + r.run(["issues", "list"]) + with mock.patch.object(admin.subprocess, "run", return_value=self.done(255, err="ssh: Could not resolve")): + with self.assertRaisesRegex(admin.FwError, r"could not reach dh \(try `ssh dh true`\)"): + r.run(["issues", "list"]) + with mock.patch.object(admin.subprocess, "run", return_value=self.done(1, err="issues: no issue demo#9\n")): + with self.assertRaisesRegex(admin.FwError, "no issue demo#9"): + r.run(["issues", "list"]) + with mock.patch.object(admin.subprocess, "run", side_effect=FileNotFoundError("ssh")): + with self.assertRaises(admin.FwError): + r.run(["issues", "list"]) + + def test_json(self): + r = admin.SshRunner("dh") + with mock.patch.object(admin.subprocess, "run", return_value=self.done(out='{"a": 1}')): + self.assertEqual(r.json(["issues", "list", "--json"]), {"a": 1}) + for out in ("not json", "[1]"): + with mock.patch.object(admin.subprocess, "run", return_value=self.done(out=out)): + with self.assertRaises(admin.FwError): + r.json(["issues", "list", "--json"]) + + +class Cli(unittest.TestCase): + def test_main_prints_the_login_url_without_opening(self): + started = {} + + def fake_serve(self): + started["url"] = self.login_url() + + import io + out = io.StringIO() + with mock.patch.object(admin.AdminServer, "serve_forever", fake_serve), mock.patch.object(admin.webbrowser, "open") as opened, \ + mock.patch("sys.stdout", out), mock.patch.dict(os.environ, {"FW_HOST": "dh"}): + self.assertEqual(admin.main(["--no-open", "--code-dir", "/x"]), 0) + opened.assert_not_called() + self.assertTrue(re.match(r"^http://127\.0\.0\.1:\d+/login\?k=[\w-]+$", started["url"])) + self.assertIn(started["url"], out.getvalue()) + + def test_bad_host_env_is_reported(self): + with mock.patch.dict(os.environ, {"FW_HOST": "-oEvil"}): + self.assertEqual(admin.main(["--no-open"]), 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_comments.py b/tests/test_comments.py index 7570bd2..b1ab777 100644 --- a/tests/test_comments.py +++ b/tests/test_comments.py @@ -83,6 +83,45 @@ class Base(unittest.TestCase): return c["id"] +class ListJson(Base): + def js(self): + code, out, err = self.run_cli("list", "--json") + self.assertEqual((code, err), (0, "")) + for word in ("sender_hash", "msg_key", "mkey"): + self.assertNotIn(word, out) + return json.loads(out) + + def test_pending_fields_post_url_and_trusted(self): + a = self.add_pending(source="email", sender_hash="h1", text="full <b>text</b>\nline two " * 30) + b = self.add_pending(source="form") + comments.write_json(comments.trusted_path(), ["h1"]) + pending = {c["id"]: c for c in self.js()["pending"]} + self.assertEqual(set(pending[a]), {"id", "site", "path", "post_url", "author", "date", "source", "parent", "text", "trusted"}) + self.assertEqual(pending[a]["post_url"], "https://w" + POST) + self.assertEqual(pending[a]["text"], "full <b>text</b>\nline two " * 30) # in full, not cut to 160 + self.assertEqual((pending[a]["trusted"], pending[b]["trusted"]), (True, False)) + + def test_approved_are_the_newest_twenty_and_removable(self): + ids = [] + for i in range(23): + ids.append(self.add_pending(date=f"2024-05-01T10:00:{i:02d}Z", text=f"c{i}", sender_hash=f"s{i}")) + self.assertEqual(self.run_cli("approve", *ids)[0], 0) + data = self.js() + self.assertEqual(data["pending"], []) + self.assertEqual(len(data["approved"]), 20) + self.assertEqual(data["approved"][0]["text"], "c22") + self.assertEqual(data["approved"][-1]["text"], "c3") + self.assertEqual(set(data["approved"][0]), {"id", "site", "path", "post_url", "author", "date", "source", "parent", "text", "trusted"}) + self.assertFalse(data["approved"][0]["trusted"]) + self.run_cli("remove", data["approved"][0]["id"]) + self.assertEqual(len(self.js()["approved"]), 20) + self.assertNotIn("c22", [c["text"] for c in self.js()["approved"]]) + + def test_empty_and_bad_flags(self): + self.assertEqual(self.js(), {"pending": [], "approved": []}) + self.assertEqual(self.run_cli("list", "--bogus")[0], 2) + + class Moderation(Base): def test_approve_publishes_without_private_fields(self): cid = self.add_pending(source="email", sender_hash="abc") diff --git a/tests/test_issues.py b/tests/test_issues.py index ea083e0..c9c8a0f 100644 --- a/tests/test_issues.py +++ b/tests/test_issues.py @@ -10,6 +10,8 @@ import itertools import json import mailbox import os +import types +import re import shutil import subprocess import tempfile @@ -885,6 +887,14 @@ class FwScript(Base): return subprocess.run(["git", "-C", os.path.join(self.git, repo + ".git"), "config", "--get", "fuzzyworld.issues"], capture_output=True, text=True).stdout.strip() + def test_repo_tabs_carry_an_issues_tab_only_when_there_are_public_issues(self): + view = types.SimpleNamespace(name="demo", url=lambda kind: f"/demo/{kind}") + def tabs(): + return re.search(r'<nav class="repo-nav">.*?</nav>', issues.gen.page("t", "b", view=view), re.S).group(0) + self.assertNotIn(">Issues</a>", tabs()) + self.ingest(mail()) + self.assertIn(">Issues</a>", tabs()) + def test_mode_setter_and_publish_one(self): self.ingest(mail()) r = self.fw("issues", "demo", "reviewed") @@ -959,6 +969,101 @@ class Replay(Base): self.assertEqual(self.summaries, ["demo", "demo"]) # appeared, then disappeared +class AdminJson(Base): + """The machine-readable commands `fw admin` runs: list --json, show --json, record.""" + PRIVATE = ("sender_hash", "mkey", "mid") + + def js(self, *argv): + code, out, err = self.cli(*argv) + self.assertEqual((code, err), (0, "")) + for word in self.PRIVATE: + self.assertNotIn(word, out) + return json.loads(out) + + def test_list_json_fields_and_waiting(self): + self.repo("rev", "public", "reviewed") + self.repo("priv", "public", "private") + self.ingest(mail(repo="rev", body="Boom.\nRun record: " + record() + "\n"), mail(), mail(repo="priv")) + data = self.js("list", "--json") + self.assertEqual(data["repos"]["rev"], {"mode": "reviewed", "public_repo": True}) + self.assertEqual(data["repos"]["priv"]["mode"], "private") + by = {(i["repo"], i["n"]): i for i in data["issues"]} + rev, pub, priv = by[("rev", 1)], by[("demo", 1)], by[("priv", 1)] + self.assertEqual((rev["public"], rev["waiting"], rev["record"], rev["messages"], rev["shown"]), (False, True, True, 1, 0)) + self.assertEqual((pub["public"], pub["waiting"], pub["shown"]), (True, False, 1)) + self.assertFalse(priv["waiting"]) + self.assertEqual(set(rev), {"repo", "n", "title", "created", "updated", "state", "approved", "hidden", "public", + "record", "messages", "shown", "waiting"}) + self.cli("approve", "rev", "1") + rev = {(i["repo"], i["n"]): i for i in self.js("list", "rev", "--json")["issues"]}[("rev", 1)] + self.assertEqual((rev["public"], rev["waiting"], rev["approved"]), (True, False, True)) + self.cli("hide", "rev", "1") + hidden = {(i["repo"], i["n"]): i for i in self.js("list", "--json")["issues"]}[("rev", 1)] + self.assertEqual((hidden["hidden"], hidden["waiting"]), (True, False)) + + def test_list_json_new_reply_waits_in_public_mode_when_unauthenticated(self): + self.ingest(mail(mid="<a@test>")) + self.ingest(mail(subject="Re: Bug: crash on start", body="more", auth=("mx; dkim=fail",), + extra_headers=("In-Reply-To: <a@test>",))) + issue = self.js("list", "--json")["issues"][0] + self.assertEqual((issue["messages"], issue["shown"], issue["waiting"]), (2, 1, True)) + + def test_show_json_full_text_of_unpublished_report(self): + self.repo("rev", "public", "reviewed") + self.ingest(mail(repo="rev", mid="<a@test>", body="Version 1.2 (rules 43), commit abcdef1\nSeed 7, level 2, 30 turns taken, 3 lives, alive; modes: none\nSecret words\n"), + mail(repo="rev", subject="Second")) + data = self.js("show", "rev", "1", "--json") + self.assertEqual((data["mode"], data["public_repo"], data["public"], data["page_url"]), ("reviewed", True, False, None)) + self.assertIn("Secret words", data["messages"][0]["text"]) + self.assertEqual(data["messages"][0], {"label": "Reporter A", "date": data["messages"][0]["date"], "authenticated": True, + "approved": False, "shown": False, "text": data["messages"][0]["text"]}) + self.assertIn(["Commit", "abcdef1"], data["details"]) + self.assertEqual((data["prev"], data["next"]), (None, 2)) + self.assertEqual(self.js("show", "rev", "2", "--json")["prev"], 1) + self.cli("approve", "rev", "1") + data = self.js("show", "rev", "1", "--json") + self.assertEqual((data["public"], data["page_url"], data["messages"][0]["shown"]), (True, "https://code.fuzzyworld.net/rev/issues/1/", True)) + + def test_show_json_labels_and_watch_url(self): + d = os.path.join(self.apps, "demo") + os.makedirs(d) + open(os.path.join(d, "index.html"), "w").write("x") + json.dump({"app": "demo", "repo": "demo", "replay": "replay.html?report={record}"}, open(os.path.join(d, ".fuzzyworld.json"), "w")) + self.ingest(mail(mid="<a@test>", body="Boom.\nRun record: " + record() + "\n")) + self.ingest(mail(sender="Other <other@example.org>", subject="Re: x", body="me too", extra_headers=("In-Reply-To: <a@test>",))) + data = self.js("show", "demo", "1", "--json") + self.assertEqual([m["label"] for m in data["messages"]], ["Reporter A", "Reporter B"]) + self.assertTrue(data["record_shown"]) + self.assertIn("replay.html?report=", data["watch_url"]) + self.assertEqual(issues.replay_href("demo", 1), data["watch_url"]) + self.assertEqual(issues.replay_href("nothing", 1), "") + + def test_show_json_errors_and_plain_show_unchanged(self): + self.ingest(mail()) + self.assertEqual(self.cli("show", "demo", "9", "--json")[0], 1) + code, out, _ = self.cli("show", "demo", "1") + self.assertIn('"sender_hash"', out) # the plain command still prints the stored file + self.assertEqual(self.cli("show", "demo", "1", "--bogus")[0], 2) + + def test_details_rows_match_the_table(self): + text = "Version 1.2 (rules 43), commit abcdef1\nSeed 7, level 2, 30 turns taken, 3 lives, alive; modes: none" + self.assertEqual(dict(issues.details_rows(text))["Seed"], "7") + self.assertIn("<th>Seed</th><td>7</td>", issues.details_table(text)) + self.assertEqual(issues.details_rows("nothing"), []) + self.assertEqual(issues.details_table("nothing"), "") + + def test_record_prints_the_stored_file_even_when_unpublished(self): + self.repo("rev", "public", "reviewed") + self.ingest(mail(repo="rev", body="Boom.\nRun record: " + record() + "\n"), mail(repo="rev", subject="No record")) + code, out, _ = self.cli("record", "rev", "1") + self.assertEqual(code, 0) + self.assertEqual(out, rd(os.path.join(self.state, "repos", "rev", "1", "record.json"))) + self.assertFalse(os.path.exists(self.page("rev"))) + code, out, err = self.cli("record", "rev", "2") + self.assertEqual((code, out), (1, "")) + self.assertIn("no record for rev#2", err) + + class ImapParameters(unittest.TestCase): def test_defaults_are_the_comments_values_and_issues_can_override(self): c = issues.comments.IMAPSource()