fuzzyworld-tools / commit
Blog comments by email, with moderation CLI (and a disabled web-form endpoint)
PLAN.md | 31 +- README.md | 35 ++ bin/fw | 5 +- server/blog.py | 68 ++- server/comments.py | 1008 ++++++++++++++++++++++++++++++++++++++++ server/comments/post.php | 356 ++++++++++++++ server/comments/sites.json | 6 + server/fw | 4 +- server/jobs/comments-ingest.sh | 9 + server/jobs/crontab | 3 +- server/site-blog/blog.css | 4 + server/site-blog/blog.js | 65 +++ server/site-blog/htaccess | 8 + tests/test_blog_comments.py | 124 +++++ tests/test_comments.py | 972 ++++++++++++++++++++++++++++++++++++++ tests/test_comments_php.py | 423 +++++++++++++++++ 16 files changed, 3101 insertions(+), 20 deletions(-) diff --git a/PLAN.md b/PLAN.md index 6e76a03..ee4510a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -106,7 +106,7 @@ WordPress text wins when a source file differs (similarity under 0.95: 100 of th into `~/Blog/wordpress/`); the old site's custom CSS is carried over (hidden `<br>`, dotted scene breaks, underlined chapter headings); a light/dark switch and reading-progress bar added (they needed `script-src 'self'`). The Atom writer exists now, so Next 1 (Feeds) can reuse it. **Done 2026-11-01 (Q5/Q6):** `blog.fuzzyworld.net` 301-redirects every path (query kept) to `writing.`; `til.fuzzyworld.net` answers 410 Gone. Both WordPress folders stay on the server, unserved; the original `.htaccess` of each is saved as `.htaccess.wp-20261101`. Backup: a WordPress export of the blog (Pavel, `~/Downloads`). The ancient `old.blog.fuzzyworld.net` copy was moved to `~/old/` on the server (it was not served). **Still to do:** building from the push hook after the move to a private repo, -the `til` blog (Q14, same importer with `--wp https://til.fuzzyworld.net`). Below, the plan as written before the +the `til` blog: **built 2026-11-01 (Q14)**, config `~/til/fw-blog.json` (4 posts, WordPress slugs kept, one image in `~/til/media/`), same generator; uploaded the same day (the WordPress folder was moved to `~/old/` on the server first, because `fw blog deploy` runs rsync `--delete`). Below, the plan as written before the build: it still says `blog.fuzzyworld.net` in places; read that as `writing.fuzzyworld.net`. **What is there** (read from its public API, 2026-10-28): @@ -156,16 +156,25 @@ basic auth first; client-side encryption only if needed (it adds a script and al ## Comments -Decided 2026-10-28: **by email first, no accounts.** Google sign-in is a 🔌 plugin for later. -- A "comment" link on each post is a `mailto:` to the comments mailbox, with the post in the subject. Same plumbing as - [Issues](#issues): a cron job reads the mailbox and renders threads as static pages. -- Identity is the sender's verified address (DKIM/DMARC must pass), which proves they control it, so no login. Only the - display name is shown, never the address. -- Moderation, light: a person's first comment waits for my approval; after that they are trusted and published on - arrival. One command removes a comment. Length and rate limits apply. A deletion request is honoured. -- **🔌 Google sign-in plugin:** needs an endpoint that checks the Google token and stores the comment (a small CGI on - DreamHost or a Cloudflare Worker), and a redirect flow so the site's CSP (`script-src 'self'`) stays. It breaks - "DreamHost only serves files", which is why it is a plugin. +Decided 2026-10-28: by email, no accounts. Revised 2026-11-01: a web form (`server/comments/post.php`, written, never run +or installed) was designed too, but **only the +email path is enabled** (Pavel: a form means code that runs on every web request, which is more attack surface than +he wants; see the tiers in the README, form = T2, email = T1). The form stays in the repo, off by default. +- A "comment by email" link on each post is a `mailto:` to the comments mailbox, with the post in the subject + (`Comment [writing:/2024/05/01/slug/]`). A cron job (`server/comments.py ingest`) reads the comments mailbox, over IMAP + (a hosted DreamHost mailbox; mail arrives through MailChannels) or from a local Maildir, and puts each comment in a pending queue. +- Identity is the sender's verified address (the topmost Authentication-Results header must show DMARC pass, or DKIM + pass for the From domain). The address is never shown or stored, only a hash; only the display name is shown. +- Moderation: a person's first comment waits for my approval (`fw comments list|approve|reject|remove|untrust`); + with `approve --trust` that sender is published on arrival from then on. Limits: 4000 characters, 5 per sender per 24 hours, 200 pending. + A deletion request is honoured. +- Approved comments are written as one static JSON file per post (`/_comments/<post path>.json`) which `blog.js` + renders with `textContent`, so a comment appears without a rebuild. `fw blog deploy` excludes `/_comments/`. + Imported WordPress comments stay baked into the pages. Readers without JavaScript see only the imported ones. +- **The form (off):** `server/comments/post.php` on its own subdomain, `comments_form` in the blog config; every + form comment is moderated. Enabling it is a decision for later; the upsides are no email client needed, the + downsides are a public PHP endpoint, spam, and unverified identities. +- **🔌 Google sign-in plugin:** would need an endpoint that checks the Google token (T2), so it stays a plugin. ## CI dashboard diff --git a/README.md b/README.md index 5b0ce7d..d9c1a8f 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,22 @@ only when I choose to. The same text is on [/why-i-left-gitlab/](https://code.fuzzyworld.net/why-i-left-gitlab/) (formerly /why/ and /principles/), and in the `/how/` draft (not live yet). +## Hosting tiers + +Each feature needs a certain kind of host. The tiers say what that is, so you can pick what your hosting allows. +A tier includes the ones above it. + +| Tier | The host must offer | Features | +| --- | --- | --- | +| **T0, files only** (ftp only) | serving static files | the built blog, Atom feeds, the landing page, the generated code and CI pages, VS Code in the browser | +| **T1, files + jobs** (ftp + python) | ssh, cron, git hooks, a mailbox reachable by IMAP or on the same host; nothing runs when a page is requested | page generation on push, the CI queue and runner API, cleanup jobs, comments by email, the `fw` commands | +| **T2, files + an endpoint** (ftp + php/cgi) | a script that answers web requests | the comment form (`post.php`: written, **not enabled, not installed**) | +| **T3, always-on process** | a long-running server | nothing; out of scope | + +T2 is new attack surface: code that runs on a web request, as the same user as everything else on the account. +The comment form is written but switched off and not installed; comments are accepted by email only (T1). To use the form you would +host `server/comments/post.php` on its own subdomain and set `comments_form` in the blog config. + ## Making a repo deploy as an app Add to the app repo's `package.json`: @@ -73,6 +89,7 @@ On the laptop, `bin/fw` runs `deploy` locally and forwards repo commands to Drea | `fw deploy [--ref B] [--as NAME] [--here] [--dry-run]` | build and publish this repo's app (above). `--as NAME` publishes it at `apps.fuzzyworld.net/NAME/` instead, a second copy of the same repo (e.g. `explodorama-next` built from `main` while `/explodorama/` is deployed from a tag with `--ref`); the build gets `FW_APP=NAME`, which the vite plugin uses for `base`, and the index title gets the suffix | | `fw repoint [DIR] [--https] [--dry-run]` | for each repo in DIR whose `origin` is on gitlab.com: rename it `gitlab`, add `origin` → `dh:git/<repo>.git` (or read-only `https://code.fuzzyworld.net/<repo>.git` with `--https`, public repos only), fetch, and retarget branch tracking. Changes only remotes; checks that uncommitted work, HEAD and stashes are unchanged | | `fw usage [--all\|--now]` | Claude Code token use per session from `~/.claude/projects` (runs locally): messages, current context size, cache-read, output (cache-read and output include subagents). `--now` prints one line for the newest session, to decide when to put it down | +| `fw comments list\|show\|approve\|reject\|remove\|untrust\|publish\|export\|ingest\|imap-check` | moderate blog comments received by email (see [Blog](#blog)); runs on the server | | `fw list` | every repo, its visibility and last commit | | `fw new <name> [--public] [--description TEXT]` | create `~/git/<name>.git` on DreamHost; **private unless `--public`** | | `fw public <name>` | publish the source on code.fuzzyworld.net right away | @@ -311,6 +328,24 @@ fw blog deploy ~/Blog/fw-blog.json --dry-run # then, for real: --yes (or answer `robots.txt` that blocks AI crawlers, comments as static text. - **Reader features** (`server/site-blog/blog.js`): a light/dark switch (the choice is kept in the browser; without one the system setting decides) and a reading-progress bar on posts. +- **Comments by email:** set `"comments_site": "writing"` (or `"til"`, a key of `server/comments/sites.json`) and + `"comments_mailbox": "<address>"` in the config: each post then gets a "Comment by email" link (subject + `Comment [writing:/2024/05/01/slug/]`) and a container that `blog.js` fills from `/_comments/<post path>.json`. + Comments are moderated on the server with `fw comments list|show|approve [--trust]|reject|remove|untrust|publish|export|ingest|imap-check`; + a cron job runs `ingest` hourly. `ingest` reads the comments mailbox: a hosted mailbox over IMAP when `~/fw/comments/imap.json` + exists (`{"host", "user", "password", "folder"}`, mode 600, created by hand on the server; TLS with certificate + check; processed mail moves to the mailbox folders `comments-done` and `comments-bad`, with the server's folder prefix such as `INBOX.` added when it needs one; `done_folder` and `bad_folder` in the file override the names), otherwise a Maildir folder + `~/Maildir/.comments`. `fw comments imap-check` prints only header names and the Authentication-Results result tokens, to + learn the right `authserv_id`. `ingest` accepts a message only when the topmost + `Authentication-Results` header, written by the receiving mail server (its name must be set as `authserv_id` in + `server/comments/sites.json`; empty means ingest refuses to read anything; set it in the repo copy, because an install + overwrites the server's), shows a DMARC pass for the From domain (or an aligned DKIM pass). That trust holds only if the + receiving server always adds its own header above anything the sender supplied. The address is never shown, only a + salted hash is kept (losing `~/fw/comments/secret` invalidates trusted senders). Limits: 4000 characters, 5 comments + per sender per 24 hours, 200 pending. State lives in `~/fw/comments/`, outside every web root; `fw blog deploy` + leaves `/_comments/` alone. `comments_form` (an `https` action) adds a plain HTML form for the optional comment + endpoint `server/comments/post.php`; it is off, and the endpoint is written (its tests pass on the server's PHP; an optional `self_origin` key in `sites.json` names the origin its no-JavaScript confirm page posts from) but is not installed (see + [Hosting tiers](#hosting-tiers)). - **Upload safety:** `deploy` audits the built folder first and refuses anything that is not a generated file or an allowlisted media type (`png jpg jpeg gif webp svg pdf epub kpf mp3 mp4 zip`), then rsyncs only that folder (it keeps `.well-known`). It asks for confirmation on every real upload; `--dry-run` shows what would be sent. diff --git a/bin/fw b/bin/fw index 96c11b5..844d0c8 100755 --- a/bin/fw +++ b/bin/fw @@ -10,6 +10,7 @@ # import the WordPress blog (see lib/wp_import.py) # fw blog check <config> [--out DIR] compare the built site with the cached WordPress text # fw describe <name> TEXT | publish <name> | publish-all | ci <name> [branch] | ci-list | ci-watch +# fw comments list|show|approve|reject|remove|untrust|publish|export|ingest|imap-check (blog comments) # (run on DreamHost over ssh) set -euo pipefail HOST=${FW_HOST:-dh} @@ -65,7 +66,7 @@ blog() { target=$(python3 "$blogpy" deploy-target "$config") python3 "$blogpy" build "$config" --out "$out" python3 "$blogpy" audit "$out" - local rsync_opts=(-rlptz --delete --exclude .well-known --exclude .dh-diag --exclude .fw-blog-out) + local rsync_opts=(-rlptz --delete --exclude .well-known --exclude .dh-diag --exclude .fw-blog-out --exclude /_comments/) if [ "$dry" = 1 ]; then exec rsync -n -v "${rsync_opts[@]}" "$out/" "$target/" fi @@ -87,7 +88,7 @@ case "${1:-help}" in shift; exec python3 "$HERE/lib/usage.py" "$@" ;; repoint) shift; exec bash "$HERE/lib/repoint.sh" "$@" ;; - list|new|public|private|describe|publish|publish-all|ci|ci-list|ci-watch|help|-h|--help) + list|new|public|private|describe|publish|publish-all|ci|ci-list|ci-watch|comments|help|-h|--help) # printf %q keeps arguments (e.g. a description with spaces) intact through ssh. exec ssh "$HOST" "~/bin/fw $(printf '%q ' "$@")" ;; *) diff --git a/server/blog.py b/server/blog.py index 13afe6c..70e53d4 100644 --- a/server/blog.py +++ b/server/blog.py @@ -63,13 +63,15 @@ POST_FIELDS = ("published", "date", "slug", "title", "tags", "body_until", "wp_i PAGE_FIELDS = ("published", "slug", "title") CONFIG_KEYS = {"title", "tagline", "base_url", "author", "copyright", "source", "include", "body_section", "line_breaks", "url", "home", "pages_dir", "nav", "tags", "feed", "reading_wpm", "media", - "rewrite_hosts", "redirects", "wp_ids", "comments", "deploy", "posts", "timezone", "site"} + "rewrite_hosts", "redirects", "wp_ids", "comments", "deploy", "posts", "timezone", "site", + "comments_site", "comments_mailbox", "comments_form"} REQUIRED_KEYS = ("title", "base_url", "source") DEFAULTS = {"tagline": "", "author": "", "copyright": "© {years} {author}. All rights reserved.", "include": ["*.md"], "body_section": "Writing", "line_breaks": "markdown", "url": DEFAULT_URL, "home": None, "pages_dir": None, "nav": [], "tags": {}, "feed": {"path": "/feed/", "entries": 20}, "reading_wpm": 200, "media": None, "rewrite_hosts": [], "redirects": {}, "wp_ids": {}, - "comments": None, "deploy": None, "posts": {}, "timezone": "UTC", "site": None} + "comments": None, "deploy": None, "posts": {}, "timezone": "UTC", "site": None, + "comments_site": None, "comments_mailbox": None, "comments_form": None} class BlogError(Exception): @@ -147,11 +149,31 @@ def load_config(path): raise BlogError("tags must map a slug to a name") for slug in cfg["tags"]: check_slug(slug, "tags") + check_comments_config(cfg) cfg["_dir"] = os.path.dirname(path) cfg["_source"] = os.path.normpath(os.path.join(cfg["_dir"], cfg["source"])) return cfg +def check_comments_config(cfg): + """Reader comments by email (and, off by default, a web form). Off unless comments_site is set.""" + site, mailbox, form = cfg["comments_site"], cfg["comments_mailbox"], cfg["comments_form"] + if site is None: + if mailbox is not None or form is not None: + raise BlogError("comments_mailbox and comments_form need comments_site") + return + if not (isinstance(site, str) and re.fullmatch(r"[A-Za-z0-9_-]+", site)): + raise BlogError('comments_site must be a short name such as "writing" (letters, digits, - and _)') + if not (isinstance(mailbox, str) and re.fullmatch(r"[^@\s<>\"',;:()\[\]\\]+@[^@\s<>\"',;:()\[\]\\]+\.[^@\s<>\"',;:()\[\]\\]+", mailbox)): + raise BlogError("comments_site needs comments_mailbox: the email address that receives comments") + if form is not None: + action = form.get("action") if isinstance(form, dict) else None + parts = urllib.parse.urlsplit(action) if isinstance(action, str) else None + if not (isinstance(form, dict) and set(form) == {"action"} and parts and parts.scheme == "https" and parts.netloc + and not re.search(r"[\s\x00-\x1f\x7f\"'<>]", action)): + raise BlogError('comments_form must be null or {"action": "https://..."} (an https URL without spaces or quotes)') + + def check_slug(slug, what): """The slug, or BlogError: a slug becomes a folder name and part of a URL (and of .htaccess), so it is ASCII letters, digits and . _ % - only, and it must not decode to `.`, `..` or something with a slash.""" @@ -609,8 +631,35 @@ def absolutize(html_text, base_url): return map_links(html_text, lambda u: base_url + u if u.startswith("/") and not u.startswith("//") else u) -def render_comments(comments): - """Threaded comments: name, date and text only. Everything is escaped; newlines become <br>. Replies nest.""" +def comments_src(path): + """Where the server publishes a post's comments: /2024/05/01/slug/ -> /_comments/2024/05/01/slug.json""" + return "/_comments/" + path.strip("/") + ".json" + + +def comments_widget(cfg, path): + """The container blog.js fills, the "Comment by email" link and (only with comments_form) the web form.""" + site = cfg["comments_site"] + subject = urllib.parse.quote(f"Comment [{site}:{path}]", safe="") + mailto = f"mailto:{urllib.parse.quote(cfg['comments_mailbox'], safe='@')}?subject={subject}" + html_ = ('<div class="comments-live"></div>' + f'<p class="comment-by-email"><a href="{esc(mailto, quote=True)}">Comment by email</a>. ' + 'Comments are moderated, and your address is never shown.</p>') + form = cfg["comments_form"] + if form: + html_ += (f'<form class="comment-form" method="post" action="{esc(form["action"], quote=True)}">' + f'<input type="hidden" name="site" value="{esc(site, quote=True)}">' + f'<input type="hidden" name="path" value="{esc(path, quote=True)}">' + '<input type="hidden" name="elapsed" value="0">' + '<div class="hp" aria-hidden="true"><input type="text" name="website" tabindex="-1" autocomplete="off"></div>' + '<p><label>Name <input type="text" name="name" maxlength="60" required></label></p>' + '<p><label>Comment<br><textarea name="text" rows="5" maxlength="4000" required></textarea></label></p>' + '<p><button type="submit">Send comment</button></p></form>') + return html_ + + +def render_comments(comments, widget=None, src=None): + """Threaded comments: name, date and text only. Everything is escaped; newlines become <br>. Replies nest. + With `src` (comments_site is set) the section is always written: it carries data-src, and `widget` follows the list.""" ids = {c.get("id") for c in comments} children = {} for c in sorted(comments, key=lambda c: str(c.get("date", ""))): @@ -627,12 +676,21 @@ def render_comments(comments): f'{esc(str(c.get("date", ""))[:10])}</p><p>{text}</p>{nested}</li>') return f'<ul class="comments">{"".join(items)}</ul>' if items else "" + if src: + return (f'<section class="comments-section" data-src="{esc(src, quote=True)}"><h2>Comments</h2>' + tree(0) + + (widget or "") + '<p class="meta">Comments © their authors.</p></section>') if not comments: return "" return ('<section class="comments-section"><h2>Comments</h2>' + tree(0) + '<p class="meta">Comments © their authors.</p></section>') +def post_comments(cfg, comments, post): + if not cfg["comments_site"]: + return render_comments(comments.get(post.path, [])) + return render_comments(comments.get(post.path, []), comments_widget(cfg, post.path), comments_src(post.path)) + + def load_comments(cfg): if not cfg["comments"]: return {} @@ -886,7 +944,7 @@ def build(cfg, out, htaccess=True): for t in p.tags: tags.setdefault(tag_slug(cfg, t), []).append(it) page_body = (f'<article class="post"><h1>{esc(p.title)}</h1>{post_meta_line(cfg, p, minutes)}' - f'<div class="post-body">{body}</div></article>{render_comments(comments.get(p.path, []))}') + f'<div class="post-body">{body}</div></article>{post_comments(cfg, comments, p)}') emit(p.path, blog_page(cfg, p.title, page_body, p.path)) years = sorted({it["post"].date.year for it in items}, reverse=True) diff --git a/server/comments.py b/server/comments.py new file mode 100644 index 0000000..0946b1a --- /dev/null +++ b/server/comments.py @@ -0,0 +1,1008 @@ +#!/usr/bin/env python3 +"""Blog comments by email, with a moderation CLI. Python standard library only. + +Usage (on the server: `fw comments <cmd>`; from the laptop: `bin/fw comments <cmd>`): + + list pending comments (id, site, path, author, source, first 160 chars) + show <id> one pending or approved comment in full + approve <id>... [--trust] publish pending comments; --trust also trusts the sender (email only) + reject <id>... discard pending comments (logged in rejected.log) + remove <id> delete an approved comment and republish its post + untrust <id> stop auto-approving the sender of that comment + publish [site] rewrite every per-post file from the approved store + export <site> print the approved store (path -> list of comments) as JSON + ingest read new mail from the comments mailbox (IMAP if imap.json exists, else Maildir) + imap-check connect to the IMAP mailbox and print what it holds (header names only) + +State, outside every web root (override the directory with FW_COMMENTS_DIR): + ~/fw/comments/pending/<id>.json waiting for moderation (may hold sender_hash, never an address) + ~/fw/comments/approved/<site>.json {path: [{author,date,id,parent,text}]}, public fields only + ~/fw/comments/approved-meta.json id -> sender_hash, so `untrust` works after approval; never published + ~/fw/comments/approved-keys.json message key (sha256 of the raw mail) -> comment id, so a mail whose comment is + already stored is never added twice; pruned on every ingest + ~/fw/comments/secret random salt for sender hashes (mode 600). Back it up: without it every + trusted.json entry and every untrust record stops matching. + ~/fw/comments/trusted.json salted sha256 (secret + lowercased sender address) hashes + ~/fw/comments/rejected.log "id date reason" per rejection (rotated to rejected.log.1 at 1 MB) + ~/fw/comments/lock held for every change to the store (taken per message by ingest) + ~/fw/comments/ingest.lock held for a whole ingest run, so a second run waits for the first +Published: <site root>/_comments/<post path minus trailing slash>.json, e.g. + /2024/05/01/slug/ -> <root>/_comments/2024/05/01/slug.json (removed when the list is empty). + +Sites: server/comments/sites.json (override the path with FW_COMMENTS_SITES). Keys "writing" and +"til" give origin and root; "mailbox" is the comments address (informational; may be empty); +"authserv_id" is the authserv-id of the receiving mail server (see below). + +Email intake refuses to run until "authserv_id" is set in sites.json (fail closed). Maildir: +~/Maildir/.comments (override with FW_COMMENTS_MAILDIR); processed mail moves to ~/Maildir/.comments.done, +rejected mail to ~/Maildir/.comments.bad. The subject must +contain [writing:/path/] or [til:/path/] ("Re:" prefixes are fine). Authentication uses ONLY the +topmost Authentication-Results header, and only if its authserv-id equals the configured "authserv_id" +(the id your mail provider puts in front of the first ";" of a real message). Headers lower in the +message may have been added by the sender, so they are never searched: a missing or foreign topmost +header rejects the message. The message needs exactly one From header with one address. Accepted when +every dmarc result passes and its header.from equals the From domain, or every dkim result passes and +one has header.d equal to, or a subdomain of, the From domain. Each sender (salted hash) may have at +most 5 pending or approved comments per 24 hours (trusted senders too); the rest goes to .bad. +The "topmost header" rule only holds if the receiving mail server always adds its own +Authentication-Results header above anything the sender supplied; check that with a real message. +Only stored comments are remembered: a rejected mail moved back from .bad is simply tried again. +Ingest does nothing (quietly) when the mail source does not exist. The message source is a small class +(MaildirSource, IMAPSource) with available(), keys(), read(key), finish(key, raw, "done"|"bad") and close(). + +IMAP source. When ~/fw/comments/imap.json exists (override the path with FW_COMMENTS_IMAP) ingest reads that +mailbox instead of the Maildir: {"host": "imap.dreamhost.com", "user": "...", "password": "...", "folder": +"INBOX"} (folder is optional; "done_folder" / "bad_folder" optionally override the target folders). The file must be mode 600; a group/world-readable file, a missing key or an empty +password is refused with an error. The connection is IMAP4_SSL on port 993 with certificate verification on and a +30 s timeout, and always logs out. The password is never printed or logged. Messages are listed by UID (with the +folder's UIDVALIDITY), fetched with BODY.PEEK[], and over 256 KB are moved to the bad folder without being +downloaded. Only undeleted messages are listed. Finished mail is copied to <prefix>comments-done / +<prefix>comments-bad (created when missing; the prefix, e.g. "INBOX.", comes from the server's NAMESPACE or LIST +delimiter), then flagged \\Deleted and expunged (UID EXPUNGE when the server's CAPABILITY, asked after login, lists UIDPLUS). A message the server +will not fetch (SkipMessage) is logged and skipped; it stays in the mailbox, is logged again on every run, and +must be removed by hand. Ingest holds a job-wide lock (ingest.lock), so overlapping runs +queue up. `comments.py imap-check` prints the message count, the folder names and NAMESPACE response, and, for +the 5 newest, only the header names plus the authserv-id and method=result tokens of each Authentication-Results +header (no bodies, subjects or addresses), to learn what the provider writes. +Post paths are stored in one canonical spelling (decoded, then re-escaped with upper-case hex), so /sl%75g/ and +/slug/ are the same post; publishing merges any spellings that map to the same folder. +""" + +import hashlib +import imaplib +import json +import mailbox +import os +import re +import secrets +import ssl +import sys +import tempfile +import time +import unicodedata +from urllib.parse import quote, unquote +from contextlib import contextmanager +from email import message_from_bytes, policy +from email.header import decode_header, make_header +from email.utils import getaddresses + +try: + import fcntl +except ImportError: # not on Windows servers; only the lock is lost + fcntl = None + +# Every URL form blog.py makes: safe segments (a url template may start a segment with _ or ~), trailing slash. +PATH_RE = re.compile(r"^/([A-Za-z0-9_~][A-Za-z0-9._%~+-]*/){1,8}$") +ID_RE = re.compile(r"^\d{8}T\d{6}Z-[0-9a-f]{6}$") +PUBLIC_FIELDS = ("author", "date", "id", "parent", "text") +PENDING_CAP = 200 +SENDER_LIMIT = 5 # pending or approved comments per sender per SENDER_WINDOW seconds +SENDER_WINDOW = 24 * 3600 +MAX_TEXT = 4000 +MAX_NAME = 60 + + +class CommentError(Exception): + pass + + +# ---- configuration and paths ---- + +def state_dir(): + return os.path.expanduser(os.environ.get("FW_COMMENTS_DIR") or "~/fw/comments") + + +def load_config(): + path = os.environ.get("FW_COMMENTS_SITES") or os.path.join(os.path.dirname(os.path.abspath(__file__)), "comments", "sites.json") + try: + with open(path) as f: + return json.load(f) + except (ValueError, OSError) as e: + raise CommentError(f"cannot read {path}: {e}") + + +def site_root(site): + cfg = load_config().get(site) + if not isinstance(cfg, dict) or "root" not in cfg: + raise CommentError(f"unknown site: {site}") + return os.path.expanduser(cfg["root"]) + + +def site_names(): + return sorted(k for k, v in load_config().items() if isinstance(v, dict) and "root" in v) + + +def valid_path(path): + if not (isinstance(path, str) and len(path) <= 300 and PATH_RE.match(path) and ".." not in path): + return False + for seg in path.strip("/").split("/"): # as the web server and blog.py decode it: no ".", "..", separator, control + decoded = unquote(seg) + if (decoded in (".", "..") or ".." in decoded or "\ufffd" in decoded or "/" in decoded or "\\" in decoded + or any(unicodedata.category(ch) in ("Cc", "Cf") for ch in decoded)): + return False + return True + + +def canonical_path(path): + """One spelling per post: decoded, then re-escaped (upper-case hex, only what must be escaped).""" + check_path(path) + canon = "/" + "/".join(quote(unquote(seg), safe="~+") for seg in path.strip("/").split("/")) + "/" + check_path(canon) + return canon + + +def fs_path(path): + """The folder a post path maps to under a site root (the web server decodes %xx first).""" + return unquote(path.strip("/")) + + +def check_path(path): + if not valid_path(path): + raise CommentError(f"invalid post path: {path!r}") + + +def check_id(cid): + if not ID_RE.match(cid): + raise CommentError(f"invalid comment id: {cid!r}") + + +def now_iso(): + return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + + +def new_id(): + return time.strftime("%Y%m%dT%H%M%SZ", time.gmtime()) + "-" + secrets.token_hex(3) + + +def secret(): + """A random per-installation secret, created once; it salts the sender hashes.""" + path = os.path.join(state_dir(), "secret") + try: + with open(path) as f: + value = f.read().strip() + except FileNotFoundError: + value = "" + if not value: + value = secrets.token_hex(32) + write_atomic(path, value + "\n") + return value + + +def sender_hash(address): + return hashlib.sha256((secret() + address.strip().lower()).encode()).hexdigest() + + +# ---- store ---- + +def write_atomic(path, data): + os.makedirs(os.path.dirname(path), exist_ok=True) + fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path), prefix=".tmp-") + try: + with os.fdopen(fd, "w") as f: + f.write(data) + f.flush() + os.fsync(f.fileno()) + os.chmod(tmp, 0o644 if "_comments" in path.split(os.sep) else 0o600) + os.replace(tmp, path) + try: + dfd = os.open(os.path.dirname(path), os.O_RDONLY) + try: + os.fsync(dfd) + finally: + os.close(dfd) + except OSError: + pass + except BaseException: + if os.path.exists(tmp): + os.unlink(tmp) + raise + + +def write_json(path, obj): + write_atomic(path, json.dumps(obj, indent=2, ensure_ascii=False, sort_keys=True) + "\n") + + +def read_json(path, default): + try: + with open(path) as f: + return json.load(f) + except FileNotFoundError: + return default + except (ValueError, OSError) as e: # empty or damaged file: say which one + raise CommentError(f"cannot read {path}: {e}") + + +@contextmanager +def ingest_lock(): + """Job-wide: one ingest at a time (a second one waits), so its snapshot of the store is never stale.""" + os.makedirs(state_dir(), exist_ok=True) + with open(os.path.join(state_dir(), "ingest.lock"), "w") as f: + if fcntl: + fcntl.flock(f, fcntl.LOCK_EX) + yield + + +@contextmanager +def locked(): + os.makedirs(state_dir(), exist_ok=True) + with open(os.path.join(state_dir(), "lock"), "w") as f: + if fcntl: + fcntl.flock(f, fcntl.LOCK_EX) + yield + + +def pending_path(cid): + check_id(cid) + return os.path.join(state_dir(), "pending", cid + ".json") + + +def pending_ids(): + d = os.path.join(state_dir(), "pending") + return sorted(n[:-5] for n in os.listdir(d) if n.endswith(".json") and not n.startswith(".")) if os.path.isdir(d) else [] + + +def load_pending(cid): + c = read_json(pending_path(cid), None) + if c is None: + raise CommentError(f"no pending comment {cid}") + return c + + +def approved_path(site): + site_root(site) # validates the site name + return os.path.join(state_dir(), "approved", site + ".json") + + +def load_approved(site): + return read_json(approved_path(site), {}) + + +def meta_path(): + return os.path.join(state_dir(), "approved-meta.json") + + +def keys_path(): + return os.path.join(state_dir(), "approved-keys.json") + + +def trusted_path(): + return os.path.join(state_dir(), "trusted.json") + + +def log_rejection(cid, reason): + os.makedirs(state_dir(), exist_ok=True) + reason = clean(reason).replace("\n", " ") + log = os.path.join(state_dir(), "rejected.log") + try: + if os.path.getsize(log) > 1_000_000: + os.replace(log, log + ".1") + except FileNotFoundError: + pass + with open(log, "a") as f: + f.write(f"{cid} {now_iso()} {reason}\n") + + +def public(c): + return {k: c.get(k) for k in PUBLIC_FIELDS} + + +# ---- publishing ---- + +def post_file(site, path): + check_path(path) + base = os.path.realpath(os.path.join(site_root(site), "_comments")) + target = os.path.normpath(os.path.join(base, fs_path(path) + ".json")) + if not target.startswith(base + os.sep): + raise CommentError(f"path escapes the comments directory: {path!r}") + return target + + +_warned = set() + + +def publish_path(site, path, approved): + target = post_file(site, path) + if not os.path.isdir(site_root(site)): # never create a site root; `publish` writes the files later + if site not in _warned: + _warned.add(site) + print(f"comments: root of site {site} does not exist; not publishing (run `publish` later)", file=sys.stderr) + return + folder = fs_path(path) # every spelling of this post, merged, oldest first + items = sorted((public(c) for key, cs in approved.items() if valid_path(key) and fs_path(key) == folder for c in cs), + key=lambda c: (c.get("date") or "", c.get("id") or "")) + if items: + write_json(target, items) + elif os.path.exists(target): + os.unlink(target) + + +def publish_site(site): + approved = load_approved(site) + done = set() + for path in sorted(approved): + if valid_path(path) and fs_path(path) not in done: + done.add(fs_path(path)) + publish_path(site, path, approved) + return len(done) + + +# ---- moderation ---- + +def add_approved(c): + """Store the comment, then publish it; a publish failure only warns (a later `publish` fixes the files).""" + site, path = c["site"], canonical_path(c["path"]) + c = {**c, "path": path} + approved = load_approved(site) + approved.setdefault(path, []).append(public(c)) + write_json(approved_path(site), approved) + if c.get("sender_hash"): + meta = read_json(meta_path(), {}) + meta[c["id"]] = c["sender_hash"] + write_json(meta_path(), meta) + if c.get("msg_key"): + keys = read_json(keys_path(), {}) + keys[c["msg_key"]] = c["id"] + write_json(keys_path(), keys) + try: + publish_path(site, path, approved) + except OSError as e: + print(f"comments: could not publish {site}:{path} ({e}); run `publish` later", file=sys.stderr) + + +def trust(h): + t = read_json(trusted_path(), []) + if h not in t: + write_json(trusted_path(), t + [h]) + + +def cmd_list(): + for cid in pending_ids(): + c = read_json(pending_path(cid), {}) + text = " ".join(c.get("text", "").split())[:160] + print(f"{cid} {c.get('site')}:{c.get('path')} {c.get('author')} [{c.get('source')}] {text}") + + +def find_approved(cid): + for site in site_names(): + for path, items in load_approved(site).items(): + for c in items: + if c.get("id") == cid: + return site, path, c + return None + + +def cmd_show(cid): + check_id(cid) + p = read_json(pending_path(cid), None) + if p is not None: + print(json.dumps(p, indent=2, ensure_ascii=False)) + return + found = find_approved(cid) + if not found: + raise CommentError(f"no comment {cid}") + print(json.dumps({"site": found[0], "path": found[1], **found[2]}, indent=2, ensure_ascii=False)) + + +def cmd_approve(ids, with_trust): + with locked(): + for cid in ids: + c = load_pending(cid) + if with_trust: + if c.get("source") != "email" or not c.get("sender_hash"): + raise CommentError(f"{cid}: --trust works only for email comments") + trust(c["sender_hash"]) + add_approved(c) + os.unlink(pending_path(cid)) + print(f"approved {cid}") + + +def cmd_reject(ids): + with locked(): + for cid in ids: + c = load_pending(cid) + if c.get("sender_hash"): + meta = read_json(meta_path(), {}) + meta[cid] = c["sender_hash"] + write_json(meta_path(), meta) + log_rejection(cid, "rejected by moderator") + os.unlink(pending_path(cid)) + print(f"rejected {cid}") + + +def cmd_remove(cid): + check_id(cid) + with locked(): + found = find_approved(cid) + if not found: + raise CommentError(f"no approved comment {cid}") + site, path, _ = found + approved = load_approved(site) + approved[path] = [c for c in approved[path] if c.get("id") != cid] + if not approved[path]: + del approved[path] + write_json(approved_path(site), approved) + publish_path(site, path, approved) + print(f"removed {cid}") + + +def cmd_untrust(cid): + check_id(cid) + with locked(): + h = (read_json(pending_path(cid), None) or {}).get("sender_hash") or read_json(meta_path(), {}).get(cid) + if not h: + raise CommentError(f"no sender recorded for {cid}") + t = read_json(trusted_path(), []) + if h in t: + write_json(trusted_path(), [x for x in t if x != h]) + print("untrusted") + else: + print("sender was not trusted") + + +def cmd_publish(site=None): + with locked(): + for s in ([site] if site else site_names()): + print(f"{s}: {publish_site(s)} post file(s)") + + +def cmd_export(site): + print(json.dumps(load_approved(site), indent=2, ensure_ascii=False, sort_keys=True)) + + +# ---- email intake ---- + +def clean(s): + """Drop control and format characters (bidi, zero-width, ...) and line/paragraph separators; keep newlines.""" + s = s.replace("\r\n", "\n").replace("\r", "\n") + return "".join(ch for ch in s + if ch == "\n" or (unicodedata.category(ch) not in ("Cc", "Cf") and ch not in "\u2028\u2029")) + + +def split_ar(text): + """Split an Authentication-Results value on ';' outside "quoted strings", dropping (comments).""" + parts, cur, quote, depth, esc = [], [], False, 0, False + for ch in text: + if esc: + esc = False + if depth == 0: + cur.append(ch) + elif ch == "\\": + esc = True + if depth == 0: + cur.append(ch) + elif quote: + cur.append(ch) + quote = ch != '"' + elif depth: + depth += (ch == "(") - (ch == ")") + elif ch == "(": + depth = 1 + elif ch == '"': + quote = True + cur.append(ch) + elif ch == ";": + parts.append("".join(cur)) + cur = [] + else: + cur.append(ch) + parts.append("".join(cur)) + return parts + + +def parse_ar(header): + """(authserv-id, {method: [(result, props)]}) of one Authentication-Results header.""" + segs = split_ar(str(header)) + head = segs[0].split() + by_method = {} + for seg in segs[1:]: + m = re.match(r"\s*([A-Za-z0-9_-]+)\s*=\s*([A-Za-z]+)(?![A-Za-z0-9])(.*)$", seg, re.S) + if not m: + continue + props = {k.lower(): v.strip('"').lower() for k, v in + re.findall(r'([A-Za-z0-9_.-]+)\s*=\s*("(?:[^"\\]|\\.)*"|[^\s"]+)', m.group(3))} + by_method.setdefault(m.group(1).lower(), []).append((m.group(2).lower(), props)) + return (head[0].lower() if head else ""), by_method + + +def authenticated(msg, from_domain, authserv_id): + """None when the message is authenticated, else the reason it is not. Only the topmost header counts.""" + headers = msg.get_all("Authentication-Results") or [] + if not headers: + return "no Authentication-Results header" + aid, by_method = parse_ar(headers[0]) + if aid != authserv_id.lower(): + return "topmost Authentication-Results header is not from our server" + dmarc = by_method.get("dmarc", []) + if dmarc and all(r == "pass" and p.get("header.from", "").rsplit("@", 1)[-1] == from_domain for r, p in dmarc): + return None + dkim = by_method.get("dkim", []) + if dkim and all(r == "pass" for r, _ in dkim) and any( + p.get("header.d") and (p["header.d"] == from_domain or p["header.d"].endswith("." + from_domain)) for _, p in dkim): + return None + return "not authenticated (no aligned dmarc=pass or dkim=pass)" + + +def extract_text(msg): + part = msg.get_body(("plain",)) + if part is None: + return "" + lines = [] + for line in clean(part.get_content()).split("\n"): + if line.rstrip() == "--": + break + if line.lstrip().startswith(">"): + continue + lines.append(line) + return "\n".join(lines).strip()[:MAX_TEXT] + + +def parse_message(msg, authserv_id): + """Return (comment dict with the sender "_address", None) or (None, reason). Content problems only.""" + subject = str(make_header(decode_header(str(msg.get("Subject", ""))))) + sites = site_names() + m = re.search(r"\[(" + "|".join(re.escape(x) for x in sites) + r"):(/[^\]\s]+)\]", subject) if sites else None + if not m: + return None, "no [site:/path/] tag in subject" + site, path = m.group(1), m.group(2) + if not valid_path(path): + return None, f"invalid post path {path!r}" + try: + path = canonical_path(path) + except CommentError: # e.g. a bare % becomes %25 and the path gets too long: a content problem, not a state one + return None, f"invalid post path {path[:60]!r}" + root = site_root(site) + if os.path.isdir(root) and not os.path.isfile(os.path.join(root, fs_path(path), "index.html")): + return None, f"no such post {site}:{path}" + froms = msg.get_all("From") or [] + addrs = [a for a in getaddresses([str(x) for x in froms]) if a[1]] + if len(froms) != 1 or len(addrs) != 1 or "@" not in addrs[0][1]: + return None, "need exactly one From header with one address" + name, addr = addrs[0] + why = authenticated(msg, addr.rsplit("@", 1)[1].lower(), authserv_id) + if why: + return None, why + text = extract_text(msg) + if not text: + return None, "empty text" + name = clean(name).replace("\n", " ").strip()[:MAX_NAME].strip() + if not name or "@" in name: + name = "A reader" + return {"id": new_id(), "site": site, "path": path, "author": name, "text": text, + "date": now_iso(), "source": "email", "parent": None, "_address": addr}, None + + +def recent_index(): + """sender hash -> dates of that sender's pending or approved comments (read once per run).""" + meta = read_json(meta_path(), {}) + index = {} + for cid in pending_ids(): + c = read_json(pending_path(cid), {}) + if c.get("sender_hash"): + index.setdefault(c["sender_hash"], []).append(c.get("date", "")) + for site in site_names(): + for items in load_approved(site).values(): + for c in items: + if meta.get(c.get("id")): + index.setdefault(meta[c["id"]], []).append(c.get("date", "")) + return index + + +def stored_keys(): + """Message keys of every stored comment (pending records carry theirs); prunes approved-keys.json.""" + approved_ids = {c.get("id") for site in site_names() for items in load_approved(site).values() for c in items} + keys = read_json(keys_path(), {}) + kept = {k: v for k, v in keys.items() if v in approved_ids} + if kept != keys: + write_json(keys_path(), kept) + for cid in pending_ids(): + k = read_json(pending_path(cid), {}).get("msg_key") + if k: + kept[k] = cid + return set(kept) + + +class MaildirSource: + """Where the mail comes from: a Maildir folder, with .done and .bad folders beside it.""" + + def __init__(self, path): + self.path = path + self.box = None + + def available(self): + return os.path.isdir(self.path) + + def close(self): + pass + + def keys(self): + self.box = mailbox.Maildir(self.path, factory=None, create=False) + return self.box.keys() + + def read(self, key): + """The raw bytes; KeyError when another run already took the message.""" + return self.box.get_bytes(key) + + def finish(self, key, raw, outcome): + """Move the message to .done or .bad (it is gone from the inbox afterwards).""" + mailbox.Maildir(self.path + "." + outcome, create=True).add(raw) + try: + self.box.remove(key) + except KeyError: # another process already took it + pass + + +IMAP_MAX_SIZE = 256 * 1024 +IMAP_TIMEOUT = 30 + + +def imap_config_path(): + return os.path.expanduser(os.environ.get("FW_COMMENTS_IMAP") or os.path.join(state_dir(), "imap.json")) + + +def load_imap_config(): + path = imap_config_path() + mode = os.stat(path).st_mode + if mode & 0o077: + raise CommentError(f"{path} must not be readable by group or others (chmod 600 it)") + cfg = read_json(path, {}) + if not isinstance(cfg, dict): + raise CommentError(f"{path} must hold a JSON object") + for key in ("host", "user", "password"): + if not isinstance(cfg.get(key), str) or not cfg[key]: + raise CommentError(f"{path}: {key!r} is missing or empty") + if not cfg[key].isascii(): + raise CommentError(f"{path}: {key!r} must be ASCII") + out = {"host": cfg["host"], "user": cfg["user"], "password": cfg["password"], "folder": cfg.get("folder") or "INBOX"} + for key in ("done_folder", "bad_folder"): + out[key] = cfg.get(key) or "" + for key in ("folder", "done_folder", "bad_folder"): + if not isinstance(out[key], str) or (out[key] and not IMAP_NAME_RE.match(out[key])): + raise CommentError(f"{path}: {key!r} must be a folder name of letters, digits and . _ / -") + return out + + +class SkipMessage(Exception): + """This message cannot be handled this run (e.g. the server refuses to fetch it); it stays where it is.""" + + +IMAP_NAME_RE = re.compile(r"^[A-Za-z0-9._/-]+$") + + +class IMAPSource: + """The comments mailbox on an IMAP server. Finished mail is copied to comments-done / comments-bad.""" + + def __init__(self): + self.m = None + self.cfg = None + self.validity = "" + self.caps = set() + + def available(self): + return os.path.exists(imap_config_path()) + + def _redact(self, text): + pw = self.cfg["password"] if self.cfg else "" + return text.replace(pw, "***") if pw else text + + def _fail(self, what, e=None, detail=""): + if e is not None and not isinstance(e, UnicodeError): # a UnicodeError could quote a password character + detail = f"{type(e).__name__} {e}" + raise CommentError(f"IMAP {what} failed" + (f": {self._redact(detail)}" if detail else "")) from None + + def _text(self, typ, data): + parts = [d.decode("utf-8", "replace") if isinstance(d, bytes) else str(d) for d in (data or []) if d] + return self._redact(f"{typ} {' '.join(parts)}".strip()) + + def _ok(self, result, what): + typ, data = result + if typ != "OK": + self._fail(what, detail=self._text(typ, data)) + return data + + def _open(self, readonly=False): + self.cfg = load_imap_config() + try: + self.m = imaplib.IMAP4_SSL(self.cfg["host"], 993, ssl_context=ssl.create_default_context(), timeout=IMAP_TIMEOUT) + self.m.login(self.cfg["user"], self.cfg["password"]) + self.caps = self._capabilities() # imaplib does not refresh these after login, and Dovecot adds UIDPLUS then + self._ok(self.m.select(self.cfg["folder"], readonly=readonly), "select") + v = self.m.response("UIDVALIDITY")[1] + self.validity = (v[0] or b"0").decode() + self._namespace() + except CommentError: + raise + except (OSError, UnicodeError, imaplib.IMAP4.error) as e: + self._fail("connection", e) + + def _capabilities(self): + try: + typ, data = self.m.capability() + except (imaplib.IMAP4.error, AttributeError): + return set() + return {t.upper() for d in (data if typ == "OK" else []) if isinstance(d, bytes) for t in d.decode("ascii", "replace").split()} + + def _list(self): + """[(delimiter, name)] from LIST (empty when the server refuses or the reply cannot be parsed).""" + try: + typ, data = self.m.list() + except imaplib.IMAP4.error: + return [] + out = [] + for item in data if typ == "OK" else []: + try: + if isinstance(item, tuple): # imaplib hands a literal mailbox name over as (head, literal) + head, name = item[0].decode("utf-8", "replace"), item[1].decode("utf-8", "replace") + m = re.match(r'^\((.*?)\) (?:"((?:[^"\\]|\\.)*)"|NIL) ', head) + out.append(((m.group(2) or "") if m else "", name)) + continue + m = re.match(r'^\((.*?)\) (?:"((?:[^"\\]|\\.)*)"|NIL) (.*)$', (item or b"").decode("utf-8", "replace").strip()) + if m: + out.append((m.group(2) or "", m.group(3).strip().strip('"'))) + except (AttributeError, IndexError, TypeError, ValueError): + continue + return out + + def _namespace_text(self): + try: + typ, data = self.m.namespace() + except (imaplib.IMAP4.error, AttributeError): + return "" + if typ != "OK": + return "" + return " ".join(d.decode("utf-8", "replace") for d in data if isinstance(d, bytes)) + + def _namespace(self): + """Learn the personal namespace prefix (e.g. "INBOX.") from NAMESPACE, else from LIST; sets self.prefix.""" + self.prefix = "" + try: + m = re.search(r'\(\("([^"]*)"', self._namespace_text()) + if m: + self.prefix = m.group(1) + return + listing = self._list() + delim = next((d for d, _ in listing if d), "") + if delim and any(n.startswith("INBOX" + delim) for _, n in listing): + self.prefix = "INBOX" + delim + except (AttributeError, IndexError, TypeError, ValueError): + self.prefix = "" + + def _folder(self, outcome): + return self.cfg[outcome + "_folder"] or f"{self.prefix}comments-{outcome}" + + def close(self): + if self.m is not None: + try: + self.m.logout() + except Exception: + pass + self.m = None + + def _uids(self): + data = self._ok(self.m.uid("SEARCH", None, "UNDELETED"), "search") + return [u.decode() for u in (data[0] or b"").split()] + + def _sizes(self, uids): + sizes = {} + for i in range(0, len(uids), 200): + data = self._ok(self.m.uid("FETCH", ",".join(uids[i:i + 200]), "(RFC822.SIZE)"), "fetch sizes") + for item in data: + text = (item[0] if isinstance(item, tuple) else item or b"").decode("ascii", "replace") + u, z = re.search(r"UID (\d+)", text), re.search(r"RFC822\.SIZE (\d+)", text) + if u and z: + sizes[u.group(1)] = int(z.group(1)) + return sizes + + def keys(self): + self._open() + uids = self._uids() + sizes = self._sizes(uids) + keys = [] + for uid in uids: + if sizes.get(uid, 0) > IMAP_MAX_SIZE: # never downloaded + try: + self._move(uid, "bad") + except (CommentError, OSError, imaplib.IMAP4.error) as e: # leave it, keep going + log_rejection(new_id(), f"cannot move oversize message {uid}: {e}") + continue + log_rejection(new_id(), f"message over {IMAP_MAX_SIZE // 1024} KB") + print(f"bad: message over {IMAP_MAX_SIZE // 1024} KB") + else: + keys.append(f"{self.validity}:{uid}") + return keys + + def _split(self, key): + validity, _, uid = key.partition(":") + if validity != self.validity or not uid.isdigit(): + raise KeyError(key) + return uid + + def read(self, key): + uid = self._split(key) + typ, data = self.m.uid("FETCH", uid, "(BODY.PEEK[])") + if typ != "OK": + raise SkipMessage(f"server refused to fetch UID {uid}: {self._text(typ, data)}") + for item in data: + if isinstance(item, tuple): + return item[1] + raise KeyError(key) + + def _move(self, uid, outcome): + folder = self._folder(outcome) + typ, _ = self.m.uid("COPY", uid, folder) + if typ != "OK": + self._ok(self.m.create(folder), "create folder " + folder) + self._ok(self.m.uid("COPY", uid, folder), "copy to " + folder) + self._ok(self.m.uid("STORE", uid, "+FLAGS", "(\\Deleted)"), "flag") + if "UIDPLUS" in self.caps: + self._ok(self.m.uid("EXPUNGE", uid), "expunge") + else: + self._ok(self.m.expunge(), "expunge") + + def finish(self, key, raw, outcome): + try: + self._move(self._split(key), outcome) + except (OSError, imaplib.IMAP4.error) as e: + self._fail("move", e) + + def check(self): + """Print the message count, folder names, NAMESPACE and, for the 5 newest messages, only header names + and Authentication-Results tokens.""" + self._open(readonly=True) + uids = self._uids() + print(f"messages in {self.cfg['folder']}: {len(uids)}") + print("folders: " + ", ".join(name for _, name in self._list())) + print("namespace: " + self._redact(self._namespace_text())) + print(f"target folders: {self._folder('done')}, {self._folder('bad')}") + for n, uid in enumerate(reversed(uids[-5:]), 1): + data = self._ok(self.m.uid("FETCH", uid, "(BODY.PEEK[HEADER])"), "fetch") + raw = next((i[1] for i in data if isinstance(i, tuple)), b"") + msg = message_from_bytes(raw, policy=policy.compat32) + print(f"message {n}: headers: " + ", ".join(sorted({k for k in msg.keys()}, key=str.lower))) + for h in msg.get_all("Authentication-Results") or []: + segs = split_ar(" ".join(str(h).split())) + tokens = [] + for seg in segs[1:]: + m = re.match(r"\s*([A-Za-z0-9_-]+)\s*=\s*([A-Za-z]+)(?![A-Za-z0-9])", seg) + if m: + tokens.append(f"{m.group(1).lower()}={m.group(2).lower()}") + head = segs[0].split() + print(f" Authentication-Results: authserv-id {head[0] if head else '(none)'}; " + " ".join(tokens)) + + +def message_source(): + if os.path.exists(imap_config_path()): + return IMAPSource() + return MaildirSource(os.path.expanduser(os.environ.get("FW_COMMENTS_MAILDIR") or "~/Maildir/.comments")) + + +def ingest_messages(source, authserv_id): + with locked(): + seen = stored_keys() + recent = recent_index() + cutoff = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() - SENDER_WINDOW)) + for key in source.keys(): + with locked(): # per message, so `fw comments approve` is not starved by a big backlog + try: + raw = source.read(key) + except KeyError: + continue + except SkipMessage as e: # leave it in the inbox and go on with the others + log_rejection(new_id(), f"skipped {key}: {e}") + print(f"skipped {key}: {e}") + continue + mkey = hashlib.sha256(raw).hexdigest() + if mkey in seen: # its comment is stored: a crash or overlap left the mail behind + log_rejection(new_id(), "duplicate of a stored comment; moved to .done") + source.finish(key, raw, "done") + continue + try: + c, why = parse_message(message_from_bytes(raw, policy=policy.default), authserv_id) + except (CommentError, OSError): # state problems abort the run and leave the mail in the inbox + raise + except Exception as e: # a malformed message must not stop the rest + c, why = None, f"unreadable message: {e}" + trusted = read_json(trusted_path(), []) + if c: + c["sender_hash"] = sender_hash(c.pop("_address")) + c["msg_key"] = mkey + h = c["sender_hash"] + if h not in trusted and len(pending_ids()) >= PENDING_CAP: + c, why = None, f"pending queue full ({PENDING_CAP})" + elif sum(1 for d in recent.get(h, []) if d >= cutoff) >= SENDER_LIMIT: + c, why = None, f"sender limit reached ({SENDER_LIMIT} per 24 hours)" + if c is None: # rejected mail is not remembered: moved back from .bad, it is tried again + cid = new_id() + log_rejection(cid, why) + print(f"bad {cid}: {why}") + source.finish(key, raw, "bad") + continue + if c["sender_hash"] in trusted: + add_approved(c) # stores first; a publish failure only warns + print(f"published {c['id']} {c['site']}:{c['path']} (trusted sender)") + else: + write_json(pending_path(c["id"]), c) + print(f"pending {c['id']} {c['site']}:{c['path']} by {c['author']}") + seen.add(mkey) + recent.setdefault(c["sender_hash"], []).append(c["date"]) + source.finish(key, raw, "done") + + +def cmd_ingest(): + source = message_source() + if not source.available(): + return + authserv_id = (load_config().get("authserv_id") or "").strip() + if not authserv_id: + raise CommentError('authserv_id is empty in sites.json; refusing to ingest (set it to the authserv-id ' + 'your mail server puts at the start of its Authentication-Results header)') + try: + with ingest_lock(): # a second run waits here, so it starts from the first run's results + ingest_messages(source, authserv_id) + finally: + source.close() + +def main(argv): + if not argv or argv[0] in ("-h", "--help", "help"): + print(__doc__) + return 0 + cmd, args = argv[0], argv[1:] + flags = [a for a in args if a.startswith("--")] + args = [a for a in args if not a.startswith("--")] + try: + if cmd == "list": + cmd_list() + elif cmd == "show" and len(args) == 1: + cmd_show(args[0]) + elif cmd == "approve" and args and set(flags) <= {"--trust"}: + cmd_approve(args, "--trust" in flags) + elif cmd == "reject" and args: + cmd_reject(args) + elif cmd == "remove" and len(args) == 1: + cmd_remove(args[0]) + elif cmd == "untrust" and len(args) == 1: + cmd_untrust(args[0]) + elif cmd == "publish" and len(args) <= 1: + cmd_publish(args[0] if args else None) + elif cmd == "export" and len(args) == 1: + cmd_export(args[0]) + elif cmd == "ingest": + cmd_ingest() + elif cmd == "imap-check": + source = IMAPSource() + if not source.available(): + raise CommentError(f"no IMAP config at {imap_config_path()}") + try: + source.check() + finally: + source.close() + else: + print("usage: comments.py list|show|approve|reject|remove|untrust|publish|export|ingest|imap-check (try --help)", file=sys.stderr) + return 2 + except (CommentError, OSError, imaplib.IMAP4.error) as e: + print(f"comments: {e}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/server/comments/post.php b/server/comments/post.php new file mode 100644 index 0000000..a75d3a2 --- /dev/null +++ b/server/comments/post.php @@ -0,0 +1,356 @@ +<?php +declare(strict_types=1); + +/* + * Comment form endpoint (T2 in the README tiers: server-side code that takes input from the + * public internet). OFF BY DEFAULT: nothing installs or enables this file; it lives in the repo + * until the comment form is switched on and this script is deployed to its own subdomain + * (e.g. comments.fuzzyworld.net, PHP 8.2, running as the account user). + * + * What it does: validates one POST from the comment form (fields site, path, elapsed, website, + * name, text), and writes ONE file, <state>/pending/<id>.json, in exactly the format comments.py + * writes for mail (source "form", sender_hash null). Nothing is ever published from here: a human + * runs `fw comments list / approve`, and `approve --trust` refuses form comments (no sender_hash). + * + * Threat model: every field is hostile. The only things the client controls that matter are + * strings, which are (1) checked against the same path rules as comments.py and sites.json, + * (2) cleaned of control/format characters, (3) length-capped, (4) stored as JSON data and + * never executed, included, or used as a file name; the only file read is + * <site root><validated path>index.html, and the redirect target is built from sites.json plus the + * validated path, never from the request. Output is fixed HTML; request values appear only through + * htmlspecialchars (the "are you human?" confirm page). Abuse is limited by a honeypot, a minimum + * time, 5 accepted comments per hour per hashed IP (IPv6: per /64), a cap of 100 pending form + * comments, and the global cap of 200 pending files. Be honest about the first two: `elapsed` is + * supplied by the client, so the minimum-time check and the confirm step only stop naive bots; a + * targeted script can sign its own confirm round trip. The real limits are the rate limit, the + * caps, and human moderation. Cross-site posts are refused by the Origin check (sites.json + * origins plus the optional "self_origin", the origin this script is served from). No cookies, + * no sessions, no error details to the client, errors go to error_log. + * + * Config (environment, for tests and odd layouts): + * FW_COMMENTS_DIR state directory, default ~/fw/comments (the same one comments.py uses) + * FW_COMMENTS_SITES sites.json, default: the one next to this file + * FW_COMMENTS_SELF_ORIGIN overrides "self_origin" in sites.json (e.g. https://comments.fuzzyworld.net) + * Needs only core PHP (no mbstring, no ctype). + * <state>/secret must already exist (comments.py creates it); without it this script fails with 500. + */ + +const PATH_RE = '#^/([A-Za-z0-9_~][A-Za-z0-9._%~+-]*/){1,8}\z#'; // same as comments.py PATH_RE +const MIN_MS = 3000; +const CONFIRM_MIN_S = 3; +const CONFIRM_MAX_S = 3600; +const RATE_MAX = 5; +const RATE_WINDOW = 3600; +const PENDING_CAP = 200; +const FORM_CAP = 100; +const MAX_NAME = 60; +const MAX_TEXT = 4000; + +ini_set('display_errors', '0'); + +function send_headers(): void +{ + header('Content-Type: text/html; charset=utf-8'); + header('X-Content-Type-Options: nosniff'); + header('Cache-Control: no-store'); + header("Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; form-action 'self'"); +} + +function page(int $code, string $body): never +{ + http_response_code($code); + send_headers(); + echo '<!doctype html><html lang="en"><meta charset="utf-8"><meta name="viewport" content="width=device-width">' + . '<title>Comment</title><style>body{font:16px/1.5 sans-serif;max-width:32em;margin:2em auto;padding:0 1em}' + . 'pre{white-space:pre-wrap}</style><body>' . $body . '</body></html>'; + exit; +} + +/** Refuse with a fixed message; $why (a fixed string, never request data) goes to the log only. */ +function refuse(int $code, string $why): never +{ + error_log("comments post.php: $code $why"); + page($code, '<p>Sorry, this comment could not be accepted.</p>'); +} + +function h(string $s): string +{ + return htmlspecialchars($s, ENT_QUOTES | ENT_SUBSTITUTE, 'UTF-8'); +} + +function home_dir(): string +{ + $home = getenv('HOME'); + if (($home === false || $home === '') && function_exists('posix_getpwuid')) { + $pw = posix_getpwuid(posix_getuid()); + $home = $pw['dir'] ?? ''; + } + return rtrim((string)$home, '/'); +} + +function expand(string $p): string +{ + return ($p === '~' || str_starts_with($p, '~/')) ? home_dir() . substr($p, 1) : $p; +} + +function is_utf8(string $s): bool +{ + return preg_match('//u', $s) === 1; +} + +/** Length in characters (the string must be valid UTF-8). */ +function ulen(string $s): int +{ + return (int)preg_match_all('/./su', $s); +} + +function valid_path(string $path): bool +{ + if (strlen($path) > 300 || !preg_match(PATH_RE, $path) || str_contains($path, '..')) { + return false; + } + foreach (explode('/', trim($path, '/')) as $seg) { + $decoded = rawurldecode($seg); // comments.py: unquote(), where bad UTF-8 becomes U+FFFD + if (!is_utf8($decoded) || $decoded === '.' || str_contains($decoded, '..') || str_contains($decoded, "\u{FFFD}") + || str_contains($decoded, '/') || str_contains($decoded, '\\') + || preg_match('/[\p{Cc}\p{Cf}]/u', $decoded)) { + return false; + } + } + return true; +} + +/** + * The canonical spelling of a validated post path, identical to comments.py canonical_path(): + * every segment decoded, then re-escaped like urllib.parse.quote(..., safe="~+"): letters, + * digits and _.-~+ stay, every other byte is %XX in upper case. So /sl%75g/ is /slug/ and + * %c3%a9 is %C3%A9. Keep the two in step. + */ +function canonical_path(string $path): string +{ + $canon = '/' . implode('/', array_map( + fn($seg) => str_replace('%2B', '+', rawurlencode(rawurldecode($seg))), + explode('/', trim($path, '/')))) . '/'; + return valid_path($canon) ? $canon : ''; +} + +/** [sites, self origin or '']. */ +function load_sites(string $file): array +{ + $data = json_decode((string)@file_get_contents($file), true); + if (!is_array($data)) { + refuse(500, 'cannot read sites.json'); + } + $sites = []; + foreach ($data as $name => $cfg) { + if (is_array($cfg) && is_string($cfg['root'] ?? null) && is_string($cfg['origin'] ?? null) + && preg_match('~^https?://[A-Za-z0-9.:-]+\z~', rtrim($cfg['origin'], '/'))) { + $sites[(string)$name] = ['root' => rtrim(expand($cfg['root']), '/'), 'origin' => rtrim($cfg['origin'], '/')]; + } + } + $self = rtrim((string)(getenv('FW_COMMENTS_SELF_ORIGIN') ?: ($data['self_origin'] ?? '')), '/'); + if ($self !== '' && !preg_match('~^https?://[A-Za-z0-9.:-]+\z~', $self)) { + $self = ''; + } + return [$sites, $self]; +} + +/** Control and format characters (bidi, zero-width, ...) and U+2028/2029 removed; "\n" kept in text. */ +function clean(string $s, bool $keep_newlines): string +{ + $s = str_replace(["\r\n", "\r"], "\n", $s); + if (!$keep_newlines) { + $s = str_replace("\n", ' ', $s); + } + $out = preg_replace('/(?!\n)[\p{Cc}\p{Cf}\x{2028}\x{2029}]/u', '', $s); + if ($out === null) { + refuse(400, 'cleaning failed'); + } + return trim($out); +} + +function write_atomic(string $path, string $data): void +{ + $dir = dirname($path); + if (!is_dir($dir) && !@mkdir($dir, 0700, true) && !is_dir($dir)) { + throw new RuntimeException('cannot create directory'); + } + $tmp = tempnam($dir, '.tmp-'); + if ($tmp === false) { + throw new RuntimeException('cannot create temp file'); + } + if (file_put_contents($tmp, $data) !== strlen($data) || !chmod($tmp, 0600) || !rename($tmp, $path)) { + @unlink($tmp); + throw new RuntimeException('cannot write file'); + } +} + +function sign(string $secret, string $ts, string $site, string $path, string $name, string $text): string +{ + return hash_hmac('sha256', $ts . '|' . $site . '|' . $path . '|' . hash('sha256', $name) . '|' . hash('sha256', $text), $secret); +} + +function main(): void +{ + if (($_SERVER['REQUEST_METHOD'] ?? '') !== 'POST') { + header('Allow: POST'); + refuse(405, 'method not allowed'); + } + + $state = rtrim(expand(getenv('FW_COMMENTS_DIR') ?: '~/fw/comments'), '/'); + [$sites, $selfOrigin] = load_sites(getenv('FW_COMMENTS_SITES') ?: __DIR__ . '/sites.json'); + + if (isset($_SERVER['HTTP_ORIGIN'])) { + $allowed = array_column($sites, 'origin'); + if ($selfOrigin !== '') { + $allowed[] = $selfOrigin; // the confirm page posts back to this origin + } + if (!in_array($_SERVER['HTTP_ORIGIN'], $allowed, true)) { + refuse(403, 'bad origin'); + } + } + + $secret = is_file("$state/secret") ? trim((string)file_get_contents("$state/secret")) : ''; + if ($secret === '') { + refuse(500, "missing $state/secret (run comments.py once)"); + } + + $in = []; + foreach (['site', 'path', 'elapsed', 'website', 'name', 'text', 'ts', 'sig'] as $k) { + $v = $_POST[$k] ?? ''; + if (!is_string($v) || strlen($v) > 16000) { + refuse(400, 'bad field'); + } + $in[$k] = $v; + } + + $site = $in['site']; + $path = $in['path']; + if (!isset($sites[$site]) || !valid_path($path) || !is_dir($sites[$site]['root']) + || !is_file($sites[$site]['root'] . '/' . rawurldecode(trim($path, '/')) . '/index.html')) { // as comments.py fs_path + refuse(400, 'unknown site or post'); + } + $path = canonical_path($path); // stored and redirected in the canonical spelling + if ($path === '') { + refuse(400, 'path not canonical'); + } + $thanks = $sites[$site]['origin'] . $path . '#comment-pending'; + + if ($in['website'] !== '') { // honeypot: looks like success, stores nothing + http_response_code(303); + header('Location: ' . $thanks); + send_headers(); + return; + } + + if (!is_utf8($in['name']) || !is_utf8($in['text'])) { + refuse(400, 'invalid utf-8'); + } + $name = clean($in['name'], false); + $text = clean($in['text'], true); + if ($name === '' || ulen($name) > MAX_NAME || $text === '' || ulen($text) > MAX_TEXT) { + refuse(400, 'bad name or text length'); + } + if (str_contains($name, '@')) { + $name = 'A reader'; + } + + $now = time(); + if ($in['sig'] !== '') { // second step: the visitor confirmed + $ok = preg_match('/^\d{1,12}\z/', $in['ts']) === 1 + && hash_equals(sign($secret, $in['ts'], $site, $path, $name, $text), $in['sig']) + && ($now - (int)$in['ts']) >= CONFIRM_MIN_S && ($now - (int)$in['ts']) <= CONFIRM_MAX_S; + if (!$ok) { + refuse(400, 'bad or expired confirmation'); + } + } elseif (preg_match('/^\d{1,9}\z/', $in['elapsed']) !== 1 || (int)$in['elapsed'] < MIN_MS) { + $ts = (string)$now; + page(200, '<p>That was quick. Please confirm that you are a person and want to post this comment:</p>' + . '<p><strong>' . h($name) . '</strong></p><pre>' . h($text) . '</pre>' + . '<form method="post" action="">' + . '<input type="hidden" name="site" value="' . h($site) . '">' + . '<input type="hidden" name="path" value="' . h($path) . '">' + . '<input type="hidden" name="name" value="' . h($name) . '">' + . '<input type="hidden" name="text" value="' . h($text) . '">' + . '<input type="hidden" name="ts" value="' . h($ts) . '">' + . '<input type="hidden" name="sig" value="' . h(sign($secret, $ts, $site, $path, $name, $text)) . '">' + . '<button type="submit">Post comment</button></form>'); + } + + // Accept. One lock (the one comments.py uses) covers rate limit, cap and write. + if (!is_dir($state) && !@mkdir($state, 0700, true) && !is_dir($state)) { + throw new RuntimeException('cannot create state directory'); + } + $lock = fopen("$state/lock", 'c'); + if ($lock === false || !flock($lock, LOCK_EX)) { + throw new RuntimeException('cannot lock'); + } + + $addr = (string)($_SERVER['REMOTE_ADDR'] ?? ''); + $packed = @inet_pton($addr); + if ($packed !== false && strlen($packed) === 16) { // IPv6: one visitor owns a whole /64 + $addr = bin2hex(substr($packed, 0, 8)); + } + $iphash = hash_hmac('sha256', $addr, $secret); + $rfile = "$state/rate/$iphash"; + $recent = []; + foreach (is_file($rfile) ? (file($rfile, FILE_IGNORE_NEW_LINES) ?: []) : [] as $line) { + if (preg_match('/^\d{1,12}\z/', $line) === 1 && (int)$line > $now - RATE_WINDOW) { + $recent[] = (int)$line; + } + } + if (count($recent) >= RATE_MAX) { + refuse(429, 'rate limit'); + } + $files = glob("$state/pending/*.json") ?: []; + if (count($files) >= PENDING_CAP) { + refuse(503, 'pending queue full'); + } + $forms = 0; + foreach ($files as $f) { + $c = json_decode((string)@file_get_contents($f), true); + if (is_array($c) && ($c['source'] ?? '') === 'form') { + $forms++; + } + } + if ($forms >= FORM_CAP) { + refuse(503, 'pending form queue full'); + } + + $id = gmdate('Ymd\THis\Z', $now) . '-' . bin2hex(random_bytes(3)); + $record = [ + 'id' => $id, 'site' => $site, 'path' => $path, 'author' => $name, 'text' => $text, + 'date' => gmdate('Y-m-d\TH:i:s\Z', $now), 'source' => 'form', 'parent' => null, 'sender_hash' => null, + ]; + ksort($record); + $json = json_encode($record, JSON_PRETTY_PRINT | JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR); + $json = preg_replace_callback('/^ +/m', fn($m) => str_repeat(' ', intdiv(strlen($m[0]), 2)), $json); // 2-space indent like comments.py + write_atomic("$state/pending/$id.json", $json . "\n"); + + try { // the comment is stored; a failed rate record must not turn that into an error + $recent[] = $now; + write_atomic($rfile, implode("\n", $recent) . "\n"); + foreach (glob("$state/rate/*") ?: [] as $f) { // forget other addresses after the window + if (is_file($f) && filemtime($f) < $now - RATE_WINDOW) { + @unlink($f); + } + } + } catch (Throwable $e) { + error_log('comments post.php: rate record not written: ' . $e->getMessage()); + } + + http_response_code(303); + header('Location: ' . $thanks); + send_headers(); +} + +try { + main(); +} catch (Throwable $e) { + error_log('comments post.php: ' . get_class($e) . ': ' . $e->getMessage()); + if (!headers_sent()) { + http_response_code(500); + send_headers(); + echo '<!doctype html><meta charset="utf-8"><p>Sorry, something went wrong.</p>'; + } +} diff --git a/server/comments/sites.json b/server/comments/sites.json new file mode 100644 index 0000000..94ebfd9 --- /dev/null +++ b/server/comments/sites.json @@ -0,0 +1,6 @@ +{ + "writing": {"origin": "https://writing.fuzzyworld.net", "root": "~/writing.fuzzyworld.net"}, + "til": {"origin": "https://til.fuzzyworld.net", "root": "~/til.fuzzyworld.net"}, + "mailbox": "comments-r5h7@fuzzyworld.net", + "authserv_id": "inbound.mailchannels.net" +} diff --git a/server/fw b/server/fw index 0ad433e..4a5cdc7 100755 --- a/server/fw +++ b/server/fw @@ -19,6 +19,7 @@ # fw ci-watch follow CI (all repos, private too) until nothing is queued or running # fw ci-pages <name> rebuild a repo's CI pages and the dashboard (runner-api runs this) # fw ci-dashboard rebuild code.fuzzyworld.net/ci/ +# fw comments <cmd> moderate blog comments (list, show, approve, reject, ... ingest, imap-check) # # Visibility is `git config fuzzyworld.visibility` inside each repo. Anything other # than exactly "public" is treated as private. @@ -245,6 +246,7 @@ case "${1:-}" in ci-watch) cmd_ci_watch ;; ci-pages) need_repo "${2:-}"; with_lock "$2" "${GEN[@]}" ci "$2" ;; ci-dashboard) "${GEN[@]}" ci-dashboard ;; - ""|-h|--help|help) sed -n '2,23p' "$0" | sed 's/^# \{0,1\}//' ;; + comments) shift; python3 "$TOOLS/server/comments.py" "$@" ;; + ""|-h|--help|help) sed -n '2,24p' "$0" | sed 's/^# \{0,1\}//' ;; *) die "unknown command: $1 (try: fw help)" ;; esac diff --git a/server/jobs/comments-ingest.sh b/server/jobs/comments-ingest.sh new file mode 100755 index 0000000..b4283ac --- /dev/null +++ b/server/jobs/comments-ingest.sh @@ -0,0 +1,9 @@ +#!/bin/bash +# Hourly: turn new mail in the comments mailbox (IMAP, or ~/Maildir/.comments) into blog comments (see server/comments.py). +# Quiet unless something happened or failed. +set -euo pipefail +mkdir -p "$HOME/fw/state/locks" +exec 9>"$HOME/fw/state/locks/job-comments-ingest.lock" +flock -n 9 || exit 0 +out=$(python3 "$HOME/fw/tools/server/comments.py" ingest 2>&1) || { echo "== $(date -u '+%F %T UTC') comments ingest failed"; echo "$out"; exit 1; } +[ -z "$out" ] || { echo "== $(date -u '+%F %T UTC') comments ingest"; echo "$out"; } diff --git a/server/jobs/crontab b/server/jobs/crontab index cbd0837..7fd77f7 100644 --- a/server/jobs/crontab +++ b/server/jobs/crontab @@ -1,4 +1,5 @@ # fw:begin (managed by fuzzyworld-tools server/install.sh; edit server/jobs/crontab, then reinstall) -# Each job script checks its own UTC schedule; cron only wakes it hourly. +# Each job script checks its own UTC schedule (if it has one); cron only wakes it hourly. 0 * * * * $HOME/fw/tools/server/jobs/north-star-cleanup.sh >> $HOME/fw/logs/cron-north-star-cleanup.log 2>&1 +0 * * * * $HOME/fw/tools/server/jobs/comments-ingest.sh >> $HOME/fw/logs/cron-comments-ingest.log 2>&1 # fw:end diff --git a/server/site-blog/blog.css b/server/site-blog/blog.css index 5c1e975..c7336b5 100644 --- a/server/site-blog/blog.css +++ b/server/site-blog/blog.css @@ -75,6 +75,10 @@ ul.comments { list-style: none; padding: 0; } ul.comments ul.comments { margin-left: var(--spacing-lg); padding-left: var(--spacing-md); border-left: 2px solid var(--color-border-primary); } .comment { margin: var(--spacing-md) 0; } .comment p { margin: 0 0 var(--spacing-xs); } +.comment-form label { display: block; } +.comment-form input[type=text], .comment-form textarea { box-sizing: border-box; width: 100%; max-width: 40rem; padding: var(--spacing-xs) var(--spacing-sm); + font: inherit; color: var(--color-fg-primary); background: var(--color-bg-secondary); border: 1px solid var(--color-border-primary); border-radius: 6px; } +.comment-form .hp { position: absolute; left: -10000px; width: 1px; height: 1px; overflow: hidden; } /* Code colors: the token classes gen.py's highlighter writes. */ .hl .c { color: var(--color-hl-comment); font-style: italic; } diff --git a/server/site-blog/blog.js b/server/site-blog/blog.js index 406f93b..7529753 100644 --- a/server/site-blog/blog.js +++ b/server/site-blog/blog.js @@ -53,5 +53,70 @@ window.addEventListener("resize", update); update(); } + + comments(); }); + + /* Reader comments: /_comments/<post>.json is written by the server. Missing (404), broken or not a list: nothing shown. + Items are built with createElement/textContent only. Comments already in the page (imported, id="comment-N") are skipped. */ + function comments() { + var started = Date.now(); + var sections = document.querySelectorAll(".comments-section[data-src]"); + Array.prototype.forEach.call(sections, function (sec) { + var form = sec.querySelector("form.comment-form"); + if (form) form.addEventListener("submit", function () { + var f = form.elements.elapsed; + if (f) f.value = String(Date.now() - started); + }); + if (!window.fetch) return; + fetch(sec.getAttribute("data-src"), { credentials: "omit", headers: { Accept: "application/json" } }) + .then(function (r) { return r.ok ? r.json() : null; }) + .then(function (list) { if (Array.isArray(list)) show(sec, list); }) + .catch(function () {}); + }); + } + + function cid(c) { return c && c.id != null ? String(c.id).replace(/[^A-Za-z0-9_-]/g, "") : ""; } + + function show(sec, list) { + var box = sec.querySelector(".comments-live") || sec; + var fresh = list.filter(function (c) { + return c && typeof c === "object" && cid(c) && !document.getElementById("comment-" + cid(c)); + }); + fresh.sort(function (a, b) { return String(a.date || "") < String(b.date || "") ? -1 : String(a.date || "") > String(b.date || "") ? 1 : 0; }); + var made = Object.create(null); + fresh.forEach(function (c) { made[cid(c)] = item(c); }); + var top = document.createElement("ul"); + top.className = "comments"; + fresh.forEach(function (c) { + var li = made[cid(c)], pid = c.parent ? cid({ id: c.parent }) : ""; + var parent = pid && pid !== cid(c) ? (made[pid] || document.getElementById("comment-" + pid)) : null; + if (!parent) { top.appendChild(li); return; } + var ul = null; + for (var n = parent.firstChild; n; n = n.nextSibling) if (n.nodeName === "UL") ul = n; + if (!ul) { ul = document.createElement("ul"); ul.className = "comments"; parent.appendChild(ul); } + ul.appendChild(li); + }); + if (top.firstChild) box.appendChild(top); + } + + function item(c) { + var li = document.createElement("li"); + li.className = "comment"; + li.id = "comment-" + cid(c); + var meta = document.createElement("p"); + meta.className = "meta"; + var who = document.createElement("strong"); + who.textContent = String(c.author == null ? "" : c.author); + meta.appendChild(who); + meta.appendChild(document.createTextNode(" · " + String(c.date == null ? "" : c.date).slice(0, 10))); + var body = document.createElement("p"); + String(c.text == null ? "" : c.text).replace(/\r\n/g, "\n").split("\n").forEach(function (line, i) { + if (i) { body.appendChild(document.createElement("br")); body.appendChild(document.createTextNode("\n")); } + body.appendChild(document.createTextNode(line)); + }); + li.appendChild(meta); + li.appendChild(body); + return li; + } })(); diff --git a/server/site-blog/htaccess b/server/site-blog/htaccess index 7d477b8..ee36cd7 100644 --- a/server/site-blog/htaccess +++ b/server/site-blog/htaccess @@ -20,6 +20,11 @@ RemoveType .php .php3 .php4 .php5 .php7 .php8 .phtml .phar ForceType application/atom+xml </FilesMatch> +# Reader comments, written by the server as /_comments/<post path>.json (never part of the build). +<FilesMatch "\.json$"> + ForceType application/json +</FilesMatch> + <IfModule mod_headers.c> Header always set Content-Security-Policy "default-src 'self'; img-src 'self' https: data:; style-src 'self' 'unsafe-inline'; script-src 'self'; frame-src 'none'; frame-ancestors 'none'" Header always set X-Content-Type-Options nosniff @@ -28,4 +33,7 @@ RemoveType .php .php3 .php4 .php5 .php7 .php8 .phtml .phar <FilesMatch "\.(html|css|js|xml)$"> Header set Cache-Control "no-cache" </FilesMatch> + <FilesMatch "\.json$"> + Header set Cache-Control "no-cache" + </FilesMatch> </IfModule> diff --git a/tests/test_blog_comments.py b/tests/test_blog_comments.py new file mode 100644 index 0000000..acbf5cc --- /dev/null +++ b/tests/test_blog_comments.py @@ -0,0 +1,124 @@ +"""Tests for reader comments in server/blog.py: email link, optional form, data-src. Run: python3 -m unittest discover tests""" +import json +import os +import sys +import unittest + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from test_blog import POST, Tree, blog, needs_markdown # noqa: E402 + +ON = {"comments_site": "writing", "comments_mailbox": "comments@example.org"} +FIRST = "2018/03/02/first-post/index.html" + + +def make(test, **config): + t = Tree(test, config) + t.file("2018-03-02-first-post.md", "---\npublished: true\n---\n" + POST) + t.file("2016-05-06-old-post.md", "---\npublished: true\n---\n# Old\n\nSome old text.\n") + return t + + +@needs_markdown +class CommentsOff(unittest.TestCase): + def test_off_by_default_nothing_changes(self): + t = make(self) + _, out = t.build() + everything = "\n".join(t.read(out, r) for r in (FIRST, "blog/index.html", "feed/index.xml")) + for word in ("comments-section", "data-src", "mailto:", "comment-form"): + self.assertNotIn(word, everything) + self.assertFalse(os.path.exists(os.path.join(out, "_comments"))) + + def test_imported_comments_markup_is_the_same_off_and_on(self): + data = {"/2018/03/02/first-post/": [ + {"id": 1, "parent": 0, "author": "Eve", "date": "2018-03-03T10:00:00", "text": "Hi"}, + {"id": 2, "parent": 1, "author": "Bob", "date": "2018-03-04T10:00:00", "text": "reply"}]} + pages = [] + for extra in ({}, ON): + t = make(self, comments="comments.json", **extra) + t.file("comments.json", json.dumps(data), base=t.dir) + _, out = t.build() + pages.append(t.read(out, FIRST)) + off, on = pages + tree = off[off.index('<ul class="comments">'):off.index('<p class="meta">Comments ©')] + self.assertIn(tree, on) + self.assertNotIn("data-src", off) + self.assertIn('data-src="/_comments/2018/03/02/first-post.json"', on) + + +@needs_markdown +class CommentsOn(unittest.TestCase): + def test_section_for_every_post_with_data_src_and_mailto(self): + t = make(self, **ON) + _, out = t.build() + page = t.read(out, FIRST) + self.assertIn('<section class="comments-section" data-src="/_comments/2018/03/02/first-post.json">', page) + self.assertIn('<div class="comments-live"></div>', page) + self.assertIn('href="mailto:comments@example.org?subject=Comment%20%5Bwriting%3A%2F2018%2F03%2F02%2Ffirst-post%2F%5D"', page) + self.assertIn("Comments are moderated, and your address is never shown.", page) + self.assertIn('data-src="/_comments/2016/05/06/old-post.json"', t.read(out, "2016/05/06/old-post/index.html")) + self.assertNotIn("<form", page) + self.assertNotIn('name="website"', page) + self.assertFalse(os.path.exists(os.path.join(out, "_comments"))) + blog.audit_output(out) + + def test_undated_url_scheme(self): + t = make(self, url="/notes/{slug}/", **ON) + _, out = t.build() + page = t.read(out, "notes/first-post/index.html") + self.assertIn('data-src="/_comments/notes/first-post.json"', page) + self.assertIn("writing%3A%2Fnotes%2Ffirst-post%2F%5D", page) + + def test_hostile_title_does_not_reach_the_widget_and_values_are_escaped(self): + t = Tree(self, {"comments_site": "writing", "comments_mailbox": "a+b&c@example.org", + "comments_form": {"action": "https://c.example.org/post.php?a=1&b=2"}}) + t.file("2018-03-02-x.md", '---\npublished: true\ntitle: "<script>alert(1)</script>"\n---\n# T\n\n## Writing\n\ntext\n') + _, out = t.build() + page = t.read(out, "2018/03/02/x/index.html") + self.assertNotIn("<script>alert", page) + self.assertIn("mailto:a%2Bb%26c@example.org?", page) + self.assertIn('action="https://c.example.org/post.php?a=1&b=2"', page) + + def test_form_only_with_comments_form(self): + t = make(self, comments_form={"action": "https://comments.example.org/post.php"}, **ON) + _, out = t.build() + page = t.read(out, FIRST) + self.assertIn('<form class="comment-form" method="post" action="https://comments.example.org/post.php">', page) + self.assertIn('<input type="hidden" name="site" value="writing">', page) + self.assertIn('<input type="hidden" name="path" value="/2018/03/02/first-post/">', page) + self.assertIn('<input type="hidden" name="elapsed" value="0">', page) + self.assertIn('name="website" tabindex="-1" autocomplete="off"', page) + self.assertIn('name="name" maxlength="60" required', page) + self.assertIn('name="text" rows="5" maxlength="4000" required', page) + self.assertIn('type="submit"', page) + + +class CommentsConfig(unittest.TestCase): + def bad(self, **config): + with self.assertRaises(blog.BlogError): + Tree(self, config).cfg() + + def test_validation(self): + self.bad(comments_site="writing") # no mailbox + self.bad(comments_site="writing", comments_mailbox="not-an-address") + self.bad(comments_site="writing", comments_mailbox="a b@example.org") + self.bad(comments_site="bad site", comments_mailbox="a@example.org") + self.bad(comments_site=5, comments_mailbox="a@example.org") + self.bad(comments_mailbox="a@example.org") # mailbox without a site + self.bad(comments_form={"action": "https://x.org/p"}) # form without a site + for form in ("https://x.org/p", {"action": "http://x.org/p"}, {"action": "https://"}, {"action": 3}, + {"action": "https://x.org/p q"}, {"action": 'https://x.org/"'}, {"url": "https://x.org/p"}, + {"action": "https://x.org/p", "extra": 1}, {}): + self.bad(comments_form=form, **ON) + + def test_valid_form_is_accepted(self): + cfg = Tree(self, {**ON, "comments_form": {"action": "https://x.org/p"}}).cfg() + self.assertEqual(cfg["comments_form"], {"action": "https://x.org/p"}) + + def test_defaults(self): + cfg = Tree(self).cfg() + self.assertEqual((cfg["comments_site"], cfg["comments_mailbox"], cfg["comments_form"]), (None, None, None)) + self.assertEqual(Tree(self, {**ON, "comments_form": None}).cfg()["comments_form"], None) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_comments.py b/tests/test_comments.py new file mode 100644 index 0000000..7570bd2 --- /dev/null +++ b/tests/test_comments.py @@ -0,0 +1,972 @@ +"""Tests for server/comments.py (blog comments by email, moderation CLI, publishing). + +Run with: python3 -m unittest discover tests +""" + +import contextlib +import importlib.util +import io +import itertools +import json +import mailbox +import os +import shutil +import tempfile +import unittest + +ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..") +spec = importlib.util.spec_from_file_location("comments", os.path.join(ROOT, "server", "comments.py")) +comments = importlib.util.module_from_spec(spec) +spec.loader.exec_module(comments) + +POST = "/2024/05/01/hello/" + + +_counter = itertools.count() + + +def mail(subject="Comment [writing:/2024/05/01/hello/]", body="Nice post.\n", sender="Ann <ann@example.org>", + auth=("mx; dkim=pass header.d=example.org; dmarc=fail",), extra_auth_below=(), extra_headers=()): + lines = [f"Message-ID: <{next(_counter)}@test>", *extra_headers] + for a in auth: + lines.append(f"Authentication-Results: {a}") + lines += [f"From: {sender}", "To: comments@fuzzyworld.net", f"Subject: {subject}"] + for a in extra_auth_below: + lines.append(f"Authentication-Results: {a}") + lines += ["Content-Type: text/plain; charset=utf-8", "", body] + return "\n".join(lines).encode() + + +class Base(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmp) + j = lambda *p: os.path.join(self.tmp, *p) + self.root = j("writing") + os.makedirs(j("writing", POST.strip("/"))) + open(j("writing", POST.strip("/"), "index.html"), "w").write("x") + self.til = j("til") # does not exist: existence check skipped + json.dump({"writing": {"origin": "https://w", "root": self.root}, "til": {"origin": "https://t", "root": self.til}, + "mailbox": "", "authserv_id": "mx"}, open(j("sites.json"), "w")) + self.maildir = j("Maildir", ".comments") + env = {"FW_COMMENTS_DIR": j("state"), "FW_COMMENTS_SITES": j("sites.json"), "FW_COMMENTS_MAILDIR": self.maildir} + old = {k: os.environ.get(k) for k in env} + os.environ.update(env) + self.addCleanup(lambda: [os.environ.pop(k) if v is None else os.environ.__setitem__(k, v) for k, v in old.items()]) + self.state = j("state") + + def run_cli(self, *argv): + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = comments.main(list(argv)) + return code, out.getvalue(), err.getvalue() + + def deliver(self, raw): + os.makedirs(os.path.dirname(self.maildir), exist_ok=True) + mailbox.Maildir(self.maildir, create=True).add(raw) + + def count(self, suffix): + d = self.maildir + suffix + return len(mailbox.Maildir(d, create=False).keys()) if os.path.isdir(d) else 0 + + def pending(self): + return comments.pending_ids() + + def pub(self, path="2024/05/01/hello.json"): + return os.path.join(self.root, "_comments", path) + + def add_pending(self, **kw): + c = {"id": comments.new_id(), "site": "writing", "path": POST, "author": "Ann", "text": "hi", + "date": comments.now_iso(), "source": "form", "parent": None} + c.update(kw) + comments.write_json(comments.pending_path(c["id"]), c) + return c["id"] + + +class Moderation(Base): + def test_approve_publishes_without_private_fields(self): + cid = self.add_pending(source="email", sender_hash="abc") + self.assertEqual(self.run_cli("approve", cid)[0], 0) + self.assertEqual(self.pending(), []) + pub = json.load(open(self.pub())) + self.assertEqual([c["id"] for c in pub], [cid]) + self.assertEqual(set(pub[0]), {"author", "date", "id", "parent", "text"}) + self.assertNotIn("abc", open(os.path.join(self.state, "approved", "writing.json")).read()) + self.assertEqual(json.load(open(os.path.join(self.state, "approved-meta.json"))), {cid: "abc"}) + + def test_reject_logs(self): + cid = self.add_pending() + self.run_cli("reject", cid) + self.assertEqual(self.pending(), []) + self.assertIn(cid, open(os.path.join(self.state, "rejected.log")).read()) + self.assertFalse(os.path.exists(self.pub())) + + def test_remove_republishes_and_empty_removes_file(self): + a, b = self.add_pending(), self.add_pending() + self.run_cli("approve", a, b) + self.run_cli("remove", a) + self.assertEqual([c["id"] for c in json.load(open(self.pub()))], [b]) + self.run_cli("remove", b) + self.assertFalse(os.path.exists(self.pub())) + self.assertEqual(self.run_cli("remove", b)[0], 1) + + def test_trust_and_untrust(self): + cid = self.add_pending(source="email", sender_hash="h1") + self.run_cli("approve", cid, "--trust") + self.assertEqual(json.load(open(os.path.join(self.state, "trusted.json"))), ["h1"]) + self.run_cli("untrust", cid) + self.assertEqual(json.load(open(os.path.join(self.state, "trusted.json"))), []) + + def test_trust_refused_for_form_comment(self): + cid = self.add_pending(source="form") + self.assertEqual(self.run_cli("approve", cid, "--trust")[0], 1) + self.assertEqual(self.pending(), [cid]) + + def test_publish_rewrites_files(self): + cid = self.add_pending() + self.run_cli("approve", cid) + os.unlink(self.pub()) + self.run_cli("publish", "writing") + self.assertTrue(os.path.exists(self.pub())) + self.assertIn(cid, self.run_cli("export", "writing")[1]) + + def test_list_and_show(self): + cid = self.add_pending(text="x" * 300) + out = self.run_cli("list")[1] + self.assertIn(cid, out) + self.assertNotIn("x" * 161, out) + self.assertIn(cid, self.run_cli("show", cid)[1]) + + def test_traversal_paths_refused(self): + for bad in ("/../../etc/", "/a/../b/", "/./", "/..%2f/", "/x", "/", "//", "/a//b/", "/%2e%2e/", "/a/%2E/", "/a%2fb/", "/a%00b/", "/caf%e9/", "/%2e/", "/" + "a/" * 9): + self.assertFalse(comments.valid_path(bad), bad) + cid = self.add_pending(path="/../../evil/") + self.assertEqual(self.run_cli("approve", cid)[0], 1) + self.assertFalse(os.path.exists(os.path.join(self.tmp, "evil.json"))) + self.assertEqual(self.run_cli("show", "../x")[0], 1) + + def test_missing_site_root_is_not_created(self): + shutil.rmtree(self.root) + cid = self.add_pending() + self.assertEqual(self.run_cli("approve", cid)[0], 0) + self.assertFalse(os.path.exists(self.root)) + self.assertEqual(len(json.load(open(os.path.join(self.state, "approved", "writing.json")))[POST]), 1) + os.makedirs(self.root) + self.run_cli("publish") + self.assertTrue(os.path.exists(self.pub())) + + def test_valid_paths(self): + for ok in ("/hello/", POST, "/about.me/", "/a%20b/", "/a/b/c/", "/2024/05/01/", "/blog/hello/", "/caf%c3%a9/", "/_posts/~me/x+y/"): + self.assertTrue(comments.valid_path(ok), ok) + + +class Ingest(Base): + def ingest(self): + return self.run_cli("ingest") + + def test_aligned_dkim_pass(self): + self.deliver(mail()) + self.ingest() + self.assertEqual(len(self.pending()), 1) + c = json.load(open(comments.pending_path(self.pending()[0]))) + self.assertEqual((c["author"], c["site"], c["path"], c["source"]), ("Ann", "writing", POST, "email")) + self.assertEqual(c["sender_hash"], comments.sender_hash("ANN@example.org")) + self.assertNotIn("ann@example.org", json.dumps(c)) + self.assertEqual((self.count(""), self.count(".done"), self.count(".bad")), (0, 1, 0)) + + def test_dkim_subdomain_of_from_domain_ok_but_not_parent(self): + self.deliver(mail(auth=("mx; dkim=pass header.d=mail.example.org",))) + self.deliver(mail(sender="Bob <bob@mail.example.org>", auth=("mx; dkim=pass header.d=example.org",))) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (1, 1)) + + def test_dmarc_pass(self): + self.deliver(mail(auth=("mx; dmarc=pass header.from=example.org",))) + self.ingest() + self.assertEqual(len(self.pending()), 1) + + def test_dmarc_fail(self): + self.deliver(mail(auth=("mx; dkim=fail; dmarc=fail",))) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (0, 1)) + self.assertIn("not authenticated", open(os.path.join(self.state, "rejected.log")).read()) + + def test_unaligned_dkim_rejected(self): + self.deliver(mail(auth=("mx; dkim=pass header.d=evil.test; dmarc=fail",))) + self.ingest() + self.assertEqual(self.count(".bad"), 1) + + def test_forged_lower_header_ignored(self): + self.deliver(mail(auth=("mx; dmarc=fail",), extra_auth_below=("mx; dmarc=pass",))) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (0, 1)) + + def set_cfg(self, **kw): + cfg = json.load(open(os.environ["FW_COMMENTS_SITES"])) + cfg.update(kw) + json.dump(cfg, open(os.environ["FW_COMMENTS_SITES"], "w")) + + def test_empty_authserv_id_refuses(self): + self.set_cfg(authserv_id="") + self.deliver(mail()) + code, out, err = self.ingest() + self.assertEqual(code, 1) + self.assertIn("authserv_id", err) + self.assertEqual((self.count(""), self.count(".done"), self.count(".bad"), self.pending()), (1, 0, 0, [])) + + def test_only_topmost_header_counts(self): + # topmost is someone else's: rejected even though a lower header has our id and passes + self.deliver(mail(auth=("evil.test; dmarc=pass header.from=example.org",), extra_auth_below=("mx; dmarc=pass header.from=example.org",))) + # topmost absent, only a forged lower header with our id (it is then the topmost, but a sender-made one + # cannot be told apart, so the real protection is the receiving server placing its own on top) + self.deliver(mail(auth=("evil.test; dmarc=pass header.from=example.org", "mx; dmarc=pass header.from=example.org"))) + self.deliver(mail(auth=("mx; dmarc=pass header.from=example.org", "evil.test; dmarc=fail"))) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (1, 2)) + + def test_no_auth_header(self): + self.deliver(mail(auth=())) + self.ingest() + self.assertEqual(self.count(".bad"), 1) + + def test_subject_without_tag(self): + self.deliver(mail(subject="hello there")) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (0, 1)) + + def test_unknown_post_when_root_exists(self): + self.deliver(mail(subject="[writing:/2024/05/01/missing/]")) + self.deliver(mail(subject="[writing:/../../etc/]")) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (0, 2)) + + def test_til_site_root_missing_skips_existence_check(self): + self.deliver(mail(subject="[til:/2024/05/01/x/]")) + self.ingest() + self.assertEqual(len(self.pending()), 1) + + def test_re_prefix(self): + self.deliver(mail(subject="Re: Fwd: Thoughts [writing:/2024/05/01/hello/] more")) + self.ingest() + self.assertEqual(len(self.pending()), 1) + + def test_quotes_and_signature_cut(self): + self.deliver(mail(body="> old text\nMy reply\n> more old\nsecond line\n\n-- \nAnn\nsecret sig\n")) + self.ingest() + c = json.load(open(comments.pending_path(self.pending()[0]))) + self.assertEqual(c["text"], "My reply\nsecond line") + + def test_oversized_body_and_name_capped_and_controls_dropped(self): + self.deliver(mail(sender='"' + "N" * 100 + '" <ann@example.org>', body="a\x01b" + "x" * 9000 + "\n")) + self.ingest() + c = json.load(open(comments.pending_path(self.pending()[0]))) + self.assertEqual(len(c["text"]), 4000) + self.assertNotIn("\x01", c["text"]) + self.assertEqual(len(c["author"]), 60) + + def test_missing_name_gives_a_reader(self): + self.deliver(mail(sender="ann@example.org")) + self.ingest() + self.assertEqual(json.load(open(comments.pending_path(self.pending()[0])))["author"], "A reader") + + def test_trusted_sender_auto_publishes(self): + comments.write_json(os.path.join(self.state, "trusted.json"), [comments.sender_hash("ann@example.org")]) + self.deliver(mail()) + self.ingest() + self.assertEqual(self.pending(), []) + self.assertEqual(json.load(open(self.pub()))[0]["text"], "Nice post.") + + def test_second_run_is_noop(self): + self.deliver(mail()) + self.deliver(mail(subject="nope")) + self.ingest() + before = (self.pending(), open(os.path.join(self.state, "rejected.log")).read()) + code, out, _ = self.ingest() + self.assertEqual((code, out), (0, "")) + self.assertEqual((self.pending(), open(os.path.join(self.state, "rejected.log")).read()), before) + + def test_pending_cap(self): + for _ in range(comments.PENDING_CAP): + self.add_pending() + self.deliver(mail()) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (comments.PENDING_CAP, 1)) + self.assertIn("queue full", open(os.path.join(self.state, "rejected.log")).read()) + + def test_multiple_from_headers_rejected(self): + self.deliver(mail(extra_headers=("From: Bob <bob@example.org>",))) + self.deliver(mail(sender="Ann <ann@example.org>, Bob <bob@example.org>")) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (0, 2)) + + def test_dmarc_pass_needs_matching_header_from(self): + self.deliver(mail(auth=("mx; dmarc=pass header.from=evil.test",))) + self.deliver(mail(auth=("mx; dmarc=pass",))) + self.deliver(mail(auth=('mx; dmarc=pass header.from="Example.org"',))) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (1, 2)) + + def test_quoted_semicolon_does_not_fake_a_result(self): + self.deliver(mail(auth=('mx; dmarc=fail; spf=pass smtp.mailfrom="x;dmarc=pass header.from=example.org@evil.test"',))) + self.deliver(mail(auth=('mx (a;dmarc=pass header.from=example.org); dmarc=fail',))) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (0, 2)) + + def test_worst_result_of_a_method_wins(self): + self.deliver(mail(auth=("mx; dmarc=pass header.from=example.org; dmarc=fail header.from=example.org",))) + self.deliver(mail(auth=("mx; dkim=pass header.d=example.org; dkim=fail header.d=other.test",))) + self.deliver(mail(auth=("mx; dkim=pass header.d=example.org; dkim=pass header.d=other.test",))) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (1, 2)) + + def test_crash_between_store_and_move_does_not_duplicate(self): + self.deliver(mail()) + real = comments.MaildirSource.finish + def crash(*a, **k): + raise RuntimeError("crash") + comments.MaildirSource.finish = crash + try: + with self.assertRaises(RuntimeError): + self.ingest() + finally: + comments.MaildirSource.finish = real + self.assertEqual((len(self.pending()), self.count("")), (1, 1)) + self.ingest() + self.assertEqual((len(self.pending()), self.count(""), self.count(".done")), (1, 0, 1)) + self.assertIn("duplicate", open(os.path.join(self.state, "rejected.log")).read()) + + def test_crash_after_approval_does_not_duplicate_approved(self): + comments.write_json(os.path.join(self.state, "trusted.json"), [comments.sender_hash("ann@example.org")]) + self.deliver(mail()) + real = comments.MaildirSource.finish + comments.MaildirSource.finish = lambda *a: (_ for _ in ()).throw(RuntimeError("crash")) + try: + with self.assertRaises(RuntimeError): + self.ingest() + finally: + comments.MaildirSource.finish = real + self.ingest() + self.assertEqual(len(json.load(open(self.pub()))), 1) + self.assertEqual(self.count(""), 0) + + def test_overlapping_run_is_a_noop(self): + self.deliver(mail()) + self.deliver(mail()) + real = comments.MaildirSource.finish + comments.MaildirSource.finish = lambda *a: None # messages stay: as if another run held them + try: + self.ingest() + finally: + comments.MaildirSource.finish = real + self.assertEqual(len(self.pending()), 2) + self.ingest() + self.assertEqual((len(self.pending()), self.count("")), (2, 0)) + + def test_rejected_mail_moved_back_is_retried(self): + self.set_cfg(authserv_id="other") + self.deliver(mail()) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (0, 1)) + self.set_cfg(authserv_id="mx") + bad = mailbox.Maildir(self.maildir + ".bad") + for k in bad.keys(): + self.deliver(bad.get_bytes(k)) + bad.remove(k) + self.ingest() + self.assertEqual((len(self.pending()), self.count("")), (1, 0)) + + def test_sender_chosen_message_id_cannot_block_other_mail(self): + self.deliver(mail(body="one\n", extra_headers=("X-Test: 1",))) + self.ingest() + self.deliver(mail(body="two\n")) + self.ingest() + self.assertEqual(len(self.pending()), 2) + + def test_keys_pruned_when_comment_removed(self): + comments.write_json(os.path.join(self.state, "trusted.json"), [comments.sender_hash("ann@example.org")]) + self.deliver(mail()) + self.ingest() + cid = json.load(open(self.pub()))[0]["id"] + self.assertEqual(len(json.load(open(os.path.join(self.state, "approved-keys.json")))), 1) + self.run_cli("remove", cid) + self.deliver(mail(body="x\n")) + self.ingest() + self.assertEqual(len(json.load(open(os.path.join(self.state, "approved-keys.json")))), 1) + + def test_publish_failure_for_trusted_sender_stores_once(self): + comments.write_json(os.path.join(self.state, "trusted.json"), [comments.sender_hash("ann@example.org")]) + self.deliver(mail()) + real = comments.publish_path + def boom(*a): + raise OSError("disk full") + comments.publish_path = boom + try: + for _ in range(3): + self.assertEqual(self.ingest()[0], 0) + finally: + comments.publish_path = real + self.assertEqual(len(json.load(open(os.path.join(self.state, "approved", "writing.json")))[POST]), 1) + self.assertEqual(self.count(""), 0) + self.run_cli("publish") + self.assertEqual(len(json.load(open(self.pub()))), 1) + + def test_state_error_leaves_mail_in_inbox(self): + os.makedirs(os.path.join(self.state, "secret")) # unreadable as a file + self.deliver(mail()) + code, _, err = self.ingest() + self.assertEqual(code, 1) + self.assertEqual((self.count(""), self.count(".bad"), self.count(".done")), (1, 0, 0)) + + def test_quiet_when_no_maildir_even_without_authserv_id(self): + self.set_cfg(authserv_id="") + self.assertEqual(self.ingest(), (0, "", "")) + + def test_rejected_log_rotates(self): + os.makedirs(self.state) + log = os.path.join(self.state, "rejected.log") + open(log, "w").write("x" * 1_000_001) + comments.log_rejection("id", "why") + self.assertTrue(os.path.exists(log + ".1")) + self.assertEqual(len(open(log).read().splitlines()), 1) + + def test_percent_encoded_slug_maps_to_decoded_folder(self): + folder = os.path.join(self.root, "2024", "05", "01", "café") + os.makedirs(folder) + open(os.path.join(folder, "index.html"), "w").close() + self.deliver(mail(subject="[writing:/2024/05/01/caf%c3%a9/]")) + self.deliver(mail(subject="[writing:/2024/05/01/caf%e9/]")) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (1, 1)) + cid = self.pending()[0] + self.assertEqual(json.load(open(comments.pending_path(cid)))["path"], "/2024/05/01/caf%C3%A9/") + self.assertIn("invalid post path", open(os.path.join(self.state, "rejected.log")).read()) + self.run_cli("approve", cid) + self.assertEqual(len(json.load(open(self.pub("2024/05/01/café.json")))), 1) + self.assertEqual(os.listdir(os.path.join(self.root, "_comments", "2024", "05", "01")), ["café.json"]) + + def test_encoded_traversal_rejected(self): + self.deliver(mail(subject="[writing:/%2e%2e/%2E%2e/etc/]")) + self.deliver(mail(subject="[writing:/a%2f..%2fb/]")) + self.deliver(mail(subject="[writing:/a%5cb/]")) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (0, 3)) + + def test_sender_limit(self): + for i in range(comments.SENDER_LIMIT + 2): + self.deliver(mail(body=f"c{i}\n")) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (comments.SENDER_LIMIT, 2)) + self.assertIn("sender limit", open(os.path.join(self.state, "rejected.log")).read()) + + def test_sender_limit_counts_trusted_and_approved(self): + comments.write_json(os.path.join(self.state, "trusted.json"), [comments.sender_hash("ann@example.org")]) + for i in range(comments.SENDER_LIMIT + 1): + self.deliver(mail(body=f"c{i}\n")) + self.ingest() + self.assertEqual(len(json.load(open(self.pub()))), comments.SENDER_LIMIT) + self.assertEqual(self.count(".bad"), 1) + + def test_format_characters_stripped(self): + self.deliver(mail(sender="Ann\u202e <ann@example.org>", body="a\u200bb\u2028c\u2066d\n")) + self.ingest() + c = json.load(open(comments.pending_path(self.pending()[0]))) + self.assertEqual((c["text"], c["author"]), ("abcd", "Ann")) + + def test_sender_hash_is_salted(self): + h = comments.sender_hash("ann@example.org") + import hashlib + self.assertNotEqual(h, hashlib.sha256(b"ann@example.org").hexdigest()) + self.assertEqual(h, comments.sender_hash(" ANN@example.org")) + self.assertEqual(os.stat(os.path.join(self.state, "secret")).st_mode & 0o777, 0o600) + + def test_unreadable_state_file_reported(self): + os.makedirs(self.state) + open(os.path.join(self.state, "trusted.json"), "w").close() + self.deliver(mail()) + code, _, err = self.ingest() + self.assertEqual(code, 1) + self.assertIn("trusted.json", err) + + def test_missing_maildir_is_fine(self): + self.assertEqual(self.ingest()[0], 0) + + def test_approve_after_ingest_then_trust_flow(self): + self.deliver(mail()) + self.ingest() + cid = self.pending()[0] + self.run_cli("approve", cid, "--trust") + self.deliver(mail(body="Second\n")) + self.ingest() + self.assertEqual(self.pending(), []) + self.assertEqual(len(json.load(open(self.pub()))), 2) + self.run_cli("untrust", cid) + self.deliver(mail(body="Third\n")) + self.ingest() + self.assertEqual(len(self.pending()), 1) + + +PASSWORD = "s3cret-pw-xyz" + + +class FakeServer: + """One IMAP account shared by all connections: folder name -> list of [uid, raw, deleted].""" + + def __init__(self): + self.folders = {"INBOX": []} + self.next_uid = 1 + self.instances = [] + self.body_fetches = 0 + self.ssl_context = None + self.fail_login = False + self.prefix = "" # namespace prefix the server demands for new folders, e.g. "INBOX." + self.namespace = True # answer NAMESPACE (else only LIST shows the delimiter) + self.uidplus = False + self.fail = {} # operation -> number of times it still fails: copy, store, expunge, create + self.refuse_fetch = set() # UIDs whose body fetch answers NO + self.expunged_by_uid = 0 + self.literal_folder = False + + def add(self, raw, folder="INBOX"): + self.folders.setdefault(folder, []).append([self.next_uid, raw, False]) + self.next_uid += 1 + + def raws(self, folder): + return [m[1] for m in self.folders.get(folder, [])] + + +class FakeIMAP: + def __init__(self, server, host, port, ssl_context=None, timeout=None): + self.s, self.host, self.port, self.timeout = server, host, port, timeout + server.ssl_context = ssl_context + self.logged_out = False + self.folder = None + server.instances.append(self) + + @property + def capabilities(self): + return ("IMAP4REV1",) # as imaplib: fixed at connect time, never refreshed after login + + def capability(self): + caps = b"IMAP4REV1 UIDPLUS" if self.s.uidplus else b"IMAP4REV1" + return "OK", [caps] + + def _fails(self, op): + if self.s.fail.get(op, 0) > 0: + self.s.fail[op] -= 1 + return True + return False + + def list(self, directory='""', pattern="*"): + out = [f'(\\HasNoChildren) "." "{name}"'.encode() for name in self.s.folders] + if self.s.literal_folder: + out.append((b'(\\HasNoChildren) "." {11}', b"INBOX.Weird\xc3\xa9")) + return "OK", out + + def namespace(self): + if not self.s.namespace: + return "NO", [b"not supported"] + return "OK", [f'(("{self.s.prefix}" ".")) NIL NIL'.encode()] + + def login(self, user, password): + if self.s.fail_login: + raise comments.imaplib.IMAP4.error(f"login failed for {user} {password}") + self.user = user + + def select(self, folder, readonly=False): + self.folder = folder + return "OK", [b"1"] + + def response(self, name): + return name, [b"777"] + + def create(self, name): + if self._fails("create") or (self.s.prefix and not name.startswith(self.s.prefix)): + return "NO", [b"[CANNOT] namespace prefix required"] + self.s.folders.setdefault(name, []) + return "OK", [b""] + + def logout(self): + self.logged_out = True + return "BYE", [b""] + + def expunge(self): + if self._fails("expunge"): + return "NO", [b"expunge failed"] + box = self.s.folders[self.folder] + box[:] = [m for m in box if not m[2]] + return "OK", [b""] + + def uid(self, cmd, *args): + box = self.s.folders[self.folder] + if cmd == "SEARCH": + assert args[1] == "UNDELETED" + return "OK", [" ".join(str(m[0]) for m in box if not m[2]).encode()] + if cmd == "FETCH": + wanted = {int(u) for u in args[0].split(",")} + found = [m for m in box if m[0] in wanted] + if args[1] == "(RFC822.SIZE)": + return "OK", [f"{i} (UID {m[0]} RFC822.SIZE {len(m[1])})".encode() for i, m in enumerate(found, 1)] + if not found: + return "OK", [None] + m = found[0] + if m[0] in self.s.refuse_fetch: + return "NO", [b"[SERVERBUG] cannot fetch"] + if args[1] == "(BODY.PEEK[HEADER])": + return "OK", [(b"1 (UID 1 BODY[HEADER] {1}", m[1].split(b"\n\n", 1)[0] + b"\n\n"), b")"] + self.s.body_fetches += 1 + return "OK", [(b"1 (UID 1 BODY[] {1}", m[1]), b")"] + if cmd == "COPY": + if self._fails("copy"): + return "NO", [b"copy failed"] + if args[1] not in self.s.folders: + return "NO", [b"[TRYCREATE]"] + m = next(m for m in box if m[0] == int(args[0])) + self.s.add(m[1], args[1]) + return "OK", [b""] + if cmd == "EXPUNGE": + assert self.s.uidplus + if self._fails("expunge"): + return "NO", [b"expunge failed"] + self.s.expunged_by_uid += 1 + box[:] = [m for m in box if not (m[2] and m[0] == int(args[0]))] + return "OK", [b""] + if cmd == "STORE": + if self._fails("store"): + return "NO", [b"store failed"] + for m in box: + if m[0] == int(args[0]): + m[2] = True + return "OK", [b""] + raise AssertionError(cmd) + + +class IMAPBase(Base): + def setUp(self): + super().setUp() + self.server = FakeServer() + self.cfgpath = os.path.join(self.tmp, "imap.json") + os.environ["FW_COMMENTS_IMAP"] = self.cfgpath + self.addCleanup(os.environ.pop, "FW_COMMENTS_IMAP", None) + self.write_cfg() + real = comments.imaplib.IMAP4_SSL + comments.imaplib.IMAP4_SSL = lambda host, port=993, **kw: FakeIMAP(self.server, host, port, **kw) + self.addCleanup(setattr, comments.imaplib, "IMAP4_SSL", real) + + def write_cfg(self, mode=0o600, **override): + cfg = {"host": "imap.example.net", "user": "c@example.net", "password": PASSWORD, "folder": "INBOX"} + cfg.update(override) + cfg = {k: v for k, v in cfg.items() if v is not None} + with open(self.cfgpath, "w") as f: + json.dump(cfg, f) + os.chmod(self.cfgpath, mode) + + +class IMAPIngest(IMAPBase): + def test_end_to_end(self): + self.server.add(mail()) + self.server.add(mail(subject="no tag")) + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 0, err) + self.assertEqual(len(self.pending()), 1) + self.assertEqual((len(self.server.raws("INBOX")), len(self.server.raws("comments-done")), len(self.server.raws("comments-bad"))), (0, 1, 1)) + self.assertTrue(all(i.logged_out for i in self.server.instances)) + self.assertEqual((self.server.instances[0].host, self.server.instances[0].port, self.server.instances[0].timeout), + ("imap.example.net", 993, 30)) + self.assertEqual(self.server.ssl_context.verify_mode, comments.ssl.CERT_REQUIRED) + self.assertEqual(self.run_cli("ingest")[1], "") # second run: nothing to do + + def test_insecure_file_mode_refused(self): + self.write_cfg(mode=0o644) + self.server.add(mail()) + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 1) + self.assertIn("readable", err) + self.assertNotIn(PASSWORD, out + err) + self.assertEqual(len(self.server.raws("INBOX")), 1) + self.assertEqual(self.server.instances, []) + + def test_missing_keys_and_empty_password_refused(self): + for bad in ({"host": None}, {"user": None}, {"password": None}, {"password": ""}): + self.write_cfg(**bad) + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 1, bad) + self.assertNotIn(PASSWORD, out + err) + + def test_password_never_in_output_on_login_failure(self): + self.server.fail_login = True + self.server.add(mail()) + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 1) + self.assertNotIn(PASSWORD, out + err) + self.assertTrue(all(i.logged_out for i in self.server.instances)) + + def test_finish_copies_creates_folder_and_expunges(self): + self.server.add(mail()) + src = comments.IMAPSource() + try: + key = src.keys()[0] + raw = src.read(key) + src.finish(key, raw, "done") + finally: + src.close() + self.assertEqual((self.server.raws("INBOX"), self.server.raws("comments-done")), ([], [raw])) + self.assertEqual(self.server.body_fetches, 1) + + def test_read_of_vanished_message_is_keyerror(self): + self.server.add(mail()) + src = comments.IMAPSource() + try: + key = src.keys()[0] + self.server.folders["INBOX"].clear() + with self.assertRaises(KeyError): + src.read(key) + with self.assertRaises(KeyError): + src.read("1:1") # other UIDVALIDITY + finally: + src.close() + + def test_oversize_skipped_without_download(self): + self.server.add(mail(body="x" * (300 * 1024))) + self.server.add(mail()) + self.run_cli("ingest") + self.assertEqual(self.server.body_fetches, 1) + self.assertEqual((len(self.pending()), len(self.server.raws("comments-bad")), len(self.server.raws("INBOX"))), (1, 1, 0)) + self.assertIn("over 256 KB", open(os.path.join(self.state, "rejected.log")).read()) + + def test_missing_config_is_quiet(self): + os.unlink(self.cfgpath) + self.assertEqual(self.run_cli("ingest"), (0, "", "")) + self.assertEqual(self.server.instances, []) + + def test_imap_check_prints_only_names_and_tokens(self): + self.server.add(mail(sender="Ann Secret <ann@example.org>", subject="Private subject [writing:/x/]", + auth=("mx (comment); dkim=pass header.d=example.org; dmarc=pass header.from=example.org",))) + code, out, err = self.run_cli("imap-check") + self.assertEqual(code, 0, err) + self.assertIn("messages in INBOX: 1", out) + self.assertIn("Authentication-Results: authserv-id mx; dkim=pass dmarc=pass", out) + for secret in ("Secret", "ann@", "Private", "example.org", PASSWORD, "Nice post"): + self.assertNotIn(secret, out + err) + self.assertTrue(self.server.instances[0].logged_out) + + +class IMAPRobustness(IMAPBase): + def run_twice(self): + self.run_cli("ingest") + return self.run_cli("ingest") + + def test_partial_failure_leaves_mail_and_next_run_finishes(self): + for op in ("copy", "store", "expunge"): + with self.subTest(op=op): + self.setUp() + self.server.add(mail()) + self.server.fail[op] = 5 # the first move attempts fail + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 1, op) + self.assertNotIn(PASSWORD, out + err) + self.assertEqual(len(self.pending()), 1) # stored once + self.server.fail[op] = 0 + self.run_cli("ingest") + self.assertEqual(len(self.pending()), 1) + inbox_live = [m for m in self.server.folders["INBOX"] if not m[2]] + self.assertEqual(inbox_live, []) + self.assertEqual(len(self.server.raws("comments-done")) >= 1, True) + + def test_namespace_prefix_used_for_folders(self): + self.server.prefix = "INBOX." + self.server.folders = {"INBOX": [], "INBOX.Sent": []} + self.server.add(mail()) + self.server.add(mail(subject="no tag")) + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 0, err) + self.assertEqual((len(self.server.raws("INBOX.comments-done")), len(self.server.raws("INBOX.comments-bad"))), (1, 1)) + self.assertNotIn("comments-done", self.server.folders) + + def test_namespace_from_list_when_namespace_command_missing(self): + self.server.prefix = "INBOX." + self.server.namespace = False + self.server.folders = {"INBOX": [], "INBOX.Sent": []} + self.server.add(mail()) + self.assertEqual(self.run_cli("ingest")[0], 0) + self.assertEqual(len(self.server.raws("INBOX.comments-done")), 1) + + def test_create_failure_reported_and_mail_kept(self): + self.server.prefix = "INBOX." + self.server.namespace = False + self.server.folders = {"INBOX": []} # no hint of the prefix anywhere: the wrong guess fails + self.server.add(mail()) + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 1) + self.assertIn("create folder comments-done", err) + self.assertIn("namespace prefix required", err) + self.assertNotIn(PASSWORD, out + err) + self.assertEqual(len(self.server.raws("INBOX")), 1) + + def test_folder_overrides(self): + self.write_cfg(done_folder="INBOX.mine-done", bad_folder="INBOX.mine-bad") + self.server.add(mail()) + self.server.folders["INBOX.mine-done"] = [] + self.run_cli("ingest") + self.assertEqual(len(self.server.raws("INBOX.mine-done")), 1) + self.write_cfg(done_folder="bad folder") + self.assertEqual(self.run_cli("ingest")[0], 1) + + def test_deleted_flagged_message_skipped(self): + self.server.add(mail()) + self.server.folders["INBOX"][0][2] = True + self.assertEqual(self.run_cli("ingest")[0], 0) + self.assertEqual(self.pending(), []) + + def test_uid_expunge_with_uidplus(self): + self.server.uidplus = True + self.server.add(mail()) + self.run_cli("ingest") + self.assertEqual((self.server.expunged_by_uid, self.server.raws("INBOX")), (1, [])) + + def test_refused_fetch_does_not_block_others(self): + self.server.add(mail(body="first\n")) + self.server.add(mail(body="second\n")) + self.server.refuse_fetch = {1} + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 0, err) + self.assertEqual(len(self.pending()), 1) + self.assertEqual(len(self.server.raws("INBOX")), 1) # the refused one stays + self.assertIn("skipped", open(os.path.join(self.state, "rejected.log")).read()) + + def test_non_ascii_credentials_refused_without_echo(self): + self.write_cfg(password="pässword-ünique") + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 1) + self.assertIn("ASCII", err) + self.assertNotIn("pässword", out + err) + + def test_imap_check_lists_folders_and_namespace(self): + self.server.prefix = "INBOX." + self.server.folders = {"INBOX": [], "INBOX.Sent": []} + self.server.add(mail()) + code, out, err = self.run_cli("imap-check") + self.assertIn("folders: INBOX, INBOX.Sent", out) + self.assertIn('namespace: (("INBOX." "."))', out) + + def test_list_with_literal_folder_name(self): + self.server.literal_folder = True + self.server.namespace = False + self.server.prefix = "INBOX." + self.server.folders = {"INBOX": [], "INBOX.Sent": []} + self.server.add(mail()) + code, out, err = self.run_cli("imap-check") + self.assertEqual(code, 0, err) + self.assertIn("INBOX.Weird", out) + self.run_cli("ingest") + self.assertEqual(len(self.server.raws("INBOX.comments-done")), 1) + + def test_unparseable_namespace_falls_back_to_empty_prefix(self): + self.server.add(mail()) + real = FakeIMAP.namespace + FakeIMAP.namespace = lambda self: ("OK", [(b"weird", b"tuple")]) + self.addCleanup(setattr, FakeIMAP, "namespace", real) + self.assertEqual(self.run_cli("ingest")[0], 0) + self.assertEqual(len(self.server.raws("comments-done")), 1) + + def test_stale_snapshot_cannot_double_store(self): + import fcntl + import threading + raw = mail() + self.server.add(raw) + os.makedirs(self.state, exist_ok=True) + lock = open(os.path.join(self.state, "ingest.lock"), "w") + fcntl.flock(lock, fcntl.LOCK_EX) # "another run" is in progress + t = threading.Thread(target=self.run_cli, args=("ingest",)) + t.start() + t.join(0.5) + self.assertTrue(t.is_alive()) # the second run waits for the job lock + # the other run finishes: stores the comment (its record carries the message key) but leaves the mail + import hashlib + self.add_pending(msg_key=hashlib.sha256(raw).hexdigest()) + fcntl.flock(lock, fcntl.LOCK_UN) + t.join(5) + self.assertFalse(t.is_alive()) + self.assertEqual(len(self.pending()), 1) # not stored twice + self.assertEqual(len(self.server.raws("comments-done")), 1) + + +class MalformedSubjects(Base): + def test_bad_subjects_go_to_bad_and_do_not_abort_the_run(self): + subjects = ["[writing:/a" + "%" * 290 + "/]", "[writing:/" + "a/" * 200 + "]", "[writing:/%/]", "[writing:/%zz/]", + "[writing:/%ff%fe/]", "[writing:/a%00b/]", "[writing:/\u00e9/]", "[nosuchsite:/x/]", "[writing:]", + "[writing:/" + "%25" * 120 + "/]"] + for sub in subjects: + self.deliver(mail(subject=sub)) + self.deliver(mail(subject="Good [writing:/2024/05/01/hello/]")) + code, out, err = self.run_cli("ingest") + self.assertEqual(code, 0, err) + self.assertEqual((len(self.pending()), self.count(".bad"), self.count("")), (1, len(subjects), 0)) + + +class Canonical(Base): + def test_equivalent_spellings_share_one_key(self): + cases = [("/sl%75g/", "/slug/"), ("/caf%c3%a9/", "/caf%C3%A9/"), ("/a%2db/", "/a-b/"), ("/x%7e/", "/x~/")] + for a, b in cases: + self.assertEqual(comments.canonical_path(a), comments.canonical_path(b), (a, b)) + self.assertEqual(comments.canonical_path("/caf%c3%a9/"), "/caf%C3%A9/") + + def test_ingest_collapses_spellings_and_publishes_one_file(self): + for d in ("slug", "café"): + os.makedirs(os.path.join(self.root, d)) + open(os.path.join(self.root, d, "index.html"), "w").close() + for subject in ("[writing:/sl%75g/]", "[writing:/slug/]", "[writing:/caf%c3%a9/]", "[writing:/caf%C3%A9/]"): + self.deliver(mail(subject=subject, body=subject + "\n")) + comments.write_json(os.path.join(self.state, "trusted.json"), [comments.sender_hash("ann@example.org")]) + self.run_cli("ingest") + approved = json.load(open(os.path.join(self.state, "approved", "writing.json"))) + self.assertEqual(sorted(approved), ["/caf%C3%A9/", "/slug/"]) + self.assertEqual(len(approved["/slug/"]), 2) + self.assertEqual(len(json.load(open(self.pub("slug.json")))), 2) + self.assertEqual(len(json.load(open(self.pub("café.json")))), 2) + + def test_existing_duplicate_spellings_are_merged_on_publish_and_remove(self): + mk = lambda cid, path, date: {"id": cid, "date": date, "author": "A", "parent": None, "text": cid} + approved = {"/slug/": [mk("20240101T000000Z-aaaaaa", "", "2024-01-01T00:00:00Z")], + "/sl%75g/": [mk("20240102T000000Z-bbbbbb", "", "2024-01-02T00:00:00Z")]} + comments.write_json(os.path.join(self.state, "approved", "writing.json"), approved) + self.run_cli("publish", "writing") + self.assertEqual([c["id"][-6:] for c in json.load(open(self.pub("slug.json")))], ["aaaaaa", "bbbbbb"]) + self.run_cli("remove", "20240101T000000Z-aaaaaa") + self.assertEqual([c["id"][-6:] for c in json.load(open(self.pub("slug.json")))], ["bbbbbb"]) + self.run_cli("remove", "20240102T000000Z-bbbbbb") + self.assertFalse(os.path.exists(self.pub("slug.json"))) + + +class MailChannelsHeaders(Base): + def ingest(self): + return self.run_cli("ingest") + + def set_cfg(self, **kw): + cfg = json.load(open(os.environ["FW_COMMENTS_SITES"])) + cfg.update(kw) + json.dump(cfg, open(os.environ["FW_COMMENTS_SITES"], "w")) + + REAL = "inbound.mailchannels.net; spf=pass dkim=pass header.d=example.org; dmarc=pass header.from=example.org" + + def setUp(self): + super().setUp() + self.set_cfg(authserv_id="inbound.mailchannels.net") + + def test_only_the_real_authentication_results_counts(self): + arc_above = ("ARC-Seal: i=1; a=rsa-sha256; d=mailchannels.net; s=arc; cv=none; b=abc", + "ARC-Authentication-Results: i=1; inbound.mailchannels.net; dmarc=pass header.from=example.org") + # real header on top, ARC headers around it: accepted + self.deliver(mail(auth=(self.REAL,), extra_headers=arc_above)) + # a forged "ARC-Authentication-Results: <our id>; dmarc=pass" with a failing real header: rejected + self.deliver(mail(auth=("inbound.mailchannels.net; dkim=fail; dmarc=fail",), extra_headers=arc_above)) + # no real header at all, only the ARC copy carrying our id: rejected + raw = mail(auth=()).replace(b"From:", b"ARC-Authentication-Results: i=1; inbound.mailchannels.net; dmarc=pass header.from=example.org\nFrom:", 1) + self.deliver(raw) + # real header below a forged foreign one: rejected (topmost is not ours) + self.deliver(mail(auth=("evil.test; dmarc=pass header.from=example.org", self.REAL))) + self.ingest() + self.assertEqual((len(self.pending()), self.count(".bad")), (1, 3)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_comments_php.py b/tests/test_comments_php.py new file mode 100644 index 0000000..589e180 --- /dev/null +++ b/tests/test_comments_php.py @@ -0,0 +1,423 @@ +"""Tests for server/comments/post.php (the off-by-default comment form endpoint). + +Skipped unless `php` is installed. Each test starts `php -S 127.0.0.1:<port>` with a temporary +state directory, sites.json and site root, and talks to it with urllib. + +Run: python3 -m unittest tests.test_comments_php (or python3 -m unittest discover tests) +""" + +import hashlib +import hmac +import json +import os +import shutil +import socket +import subprocess +import sys +import tempfile +import time +import unittest +import urllib.error +import urllib.parse +import urllib.request + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +POST_PHP = os.path.join(ROOT, "server", "comments", "post.php") +COMMENTS_PY = os.path.join(ROOT, "server", "comments.py") +POST_PATH = "/2024/05/01/slug/" +ORIGIN = "https://writing.example" +SECRET = "test-secret-0123456789" + + +class NoRedirect(urllib.request.HTTPRedirectHandler): + def redirect_request(self, *args, **kwargs): + return None + + +def free_port(): + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +@unittest.skipUnless(shutil.which("php"), "php is not installed") +class PostPhpTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp() + self.state = os.path.join(self.tmp, "state") + self.docroot = os.path.join(self.tmp, "web") + self.siteroot = os.path.join(self.tmp, "writing") + os.makedirs(self.state) + os.makedirs(self.docroot) + os.makedirs(self.siteroot + POST_PATH) + with open(self.siteroot + POST_PATH + "index.html", "w") as f: + f.write("<html></html>") + with open(os.path.join(self.state, "secret"), "w") as f: + f.write(SECRET + "\n") + self.sites = os.path.join(self.tmp, "sites.json") + with open(self.sites, "w") as f: + json.dump({"writing": {"origin": ORIGIN, "root": self.siteroot}, + "mailbox": "", "authserv_id": ""}, f) + shutil.copy(POST_PHP, self.docroot) + self.port = free_port() + self.self_origin = f"http://127.0.0.1:{self.port}" + self.env = dict(os.environ, FW_COMMENTS_DIR=self.state, FW_COMMENTS_SITES=self.sites, + FW_COMMENTS_SELF_ORIGIN=self.self_origin) + self.server = subprocess.Popen( + ["php", "-S", f"127.0.0.1:{self.port}", "-t", self.docroot], + env=self.env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + for _ in range(100): + try: + socket.create_connection(("127.0.0.1", self.port), timeout=0.2).close() + break + except OSError: + time.sleep(0.05) + self.opener = urllib.request.build_opener(NoRedirect) + + def tearDown(self): + self.server.terminate() + self.server.wait() + shutil.rmtree(self.tmp, ignore_errors=True) + + # ---- helpers ---- + + def request(self, method="POST", fields=None, headers=None): + url = f"http://127.0.0.1:{self.port}/post.php" + data = urllib.parse.urlencode(fields).encode() if fields is not None else None + req = urllib.request.Request(url, data=data, method=method, headers=headers or {}) + try: + r = self.opener.open(req) + except urllib.error.HTTPError as e: + r = e + with r: + return r.status, r.headers, r.read().decode("utf-8", "replace") + + def post(self, headers=None, **over): + fields = {"site": "writing", "path": POST_PATH, "elapsed": "5000", "website": "", + "name": "Ann", "text": "Nice post."} + fields.update(over) + fields = {k: v for k, v in fields.items() if v is not None} + return self.request("POST", fields, headers) + + def pending(self): + d = os.path.join(self.state, "pending") + return sorted(os.path.join(d, n) for n in os.listdir(d)) if os.path.isdir(d) else [] + + def load(self, path): + with open(path, encoding="utf-8") as f: + return json.load(f) + + def sign(self, ts, site, path, name, text): + msg = f"{ts}|{site}|{path}|{hashlib.sha256(name.encode()).hexdigest()}|{hashlib.sha256(text.encode()).hexdigest()}" + return hmac.new(SECRET.encode(), msg.encode(), hashlib.sha256).hexdigest() + + def assert_ok(self, resp): + status, headers, _ = resp + self.assertEqual(status, 303) + self.assertEqual(headers["Location"], ORIGIN + POST_PATH + "#comment-pending") + + def assert_refused(self, resp, code=None): + status, _, _ = resp + self.assertTrue(status >= 400 if code is None else status == code, status) + self.assertEqual(self.pending(), []) + + # ---- tests ---- + + def test_get_refused(self): + status, headers, body = self.request("GET") + self.assertEqual(status, 405) + self.assertEqual(headers["Allow"], "POST") + self.assertEqual(self.pending(), []) + + def test_security_headers(self): + _, headers, _ = self.request("GET") + self.assertEqual(headers["X-Content-Type-Options"], "nosniff") + self.assertEqual(headers["Cache-Control"], "no-store") + self.assertIn("default-src 'none'", headers["Content-Security-Policy"]) + self.assertIn("text/html; charset=utf-8", headers["Content-Type"]) + self.assertIsNone(headers["Set-Cookie"]) + + def test_bad_origin(self): + self.assert_refused(self.post(headers={"Origin": "https://evil.example"}), 403) + self.assert_refused(self.post(headers={"Origin": "null"}), 403) + self.assert_refused(self.post(headers={"Origin": ORIGIN + ".evil.example"}), 403) + + def test_good_origin_and_none(self): + self.assert_ok(self.post(headers={"Origin": ORIGIN})) + self.assert_ok(self.post()) + self.assertEqual(len(self.pending()), 2) + + def test_unknown_site(self): + self.assert_refused(self.post(site="nope")) + self.assert_refused(self.post(site="mailbox")) + self.assert_refused(self.post(site="authserv_id")) + self.assert_refused(self.post(site="")) + + def test_traversal_paths(self): + for bad in ["/../", "/2024/../", "/%2e%2e/x/", "/2024/%2E%2E/", "/%2e/x/", "//", "/2024//05/", + "2024/05/01/slug/", "/2024/05/01/slug", "/2024\\05/01/slug/", "/2024/a%2Fb/", + "/2024/a%5Cb/", "/2024/a%00b/", "/2024/05/01/slug/\n", "/" + "a/" * 9, "/.hidden/", "", + "/etc/passwd/"]: + with self.subTest(path=bad): + self.assert_refused(self.post(path=bad)) + + def test_nonexistent_post(self): + self.assert_refused(self.post(path="/2024/05/01/missing/")) + + def test_missing_site_root(self): + shutil.rmtree(self.siteroot) + self.assert_refused(self.post()) + + def test_honeypot_stores_nothing(self): + self.assert_ok(self.post(website="http://spam.example")) + self.assertEqual(self.pending(), []) + + def test_too_fast_gives_confirm_page(self): + for elapsed in ("1000", "0", "", "abc", "2999", None): + with self.subTest(elapsed=elapsed): + status, _, body = self.post(elapsed=elapsed) + self.assertEqual(status, 200) + self.assertIn('name="sig"', body) + self.assertEqual(self.pending(), []) + + def test_confirm_flow_stores_one_comment(self): + ts = str(int(time.time()) - 10) + sig = self.sign(ts, "writing", POST_PATH, "Ann", "Nice post.") + self.assert_ok(self.request("POST", {"site": "writing", "path": POST_PATH, "name": "Ann", + "text": "Nice post.", "ts": ts, "sig": sig})) + files = self.pending() + self.assertEqual(len(files), 1) + c = self.load(files[0]) + self.assertRegex(c["id"], r"^\d{8}T\d{6}Z-[0-9a-f]{6}$") + self.assertEqual(os.path.basename(files[0]), c["id"] + ".json") + self.assertEqual({k: v for k, v in c.items() if k not in ("id", "date")}, + {"site": "writing", "path": POST_PATH, "author": "Ann", "text": "Nice post.", + "source": "form", "parent": None, "sender_hash": None}) + self.assertRegex(c["date"], r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$") + self.assertEqual(os.stat(files[0]).st_mode & 0o777, 0o600) + out = subprocess.run([sys.executable, COMMENTS_PY, "list"], env=self.env, capture_output=True, text=True) + self.assertEqual(out.returncode, 0, out.stderr) + self.assertIn(c["id"], out.stdout) + self.assertIn("[form]", out.stdout) + self.assertIn("Nice post.", out.stdout) + # form comments cannot be trusted, but can be approved + trust = subprocess.run([sys.executable, COMMENTS_PY, "approve", c["id"], "--trust"], env=self.env, + capture_output=True, text=True) + self.assertNotEqual(trust.returncode, 0) + ok = subprocess.run([sys.executable, COMMENTS_PY, "approve", c["id"]], env=self.env, + capture_output=True, text=True) + self.assertEqual(ok.returncode, 0, ok.stderr) + + def test_confirm_page_then_confirm_roundtrip(self): + status, _, body = self.post(elapsed="10") + self.assertEqual(status, 200) + fields = dict(urllib.parse.parse_qsl( + "&".join(f"{n}={urllib.parse.quote(v)}" for n, v in + __import__("re").findall(r'name="(\w+)" value="([^"]*)"', body)))) + fields = {k: __import__("html").unescape(v) for k, v in fields.items()} + time.sleep(3.2) + self.assert_ok(self.request("POST", fields, {"Origin": self.self_origin})) + self.assertEqual(len(self.pending()), 1) + + def test_bad_and_expired_sig(self): + now = int(time.time()) + base = {"site": "writing", "path": POST_PATH, "name": "Ann", "text": "Nice post."} + good = self.sign(str(now - 10), "writing", POST_PATH, "Ann", "Nice post.") + self.assert_refused(self.request("POST", dict(base, ts=str(now - 10), sig="0" * 64))) + self.assert_refused(self.request("POST", dict(base, ts=str(now - 10), sig=good, text="Other"))) + self.assert_refused(self.request("POST", dict(base, ts=str(now - 11), sig=good))) + for ts in (now - 4000, now, now + 100): + sig = self.sign(str(ts), "writing", POST_PATH, "Ann", "Nice post.") + self.assert_refused(self.request("POST", dict(base, ts=str(ts), sig=sig))) + self.assert_refused(self.request("POST", dict(base, ts="", sig="x"))) + + def test_length_caps(self): + self.assert_ok(self.post(name="n" * 60, text="t" * 4000)) + self.assertEqual(len(self.pending()), 1) + for p in self.pending(): + os.unlink(p) + self.assert_refused(self.post(name="n" * 61)) + self.assert_refused(self.post(text="t" * 4001)) + self.assert_refused(self.post(name="")) + self.assert_refused(self.post(text="")) + self.assert_refused(self.post(text=" \n ")) + self.assert_refused(self.post(name="\u200b\u202e")) + self.assert_ok(self.post(name="\u00e9" * 60)) # characters, not bytes + + def test_invalid_utf8(self): + body = b"site=writing&path=%2F2024%2F05%2F01%2Fslug%2F&elapsed=5000&website=&name=Ann&text=bad%ff%fe" + req = urllib.request.Request(f"http://127.0.0.1:{self.port}/post.php", data=body, method="POST") + with self.assertRaises(urllib.error.HTTPError) as cm: + self.opener.open(req) + self.assertEqual(cm.exception.code, 400) + cm.exception.close() + self.assertEqual(self.pending(), []) + + def test_array_fields_refused(self): + body = b"site=writing&path=%2F2024%2F05%2F01%2Fslug%2F&elapsed=5000&name%5B%5D=Ann&text=hi" + req = urllib.request.Request(f"http://127.0.0.1:{self.port}/post.php", data=body, method="POST") + with self.assertRaises(urllib.error.HTTPError) as cm: + self.opener.open(req) + cm.exception.close() + self.assertEqual(self.pending(), []) + + def test_at_sign_name(self): + self.assert_ok(self.post(name="me@example.com")) + self.assertEqual(self.load(self.pending()[0])["author"], "A reader") + + def test_control_and_bidi_stripped(self): + self.assert_ok(self.post(name="Bo\u202eb\x00\n x", text=" a\u202eb\u200bc\x00d\r\ne\u2028f\tg\x7f ")) + c = self.load(self.pending()[0]) + self.assertEqual(c["author"], "Bob x") # newline becomes a space, then the input's own space + self.assertEqual(c["text"], "abcd\nefg") + + def test_hostile_html_stored_raw_never_echoed(self): + name = "<script>alert(1)</script>" + text = "<img src=x onerror=alert(1)>\"'& </textarea>" + status, _, body = self.post(elapsed="1", name=name, text=text) + self.assertEqual(status, 200) + self.assertNotIn("<script>", body) + self.assertNotIn("<img", body) + self.assertNotIn("</textarea>", body) + self.assertIn("<script>", body) + self.assertIn("<img src=x", body) + for other in (self.post(path="/nope/", name=name, text=text), + self.post(site="<b>x</b>", name=name, text=text)): + self.assertNotIn("<b>", other[2]) + self.assertNotIn("<script>", other[2]) + self.assert_ok(self.post(name=name, text=text)) + c = self.load(self.pending()[0]) + self.assertEqual((c["author"], c["text"]), (name, text)) + + def test_rate_limit(self): + for _ in range(5): + self.assert_ok(self.post()) + status, _, _ = self.post() + self.assertEqual(status, 429) + self.assertEqual(len(self.pending()), 5) + rate_dir = os.path.join(self.state, "rate") + self.assertEqual(len(os.listdir(rate_dir)), 1) + self.assertNotIn("127.0.0.1", os.listdir(rate_dir)[0]) + + def test_pending_cap(self): + d = os.path.join(self.state, "pending") + os.makedirs(d) + for i in range(200): + with open(os.path.join(d, f"20240101T000000Z-{i:06x}.json"), "w") as f: + f.write("{}") + status, _, _ = self.post() + self.assertEqual(status, 503) + self.assertEqual(len(os.listdir(d)), 200) + + def make_post(self, rel): + os.makedirs(os.path.join(self.siteroot, rel), exist_ok=True) + with open(os.path.join(self.siteroot, rel, "index.html"), "w") as f: + f.write("<html></html>") + + def test_percent_encoded_slug_uses_decoded_folder(self): + self.make_post("2024/caf\u00e9") + self.assert_ok_path(self.post(path="/2024/caf%C3%A9/"), "/2024/caf%C3%A9/") + self.assertEqual(self.load(self.pending()[0])["path"], "/2024/caf%C3%A9/") + for p in self.pending(): + os.unlink(p) + self.assert_refused(self.post(path="/2024/caf%C3%A8/")) # decoded folder missing + self.assert_refused(self.post(path="/2024/cafe%CC%81/")) + + def assert_ok_path(self, resp, path): + self.assertEqual(resp[0], 303) + self.assertEqual(resp[1]["Location"], ORIGIN + path + "#comment-pending") + + def test_path_is_stored_canonical(self): + self.make_post("2024/caf\u00e9") + for spelled in ("/2024/05/01/sl%75g/", "/2024/05/01/s%6C%75g/"): + with self.subTest(path=spelled): + self.assert_ok_path(self.post(path=spelled), POST_PATH) + for spelled in ("/2024/caf%c3%a9/", "/2024/caf%C3%A9/", "/2024/caf%c3%A9/"): + with self.subTest(path=spelled): + self.assert_ok_path(self.post(path=spelled), "/2024/caf%C3%A9/") + self.assertEqual(sorted(self.load(p)["path"] for p in self.pending()), + [POST_PATH] * 2 + ["/2024/caf%C3%A9/"] * 3) + + def test_canonical_path_parity_with_comments_py(self): + sys.path.insert(0, os.path.join(ROOT, "server")) + try: + import comments + finally: + sys.path.pop(0) + tricky = ["/2024/05/01/sl%75g/", "/2024/caf%c3%a9/", "/2024/caf%C3%A9/", "/2024/a+b/", "/2024/a%2bb/", + "/~me/", "/_notes/", "/2024/caf%e9/", "/2024/a%20b/", "/2024/a%7eb/", "/2024/a%2Cb/", "/2024/a%25b/", + "/2024/a%41/", "/2024/x.y_z-w/", "/2024/%e2%82%ac/"] + checked = 0 + for p in tricky: + if not comments.valid_path(p): + n = len(self.pending()) + self.assertGreaterEqual(self.post(path=p)[0], 400) # also invalid here + self.assertEqual(len(self.pending()), n) + continue + expected = comments.canonical_path(p) + self.make_post(urllib.parse.unquote(expected.strip("/"))) + before = set(self.pending()) + with self.subTest(path=p): + self.assert_ok_path(self.post(path=p, name=f"n{checked}"), expected) + stored = [self.load(f)["path"] for f in set(self.pending()) - before] + self.assertEqual(stored, [expected]) + checked += 1 + if checked % 4 == 0: # stay under the rate limit + os.unlink(os.path.join(self.state, "rate", os.listdir(os.path.join(self.state, "rate"))[0])) + self.assertGreater(checked, 8) + + def test_underscore_and_tilde_segments(self): + self.make_post("_x/~u") + self.assert_ok_path(self.post(path="/_x/~u/"), "/_x/~u/") + self.make_post("2024/a+b") + self.assert_ok_path(self.post(path="/2024/a+b/"), "/2024/a+b/") + + def test_decoded_segment_rules(self): + for bad in ["/2024/a%2e%2eb/", "/2024/a%EF%BF%BDb/", "/2024/a%E2%80%AEb/", "/2024/%ff/", + "/2024/a%0Ab/", "/2024/a%C2%ADb/", "/2024/%2e/"]: + with self.subTest(path=bad): + self.assert_refused(self.post(path=bad)) + + def test_self_origin_allowed_but_not_by_default(self): + self.assert_ok(self.post(headers={"Origin": self.self_origin})) + + def test_double_submit_of_confirm_token_stores_two(self): + # There is no one-time nonce: a signed token can be replayed within an hour. + # Accepted trade-off (the rate limit and moderation bound the damage); this documents it. + ts = str(int(time.time()) - 10) + fields = {"site": "writing", "path": POST_PATH, "name": "Ann", "text": "Nice post.", "ts": ts, + "sig": self.sign(ts, "writing", POST_PATH, "Ann", "Nice post.")} + self.assert_ok(self.request("POST", fields)) + self.assert_ok(self.request("POST", fields)) + self.assertEqual(len(self.pending()), 2) + + def test_form_cap(self): + d = os.path.join(self.state, "pending") + os.makedirs(d) + for i in range(100): + with open(os.path.join(d, f"20240101T000000Z-{i:06x}.json"), "w") as f: + json.dump({"source": "email"}, f) + self.assert_ok(self.post()) # 100 email comments do not count + for i in range(100, 199): + with open(os.path.join(d, f"20240101T000000Z-{i:06x}.json"), "w") as f: + json.dump({"source": "form"}, f) + status, _, _ = self.post() # now 100 form files (99 + the one just stored) + self.assertEqual(status, 503) + + def test_json_indent_matches_comments_py(self): + self.assert_ok(self.post()) + with open(self.pending()[0], encoding="utf-8") as f: + raw = f.read() + self.assertIn('\n "author": "Ann",\n', raw) + self.assertTrue(raw.endswith("}\n")) + + def test_missing_secret_fails(self): + os.unlink(os.path.join(self.state, "secret")) + status, _, body = self.post() + self.assertEqual(status, 500) + self.assertNotIn("secret", body) + self.assertEqual(self.pending(), []) + + +if __name__ == "__main__": + unittest.main()