fuzzyworld

fuzzyworld-tools / files / server / comments.py

comments.py

41.5 KB · history · blame

1#!/usr/bin/env python3
2"""Blog comments by email, with a moderation CLI. Python standard library only.
3
4Usage (on the server: `fw comments <cmd>`; from the laptop: `bin/fw comments <cmd>`):
5
6 list [--json] pending comments (id, site, path, author, source, first 160 chars); --json (for `fw admin`):
7 {"pending": [full comments + post_url, trusted], "approved": [the 20 newest approved]}
8 show <id> one pending or approved comment in full
9 approve <id>... [--trust] publish pending comments; --trust also trusts the sender (email only)
10 reject <id>... discard pending comments (logged in rejected.log)
11 remove <id> delete an approved comment and republish its post
12 untrust <id> stop auto-approving the sender of that comment
13 publish [site] rewrite every per-post file from the approved store
14 export <site> print the approved store (path -> list of comments) as JSON
15 ingest read new mail from the comments mailbox (IMAP if imap.json exists, else Maildir)
16 imap-check connect to the IMAP mailbox and print what it holds (header names only)
17
18State, outside every web root (override the directory with FW_COMMENTS_DIR):
19 ~/fw/comments/pending/<id>.json waiting for moderation (may hold sender_hash, never an address)
20 ~/fw/comments/approved/<site>.json {path: [{author,date,id,parent,text}]}, public fields only
21 ~/fw/comments/approved-meta.json id -> sender_hash, so `untrust` works after approval; never published
22 ~/fw/comments/approved-keys.json message key (sha256 of the raw mail) -> comment id, so a mail whose comment is
23 already stored is never added twice; pruned on every ingest
24 ~/fw/comments/secret random salt for sender hashes (mode 600). Back it up: without it every
25 trusted.json entry and every untrust record stops matching.
26 ~/fw/comments/trusted.json salted sha256 (secret + lowercased sender address) hashes
27 ~/fw/comments/rejected.log "id date reason" per rejection (rotated to rejected.log.1 at 1 MB)
28 ~/fw/comments/lock held for every change to the store (taken per message by ingest)
29 ~/fw/comments/ingest.lock held for a whole ingest run, so a second run waits for the first
30Published: <site root>/_comments/<post path minus trailing slash>.json, e.g.
31 /2024/05/01/slug/ -> <root>/_comments/2024/05/01/slug.json (removed when the list is empty).
32
33Sites: server/comments/sites.json (override the path with FW_COMMENTS_SITES). Keys "writing" and
34"til" give origin and root; "mailbox" is the comments address (informational; may be empty);
35"authserv_id" is the authserv-id of the receiving mail server (see below).
36
37Email intake refuses to run until "authserv_id" is set in sites.json (fail closed). Maildir:
38~/Maildir/.comments (override with FW_COMMENTS_MAILDIR); processed mail moves to ~/Maildir/.comments.done,
39rejected mail to ~/Maildir/.comments.bad. The subject must
40contain [writing:/path/] or [til:/path/] ("Re:" prefixes are fine). Authentication uses ONLY the
41topmost Authentication-Results header, and only if its authserv-id equals the configured "authserv_id"
42(the id your mail provider puts in front of the first ";" of a real message). Headers lower in the
43message may have been added by the sender, so they are never searched: a missing or foreign topmost
44header rejects the message. The message needs exactly one From header with one address. Accepted when
45every dmarc result passes and its header.from equals the From domain, or every dkim result passes and
46one has header.d equal to, or a subdomain of, the From domain. Each sender (salted hash) may have at
47most 5 pending or approved comments per 24 hours (trusted senders too); the rest goes to .bad.
48The "topmost header" rule only holds if the receiving mail server always adds its own
49Authentication-Results header above anything the sender supplied; check that with a real message.
50Only stored comments are remembered: a rejected mail moved back from .bad is simply tried again.
51Ingest does nothing (quietly) when the mail source does not exist. The message source is a small class
52(MaildirSource, IMAPSource) with available(), keys(), read(key), finish(key, raw, "done"|"bad") and close().
53
54IMAP source. When ~/fw/comments/imap.json exists (override the path with FW_COMMENTS_IMAP) ingest reads that
55mailbox instead of the Maildir: {"host": "imap.dreamhost.com", "user": "...", "password": "...", "folder":
56"INBOX"} (folder is optional; "done_folder" / "bad_folder" optionally override the target folders). The file must be mode 600; a group/world-readable file, a missing key or an empty
57password is refused with an error. The connection is IMAP4_SSL on port 993 with certificate verification on and a
5830 s timeout, and always logs out. The password is never printed or logged. Messages are listed by UID (with the
59folder's UIDVALIDITY), fetched with BODY.PEEK[], and over 256 KB are moved to the bad folder without being
60downloaded. Only undeleted messages are listed. Finished mail is copied to <prefix>comments-done /
61<prefix>comments-bad (created when missing; the prefix, e.g. "INBOX.", comes from the server's NAMESPACE or LIST
62delimiter), then flagged \\Deleted and expunged (UID EXPUNGE when the server's CAPABILITY, asked after login, lists UIDPLUS). A message the server
63will not fetch (SkipMessage) is logged and skipped; it stays in the mailbox, is logged again on every run, and
64must be removed by hand. Ingest holds a job-wide lock (ingest.lock), so overlapping runs
65queue up. `comments.py imap-check` prints the message count, the folder names and NAMESPACE response, and, for
66the 5 newest, only the header names plus the authserv-id and method=result tokens of each Authentication-Results
67header (no bodies, subjects or addresses), to learn what the provider writes.
68Post paths are stored in one canonical spelling (decoded, then re-escaped with upper-case hex), so /sl%75g/ and
69/slug/ are the same post; publishing merges any spellings that map to the same folder.
70"""
71
72import hashlib
73import imaplib
74import json
75import mailbox
76import os
77import re
78import secrets
79import ssl
80import sys
81import tempfile
82import time
83import unicodedata
84from urllib.parse import quote, unquote
85from contextlib import contextmanager
86from email import message_from_bytes, policy
87from email.header import decode_header, make_header
88from email.utils import getaddresses
89
90try:
91 import fcntl
92except ImportError: # not on Windows servers; only the lock is lost
93 fcntl = None
94
95# Every URL form blog.py makes: safe segments (a url template may start a segment with _ or ~), trailing slash.
96PATH_RE = re.compile(r"^/([A-Za-z0-9_~][A-Za-z0-9._%~+-]*/){1,8}$")
97ID_RE = re.compile(r"^\d{8}T\d{6}Z-[0-9a-f]{6}$")
98PUBLIC_FIELDS = ("author", "date", "id", "parent", "text")
99PENDING_CAP = 200
100SENDER_LIMIT = 5 # pending or approved comments per sender per SENDER_WINDOW seconds
101SENDER_WINDOW = 24 * 3600
102MAX_TEXT = 4000
103MAX_NAME = 60
104
105
106class CommentError(Exception):
107 pass
108
109
110# ---- configuration and paths ----
111
112def state_dir():
113 return os.path.expanduser(os.environ.get("FW_COMMENTS_DIR") or "~/fw/comments")
114
115
116def load_config():
117 path = os.environ.get("FW_COMMENTS_SITES") or os.path.join(os.path.dirname(os.path.abspath(__file__)), "comments", "sites.json")
118 try:
119 with open(path) as f:
120 return json.load(f)
121 except (ValueError, OSError) as e:
122 raise CommentError(f"cannot read {path}: {e}")
123
124
125def site_root(site):
126 cfg = load_config().get(site)
127 if not isinstance(cfg, dict) or "root" not in cfg:
128 raise CommentError(f"unknown site: {site}")
129 return os.path.expanduser(cfg["root"])
130
131
132def site_names():
133 return sorted(k for k, v in load_config().items() if isinstance(v, dict) and "root" in v)
134
135
136def valid_path(path):
137 if not (isinstance(path, str) and len(path) <= 300 and PATH_RE.match(path) and ".." not in path):
138 return False
139 for seg in path.strip("/").split("/"): # as the web server and blog.py decode it: no ".", "..", separator, control
140 decoded = unquote(seg)
141 if (decoded in (".", "..") or ".." in decoded or "\ufffd" in decoded or "/" in decoded or "\\" in decoded
142 or any(unicodedata.category(ch) in ("Cc", "Cf") for ch in decoded)):
143 return False
144 return True
145
146
147def canonical_path(path):
148 """One spelling per post: decoded, then re-escaped (upper-case hex, only what must be escaped)."""
149 check_path(path)
150 canon = "/" + "/".join(quote(unquote(seg), safe="~+") for seg in path.strip("/").split("/")) + "/"
151 check_path(canon)
152 return canon
153
154
155def fs_path(path):
156 """The folder a post path maps to under a site root (the web server decodes %xx first)."""
157 return unquote(path.strip("/"))
158
159
160def check_path(path):
161 if not valid_path(path):
162 raise CommentError(f"invalid post path: {path!r}")
163
164
165def check_id(cid):
166 if not ID_RE.match(cid):
167 raise CommentError(f"invalid comment id: {cid!r}")
168
169
170def now_iso():
171 return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
172
173
174def new_id():
175 return time.strftime("%Y%m%dT%H%M%SZ", time.gmtime()) + "-" + secrets.token_hex(3)
176
177
178def secret():
179 """A random per-installation secret, created once; it salts the sender hashes."""
180 path = os.path.join(state_dir(), "secret")
181 try:
182 with open(path) as f:
183 value = f.read().strip()
184 except FileNotFoundError:
185 value = ""
186 if not value:
187 value = secrets.token_hex(32)
188 write_atomic(path, value + "\n")
189 return value
190
191
192def sender_hash(address):
193 return hashlib.sha256((secret() + address.strip().lower()).encode()).hexdigest()
194
195
196# ---- store ----
197
198def write_atomic(path, data):
199 os.makedirs(os.path.dirname(path), exist_ok=True)
200 fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path), prefix=".tmp-")
201 try:
202 with os.fdopen(fd, "w") as f:
203 f.write(data)
204 f.flush()
205 os.fsync(f.fileno())
206 os.chmod(tmp, 0o644 if "_comments" in path.split(os.sep) else 0o600)
207 os.replace(tmp, path)
208 try:
209 dfd = os.open(os.path.dirname(path), os.O_RDONLY)
210 try:
211 os.fsync(dfd)
212 finally:
213 os.close(dfd)
214 except OSError:
215 pass
216 except BaseException:
217 if os.path.exists(tmp):
218 os.unlink(tmp)
219 raise
220
221
222def write_json(path, obj):
223 write_atomic(path, json.dumps(obj, indent=2, ensure_ascii=False, sort_keys=True) + "\n")
224
225
226def read_json(path, default):
227 try:
228 with open(path) as f:
229 return json.load(f)
230 except FileNotFoundError:
231 return default
232 except (ValueError, OSError) as e: # empty or damaged file: say which one
233 raise CommentError(f"cannot read {path}: {e}")
234
235
236@contextmanager
237def ingest_lock():
238 """Job-wide: one ingest at a time (a second one waits), so its snapshot of the store is never stale."""
239 os.makedirs(state_dir(), exist_ok=True)
240 with open(os.path.join(state_dir(), "ingest.lock"), "w") as f:
241 if fcntl:
242 fcntl.flock(f, fcntl.LOCK_EX)
243 yield
244
245
246@contextmanager
247def locked():
248 os.makedirs(state_dir(), exist_ok=True)
249 with open(os.path.join(state_dir(), "lock"), "w") as f:
250 if fcntl:
251 fcntl.flock(f, fcntl.LOCK_EX)
252 yield
253
254
255def pending_path(cid):
256 check_id(cid)
257 return os.path.join(state_dir(), "pending", cid + ".json")
258
259
260def pending_ids():
261 d = os.path.join(state_dir(), "pending")
262 return sorted(n[:-5] for n in os.listdir(d) if n.endswith(".json") and not n.startswith(".")) if os.path.isdir(d) else []
263
264
265def load_pending(cid):
266 c = read_json(pending_path(cid), None)
267 if c is None:
268 raise CommentError(f"no pending comment {cid}")
269 return c
270
271
272def approved_path(site):
273 site_root(site) # validates the site name
274 return os.path.join(state_dir(), "approved", site + ".json")
275
276
277def load_approved(site):
278 return read_json(approved_path(site), {})
279
280
281def meta_path():
282 return os.path.join(state_dir(), "approved-meta.json")
283
284
285def keys_path():
286 return os.path.join(state_dir(), "approved-keys.json")
287
288
289def trusted_path():
290 return os.path.join(state_dir(), "trusted.json")
291
292
293def log_rejection(cid, reason):
294 os.makedirs(state_dir(), exist_ok=True)
295 reason = clean(reason).replace("\n", " ")
296 log = os.path.join(state_dir(), "rejected.log")
297 try:
298 if os.path.getsize(log) > 1_000_000:
299 os.replace(log, log + ".1")
300 except FileNotFoundError:
301 pass
302 with open(log, "a") as f:
303 f.write(f"{cid} {now_iso()} {reason}\n")
304
305
306def public(c):
307 return {k: c.get(k) for k in PUBLIC_FIELDS}
308
309
310# ---- publishing ----
311
312def post_file(site, path):
313 check_path(path)
314 base = os.path.realpath(os.path.join(site_root(site), "_comments"))
315 target = os.path.normpath(os.path.join(base, fs_path(path) + ".json"))
316 if not target.startswith(base + os.sep):
317 raise CommentError(f"path escapes the comments directory: {path!r}")
318 return target
319
320
321_warned = set()
322
323
324def publish_path(site, path, approved):
325 target = post_file(site, path)
326 if not os.path.isdir(site_root(site)): # never create a site root; `publish` writes the files later
327 if site not in _warned:
328 _warned.add(site)
329 print(f"comments: root of site {site} does not exist; not publishing (run `publish` later)", file=sys.stderr)
330 return
331 folder = fs_path(path) # every spelling of this post, merged, oldest first
332 items = sorted((public(c) for key, cs in approved.items() if valid_path(key) and fs_path(key) == folder for c in cs),
333 key=lambda c: (c.get("date") or "", c.get("id") or ""))
334 if items:
335 write_json(target, items)
336 elif os.path.exists(target):
337 os.unlink(target)
338
339
340def publish_site(site):
341 approved = load_approved(site)
342 done = set()
343 for path in sorted(approved):
344 if valid_path(path) and fs_path(path) not in done:
345 done.add(fs_path(path))
346 publish_path(site, path, approved)
347 return len(done)
348
349
350# ---- moderation ----
351
352def add_approved(c):
353 """Store the comment, then publish it; a publish failure only warns (a later `publish` fixes the files)."""
354 site, path = c["site"], canonical_path(c["path"])
355 c = {**c, "path": path}
356 approved = load_approved(site)
357 approved.setdefault(path, []).append(public(c))
358 write_json(approved_path(site), approved)
359 if c.get("sender_hash"):
360 meta = read_json(meta_path(), {})
361 meta[c["id"]] = c["sender_hash"]
362 write_json(meta_path(), meta)
363 if c.get("msg_key"):
364 keys = read_json(keys_path(), {})
365 keys[c["msg_key"]] = c["id"]
366 write_json(keys_path(), keys)
367 try:
368 publish_path(site, path, approved)
369 except OSError as e:
370 print(f"comments: could not publish {site}:{path} ({e}); run `publish` later", file=sys.stderr)
371
372
373def trust(h):
374 t = read_json(trusted_path(), [])
375 if h not in t:
376 write_json(trusted_path(), t + [h])
377
378
379APPROVED_LISTED = 20
380
381
382def post_url(site, path):
383 origin = (load_config().get(site) or {}).get("origin") or ""
384 return origin.rstrip("/") + path if origin else ""
385
386
387def cmd_list_json():
388 """For `fw admin`: pending comments in full and the newest approved ones. Never a sender hash, only whether it is trusted."""
389 trusted = set(read_json(trusted_path(), []))
390 meta = read_json(meta_path(), {})
391 pending = []
392 for cid in pending_ids():
393 c = read_json(pending_path(cid), {})
394 pending.append({"id": cid, "site": c.get("site"), "path": c.get("path"), "post_url": post_url(c.get("site"), c.get("path") or ""),
395 "author": c.get("author"), "date": c.get("date"), "source": c.get("source"), "parent": c.get("parent"),
396 "text": c.get("text", ""), "trusted": bool(c.get("sender_hash")) and c["sender_hash"] in trusted})
397 approved = []
398 for site in site_names():
399 for path, items in load_approved(site).items():
400 for c in items:
401 approved.append({"id": c.get("id"), "site": site, "path": path, "post_url": post_url(site, path),
402 "author": c.get("author"), "date": c.get("date"), "source": None, "parent": c.get("parent"),
403 "text": c.get("text", ""), "trusted": meta.get(c.get("id")) in trusted and c.get("id") in meta})
404 approved.sort(key=lambda c: (c["date"] or "", c["id"] or ""), reverse=True)
405 print(json.dumps({"pending": pending, "approved": approved[:APPROVED_LISTED]}, ensure_ascii=False))
406
407
408def cmd_list():
409 for cid in pending_ids():
410 c = read_json(pending_path(cid), {})
411 text = " ".join(c.get("text", "").split())[:160]
412 print(f"{cid} {c.get('site')}:{c.get('path')} {c.get('author')} [{c.get('source')}] {text}")
413
414
415def find_approved(cid):
416 for site in site_names():
417 for path, items in load_approved(site).items():
418 for c in items:
419 if c.get("id") == cid:
420 return site, path, c
421 return None
422
423
424def cmd_show(cid):
425 check_id(cid)
426 p = read_json(pending_path(cid), None)
427 if p is not None:
428 print(json.dumps(p, indent=2, ensure_ascii=False))
429 return
430 found = find_approved(cid)
431 if not found:
432 raise CommentError(f"no comment {cid}")
433 print(json.dumps({"site": found[0], "path": found[1], **found[2]}, indent=2, ensure_ascii=False))
434
435
436def cmd_approve(ids, with_trust):
437 with locked():
438 for cid in ids:
439 c = load_pending(cid)
440 if with_trust:
441 if c.get("source") != "email" or not c.get("sender_hash"):
442 raise CommentError(f"{cid}: --trust works only for email comments")
443 trust(c["sender_hash"])
444 add_approved(c)
445 os.unlink(pending_path(cid))
446 print(f"approved {cid}")
447
448
449def cmd_reject(ids):
450 with locked():
451 for cid in ids:
452 c = load_pending(cid)
453 if c.get("sender_hash"):
454 meta = read_json(meta_path(), {})
455 meta[cid] = c["sender_hash"]
456 write_json(meta_path(), meta)
457 log_rejection(cid, "rejected by moderator")
458 os.unlink(pending_path(cid))
459 print(f"rejected {cid}")
460
461
462def cmd_remove(cid):
463 check_id(cid)
464 with locked():
465 found = find_approved(cid)
466 if not found:
467 raise CommentError(f"no approved comment {cid}")
468 site, path, _ = found
469 approved = load_approved(site)
470 approved[path] = [c for c in approved[path] if c.get("id") != cid]
471 if not approved[path]:
472 del approved[path]
473 write_json(approved_path(site), approved)
474 publish_path(site, path, approved)
475 print(f"removed {cid}")
476
477
478def cmd_untrust(cid):
479 check_id(cid)
480 with locked():
481 h = (read_json(pending_path(cid), None) or {}).get("sender_hash") or read_json(meta_path(), {}).get(cid)
482 if not h:
483 raise CommentError(f"no sender recorded for {cid}")
484 t = read_json(trusted_path(), [])
485 if h in t:
486 write_json(trusted_path(), [x for x in t if x != h])
487 print("untrusted")
488 else:
489 print("sender was not trusted")
490
491
492def cmd_publish(site=None):
493 with locked():
494 for s in ([site] if site else site_names()):
495 print(f"{s}: {publish_site(s)} post file(s)")
496
497
498def cmd_export(site):
499 print(json.dumps(load_approved(site), indent=2, ensure_ascii=False, sort_keys=True))
500
501
502# ---- email intake ----
503
504def clean(s):
505 """Drop control and format characters (bidi, zero-width, ...) and line/paragraph separators; keep newlines."""
506 s = s.replace("\r\n", "\n").replace("\r", "\n")
507 return "".join(ch for ch in s
508 if ch == "\n" or (unicodedata.category(ch) not in ("Cc", "Cf") and ch not in "\u2028\u2029"))
509
510
511def split_ar(text):
512 """Split an Authentication-Results value on ';' outside "quoted strings", dropping (comments)."""
513 parts, cur, quote, depth, esc = [], [], False, 0, False
514 for ch in text:
515 if esc:
516 esc = False
517 if depth == 0:
518 cur.append(ch)
519 elif ch == "\\":
520 esc = True
521 if depth == 0:
522 cur.append(ch)
523 elif quote:
524 cur.append(ch)
525 quote = ch != '"'
526 elif depth:
527 depth += (ch == "(") - (ch == ")")
528 elif ch == "(":
529 depth = 1
530 elif ch == '"':
531 quote = True
532 cur.append(ch)
533 elif ch == ";":
534 parts.append("".join(cur))
535 cur = []
536 else:
537 cur.append(ch)
538 parts.append("".join(cur))
539 return parts
540
541
542def parse_ar(header):
543 """(authserv-id, {method: [(result, props)]}) of one Authentication-Results header."""
544 segs = split_ar(str(header))
545 head = segs[0].split()
546 by_method = {}
547 for seg in segs[1:]:
548 m = re.match(r"\s*([A-Za-z0-9_-]+)\s*=\s*([A-Za-z]+)(?![A-Za-z0-9])(.*)$", seg, re.S)
549 if not m:
550 continue
551 props = {k.lower(): v.strip('"').lower() for k, v in
552 re.findall(r'([A-Za-z0-9_.-]+)\s*=\s*("(?:[^"\\]|\\.)*"|[^\s"]+)', m.group(3))}
553 by_method.setdefault(m.group(1).lower(), []).append((m.group(2).lower(), props))
554 return (head[0].lower() if head else ""), by_method
555
556
557def authenticated(msg, from_domain, authserv_id):
558 """None when the message is authenticated, else the reason it is not. Only the topmost header counts."""
559 headers = msg.get_all("Authentication-Results") or []
560 if not headers:
561 return "no Authentication-Results header"
562 aid, by_method = parse_ar(headers[0])
563 if aid != authserv_id.lower():
564 return "topmost Authentication-Results header is not from our server"
565 dmarc = by_method.get("dmarc", [])
566 if dmarc and all(r == "pass" and p.get("header.from", "").rsplit("@", 1)[-1] == from_domain for r, p in dmarc):
567 return None
568 dkim = by_method.get("dkim", [])
569 if dkim and all(r == "pass" for r, _ in dkim) and any(
570 p.get("header.d") and (p["header.d"] == from_domain or p["header.d"].endswith("." + from_domain)) for _, p in dkim):
571 return None
572 return "not authenticated (no aligned dmarc=pass or dkim=pass)"
573
574
575def extract_text(msg):
576 part = msg.get_body(("plain",))
577 if part is None:
578 return ""
579 lines = []
580 for line in clean(part.get_content()).split("\n"):
581 if line.rstrip() == "--":
582 break
583 if line.lstrip().startswith(">"):
584 continue
585 lines.append(line)
586 return "\n".join(lines).strip()[:MAX_TEXT]
587
588
589def parse_message(msg, authserv_id):
590 """Return (comment dict with the sender "_address", None) or (None, reason). Content problems only."""
591 subject = str(make_header(decode_header(str(msg.get("Subject", "")))))
592 sites = site_names()
593 m = re.search(r"\[(" + "|".join(re.escape(x) for x in sites) + r"):(/[^\]\s]+)\]", subject) if sites else None
594 if not m:
595 return None, "no [site:/path/] tag in subject"
596 site, path = m.group(1), m.group(2)
597 if not valid_path(path):
598 return None, f"invalid post path {path!r}"
599 try:
600 path = canonical_path(path)
601 except CommentError: # e.g. a bare % becomes %25 and the path gets too long: a content problem, not a state one
602 return None, f"invalid post path {path[:60]!r}"
603 root = site_root(site)
604 if os.path.isdir(root) and not os.path.isfile(os.path.join(root, fs_path(path), "index.html")):
605 return None, f"no such post {site}:{path}"
606 froms = msg.get_all("From") or []
607 addrs = [a for a in getaddresses([str(x) for x in froms]) if a[1]]
608 if len(froms) != 1 or len(addrs) != 1 or "@" not in addrs[0][1]:
609 return None, "need exactly one From header with one address"
610 name, addr = addrs[0]
611 why = authenticated(msg, addr.rsplit("@", 1)[1].lower(), authserv_id)
612 if why:
613 return None, why
614 text = extract_text(msg)
615 if not text:
616 return None, "empty text"
617 name = clean(name).replace("\n", " ").strip()[:MAX_NAME].strip()
618 if not name or "@" in name:
619 name = "A reader"
620 return {"id": new_id(), "site": site, "path": path, "author": name, "text": text,
621 "date": now_iso(), "source": "email", "parent": None, "_address": addr}, None
622
623
624def recent_index():
625 """sender hash -> dates of that sender's pending or approved comments (read once per run)."""
626 meta = read_json(meta_path(), {})
627 index = {}
628 for cid in pending_ids():
629 c = read_json(pending_path(cid), {})
630 if c.get("sender_hash"):
631 index.setdefault(c["sender_hash"], []).append(c.get("date", ""))
632 for site in site_names():
633 for items in load_approved(site).values():
634 for c in items:
635 if meta.get(c.get("id")):
636 index.setdefault(meta[c["id"]], []).append(c.get("date", ""))
637 return index
638
639
640def stored_keys():
641 """Message keys of every stored comment (pending records carry theirs); prunes approved-keys.json."""
642 approved_ids = {c.get("id") for site in site_names() for items in load_approved(site).values() for c in items}
643 keys = read_json(keys_path(), {})
644 kept = {k: v for k, v in keys.items() if v in approved_ids}
645 if kept != keys:
646 write_json(keys_path(), kept)
647 for cid in pending_ids():
648 k = read_json(pending_path(cid), {}).get("msg_key")
649 if k:
650 kept[k] = cid
651 return set(kept)
652
653
654class MaildirSource:
655 """Where the mail comes from: a Maildir folder, with .done and .bad folders beside it."""
656
657 def __init__(self, path):
658 self.path = path
659 self.box = None
660
661 def available(self):
662 return os.path.isdir(self.path)
663
664 def close(self):
665 pass
666
667 def keys(self):
668 self.box = mailbox.Maildir(self.path, factory=None, create=False)
669 return self.box.keys()
670
671 def read(self, key):
672 """The raw bytes; KeyError when another run already took the message."""
673 return self.box.get_bytes(key)
674
675 def finish(self, key, raw, outcome):
676 """Move the message to .done or .bad (it is gone from the inbox afterwards)."""
677 mailbox.Maildir(self.path + "." + outcome, create=True).add(raw)
678 try:
679 self.box.remove(key)
680 except KeyError: # another process already took it
681 pass
682
683
684IMAP_MAX_SIZE = 256 * 1024
685IMAP_TIMEOUT = 30
686
687
688def imap_config_path():
689 return os.path.expanduser(os.environ.get("FW_COMMENTS_IMAP") or os.path.join(state_dir(), "imap.json"))
690
691
692def load_imap_config(path=None):
693 path = path or imap_config_path()
694 mode = os.stat(path).st_mode
695 if mode & 0o077:
696 raise CommentError(f"{path} must not be readable by group or others (chmod 600 it)")
697 cfg = read_json(path, {})
698 if not isinstance(cfg, dict):
699 raise CommentError(f"{path} must hold a JSON object")
700 for key in ("host", "user", "password"):
701 if not isinstance(cfg.get(key), str) or not cfg[key]:
702 raise CommentError(f"{path}: {key!r} is missing or empty")
703 if not cfg[key].isascii():
704 raise CommentError(f"{path}: {key!r} must be ASCII")
705 out = {"host": cfg["host"], "user": cfg["user"], "password": cfg["password"], "folder": cfg.get("folder") or "INBOX"}
706 for key in ("done_folder", "bad_folder"):
707 out[key] = cfg.get(key) or ""
708 for key in ("folder", "done_folder", "bad_folder"):
709 if not isinstance(out[key], str) or (out[key] and not IMAP_NAME_RE.match(out[key])):
710 raise CommentError(f"{path}: {key!r} must be a folder name of letters, digits and . _ / -")
711 return out
712
713
714class SkipMessage(Exception):
715 """This message cannot be handled this run (e.g. the server refuses to fetch it); it stays where it is."""
716
717
718IMAP_NAME_RE = re.compile(r"^[A-Za-z0-9._/-]+$")
719
720
721class IMAPSource:
722 """The comments mailbox on an IMAP server. Finished mail is copied to comments-done / comments-bad.
723 issues.py reuses this class for the bugs mailbox: its own config file (cfg_path), folder stem, size limit and logger."""
724
725 def __init__(self, cfg_path=None, stem="comments", max_size=IMAP_MAX_SIZE, log=None):
726 self.cfg_path = cfg_path
727 self.stem = stem
728 self.max_size = max_size
729 self.log = log or log_rejection
730 self.m = None
731 self.cfg = None
732 self.validity = ""
733 self.caps = set()
734
735 def available(self):
736 return os.path.exists(self.cfg_path or imap_config_path())
737
738 def _redact(self, text):
739 pw = self.cfg["password"] if self.cfg else ""
740 return text.replace(pw, "***") if pw else text
741
742 def _fail(self, what, e=None, detail=""):
743 if e is not None and not isinstance(e, UnicodeError): # a UnicodeError could quote a password character
744 detail = f"{type(e).__name__} {e}"
745 raise CommentError(f"IMAP {what} failed" + (f": {self._redact(detail)}" if detail else "")) from None
746
747 def _text(self, typ, data):
748 parts = [d.decode("utf-8", "replace") if isinstance(d, bytes) else str(d) for d in (data or []) if d]
749 return self._redact(f"{typ} {' '.join(parts)}".strip())
750
751 def _ok(self, result, what):
752 typ, data = result
753 if typ != "OK":
754 self._fail(what, detail=self._text(typ, data))
755 return data
756
757 def _open(self, readonly=False):
758 self.cfg = load_imap_config(self.cfg_path)
759 try:
760 self.m = imaplib.IMAP4_SSL(self.cfg["host"], 993, ssl_context=ssl.create_default_context(), timeout=IMAP_TIMEOUT)
761 self.m.login(self.cfg["user"], self.cfg["password"])
762 self.caps = self._capabilities() # imaplib does not refresh these after login, and Dovecot adds UIDPLUS then
763 self._ok(self.m.select(self.cfg["folder"], readonly=readonly), "select")
764 v = self.m.response("UIDVALIDITY")[1]
765 self.validity = (v[0] or b"0").decode()
766 self._namespace()
767 except CommentError:
768 raise
769 except (OSError, UnicodeError, imaplib.IMAP4.error) as e:
770 self._fail("connection", e)
771
772 def _capabilities(self):
773 try:
774 typ, data = self.m.capability()
775 except (imaplib.IMAP4.error, AttributeError):
776 return set()
777 return {t.upper() for d in (data if typ == "OK" else []) if isinstance(d, bytes) for t in d.decode("ascii", "replace").split()}
778
779 def _list(self):
780 """[(delimiter, name)] from LIST (empty when the server refuses or the reply cannot be parsed)."""
781 try:
782 typ, data = self.m.list()
783 except imaplib.IMAP4.error:
784 return []
785 out = []
786 for item in data if typ == "OK" else []:
787 try:
788 if isinstance(item, tuple): # imaplib hands a literal mailbox name over as (head, literal)
789 head, name = item[0].decode("utf-8", "replace"), item[1].decode("utf-8", "replace")
790 m = re.match(r'^\((.*?)\) (?:"((?:[^"\\]|\\.)*)"|NIL) ', head)
791 out.append(((m.group(2) or "") if m else "", name))
792 continue
793 m = re.match(r'^\((.*?)\) (?:"((?:[^"\\]|\\.)*)"|NIL) (.*)$', (item or b"").decode("utf-8", "replace").strip())
794 if m:
795 out.append((m.group(2) or "", m.group(3).strip().strip('"')))
796 except (AttributeError, IndexError, TypeError, ValueError):
797 continue
798 return out
799
800 def _namespace_text(self):
801 try:
802 typ, data = self.m.namespace()
803 except (imaplib.IMAP4.error, AttributeError):
804 return ""
805 if typ != "OK":
806 return ""
807 return " ".join(d.decode("utf-8", "replace") for d in data if isinstance(d, bytes))
808
809 def _namespace(self):
810 """Learn the personal namespace prefix (e.g. "INBOX.") from NAMESPACE, else from LIST; sets self.prefix."""
811 self.prefix = ""
812 try:
813 m = re.search(r'\(\("([^"]*)"', self._namespace_text())
814 if m:
815 self.prefix = m.group(1)
816 return
817 listing = self._list()
818 delim = next((d for d, _ in listing if d), "")
819 if delim and any(n.startswith("INBOX" + delim) for _, n in listing):
820 self.prefix = "INBOX" + delim
821 except (AttributeError, IndexError, TypeError, ValueError):
822 self.prefix = ""
823
824 def _folder(self, outcome):
825 return self.cfg[outcome + "_folder"] or f"{self.prefix}{self.stem}-{outcome}"
826
827 def close(self):
828 if self.m is not None:
829 try:
830 self.m.logout()
831 except Exception:
832 pass
833 self.m = None
834
835 def _uids(self):
836 data = self._ok(self.m.uid("SEARCH", None, "UNDELETED"), "search")
837 return [u.decode() for u in (data[0] or b"").split()]
838
839 def _sizes(self, uids):
840 sizes = {}
841 for i in range(0, len(uids), 200):
842 data = self._ok(self.m.uid("FETCH", ",".join(uids[i:i + 200]), "(RFC822.SIZE)"), "fetch sizes")
843 for item in data:
844 text = (item[0] if isinstance(item, tuple) else item or b"").decode("ascii", "replace")
845 u, z = re.search(r"UID (\d+)", text), re.search(r"RFC822\.SIZE (\d+)", text)
846 if u and z:
847 sizes[u.group(1)] = int(z.group(1))
848 return sizes
849
850 def keys(self):
851 self._open()
852 uids = self._uids()
853 sizes = self._sizes(uids)
854 keys = []
855 for uid in uids:
856 if sizes.get(uid, 0) > self.max_size: # never downloaded
857 try:
858 self._move(uid, "bad")
859 except (CommentError, OSError, imaplib.IMAP4.error) as e: # leave it, keep going
860 self.log(new_id(), f"cannot move oversize message {uid}: {e}")
861 continue
862 self.log(new_id(), f"message over {self.max_size // 1024} KB")
863 print(f"bad: message over {self.max_size // 1024} KB")
864 else:
865 keys.append(f"{self.validity}:{uid}")
866 return keys
867
868 def _split(self, key):
869 validity, _, uid = key.partition(":")
870 if validity != self.validity or not uid.isdigit():
871 raise KeyError(key)
872 return uid
873
874 def read(self, key):
875 uid = self._split(key)
876 typ, data = self.m.uid("FETCH", uid, "(BODY.PEEK[])")
877 if typ != "OK":
878 raise SkipMessage(f"server refused to fetch UID {uid}: {self._text(typ, data)}")
879 for item in data:
880 if isinstance(item, tuple):
881 return item[1]
882 raise KeyError(key)
883
884 def _move(self, uid, outcome):
885 folder = self._folder(outcome)
886 typ, _ = self.m.uid("COPY", uid, folder)
887 if typ != "OK":
888 self._ok(self.m.create(folder), "create folder " + folder)
889 self._ok(self.m.uid("COPY", uid, folder), "copy to " + folder)
890 self._ok(self.m.uid("STORE", uid, "+FLAGS", "(\\Deleted)"), "flag")
891 if "UIDPLUS" in self.caps:
892 self._ok(self.m.uid("EXPUNGE", uid), "expunge")
893 else:
894 self._ok(self.m.expunge(), "expunge")
895
896 def finish(self, key, raw, outcome):
897 try:
898 self._move(self._split(key), outcome)
899 except (OSError, imaplib.IMAP4.error) as e:
900 self._fail("move", e)
901
902 def check(self):
903 """Print the message count, folder names, NAMESPACE and, for the 5 newest messages, only header names
904 and Authentication-Results tokens."""
905 self._open(readonly=True)
906 uids = self._uids()
907 print(f"messages in {self.cfg['folder']}: {len(uids)}")
908 print("folders: " + ", ".join(name for _, name in self._list()))
909 print("namespace: " + self._redact(self._namespace_text()))
910 print(f"target folders: {self._folder('done')}, {self._folder('bad')}")
911 for n, uid in enumerate(reversed(uids[-5:]), 1):
912 data = self._ok(self.m.uid("FETCH", uid, "(BODY.PEEK[HEADER])"), "fetch")
913 raw = next((i[1] for i in data if isinstance(i, tuple)), b"")
914 msg = message_from_bytes(raw, policy=policy.compat32)
915 print(f"message {n}: headers: " + ", ".join(sorted({k for k in msg.keys()}, key=str.lower)))
916 for h in msg.get_all("Authentication-Results") or []:
917 segs = split_ar(" ".join(str(h).split()))
918 tokens = []
919 for seg in segs[1:]:
920 m = re.match(r"\s*([A-Za-z0-9_-]+)\s*=\s*([A-Za-z]+)(?![A-Za-z0-9])", seg)
921 if m:
922 tokens.append(f"{m.group(1).lower()}={m.group(2).lower()}")
923 head = segs[0].split()
924 print(f" Authentication-Results: authserv-id {head[0] if head else '(none)'}; " + " ".join(tokens))
925
926
927def message_source():
928 if os.path.exists(imap_config_path()):
929 return IMAPSource()
930 return MaildirSource(os.path.expanduser(os.environ.get("FW_COMMENTS_MAILDIR") or "~/Maildir/.comments"))
931
932
933def ingest_messages(source, authserv_id):
934 with locked():
935 seen = stored_keys()
936 recent = recent_index()
937 cutoff = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() - SENDER_WINDOW))
938 for key in source.keys():
939 with locked(): # per message, so `fw comments approve` is not starved by a big backlog
940 try:
941 raw = source.read(key)
942 except KeyError:
943 continue
944 except SkipMessage as e: # leave it in the inbox and go on with the others
945 log_rejection(new_id(), f"skipped {key}: {e}")
946 print(f"skipped {key}: {e}")
947 continue
948 mkey = hashlib.sha256(raw).hexdigest()
949 if mkey in seen: # its comment is stored: a crash or overlap left the mail behind
950 log_rejection(new_id(), "duplicate of a stored comment; moved to .done")
951 source.finish(key, raw, "done")
952 continue
953 try:
954 c, why = parse_message(message_from_bytes(raw, policy=policy.default), authserv_id)
955 except (CommentError, OSError): # state problems abort the run and leave the mail in the inbox
956 raise
957 except Exception as e: # a malformed message must not stop the rest
958 c, why = None, f"unreadable message: {e}"
959 trusted = read_json(trusted_path(), [])
960 if c:
961 c["sender_hash"] = sender_hash(c.pop("_address"))
962 c["msg_key"] = mkey
963 h = c["sender_hash"]
964 if h not in trusted and len(pending_ids()) >= PENDING_CAP:
965 c, why = None, f"pending queue full ({PENDING_CAP})"
966 elif sum(1 for d in recent.get(h, []) if d >= cutoff) >= SENDER_LIMIT:
967 c, why = None, f"sender limit reached ({SENDER_LIMIT} per 24 hours)"
968 if c is None: # rejected mail is not remembered: moved back from .bad, it is tried again
969 cid = new_id()
970 log_rejection(cid, why)
971 print(f"bad {cid}: {why}")
972 source.finish(key, raw, "bad")
973 continue
974 if c["sender_hash"] in trusted:
975 add_approved(c) # stores first; a publish failure only warns
976 print(f"published {c['id']} {c['site']}:{c['path']} (trusted sender)")
977 else:
978 write_json(pending_path(c["id"]), c)
979 print(f"pending {c['id']} {c['site']}:{c['path']} by {c['author']}")
980 seen.add(mkey)
981 recent.setdefault(c["sender_hash"], []).append(c["date"])
982 source.finish(key, raw, "done")
983
984
985def cmd_ingest():
986 source = message_source()
987 if not source.available():
988 return
989 authserv_id = (load_config().get("authserv_id") or "").strip()
990 if not authserv_id:
991 raise CommentError('authserv_id is empty in sites.json; refusing to ingest (set it to the authserv-id '
992 'your mail server puts at the start of its Authentication-Results header)')
993 try:
994 with ingest_lock(): # a second run waits here, so it starts from the first run's results
995 ingest_messages(source, authserv_id)
996 finally:
997 source.close()
998
999def main(argv):
1000 if not argv or argv[0] in ("-h", "--help", "help"):
1001 print(__doc__)
1002 return 0
1003 cmd, args = argv[0], argv[1:]
1004 flags = [a for a in args if a.startswith("--")]
1005 args = [a for a in args if not a.startswith("--")]
1006 try:
1007 if cmd == "list" and not args and set(flags) <= {"--json"}:
1008 (cmd_list_json if "--json" in flags else cmd_list)()
1009 elif cmd == "show" and len(args) == 1:
1010 cmd_show(args[0])
1011 elif cmd == "approve" and args and set(flags) <= {"--trust"}:
1012 cmd_approve(args, "--trust" in flags)
1013 elif cmd == "reject" and args:
1014 cmd_reject(args)
1015 elif cmd == "remove" and len(args) == 1:
1016 cmd_remove(args[0])
1017 elif cmd == "untrust" and len(args) == 1:
1018 cmd_untrust(args[0])
1019 elif cmd == "publish" and len(args) <= 1:
1020 cmd_publish(args[0] if args else None)
1021 elif cmd == "export" and len(args) == 1:
1022 cmd_export(args[0])
1023 elif cmd == "ingest":
1024 cmd_ingest()
1025 elif cmd == "imap-check":
1026 source = IMAPSource()
1027 if not source.available():
1028 raise CommentError(f"no IMAP config at {imap_config_path()}")
1029 try:
1030 source.check()
1031 finally:
1032 source.close()
1033 else:
1034 print("usage: comments.py list|show|approve|reject|remove|untrust|publish|export|ingest|imap-check (try --help)", file=sys.stderr)
1035 return 2
1036 except (CommentError, OSError, imaplib.IMAP4.error) as e:
1037 print(f"comments: {e}", file=sys.stderr)
1038 return 1
1039 return 0
1040
1041
1042if __name__ == "__main__":
1043 sys.exit(main(sys.argv[1:]))