Blog (fw blog)

A generic static-blog generator: a folder of Markdown files and one JSON config become a site (server/blog.py). It is used for writing.fuzzyworld.net (and til.fuzzyworld.net).

It runs on the laptop, builds into a clean temporary folder and uploads only that folder with rsync, so nothing in the source folder (drafts, notes, config) is ever published.

fw blog list ~/Blog/fw-blog.json              # every source file: URL, title, date, tags, published or not
fw blog build ~/Blog/fw-blog.json --out DIR [--no-htaccess]   # write the site into DIR (other options are refused)
fw blog preview ~/Blog/fw-blog.json [port]    # build and serve it locally (no redirects, no feed content type)
fw blog deploy ~/Blog/fw-blog.json --dry-run  # then, for real: --yes (or answer the prompt)
fw blog watch ~/Blog/fw-blog.json [--once] [--dry-run]         # mirror to git and auto-deploy edits
fw blog watch install|uninstall ~/Blog/fw-blog.json [--print]  # the launchd agent that runs it

Python-Markdown and Pygments are installed into ~/fw/pylib on the laptop the first time fw blog runs.

What is published

• Posts: only those with published: true, in the file's front matter or in the config's posts table keyed by file name (front matter wins).
• Pages (files in pages_dir) are opt-in the same way: a page needs published: true in its front matter (the importer writes it).
• Never reaches the output: text under ## Spelling and ## Removed writing, and every HTML comment (except <!--more-->).
• The build stops (rather than risk a leak) when a heading that looks private is written another way (##Spelling, an underlined heading) or a code fence is never closed.

What it builds

Reader features (server/site-blog/blog.js): a light/dark switch (the choice is kept in the browser; without one the system setting decides) and a reading-progress bar on posts.

Config

title, base_url and source are required; unknown keys are an error. The full list is at the top of server/blog.py.

Paths: source and comments are relative to the config file's folder; pages_dir and media are relative to source.

site puts the site-wide bar above the blog's own bar (as on code and apps): {"name": "fuzzyworld", "url": "https://fuzzyworld.net/", "nav": [[label, url], ...], "current": "Writing"}. "nav": "gen" takes the fuzzyworld nav from NAV in server/gen.py (the one source) and also shows the shared four-column footer (gen.site_footer(), with the blog's copyright and Feed line under it). The landing page's nav and footer are built from the same functions by gen.py landing in install.sh (<!--SITE-NAV-->, <!--SITE-FOOTER-->). The landing page's text is Markdown in server/site/landing.md, rendered into the <!--LANDING-TEXT--> spot of server/site/index.html.

Safety checks

Deploy and upload safety

fw blog deploy audits the built folder first and refuses anything that is not a generated file or an allowlisted media type (png jpg jpeg gif webp svg pdf epub kpf mp3 mp4 zip). It then rsyncs only that folder, and keeps .well-known.

• It asks for confirmation on every real upload. It holds the watcher's lock only for the upload itself, not while the question is open. Ctrl-C at the question exits quietly with 1.
• --dry-run shows what would be sent.
• The watcher limits the upload to 600 s; a manual deploy has no time limit.

Comments by email

Set "comments_site": "writing" (or "til"; a key of server/comments/sites.json) and "comments_mailbox": "<address>" in the config. Each post then gets a "Comment by email" link (subject Comment [writing:/2024/05/01/slug/]) and a container that blog.js fills from /_comments/<post path>.json.

Moderating

fw comments list|show|approve [--trust]|reject|remove|untrust|publish|export|ingest|imap-check runs on the server (or use the admin panel).

• list --json prints the pending comments in full plus the 20 newest approved ones, with each post's URL and whether the sender is trusted, never a sender hash. fw admin uses it.
• A cron job runs ingest hourly (operations.md).
• fw blog deploy leaves /_comments/ alone.

The mailbox

ingest reads the comments mailbox:

• Hosted mailbox over IMAP when ~/fw/comments/imap.json exists: {"host", "user", "password", "folder"}, mode 600, created by hand on the server. TLS with certificate check. Processed mail moves to the folders comments-done and comments-bad (with the server's folder prefix such as INBOX. added when it needs one); done_folder and bad_folder in the file override the names.
• Otherwise a Maildir folder ~/Maildir/.comments.
• fw comments imap-check prints only header names and the Authentication-Results result tokens, to learn the right authserv_id.

Trust and limits

• ingest accepts a message only when the topmost Authentication-Results header shows a DMARC pass for the From domain (or an aligned DKIM pass), and was written by the receiving mail server. That server's name must be set as authserv_id in server/comments/sites.json. Empty means ingest refuses to read anything. Set it in the repo copy, because an install overwrites the server's.
• That trust holds only if the receiving server always adds its own header above anything the sender supplied.
• The address is never shown. Only a salted hash is kept (losing ~/fw/comments/secret invalidates trusted senders).
• Limits: 4000 characters, 5 comments per sender per 24 hours, 200 pending.
• State lives in ~/fw/comments/, outside every web root.

The optional form

comments_form (an https action) adds a plain HTML form for the endpoint server/comments/post.php. It is off, and the endpoint is not installed (hosting tiers). It is written, and its tests pass on the server's PHP; an optional self_origin key in sites.json names the origin its no-JavaScript confirm page posts from.

The watcher (fw blog watch)

server/blog_watch.py mirrors the blog source into a git repo and auto-deploys edits that change no URLs, for ONE blog config. Run it on ONE machine only. The MacBook Air just keeps writing through Dropbox, and its edits are captured by the mirror on the watching machine. A manual fw blog deploy on another machine does not update this machine's baseline; the result is a harmless re-deploy or a hold.

• One tick every 60 s: the launchd agent runs fw blog watch <config> --once. Without --once it loops.
• --dry-run shows what a tick would do now and writes nothing.
• --once never runs pip: if Markdown or Pygments are missing it exits 1. Run another fw blog command once to install them.

Config keys (top level only)

"autodeploy": { "quiet_seconds": 300 },     // whole number, 60 or more; turns the deploy guard on
"mirror": {
  "repo": "~/code/blog-mirror",             // required; ~ expanded; relative = relative to the config's folder
  "branch": "dropbox-mirror",               // default; may not be main, master or HEAD
  "remote": "origin",                       // no remote means no push
  "exclude": ["*.tmp"]                      // extra rsync patterns; no `..`, none starting with ! + - :
}

autodeploy and mirror each work without the other. The quiet period is autodeploy.quiet_seconds; without an autodeploy key it is a fixed 300 s, also for the mirror.

State

In ~/.fw-blog-watch/<config folder, lowercased>-<8 hex of the config path's hash>/ (FW_BLOG_WATCH_DIR replaces the whole path; FW_BLOG_WATCH_NOTIFY=0 turns the macOS notifications off, and the tests set it).

One tick

1. Lock. Take a non-blocking flock on lock. If it is held, exit quietly; held for over 30 minutes gives one "watch tick stuck" notification.
2. Fingerprint the source (sha256 of every file's path and content). .DS_Store, .dropbox*, .git and Dropbox "conflicted copy" files are ignored; links are not followed.
3. Wait for quiet. A changed fingerprint restarts the timer; nothing happens until the folder has been quiet for the quiet period. - Only source is fingerprinted. Editing the config or the comments file does not start a tick (the next source change, or a manual fw blog deploy, picks it up). - A "conflicted copy" file blocks everything (notification, retried each tick). - If the source changes while the mirror or the build runs, the tick stops ("changed during build") and the new quiet period applies.
4. Mirror (if configured), below.
5. Deploy guard (only with autodeploy), below.

Mirror

The repo must be a git repo with the mirror branch checked out, no merge or rebase in progress, and a clean tree. Git-ignored files count as changes, so rsync --delete can never remove them. Otherwise the tool notifies and stops the tick (it is retried; it never overwrites). Only an absent or empty folder is auto-initialised (git init -b <branch>).

Then:

1. rsync -a --delete source into the repo (your excludes, then the protected .git);
2. git add -A;
3. a commit Mirror of <source folder> at <UTC time> if anything changed (your git identity; no trailers; hooks skipped; other branches are never touched);
4. with remote, push refs/heads/<branch>:refs/heads/<branch> (never forced). A failed push is notified, but the deploy goes on.

If a step fails halfway (rsync exit 23/24 is normal while Dropbox syncs, and 24 is retried silently; git add or the commit can also fail), the tool records exactly what it left behind in state.json as mirror_dirty: the repo's real path, HEAD, the git status --porcelain --ignored listing, and a hash of those files' contents. The same record is made after a successful run that leaves git-ignored files copied from a source .gitignore.

• A dirty tree is accepted on a later tick only if the repo, HEAD, the listing and the contents are all still exactly that.
• The record disappears after a run that leaves nothing behind.
• Any other difference (an edited tracked file, a new path) is refused until you look at it, and your edit is never overwritten.

Deploy guard

Builds into a temporary folder (same code as fw blog build) and compares with deployed.json.

Held-back changes wait for you to run fw blog deploy <config>, which rewrites all of these baselines. So code, template and config changes always wait for a manual deploy.

Notifications and failures

• macOS notifications (osascript) are sent for held-back changes, failures, conflict files, mirror problems and unexpected errors, never for a successful deploy.
• A build or deploy failure notifies once and is not retried until the next source change.
• An unexpected exception is logged and notified once, and the tick still exits 0 (so launchd does not throttle the agent).
• Every subprocess has a timeout (git 120 s, mirror and deploy rsync 600 s, osascript 10 s). A timeout is an ordinary failure.

Manual deploys while the watcher runs

A manual real fw blog deploy (not --dry-run):

• takes the same lock (if a tick holds it: "a watch tick is running, try again in a minute");
• refuses if the watcher deployed while you were answering the question (run it again);
• rewrites deployed.json, which keeps the baseline current;
• clears the watcher's "handled" state so the next tick re-evaluates.

The launchd agent

install writes ~/Library/LaunchAgents/net.fuzzyworld.fw-blog-watch-<folder>-<hash>.plist (every 60 s and at load, low priority) and loads it with launchctl bootstrap gui/$UID (retried up to 5 times). uninstall unloads and removes it. install --print only prints the plist.

WordPress import

lib/wp_import.py (fw blog import-wp <config> --wp URL) reads the public REST API.

• Skips password-protected posts.
• Matches each post to a source file: similarity of 0.95 or more uses your file; otherwise the WordPress text is written to wordpress/<date>-<slug>.md. A non-ASCII slug gives a readable file name but keeps its URL.
• Writes comments (name, date and text only, into the comments file next to the config), pages (with published: true), media and the config entries.
• --dry-run writes only a cache and wordpress/_report.txt. A second run changes nothing.

fw blog check <config> lists posts whose built text differs from WordPress.

Tests

python3 -m unittest discover tests covers the blog, the WordPress importer, the watcher, the comments and the runner.