fuzzyworld-tools / files / docs / blog.md
blog.md
bb7e1f57 2026-11-06 Pavel Safronov | 1 | # Blog (`fw blog`) |
| 2 | ||
| 3 | A generic static-blog generator: a folder of Markdown files and one JSON config become a site (`server/blog.py`). | |
| 4 | It is used for writing.fuzzyworld.net (and til.fuzzyworld.net). | |
| 5 | ||
| 6 | It runs on the **laptop**, builds into a clean temporary folder and uploads only that folder with `rsync`, so | |
| 7 | nothing in the source folder (drafts, notes, config) is ever published. | |
| 8 | ||
| 9 | ```bash | |
| 10 | fw blog list ~/Blog/fw-blog.json # every source file: URL, title, date, tags, published or not | |
| 11 | fw blog build ~/Blog/fw-blog.json --out DIR [--no-htaccess] # write the site into DIR (other options are refused) | |
| 12 | fw blog preview ~/Blog/fw-blog.json [port] # build and serve it locally (no redirects, no feed content type) | |
| 13 | fw blog deploy ~/Blog/fw-blog.json --dry-run # then, for real: --yes (or answer the prompt) | |
| 14 | fw blog watch ~/Blog/fw-blog.json [--once] [--dry-run] # mirror to git and auto-deploy edits | |
| 15 | fw blog watch install|uninstall ~/Blog/fw-blog.json [--print] # the launchd agent that runs it | |
| 16 | ``` | |
| 17 | ||
| 18 | Python-Markdown and Pygments are installed into `~/fw/pylib` on the laptop the first time `fw blog` runs. | |
| 19 | ||
| 20 | ## What is published | |
| 21 | ||
| 22 | - **Posts:** only those with `published: true`, in the file's front matter or in the config's `posts` table keyed by file name (front matter wins). | |
| 23 | - **Pages** (files in `pages_dir`) are opt-in the same way: a page needs `published: true` in its front matter (the importer writes it). | |
| 24 | - **Never reaches the output:** text under `## Spelling` and `## Removed writing`, and every HTML comment (except `<!--more-->`). | |
| 25 | - **The build stops** (rather than risk a leak) when a heading that looks private is written another way (`##Spelling`, an underlined heading) or a code fence is never closed. | |
| 26 | ||
| 27 | ## What it builds | |
| 28 | ||
| 29 | | Output | Notes | | |
| 30 | |---|---| | |
| 31 | | Posts at `/YYYY/MM/DD/<slug>/` | | | |
| 32 | | `/blog/` | every post by year and month, with excerpts and a "Continue reading" link | | |
| 33 | | `/tag/<slug>/` | | | |
| 34 | | Pages | | | |
| 35 | | Atom feed at `/feed/` | `server/atom.py` | | |
| 36 | | `.htaccess` | redirects from `redirects` and `wp_ids`, no script execution, a security policy that allows only the site's own `blog.js` | | |
| 37 | | `robots.txt` | blocks AI crawlers | | |
| 38 | | Comments | as static text | | |
| 39 | ||
| 40 | **Reader features** (`server/site-blog/blog.js`): a light/dark switch (the choice is kept in the browser; without | |
| 41 | one the system setting decides) and a reading-progress bar on posts. | |
| 42 | ||
| 43 | ## Config | |
| 44 | ||
| 45 | `title`, `base_url` and `source` are required; unknown keys are an error. The full list is at the top of | |
| 46 | `server/blog.py`. | |
| 47 | ||
| 48 | | Group | Keys | | |
| 49 | |---|---| | |
| 50 | | Identity | `title`, `tagline`, `author`, `copyright` (`{years}` and `{author}` are filled in), `url`, `base_url` | | |
| 51 | | Content | `source`, `include` (source file patterns, default `*.md`), `pages_dir`, `media`, `body_section`, `line_breaks`, `posts`, `tags`, `home`, `nav`, `reading_wpm`, `timezone` | | |
| 52 | | Output | `feed`, `redirects`, `wp_ids`, `rewrite_hosts`, `site` | | |
| 53 | | Comments | `comments`, `comments_site`, `comments_mailbox`, `comments_form` | | |
| 54 | | Deploy | `deploy`, `mirror`, `autodeploy` | | |
| 55 | ||
| 56 | **Paths:** `source` and `comments` are relative to the config file's folder; `pages_dir` and `media` are relative to `source`. | |
| 57 | ||
| 58 | **`site`** puts the site-wide bar above the blog's own bar (as on code and apps): | |
| 59 | `{"name": "fuzzyworld", "url": "https://fuzzyworld.net/", "nav": [[label, url], ...], "current": "Writing"}`. | |
| 60 | `"nav": "gen"` takes the fuzzyworld nav from `NAV` in `server/gen.py` (the one source) and also shows the shared | |
| 61 | four-column footer (`gen.site_footer()`, with the blog's copyright and Feed line under it). The landing page's nav | |
| 62 | and footer are built from the same functions by `gen.py landing` in `install.sh` (`<!--SITE-NAV-->`, | |
| 63 | `<!--SITE-FOOTER-->`). The landing page's text is Markdown in `server/site/landing.md`, rendered into the | |
| 64 | `<!--LANDING-TEXT-->` spot of `server/site/index.html`. | |
| 65 | ||
| 66 | ## Safety checks | |
| 67 | ||
| 68 | | Check | Rule | | |
| 69 | |---|---| | |
| 70 | | Slugs | posts, pages, tags and the feed path may use only letters, digits and `. _ % -` | | |
| 71 | | `.htaccess` values | `redirects`, `wp_ids`, `home` and `feed.path` may not contain spaces or control characters, even percent-encoded | | |
| 72 | | `deploy.dir` | must be a site folder such as `writing.fuzzyworld.net` (relative to the login folder); `~`, `.`, `/`, `..` are refused | | |
| 73 | | Media | symbolic links in the media folder are skipped | | |
| 74 | ||
| 75 | ## Deploy and upload safety | |
| 76 | ||
| 77 | `fw blog deploy` audits the built folder first and refuses anything that is not a generated file or an allowlisted | |
| 78 | media type (`png jpg jpeg gif webp svg pdf epub kpf mp3 mp4 zip`). It then rsyncs only that folder, and keeps | |
| 79 | `.well-known`. | |
| 80 | ||
| 81 | - It asks for confirmation on every real upload. It holds the watcher's lock only for the upload itself, not while the question is open. Ctrl-C at the question exits quietly with 1. | |
| 82 | - `--dry-run` shows what would be sent. | |
| 83 | - The watcher limits the upload to 600 s; a manual deploy has no time limit. | |
| 84 | ||
| 85 | ## Comments by email | |
| 86 | ||
| 87 | Set `"comments_site": "writing"` (or `"til"`; a key of `server/comments/sites.json`) and | |
| 88 | `"comments_mailbox": "<address>"` in the config. Each post then gets a "Comment by email" link (subject | |
| 89 | `Comment [writing:/2024/05/01/slug/]`) and a container that `blog.js` fills from `/_comments/<post path>.json`. | |
| 90 | ||
| 91 | ### Moderating | |
| 92 | ||
| 93 | `fw comments list|show|approve [--trust]|reject|remove|untrust|publish|export|ingest|imap-check` runs on the server | |
| 94 | (or use the [admin panel](admin.md)). | |
| 95 | ||
| 96 | - `list --json` prints the pending comments in full plus the 20 newest approved ones, with each post's URL and whether the sender is trusted, never a sender hash. `fw admin` uses it. | |
| 97 | - A cron job runs `ingest` hourly ([operations.md](operations.md#scheduled-jobs)). | |
| 98 | - `fw blog deploy` leaves `/_comments/` alone. | |
| 99 | ||
| 100 | ### The mailbox | |
| 101 | ||
| 102 | `ingest` reads the comments mailbox: | |
| 103 | ||
| 104 | - **Hosted mailbox over IMAP** when `~/fw/comments/imap.json` exists: `{"host", "user", "password", "folder"}`, mode 600, created by hand on the server. TLS with certificate check. Processed mail moves to the folders `comments-done` and `comments-bad` (with the server's folder prefix such as `INBOX.` added when it needs one); `done_folder` and `bad_folder` in the file override the names. | |
| 105 | - **Otherwise a Maildir** folder `~/Maildir/.comments`. | |
| 106 | - `fw comments imap-check` prints only header names and the Authentication-Results result tokens, to learn the right `authserv_id`. | |
| 107 | ||
| 108 | ### Trust and limits | |
| 109 | ||
| 110 | - `ingest` accepts a message only when the topmost `Authentication-Results` header shows a DMARC pass for the From domain (or an aligned DKIM pass), and was written by the receiving mail server. That server's name must be set as `authserv_id` in `server/comments/sites.json`. Empty means ingest refuses to read anything. Set it in the repo copy, because an install overwrites the server's. | |
| 111 | - That trust holds only if the receiving server always adds its own header above anything the sender supplied. | |
| 112 | - The address is never shown. Only a salted hash is kept (losing `~/fw/comments/secret` invalidates trusted senders). | |
| 113 | - Limits: 4000 characters, 5 comments per sender per 24 hours, 200 pending. | |
| 114 | - State lives in `~/fw/comments/`, outside every web root. | |
| 115 | ||
| 116 | ### The optional form | |
| 117 | ||
| 118 | `comments_form` (an `https` action) adds a plain HTML form for the endpoint `server/comments/post.php`. It is **off, | |
| 119 | and the endpoint is not installed** ([hosting tiers](../README.md#hosting-tiers)). It is written, and its tests pass | |
| 120 | on the server's PHP; an optional `self_origin` key in `sites.json` names the origin its no-JavaScript confirm page | |
| 121 | posts from. | |
| 122 | ||
| 123 | ## The watcher (`fw blog watch`) | |
| 124 | ||
| 125 | `server/blog_watch.py` mirrors the blog source into a git repo and auto-deploys edits that change no URLs, for ONE | |
| 126 | blog config. **Run it on ONE machine only.** The MacBook Air just keeps writing through Dropbox, and its edits are | |
| 127 | captured by the mirror on the watching machine. A manual `fw blog deploy` on another machine does not update this | |
| 128 | machine's baseline; the result is a harmless re-deploy or a hold. | |
| 129 | ||
| 130 | - One tick every 60 s: the launchd agent runs `fw blog watch <config> --once`. Without `--once` it loops. | |
| 131 | - `--dry-run` shows what a tick would do now and writes nothing. | |
| 132 | - `--once` never runs `pip`: if Markdown or Pygments are missing it exits 1. Run another `fw blog` command once to install them. | |
| 133 | ||
| 134 | ### Config keys (top level only) | |
| 135 | ||
| 136 | ```jsonc | |
| 137 | "autodeploy": { "quiet_seconds": 300 }, // whole number, 60 or more; turns the deploy guard on | |
| 138 | "mirror": { | |
| 139 | "repo": "~/code/blog-mirror", // required; ~ expanded; relative = relative to the config's folder | |
| 140 | "branch": "dropbox-mirror", // default; may not be main, master or HEAD | |
| 141 | "remote": "origin", // no remote means no push | |
| 142 | "exclude": ["*.tmp"] // extra rsync patterns; no `..`, none starting with ! + - : | |
| 143 | } | |
| 144 | ``` | |
| 145 | ||
| 146 | `autodeploy` and `mirror` each work without the other. The quiet period is `autodeploy.quiet_seconds`; without an | |
| 147 | `autodeploy` key it is a fixed 300 s, also for the mirror. | |
| 148 | ||
| 149 | ### State | |
| 150 | ||
| 151 | In `~/.fw-blog-watch/<config folder, lowercased>-<8 hex of the config path's hash>/` (`FW_BLOG_WATCH_DIR` replaces | |
| 152 | the whole path; `FW_BLOG_WATCH_NOTIFY=0` turns the macOS notifications off, and the tests set it). | |
| 153 | ||
| 154 | | File | Contents | | |
| 155 | |---|---| | |
| 156 | | `state.json` | | | |
| 157 | | `watch.log` | one line per decision | | |
| 158 | | `deployed.json` | the baseline: `{files: {path: sha256}, time, generator, config, target}` | | |
| 159 | | `hashes.json` | a size+mtime cache so media are not read every minute | | |
| 160 | | `lock` | | | |
| 161 | | `launchd.log` | | | |
| 162 | ||
| 163 | ### One tick | |
| 164 | ||
| 165 | 1. **Lock.** Take a non-blocking `flock` on `lock`. If it is held, exit quietly; held for over 30 minutes gives one "watch tick stuck" notification. | |
| 166 | 2. **Fingerprint** the source (sha256 of every file's path and content). `.DS_Store`, `.dropbox*`, `.git` and Dropbox "conflicted copy" files are ignored; links are not followed. | |
| 167 | 3. **Wait for quiet.** A changed fingerprint restarts the timer; nothing happens until the folder has been quiet for the quiet period. | |
| 168 | - **Only `source` is fingerprinted.** Editing the config or the comments file does not start a tick (the next source change, or a manual `fw blog deploy`, picks it up). | |
| 169 | - A "conflicted copy" file blocks everything (notification, retried each tick). | |
| 170 | - If the source changes while the mirror or the build runs, the tick stops ("changed during build") and the new quiet period applies. | |
| 171 | 4. **Mirror** (if configured), below. | |
| 172 | 5. **Deploy guard** (only with `autodeploy`), below. | |
| 173 | ||
| 174 | ### Mirror | |
| 175 | ||
| 176 | The repo must be a git repo with the mirror branch checked out, no merge or rebase in progress, and a clean tree. | |
| 177 | **Git-ignored files count as changes**, so `rsync --delete` can never remove them. Otherwise the tool notifies and | |
| 178 | stops the tick (it is retried; it never overwrites). Only an absent or empty folder is auto-initialised | |
| 179 | (`git init -b <branch>`). | |
| 180 | ||
| 181 | Then: | |
| 182 | ||
| 183 | 1. `rsync -a --delete` source into the repo (your excludes, then the protected `.git`); | |
| 184 | 2. `git add -A`; | |
| 185 | 3. a commit `Mirror of <source folder> at <UTC time>` if anything changed (your git identity; no trailers; hooks skipped; other branches are never touched); | |
| 186 | 4. with `remote`, push `refs/heads/<branch>:refs/heads/<branch>` (never forced). A failed push is notified, but the deploy goes on. | |
| 187 | ||
| 188 | **If a step fails halfway** (rsync exit 23/24 is normal while Dropbox syncs, and 24 is retried silently; `git add` | |
| 189 | or the commit can also fail), the tool records exactly what it left behind in `state.json` as `mirror_dirty`: the | |
| 190 | repo's real path, HEAD, the `git status --porcelain --ignored` listing, and a hash of those files' contents. The same | |
| 191 | record is made after a successful run that leaves git-ignored files copied from a source `.gitignore`. | |
| 192 | ||
| 193 | - A dirty tree is accepted on a later tick only if the repo, HEAD, the listing and the contents are all still exactly that. | |
| 194 | - The record disappears after a run that leaves nothing behind. | |
| 195 | - Any other difference (an edited tracked file, a new path) is refused until you look at it, and your edit is never overwritten. | |
| 196 | ||
| 197 | ### Deploy guard | |
| 198 | ||
| 199 | Builds into a temporary folder (same code as `fw blog build`) and compares with `deployed.json`. | |
| 200 | ||
| 201 | | Situation | Result | | |
| 202 | |---|---| | |
| 203 | | no baseline | **held back** with a notification | | |
| 204 | | the **generator hash** changed (`blog.py`, `atom.py`, `gen.py`, the shared CSS/icon, `site-blog/*`; fixed when the process starts) | held back | | |
| 205 | | the **config** changed (hash of the loaded config) | held back | | |
| 206 | | the **deploy target** changed | held back | | |
| 207 | | any output file added or removed | held back | | |
| 208 | | same paths, no content difference | logged "no visible change" | | |
| 209 | | same paths, some content changed | deployed through the same function as `fw blog deploy` (audit and rsync, no question) | | |
| 210 | ||
| 211 | Held-back changes wait for you to run `fw blog deploy <config>`, which rewrites all of these baselines. So code, | |
| 212 | template and config changes always wait for a manual deploy. | |
| 213 | ||
| 214 | ### Notifications and failures | |
| 215 | ||
| 216 | - macOS notifications (`osascript`) are sent for held-back changes, failures, conflict files, mirror problems and unexpected errors, **never for a successful deploy**. | |
| 217 | - A build or deploy failure notifies once and is **not retried until the next source change**. | |
| 218 | - An unexpected exception is logged and notified once, and the tick still exits 0 (so launchd does not throttle the agent). | |
| 219 | - Every subprocess has a timeout (git 120 s, mirror and deploy rsync 600 s, osascript 10 s). A timeout is an ordinary failure. | |
| 220 | ||
| 221 | ### Manual deploys while the watcher runs | |
| 222 | ||
| 223 | A manual real `fw blog deploy` (not `--dry-run`): | |
| 224 | ||
| 225 | - takes the same lock (if a tick holds it: "a watch tick is running, try again in a minute"); | |
| 226 | - refuses if the watcher deployed while you were answering the question (run it again); | |
| 227 | - rewrites `deployed.json`, which keeps the baseline current; | |
| 228 | - clears the watcher's "handled" state so the next tick re-evaluates. | |
| 229 | ||
| 230 | ### The launchd agent | |
| 231 | ||
| 232 | `install` writes `~/Library/LaunchAgents/net.fuzzyworld.fw-blog-watch-<folder>-<hash>.plist` (every 60 s and at | |
| 233 | load, low priority) and loads it with `launchctl bootstrap gui/$UID` (retried up to 5 times). `uninstall` unloads | |
| 234 | and removes it. `install --print` only prints the plist. | |
| 235 | ||
| 236 | ## WordPress import | |
| 237 | ||
| 238 | `lib/wp_import.py` (`fw blog import-wp <config> --wp URL`) reads the public REST API. | |
| 239 | ||
| 240 | - Skips password-protected posts. | |
| 241 | - Matches each post to a source file: similarity of 0.95 or more uses your file; otherwise the WordPress text is written to `wordpress/<date>-<slug>.md`. A non-ASCII slug gives a readable file name but keeps its URL. | |
| 242 | - Writes comments (name, date and text only, into the `comments` file next to the config), pages (with `published: true`), media and the config entries. | |
| 243 | - `--dry-run` writes only a cache and `wordpress/_report.txt`. A second run changes nothing. | |
| 244 | ||
| 245 | `fw blog check <config>` lists posts whose built text differs from WordPress. | |
| 246 | ||
| 247 | ## Tests | |
| 248 | ||
| 249 | `python3 -m unittest discover tests` covers the blog, the WordPress importer, the watcher, the comments and the runner. |