fuzzyworld-tools / commit
fw blog watch: mirror a blog's source into a git repo and auto-deploy edits to published posts
PLAN.md | 9 + README.md | 57 ++- bin/fw | 38 +- package.json | 2 +- server/blog.py | 233 +++++++++- server/blog_watch.py | 735 ++++++++++++++++++++++++++++++++ tests/test_blog_watch.py | 1057 ++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 2106 insertions(+), 25 deletions(-) diff --git a/PLAN.md b/PLAN.md index ee4510a..44459b3 100644 --- a/PLAN.md +++ b/PLAN.md @@ -152,6 +152,15 @@ basic auth first; client-side encryption only if needed (it adds a script and al - Reuse what exists: Python-Markdown, Pygments, the site CSS, the feed code from step 1. - **Import (🔌 plugin):** one-time script that reads the WordPress API: posts, comments, media, and a redirect map. - The editor and the Dropbox → private-git-repo move come after the generator, not with it. +- **Built 2026-11-02 (Q91/Q92):** `fw blog watch <config>` (`server/blog_watch.py`; README "Blog", "Watcher"): a laptop + launchd agent (`fw blog watch install`) runs one tick every 60 s per blog config. After a quiet period it mirrors the + Dropbox source into a local git repo (snapshot commits on `dropbox-mirror`, never main) and, with `autodeploy`, deploys + edits only when the set of output files, the generator (code/templates), the config and the deploy target are + unchanged (anything else is held back with a notification; the baseline is `deployed.json`, also written by a manual `fw blog deploy` and guarded by the + state folder's lock). Run on ONE machine only. **Not built yet:** + the cutover `fw blog move <config> --to repo|dropbox` (copy, build both, require identical output, repoint the `~/Blog` + symlink, keep the old copy), then Dropbox is dropped and the Air uses a clone plus a simple `fw blog sync`; the server + remote and the push-hook build (Q25). - Needs the `blog.fuzzyworld.net` subdomain pointed at a folder (DreamHost panel). ## Comments diff --git a/README.md b/README.md index b630563..eb53309 100644 --- a/README.md +++ b/README.md @@ -101,6 +101,7 @@ On the laptop, `bin/fw` runs `deploy` locally and forwards repo commands to Drea | `fw ci-list` | queued and running CI jobs | | `fw ci-watch` | follow CI in the terminal (private repos too): running jobs with each job's state, the queue, the last 5 results; ends when nothing is queued or running | | `fw blog list\|build\|preview\|deploy <config>` | the static blog (see [Blog](#blog)); runs on the laptop | +| `fw blog watch <config> [--once] [--dry-run]` / `fw blog watch install\|uninstall <config>` | mirror the blog source into a git repo and auto-deploy edits that change no URLs (see [Blog](#blog)); one machine only | | `fw blog import-wp <config> --wp URL` / `fw blog check <config>` | one-time WordPress import / compare the built site with the WordPress text | Push to a repo with `git push dh:git/<name>.git` (`dh` is the SSH alias in `~/.ssh/config`). @@ -301,6 +302,8 @@ fw blog list ~/Blog/fw-blog.json # every source file: URL, title, dat fw blog build ~/Blog/fw-blog.json --out DIR [--no-htaccess] # write the site into DIR (other options are refused) fw blog preview ~/Blog/fw-blog.json [port] # build and serve it locally (no redirects, no feed content type) fw blog deploy ~/Blog/fw-blog.json --dry-run # then, for real: --yes (or answer the prompt) +fw blog watch ~/Blog/fw-blog.json [--once] [--dry-run] # mirror to git and auto-deploy edits (below) +fw blog watch install|uninstall ~/Blog/fw-blog.json [--print] # the launchd agent that runs it ``` - **What is published:** only posts with `published: true` (front matter in the file, or the config's `posts` table keyed @@ -315,7 +318,8 @@ fw blog deploy ~/Blog/fw-blog.json --dry-run # then, for real: --yes (or answer `~`, `.`, `/`, `..` are refused); symbolic links in the media folder are skipped. - **Config:** `title`, `base_url` and `source` are required; unknown keys are an error. Others: `tagline`, `author`, `copyright` (`{years}` and `{author}` are filled in), `body_section`, `line_breaks`, `url`, `home`, `pages_dir`, `nav`, `tags`, `feed`, - `reading_wpm`, `media`, `rewrite_hosts`, `redirects`, `wp_ids`, `comments`, `timezone`, `deploy`, `posts`, `site`; see the + `reading_wpm`, `media`, `rewrite_hosts`, `redirects`, `wp_ids`, `comments`, `timezone`, `deploy`, `posts`, `site`, + `mirror`, `autodeploy` (see "Watcher" below); see the top of `server/blog.py`. `source` and `comments` are relative to the config file's folder; `pages_dir` and `media` are relative to `source`. `site` puts the site-wide bar above the blog's own bar (as on code and apps): `{"name": "fuzzyworld", "url": "https://fuzzyworld.net/", "nav": [[label, url], ...], "current": "Writing"}`; @@ -350,7 +354,56 @@ fw blog deploy ~/Blog/fw-blog.json --dry-run # then, for real: --yes (or answer [Hosting tiers](#hosting-tiers)). - **Upload safety:** `deploy` audits the built folder first and refuses anything that is not a generated file or an allowlisted media type (`png jpg jpeg gif webp svg pdf epub kpf mp3 mp4 zip`), then rsyncs only that folder (it - keeps `.well-known`). It asks for confirmation on every real upload; `--dry-run` shows what would be sent. + keeps `.well-known`; the watcher limits it to 600 s, a manual deploy has no time limit). It asks for confirmation on every real upload (it holds the watcher's lock only for the upload itself, not while the question is open; ctrl-c at the question exits quietly with 1); `--dry-run` shows what would be sent. +- **Watcher** (`server/blog_watch.py`, `fw blog watch <config>`): for ONE blog config, one tick every 60 s (the launchd + agent runs `fw blog watch <config> --once`; without `--once` it loops; `--dry-run` shows what a tick would do now and + writes nothing). **Run it on ONE machine only**: the MacBook Air just keeps writing through Dropbox, and its edits are + captured by the mirror on the watching machine. A manual `fw blog deploy` on another machine does not update this + machine's baseline; the result is a harmless re-deploy or a hold. Config keys (top level only): + `"autodeploy": {"quiet_seconds": 300}` (whole number, 60 or more; turns the deploy guard on) and + `"mirror": {"repo": "~/code/blog-mirror", "branch": "dropbox-mirror", "remote": "origin", "exclude": ["*.tmp"]}` + (`repo` required, `~` expanded, a relative path is relative to the config's folder; `branch` defaults to + `dropbox-mirror` and may not be `main`, `master` or `HEAD`; no `remote` means no push; `exclude` adds rsync patterns, + no `..`, none starting with `! + - :`). Each works without the other. The quiet period is `autodeploy.quiet_seconds`; + without an `autodeploy` key it is a fixed 300 s, also for the mirror. State is in + `~/.fw-blog-watch/<config folder, lowercased>-<8 hex of the config path's hash>/` (`FW_BLOG_WATCH_DIR` replaces the + whole path; `FW_BLOG_WATCH_NOTIFY=0` turns the macOS notifications off, and the tests set it): `state.json`, `watch.log` (one line per decision), `deployed.json`, `hashes.json` (a size+mtime cache so + media are not read every minute), `lock`, `launchd.log`. A tick: take a non-blocking `flock` on `lock` (held: exit + quietly; held for over 30 minutes: one "watch tick stuck" notification); fingerprint the source (sha256 of every + file's path and content; `.DS_Store`, `.dropbox*`, `.git` and Dropbox "conflicted copy" files are ignored; links are not + followed); a changed fingerprint restarts the timer; nothing happens until the folder has been quiet for the quiet + period. **Only `source` is fingerprinted**: editing the config or the comments file does not start a tick (the next + source change, or a manual `fw blog deploy`, picks it up). A "conflicted copy" file blocks everything (notification, + retried each tick). If the source changes while the mirror or the build runs, the tick stops ("changed during build") + and the new quiet period applies. **Mirror:** the repo must be a git repo with the mirror branch checked out, no merge + or rebase in progress, and a clean tree where **git-ignored files count as changes** (so `rsync --delete` can never + remove them); otherwise it notifies and stops the tick (retried; it never overwrites). Only an absent or empty folder + is auto-initialised (`git init -b <branch>`). Then `rsync -a --delete` source into the repo (your excludes, then the + protected `.git`), `git add -A`, and a commit `Mirror of <source folder> at <UTC time>` if anything changed (your git + identity; no trailers; hooks skipped; other branches are never touched). With `remote`, it pushes + `refs/heads/<branch>:refs/heads/<branch>` (never forced); a failed push is notified but the deploy goes on. If the + rsync (exit 23/24 are normal while Dropbox syncs; 24 is retried silently), `git add` or the commit fails halfway, the + tool records exactly what it left behind in `state.json` (`mirror_dirty`: the repo's real path, HEAD, the + `git status --porcelain --ignored` listing and a hash of those files' contents; the same record is made after a + successful run that leaves git-ignored files copied from a source `.gitignore`). A dirty tree is accepted on a later + tick only if the repo, HEAD, the listing and the contents are all still exactly that; the record disappears after a + run that leaves nothing behind. Any other difference (an edited tracked file, a new path) is refused until you look + at it, and your edit is never overwritten. **Deploy guard** (only with `autodeploy`): build into a temporary folder (same code as + `fw blog build`), compare with `deployed.json` (`{files: {path: sha256}, time, generator, config, target}`): no baseline, a changed + **generator hash** (blog.py, atom.py, gen.py, the shared CSS/icon, `site-blog/*`, fixed when the process starts), a + changed **config** (hash of the loaded config), a changed **deploy target**, or any added or removed output file is **held back** with a notification (run `fw blog deploy + <config>` yourself; so code, template and config changes wait for a manual deploy, which rewrites all of these); the same paths with no content difference is logged "no visible change"; the same paths with + some changed content is deployed through the same function as `fw blog deploy` (audit and rsync, no question). A build + or deploy failure notifies once and is **not retried until the next source change**. Notifications (macOS, + `osascript`) are sent for held-back changes, failures, conflict files, mirror problems and unexpected errors, never + for a successful deploy; an unexpected exception is logged and notified once and the tick still exits 0 (so launchd + does not throttle the agent). Every subprocess has a timeout (git 120 s, mirror and deploy rsync 600 s, osascript + 10 s); a timeout is an ordinary failure. A manual real `fw blog deploy` (not `--dry-run`) takes the same lock (if a + tick holds it: "a watch tick is running, try again in a minute"), refuses if the watcher deployed while you were answering the question (run it again), rewrites `deployed.json` (which keeps the + baseline current) and clears the watcher's "handled" state so the next tick re-evaluates. `install` writes `~/Library/LaunchAgents/net.fuzzyworld.fw-blog-watch-<folder>-<hash>.plist` + (every 60 s and at load, low priority) and loads it with `launchctl bootstrap gui/$UID` (retried up to 5 times); + `uninstall` unloads and removes it; `install --print` only prints the plist. `fw blog watch ... --once` never runs + `pip`: if Markdown or Pygments are missing it exits 1; run another `fw blog` command once to install them. - **WordPress import** (`lib/wp_import.py`, `fw blog import-wp`): reads the public REST API, skips password-protected posts, matches each post to a source file (similarity of 0.95 or more uses your file; otherwise the WordPress text is written to `wordpress/<date>-<slug>.md`; a non-ASCII slug gives a readable file name but keeps its URL), and writes diff --git a/bin/fw b/bin/fw index 844d0c8..1e24f8a 100755 --- a/bin/fw +++ b/bin/fw @@ -5,10 +5,13 @@ # fw list | new <name> [--public] [--description TEXT] | public <name> | private <name> # fw usage [--all|--now] Claude Code token use per session (see lib/usage.py) # fw blog list|build|preview|deploy <config> static blog from Markdown (see server/blog.py); -# build takes --out DIR and --no-htaccess +# build takes --out DIR and --no-htaccess; +# deploy takes --dry-run and --yes (else it asks) # fw blog import-wp <config> --wp URL [--dry-run] [--force] [--refresh] [--probe-links] # import the WordPress blog (see lib/wp_import.py) # fw blog check <config> [--out DIR] compare the built site with the cached WordPress text +# fw blog watch <config> [--once] [--dry-run] mirror the source into git and auto-deploy edits (see server/blog_watch.py); +# fw blog watch install|uninstall <config> [--print] the launchd agent that runs it every 60 s (ONE machine only) # fw describe <name> TEXT | publish <name> | publish-all | ci <name> [branch] | ci-list | ci-watch # fw comments list|show|approve|reject|remove|untrust|publish|export|ingest|imap-check (blog comments) # (run on DreamHost over ssh) @@ -23,12 +26,17 @@ blog() { tmp=${tmp%/} local pylib=${FW_PYLIB:-$HOME/fw/pylib} blogpy="$HERE/server/blog.py" case "$sub" in - list|build|preview|deploy|import-wp|check) ;; - *) echo "usage: fw blog list|build|preview|deploy|import-wp|check <config> [options]" >&2; exit 2 ;; + list|build|preview|deploy|import-wp|check|watch) ;; + *) echo "usage: fw blog list|build|preview|deploy|import-wp|check|watch <config> [options]" >&2; exit 2 ;; esac [ -n "$config" ] || { echo "fw blog $sub: needs the config file (e.g. ~/Blog/fw-blog.json)" >&2; exit 2; } shift 2 if ! PYTHONPATH="$pylib" python3 -c 'import markdown, pygments' 2>/dev/null; then + # The launchd agent (watch --once) must never run pip: only an interactive command installs the libraries. + if [ "$sub" = watch ] && [[ " $* " == *" --once "* ]]; then + echo "fw: Markdown and Pygments are missing from $pylib; run any other fw blog command once to install them" >&2 + exit 1 + fi echo "fw: installing Markdown and Pygments into $pylib" >&2 python3 -m pip install --quiet --target "$pylib" Markdown==3.7 Pygments==2.21.0 fi @@ -56,26 +64,16 @@ blog() { python3 "$blogpy" build "$config" --out "$out" --no-htaccess echo "preview: http://127.0.0.1:$port/ (ctrl-c to stop)" exec python3 -m http.server --bind 127.0.0.1 --directory "$out" "$port" ;; + watch) + # One tick (--once, what the launchd agent runs) or a loop; or install/uninstall the agent: `watch install <config>`. + exec python3 "$HERE/server/blog_watch.py" "$config" "$@" ;; deploy) - # Uploads ONLY a freshly built, audited output folder: never the source folder. - local dry=0 yes=0 a + # Uploads ONLY a freshly built, audited output folder: never the source folder. Asks first unless --yes. + local args=() a for a in "$@"; do - case "$a" in --dry-run) dry=1 ;; --yes) yes=1 ;; *) echo "fw blog deploy: unknown option $a" >&2; exit 2 ;; esac + case "$a" in --dry-run|--yes) args+=("$a") ;; *) echo "fw blog deploy: unknown option $a" >&2; exit 2 ;; esac done - local out=$tmp/fw-blog-deploy target - target=$(python3 "$blogpy" deploy-target "$config") - python3 "$blogpy" build "$config" --out "$out" - python3 "$blogpy" audit "$out" - local rsync_opts=(-rlptz --delete --exclude .well-known --exclude .dh-diag --exclude .fw-blog-out --exclude /_comments/) - if [ "$dry" = 1 ]; then - exec rsync -n -v "${rsync_opts[@]}" "$out/" "$target/" - fi - if [ "$yes" != 1 ]; then - [ -t 0 ] || { echo "fw blog deploy: not a terminal; pass --yes to upload to $target" >&2; exit 1; } - read -r -p "Upload the output above to $target, deleting other files there? [y/N] " reply - [ "$reply" = y ] || [ "$reply" = Y ] || { echo "not uploaded" >&2; exit 1; } - fi - exec rsync "${rsync_opts[@]}" "$out/" "$target/" ;; + exec python3 "$blogpy" deploy "$config" --out "$tmp/fw-blog-deploy" ${args[@]+"${args[@]}"} ;; esac } diff --git a/package.json b/package.json index 3abe84e..28d204d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "fuzzyworld-tools", - "version": "1.3.0", + "version": "1.4.0", "description": "Tooling for code.fuzzyworld.net and apps.fuzzyworld.net: fw command, deploy, vite plugin.", "type": "module", "scripts": { diff --git a/server/blog.py b/server/blog.py index 70e53d4..2bcef65 100644 --- a/server/blog.py +++ b/server/blog.py @@ -4,6 +4,7 @@ blog.py list <config> every source file: path -> URL, title, date, tags (and whether it is published) blog.py build <config> --out DIR write the site into DIR (--no-htaccess for a local preview) blog.py deploy-target <config> print host:dir from the config (used by `fw blog deploy`) + blog.py deploy <config> --out DIR build, audit, ask, rsync (`fw blog deploy`); --dry-run, --yes blog.py audit DIR refuse (exit 1) if DIR holds anything that must not be uploaded `fw blog ...` is the usual way in (it also installs Markdown and Pygments into ~/fw/pylib). The config is @@ -28,14 +29,18 @@ import argparse import calendar import dataclasses import datetime +import fcntl import glob import html import json import math import os import re +import hashlib import shutil +import subprocess import sys +import time import urllib.parse import zoneinfo @@ -64,14 +69,15 @@ PAGE_FIELDS = ("published", "slug", "title") CONFIG_KEYS = {"title", "tagline", "base_url", "author", "copyright", "source", "include", "body_section", "line_breaks", "url", "home", "pages_dir", "nav", "tags", "feed", "reading_wpm", "media", "rewrite_hosts", "redirects", "wp_ids", "comments", "deploy", "posts", "timezone", "site", - "comments_site", "comments_mailbox", "comments_form"} + "comments_site", "comments_mailbox", "comments_form", "mirror", "autodeploy"} REQUIRED_KEYS = ("title", "base_url", "source") DEFAULTS = {"tagline": "", "author": "", "copyright": "© {years} {author}. All rights reserved.", "include": ["*.md"], "body_section": "Writing", "line_breaks": "markdown", "url": DEFAULT_URL, "home": None, "pages_dir": None, "nav": [], "tags": {}, "feed": {"path": "/feed/", "entries": 20}, "reading_wpm": 200, "media": None, "rewrite_hosts": [], "redirects": {}, "wp_ids": {}, "comments": None, "deploy": None, "posts": {}, "timezone": "UTC", "site": None, - "comments_site": None, "comments_mailbox": None, "comments_form": None} + "comments_site": None, "comments_mailbox": None, "comments_form": None, + "mirror": None, "autodeploy": None} class BlogError(Exception): @@ -150,6 +156,8 @@ def load_config(path): for slug in cfg["tags"]: check_slug(slug, "tags") check_comments_config(cfg) + check_watch_config(cfg) + cfg["_path"] = path cfg["_dir"] = os.path.dirname(path) cfg["_source"] = os.path.normpath(os.path.join(cfg["_dir"], cfg["source"])) return cfg @@ -174,6 +182,40 @@ def check_comments_config(cfg): raise BlogError('comments_form must be null or {"action": "https://..."} (an https URL without spaces or quotes)') +_BRANCH = re.compile(r"^[A-Za-z0-9._/][A-Za-z0-9._/-]*$") +_REMOTE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$") + + +def check_watch_config(cfg): + """`mirror` and `autodeploy` (both off unless set) drive `fw blog watch` (see blog_watch.py).""" + auto, mirror = cfg["autodeploy"], cfg["mirror"] + if auto is not None: + quiet = auto.get("quiet_seconds", 300) if isinstance(auto, dict) else None + if not (isinstance(auto, dict) and set(auto) <= {"quiet_seconds"} + and isinstance(quiet, int) and not isinstance(quiet, bool) and quiet >= 60): + raise BlogError('autodeploy must be {"quiet_seconds": whole number, at least 60 (optional, default 300)}') + if mirror is None: + return + if not (isinstance(mirror, dict) and set(mirror) <= {"repo", "branch", "remote", "exclude"}): + raise BlogError('mirror must be {"repo", "branch" (optional), "remote" (optional), "exclude" (optional)}') + repo = mirror.get("repo") + if not (isinstance(repo, str) and repo.strip()): + raise BlogError("mirror.repo must be a folder path (non-empty text)") + branch = mirror.get("branch", "dropbox-mirror") + if (not isinstance(branch, str) or not _BRANCH.match(branch) or ".." in branch or "//" in branch + or branch.endswith(("/", ".", ".lock")) or branch in ("main", "master", "HEAD")): + raise BlogError("mirror.branch must be a plain git branch name (letters, digits . _ / -; not starting with -), " + "and not main, master or HEAD") + remote = mirror.get("remote") + if remote is not None and not (isinstance(remote, str) and _REMOTE.match(remote)): + raise BlogError("mirror.remote must be a plain remote name (letters, digits . _ -)") + exclude = mirror.get("exclude", []) + if not (isinstance(exclude, list) and all(isinstance(x, str) and x.strip() and ".." not in x + and not x.startswith("/..") and "\n" not in x + and x[0] not in "!+-:" for x in exclude)): + raise BlogError('mirror.exclude must be a list of rsync patterns (text; no "..", not starting with ! + - :)') + + def check_slug(slug, what): """The slug, or BlogError: a slug becomes a folder name and part of a URL (and of .htaccess), so it is ASCII letters, digits and . _ % - only, and it must not decode to `.`, `..` or something with a slash.""" @@ -1005,6 +1047,185 @@ def build(cfg, out, htaccess=True): return f"{len(posts)} posts, {len(tags)} tags, {len(written)} pages, {media} media files -> {out}" +# ---------------------------------------------------------------- deploy + +# --delete removes whatever else is in the target folder, except these. +RSYNC_OPTS = ["-rlptz", "--delete", "--exclude", ".well-known", "--exclude", ".dh-diag", "--exclude", ".fw-blog-out", + "--exclude", "/_comments/"] + + +def watch_name(config_path): + """<folder name of the config, lowercased>-<first 8 hex of sha256 of its real path>: names the state folder + and the launchd label, so two configs in same-named folders never share state.""" + real = os.path.realpath(config_path) + return f"{os.path.basename(os.path.dirname(real)).lower()}-{hashlib.sha256(real.encode()).hexdigest()[:8]}" + + +def state_dir(config_path): + """Where `fw blog watch` and `fw blog deploy` keep their state for one blog: ~/.fw-blog-watch/<watch_name>/ + (the environment variable FW_BLOG_WATCH_DIR replaces the whole path, for tests).""" + return os.environ.get("FW_BLOG_WATCH_DIR") or os.path.join(os.path.expanduser("~"), ".fw-blog-watch", + watch_name(config_path)) + + +def acquire_lock(directory, now=time.time): + """The exclusive lock of one blog's state folder (a tick of the watcher, or a real manual deploy), as an open + file to close when done; None when someone holds it. The holder's start time is written into the file.""" + os.makedirs(directory, exist_ok=True) + f = open(os.path.join(directory, "lock"), "a+") + try: + fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + f.close() + return None + f.seek(0) + f.truncate(0) + f.write(repr(float(now()))) + f.flush() + return f + + +def lock_since(directory): + """The start time (epoch seconds) the current lock holder wrote, or None.""" + try: + with open(os.path.join(directory, "lock")) as f: + return float(f.read().strip()) + except (OSError, ValueError): + return None + + +def generator_hash(): + """sha256 of everything that shapes the output besides the posts: this generator, the feed code, gen.py (nav and + footer), the shared CSS/icon and the blog's own CSS/JS/robots/htaccess. `fw blog watch` deploys on its own only + while this equals the value stored in deployed.json.""" + files = [os.path.join(HERE, n) for n in ("blog.py", "atom.py", "gen.py")] + files += [os.path.join(SITE_DIR, n) for n in ("_shared.css", "favicon.ico")] + files += sorted(glob.glob(os.path.join(BLOG_DIR, "*"))) + h = hashlib.sha256() + for path in files: + if os.path.isfile(path): + h.update(os.path.relpath(path, HERE).encode() + b"\0") + with open(path, "rb") as f: + h.update(hashlib.sha256(f.read()).digest()) + return h.hexdigest() + + +def manifest(out): + """{relative path: sha256 of the content} of a built folder (the build's marker file is not part of the site).""" + files = {} + for dirpath, _, names in os.walk(out): + for name in names: + path = os.path.join(dirpath, name) + rel = os.path.relpath(path, out).replace(os.sep, "/") + if rel != MARKER: + with open(path, "rb") as f: + files[rel] = hashlib.sha256(f.read()).hexdigest() + return dict(sorted(files.items())) + + +def config_hash(cfg): + """sha256 of the loaded config (keys starting with _ are internal and left out).""" + public = {k: v for k, v in cfg.items() if not k.startswith("_")} + return hashlib.sha256(json.dumps(public, sort_keys=True, default=str).encode()).hexdigest() + + +def write_deployed(directory, files, when=None, generator=None, cfg=None): + """deployed.json: what the last real upload sent, the generator hash it was built with and (with `cfg`) the hash of + the config and the deploy target. `fw blog watch` compares the next build with it.""" + os.makedirs(directory, exist_ok=True) + when = when or datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + tmp = os.path.join(directory, "deployed.json.tmp") + with open(tmp, "w", encoding="utf-8") as f: + json.dump({"files": files, "time": when, "generator": generator or GENERATOR_HASH, + **({"config": config_hash(cfg), "target": deploy_target(cfg)} if cfg else {})}, f, indent=1) + os.replace(tmp, os.path.join(directory, "deployed.json")) + + +def _read_bytes(path): + try: + with open(path, "rb") as f: + return f.read() + except OSError: + return None + + +def _forget_fingerprint(directory): + """After a manual upload the watcher's idea of "handled" may be stale: drop it so the next tick re-evaluates.""" + path = os.path.join(directory, "state.json") + try: + with open(path, encoding="utf-8") as f: + state = json.load(f) + if not isinstance(state, dict) or not ({"fingerprint", "handled"} & set(state)): + return + state.pop("fingerprint", None) + state.pop("handled", None) + with open(path + ".tmp", "w", encoding="utf-8") as f: + json.dump(state, f, indent=1) + os.replace(path + ".tmp", path) + except (OSError, ValueError): + pass + + +def deploy_site(cfg, out, dry_run=False, confirm=None, run=subprocess.run, prebuilt=False, locked=False, timeout=None): + """Uploads ONLY a freshly built, audited output folder, never the source folder. Builds into `out` (unless + `prebuilt`: the caller just built it), audits it, then (not for a dry run) asks `confirm(target)` if given and + runs rsync (no time limit unless `timeout`). A real upload takes the state folder's lock around the upload only + (unless the caller `locked` it already: the watcher), and writes deployed.json. True when something was + uploaded; BlogError on any failure.""" + target = deploy_target(cfg) + directory = state_dir(cfg["_path"]) + baseline = _read_bytes(os.path.join(directory, "deployed.json")) # to notice a deploy by the watcher meanwhile + if not prebuilt: + print(build(cfg, out)) + count, size = audit_output(out) + print(f"audit ok: {count} files, {size / 1024:.0f} KB") + lock = None + if dry_run: + cmd = ["rsync", "-n", "-v", *RSYNC_OPTS, out + "/", target + "/"] + else: + if confirm and not confirm(target): + raise BlogError("not uploaded") + if not locked: + lock = acquire_lock(directory) + if lock is None: + raise BlogError("a watch tick is running, try again in a minute") + if _read_bytes(os.path.join(directory, "deployed.json")) != baseline: + lock.close() + raise BlogError("the watcher deployed while you were answering; run fw blog deploy again") + cmd = ["rsync", *RSYNC_OPTS, out + "/", target + "/"] + try: + try: + code = run(cmd, timeout=timeout).returncode + except subprocess.TimeoutExpired: + raise BlogError(f"rsync timed out after {timeout} s") + if code: + raise BlogError(f"rsync failed (exit {code})") + if not dry_run: + write_deployed(directory, manifest(out), cfg=cfg) + if not locked: + _forget_fingerprint(directory) + return not dry_run + finally: + if lock: + lock.close() + + +def ask_to_upload(target): + """The prompt of `fw blog deploy` (needs a terminal).""" + if not sys.stdin.isatty(): + raise BlogError(f"not a terminal; pass --yes to upload to {target}") + try: + return input(f"Upload the output above to {target}, deleting other files there? [y/N] ").strip() in ("y", "Y") + except EOFError: + return False + except KeyboardInterrupt: + print(file=sys.stderr) + raise SystemExit(1) + + +GENERATOR_HASH = generator_hash() # computed once, when the module loads: the code that is actually running + + # ---------------------------------------------------------------- command line def main(argv): @@ -1017,6 +1238,11 @@ def main(argv): b.add_argument("--no-htaccess", action="store_true", help="skip .htaccess (local preview)") sub.add_parser("audit").add_argument("out") sub.add_parser("deploy-target").add_argument("config") + d = sub.add_parser("deploy") + d.add_argument("config") + d.add_argument("--out", required=True) + d.add_argument("--dry-run", action="store_true") + d.add_argument("--yes", action="store_true", help="upload without asking") args = ap.parse_args(argv[1:]) try: if args.cmd == "audit": @@ -1028,6 +1254,9 @@ def main(argv): print("\n".join(list_posts(cfg))) elif args.cmd == "build": print(build(cfg, os.path.abspath(args.out), htaccess=not args.no_htaccess)) + elif args.cmd == "deploy": + deploy_site(cfg, os.path.abspath(args.out), dry_run=args.dry_run, + confirm=None if args.yes else ask_to_upload) else: print(deploy_target(cfg)) except BlogError as e: diff --git a/server/blog_watch.py b/server/blog_watch.py new file mode 100644 index 0000000..48f93fd --- /dev/null +++ b/server/blog_watch.py @@ -0,0 +1,735 @@ +#!/usr/bin/env python3 +"""`fw blog watch`: mirror a blog's source folder into a local git repo and deploy edits to published posts. + + blog_watch.py <config> [--once] [--dry-run] one tick (--once, what launchd runs every 60 s), or a loop + blog_watch.py install|uninstall <config> [--print] the launchd agent (--print only shows the plist) + +Run it on ONE machine. Per tick, for one blog config: take a lock, fingerprint the source folder, wait until the +folder has been quiet for `autodeploy.quiet_seconds` (default 300), then (1) mirror it into `mirror.repo` as a +commit on a separate branch, (2) build the site and deploy it only when the set of output files is the same as in +the last deploy and some file's content differs. Anything else (a new or removed page, no baseline, or a changed +generator, config or deploy target) is held back with a notification: run `fw blog deploy <config>` yourself. State: ~/.fw-blog-watch/<name>-<hash>/ (see blog.watch_name): +state.json, watch.log, deployed.json, hashes.json, lock, launchd.log. Rules and config keys are in the README ("Blog"). + +Everything with side effects (clock, notification, running commands, the deploy) is injected through `Env`, so +the tests use fakes. Stdlib only. +""" + +import argparse +import dataclasses +import datetime +import hashlib +import json +import os +import plistlib +import shutil +import subprocess +import sys +import tempfile +import time +import traceback + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import blog # noqa: E402 + +JUNK_NAMES = {".DS_Store", ".dropbox", ".dropbox.cache", ".git"} +RSYNC_JUNK = [".DS_Store", ".dropbox", ".dropbox.cache", "*conflicted copy*", ".git"] +DEFAULT_QUIET = 300 +TICK_SECONDS = 60 +GIT_TIMEOUT, RSYNC_TIMEOUT, NOTIFY_TIMEOUT = 120, 600, 10 +STUCK_AFTER = 1800 # a lock held this long (seconds) is reported as a stuck tick +LABEL = "net.fuzzyworld.fw-blog-watch-" +LAUNCH_PATH = "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" + + +def is_conflict(name): + return "conflicted copy" in name.lower() + + +def is_junk(name): + return name in JUNK_NAMES or is_conflict(name) + + +# ---------------------------------------------------------------- injected effects + +def notify_macos(message, title="fw blog watch"): + """A macOS notification; any failure is ignored. FW_BLOG_WATCH_NOTIFY=0 turns it off (the tests do).""" + if os.environ.get("FW_BLOG_WATCH_NOTIFY") == "0": + return + def quote(text): + return '"' + text.replace("\\", "\\\\").replace('"', '\\"') + '"' + try: + subprocess.run(["osascript", "-e", f"display notification {quote(message[:230])} with title {quote(title)}"], + capture_output=True, timeout=NOTIFY_TIMEOUT) + except Exception: + pass + + +def run_command(cmd, cwd=None, timeout=GIT_TIMEOUT): + """subprocess.run with captured text output. git never asks anything on the terminal. A command that runs + too long is killed and reported as a failure (exit 124), like any other.""" + try: + return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, encoding="utf-8", errors="surrogateescape", + stdin=subprocess.DEVNULL, timeout=timeout, + env={**os.environ, "GIT_TERMINAL_PROMPT": "0"}) + except subprocess.TimeoutExpired: + return subprocess.CompletedProcess(cmd, 124, "", f"{cmd[0]} timed out after {timeout} s") + + +def build_site(cfg, out): + return blog.build(cfg, out) + + +def deploy_built(cfg, out): + """The same function `fw blog deploy` uses, without the question; it audits `out` and runs rsync.""" + return blog.deploy_site(cfg, out, prebuilt=True, locked=True, timeout=RSYNC_TIMEOUT) # the tick holds the lock + + +@dataclasses.dataclass +class Env: + now: object = time.time # () -> seconds since the epoch + notify: object = notify_macos # (message) -> None + run: object = run_command # (argv, cwd=None, timeout=...) -> object with returncode, stdout, stderr + build: object = build_site # (cfg, out) -> None + deploy: object = deploy_built # (cfg, out) -> None, raises when the upload fails + generator: object = lambda: blog.GENERATOR_HASH # () -> hash of the code loaded now (fixed at import) + echo: object = print # where --dry-run output goes + + +def iso(seconds): + return datetime.datetime.fromtimestamp(seconds, datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +# ---------------------------------------------------------------- fingerprint + +def file_hash(path): + h = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def scan(src, cache=None): + """(fingerprint, conflict paths) of a source folder. With a `cache` dict ({path: [size, mtime_ns, inode, ctime_ns, + sha256]}, updated in place) a file whose first four values are unchanged is not read again. The fingerprint is a sha256 over the sorted (relative path, + content sha256) pairs of every file except junk (.DS_Store, .dropbox*, .git, Dropbox "conflicted copy" files). + Symbolic links are not followed: a link counts by its target text.""" + entries, conflicts, fresh = [], [], {} + for dirpath, dirs, files in os.walk(src): + keep = [] + for name in sorted(dirs): + path = os.path.join(dirpath, name) + rel = os.path.relpath(path, src).replace(os.sep, "/") + if is_conflict(name): + conflicts.append(rel) + elif name in JUNK_NAMES: + pass + elif os.path.islink(path): + entries.append((rel, "link:" + os.readlink(path))) + else: + keep.append(name) + dirs[:] = keep + for name in sorted(files): + path = os.path.join(dirpath, name) + rel = os.path.relpath(path, src).replace(os.sep, "/") + if is_conflict(name): + conflicts.append(rel) + elif is_junk(name): + continue + elif os.path.islink(path): + entries.append((rel, "link:" + os.readlink(path))) + else: + try: + st = os.stat(path) + stamp = [st.st_size, st.st_mtime_ns, st.st_ino, st.st_ctime_ns] + old = cache.get(rel) if isinstance(cache, dict) else None + if isinstance(old, list) and len(old) == 5 and old[:4] == stamp and isinstance(old[4], str): + digest = old[4] + else: + digest = file_hash(path) + fresh[rel] = stamp + [digest] + entries.append((rel, digest)) + except FileNotFoundError: # Dropbox moved it while we looked: the next tick sees the result + continue + except OSError as e: + entries.append((rel, f"unreadable:{e.errno}")) + if cache is not None: + cache.clear() + cache.update(fresh) + text = "\n".join(f"{rel}\0{digest}" for rel, digest in sorted(entries)) + return hashlib.sha256(text.encode("utf-8", "surrogateescape")).hexdigest(), sorted(conflicts) + + +# ---------------------------------------------------------------- state and log + +def read_json(path, default): + try: + with open(path, encoding="utf-8") as f: + data = json.load(f) + return data if isinstance(data, type(default)) else default + except (OSError, ValueError): + return default + + +def save_json(sd, name, data): + tmp = os.path.join(sd, name + ".tmp") + with open(tmp, "w", encoding="utf-8") as f: + json.dump(data, f, indent=1) + os.replace(tmp, os.path.join(sd, name)) + + +def save_state(sd, state): + save_json(sd, "state.json", state) + + +def current(sd, cfg): + """(fingerprint, conflicts) of the source now, using (and refreshing) the hash cache in the state folder.""" + cache = read_json(os.path.join(sd, "hashes.json"), {}) # a damaged cache file is just an empty one + result = scan(cfg["_source"], cache) + save_json(sd, "hashes.json", cache) + return result + + +def changed_since(sd, cfg, state): + return current(sd, cfg) != (state.get("fingerprint"), state.get("conflicts")) + + +def log(sd, env, message): + with open(os.path.join(sd, "watch.log"), "a", encoding="utf-8") as f: + f.write(f"{iso(env.now())} {message}\n") + + +def once(sd, env, state, key, message, notify=True): + """Log (and notify) a message the first time `key` happens after a change of the source, not on every tick.""" + seen = state.setdefault("notified", []) + if key in seen: + return + seen.append(key) + log(sd, env, message) + if notify: + env.notify(message) + + +# ---------------------------------------------------------------- mirror + +def git(env, repo, *args): + return env.run(["git", "--no-optional-locks", "-C", repo, *args], timeout=GIT_TIMEOUT) + + +def problem(result, what): + lines = (result.stderr or result.stdout or "").strip().splitlines() + return f"{what}: {lines[-1] if lines else 'failed'}" + + +def mirror_settings(cfg): + """(repo folder, branch, remote, extra excludes). A relative `repo` is relative to the config's folder.""" + m = cfg["mirror"] + repo = os.path.abspath(os.path.join(cfg["_dir"], os.path.expanduser(m["repo"]))) + return repo, m.get("branch", "dropbox-mirror"), m.get("remote"), list(m.get("exclude", [])) + + +def rsync_command(src, repo, extra, dry_run=False): + """The user's excludes come before the protective /.git (rsync uses the first pattern that matches).""" + cmd = ["rsync", "-a", "--delete"] + (["-n", "-v"] if dry_run else []) + for pattern in RSYNC_JUNK + extra + ["/.git"]: + cmd.append("--exclude=" + pattern) + return cmd + [src.rstrip("/") + "/", repo.rstrip("/") + "/"] + + +def head_sha(env, repo): + result = git(env, repo, "rev-parse", "-q", "--verify", "HEAD") + return result.stdout.strip() if result.returncode == 0 and result.stdout.strip() else "unborn" + + +def current_branch(env, repo): + result = git(env, repo, "symbolic-ref", "--short", "-q", "HEAD") + return result.stdout.strip() if result.returncode == 0 else "" + + +def status_lines(env, repo): + """(exit code, sorted lines) of `git status --porcelain=v1 -z --ignored`: "XY path" with the real (unquoted) path; + a rename or copy adds a second line "-> origin path" so both names are recorded.""" + result = git(env, repo, "status", "--porcelain=v1", "-z", "--ignored") + fields = result.stdout.split("\0") + lines, i = [], 0 + while i < len(fields): + entry = fields[i] + i += 1 + if len(entry) < 4: + continue + lines.append(entry) + if entry[0] in "RC" or entry[1] in "RC": # the next field is the origin path + if i < len(fields) and fields[i]: + lines.append("-> " + fields[i]) + i += 1 + return result.returncode, sorted(lines) + + +def content_digest(repo, lines): + """sha256 over the contents of the paths a status listing names (folders: every file in them), so an edit to a file + that was already listed is noticed too.""" + h = hashlib.sha256() + for line in lines: + rel = line[3:] + top = os.path.join(repo, rel) + paths = ([os.path.join(d, n) for d, _, names in os.walk(top) for n in sorted(names)] + if os.path.isdir(top) else [top]) + for path in sorted(paths): + try: + h.update(path.encode() + b"\0" + file_hash(path).encode()) + except OSError: + h.update(path.encode() + b"\0missing") + return h.hexdigest() + + +def record_leftover(env, repo, state): + """Remember exactly what our own run left in the repo (HEAD, the status listing, the files' content): a later tick + accepts a dirty tree only if it is still exactly that. Nothing left: no record.""" + code, lines = status_lines(env, repo) + if code == 0 and lines: + state["mirror_dirty"] = {"repo": os.path.realpath(repo), "head": head_sha(env, repo), "status": lines, + "content": content_digest(repo, lines)} + else: + state.pop("mirror_dirty", None) + + +def mirror_blocker(env, cfg, repo, branch, dirty_ok=None): + """Why the mirror must not run now (a sentence), or None when the repo is ready (or about to be created). + Git-ignored files count as changes (rsync --delete could remove them). A dirty tree is accepted only when + `dirty_ok` (what our own run recorded with record_leftover) is exactly what is there now: same repo, same HEAD, + same status listing, same file contents. Anything else is somebody else's work.""" + src = os.path.realpath(cfg["_source"]) + real = os.path.realpath(repo) + if real == src or real.startswith(src + os.sep) or src.startswith(real + os.sep): + return f"mirror.repo {repo} and the source folder contain each other" + if not os.path.exists(repo) or (os.path.isdir(repo) and not os.listdir(repo)): + return None # will be created with `git init` + if not os.path.isdir(repo): + return f"mirror.repo {repo} is not a folder" + top = git(env, repo, "rev-parse", "--show-toplevel") + if top.returncode or os.path.realpath(top.stdout.strip()) != real: + return f"mirror.repo {repo} is not a git repository (its own top folder)" + here = current_branch(env, repo) + if here != branch: + return f"mirror.repo {repo} has {here or 'no branch'} checked out, not {branch}" + gitdir = git(env, repo, "rev-parse", "--absolute-git-dir").stdout.strip() + for name in ("MERGE_HEAD", "rebase-merge", "rebase-apply", "CHERRY_PICK_HEAD", "REVERT_HEAD", "BISECT_LOG"): + if gitdir and os.path.exists(os.path.join(gitdir, name)): + return f"mirror.repo {repo} is in the middle of a merge, rebase or similar ({name})" + code, lines = status_lines(env, repo) + if code == 0 and not lines: + return None + if (code == 0 and isinstance(dirty_ok, dict) and dirty_ok.get("repo") == real and dirty_ok.get("head") == head_sha(env, repo) + and dirty_ok.get("status") == lines and dirty_ok.get("content") == content_digest(repo, lines)): + return None + return (f"mirror.repo {repo} has uncommitted or git-ignored files (or git status failed); " + "the mirror will not overwrite them") + + +def mirror_step(sd, env, state, cfg): + """Mirror the source into the repo. 'ok' or 'blocked' (try again next tick; nothing in the repo is overwritten + that the mirror did not write itself).""" + repo, branch, remote, extra = mirror_settings(cfg) + blocked = mirror_blocker(env, cfg, repo, branch, state.get("mirror_dirty")) + if blocked: + once(sd, env, state, "mirror:" + blocked, "mirror skipped: " + blocked) + return "blocked" + if not os.path.isdir(repo) or not os.listdir(repo): + os.makedirs(repo, exist_ok=True) + init = env.run(["git", "init", "-b", branch, repo], timeout=GIT_TIMEOUT) + if init.returncode: + once(sd, env, state, "mirror:init", "mirror skipped: " + problem(init, "git init")) + return "blocked" + log(sd, env, f"mirror: created git repo {repo} on branch {branch}") + for key in ("user.name", "user.email"): + if not git(env, repo, "config", key).stdout.strip(): + once(sd, env, state, "mirror:identity", + f"mirror skipped: git has no {key}; run: git config --global {key} ...") + return "blocked" + + def failed(key, message): + record_leftover(env, repo, state) # the dirt in the repo is ours: allow a retry over it + once(sd, env, state, key, message) + return "blocked" + + sync = env.run(rsync_command(cfg["_source"], repo, extra), timeout=RSYNC_TIMEOUT) + if sync.returncode == 24: # files vanished while Dropbox was syncing: normal + record_leftover(env, repo, state) + log(sd, env, "mirror: files vanished while copying (rsync 24); trying again next tick") + return "blocked" + if sync.returncode: + return failed("mirror:rsync", "mirror failed: " + problem(sync, "rsync") + + f" ({repo} may be partly updated; the next tick tries again)") + added = git(env, repo, "add", "-A") + if added.returncode: + return failed("mirror:add", "mirror failed: " + problem(added, "git add")) + status = git(env, repo, "status", "--porcelain") + if status.returncode: + return failed("mirror:status", "mirror failed: " + problem(status, "git status")) + if status.stdout.strip(): + if current_branch(env, repo) != branch: # someone switched branches since the check + return failed("mirror:branch", f"mirror skipped: {repo} no longer has {branch} checked out") + stamp = iso(env.now()) + message = f"Mirror of {os.path.basename(os.path.normpath(cfg['_source']))} at {stamp}" + commit = git(env, repo, "commit", "--no-verify", "-m", message) + if commit.returncode: + return failed("mirror:commit", "mirror failed: " + problem(commit, "git commit")) + log(sd, env, f"mirror: committed ({len(status.stdout.splitlines())} paths changed)") + else: + log(sd, env, "mirror: nothing new to commit") + record_leftover(env, repo, state) + if state.get("mirror_dirty"): + # git-ignored files copied from the source (it has a .gitignore?): ours, recorded exactly + once(sd, env, state, "mirror:ignored", "mirror: the repo holds git-ignored files copied from the source") + if remote: + ref = f"refs/heads/{branch}:refs/heads/{branch}" + push = git(env, repo, "push", remote, ref) + if push.returncode: + once(sd, env, state, "mirror:push", f"mirror: push to {remote} failed: " + problem(push, "git push")) + else: + log(sd, env, f"mirror: pushed {branch} to {remote}") + return "ok" + + +# ---------------------------------------------------------------- deploy guard + +def compare(built, baseline, generator=None, config=None, target=None): + """('no-baseline'|'generator'|'target'|'config', {}) | ('paths', {added, removed}) | ('same', {}) | + ('changed', {changed}). A baseline made by another version of the generator (its code, templates, CSS), with another + deploy target or another config holds back, so an automatic deploy never carries a code, template or config change.""" + if baseline is None: + return "no-baseline", {} + if generator is not None and baseline.get("generator") != generator: + return "generator", {} + if target is not None and baseline.get("target") != target: + return "target", {} + if config is not None and baseline.get("config") != config: + return "config", {} + old = baseline["files"] + added, removed = sorted(set(built) - set(old)), sorted(set(old) - set(built)) + if added or removed: + return "paths", {"added": added, "removed": removed} + changed = sorted(p for p in built if built[p] != old[p]) + return ("changed", {"changed": changed}) if changed else ("same", {}) + + +def load_baseline(sd): + data = read_json(os.path.join(sd, "deployed.json"), {}) + return data if isinstance(data.get("files"), dict) else None + + +def build_to_temp(env, cfg): + """(temp folder, built folder, manifest) or raises. The caller removes the temp folder.""" + tmp = tempfile.mkdtemp(prefix="fw-blog-watch-") + out = os.path.join(tmp, "site") + try: + env.build(cfg, out) + return tmp, out, blog.manifest(out) + except BaseException: + shutil.rmtree(tmp, ignore_errors=True) + raise + + +def few(paths, n=5): + return ", ".join(paths[:n]) + (f" (+{len(paths) - n} more)" if len(paths) > n else "") + + +def guard_step(sd, env, state, cfg, config_path): + """Build, compare with deployed.json, deploy when it is safe. Ends the change (the caller marks it handled), + except 'changed-during-build': the source moved, so the new quiet period applies.""" + try: + tmp, out, built = build_to_temp(env, cfg) + except Exception as e: + once(sd, env, state, "build", f"build failed, nothing deployed: {e}") + return "build-error" + try: + if changed_since(sd, cfg, state): + log(sd, env, "changed during build; nothing deployed") + return "changed-during-build" + generator = env.generator() + kind, info = compare(built, load_baseline(sd), generator, blog.config_hash(cfg), blog.deploy_target(cfg)) + if kind == "no-baseline": + once(sd, env, state, "hold", "held back: no baseline: run fw blog deploy once") + return "held" + if kind in ("generator", "target", "config"): + what = {"generator": "generator changed", "target": "deploy target changed", "config": "config changed"}[kind] + once(sd, env, state, "hold", f"held back: {what}: run `fw blog deploy {config_path}`") + return "held" + if kind == "paths": + a, r = info["added"], info["removed"] + once(sd, env, state, "hold", f"held back: {len(a)} added, {len(r)} removed: run `fw blog deploy {config_path}`") + log(sd, env, "held paths: " + few(["+" + p for p in a] + ["-" + p for p in r])) + return "held" + if kind == "same": + log(sd, env, "no visible change") + return "unchanged" + try: + env.deploy(cfg, out) + except Exception as e: + once(sd, env, state, "deploy", f"deploy failed: {e}") + return "deploy-error" + blog.write_deployed(sd, built, iso(env.now()), generator, cfg) + log(sd, env, f"deployed: {len(info['changed'])} files changed") + return "deployed" + finally: + shutil.rmtree(tmp, ignore_errors=True) + + +# ---------------------------------------------------------------- one tick + +def tick(config_path, env=None): + """One pass. Returns a word saying what happened (for the tests and --dry-run): locked, changed, idle, waiting, + conflict, blocked, deployed, held, unchanged, build-error, deploy-error, changed-during-build, mirrored, + config-error, noop, error (an unexpected exception: logged, notified once, never raised).""" + env = env or Env() + config_path = os.path.abspath(config_path) + sd = blog.state_dir(config_path) + os.makedirs(sd, exist_ok=True) + lock = blog.acquire_lock(sd, env.now) + if lock is None: + since = blog.lock_since(sd) + marker = os.path.join(sd, "stuck-notified") + if since and env.now() - since > STUCK_AFTER and read_text(marker) != repr(since): + with open(marker, "w") as f: + f.write(repr(since)) + text = "watch tick stuck: the lock has been held for over 30 minutes" + log(sd, env, text) + env.notify(text) + return "locked" + try: + result = locked_tick(sd, env, config_path) + state = read_json(os.path.join(sd, "state.json"), {}) + if state.pop("last_error", None): # a clean tick: the next unexpected error notifies again + save_state(sd, state) + return result + except Exception as e: + frame = traceback.extract_tb(e.__traceback__)[-1] + text = f"unexpected error: {type(e).__name__}: {e} ({os.path.basename(frame.filename)}:{frame.lineno})" + state = read_json(os.path.join(sd, "state.json"), {}) + if state.get("last_error") != text: + state["last_error"] = text + save_state(sd, state) + log(sd, env, text) + env.notify(text) + return "error" + finally: + lock.close() + + +def read_text(path): + try: + with open(path) as f: + return f.read() + except OSError: + return None + + +def locked_tick(sd, env, config_path): + state = read_json(os.path.join(sd, "state.json"), {}) + earlier = state.pop("config_error", None) + + def config_problem(text): + if earlier != text: + log(sd, env, text) + env.notify(text) + state["config_error"] = text + save_state(sd, state) + return "config-error" + + try: + cfg = blog.load_config(config_path) + except blog.BlogError as e: + return config_problem(f"config error: {e}") + if cfg["mirror"] is None and cfg["autodeploy"] is None: + if earlier: + save_state(sd, state) + return "noop" + if not os.path.isdir(cfg["_source"]): + return config_problem(f"source folder {cfg['_source']} not found") + if earlier: + save_state(sd, state) # recovered: the next problem notifies again + now = env.now() + fingerprint, conflicts = current(sd, cfg) + if state.get("fingerprint") != fingerprint or state.get("conflicts") != conflicts: + save_state(sd, {"fingerprint": fingerprint, "conflicts": conflicts, "changed_at": now, "handled": False, + "notified": [], **{k: state[k] for k in ("mirror_dirty",) if k in state}}) + log(sd, env, "changed") + return "changed" + if state.get("handled"): + return "idle" + quiet = (cfg["autodeploy"] or {}).get("quiet_seconds", DEFAULT_QUIET) + if now - state.get("changed_at", now) < quiet: + if "waiting" not in state.setdefault("notified", []): + state["notified"].append("waiting") + log(sd, env, f"waiting: {quiet} s without changes needed") + save_state(sd, state) + return "waiting" + try: + return decide(sd, env, state, cfg, config_path, conflicts) + finally: + save_state(sd, state) + + +def decide(sd, env, state, cfg, config_path, conflicts): + if conflicts: + once(sd, env, state, "conflict", "Dropbox conflict file(s) in the source, nothing mirrored or deployed: " + + few(conflicts, 3)) + return "conflict" + result = "mirrored" + if cfg["mirror"] is not None: + if mirror_step(sd, env, state, cfg) == "blocked": + return "blocked" + if changed_since(sd, cfg, state): + log(sd, env, "changed during mirror; nothing built") + return "changed-during-build" + if cfg["autodeploy"] is not None: + result = guard_step(sd, env, state, cfg, config_path) + if result == "changed-during-build": + return result + state["handled"] = True + return result + + +# ---------------------------------------------------------------- --dry-run + +def dry_run(config_path, env): + """Shows what a tick would do right now (ignoring the quiet period); writes nothing.""" + cfg = blog.load_config(config_path) + say = env.echo + if cfg["mirror"] is None and cfg["autodeploy"] is None: + say("the config has neither mirror nor autodeploy: nothing to do") + return + if not os.path.isdir(cfg["_source"]): + say(f"source folder {cfg['_source']} not found") + return + _, conflicts = scan(cfg["_source"]) + if conflicts: + say("would stop: Dropbox conflict file(s): " + few(conflicts)) + return + if cfg["mirror"] is not None: + repo, branch, remote, extra = mirror_settings(cfg) + dirty_ok = read_json(os.path.join(blog.state_dir(cfg["_path"]), "state.json"), {}).get("mirror_dirty") + blocked = mirror_blocker(env, cfg, repo, branch, dirty_ok) + if blocked: + say("mirror would be skipped: " + blocked) + return + if not os.path.isdir(repo) or not os.listdir(repo): + say(f"mirror: would run git init -b {branch} {repo}") + else: + say(f"mirror: rsync would change (source -> {repo}):") + result = env.run(rsync_command(cfg["_source"], repo, extra, dry_run=True), timeout=RSYNC_TIMEOUT) + say(result.stdout.rstrip() or " (nothing)") + say("mirror: would git add -A, commit if anything changed" + (f", and push to {remote}" if remote else "")) + if cfg["autodeploy"] is not None: + try: + tmp, _, built = build_to_temp(env, cfg) + except Exception as e: + say(f"deploy: the build fails, nothing would be deployed: {e}") + return + shutil.rmtree(tmp, ignore_errors=True) + kind, info = compare(built, load_baseline(blog.state_dir(cfg["_path"])), env.generator(), + blog.config_hash(cfg), blog.deploy_target(cfg)) + say({"no-baseline": "deploy: would hold back (no baseline: run fw blog deploy once)", + "generator": "deploy: would hold back (generator changed: run fw blog deploy)", + "target": "deploy: would hold back (deploy target changed: run fw blog deploy)", + "config": "deploy: would hold back (config changed: run fw blog deploy)", + "paths": f"deploy: would hold back ({len(info.get('added', []))} added, {len(info.get('removed', []))} " + "removed paths)", + "same": "deploy: no visible change, nothing to deploy", + "changed": f"deploy: would upload ({len(info.get('changed', []))} files changed): " + + few(info.get("changed", []))}[kind]) + + +# ---------------------------------------------------------------- launchd agent + +def plist_data(config_path, name, sd, fw=None): + fw = fw or os.path.realpath(os.path.join(HERE, "..", "bin", "fw")) + return {"Label": LABEL + name, + "ProgramArguments": [fw, "blog", "watch", os.path.abspath(config_path), "--once"], + "StartInterval": TICK_SECONDS, "RunAtLoad": True, + "EnvironmentVariables": {"PATH": LAUNCH_PATH}, + "StandardOutPath": os.path.join(sd, "launchd.log"), "StandardErrorPath": os.path.join(sd, "launchd.log"), + "ProcessType": "Background", "Nice": 10} + + +def plist_text(config_path, fw=None): + config_path = os.path.abspath(config_path) + return plistlib.dumps(plist_data(config_path, blog.watch_name(config_path), blog.state_dir(config_path), fw)).decode("utf-8") + + +def agent_path(config_path, agents_dir=None): + return os.path.join(agents_dir or os.path.expanduser("~/Library/LaunchAgents"), + LABEL + blog.watch_name(config_path) + ".plist") + + +def install(config_path, run=run_command, agents_dir=None, uid=None, fw=None, sleep=time.sleep): + """Writes the plist and loads it. Reinstalling replaces a loaded agent.""" + blog.load_config(config_path) # a typo in the config is better found now than every minute + path = agent_path(config_path, agents_dir) + os.makedirs(os.path.dirname(path), exist_ok=True) + os.makedirs(blog.state_dir(config_path), exist_ok=True) + domain = f"gui/{os.getuid() if uid is None else uid}" + with open(path, "w", encoding="utf-8") as f: + f.write(plist_text(config_path, fw)) + run(["launchctl", "bootout", domain, path]) # fails when it was not loaded: fine + for attempt in range(5): # bootstrap can fail for a moment right after a bootout + result = run(["launchctl", "bootstrap", domain, path]) + if not result.returncode: + break + if attempt < 4: + sleep(1) + if result.returncode: + raise blog.BlogError(f"launchctl bootstrap failed: {(result.stderr or '').strip()}") + return path + + +def uninstall(config_path, run=run_command, agents_dir=None, uid=None): + path = agent_path(config_path, agents_dir) + run(["launchctl", "bootout", f"gui/{os.getuid() if uid is None else uid}", path]) + if os.path.exists(path): + os.remove(path) + return path + + +# ---------------------------------------------------------------- command line + +def main(argv): + ap = argparse.ArgumentParser(prog="blog_watch.py", description=__doc__.split("\n")[0]) + ap.add_argument("words", nargs="+", metavar="[install|uninstall] config") + ap.add_argument("--once", action="store_true", help="one tick, then exit (what launchd runs)") + ap.add_argument("--dry-run", action="store_true", help="show what a tick would do now; write nothing") + ap.add_argument("--print", action="store_true", dest="show", help="with install: only print the launchd plist") + args = ap.parse_args(argv[1:]) + action = args.words[0] if args.words[0] in ("install", "uninstall") else None + rest = args.words[1:] if action else args.words + if len(rest) != 1: + ap.error("needs exactly one config file") + config = os.path.abspath(os.path.expanduser(rest[0])) + try: + if action == "install": + if args.show: + print(plist_text(config), end="") + else: + print("installed " + install(config)) + elif action == "uninstall": + print("removed " + uninstall(config)) + elif args.dry_run: + dry_run(config, Env()) + elif args.once: + result = tick(config) + if result in ("config-error", "build-error", "deploy-error", "blocked", "error"): # the exit code stays 0: launchd must not throttle the agent + print(f"blog_watch: {result} (see {blog.state_dir(config)}/watch.log)", file=sys.stderr) + else: + while True: + tick(config) + time.sleep(TICK_SECONDS) + except blog.BlogError as e: + raise SystemExit(f"blog_watch.py: {e}") + except KeyboardInterrupt: + pass + + +if __name__ == "__main__": + main(sys.argv) diff --git a/tests/test_blog_watch.py b/tests/test_blog_watch.py new file mode 100644 index 0000000..839fa44 --- /dev/null +++ b/tests/test_blog_watch.py @@ -0,0 +1,1057 @@ +"""Tests for server/blog_watch.py (`fw blog watch`). Run: python3 -m unittest discover tests + +Every blog, repo and state folder here is made up, in a temporary folder. Git and rsync are real (local bare +repo as the "remote"); the clock, notifications and deploy are fakes. Nothing touches the network or ~/Library. +""" +import io +import json +import os +import plistlib +import shutil +import subprocess +import sys +import tempfile +import unittest +from unittest import mock + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "server")) +import blog # noqa: E402 +import blog_watch as bw # noqa: E402 + +# No real macOS notifications from a test run (some tests go through the default notifier). +os.environ["FW_BLOG_WATCH_NOTIFY"] = "0" + +needs_markdown = unittest.skipUnless(blog.markdown, "Python-Markdown is not installed (see FW_PYLIB)") + + +class Clock: + def __init__(self, t=1_800_000_000.0): + self.t = t + + def __call__(self): + return self.t + + def advance(self, seconds): + self.t += seconds + + +def git(repo, *args): + return subprocess.run(["git", "-C", repo, *args], capture_output=True, text=True, check=True).stdout + + +class WatchCase(unittest.TestCase): + """A made-up blog folder `MyBlog/` (config + src/), a state dir, a fake clock and notifier.""" + + def setUp(self): + self.dir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.dir, ignore_errors=True) + self.root = os.path.join(self.dir, "MyBlog") + self.src = os.path.join(self.root, "src") + self.sd = os.path.join(self.dir, "state") + os.makedirs(self.src) + gitconfig = os.path.join(self.dir, "gitconfig") + with open(gitconfig, "w") as f: + f.write("[user]\n\tname = Test Writer\n\temail = test@example.org\n[init]\n\tdefaultBranch = main\n") + self.gitconfig = gitconfig + saved = {k: os.environ.get(k) for k in ("FW_BLOG_WATCH_DIR", "GIT_CONFIG_GLOBAL", "GIT_CONFIG_NOSYSTEM")} + os.environ.update(FW_BLOG_WATCH_DIR=self.sd, GIT_CONFIG_GLOBAL=gitconfig, GIT_CONFIG_NOSYSTEM="1") + self.addCleanup(lambda: [os.environ.pop(k, None) if v is None else os.environ.__setitem__(k, v) + for k, v in saved.items()]) + self.clock = Clock() + self.notes = [] + self.deploys = [] + self.built = {"index.html": "<p>home</p>", "a/index.html": "<p>a</p>"} + self.config = {"title": "T", "base_url": "https://blog.example.org", "source": "src", + "deploy": {"host": "dh", "dir": "writing.example.org"}} + self.write("one.md", "# One\n\nbody\n") + + def write(self, rel, text, base=None): + path = os.path.join(base or self.src, rel) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as f: + f.write(text) + + def cfg_path(self, **extra): + self.config.update(extra) + path = os.path.join(self.root, "fw-blog.json") + with open(path, "w") as f: + json.dump(self.config, f) + self.cfg_file = path + return path + + def fake_build(self, cfg, out): + for rel, text in self.built.items(): + self.write(rel, text, base=out) + + def fake_deploy(self, cfg, out): + self.deploys.append(sorted(os.listdir(out))) + + def env(self, **kw): + kw.setdefault("build", self.fake_build) + kw.setdefault("deploy", self.fake_deploy) + return bw.Env(now=self.clock, notify=self.notes.append, **kw) + + def tick(self, path, **kw): + return bw.tick(path, self.env(**kw)) + + def settle(self, path, seconds=301, **kw): + """The first tick sees a change; after the quiet period the next one acts.""" + self.assertEqual(self.tick(path, **kw), "changed") + self.clock.advance(seconds) + return self.tick(path, **kw) + + def log(self): + with open(os.path.join(self.sd, "watch.log")) as f: + return f.read() + + def state(self): + with open(os.path.join(self.sd, "state.json")) as f: + return json.load(f) + + def baseline(self, files=None, **kw): + """deployed.json as a manual deploy of the current config and code would have left it.""" + blog.write_deployed(self.sd, files if files is not None else + {rel: __import__("hashlib").sha256(t.encode()).hexdigest() for rel, t in self.built.items()}, + cfg=kw.pop("cfg", None) or blog.load_config(self.cfg_file), **kw) + + +class Fingerprint(WatchCase): + def test_ignores_junk(self): + before, conflicts = bw.scan(self.src) + self.write(".DS_Store", "x") + self.write(".dropbox", "x") + self.write(".dropbox.cache/a", "x") + self.write(".git/config", "x") + self.write("sub/.git/HEAD", "x") + self.write("one (conflicted copy 2026-11-02).md", "x") + after, conflicts = bw.scan(self.src) + self.assertEqual(before, after) + self.assertEqual(conflicts, ["one (conflicted copy 2026-11-02).md"]) + + def test_content_and_name_change_it(self): + a, _ = bw.scan(self.src) + self.write("one.md", "# One\n\nbody!\n") + b, _ = bw.scan(self.src) + os.rename(os.path.join(self.src, "one.md"), os.path.join(self.src, "two.md")) + c, _ = bw.scan(self.src) + self.assertEqual(len({a, b, c}), 3) + + def test_symlinks_are_not_followed(self): + outside = os.path.join(self.dir, "outside") + os.makedirs(outside) + self.write("x.txt", "1", base=outside) + os.symlink(outside, os.path.join(self.src, "link")) + a, _ = bw.scan(self.src) + self.write("x.txt", "2", base=outside) + b, _ = bw.scan(self.src) + self.assertEqual(a, b) + + +class Quiet(WatchCase): + def test_waits_for_quiet_period_then_acts(self): + path = self.cfg_path(autodeploy={}) + self.assertEqual(self.tick(path), "changed") + self.clock.advance(100) + self.assertEqual(self.tick(path), "waiting") + self.assertEqual(self.tick(path), "waiting") + self.assertEqual(self.log().count("waiting"), 1) # logged once per change + self.clock.advance(201) + self.assertEqual(self.tick(path), "held") # no baseline + self.assertTrue(self.state()["handled"]) + self.assertEqual(self.tick(path), "idle") + + def test_change_during_wait_resets_the_timer(self): + path = self.cfg_path(autodeploy={"quiet_seconds": 120}) + self.tick(path) + self.clock.advance(100) + self.write("one.md", "# One\n\nmore\n") + self.assertEqual(self.tick(path), "changed") + self.clock.advance(100) + self.assertEqual(self.tick(path), "waiting") + self.clock.advance(21) + self.assertEqual(self.tick(path), "held") + + def test_conflict_file_blocks_and_notifies_once(self): + path = self.cfg_path(autodeploy={}) + self.baseline() + self.write("one (conflicted copy).md", "x") + self.assertEqual(self.settle(path), "conflict") + self.assertFalse(self.state()["handled"]) + self.clock.advance(60) + self.assertEqual(self.tick(path), "conflict") + self.assertEqual(len(self.notes), 1) + self.assertEqual(self.deploys, []) + os.remove(os.path.join(self.src, "one (conflicted copy).md")) + self.assertEqual(self.tick(path), "changed") # resolved: the timer starts over + + def test_nothing_configured(self): + self.assertEqual(self.tick(self.cfg_path()), "noop") + + def test_config_error_notifies_once(self): + path = self.cfg_path(autodeploy={"quiet_seconds": 5}) + self.assertEqual(self.tick(path), "config-error") + self.assertEqual(self.tick(path), "config-error") + self.assertEqual(len(self.notes), 1) + + +class Lock(WatchCase): + def test_second_instance_exits_quietly(self): + path = self.cfg_path(autodeploy={}) + os.makedirs(self.sd) + with open(os.path.join(self.sd, "lock"), "a") as held: + import fcntl + fcntl.flock(held, fcntl.LOCK_EX | fcntl.LOCK_NB) + self.assertEqual(self.tick(path), "locked") + self.assertFalse(os.path.exists(os.path.join(self.sd, "state.json"))) + self.assertEqual(self.tick(path), "changed") # released: works again + + +class Mirror(WatchCase): + def setUp(self): + super().setUp() + self.repo = os.path.join(self.dir, "repo") + self.path = self.cfg_path(mirror={"repo": self.repo}) + + def commits(self): + return git(self.repo, "log", "--format=%s").splitlines() + + def test_first_run_inits_and_commits(self): + self.write("img/a.png", "png") + self.write(".DS_Store", "x") + self.assertEqual(self.settle(self.path), "mirrored") + self.assertEqual(git(self.repo, "symbolic-ref", "--short", "HEAD").strip(), "dropbox-mirror") + self.assertEqual(sorted(git(self.repo, "ls-files").split()), ["img/a.png", "one.md"]) + self.assertRegex(self.commits()[0], r"^Mirror of src at \d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$") + self.assertEqual(git(self.repo, "log", "-1", "--format=%an <%ae>").strip(), "Test Writer <test@example.org>") + self.assertEqual(git(self.repo, "log", "-1", "--format=%b").strip(), "") + self.assertEqual(self.notes, []) + + def test_empty_existing_folder_is_initialised(self): + os.makedirs(self.repo) + self.assertEqual(self.settle(self.path), "mirrored") + self.assertEqual(len(self.commits()), 1) + + def test_no_change_makes_no_commit_and_edit_and_delete_do(self): + self.settle(self.path) + self.write("one.md", "# One\n\nsame fingerprint? no, changed\n") + self.assertEqual(self.settle(self.path), "mirrored") + self.assertEqual(len(self.commits()), 2) + os.remove(os.path.join(self.src, "one.md")) + self.write("two.md", "x") + self.assertEqual(self.settle(self.path), "mirrored") + self.assertEqual(git(self.repo, "ls-files").split(), ["two.md"]) # deleted file is deleted in the repo + self.assertEqual(len(self.commits()), 3) + # a junk-only change alters nothing: not even a tick's worth of work + self.write(".DS_Store", "x") + self.assertEqual(self.tick(self.path), "idle") + + def test_noop_when_content_is_the_same(self): + self.settle(self.path) + # force another pass without a source change: no commit + state = self.state() + state["handled"] = False + with open(os.path.join(self.sd, "state.json"), "w") as f: + json.dump(state, f) + self.clock.advance(400) + self.assertEqual(self.tick(self.path), "mirrored") + self.assertEqual(len(self.commits()), 1) + + def test_dirty_tree_is_refused(self): + self.settle(self.path) + self.write("scratch.txt", "mine", base=self.repo) + self.write("one.md", "# One\n\nedited\n") + self.assertEqual(self.settle(self.path), "blocked") + self.assertTrue(os.path.exists(os.path.join(self.repo, "scratch.txt"))) # never overwritten or deleted + self.assertEqual(len(self.commits()), 1) + self.assertFalse(self.state()["handled"]) + self.assertEqual(len(self.notes), 1) + self.assertIn("uncommitted", self.notes[0]) + self.clock.advance(60) + self.tick(self.path) + self.assertEqual(len(self.notes), 1) # once + + def test_wrong_branch_is_refused(self): + self.settle(self.path) + git(self.repo, "checkout", "-q", "-b", "other") + self.write("one.md", "# One\n\nedited\n") + self.assertEqual(self.settle(self.path), "blocked") + self.assertIn("other", self.notes[0]) + self.assertEqual(git(self.repo, "log", "--format=%s", "other").count("Mirror"), 1) + + def test_non_git_folder_is_refused(self): + self.write("something", "x", base=self.repo) + self.assertEqual(self.settle(self.path), "blocked") + self.assertIn("not a git repository", self.notes[0]) + self.assertFalse(os.path.exists(os.path.join(self.repo, ".git"))) + + def test_in_progress_merge_is_refused(self): + self.settle(self.path) + gitdir = git(self.repo, "rev-parse", "--absolute-git-dir").strip() + with open(os.path.join(gitdir, "MERGE_HEAD"), "w") as f: + f.write(git(self.repo, "rev-parse", "HEAD")) + self.write("one.md", "# One\n\nedited\n") + self.assertEqual(self.settle(self.path), "blocked") + self.assertIn("MERGE_HEAD", self.notes[0]) + + def test_missing_git_identity_fails_clearly(self): + open(self.gitconfig, "w").close() + self.assertEqual(self.settle(self.path), "blocked") + self.assertIn("user.name", self.notes[0]) + + def test_extra_excludes(self): + self.path = self.cfg_path(mirror={"repo": self.repo, "exclude": ["secret/"]}) + self.write("secret/x.md", "x") + self.settle(self.path) + self.assertEqual(git(self.repo, "ls-files").split(), ["one.md"]) + + def test_push_goes_to_the_remote_and_failure_does_not_stop_deploy(self): + remote = os.path.join(self.dir, "remote.git") + subprocess.run(["git", "init", "-q", "--bare", remote], check=True) + self.path = self.cfg_path(mirror={"repo": self.repo, "remote": "origin"}, autodeploy={}) + self.baseline() + self.write("seed", "x", base=self.repo) + subprocess.run(["git", "init", "-q", "-b", "dropbox-mirror", self.repo], check=True) + git(self.repo, "add", "-A") + git(self.repo, "commit", "-q", "-m", "seed") + git(self.repo, "remote", "add", "origin", remote) + self.built["index.html"] = "<p>changed</p>" + self.assertEqual(self.settle(self.path), "deployed") + self.assertEqual(git(remote, "log", "--format=%s", "dropbox-mirror").splitlines()[0][:9], "Mirror of") + self.assertEqual(git(remote, "branch", "--format=%(refname:short)").split(), ["dropbox-mirror"]) + # now the remote disappears: the push fails, the notification says so, the deploy still happens + shutil.rmtree(remote) + self.notes.clear() + self.write("one.md", "# One\n\nagain\n") + self.built["index.html"] = "<p>changed again</p>" + self.assertEqual(self.settle(self.path), "deployed") + self.assertEqual(len(self.deploys), 2) + self.assertEqual(len(self.notes), 1) + self.assertIn("push", self.notes[0]) + + def test_dry_run_writes_nothing(self): + lines = [] + env = self.env(echo=lines.append) + bw.dry_run(self.path, env) + self.assertIn("git init", "\n".join(lines)) + self.assertFalse(os.path.exists(self.repo)) + self.assertFalse(os.path.exists(os.path.join(self.sd, "state.json"))) + self.settle(self.path) + self.write("two.md", "x") + lines.clear() + bw.dry_run(self.path, env) + self.assertIn("two.md", "\n".join(lines)) + self.assertEqual(len(self.commits()), 1) + self.assertFalse(os.path.exists(os.path.join(self.repo, "two.md"))) + + def test_repo_inside_source_is_refused(self): + path = self.cfg_path(mirror={"repo": os.path.join(self.src, "repo")}) + self.assertEqual(self.settle(path), "blocked") + + +class Guard(WatchCase): + def setUp(self): + super().setUp() + self.path = self.cfg_path(autodeploy={}) + + def test_no_baseline_holds(self): + self.assertEqual(self.settle(self.path), "held") + self.assertIn("no baseline", self.notes[0]) + self.assertEqual(self.deploys, []) + self.assertTrue(self.state()["handled"]) + + def test_added_file_holds(self): + self.baseline() + self.built["new/index.html"] = "<p>new</p>" + self.assertEqual(self.settle(self.path), "held") + self.assertIn("1 added, 0 removed", self.notes[0]) + self.assertIn("fw blog deploy", self.notes[0]) + self.assertIn("+new/index.html", self.log()) + self.assertEqual(self.deploys, []) + + def test_removed_file_holds(self): + self.baseline() + del self.built["a/index.html"] + self.assertEqual(self.settle(self.path), "held") + self.assertIn("0 added, 1 removed", self.notes[0]) + self.assertEqual(self.deploys, []) + + def test_same_paths_changed_content_deploys_once(self): + self.baseline() + self.built["a/index.html"] = "<p>a, edited</p>" + self.assertEqual(self.settle(self.path), "deployed") + self.assertEqual(len(self.deploys), 1) + self.assertEqual(self.notes, []) # no notification for a success + self.assertIn("deployed: 1 files changed", self.log()) + self.clock.advance(1000) + self.assertEqual(self.tick(self.path), "idle") + self.assertEqual(len(self.deploys), 1) + # the baseline moved on: an identical build later changes nothing + self.write("one.md", "# One\n\nsmall edit\n") + self.assertEqual(self.settle(self.path), "unchanged") + self.assertEqual(len(self.deploys), 1) + + def test_identical_output_does_not_deploy(self): + self.baseline() + self.assertEqual(self.settle(self.path), "unchanged") + self.assertEqual(self.deploys, []) + self.assertEqual(self.notes, []) + self.assertIn("no visible change", self.log()) + self.assertTrue(self.state()["handled"]) + + def test_deploy_failure_notifies_once_and_is_handled(self): + self.baseline() + self.built["a/index.html"] = "<p>edited</p>" + + def failing(cfg, out): + raise blog.BlogError("rsync failed (exit 12)") + self.assertEqual(self.settle(self.path, deploy=failing), "deploy-error") + self.assertEqual(len(self.notes), 1) + self.assertTrue(self.state()["handled"]) + self.assertEqual(self.tick(self.path), "idle") + self.assertEqual(len(self.notes), 1) + + def test_build_error_notifies_once(self): + self.baseline() + + def broken(cfg, out): + raise blog.BlogError("post.md: a code fence is never closed") + self.assertEqual(self.settle(self.path, build=broken), "build-error") + self.assertEqual(self.tick(self.path, build=broken), "idle") + self.assertEqual(len(self.notes), 1) + self.assertIn("code fence", self.notes[0]) + self.assertTrue(self.state()["handled"]) + self.write("one.md", "# One\n\nfixed\n") # a later change retries + self.assertEqual(self.settle(self.path), "unchanged") + + @needs_markdown + def test_real_build_with_unclosed_fence_is_refused(self): + self.config["posts"] = {"2026-01-01-p.md": {"published": True}} + self.path = self.cfg_path() + self.write("2026-01-01-p.md", "# P\n\n## Writing\n\n```\nunclosed\n\n## Spelling\n\nsecret\n") + self.baseline() + self.assertEqual(self.settle(self.path, build=bw.build_site), "build-error") + self.assertIn("code fence", self.notes[0]) + self.assertEqual(self.deploys, []) + + @needs_markdown + def test_real_build_is_deterministic_and_manifest_matches(self): + self.config["posts"] = {"2026-01-01-p.md": {"published": True}} + self.path = self.cfg_path() + self.write("2026-01-01-p.md", "# P\n\n## Writing\n\nhello\n") + cfg = blog.load_config(self.path) + a, b = os.path.join(self.dir, "a"), os.path.join(self.dir, "b") + blog.build(cfg, a) + blog.build(cfg, b) + self.assertEqual(blog.manifest(a), blog.manifest(b)) + self.assertNotIn(blog.MARKER, blog.manifest(a)) + + +class ManualDeploy(WatchCase): + def test_real_deploy_writes_baseline_and_dry_run_does_not(self): + path = self.cfg_path() + cfg = blog.load_config(path) + out = os.path.join(self.dir, "out") + calls = [] + + class Done: + returncode = 0 + run = lambda cmd, **kw: calls.append(cmd) or Done() # noqa: E731 + self.built = {"index.html": "<p>x</p>", "robots.txt": "User-agent: *\n"} + os.makedirs(out) + blog.prepare_out(out) + self.fake_build(cfg, out) + blog.deploy_site(cfg, out, dry_run=True, prebuilt=True, run=run) + self.assertFalse(os.path.exists(os.path.join(self.sd, "deployed.json"))) + self.assertIn("-n", calls[0]) + self.assertEqual(calls[0][0], "rsync") + asked = [] + blog.deploy_site(cfg, out, prebuilt=True, run=run, confirm=lambda t: asked.append(t) or True) + self.assertEqual(asked, ["dh:writing.example.org"]) + cmd = calls[1] + self.assertEqual(cmd[1:1 + len(blog.RSYNC_OPTS)], blog.RSYNC_OPTS) # the protective excludes stay + self.assertIn("/_comments/", cmd) + self.assertIn(".well-known", cmd) + self.assertEqual(cmd[-2:], [out + "/", "dh:writing.example.org/"]) # source first, then target + with open(os.path.join(self.sd, "deployed.json")) as f: + data = json.load(f) + self.assertEqual(sorted(data["files"]), ["index.html", "robots.txt"]) + self.assertIn("time", data) + + def test_declined_or_failed_upload_leaves_baseline_alone(self): + path = self.cfg_path() + cfg = blog.load_config(path) + out = os.path.join(self.dir, "out") + blog.prepare_out(out) + self.fake_build(cfg, out) + + class Failed: + returncode = 12 + with self.assertRaises(blog.BlogError): + blog.deploy_site(cfg, out, prebuilt=True, run=lambda cmd, **kw: Failed(), confirm=lambda t: False) + with self.assertRaises(blog.BlogError): + blog.deploy_site(cfg, out, prebuilt=True, run=lambda cmd, **kw: Failed()) + self.assertFalse(os.path.exists(os.path.join(self.sd, "deployed.json"))) + + def test_audit_still_runs_before_upload(self): + cfg = blog.load_config(self.cfg_path()) + out = os.path.join(self.dir, "out") + blog.prepare_out(out) + self.write("notes.md", "private", base=out) + with self.assertRaises(blog.BlogError): + blog.deploy_site(cfg, out, prebuilt=True, run=lambda cmd, **kw: self.fail("rsync must not run")) + + +class ConfigKeys(WatchCase): + def bad(self, **extra): + with self.assertRaises(blog.BlogError): + blog.load_config(self.cfg_path(**extra)) + + def test_valid(self): + cfg = blog.load_config(self.cfg_path(autodeploy={"quiet_seconds": 60}, + mirror={"repo": "~/x", "branch": "a/b-c_1.2", "remote": "origin", + "exclude": ["*.tmp", "/drafts"]})) + self.assertEqual(cfg["autodeploy"]["quiet_seconds"], 60) + + def test_autodeploy(self): + for value in ({"quiet_seconds": 59}, {"quiet_seconds": "300"}, {"quiet_seconds": True}, {"other": 1}, []): + self.bad(autodeploy=value) + + def test_mirror(self): + for value in ({}, {"repo": ""}, {"repo": 5}, {"repo": "r", "branch": "-x"}, {"repo": "r", "branch": "a b"}, + {"repo": "r", "branch": "a..b"}, {"repo": "r", "remote": "a/b"}, {"repo": "r", "remote": "-x"}, + {"repo": "r", "exclude": "x"}, {"repo": "r", "exclude": ["../x"]}, + {"repo": "r", "exclude": ["/../x"]}, {"repo": "r", "exclude": [1]}, {"repo": "r", "extra": 1}, + {"repo": "r", "exclude": ["!x"]}, {"repo": "r", "exclude": ["+ x"]}, + {"repo": "r", "exclude": ["-x"]}, {"repo": "r", "exclude": [":x"]}, + {"repo": "r", "branch": "main"}, {"repo": "r", "branch": "master"}, {"repo": "r", "branch": "HEAD"}): + self.config.pop("mirror", None) + self.bad(mirror=value) + + +class Plist(WatchCase): + def test_content(self): + path = self.cfg_path() + data = plistlib.loads(bw.plist_text(path, fw="/opt/fw/bin/fw").encode()) + self.assertEqual(data["Label"], "net.fuzzyworld.fw-blog-watch-" + blog.watch_name(path)) + self.assertEqual(data["ProgramArguments"], ["/opt/fw/bin/fw", "blog", "watch", os.path.abspath(path), "--once"]) + self.assertEqual(data["StartInterval"], 60) + self.assertTrue(data["RunAtLoad"]) + self.assertEqual(data["EnvironmentVariables"]["PATH"], "/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin") + self.assertEqual(data["StandardOutPath"], os.path.join(self.sd, "launchd.log")) + self.assertEqual(data["StandardErrorPath"], os.path.join(self.sd, "launchd.log")) + self.assertEqual((data["ProcessType"], data["Nice"]), ("Background", 10)) + + def test_install_and_uninstall_with_a_fake_launchctl(self): + path = self.cfg_path() + agents = os.path.join(self.dir, "LaunchAgents") + calls = [] + + class Ok: + returncode, stderr = 0, "" + run = lambda cmd, **kw: calls.append(cmd) or Ok() # noqa: E731 + plist = bw.install(path, run=run, agents_dir=agents, uid=501) + self.assertEqual(plist, os.path.join(agents, "net.fuzzyworld.fw-blog-watch-" + blog.watch_name(path) + ".plist")) + self.assertTrue(os.path.exists(plist)) + self.assertEqual(calls[-1], ["launchctl", "bootstrap", "gui/501", plist]) + bw.uninstall(path, run=run, agents_dir=agents, uid=501) + self.assertEqual(calls[-1], ["launchctl", "bootout", "gui/501", plist]) + self.assertFalse(os.path.exists(plist)) + + def test_print_flag_writes_nothing(self): + path = self.cfg_path() + proc = subprocess.run([sys.executable, os.path.join(bw.HERE, "blog_watch.py"), "install", path, "--print"], + capture_output=True, text=True, env={**os.environ, "HOME": self.dir}) + self.assertEqual(proc.returncode, 0, proc.stderr) + self.assertIn("net.fuzzyworld.fw-blog-watch-" + blog.watch_name(path), proc.stdout) + self.assertFalse(os.path.exists(os.path.join(self.dir, "Library"))) + + +class Robustness(WatchCase): + def test_hash_cache_skips_unchanged_files(self): + cache = {} + a, _ = bw.scan(self.src, cache) + with mock.patch.object(bw, "file_hash", side_effect=AssertionError("read again")): + b, _ = bw.scan(self.src, cache) + self.assertEqual(a, b) + + def test_cache_entry_is_reused_only_for_an_identical_stamp(self): + cache = {} + bw.scan(self.src, cache) + for i in range(4): # size, mtime, inode, ctime + tampered = {k: list(v) for k, v in cache.items()} + tampered["one.md"][i] += 1 + tampered["one.md"][4] = "stale" + bw.scan(self.src, tampered) + self.assertNotEqual(tampered["one.md"][4], "stale", i) + again = {k: list(v) for k, v in cache.items()} + again["one.md"][4] = "kept" + bw.scan(self.src, again) + self.assertEqual(again["one.md"][4], "kept") # untouched stamp: not re-read + + def test_malformed_hash_cache_is_ignored(self): + path = self.cfg_path(autodeploy={}) + os.makedirs(self.sd) + for junk in ("{not json", "[1, 2]", '{"one.md": "x"}', '{"one.md": [1, 2]}', '{"one.md": [0, 0, 0, 0, 5]}'): + with open(os.path.join(self.sd, "hashes.json"), "w") as f: + f.write(junk) + self.assertIn(self.tick(path), ("changed", "waiting", "idle")) + with open(os.path.join(self.sd, "hashes.json")) as f: + self.assertEqual(sorted(json.load(f)), ["one.md"]) + + def test_run_command_times_out_as_a_failure(self): + result = bw.run_command(["sleep", "5"], timeout=0.2) + self.assertEqual(result.returncode, 124) + self.assertIn("timed out", result.stderr) + + def test_unexpected_exception_is_logged_and_notified_once(self): + path = self.cfg_path(autodeploy={}) + with mock.patch.object(bw, "scan", side_effect=RuntimeError("boom")): + self.assertEqual(self.tick(path), "error") + self.assertEqual(self.tick(path), "error") + self.assertEqual(len(self.notes), 1) + self.assertIn("boom", self.notes[0]) + self.assertIn("boom", self.log()) + self.assertEqual(self.tick(path), "changed") # and the next tick works again + # --once exits 0 even then + with mock.patch.object(bw, "scan", side_effect=RuntimeError("boom")), \ + mock.patch.object(sys, "stderr", new=io.StringIO()): + bw.main(["blog_watch.py", path, "--once"]) + + def test_config_error_recovery_notifies_again_next_time(self): + good = {"quiet_seconds": 300} + path = self.cfg_path(autodeploy={"quiet_seconds": 5}) + self.tick(path) + self.tick(path) + self.assertEqual(len(self.notes), 1) + path = self.cfg_path(autodeploy=good) + self.assertEqual(self.tick(path), "changed") + self.assertNotIn("config_error", self.state()) + path = self.cfg_path(autodeploy={"quiet_seconds": 5}) + self.tick(path) + self.assertEqual(len(self.notes), 2) + + def test_missing_source_folder_notifies_once(self): + path = self.cfg_path(autodeploy={}) + shutil.rmtree(self.src) + self.assertEqual(self.tick(path), "config-error") + self.assertEqual(self.tick(path), "config-error") + self.assertEqual(len(self.notes), 1) + self.assertIn("not found", self.notes[0]) + lines = [] + bw.dry_run(path, self.env(echo=lines.append)) + self.assertIn("not found", lines[0]) + + def test_source_change_during_build_aborts_the_deploy(self): + path = self.cfg_path(autodeploy={}) + self.baseline() + self.built["a/index.html"] = "<p>edited</p>" + + def build_and_edit(cfg, out): + self.fake_build(cfg, out) + self.write("one.md", "# One\n\nedited while building\n") + self.assertEqual(self.settle(path, build=build_and_edit), "changed-during-build") + self.assertEqual(self.deploys, []) + self.assertFalse(self.state()["handled"]) + self.assertIn("changed during build", self.log()) + self.assertEqual(self.tick(path), "changed") # a new quiet period starts + self.clock.advance(301) + self.assertEqual(self.tick(path), "deployed") + + def test_stuck_lock_holder_is_reported_once(self): + path = self.cfg_path(autodeploy={}) + sd = self.sd + held = blog.acquire_lock(sd, now=lambda: self.clock.t - 4000) + self.addCleanup(held.close) + self.assertEqual(self.tick(path), "locked") + self.assertEqual(self.tick(path), "locked") + self.assertEqual(len(self.notes), 1) + self.assertIn("stuck", self.notes[0]) + + def test_state_dir_is_keyed_by_folder_and_path(self): + del os.environ["FW_BLOG_WATCH_DIR"] + a, b = os.path.join(self.dir, "x", "Blog", "fw-blog.json"), os.path.join(self.dir, "y", "Blog", "fw-blog.json") + self.assertRegex(blog.watch_name(a), r"^blog-[0-9a-f]{8}$") + self.assertNotEqual(blog.watch_name(a), blog.watch_name(b)) + self.assertTrue(blog.state_dir(a).endswith(os.path.join(".fw-blog-watch", blog.watch_name(a)))) + + def test_install_retries_bootstrap(self): + path = self.cfg_path() + calls, naps = [], [] + + class R: + def __init__(self, code): + self.returncode, self.stderr = code, "busy" + + def run(cmd, **kw): + calls.append(cmd[1]) + return R(5 if cmd[1] == "bootstrap" and calls.count("bootstrap") < 3 else 0) + bw.install(path, run=run, agents_dir=os.path.join(self.dir, "LA"), uid=1, sleep=naps.append) + self.assertEqual(calls.count("bootstrap"), 3) + self.assertEqual(naps, [1, 1]) + calls.clear() + with self.assertRaises(blog.BlogError): + bw.install(path, run=lambda cmd, **kw: R(5), agents_dir=os.path.join(self.dir, "LA"), uid=1, + sleep=naps.append) + + +class GeneratorGuard(WatchCase): + def setUp(self): + super().setUp() + self.path = self.cfg_path(autodeploy={}) + + def test_generator_change_holds(self): + self.baseline(generator="old") + self.built["a/index.html"] = "<p>edited</p>" + self.assertEqual(self.settle(self.path), "held") + self.assertIn("generator changed", self.notes[0]) + self.assertEqual(self.deploys, []) + + def test_deploy_target_change_holds(self): + self.baseline() + self.path = self.cfg_path(deploy={"host": "dh", "dir": "other.example.org"}, autodeploy={}) + self.built["a/index.html"] = "<p>edited</p>" + self.assertEqual(self.settle(self.path), "held") + self.assertIn("deploy target changed", self.notes[0]) + self.assertEqual(self.deploys, []) + + def test_config_change_holds(self): + self.baseline() + self.path = self.cfg_path(title="A new title", autodeploy={}) + self.built["a/index.html"] = "<p>edited</p>" + self.assertEqual(self.settle(self.path), "held") + self.assertIn("config changed", self.notes[0]) + self.assertEqual(self.deploys, []) + + def test_deployed_json_records_generator_config_and_target(self): + self.baseline() + self.built["a/index.html"] = "<p>edited</p>" + self.assertEqual(self.settle(self.path), "deployed") # the watcher's own deploy keeps them current + with open(os.path.join(self.sd, "deployed.json")) as f: + data = json.load(f) + self.assertEqual(data["generator"], blog.GENERATOR_HASH) + self.assertEqual(data["config"], blog.config_hash(blog.load_config(self.path))) + self.assertEqual(data["target"], "dh:writing.example.org") + + def test_generator_hash_is_fixed_when_the_module_loads(self): + before = blog.GENERATOR_HASH + other = os.path.join(self.dir, "blogdir") + os.makedirs(other) + self.write("x.css", "changed", base=other) + with mock.patch.object(blog, "BLOG_DIR", other): + self.assertNotEqual(blog.generator_hash(), before) # a fresh computation sees the edit ... + self.assertEqual(blog.GENERATOR_HASH, before) # ... the value the running code uses does not + self.assertEqual(bw.Env().generator(), before) + + def test_generator_hash_follows_each_input(self): + fake = os.path.join(self.dir, "srv") + for rel in ("blog.py", "atom.py", "gen.py", "site/_shared.css", "site/favicon.ico", "site-blog/blog.css"): + self.write(rel, "v1", base=fake) + with mock.patch.object(blog, "HERE", fake), mock.patch.object(blog, "SITE_DIR", os.path.join(fake, "site")), \ + mock.patch.object(blog, "BLOG_DIR", os.path.join(fake, "site-blog")): + seen = {blog.generator_hash()} + for rel in ("blog.py", "atom.py", "gen.py", "site/_shared.css", "site-blog/blog.css"): + self.write(rel, "v2 " + rel, base=fake) + seen.add(blog.generator_hash()) + self.assertEqual(len(seen), 6) # every edit changed the hash + + def test_dry_run_reports_each_decision(self): + lines = [] + env = self.env(echo=lines.append) + bw.dry_run(self.path, env) + self.assertIn("no baseline", lines[-1]) + self.baseline() + lines.clear() + self.built["a/index.html"] = "<p>edited</p>" + bw.dry_run(self.path, env) + self.assertIn("would upload (1 files changed)", lines[-1]) + self.assertFalse(os.path.exists(os.path.join(self.sd, "state.json"))) + self.assertEqual(self.deploys, []) + + +class MirrorRecovery(WatchCase): + def setUp(self): + super().setUp() + self.repo = os.path.join(self.dir, "repo") + self.path = self.cfg_path(mirror={"repo": self.repo}) + + def commits(self): + return git(self.repo, "log", "--format=%s").splitlines() + + def run_with_rsync_code(self, code): + def run(cmd, cwd=None, timeout=None): + result = bw.run_command(cmd, cwd=cwd, timeout=timeout or 120) + if cmd[0] == "rsync" and result.returncode == 0: + return subprocess.CompletedProcess(cmd, code, result.stdout, "some files vanished") + return result + return run + + def test_rsync_24_is_retried_over_our_own_dirt(self): + self.write("one.md", "# One\n\nv2\n") + self.assertEqual(self.settle(self.path, run=self.run_with_rsync_code(24)), "blocked") + self.assertEqual(self.notes, []) # 24 is normal while Dropbox syncs + self.assertIn("mirror_dirty", self.state()) + self.assertFalse(self.state()["handled"]) + self.assertEqual(self.tick(self.path, run=self.run_with_rsync_code(24)), "blocked") # still ours + self.assertEqual(self.tick(self.path), "mirrored") + self.assertNotIn("mirror_dirty", self.state()) + self.assertEqual(len(self.commits()), 1) + + def test_rsync_23_notifies_once_and_recovers(self): + self.assertEqual(self.settle(self.path, run=self.run_with_rsync_code(23)), "blocked") + self.tick(self.path, run=self.run_with_rsync_code(23)) + self.assertEqual(len(self.notes), 1) + self.assertIn("rsync", self.notes[0]) + self.assertEqual(self.tick(self.path), "mirrored") + + def ignoring_source(self): + self.write(".gitignore", "*.log\n") + self.write("run.log", "kept out of git\n") + + def test_ignored_leftovers_are_accepted_only_as_recorded(self): + self.ignoring_source() + self.assertEqual(self.settle(self.path), "mirrored") + self.assertIn("run.log", json.dumps(self.state()["mirror_dirty"])) + self.write("one.md", "# One\n\nsecond edit\n") + self.assertEqual(self.settle(self.path), "mirrored") # the recorded leftovers are ours + self.assertEqual(len(self.commits()), 2) + + def test_human_edit_of_a_tracked_file_blocks_despite_the_record(self): + self.ignoring_source() + self.settle(self.path) + with open(os.path.join(self.repo, "one.md"), "w") as f: + f.write("my own work in the repo\n") + self.write("one.md", "# One\n\nsource edit\n") + self.assertEqual(self.settle(self.path), "blocked") + with open(os.path.join(self.repo, "one.md")) as f: + self.assertEqual(f.read(), "my own work in the repo\n") # preserved + self.assertIn("uncommitted", self.notes[-1]) + + def test_human_edit_of_a_leftover_file_blocks(self): + self.ignoring_source() + self.settle(self.path) + self.write("run.log", "my notes\n", base=self.repo) + self.write("one.md", "# One\n\nsource edit\n") + self.assertEqual(self.settle(self.path), "blocked") + + def test_new_path_beside_the_record_blocks(self): + self.ignoring_source() + self.settle(self.path) + self.write("other.log", "new\n", base=self.repo) + self.write("one.md", "# One\n\nsource edit\n") + self.assertEqual(self.settle(self.path), "blocked") + + def test_non_ascii_and_odd_names_in_the_record(self): + names = ["заметки.log", "a -> b.log", "with space.log"] + self.write(".gitignore", "*.log\n") + for name in names: + self.write(name, "kept out of git\n") + self.assertEqual(self.settle(self.path), "mirrored") + recorded = self.state()["mirror_dirty"]["status"] + for name in names: + self.assertIn("!! " + name, recorded) + self.write("one.md", "# One\n\nsecond edit\n") + self.assertEqual(self.settle(self.path), "mirrored") # untouched leftovers are accepted + for name in names: + self.write(name, "my notes\n", base=self.repo) + self.write("one.md", "# One\n\nedit after " + name + "\n") + self.assertEqual(self.settle(self.path), "blocked", name) + with open(os.path.join(self.repo, name), encoding="utf-8") as f: + self.assertEqual(f.read(), "my notes\n") # preserved + os.remove(os.path.join(self.repo, name)) + self.write(name, "kept out of git\n", base=self.repo) # put the leftover back as it was + + def test_record_for_another_repo_is_ignored(self): + self.ignoring_source() + self.settle(self.path) + state = self.state() + state["mirror_dirty"]["repo"] = "/somewhere/else" + with open(os.path.join(self.sd, "state.json"), "w") as f: + json.dump(state, f) + self.write("one.md", "# One\n\nsource edit\n") + self.assertEqual(self.settle(self.path), "blocked") + + def test_marker_does_not_excuse_a_changed_head(self): + self.settle(self.path) + self.write("one.md", "# One\n\nv2\n") + self.assertEqual(self.settle(self.path, run=self.run_with_rsync_code(24)), "blocked") + git(self.repo, "commit", "-q", "-a", "-m", "by hand") # HEAD moved: the dirt is no longer provably ours + self.write("stray.txt", "x", base=self.repo) + self.assertEqual(self.tick(self.path), "blocked") + self.assertIn("uncommitted", self.notes[-1]) + + def test_relative_repo_is_relative_to_the_config_folder(self): + path = self.cfg_path(mirror={"repo": "../repo-rel"}) + self.assertEqual(self.settle(path), "mirrored") + self.assertTrue(os.path.isdir(os.path.join(self.dir, "repo-rel", ".git"))) + + def test_git_ignored_files_count_as_dirty(self): + self.settle(self.path) + self.write(".gitignore", "*.log\n", base=self.repo) + git(self.repo, "add", ".gitignore") + git(self.repo, "commit", "-q", "-m", "ignore") + self.write("keep.log", "mine", base=self.repo) + self.write("one.md", "# One\n\nv2\n") + self.assertEqual(self.settle(self.path), "blocked") + self.assertTrue(os.path.exists(os.path.join(self.repo, "keep.log"))) + + def test_user_excludes_come_before_the_protective_git_exclude(self): + cmd = bw.rsync_command("/s", "/r", ["secret/"]) + self.assertLess(cmd.index("--exclude=secret/"), cmd.index("--exclude=/.git")) + self.assertEqual(cmd[-2:], ["/s/", "/r/"]) + + def test_push_uses_explicit_refspec(self): + calls = [] + base = bw.run_command + + def run(cmd, cwd=None, timeout=None): + calls.append(cmd) + return base(cmd, cwd=cwd, timeout=timeout or 120) + remote = os.path.join(self.dir, "r.git") + subprocess.run(["git", "init", "-q", "--bare", remote], check=True) + path = self.cfg_path(mirror={"repo": self.repo, "remote": "origin"}) + os.makedirs(self.repo) + subprocess.run(["git", "init", "-q", "-b", "dropbox-mirror", self.repo], check=True) + git(self.repo, "remote", "add", "origin", remote) + self.settle(path, run=run) + pushes = [c for c in calls if "push" in c] + self.assertEqual(pushes[0][-2:], ["origin", "refs/heads/dropbox-mirror:refs/heads/dropbox-mirror"]) + + +class DeployCommand(WatchCase): + def test_main_passes_flags_and_exit_codes(self): + path = self.cfg_path() + out = os.path.join(self.dir, "o") + with mock.patch.object(blog, "deploy_site") as deploy: + blog.main(["blog.py", "deploy", path, "--out", out, "--yes"]) + self.assertIsNone(deploy.call_args.kwargs["confirm"]) + self.assertFalse(deploy.call_args.kwargs["dry_run"]) + blog.main(["blog.py", "deploy", path, "--out", out, "--dry-run"]) + self.assertTrue(deploy.call_args.kwargs["dry_run"]) + self.assertIs(deploy.call_args.kwargs["confirm"], blog.ask_to_upload) + deploy.side_effect = blog.BlogError("rsync failed (exit 12)") + with self.assertRaises(SystemExit) as caught: + blog.main(["blog.py", "deploy", path, "--out", out, "--yes"]) + self.assertIn("rsync failed", str(caught.exception.code)) + + @needs_markdown + def test_not_a_terminal_refuses_without_yes(self): + path = self.cfg_path() + out = os.path.join(self.dir, "o") + with mock.patch.object(sys.stdin, "isatty", return_value=False), \ + mock.patch.object(blog.subprocess, "run", side_effect=AssertionError("rsync must not run")): + with self.assertRaises(SystemExit) as caught: + blog.main(["blog.py", "deploy", path, "--out", out]) + self.assertIn("not a terminal", str(caught.exception.code)) + + def test_eof_at_the_prompt_declines(self): + with mock.patch.object(sys.stdin, "isatty", return_value=True), mock.patch("builtins.input", side_effect=EOFError): + self.assertFalse(blog.ask_to_upload("dh:x")) + + def test_ctrl_c_at_the_prompt_exits_quietly_with_1(self): + with mock.patch.object(sys.stdin, "isatty", return_value=True), \ + mock.patch("builtins.input", side_effect=KeyboardInterrupt), mock.patch.object(sys, "stderr", new=io.StringIO()): + with self.assertRaises(SystemExit) as caught: + blog.ask_to_upload("dh:x") + self.assertEqual(caught.exception.code, 1) + + def test_lock_is_taken_after_the_answer_and_manual_rsync_has_no_timeout(self): + cfg = blog.load_config(self.cfg_path()) + out = os.path.join(self.dir, "out") + blog.prepare_out(out) + self.fake_build(cfg, out) + seen = {} + + class Done: + returncode = 0 + + def confirm(target): + lock = blog.acquire_lock(self.sd) # free while the question is open + seen["free"] = lock is not None + if lock: + lock.close() + return True + blog.deploy_site(cfg, out, prebuilt=True, confirm=confirm, + run=lambda cmd, **kw: seen.update(kw) or Done()) + self.assertTrue(seen["free"]) + self.assertIsNone(seen["timeout"]) + with mock.patch.object(blog, "deploy_site") as deploy: + bw.deploy_built(cfg, out) + self.assertEqual(deploy.call_args.kwargs["timeout"], 600) + self.assertTrue(deploy.call_args.kwargs["locked"]) + + def test_watcher_deploy_while_answering_refuses_the_manual_upload(self): + cfg = blog.load_config(self.cfg_path()) + out = os.path.join(self.dir, "out") + blog.prepare_out(out) + self.fake_build(cfg, out) + self.baseline() + + def confirm(target): # the watcher deploys while the question is open + blog.write_deployed(self.sd, {"index.html": "x"}, "2030-01-01T00:00:00Z", cfg=cfg) + return True + with self.assertRaises(blog.BlogError) as caught: + blog.deploy_site(cfg, out, prebuilt=True, confirm=confirm, run=lambda cmd, **kw: self.fail("no rsync")) + self.assertIn("watcher deployed while you were answering", str(caught.exception)) + self.assertIsNotNone(blog.acquire_lock(self.sd)) # and the lock was released + + def test_manual_upload_resets_the_watchers_handled_state(self): + cfg = blog.load_config(self.cfg_path()) + out = os.path.join(self.dir, "out") + blog.prepare_out(out) + self.fake_build(cfg, out) + os.makedirs(self.sd, exist_ok=True) + with open(os.path.join(self.sd, "state.json"), "w") as f: + json.dump({"fingerprint": "x", "handled": True, "conflicts": [], "changed_at": 5}, f) + + class Done: + returncode = 0 + blog.deploy_site(cfg, out, prebuilt=True, run=lambda cmd, **kw: Done()) + with open(os.path.join(self.sd, "state.json")) as f: + state = json.load(f) + self.assertNotIn("fingerprint", state) + self.assertNotIn("handled", state) + self.assertEqual(state["changed_at"], 5) + self.cfg_path(autodeploy={}) + self.assertEqual(self.tick(self.cfg_file), "changed") # the next tick re-evaluates + + def test_manual_deploy_is_refused_while_a_tick_holds_the_lock(self): + cfg = blog.load_config(self.cfg_path()) + out = os.path.join(self.dir, "out") + blog.prepare_out(out) + self.fake_build(cfg, out) + held = blog.acquire_lock(self.sd) + self.addCleanup(held.close) + with self.assertRaises(blog.BlogError) as caught: + blog.deploy_site(cfg, out, prebuilt=True, run=lambda cmd, **kw: self.fail("no rsync")) + self.assertIn("try again in a minute", str(caught.exception)) + + class Done: + returncode = 0 + # the watcher itself holds the lock, so its own deploy path must not take it again; a dry run never needs it + blog.deploy_site(cfg, out, prebuilt=True, locked=True, run=lambda cmd, **kw: Done()) + blog.deploy_site(cfg, out, prebuilt=True, dry_run=True, run=lambda cmd, **kw: Done()) + + @needs_markdown + def test_bin_fw_deploy_without_flags_reaches_the_prompt_check(self): + path = self.cfg_path() + env = {**os.environ, "TMPDIR": self.dir, "FW_PYLIB": blog.gen.PYLIB} + proc = subprocess.run([os.path.join(bw.HERE, "..", "bin", "fw"), "blog", "deploy", path], + capture_output=True, text=True, stdin=subprocess.DEVNULL, env=env) + self.assertEqual(proc.returncode, 1, proc.stderr) + self.assertIn("not a terminal", proc.stderr) + proc = subprocess.run([os.path.join(bw.HERE, "..", "bin", "fw"), "blog", "deploy", path, "--bogus"], + capture_output=True, text=True, stdin=subprocess.DEVNULL, env=env) + self.assertEqual(proc.returncode, 2) + + def test_bin_fw_watch_once_never_installs_libraries(self): + path = self.cfg_path() + env = {**os.environ, "FW_PYLIB": os.path.join(self.dir, "empty-pylib"), "PATH": os.environ["PATH"]} + proc = subprocess.run([os.path.join(bw.HERE, "..", "bin", "fw"), "blog", "watch", path, "--once"], + capture_output=True, text=True, stdin=subprocess.DEVNULL, env=env, timeout=60) + self.assertEqual(proc.returncode, 1, proc.stderr) + self.assertIn("missing", proc.stderr) + self.assertFalse(os.path.exists(env["FW_PYLIB"])) + + +if __name__ == "__main__": + unittest.main()