fuzzyworld-tools / commit
README: rewrite as an overview; detail moved to docs/
PLAN.md | 9 +- README.md | 572 +++++++++++---------------------------------------- docs/admin.md | 77 +++++++ docs/apps.md | 84 ++++++++ docs/blog.md | 249 ++++++++++++++++++++++ docs/ci.md | 96 +++++++++ docs/code-site.md | 86 ++++++++ docs/issues.md | 148 +++++++++++++ docs/operations.md | 122 +++++++++++ docs/usage.md | 70 +++++++ server/blog_watch.py | 2 +- 11 files changed, 1059 insertions(+), 456 deletions(-) diff --git a/PLAN.md b/PLAN.md index 2d5c4a4..e49b4a7 100644 --- a/PLAN.md +++ b/PLAN.md @@ -100,7 +100,7 @@ always-on machine; a small setup is a DreamHost account and a laptop. **Built and live 2026-10-31 at `writing.fuzzyworld.net`** (queue item Q4; not linked from anywhere yet). The generator is `server/blog.py` (+ `server/atom.py`, `server/site-blog/`), the importer `lib/wp_import.py`; commands and behavior are in -the README ("Blog"). Decisions taken: build on the laptop from Dropbox and upload only the audited output folder (never +docs/blog.md. Decisions taken: build on the laptop from Dropbox and upload only the audited output folder (never the source); footer "All rights reserved" notice with an AI-training line and a `robots.txt` blocking AI crawlers; the WordPress text wins when a source file differs (similarity under 0.95: 100 of the 142 posts are written from WordPress into `~/Blog/wordpress/`); the old site's custom CSS is carried over (hidden `<br>`, dotted scene breaks, underlined @@ -152,7 +152,7 @@ basic auth first; client-side encryption only if needed (it adds a script and al - Reuse what exists: Python-Markdown, Pygments, the site CSS, the feed code from step 1. - **Import (🔌 plugin):** one-time script that reads the WordPress API: posts, comments, media, and a redirect map. - The editor and the Dropbox → private-git-repo move come after the generator, not with it. -- **Built 2026-11-02 (Q91/Q92):** `fw blog watch <config>` (`server/blog_watch.py`; README "Blog", "Watcher"): a laptop +- **Built 2026-11-02 (Q91/Q92):** `fw blog watch <config>` (`server/blog_watch.py`; docs/blog.md, "The watcher"): a laptop launchd agent (`fw blog watch install`) runs one tick every 60 s per blog config. After a quiet period it mirrors the Dropbox source into a local git repo (snapshot commits on `dropbox-mirror`, never main) and, with `autodeploy`, deploys edits only when the set of output files, the generator (code/templates), the config and the deploy target are @@ -187,7 +187,7 @@ he wants; see the tiers in the README, form = T2, email = T1). The form stays in ## CI dashboard -✅ Built 2026-10-27 as proposed (README: "Watching CI"): +✅ Built 2026-10-27 as proposed (docs/ci.md: "Watching CI"): - `code.fuzzyworld.net/ci/`: one static page for all public repos: queued jobs, running pipelines (which job, for how long), and the last 50 results. Rebuilt on every queue change and every upload from a runner. @@ -283,7 +283,7 @@ Wording approved 2026-10-26: ## CI -✅ Live since 2026-10-27. The README's CI section is the full reference; this is the summary. It is the same +✅ Live since 2026-10-27. docs/ci.md is the full reference; this is the summary. It is the same pattern as GitLab runners: **a runner registers, pulls jobs, runs them in containers, and reports back.** The difference is that there's no coordinator service: the coordinator is files on DreamHost. @@ -423,6 +423,7 @@ to the plain recorded run, with comparison as an option (game PLAN §20.2 step 5 - **Watch = dev replay:** a link to the deployed viewer when the record is public, "Download record" always, and "Prepare dev replay" (runs the app's `report-replay` npm script with the record and `--rules <commit>`; needs `npm run dev` running in the app). - **Recently approved comments** (the 20 newest) with a Remove button are in v1, not later. - **`FW_CODE_DIR`** (also `--code-dir`, default `~/code`) is configurable: where the apps' working copies live. +- **Deploy tab (Q111 stage 1, built 2026-11-04):** read-only; `fw deploys --json` (server/deploys.py) joins the deploy ledger (~/fw/state/deployed.json, written by `bin/deploy-tools`, keyed `<repo>/<target>`) and each app's `.fuzzyworld.json` with the repo's HEAD. Targets are declared in package.json `fuzzyworld.deploy`. Next: stage 2, a "Deploy now" button that deploys `origin/main` from a clean export (confirm page with details, background run, progress page); stage 3, any repo declares targets, apps included, tested on nordic-skating. - Still not in v1: "make this a regression fixture", pull requests, CI, blog state. A **local-only** web page over `fw` commands (see "Report replays": never hosted on DreamHost). `fw admin` starts a diff --git a/README.md b/README.md index fa46259..9791ddd 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,24 @@ # fuzzyworld-tools -Tooling for Pavel's self-hosted code and apps on DreamHost: +Tooling for Pavel's self-hosted code and apps on DreamHost. It replaces GitLab (code, CI, Pages) and WordPress +(the blog) with static files plus a few helper machines. -- **code.fuzzyworld.net**: static browse pages and read-only `git clone` for public repos: every unmerged branch, syntax highlighting, file history, blame, CI results -- **writing.fuzzyworld.net**: the writing blog, a static site built from Markdown by `fw blog` (see [Blog](#blog)); with `site` set in its config it shows the same site-wide header as the other sites -- **apps.fuzzyworld.net**: deployed web apps, one per path (`/bubbles/`, `/slide/`, ...), published with `fw deploy`, with screenshots taken by CI -- **VS Code in the browser** (`apps.fuzzyworld.net/vscode/?repo=<name>`): any public repo, read-only, with quick open and search -- **CI runners** (TrueNAS, and the laptop when it is idle): run each push's pipeline (`.fw-ci.yml`) in Docker, like a GitLab runner +**The one rule: DreamHost only serves files.** There is no app server. Pages are generated when you push, and +Apache just sends them. Work that needs a real machine (CI, screenshots) runs on a *runner* (a home TrueNAS box, or +the laptop when it is idle), which pulls jobs from DreamHost over ssh. No runner is a normal state: jobs wait. -There is no app server. Pages are generated when you push; Apache only serves files. Work that needs a real -machine (CI, screenshots) runs on a runner (TrueNAS, or the laptop), which pulls jobs from DreamHost over ssh. No runner is a normal state: jobs wait. The plan and backlog are in -[PLAN.md](PLAN.md). +The roadmap and backlog are in [PLAN.md](PLAN.md). + +## What it provides + +| Site | What it is | Details | +|---|---|---| +| **code.fuzzyworld.net** | Browse pages and read-only `git clone` for public repos: every unmerged branch, syntax highlighting, history, blame, CI results, bug-report issues, AI-usage charts | [docs/code-site.md](docs/code-site.md) | +| **apps.fuzzyworld.net** | Deployed web apps, one per path (`/bubbles/`, `/slide/`, ...), published with `fw deploy`, with CI screenshots | [docs/apps.md](docs/apps.md) | +| **writing.fuzzyworld.net** | The writing blog: a static site built from Markdown by `fw blog`, with comments by email | [docs/blog.md](docs/blog.md) | +| **VS Code in the browser** | Any public repo, read-only, at `apps.fuzzyworld.net/vscode/?repo=<name>` | [docs/operations.md](docs/operations.md#vs-code-in-the-browser) | +| **CI runners** | Run each push's pipeline (`.fw-ci.yml`) in Docker, like a GitLab runner | [docs/ci.md](docs/ci.md) | +| **Admin panel** | `fw admin`: a local web page for bug reports, blog comments and deploy state | [docs/admin.md](docs/admin.md) | ## Philosophy @@ -20,502 +28,164 @@ are MIT-licensed: use them, change them, share them. My conversations with the AI are different. My words and my way of working are mine. I publish a conversation only when I choose to. -The same text is on [/why-i-left-gitlab/](https://code.fuzzyworld.net/why-i-left-gitlab/) (formerly /why/ and /principles/), and in the `/how/` draft (not live yet). +The same text is on [/why-i-left-gitlab/](https://code.fuzzyworld.net/why-i-left-gitlab/). ## Hosting tiers -Each feature needs a certain kind of host. The tiers say what that is, so you can pick what your hosting allows. -A tier includes the ones above it. +Each feature needs a certain kind of host. A tier includes the ones above it, so you can pick what your hosting allows. | Tier | The host must offer | Features | -| --- | --- | --- | +|---|---|---| | **T0, files only** (ftp only) | serving static files | the built blog, Atom feeds, the landing page, the generated code and CI pages, VS Code in the browser | -| **T1, files + jobs** (ftp + python) | ssh, cron, git hooks, a mailbox reachable by IMAP or on the same host; nothing runs when a page is requested | page generation on push, the CI queue and runner API, cleanup jobs, comments by email, the `fw` commands | -| **T2, files + an endpoint** (ftp + php/cgi) | a script that answers web requests | the comment form (`post.php`: written, **not enabled, not installed**) | +| **T1, files + jobs** (ftp + python) | ssh, cron, git hooks, a mailbox reachable by IMAP (or on the same host); nothing runs when a page is requested | page generation on push, the CI queue and runner API, cleanup jobs, comments and bug reports by email, the `fw` commands | +| **T2, files + an endpoint** (ftp + php/cgi) | a script that answers web requests | the comment form (`post.php`): written, **not enabled, not installed** | | **T3, always-on process** | a long-running server | nothing; out of scope | -T2 is new attack surface: code that runs on a web request, as the same user as everything else on the account. -The comment form is written but switched off and not installed; comments are accepted by email only (T1). To use the form you would -host `server/comments/post.php` on its own subdomain and set `comments_form` in the blog config. +T2 is new attack surface (code that runs on a web request, as the same user as everything else on the account), +so comments are accepted by email only. To use the form you would host `server/comments/post.php` on its own +subdomain and set `comments_form` in the blog config. -## Making a repo deploy as an app +## Quick start -Add to the app repo's `package.json`: +First create the repo on DreamHost (private unless `--public`): + +```bash +fw new my-game --public --description "One line." +``` + +Then make it an app (details and every option: [docs/apps.md](docs/apps.md)): ```jsonc +// package.json of the app repo "scripts": { "deploy": "fw deploy" }, -"fuzzyworld": { - "app": "my-game", // URL: apps.fuzzyworld.net/my-game/ (required) - "title": "My Game", // apps index - "description": "One line.", // apps index - "repo": "my-game", // DreamHost repo name, if different from app - "prebuild": ["pnpm gen-assets"], // commands before checks and build - "checks": ["typecheck", "test"], // scripts that must pass (default: whichever of typecheck/lint/test exist) - "build": "make public", // default: `<pm> run build` - "out": "dist", // default: dist, else public, else build - "keep": ["my-game.apk"], // files on the server a deploy must not delete or replace (e.g. a download built elsewhere) - "extra": ["sim/*.js"], // build-output files every later deploy keeps: uploaded and updated, never deleted (e.g. old rules versions) - "replay": "replay.html?report={record}" // a page of this app that plays a bug report's run record; issue pages link to it (see Issues) -}, +"fuzzyworld": { "app": "my-game", "title": "My Game", "description": "One line." }, "devDependencies": { "fuzzyworld-tools": "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0" } ``` -For vite, add the plugin; it sets `base` to `/<app>/` for builds (dev server unchanged; `FW_APP`, set by `fw deploy --as`, overrides the app name): - -```ts -import fuzzyworld from 'fuzzyworld-tools/vite'; -export default defineConfig({ plugins: [fuzzyworld()] }); +```bash +git push dh:git/my-game.git # `dh` is the SSH alias in ~/.ssh/config +npm run deploy # clone main, install, check, build, upload, print the URL ``` -Then `pnpm run deploy` / `npm run deploy`. **`fw deploy`**: - -1. clones branch `main` of `dh:git/<repo>.git` into a temp dir. It builds what is pushed, never your working copy (`--ref <branch>` for another branch, `--here` to build the current directory as is) -2. installs with the lockfile's package manager (`pnpm install --frozen-lockfile`, `npm ci`, `yarn install --frozen-lockfile`) -3. runs `prebuild`, then the `checks`, then the build. Any failure stops the deploy. -4. checks the output: every same-site URL in `index.html`, the web manifest's `id`/`scope`/`start_url`, and every `serviceWorker.register(...)` script and scope must stay inside `/<app>/`. That keeps apps on the shared origin from capturing each other, and makes each one install separately on Android. Any problem stops the deploy. -5. `rsync --delete` to `apps.fuzzyworld.net/<app>/` (except `keep` paths; `extra` files are protected from `--delete` by an rsync filter, so they are uploaded and updated but never removed), writes `.fuzzyworld.json` (commit, time, and the `replay` template; not web-readable), and refreshes the apps index and the repo's code page -6. prints the URL - -`--dry-run` does everything except upload. +Use `fw` outside an app repo (for example on a second laptop): -The apps index is a table, ordered by the newest commit of each app's repo (not by deploy date, since an app can be redeployed often without its code changing; the deploy date is shown in its own column). Columns: a screenshot thumbnail if the runner has taken one, name, description, last commit, deployed, and a source link for apps from public repos. The code index (code.fuzzyworld.net) is likewise a table of public repos, newest commit first, with description and an app link. - -Apps hosted elsewhere are listed on the index from `server/site-apps/extra.json` (engye: Cloudflare Pages; `apps.fuzzyworld.net/engye/...` redirects there). +```bash +npm install -g "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0" +``` ## Commands -On the laptop, `bin/fw` runs `deploy` locally and forwards repo commands to DreamHost (`ssh dh ~/bin/fw ...`): +`bin/fw` runs some commands on the laptop (`deploy`, `repoint`, `usage`, `admin`, `blog`) and forwards the rest to +DreamHost (`ssh dh ~/bin/fw ...`). + +### Repos | Command | What it does | |---|---| -| `fw deploy [--ref B] [--as NAME] [--here] [--dry-run]` | build and publish this repo's app (above). `--as NAME` publishes it at `apps.fuzzyworld.net/NAME/` instead, a second copy of the same repo (e.g. `explodorama-next` built from `main` while `/explodorama/` is deployed from a tag with `--ref`); the build gets `FW_APP=NAME`, which the vite plugin uses for `base`, and the index title gets the suffix | -| `fw repoint [DIR] [--https] [--dry-run]` | for each repo in DIR whose `origin` is on gitlab.com: rename it `gitlab`, add `origin` → `dh:git/<repo>.git` (or read-only `https://code.fuzzyworld.net/<repo>.git` with `--https`, public repos only), fetch, and retarget branch tracking. Changes only remotes; checks that uncommitted work, HEAD and stashes are unchanged | -| `fw usage [--all\|--now\|--trailers [--session ID]]` | Claude Code token use per session from `~/.claude/projects` (runs locally): messages, current context size, cache-read, output (cache-read and output include subagents). `--now` prints one line for the newest session, to decide when to put it down. `--trailers` prints the commit-message trailers (exact integers, one `FW-Model-Tokens` line per model, `FW-Tokens-Method: 2`) for the newest session here or the one named; it warns on stderr when that transcript is over 10 minutes old | -| `fw admin [--port N] [--no-open] [--code-dir DIR]` | a local web page for reports and blog comments, over the commands below (runs locally; see [Admin panel](#admin-panel)) | -| `fw comments list\|show\|approve\|reject\|remove\|untrust\|publish\|export\|ingest\|imap-check` | moderate blog comments received by email (see [Blog](#blog)); runs on the server | -| `fw issues <repo> public\|reviewed\|private` | set who sees the repo's bug-report issue pages (`git config fuzzyworld.issues`; default private) and republish them | -| `fw issues list\|show\|approve\|hide\|close\|reopen\|publish\|publish-all\|ingest\|imap-check` | moderate bug-report issues received by email (see [Issues](#issues)); runs on the server | | `fw list` | every repo, its visibility and last commit | | `fw new <name> [--public] [--description TEXT]` | create `~/git/<name>.git` on DreamHost; **private unless `--public`** | -| `fw public <name>` | publish the source on code.fuzzyworld.net right away | -| `fw private <name>` | unpublish right away: pages, clone URL and index entry are removed | +| `fw public <name>` / `fw private <name>` | publish the source on code.fuzzyworld.net right away / unpublish it (pages, clone URL and index entry are removed) | | `fw describe <name> TEXT` | set the description shown on the index and summary page | | `fw publish <name>` | refresh one repo's public copy and pages (the push hook does this) | -| `fw publish-all` | refresh every repo (printing `[n/total] name (seconds)` as each finishes), then the index and the CI dashboard | -| `fw ci <name> [branch]` | queue a CI run of a branch's latest commit (default: the default branch) | -| `fw ci-list` | queued and running CI jobs | -| `fw ci-watch` | follow CI in the terminal (private repos too): running jobs with each job's state, the queue, the last 5 results; ends when nothing is queued or running | -| `fw blog list\|build\|preview\|deploy <config>` | the static blog (see [Blog](#blog)); runs on the laptop | -| `fw blog watch <config> [--once] [--dry-run]` / `fw blog watch install\|uninstall <config>` | mirror the blog source into a git repo and auto-deploy edits that change no URLs (see [Blog](#blog)); one machine only | -| `fw blog import-wp <config> --wp URL` / `fw blog check <config>` | one-time WordPress import / compare the built site with the WordPress text | - -Push to a repo with `git push dh:git/<name>.git` (`dh` is the SSH alias in `~/.ssh/config`). - -## Visibility - -Each bare repo stores its own setting: `git config fuzzyworld.visibility public|private`. -**Anything other than exactly `public` is private**, including a missing value, so a repo is never published by accident. -A private repo has no pages, no clone URL and no index entry. Pushing to it publishes nothing. +| `fw publish-all` | refresh every repo, printing `[n/total] name (seconds)`, then the index and the CI dashboard | +| `fw repoint [DIR] [--https] [--dry-run]` | for each repo in DIR whose `origin` is on gitlab.com: rename it `gitlab`, add `origin` on DreamHost (or the read-only https URL with `--https`, public repos only), fetch, retarget branch tracking. Changes only remotes, and checks that uncommitted work, HEAD and stashes are unchanged | -Other settings in the repo config: `fuzzyworld.issues` (`public`, `reviewed` or anything else = private: who sees bug-report issue pages, see [Issues](#issues); a private repo never shows issues), `fuzzyworld.description`, `fuzzyworld.ci` (`off`: pushes queue no CI jobs), plus `fuzzyworld.gitlab-id` / `fuzzyworld.gitlab-path` for repos migrated from GitLab. +### Apps and deploys -## Layout on DreamHost - -| Path | Contents | -|---|---| -| `~/git/<name>.git` | canonical bare repos (push here; not web-visible) | -| `~/code.fuzzyworld.net/<name>/` | generated pages for public repos | -| `~/code.fuzzyworld.net/<name>.git` | public clone copy (dumb HTTP), refreshed on push | -| `~/apps.fuzzyworld.net/<app>/` | deployed apps (`fw deploy`); `index.html` at the root is generated | -| `~/writing.fuzzyworld.net/` | the blog's built site, uploaded by `fw blog deploy` from the laptop (never edited there) | -| `~/fw/tools/server/` | this repo's `server/` directory, installed by `server/install.sh` | -| `~/bin/fw` | symlink to `~/fw/tools/server/fw` | -| `~/fw/state/` | per-repo page manifests and locks | -| `~/fw/logs/<name>.log` | output of each push's publish run | -| `~/fw/pylib/` | Python-Markdown 3.7 (README rendering) and Pygments 2.21.0 (syntax highlighting), `pip install --target`; `install.sh` adds both (the landing page needs Markdown) | -| `~/fw/queue/pending/`, `running/` | CI jobs, one JSON file each (see [CI](#ci)); `running/<job>.beat` is the runner's heartbeat, `declined/` the runners that gave a job back | -| `~/fw/runners/<runner>` | one file per runner: its tags, and (by its time) when it last checked in | -| `~/fw/issues/` | bug-report issues: stored reports, `secret`, `imap.json`, locks (see [Issues](#issues)); outside every web root | -| `~/fw/ci/<name>/<sha>/` | CI results uploaded by runners: `pipeline.json` and one log per job | -| `~/apps.fuzzyworld.net/_shots/<app>/` | screenshots of each app, uploaded by runners (`shots.json` lists them) | -| `~/apps.fuzzyworld.net/vscode/` | VS Code in the browser (`vscode-web/install.sh`; not listed on the apps index) | - -Every repo uses the shared hook via `core.hooksPath=~/fw/tools/server/hooks`. `post-receive` starts `fw publish <name>` in the background, so a push returns immediately; pages update about 1–5 seconds later. - -## Page URLs - -| URL | Page | +| Command | What it does | |---|---| -| `/` | index of public repos, newest first | -| `/<name>/` | summary: description, clone command, "Open in VS Code" links, license, CI status, screenshots, recent commits, README | -| `/<name>/tree/<dir>/` | directory listing (default branch) | -| `/<name>/blob/<path>.html` | file view with line numbers and syntax highlighting (Pygments; files ≤ 512 KB); `.md` files rendered | -| `/<name>/history/<path>.html` | commits that changed the file (up to 200; renames not followed) | -| `/<name>/blame/<path>.html` | the commit that last changed each line (text files ≤ 256 KB) | -| `/<name>/raw/<path>` | raw copies of images only (png, jpg, gif, webp, ico, avif, bmp, svg; ≤ 2 MB) | -| `/<name>/log/` | the branch's commits (up to 2000) | -| `/<name>/issues/`, `/<name>/issues/<n>/` | bug-report issues and one issue's page (only when the repo's `fuzzyworld.issues` mode and the issue's state allow it; see [Issues](#issues)) | -| `/<name>/commit/<sha>.html` | one commit with its diff (truncated at 300 KB); a page for every commit on every branch, up to 2000 per repo | -| `/<name>/refs/` | branches and tags | -| `/<name>/usage/` | Claude Code token usage per commit and per session, read from `Usage:` trailers in commit messages (only for repos that have them; see below) | -| `/usage/` | (not a repo name) Claude Code usage summed over the public published repos: the same charts as a repo's Usage tab (output tokens, estimated cost, cached reads, cost per commit, AI-marked commits, per week or month) plus a table with one row per repo, written by `build_index` | -| `/<name>/branch/<branch>/...` | the same tree/blob/history/blame/log pages for another branch; `/` in a branch name is written `~` | -| `/<name>/ci/`, `/<name>/ci/<sha>.html` | CI pipelines (newest 100) and one pipeline with its job logs | -| `/<name>/ci/badge.svg` | status of the latest pipeline on the default branch | -| `/ci/` | CI dashboard: running pipelines (each job's state and time), the queue, the last 50 results, for public repos | -| `/<name>/files.json`, `/<name>/objects/<sha>` | for VS Code in the browser: every file's blob SHA and size, and each file's contents (≤ 1 MB) | -| `/projects/`, `/why-i-left-gitlab/`, `/my-ai-use/`, `/contribute/` | projects, and about the site (`/why/` and `/principles/` redirect to `/why-i-left-gitlab/`; the old `fuzzyworld.net/projects/` redirects to `/projects/`). All are files in `server/content/` (Markdown, or an HTML fragment with a leading `<!-- title: … -->`), rendered by `fw content` into the common page layout. An image in that folder that a page mentions (``) is copied next to the page. A file starting with `_` (the `/how/` draft) is not built. `fw content` also writes `404.html` | - -**Branches:** the default branch, and every branch **not yet merged** into it, get browse pages. A merged branch -adds nothing new to read and would duplicate every page, so the branches page lists it with a link to its last -commit. Every branch and tag can be cloned. Author email addresses are not shown. - -**Usage trailers:** a commit message can carry a Claude Code session's running token tally as git trailers: -`FW-Tokens-In`, `FW-Tokens-Out`, `FW-Tokens-Cached` (exact integers), `FW-Tokens-Method: 2`, one -`FW-Model-Tokens: <model> in=… out=… read=… write5m=… write1h=…` line per model (so a cost can be computed; -requests with a prompt over 100K tokens on a model with a higher tier are filed under `<model>@over100k`; a cache write without a 5m/1h split is counted as 5m), plus -`FW-Session` (the session id, for provenance only) and `FW-Rtk-Saved`/`FW-Rtk-Cmds` (rtk's lifetime savings, not -read by the site). `npm run prepare-usage-for-pr` (same as `fw usage --trailers`) writes them. Commits without `FW-Tokens-Method` are method 1: that jq -count double counted repeated transcript lines (roughly 1.5-2× too high) and missed subagents; they are left as -written, shown with tokens but with the cost unknown. Even older commits have one rounded line, -`Usage: 10.6K in / 101.3K out / 5.1M cached (session tally)`, which is still read. -A commit's cost is the step from the previous tally; a new session starts when a tally (or any model's counter) -falls or the method changes. Dollar figures are estimates: `server/prices.json` holds the list prices per model -(with an `as_of` date) and the page says "estimated API list-price equivalent, not what was paid". The log shows a -bar of output tokens per commit, highlighted at 2× the median or more; each commit page shows its step, estimated -cost and the session tally; `/<name>/usage/` ranks commits, totals sessions (with an estimated cost per session, -"unknown" unless every commit in it has one) and charts, per week (per month for histories over 104 weeks): output -tokens, estimated cost, cached reads, average cost per commit (the tooltip gives n) and AI-marked commits (two lines: all commits and AI-marked ones; in the token and cost charts hatched = commits without usage data or without a known cost, unknown -rather than zero; the AI-marked chart has no hatching). "AI-marked" -means an AI `Co-Authored-By` trailer or a usage trailer; a commit without one is "no AI marker", not "human" -(see Provenance in PLAN.md). `/usage/` rolls the same up over the public published repos (charts start at the first AI-marked commit; dates before -2000 or after tomorrow are ignored), with a table with one row per repo; a repo's cost shows "(n of m commits)" when -only some commits have a known cost. It is rebuilt on every `fw publish`, from a small per-repo summary that -`build_repo` saves in the state dir (`<name>.usage.json`, keyed by the repo's HEAD), so it does not re-run `git log` for -every repo. Like `/ci/` and the other site pages, `/usage/` cannot be a repo name (such repos are skipped). Steps between old-style lines are only known -to within the rounding (about two tenths of K/M/G), so small ones show as ≈0; commits without a trailer show no bar. - -**License:** the summary page names the repo's license, read from a root `LICENSE`/`LICENCE`/`COPYING` file -(MIT, Apache-2.0, GPL, LGPL, AGPL, MPL-2.0, BSD, ISC, Unlicense, CC0), else from `package.json`. - -**Preview locally:** `npm run dev` builds this repo's pages into a temp directory and serves them at -http://localhost:8000/. Pass other working copies after `--` (`npm run dev -- ~/code/explodorama`), and set -`PORT` to change the port. It uses each repo's committed state, and `server/site/*.css` edits show on reload; -a change to `gen.py` needs a restart. - -## Security - -The site never executes anything from a repo: - -- pages are pre-generated HTML with all repo content escaped -- raw copies are written only for image extensions; `.htaccess` also denies `.php`, `.shtml`, `.cgi`, `.pl`, `.py` and `.sh` and disables SSI/CGI (DreamHost runs `.php` files and SSI otherwise; tested 2026-10-26) -- `Content-Security-Policy: script-src 'none'` on every response (this also covers SVG images) -- inside public clone copies, `config`, `description`, `hooks/` and `logs/` return 404 -- `objects/<sha>` (file contents for VS Code) is always sent as `application/octet-stream` with `Content-Disposition: attachment`, so a repo's `.html` or `.svg` file can never render as a page; `files.json` and `objects/` allow cross-origin reads, because VS Code runs on apps.fuzzyworld.net -- CI logs are escaped like everything else; CI pages exist only for public repos -- apps.fuzzyworld.net gets the same no-execution rules. HTML, service workers and manifests are sent with `no-cache` (a deploy is seen immediately); `/<app>/assets/` (hashed by vite) is cached for a year. - -## Scheduled jobs - -Jobs live in `server/jobs/`. `install.sh` writes `server/jobs/crontab` into the DreamHost crontab, between `# fw:begin` and `# fw:end`; it replaces only that block. Cron wakes each job hourly. DreamHost's clock is Pacific time, so **each job checks its own UTC schedule**, and daylight saving can't shift it. - -| Job | Schedule | What it does | Log | -|---|---|---|---| -| `north-star-cleanup.sh` | Tuesdays 06:00 UTC (was the GitLab schedule) | updates `~/fw/jobs/north-star` to `main`, then runs `scripts/clean-old-firebase.sh --days=30` as a preview and then `--apply` | `~/fw/logs/cron-north-star-cleanup.log` | - -| `issues-ingest.sh` | hourly | `issues.py ingest`: turns new mail in the bugs mailbox into issues and republishes the touched repos (quiet unless something happened) | `~/fw/logs/cron-issues-ingest.log` | - -Run one by hand: `ssh dh '~/fw/tools/server/jobs/north-star-cleanup.sh --now --preview-only'` (or without `--preview-only` for the real thing). - -Secrets are in `~/.config/<project>.env` on DreamHost (mode 600, never in a repo or the web root): `north-star.env` holds `FIREBASE_TOKEN`. - -**Node on DreamHost:** the system Node is 18. Jobs use Node 22 from `~/fw/node` (the official linux-x64 build; checksum-verified), always with `NODE_OPTIONS=--jitless`. DreamHost's grsec kernel refuses executable memory, so V8's JIT crashes without it. npm installs use `--ignore-scripts`, because native addons can't be built there. `jq` 1.7.1 (the official static build) is in `~/bin`. - -## CI - -The same pattern as GitLab runners: a runner pulls jobs, runs them in containers, and reports back. There is no -coordinator service; the coordinator is files on DreamHost. - -1. **Queue.** `post-receive` queues a job (`~/fw/queue/pending/<id>.json`) for each pushed branch or tag whose - commit has a `.fw-ci.yml` (or `.gitlab-ci.yml`, see below) or `package.json` (unless `fuzzyworld.ci` is `off`). `fw ci <name>` queues one by hand. -2. **Claim.** A runner (`runner/fw-runner.py`; on TrueNAS from cron, on a laptop from launchd, every 2 minutes) runs - `ssh dh claim <runner> <tags>`. The server moves the job into `running/` with one `rename`, which is atomic, so two - runners never take the same job. While it works, the runner sends a heartbeat every 30 seconds - (`beat <job> <runner>`); a job whose runner has been quiet for about 2 minutes (lid closed, network gone) goes - back to `pending/` (a runner that lost its job this way can't mark it done). (A runner that sends no heartbeats, i.e. an old one, is given 2 hours.) With no runner at - all the jobs simply stay in `pending/`, and the CI dashboard says "Waiting for a runner". -2b. **Tags.** A runner has tags (`FW_RUNNER_TAGS`, default `docker,chrome`); a job can ask for more with GitLab's - `tags:`. After cloning, a runner that lacks a tag the pipeline needs gives the job back (`release`) before running - anything; the server then never offers that job to that runner again. Screenshots need `chrome`. The tag - `secrets` means "this repo's secrets": a job tagged `secrets` on the default branch or a tag runs only on a runner - that has `secrets/<repo>.env`. A job nobody can run waits in the queue. -3. **The runner's key** is limited in `authorized_keys` to `server/runner-api` - (`restrict,command="…/runner-api"`): it can claim jobs, fetch repos, and upload results; it can't get a shell. -4. **Run.** The runner clones the commit and runs each job in a fresh Docker container (`--cpus 4 --memory 4g`, - 30-minute timeout unless the job sets `timeout`). All jobs of a pipeline share one workspace, so artifacts are - already there for later stages. Inside jobs, `origin` is the read-only https clone URL - (`https://code.fuzzyworld.net/<name>.git`), as on GitLab; the runner's ssh key never enters a container. -5. **Report.** Logs and `pipeline.json` go to `~/fw/ci/<name>/<sha>/`. While a job runs, the runner marks it - running and uploads its log every 15 seconds; each upload rebuilds that repo's CI pages and the dashboard - (`fw ci-pages`). When the pipeline finishes, the repo's pages are republished, showing ✓/✗ on the log, - the summary and `/<name>/ci/`. - -**Watching CI:** `code.fuzzyworld.net/ci/` shows what is running (each job ✓/✗/● with its time), what is queued -and the last 50 results. While anything is queued or running, that page and the running pipeline's page reload -themselves every 15 seconds (`<meta http-equiv="refresh">`; no JavaScript). A thin bar along the top empties as the reload nears, and "Next reload in N s." -counts down beside the reload note, both done in CSS (the site allows no scripts; the countdown needs a browser with -`@property`, otherwise only the bar shows; the bar is hidden for `prefers-reduced-motion`). Private repos are never named there, -only counted. In the terminal, `fw ci-watch` shows everything, private repos included. - -**Job definitions: `.fw-ci.yml`**, at the repo root, in a subset of GitLab CI's YAML format, so an old -`.gitlab-ci.yml` works after a rename (`git mv .gitlab-ci.yml .fw-ci.yml`). Until a repo is renamed, the runner -still reads `.gitlab-ci.yml`; `.fw-ci.yml` wins if both exist. Supported: `stages`, `variables`, `image` (string or `name:`), -`default:` (image, before_script, after_script), `workflow:rules`, `extends` (hidden jobs), and per job `stage`, -`image`, `script`, `before_script`, `after_script`, `variables`, `rules` (`if`, `when`, `exists`; `changes` always -matches), `only`/`except` (refs), `when` (`on_success`, `on_failure`, `always`, `manual`, `never`), -`allow_failure`, `timeout`, `needs` (ordering comes from stages), `tags` (see Tags above). Ignored, because they change nothing here: -`artifacts`, `cache`, `dependencies`, `environment`, `interruptible`, `retry`, `coverage`, `resource_group`. -**Anything else** (e.g. `include`, `services`, `release`, `trigger`, `parallel`) makes that job fail with a message -naming the key; nothing is silently skipped. `manual` jobs are listed but not run. The usual `CI_*` variables are -set (`CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`/`CI_COMMIT_TAG`, `CI_COMMIT_REF_SLUG`, `CI_DEFAULT_BRANCH`, -`CI_PIPELINE_SOURCE=push`, ...), plus `FUZZYWORLD_CI=true`. Jobs without an image use `node:22-bookworm`. A `pages` -job runs its build as a check; publishing is `fw deploy`'s job. - -**Without a pipeline file:** an app (`package.json` with `fuzzyworld.app`) gets one job, `fw deploy --here --dry-run` -(install, prebuild, checks, build, output checks); other node repos get their `typecheck`/`lint`/`test` scripts. - -**Secrets:** `<runner dir>/secrets/<repo>.env` on each runner that should deploy (TrueNAS: `/mnt/main-2025/fw-runner/secrets/`; -laptop: `~/.fw-runner/secrets/`), passed (`--env-file`) only to jobs on the default branch and tags. A runner without -the file still runs that repo's other jobs; tag a deploy job `secrets` to keep it for a runner that has the file. -Without any, jobs that deploy elsewhere (e.g. `dreamhost`'s rsync, engye's store uploads) fail. Anything on the laptop -is as safe as the laptop (FileVault on); if it is lost, rotate the secrets. - -**The laptop as a runner:** `runner/install-laptop.sh` (`bin/deploy-tools laptop-runner`) installs a launchd job (every 2 minutes) -with `FW_RUNNER_POLITE=1`, so the runner claims a job only when the Mac is on AC power, at least `FW_RUNNER_MIN_IDLE` -(50) percent of the CPU is idle, and Docker is running; it checks before each job, never during one, and its containers -get a low CPU share (`FW_RUNNER_CPUS`, default 2 there, is the real limit). Settings live in `~/.fw-runner/runner.env`, -the log in `~/.fw-runner/runner.log`. `--authorize` allows its key on DreamHost; `--remove` stops it. It assumes the -laptop is on for a few hours a day, which is enough when TrueNAS isn't there. - -Check what a checkout would run, without running it: `runner/fw-runner.py --plan <dir> [refs/heads/<branch>]`. - -### Screenshots - -After a successful pipeline on the default branch, an app (`fuzzyworld.app` in `package.json`) is built the way -`fw deploy` builds it, served at `/<app>/` inside a Playwright container with no network, and photographed. The -pictures go to `apps.fuzzyworld.net/_shots/<app>/` and appear on the apps index and the repo's summary page. +| `fw deploy [--ref B] [--as NAME] [--here] [--dry-run]` | build and publish this repo's app ([docs/apps.md](docs/apps.md)) | +| `fw deploys [--json]` | what is deployed where: the deploy ledger and each deployed app against its repo's default branch. **Known gap: `bin/fw` does not forward it yet (see [Known gaps](#known-gaps)); until then run `ssh dh '~/bin/fw deploys'`.** | -```jsonc -"fuzzyworld": { - "screenshots": [{ "path": "", "size": "phone", "wait": 1500 }, { "path": "?demo", "size": "desktop" }], - // sizes: phone 390×844, tablet 820×1180, desktop 1280×800. Default: one phone shot of the start page. - // or a command that writes its own PNGs to $FW_SHOTS_DIR, for apps that need clicks first: - // "screenshots": "pnpm shots", - // or false: no screenshots -} -``` - -Without runner screenshots, the images in a public repo's `screenshots/` folder are shown instead. - -## Issues +### CI -Bug reports arrive by email at `bugs-k7q4+<repo>@fuzzyworld.net` (the address is on every repo's summary page; the repo name is part of it, and a repo that does not exist under `~/git` sends the mail to the bad folder). `server/issues.py` (`fw issues ...`, Python standard library only) turns each report into an issue and, when allowed, a static page. It reuses the comments plumbing in `comments.py` (IMAP or Maildir source, `Authentication-Results` check, locks, atomic JSON files). - -**Modes.** `fw issues <repo> public|reviewed|private` sets `git config fuzzyworld.issues` in the bare repo and republishes. Anything other than exactly `public` or `reviewed` is `private`, and a repo that is itself private (`fuzzyworld.visibility`) never shows issues, whatever its mode. - -| Mode | An issue's page appears | +| Command | What it does | |---|---| -| `private` | never; reports are stored and listed by `fw issues list` only | -| `reviewed` | after `fw issues approve <repo> <n>` | -| `public` | as soon as its first message passes email authentication (unauthenticated mail is stored with `authenticated: false` and waits for `approve`) | +| `fw ci <name> [branch]` | queue a CI run of a branch's latest commit (default: the default branch) | +| `fw ci-list` | queued and running CI jobs | +| `fw ci-watch` | follow CI in the terminal (private repos too): running jobs with each job's state, the queue, the last 5 results; ends when nothing is queued or running | -In `public` and `reviewed` mode each sender may send 5 messages per 24 hours (the rest goes to the bad folder). +### Bug reports and comments -`fw issues hide <repo> <n>` takes an issue off the site in any mode and keeps it off until `approve`. `close` / `reopen` change the state shown on the page. A reply (its `In-Reply-To` or `References` matches an earlier message of an issue of the same repo) is added to that issue. `approve` marks the messages the issue has at that moment; a message that arrives later is shown only in `public` mode and only if its sender passed authentication, or after another `approve`. An unauthenticated reply is never shown automatically. A mail that repeats a stored Message-ID or a stored byte-for-byte copy is ignored. +| Command | What it does | +|---|---| +| `fw issues <repo> public\|reviewed\|private` | set who sees the repo's bug-report pages (default private) and republish them | +| `fw issues list\|show\|record\|approve\|hide\|close\|reopen\|publish\|publish-all\|ingest\|imap-check` | moderate bug reports ([docs/issues.md](docs/issues.md)) | +| `fw comments list\|show\|approve\|reject\|remove\|untrust\|publish\|export\|ingest\|imap-check` | moderate blog comments ([docs/blog.md](docs/blog.md#comments-by-email)) | -**Pages.** `/<repo>/issues/` lists the visible issues, `/<repo>/issues/<n>/` shows one. Every publish rebuilds the whole `issues/` folder, so hiding removes the page; when the folder appears or disappears the summary page is rebuilt so its "Issues" link follows. Each issue page has an open/closed badge (also in the list), a small details table (version, rules, commit, seed, level, turns, lives, status, modes) filled from the game's own report lines when they match exactly, the messages in wrapping text boxes (a line of three or more dashes becomes a thin rule between boxes), and links to the previous and next visible issue. Senders are labelled "Reporter A", "Reporter B", ... by first appearance within that one issue (the stored author stays "Reporter"; hashes are never written). **Privacy rules:** no address, name, Message-ID or sender hash is written to a page; every address-shaped string in the title and text is replaced by `[address hidden]`; lines starting `Device:` or `Run record:` are left out, and so are the lines that wrap a long `Device:` value (following lines are dropped while they are indented or look like a user agent: `token/version`, parentheses or semicolons; a blank line, a template label such as `Seed ...`, or plain prose ends the block, so the reporter's own words are kept); format=flowed text is re-joined first (an empty line always ends a paragraph) and the text is NFKC-normalized before any filtering (so fullwidth `Ｄevice:` or `＠` count); the text is cut at the start of a quoted reply so quoted names never publish: an attribution line in any language (a colon line with an address, or an `On`/`Am`/`Le`/`El`/`Op`... line with a date that says wrote/schrieb/a écrit/schreef/escribió; any such line directly followed by a quoted `>` line), a wrapped attribution (up to three lines ending in a colon, directly followed by a quoted line, or ending in a separate `wrote:` line), a `-----Original Message-----` line, or a block of at least three `Label: value` lines that is a mail header (`From:`/`Von:`/`De :`..., with an address or three header labels; an Outlook `____` line before it is dropped); a colon line followed only by quoted lines at the end of the text is cut too, but never when it would leave nothing, and prose such as `I pressed <Enter> and wrote:` stays; `delsp=yes` is honoured when re-joining flowed text; all text is HTML-escaped inside `<pre class="issue-text">` and no page has a script. +### Laptop-side tools -**Run records.** If the text part holds a line `Run record: {json}` (the Explodorama bug-report mail does), it is read from the full decoded part (the text shown is cut to 4000 characters, the record is not) and checked: at most 64 KB, valid JSON, nesting at most 32 deep, an object with integer `seed` and `simVersion` and a list `actions`, finite numbers only. Only the keys `simVersion`, `seed`, `startLevel`, `startCharge`, `options` and `actions` are kept (all others are dropped), and a record with an address-like string anywhere in it is refused. A run record split over several lines by a mail client is not recognised. The first line that passes wins, so the mail's later "too long for an email link" footer is ignored; if the decoded text has none, the raw mail is decoded as quoted-printable and searched again. The record is stored re-serialized, never as mail text, and only the first record of an issue is kept (a record in an unauthenticated reply is ignored; every refusal is logged in `rejected.log`). A record is as public as the message that carried it: one in a reply is published only once that reply is shown (after `approve` in `reviewed` mode; at once in `public` mode for an authenticated sender). It is published as `/<repo>/issues/<n>/record.json` at the same moment as the issue page (`approve` publishes both in `reviewed` mode; `hide` removes both). `server/site/htaccess` lets `https://apps.fuzzyworld.net` read it (CORS) and sends `no-cache`. +| Command | What it does | +|---|---| +| `fw admin [--port N] [--no-open] [--code-dir DIR]` | the local admin web page ([docs/admin.md](docs/admin.md)) | +| `fw blog list\|build\|preview\|deploy <config>` | the static blog ([docs/blog.md](docs/blog.md)) | +| `fw blog watch <config> [--once] [--dry-run]`, `fw blog watch install\|uninstall <config>` | mirror the blog source into git and auto-deploy safe edits (one machine only) | +| `fw blog import-wp <config> --wp URL`, `fw blog check <config>` | one-time WordPress import / compare the built site with the WordPress text | +| `fw usage [--all\|--now\|--trailers [--session ID]]` | Claude Code token use per session; `--trailers` prints commit-message trailers ([docs/usage.md](docs/usage.md)) | -**Replay link.** If a deployed app of the repo (`apps.fuzzyworld.net/<app>/.fuzzyworld.json` with `repo` equal to the repo and an `index.html`) has a `replay` template (the `"fuzzyworld"` block of its `package.json`, e.g. `"replay": "replay.html?report={record}"`), a visible issue with a record shows "Watch what happened": a plain link to `https://apps.fuzzyworld.net/<app>/<replay>` with `{record}` replaced by the URL-encoded address of `record.json`. The app named like the repo is preferred, so a `-next` copy is not chosen. It is a link, not an iframe (the site's CSP blocks framing). No `replay` key, no link. +`npm run` scripts in this repo: `dev` (preview the pages locally), `deploy` and `deploy-status` (deploy this repo's own +server by default; `-- all` or a target name adds the runner and VS Code web; [docs/operations.md](docs/operations.md#install-and-update)), `list-repos` +(`fw list`) and `prepare-usage-for-pr` (`fw usage --trailers`). -**Machine-readable commands** (used by [`fw admin`](#admin-panel); the text of unpublished reports IS included, it is a private tool; sender hashes, `mkey` and `mid` never are): `fw issues list [repo] --json` prints every stored issue (`repo`, `n`, `title`, `created`, `updated`, `state`, `approved`, `hidden`, `public`, `record`, `messages`, `shown`, `waiting`) and each repo's `mode` and `public_repo`; `waiting` means a public or reviewed repo's issue that is not hidden and has messages not yet shown. `fw issues show <repo> <n> --json` prints one issue with the full text of every message (label, date, authenticated, approved, shown), the details table rows, prev/next numbers, the public page URL (only when public) and the replay URL (only when the record is shown). `fw issues record <repo> <n>` prints `record.json` as stored, even for an unpublished issue (error `no record for repo#n` when there is none). Plain `list` and `show` are unchanged. +## Visibility -**State** is in `~/fw/issues/` (override with `FW_ISSUES_DIR`), outside every web root: `repos/<repo>/<n>/issue.json` and `record.json`, `secret` (salt for sender hashes; mode 600), `msgids.json` (threading), `keys.json` (a mail is never stored twice), `rejected.log`, the locks. Mail source: `~/fw/issues/imap.json` (mode 600, the same format as the comments one; finished mail goes to `issues-done` / `issues-bad`) or, without it, the Maildir `~/Maildir/.bugs` (`.bugs.done`, `.bugs.bad`). **The bugs mailbox needs its own account or its own `folder`** in `imap.json`: ingest and `imap-check` refuse to run when it has the same host, user and folder as the comments `imap.json` (the two jobs would move each other's mail to their bad folders). Ingest also refuses to run until `authserv_id` is set in `server/comments/sites.json`. The hourly job is `issues-ingest.sh`. +Each bare repo stores its own setting: `git config fuzzyworld.visibility public|private`. +**Anything other than exactly `public` is private**, including a missing value, so a repo is never published by +accident. A private repo has no pages, no clone URL and no index entry, and pushing to it publishes nothing. -## Admin panel +Other settings in the repo config: -`fw admin [--port N] [--no-open] [--code-dir DIR]` (`lib/admin.py`, Python standard library only, runs on the laptop) starts a small web page on `127.0.0.1` for reports (bug-report issues) and blog comments, prints a one-time login URL and opens it in the browser. The pages are server-rendered HTML with plain `POST` forms and **no JavaScript**. Every button runs the matching `~/bin/fw ...` command on the server over ssh, exactly what you would type (`FW_HOST`, default `dh`); the panel keeps no state and never shows a password. It is local-only on purpose: it is never hosted on DreamHost. +| Key | Meaning | +|---|---| +| `fuzzyworld.issues` | `public`, `reviewed`, or anything else = private: who sees bug-report pages ([docs/issues.md](docs/issues.md)) | +| `fuzzyworld.description` | the one-line description | +| `fuzzyworld.ci` | `off`: pushes queue no CI jobs | +| `fuzzyworld.gitlab-id`, `fuzzyworld.gitlab-path` | for repos migrated from GitLab | -**Security model.** The server binds to `127.0.0.1` only. The login URL carries a random key that works once; it sets a session cookie named `fwadmin-<port>` (`HttpOnly`, `SameSite=Strict`), and every other request without that cookie gets 403. The `Host` header must be exactly `127.0.0.1:<port>` (DNS rebinding), else 421. Every `POST` also needs a random per-launch token in its form (a separate secret from the cookie), an `Origin` (if sent) equal to the page's own, and `Sec-Fetch-Site` (if sent) `same-origin` or `none`; otherwise 403 and no command runs. Responses carry a strict `Content-Security-Policy` (no scripts, forms only to itself, no framing), `nosniff`, no referrer and no caching. Mail text never reaches a command line: only a validated repo name, issue number, comment id and a fixed set of action words, each checked again before ssh is run. Hide, reject, untrust, remove and every mode change ask for confirmation first. Query strings are never logged. +## Repository layout -**Pages.** -- **Reports** (`/reports`): one table across repos (repo, number, title, state, public, mode, record, messages shown/total, age) with the filters Waiting (default when anything waits), Has replay and All. -- **A report** (`/reports/<repo>/<n>`): the details table and every message in full, **including reports that are not published**, with who sent it (Reporter A, B, ...), whether it passed authentication and whether it is shown; buttons Approve, Hide, Close / Reopen; radio buttons for the repo's mode (public, reviewed, private) with Set; previous / next links; a link to the public page when there is one. -- **Watch** (on a report with a run record): a link to the deployed replay viewer when the record is public; always "Download record" (`issues record`); and, when `<code dir>/<repo>/package.json` has a `report-replay` script, "Prepare dev replay": it saves the record to a fresh temporary folder and runs `npm run --silent report-replay -- <record.json>` in `<code dir>/<repo>` (with `--rules <commit>` when the report's details have a commit), 60 s at most. The output is shown, with the `http://localhost:5173/?replay=...` address as a link; it works only while `npm run dev` runs in that repo. The record folder is kept (the dev server needs it): one `fw-admin-<repo>-<n>-...` folder per replay under the system temp dir, mode 700; delete them when you like. -- **Comments** (`/comments`): pending comments in full with the post link, author, date and source, and Approve, Approve + trust (email comments only), Reject and Untrust (trusted senders only); below, "Recently approved" (the 20 newest) with a Remove button each. +| Folder | Contents | +|---|---| +| `bin/` | `fw` (laptop side) and `deploy-tools` | +| `lib/` | laptop-side programs: `deploy.mjs`, `admin.py`, `usage.py`, `wp_import.py`, `repoint.sh` | +| `server/` | everything installed on DreamHost: `fw`, `gen.py` (page generator), `issues.py`, `comments.py`, `blog.py`, `blog_watch.py`, `deploys.py`, `runner-api`, hooks, jobs, site templates | +| `runner/` | the CI runner (`fw-runner.py`), its installers and the screenshot script | +| `vite/` | the vite plugin for apps (`fuzzyworld-tools/vite`) | +| `vscode-web/` | VS Code in the browser: installer and the `fw-fs` extension | +| `tests/` | the test suite | -The server handles one request at a time (an idle connection cannot stall it). After each action the page reloads with the command's output (or its error) at the top (these messages are server-wide, not per browser tab). `FW_CODE_DIR` (or `--code-dir`, default `~/code`) is the folder holding the apps' working copies. Needs the server-side `fw` to know `issues list --json`, `issues show --json`, `issues record` and `comments list --json` (deploy the server scripts first). +Where things live on DreamHost: [docs/operations.md](docs/operations.md#layout-on-dreamhost). -## Blog +## Security in one paragraph -A generic static-blog generator: a folder of Markdown files and one JSON config become a site (`server/blog.py`). -It runs on the **laptop**, builds into a clean temporary folder and uploads only that folder with `rsync`, so -nothing in the source folder (drafts, notes, config) is ever published. It is used for `writing.fuzzyworld.net`. +The site never executes anything from a repo: pages are pre-generated HTML with all repo content escaped, only +image files are copied raw, scripts are denied by `.htaccess`, and the code site sends +`Content-Security-Policy: script-src 'none'`. The full list is in [docs/code-site.md](docs/code-site.md#security). -``` -fw blog list ~/Blog/fw-blog.json # every source file: URL, title, date, tags, published or not -fw blog build ~/Blog/fw-blog.json --out DIR [--no-htaccess] # write the site into DIR (other options are refused) -fw blog preview ~/Blog/fw-blog.json [port] # build and serve it locally (no redirects, no feed content type) -fw blog deploy ~/Blog/fw-blog.json --dry-run # then, for real: --yes (or answer the prompt) -fw blog watch ~/Blog/fw-blog.json [--once] [--dry-run] # mirror to git and auto-deploy edits (below) -fw blog watch install|uninstall ~/Blog/fw-blog.json [--print] # the launchd agent that runs it -``` +## Documentation -- **What is published:** only posts with `published: true` (front matter in the file, or the config's `posts` table keyed - by file name; front matter wins over the table). Pages (files in `pages_dir`) are opt-in the same way: a page needs - `published: true` in its front matter (the importer writes it). Text under `## Spelling` and - `## Removed writing` and every HTML comment (except `<!--more-->`) never reach the output. The build stops (rather - than risk a leak) when a heading that looks private is written another way (`##Spelling`, an underlined heading) or a - code fence is never closed. -- **Safety checks:** slugs (posts, pages, tags, the feed path) may use only letters, digits and `. _ % -`; values that - go into `.htaccess` (`redirects`, `wp_ids`, `home`, `feed.path`) may not contain spaces or control characters, even - percent-encoded; `deploy.dir` must be a site folder such as `writing.fuzzyworld.net` (relative to the login folder; - `~`, `.`, `/`, `..` are refused); symbolic links in the media folder are skipped. -- **Config:** `title`, `base_url` and `source` are required; unknown keys are an error. Others: `tagline`, `author`, - `copyright` (`{years}` and `{author}` are filled in), `body_section`, `line_breaks`, `url`, `home`, `pages_dir`, `nav`, `tags`, `feed`, - `reading_wpm`, `media`, `rewrite_hosts`, `redirects`, `wp_ids`, `comments`, `timezone`, `deploy`, `posts`, `site`, - `mirror`, `autodeploy` (see "Watcher" below); see the - top of `server/blog.py`. `source` and `comments` are relative to the config file's folder; `pages_dir` and `media` - are relative to `source`. `site` puts the site-wide bar above the blog's own bar (as on code and apps): - `{"name": "fuzzyworld", "url": "https://fuzzyworld.net/", "nav": [[label, url], ...], "current": "Writing"}`; - `"nav": "gen"` takes the fuzzyworld nav from `NAV` in `server/gen.py`, the one source, and also shows the shared - four-column footer (`gen.site_footer()`, with the blog's copyright and Feed line under it). The landing page's nav and - footer are built from the same functions by `gen.py landing` in `install.sh` (`<!--SITE-NAV-->`, `<!--SITE-FOOTER-->`). - The landing page's text is Markdown in `server/site/landing.md`, rendered into the `<!--LANDING-TEXT-->` spot of - `server/site/index.html` by the same command. -- **Output:** posts at `/YYYY/MM/DD/<slug>/`, `/blog/` (every post by year and month, with excerpts and a "Continue - reading" link), `/tag/<slug>/`, pages, an Atom feed at `/feed/` (`server/atom.py`), `.htaccess` (redirects from - `redirects` and `wp_ids`, no script execution, a security policy that allows only the site's own `blog.js`), - `robots.txt` that blocks AI crawlers, comments as static text. -- **Reader features** (`server/site-blog/blog.js`): a light/dark switch (the choice is kept in the browser; without one the - system setting decides) and a reading-progress bar on posts. -- **Comments by email:** set `"comments_site": "writing"` (or `"til"`, a key of `server/comments/sites.json`) and - `"comments_mailbox": "<address>"` in the config: each post then gets a "Comment by email" link (subject - `Comment [writing:/2024/05/01/slug/]`) and a container that `blog.js` fills from `/_comments/<post path>.json`. - Comments are moderated on the server with `fw comments list|show|approve [--trust]|reject|remove|untrust|publish|export|ingest|imap-check` - (`list --json` prints the pending comments in full plus the 20 newest approved ones, with each post's URL and whether the sender is trusted, never a sender hash; `fw admin` uses it); - a cron job runs `ingest` hourly. `ingest` reads the comments mailbox: a hosted mailbox over IMAP when `~/fw/comments/imap.json` - exists (`{"host", "user", "password", "folder"}`, mode 600, created by hand on the server; TLS with certificate - check; processed mail moves to the mailbox folders `comments-done` and `comments-bad`, with the server's folder prefix such as `INBOX.` added when it needs one; `done_folder` and `bad_folder` in the file override the names), otherwise a Maildir folder - `~/Maildir/.comments`. `fw comments imap-check` prints only header names and the Authentication-Results result tokens, to - learn the right `authserv_id`. `ingest` accepts a message only when the topmost - `Authentication-Results` header, written by the receiving mail server (its name must be set as `authserv_id` in - `server/comments/sites.json`; empty means ingest refuses to read anything; set it in the repo copy, because an install - overwrites the server's), shows a DMARC pass for the From domain (or an aligned DKIM pass). That trust holds only if the - receiving server always adds its own header above anything the sender supplied. The address is never shown, only a - salted hash is kept (losing `~/fw/comments/secret` invalidates trusted senders). Limits: 4000 characters, 5 comments - per sender per 24 hours, 200 pending. State lives in `~/fw/comments/`, outside every web root; `fw blog deploy` - leaves `/_comments/` alone. `comments_form` (an `https` action) adds a plain HTML form for the optional comment - endpoint `server/comments/post.php`; it is off, and the endpoint is written (its tests pass on the server's PHP; an optional `self_origin` key in `sites.json` names the origin its no-JavaScript confirm page posts from) but is not installed (see - [Hosting tiers](#hosting-tiers)). -- **Upload safety:** `deploy` audits the built folder first and refuses anything that is not a generated file or an - allowlisted media type (`png jpg jpeg gif webp svg pdf epub kpf mp3 mp4 zip`), then rsyncs only that folder (it - keeps `.well-known`; the watcher limits it to 600 s, a manual deploy has no time limit). It asks for confirmation on every real upload (it holds the watcher's lock only for the upload itself, not while the question is open; ctrl-c at the question exits quietly with 1); `--dry-run` shows what would be sent. -- **Watcher** (`server/blog_watch.py`, `fw blog watch <config>`): for ONE blog config, one tick every 60 s (the launchd - agent runs `fw blog watch <config> --once`; without `--once` it loops; `--dry-run` shows what a tick would do now and - writes nothing). **Run it on ONE machine only**: the MacBook Air just keeps writing through Dropbox, and its edits are - captured by the mirror on the watching machine. A manual `fw blog deploy` on another machine does not update this - machine's baseline; the result is a harmless re-deploy or a hold. Config keys (top level only): - `"autodeploy": {"quiet_seconds": 300}` (whole number, 60 or more; turns the deploy guard on) and - `"mirror": {"repo": "~/code/blog-mirror", "branch": "dropbox-mirror", "remote": "origin", "exclude": ["*.tmp"]}` - (`repo` required, `~` expanded, a relative path is relative to the config's folder; `branch` defaults to - `dropbox-mirror` and may not be `main`, `master` or `HEAD`; no `remote` means no push; `exclude` adds rsync patterns, - no `..`, none starting with `! + - :`). Each works without the other. The quiet period is `autodeploy.quiet_seconds`; - without an `autodeploy` key it is a fixed 300 s, also for the mirror. State is in - `~/.fw-blog-watch/<config folder, lowercased>-<8 hex of the config path's hash>/` (`FW_BLOG_WATCH_DIR` replaces the - whole path; `FW_BLOG_WATCH_NOTIFY=0` turns the macOS notifications off, and the tests set it): `state.json`, `watch.log` (one line per decision), `deployed.json`, `hashes.json` (a size+mtime cache so - media are not read every minute), `lock`, `launchd.log`. A tick: take a non-blocking `flock` on `lock` (held: exit - quietly; held for over 30 minutes: one "watch tick stuck" notification); fingerprint the source (sha256 of every - file's path and content; `.DS_Store`, `.dropbox*`, `.git` and Dropbox "conflicted copy" files are ignored; links are not - followed); a changed fingerprint restarts the timer; nothing happens until the folder has been quiet for the quiet - period. **Only `source` is fingerprinted**: editing the config or the comments file does not start a tick (the next - source change, or a manual `fw blog deploy`, picks it up). A "conflicted copy" file blocks everything (notification, - retried each tick). If the source changes while the mirror or the build runs, the tick stops ("changed during build") - and the new quiet period applies. **Mirror:** the repo must be a git repo with the mirror branch checked out, no merge - or rebase in progress, and a clean tree where **git-ignored files count as changes** (so `rsync --delete` can never - remove them); otherwise it notifies and stops the tick (retried; it never overwrites). Only an absent or empty folder - is auto-initialised (`git init -b <branch>`). Then `rsync -a --delete` source into the repo (your excludes, then the - protected `.git`), `git add -A`, and a commit `Mirror of <source folder> at <UTC time>` if anything changed (your git - identity; no trailers; hooks skipped; other branches are never touched). With `remote`, it pushes - `refs/heads/<branch>:refs/heads/<branch>` (never forced); a failed push is notified but the deploy goes on. If the - rsync (exit 23/24 are normal while Dropbox syncs; 24 is retried silently), `git add` or the commit fails halfway, the - tool records exactly what it left behind in `state.json` (`mirror_dirty`: the repo's real path, HEAD, the - `git status --porcelain --ignored` listing and a hash of those files' contents; the same record is made after a - successful run that leaves git-ignored files copied from a source `.gitignore`). A dirty tree is accepted on a later - tick only if the repo, HEAD, the listing and the contents are all still exactly that; the record disappears after a - run that leaves nothing behind. Any other difference (an edited tracked file, a new path) is refused until you look - at it, and your edit is never overwritten. **Deploy guard** (only with `autodeploy`): build into a temporary folder (same code as - `fw blog build`), compare with `deployed.json` (`{files: {path: sha256}, time, generator, config, target}`): no baseline, a changed - **generator hash** (blog.py, atom.py, gen.py, the shared CSS/icon, `site-blog/*`, fixed when the process starts), a - changed **config** (hash of the loaded config), a changed **deploy target**, or any added or removed output file is **held back** with a notification (run `fw blog deploy - <config>` yourself; so code, template and config changes wait for a manual deploy, which rewrites all of these); the same paths with no content difference is logged "no visible change"; the same paths with - some changed content is deployed through the same function as `fw blog deploy` (audit and rsync, no question). A build - or deploy failure notifies once and is **not retried until the next source change**. Notifications (macOS, - `osascript`) are sent for held-back changes, failures, conflict files, mirror problems and unexpected errors, never - for a successful deploy; an unexpected exception is logged and notified once and the tick still exits 0 (so launchd - does not throttle the agent). Every subprocess has a timeout (git 120 s, mirror and deploy rsync 600 s, osascript - 10 s); a timeout is an ordinary failure. A manual real `fw blog deploy` (not `--dry-run`) takes the same lock (if a - tick holds it: "a watch tick is running, try again in a minute"), refuses if the watcher deployed while you were answering the question (run it again), rewrites `deployed.json` (which keeps the - baseline current) and clears the watcher's "handled" state so the next tick re-evaluates. `install` writes `~/Library/LaunchAgents/net.fuzzyworld.fw-blog-watch-<folder>-<hash>.plist` - (every 60 s and at load, low priority) and loads it with `launchctl bootstrap gui/$UID` (retried up to 5 times); - `uninstall` unloads and removes it; `install --print` only prints the plist. `fw blog watch ... --once` never runs - `pip`: if Markdown or Pygments are missing it exits 1; run another `fw blog` command once to install them. -- **WordPress import** (`lib/wp_import.py`, `fw blog import-wp`): reads the public REST API, skips password-protected - posts, matches each post to a source file (similarity of 0.95 or more uses your file; otherwise the WordPress text is - written to `wordpress/<date>-<slug>.md`; a non-ASCII slug gives a readable file name but keeps its URL), and writes - comments (name, date and text only, into the `comments` file next to the config), pages (with `published: true`), - media and the config entries. `--dry-run` writes only a cache and `wordpress/_report.txt`; a second run changes nothing. - `fw blog check` lists posts whose built text differs from WordPress. -- Python-Markdown and Pygments are installed into `~/fw/pylib` on the laptop the first time `fw blog` runs. -- Tests: `python3 -m unittest discover tests` (the blog, the WordPress importer, and the runner with a throwaway-`HOME` run of `server/runner-api`). - -## VS Code in the browser - -`apps.fuzzyworld.net/vscode/?repo=<name>` (and `&branch=<branch>` for an unmerged branch) opens a public repo in -VS Code (add `&file=<path>` to open a file, `&file=<path>:<line>` to jump to a line), linked from each summary page. It is the open-source VS Code web build (the `vscode-web` npm package, -1.91.1, MIT) plus `vscode-web/fw-fs/`, an extension that reads `files.json` and `objects/<sha>` from -code.fuzzyworld.net. Everything is static files; nothing can be changed (clone to edit). Quick open (Ctrl/Cmd+P) -and search across files work; search reads text files up to 256 KB. Not yet: Markdown preview and other webviews -(they need a separate domain per view), and project-wide TypeScript IntelliSense (turned off; syntax colors work). - -The plain "Open in VS Code" link (`vscode://vscode.git/clone?url=…`) clones the repo into the visitor's own VS Code. - -## Install / update +| File | Covers | +|---|---| +| [docs/code-site.md](docs/code-site.md) | page URLs, branches, licences, local preview, security | +| [docs/apps.md](docs/apps.md) | `package.json` keys, `fw deploy`, the vite plugin, the apps index, screenshots | +| [docs/ci.md](docs/ci.md) | queue, runners, `.fw-ci.yml`, secrets, watching CI | +| [docs/issues.md](docs/issues.md) | bug reports by email: modes, privacy rules, run records, replay links | +| [docs/admin.md](docs/admin.md) | `fw admin`: security model and pages | +| [docs/blog.md](docs/blog.md) | `fw blog`, comments by email, the watcher, the WordPress import | +| [docs/usage.md](docs/usage.md) | Claude Code usage trailers, cost estimates, usage charts | +| [docs/operations.md](docs/operations.md) | DreamHost layout, scheduled jobs, installing and updating, VS Code in the browser, requirements | +| [PLAN.md](PLAN.md) | roadmap, decisions, backlog | + +## Tests ```bash -server/install.sh # copy server/ to dh, install site files and hooks -server/install.sh --publish # ...and republish every repo -runner/install.sh # CI runner on TrueNAS: copy it, make its ssh key, add the cron job -runner/install.sh --authorize # ...and allow that key on DreamHost (runner-api only) -runner/install-laptop.sh # the laptop as a polite second runner (launchd, Docker, its own venv) -vscode-web/install.sh # VS Code in the browser at apps.fuzzyworld.net/vscode/ +python3 -m unittest discover tests # the whole suite: pages, blog, comments, issues, runner, admin, deploys +node --test tests/deploy.test.mjs # fw deploy's config, rsync filters and replay-template checks ``` -npm scripts (`npm run` lists them): - -| Script | Runs | -|---|---| -| `npm run deploy` | `bin/deploy-tools`: install the server and republish every repo, but only if `server/` changed since the last deploy | -| `npm run deploy-status` | `bin/deploy-tools status`: what is deployed where, and what is pending | -| `npm run list-repos` | `bin/fw list` | - -`bin/deploy-tools [server|runner|laptop-runner|vscode|all] [--force]` wraps the install scripts (`server/install.sh`, `runner/install.sh`, `runner/install-laptop.sh`, `vscode-web/install.sh`) with a ledger kept on the server, `~/fw/state/deployed.json`. For each target it records the commit and a content hash of the folders that target ships (`server/`, `runner/`, `vscode-web/`). A target is redeployed only when that hash differs, so uncommitted edits count, unrelated commits (README, tests) do not, and committing the files you already deployed does not trigger a second deploy. The ledger is written only after the script succeeds, and any laptop sees the same state, except `laptop-runner`, which is installed per machine and so is recorded per hostname. Gitignored files in those folders are not hashed. `server` includes the republish, since pages are generated by the installed `gen.py`. For flags `deploy-tools` does not pass (`--authorize`, `--remove`, `--out`) run the install script directly; to regenerate pages without installing, run `bin/fw publish-all`. - -`install.sh` copies `server/` from the working copy, not from a commit, so uncommitted changes there go live too. +## Known gaps -Requirements on DreamHost: git, Python 3.12, `flock`, `setsid`. On the laptop: Node ≥ 18, git, rsync, ssh with the `dh` alias. -On TrueNAS (runner): Docker, Python 3 with PyYAML (both part of TrueNAS SCALE), and the `truenas` ssh alias on the laptop. -On the laptop (runner, optional): Docker (Docker Desktop, OrbStack, ...) and Python 3; the installer makes a venv with PyYAML. +- `fw deploys` is documented above but `bin/fw` on the laptop rejects it ("unknown command"). `fw admin` is not affected. +- The npm package does not ship `server/` (`files` in `package.json`), but `fw blog` (including `import-wp` and `check`) needs it, and `fw usage --trailers` reads `server/prices.json` (without it the trailers lose the `@over100k` split). So these work fully only from a checkout of this repo. -Using `fw` outside an app repo (e.g. on a second laptop): `npm install -g "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0"`. +Other problems found are tracked in the global queue, not here. diff --git a/docs/admin.md b/docs/admin.md new file mode 100644 index 0000000..3e3b513 --- /dev/null +++ b/docs/admin.md @@ -0,0 +1,77 @@ +# Admin panel (`fw admin`) + +```bash +fw admin [--port N] [--no-open] [--code-dir DIR] +``` + +`lib/admin.py` (Python standard library only) runs **on the laptop**. It starts a small web page on `127.0.0.1` for +bug reports, blog comments and what is deployed where, prints a one-time login URL and opens it in the browser. + +- The pages are server-rendered HTML with plain `POST` forms and **no JavaScript**. +- Every button runs the matching `~/bin/fw ...` command on the server over ssh, exactly what you would type (`FW_HOST`, default `dh`). +- The panel keeps no state and never shows a password. +- It is local-only on purpose: it is never hosted on DreamHost. + +`FW_CODE_DIR` (or `--code-dir`, default `~/code`) is the folder holding the apps' working copies. + +The server-side `fw` must know `issues list --json`, `issues show --json`, `issues record`, `comments list --json` +and `deploys --json`, so deploy the server scripts first (`npm run deploy`). + +## Security model + +| Threat | Defence | +|---|---| +| Other machines | binds to `127.0.0.1` only | +| Strangers on this machine | the login URL carries a random key that works once and sets a session cookie `fwadmin-<port>` (`HttpOnly`, `SameSite=Strict`); any other request without the cookie gets 403 | +| DNS rebinding | the `Host` header must be exactly `127.0.0.1:<port>`, else 421 | +| Cross-site requests | every `POST` needs a random per-launch token in its form (a separate secret from the cookie), an `Origin` (if sent) equal to the page's own, and `Sec-Fetch-Site` (if sent) `same-origin` or `none`; otherwise 403 and no command runs | +| Script injection | strict `Content-Security-Policy` (no scripts, forms only to itself, no framing), `nosniff`, no referrer, no caching | +| Command injection | mail text never reaches a command line: only a validated repo name, issue number, comment id and a fixed set of action words, each checked again before ssh is run | +| Mistakes | hide, reject, untrust, remove and every mode change ask for confirmation first | +| Leaks | query strings are never logged | + +## Pages + +### Reports (`/reports`) + +One table across repos: repo, number, title, state, public, mode, record, messages shown/total, age. Filters: +**Waiting** (the default when anything waits), **Has replay** and **All**. + +### A report (`/reports/<repo>/<n>`) + +- The details table and every message in full, **including reports that are not published**, with who sent it (Reporter A, B, ...), whether it passed authentication and whether it is shown. +- Buttons: Approve, Hide, Close / Reopen. +- Radio buttons for the repo's mode (public, reviewed, private) with Set. +- Previous / next links, and a link to the public page when there is one. + +### Watch (on a report with a run record) + +- A link to the deployed replay viewer, when the record is public. +- Always "Download record" (`issues record`). +- When `<code dir>/<repo>/package.json` has a `report-replay` script, "Prepare dev replay": + 1. it saves the record to a fresh temporary folder, + 2. runs `npm run --silent report-replay -- <record.json>` in `<code dir>/<repo>` (with `--rules <commit>` when the report's details have a commit), 60 s at most, + 3. shows the output, with the `http://localhost:5173/?replay=...` address as a link. + + It works only while `npm run dev` runs in that repo. The record folder is kept, because the dev server needs it: + one `fw-admin-<repo>-<n>-...` folder per replay under the system temp dir, mode 700. Delete them when you like. + +### Comments (`/comments`) + +Pending comments in full, with the post link, author, date and source. Buttons: Approve, Approve + trust (email +comments only), Reject, and Untrust (trusted senders only). Below, "Recently approved" (the 20 newest) with a Remove +button each. + +### Deploy (`/deploy`, read-only for now) + +Uses `fw deploys --json`. There is no button yet: deploy with `npm run deploy`. + +- **Tools:** per repo that declares deploy targets (in `package.json` under `fuzzyworld.deploy`), each target's state (up to date, pending, never deployed), the commit and time it was deployed from, and the commits on the default branch since that touch its folders; per laptop for `perMachine` targets. +- **Apps:** every deployed app against its repo: state, deployed commit, time, and the commits since. + - Commits that only touch Markdown, CI config, the licence, `.gitignore`, `docs/` or `screenshots/` are ignored, so a repo-wide CI-file rename does not mark every app pending. + - The state is "unknown" when the repo or the commit is not on the server. + +## Behaviour + +The server handles one request at a time (an idle connection cannot stall it). After each action the page reloads +with the command's output (or its error) at the top. These messages are server-wide, not per browser tab. diff --git a/docs/apps.md b/docs/apps.md new file mode 100644 index 0000000..c430560 --- /dev/null +++ b/docs/apps.md @@ -0,0 +1,84 @@ +# Apps (apps.fuzzyworld.net) + +Each app lives at `apps.fuzzyworld.net/<app>/` and is a static build uploaded by `fw deploy`. + +## Making a repo deploy as an app + +Add to the app repo's `package.json`: + +```jsonc +"scripts": { "deploy": "fw deploy" }, +"fuzzyworld": { + "app": "my-game", // URL: apps.fuzzyworld.net/my-game/ (required) + "title": "My Game", // apps index + "description": "One line.", // apps index + "repo": "my-game", // DreamHost repo name, if different from app + "prebuild": ["pnpm gen-assets"], // commands before checks and build + "checks": ["typecheck", "test"], // scripts that must pass (default: whichever of typecheck/lint/test exist) + "build": "make public", // default: `<pm> run build` + "out": "dist", // default: dist, else public, else build + "keep": ["my-game.apk"], // files on the server a deploy must not delete or replace (e.g. a download built elsewhere) + "extra": ["sim/*.js"], // build-output files every later deploy keeps: uploaded and updated, never deleted (e.g. old rules versions) + "replay": "replay.html?report={record}" // a page of this app that plays a bug report's run record (see issues.md) +}, +"devDependencies": { "fuzzyworld-tools": "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0" } +``` + +For vite, add the plugin. It sets `base` to `/<app>/` for builds (the dev server is unchanged). `FW_APP`, set by +`fw deploy --as`, overrides the app name. + +```ts +import fuzzyworld from 'fuzzyworld-tools/vite'; +export default defineConfig({ plugins: [fuzzyworld()] }); +``` + +## `fw deploy` + +Run it with `pnpm run deploy` or `npm run deploy`. + +1. **Clone.** Clones branch `main` of `dh:git/<repo>.git` into a temp dir. It builds what is pushed, never your working copy (`--ref <branch>` for another branch, `--here` to build the current directory as is). +2. **Install** with the lockfile's package manager (`pnpm install --frozen-lockfile`, `npm ci`, `yarn install --frozen-lockfile`). +3. **Prebuild, checks, build.** Any failure stops the deploy. +4. **Check the output.** Every same-site URL in `index.html`, the web manifest's `id`/`scope`/`start_url`, and every `serviceWorker.register(...)` script and scope must stay inside `/<app>/`. That keeps apps on the shared origin from capturing each other, and makes each one install separately on Android. Any problem stops the deploy. +5. **Upload.** `rsync --delete` to `apps.fuzzyworld.net/<app>/`. + - `keep` paths are excluded. + - `extra` files are protected from `--delete` by an rsync filter: uploaded and updated, never removed. + - It writes `.fuzzyworld.json` (commit, time, and the `replay` template; not web-readable). + - It refreshes the apps index and the repo's code page. +6. **Print the URL.** + +`--dry-run` does everything except upload. + +`--as NAME` publishes the app at `apps.fuzzyworld.net/NAME/` instead: a second copy of the same repo (for example +`explodorama-next` built from `main` while `/explodorama/` is deployed from a tag with `--ref`). The build gets +`FW_APP=NAME`, which the vite plugin uses for `base`, and the index title gets a suffix. + +## The apps index + +A table ordered by the newest commit of each app's repo (not by deploy date: an app can be redeployed often +without its code changing, so the deploy date has its own column). + +Columns: screenshot thumbnail (if the runner has taken one), name, description, last commit, deployed, and a source +link for apps from public repos. The code index on code.fuzzyworld.net is likewise a table of public repos. + +Apps hosted elsewhere are listed from `server/site-apps/extra.json` (today: engye on Cloudflare Pages; +`apps.fuzzyworld.net/engye/...` redirects there). + +## Screenshots + +After a successful pipeline on the default branch, an app is built the way `fw deploy` builds it, served at +`/<app>/` inside a Playwright container with no network, and photographed. The pictures go to +`apps.fuzzyworld.net/_shots/<app>/` and appear on the apps index and the repo's summary page. They need a runner +with the `chrome` tag ([ci.md](ci.md#tags)). + +```jsonc +"fuzzyworld": { + "screenshots": [{ "path": "", "size": "phone", "wait": 1500 }, { "path": "?demo", "size": "desktop" }], + // sizes: phone 390×844, tablet 820×1180, desktop 1280×800. Default: one phone shot of the start page. + // or a command that writes its own PNGs to $FW_SHOTS_DIR, for apps that need clicks first: + // "screenshots": "pnpm shots", + // or false: no screenshots +} +``` + +Without runner screenshots, the images in a public repo's `screenshots/` folder are shown instead. diff --git a/docs/blog.md b/docs/blog.md new file mode 100644 index 0000000..375b455 --- /dev/null +++ b/docs/blog.md @@ -0,0 +1,249 @@ +# Blog (`fw blog`) + +A generic static-blog generator: a folder of Markdown files and one JSON config become a site (`server/blog.py`). +It is used for writing.fuzzyworld.net (and til.fuzzyworld.net). + +It runs on the **laptop**, builds into a clean temporary folder and uploads only that folder with `rsync`, so +nothing in the source folder (drafts, notes, config) is ever published. + +```bash +fw blog list ~/Blog/fw-blog.json # every source file: URL, title, date, tags, published or not +fw blog build ~/Blog/fw-blog.json --out DIR [--no-htaccess] # write the site into DIR (other options are refused) +fw blog preview ~/Blog/fw-blog.json [port] # build and serve it locally (no redirects, no feed content type) +fw blog deploy ~/Blog/fw-blog.json --dry-run # then, for real: --yes (or answer the prompt) +fw blog watch ~/Blog/fw-blog.json [--once] [--dry-run] # mirror to git and auto-deploy edits +fw blog watch install|uninstall ~/Blog/fw-blog.json [--print] # the launchd agent that runs it +``` + +Python-Markdown and Pygments are installed into `~/fw/pylib` on the laptop the first time `fw blog` runs. + +## What is published + +- **Posts:** only those with `published: true`, in the file's front matter or in the config's `posts` table keyed by file name (front matter wins). +- **Pages** (files in `pages_dir`) are opt-in the same way: a page needs `published: true` in its front matter (the importer writes it). +- **Never reaches the output:** text under `## Spelling` and `## Removed writing`, and every HTML comment (except `<!--more-->`). +- **The build stops** (rather than risk a leak) when a heading that looks private is written another way (`##Spelling`, an underlined heading) or a code fence is never closed. + +## What it builds + +| Output | Notes | +|---|---| +| Posts at `/YYYY/MM/DD/<slug>/` | | +| `/blog/` | every post by year and month, with excerpts and a "Continue reading" link | +| `/tag/<slug>/` | | +| Pages | | +| Atom feed at `/feed/` | `server/atom.py` | +| `.htaccess` | redirects from `redirects` and `wp_ids`, no script execution, a security policy that allows only the site's own `blog.js` | +| `robots.txt` | blocks AI crawlers | +| Comments | as static text | + +**Reader features** (`server/site-blog/blog.js`): a light/dark switch (the choice is kept in the browser; without +one the system setting decides) and a reading-progress bar on posts. + +## Config + +`title`, `base_url` and `source` are required; unknown keys are an error. The full list is at the top of +`server/blog.py`. + +| Group | Keys | +|---|---| +| Identity | `title`, `tagline`, `author`, `copyright` (`{years}` and `{author}` are filled in), `url`, `base_url` | +| Content | `source`, `include` (source file patterns, default `*.md`), `pages_dir`, `media`, `body_section`, `line_breaks`, `posts`, `tags`, `home`, `nav`, `reading_wpm`, `timezone` | +| Output | `feed`, `redirects`, `wp_ids`, `rewrite_hosts`, `site` | +| Comments | `comments`, `comments_site`, `comments_mailbox`, `comments_form` | +| Deploy | `deploy`, `mirror`, `autodeploy` | + +**Paths:** `source` and `comments` are relative to the config file's folder; `pages_dir` and `media` are relative to `source`. + +**`site`** puts the site-wide bar above the blog's own bar (as on code and apps): +`{"name": "fuzzyworld", "url": "https://fuzzyworld.net/", "nav": [[label, url], ...], "current": "Writing"}`. +`"nav": "gen"` takes the fuzzyworld nav from `NAV` in `server/gen.py` (the one source) and also shows the shared +four-column footer (`gen.site_footer()`, with the blog's copyright and Feed line under it). The landing page's nav +and footer are built from the same functions by `gen.py landing` in `install.sh` (`<!--SITE-NAV-->`, +`<!--SITE-FOOTER-->`). The landing page's text is Markdown in `server/site/landing.md`, rendered into the +`<!--LANDING-TEXT-->` spot of `server/site/index.html`. + +## Safety checks + +| Check | Rule | +|---|---| +| Slugs | posts, pages, tags and the feed path may use only letters, digits and `. _ % -` | +| `.htaccess` values | `redirects`, `wp_ids`, `home` and `feed.path` may not contain spaces or control characters, even percent-encoded | +| `deploy.dir` | must be a site folder such as `writing.fuzzyworld.net` (relative to the login folder); `~`, `.`, `/`, `..` are refused | +| Media | symbolic links in the media folder are skipped | + +## Deploy and upload safety + +`fw blog deploy` audits the built folder first and refuses anything that is not a generated file or an allowlisted +media type (`png jpg jpeg gif webp svg pdf epub kpf mp3 mp4 zip`). It then rsyncs only that folder, and keeps +`.well-known`. + +- It asks for confirmation on every real upload. It holds the watcher's lock only for the upload itself, not while the question is open. Ctrl-C at the question exits quietly with 1. +- `--dry-run` shows what would be sent. +- The watcher limits the upload to 600 s; a manual deploy has no time limit. + +## Comments by email + +Set `"comments_site": "writing"` (or `"til"`; a key of `server/comments/sites.json`) and +`"comments_mailbox": "<address>"` in the config. Each post then gets a "Comment by email" link (subject +`Comment [writing:/2024/05/01/slug/]`) and a container that `blog.js` fills from `/_comments/<post path>.json`. + +### Moderating + +`fw comments list|show|approve [--trust]|reject|remove|untrust|publish|export|ingest|imap-check` runs on the server +(or use the [admin panel](admin.md)). + +- `list --json` prints the pending comments in full plus the 20 newest approved ones, with each post's URL and whether the sender is trusted, never a sender hash. `fw admin` uses it. +- A cron job runs `ingest` hourly ([operations.md](operations.md#scheduled-jobs)). +- `fw blog deploy` leaves `/_comments/` alone. + +### The mailbox + +`ingest` reads the comments mailbox: + +- **Hosted mailbox over IMAP** when `~/fw/comments/imap.json` exists: `{"host", "user", "password", "folder"}`, mode 600, created by hand on the server. TLS with certificate check. Processed mail moves to the folders `comments-done` and `comments-bad` (with the server's folder prefix such as `INBOX.` added when it needs one); `done_folder` and `bad_folder` in the file override the names. +- **Otherwise a Maildir** folder `~/Maildir/.comments`. +- `fw comments imap-check` prints only header names and the Authentication-Results result tokens, to learn the right `authserv_id`. + +### Trust and limits + +- `ingest` accepts a message only when the topmost `Authentication-Results` header shows a DMARC pass for the From domain (or an aligned DKIM pass), and was written by the receiving mail server. That server's name must be set as `authserv_id` in `server/comments/sites.json`. Empty means ingest refuses to read anything. Set it in the repo copy, because an install overwrites the server's. +- That trust holds only if the receiving server always adds its own header above anything the sender supplied. +- The address is never shown. Only a salted hash is kept (losing `~/fw/comments/secret` invalidates trusted senders). +- Limits: 4000 characters, 5 comments per sender per 24 hours, 200 pending. +- State lives in `~/fw/comments/`, outside every web root. + +### The optional form + +`comments_form` (an `https` action) adds a plain HTML form for the endpoint `server/comments/post.php`. It is **off, +and the endpoint is not installed** ([hosting tiers](../README.md#hosting-tiers)). It is written, and its tests pass +on the server's PHP; an optional `self_origin` key in `sites.json` names the origin its no-JavaScript confirm page +posts from. + +## The watcher (`fw blog watch`) + +`server/blog_watch.py` mirrors the blog source into a git repo and auto-deploys edits that change no URLs, for ONE +blog config. **Run it on ONE machine only.** The MacBook Air just keeps writing through Dropbox, and its edits are +captured by the mirror on the watching machine. A manual `fw blog deploy` on another machine does not update this +machine's baseline; the result is a harmless re-deploy or a hold. + +- One tick every 60 s: the launchd agent runs `fw blog watch <config> --once`. Without `--once` it loops. +- `--dry-run` shows what a tick would do now and writes nothing. +- `--once` never runs `pip`: if Markdown or Pygments are missing it exits 1. Run another `fw blog` command once to install them. + +### Config keys (top level only) + +```jsonc +"autodeploy": { "quiet_seconds": 300 }, // whole number, 60 or more; turns the deploy guard on +"mirror": { + "repo": "~/code/blog-mirror", // required; ~ expanded; relative = relative to the config's folder + "branch": "dropbox-mirror", // default; may not be main, master or HEAD + "remote": "origin", // no remote means no push + "exclude": ["*.tmp"] // extra rsync patterns; no `..`, none starting with ! + - : +} +``` + +`autodeploy` and `mirror` each work without the other. The quiet period is `autodeploy.quiet_seconds`; without an +`autodeploy` key it is a fixed 300 s, also for the mirror. + +### State + +In `~/.fw-blog-watch/<config folder, lowercased>-<8 hex of the config path's hash>/` (`FW_BLOG_WATCH_DIR` replaces +the whole path; `FW_BLOG_WATCH_NOTIFY=0` turns the macOS notifications off, and the tests set it). + +| File | Contents | +|---|---| +| `state.json` | | +| `watch.log` | one line per decision | +| `deployed.json` | the baseline: `{files: {path: sha256}, time, generator, config, target}` | +| `hashes.json` | a size+mtime cache so media are not read every minute | +| `lock` | | +| `launchd.log` | | + +### One tick + +1. **Lock.** Take a non-blocking `flock` on `lock`. If it is held, exit quietly; held for over 30 minutes gives one "watch tick stuck" notification. +2. **Fingerprint** the source (sha256 of every file's path and content). `.DS_Store`, `.dropbox*`, `.git` and Dropbox "conflicted copy" files are ignored; links are not followed. +3. **Wait for quiet.** A changed fingerprint restarts the timer; nothing happens until the folder has been quiet for the quiet period. + - **Only `source` is fingerprinted.** Editing the config or the comments file does not start a tick (the next source change, or a manual `fw blog deploy`, picks it up). + - A "conflicted copy" file blocks everything (notification, retried each tick). + - If the source changes while the mirror or the build runs, the tick stops ("changed during build") and the new quiet period applies. +4. **Mirror** (if configured), below. +5. **Deploy guard** (only with `autodeploy`), below. + +### Mirror + +The repo must be a git repo with the mirror branch checked out, no merge or rebase in progress, and a clean tree. +**Git-ignored files count as changes**, so `rsync --delete` can never remove them. Otherwise the tool notifies and +stops the tick (it is retried; it never overwrites). Only an absent or empty folder is auto-initialised +(`git init -b <branch>`). + +Then: + +1. `rsync -a --delete` source into the repo (your excludes, then the protected `.git`); +2. `git add -A`; +3. a commit `Mirror of <source folder> at <UTC time>` if anything changed (your git identity; no trailers; hooks skipped; other branches are never touched); +4. with `remote`, push `refs/heads/<branch>:refs/heads/<branch>` (never forced). A failed push is notified, but the deploy goes on. + +**If a step fails halfway** (rsync exit 23/24 is normal while Dropbox syncs, and 24 is retried silently; `git add` +or the commit can also fail), the tool records exactly what it left behind in `state.json` as `mirror_dirty`: the +repo's real path, HEAD, the `git status --porcelain --ignored` listing, and a hash of those files' contents. The same +record is made after a successful run that leaves git-ignored files copied from a source `.gitignore`. + +- A dirty tree is accepted on a later tick only if the repo, HEAD, the listing and the contents are all still exactly that. +- The record disappears after a run that leaves nothing behind. +- Any other difference (an edited tracked file, a new path) is refused until you look at it, and your edit is never overwritten. + +### Deploy guard + +Builds into a temporary folder (same code as `fw blog build`) and compares with `deployed.json`. + +| Situation | Result | +|---|---| +| no baseline | **held back** with a notification | +| the **generator hash** changed (`blog.py`, `atom.py`, `gen.py`, the shared CSS/icon, `site-blog/*`; fixed when the process starts) | held back | +| the **config** changed (hash of the loaded config) | held back | +| the **deploy target** changed | held back | +| any output file added or removed | held back | +| same paths, no content difference | logged "no visible change" | +| same paths, some content changed | deployed through the same function as `fw blog deploy` (audit and rsync, no question) | + +Held-back changes wait for you to run `fw blog deploy <config>`, which rewrites all of these baselines. So code, +template and config changes always wait for a manual deploy. + +### Notifications and failures + +- macOS notifications (`osascript`) are sent for held-back changes, failures, conflict files, mirror problems and unexpected errors, **never for a successful deploy**. +- A build or deploy failure notifies once and is **not retried until the next source change**. +- An unexpected exception is logged and notified once, and the tick still exits 0 (so launchd does not throttle the agent). +- Every subprocess has a timeout (git 120 s, mirror and deploy rsync 600 s, osascript 10 s). A timeout is an ordinary failure. + +### Manual deploys while the watcher runs + +A manual real `fw blog deploy` (not `--dry-run`): + +- takes the same lock (if a tick holds it: "a watch tick is running, try again in a minute"); +- refuses if the watcher deployed while you were answering the question (run it again); +- rewrites `deployed.json`, which keeps the baseline current; +- clears the watcher's "handled" state so the next tick re-evaluates. + +### The launchd agent + +`install` writes `~/Library/LaunchAgents/net.fuzzyworld.fw-blog-watch-<folder>-<hash>.plist` (every 60 s and at +load, low priority) and loads it with `launchctl bootstrap gui/$UID` (retried up to 5 times). `uninstall` unloads +and removes it. `install --print` only prints the plist. + +## WordPress import + +`lib/wp_import.py` (`fw blog import-wp <config> --wp URL`) reads the public REST API. + +- Skips password-protected posts. +- Matches each post to a source file: similarity of 0.95 or more uses your file; otherwise the WordPress text is written to `wordpress/<date>-<slug>.md`. A non-ASCII slug gives a readable file name but keeps its URL. +- Writes comments (name, date and text only, into the `comments` file next to the config), pages (with `published: true`), media and the config entries. +- `--dry-run` writes only a cache and `wordpress/_report.txt`. A second run changes nothing. + +`fw blog check <config>` lists posts whose built text differs from WordPress. + +## Tests + +`python3 -m unittest discover tests` covers the blog, the WordPress importer, the watcher, the comments and the runner. diff --git a/docs/ci.md b/docs/ci.md new file mode 100644 index 0000000..bdfeb4a --- /dev/null +++ b/docs/ci.md @@ -0,0 +1,96 @@ +# CI + +The same pattern as GitLab runners: a runner pulls jobs, runs them in containers, and reports back. There is no +coordinator service; the coordinator is files on DreamHost. + +## How a job flows + +1. **Queue.** `post-receive` queues a job (`~/fw/queue/pending/<id>.json`) for each pushed branch or tag whose commit has a `.fw-ci.yml` (or `.gitlab-ci.yml`) or a `package.json`, unless `fuzzyworld.ci` is `off`. `fw ci <name>` queues one by hand. +2. **Claim.** A runner (`runner/fw-runner.py`; on TrueNAS from cron, on a laptop from launchd, every 2 minutes) runs `ssh dh claim <runner> <tags>`. The server moves the job into `running/` with one `rename`, which is atomic, so two runners never take the same job. +3. **Heartbeat.** While a job runs, the runner sends a heartbeat every 30 seconds (`beat <job> <runner>`). A job whose runner has been quiet for about 2 minutes (lid closed, network gone) goes back to `pending/`, and a runner that lost its job this way can't mark it done. A runner that sends no heartbeats (an old one) is given 2 hours. +4. **Run.** The runner clones the commit and runs each job in a fresh Docker container (`--cpus 4 --memory 4g`, 30-minute timeout unless the job sets `timeout`). + - All jobs of a pipeline share one workspace, so artifacts are already there for later stages. + - Inside jobs, `origin` is the read-only https clone URL (`https://code.fuzzyworld.net/<name>.git`), as on GitLab. The runner's ssh key never enters a container. +5. **Report.** Logs and `pipeline.json` go to `~/fw/ci/<name>/<sha>/`. While a job runs, the runner marks it running and uploads its log every 15 seconds; each upload rebuilds that repo's CI pages and the dashboard (`fw ci-pages`). When the pipeline finishes, the repo's pages are republished, showing ✓/✗ on the log, the summary and `/<name>/ci/`. + +With no runner at all, jobs stay in `pending/` and the CI dashboard says "Waiting for a runner". + +### Tags + +A runner has tags (`FW_RUNNER_TAGS`, default `docker,chrome`); a job can ask for more with GitLab's `tags:`. + +- After cloning, a runner that lacks a tag the pipeline needs gives the job back (`release`) before running anything. The server then never offers that job to that runner again. +- Screenshots need `chrome`. +- The tag `secrets` means "this repo's secrets": a job tagged `secrets` on the default branch or a tag runs only on a runner that has `secrets/<repo>.env`. +- A job nobody can run waits in the queue. + +### The runner's key + +The runner's ssh key is limited in `authorized_keys` to `server/runner-api` (`restrict,command="…/runner-api"`). It can claim jobs, fetch repos and upload results. It can't get a shell. + +## Watching CI + +- **Web:** `code.fuzzyworld.net/ci/` shows what is running (each job ✓/✗/● with its time), what is queued and the last 50 results. Private repos are never named there, only counted. +- **Terminal:** `fw ci-watch` shows everything, private repos included (running jobs with each job's state, the queue, the last 5 results), and ends when nothing is queued or running. +- **Self-refresh without JavaScript:** while anything is queued or running, the dashboard and the running pipeline's page reload every 15 seconds (`<meta http-equiv="refresh">`). A thin bar along the top empties as the reload nears, and "Next reload in N s." counts down beside the reload note. Both are CSS only. The countdown needs a browser with `@property` (otherwise only the bar shows), and the bar is hidden for `prefers-reduced-motion`. + +## Job definitions: `.fw-ci.yml` + +A file at the repo root, in a subset of GitLab CI's YAML format, so an old `.gitlab-ci.yml` works after a rename +(`git mv .gitlab-ci.yml .fw-ci.yml`). Until a repo is renamed, the runner still reads `.gitlab-ci.yml`; `.fw-ci.yml` +wins if both exist. + +| Supported | Keys | +|---|---| +| Top level | `stages`, `variables`, `image` (string or `name:`), `default:` (image, before_script, after_script), `workflow:rules` | +| Reuse | `extends` (hidden jobs) | +| Per job | `stage`, `image`, `script`, `before_script`, `after_script`, `variables`, `rules` (`if`, `when`, `exists`; `changes` always matches), `only`/`except` (refs), `when` (`on_success`, `on_failure`, `always`, `manual`, `never`), `allow_failure`, `timeout`, `needs` (ordering comes from stages), `tags` | +| Ignored (change nothing here) | `artifacts`, `cache`, `dependencies`, `environment`, `interruptible`, `retry`, `coverage`, `resource_group` | + +**Anything else** (for example `include`, `services`, `release`, `trigger`, `parallel`) makes that job fail with a +message naming the key; nothing is silently skipped. `manual` jobs are listed but not run. + +- The usual `CI_*` variables are set (`CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`/`CI_COMMIT_TAG`, `CI_COMMIT_REF_SLUG`, `CI_DEFAULT_BRANCH`, `CI_PIPELINE_SOURCE=push`, ...), plus `FUZZYWORLD_CI=true`. +- Jobs without an image use `node:22-bookworm`. +- A `pages` job runs its build as a check; publishing is `fw deploy`'s job. + +### Without a pipeline file + +- An app (`package.json` with `fuzzyworld.app`) gets one job, `fw deploy --here --dry-run` (install, prebuild, checks, build, output checks). +- Other node repos get their `typecheck`/`lint`/`test` scripts. + +### Check what a checkout would run + +```bash +runner/fw-runner.py --plan <dir> [refs/heads/<branch>] +``` + +## Secrets + +`<runner dir>/secrets/<repo>.env` on each runner that should deploy (TrueNAS: `/mnt/main-2025/fw-runner/secrets/`; +laptop: `~/.fw-runner/secrets/`). The file is passed (`--env-file`) only to jobs on the default branch and tags. + +- A runner without the file still runs that repo's other jobs. Tag a deploy job `secrets` to keep it for a runner that has the file. +- Without any secrets, jobs that deploy elsewhere (for example `dreamhost`'s rsync, engye's store uploads) fail. +- Anything on the laptop is as safe as the laptop (FileVault on). If it is lost, rotate the secrets. + +## The laptop as a runner + +`runner/install-laptop.sh` (or `bin/deploy-tools laptop-runner`) installs a launchd job that runs every 2 minutes +with `FW_RUNNER_POLITE=1`. A polite runner claims a job only when: + +- the Mac is on AC power, +- at least `FW_RUNNER_MIN_IDLE` (default 50) percent of the CPU is idle, and +- Docker is running. + +It checks before each job, never during one. Its containers get a low CPU share (`FW_RUNNER_CPUS`, default 2 there, +is the real limit). + +| Item | Where | +|---|---| +| Settings | `~/.fw-runner/runner.env` | +| Log | `~/.fw-runner/runner.log` | +| Allow its key on DreamHost | `runner/install-laptop.sh --authorize` | +| Stop it | `runner/install-laptop.sh --remove` | + +It assumes the laptop is on for a few hours a day, which is enough when TrueNAS isn't there. diff --git a/docs/code-site.md b/docs/code-site.md new file mode 100644 index 0000000..ef636e4 --- /dev/null +++ b/docs/code-site.md @@ -0,0 +1,86 @@ +# The code site (code.fuzzyworld.net) + +Static pages for every public repo, generated by `server/gen.py` on each push (the `post-receive` hook starts +`fw publish <name>` in the background, so a push returns at once and pages update 1-5 seconds later). Every repo uses +the shared hook through `core.hooksPath=~/fw/tools/server/hooks`. + +Who sees what is set per repo; see [Visibility](../README.md#visibility). + +## Page URLs + +### Repo pages + +| URL | Page | +|---|---| +| `/` | index of public repos: a table, newest code commit first, with description and an app link | +| `/<name>/` | summary: description, clone command, "Open in VS Code" links, license, CI status, screenshots, recent commits, README | +| `/<name>/tree/<dir>/` | directory listing (default branch) | +| `/<name>/blob/<path>.html` | file view with line numbers and syntax highlighting (Pygments; files ≤ 512 KB); `.md` files rendered | +| `/<name>/history/<path>.html` | commits that changed the file (up to 200; renames not followed) | +| `/<name>/blame/<path>.html` | the commit that last changed each line (text files ≤ 256 KB) | +| `/<name>/raw/<path>` | raw copies of images only (png, jpg, gif, webp, ico, avif, bmp, svg; ≤ 2 MB) | +| `/<name>/log/` | the branch's commits (up to 2000) | +| `/<name>/commit/<sha>.html` | one commit with its diff (truncated at 300 KB); a page for every commit on every branch, up to 2000 per repo | +| `/<name>/refs/` | branches and tags | +| `/<name>/branch/<branch>/...` | the same tree/blob/history/blame/log pages for another branch; `/` in a branch name is written `~` | + +### Pages for features + +| URL | Page | +|---|---| +| `/<name>/ci/`, `/<name>/ci/<sha>.html` | CI pipelines (newest 100) and one pipeline with its job logs ([ci.md](ci.md)) | +| `/<name>/ci/badge.svg` | status of the latest pipeline on the default branch | +| `/ci/` | CI dashboard: running pipelines, the queue, the last 50 results, for public repos | +| `/<name>/issues/`, `/<name>/issues/<n>/` | bug-report issues, when the repo's mode and the issue's state allow it ([issues.md](issues.md)) | +| `/<name>/usage/` | Claude Code usage per commit and per session, only for repos with usage trailers ([usage.md](usage.md)) | +| `/usage/` | the same, summed over all public repos ([usage.md](usage.md)) | +| `/<name>/files.json`, `/<name>/objects/<sha>` | for VS Code in the browser: every file's blob SHA and size, and each file's contents (≤ 1 MB) | + +### Site pages + +`/projects/`, `/why-i-left-gitlab/`, `/my-ai-use/` and `/contribute/` are files in `server/content/`, rendered by +`fw content` into the common page layout. + +- A file is Markdown, or an HTML fragment with a leading `<!-- title: … -->`. +- An image in that folder that a page mentions (``) is copied next to the page. +- A file starting with `_` (the `/how/` draft) is not built. +- `/why/` and `/principles/` redirect to `/why-i-left-gitlab/`; the old `fuzzyworld.net/projects/` redirects to `/projects/`. +- `fw content` also writes `404.html`. + +`/ci/`, `/usage/` and the other site pages cannot be repo names (repos with those names are skipped). + +## Branches + +The default branch, and every branch **not yet merged** into it, get browse pages. A merged branch adds nothing new +to read and would duplicate every page, so the branches page lists it with a link to its last commit. Every branch +and tag can be cloned. Author email addresses are not shown. + +## License detection + +The summary page names the repo's license, read from a root `LICENSE`, `LICENCE` or `COPYING` file (MIT, Apache-2.0, +GPL, LGPL, AGPL, MPL-2.0, BSD, ISC, Unlicense, CC0), else from `package.json`. + +## Preview locally + +`npm run dev` builds this repo's pages into a temp directory and serves them at <http://localhost:8000/>. + +- Pass other working copies after `--`: `npm run dev -- ~/code/explodorama`. +- Set `PORT` to change the port. +- It uses each repo's committed state. `server/site/*.css` edits show on reload; a change to `gen.py` needs a restart. + +## Security + +The site never executes anything from a repo. + +| Rule | How | +|---|---| +| Pages are pre-generated | all repo content is HTML-escaped | +| Raw files are images only | written only for image extensions; `.htaccess` also denies `.php`, `.shtml`, `.cgi`, `.pl`, `.py` and `.sh` and disables SSI/CGI (DreamHost runs `.php` files and SSI otherwise; tested 2026-10-26) | +| No scripts | `Content-Security-Policy: script-src 'none'` on every response, which also covers SVG images | +| Clone copies hide internals | inside public clone copies, `config`, `description`, `hooks/` and `logs/` return 404 | +| File contents never render | `objects/<sha>` is always sent as `application/octet-stream` with `Content-Disposition: attachment`, so a repo's `.html` or `.svg` can never render as a page. `files.json` and `objects/` allow cross-origin reads, because VS Code runs on apps.fuzzyworld.net | +| CI logs are escaped | CI pages exist only for public repos | + +apps.fuzzyworld.net gets the same server-side no-execution rules (no PHP, SSI or CGI). Apps themselves run +JavaScript in the browser, so the apps site sends no script-blocking policy. HTML, service workers and manifests are +sent with `no-cache` (a deploy is seen at once); `/<app>/assets/` (hashed by vite) is cached for a year. diff --git a/docs/issues.md b/docs/issues.md new file mode 100644 index 0000000..bd70898 --- /dev/null +++ b/docs/issues.md @@ -0,0 +1,148 @@ +# Issues (bug reports by email) + +Bug reports arrive by email at `bugs-k7q4+<repo>@fuzzyworld.net`. The address is on every repo's summary page. The +repo name is part of it, and a repo that does not exist under `~/git` sends the mail to the bad folder. + +`server/issues.py` (`fw issues ...`, Python standard library only) turns each report into an issue and, when +allowed, a static page. It reuses the comments plumbing in `comments.py` (IMAP or Maildir source, +`Authentication-Results` check, locks, atomic JSON files). + +Moderate from the terminal with `fw issues`, or from the local web page ([admin.md](admin.md)). + +## Modes + +`fw issues <repo> public|reviewed|private` sets `git config fuzzyworld.issues` in the bare repo and republishes. +Anything other than exactly `public` or `reviewed` is `private`, and a repo that is itself private +(`fuzzyworld.visibility`) never shows issues, whatever its mode. + +| Mode | An issue's page appears | +|---|---| +| `private` | never; reports are stored and listed by `fw issues list` only | +| `reviewed` | after `fw issues approve <repo> <n>` | +| `public` | as soon as its first message passes email authentication (unauthenticated mail is stored with `authenticated: false` and waits for `approve`) | + +In `public` and `reviewed` mode each sender may send 5 messages per 24 hours (the rest goes to the bad folder). + +## Moderating + +| Command | Effect | +|---|---| +| `approve <repo> <n>` | marks the messages the issue has at that moment as shown | +| `hide <repo> <n>` | takes the issue off the site in any mode and keeps it off until `approve` | +| `close` / `reopen` | change the state shown on the page | +| `list`, `show` | read stored issues (plain text) | +| `publish`, `publish-all` | rebuild the pages | +| `ingest`, `imap-check` | read the mailbox / check the mailbox settings | + +Rules for messages: + +- A reply (its `In-Reply-To` or `References` matches an earlier message of an issue of the same repo) is added to that issue. +- A message that arrives after `approve` is shown only in `public` mode, and only if its sender passed authentication, or after another `approve`. An unauthenticated reply is never shown automatically. +- A mail that repeats a stored Message-ID or a stored byte-for-byte copy is ignored. + +## Pages + +`/<repo>/issues/` lists the visible issues; `/<repo>/issues/<n>/` shows one. Every publish rebuilds the whole +`issues/` folder, so hiding removes the page. When the folder appears or disappears, the summary page is rebuilt so +its "Issues" link follows. + +An issue page has: + +- an open/closed badge (also in the list), +- a small details table (version, rules, commit, seed, level, turns, lives, status, modes), filled from the game's own report lines when they match exactly, +- the messages in wrapping text boxes (a line of three or more dashes becomes a thin rule between boxes), +- links to the previous and next visible issue. + +Senders are labelled "Reporter A", "Reporter B", ... by first appearance within that one issue. The stored author +stays "Reporter"; hashes are never written. + +## Privacy rules for published text + +**Never written to a page:** any address, name, Message-ID or sender hash. + +**Replaced or dropped:** + +- every address-shaped string in the title and text becomes `[address hidden]`; +- lines starting `Device:` or `Run record:` are left out. + +**Device lines that wrap:** following lines are dropped while they are indented or look like a user agent +(`token/version`, parentheses or semicolons). A blank line, a template label such as `Seed ...`, or plain prose ends +the block, so the reporter's own words are kept. + +**Normalisation before filtering:** format=flowed text is re-joined first (an empty line always ends a paragraph; `delsp=yes` is honoured), and the text is NFKC-normalised, so fullwidth `Ｄevice:` or `＠` count. + +**Quoted replies are cut,** so quoted names never publish. The text is cut at the first of: + +- an attribution line in any language: a colon line with an address, or an `On`/`Am`/`Le`/`El`/`Op`... line with a date that says wrote/schrieb/a écrit/schreef/escribió; any such line directly followed by a quoted `>` line; +- a wrapped attribution: up to three lines ending in a colon, directly followed by a quoted line, or ending in a separate `wrote:` line; +- a `-----Original Message-----` line; +- a block of at least three `Label: value` lines that is a mail header (`From:`/`Von:`/`De :`..., with an address or three header labels; an Outlook `____` line before it is dropped); +- a colon line followed only by quoted lines at the end of the text. + +The last cut is never made when it would leave nothing, and prose such as `I pressed <Enter> and wrote:` stays. + +**Output:** all text is HTML-escaped inside `<pre class="issue-text">`, and no page has a script. + +## Run records + +If the text part holds a line `Run record: {json}` (the Explodorama bug-report mail does), it is read and checked. + +| Rule | Detail | +|---|---| +| Source | the full decoded part (the text shown is cut to 4000 characters; the record is not). If the decoded text has none, the raw mail is decoded as quoted-printable and searched again | +| Validity | at most 64 KB, valid JSON, nesting at most 32 deep, an object with integer `seed` and `simVersion` and a list `actions`, finite numbers only | +| Kept keys | only `simVersion`, `seed`, `startLevel`, `startCharge`, `options` and `actions`; all others are dropped | +| Refused | a record with an address-like string anywhere in it. Every refusal is logged in `rejected.log` | +| Not recognised | a record split over several lines by a mail client | +| Which one wins | the first line that passes (so the mail's later "too long for an email link" footer is ignored). Only the first record of an issue is kept; a record in an unauthenticated reply is ignored | +| Storage | re-serialised, never as mail text | + +**Publishing.** A record is as public as the message that carried it: one in a reply is published only once that +reply is shown (after `approve` in `reviewed` mode; at once in `public` mode for an authenticated sender). It is published as `/<repo>/issues/<n>/record.json` at the same moment as the issue page +(`approve` publishes both in `reviewed` mode; `hide` removes both). `server/site/htaccess` lets +`https://apps.fuzzyworld.net` read it (CORS) and sends `no-cache`. + +## Replay link + +If a deployed app of the repo has a `replay` template, a visible issue with a record shows **Watch what happened**. + +- "A deployed app of the repo" means `apps.fuzzyworld.net/<app>/.fuzzyworld.json` has `repo` equal to the repo, and the app has an `index.html`. +- The template comes from the `"fuzzyworld"` block of the app's `package.json`, for example `"replay": "replay.html?report={record}"` ([apps.md](apps.md)). +- The link goes to `https://apps.fuzzyworld.net/<app>/<replay>` with `{record}` replaced by the URL-encoded address of `record.json`. +- The app named like the repo is preferred, so a `-next` copy is not chosen. +- It is a plain link, not an iframe (the site's CSP blocks framing). No `replay` key, no link. + +## Machine-readable commands + +Used by [`fw admin`](admin.md). The text of unpublished reports **is** included (it is a private tool); sender +hashes, `mkey` and `mid` never are. + +| Command | Prints | +|---|---| +| `fw issues list [repo] --json` | every stored issue (`repo`, `n`, `title`, `created`, `updated`, `state`, `approved`, `hidden`, `public`, `record`, `messages`, `shown`, `waiting`) and each repo's `mode` and `public_repo`. `waiting` means a public or reviewed repo's issue that is not hidden and has messages not yet shown | +| `fw issues show <repo> <n> --json` | one issue with the full text of every message (label, date, authenticated, approved, shown), the details-table rows, previous/next numbers, the public page URL (only when public) and the replay URL (only when the record is shown) | +| `fw issues record <repo> <n>` | `record.json` as stored, even for an unpublished issue (error `no record for repo#n` when there is none) | + +Plain `list` and `show` are unchanged. + +## State and mail source + +State is in `~/fw/issues/` (override with `FW_ISSUES_DIR`), outside every web root: + +| File | Purpose | +|---|---| +| `repos/<repo>/<n>/issue.json`, `record.json` | the stored issue and its run record | +| `secret` | salt for sender hashes (mode 600) | +| `msgids.json` | threading | +| `keys.json` | a mail is never stored twice | +| `rejected.log` | every refusal | +| locks | | + +**Mail source:** `~/fw/issues/imap.json` (mode 600, the same format as the comments one; finished mail goes to +`issues-done` / `issues-bad`), or, without it, the Maildir `~/Maildir/.bugs` (`.bugs.done`, `.bugs.bad`). + +**Requirements:** + +- The bugs mailbox needs its own account or its own `folder` in `imap.json`. Ingest and `imap-check` refuse to run when it has the same host, user and folder as the comments `imap.json` (the two jobs would move each other's mail to their bad folders). +- Ingest refuses to run until `authserv_id` is set in `server/comments/sites.json`. +- The hourly job is `issues-ingest.sh` ([operations.md](operations.md#scheduled-jobs)). diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 0000000..84f3375 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,122 @@ +# Operations: installing, layout and jobs + +## Requirements + +| Machine | Needs | +|---|---| +| DreamHost | git, Python 3.12, `flock`, `setsid` | +| Laptop | Node ≥ 18, git, rsync, ssh with the `dh` alias | +| TrueNAS (runner) | Docker, Python 3 with PyYAML (both part of TrueNAS SCALE), and the `truenas` ssh alias on the laptop | +| Laptop as a runner (optional) | Docker (Docker Desktop, OrbStack, ...) and Python 3; the installer makes a venv with PyYAML | + +## Install and update + +```bash +server/install.sh # copy server/ to dh, install site files and hooks +server/install.sh --publish # ...and republish every repo +runner/install.sh # CI runner on TrueNAS: copy it, make its ssh key, add the cron job +runner/install.sh --authorize # ...and allow that key on DreamHost (runner-api only) +runner/install-laptop.sh # the laptop as a polite second runner (launchd, Docker, its own venv) +vscode-web/install.sh # VS Code in the browser at apps.fuzzyworld.net/vscode/ +``` + +`install.sh` copies `server/` from the working copy, not from a commit, so uncommitted changes there go live too. + +### `npm run deploy` (the usual way) + +| Script | Runs | +|---|---| +| `npm run deploy` | `bin/deploy-tools`: install the server and republish every repo, but only if `server/` changed since the last deploy | +| `npm run deploy-status` | `bin/deploy-tools status`: what is deployed where, and what is pending | +| `npm run list-repos` | `bin/fw list` | + +`bin/deploy-tools [server|runner|laptop-runner|vscode|all] [--force]` wraps the install scripts with a **ledger** +kept on the server, `~/fw/state/deployed.json`. + +- **Targets** are declared in this repo's `package.json`: `"fuzzyworld": {"deploy": {"<target>": {"paths": [folders it ships], "command": [script, args], "description": "...", "perMachine": true}}}`. + + | Target | Ships | Runs | + |---|---|---| + | `server` | `server/` | `server/install.sh --publish` (includes the republish, since pages are generated by the installed `gen.py`) | + | `runner` | `runner/` | `runner/install.sh` | + | `laptop-runner` | `runner/` | `runner/install-laptop.sh` (per machine) | + | `vscode` | `vscode-web/` | `vscode-web/install.sh` | + +- **The ledger** records, for each `<repo>/<target>`, the commit and a content hash of the folders that target ships. A target is redeployed only when that hash differs. So uncommitted edits count, unrelated commits (README, tests) do not, and committing the files you already deployed does not trigger a second deploy. Gitignored files in those folders are not hashed. +- The ledger is written only after the script succeeds, and any laptop sees the same state. The exception is `laptop-runner`, which is installed per machine and so is recorded per hostname (key `<repo>/laptop-runner@<hostname>`). +- `fw deploys` and the [admin panel](admin.md#deploy-deploy-read-only-for-now) read the same ledger. +- For flags `deploy-tools` does not pass (`--authorize`, `--remove`, `--out`) run the install script directly. To regenerate pages without installing, run `bin/fw publish-all`. + +## Layout on DreamHost + +### Sites and code + +| Path | Contents | +|---|---| +| `~/git/<name>.git` | canonical bare repos (push here; not web-visible) | +| `~/code.fuzzyworld.net/<name>/` | generated pages for public repos | +| `~/code.fuzzyworld.net/<name>.git` | public clone copy (dumb HTTP), refreshed on push | +| `~/apps.fuzzyworld.net/<app>/` | deployed apps (`fw deploy`); `index.html` at the root is generated | +| `~/apps.fuzzyworld.net/_shots/<app>/` | screenshots of each app, uploaded by runners (`shots.json` lists them) | +| `~/apps.fuzzyworld.net/vscode/` | VS Code in the browser (not listed on the apps index) | +| `~/writing.fuzzyworld.net/` | the blog's built site, uploaded by `fw blog deploy` from the laptop (never edited there) | + +### Tools and state (outside every web root) + +| Path | Contents | +|---|---| +| `~/fw/tools/server/` | this repo's `server/` directory, installed by `server/install.sh` | +| `~/bin/fw` | symlink to `~/fw/tools/server/fw` | +| `~/fw/state/` | per-repo page manifests and locks; `deployed.json` is the deploy ledger | +| `~/fw/logs/<name>.log` | output of each push's publish run | +| `~/fw/pylib/` | Python-Markdown 3.7 (README rendering) and Pygments 2.21.0 (syntax highlighting), `pip install --target`; `install.sh` adds both (the landing page needs Markdown) | +| `~/fw/queue/pending/`, `running/` | CI jobs, one JSON file each ([ci.md](ci.md)). `running/<job>.beat` is the runner's heartbeat; `declined/` holds the runners that gave a job back | +| `~/fw/runners/<runner>` | one file per runner: its tags, and (by its time) when it last checked in | +| `~/fw/ci/<name>/<sha>/` | CI results uploaded by runners: `pipeline.json` and one log per job | +| `~/fw/issues/` | bug-report issues: stored reports, `secret`, `imap.json`, locks ([issues.md](issues.md)) | +| `~/fw/comments/` | blog comments state and `imap.json` ([blog.md](blog.md#comments-by-email)) | + +## Scheduled jobs + +Jobs live in `server/jobs/`. `install.sh` writes `server/jobs/crontab` into the DreamHost crontab, between +`# fw:begin` and `# fw:end`; it replaces only that block. Cron wakes each job hourly. DreamHost's clock is Pacific +time, so **each job checks its own UTC schedule**, and daylight saving can't shift it. + +| Job | Schedule | What it does | Log | +|---|---|---|---| +| `north-star-cleanup.sh` | Tuesdays 06:00 UTC (was the GitLab schedule) | updates `~/fw/jobs/north-star` to `main`, then runs `scripts/clean-old-firebase.sh --days=30` as a preview and then `--apply` | `~/fw/logs/cron-north-star-cleanup.log` | +| `comments-ingest.sh` | hourly | `comments.py ingest`: turns new mail in the comments mailbox into blog comments (quiet unless something happened) | `~/fw/logs/cron-comments-ingest.log` | +| `issues-ingest.sh` | hourly | `issues.py ingest`: turns new mail in the bugs mailbox into issues and republishes the touched repos (quiet unless something happened) | `~/fw/logs/cron-issues-ingest.log` | + +Run one by hand: `ssh dh '~/fw/tools/server/jobs/north-star-cleanup.sh --now --preview-only'` (without +`--preview-only` for the real thing). + +**Secrets** are in `~/.config/<project>.env` on DreamHost (mode 600, never in a repo or the web root). +`north-star.env` holds `FIREBASE_TOKEN`. + +## Node on DreamHost + +The system Node is 18. Jobs use Node 22 from `~/fw/node` (the official linux-x64 build; checksum-verified), always +with `NODE_OPTIONS=--jitless`: DreamHost's grsec kernel refuses executable memory, so V8's JIT crashes without it. +npm installs use `--ignore-scripts`, because native addons can't be built there. `jq` 1.7.1 (the official static +build) is in `~/bin`. + +## VS Code in the browser + +`apps.fuzzyworld.net/vscode/?repo=<name>` opens a public repo in VS Code. It is linked from each summary page. + +| Parameter | Effect | +|---|---| +| `repo=<name>` | the repo | +| `branch=<branch>` | an unmerged branch | +| `file=<path>` | open a file | +| `file=<path>:<line>` | jump to a line | + +What it is: the open-source VS Code web build (the `vscode-web` npm package, 1.91.1, MIT) plus `vscode-web/fw-fs/`, +an extension that reads `files.json` and `objects/<sha>` from code.fuzzyworld.net. Everything is static files. + +- **Works:** quick open (Ctrl/Cmd+P), and search across files (text files up to 256 KB). +- **Read-only:** nothing can be changed; clone to edit. +- **Not yet:** Markdown preview and other webviews (they need a separate domain per view), and project-wide TypeScript IntelliSense (turned off; syntax colors work). + +The plain "Open in VS Code" link (`vscode://vscode.git/clone?url=…`) clones the repo into the visitor's own VS Code. diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..ea07dc3 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,70 @@ +# Claude Code usage: trailers, costs and charts + +A commit message can carry a Claude Code session's running token tally as git trailers. The code site reads them to +show usage per commit, per session and per week, per repo (`/<name>/usage/`) and across all public repos (`/usage/`). + +## Writing the trailers + +```bash +npm run prepare-usage-for-pr # same as: fw usage --trailers (python3 lib/usage.py --trailers) +``` + +`fw usage` also reads Claude Code's transcripts in `~/.claude/projects` for other views: + +| Command | Prints | +|---|---| +| `fw usage` | token use per session of this project: messages, current context size, cache-read, output (cache-read and output include subagents) | +| `fw usage --all` | the same for every project | +| `fw usage --now` | one line for the newest session, to decide when to put it down | +| `fw usage --trailers [--session ID]` | the commit-message trailers for the newest session here, or the one named. It warns on stderr when that transcript is over 10 minutes old | + +## The trailers + +| Trailer | Meaning | +|---|---| +| `FW-Tokens-In`, `FW-Tokens-Out`, `FW-Tokens-Cached` | exact integers | +| `FW-Tokens-Method: 2` | how the count was made (see "Older commits") | +| `FW-Model-Tokens: <model> in=… out=… read=… write5m=… write1h=…` | one line per model, so a cost can be computed | +| `FW-Session` | the session id, for provenance only | +| `FW-Rtk-Saved`, `FW-Rtk-Cmds` | rtk's lifetime savings; not read by the site | + +Notes on `FW-Model-Tokens`: requests with a prompt over 100K tokens on a model with a higher tier are filed under +`<model>@over100k`; a cache write without a 5m/1h split is counted as 5m. + +### Older commits + +- **Method 1** (no `FW-Tokens-Method`): that `jq` count double counted repeated transcript lines (roughly 1.5-2× too high) and missed subagents. They are left as written, shown with tokens but with the cost unknown. +- **Rounded `Usage:` line** (even older): `Usage: 10.6K in / 101.3K out / 5.1M cached (session tally)`. Still read. Steps between such lines are only known to within the rounding (about two tenths of K/M/G), so small ones show as ≈0. +- A commit without a trailer shows no bar. + +## How the site reads them + +- **A commit's cost** is the step from the previous tally. A new session starts when a tally (or any model's counter) falls, or the method changes. +- **Dollar figures are estimates.** `server/prices.json` holds the list prices per model (with an `as_of` date), and the page says "estimated API list-price equivalent, not what was paid". +- **"AI-marked"** means an AI `Co-Authored-By` trailer or a usage trailer. A commit without one is "no AI marker", not "human" (see Provenance in [PLAN.md](../PLAN.md)). + +## What the pages show + +**The commit log** has a bar of output tokens per commit, highlighted at 2× the median or more. Each commit page +shows its step, estimated cost and the session tally. + +**`/<name>/usage/`** ranks commits, totals sessions (with an estimated cost per session, "unknown" unless every +commit in it has one) and draws five charts, per week (per month for histories over 104 weeks): + +1. output tokens, +2. estimated cost, +3. cached reads, +4. average cost per commit (the tooltip gives n), +5. AI-marked commits: two lines, all commits and AI-marked ones. + +In the token and cost charts, hatched means commits without usage data or without a known cost: unknown, rather +than zero. The AI-marked chart has no hatching. + +**`/usage/`** rolls the same up over the public published repos, with a table of one row per repo. A repo's cost +shows "(n of m commits)" when only some commits have a known cost. Charts start at the first AI-marked commit; dates +before 2000 or after tomorrow are ignored. + +### How it is built + +`/usage/` is rebuilt on every `fw publish`, from a small per-repo summary that `build_repo` saves in the state dir +(`<name>.usage.json`, keyed by the repo's HEAD), so it does not re-run `git log` for every repo. diff --git a/server/blog_watch.py b/server/blog_watch.py index 48f93fd..d867b4c 100644 --- a/server/blog_watch.py +++ b/server/blog_watch.py @@ -9,7 +9,7 @@ folder has been quiet for `autodeploy.quiet_seconds` (default 300), then (1) mir commit on a separate branch, (2) build the site and deploy it only when the set of output files is the same as in the last deploy and some file's content differs. Anything else (a new or removed page, no baseline, or a changed generator, config or deploy target) is held back with a notification: run `fw blog deploy <config>` yourself. State: ~/.fw-blog-watch/<name>-<hash>/ (see blog.watch_name): -state.json, watch.log, deployed.json, hashes.json, lock, launchd.log. Rules and config keys are in the README ("Blog"). +state.json, watch.log, deployed.json, hashes.json, lock, launchd.log. Rules and config keys are in docs/blog.md. Everything with side effects (clock, notification, running commands, the deploy) is injected through `Env`, so the tests use fakes. Stdlib only.