fuzzyworld

fuzzyworld-tools / files / PLAN.md

PLAN.md

Blame: the commit that last changed each line · file · history

1758548c 2026-10-27 Pavel Safronov1# Plan
2
3Where fuzzyworld-tools goes from here: from "a static mirror of my repos" to a small, open-source
4code host built on one rule: **DreamHost only serves files; other machines do the work.**
5
6This is a planning document. Nothing in it is promised; the order is the current priority.
a890b8a6 2026-11-04 Pavel Safronov7Last updated 2026-11-03.
77346d8a 2026-10-29 Pavel Safronov8
9**Who it is for.** First, me: I build the site I want and need (no GitLab, no WordPress, as few machines as possible).
10Second, anyone who wants to get going with the fewest resources. Anything that only makes sense for my setup is
11tagged 🔌 **plugin**: it should be optional and separable, so that the core stays small enough for most people.
1758548c 2026-10-27 Pavel Safronov12
13## The machines
14
15| Machine | Does | Can't do |
16|---|---|---|
17| DreamHost (shared hosting) | serves static files with no rate limits; git over ssh; hourly cron; `post-receive` hooks | long-running processes, Chromium, V8's JIT (Node runs `--jitless`), native addons |
18| TrueNAS (home, always on) | Docker: CI runner, headless browsers, screenshots, code-server; already mirrors every repo | be reached from the internet. It **pulls** work from DreamHost over ssh instead |
19| Laptop | `fw deploy`, interactive work; can act as a second runner | stay on |
20
21Everything on the sites stays pre-generated: no app server, no database, no accounts, repo content is
22never executed. Where a feature needs JavaScript, it is one script **we** ship (CSP `script-src 'self'`,
23never anything from a repo).
24
25## Now (prioritized)
26
9eb01df7 2026-10-27 Pavel Safronov27All done: ✅ live since 2026-10-27 (v1.1.0), unless noted.
1758548c 2026-10-27 Pavel Safronov28
392edf54 2026-11-04 Pavel Safronov291. ✅ Apps index: newest deploy first (2026-11-04: now newest code commit first, and both the code and apps indexes are tables, not tiles). (It once listed the commits of each deploy; removed 2026-10-27 as noise.)
9eb01df7 2026-10-27 Pavel Safronov302. ✅ **LICENSE file** (MIT) in this repo. Each repo's license on its summary page.
313. ✅ **Principles** in four places: `/why/`, `/how/`, a new `/principles/` page, and a "Philosophy" section in the
32 README. `/how/` is still a draft that `install.sh` doesn't install (see Backlog); the principle is in it for
33 when it goes live.
344. ✅ **File navigation**
1758548c 2026-10-27 Pavel Safronov35 1. "Open in VS Code" link on every summary page (`vscode://vscode.git/clone?url=…`).
36 2. Syntax highlighting at generation time (Pygments in `~/fw/pylib`; no JavaScript).
37 3. Per-file history, blame, and browsing every branch not yet merged into the default branch.
38 4. VS Code in the browser at `apps.fuzzyworld.net/vscode/?repo=<name>`: the `vscode-web` build plus a
39 read-only file-system extension reading `files.json` and `objects/<sha>` from code.fuzzyworld.net.
9eb01df7 2026-10-27 Pavel Safronov405. ✅ **CI runner** on TrueNAS. See [CI](#ci). Verified live with nordic-skating and tower.
416. ✅ **Screenshots** as a built-in runner job. See [Screenshots](#screenshots). Live for nordic-skating and tower.
1758548c 2026-10-27 Pavel Safronov42
43## Next (in this order)
44
45The order goes from cheap to expensive, and each step reuses what the one before it built.
46
77346d8a 2026-10-29 Pavel Safronov470. **Runners without TrueNAS** (decided 2026-10-28; see [Runners](#runners)): runner tags, the laptop as a runner when
48 idle and plugged in, secrets in a folder on each runner, and "no runners" as a normal state. Goes first.
73733e6e 2026-11-01 Pavel Safronov49 **Built and committed 2026-10-31 (a499715); deployed 2026-10-31** (server, then the TrueNAS runner, then the laptop runner with `--authorize`,
462e50c2 2026-10-31 Pavel Safronov50 in that order: a new runner against the old server has every heartbeat refused). Smoke-tested: both runners poll
51 DreamHost and find nothing to run; no real job has gone through them yet. Review follow-ups (claim check on `done`,
0e603cf7 2026-10-31 Pavel Safronov52 tolerant `release`) deployed the same day.
1758548c 2026-10-27 Pavel Safronov531. **Feeds**: Atom per repo (commits, releases), for app deploys, and site-wide. Smallest step: the data is already
77346d8a 2026-10-29 Pavel Safronov54 gathered for the log pages and the apps index. The blog reuses it.
551b. **Blog** at `blog.fuzzyworld.net` (see [Blog](#blog)): a generator that turns a folder or repo of Markdown into a
56 blog from one JSON file, plus a one-time WordPress import. Replaces WordPress.
1758548c 2026-10-27 Pavel Safronov572. **Releases**: a page per tag (the tag's message as notes), `git archive` `.zip`/`.tar.gz`, and files the
58 runner builds (APKs, extensions). Import the release assets already in `~/code/gitlab-export/*/release-assets/`.
59 Comes early because some of those files exist nowhere else. Adds a releases feed to step 1.
603. **Stats and heatmap**, built together, since both are computed from the same pass over the history when pages are generated:
61 - stats: languages by bytes, size, commit count, share of commits made with AI (see [Provenance](#provenance))
62 - heatmap: a commit-activity SVG per repo and site-wide
634. **Wiki**: render a repo's `docs/` (or `wiki/`) folder with a sidebar. Import the GitLab wikis in
64 `~/code/gitlab-export/*/wikis.json`. Static rendering only; editing comes with pull requests.
55e5fc3e 2026-11-04 Pavel Safronov655. **Issues** (built 2026-11-03): see [Issues](#issues). The first feature that reads email; builds the mailbox plumbing.
77346d8a 2026-10-29 Pavel Safronov665b. **Blog comments** by email (see [Comments](#comments)): right after Issues, because it reuses the mailbox plumbing.
1758548c 2026-10-27 Pavel Safronov676. **Pull requests**: see [Pull requests](#pull-requests). Last, because it reuses the issues mailbox plumbing
68 and needs the CI runner (approved CI runs on contributed code).
69
77346d8a 2026-10-29 Pavel Safronov70## Runners
71
72Goal (2026-10-28): TrueNAS becomes optional (🔌 plugin), and the laptop can do the work. Running jobs needs no
73always-on machine; a small setup is a DreamHost account and a laptop.
74
75- **No runners is a normal state.** Jobs stay in `pending/` until a runner claims them (the queue is files, so this
a4997156 2026-10-31 Pavel Safronov76 already works). The dashboard says "waiting for a runner" instead of looking broken (a runner counts as gone after
77 15 minutes without a claim; each claim writes `~/fw/runners/<name>`).
78- **Laptop runner** (launchd via `runner/install-laptop.sh`, every 2 minutes), only when it will not hurt my work: on AC
79 power, CPU at least half idle, Docker running, containers at a low CPU share, no new claims once I get busy (checked
80 between jobs; a job already running is not stopped). Everything scheduled runs then. It assumes the laptop
77346d8a 2026-10-29 Pavel Safronov81 is on a few hours a day; that is enough for me.
a4997156 2026-10-31 Pavel Safronov82- **Lease.** A claimed job needs a heartbeat (`beat`, every 30 seconds, from a thread in the runner). A job whose runner
83 went quiet for ~2 minutes (lid closed) goes back to `pending/` instead of waiting 2 hours. If that runner wakes up and
462e50c2 2026-10-31 Pavel Safronov84 finishes anyway, its results are uploaded too, and the job may have run twice: accepted. Its `done` is refused,
85 so it cannot remove the new holder's claim. Runners that send no
a4997156 2026-10-31 Pavel Safronov86 heartbeat (old versions) keep the 2-hour rule.
87- **Tags** (`docker`, `chrome`, `android`): job `tags:` are honoured, and screenshots need `chrome`. The runner learns what
88 a pipeline needs only after cloning, so a runner that lacks a tag gives the job back (`release`); the server
89 remembers and does not offer it to that runner again. Simple, at the cost of one clone per declined job.
77346d8a 2026-10-29 Pavel Safronov90- **Secrets.** Each runner has a folder (`…/secrets/<repo>.env`; on TrueNAS `/mnt/main-2025/fw-runner/secrets/`).
a4997156 2026-10-31 Pavel Safronov91 A deploy job tagged `secrets` (a reserved tag: "this repo's secrets") runs only on a runner that has the file; no repo
92 uses it yet. Only default-branch and tag jobs get secrets. Put them on the laptop and TrueNAS is not needed. Laptop theft means rotating them (FileVault on).
77346d8a 2026-10-29 Pavel Safronov93- **Stretch goal: a secure secrets API** (🔌): `runner-api` hands each job its secrets over ssh, per runner key, so
94 no runner stores them. Wait for user requests or downtime.
95
96## Blog
97
98`blog.fuzzyworld.net` is WordPress today (theme Scrawl, 144 posts, 4 pages, 45 comments, 15 media files, 3 categories,
9916 tags, one author). Goal: the same site, generated by this repo, with no WordPress.
100
0e603cf7 2026-10-31 Pavel Safronov101**Built and live 2026-10-31 at `writing.fuzzyworld.net`** (queue item Q4; not linked from anywhere yet). The generator is
102`server/blog.py` (+ `server/atom.py`, `server/site-blog/`), the importer `lib/wp_import.py`; commands and behavior are in
bb7e1f57 2026-11-06 Pavel Safronov103docs/blog.md. Decisions taken: build on the laptop from Dropbox and upload only the audited output folder (never
0e603cf7 2026-10-31 Pavel Safronov104the source); footer "All rights reserved" notice with an AI-training line and a `robots.txt` blocking AI crawlers; the
105WordPress text wins when a source file differs (similarity under 0.95: 100 of the 142 posts are written from WordPress
106into `~/Blog/wordpress/`); the old site's custom CSS is carried over (hidden `<br>`, dotted scene breaks, underlined
107chapter headings); a light/dark switch and reading-progress bar added (they needed `script-src 'self'`). The Atom
73733e6e 2026-11-01 Pavel Safronov108writer exists now, so Next 1 (Feeds) can reuse it. **Done 2026-11-01 (Q5/Q6):** `blog.fuzzyworld.net` 301-redirects every path (query kept) to `writing.`; `til.fuzzyworld.net` answers 410 Gone. Both WordPress folders stay on the server, unserved; the original `.htaccess` of each is saved as `.htaccess.wp-20261101`. Backup: a WordPress export of the blog (Pavel, `~/Downloads`). The ancient `old.blog.fuzzyworld.net` copy was moved to `~/old/` on the server (it was not served). **Still to do:** building from the push hook after the move to a private repo,
e9fdc5db 2026-11-01 Pavel Safronov109the `til` blog: **built 2026-11-01 (Q14)**, config `~/til/fw-blog.json` (4 posts, WordPress slugs kept, one image in `~/til/media/`), same generator; uploaded the same day (the WordPress folder was moved to `~/old/` on the server first, because `fw blog deploy` runs rsync `--delete`). Below, the plan as written before the
0e603cf7 2026-10-31 Pavel Safronov110build: it still says `blog.fuzzyworld.net` in places; read that as `writing.fuzzyworld.net`.
a4997156 2026-10-31 Pavel Safronov111
77346d8a 2026-10-29 Pavel Safronov112**What is there** (read from its public API, 2026-10-28):
113- Posts at `/YYYY/MM/DD/<slug>/`, an RSS feed at `/feed/`, and monthly archives. Categories hardly matter ("Uncategorized"
114 has 138 posts). Tags are the real structure (a story series is a tag, e.g. Dice, Walkabout, Writing Excuses).
115- Pages: **Highlights** (the front page: a chronological list of the work I consider good, with an AI-use note),
116 **Blog** (all posts), **Collections** (series in reading order, a hand-written list), **Upcoming**.
795e51b3 2026-10-29 Pavel Safronov117- 2 posts are password-protected (Rasputin, Belfast). They are not carried over (see Protected posts below).
77346d8a 2026-10-29 Pavel Safronov118- A reading time on each post, media (images, plus epub/kpf/pdf downloads of Wick Canyon), and comments.
119- Comments: 45 (36 top-level, 8 replies). They get imported, as static text with the author's name (no emails).
120
121**Source: `~/Blog`** (Dropbox for now; a private git repo later): 286 Markdown files, `YYYY-MM-DD-name.md`, no front
122matter. The title is the first `# heading`. The post is only the **`## Writing`** section; `## Spelling` (cspell words)
123and `## Removed writing` (cut text) are private notes and are never published. 164 files have a `## Writing` section;
124the others are notes. WordPress turns single line breaks into `<br>`, so the renderer must too.
125
126**Findings that shape the design:**
127- File names do **not** match published URLs. Only 13 of 144 posts match `date-slug`: the publish date differs from the
128 draft date, and some names differ (`belfast-porn.md` is published as `/belfast/`). The import must record the mapping
795e51b3 2026-10-29 Pavel Safronov129 (published date, slug, tags, highlighted) somewhere.
77346d8a 2026-10-29 Pavel Safronov130- So the generator must keep the old URLs working: same path pattern and the same slugs.
131
795e51b3 2026-10-29 Pavel Safronov132**Post metadata** (decided 2026-10-28): front matter *and* config, so the tool does not force edits to source files.
133Precedence, highest first:
134
1351. front matter in the file (`date`, `slug`, `tags`, `highlight`, `published`);
1362. per-file overrides in the blog's JSON file, keyed by file name (a "sidecar" for people who will not or cannot edit
137 their files);
1383. defaults from the file: date and name from `YYYY-MM-DD-name.md`, title from the first `# heading`.
139
140Nothing is required: a plain folder of Markdown works. **Nothing is published unless marked** (`published` in front
141matter or an entry in the config): `~/Blog` has 286 files and 144 posts are live. This is the same rule as repo
142visibility (private unless exactly `public`). For `~/Blog`, the import writes the sidecar (your files stay untouched;
143easy to undo), and a later command can convert it to front matter, e.g. when the posts move to a private git repo.
144
145**Protected posts** (decided 2026-10-28): not carried over. The importer skips password-protected posts (Rasputin,
146Belfast) and the comments on them, and says so. They stay drafts in `~/Blog`. 🔌 plugin idea for later: `.htaccess`
147basic auth first; client-side encryption only if needed (it adds a script and allows offline guessing).
148
77346d8a 2026-10-29 Pavel Safronov149**Design:**
150- One JSON file describes a blog (title, base URL, source folder, which section is the post body, URL pattern, tag
151 list, highlight list, collections, feed). The tool is `gen.py` run on a repo of Markdown, from the push hook.
152- Reuse what exists: Python-Markdown, Pygments, the site CSS, the feed code from step 1.
153- **Import (🔌 plugin):** one-time script that reads the WordPress API: posts, comments, media, and a redirect map.
154- The editor and the Dropbox → private-git-repo move come after the generator, not with it.
bb7e1f57 2026-11-06 Pavel Safronov155- **Built 2026-11-02 (Q91/Q92):** `fw blog watch <config>` (`server/blog_watch.py`; docs/blog.md, "The watcher"): a laptop
9bb22e48 2026-11-03 Pavel Safronov156 launchd agent (`fw blog watch install`) runs one tick every 60 s per blog config. After a quiet period it mirrors the
157 Dropbox source into a local git repo (snapshot commits on `dropbox-mirror`, never main) and, with `autodeploy`, deploys
158 edits only when the set of output files, the generator (code/templates), the config and the deploy target are
159 unchanged (anything else is held back with a notification; the baseline is `deployed.json`, also written by a manual `fw blog deploy` and guarded by the
160 state folder's lock). Run on ONE machine only. **Not built yet:**
161 the cutover `fw blog move <config> --to repo|dropbox` (copy, build both, require identical output, repoint the `~/Blog`
162 symlink, keep the old copy), then Dropbox is dropped and the Air uses a clone plus a simple `fw blog sync`; the server
163 remote and the push-hook build (Q25).
77346d8a 2026-10-29 Pavel Safronov164- Needs the `blog.fuzzyworld.net` subdomain pointed at a folder (DreamHost panel).
165
166## Comments
167
e9fdc5db 2026-11-01 Pavel Safronov168Decided 2026-10-28: by email, no accounts. Revised 2026-11-01: a web form (`server/comments/post.php`, written, never run
169or installed) was designed too, but **only the
170email path is enabled** (Pavel: a form means code that runs on every web request, which is more attack surface than
171he wants; see the tiers in the README, form = T2, email = T1). The form stays in the repo, off by default.
172- A "comment by email" link on each post is a `mailto:` to the comments mailbox, with the post in the subject
173 (`Comment [writing:/2024/05/01/slug/]`). A cron job (`server/comments.py ingest`) reads the comments mailbox, over IMAP
174 (a hosted DreamHost mailbox; mail arrives through MailChannels) or from a local Maildir, and puts each comment in a pending queue.
175- Identity is the sender's verified address (the topmost Authentication-Results header must show DMARC pass, or DKIM
176 pass for the From domain). The address is never shown or stored, only a hash; only the display name is shown.
177- Moderation: a person's first comment waits for my approval (`fw comments list|approve|reject|remove|untrust`);
178 with `approve --trust` that sender is published on arrival from then on. Limits: 4000 characters, 5 per sender per 24 hours, 200 pending.
179 A deletion request is honoured.
180- Approved comments are written as one static JSON file per post (`/_comments/<post path>.json`) which `blog.js`
181 renders with `textContent`, so a comment appears without a rebuild. `fw blog deploy` excludes `/_comments/`.
182 Imported WordPress comments stay baked into the pages. Readers without JavaScript see only the imported ones.
183- **The form (off):** `server/comments/post.php` on its own subdomain, `comments_form` in the blog config; every
184 form comment is moderated. Enabling it is a decision for later; the upsides are no email client needed, the
185 downsides are a public PHP endpoint, spam, and unverified identities.
186- **🔌 Google sign-in plugin:** would need an endpoint that checks the Google token (T2), so it stays a plugin.
77346d8a 2026-10-29 Pavel Safronov187
a5a72f55 2026-10-27 Pavel Safronov188## CI dashboard
1758548c 2026-10-27 Pavel Safronov189
bb7e1f57 2026-11-06 Pavel Safronov190✅ Built 2026-10-27 as proposed (docs/ci.md: "Watching CI"):
1758548c 2026-10-27 Pavel Safronov191
192- `code.fuzzyworld.net/ci/`: one static page for all public repos: queued jobs, running pipelines (which job,
193 for how long), and the last 50 results. Rebuilt on every queue change and every upload from a runner.
194- Live logs: while a job runs, the runner uploads its log every ~15 seconds, so the job's page shows it growing.
195- Pages that show something running reload themselves every 15 seconds (`<meta http-equiv="refresh">`), so no
196 JavaScript is needed and the site's `script-src 'none'` stays.
197- Private repos: never named on the public page. For those, `fw ci-list` in the terminal (and a `fw ci-watch`
198 that follows a run and prints each job's result).
199
200## Backlog
201
85702b14 2026-11-04 Pavel Safronov202- **URGENT (Pavel, 2026-11-04): AI usage analysis per repo, on the CI/usage pages.** There is a lot of public chatter
203 about Claude's usage limits, so each repo should show what its AI use looks like over time. It builds on the
204 existing per-commit "Usage" tab (`repo_usage` / `usage_costs` in `server/gen.py`, from the `FW-Tokens-*` commit
205 trailers) and on the planned "Stats and heatmap" step (Next 3). Candidate views, as static SVG charts (no JS):
206 AI usage across time (output tokens, cached reads, cost per week or per session); AI share of commits and of pull
207 requests over time (`Co-Authored-By`, with the Provenance caveats); lines of code added and removed over time;
208 **usage per line of code** (output tokens, and estimated cost, per added line, per week and per commit, with the
209 median and the hot commits already marked on the Usage tab); and a site-wide roll-up across repos. Open points:
210 how to price tokens (a per-model price table kept in the repo, stated as an estimate); what a "line" is (added
211 lines in non-generated, non-vendored files); sessions that span several repos; commits without trailers (the
212 ChatGPT era) shown as unknown, never as zero. Data stays derived from git; no transcripts are published.
ced0eafa 2026-11-04 Pavel Safronov213 **Q107 stage 1 (done 2026-11-04):** trailers that make cost computable (`FW-Tokens-Method: 2`, per-model
214 `FW-Model-Tokens`, written by `fw usage --trailers`; each message id counted once, subagents included); per-commit
215 estimated cost from `server/prices.json` (list prices, `as_of`; unknown stays unknown); one weekly "output tokens"
216 SVG chart on the Usage tab with a hatched strip for commits without data. Old method-1 trailers are left alone
217 (they ran about 1.5-2x high).
799905ae 2026-11-04 Pavel Safronov218 **Q107 stages 2 and 3 (done 2026-11-04):** the Usage tab also charts estimated cost, cached reads, average cost per
219 commit and AI-marked commits (two lines) per week or month, and the Sessions table has an est. cost column; the
220 site-wide `/usage/` page (`build_site_usage`, called from `build_index`, public published repos only) has the same
221 charts plus a per-repo table (unknown costs are hatched, never zero; the page is rebuilt on every `fw publish`
222 from a per-repo summary cached in the state dir by HEAD sha). Reserved top-level names (`usage`, `ci`, `projects`,
223 ...: `SITE_RESERVED` in `server/gen.py`) are never treated as repos by the index or the roll-up.
224 Remaining: (4) optional: lines added/removed per week and usage per added line.
1758548c 2026-10-27 Pavel Safronov225- File finder and code search (one shipped script plus a prebuilt index per repo)
226- code-server on TrueNAS, for editing in the browser (private, over Tailscale)
227- Provenance stats and transcript publishing (see below)
228- An app at every commit: `apps.fuzzyworld.net/<app>/@<sha>/` (a service worker at `/<app>/` would cover those paths; must be solved first)
229- OpenGraph images (link previews) from the screenshots
230- Deploy automatically when CI passes on `main`
231- Run GitHub Actions workflows through `act` (10 repos have `.github/workflows`)
232- Publish the shared-origin deploy checks as a reusable lint
c7da2b90 2026-10-29 Pavel Safronov233- Discuss: an Advent of Code wrapper app (idea added 2026-10-28; Pavel thinks it has merit). A static page: sign in
234 with GitHub, fetch the user's own puzzle data from adventofcode.com (it differs per user), show the puzzle beside
235 an in-browser editor holding a TypeScript function that takes the logically named test data, run the code on a
236 button press, submit the answer to adventofcode and show whether it is right. To settle: how to sign in to
237 adventofcode from a static page (its session cookie, CORS, and its automation rules), where the code runs
238 (a worker in the browser), and how it relates to the existing AoC harness
1758548c 2026-10-27 Pavel Safronov239- Found while building "Now" (2026-10-27):
240 - VS Code in the browser: Markdown preview and other webviews need a separate domain per view
241 (a wildcard subdomain, e.g. `*.vscode.fuzzyworld.net`, set up in the DreamHost panel)
242 - VS Code in the browser: build from current VS Code source instead of the `vscode-web` 1.91.1 package (2024)
243 - VS Code in the browser: project-wide TypeScript IntelliSense (turned off; the web TS server rejects our URI scheme)
a5a72f55 2026-10-27 Pavel Safronov244 - ✅ Renamed `.gitlab-ci.yml` → `.fw-ci.yml` in 37 repos (2026-10-27; blister's is only local, never pushed).
77346d8a 2026-10-29 Pavel Safronov245 READMEs that said "deployed to GitLab Pages" are fixed in 11 repos (2026-10-28; edits waiting to be committed in each).
246 Still to do: the same in basin, chance-it, flight, herd, nuts, takeover (their local copies are behind, see below),
247 the `.md` plans/designs that still say it (e.g. tower's design.md, decisions.md), and dropping `pages` jobs that only
248 published to GitLab Pages
1758548c 2026-10-27 Pavel Safronov249 - Old `.gitlab-ci.yml` files: `node:*-alpine` images have no `git`, so installing the
250 `fuzzyworld-tools` git dependency fails (e.g. bubbles). Switch those jobs to a `node:22` image, or delete
251 `pages` jobs that `fw deploy` replaced
77346d8a 2026-10-29 Pavel Safronov252 - ✅ The 34 merged `fuzzyworld-deploy` branches on DreamHost were deleted (2026-10-28). Local clones and the TrueNAS
253 mirror may still list `origin/fuzzyworld-deploy` until pruned (`git fetch --prune`)
1758548c 2026-10-27 Pavel Safronov254 - CI `cache:` support (a per-repo volume on TrueNAS), and `services:`
255 - Branch pages: link to the default branch's page when a file is identical, instead of a copy
9eb01df7 2026-10-27 Pavel Safronov256- Found in review (2026-10-27): discussed, not done
0a08554d 2026-10-27 Pavel Safronov257 - Site pages: all now in `server/content/` (`gen.py content`), including `404.html`; `/how/` is `_how.html` (draft, not built)
258 - `/how/` goes live: finish the draft and rename `server/content/_how.html` to `how.html` (`/why/` already says it is coming)
9eb01df7 2026-10-27 Pavel Safronov259 - CI secrets: none are set, so jobs that deploy elsewhere fail (`dreamhost`'s rsync, engye's store uploads).
a4997156 2026-10-31 Pavel Safronov260 They go in `<runner>/secrets/<repo>.env` (TrueNAS: `/mnt/main-2025/fw-runner/secrets/`; default branch and tag jobs only)
9eb01df7 2026-10-27 Pavel Safronov261 - Look at the results of the 37 rename pushes' CI runs (north-star and nuts failed; not looked at yet)
262 - explodorama: `test/demo.test.ts` "decoy" timed out at vitest's 5-second default on TrueNAS (passed the day
263 before); a longer `testTimeout` for that test, or a faster test
77346d8a 2026-10-29 Pavel Safronov264 - tower: its screenshot caught the loading screen. `"screenshots": [{ "path": "", "size": "phone", "wait": 8000 }]`
265 is added to `fuzzyworld` in its `package.json` (2026-10-28); it takes effect once committed and pushed, and the
266 next CI run takes the new shot
9eb01df7 2026-10-27 Pavel Safronov267 - Local copies behind DreamHost, with uncommitted edits in the same files as the incoming commits:
268 basin, chance-it, flight, herd; takeover has staged changes. Pull them by hand
269- Housekeeping (noticed 2026-10-27, never discussed): nothing is cleaned up yet
270 - `~/fw/ci/<repo>/<sha>/` on DreamHost keeps every pipeline (pages show only the newest 100 per repo)
271 - `runner.log` on TrueNAS grows forever
272 - the runner's Docker images on TrueNAS (about 5 GB so far); a periodic `docker image prune`
1758548c 2026-10-27 Pavel Safronov273
274## Principles
275
276Wording approved 2026-10-26:
277
278> Things made with AI should be shared freely with the world. The code and apps here were made with
279> AI, so they are MIT-licensed: use them, change them, share them.
280>
ec9751fa 2026-10-27 Pavel Safronov281> My conversations with the AI are different. My words and my way of working are mine to publish or not.
282> I will publish a conversation only when I choose to.
1758548c 2026-10-27 Pavel Safronov283
284## CI
285
bb7e1f57 2026-11-06 Pavel Safronov286✅ Live since 2026-10-27. docs/ci.md is the full reference; this is the summary. It is the same
9eb01df7 2026-10-27 Pavel Safronov287pattern as GitLab runners: **a runner registers, pulls jobs, runs them in containers, and reports back.** The
288difference is that there's no coordinator service: the coordinator is files on DreamHost.
289
2901. **Queue.** `post-receive` runs `fw ci-queue`, which writes `~/fw/queue/pending/<id>.json` (repo, ref, sha,
291 default branch, time) for each pushed branch or tag with a `.fw-ci.yml`, `.gitlab-ci.yml` or `package.json`.
2922. **Claim.** The runner (`runner/fw-runner.py`, TrueNAS cron every 2 minutes) runs `ssh dh claim <runner>`
293 through `server/runner-api`. The server moves the job to `running/` with one `rename`, which is atomic, so
294 two runners can never take the same job. The same step puts jobs running for over 2 hours back in `pending/`.
2953. **Registration = an ssh key.** The runner's key is limited in `authorized_keys` to
296 `command=".../server/runner-api"`: it can claim jobs, fetch repos and upload results, and can't get a shell.
2974. **Run.** The runner clones over ssh and runs each job in a throwaway Docker container from the job's
298 `image`. Inside jobs, `origin` is the read-only https clone URL; the runner's key never enters a container.
2995. **Report.** Logs and `pipeline.json` are uploaded to `~/fw/ci/<repo>/<sha>/` (a running job's log every 15
300 seconds); `gen.py` renders ✓/✗ on the log and summary pages, a page per pipeline with its logs, a badge,
301 and the dashboard at `/ci/`.
3026. **Updates.** The runner restarts itself between jobs when `runner/install.sh` has replaced it.
462e50c2 2026-10-31 Pavel Safronov3037. **Two runners.** Tags, the heartbeat lease and the laptop runner are deployed (2026-10-31; see [Runners](#runners)).
1758548c 2026-10-27 Pavel Safronov304
305**Job definitions: `.fw-ci.yml`** (decided 2026-10-27: our own name; the format stays a subset of GitLab CI's,
306so moving a repo over is a rename; the old `.gitlab-ci.yml` name still works until then). 42 repos have one, and they use only `image`,
307`stages`, `script`, `before_script`, `variables`, `rules` (branch conditions), `needs`, `artifacts`, `cache`.
30836 of them are GitLab Pages jobs; a `pages` job maps to `fw deploy`. An unsupported keyword fails the job with
309a clear message; it is never silently ignored. Repos without the file fall back to `fuzzyworld.checks` from
310`package.json`. engye (20 KB: Android builds, Play Store, Firefox extension) is the hard case and the test of
311how far the subset needs to go.
312
9eb01df7 2026-10-27 Pavel Safronov313**Security.** Secrets (`/mnt/main-2025/fw-runner/secrets/<repo>.env` on TrueNAS) are given only to jobs on the
314default branch and tags. None are set yet (see Backlog). Code from contributors will run only after I approve it,
315because it runs on a machine in my home (part of Pull requests).
1758548c 2026-10-27 Pavel Safronov316
317## Screenshots
318
9eb01df7 2026-10-27 Pavel Safronov319✅ Live since 2026-10-27. A built-in runner job, after a successful pipeline on the default branch of an app
320(`fuzzyworld.app` in `package.json`):
1758548c 2026-10-27 Pavel Safronov321
9eb01df7 2026-10-27 Pavel Safronov322- builds the latest `main` the way `fw deploy` does (`fw deploy --here --dry-run`), serves it at `/<app>/` in a
323 Playwright container with no network, and photographs it. This is the commit CI just tested, which may be newer
324 than what is deployed
1758548c 2026-10-27 Pavel Safronov325- configured in `package.json`:
326 `"screenshots": [{ "path": "", "size": "phone", "wait": 1500 }, { "path": "?demo", "size": "desktop" }]`,
9eb01df7 2026-10-27 Pavel Safronov327 or a command for apps that need clicks first (`"screenshots": "pnpm shots"`, writing PNGs to `$FW_SHOTS_DIR`),
328 or `false`. With no config: one phone shot of the start page
329- apps without runner screenshots show the images in the repo's `screenshots/` folder instead (public repos)
330- shown on the apps index and repo summary pages; OpenGraph images are in the Backlog
1758548c 2026-10-27 Pavel Safronov331
332## Issues
333
55e5fc3e 2026-11-04 Pavel Safronov334- Reports arrive at `bugs-k7q4+<repo>@fuzzyworld.net` (only the address was live before 2026-11-03; the ingest and
335 the pages were built that day, see the task list under Report replays).
1758548c 2026-10-27 Pavel Safronov336- A cron job reads the mailbox (IMAP, Python standard library) and renders each thread as static pages.
337- Reporters' email addresses are never shown.
338- **Configurable per repo**, the same way as visibility: `git config fuzzyworld.issues public|reviewed|private`,
339 set with `fw issues <repo> <mode>`:
55e5fc3e 2026-11-04 Pavel Safronov340 - `public`: a report whose sender passed email authentication is published as it arrives (an unauthenticated one waits for approval)
1758548c 2026-10-27 Pavel Safronov341 - `reviewed`: a report is published only after I mark it public
55e5fc3e 2026-11-04 Pavel Safronov342 - `private`: nothing is published; reports are still stored in `~/fw/issues` (and listed by `fw issues list`)
1758548c 2026-10-27 Pavel Safronov343 - anything other than exactly `public` or `reviewed` counts as `private`, so nothing is published by accident.
344 A private repo's issues are always private, whatever this says.
345- Import the GitLab issues and notes in `~/code/gitlab-export/*/issues.json`.
346
55e5fc3e 2026-11-04 Pavel Safronov347### Report replays (Q104; planned 2026-11-03, built 2026-11-03 in this repo)
a890b8a6 2026-11-04 Pavel Safronov348
349Idea from the Explodorama bug mails: a report can carry a **machine-readable payload** (for the game: the run
350record, a few KB of JSON: seed, rules version and actions), and the issue page can then show *what the reporter
351saw* as a looping animation. The game side (the replay page, its limits, how old rules are loaded) is in
352`~/code/explodorama/PLAN.md` §20. This repo's part:
353
354- **Intake stays email.** DreamHost only serves files, so no upload endpoint (no abuse surface, no server code).
355 The existing mailbox cron reads `Run record: {…}` (or whatever line an app agrees on) out of a report.
356 The mail's quoted-printable text is decoded first (the game's `tools/report-replay.mjs` already does it in
357 Node; port the same rule to Python, and keep a test with a real report).
358- **Store it beside the issue.** For a repo whose issue mode allows publishing, the cron writes the payload as
359 a static file next to the issue page (`…/issues/<n>/record.json`), under the same visibility rule as the
360 report itself: `public` as it arrives, `reviewed` only after I mark it public, `private` never. The payload
55e5fc3e 2026-11-04 Pavel Safronov361 is validated first (size ≤ 64 KB, parses as finite JSON, an integer `seed` and `simVersion`, a list `actions`) and only the keys
362 `simVersion, seed, startLevel, startCharge, options, actions` are kept; a record with an address-like string is refused.
363 Anything else stays only in my mailbox.
a890b8a6 2026-11-04 Pavel Safronov364 Reporter emails are still never shown; the device string stays out of the page by default.
365- **Link a viewer, don't build one.** An app announces its viewer in `package.json`, for example
366 `"fuzzyworld": { "replay": "replay.html?report={record}" }`; with it, an issue page that has a payload shows
55e5fc3e 2026-11-04 Pavel Safronov367 "Watch what happened", a plain link to that page of the app, with `{record}` set
a890b8a6 2026-11-04 Pavel Safronov368 to the stored file. Without the key, nothing changes. This keeps the core free of any game knowledge (🔌
369 plugin-shaped: an app opts in).
370- **CSP.** The viewer is the app's own script on the app's own origin, so the site-wide `script-src 'self'`
371 policy (Q99) holds; the record is data, never inserted as HTML. The issue page itself needs no script.
372- **Old rules.** A viewer may need the sim of the report's own rules version. An app can publish those as
373 static files at deploy time (Explodorama: `/explodorama/sim/<simVersion>.js`); fuzzyworld-tools only has to
374 let `fw deploy` carry such extra files (a per-app list of globs in `package.json`) and keep old ones. The
375 runner (TrueNAS) can build them on a tag; DreamHost has no Node.
376- **Admin panel (Q102).** On the laptop, the inbox lists reports with a Replay link per report (the same viewer,
377 served locally with the dev page's `&rules=` mode, which can use any commit from git), and the buttons that
378 mark a report public, close it, or turn it into a regression fixture in the app's repo (`test/fixtures/`).
379 The panel is a thin local web page over `fw` commands; it holds no state of its own.
55e5fc3e 2026-11-04 Pavel Safronov380 **Decided 2026-11-03: local-only, never hosted on DreamHost.** Every panel action is privileged (publish,
381 merge, deploy), and a login form on shared hosting would make the most valuable target on the site; the
382 local panel's security is that it is unreachable, and its authority is the ssh key already held. It also
383 keeps "DreamHost only serves files". Cost: no triage from a phone (reports still arrive as email, and the
384 replay link works anywhere because it is a static page). Possible later add, not planned: one narrow
385 endpoint behind a per-report signed link in the report mail ("mark this report public"), with no login
386 and no general admin surface.
387
388**This repo's task list (2026-11-03; Pavel: "add to the plan, start implementing").** Items 1-4 were built
389on 2026-11-03 (`server/issues.py`, `lib/deploy.mjs`); the first line of each says so. Not yet committed or
390deployed is deliberately not recorded here. In dependency order:
391
3921. **Issues** (built 2026-11-03; Next 5): cron ingest of the bugs mailbox over IMAP (reuse the `comments.py` /
393 `comments-ingest.sh` plumbing), one static page per thread under `…/<repo>/issues/<n>/`,
394 `fw issues <repo> public|reviewed|private` (anything else = private), reporter addresses never shown, and
395 a "mark public" action for `reviewed` mode.
3962. **Record extraction** (built 2026-11-03): decode the quoted-printable mail, find the `Run record: {…}` line, validate
397 (≤ 64 KB, valid JSON, expected keys), write `record.json` beside the issue under the issue's visibility
398 rule; device string stays out of the page. Test with a real report (Explodorama's
399 `test/fixtures/q101-bomb-pass.json` and its `.eml`).
4003. **App opt-in** (built 2026-11-03): a `replay` key in the `"fuzzyworld"` block of `package.json`
401 (`"replay": "replay.html?report={record}"`); the issue-page generator reads it and shows "Watch what
402 happened". No key, no change.
4034. **`fw deploy` carries extra files, kept forever** (built 2026-11-03): a per-app glob list
404 (`"extra": ["sim/*.js"]`) published with the deploy; extra files are protected from `--delete` through an rsync
405 filter (uploaded and updated, never removed), so a later deploy never deletes old rules versions.
406
407No CSP change (the viewer is the app's own script) and no server code on DreamHost. The admin panel (Q102,
408local-only) comes after all four.
a890b8a6 2026-11-04 Pavel Safronov409
410**Order:** (1) the game's production replay page and limits (its PLAN §20.2 steps 1-2), by hand, with a stored
411record; (2) the cron extracting and publishing `record.json` and the `replay` key; (3) per-rules-version sim
412files in `fw deploy`; (4) the panel. Each step is useful alone.
413
55e5fc3e 2026-11-04 Pavel Safronov414**Decided 2026-11-03 (Pavel):** the viewer opens as its own page (a plain link, no iframe); a `reviewed`
415report's record is published at the same moment as its text; old `sim/` files are kept forever; compact records
416in the game's `mailto:` link (game PLAN §20.2 step 4) come after the cron (step 2); the published page defaults
417to the plain recorded run, with comparison as an option (game PLAN §20.2 step 5). Nothing open here.
a890b8a6 2026-11-04 Pavel Safronov418
85702b14 2026-11-04 Pavel Safronov419## Admin panel (`fw admin`, Q102; designed 2026-11-04, decisions by Pavel)
420
421**Version 1 built 2026-11-04** (`lib/admin.py`, `tests/test_admin.py`; server side: `issues list|show --json`, `issues record`, `comments list --json`). Not yet tried against the real server (tested only with a fake runner). Decisions made while building:
422- **Login:** a one-time login URL (random key) that sets a session cookie; everything else needs the cookie, plus the per-launch POST token and Host / Origin / Sec-Fetch-Site checks.
423- **Watch = dev replay:** a link to the deployed viewer when the record is public, "Download record" always, and "Prepare dev replay" (runs the app's `report-replay` npm script with the record and `--rules <commit>`; needs `npm run dev` running in the app).
424- **Recently approved comments** (the 20 newest) with a Remove button are in v1, not later.
425- **`FW_CODE_DIR`** (also `--code-dir`, default `~/code`) is configurable: where the apps' working copies live.
bb7e1f57 2026-11-06 Pavel Safronov426- **Deploy tab (Q111 stage 1, built 2026-11-04):** read-only; `fw deploys --json` (server/deploys.py) joins the deploy ledger (~/fw/state/deployed.json, written by `bin/deploy-tools`, keyed `<repo>/<target>`) and each app's `.fuzzyworld.json` with the repo's HEAD. Targets are declared in package.json `fuzzyworld.deploy`. Next: stage 2, a "Deploy now" button that deploys `origin/main` from a clean export (confirm page with details, background run, progress page); stage 3, any repo declares targets, apps included, tested on nordic-skating.
85702b14 2026-11-04 Pavel Safronov427- Still not in v1: "make this a regression fixture", pull requests, CI, blog state.
428
429A **local-only** web page over `fw` commands (see "Report replays": never hosted on DreamHost). `fw admin` starts a
430small Python (stdlib) server on `127.0.0.1` and opens the browser; every button runs the same `fw ...` command over
431ssh that Pavel would type, so the panel holds no state and has no authority beyond his ssh key.
432
433- **Decided:** version 1 is **reports and blog comments** together (both are "read it, then approve or reject");
434 **no JavaScript at first** (server-rendered pages and `POST` forms, a full reload after each action); the command
435 is `fw admin`.
436- **Safety:** bind to 127.0.0.1 only; check the `Host` header (DNS rebinding); every action is a `POST` carrying a
437 random token made at launch; a confirm step for destructive actions; never show passwords.
438- **Version 1:** (1) one list of reports across repos (repo, number, title, state, visibility, record, age;
439 filters "waiting for approval" and "has a replay"); (2) a report page with the full stored text, **including
440 unpublished reports**, the details table and thread, and Approve / Hide / Close / Reopen and the repo's mode
441 switch; (3) blog comments: pending ones with approve, reject, trust sender; (4) a Watch link per report with a
442 record (the dev replay page until the game's production viewer exists).
443- **Needs from the existing code:** `fw issues list --json`; `fw issues record <repo> <n>` (read an unpublished
444 record); a machine-readable `fw comments list`.
445- **Later, each useful alone:** pull requests (after contrib intake), CI and runners, blog build/deploy and watcher
446 state (Q100), "make this report a regression fixture" (writes `test/fixtures/` in the app's repo).
799905ae 2026-11-04 Pavel Safronov447- **Backlog (Q108, Pavel 2026-11-04): the admin page on the live site, working with the local `fw admin`.** A page on
448 code.fuzzyworld.net checks whether the visitor's machine runs `fw admin` (a probe of `http://127.0.0.1:<port>`) and,
449 if so, shows the admin UI there while all data and actions go browser to localhost; the live site never sees
450 reports, addresses or tokens (the hosted page is a static shell). Feasible, with conditions: (1) it needs JavaScript
451 on that one page (allowed by the "local JS is OK" policy, nothing else on the site changes); (2) `fw admin` gains a
452 JSON API with CORS for exactly the live origin, a bearer token (the current cookie login does not work
453 cross-site), strict Origin / Host checks and the Private Network Access preflight header; (3) browsers differ:
454 Chrome and Firefox allow an https page to call 127.0.0.1, newer Chrome asks the user once for "local network
455 access", Safari may block it as mixed content, so test all three and keep the local page (`fw admin` as today) as
456 the fallback; (4) the token reaches the live page through a link `fw admin` prints (token in the URL fragment,
457 which is never sent to a server), kept in `sessionStorage`. Not started.
85702b14 2026-11-04 Pavel Safronov458
1758548c 2026-10-27 Pavel Safronov459## Pull requests
460
461- Contributions arrive at `contrib-68uq+<repo>@fuzzyworld.net` as a branch URL or patches (already live).
55e5fc3e 2026-11-04 Pavel Safronov462 **Forward-only today** (2026-11-03): like `bugs-k7q4`, it must be recreated as a fully hosted mailbox before a
463 cron job can read it over IMAP, and so that the receiving server's `Authentication-Results` header is trusted
464 (a forwarded copy carries the destination's header instead). Give it its own account, never the comments one.
1758548c 2026-10-27 Pavel Safronov465- A cron job fetches each one into `refs/contrib/<n>` in the bare repo, never onto `main`.
466- It renders a review page: the diff, CI results (after I approve the run), and the discussion from email replies.
467- **Transcripts.** A submitter can attach the AI conversation behind the change, so the reviewer sees exactly
468 where they're coming from. It is the submitter's choice, published under the submitter's terms, and only on that pull request.
469- I merge locally, as now.
470- Import the GitLab merge requests in `~/code/gitlab-export/*/merge_requests.json`.
471
472## Provenance
473
474- **AI share.** Count commits with an AI `Co-Authored-By:` trailer, per repo and site-wide.
475- **Transcripts** (Claude Code session → rendered page):
476 - opt-in, one conversation at a time
477 - with a redaction pass (paths, email addresses, tokens) and my review before anything is published
478 - each published transcript has a consent record: who gave permission, when, for which conversation
479 - `fw transcript withdraw <id>` removes it from the sites
480 - transcripts are **not** MIT; they say "shown with permission, all rights reserved"
481- **First candidate:** the Firefox bug-fix conversation. Permission was given 2026-10-26 and can be withdrawn.
482 It is in a different session, so it still goes through redaction and review first.
483- **My own conversations** (like the ones that built this tool) are not published.