fuzzyworld

fuzzyworld-tools / files / docs / operations.md

operations.md

7.8 KB · history · blame

Operations: installing, layout and jobs

Requirements

Machine Needs
DreamHost git, Python 3.12, flock, setsid
Laptop Node ≥ 18, git, rsync, ssh with the dh alias
TrueNAS (runner) Docker, Python 3 with PyYAML (both part of TrueNAS SCALE), and the truenas ssh alias on the laptop
Laptop as a runner (optional) Docker (Docker Desktop, OrbStack, ...) and Python 3; the installer makes a venv with PyYAML

Install and update

server/install.sh              # copy server/ to dh, install site files and hooks
server/install.sh --publish    # ...and republish every repo
runner/install.sh              # CI runner on TrueNAS: copy it, make its ssh key, add the cron job
runner/install.sh --authorize  # ...and allow that key on DreamHost (runner-api only)
runner/install-laptop.sh       # the laptop as a polite second runner (launchd, Docker, its own venv)
vscode-web/install.sh          # VS Code in the browser at apps.fuzzyworld.net/vscode/

install.sh copies server/ from the working copy, not from a commit, so uncommitted changes there go live too.

npm run deploy (the usual way)

Script Runs
npm run deploy bin/deploy-tools: install the server and republish every repo, but only if server/ changed since the last deploy
npm run deploy-status bin/deploy-tools status: what is deployed where, and what is pending
npm run list-repos bin/fw list

bin/deploy-tools [server|runner|laptop-runner|vscode|all] [--force] wraps the install scripts with a ledger kept on the server, ~/fw/state/deployed.json.

Target Ships Runs
server server/ server/install.sh --publish (includes the republish, since pages are generated by the installed gen.py)
runner runner/ runner/install.sh
laptop-runner runner/ runner/install-laptop.sh (per machine)
vscode vscode-web/ vscode-web/install.sh

Layout on DreamHost

Sites and code

Path Contents
~/git/<name>.git canonical bare repos (push here; not web-visible)
~/code.fuzzyworld.net/<name>/ generated pages for public repos
~/code.fuzzyworld.net/<name>.git public clone copy (dumb HTTP), refreshed on push
~/apps.fuzzyworld.net/<app>/ deployed apps (fw deploy); index.html at the root is generated
~/apps.fuzzyworld.net/_shots/<app>/ screenshots of each app, uploaded by runners (shots.json lists them)
~/apps.fuzzyworld.net/vscode/ VS Code in the browser (not listed on the apps index)
~/writing.fuzzyworld.net/ the blog's built site, uploaded by fw blog deploy from the laptop (never edited there)

Tools and state (outside every web root)

Path Contents
~/fw/tools/server/ this repo's server/ directory, installed by server/install.sh
~/bin/fw symlink to ~/fw/tools/server/fw
~/fw/state/ per-repo page manifests and locks; deployed.json is the deploy ledger
~/fw/logs/<name>.log output of each push's publish run
~/fw/pylib/ Python-Markdown 3.7 (README rendering) and Pygments 2.21.0 (syntax highlighting), pip install --target; install.sh adds both (the landing page needs Markdown)
~/fw/queue/pending/, running/ CI jobs, one JSON file each (ci.md). running/<job>.beat is the runner's heartbeat; declined/ holds the runners that gave a job back
~/fw/runners/<runner> one file per runner: its tags, and (by its time) when it last checked in
~/fw/ci/<name>/<sha>/ CI results uploaded by runners: pipeline.json and one log per job
~/fw/issues/ bug-report issues: stored reports, secret, imap.json, locks (issues.md)
~/fw/comments/ blog comments state and imap.json (blog.md)

Scheduled jobs

Jobs live in server/jobs/. install.sh writes server/jobs/crontab into the DreamHost crontab, between # fw:begin and # fw:end; it replaces only that block. Cron wakes each job hourly. DreamHost's clock is Pacific time, so each job checks its own UTC schedule, and daylight saving can't shift it.

Job Schedule What it does Log
north-star-cleanup.sh Tuesdays 06:00 UTC (was the GitLab schedule) updates ~/fw/jobs/north-star to main, then runs scripts/clean-old-firebase.sh --days=30 as a preview and then --apply ~/fw/logs/cron-north-star-cleanup.log
comments-ingest.sh hourly comments.py ingest: turns new mail in the comments mailbox into blog comments (quiet unless something happened) ~/fw/logs/cron-comments-ingest.log
issues-ingest.sh hourly issues.py ingest: turns new mail in the bugs mailbox into issues and republishes the touched repos (quiet unless something happened) ~/fw/logs/cron-issues-ingest.log

Run one by hand: ssh dh '~/fw/tools/server/jobs/north-star-cleanup.sh --now --preview-only' (without --preview-only for the real thing).

Secrets are in ~/.config/<project>.env on DreamHost (mode 600, never in a repo or the web root). north-star.env holds FIREBASE_TOKEN.

Node on DreamHost

The system Node is 18. Jobs use Node 22 from ~/fw/node (the official linux-x64 build; checksum-verified), always with NODE_OPTIONS=--jitless: DreamHost's grsec kernel refuses executable memory, so V8's JIT crashes without it. npm installs use --ignore-scripts, because native addons can't be built there. jq 1.7.1 (the official static build) is in ~/bin.

VS Code in the browser

apps.fuzzyworld.net/vscode/?repo=<name> opens a public repo in VS Code. It is linked from each summary page.

Parameter Effect
repo=<name> the repo
branch=<branch> an unmerged branch
file=<path> open a file
file=<path>:<line> jump to a line

What it is: the open-source VS Code web build (the vscode-web npm package, 1.91.1, MIT) plus vscode-web/fw-fs/, an extension that reads files.json and objects/<sha> from code.fuzzyworld.net. Everything is static files.

The plain "Open in VS Code" link (vscode://vscode.git/clone?url=…) clones the repo into the visitor's own VS Code.

Source
1# Operations: installing, layout and jobs
2
3## Requirements
4
5| Machine | Needs |
6|---|---|
7| DreamHost | git, Python 3.12, `flock`, `setsid` |
8| Laptop | Node ≥ 18, git, rsync, ssh with the `dh` alias |
9| TrueNAS (runner) | Docker, Python 3 with PyYAML (both part of TrueNAS SCALE), and the `truenas` ssh alias on the laptop |
10| Laptop as a runner (optional) | Docker (Docker Desktop, OrbStack, ...) and Python 3; the installer makes a venv with PyYAML |
11
12## Install and update
13
14```bash
15server/install.sh # copy server/ to dh, install site files and hooks
16server/install.sh --publish # ...and republish every repo
17runner/install.sh # CI runner on TrueNAS: copy it, make its ssh key, add the cron job
18runner/install.sh --authorize # ...and allow that key on DreamHost (runner-api only)
19runner/install-laptop.sh # the laptop as a polite second runner (launchd, Docker, its own venv)
20vscode-web/install.sh # VS Code in the browser at apps.fuzzyworld.net/vscode/
21```
22
23`install.sh` copies `server/` from the working copy, not from a commit, so uncommitted changes there go live too.
24
25### `npm run deploy` (the usual way)
26
27| Script | Runs |
28|---|---|
29| `npm run deploy` | `bin/deploy-tools`: install the server and republish every repo, but only if `server/` changed since the last deploy |
30| `npm run deploy-status` | `bin/deploy-tools status`: what is deployed where, and what is pending |
31| `npm run list-repos` | `bin/fw list` |
32
33`bin/deploy-tools [server|runner|laptop-runner|vscode|all] [--force]` wraps the install scripts with a **ledger**
34kept on the server, `~/fw/state/deployed.json`.
35
36- **Targets** are declared in this repo's `package.json`: `"fuzzyworld": {"deploy": {"<target>": {"paths": [folders it ships], "command": [script, args], "description": "...", "perMachine": true}}}`.
37
38 | Target | Ships | Runs |
39 |---|---|---|
40 | `server` | `server/` | `server/install.sh --publish` (includes the republish, since pages are generated by the installed `gen.py`) |
41 | `runner` | `runner/` | `runner/install.sh` |
42 | `laptop-runner` | `runner/` | `runner/install-laptop.sh` (per machine) |
43 | `vscode` | `vscode-web/` | `vscode-web/install.sh` |
44
45- **The ledger** records, for each `<repo>/<target>`, the commit and a content hash of the folders that target ships. A target is redeployed only when that hash differs. So uncommitted edits count, unrelated commits (README, tests) do not, and committing the files you already deployed does not trigger a second deploy. Gitignored files in those folders are not hashed.
46- Ledger keys used to be bare (`server`, `laptop-runner@<hostname>`). `bin/deploy-tools` renames such a key to `<repo>/<target>` in memory when it loads the ledger (an existing new key wins), and the next successful deploy saves the result. Times in the ledger are the laptop's local time.
47- The ledger is written only after the script succeeds, and any laptop sees the same state. The exception is `laptop-runner`, which is installed per machine and so is recorded per hostname (key `<repo>/laptop-runner@<hostname>`).
48- `fw deploys` and the [admin panel](admin.md#deploy-deploy-read-only-for-now) read the same ledger.
49- For flags `deploy-tools` does not pass (`--authorize`, `--remove`, `--out`) run the install script directly. To regenerate pages without installing, run `bin/fw publish-all`.
50
51## Layout on DreamHost
52
53### Sites and code
54
55| Path | Contents |
56|---|---|
57| `~/git/<name>.git` | canonical bare repos (push here; not web-visible) |
58| `~/code.fuzzyworld.net/<name>/` | generated pages for public repos |
59| `~/code.fuzzyworld.net/<name>.git` | public clone copy (dumb HTTP), refreshed on push |
60| `~/apps.fuzzyworld.net/<app>/` | deployed apps (`fw deploy`); `index.html` at the root is generated |
61| `~/apps.fuzzyworld.net/_shots/<app>/` | screenshots of each app, uploaded by runners (`shots.json` lists them) |
62| `~/apps.fuzzyworld.net/vscode/` | VS Code in the browser (not listed on the apps index) |
63| `~/writing.fuzzyworld.net/` | the blog's built site, uploaded by `fw blog deploy` from the laptop (never edited there) |
64
65### Tools and state (outside every web root)
66
67| Path | Contents |
68|---|---|
69| `~/fw/tools/server/` | this repo's `server/` directory, installed by `server/install.sh` |
70| `~/bin/fw` | symlink to `~/fw/tools/server/fw` |
71| `~/fw/state/` | per-repo page manifests and locks; `deployed.json` is the deploy ledger |
72| `~/fw/logs/<name>.log` | output of each push's publish run |
73| `~/fw/pylib/` | Python-Markdown 3.7 (README rendering) and Pygments 2.21.0 (syntax highlighting), `pip install --target`; `install.sh` adds both (the landing page needs Markdown) |
74| `~/fw/queue/pending/`, `running/` | CI jobs, one JSON file each ([ci.md](ci.md)). `running/<job>.beat` is the runner's heartbeat; `declined/` holds the runners that gave a job back |
75| `~/fw/runners/<runner>` | one file per runner: its tags, and (by its time) when it last checked in |
76| `~/fw/ci/<name>/<sha>/` | CI results uploaded by runners: `pipeline.json` and one log per job |
77| `~/fw/issues/` | bug-report issues: stored reports, `secret`, `imap.json`, locks ([issues.md](issues.md)) |
78| `~/fw/comments/` | blog comments state and `imap.json` ([blog.md](blog.md#comments-by-email)) |
79
80## Scheduled jobs
81
82Jobs live in `server/jobs/`. `install.sh` writes `server/jobs/crontab` into the DreamHost crontab, between
83`# fw:begin` and `# fw:end`; it replaces only that block. Cron wakes each job hourly. DreamHost's clock is Pacific
84time, so **each job checks its own UTC schedule**, and daylight saving can't shift it.
85
86| Job | Schedule | What it does | Log |
87|---|---|---|---|
88| `north-star-cleanup.sh` | Tuesdays 06:00 UTC (was the GitLab schedule) | updates `~/fw/jobs/north-star` to `main`, then runs `scripts/clean-old-firebase.sh --days=30` as a preview and then `--apply` | `~/fw/logs/cron-north-star-cleanup.log` |
89| `comments-ingest.sh` | hourly | `comments.py ingest`: turns new mail in the comments mailbox into blog comments (quiet unless something happened) | `~/fw/logs/cron-comments-ingest.log` |
90| `issues-ingest.sh` | hourly | `issues.py ingest`: turns new mail in the bugs mailbox into issues and republishes the touched repos (quiet unless something happened) | `~/fw/logs/cron-issues-ingest.log` |
91
92Run one by hand: `ssh dh '~/fw/tools/server/jobs/north-star-cleanup.sh --now --preview-only'` (without
93`--preview-only` for the real thing).
94
95**Secrets** are in `~/.config/<project>.env` on DreamHost (mode 600, never in a repo or the web root).
96`north-star.env` holds `FIREBASE_TOKEN`.
97
98## Node on DreamHost
99
100The system Node is 18. Jobs use Node 22 from `~/fw/node` (the official linux-x64 build; checksum-verified), always
101with `NODE_OPTIONS=--jitless`: DreamHost's grsec kernel refuses executable memory, so V8's JIT crashes without it.
102npm installs use `--ignore-scripts`, because native addons can't be built there. `jq` 1.7.1 (the official static
103build) is in `~/bin`.
104
105## VS Code in the browser
106
107`apps.fuzzyworld.net/vscode/?repo=<name>` opens a public repo in VS Code. It is linked from each summary page.
108
109| Parameter | Effect |
110|---|---|
111| `repo=<name>` | the repo |
112| `branch=<branch>` | an unmerged branch |
113| `file=<path>` | open a file |
114| `file=<path>:<line>` | jump to a line |
115
116What it is: the open-source VS Code web build (the `vscode-web` npm package, 1.91.1, MIT) plus `vscode-web/fw-fs/`,
117an extension that reads `files.json` and `objects/<sha>` from code.fuzzyworld.net. Everything is static files.
118
119- **Works:** quick open (Ctrl/Cmd+P), and search across files (text files up to 256 KB).
120- **Read-only:** nothing can be changed; clone to edit.
121- **Not yet:** Markdown preview and other webviews (they need a separate domain per view), and project-wide TypeScript IntelliSense (turned off; syntax colors work).
122
123The plain "Open in VS Code" link (`vscode://vscode.git/clone?url=…`) clones the repo into the visitor's own VS Code.