fuzzyworld-tools / files / docs / operations.md

operations.md

Blame: the commit that last changed each line · file · history

`bb7e1f57` 2026-11-06 Pavel Safronov	1	# Operations: installing, layout and jobs
	2
	3	## Requirements
	4
	5	\| Machine \| Needs \|
	6	\|---\|---\|
	7	\| DreamHost \| git, Python 3.12, `flock`, `setsid` \|
	8	\| Laptop \| Node ≥ 18, git, rsync, ssh with the `dh` alias \|
	9	\| TrueNAS (runner) \| Docker, Python 3 with PyYAML (both part of TrueNAS SCALE), and the `truenas` ssh alias on the laptop \|
	10	\| Laptop as a runner (optional) \| Docker (Docker Desktop, OrbStack, ...) and Python 3; the installer makes a venv with PyYAML \|
	11
	12	## Install and update
	13
	14	```bash
	15	server/install.sh # copy server/ to dh, install site files and hooks
	16	server/install.sh --publish # ...and republish every repo
	17	runner/install.sh # CI runner on TrueNAS: copy it, make its ssh key, add the cron job
	18	runner/install.sh --authorize # ...and allow that key on DreamHost (runner-api only)
	19	runner/install-laptop.sh # the laptop as a polite second runner (launchd, Docker, its own venv)
	20	vscode-web/install.sh # VS Code in the browser at apps.fuzzyworld.net/vscode/
	21	```
	22
	23	`install.sh` copies `server/` from the working copy, not from a commit, so uncommitted changes there go live too.
	24
	25	### `npm run deploy` (the usual way)
	26
	27	\| Script \| Runs \|
	28	\|---\|---\|
	29	\| `npm run deploy` \| `bin/deploy-tools`: install the server and republish every repo, but only if `server/` changed since the last deploy \|
	30	\| `npm run deploy-status` \| `bin/deploy-tools status`: what is deployed where, and what is pending \|
	31	\| `npm run list-repos` \| `bin/fw list` \|
	32
	33	`bin/deploy-tools [server\|runner\|laptop-runner\|vscode\|all] [--force]` wraps the install scripts with a ledger
	34	kept on the server, `~/fw/state/deployed.json`.
	35
	36	- Targets are declared in this repo's `package.json`: `"fuzzyworld": {"deploy": {"<target>": {"paths": [folders it ships], "command": [script, args], "description": "...", "perMachine": true}}}`.
	37
	38	\| Target \| Ships \| Runs \|
	39	\|---\|---\|---\|
	40	\| `server` \| `server/` \| `server/install.sh --publish` (includes the republish, since pages are generated by the installed `gen.py`) \|
	41	\| `runner` \| `runner/` \| `runner/install.sh` \|
	42	\| `laptop-runner` \| `runner/` \| `runner/install-laptop.sh` (per machine) \|
	43	\| `vscode` \| `vscode-web/` \| `vscode-web/install.sh` \|
	44
	45	- The ledger records, for each `<repo>/<target>`, the commit and a content hash of the folders that target ships. A target is redeployed only when that hash differs. So uncommitted edits count, unrelated commits (README, tests) do not, and committing the files you already deployed does not trigger a second deploy. Gitignored files in those folders are not hashed.
`ab35fc22` 2026-11-07 Pavel Safronov	46	- Ledger keys used to be bare (`server`, `laptop-runner@<hostname>`). `bin/deploy-tools` renames such a key to `<repo>/<target>` in memory when it loads the ledger (an existing new key wins), and the next successful deploy saves the result. Times in the ledger are the laptop's local time.
`bb7e1f57` 2026-11-06 Pavel Safronov	47	- The ledger is written only after the script succeeds, and any laptop sees the same state. The exception is `laptop-runner`, which is installed per machine and so is recorded per hostname (key `<repo>/laptop-runner@<hostname>`).
	48	- `fw deploys` and the [admin panel](admin.md#deploy-deploy-read-only-for-now) read the same ledger.
	49	- For flags `deploy-tools` does not pass (`--authorize`, `--remove`, `--out`) run the install script directly. To regenerate pages without installing, run `bin/fw publish-all`.
	50
	51	## Layout on DreamHost
	52
	53	### Sites and code
	54
	55	\| Path \| Contents \|
	56	\|---\|---\|
	57	\| `~/git/<name>.git` \| canonical bare repos (push here; not web-visible) \|
	58	\| `~/code.fuzzyworld.net/<name>/` \| generated pages for public repos \|
	59	\| `~/code.fuzzyworld.net/<name>.git` \| public clone copy (dumb HTTP), refreshed on push \|
	60	\| `~/apps.fuzzyworld.net/<app>/` \| deployed apps (`fw deploy`); `index.html` at the root is generated \|
	61	\| `~/apps.fuzzyworld.net/_shots/<app>/` \| screenshots of each app, uploaded by runners (`shots.json` lists them) \|
	62	\| `~/apps.fuzzyworld.net/vscode/` \| VS Code in the browser (not listed on the apps index) \|
	63	\| `~/writing.fuzzyworld.net/` \| the blog's built site, uploaded by `fw blog deploy` from the laptop (never edited there) \|
	64
	65	### Tools and state (outside every web root)
	66
	67	\| Path \| Contents \|
	68	\|---\|---\|
	69	\| `~/fw/tools/server/` \| this repo's `server/` directory, installed by `server/install.sh` \|
	70	\| `~/bin/fw` \| symlink to `~/fw/tools/server/fw` \|
	71	\| `~/fw/state/` \| per-repo page manifests and locks; `deployed.json` is the deploy ledger \|
	72	\| `~/fw/logs/<name>.log` \| output of each push's publish run \|
	73	\| `~/fw/pylib/` \| Python-Markdown 3.7 (README rendering) and Pygments 2.21.0 (syntax highlighting), `pip install --target`; `install.sh` adds both (the landing page needs Markdown) \|
	74	\| `~/fw/queue/pending/`, `running/` \| CI jobs, one JSON file each ([ci.md](ci.md)). `running/<job>.beat` is the runner's heartbeat; `declined/` holds the runners that gave a job back \|
	75	\| `~/fw/runners/<runner>` \| one file per runner: its tags, and (by its time) when it last checked in \|
	76	\| `~/fw/ci/<name>/<sha>/` \| CI results uploaded by runners: `pipeline.json` and one log per job \|
	77	\| `~/fw/issues/` \| bug-report issues: stored reports, `secret`, `imap.json`, locks ([issues.md](issues.md)) \|
	78	\| `~/fw/comments/` \| blog comments state and `imap.json` ([blog.md](blog.md#comments-by-email)) \|
	79
	80	## Scheduled jobs
	81
	82	Jobs live in `server/jobs/`. `install.sh` writes `server/jobs/crontab` into the DreamHost crontab, between
	83	`# fw:begin` and `# fw:end`; it replaces only that block. Cron wakes each job hourly. DreamHost's clock is Pacific
	84	time, so each job checks its own UTC schedule, and daylight saving can't shift it.
	85
	86	\| Job \| Schedule \| What it does \| Log \|
	87	\|---\|---\|---\|---\|
	88	\| `north-star-cleanup.sh` \| Tuesdays 06:00 UTC (was the GitLab schedule) \| updates `~/fw/jobs/north-star` to `main`, then runs `scripts/clean-old-firebase.sh --days=30` as a preview and then `--apply` \| `~/fw/logs/cron-north-star-cleanup.log` \|
	89	\| `comments-ingest.sh` \| hourly \| `comments.py ingest`: turns new mail in the comments mailbox into blog comments (quiet unless something happened) \| `~/fw/logs/cron-comments-ingest.log` \|
	90	\| `issues-ingest.sh` \| hourly \| `issues.py ingest`: turns new mail in the bugs mailbox into issues and republishes the touched repos (quiet unless something happened) \| `~/fw/logs/cron-issues-ingest.log` \|
	91
	92	Run one by hand: `ssh dh '~/fw/tools/server/jobs/north-star-cleanup.sh --now --preview-only'` (without
	93	`--preview-only` for the real thing).
	94
	95	Secrets are in `~/.config/<project>.env` on DreamHost (mode 600, never in a repo or the web root).
	96	`north-star.env` holds `FIREBASE_TOKEN`.
	97
	98	## Node on DreamHost
	99
	100	The system Node is 18. Jobs use Node 22 from `~/fw/node` (the official linux-x64 build; checksum-verified), always
	101	with `NODE_OPTIONS=--jitless`: DreamHost's grsec kernel refuses executable memory, so V8's JIT crashes without it.
	102	npm installs use `--ignore-scripts`, because native addons can't be built there. `jq` 1.7.1 (the official static
	103	build) is in `~/bin`.
	104
	105	## VS Code in the browser
	106
	107	`apps.fuzzyworld.net/vscode/?repo=<name>` opens a public repo in VS Code. It is linked from each summary page.
	108
	109	\| Parameter \| Effect \|
	110	\|---\|---\|
	111	\| `repo=<name>` \| the repo \|
	112	\| `branch=<branch>` \| an unmerged branch \|
	113	\| `file=<path>` \| open a file \|
	114	\| `file=<path>:<line>` \| jump to a line \|
	115
	116	What it is: the open-source VS Code web build (the `vscode-web` npm package, 1.91.1, MIT) plus `vscode-web/fw-fs/`,
	117	an extension that reads `files.json` and `objects/<sha>` from code.fuzzyworld.net. Everything is static files.
	118
	119	- Works: quick open (Ctrl/Cmd+P), and search across files (text files up to 256 KB).
	120	- Read-only: nothing can be changed; clone to edit.
	121	- Not yet: Markdown preview and other webviews (they need a separate domain per view), and project-wide TypeScript IntelliSense (turned off; syntax colors work).
	122
	123	The plain "Open in VS Code" link (`vscode://vscode.git/clone?url=…`) clones the repo into the visitor's own VS Code.