CI

The same pattern as GitLab runners: a runner pulls jobs, runs them in containers, and reports back. There is no coordinator service; the coordinator is files on DreamHost.

How a job flows

1. Queue. post-receive queues a job (~/fw/queue/pending/<id>.json) for each pushed branch or tag whose commit has a .fw-ci.yml (or .gitlab-ci.yml) or a package.json, unless fuzzyworld.ci is off. fw ci <name> queues one by hand.
2. Claim. A runner (runner/fw-runner.py; on TrueNAS from cron, on a laptop from launchd, every 2 minutes) runs ssh dh claim <runner> <tags>. The server moves the job into running/ with one rename, which is atomic, so two runners never take the same job.
3. Heartbeat. While a job runs, the runner sends a heartbeat every 30 seconds (beat <job> <runner>). A job whose runner has been quiet for about 2 minutes (lid closed, network gone) goes back to pending/, and a runner that lost its job this way can't mark it done. A runner that sends no heartbeats (an old one) is given 2 hours.
4. Run. The runner clones the commit and runs each job in a fresh Docker container (--cpus 4 --memory 4g, 30-minute timeout unless the job sets timeout). - All jobs of a pipeline share one workspace, so artifacts are already there for later stages. - Inside jobs, origin is the read-only https clone URL (https://code.fuzzyworld.net/<name>.git), as on GitLab. The runner's ssh key never enters a container.
5. Report. Logs and pipeline.json go to ~/fw/ci/<name>/<sha>/. While a job runs, the runner marks it running and uploads its log every 15 seconds; each upload rebuilds that repo's CI pages and the dashboard (fw ci-pages). When the pipeline finishes, the repo's pages are republished, showing ✓/✗ on the log, the summary and /<name>/ci/.

With no runner at all, jobs stay in pending/ and the CI dashboard says "Waiting for a runner".

Tags

A runner has tags (FW_RUNNER_TAGS, default docker,chrome); a job can ask for more with GitLab's tags:.

• After cloning, a runner that lacks a tag the pipeline needs gives the job back (release) before running anything. The server then never offers that job to that runner again.
• Screenshots need chrome.
• The tag secrets means "this repo's secrets": a job tagged secrets on the default branch or a tag runs only on a runner that has secrets/<repo>.env.
• A job nobody can run waits in the queue.

The runner's key

The runner's ssh key is limited in authorized_keys to server/runner-api (restrict,command="…/runner-api"). It can claim jobs, fetch repos and upload results. It can't get a shell.

Watching CI

• Web: code.fuzzyworld.net/ci/ shows what is running (each job ✓/✗/● with its time), what is queued and the last 50 results. Private repos are never named there, only counted.
• Terminal: fw ci-watch shows everything, private repos included (running jobs with each job's state, the queue, the last 5 results), and ends when nothing is queued or running.
• Self-refresh without JavaScript: while anything is queued or running, the dashboard and the running pipeline's page reload every 15 seconds (<meta http-equiv="refresh">). A thin bar along the top empties as the reload nears, and "Next reload in N s." counts down beside the reload note. Both are CSS only. The countdown needs a browser with @property (otherwise only the bar shows), and the bar is hidden for prefers-reduced-motion.

Job definitions: .fw-ci.yml

A file at the repo root, in a subset of GitLab CI's YAML format, so an old .gitlab-ci.yml works after a rename (git mv .gitlab-ci.yml .fw-ci.yml). Until a repo is renamed, the runner still reads .gitlab-ci.yml; .fw-ci.yml wins if both exist.

Anything else (for example include, services, release, trigger, parallel) makes that job fail with a message naming the key; nothing is silently skipped. manual jobs are listed but not run.

• The usual CI_* variables are set (CI_COMMIT_SHA, CI_COMMIT_BRANCH/CI_COMMIT_TAG, CI_COMMIT_REF_SLUG, CI_DEFAULT_BRANCH, CI_PIPELINE_SOURCE=push, ...), plus FUZZYWORLD_CI=true.
• Jobs without an image use node:22-bookworm.
• A pages job runs its build as a check; publishing is fw deploy's job.

Without a pipeline file

• An app (package.json with fuzzyworld.app) gets one job, fw deploy --here --dry-run (install, prebuild, checks, build, output checks).
• Other node repos get their typecheck/lint/test scripts.

Check what a checkout would run

runner/fw-runner.py --plan <dir> [refs/heads/<branch>]

Secrets

<runner dir>/secrets/<repo>.env on each runner that should deploy (TrueNAS: /mnt/main-2025/fw-runner/secrets/; laptop: ~/.fw-runner/secrets/). The file is passed (--env-file) only to jobs on the default branch and tags.

• A runner without the file still runs that repo's other jobs. Tag a deploy job secrets to keep it for a runner that has the file.
• Without any secrets, jobs that deploy elsewhere (for example dreamhost's rsync, engye's store uploads) fail.
• Anything on the laptop is as safe as the laptop (FileVault on). If it is lost, rotate the secrets.

The laptop as a runner

runner/install-laptop.sh (or bin/deploy-tools laptop-runner) installs a launchd job that runs every 2 minutes with FW_RUNNER_POLITE=1. A polite runner claims a job only when:

• the Mac is on AC power,
• at least FW_RUNNER_MIN_IDLE (default 50) percent of the CPU is idle, and
• Docker is running.

It checks before each job, never during one. Its containers get a low CPU share (FW_RUNNER_CPUS, default 2 there, is the real limit).

It assumes the laptop is on for a few hours a day, which is enough when TrueNAS isn't there.