fuzzyworld-tools / files / docs / ci.md
ci.md
bb7e1f57 2026-11-06 Pavel Safronov | 1 | # CI |
| 2 | ||
| 3 | The same pattern as GitLab runners: a runner pulls jobs, runs them in containers, and reports back. There is no | |
| 4 | coordinator service; the coordinator is files on DreamHost. | |
| 5 | ||
| 6 | ## How a job flows | |
| 7 | ||
| 8 | 1. **Queue.** `post-receive` queues a job (`~/fw/queue/pending/<id>.json`) for each pushed branch or tag whose commit has a `.fw-ci.yml` (or `.gitlab-ci.yml`) or a `package.json`, unless `fuzzyworld.ci` is `off`. `fw ci <name>` queues one by hand. | |
| 9 | 2. **Claim.** A runner (`runner/fw-runner.py`; on TrueNAS from cron, on a laptop from launchd, every 2 minutes) runs `ssh dh claim <runner> <tags>`. The server moves the job into `running/` with one `rename`, which is atomic, so two runners never take the same job. | |
| 10 | 3. **Heartbeat.** While a job runs, the runner sends a heartbeat every 30 seconds (`beat <job> <runner>`). A job whose runner has been quiet for about 2 minutes (lid closed, network gone) goes back to `pending/`, and a runner that lost its job this way can't mark it done. A runner that sends no heartbeats (an old one) is given 2 hours. | |
| 11 | 4. **Run.** The runner clones the commit and runs each job in a fresh Docker container (`--cpus 4 --memory 4g`, 30-minute timeout unless the job sets `timeout`). | |
| 12 | - All jobs of a pipeline share one workspace, so artifacts are already there for later stages. | |
| 13 | - Inside jobs, `origin` is the read-only https clone URL (`https://code.fuzzyworld.net/<name>.git`), as on GitLab. The runner's ssh key never enters a container. | |
| 14 | 5. **Report.** Logs and `pipeline.json` go to `~/fw/ci/<name>/<sha>/`. While a job runs, the runner marks it running and uploads its log every 15 seconds; each upload rebuilds that repo's CI pages and the dashboard (`fw ci-pages`). When the pipeline finishes, the repo's pages are republished, showing ✓/✗ on the log, the summary and `/<name>/ci/`. | |
| 15 | ||
| 16 | With no runner at all, jobs stay in `pending/` and the CI dashboard says "Waiting for a runner". | |
| 17 | ||
| 18 | ### Tags | |
| 19 | ||
| 20 | A runner has tags (`FW_RUNNER_TAGS`, default `docker,chrome`); a job can ask for more with GitLab's `tags:`. | |
| 21 | ||
| 22 | - After cloning, a runner that lacks a tag the pipeline needs gives the job back (`release`) before running anything. The server then never offers that job to that runner again. | |
| 23 | - Screenshots need `chrome`. | |
| 24 | - The tag `secrets` means "this repo's secrets": a job tagged `secrets` on the default branch or a tag runs only on a runner that has `secrets/<repo>.env`. | |
| 25 | - A job nobody can run waits in the queue. | |
| 26 | ||
| 27 | ### The runner's key | |
| 28 | ||
| 29 | The runner's ssh key is limited in `authorized_keys` to `server/runner-api` (`restrict,command="…/runner-api"`). It can claim jobs, fetch repos and upload results. It can't get a shell. | |
| 30 | ||
| 31 | ## Watching CI | |
| 32 | ||
| 33 | - **Web:** `code.fuzzyworld.net/ci/` shows what is running (each job ✓/✗/● with its time), what is queued and the last 50 results. Private repos are never named there, only counted. | |
| 34 | - **Terminal:** `fw ci-watch` shows everything, private repos included (running jobs with each job's state, the queue, the last 5 results), and ends when nothing is queued or running. | |
| 35 | - **Self-refresh without JavaScript:** while anything is queued or running, the dashboard and the running pipeline's page reload every 15 seconds (`<meta http-equiv="refresh">`). A thin bar along the top empties as the reload nears, and "Next reload in N s." counts down beside the reload note. Both are CSS only. The countdown needs a browser with `@property` (otherwise only the bar shows), and the bar is hidden for `prefers-reduced-motion`. | |
| 36 | ||
| 37 | ## Job definitions: `.fw-ci.yml` | |
| 38 | ||
| 39 | A file at the repo root, in a subset of GitLab CI's YAML format, so an old `.gitlab-ci.yml` works after a rename | |
| 40 | (`git mv .gitlab-ci.yml .fw-ci.yml`). Until a repo is renamed, the runner still reads `.gitlab-ci.yml`; `.fw-ci.yml` | |
| 41 | wins if both exist. | |
| 42 | ||
| 43 | | Supported | Keys | | |
| 44 | |---|---| | |
| 45 | | Top level | `stages`, `variables`, `image` (string or `name:`), `default:` (image, before_script, after_script), `workflow:rules` | | |
| 46 | | Reuse | `extends` (hidden jobs) | | |
| 47 | | Per job | `stage`, `image`, `script`, `before_script`, `after_script`, `variables`, `rules` (`if`, `when`, `exists`; `changes` always matches), `only`/`except` (refs), `when` (`on_success`, `on_failure`, `always`, `manual`, `never`), `allow_failure`, `timeout`, `needs` (ordering comes from stages), `tags` | | |
| 48 | | Ignored (change nothing here) | `artifacts`, `cache`, `dependencies`, `environment`, `interruptible`, `retry`, `coverage`, `resource_group` | | |
| 49 | ||
| 50 | **Anything else** (for example `include`, `services`, `release`, `trigger`, `parallel`) makes that job fail with a | |
| 51 | message naming the key; nothing is silently skipped. `manual` jobs are listed but not run. | |
| 52 | ||
| 53 | - The usual `CI_*` variables are set (`CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`/`CI_COMMIT_TAG`, `CI_COMMIT_REF_SLUG`, `CI_DEFAULT_BRANCH`, `CI_PIPELINE_SOURCE=push`, ...), plus `FUZZYWORLD_CI=true`. | |
| 54 | - Jobs without an image use `node:22-bookworm`. | |
| 55 | - A `pages` job runs its build as a check; publishing is `fw deploy`'s job. | |
| 56 | ||
| 57 | ### Without a pipeline file | |
| 58 | ||
| 59 | - An app (`package.json` with `fuzzyworld.app`) gets one job, `fw deploy --here --dry-run` (install, prebuild, checks, build, output checks). | |
| 60 | - Other node repos get their `typecheck`/`lint`/`test` scripts. | |
| 61 | ||
| 62 | ### Check what a checkout would run | |
| 63 | ||
| 64 | ```bash | |
| 65 | runner/fw-runner.py --plan <dir> [refs/heads/<branch>] | |
| 66 | ``` | |
| 67 | ||
| 68 | ## Secrets | |
| 69 | ||
| 70 | `<runner dir>/secrets/<repo>.env` on each runner that should deploy (TrueNAS: `/mnt/main-2025/fw-runner/secrets/`; | |
| 71 | laptop: `~/.fw-runner/secrets/`). The file is passed (`--env-file`) only to jobs on the default branch and tags. | |
| 72 | ||
| 73 | - A runner without the file still runs that repo's other jobs. Tag a deploy job `secrets` to keep it for a runner that has the file. | |
| 74 | - Without any secrets, jobs that deploy elsewhere (for example `dreamhost`'s rsync, engye's store uploads) fail. | |
| 75 | - Anything on the laptop is as safe as the laptop (FileVault on). If it is lost, rotate the secrets. | |
| 76 | ||
| 77 | ## The laptop as a runner | |
| 78 | ||
| 79 | `runner/install-laptop.sh` (or `bin/deploy-tools laptop-runner`) installs a launchd job that runs every 2 minutes | |
| 80 | with `FW_RUNNER_POLITE=1`. A polite runner claims a job only when: | |
| 81 | ||
| 82 | - the Mac is on AC power, | |
| 83 | - at least `FW_RUNNER_MIN_IDLE` (default 50) percent of the CPU is idle, and | |
| 84 | - Docker is running. | |
| 85 | ||
| 86 | It checks before each job, never during one. Its containers get a low CPU share (`FW_RUNNER_CPUS`, default 2 there, | |
| 87 | is the real limit). | |
| 88 | ||
| 89 | | Item | Where | | |
| 90 | |---|---| | |
| 91 | | Settings | `~/.fw-runner/runner.env` | | |
| 92 | | Log | `~/.fw-runner/runner.log` | | |
| 93 | | Allow its key on DreamHost | `runner/install-laptop.sh --authorize` | | |
| 94 | | Stop it | `runner/install-laptop.sh --remove` | | |
| 95 | ||
| 96 | It assumes the laptop is on for a few hours a day, which is enough when TrueNAS isn't there. |