fuzzyworld-tools / commit
Runners: tags, laptop runner when idle, job lease; deploy server first
PLAN.md | 33 +++++---- README.md | 42 ++++++++--- package.json | 1 + runner/fw-runner.py | 180 +++++++++++++++++++++++++++++++++++++++++++---- runner/install-laptop.sh | 70 ++++++++++++++++++ server/gen.py | 33 ++++++++- server/runner-api | 53 +++++++++++--- 7 files changed, 366 insertions(+), 46 deletions(-) diff --git a/PLAN.md b/PLAN.md index 9ab37d5..ad845c5 100644 --- a/PLAN.md +++ b/PLAN.md @@ -46,6 +46,8 @@ The order goes from cheap to expensive, and each step reuses what the one before 0. **Runners without TrueNAS** (decided 2026-10-28; see [Runners](#runners)): runner tags, the laptop as a runner when idle and plugged in, secrets in a folder on each runner, and "no runners" as a normal state. Goes first. + **Built 2026-10-28, not yet deployed** (needs `npm run update-server`, then `update-runner`, in that order: a new runner against the old server has every heartbeat refused; then + `update-laptop-runner --authorize` on the laptop; nothing has run on real machines yet). 1. **Feeds**: Atom per repo (commits, releases), for app deploys, and site-wide. Smallest step: the data is already gathered for the log pages and the apps index. The blog reuses it. 1b. **Blog** at `blog.fuzzyworld.net` (see [Blog](#blog)): a generator that turns a folder or repo of Markdown into a @@ -69,16 +71,22 @@ Goal (2026-10-28): TrueNAS becomes optional (🔌 plugin), and the laptop can do always-on machine; a small setup is a DreamHost account and a laptop. - **No runners is a normal state.** Jobs stay in `pending/` until a runner claims them (the queue is files, so this - already works). The dashboard says "waiting for a runner" instead of looking broken. -- **Laptop runner** (launchd, every 2 minutes), only when it will not hurt my work: on AC power, CPU mostly idle, - containers at low priority, no new claims once I get busy. Everything scheduled runs then. It assumes the laptop + already works). The dashboard says "waiting for a runner" instead of looking broken (a runner counts as gone after + 15 minutes without a claim; each claim writes `~/fw/runners/<name>`). +- **Laptop runner** (launchd via `runner/install-laptop.sh`, every 2 minutes), only when it will not hurt my work: on AC + power, CPU at least half idle, Docker running, containers at a low CPU share, no new claims once I get busy (checked + between jobs; a job already running is not stopped). Everything scheduled runs then. It assumes the laptop is on a few hours a day; that is enough for me. -- **Lease.** A claimed job needs a heartbeat (uploaded with the log). A job whose runner went quiet for ~2 minutes - (lid closed) goes back to `pending/` instead of waiting 2 hours. -- **Tags** (`docker`, `chrome`, `android`): a runner takes only jobs it can run. +- **Lease.** A claimed job needs a heartbeat (`beat`, every 30 seconds, from a thread in the runner). A job whose runner + went quiet for ~2 minutes (lid closed) goes back to `pending/` instead of waiting 2 hours. If that runner wakes up and + finishes anyway, its results are uploaded too, and the job may have run twice: accepted. Runners that send no + heartbeat (old versions) keep the 2-hour rule. +- **Tags** (`docker`, `chrome`, `android`): job `tags:` are honoured, and screenshots need `chrome`. The runner learns what + a pipeline needs only after cloning, so a runner that lacks a tag gives the job back (`release`); the server + remembers and does not offer it to that runner again. Simple, at the cost of one clone per declined job. - **Secrets.** Each runner has a folder (`…/secrets/<repo>.env`; on TrueNAS `/mnt/main-2025/fw-runner/secrets/`). - A runner without a repo's secrets leaves that repo's deploy jobs for one that has them. Only default-branch and - tag jobs get secrets. Put them on the laptop and TrueNAS is not needed. Laptop theft means rotating them (FileVault on). + A deploy job tagged `secrets` (a reserved tag: "this repo's secrets") runs only on a runner that has the file; no repo + uses it yet. Only default-branch and tag jobs get secrets. Put them on the laptop and TrueNAS is not needed. Laptop theft means rotating them (FileVault on). - **Stretch goal: a secure secrets API** (🔌): `runner-api` hands each job its secrets over ssh, per runner key, so no runner stores them. Wait for user requests or downtime. @@ -87,6 +95,8 @@ always-on machine; a small setup is a DreamHost account and a laptop. `blog.fuzzyworld.net` is WordPress today (theme Scrawl, 144 posts, 4 pages, 45 comments, 15 media files, 3 categories, 16 tags, one author). Goal: the same site, generated by this repo, with no WordPress. +**Rename requested 2026-10-31:** the writing blog moves to `writing.fuzzyworld.net` (queue item Q4 in `~/QUEUE.md`). This section still says `blog.fuzzyworld.net`; it is updated once Pavel confirms the new name and what the old site becomes. + **What is there** (read from its public API, 2026-10-28): - Posts at `/YYYY/MM/DD/<slug>/`, an RSS feed at `/feed/`, and monthly archives. Categories hardly matter ("Uncategorized" has 138 posts). Tags are the real structure (a story series is a tag, e.g. Dice, Walkabout, Writing Excuses). @@ -193,10 +203,8 @@ Decided 2026-10-28: **by email first, no accounts.** Google sign-in is a 🔌 pl - Found in review (2026-10-27): discussed, not done - Site pages: all now in `server/content/` (`gen.py content`), including `404.html`; `/how/` is `_how.html` (draft, not built) - `/how/` goes live: finish the draft and rename `server/content/_how.html` to `how.html` (`/why/` already says it is coming) - - CI runner tags (`docker`, `chrome`, `android`) and the laptop as a second runner (`fw-runner.py --once`): - designed, not built; there is one runner, and it takes every job - CI secrets: none are set, so jobs that deploy elsewhere fail (`dreamhost`'s rsync, engye's store uploads). - They go in `/mnt/main-2025/fw-runner/secrets/<repo>.env` on TrueNAS (default branch and tag jobs only) + They go in `<runner>/secrets/<repo>.env` (TrueNAS: `/mnt/main-2025/fw-runner/secrets/`; default branch and tag jobs only) - Look at the results of the 37 rename pushes' CI runs (north-star and nuts failed; not looked at yet) - explodorama: `test/demo.test.ts` "decoy" timed out at vitest's 5-second default on TrueNAS (passed the day before); a longer `testTimeout` for that test, or a faster test @@ -239,7 +247,8 @@ difference is that there's no coordinator service: the coordinator is files on D seconds); `gen.py` renders ✓/✗ on the log and summary pages, a page per pipeline with its logs, a badge, and the dashboard at `/ci/`. 6. **Updates.** The runner restarts itself between jobs when `runner/install.sh` has replaced it. -7. **One runner so far.** Runner tags and a second runner on the laptop are in the Backlog. +7. **One runner so far.** Tags, the heartbeat lease and the laptop runner are built (see [Runners](#runners)) but not + deployed yet. **Job definitions: `.fw-ci.yml`** (decided 2026-10-27: our own name; the format stays a subset of GitLab CI's, so moving a repo over is a rename; the old `.gitlab-ci.yml` name still works until then). 42 repos have one, and they use only `image`, diff --git a/README.md b/README.md index 05401eb..5c6dba8 100644 --- a/README.md +++ b/README.md @@ -5,10 +5,10 @@ Tooling for Pavel's self-hosted code and apps on DreamHost: - **code.fuzzyworld.net**: static browse pages and read-only `git clone` for public repos: every unmerged branch, syntax highlighting, file history, blame, CI results - **apps.fuzzyworld.net**: deployed web apps, one per path (`/bubbles/`, `/slide/`, ...), published with `fw deploy`, with screenshots taken by CI - **VS Code in the browser** (`apps.fuzzyworld.net/vscode/?repo=<name>`): any public repo, read-only, with quick open and search -- **CI runner** on TrueNAS: runs each push's pipeline (`.fw-ci.yml`) in Docker, like a GitLab runner +- **CI runners** (TrueNAS, and the laptop when it is idle): run each push's pipeline (`.fw-ci.yml`) in Docker, like a GitLab runner There is no app server. Pages are generated when you push; Apache only serves files. Work that needs a real -machine (CI, screenshots) runs on TrueNAS, which pulls jobs from DreamHost over ssh. The plan and backlog are in +machine (CI, screenshots) runs on a runner (TrueNAS, or the laptop), which pulls jobs from DreamHost over ssh. No runner is a normal state: jobs wait. The plan and backlog are in [PLAN.md](PLAN.md). ## Philosophy @@ -105,7 +105,8 @@ Other settings in the repo config: `fuzzyworld.description`, `fuzzyworld.ci` (`o | `~/fw/state/` | per-repo page manifests and locks | | `~/fw/logs/<name>.log` | output of each push's publish run | | `~/fw/pylib/` | Python-Markdown 3.7 (README rendering) and Pygments 2.21.0 (syntax highlighting), `pip install --target`; `install.sh` adds Pygments | -| `~/fw/queue/pending/`, `running/` | CI jobs, one JSON file each (see [CI](#ci)) | +| `~/fw/queue/pending/`, `running/` | CI jobs, one JSON file each (see [CI](#ci)); `running/<job>.beat` is the runner's heartbeat, `declined/` the runners that gave a job back | +| `~/fw/runners/<runner>` | one file per runner: its tags, and (by its time) when it last checked in | | `~/fw/ci/<name>/<sha>/` | CI results uploaded by runners: `pipeline.json` and one log per job | | `~/apps.fuzzyworld.net/_shots/<app>/` | screenshots of each app, uploaded by runners (`shots.json` lists them) | | `~/apps.fuzzyworld.net/vscode/` | VS Code in the browser (`vscode-web/install.sh`; not listed on the apps index) | @@ -188,9 +189,17 @@ coordinator service; the coordinator is files on DreamHost. 1. **Queue.** `post-receive` queues a job (`~/fw/queue/pending/<id>.json`) for each pushed branch or tag whose commit has a `.fw-ci.yml` (or `.gitlab-ci.yml`, see below) or `package.json` (unless `fuzzyworld.ci` is `off`). `fw ci <name>` queues one by hand. -2. **Claim.** The runner (`runner/fw-runner.py`, on TrueNAS, from cron every 2 minutes) runs - `ssh dh claim <runner>`. The server moves the job into `running/` with one `rename`, which is atomic, so two - runners never take the same job. A job running for over 2 hours goes back to `pending/`. +2. **Claim.** A runner (`runner/fw-runner.py`; on TrueNAS from cron, on a laptop from launchd, every 2 minutes) runs + `ssh dh claim <runner> <tags>`. The server moves the job into `running/` with one `rename`, which is atomic, so two + runners never take the same job. While it works, the runner sends a heartbeat every 30 seconds + (`beat <job> <runner>`); a job whose runner has been quiet for about 2 minutes (lid closed, network gone) goes + back to `pending/`. (A runner that sends no heartbeats, i.e. an old one, is given 2 hours.) With no runner at + all the jobs simply stay in `pending/`, and the CI dashboard says "Waiting for a runner". +2b. **Tags.** A runner has tags (`FW_RUNNER_TAGS`, default `docker,chrome`); a job can ask for more with GitLab's + `tags:`. After cloning, a runner that lacks a tag the pipeline needs gives the job back (`release`) before running + anything; the server then never offers that job to that runner again. Screenshots need `chrome`. The tag + `secrets` means "this repo's secrets": a job tagged `secrets` on the default branch or a tag runs only on a runner + that has `secrets/<repo>.env`. A job nobody can run waits in the queue. 3. **The runner's key** is limited in `authorized_keys` to `server/runner-api` (`restrict,command="…/runner-api"`): it can claim jobs, fetch repos, and upload results; it can't get a shell. 4. **Run.** The runner clones the commit and runs each job in a fresh Docker container (`--cpus 4 --memory 4g`, @@ -213,8 +222,8 @@ still reads `.gitlab-ci.yml`; `.fw-ci.yml` wins if both exist. Supported: `stage `default:` (image, before_script, after_script), `workflow:rules`, `extends` (hidden jobs), and per job `stage`, `image`, `script`, `before_script`, `after_script`, `variables`, `rules` (`if`, `when`, `exists`; `changes` always matches), `only`/`except` (refs), `when` (`on_success`, `on_failure`, `always`, `manual`, `never`), -`allow_failure`, `timeout`, `needs` (ordering comes from stages). Ignored, because they change nothing here: -`artifacts`, `cache`, `dependencies`, `environment`, `tags`, `interruptible`, `retry`, `coverage`, `resource_group`. +`allow_failure`, `timeout`, `needs` (ordering comes from stages), `tags` (see Tags above). Ignored, because they change nothing here: +`artifacts`, `cache`, `dependencies`, `environment`, `interruptible`, `retry`, `coverage`, `resource_group`. **Anything else** (e.g. `include`, `services`, `release`, `trigger`, `parallel`) makes that job fail with a message naming the key; nothing is silently skipped. `manual` jobs are listed but not run. The usual `CI_*` variables are set (`CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`/`CI_COMMIT_TAG`, `CI_COMMIT_REF_SLUG`, `CI_DEFAULT_BRANCH`, @@ -224,8 +233,18 @@ job runs its build as a check; publishing is `fw deploy`'s job. **Without a pipeline file:** an app (`package.json` with `fuzzyworld.app`) gets one job, `fw deploy --here --dry-run` (install, prebuild, checks, build, output checks); other node repos get their `typecheck`/`lint`/`test` scripts. -**Secrets:** `<runner dir>/secrets/<repo>.env` on TrueNAS, passed (`--env-file`) only to jobs on the default branch -and tags. Without one, jobs that deploy elsewhere (e.g. `dreamhost`'s rsync, engye's store uploads) fail. +**Secrets:** `<runner dir>/secrets/<repo>.env` on each runner that should deploy (TrueNAS: `/mnt/main-2025/fw-runner/secrets/`; +laptop: `~/.fw-runner/secrets/`), passed (`--env-file`) only to jobs on the default branch and tags. A runner without +the file still runs that repo's other jobs; tag a deploy job `secrets` to keep it for a runner that has the file. +Without any, jobs that deploy elsewhere (e.g. `dreamhost`'s rsync, engye's store uploads) fail. Anything on the laptop +is as safe as the laptop (FileVault on); if it is lost, rotate the secrets. + +**The laptop as a runner:** `runner/install-laptop.sh` (`npm run update-laptop-runner`) installs a launchd job (every 2 minutes) +with `FW_RUNNER_POLITE=1`, so the runner claims a job only when the Mac is on AC power, at least `FW_RUNNER_MIN_IDLE` +(50) percent of the CPU is idle, and Docker is running; it checks before each job, never during one, and its containers +get a low CPU share (`FW_RUNNER_CPUS`, default 2 there, is the real limit). Settings live in `~/.fw-runner/runner.env`, +the log in `~/.fw-runner/runner.log`. `--authorize` allows its key on DreamHost; `--remove` stops it. It assumes the +laptop is on for a few hours a day, which is enough when TrueNAS isn't there. Check what a checkout would run, without running it: `runner/fw-runner.py --plan <dir> [refs/heads/<branch>]`. @@ -265,6 +284,7 @@ server/install.sh # copy server/ to dh, install site files and hooks server/install.sh --publish # ...and republish every repo runner/install.sh # CI runner on TrueNAS: copy it, make its ssh key, add the cron job runner/install.sh --authorize # ...and allow that key on DreamHost (runner-api only) +runner/install-laptop.sh # the laptop as a polite second runner (launchd, Docker, its own venv) vscode-web/install.sh # VS Code in the browser at apps.fuzzyworld.net/vscode/ ``` @@ -277,11 +297,13 @@ The same actions as npm scripts (`npm run` lists them): | `npm run republish-all` | `bin/fw publish-all` | | `npm run list-repos` | `bin/fw list` | | `npm run update-runner` | `runner/install.sh` | +| `npm run update-laptop-runner` | `runner/install-laptop.sh` | | `npm run update-vscode` | `vscode-web/install.sh` | `install.sh` copies `server/` from the working copy, not from a commit, so uncommitted changes there go live too. Requirements on DreamHost: git, Python 3.12, `flock`, `setsid`. On the laptop: Node ≥ 18, git, rsync, ssh with the `dh` alias. On TrueNAS (runner): Docker, Python 3 with PyYAML (both part of TrueNAS SCALE), and the `truenas` ssh alias on the laptop. +On the laptop (runner, optional): Docker (Docker Desktop, OrbStack, ...) and Python 3; the installer makes a venv with PyYAML. Using `fw` outside an app repo (e.g. on a second laptop): `npm install -g "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0"`. diff --git a/package.json b/package.json index e92719b..29fffb4 100644 --- a/package.json +++ b/package.json @@ -10,6 +10,7 @@ "republish-all": "bin/fw publish-all", "list-repos": "bin/fw list", "update-runner": "runner/install.sh", + "update-laptop-runner": "runner/install-laptop.sh", "update-vscode": "vscode-web/install.sh" }, "bin": { diff --git a/runner/fw-runner.py b/runner/fw-runner.py index d2b194d..3a9f744 100755 --- a/runner/fw-runner.py +++ b/runner/fw-runner.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """fw-runner: CI for the repos on DreamHost, in the style of a GitLab runner. - fw-runner.py [--once] [--keep] take queued jobs and run them (cron runs this every minute) + fw-runner.py [--once] [--keep] take queued jobs and run them (cron or launchd runs this every 2 minutes) fw-runner.py --plan <dir> [ref] print the jobs a checkout would run, and run nothing A push to DreamHost queues a job (server/hooks/post-receive). This runner claims jobs over @@ -23,8 +23,16 @@ Configuration (environment, or <dir>/runner.env next to this script): FW_RUNNER_WORK working directory for clones (default <dir>/work) FW_RUNNER_SECRETS directory of <repo>.env files, given to default-branch and tag jobs only (default <dir>/secrets) FW_RUNNER_CPUS / FW_RUNNER_MEMORY container limits (default 4 / 4g) + FW_RUNNER_TAGS what this runner can do, comma-separated (default docker,chrome). A job whose `tags:` or + screenshots need something else is given back to the queue for a runner that has it. The + tag `secrets` means "this repo's secrets": a default-branch or tag job tagged `secrets` runs + only on a runner that has <repo>.env. + FW_RUNNER_POLITE 1 for a runner on a laptop: claim jobs only on AC power, with the CPU mostly idle and + Docker running; containers get a low CPU share. Checked before each job, never during one. + FW_RUNNER_MIN_IDLE percent of CPU that must be idle for a polite runner to claim a job (default 50) """ +import fcntl import fnmatch import json import os @@ -34,6 +42,7 @@ import shutil import socket import subprocess import sys +import threading import time HERE = os.path.dirname(os.path.abspath(__file__)) @@ -58,6 +67,9 @@ WORK = os.environ.get("FW_RUNNER_WORK", os.path.join(HERE, "work")) SECRETS = os.environ.get("FW_RUNNER_SECRETS", os.path.join(HERE, "secrets")) CPUS = os.environ.get("FW_RUNNER_CPUS", "4") MEMORY = os.environ.get("FW_RUNNER_MEMORY", "4g") +TAGS = {t for t in re.split(r"[,\s]+", os.environ.get("FW_RUNNER_TAGS", "docker,chrome")) if t} +POLITE = os.environ.get("FW_RUNNER_POLITE", "") not in ("", "0", "false") +MIN_IDLE = float(os.environ.get("FW_RUNNER_MIN_IDLE", "50")) APPS_URL = "https://apps.fuzzyworld.net" CODE_URL = "https://code.fuzzyworld.net" @@ -67,6 +79,7 @@ PLAYWRIGHT = "1.49.1" SHOOT_IMAGE = f"fw-shoot:{PLAYWRIGHT}" # `fw deploy --dry-run` from the published fuzzyworld-tools, whether or not the app depends on it. FW_DEPLOY_CHECK = "npx --yes --package=git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0 fw deploy --here --dry-run" +BEAT_EVERY = 30 # seconds between heartbeats; DreamHost gives a job back after ~2 minutes without one LOG_EVERY = 15 # seconds between uploads of a running job's log (live CI pages) JOB_TIMEOUT = 30 * 60 # seconds, unless the job sets `timeout` MAX_LOG = 20 * 1024 * 1024 # runner-api refuses bigger files @@ -79,8 +92,8 @@ SSH = ["ssh", "-i", KEY, "-o", "IdentitiesOnly=yes", "-o", "BatchMode=yes", # IGNORED_JOB_KEYS don't change what a job does here (every job of a pipeline shares one workspace, # so artifacts are already there for later stages). SUPPORTED_JOB_KEYS = {"stage", "image", "script", "before_script", "after_script", "variables", "rules", - "only", "except", "when", "allow_failure", "extends", "timeout", "needs"} -IGNORED_JOB_KEYS = {"artifacts", "cache", "dependencies", "environment", "tags", "interruptible", "retry", + "only", "except", "when", "allow_failure", "extends", "timeout", "needs", "tags"} +IGNORED_JOB_KEYS = {"artifacts", "cache", "dependencies", "environment", "interruptible", "retry", "coverage", "resource_group", "pages", "id_tokens", "publish"} TOP_LEVEL_KEYS = {"stages", "variables", "image", "default", "workflow", "before_script", "after_script", "cache"} RESERVED = TOP_LEVEL_KEYS | {"include", "types"} @@ -338,6 +351,7 @@ def yaml_pipeline(src, env): "script": script_lines(before) + script_lines(job.get("script")), "after": script_lines(after), "when": when, "allow_failure": bool(job.get("allow_failure", False)), "timeout": parse_timeout(job.get("timeout")), + "tags": [str(t) for t in (job.get("tags") or [])], "unsupported": sorted(unknown), "note": "GitLab Pages: the build runs as a check here; `fw deploy` publishes apps." if name == "pages" else "", }) @@ -390,7 +404,7 @@ def install_command(src): def job(name, stage, script, env): return {"name": name, "stage": stage, "image": DEFAULT_IMAGE, "env": {**env, "CI_JOB_NAME": name, "CI_JOB_STAGE": stage}, "script": script, "after": [], "when": "on_success", "allow_failure": False, - "timeout": JOB_TIMEOUT, "unsupported": [], "note": ""} + "timeout": JOB_TIMEOUT, "tags": [], "unsupported": [], "note": ""} def ci_env(j, src): @@ -413,6 +427,25 @@ def ci_env(j, src): return env +# ---------------------------------------------------------------- tags + +class Declined(Exception): + """This runner can't run the job; it went back in the queue for another.""" + + +def missing_tags(j, jobs, src, protected, has_secrets): + """What this pipeline needs that this runner lacks: job `tags:`, `chrome` if it will take screenshots, + and `secrets` (the repo's <repo>.env) for a default-branch or tag job tagged `secrets`.""" + need = {t for x in jobs for t in x["tags"]} + if protected and not j["ref"].startswith("refs/tags/") and shot_plan(src): + need.add("chrome") + # `secrets` is not a runner tag: the runner has them if it has the repo's file, and a job off the + # default branch never gets them, so for that job the tag is met by any runner. + wants_secrets = "secrets" in need + need.discard("secrets") + return sorted(need - TAGS) + (["secrets"] if wants_secrets and protected and not has_secrets else []) + + # ---------------------------------------------------------------- running def git(cwd, *args): @@ -424,6 +457,8 @@ def docker_run(name, image, workdir, env, script, timeout, logf, env_file=None, progress() is called every LOG_EVERY seconds while it runs.""" cmd = ["docker", "run", "--rm", "--name", name, "--cpus", CPUS, "--memory", MEMORY, "-v", f"{workdir}:/builds/{env['CI_PROJECT_NAME']}", "-w", f"/builds/{env['CI_PROJECT_NAME']}", *extra] + if POLITE: + cmd[2:2] = ["--cpu-shares", "128"] # the default is 1024: other containers and the Docker VM's own work win for k, v in env.items(): cmd += ["-e", f"{k}={v}"] if env_file: @@ -451,10 +486,19 @@ def docker_run(name, image, workdir, env, script, timeout, logf, env_file=None, return code, time.time() - start +def clear_workdir(workdir): + """Remove a job's workspace. Containers ran as root, so it may hold root-owned files: remove it from a container too.""" + if os.path.isdir(workdir): + subprocess.run(["docker", "run", "--rm", "-v", f"{workdir}:/w", "alpine:3.20", "rm", "-rf", "/w/src"], + capture_output=True) + shutil.rmtree(workdir, ignore_errors=True) + + def run_pipeline(j, keep=False): repo, sha = j["repo"], j["sha"] workdir = os.path.join(WORK, j["id"]) src = os.path.join(workdir, "src") + clear_workdir(workdir) # left behind if this runner crashed on the same job (a requeued job can come back to it) os.makedirs(workdir, exist_ok=True) pipeline = {"repo": repo, "sha": sha, "ref": j["ref"].split("/", 2)[2], "runner": RUNNER, "started": now_iso(), "status": "running", "jobs": []} @@ -486,12 +530,17 @@ def run_pipeline(j, keep=False): if not jobs: log(f"{repo} {sha[:12]}: nothing to run ({why})") return None - pipeline["jobs"] = [{"name": x["name"], "stage": x["stage"], "status": "pending"} for x in jobs] - put(repo, sha, "pipeline.json", json.dumps(pipeline)) - protected = j["ref"] == f"refs/heads/{j['default_branch']}" or j["ref"].startswith("refs/tags/") secret = os.path.join(SECRETS, f"{repo}.env") env_file = secret if protected and os.path.isfile(secret) else None + lacking = missing_tags(j, jobs, src, protected, os.path.isfile(secret)) + if lacking: + # Nothing has run and nothing is uploaded: another runner takes it (or it waits for one). + api("release", j["id"], RUNNER) + raise Declined(f"needs {', '.join(lacking)}, which {RUNNER} lacks") + pipeline["jobs"] = [{"name": x["name"], "stage": x["stage"], "status": "pending"} for x in jobs] + put(repo, sha, "pipeline.json", json.dumps(pipeline)) + failed = False for i, x in enumerate(jobs): rec = pipeline["jobs"][i] @@ -541,10 +590,7 @@ def run_pipeline(j, keep=False): return pipeline finally: if not keep: - # Containers ran as root, so the workspace may hold root-owned files: remove it from a container too. - subprocess.run(["docker", "run", "--rm", "-v", f"{workdir}:/w", "alpine:3.20", "rm", "-rf", "/w/src"], - capture_output=True) - shutil.rmtree(workdir, ignore_errors=True) + clear_workdir(workdir) # ---------------------------------------------------------------- screenshots @@ -560,16 +606,25 @@ def shoot_image(): return SHOOT_IMAGE -def screenshots(j, src, workdir): - """Build the app and photograph it, for apps (package.json "fuzzyworld.app") on the default branch.""" +def shot_plan(src): + """(package.json's "fuzzyworld" section, app, screenshot config) for an app that wants screenshots, else None.""" try: with open(os.path.join(src, "package.json")) as f: cfg = json.load(f).get("fuzzyworld") or {} except (OSError, ValueError): - return + return None app, shots = cfg.get("app"), cfg.get("screenshots", [{"path": "", "size": "phone"}]) if not app or shots is False or shots == []: + return None + return cfg, app, shots + + +def screenshots(j, src, workdir): + """Build the app and photograph it, for apps (package.json "fuzzyworld.app") on the default branch.""" + plan = shot_plan(src) + if not plan: return + cfg, app, shots = plan repo, sha = j["repo"], j["sha"] env = {**ci_env(j, src), "FW_APP": app, "FW_BASE": f"/{app}/"} log_path = os.path.join(workdir, "screenshots.log") @@ -634,6 +689,7 @@ def plan(src, ref): print(f"nothing to run: {why}") for x in jobs: print(f"[{x['stage']}] {x['name']} image={x['image']} when={x['when']}" + + (f" tags={','.join(x['tags'])}" if x["tags"] else "") + (f" UNSUPPORTED: {', '.join(x['unsupported'])}" if x["unsupported"] else "") + (f"\n {x['note']}" if x["note"] else "")) for line in x["script"]: @@ -641,25 +697,118 @@ def plan(src, ref): return 0 +class Heartbeat(threading.Thread): + """Tells DreamHost every BEAT_EVERY seconds that this runner is still working on a job. Without it the + job goes back in the queue after ~2 minutes, so a closed laptop doesn't hold a pipeline for hours.""" + + def __init__(self, job_id): + super().__init__(daemon=True) + self.job_id, self.stopped = job_id, threading.Event() + + def run(self): + while not self.stopped.wait(BEAT_EVERY): + try: + api("beat", self.job_id, RUNNER) + except Exception as e: # a network blip: the lease is ~2 minutes, so the next beat may still be in time + log(f"heartbeat failed: {e}") + + def stop(self): + self.stopped.set() + + +def cpu_idle_percent(): + """Percent of CPU idle over the last second or so, or None if this system doesn't say.""" + try: + if sys.platform == "darwin": + out = subprocess.run(["top", "-l", "2", "-n", "0", "-s", "1"], capture_output=True, text=True, timeout=20).stdout + m = re.findall(r"CPU usage:.*?([\d.]+)% idle", out) + return float(m[-1]) if m else None + def times(): + f = [int(x) for x in open("/proc/stat").readline().split()[1:]] + return f[3] + f[4], sum(f) # idle + iowait, total + i0, t0 = times() + time.sleep(1) + i1, t1 = times() + return 100.0 * (i1 - i0) / (t1 - t0) if t1 > t0 else None + except (OSError, ValueError, subprocess.SubprocessError): + return None + + +def on_ac_power(): + """False only when the system says it is on battery; True when it is on AC or has no battery to report.""" + try: + if sys.platform == "darwin": + out = subprocess.run(["pmset", "-g", "batt"], capture_output=True, text=True, timeout=10).stdout + return "Battery Power" not in out + base = "/sys/class/power_supply" + mains = [os.path.join(base, d, "online") for d in os.listdir(base) if open(os.path.join(base, d, "type")).read().strip() == "Mains"] + return not mains or any(open(m).read().strip() == "1" for m in mains) + except (OSError, subprocess.SubprocessError): + return True + + +def busy_reason(): + """Why a polite runner should not claim a job right now, or "" if it may.""" + if not on_ac_power(): + return "on battery" + if subprocess.run(["docker", "info"], capture_output=True).returncode != 0: + return "Docker is not running" + idle = cpu_idle_percent() + if idle is not None and idle < MIN_IDLE: + return f"CPU busy ({100 - idle:.0f}% used)" + return "" + + +def say_once(reason): + """Log a skip reason only when it changes, so a polite runner polling every 2 minutes doesn't fill the log.""" + path = os.path.join(HERE, ".last-skip") + try: + last = open(path).read() + except OSError: + last = "" + if reason != last: + log(f"not claiming: {reason}" if reason else "ready to claim again") + with open(path, "w") as f: + f.write(reason) + + def main(argv): if argv[1:2] == ["--plan"]: return plan(argv[2], argv[3] if len(argv) > 3 else "refs/heads/main") once, keep = "--once" in argv, "--keep" in argv os.makedirs(WORK, exist_ok=True) + # One runner process at a time. (TrueNAS cron also wraps it in `flock -n .lock`; launchd on a Mac has no flock.) + lock = open(os.path.join(HERE, ".runner.lock"), "a") + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + return 0 started_as = os.path.getmtime(__file__) while True: # runner/install.sh replaced this file: continue as the new version (between jobs, never during one). if os.path.getmtime(__file__) != started_as: log("fw-runner.py was updated; restarting") + lock.close() os.execv(sys.executable, [sys.executable, os.path.abspath(__file__)] + argv[1:]) - raw = api("claim", RUNNER).decode().strip() + if POLITE: + reason = busy_reason() + say_once(reason) + if reason: + return 0 + raw = api("claim", RUNNER, ",".join(sorted(TAGS)) or "-").decode().strip() if not raw: return 0 j = json.loads(raw) log(f"claimed {j['id']}: {j['repo']} {j['ref']} {j['sha'][:12]}") + beat = Heartbeat(j["id"]) + beat.start() try: p = run_pipeline(j, keep) log(f"{j['id']}: {p['status'] if p else 'nothing to run'}") + except Declined as e: + beat.stop() + log(f"{j['id']}: {j['repo']} given back: {e}") + continue except Exception as e: log(f"{j['id']}: runner error: {e}") try: @@ -668,6 +817,7 @@ def main(argv): "started": now_iso(), "status": "error", "message": f"runner error: {e}", "jobs": []})) except Exception: pass + beat.stop() api("done", j["repo"], j["sha"], j["id"]) if once: return 0 diff --git a/runner/install-laptop.sh b/runner/install-laptop.sh new file mode 100755 index 0000000..63d4d28 --- /dev/null +++ b/runner/install-laptop.sh @@ -0,0 +1,70 @@ +#!/bin/bash +# Install or update the CI runner on this Mac, as a polite second runner. Run on the laptop: +# +# runner/install-laptop.sh copy the runner, make its ssh key, add the launchd job +# runner/install-laptop.sh --authorize ...and allow that key on DreamHost (restricted to server/runner-api) +# runner/install-laptop.sh --remove stop it and remove the launchd job (files stay) +# +# The runner wakes every 2 minutes and claims a job only when the Mac is on AC power, the CPU is mostly idle +# and Docker is running (FW_RUNNER_POLITE=1). Containers get a low CPU share and FW_RUNNER_CPUS cores. +# Settings (environment): FW_RUNNER_DIR (default ~/.fw-runner), FW_HOST (DreamHost ssh alias, default: dh). +# Needs Docker (Docker Desktop, OrbStack, ...) and Python 3 with `venv`; PyYAML is installed into the runner's own venv. +set -euo pipefail +cd "$(dirname "$0")" +DIR=${FW_RUNNER_DIR:-$HOME/.fw-runner} +HOST=${FW_HOST:-dh} +LABEL=net.fuzzyworld.fw-runner +PLIST=$HOME/Library/LaunchAgents/$LABEL.plist + +if [ "${1:-}" = --remove ]; then + launchctl bootout "gui/$(id -u)/$LABEL" 2>/dev/null || true + rm -f "$PLIST" + echo "launchd job removed; $DIR is untouched (delete it to forget the key, secrets and logs)" + exit 0 +fi + +mkdir -p "$DIR/.ssh" "$DIR/secrets" "$DIR/work" +chmod 700 "$DIR/.ssh" "$DIR/secrets" +cp fw-runner.py shoot.py "$DIR/" +chmod +x "$DIR/fw-runner.py" +[ -x "$DIR/venv/bin/python" ] || python3 -m venv "$DIR/venv" +"$DIR/venv/bin/python" -c 'import yaml' 2>/dev/null || "$DIR/venv/bin/pip" -q install pyyaml +[ -f "$DIR/.ssh/id_ed25519" ] || ssh-keygen -q -t ed25519 -N '' -C "fw-runner $(scutil --get LocalHostName) (CI only on DreamHost)" -f "$DIR/.ssh/id_ed25519" +# The first install writes the settings; later installs leave your edits alone. +[ -f "$DIR/runner.env" ] || cat > "$DIR/runner.env" <<ENV +FW_RUNNER_NAME=$(scutil --get LocalHostName) +FW_RUNNER_TAGS=docker,chrome +FW_RUNNER_POLITE=1 +FW_RUNNER_CPUS=2 +FW_RUNNER_MEMORY=4g +ENV + +cat > "$PLIST" <<XML +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> +<plist version="1.0"><dict> + <key>Label</key><string>$LABEL</string> + <key>ProgramArguments</key><array><string>$DIR/venv/bin/python</string><string>$DIR/fw-runner.py</string></array> + <key>StartInterval</key><integer>120</integer> + <key>EnvironmentVariables</key><dict><key>PATH</key><string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$HOME/.orbstack/bin:/Applications/Docker.app/Contents/Resources/bin</string></dict> + <key>StandardOutPath</key><string>$DIR/runner.log</string> + <key>StandardErrorPath</key><string>$DIR/runner.log</string> + <key>ProcessType</key><string>Background</string> + <key>LowPriorityIO</key><true/> + <key>Nice</key><integer>10</integer> +</dict></plist> +XML +launchctl bootout "gui/$(id -u)/$LABEL" 2>/dev/null || true +launchctl bootstrap "gui/$(id -u)" "$PLIST" +echo "runner installed in $DIR (launchd: every 2 minutes, polite; settings in $DIR/runner.env, log in $DIR/runner.log)" + +pub=$(cat "$DIR/.ssh/id_ed25519.pub") +line="restrict,command=\"/home/fuzzygamer/fw/tools/server/runner-api\" $pub" +if [ "${1:-}" = --authorize ]; then + ssh "$HOST" "grep -qF '$pub' ~/.ssh/authorized_keys || echo '$line' >> ~/.ssh/authorized_keys" + echo "key allowed on $HOST (runner-api only)" +else + echo "To let it take jobs, add this line to ~/.ssh/authorized_keys on DreamHost (or rerun with --authorize):" + echo " $line" +fi +echo "Repo secrets for jobs that deploy go in $DIR/secrets/<repo>.env" diff --git a/server/gen.py b/server/gen.py index 5cacbb2..37dcc29 100755 --- a/server/gen.py +++ b/server/gen.py @@ -37,6 +37,7 @@ STATE = os.environ.get("FW_STATE", os.path.join(HOME, "fw", "state")) PYLIB = os.environ.get("FW_PYLIB", os.path.join(HOME, "fw", "pylib")) CI_ROOT = os.environ.get("FW_CI", os.path.join(HOME, "fw", "ci")) QUEUE = os.environ.get("FW_QUEUE", os.path.join(HOME, "fw", "queue")) +RUNNERS = os.environ.get("FW_RUNNERS", os.path.join(HOME, "fw", "runners")) # one file per runner; its time is its last check-in BASE_URL = "https://code.fuzzyworld.net" APPS_URL = "https://apps.fuzzyworld.net" @@ -47,6 +48,7 @@ COMMIT_LIMIT = 2000 # commit pages per repo, across all branches; older commi HISTORY_LIMIT = 200 # commits listed on a file's history page CI_LIMIT = 100 # pipelines listed on a repo's CI page CI_RECENT = 50 # finished pipelines on the CI dashboard +RUNNER_GONE = 15 * 60 # seconds without a check-in before a runner doesn't count (runners ask every 2 minutes) CI_REFRESH = 15 # seconds: pages showing something running reload themselves SHOTS = "_shots" # apps.fuzzyworld.net/_shots/<app>/: screenshots from CI runners (no app name starts with "_") SHOT_EXTENSIONS = (".png", ".jpg", ".jpeg", ".webp") @@ -389,6 +391,31 @@ def queue_jobs(kind): return out +def runners_seen(): + """[(name, tags, seconds since it last checked in)], most recent first.""" + out = [] + for name in os.listdir(RUNNERS) if os.path.isdir(RUNNERS) else []: + try: + with open(os.path.join(RUNNERS, name)) as f: + tags = f.read().strip().strip("-") + out.append((name, tags, time.time() - os.path.getmtime(os.path.join(RUNNERS, name)))) + except OSError: + continue + return sorted(out, key=lambda r: r[2]) + + +def runner_notice(pending): + """One sentence for the CI pages: "" if some runner is around, else why queued jobs are not moving.""" + if not pending: + return "" + seen = runners_seen() + if any(age < RUNNER_GONE for _, _, age in seen): + return "" + if seen: + return f"Waiting for a runner: the last one checked in {duration(seen[0][2])} ago. Jobs wait here until one does." + return "Waiting for a runner: none has checked in yet. Jobs wait here until one does." + + def public_repo(name, _cache={}): if name not in _cache: repo = os.path.join(GIT_ROOT, name + ".git") @@ -443,10 +470,12 @@ def build_ci_dashboard(): f'<td>{esc(CI_STATES[p["status"]][1])}</td><td class="num">{duration(p.get("duration"))}</td></tr>' for started, name, sha, p in recent) busy = bool(pending or running) + notice = runner_notice(pending) body = ('<h1>CI</h1>' - '<p class="meta">Pipelines run on a runner at home, one at a time, as pushes queue them. ' + '<p class="meta">Pipelines run on whichever runner is available (a home server, sometimes a laptop), one at a time, as pushes queue them. ' + (f'This page reloads every {CI_REFRESH} seconds while anything is queued or running.' if busy else "Nothing is running.") + (f' Also {hidden} job{"s" if hidden != 1 else ""} for private repositories.' if hidden else "") + '</p>' + + (f'<p><strong>{esc(notice)}</strong></p>' if notice else "") + (f'<h2>Running</h2><table class="log">{run_rows}</table>' if run_rows else "") + (f'<h2>Queued</h2><table class="log">{queue_rows}</table>' if queue_rows else "") + (f'<h2>Recent</h2><table class="log">{done_rows}</table>' if done_rows else "<p>No pipelines yet.</p>")) @@ -464,6 +493,8 @@ def ci_status_text(): for started, name, sha, p in recent[:5]: mark, words, _ = CI_STATES[p["status"]] lines.append(f'{mark} {words:<6} {name} {p.get("ref", "")} {sha[:12]} {started[:16].replace("T", " ")} {duration(p.get("duration"))}') + if runner_notice(pending): + lines.insert(0, runner_notice(pending)) return "\n".join(lines) if lines else "no CI jobs yet" diff --git a/server/runner-api b/server/runner-api index 344c4dd..1faefc3 100755 --- a/server/runner-api +++ b/server/runner-api @@ -4,22 +4,30 @@ # restrict,command="/home/fuzzygamer/fw/tools/server/runner-api" ssh-ed25519 AAAA... truenas-runner # # A runner (runner/fw-runner.py) can then only: -# claim <runner> take the oldest queued job; prints its JSON, or nothing +# claim <runner> [tags] take the oldest queued job this runner hasn't declined; prints its +# JSON, or nothing. `tags` (comma-separated) is what the runner can do. +# beat <job> <runner> "still working on it": a job whose runner is quiet for ~2 minutes +# (lid closed, network gone) goes back in the queue +# release <job> <runner> "I can't run this one" (missing tag or secrets): back in the queue, +# and this runner is not offered it again # git-upload-pack '<repo>' clone/fetch ~/git/<name>.git (the job's source) # put <repo> <sha> <file> store one result file (stdin) in ~/fw/ci/<repo>/<sha>/ # done <repo> <sha> <job-id> the job is finished: remove it from the queue, refresh pages # put-shot <app> <file> store one screenshot (stdin) in apps.fuzzyworld.net/_shots/<app>/ # shots-done <app> <repo> refresh the apps index and the repo's page # Anything else is refused. Every call is logged to ~/fw/logs/runner-api.log, except a claim that -# finds nothing (runners ask every few minutes). +# finds nothing (runners ask every few minutes). A claim also records the runner in ~/fw/runners/<runner> +# (its file's time is when it last checked in; the CI dashboard says "waiting for a runner" when none has). set -u cmd=${SSH_ORIGINAL_COMMAND:-} QUEUE=$HOME/fw/queue +RUNNERS=$HOME/fw/runners +LEASE_MIN=2 # a job whose runner has not sent a heartbeat for this long goes back to pending/ CI=$HOME/fw/ci SHOTS=$HOME/apps.fuzzyworld.net/_shots MAX_FILE=$((20 * 1024 * 1024)) log="$HOME/fw/logs/runner-api.log" -mkdir -p "$(dirname "$log")" "$QUEUE/pending" "$QUEUE/running" +mkdir -p "$(dirname "$log")" "$QUEUE/pending" "$QUEUE/running" "$QUEUE/declined" "$RUNNERS" note() { echo "$(date -u '+%F %T') ${SSH_CLIENT%% *} $*" >> "$log"; } [[ "$cmd" == claim* ]] || note "$cmd" @@ -29,6 +37,7 @@ bg() { setsid nohup "$@" >> "$HOME/fw/logs/runner-api-pages.log" 2>&1 < /dev/nul valid_name() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] && [[ "$1" != *.git ]]; } valid_repo() { valid_name "$1" && [ -d "$HOME/git/$1.git" ]; } valid_sha() { [[ "$1" =~ ^[0-9a-f]{40}$ ]]; } +valid_tags() { [[ "$1" =~ ^[A-Za-z0-9,_-]*$ ]]; } valid_job() { [[ "$1" =~ ^[0-9]+-[A-Za-z0-9._-]+-[0-9a-f]{12}$ ]]; } # stdin to a file, refusing anything over MAX_FILE; written beside the target, then renamed. @@ -42,20 +51,48 @@ store() { read -r -a w <<< "$cmd" case "${w[0]:-}" in claim) - runner=${w[1]:-} - valid_name "$runner" || die "usage: claim <runner>" - # Jobs a runner took more than 2 hours ago (it crashed or lost the connection) go back in the queue. - find "$QUEUE/running" -name '*.json' -mmin +120 -exec mv -t "$QUEUE/pending/" {} + 2>/dev/null + runner=${w[1]:-} tags=${w[2]:-} + valid_name "$runner" && valid_tags "$tags" || die "usage: claim <runner> [tag,tag,...]" + echo "$tags" > "$RUNNERS/$runner" + # Jobs whose runner went quiet go back in the queue. A runner that sends heartbeats (it claimed with + # tags) has a <job>.beat file, and is given LEASE_MIN minutes; an older runner is given 2 hours. + for f in "$QUEUE"/running/*.json; do + [ -e "$f" ] || continue + id=$(basename "$f" .json) + if [ -e "$QUEUE/running/$id.beat" ]; then + [ -n "$(find "$QUEUE/running/$id.beat" -mmin +$LEASE_MIN)" ] || continue + else + [ -n "$(find "$f" -mmin +120)" ] || continue + fi + mv "$f" "$QUEUE/pending/" 2>/dev/null && { rm -f "$QUEUE/running/$id.beat"; note "requeued $id (its runner went quiet)"; } + done for f in $(ls "$QUEUE/pending" 2>/dev/null | sort); do + id=${f%.json} + [ -e "$QUEUE/declined/$id.$runner" ] && continue # rename(2) is atomic: if two runners race for the same job, only one mv succeeds. if mv "$QUEUE/pending/$f" "$QUEUE/running/$f" 2>/dev/null; then touch "$QUEUE/running/$f" + [ -n "$tags" ] && echo "$runner" > "$QUEUE/running/$id.beat" note "$cmd: $f" bg "$HOME/bin/fw" ci-dashboard cat "$QUEUE/running/$f" exit 0 fi done ;; + beat) + job=${w[1]:-} runner=${w[2]:-} + valid_job "$job" && valid_name "$runner" || die "usage: beat <job-id> <runner>" + [ -e "$QUEUE/running/$job.json" ] && [ "$(cat "$QUEUE/running/$job.beat" 2>/dev/null)" = "$runner" ] \ + || die "job $job is not running on $runner (it was put back in the queue)" + touch "$QUEUE/running/$job.beat" ;; + release) + job=${w[1]:-} runner=${w[2]:-} + valid_job "$job" && valid_name "$runner" || die "usage: release <job-id> <runner>" + [ -e "$QUEUE/running/$job.json" ] && [ "$(cat "$QUEUE/running/$job.beat" 2>/dev/null)" = "$runner" ] \ + || die "job $job is not running on $runner" + : > "$QUEUE/declined/$job.$runner" + mv "$QUEUE/running/$job.json" "$QUEUE/pending/$job.json" && rm -f "$QUEUE/running/$job.beat" + bg "$HOME/bin/fw" ci-dashboard ;; git-upload-pack) repo=${cmd#git-upload-pack \'}; repo=${repo%\'}; repo=${repo#/home/fuzzygamer/}; repo=${repo#\~/} [[ "$repo" =~ ^git/([A-Za-z0-9][A-Za-z0-9._-]*)\.git$ ]] && valid_repo "${BASH_REMATCH[1]}" || die "no such repo" @@ -75,7 +112,7 @@ case "${w[0]:-}" in done) repo=${w[1]:-} sha=${w[2]:-} job=${w[3]:-} valid_repo "$repo" && valid_sha "$sha" && valid_job "$job" || die "usage: done <repo> <sha> <job-id>" - rm -f "$QUEUE/running/$job.json" + rm -f "$QUEUE/running/$job.json" "$QUEUE/running/$job.beat" "$QUEUE/declined/$job".* bg "$HOME/bin/fw" ci-dashboard ;; put-shot) app=${w[1]:-} file=${w[2]:-}