fuzzyworld-tools / commit
Runners: deployed; `done` refused from a runner that lost its lease; release retries
PLAN.md | 12 +++++++----- README.md | 2 +- runner/fw-runner.py | 21 +++++++++++++++++++-- server/runner-api | 13 ++++++++++--- 4 files changed, 37 insertions(+), 11 deletions(-) diff --git a/PLAN.md b/PLAN.md index ad845c5..9acd30d 100644 --- a/PLAN.md +++ b/PLAN.md @@ -46,8 +46,10 @@ The order goes from cheap to expensive, and each step reuses what the one before 0. **Runners without TrueNAS** (decided 2026-10-28; see [Runners](#runners)): runner tags, the laptop as a runner when idle and plugged in, secrets in a folder on each runner, and "no runners" as a normal state. Goes first. - **Built 2026-10-28, not yet deployed** (needs `npm run update-server`, then `update-runner`, in that order: a new runner against the old server has every heartbeat refused; then - `update-laptop-runner --authorize` on the laptop; nothing has run on real machines yet). + **Built 2026-10-28; deployed 2026-10-31** (server, then the TrueNAS runner, then the laptop runner with `--authorize`, + in that order: a new runner against the old server has every heartbeat refused). Smoke-tested: both runners poll + DreamHost and find nothing to run; no real job has gone through them yet. Review follow-ups (claim check on `done`, + tolerant `release`) are in the working tree, not yet deployed. 1. **Feeds**: Atom per repo (commits, releases), for app deploys, and site-wide. Smallest step: the data is already gathered for the log pages and the apps index. The blog reuses it. 1b. **Blog** at `blog.fuzzyworld.net` (see [Blog](#blog)): a generator that turns a folder or repo of Markdown into a @@ -79,7 +81,8 @@ always-on machine; a small setup is a DreamHost account and a laptop. is on a few hours a day; that is enough for me. - **Lease.** A claimed job needs a heartbeat (`beat`, every 30 seconds, from a thread in the runner). A job whose runner went quiet for ~2 minutes (lid closed) goes back to `pending/` instead of waiting 2 hours. If that runner wakes up and - finishes anyway, its results are uploaded too, and the job may have run twice: accepted. Runners that send no + finishes anyway, its results are uploaded too, and the job may have run twice: accepted. Its `done` is refused, + so it cannot remove the new holder's claim. Runners that send no heartbeat (old versions) keep the 2-hour rule. - **Tags** (`docker`, `chrome`, `android`): job `tags:` are honoured, and screenshots need `chrome`. The runner learns what a pipeline needs only after cloning, so a runner that lacks a tag gives the job back (`release`); the server @@ -247,8 +250,7 @@ difference is that there's no coordinator service: the coordinator is files on D seconds); `gen.py` renders ✓/✗ on the log and summary pages, a page per pipeline with its logs, a badge, and the dashboard at `/ci/`. 6. **Updates.** The runner restarts itself between jobs when `runner/install.sh` has replaced it. -7. **One runner so far.** Tags, the heartbeat lease and the laptop runner are built (see [Runners](#runners)) but not - deployed yet. +7. **Two runners.** Tags, the heartbeat lease and the laptop runner are deployed (2026-10-31; see [Runners](#runners)). **Job definitions: `.fw-ci.yml`** (decided 2026-10-27: our own name; the format stays a subset of GitLab CI's, so moving a repo over is a rename; the old `.gitlab-ci.yml` name still works until then). 42 repos have one, and they use only `image`, diff --git a/README.md b/README.md index 7678beb..69f776a 100644 --- a/README.md +++ b/README.md @@ -194,7 +194,7 @@ coordinator service; the coordinator is files on DreamHost. `ssh dh claim <runner> <tags>`. The server moves the job into `running/` with one `rename`, which is atomic, so two runners never take the same job. While it works, the runner sends a heartbeat every 30 seconds (`beat <job> <runner>`); a job whose runner has been quiet for about 2 minutes (lid closed, network gone) goes - back to `pending/`. (A runner that sends no heartbeats, i.e. an old one, is given 2 hours.) With no runner at + back to `pending/` (a runner that lost its job this way can't mark it done). (A runner that sends no heartbeats, i.e. an old one, is given 2 hours.) With no runner at all the jobs simply stay in `pending/`, and the CI dashboard says "Waiting for a runner". 2b. **Tags.** A runner has tags (`FW_RUNNER_TAGS`, default `docker,chrome`); a job can ask for more with GitLab's `tags:`. After cloning, a runner that lacks a tag the pipeline needs gives the job back (`release`) before running diff --git a/runner/fw-runner.py b/runner/fw-runner.py index 3a9f744..7d2d08c 100755 --- a/runner/fw-runner.py +++ b/runner/fw-runner.py @@ -429,6 +429,20 @@ def ci_env(j, src): # ---------------------------------------------------------------- tags +def release(job_id, tries=3): + """Give a claimed job back. A network blip must not drop the job into the generic error path (which would + upload an "error" result for a job that never ran): retry, then give up quietly. Unreleased, the job goes + back in the queue by itself when the heartbeat has been quiet for ~2 minutes.""" + for n in range(tries): + try: + api("release", job_id, RUNNER) + return + except Exception as e: + log(f"{job_id}: release failed ({e})" + ("; retrying" if n + 1 < tries else "; the lease will expire")) + if n + 1 < tries: + time.sleep(5) + + class Declined(Exception): """This runner can't run the job; it went back in the queue for another.""" @@ -536,7 +550,7 @@ def run_pipeline(j, keep=False): lacking = missing_tags(j, jobs, src, protected, os.path.isfile(secret)) if lacking: # Nothing has run and nothing is uploaded: another runner takes it (or it waits for one). - api("release", j["id"], RUNNER) + release(j["id"]) raise Declined(f"needs {', '.join(lacking)}, which {RUNNER} lacks") pipeline["jobs"] = [{"name": x["name"], "stage": x["stage"], "status": "pending"} for x in jobs] put(repo, sha, "pipeline.json", json.dumps(pipeline)) @@ -818,7 +832,10 @@ def main(argv): except Exception: pass beat.stop() - api("done", j["repo"], j["sha"], j["id"]) + try: + api("done", j["repo"], j["sha"], j["id"], RUNNER) + except Exception as e: # refused when the lease was lost and another runner holds the job: expected, not a crash + log(f"{j['id']}: not marked done ({e})") if once: return 0 diff --git a/server/runner-api b/server/runner-api index 1faefc3..d92fbf0 100755 --- a/server/runner-api +++ b/server/runner-api @@ -12,7 +12,9 @@ # and this runner is not offered it again # git-upload-pack '<repo>' clone/fetch ~/git/<name>.git (the job's source) # put <repo> <sha> <file> store one result file (stdin) in ~/fw/ci/<repo>/<sha>/ -# done <repo> <sha> <job-id> the job is finished: remove it from the queue, refresh pages +# done <repo> <sha> <job-id> [runner] +# the job is finished: remove it from the queue, refresh pages. With +# `runner`, refused if the job was meanwhile given to another runner # put-shot <app> <file> store one screenshot (stdin) in apps.fuzzyworld.net/_shots/<app>/ # shots-done <app> <repo> refresh the apps index and the repo's page # Anything else is refused. Every call is logged to ~/fw/logs/runner-api.log, except a claim that @@ -110,8 +112,13 @@ case "${w[0]:-}" in bg "$HOME/bin/fw" ci-pages "$repo" fi ;; done) - repo=${w[1]:-} sha=${w[2]:-} job=${w[3]:-} - valid_repo "$repo" && valid_sha "$sha" && valid_job "$job" || die "usage: done <repo> <sha> <job-id>" + repo=${w[1]:-} sha=${w[2]:-} job=${w[3]:-} runner=${w[4]:-} + valid_repo "$repo" && valid_sha "$sha" && valid_job "$job" && { [ -z "$runner" ] || valid_name "$runner"; } \ + || die "usage: done <repo> <sha> <job-id> [runner]" + # A runner that lost its lease (laptop slept, job re-claimed by another runner) must not remove the new holder's claim. + holder=$(cat "$QUEUE/running/$job.beat" 2>/dev/null) + # (Every current runner has a .beat file, even without tags; only an old runner, which sends no runner name, has none.) + [ -z "$runner" ] || [ "$holder" = "$runner" ] || die "job $job is not held by $runner (held by: ${holder:-nobody})" rm -f "$QUEUE/running/$job.json" "$QUEUE/running/$job.beat" "$QUEUE/declined/$job".* bg "$HOME/bin/fw" ci-dashboard ;; put-shot)