fuzzyworld-tools / commit
CI dashboard and live CI progress; fw ci-watch; v1.2.0
PLAN.md | 9 ++-- README.md | 13 ++++- bin/fw | 4 +- package.json | 2 +- runner/fw-runner.py | 47 ++++++++++++----- server/fw | 22 +++++++- server/gen.py | 145 ++++++++++++++++++++++++++++++++++++++++++++++++---- server/runner-api | 15 ++++-- 8 files changed, 222 insertions(+), 35 deletions(-) diff --git a/PLAN.md b/PLAN.md index 4739bcd..7e87d05 100644 --- a/PLAN.md +++ b/PLAN.md @@ -54,9 +54,9 @@ The order goes from cheap to expensive, and each step reuses what the one before 6. **Pull requests**: see [Pull requests](#pull-requests). Last, because it reuses the issues mailbox plumbing and needs the CI runner (approved CI runs on contributed code). -## Proposed: CI dashboard +## CI dashboard -Asked 2026-10-27: a way to watch CI progress. Proposal, not built: +✅ Built 2026-10-27 as proposed (README: "Watching CI"): - `code.fuzzyworld.net/ci/`: one static page for all public repos: queued jobs, running pipelines (which job, for how long), and the last 50 results. Rebuilt on every queue change and every upload from a runner. @@ -81,8 +81,9 @@ Asked 2026-10-27: a way to watch CI progress. Proposal, not built: (a wildcard subdomain, e.g. `*.vscode.fuzzyworld.net`, set up in the DreamHost panel) - VS Code in the browser: build from current VS Code source instead of the `vscode-web` 1.91.1 package (2024) - VS Code in the browser: project-wide TypeScript IntelliSense (turned off; the web TS server rejects our URI scheme) - - Rename `.gitlab-ci.yml` → `.fw-ci.yml` in the 42 repos (one commit each), dropping `pages` jobs that - only published to GitLab Pages + - ✅ Renamed `.gitlab-ci.yml` → `.fw-ci.yml` in 37 repos (2026-10-27; blister's is only local, never pushed). + Still to do: READMEs and plans that say "deployed to GitLab Pages by `.gitlab-ci.yml`" (stale since the move), + and dropping `pages` jobs that only published to GitLab Pages - Old `.gitlab-ci.yml` files: `node:*-alpine` images have no `git`, so installing the `fuzzyworld-tools` git dependency fails (e.g. bubbles). Switch those jobs to a `node:22` image, or delete `pages` jobs that `fw deploy` replaced diff --git a/README.md b/README.md index 510d495..a99ca68 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,7 @@ On the laptop, `bin/fw` runs `deploy` locally and forwards repo commands to Drea | `fw publish-all` | refresh every repo, then the index | | `fw ci <name> [branch]` | queue a CI run of a branch's latest commit (default: the default branch) | | `fw ci-list` | queued and running CI jobs | +| `fw ci-watch` | follow CI in the terminal (private repos too): running jobs with each job's state, the queue, the last 5 results; ends when nothing is queued or running | Push to a repo with `git push dh:git/<name>.git` (`dh` is the SSH alias in `~/.ssh/config`). @@ -130,6 +131,7 @@ Every repo uses the shared hook via `core.hooksPath=~/fw/tools/server/hooks`. `p | `/<name>/branch/<branch>/...` | the same tree/blob/history/blame/log pages for another branch; `/` in a branch name is written `~` | | `/<name>/ci/`, `/<name>/ci/<sha>.html` | CI pipelines (newest 100) and one pipeline with its job logs | | `/<name>/ci/badge.svg` | status of the latest pipeline on the default branch | +| `/ci/` | CI dashboard: running pipelines (each job's state and time), the queue, the last 50 results, for public repos | | `/<name>/files.json`, `/<name>/objects/<sha>` | for VS Code in the browser: every file's blob SHA and size, and each file's contents (≤ 1 MB) | | `/why/`, `/principles/`, `/contribute/` | about the site | @@ -182,8 +184,15 @@ coordinator service; the coordinator is files on DreamHost. 30-minute timeout unless the job sets `timeout`). All jobs of a pipeline share one workspace, so artifacts are already there for later stages. Inside jobs, `origin` is the read-only https clone URL (`https://code.fuzzyworld.net/<name>.git`), as on GitLab; the runner's ssh key never enters a container. -5. **Report.** Logs and `pipeline.json` go to `~/fw/ci/<name>/<sha>/`; each `pipeline.json` upload republishes the - repo's pages, which show ✓/✗ on the log, the summary and `/<name>/ci/`. +5. **Report.** Logs and `pipeline.json` go to `~/fw/ci/<name>/<sha>/`. While a job runs, the runner marks it + running and uploads its log every 15 seconds; each upload rebuilds that repo's CI pages and the dashboard + (`fw ci-pages`). When the pipeline finishes, the repo's pages are republished, showing ✓/✗ on the log, + the summary and `/<name>/ci/`. + +**Watching CI:** `code.fuzzyworld.net/ci/` shows what is running (each job ✓/✗/● with its time), what is queued +and the last 50 results. While anything is queued or running, that page and the running pipeline's page reload +themselves every 15 seconds (`<meta http-equiv="refresh">`; no JavaScript). Private repos are never named there, +only counted. In the terminal, `fw ci-watch` shows everything, private repos included. **Job definitions: `.fw-ci.yml`**, at the repo root, in a subset of GitLab CI's YAML format, so an old `.gitlab-ci.yml` works after a rename (`git mv .gitlab-ci.yml .fw-ci.yml`). Until a repo is renamed, the runner diff --git a/bin/fw b/bin/fw index 7b33a7c..43cfa89 100755 --- a/bin/fw +++ b/bin/fw @@ -3,7 +3,7 @@ # fw deploy [--ref <branch>] [--here] [--dry-run] build and publish this repo's app (see lib/deploy.mjs) # fw repoint [DIR] [--https] [--dry-run] move working copies' origin from GitLab to DreamHost # fw list | new <name> [--public] [--description TEXT] | public <name> | private <name> -# fw describe <name> TEXT | publish <name> | publish-all | ci <name> [branch] | ci-list +# fw describe <name> TEXT | publish <name> | publish-all | ci <name> [branch] | ci-list | ci-watch # (run on DreamHost over ssh) set -euo pipefail HOST=${FW_HOST:-dh} @@ -15,7 +15,7 @@ case "${1:-help}" in shift; exec node "$HERE/lib/deploy.mjs" "$@" ;; repoint) shift; exec bash "$HERE/lib/repoint.sh" "$@" ;; - list|new|public|private|describe|publish|publish-all|ci|ci-list|help|-h|--help) + list|new|public|private|describe|publish|publish-all|ci|ci-list|ci-watch|help|-h|--help) # printf %q keeps arguments (e.g. a description with spaces) intact through ssh. exec ssh "$HOST" "~/bin/fw $(printf '%q ' "$@")" ;; *) diff --git a/package.json b/package.json index 6ec8afc..7f5c732 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "fuzzyworld-tools", - "version": "1.1.0", + "version": "1.2.0", "description": "Tooling for code.fuzzyworld.net and apps.fuzzyworld.net: fw command, deploy, vite plugin.", "type": "module", "scripts": { diff --git a/runner/fw-runner.py b/runner/fw-runner.py index ea4e08a..d2b194d 100755 --- a/runner/fw-runner.py +++ b/runner/fw-runner.py @@ -67,6 +67,7 @@ PLAYWRIGHT = "1.49.1" SHOOT_IMAGE = f"fw-shoot:{PLAYWRIGHT}" # `fw deploy --dry-run` from the published fuzzyworld-tools, whether or not the app depends on it. FW_DEPLOY_CHECK = "npx --yes --package=git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0 fw deploy --here --dry-run" +LOG_EVERY = 15 # seconds between uploads of a running job's log (live CI pages) JOB_TIMEOUT = 30 * 60 # seconds, unless the job sets `timeout` MAX_LOG = 20 * 1024 * 1024 # runner-api refuses bigger files @@ -418,8 +419,9 @@ def git(cwd, *args): return subprocess.run(["git", "-C", cwd, *args], capture_output=True, text=True, check=True).stdout.strip() -def docker_run(name, image, workdir, env, script, timeout, logf, env_file=None, extra=()): - """Run a shell script in a fresh container; returns (exit code, seconds). Output goes to logf.""" +def docker_run(name, image, workdir, env, script, timeout, logf, env_file=None, extra=(), progress=None): + """Run a shell script in a fresh container; returns (exit code, seconds). Output goes to logf; + progress() is called every LOG_EVERY seconds while it runs.""" cmd = ["docker", "run", "--rm", "--name", name, "--cpus", CPUS, "--memory", MEMORY, "-v", f"{workdir}:/builds/{env['CI_PROJECT_NAME']}", "-w", f"/builds/{env['CI_PROJECT_NAME']}", *extra] for k, v in env.items(): @@ -430,13 +432,22 @@ def docker_run(name, image, workdir, env, script, timeout, logf, env_file=None, cmd += ["--entrypoint", "/bin/sh", image, "-c", "\n".join(lines)] start = time.time() p = subprocess.Popen(cmd, stdout=logf, stderr=subprocess.STDOUT) - try: - code = p.wait(timeout=timeout) - except subprocess.TimeoutExpired: - subprocess.run(["docker", "kill", name], capture_output=True) - p.wait() - logf.write(f"\n\nfw-runner: stopped after {timeout} seconds (the job's timeout)\n".encode()) - code = 124 + while True: + try: + code = p.wait(timeout=LOG_EVERY) + break + except subprocess.TimeoutExpired: + if time.time() - start >= timeout: + subprocess.run(["docker", "kill", name], capture_output=True) + p.wait() + logf.write(f"\n\nfw-runner: stopped after {timeout} seconds (the job's timeout)\n".encode()) + code = 124 + break + if progress: + try: + progress() + except Exception as e: # a failed upload must not stop the job + log(f"progress upload failed: {e}") return code, time.time() - start @@ -498,23 +509,30 @@ def run_pipeline(j, keep=False): rec.update(status="skipped", message="an earlier job failed", duration=0) continue log_path = os.path.join(workdir, rec["log"]) + rec.update(status="running", started=now_iso()) + put(repo, sha, "pipeline.json", json.dumps(pipeline)) + + def upload_log(path=log_path, name=rec["log"]): + with open(path, "rb") as f: + put(repo, sha, name, f.read()) with open(log_path, "wb") as logf: logf.write(f"fw-runner {RUNNER}: {x['name']} ({x['stage']}) in {x['image']}\n".encode()) if x["note"]: logf.write(f"note: {x['note']}\n".encode()) logf.flush() code, secs = docker_run(f"fw-{j['id']}-{i}", x["image"], src, x["env"], x["script"], - x["timeout"], logf, env_file) + x["timeout"], logf, env_file, progress=lambda: (logf.flush(), upload_log())) if x["after"]: logf.write(b"\n--- after_script\n") logf.flush() docker_run(f"fw-{j['id']}-{i}-after", x["image"], src, x["env"], x["after"], 300, logf, env_file) ok = code == 0 + rec.pop("started", None) rec.update(status="success" if ok else ("success" if x["allow_failure"] else "failed"), duration=round(secs), **({} if ok else {"message": f"exit code {code}" + (" (allowed to fail)" if x["allow_failure"] else "")})) failed = failed or (not ok and not x["allow_failure"]) - with open(log_path, "rb") as f: - put(repo, sha, rec["log"], f.read()) + upload_log() + put(repo, sha, "pipeline.json", json.dumps(pipeline)) pipeline.update(status="failed" if failed else "success", duration=round(time.time() - t0)) put(repo, sha, "pipeline.json", json.dumps(pipeline)) @@ -628,7 +646,12 @@ def main(argv): return plan(argv[2], argv[3] if len(argv) > 3 else "refs/heads/main") once, keep = "--once" in argv, "--keep" in argv os.makedirs(WORK, exist_ok=True) + started_as = os.path.getmtime(__file__) while True: + # runner/install.sh replaced this file: continue as the new version (between jobs, never during one). + if os.path.getmtime(__file__) != started_as: + log("fw-runner.py was updated; restarting") + os.execv(sys.executable, [sys.executable, os.path.abspath(__file__)] + argv[1:]) raw = api("claim", RUNNER).decode().strip() if not raw: return 0 diff --git a/server/fw b/server/fw index 8c2b3b2..a48aef7 100755 --- a/server/fw +++ b/server/fw @@ -15,6 +15,9 @@ # fw ci <name> [branch] queue a CI run of a branch's latest commit (default: the default branch) # fw ci-queue <name> <ref> <sha> queue a CI run (the push hook runs this) # fw ci-list queued and running CI jobs +# fw ci-watch follow CI (all repos, private too) until nothing is queued or running +# fw ci-pages <name> rebuild a repo's CI pages and the dashboard (runner-api runs this) +# fw ci-dashboard rebuild code.fuzzyworld.net/ci/ # # Visibility is `git config fuzzyworld.visibility` inside each repo. Anything other # than exactly "public" is treated as private. @@ -182,6 +185,7 @@ json.dump({"id": id, "repo": name, "ref": ref, "sha": sha, "default_branch": def "$id" "$name" "$ref" "$sha" "$default" "$QUEUE/$id.json.tmp" mv "$QUEUE/$id.json.tmp" "$QUEUE/pending/$id.json" echo "fw: CI queued for $name ${ref#refs/*/} ${sha:0:12} ($id)" + "${GEN[@]}" ci-dashboard || true } cmd_ci() { @@ -204,6 +208,19 @@ cmd_ci_list() { done } +cmd_ci_watch() { + local last="" now + while true; do + now=$("${GEN[@]}" ci-status) + if [ "$now" != "$last" ]; then + printf '\n%s\n%s\n' "$(date '+%T')" "$now" + last=$now + fi + grep -qE '^(running|queued) ' <<< "$now" || { echo "(nothing queued or running)"; return; } + sleep 10 + done +} + mkdir -p "$STATE" "$LOGS" case "${1:-}" in apps-index) "${GEN[@]}" apps ;; @@ -218,6 +235,9 @@ case "${1:-}" in ci) cmd_ci "${2:-}" "${3:-}" ;; ci-queue) cmd_ci_queue "${2:-}" "${3:-}" "${4:-}" ;; ci-list) cmd_ci_list ;; - ""|-h|--help|help) sed -n '2,20p' "$0" | sed 's/^# \{0,1\}//' ;; + ci-watch) cmd_ci_watch ;; + ci-pages) need_repo "${2:-}"; with_lock "$2" "${GEN[@]}" ci "$2" ;; + ci-dashboard) "${GEN[@]}" ci-dashboard ;; + ""|-h|--help|help) sed -n '2,23p' "$0" | sed 's/^# \{0,1\}//' ;; *) die "unknown command: $1 (try: fw help)" ;; esac diff --git a/server/gen.py b/server/gen.py index 0f72d1f..dfba4c8 100755 --- a/server/gen.py +++ b/server/gen.py @@ -4,6 +4,9 @@ gen.py repo <name> build or update the pages of one public repo gen.py index rebuild the site index (public repos only) gen.py apps rebuild the apps.fuzzyworld.net index from deployed apps + gen.py ci <name> rebuild one public repo's CI pages, then the CI dashboard + gen.py ci-dashboard rebuild code.fuzzyworld.net/ci/ (queue, running, recent; public repos only) + gen.py ci-status the same, as text for the terminal, private repos included (fw ci-watch) Only the git CLI and the standard library are required. From ~/fw/pylib, when importable: Python-Markdown renders README/.md files (otherwise plain text), and @@ -32,6 +35,7 @@ APPS = os.environ.get("FW_APPS_SITE", os.path.join(HOME, "apps.fuzzyworld.net")) STATE = os.environ.get("FW_STATE", os.path.join(HOME, "fw", "state")) PYLIB = os.environ.get("FW_PYLIB", os.path.join(HOME, "fw", "pylib")) CI_ROOT = os.environ.get("FW_CI", os.path.join(HOME, "fw", "ci")) +QUEUE = os.environ.get("FW_QUEUE", os.path.join(HOME, "fw", "queue")) BASE_URL = "https://code.fuzzyworld.net" APPS_URL = "https://apps.fuzzyworld.net" @@ -41,6 +45,8 @@ GEN_VERSION = 7 COMMIT_LIMIT = 2000 # commit pages per repo, across all branches; older commits: clone HISTORY_LIMIT = 200 # commits listed on a file's history page CI_LIMIT = 100 # pipelines listed on a repo's CI page +CI_RECENT = 50 # finished pipelines on the CI dashboard +CI_REFRESH = 15 # seconds: pages showing something running reload themselves SHOTS = "_shots" # apps.fuzzyworld.net/_shots/<app>/: screenshots from CI runners (no app name starts with "_") SHOT_EXTENSIONS = (".png", ".jpg", ".jpeg", ".webp") VSCODE_APP = "vscode" # apps.fuzzyworld.net/<this>/: the static VS Code web build (vscode-web/) @@ -136,14 +142,14 @@ def url(*parts): return "/" + "/".join(urllib.parse.quote(s) for s in path.split("/")) -def page(title, body, crumbs=None, view=None, site="code"): +def page(title, body, crumbs=None, view=None, site="code", refresh=None): if site == "apps": host, footer = "apps.fuzzyworld.net", (f'<a href="/">All apps</a> · <a href="{BASE_URL}/">Code</a> · ' f'<a href="{BASE_URL}/why/">Why this site exists</a> · ' f'<a href="{BASE_URL}/principles/">Principles</a>') else: host, footer = "code.fuzzyworld.net", (f'<a href="/">All repositories</a> · <a href="{APPS_URL}/">Apps</a> · ' - f'<a href="/contribute/">Contribute</a> · <a href="/why/">Why this site exists</a> · ' + f'<a href="/contribute/">Contribute</a> · <a href="/ci/">CI</a> · <a href="/why/">Why this site exists</a> · ' f'<a href="/principles/">Principles</a>') nav = "" if view: @@ -161,6 +167,7 @@ def page(title, body, crumbs=None, view=None, site="code"): <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> +{f'<meta http-equiv="refresh" content="{refresh}">' if refresh else ""} <title>{esc(title)} · {host}</title> <link rel="stylesheet" href="/style.css"> </head> @@ -288,12 +295,17 @@ def build_ci_pages(view): f'<td><code>{esc(p.get("ref", ""))}</code></td><td>{esc(CI_STATES[p["status"]][1])}</td>' f'<td class="num">{duration(p.get("duration"))}</td></tr>') page_path = os.path.join(out, sha + ".html") - src = os.path.join(CI_ROOT, name, sha, "pipeline.json") - if os.path.exists(page_path) and os.path.getmtime(page_path) >= os.path.getmtime(src): + # Rebuilt when pipeline.json or a log is newer: a runner uploads a running job's log every few seconds. + src_dir = os.path.join(CI_ROOT, name, sha) + newest = max(os.path.getmtime(os.path.join(src_dir, f)) for f in os.listdir(src_dir)) + if os.path.exists(page_path) and os.path.getmtime(page_path) >= newest: continue jobs = [] for j in p.get("jobs", []): st = j.get("status", "error") + secs = j.get("duration") + if st == "running" and j.get("started"): + secs = time.time() - iso_time(j["started"]) log, log_name = "", j.get("log", "") if re.fullmatch(r"[A-Za-z0-9._-]+\.log", log_name or ""): try: @@ -307,20 +319,25 @@ def build_ci_pages(view): except OSError: pass mark = CI_STATES.get(st, ("–", st, "")) - jobs.append(f'<details{" open" if st in ("failed", "error") else ""}><summary>' + jobs.append(f'<details{" open" if st in ("failed", "error", "running") else ""}><summary>' f'<span class="ci ci-{esc(st)}">{mark[0]}</span> <strong>{esc(j.get("name", "?"))}</strong>' - f' · {esc(j.get("stage", ""))} · {esc(mark[1])} · {duration(j.get("duration"))}' + f' · {esc(j.get("stage", ""))} · {esc(mark[1])} · {duration(secs)}' f'{" · " + esc(j["message"]) if j.get("message") else ""}</summary>{log}</details>') mark, words, _ = CI_STATES[p["status"]] + running = p["status"] == "running" body = (f'<h1>CI {esc(words)}: <code>{sha[:12]}</code></h1>' f'<p class="meta">Branch <code>{esc(p.get("ref", ""))}</code> · ' f'<a href="{url(name, "commit", sha)}.html">commit</a> · started {esc(p.get("started", "").replace("T", " ").replace("Z", " UTC"))} · ' - f'{duration(p.get("duration"))} · runner {esc(p.get("runner", ""))}</p>' + f'{"running" if running else duration(p.get("duration"))} · runner {esc(p.get("runner", ""))}</p>' + + (f'<p class="meta">This page reloads every {CI_REFRESH} seconds while the pipeline runs.</p>' if running else "") + (f'<p>{esc(p["message"])}</p>' if p.get("message") else "") + "".join(jobs)) - write(page_path, page(f"{name} CI {sha[:12]}", body, [f'<a href="{url(name)}/">{esc(name)}</a>', "CI"], view=view)) + write(page_path, page(f"{name} CI {sha[:12]}", body, [f'<a href="{url(name)}/">{esc(name)}</a>', "CI"], view=view, + refresh=CI_REFRESH if running else None)) + any_running = any(p["status"] == "running" for _, p in ordered) write(os.path.join(out, "index.html"), - page(f"{name} CI", f'<h1>CI</h1><table class="log">{"".join(rows)}</table>', view=view)) + page(f"{name} CI", f'<h1>CI</h1><p class="meta"><a href="/ci/">All CI</a></p><table class="log">{"".join(rows)}</table>', + view=view, refresh=CI_REFRESH if any_running else None)) latest = next((p for _, p in ordered if p.get("ref") == view.branch), None) if latest: write(os.path.join(out, "badge.svg"), badge_svg(latest["status"])) @@ -328,6 +345,105 @@ def build_ci_pages(view): remove(os.path.join(out, "badge.svg")) +def iso_time(s): + try: + return datetime.datetime.fromisoformat(s.replace("Z", "+00:00")).timestamp() + except (ValueError, AttributeError): + return 0 + + +def queue_jobs(kind): + """Jobs in ~/fw/queue/<kind>/ (pending or running), oldest first, with "since": the file's time.""" + d, out = os.path.join(QUEUE, kind), [] + for f in sorted(os.listdir(d)) if os.path.isdir(d) else []: + try: + with open(os.path.join(d, f)) as fh: + j = json.load(fh) + j["since"] = os.path.getmtime(os.path.join(d, f)) + out.append(j) + except (OSError, ValueError): + continue + return out + + +def public_repo(name, _cache={}): + if name not in _cache: + repo = os.path.join(GIT_ROOT, name + ".git") + _cache[name] = os.path.isdir(repo) and is_public(repo) + return _cache[name] + + +def ci_overview(include_private): + """(pending jobs, running jobs with their pipelines, recent finished pipelines) across all repos.""" + pending, running = queue_jobs("pending"), queue_jobs("running") + for j in running: + j["pipeline"] = ci_pipelines(j["repo"]).get(j["sha"], {}) + recent = [] + for name in (os.listdir(CI_ROOT) if os.path.isdir(CI_ROOT) else []): + if include_private or public_repo(name): + recent += [(p.get("started", ""), name, sha, p) for sha, p in ci_pipelines(name).items() if p["status"] != "running"] + recent.sort(reverse=True) + return pending, running, recent[:CI_RECENT] + + +def job_progress(p): + """ "test ✓ 44s · pages ● 12s" for a running pipeline.""" + parts = [] + for j in p.get("jobs", []): + st = j.get("status", "pending") + secs = time.time() - iso_time(j["started"]) if st == "running" and j.get("started") else j.get("duration") + mark = CI_STATES.get(st, ("·" if st == "pending" else "–", st, ""))[0] + parts.append(f'{j.get("name", "?")} {mark}' + (f" {duration(secs)}" if st not in ("pending", "skipped") else "")) + return " · ".join(parts) + + +def build_ci_dashboard(): + """code.fuzzyworld.net/ci/: what is queued, running and recently finished. Private repos are never named.""" + pending, running, recent = ci_overview(include_private=False) + def ref(j): + return j["ref"].split("/", 2)[-1] + hidden = sum(1 for j in pending + running if not public_repo(j["repo"])) + run_rows = "".join( + f'<tr><td><a href="{url(j["repo"])}/">{esc(j["repo"])}</a></td><td><code>{esc(ref(j))}</code></td>' + f'<td><a href="{url(j["repo"], "ci", j["sha"])}.html"><code>{j["sha"][:12]}</code></a></td>' + f'<td>{esc(job_progress(j["pipeline"]) or "starting")}</td>' + f'<td class="num">{duration(time.time() - j["since"])}</td></tr>' + for j in running if public_repo(j["repo"])) + queue_rows = "".join( + f'<tr><td><a href="{url(j["repo"])}/">{esc(j["repo"])}</a></td><td><code>{esc(ref(j))}</code></td>' + f'<td><code>{j["sha"][:12]}</code></td><td class="num">waiting {duration(time.time() - j["since"])}</td></tr>' + for j in pending if public_repo(j["repo"])) + done_rows = "".join( + f'<tr><td class="date">{esc(started[:16].replace("T", " "))}</td><td><a href="{url(name)}/">{esc(name)}</a></td>' + f'<td><code>{esc(p.get("ref", ""))}</code></td>' + f'<td>{ci_mark(name, sha, p)}<a href="{url(name, "ci", sha)}.html"><code>{sha[:12]}</code></a></td>' + f'<td>{esc(CI_STATES[p["status"]][1])}</td><td class="num">{duration(p.get("duration"))}</td></tr>' + for started, name, sha, p in recent) + busy = bool(pending or running) + body = ('<h1>CI</h1>' + '<p class="meta">Pipelines run on a runner at home, one at a time, as pushes queue them. ' + + (f'This page reloads every {CI_REFRESH} seconds while anything is queued or running.' if busy else "Nothing is running.") + + (f' Also {hidden} job{"s" if hidden != 1 else ""} for private repositories.' if hidden else "") + '</p>' + + (f'<h2>Running</h2><table class="log">{run_rows}</table>' if run_rows else "") + + (f'<h2>Queued</h2><table class="log">{queue_rows}</table>' if queue_rows else "") + + (f'<h2>Recent</h2><table class="log">{done_rows}</table>' if done_rows else "<p>No pipelines yet.</p>")) + write(os.path.join(SITE, "ci", "index.html"), page("CI", body, refresh=CI_REFRESH if busy else None)) + + +def ci_status_text(): + """For the terminal (fw ci-watch): queue, running and the last few results, private repos included.""" + pending, running, recent = ci_overview(include_private=True) + lines = [] + for j in running: + lines.append(f'running {j["repo"]} {j["ref"].split("/", 2)[-1]} {j["sha"][:12]} {job_progress(j["pipeline"]) or "starting"}') + for j in pending: + lines.append(f'queued {j["repo"]} {j["ref"].split("/", 2)[-1]} {j["sha"][:12]}') + for started, name, sha, p in recent[:5]: + mark, words, _ = CI_STATES[p["status"]] + lines.append(f'{mark} {words:<6} {name} {p.get("ref", "")} {sha[:12]} {started[:16].replace("T", " ")} {duration(p.get("duration"))}') + return "\n".join(lines) if lines else "no CI jobs yet" + + # ---------------------------------------------------------------- views class View: @@ -998,6 +1114,7 @@ def build_repo(name): def build_index(): + build_ci_dashboard() rows = [] for entry in sorted(os.listdir(GIT_ROOT)): if not entry.endswith(".git"): @@ -1132,6 +1249,16 @@ def main(argv): build_index() elif len(argv) == 2 and argv[1] == "apps": build_apps_index() + elif len(argv) == 3 and argv[1] == "ci": + name = argv[2] + if public_repo(name): + branch = git_text(os.path.join(GIT_ROOT, name + ".git"), "symbolic-ref", "--short", "HEAD", check=False).strip() + build_ci_pages(View(name, branch or "main", True)) + build_ci_dashboard() + elif len(argv) == 2 and argv[1] == "ci-dashboard": + build_ci_dashboard() + elif len(argv) == 2 and argv[1] == "ci-status": + print(ci_status_text()) else: raise SystemExit(__doc__) diff --git a/server/runner-api b/server/runner-api index 27a3d5d..344c4dd 100755 --- a/server/runner-api +++ b/server/runner-api @@ -24,6 +24,8 @@ note() { echo "$(date -u '+%F %T') ${SSH_CLIENT%% *} $*" >> "$log"; } [[ "$cmd" == claim* ]] || note "$cmd" die() { echo "runner-api: $*" >&2; exit 1; } +# Page updates run in the background, so the runner never waits for them. +bg() { setsid nohup "$@" >> "$HOME/fw/logs/runner-api-pages.log" 2>&1 < /dev/null & } valid_name() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] && [[ "$1" != *.git ]]; } valid_repo() { valid_name "$1" && [ -d "$HOME/git/$1.git" ]; } valid_sha() { [[ "$1" =~ ^[0-9a-f]{40}$ ]]; } @@ -49,6 +51,7 @@ case "${w[0]:-}" in if mv "$QUEUE/pending/$f" "$QUEUE/running/$f" 2>/dev/null; then touch "$QUEUE/running/$f" note "$cmd: $f" + bg "$HOME/bin/fw" ci-dashboard cat "$QUEUE/running/$f" exit 0 fi @@ -62,14 +65,18 @@ case "${w[0]:-}" in valid_repo "$repo" && valid_sha "$sha" && [[ "$file" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*\.(log|json)$ ]] || die "usage: put <repo> <sha> <file.log|file.json>" mkdir -p "$CI/$repo/$sha" store "$CI/$repo/$sha/$file" - # A pipeline.json means the status changed: refresh the pages (in the background; publishing is locked per repo). - if [ "$file" = pipeline.json ]; then - setsid nohup "$HOME/bin/fw" publish "$repo" >> "$HOME/fw/logs/$repo.log" 2>&1 < /dev/null & + # A finished pipeline refreshes all the repo's pages (✓/✗ on its log and summary); anything else while + # it runs (a job starting, a growing log) only its CI pages and the dashboard. Both are locked per repo. + if [ "$file" = pipeline.json ] && ! grep -q '"status": "running"' "$CI/$repo/$sha/pipeline.json"; then + bg "$HOME/bin/fw" publish "$repo" + else + bg "$HOME/bin/fw" ci-pages "$repo" fi ;; done) repo=${w[1]:-} sha=${w[2]:-} job=${w[3]:-} valid_repo "$repo" && valid_sha "$sha" && valid_job "$job" || die "usage: done <repo> <sha> <job-id>" - rm -f "$QUEUE/running/$job.json" ;; + rm -f "$QUEUE/running/$job.json" + bg "$HOME/bin/fw" ci-dashboard ;; put-shot) app=${w[1]:-} file=${w[2]:-} valid_name "$app" && [[ "$file" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*\.(png|json)$ ]] || die "usage: put-shot <app> <file.png|shots.json>"