fuzzyworld-tools / commit
Highlighting, history, blame, branches, CI runner, screenshots, VS Code web; v1.1.0
.gitignore | 1 + LICENSE | 29 ++ PLAN.md | 181 ++++++++ README.md | 148 ++++++- bin/fw | 5 +- package.json | 10 +- runner/fw-runner.py | 654 +++++++++++++++++++++++++++++ runner/install.sh | 38 ++ runner/shoot.py | 64 +++ server/fw | 46 ++- server/gen.py | 841 +++++++++++++++++++++++++++++++------- server/hooks/post-receive | 18 +- server/install.sh | 6 +- server/runner-api | 93 +++++ server/site-apps/htaccess | 4 + server/site/contribute/index.html | 2 +- server/site/how/index.html | 10 +- server/site/htaccess | 18 + server/site/principles/index.html | 30 ++ server/site/style.css | 25 ++ server/site/why/index.html | 10 +- vscode-web/fw-fs/extension.js | 184 +++++++++ vscode-web/fw-fs/package.json | 21 + vscode-web/fw-fs/package.nls.json | 1 + vscode-web/index.html | 69 ++++ vscode-web/install.sh | 32 ++ 26 files changed, 2372 insertions(+), 168 deletions(-) diff --git a/.gitignore b/.gitignore index b908d4c..ae7c391 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ __pycache__/ *.pyc .DS_Store +.playwright-mcp/ diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..92e26f9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,29 @@ +MIT License + +Copyright (c) 2026 Pavel Safronov + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +--- + +The art in assets/ is by Kenney (https://www.kenney.nl) and is not covered by +the licence above. It is released under CC0 1.0 (public domain). The folders +whose names start with "kenney" hold the packs' files unchanged, each with its +pack's own licence file; assets/composed/ holds icons built from those files by +tools/compose-icons.sh. See the Attribution section of README.md. diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..4739bcd --- /dev/null +++ b/PLAN.md @@ -0,0 +1,181 @@ +# Plan + +Where fuzzyworld-tools goes from here: from "a static mirror of my repos" to a small, open-source +code host built on one rule: **DreamHost only serves files; other machines do the work.** + +This is a planning document. Nothing in it is promised; the order is the current priority. +Last updated 2026-10-27. + +## The machines + +| Machine | Does | Can't do | +|---|---|---| +| DreamHost (shared hosting) | serves static files with no rate limits; git over ssh; hourly cron; `post-receive` hooks | long-running processes, Chromium, V8's JIT (Node runs `--jitless`), native addons | +| TrueNAS (home, always on) | Docker: CI runner, headless browsers, screenshots, code-server; already mirrors every repo | be reached from the internet. It **pulls** work from DreamHost over ssh instead | +| Laptop | `fw deploy`, interactive work; can act as a second runner | stay on | + +Everything on the sites stays pre-generated: no app server, no database, no accounts, repo content is +never executed. Where a feature needs JavaScript, it is one script **we** ship (CSP `script-src 'self'`, +never anything from a repo). + +## Now (prioritized) + +Status: ✅ live · 🧪 built and tested, not installed yet · ⏳ not started. + +1. ✅ Apps index: newest deploy first, with what changed. Live since 2026-10-26. +2. 🧪 **LICENSE file** (MIT) in this repo. Each repo's license on its summary page. +3. 🧪 **Principles** in four places: `/why/`, `/how/`, a new `/principles/` page, and a "Philosophy" section in the + README. `/how/` is still a draft that `install.sh` doesn't install; the principle is in it for when it goes live. +4. 🧪 **File navigation** + 1. "Open in VS Code" link on every summary page (`vscode://vscode.git/clone?url=…`). + 2. Syntax highlighting at generation time (Pygments in `~/fw/pylib`; no JavaScript). + 3. Per-file history, blame, and browsing every branch not yet merged into the default branch. + 4. VS Code in the browser at `apps.fuzzyworld.net/vscode/?repo=<name>`: the `vscode-web` build plus a + read-only file-system extension reading `files.json` and `objects/<sha>` from code.fuzzyworld.net. +5. 🧪 **CI runner** on TrueNAS. See [CI](#ci). Tested on TrueNAS against the git mirrors (explodorama passed, + bubbles failed for a real reason: see below). +6. 🧪 **Screenshots** as a built-in runner job. See [Screenshots](#screenshots). Tested: explodorama. + +## Next (in this order) + +The order goes from cheap to expensive, and each step reuses what the one before it built. + +1. **Feeds**: Atom per repo (commits, releases), for app deploys, and site-wide. Smallest step: the data is already + gathered for the log pages and the apps index. +2. **Releases**: a page per tag (the tag's message as notes), `git archive` `.zip`/`.tar.gz`, and files the + runner builds (APKs, extensions). Import the release assets already in `~/code/gitlab-export/*/release-assets/`. + Comes early because some of those files exist nowhere else. Adds a releases feed to step 1. +3. **Stats and heatmap**, built together, since both are computed from the same pass over the history when pages are generated: + - stats: languages by bytes, size, commit count, share of commits made with AI (see [Provenance](#provenance)) + - heatmap: a commit-activity SVG per repo and site-wide +4. **Wiki**: render a repo's `docs/` (or `wiki/`) folder with a sidebar. Import the GitLab wikis in + `~/code/gitlab-export/*/wikis.json`. Static rendering only; editing comes with pull requests. +5. **Issues**: see [Issues](#issues). The first feature that reads email; builds the mailbox plumbing. +6. **Pull requests**: see [Pull requests](#pull-requests). Last, because it reuses the issues mailbox plumbing + and needs the CI runner (approved CI runs on contributed code). + +## Proposed: CI dashboard + +Asked 2026-10-27: a way to watch CI progress. Proposal, not built: + +- `code.fuzzyworld.net/ci/`: one static page for all public repos: queued jobs, running pipelines (which job, + for how long), and the last 50 results. Rebuilt on every queue change and every upload from a runner. +- Live logs: while a job runs, the runner uploads its log every ~15 seconds, so the job's page shows it growing. +- Pages that show something running reload themselves every 15 seconds (`<meta http-equiv="refresh">`), so no + JavaScript is needed and the site's `script-src 'none'` stays. +- Private repos: never named on the public page. For those, `fw ci-list` in the terminal (and a `fw ci-watch` + that follows a run and prints each job's result). + +## Backlog + +- File finder and code search (one shipped script plus a prebuilt index per repo) +- code-server on TrueNAS, for editing in the browser (private, over Tailscale) +- Provenance stats and transcript publishing (see below) +- An app at every commit: `apps.fuzzyworld.net/<app>/@<sha>/` (a service worker at `/<app>/` would cover those paths; must be solved first) +- OpenGraph images (link previews) from the screenshots +- Deploy automatically when CI passes on `main` +- Run GitHub Actions workflows through `act` (10 repos have `.github/workflows`) +- Publish the shared-origin deploy checks as a reusable lint +- Found while building "Now" (2026-10-27): + - VS Code in the browser: Markdown preview and other webviews need a separate domain per view + (a wildcard subdomain, e.g. `*.vscode.fuzzyworld.net`, set up in the DreamHost panel) + - VS Code in the browser: build from current VS Code source instead of the `vscode-web` 1.91.1 package (2024) + - VS Code in the browser: project-wide TypeScript IntelliSense (turned off; the web TS server rejects our URI scheme) + - Rename `.gitlab-ci.yml` → `.fw-ci.yml` in the 42 repos (one commit each), dropping `pages` jobs that + only published to GitLab Pages + - Old `.gitlab-ci.yml` files: `node:*-alpine` images have no `git`, so installing the + `fuzzyworld-tools` git dependency fails (e.g. bubbles). Switch those jobs to a `node:22` image, or delete + `pages` jobs that `fw deploy` replaced + - 34 repos still have a `fuzzyworld-deploy` branch from the migration, fully merged; they can be deleted + - CI `cache:` support (a per-repo volume on TrueNAS), and `services:` + - Branch pages: link to the default branch's page when a file is identical, instead of a copy + +## Principles + +Wording approved 2026-10-26: + +> Things made with AI should be shared freely with the world. The code and apps here were made with +> AI, so they are MIT-licensed: use them, change them, share them. +> +> My conversations with the AI are different. My words and my way of working are mine. I publish a +> conversation only when I choose to, I can take that permission back, and nobody may copy my words or +> my style of work without my express permission. + +**Style** (keep this note with the wording). Copyright protects my words; it generally doesn't protect a +style or a way of working. "Nobody may copy my style" is therefore a statement of my terms and wishes, +not something a license can enforce. The wording is written to be honest about that. + +## CI + +Built as described here; the README's CI section is the reference. Yes, this is the same pattern as GitLab runners: **a runner registers, pulls jobs, runs them in containers, +and reports back.** The difference is that there's no coordinator service: the coordinator is files on DreamHost. + +1. **Queue.** `post-receive` writes `~/fw/queue/pending/<repo>@<sha>.json` (repo, ref, sha, time). +2. **Claim.** Each runner polls `ssh dh fw runner claim --tags docker,chrome`. The server moves the job to + `running/` with one `rename`, which is atomic, so two runners can never take the same job. +3. **Registration = an ssh key.** A runner's key is limited in `authorized_keys` with `command="fw runner-api"`: + it can claim jobs and upload results, and can't get a shell. +4. **Run.** The runner clones over ssh and runs each job in a throwaway Docker container from the job's `image`. +5. **Report.** Logs and `status.json` are rsynced to `~/fw/ci/<repo>/<sha>/`; `gen.py` renders ✅/❌ on the + log and commit pages, a page per run with its log, and a badge. +6. **Stale jobs.** A cron job puts jobs that have been running too long back into `pending/`. +7. **More runners.** The laptop can run `fw runner --once`. Tags say what a runner can do (`docker`, `chrome`, `android`). + +**Job definitions: `.fw-ci.yml`** (decided 2026-10-27: our own name; the format stays a subset of GitLab CI's, +so moving a repo over is a rename; the old `.gitlab-ci.yml` name still works until then). 42 repos have one, and they use only `image`, +`stages`, `script`, `before_script`, `variables`, `rules` (branch conditions), `needs`, `artifacts`, `cache`. +36 of them are GitLab Pages jobs; a `pages` job maps to `fw deploy`. An unsupported keyword fails the job with +a clear message; it is never silently ignored. Repos without the file fall back to `fuzzyworld.checks` from +`package.json`. engye (20 KB: Android builds, Play Store, Firefox extension) is the hard case and the test of +how far the subset needs to go. + +**Security.** Secrets (`~/.config/<project>.env` on the runner) are given only to jobs on branches I push. +Code from contributors runs only after I approve it, because it runs on a machine in my home. + +## Screenshots + +A built-in runner job (tag `chrome`), run after a successful build: + +- photographs the **exact build** being deployed, served locally at `/<app>/`, so it matches the deploy +- configured in `package.json`: + `"screenshots": [{ "path": "", "size": "phone", "wait": 1500 }, { "path": "?demo", "size": "desktop" }]`, + or a script name for apps that need clicks first (`"screenshots": "pnpm shots"`) +- with no config: falls back to a `screenshots/` folder in the repo, or the README's first image +- used on the apps index, repo summary pages, and (later) OpenGraph images + +## Issues + +- Reports arrive at `bugs-k7q4+<repo>@fuzzyworld.net` (already live). +- A cron job reads the mailbox (IMAP, Python standard library) and renders each thread as static pages. +- Reporters' email addresses are never shown. +- **Configurable per repo**, the same way as visibility: `git config fuzzyworld.issues public|reviewed|private`, + set with `fw issues <repo> <mode>`: + - `public`: every report is published as it arrives + - `reviewed`: a report is published only after I mark it public + - `private`: nothing is published; reports only reach my mailbox + - anything other than exactly `public` or `reviewed` counts as `private`, so nothing is published by accident. + A private repo's issues are always private, whatever this says. +- Import the GitLab issues and notes in `~/code/gitlab-export/*/issues.json`. + +## Pull requests + +- Contributions arrive at `contrib-68uq+<repo>@fuzzyworld.net` as a branch URL or patches (already live). +- A cron job fetches each one into `refs/contrib/<n>` in the bare repo, never onto `main`. +- It renders a review page: the diff, CI results (after I approve the run), and the discussion from email replies. +- **Transcripts.** A submitter can attach the AI conversation behind the change, so the reviewer sees exactly + where they're coming from. It is the submitter's choice, published under the submitter's terms, and only on that pull request. +- I merge locally, as now. +- Import the GitLab merge requests in `~/code/gitlab-export/*/merge_requests.json`. + +## Provenance + +- **AI share.** Count commits with an AI `Co-Authored-By:` trailer, per repo and site-wide. +- **Transcripts** (Claude Code session → rendered page): + - opt-in, one conversation at a time + - with a redaction pass (paths, email addresses, tokens) and my review before anything is published + - each published transcript has a consent record: who gave permission, when, for which conversation + - `fw transcript withdraw <id>` removes it from the sites + - transcripts are **not** MIT; they say "shown with permission, all rights reserved" +- **First candidate:** the Firefox bug-fix conversation. Permission was given 2026-10-26 and can be withdrawn. + It is in a different session, so it still goes through redaction and review first. +- **My own conversations** (like the ones that built this tool) are not published. diff --git a/README.md b/README.md index 4558830..510d495 100644 --- a/README.md +++ b/README.md @@ -2,10 +2,26 @@ Tooling for Pavel's self-hosted code and apps on DreamHost: -- **code.fuzzyworld.net**: static browse pages and read-only `git clone` for public repos -- **apps.fuzzyworld.net**: deployed web apps, one per path (`/bubbles/`, `/slide/`, ...), published with `fw deploy` +- **code.fuzzyworld.net**: static browse pages and read-only `git clone` for public repos: every unmerged branch, syntax highlighting, file history, blame, CI results +- **apps.fuzzyworld.net**: deployed web apps, one per path (`/bubbles/`, `/slide/`, ...), published with `fw deploy`, with screenshots taken by CI +- **VS Code in the browser** (`apps.fuzzyworld.net/vscode/?repo=<name>`): any public repo, read-only, with quick open and search +- **CI runner** on TrueNAS: runs each push's pipeline (`.fw-ci.yml`) in Docker, like a GitLab runner -There is no app server. Pages are generated when you push; Apache only serves files. +There is no app server. Pages are generated when you push; Apache only serves files. Work that needs a real +machine (CI, screenshots) runs on TrueNAS, which pulls jobs from DreamHost over ssh. The plan and backlog are in +[PLAN.md](PLAN.md). + +## Philosophy + +Things made with AI should be shared freely with the world. The code and apps here were made with AI, so they +are MIT-licensed: use them, change them, share them. + +My conversations with the AI are different. My words and my way of working are mine. I publish a conversation +only when I choose to, I can take that permission back, and nobody may copy my words or my style of work without +my express permission. (Copyright protects my words. It generally doesn't protect a style or a way of working, +so that part is a statement of my terms and wishes, not something a license can enforce.) + +The same text is on [/principles/](https://code.fuzzyworld.net/principles/) and `/why/`, and in the `/how/` draft (not live yet). ## Making a repo deploy as an app @@ -45,6 +61,8 @@ Then `pnpm run deploy` / `npm run deploy`. **`fw deploy`**: `--dry-run` does everything except upload. +The apps index lists the most recently deployed app first. Under each app from a **public** repo it shows the commit subjects deployed since its previous deploy (up to 5, linked to their commit pages); apps from private repos show only the date, so their history stays private. `.fuzzyworld.json` holds only the latest deploy, so the index remembers earlier ones in `~/fw/state/apps.json`. + Apps hosted elsewhere are listed on the index from `server/site-apps/extra.json` (engye: Cloudflare Pages; `apps.fuzzyworld.net/engye/...` redirects there). ## Commands @@ -62,6 +80,8 @@ On the laptop, `bin/fw` runs `deploy` locally and forwards repo commands to Drea | `fw describe <name> TEXT` | set the description shown on the index and summary page | | `fw publish <name>` | refresh one repo's public copy and pages (the push hook does this) | | `fw publish-all` | refresh every repo, then the index | +| `fw ci <name> [branch]` | queue a CI run of a branch's latest commit (default: the default branch) | +| `fw ci-list` | queued and running CI jobs | Push to a repo with `git push dh:git/<name>.git` (`dh` is the SSH alias in `~/.ssh/config`). @@ -71,7 +91,7 @@ Each bare repo stores its own setting: `git config fuzzyworld.visibility public| **Anything other than exactly `public` is private**, including a missing value, so a repo is never published by accident. A private repo has no pages, no clone URL and no index entry. Pushing to it publishes nothing. -Other settings in the repo config: `fuzzyworld.description`, plus `fuzzyworld.gitlab-id` / `fuzzyworld.gitlab-path` for repos migrated from GitLab. +Other settings in the repo config: `fuzzyworld.description`, `fuzzyworld.ci` (`off`: pushes queue no CI jobs), plus `fuzzyworld.gitlab-id` / `fuzzyworld.gitlab-path` for repos migrated from GitLab. ## Layout on DreamHost @@ -83,9 +103,13 @@ Other settings in the repo config: `fuzzyworld.description`, plus `fuzzyworld.gi | `~/apps.fuzzyworld.net/<app>/` | deployed apps (`fw deploy`); `index.html` at the root is generated | | `~/fw/tools/server/` | this repo's `server/` directory, installed by `server/install.sh` | | `~/bin/fw` | symlink to `~/fw/tools/server/fw` | -| `~/fw/state/` | per-repo page manifests and locks | +| `~/fw/state/` | per-repo page manifests and locks; `apps.json`: recent deploys per app (for the apps index) | | `~/fw/logs/<name>.log` | output of each push's publish run | -| `~/fw/pylib/` | Python-Markdown 3.7 (`pip install --target`), used for README rendering | +| `~/fw/pylib/` | Python-Markdown 3.7 (README rendering) and Pygments 2.21.0 (syntax highlighting), `pip install --target`; `install.sh` adds Pygments | +| `~/fw/queue/pending/`, `running/` | CI jobs, one JSON file each (see [CI](#ci)) | +| `~/fw/ci/<name>/<sha>/` | CI results uploaded by runners: `pipeline.json` and one log per job | +| `~/apps.fuzzyworld.net/_shots/<app>/` | screenshots of each app, uploaded by runners (`shots.json` lists them) | +| `~/apps.fuzzyworld.net/vscode/` | VS Code in the browser (`vscode-web/install.sh`; not listed on the apps index) | Every repo uses the shared hook via `core.hooksPath=~/fw/tools/server/hooks`. `post-receive` starts `fw publish <name>` in the background, so a push returns immediately; pages update about 1–5 seconds later. @@ -94,16 +118,27 @@ Every repo uses the shared hook via `core.hooksPath=~/fw/tools/server/hooks`. `p | URL | Page | |---|---| | `/` | index of public repos, newest first | -| `/<name>/` | summary: description, clone command, recent commits, README | +| `/<name>/` | summary: description, clone command, "Open in VS Code" links, license, CI status, screenshots, recent commits, README | | `/<name>/tree/<dir>/` | directory listing (default branch) | -| `/<name>/blob/<path>.html` | file view with line numbers; `.md` files rendered | +| `/<name>/blob/<path>.html` | file view with line numbers and syntax highlighting (Pygments; files ≤ 512 KB); `.md` files rendered | +| `/<name>/history/<path>.html` | commits that changed the file (up to 200; renames not followed) | +| `/<name>/blame/<path>.html` | the commit that last changed each line (text files ≤ 256 KB) | | `/<name>/raw/<path>` | raw copies of images only (png, jpg, gif, webp, ico, avif, bmp, svg; ≤ 2 MB) | -| `/<name>/log/` | last 100 commits | -| `/<name>/commit/<sha>.html` | one commit with its diff (truncated at 300 KB) | +| `/<name>/log/` | the branch's commits (up to 2000) | +| `/<name>/commit/<sha>.html` | one commit with its diff (truncated at 300 KB); a page for every commit on every branch, up to 2000 per repo | | `/<name>/refs/` | branches and tags | -| `/why/` | why the site exists | +| `/<name>/branch/<branch>/...` | the same tree/blob/history/blame/log pages for another branch; `/` in a branch name is written `~` | +| `/<name>/ci/`, `/<name>/ci/<sha>.html` | CI pipelines (newest 100) and one pipeline with its job logs | +| `/<name>/ci/badge.svg` | status of the latest pipeline on the default branch | +| `/<name>/files.json`, `/<name>/objects/<sha>` | for VS Code in the browser: every file's blob SHA and size, and each file's contents (≤ 1 MB) | +| `/why/`, `/principles/`, `/contribute/` | about the site | -Only the default branch can be browsed. Every branch and tag can be cloned. Author email addresses are not shown. +**Branches:** the default branch, and every branch **not yet merged** into it, get browse pages. A merged branch +adds nothing new to read and would duplicate every page, so the branches page lists it with a link to its last +commit. Every branch and tag can be cloned. Author email addresses are not shown. + +**License:** the summary page names the repo's license, read from a root `LICENSE`/`LICENCE`/`COPYING` file +(MIT, Apache-2.0, GPL, LGPL, AGPL, MPL-2.0, BSD, ISC, Unlicense, CC0), else from `package.json`. ## Security @@ -113,6 +148,8 @@ The site never executes anything from a repo: - raw copies are written only for image extensions; `.htaccess` also denies `.php`, `.shtml`, `.cgi`, `.pl`, `.py` and `.sh` and disables SSI/CGI (DreamHost runs `.php` files and SSI otherwise; tested 2026-10-26) - `Content-Security-Policy: script-src 'none'` on every response (this also covers SVG images) - inside public clone copies, `config`, `description`, `hooks/` and `logs/` return 404 +- `objects/<sha>` (file contents for VS Code) is always sent as `application/octet-stream` with `Content-Disposition: attachment`, so a repo's `.html` or `.svg` file can never render as a page; `files.json` and `objects/` allow cross-origin reads, because VS Code runs on apps.fuzzyworld.net +- CI logs are escaped like everything else; CI pages exist only for public repos - apps.fuzzyworld.net gets the same no-execution rules. HTML, service workers and manifests are sent with `no-cache` (a deploy is seen immediately); `/<app>/assets/` (hashed by vite) is cached for a year. ## Scheduled jobs @@ -129,13 +166,100 @@ Secrets are in `~/.config/<project>.env` on DreamHost (mode 600, never in a repo **Node on DreamHost:** the system Node is 18. Jobs use Node 22 from `~/fw/node` (the official linux-x64 build; checksum-verified), always with `NODE_OPTIONS=--jitless`. DreamHost's grsec kernel refuses executable memory, so V8's JIT crashes without it. npm installs use `--ignore-scripts`, because native addons can't be built there. `jq` 1.7.1 (the official static build) is in `~/bin`. +## CI + +The same pattern as GitLab runners: a runner pulls jobs, runs them in containers, and reports back. There is no +coordinator service; the coordinator is files on DreamHost. + +1. **Queue.** `post-receive` queues a job (`~/fw/queue/pending/<id>.json`) for each pushed branch or tag whose + commit has a `.fw-ci.yml` (or `.gitlab-ci.yml`, see below) or `package.json` (unless `fuzzyworld.ci` is `off`). `fw ci <name>` queues one by hand. +2. **Claim.** The runner (`runner/fw-runner.py`, on TrueNAS, from cron every 2 minutes) runs + `ssh dh claim <runner>`. The server moves the job into `running/` with one `rename`, which is atomic, so two + runners never take the same job. A job running for over 2 hours goes back to `pending/`. +3. **The runner's key** is limited in `authorized_keys` to `server/runner-api` + (`restrict,command="…/runner-api"`): it can claim jobs, fetch repos, and upload results; it can't get a shell. +4. **Run.** The runner clones the commit and runs each job in a fresh Docker container (`--cpus 4 --memory 4g`, + 30-minute timeout unless the job sets `timeout`). All jobs of a pipeline share one workspace, so artifacts are + already there for later stages. Inside jobs, `origin` is the read-only https clone URL + (`https://code.fuzzyworld.net/<name>.git`), as on GitLab; the runner's ssh key never enters a container. +5. **Report.** Logs and `pipeline.json` go to `~/fw/ci/<name>/<sha>/`; each `pipeline.json` upload republishes the + repo's pages, which show ✓/✗ on the log, the summary and `/<name>/ci/`. + +**Job definitions: `.fw-ci.yml`**, at the repo root, in a subset of GitLab CI's YAML format, so an old +`.gitlab-ci.yml` works after a rename (`git mv .gitlab-ci.yml .fw-ci.yml`). Until a repo is renamed, the runner +still reads `.gitlab-ci.yml`; `.fw-ci.yml` wins if both exist. Supported: `stages`, `variables`, `image` (string or `name:`), +`default:` (image, before_script, after_script), `workflow:rules`, `extends` (hidden jobs), and per job `stage`, +`image`, `script`, `before_script`, `after_script`, `variables`, `rules` (`if`, `when`, `exists`; `changes` always +matches), `only`/`except` (refs), `when` (`on_success`, `on_failure`, `always`, `manual`, `never`), +`allow_failure`, `timeout`, `needs` (ordering comes from stages). Ignored, because they change nothing here: +`artifacts`, `cache`, `dependencies`, `environment`, `tags`, `interruptible`, `retry`, `coverage`, `resource_group`. +**Anything else** (e.g. `include`, `services`, `release`, `trigger`, `parallel`) makes that job fail with a message +naming the key; nothing is silently skipped. `manual` jobs are listed but not run. The usual `CI_*` variables are +set (`CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`/`CI_COMMIT_TAG`, `CI_COMMIT_REF_SLUG`, `CI_DEFAULT_BRANCH`, +`CI_PIPELINE_SOURCE=push`, ...), plus `FUZZYWORLD_CI=true`. Jobs without an image use `node:22-bookworm`. A `pages` +job runs its build as a check; publishing is `fw deploy`'s job. + +**Without a pipeline file:** an app (`package.json` with `fuzzyworld.app`) gets one job, `fw deploy --here --dry-run` +(install, prebuild, checks, build, output checks); other node repos get their `typecheck`/`lint`/`test` scripts. + +**Secrets:** `<runner dir>/secrets/<repo>.env` on TrueNAS, passed (`--env-file`) only to jobs on the default branch +and tags. Without one, jobs that deploy elsewhere (e.g. `dreamhost`'s rsync, engye's store uploads) fail. + +Check what a checkout would run, without running it: `runner/fw-runner.py --plan <dir> [refs/heads/<branch>]`. + +### Screenshots + +After a successful pipeline on the default branch, an app (`fuzzyworld.app` in `package.json`) is built the way +`fw deploy` builds it, served at `/<app>/` inside a Playwright container with no network, and photographed. The +pictures go to `apps.fuzzyworld.net/_shots/<app>/` and appear on the apps index and the repo's summary page. + +```jsonc +"fuzzyworld": { + "screenshots": [{ "path": "", "size": "phone", "wait": 1500 }, { "path": "?demo", "size": "desktop" }], + // sizes: phone 390×844, tablet 820×1180, desktop 1280×800. Default: one phone shot of the start page. + // or a command that writes its own PNGs to $FW_SHOTS_DIR, for apps that need clicks first: + // "screenshots": "pnpm shots", + // or false: no screenshots +} +``` + +Without runner screenshots, the images in a public repo's `screenshots/` folder are shown instead. + +## VS Code in the browser + +`apps.fuzzyworld.net/vscode/?repo=<name>` (and `&branch=<branch>` for an unmerged branch) opens a public repo in +VS Code, linked from each summary page. It is the open-source VS Code web build (the `vscode-web` npm package, +1.91.1, MIT) plus `vscode-web/fw-fs/`, an extension that reads `files.json` and `objects/<sha>` from +code.fuzzyworld.net. Everything is static files; nothing can be changed (clone to edit). Quick open (Ctrl/Cmd+P) +and search across files work; search reads text files up to 256 KB. Not yet: Markdown preview and other webviews +(they need a separate domain per view), and project-wide TypeScript IntelliSense (turned off; syntax colors work). + +The plain "Open in VS Code" link (`vscode://vscode.git/clone?url=…`) clones the repo into the visitor's own VS Code. + ## Install / update ```bash server/install.sh # copy server/ to dh, install site files and hooks server/install.sh --publish # ...and republish every repo +runner/install.sh # CI runner on TrueNAS: copy it, make its ssh key, add the cron job +runner/install.sh --authorize # ...and allow that key on DreamHost (runner-api only) +vscode-web/install.sh # VS Code in the browser at apps.fuzzyworld.net/vscode/ ``` +The same actions as npm scripts (`npm run` lists them): + +| Script | Runs | +|---|---| +| `npm run update-server` | `server/install.sh` | +| `npm run update-server-and-republish` | `server/install.sh --publish` | +| `npm run republish-all` | `bin/fw publish-all` | +| `npm run list-repos` | `bin/fw list` | +| `npm run update-runner` | `runner/install.sh` | +| `npm run update-vscode` | `vscode-web/install.sh` | + +`install.sh` copies `server/` from the working copy, not from a commit, so uncommitted changes there go live too. + Requirements on DreamHost: git, Python 3.12, `flock`, `setsid`. On the laptop: Node ≥ 18, git, rsync, ssh with the `dh` alias. +On TrueNAS (runner): Docker, Python 3 with PyYAML (both part of TrueNAS SCALE), and the `truenas` ssh alias on the laptop. Using `fw` outside an app repo (e.g. on a second laptop): `npm install -g "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0"`. diff --git a/bin/fw b/bin/fw index 3815490..7b33a7c 100755 --- a/bin/fw +++ b/bin/fw @@ -3,7 +3,8 @@ # fw deploy [--ref <branch>] [--here] [--dry-run] build and publish this repo's app (see lib/deploy.mjs) # fw repoint [DIR] [--https] [--dry-run] move working copies' origin from GitLab to DreamHost # fw list | new <name> [--public] [--description TEXT] | public <name> | private <name> -# fw describe <name> TEXT | publish <name> | publish-all (run on DreamHost over ssh) +# fw describe <name> TEXT | publish <name> | publish-all | ci <name> [branch] | ci-list +# (run on DreamHost over ssh) set -euo pipefail HOST=${FW_HOST:-dh} # Follow the symlink npm creates in node_modules/.bin to find this package. @@ -14,7 +15,7 @@ case "${1:-help}" in shift; exec node "$HERE/lib/deploy.mjs" "$@" ;; repoint) shift; exec bash "$HERE/lib/repoint.sh" "$@" ;; - list|new|public|private|describe|publish|publish-all|help|-h|--help) + list|new|public|private|describe|publish|publish-all|ci|ci-list|help|-h|--help) # printf %q keeps arguments (e.g. a description with spaces) intact through ssh. exec ssh "$HOST" "~/bin/fw $(printf '%q ' "$@")" ;; *) diff --git a/package.json b/package.json index 4749b64..6ec8afc 100644 --- a/package.json +++ b/package.json @@ -1,8 +1,16 @@ { "name": "fuzzyworld-tools", - "version": "1.0.5", + "version": "1.1.0", "description": "Tooling for code.fuzzyworld.net and apps.fuzzyworld.net: fw command, deploy, vite plugin.", "type": "module", + "scripts": { + "update-server": "server/install.sh", + "update-server-and-republish": "server/install.sh --publish", + "republish-all": "bin/fw publish-all", + "list-repos": "bin/fw list", + "update-runner": "runner/install.sh", + "update-vscode": "vscode-web/install.sh" + }, "bin": { "fw": "bin/fw" }, diff --git a/runner/fw-runner.py b/runner/fw-runner.py new file mode 100755 index 0000000..ea4e08a --- /dev/null +++ b/runner/fw-runner.py @@ -0,0 +1,654 @@ +#!/usr/bin/env python3 +"""fw-runner: CI for the repos on DreamHost, in the style of a GitLab runner. + + fw-runner.py [--once] [--keep] take queued jobs and run them (cron runs this every minute) + fw-runner.py --plan <dir> [ref] print the jobs a checkout would run, and run nothing + +A push to DreamHost queues a job (server/hooks/post-receive). This runner claims jobs over +ssh with a key that may only use server/runner-api, clones the commit, runs the pipeline in +throwaway Docker containers, and uploads the logs and pipeline.json. DreamHost turns those +into static pages (code.fuzzyworld.net/<repo>/ci/). For apps, a successful pipeline on the +default branch is followed by screenshots of the build (shoot.py in a Playwright container). + +Pipelines come from .fw-ci.yml, written in a subset of GitLab CI's format (see SUPPORTED_JOB_KEYS); +a repo not yet moved over can still use the old name, .gitlab-ci.yml. Without either, a repo +whose package.json has a "fuzzyworld" section gets one job: `fw deploy --here --dry-run` +(install, checks, build, and the output checks), and other node repos get their +typecheck/lint/test scripts. + +Configuration (environment, or <dir>/runner.env next to this script): + FW_RUNNER_NAME name shown on the CI pages (default: the host name) + FW_RUNNER_HOST ssh destination of DreamHost (default fuzzygamer@iad1-shared-b7-30.dreamhost.com) + FW_RUNNER_KEY ssh key allowed only to run server/runner-api (default <dir>/.ssh/id_ed25519) + FW_RUNNER_WORK working directory for clones (default <dir>/work) + FW_RUNNER_SECRETS directory of <repo>.env files, given to default-branch and tag jobs only (default <dir>/secrets) + FW_RUNNER_CPUS / FW_RUNNER_MEMORY container limits (default 4 / 4g) +""" + +import fnmatch +import json +import os +import re +import shlex +import shutil +import socket +import subprocess +import sys +import time + +HERE = os.path.dirname(os.path.abspath(__file__)) +# The pipeline file: our own name first; GitLab's name still works for repos not moved over yet. +CI_FILES = (".fw-ci.yml", ".gitlab-ci.yml") + + +def load_env_file(path): + if os.path.isfile(path): + for line in open(path): + line = line.strip() + if line and not line.startswith("#") and "=" in line: + k, v = line.split("=", 1) + os.environ.setdefault(k.strip(), v.strip().strip('"')) + + +load_env_file(os.path.join(HERE, "runner.env")) +RUNNER = os.environ.get("FW_RUNNER_NAME") or socket.gethostname().split(".")[0] +HOST = os.environ.get("FW_RUNNER_HOST", "fuzzygamer@iad1-shared-b7-30.dreamhost.com") +KEY = os.environ.get("FW_RUNNER_KEY", os.path.join(HERE, ".ssh", "id_ed25519")) +WORK = os.environ.get("FW_RUNNER_WORK", os.path.join(HERE, "work")) +SECRETS = os.environ.get("FW_RUNNER_SECRETS", os.path.join(HERE, "secrets")) +CPUS = os.environ.get("FW_RUNNER_CPUS", "4") +MEMORY = os.environ.get("FW_RUNNER_MEMORY", "4g") +APPS_URL = "https://apps.fuzzyworld.net" +CODE_URL = "https://code.fuzzyworld.net" + +DEFAULT_IMAGE = "node:22-bookworm" # for jobs that name no image (GitLab leaves this to the runner) +# Playwright's image has the browsers but not the Python package; fw-runner adds it once (shoot_image()). +PLAYWRIGHT = "1.49.1" +SHOOT_IMAGE = f"fw-shoot:{PLAYWRIGHT}" +# `fw deploy --dry-run` from the published fuzzyworld-tools, whether or not the app depends on it. +FW_DEPLOY_CHECK = "npx --yes --package=git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0 fw deploy --here --dry-run" +JOB_TIMEOUT = 30 * 60 # seconds, unless the job sets `timeout` +MAX_LOG = 20 * 1024 * 1024 # runner-api refuses bigger files + +SSH = ["ssh", "-i", KEY, "-o", "IdentitiesOnly=yes", "-o", "BatchMode=yes", + "-o", "StrictHostKeyChecking=accept-new", "-o", f"UserKnownHostsFile={os.path.join(HERE, '.ssh', 'known_hosts')}"] + +# Keys of a pipeline job that this runner understands. Unsupported keys stop the job with a +# message saying so: a job is never run with part of its definition silently ignored. Keys in +# IGNORED_JOB_KEYS don't change what a job does here (every job of a pipeline shares one workspace, +# so artifacts are already there for later stages). +SUPPORTED_JOB_KEYS = {"stage", "image", "script", "before_script", "after_script", "variables", "rules", + "only", "except", "when", "allow_failure", "extends", "timeout", "needs"} +IGNORED_JOB_KEYS = {"artifacts", "cache", "dependencies", "environment", "tags", "interruptible", "retry", + "coverage", "resource_group", "pages", "id_tokens", "publish"} +TOP_LEVEL_KEYS = {"stages", "variables", "image", "default", "workflow", "before_script", "after_script", "cache"} +RESERVED = TOP_LEVEL_KEYS | {"include", "types"} + + +class Unsupported(Exception): + pass + + +def log(msg): + print(f"{time.strftime('%F %T')} {msg}", flush=True) + + +def now_iso(): + return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + + +# ---------------------------------------------------------------- DreamHost API + +def api(*args, data=None): + r = subprocess.run(SSH + [HOST, " ".join(args)], input=data, capture_output=True, timeout=300) + if r.returncode != 0: + raise RuntimeError(f"runner-api {args[0]}: {r.stderr.decode(errors='replace').strip()}") + return r.stdout + + +def clone_url(repo): + return f"{HOST}:git/{repo}.git" + + +def put(repo, sha, name, data): + if isinstance(data, str): + data = data.encode() + if len(data) > MAX_LOG: + data = b"(start of log cut: it was over 20 MB)\n" + data[-(MAX_LOG - 100):] + api("put", repo, sha, name, data=data) + + +# ---------------------------------------------------------------- CI variables and expressions + +def slug(s): + return re.sub(r"[^a-z0-9]+", "-", s.lower()).strip("-")[:63].rstrip("-") + + +def expand(value, env): + """$VAR and ${VAR} from env; unknown variables become empty, as in GitLab.""" + if not isinstance(value, str): + return value + return re.sub(r"\$\{(\w+)\}|\$(\w+)", lambda m: env.get(m.group(1) or m.group(2), ""), value) + + +_TOKEN = re.compile(r"""\s*(?:(?P<op>==|!=|=~|!~|&&|\|\||\(|\))|(?P<var>\$\{?\w+\}?)|(?P<str>"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*') + |(?P<re>/(?:[^/\\]|\\.)*/[a-z]*)|(?P<null>null)|(?P<bad>\S))""", re.X) + + +def evaluate(expr, env): + """A GitLab `rules:if` expression: variables, strings, null, regexes, == != =~ !~ && || and ().""" + tokens = [] + for m in _TOKEN.finditer(expr): + if m.group("bad"): + raise Unsupported(f"can't read the expression {expr!r}") + kind = next(k for k in ("op", "var", "str", "re", "null") if m.group(k) is not None) + tokens.append((kind, m.group(kind))) + pos = 0 + + def peek(): + return tokens[pos] if pos < len(tokens) else (None, None) + + def take(): + nonlocal pos + pos += 1 + return tokens[pos - 1] + + def value(): + kind, v = take() + if kind == "op" and v == "(": + r = disj() + if take() != ("op", ")"): + raise Unsupported(f"unbalanced parentheses in {expr!r}") + return r + if kind == "var": + name = v.strip("${}") + return env.get(name) + if kind == "str": + return v[1:-1] + if kind == "re": + body, flags = v[1:].rsplit("/", 1) + return re.compile(body, re.I if "i" in flags else 0) + if kind == "null": + return None + raise Unsupported(f"can't read the expression {expr!r}") + + def comparison(): + left = value() + kind, op = peek() + if kind == "op" and op in ("==", "!=", "=~", "!~"): + take() + right = value() + if op in ("==", "!="): + return (left == right) == (op == "==") + matched = bool(isinstance(right, re.Pattern) and left is not None and right.search(left)) + return matched if op == "=~" else not matched + return bool(left) + + def conj(): + r = comparison() + while peek() == ("op", "&&"): + take() + r = comparison() and r + return r + + def disj(): + r = conj() + while peek() == ("op", "||"): + take() + r = conj() or r + return r + + result = disj() + if pos != len(tokens): + raise Unsupported(f"can't read the expression {expr!r}") + return result + + +def rules_when(rules, env, src): + """The `when` of the first matching rule, or None if no rule matches.""" + for rule in rules: + if not isinstance(rule, dict): + raise Unsupported(f"can't read rule {rule!r}") + unknown = set(rule) - {"if", "when", "exists", "changes", "allow_failure", "variables"} + if unknown: + raise Unsupported(f"rules: {', '.join(sorted(unknown))} not supported") + if "if" in rule and not evaluate(rule["if"], env): + continue + if "exists" in rule: + pats = rule["exists"] if isinstance(rule["exists"], list) else rule["exists"].get("paths", []) + if not any(glob_exists(src, expand(p, env)) for p in pats): + continue + # `changes` compares against the previous push in GitLab; every push is treated as a change here. + return rule.get("when", "on_success") + return None + + +def glob_exists(src, pattern): + for root, dirs, files in os.walk(src): + dirs[:] = [d for d in dirs if d != ".git"] + for f in files: + rel = os.path.relpath(os.path.join(root, f), src) + if fnmatch.fnmatch(rel, pattern): + return True + return False + + +def only_except_match(spec, env): + """Old-style `only`/`except` with refs: branches, tags, branch names and /regexes/.""" + refs = spec if isinstance(spec, list) else (spec or {}).get("refs", []) + if isinstance(spec, dict) and set(spec) - {"refs"}: + raise Unsupported(f"only/except: {', '.join(sorted(set(spec) - {'refs'}))} not supported") + name = env.get("CI_COMMIT_REF_NAME", "") + for r in refs: + if r == "branches" and env.get("CI_COMMIT_BRANCH"): + return True + if r == "tags" and env.get("CI_COMMIT_TAG"): + return True + if r in ("pushes", "web", "api"): + return True + m = re.fullmatch(r"/(.*)/(i?)", r) if isinstance(r, str) else None + if m and re.search(m.group(1), name, re.I if m.group(2) else 0): + return True + if r == name: + return True + return False + + +# ---------------------------------------------------------------- pipeline definition + +def merge(base, over): + out = dict(base) + for k, v in over.items(): + out[k] = merge(out[k], v) if isinstance(v, dict) and isinstance(out.get(k), dict) else v + return out + + +def resolve_extends(name, defs, seen=()): + job = defs[name] + ext = job.get("extends") + if not ext: + return job + base = {} + for e in ([ext] if isinstance(ext, str) else ext): + if e in seen or e not in defs: + raise Unsupported(f"{name}: extends {e!r}, which doesn't exist or loops") + base = merge(base, resolve_extends(e, defs, seen + (name,))) + return merge(base, {k: v for k, v in job.items() if k != "extends"}) + + +def script_lines(v): + if v is None: + return [] + items = v if isinstance(v, list) else [v] + out = [] + for i in items: + out.extend(script_lines(i) if isinstance(i, list) else [str(i)]) + return out + + +def ci_file(src): + return next((f for f in CI_FILES if os.path.exists(os.path.join(src, f))), None) + + +def yaml_pipeline(src, env): + """[(job dict)] from the pipeline file, in run order, or raise Unsupported.""" + import yaml # PyYAML; on TrueNAS it is part of the system Python + + with open(os.path.join(src, ci_file(src))) as f: + doc = yaml.safe_load(f) or {} + if "include" in doc: + raise Unsupported(f"include: is not supported; put the jobs in {ci_file(src)} itself") + variables = {k: str(expand(v.get("value", "") if isinstance(v, dict) else v, env)) + for k, v in (doc.get("variables") or {}).items()} + env = {**env, **variables} + wf = (doc.get("workflow") or {}).get("rules") + if wf is not None and rules_when(wf, env, src) in (None, "never"): + return [], "workflow:rules: no pipeline for this push" + default = doc.get("default") or {} + stages = doc.get("stages") or ["build", "test", "deploy"] + stages = [".pre"] + [s for s in stages if s not in (".pre", ".post")] + [".post"] + defs = {k: v for k, v in doc.items() if isinstance(v, dict) and k not in RESERVED} + jobs = [] + for name in defs: + if name.startswith("."): + continue # hidden: only used through extends + job = resolve_extends(name, defs) + unknown = set(job) - SUPPORTED_JOB_KEYS - IGNORED_JOB_KEYS + jenv = {**env, **{k: str(expand(v, env)) for k, v in (job.get("variables") or {}).items()}, + "CI_JOB_NAME": name, "CI_JOB_STAGE": job.get("stage", "test")} + if "rules" in job: + when = rules_when(job["rules"], jenv, src) + else: + only = job.get("only", ["branches", "tags"]) + when = job.get("when", "on_success") if only_except_match(only, jenv) and not ( + "except" in job and only_except_match(job["except"], jenv)) else None + if when in (None, "never"): + continue + stage = job.get("stage", "test") + image = job.get("image", default.get("image", doc.get("image", DEFAULT_IMAGE))) + if isinstance(image, dict): + if set(image) - {"name"}: + raise Unsupported(f"{name}: image: {', '.join(sorted(set(image) - {'name'}))} not supported") + image = image["name"] + before = job["before_script"] if "before_script" in job else default.get("before_script", doc.get("before_script")) + after = job["after_script"] if "after_script" in job else default.get("after_script", doc.get("after_script")) + jobs.append({ + "name": name, "stage": stage, "image": expand(image, jenv), "env": jenv, + "script": script_lines(before) + script_lines(job.get("script")), + "after": script_lines(after), "when": when, "allow_failure": bool(job.get("allow_failure", False)), + "timeout": parse_timeout(job.get("timeout")), + "unsupported": sorted(unknown), + "note": "GitLab Pages: the build runs as a check here; `fw deploy` publishes apps." if name == "pages" else "", + }) + order = {s: i for i, s in enumerate(stages)} + missing = sorted({j["stage"] for j in jobs if j["stage"] not in order}) + if missing: + raise Unsupported(f"stage(s) {', '.join(missing)} not listed under stages:") + jobs.sort(key=lambda j: order[j["stage"]]) + return jobs, "" if jobs else "no job's rules or only/except match this push" + + +def parse_timeout(v): + if not v: + return JOB_TIMEOUT + total = 0 + for n, unit in re.findall(r"(\d+)\s*(h|hours?|m|min(?:utes?)?|s|sec(?:onds?)?)", str(v)): + total += int(n) * {"h": 3600, "m": 60, "s": 1}[unit[0]] + return total or JOB_TIMEOUT + + +def package_pipeline(src, env): + """The pipeline for a repo without a pipeline file, from package.json.""" + try: + with open(os.path.join(src, "package.json")) as f: + pkg = json.load(f) + except (OSError, ValueError): + return [], "no .fw-ci.yml, and no readable package.json" + if (pkg.get("fuzzyworld") or {}).get("app"): + # The same steps `fw deploy` takes, without uploading: install, prebuild, checks, build, output checks. + return [job("deploy-check", "test", ["corepack enable", FW_DEPLOY_CHECK], env)], "" + scripts = pkg.get("scripts") or {} + checks = [s for s in ("typecheck", "lint", "test") if scripts.get(s) and "no test specified" not in scripts[s]] + if not checks: + return [], "no .fw-ci.yml, and package.json has no typecheck, lint or test script" + install = install_command(src) + pm = install.split()[0] + return [job("checks", "test", ["corepack enable", install] + [f"{pm} run {c}" for c in checks], env)], "" + + +def install_command(src): + if os.path.exists(os.path.join(src, "pnpm-lock.yaml")): + return "pnpm install --frozen-lockfile" + if os.path.exists(os.path.join(src, "yarn.lock")): + return "yarn install --frozen-lockfile" + if os.path.exists(os.path.join(src, "package-lock.json")): + return "npm ci" + return "npm install" + + +def job(name, stage, script, env): + return {"name": name, "stage": stage, "image": DEFAULT_IMAGE, "env": {**env, "CI_JOB_NAME": name, "CI_JOB_STAGE": stage}, + "script": script, "after": [], "when": "on_success", "allow_failure": False, + "timeout": JOB_TIMEOUT, "unsupported": [], "note": ""} + + +def ci_env(j, src): + ref, sha, repo, default = j["ref"], j["sha"], j["repo"], j["default_branch"] + name = ref.split("/", 2)[2] + msg = git(src, "log", "-1", "--format=%B", sha) + env = { + "CI": "true", "FUZZYWORLD_CI": "true", "CI_SERVER_NAME": "fuzzyworld", "CI_RUNNER_DESCRIPTION": RUNNER, + "CI_PROJECT_NAME": repo, "CI_PROJECT_PATH": repo, "CI_PROJECT_DIR": f"/builds/{repo}", + "CI_PROJECT_URL": f"{CODE_URL}/{repo}", "CI_REPOSITORY_URL": f"{CODE_URL}/{repo}.git", + "CI_DEFAULT_BRANCH": default, "CI_COMMIT_SHA": sha, "CI_COMMIT_SHORT_SHA": sha[:8], + "CI_COMMIT_REF_NAME": name, "CI_COMMIT_REF_SLUG": slug(name), + "CI_COMMIT_MESSAGE": msg, "CI_COMMIT_TITLE": msg.split("\n", 1)[0], + "CI_PIPELINE_SOURCE": "push", "CI_PIPELINE_ID": j["id"], "CI_PAGES_URL": f"{APPS_URL}/{repo}", + } + if ref.startswith("refs/tags/"): + env["CI_COMMIT_TAG"] = name + else: + env["CI_COMMIT_BRANCH"] = name + return env + + +# ---------------------------------------------------------------- running + +def git(cwd, *args): + return subprocess.run(["git", "-C", cwd, *args], capture_output=True, text=True, check=True).stdout.strip() + + +def docker_run(name, image, workdir, env, script, timeout, logf, env_file=None, extra=()): + """Run a shell script in a fresh container; returns (exit code, seconds). Output goes to logf.""" + cmd = ["docker", "run", "--rm", "--name", name, "--cpus", CPUS, "--memory", MEMORY, + "-v", f"{workdir}:/builds/{env['CI_PROJECT_NAME']}", "-w", f"/builds/{env['CI_PROJECT_NAME']}", *extra] + for k, v in env.items(): + cmd += ["-e", f"{k}={v}"] + if env_file: + cmd += ["--env-file", env_file] + lines = ["set -e"] + [f"printf '%s\\n' {shlex.quote('$ ' + line)}\n{line}" for line in script] + cmd += ["--entrypoint", "/bin/sh", image, "-c", "\n".join(lines)] + start = time.time() + p = subprocess.Popen(cmd, stdout=logf, stderr=subprocess.STDOUT) + try: + code = p.wait(timeout=timeout) + except subprocess.TimeoutExpired: + subprocess.run(["docker", "kill", name], capture_output=True) + p.wait() + logf.write(f"\n\nfw-runner: stopped after {timeout} seconds (the job's timeout)\n".encode()) + code = 124 + return code, time.time() - start + + +def run_pipeline(j, keep=False): + repo, sha = j["repo"], j["sha"] + workdir = os.path.join(WORK, j["id"]) + src = os.path.join(workdir, "src") + os.makedirs(workdir, exist_ok=True) + pipeline = {"repo": repo, "sha": sha, "ref": j["ref"].split("/", 2)[2], "runner": RUNNER, + "started": now_iso(), "status": "running", "jobs": []} + t0 = time.time() + try: + env_git = {**os.environ, "GIT_SSH_COMMAND": " ".join(shlex.quote(a) for a in SSH)} + subprocess.run(["git", "clone", "--quiet", "--no-checkout", clone_url(repo), src], + check=True, env=env_git, capture_output=True) + git(src, "checkout", "--quiet", "--detach", sha) + # Jobs that fetch (e.g. to build every branch) reach the repo the way GitLab jobs did: over https, + # read-only. The runner's own ssh key never goes into a container. (Private repos can't be fetched this way.) + git(src, "remote", "set-url", "origin", f"{CODE_URL}/{repo}.git") + subprocess.run(["git", "-C", src, "submodule", "update", "--init", "--recursive", "--quiet"], + env=env_git, capture_output=True) + env = ci_env(j, src) + try: + if ci_file(src): + jobs, why = yaml_pipeline(src, env) + else: + jobs, why = package_pipeline(src, env) + except Unsupported as e: + jobs, why = None, str(e) + except Exception as e: # e.g. YAML syntax + jobs, why = None, f"{ci_file(src)} could not be read: {e}" + if jobs is None: + pipeline.update(status="error", message=why, duration=time.time() - t0) + put(repo, sha, "pipeline.json", json.dumps(pipeline)) + return pipeline + if not jobs: + log(f"{repo} {sha[:12]}: nothing to run ({why})") + return None + pipeline["jobs"] = [{"name": x["name"], "stage": x["stage"], "status": "pending"} for x in jobs] + put(repo, sha, "pipeline.json", json.dumps(pipeline)) + + protected = j["ref"] == f"refs/heads/{j['default_branch']}" or j["ref"].startswith("refs/tags/") + secret = os.path.join(SECRETS, f"{repo}.env") + env_file = secret if protected and os.path.isfile(secret) else None + failed = False + for i, x in enumerate(jobs): + rec = pipeline["jobs"][i] + rec["log"] = f"{i:02d}-{re.sub(r'[^A-Za-z0-9._-]+', '-', x['name'])[:60]}.log" + if x["unsupported"]: + rec.update(status="error", message=f"not supported here: {', '.join(x['unsupported'])}", duration=0) + put(repo, sha, rec["log"], f"This job uses {', '.join(x['unsupported'])}, which fw-runner doesn't support.\n") + failed = failed or not x["allow_failure"] + continue + run_it = x["when"] == "always" or (x["when"] == "on_success" and not failed) or (x["when"] == "on_failure" and failed) + if x["when"] == "manual": + rec.update(status="skipped", message="manual job (not run automatically)", duration=0) + continue + if not run_it: + rec.update(status="skipped", message="an earlier job failed", duration=0) + continue + log_path = os.path.join(workdir, rec["log"]) + with open(log_path, "wb") as logf: + logf.write(f"fw-runner {RUNNER}: {x['name']} ({x['stage']}) in {x['image']}\n".encode()) + if x["note"]: + logf.write(f"note: {x['note']}\n".encode()) + logf.flush() + code, secs = docker_run(f"fw-{j['id']}-{i}", x["image"], src, x["env"], x["script"], + x["timeout"], logf, env_file) + if x["after"]: + logf.write(b"\n--- after_script\n") + logf.flush() + docker_run(f"fw-{j['id']}-{i}-after", x["image"], src, x["env"], x["after"], 300, logf, env_file) + ok = code == 0 + rec.update(status="success" if ok else ("success" if x["allow_failure"] else "failed"), + duration=round(secs), **({} if ok else {"message": f"exit code {code}" + (" (allowed to fail)" if x["allow_failure"] else "")})) + failed = failed or (not ok and not x["allow_failure"]) + with open(log_path, "rb") as f: + put(repo, sha, rec["log"], f.read()) + pipeline.update(status="failed" if failed else "success", duration=round(time.time() - t0)) + put(repo, sha, "pipeline.json", json.dumps(pipeline)) + + if not failed and protected and not j["ref"].startswith("refs/tags/"): + screenshots(j, src, workdir) + return pipeline + finally: + if not keep: + # Containers ran as root, so the workspace may hold root-owned files: remove it from a container too. + subprocess.run(["docker", "run", "--rm", "-v", f"{workdir}:/w", "alpine:3.20", "rm", "-rf", "/w/src"], + capture_output=True) + shutil.rmtree(workdir, ignore_errors=True) + + +# ---------------------------------------------------------------- screenshots + +SIZES = {"phone": (390, 844), "tablet": (820, 1180), "desktop": (1280, 800)} + + +def shoot_image(): + if subprocess.run(["docker", "image", "inspect", SHOOT_IMAGE], capture_output=True).returncode != 0: + dockerfile = (f"FROM mcr.microsoft.com/playwright/python:v{PLAYWRIGHT}-noble\n" + f"RUN pip install --no-cache-dir --break-system-packages playwright=={PLAYWRIGHT}\n") + subprocess.run(["docker", "build", "-q", "-t", SHOOT_IMAGE, "-"], input=dockerfile.encode(), check=True, capture_output=True) + return SHOOT_IMAGE + + +def screenshots(j, src, workdir): + """Build the app and photograph it, for apps (package.json "fuzzyworld.app") on the default branch.""" + try: + with open(os.path.join(src, "package.json")) as f: + cfg = json.load(f).get("fuzzyworld") or {} + except (OSError, ValueError): + return + app, shots = cfg.get("app"), cfg.get("screenshots", [{"path": "", "size": "phone"}]) + if not app or shots is False or shots == []: + return + repo, sha = j["repo"], j["sha"] + env = {**ci_env(j, src), "FW_APP": app, "FW_BASE": f"/{app}/"} + log_path = os.path.join(workdir, "screenshots.log") + shots_dir = os.path.join(workdir, "shots") + os.makedirs(shots_dir, exist_ok=True) + with open(log_path, "wb") as logf: + # 1. Build exactly as `fw deploy` would (fresh install, prebuild, checks, build, output checks). + script = ["corepack enable", FW_DEPLOY_CHECK] + if isinstance(shots, str): # the app takes its own pictures: a command that writes PNGs to $FW_SHOTS_DIR + script += [f"FW_SHOTS_DIR=/builds/{repo}/.fw-shots {shots}"] + code, _ = docker_run(f"fw-{j['id']}-build", DEFAULT_IMAGE, src, env, script, JOB_TIMEOUT, logf) + if code != 0: + log(f"{repo}: screenshot build failed (exit {code}); see {log_path}") + return + if isinstance(shots, str): + own = os.path.join(src, ".fw-shots") + files = sorted(f for f in os.listdir(own) if f.endswith(".png")) if os.path.isdir(own) else [] + for f in files: + shutil.copy(os.path.join(own, f), os.path.join(shots_dir, f)) + entries = [{"file": f, "label": os.path.splitext(f)[0]} for f in files] + else: + out = cfg.get("out") or next((d for d in ("dist", "public", "build") if os.path.exists(os.path.join(src, d, "index.html"))), None) + if not out: + return + spec = [{"path": s.get("path", ""), "size": s.get("size", "phone"), "wait": int(s.get("wait", 1500))} + for s in shots if isinstance(s, dict) and s.get("size", "phone") in SIZES] + entries = [{"file": f"{i + 1:02d}-{s['size']}.png", "label": s["path"] or s["size"], **s} for i, s in enumerate(spec)] + with open(os.path.join(shots_dir, "spec.json"), "w") as f: + json.dump({"app": app, "shots": [{**e, "w": SIZES[e["size"]][0], "h": SIZES[e["size"]][1]} for e in entries]}, f) + # 2. Serve the build at /<app>/ and photograph it, in a Playwright container without network access. + code, _ = docker_run(f"fw-{j['id']}-shoot", shoot_image(), src, env, + [f"python3 /fw/shoot.py /shots /builds/{repo}/{out}"], + 600, logf, extra=["--network", "none", "-v", f"{HERE}:/fw:ro", "-v", f"{shots_dir}:/shots"]) + if code != 0: + log(f"{repo}: screenshots failed (exit {code}); see {log_path}") + return + taken = [e for e in entries if os.path.exists(os.path.join(shots_dir, e["file"]))] + if not taken: + return + for e in taken: + with open(os.path.join(shots_dir, e["file"]), "rb") as f: + api("put-shot", app, e["file"], data=f.read()) + meta = {"commit": sha, "taken": now_iso(), "shots": [{"file": e["file"], "label": e["label"]} for e in taken]} + api("put-shot", app, "shots.json", data=json.dumps(meta).encode()) + api("shots-done", app, repo) + log(f"{repo}: {len(taken)} screenshot(s) of {app}") + + +# ---------------------------------------------------------------- main + +def plan(src, ref): + """--plan: what a push of this checkout would run (for checking a .fw-ci.yml).""" + sha = git(src, "rev-parse", "HEAD") + j = {"id": "plan", "repo": os.path.basename(os.path.abspath(src)), "sha": sha, "ref": ref, "default_branch": "main"} + env = ci_env(j, src) + try: + jobs, why = yaml_pipeline(src, env) if ci_file(src) else package_pipeline(src, env) + except Unsupported as e: + print(f"error: {e}") + return 1 + if not jobs: + print(f"nothing to run: {why}") + for x in jobs: + print(f"[{x['stage']}] {x['name']} image={x['image']} when={x['when']}" + + (f" UNSUPPORTED: {', '.join(x['unsupported'])}" if x["unsupported"] else "") + + (f"\n {x['note']}" if x["note"] else "")) + for line in x["script"]: + print(f" $ {line}") + return 0 + + +def main(argv): + if argv[1:2] == ["--plan"]: + return plan(argv[2], argv[3] if len(argv) > 3 else "refs/heads/main") + once, keep = "--once" in argv, "--keep" in argv + os.makedirs(WORK, exist_ok=True) + while True: + raw = api("claim", RUNNER).decode().strip() + if not raw: + return 0 + j = json.loads(raw) + log(f"claimed {j['id']}: {j['repo']} {j['ref']} {j['sha'][:12]}") + try: + p = run_pipeline(j, keep) + log(f"{j['id']}: {p['status'] if p else 'nothing to run'}") + except Exception as e: + log(f"{j['id']}: runner error: {e}") + try: + put(j["repo"], j["sha"], "pipeline.json", json.dumps({ + "repo": j["repo"], "sha": j["sha"], "ref": j["ref"].split("/", 2)[2], "runner": RUNNER, + "started": now_iso(), "status": "error", "message": f"runner error: {e}", "jobs": []})) + except Exception: + pass + api("done", j["repo"], j["sha"], j["id"]) + if once: + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/runner/install.sh b/runner/install.sh new file mode 100755 index 0000000..a9892d2 --- /dev/null +++ b/runner/install.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# Install or update the CI runner on TrueNAS. Run from the laptop: +# +# runner/install.sh copy the runner, make its ssh key, add the cron job +# runner/install.sh --authorize ...and allow that key on DreamHost (restricted to server/runner-api) +# +# Settings (environment): FW_RUNNER_SSH (default: truenas), FW_RUNNER_DIR (default: +# /mnt/main-2025/fw-runner), FW_HOST (DreamHost ssh alias, default: dh). +# The runner needs Docker and Python 3 with PyYAML on TrueNAS (both are part of TrueNAS SCALE). +set -euo pipefail +cd "$(dirname "$0")" +NAS=${FW_RUNNER_SSH:-truenas} +DIR=${FW_RUNNER_DIR:-/mnt/main-2025/fw-runner} +HOST=${FW_HOST:-dh} + +ssh "$NAS" "mkdir -p '$DIR/.ssh' '$DIR/secrets' '$DIR/work' && chmod 700 '$DIR/.ssh' '$DIR/secrets'" +rsync -rlpt fw-runner.py shoot.py "$NAS:$DIR/" +ssh "$NAS" "chmod +x '$DIR/fw-runner.py'; [ -f '$DIR/.ssh/id_ed25519' ] || ssh-keygen -q -t ed25519 -N '' -C 'fw-runner (CI only on DreamHost)' -f '$DIR/.ssh/id_ed25519'" +pub=$(ssh "$NAS" "cat '$DIR/.ssh/id_ed25519.pub'") +line="restrict,command=\"/home/fuzzygamer/fw/tools/server/runner-api\" $pub" + +# Cron job every 2 minutes, through TrueNAS's own cron settings so it survives updates. +# flock: a run that is still busy with a long job makes the next one exit at once. +cmd="flock -n $DIR/.lock python3 $DIR/fw-runner.py >> $DIR/runner.log 2>&1" +ssh "$NAS" "midclt call cronjob.query '[[\"description\",\"=\",\"fw-runner\"]]'" | grep -q '"id"' || \ + ssh "$NAS" "midclt call cronjob.create '$(python3 -c 'import json,sys; print(json.dumps({ + "description": "fw-runner", "user": "root", "command": sys.argv[1], "enabled": True, + "stdout": False, "stderr": False, # True would make TrueNAS append "> /dev/null", undoing the redirect to runner.log + "schedule": {"minute": "*/2", "hour": "*", "dom": "*", "month": "*", "dow": "*"}}))' "$cmd")'" >/dev/null +echo "runner installed in $NAS:$DIR (cron: every 2 minutes)" + +if [ "${1:-}" = --authorize ]; then + ssh "$HOST" "grep -qF '$pub' ~/.ssh/authorized_keys || echo '$line' >> ~/.ssh/authorized_keys" + echo "key allowed on $HOST (runner-api only)" +else + echo "To let it take jobs, add this line to ~/.ssh/authorized_keys on DreamHost (or rerun with --authorize):" + echo " $line" +fi diff --git a/runner/shoot.py b/runner/shoot.py new file mode 100755 index 0000000..56ef443 --- /dev/null +++ b/runner/shoot.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +"""Screenshots of a built app, run by fw-runner.py inside a Playwright container (no network). + + shoot.py <shots dir> <build output dir> + +<shots dir>/spec.json: {"app": "<app>", "shots": [{"file", "path", "w", "h", "wait"}, ...]}. +The build is served at http://127.0.0.1:<port>/<app>/, the same path it has on +apps.fuzzyworld.net, and each shot is written to <shots dir>/<file>. +""" + +import functools +import http.server +import json +import os +import sys +import threading + +from playwright.sync_api import sync_playwright + + +class AppHandler(http.server.SimpleHTTPRequestHandler): + """Serves the build directory under /<app>/, and nothing else.""" + + def __init__(self, *args, app, **kw): + self.app = app + super().__init__(*args, **kw) + + def translate_path(self, path): + prefix = f"/{self.app}/" + path = path.split("?", 1)[0].split("#", 1)[0] + if path == f"/{self.app}": + path = prefix + if not path.startswith(prefix): + return os.path.join(self.directory, ".fw-not-found") + return super().translate_path("/" + path[len(prefix):]) + + def log_message(self, *args): + pass + + +def main(shots_dir, out_dir): + with open(os.path.join(shots_dir, "spec.json")) as f: + spec = json.load(f) + app = spec["app"] + handler = functools.partial(AppHandler, app=app, directory=out_dir) + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler) + threading.Thread(target=server.serve_forever, daemon=True).start() + base = f"http://127.0.0.1:{server.server_address[1]}/{app}/" + with sync_playwright() as p: + browser = p.chromium.launch() + for s in spec["shots"]: + page = browser.new_page(viewport={"width": s["w"], "height": s["h"]}, device_scale_factor=1, + is_mobile=s["w"] < 600, has_touch=s["w"] < 600) + page.goto(base + s["path"].lstrip("/"), wait_until="load") + page.wait_for_timeout(s["wait"]) + page.screenshot(path=os.path.join(shots_dir, s["file"])) + page.close() + print(f"shot {s['file']}: {base + s['path'].lstrip('/')}", flush=True) + browser.close() + server.shutdown() + + +if __name__ == "__main__": + main(sys.argv[1], sys.argv[2]) diff --git a/server/fw b/server/fw index c2ca242..8c2b3b2 100755 --- a/server/fw +++ b/server/fw @@ -12,6 +12,9 @@ # fw install-hooks point every repo at the shared hooks # fw apps-index rebuild the apps.fuzzyworld.net index # fw app-deployed <app> <repo> after `fw deploy`: apps index, plus the repo's code page +# fw ci <name> [branch] queue a CI run of a branch's latest commit (default: the default branch) +# fw ci-queue <name> <ref> <sha> queue a CI run (the push hook runs this) +# fw ci-list queued and running CI jobs # # Visibility is `git config fuzzyworld.visibility` inside each repo. Anything other # than exactly "public" is treated as private. @@ -22,6 +25,7 @@ SITE=${FW_CODE_SITE:-$HOME/code.fuzzyworld.net} TOOLS=${FW_TOOLS:-$HOME/fw/tools} STATE=${FW_STATE:-$HOME/fw/state} LOGS=${FW_LOGS:-$HOME/fw/logs} +QUEUE=${FW_QUEUE:-$HOME/fw/queue} HOOKS=$TOOLS/server/hooks GEN=(python3 "$TOOLS/server/gen.py") @@ -163,6 +167,43 @@ cmd_app_deployed() { fi } +cmd_ci_queue() { + local name=${1:-} ref=${2:-} sha=${3:-} + [ -n "$name" ] && [ -n "$ref" ] && [[ "$sha" =~ ^[0-9a-f]{40}$ ]] || die "usage: fw ci-queue <name> <ref> <sha>" + need_repo "$name" + mkdir -p "$QUEUE/pending" "$QUEUE/running" + local id; id="$(date +%s%N | cut -c1-16)-$name-${sha:0:12}" + local default; default=$(git -C "$(repo_dir "$name")" symbolic-ref --short HEAD 2>/dev/null || echo main) + # Written beside the queue, then renamed in: a runner never sees a half-written job. + python3 -c 'import json, sys, time +id, name, ref, sha, default, out = sys.argv[1:] +json.dump({"id": id, "repo": name, "ref": ref, "sha": sha, "default_branch": default, + "queued": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}, open(out, "w"))' \ + "$id" "$name" "$ref" "$sha" "$default" "$QUEUE/$id.json.tmp" + mv "$QUEUE/$id.json.tmp" "$QUEUE/pending/$id.json" + echo "fw: CI queued for $name ${ref#refs/*/} ${sha:0:12} ($id)" +} + +cmd_ci() { + local name=${1:-} branch=${2:-} + [ -n "$name" ] || die "usage: fw ci <name> [branch]" + need_repo "$name" + local r; r=$(repo_dir "$name") + [ -n "$branch" ] || branch=$(git -C "$r" symbolic-ref --short HEAD) + local sha; sha=$(git -C "$r" rev-parse -q --verify "refs/heads/$branch^{commit}") || die "no branch $branch in $name" + cmd_ci_queue "$name" "refs/heads/$branch" "$sha" +} + +cmd_ci_list() { + local d f + for d in pending running; do + for f in "$QUEUE/$d"/*.json; do + [ -e "$f" ] || continue + printf '%-8s %s %s\n' "$d" "$(date -r "$f" '+%F %T')" "$(basename "$f" .json)" + done + done +} + mkdir -p "$STATE" "$LOGS" case "${1:-}" in apps-index) "${GEN[@]}" apps ;; @@ -174,6 +215,9 @@ case "${1:-}" in publish) cmd_publish "${2:-}" ;; publish-all) cmd_publish_all ;; install-hooks) cmd_install_hooks ;; - ""|-h|--help|help) sed -n '2,17p' "$0" | sed 's/^# \{0,1\}//' ;; + ci) cmd_ci "${2:-}" "${3:-}" ;; + ci-queue) cmd_ci_queue "${2:-}" "${3:-}" "${4:-}" ;; + ci-list) cmd_ci_list ;; + ""|-h|--help|help) sed -n '2,20p' "$0" | sed 's/^# \{0,1\}//' ;; *) die "unknown command: $1 (try: fw help)" ;; esac diff --git a/server/gen.py b/server/gen.py index 9fa1db6..0f72d1f 100755 --- a/server/gen.py +++ b/server/gen.py @@ -5,19 +5,21 @@ gen.py index rebuild the site index (public repos only) gen.py apps rebuild the apps.fuzzyworld.net index from deployed apps -Only the git CLI and the standard library are required. Python-Markdown is used -for README/.md rendering when importable (from ~/fw/pylib); otherwise those files -are shown as plain text. +Only the git CLI and the standard library are required. From ~/fw/pylib, when +importable: Python-Markdown renders README/.md files (otherwise plain text), and +Pygments colors source code (otherwise uncolored). Pages are incremental: a manifest per repo (~/fw/state/<name>.json) records the blob SHA behind every file page, so a push rewrites only what changed. """ +import datetime import html import json import os import posixpath import re +import shutil import subprocess import sys import time @@ -29,16 +31,28 @@ SITE = os.environ.get("FW_CODE_SITE", os.path.join(HOME, "code.fuzzyworld.net")) APPS = os.environ.get("FW_APPS_SITE", os.path.join(HOME, "apps.fuzzyworld.net")) STATE = os.environ.get("FW_STATE", os.path.join(HOME, "fw", "state")) PYLIB = os.environ.get("FW_PYLIB", os.path.join(HOME, "fw", "pylib")) +CI_ROOT = os.environ.get("FW_CI", os.path.join(HOME, "fw", "ci")) BASE_URL = "https://code.fuzzyworld.net" APPS_URL = "https://apps.fuzzyworld.net" # Bump when page templates change, so every repo gets a full rebuild once. -GEN_VERSION = 6 - -LOG_LIMIT = 100 +GEN_VERSION = 7 + +COMMIT_LIMIT = 2000 # commit pages per repo, across all branches; older commits: clone +HISTORY_LIMIT = 200 # commits listed on a file's history page +CI_LIMIT = 100 # pipelines listed on a repo's CI page +SHOTS = "_shots" # apps.fuzzyworld.net/_shots/<app>/: screenshots from CI runners (no app name starts with "_") +SHOT_EXTENSIONS = (".png", ".jpg", ".jpeg", ".webp") +VSCODE_APP = "vscode" # apps.fuzzyworld.net/<this>/: the static VS Code web build (vscode-web/) +APP_HISTORY = 20 # deploys remembered per app (~/fw/state/apps.json) +APP_CHANGES = 5 # commit subjects shown per app on the apps index +APP_FIRST_CHANGES = 3 # shown for an app's first recorded deploy, which has no previous one MAX_TEXT_BYTES = 1024 * 1024 MAX_DIFF_BYTES = 300 * 1024 MAX_RAW_BYTES = 2 * 1024 * 1024 +MAX_HIGHLIGHT_BYTES = 512 * 1024 +MAX_BLAME_BYTES = 256 * 1024 +MAX_LOG_BYTES = 256 * 1024 # the end of each CI job log is shown # Raw copies are written only for these: other extensions (.php, .shtml, ...) could be # executed by the web server, and the browse pages already show text files. # SVG is safe here because the site's CSP header (script-src 'none') covers SVG documents too. @@ -50,12 +64,26 @@ try: import markdown # type: ignore except ImportError: # pragma: no cover markdown = None +try: + import pygments.lexers as pygments_lexers # type: ignore + import pygments.token as pygments_token # type: ignore +except ImportError: # pragma: no cover + pygments_lexers = pygments_token = None # ---------------------------------------------------------------- git helpers def git(repo, *args, check=True): - r = subprocess.run(["git", "-C", repo, *args], capture_output=True, check=False) + # DreamHost sometimes refuses to start a process for a moment (EAGAIN) when many run in a row, + # e.g. one `git blame` per file on a first build; wait and try again. + for attempt in range(5): + try: + r = subprocess.run(["git", "-C", repo, *args], capture_output=True, check=False) + break + except BlockingIOError: + if attempt == 4: + raise + time.sleep(1 + attempt) if check and r.returncode != 0: raise RuntimeError(f"git {' '.join(args)}: {r.stderr.decode(errors='replace').strip()}") return r.stdout @@ -108,19 +136,23 @@ def url(*parts): return "/" + "/".join(urllib.parse.quote(s) for s in path.split("/")) -def page(title, body, crumbs=None, repo=None, site="code"): +def page(title, body, crumbs=None, view=None, site="code"): if site == "apps": host, footer = "apps.fuzzyworld.net", (f'<a href="/">All apps</a> · <a href="{BASE_URL}/">Code</a> · ' - f'<a href="{BASE_URL}/why/">Why this site exists</a>') + f'<a href="{BASE_URL}/why/">Why this site exists</a> · ' + f'<a href="{BASE_URL}/principles/">Principles</a>') else: host, footer = "code.fuzzyworld.net", (f'<a href="/">All repositories</a> · <a href="{APPS_URL}/">Apps</a> · ' - f'<a href="/contribute/">Contribute</a> · <a href="/why/">Why this site exists</a>') + f'<a href="/contribute/">Contribute</a> · <a href="/why/">Why this site exists</a> · ' + f'<a href="/principles/">Principles</a>') nav = "" - if repo: - nav = (f'<nav class="repo-nav"><a href="{url(repo)}/">Summary</a>' - f'<a href="{url(repo, "tree")}/">Files</a>' - f'<a href="{url(repo, "log")}/">Log</a>' - f'<a href="{url(repo, "refs")}/">Branches & tags</a></nav>') + if view: + nav = (f'<nav class="repo-nav"><a href="{url(view.name)}/">Summary</a>' + f'<a href="{view.url("tree")}/">Files</a>' + f'<a href="{view.url("log")}/">Log</a>' + f'<a href="{url(view.name, "refs")}/">Branches & tags</a>' + + (f'<a href="{url(view.name, "ci")}/">CI</a>' if ci_pipelines(view.name) else "") + + "</nav>") crumb_html = "" if crumbs: crumb_html = '<p class="crumbs">' + " / ".join(crumbs) + "</p>" @@ -177,13 +209,158 @@ def safe_path(p): return all(part not in ("", ".", "..") for part in parts) +# ---------------------------------------------------------------- CI results + +# CI runners (server/runner/) upload each pipeline to ~/fw/ci/<repo>/<sha>/: pipeline.json +# plus one log file per job. These pages show them for public repos only. +CI_STATES = { # status: (mark, words, badge color) + "success": ("✓", "passed", "#2da44e"), "failed": ("✗", "failed", "#cf222e"), + "running": ("●", "running", "#bf8700"), "error": ("!", "could not run", "#6e7781"), +} +_ci_cache = {} + + +def ci_pipelines(name): + """{sha: pipeline} for one repo, read once per run.""" + if name not in _ci_cache: + out, d = {}, os.path.join(CI_ROOT, name) + for sha in (os.listdir(d) if os.path.isdir(d) else []): + if not re.fullmatch(r"[0-9a-f]{40}", sha): + continue + try: + with open(os.path.join(d, sha, "pipeline.json")) as f: + p = json.load(f) + except (OSError, ValueError): + continue + if isinstance(p, dict) and p.get("status") in CI_STATES: + out[sha] = p + _ci_cache[name] = out + return _ci_cache[name] + + +def ci_mark(name, sha, p): + if not p: + return "" + mark, words, _ = CI_STATES[p["status"]] + return f'<a class="ci ci-{p["status"]}" href="{url(name, "ci", sha)}.html" title="CI {words}">{mark}</a> ' + + +def ci_badge(name, sha): + p = ci_pipelines(name).get(sha) + if not p: + return "" + return f' · CI: {ci_mark(name, sha, p)}<a href="{url(name, "ci", sha)}.html">{CI_STATES[p["status"]][1]}</a>' + + +def badge_svg(status): + _, words, color = CI_STATES[status] + w = 18 + 7 * len(words) + return (f'<svg xmlns="http://www.w3.org/2000/svg" width="{28 + w}" height="20" role="img" aria-label="CI: {words}">' + f'<rect width="28" height="20" fill="#555"/><rect x="28" width="{w}" height="20" fill="{color}"/>' + f'<g fill="#fff" font-family="Verdana,DejaVu Sans,sans-serif" font-size="11">' + f'<text x="6" y="14">CI</text><text x="37" y="14">{words}</text></g></svg>') + + +_ANSI = re.compile(r"\x1b\[[0-9;?]*[A-Za-z]|\x1b\][^\x07]*\x07|\r(?!\n)") + + +def duration(seconds): + s = int(seconds or 0) + return f"{s // 60}m {s % 60:02d}s" if s >= 60 else f"{s}s" + + +def build_ci_pages(view): + """ci/index.html, one page per pipeline, and ci/badge.svg (latest pipeline on the default branch).""" + name, out = view.name, os.path.join(SITE, view.name, "ci") + pipelines = ci_pipelines(name) + if not pipelines: + shutil.rmtree(out, ignore_errors=True) + return + ordered = sorted(pipelines.items(), key=lambda kv: kv[1].get("started", ""), reverse=True)[:CI_LIMIT] + keep = {sha + ".html" for sha, _ in ordered} | {"index.html", "badge.svg"} + for f in (os.listdir(out) if os.path.isdir(out) else []): + if f not in keep: + remove(os.path.join(out, f)) + rows = [] + for sha, p in ordered: + rows.append(f'<tr><td class="date">{esc(p.get("started", "")[:16].replace("T", " "))}</td>' + f'<td>{ci_mark(name, sha, p)}<a href="{url(name, "ci", sha)}.html"><code>{sha[:12]}</code></a></td>' + f'<td><code>{esc(p.get("ref", ""))}</code></td><td>{esc(CI_STATES[p["status"]][1])}</td>' + f'<td class="num">{duration(p.get("duration"))}</td></tr>') + page_path = os.path.join(out, sha + ".html") + src = os.path.join(CI_ROOT, name, sha, "pipeline.json") + if os.path.exists(page_path) and os.path.getmtime(page_path) >= os.path.getmtime(src): + continue + jobs = [] + for j in p.get("jobs", []): + st = j.get("status", "error") + log, log_name = "", j.get("log", "") + if re.fullmatch(r"[A-Za-z0-9._-]+\.log", log_name or ""): + try: + with open(os.path.join(CI_ROOT, name, sha, log_name), "rb") as f: + f.seek(0, 2) + size = f.tell() + f.seek(max(0, size - MAX_LOG_BYTES)) + text = f.read().decode("utf-8", errors="replace") + cut = "(start of log cut; showing the last part)\n" if size > MAX_LOG_BYTES else "" + log = f'<pre class="ci-log">{esc(cut + _ANSI.sub("", text))}</pre>' + except OSError: + pass + mark = CI_STATES.get(st, ("–", st, "")) + jobs.append(f'<details{" open" if st in ("failed", "error") else ""}><summary>' + f'<span class="ci ci-{esc(st)}">{mark[0]}</span> <strong>{esc(j.get("name", "?"))}</strong>' + f' · {esc(j.get("stage", ""))} · {esc(mark[1])} · {duration(j.get("duration"))}' + f'{" · " + esc(j["message"]) if j.get("message") else ""}</summary>{log}</details>') + mark, words, _ = CI_STATES[p["status"]] + body = (f'<h1>CI {esc(words)}: <code>{sha[:12]}</code></h1>' + f'<p class="meta">Branch <code>{esc(p.get("ref", ""))}</code> · ' + f'<a href="{url(name, "commit", sha)}.html">commit</a> · started {esc(p.get("started", "").replace("T", " ").replace("Z", " UTC"))} · ' + f'{duration(p.get("duration"))} · runner {esc(p.get("runner", ""))}</p>' + + (f'<p>{esc(p["message"])}</p>' if p.get("message") else "") + + "".join(jobs)) + write(page_path, page(f"{name} CI {sha[:12]}", body, [f'<a href="{url(name)}/">{esc(name)}</a>', "CI"], view=view)) + write(os.path.join(out, "index.html"), + page(f"{name} CI", f'<h1>CI</h1><table class="log">{"".join(rows)}</table>', view=view)) + latest = next((p for _, p in ordered if p.get("ref") == view.branch), None) + if latest: + write(os.path.join(out, "badge.svg"), badge_svg(latest["status"])) + else: + remove(os.path.join(out, "badge.svg")) + + +# ---------------------------------------------------------------- views + +class View: + """One browsable branch. + + The default branch keeps the short URLs (/<name>/tree/...). Other branches live under + /<name>/branch/<branch>/, with each '/' in the branch name written as '~': git forbids + '~' in ref names, so two branches can never share a directory. + """ + + def __init__(self, name, branch, is_default): + self.name, self.branch, self.is_default = name, branch, is_default + self.ref = "refs/heads/" + branch + self.base = [name] if is_default else [name, "branch", branch.replace("/", "~")] + + def url(self, *parts): + return url(*self.base, *parts) + + def path(self, *parts): + return os.path.join(SITE, *self.base, *parts) + + @property + def key(self): + return "/".join(self.base[1:]) or "." + + # ---------------------------------------------------------------- markdown class LinkContext: """Rewrites relative links in a repo's markdown to this site's pages.""" - def __init__(self, name, base_dir, files, dirs, raw): - self.name, self.base_dir, self.files, self.dirs, self.raw = name, base_dir, files, dirs, raw + def __init__(self, view, base_dir, files, dirs, raw): + self.view, self.base_dir, self.files, self.dirs, self.raw = view, base_dir, files, dirs, raw def rewrite(self, href, image): parsed = urllib.parse.urlsplit(href) @@ -192,11 +369,11 @@ class LinkContext: target = posixpath.normpath(posixpath.join(self.base_dir, urllib.parse.unquote(parsed.path))) frag = ("#" + parsed.fragment) if parsed.fragment else "" if image and target in self.raw: - return url(self.name, "raw", target) + return self.view.url("raw", target) if target in self.files: - return url(self.name, "blob", target) + ".html" + frag + return self.view.url("blob", target) + ".html" + frag if target in self.dirs or target == ".": - return url(self.name, "tree", "" if target == "." else target) + "/" + frag + return self.view.url("tree", "" if target == "." else target) + "/" + frag return href @@ -215,12 +392,81 @@ def render_markdown(text, ctx): return _ATTR.sub(fix, out) +# ---------------------------------------------------------------- highlighting + +# Pygments token types folded into a few CSS classes (colors in style.css). The most +# specific match wins, so Name.Function gets "nf" and other names get nothing. +_TOKEN_CLASSES = [] +if pygments_token: + T = pygments_token + _TOKEN_CLASSES = [(T.Comment.Preproc, "cp"), (T.Comment, "c"), (T.Keyword, "k"), (T.String, "s"), + (T.Number, "m"), (T.Name.Function, "nf"), (T.Name.Class, "nc"), (T.Name.Builtin, "nb"), + (T.Name.Tag, "nt"), (T.Name.Attribute, "na"), (T.Name.Decorator, "nd"), + (T.Generic.Inserted, "gi"), (T.Generic.Deleted, "gd"), (T.Generic.Heading, "gh"), + (T.Error, "")] +_class_cache = {} + + +def token_class(ttype): + if ttype not in _class_cache: + _class_cache[ttype] = next((c for t, c in _TOKEN_CLASSES if ttype in t), "") + return _class_cache[ttype] + + +def highlight_lines(path, text): + """The text as escaped HTML lines, colored when Pygments knows the file type.""" + lines = text.split("\n") + if lines and lines[-1] == "": + lines.pop() + plain = [esc(l) for l in lines] + if not pygments_lexers or len(text) > MAX_HIGHLIGHT_BYTES: + return plain + try: + lexer = pygments_lexers.get_lexer_for_filename(posixpath.basename(path), stripnl=False, ensurenl=False) + except Exception: # ClassNotFound, or a lexer that fails to load + return plain + out, cur = [], [] + try: + for ttype, value in lexer.get_tokens(text): + cls = token_class(ttype) + for i, part in enumerate(value.split("\n")): + if i: + out.append("".join(cur)) + cur = [] + if part: + cur.append(f'<span class="{cls}">{esc(part)}</span>' if cls else esc(part)) + except Exception: # a lexer bug must not stop the page + return plain + out.append("".join(cur)) + # Pygments turns \r\n into \n; if that changed the line count, fall back to the plain lines. + while len(out) > len(lines) and out[-1] == "": + out.pop() + return out if len(out) == len(lines) else plain + + +def source_table(lines, blame=None): + """Numbered source lines; with blame, a column naming the commit behind each run of lines.""" + rows = [] + for i, code in enumerate(lines, 1): + cell = "" + if blame is not None: + b = blame[i - 1] if i - 1 < len(blame) else None + starts = b and (i == 1 or blame[i - 2] is None or blame[i - 2]["sha"] != b["sha"]) + cell = f'<td class="blame">{b["html"] if starts else ""}</td>' + run = ' class="run"' if starts and i > 1 else "" + rows.append(f'<tr id="L{i}"{run}>{cell}' + f'<td class="ln"><a href="#L{i}">{i}</a></td><td class="code">{code or " "}</td></tr>') + else: + rows.append(f'<tr id="L{i}"><td class="ln"><a href="#L{i}">{i}</a></td><td class="code">{code or " "}</td></tr>') + return f'<table class="source{" blamed" if blame is not None else ""}">{"".join(rows)}</table>' + + # ---------------------------------------------------------------- repo pages -def tree_entries(repo): - """Default-branch files as {path: (mode, sha, size)}; submodules get size -1.""" +def tree_entries(repo, ref): + """Files at `ref` as {path: (mode, sha, size)}; submodules get size -1.""" out = {} - data = git(repo, "ls-tree", "-r", "-l", "-z", "HEAD") + data = git(repo, "ls-tree", "-r", "-l", "-z", ref) for rec in data.split(b"\0"): if not rec: continue @@ -243,84 +489,168 @@ def all_dirs(files): return dirs -def crumbs_for(name, path, last_is_file): - crumbs = [f'<a href="{url(name)}/">{esc(name)}</a>', f'<a href="{url(name, "tree")}/">files</a>'] +def branch_note(view): + if view.is_default: + return "" + return (f'<p class="branch-note">Branch <code>{esc(view.branch)}</code> · ' + f'<a href="{url(view.name, "tree")}/">default branch</a> · <a href="{url(view.name, "refs")}/">all branches</a></p>') + + +def crumbs_for(view, path): + crumbs = [f'<a href="{url(view.name)}/">{esc(view.name)}</a>'] + if not view.is_default: + crumbs.append(f"<code>{esc(view.branch)}</code>") + crumbs.append(f'<a href="{view.url("tree")}/">files</a>') parts = path.split("/") if path else [] for i, part in enumerate(parts): sub = "/".join(parts[: i + 1]) if i == len(parts) - 1: crumbs.append(f"<strong>{esc(part)}</strong>") else: - crumbs.append(f'<a href="{url(name, "tree", sub)}/">{esc(part)}</a>') + crumbs.append(f'<a href="{view.url("tree", sub)}/">{esc(part)}</a>') return crumbs -def build_tree_page(out, name, d, files, dirs): +def build_tree_page(view, d, files, dirs): prefix = d + "/" if d else "" subdirs = sorted({p[len(prefix):].split("/")[0] for p in dirs if p.startswith(prefix) and p != d and "/" not in p[len(prefix):]}) here = sorted(p for p in files if posixpath.dirname(p) == d) rows = [] if d: parent = posixpath.dirname(d) - rows.append(f'<tr><td><a href="{url(name, "tree", parent)}/">..</a></td><td></td></tr>') + rows.append(f'<tr><td><a href="{view.url("tree", parent)}/">..</a></td><td></td></tr>') for s in subdirs: - rows.append(f'<tr><td class="dir"><a href="{url(name, "tree", prefix + s)}/">{esc(s)}/</a></td><td></td></tr>') + rows.append(f'<tr><td class="dir"><a href="{view.url("tree", prefix + s)}/">{esc(s)}/</a></td><td></td></tr>') for p in here: size = files[p][2] label = "submodule" if size < 0 else human_size(size) - rows.append(f'<tr><td><a href="{url(name, "blob", p)}.html">{esc(posixpath.basename(p))}</a></td><td class="num">{label}</td></tr>') - body = f'<h1>{esc(d or name)}</h1>\n<table class="files">{"".join(rows)}</table>' - write(os.path.join(out, "tree", d, "index.html"), - page(f"{name}/{d}" if d else f"{name} files", body, crumbs_for(name, d, False) if d else None, repo=name)) - - -def build_blob_page(out, name, path, entry, blobs, ctx, raw_set): + rows.append(f'<tr><td><a href="{view.url("blob", p)}.html">{esc(posixpath.basename(p))}</a></td><td class="num">{label}</td></tr>') + title = d or (view.name if view.is_default else view.branch) + body = f'{branch_note(view)}<h1>{esc(title)}</h1>\n<table class="files">{"".join(rows)}</table>' + write(view.path("tree", d, "index.html"), + page(f"{view.name}/{d}" if d else f"{view.name} files", body, crumbs_for(view, d) if d or not view.is_default else None, view=view)) + + +def blame_lines(repo, view, path, commit_set): + """[{sha, html}] per line of the file at the view's branch, from `git blame --porcelain`.""" + data = git(repo, "blame", "--porcelain", view.ref, "--", path, check=False) + info, out, cur = {}, [], None + for raw in data.split(b"\n"): + if raw.startswith(b"\t"): + out.append(cur) + continue + head = raw.split(b" ", 1)[0] + if len(head) == 40 and re.fullmatch(rb"[0-9a-f]{40}", head): + cur = head.decode() + info.setdefault(cur, {}) + elif cur and raw: + k, _, v = raw.decode("utf-8", errors="replace").partition(" ") + info[cur].setdefault(k, v) + cells = {} + for sha, i in info.items(): + when = time.strftime("%Y-%m-%d", time.gmtime(int(i.get("author-time", "0") or 0))) + short = f'<code>{sha[:8]}</code>' + link = f'<a href="{url(view.name, "commit", sha)}.html" title="{esc(i.get("summary", ""), quote=True)}">{short}</a>' if sha in commit_set else short + cells[sha] = {"sha": sha, "html": f'{link} <span class="when">{when}</span> <span class="who">{esc(i.get("author", ""))}</span>'} + return [cells.get(s) for s in out] + + +def build_blob_page(repo, view, path, entry, blobs, ctx, raw_set, commit_set): + """The file page, plus its blame page for text files. Returns True if a blame page was written.""" mode, sha, size = entry - title = f"{name}/{path}" - crumbs = crumbs_for(name, path, True) + title = f"{view.name}/{path}" + crumbs = crumbs_for(view, path) + blamed = False + links = f' · <a href="{view.url("history", path)}.html">history</a>' if size < 0: body = f"<h1>{esc(posixpath.basename(path))}</h1><p>Git submodule at commit <code>{esc(sha)}</code>.</p>" else: data = blobs.read(sha) - raw_link = f' · <a href="{url(name, "raw", path)}">raw</a>' if path in raw_set else "" - meta = f'<p class="meta">{human_size(size)}{raw_link}</p>' + if path in raw_set: + links += f' · <a href="{view.url("raw", path)}">raw</a>' ext = posixpath.splitext(path)[1].lower() + text = None if path in raw_set: - content = f'<p><img class="blob" src="{url(name, "raw", path)}" alt="{esc(posixpath.basename(path))}"></p>' + content = f'<p><img class="blob" src="{view.url("raw", path)}" alt="{esc(posixpath.basename(path))}"></p>' elif is_binary(data): content = "<p>Binary file, not shown. Clone the repository to get it.</p>" elif size > MAX_TEXT_BYTES: content = "<p>File too large to show here. Clone the repository to get it.</p>" else: text = data.decode("utf-8", errors="replace") - lines = text.split("\n") - if lines and lines[-1] == "": - lines.pop() - numbered = "".join(f'<tr id="L{i}"><td class="ln"><a href="#L{i}">{i}</a></td><td class="code">{esc(l) or " "}</td></tr>' - for i, l in enumerate(lines, 1)) - source = f'<table class="source">{numbered}</table>' + lines = highlight_lines(path, text) + source = source_table(lines) if ext in (".md", ".markdown") and markdown: ctx.base_dir = posixpath.dirname(path) content = (f'<article class="markdown">{render_markdown(text, ctx)}</article>' f"<details><summary>Source</summary>{source}</details>") else: content = source - body = f"<h1>{esc(posixpath.basename(path))}</h1>{meta}{content}" - write(os.path.join(out, "blob", path + ".html"), page(title, body, crumbs, repo=name)) - - -def commit_list(repo, limit): - fmt = "%H%x1f%h%x1f%an%x1f%ad%x1f%s" - raw = git_text(repo, "log", f"-n{limit}", "--date=short", f"--format={fmt}", "HEAD", check=False) + if size <= MAX_BLAME_BYTES: + blame = blame_lines(repo, view, path, commit_set) + if blame: + links += f' · <a href="{view.url("blame", path)}.html">blame</a>' + bbody = (f'{branch_note(view)}<h1>{esc(posixpath.basename(path))}</h1>' + f'<p class="meta">Blame: the commit that last changed each line · ' + f'<a href="{view.url("blob", path)}.html">file</a> · ' + f'<a href="{view.url("history", path)}.html">history</a></p>{source_table(lines, blame)}') + write(view.path("blame", path + ".html"), page(f"{title} blame", bbody, crumbs, view=view)) + blamed = True + meta = f'<p class="meta">{human_size(size)}{links}</p>' + body = f"{branch_note(view)}<h1>{esc(posixpath.basename(path))}</h1>{meta}{content}" + if not blamed: + remove(view.path("blame", path + ".html")) + write(view.path("blob", path + ".html"), page(title, body, crumbs, view=view)) + return blamed + + +COMMIT_FORMAT = "%H%x1f%h%x1f%an%x1f%ad%x1f%s" + + +def commit_list(repo, ref, limit): + raw = git_text(repo, "log", f"-n{limit}", "--date=short", f"--format={COMMIT_FORMAT}", ref, check=False) return [line.split("\x1f") for line in raw.splitlines() if line.count("\x1f") == 4] -def commits_table(name, commits): - rows = "".join(f'<tr><td class="date">{esc(d)}</td><td><a href="{url(name, "commit", h)}.html"><code>{esc(s)}</code></a></td>' - f"<td>{esc(subj)}</td><td class=\"author\">{esc(a)}</td></tr>" for h, s, a, d, subj in commits) +def file_histories(repo, ref, paths): + """{path: [commit, ...]} newest first, at most HISTORY_LIMIT each, from one `git log` pass.""" + raw = git(repo, "-c", "core.quotePath=false", "log", "--no-renames", "--name-only", "--date=short", + f"-n{COMMIT_LIMIT}", f"--format=%x1e{COMMIT_FORMAT}", ref, check=False).decode("utf-8", errors="replace") + hist = {} + for chunk in raw.split("\x1e"): + lines = chunk.split("\n") + commit = lines[0].split("\x1f") + if len(commit) != 5: + continue + for p in lines[1:]: + if p in paths: + h = hist.setdefault(p, []) + if len(h) < HISTORY_LIMIT: + h.append(commit) + return hist + + +def commits_table(name, commits, commit_set=None): + def sha_cell(h, s): + if commit_set is not None and h not in commit_set: + return f"<code>{esc(s)}</code>" + return f'<a href="{url(name, "commit", h)}.html"><code>{esc(s)}</code></a>' + ci = ci_pipelines(name) + rows = "".join(f'<tr><td class="date">{esc(d)}</td><td>{sha_cell(h, s)}</td>' + f'<td>{ci_mark(name, h, ci.get(h))}{esc(subj)}</td><td class="author">{esc(a)}</td></tr>' + for h, s, a, d, subj in commits) return f'<table class="log">{rows}</table>' +def build_history_page(view, path, commits, commit_set): + body = (f'{branch_note(view)}<h1>{esc(posixpath.basename(path))}</h1>' + f'<p class="meta">Commits that changed this file, newest first · ' + f'<a href="{view.url("blob", path)}.html">file</a></p>' + + (commits_table(view.name, commits, commit_set) if commits else + "<p>No recent commits changed this file (it may have been renamed or only changed long ago).</p>")) + write(view.path("history", path + ".html"), page(f"{view.name}/{path} history", body, crumbs_for(view, path), view=view)) + + SHORT_SHA_LENGTHS = range(7, 13) @@ -330,24 +660,25 @@ def short_sha_aliases(commits): Apps link to `commit/<git rev-parse --short>.html`, and the short length varies by clone. """ seen = {} - for full in (c[0] for c in commits): + for full in commits: for n in SHORT_SHA_LENGTHS: seen.setdefault(full[:n], set()).add(full) return {alias: fulls.pop() for alias, fulls in seen.items() if len(fulls) == 1} -def write_short_sha_alias(out, name, alias, full): +def write_short_sha_alias(out, view, alias, full): target = f"{full}.html" body = f'<h1>Commit {esc(alias)}</h1><p>See <a href="{esc(target)}"><code>{esc(full)}</code></a>.</p>' - html_out = page(f"{name} {alias}", body, repo=name).replace( + html_out = page(f"{view.name} {alias}", body, view=view).replace( "<head>\n", f'<head>\n<meta http-equiv="refresh" content="0; url={esc(target)}">\n', 1) write(os.path.join(out, "commit", alias + ".html"), html_out) -def build_commit_page(repo, out, name, sha): +def build_commit_page(repo, out, view, sha): path = os.path.join(out, "commit", sha + ".html") if os.path.exists(path): return # commits never change + name = view.name head = git_text(repo, "show", "-s", "--format=%H%n%an <%ae>%n%ad%n%P%n%B", "--date=iso", sha) full, author, date, parents, *msg = head.split("\n") author = author.split(" <")[0] # don't publish email addresses @@ -373,108 +704,167 @@ def build_commit_page(repo, out, name, sha): f'<pre class="message">{esc(chr(10).join(msg).strip())}</pre>' f'<pre class="diff">{chr(10).join(lines)}</pre>' + ("<p>Diff truncated. Clone the repository to see all of it.</p>" if truncated else "")) - write(path, page(f"{name} {full[:12]}", body, [f'<a href="{url(name)}/">{esc(name)}</a>', "commit"], repo=name)) + write(path, page(f"{name} {full[:12]}", body, [f'<a href="{url(name)}/">{esc(name)}</a>', "commit"], view=view)) def refs_rows(repo, pattern, datefield): - fmt = f"%(refname:short)%1f%({datefield}:short)%1f%(subject)" + fmt = f"%(refname:short)%1f%({datefield}:short)%1f%(subject)%1f%(objectname)" raw = git_text(repo, "for-each-ref", f"--sort=-{datefield}", f"--format={fmt}", pattern, check=False) - return [l.split("\x1f") for l in raw.splitlines() if l.count("\x1f") == 2] + return [l.split("\x1f") for l in raw.splitlines() if l.count("\x1f") == 3] -def build_refs_page(repo, out, name, default_branch): +def build_refs_page(repo, view, branch_views, commit_set): branches = refs_rows(repo, "refs/heads", "committerdate") tags = refs_rows(repo, "refs/tags", "creatordate") - def table(rows, mark_default): + def branch_cell(n, sha): + v = branch_views.get(n) + label = f"<code>{esc(n)}</code>" + if v: + return f'<a href="{v.url("tree")}/">{label}</a>{" <em>(default)</em>" if v.is_default else ""}' + if sha in commit_set: # merged into the default branch: no pages of its own + return f'{label} <em>(merged, <a href="{url(view.name, "commit", sha)}.html">last commit</a>)</em>' + return label + def table(rows, cell): if not rows: return "<p>None.</p>" return '<table class="log">' + "".join( - f'<tr><td><code>{esc(n)}</code>{" <em>(default)</em>" if mark_default and n == default_branch else ""}</td>' - f'<td class="date">{esc(d)}</td><td>{esc(s)}</td></tr>' for n, d, s in rows) + "</table>" - body = (f"<h1>Branches</h1>{table(branches, True)}<h1>Tags</h1>{table(tags, False)}" - f"<p>Only the default branch is browsable here. Clone the repository for the others.</p>") - write(os.path.join(out, "refs", "index.html"), page(f"{name} branches & tags", body, repo=name)) + f'<tr><td>{cell(n, h)}</td><td class="date">{esc(d)}</td><td>{esc(s)}</td></tr>' for n, d, s, h in rows) + "</table>" + body = (f"<h1>Branches</h1>{table(branches, branch_cell)}<h1>Tags</h1>{table(tags, lambda n, h: f'<code>{esc(n)}</code>')}" + f"<p>Tags can be cloned: <code>git clone --branch <tag> {BASE_URL}/{esc(view.name)}.git</code></p>") + write(os.path.join(SITE, view.name, "refs", "index.html"), page(f"{view.name} branches & tags", body, view=view)) + + +def app_shots(app, repo=None): + """[(url, label)]: the screenshots a CI runner took of the app; otherwise (public repos) + the images in the repo's screenshots/ folder.""" + try: + with open(os.path.join(APPS, SHOTS, app, "shots.json")) as f: + listed = json.load(f).get("shots", []) + found = [(f"{APPS_URL}/{SHOTS}/{urllib.parse.quote(app)}/{urllib.parse.quote(s['file'])}", s.get("label", "")) + for s in listed if isinstance(s, dict) and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]*\.png", s.get("file", "")) + and os.path.exists(os.path.join(APPS, SHOTS, app, s["file"]))] + if found: + return found + except (OSError, ValueError, AttributeError): + pass + d = os.path.join(SITE, repo or "", "raw", "screenshots") + if repo and os.path.isdir(d): + return [(f"{BASE_URL}{url(repo, 'raw', 'screenshots', f)}", os.path.splitext(f)[0]) + for f in sorted(os.listdir(d)) if f.lower().endswith(SHOT_EXTENSIONS)][:4] + return [] def app_url(name): return f"{APPS_URL}/{name}/" if os.path.exists(os.path.join(APPS, name, "index.html")) else None -def build_summary(repo, out, name, desc, files, ctx, commits, empty): - clone = f"git clone {BASE_URL}/{name}.git" +LICENSE_NAMES = ["license", "license.md", "license.txt", "licence", "licence.md", "licence.txt", "copying", "copying.md", "copying.txt"] +# (SPDX id, phrases that must all appear), checked in order; first match wins. +LICENSE_PATTERNS = [ + ("AGPL-3.0", ["gnu affero general public license"]), + ("LGPL-3.0", ["gnu lesser general public license", "version 3"]), + ("LGPL-2.1", ["gnu lesser general public license", "version 2.1"]), + ("GPL-3.0", ["gnu general public license", "version 3"]), + ("GPL-2.0", ["gnu general public license", "version 2"]), + ("Apache-2.0", ["apache license", "version 2.0"]), + ("MPL-2.0", ["mozilla public license", "2.0"]), + ("MIT", ["permission is hereby granted, free of charge"]), + ("ISC", ["permission to use, copy, modify, and/or distribute this software for any purpose"]), + ("BSD-3-Clause", ["redistribution and use in source and binary forms", "neither the name"]), + ("BSD-2-Clause", ["redistribution and use in source and binary forms"]), + ("Unlicense", ["this is free and unencumbered software released into the public domain"]), + ("CC0-1.0", ["cc0 1.0 universal"]), +] + + +def detect_license(files, blobs): + """(license name, path of the license file) from a root LICENSE/COPYING file or package.json, or None.""" + lower = {p.lower(): p for p in files if "/" not in p} + for cand in LICENSE_NAMES: + if cand in lower: + p = lower[cand] + text = " ".join(blobs.read(files[p][1])[:20000].decode("utf-8", errors="replace").lower().split()) + for spdx, phrases in LICENSE_PATTERNS: + if all(ph in text for ph in phrases): + return spdx, p + return "see license file", p + if "package.json" in files: + try: + lic = json.loads(blobs.read(files["package.json"][1])).get("license") + except (ValueError, AttributeError): + lic = None + if isinstance(lic, str) and lic.strip() and lic.strip().upper() != "UNLICENSED": + return lic.strip(), "package.json" + return None + + +def build_summary(repo, view, desc, files, ctx, commits, empty, commit_set): + name = view.name + clone_url = f"{BASE_URL}/{name}.git" app = app_url(name) links = f'<p class="meta">App: <a href="{app}">{esc(app)}</a></p>' if app else "" contrib, bugs = f"contrib-68uq+{name}@fuzzyworld.net", f"bugs-k7q4+{name}@fuzzyworld.net" links += (f'<p class="meta">Contribute: <a href="mailto:{esc(contrib)}">{esc(contrib)}</a>' f' (<a href="/contribute/">how</a>) · Report a bug: <a href="mailto:{esc(bugs)}">{esc(bugs)}</a></p>') - readme_html = "" + readme_html, license_html = "", "" if not empty: + blobs = BlobReader(repo) + lic = detect_license(files, blobs) + if lic: + license_html = f' · License: <a href="{view.url("blob", lic[1])}.html">{esc(lic[0])}</a>' lower = {p.lower(): p for p in files if "/" not in p} for cand in README_NAMES: if cand in lower: p = lower[cand] - blobs = BlobReader(repo) text = blobs.read(files[p][1]).decode("utf-8", errors="replace") - blobs.close() ctx.base_dir = "" rendered = render_markdown(text, ctx) if p.lower().endswith((".md", ".markdown")) else f"<pre>{esc(text)}</pre>" readme_html = f'<article class="markdown readme">{rendered}</article>' break - recent = commits_table(name, commits[:5]) if commits else "" + blobs.close() + # "Open in VS Code" clones into the visitor's own VS Code; "in the browser" opens our static VS Code web build. + vscode = f'<a href="vscode://vscode.git/clone?url={urllib.parse.quote(clone_url, safe="")}">Open in VS Code</a>' + if not empty and os.path.exists(os.path.join(APPS, VSCODE_APP, "index.html")): + vscode += f' · <a href="{APPS_URL}/{VSCODE_APP}/?repo={urllib.parse.quote(name)}">Open in VS Code in the browser</a>' + recent = commits_table(name, commits[:5], commit_set) if commits else "" + shots = app_shots(name, name) if app else [] + shots_html = ('<p class="shots">' + "".join(f'<a href="{app}"><img src="{esc(u)}" alt="{esc(l)}" loading="lazy"></a>' for u, l in shots[:4]) + + "</p>") if shots else "" body = (f"<h1>{esc(name)}</h1>" + (f'<p class="desc">{esc(desc)}</p>' if desc else "") - + links - + f'<pre class="clone">{esc(clone)}</pre>' + + links + shots_html + + f'<pre class="clone">git clone {esc(clone_url)}</pre>' + + f'<p class="meta">{vscode}{license_html}{ci_badge(name, commits[0][0]) if commits else ""}</p>' + ("<p>This repository is empty.</p>" if empty else f"<h2>Recent commits</h2>{recent}<p><a href=\"{url(name, 'log')}/\">Full log</a></p>") + readme_html) - write(os.path.join(out, "index.html"), page(name, body, repo=None if empty else name)) + write(os.path.join(SITE, name, "index.html"), page(name, body, view=None if empty else view)) -def build_repo(name): - repo = os.path.join(GIT_ROOT, name + ".git") - if not os.path.isdir(repo): - raise SystemExit(f"no such repo: {name}") - if not is_public(repo): - raise SystemExit(f"{name} is not public; refusing to publish") - out = os.path.join(SITE, name) - os.makedirs(STATE, exist_ok=True) - state_path = os.path.join(STATE, name + ".json") - try: - with open(state_path) as f: - state = json.load(f) - except (FileNotFoundError, ValueError): - state = {} - if state.get("version") != GEN_VERSION: - state = {} +def is_merged(repo, branch): + r = subprocess.run(["git", "-C", repo, "merge-base", "--is-ancestor", "refs/heads/" + branch, "HEAD"], capture_output=True) + return r.returncode == 0 - desc = repo_config(repo, "fuzzyworld.description") - empty = not git(repo, "rev-parse", "-q", "--verify", "HEAD", check=False).strip() - files = {} if empty else tree_entries(repo) + +def build_view(repo, view, vstate, commit_set): + """Tree, file, history, blame and log pages for one branch. Returns the view's new state.""" + files = tree_entries(repo, view.ref) dirs = all_dirs(files) raw_set = {p for p, (m, s, size) in files.items() if 0 <= size <= MAX_RAW_BYTES and posixpath.splitext(p)[1].lower() in RAW_EXTENSIONS} - ctx = LinkContext(name, "", set(files), dirs, raw_set) - commits = [] if empty else commit_list(repo, LOG_LIMIT) + ctx = LinkContext(view, "", set(files), dirs, raw_set) + commits = commit_list(repo, view.ref, COMMIT_LIMIT) + head = commits[0][0] if commits else None + head_changed = vstate.get("head") != head - old_files = state.get("files", {}) - old_dirs = set(state.get("dirs", [])) - old_commits = set(state.get("commits", [])) - head_changed = state.get("head") != (commits[0][0] if commits else None) - - # Remove pages for files, directories and commits that are gone. + old_files = vstate.get("files", {}) for p in set(old_files) - set(files): - remove(os.path.join(out, "blob", p + ".html")) - remove(os.path.join(out, "raw", p)) - for d in old_dirs - dirs: - remove(os.path.join(out, "tree", d, "index.html")) - for h in old_commits - {c[0] for c in commits}: - remove(os.path.join(out, "commit", h + ".html")) - aliases = short_sha_aliases(commits) - for a in set(state.get("aliases", [])) - set(aliases): - remove(os.path.join(out, "commit", a + ".html")) + for kind, suffix in (("blob", ".html"), ("blame", ".html"), ("history", ".html"), ("raw", "")): + remove(view.path(kind, p + suffix)) + for d in set(vstate.get("dirs", [])) - dirs: + remove(view.path("tree", d, "index.html")) - # File pages and raw copies: only where the blob changed. Markdown pages are - # rebuilt whenever HEAD moves, because their links depend on the whole tree. + # File and blame pages and raw copies: only where the blob changed. Markdown pages are + # rebuilt whenever the branch moves, because their links depend on the whole tree. blobs = BlobReader(repo) written = 0 for p, entry in files.items(): @@ -482,35 +872,129 @@ def build_repo(name): if old_files.get(p) == entry[1] and not (is_md and head_changed): continue try: - build_blob_page(out, name, p, entry, blobs, ctx, raw_set) + build_blob_page(repo, view, p, entry, blobs, ctx, raw_set, commit_set) if p in raw_set: - write(os.path.join(out, "raw", p), blobs.read(entry[1])) + write(view.path("raw", p), blobs.read(entry[1])) else: - remove(os.path.join(out, "raw", p)) + remove(view.path("raw", p)) written += 1 except OSError as e: # e.g. a path component too long for the filesystem print(f"skip {p}: {e}", file=sys.stderr) blobs.close() - if not empty: - build_tree_page(out, name, "", files, dirs) + # History pages: only where the newest commit touching the file changed. + hist = file_histories(repo, view.ref, set(files)) if head_changed or not vstate.get("hist") else None + new_hist = vstate.get("hist", {}) if hist is None else {} + if hist is not None: + old_hist = vstate.get("hist", {}) + for p in files: + commits_p = hist.get(p, []) + newest = commits_p[0][0] if commits_p else "" + new_hist[p] = newest + if old_hist.get(p) != newest or not os.path.exists(view.path("history", p + ".html")): + try: + build_history_page(view, p, commits_p, commit_set) + except OSError as e: + print(f"skip history of {p}: {e}", file=sys.stderr) + + # For VS Code in the browser (vscode-web/): every file's blob SHA and size. The contents are + # objects/<sha>, written by build_repo for files up to MAX_TEXT_BYTES. + write(view.path("files.json"), json.dumps({"repo": view.name, "branch": view.branch, "commit": head, + "maxBytes": MAX_TEXT_BYTES, + "files": {p: [e[1], e[2]] for p, e in sorted(files.items())}})) + if files: + build_tree_page(view, "", files, dirs) for d in dirs: - build_tree_page(out, name, d, files, dirs) - for c in commits: - build_commit_page(repo, out, name, c[0]) - for alias, full in aliases.items(): - write_short_sha_alias(out, name, alias, full) - default_branch = git_text(repo, "symbolic-ref", "--short", "HEAD", check=False).strip() - build_refs_page(repo, out, name, default_branch) - write(os.path.join(out, "log", "index.html"), - page(f"{name} log", f"<h1>Last {len(commits)} commits</h1>{commits_table(name, commits)}", repo=name)) - build_summary(repo, out, name, desc, files, ctx, commits, empty) - - state = {"version": GEN_VERSION, "head": commits[0][0] if commits else None, - "files": {p: e[1] for p, e in files.items()}, "dirs": sorted(dirs), - "commits": [c[0] for c in commits], "aliases": sorted(aliases), "built": int(time.time())} + build_tree_page(view, d, files, dirs) + log_title = f"Last {len(commits)} commits" + ("" if view.is_default else f" on {esc(view.branch)}") + write(view.path("log", "index.html"), + page(f"{view.name} log", f"{branch_note(view)}<h1>{log_title}</h1>{commits_table(view.name, commits, commit_set)}", view=view)) + print(f"{view.name} [{view.branch}]: {len(files)} files, {written} file pages written, {len(commits)} commits") + return {"head": head, "files": {p: e[1] for p, e in files.items()}, "dirs": sorted(dirs), "hist": new_hist, + "_files": files, "_ctx": ctx, "_commits": commits} + + +def build_repo(name): + repo = os.path.join(GIT_ROOT, name + ".git") + if not os.path.isdir(repo): + raise SystemExit(f"no such repo: {name}") + if not is_public(repo): + raise SystemExit(f"{name} is not public; refusing to publish") + out = os.path.join(SITE, name) + os.makedirs(STATE, exist_ok=True) + state_path = os.path.join(STATE, name + ".json") + try: + with open(state_path) as f: + state = json.load(f) + except (FileNotFoundError, ValueError): + state = {} + if state.get("version") != GEN_VERSION: + state = {} + + desc = repo_config(repo, "fuzzyworld.description") + default_branch = git_text(repo, "symbolic-ref", "--short", "HEAD", check=False).strip() + branch_names = [b for b in git_text(repo, "for-each-ref", "--format=%(refname:short)", "refs/heads", check=False).split() + if safe_path(b)] + empty = default_branch not in branch_names + # Browse pages for the default branch and for branches not yet merged into it. A merged + # branch adds nothing new to read, and would duplicate every page. + views = {b: View(name, b, b == default_branch) for b in branch_names + if b == default_branch or empty or not is_merged(repo, b)} + default_view = views.get(default_branch) or View(name, default_branch or "main", True) + + # Commit pages for every commit on every branch (up to COMMIT_LIMIT), shared by all branches. + commit_shas = git_text(repo, "rev-list", f"-n{COMMIT_LIMIT}", "--branches", check=False).split() + commit_set = set(commit_shas) + for h in set(state.get("commits", [])) - commit_set: + remove(os.path.join(out, "commit", h + ".html")) + aliases = short_sha_aliases(commit_shas) + for a in set(state.get("aliases", [])) - set(aliases): + remove(os.path.join(out, "commit", a + ".html")) + for h in commit_shas: + build_commit_page(repo, out, default_view, h) + for alias, full in aliases.items(): + write_short_sha_alias(out, default_view, alias, full) + + # Branches that are gone lose their whole directory. + old_views = state.get("views", {}) + keep = {v.key for v in views.values()} + branch_root = os.path.join(out, "branch") + if os.path.isdir(branch_root): + for d in os.listdir(branch_root): + if f"branch/{d}" not in keep: + shutil.rmtree(os.path.join(branch_root, d), ignore_errors=True) + if empty: + remove(os.path.join(out, "files.json")) + for kind in ("tree", "blob", "blame", "history", "raw", "log", "objects"): + shutil.rmtree(os.path.join(out, kind), ignore_errors=True) + + new_views, built = {}, {} + for b, v in views.items(): + built[b] = build_view(repo, v, old_views.get(v.key, {}), commit_set) + new_views[v.key] = {k: val for k, val in built[b].items() if not k.startswith("_")} + + # objects/<sha>: file contents by blob SHA (served as downloads, never as pages), for VS Code in the browser. + wanted = {e[1] for b in built.values() for e in b["_files"].values() if 0 <= e[2] <= MAX_TEXT_BYTES} + objects = os.path.join(out, "objects") + have = set(os.listdir(objects)) if os.path.isdir(objects) else set() + for sha in have - wanted: + remove(os.path.join(objects, sha)) + if wanted - have: + blobs = BlobReader(repo) + for sha in wanted - have: + write(os.path.join(objects, sha), blobs.read(sha)) + blobs.close() + + d = built.get(default_branch, {}) + if not empty: + build_refs_page(repo, default_view, views, commit_set) + build_ci_pages(default_view) + build_summary(repo, default_view, desc, d.get("_files", {}), d.get("_ctx"), d.get("_commits", []), empty, commit_set) + + state = {"version": GEN_VERSION, "views": new_views, "commits": commit_shas, + "aliases": sorted(aliases), "built": int(time.time())} write(state_path, json.dumps(state)) - print(f"{name}: {len(files)} files, {written} file pages written, {len(commits)} commits") + print(f"{name}: {len(views)} branches, {len(commit_shas)} commits") def build_index(): @@ -542,8 +1026,53 @@ def build_index(): print(f"index: {len(rows)} public repos") +def app_history(apps): + """Deploys seen per app, newest first, in ~/fw/state/apps.json. + + .fuzzyworld.json holds only the latest deploy, so the previous one (needed to say + what changed) is remembered here each time the index is rebuilt. + """ + path = os.path.join(STATE, "apps.json") + try: + with open(path) as f: + hist = json.load(f) + except (FileNotFoundError, ValueError): + hist = {} + for a in apps: + deploys = hist.setdefault(a["app"], []) + if not deploys or deploys[0]["deployed"] != a["deployed"]: + deploys.insert(0, {"commit": a["commit"], "deployed": a["deployed"]}) + del deploys[APP_HISTORY:] + write(path, json.dumps(hist)) + return hist + + +def app_changes(repo, name, deploys): + """Commits in the latest deploy since the previous one with a different commit + (or the last few, the first time an app is seen), as list items.""" + cur = deploys[0]["commit"] + if not git_text(repo, "rev-parse", "-q", "--verify", cur + "^{commit}", check=False).strip(): + return "" # e.g. `fw deploy --here` from unpushed work + prev = next((d["commit"] for d in deploys[1:] if d["commit"] != cur), None) + if prev and git_text(repo, "rev-parse", "-q", "--verify", prev + "^{commit}", check=False).strip(): + rng, limit = f"{prev}..{cur}", APP_CHANGES + 1 + else: + rng, limit = cur, APP_FIRST_CHANGES + lines = git_text(repo, "log", "--format=%H%x09%s", "-n", str(limit), rng, check=False).splitlines() + items = [] + for line in lines[:APP_CHANGES]: + sha, _, subject = line.partition("\t") + page_ = os.path.join(SITE, name, "commit", sha + ".html") + subj = (f'<a href="{BASE_URL}{url(name, "commit", sha)}.html">{esc(subject)}</a>' + if os.path.exists(page_) else esc(subject)) + items.append(f"<li>{subj}</li>") + if len(lines) > APP_CHANGES: + items.append(f'<li><a href="{BASE_URL}{url(name, "log")}/">and more</a></li>') + return f'<ul class="changes">{"".join(items)}</ul>' if items else "" + + def build_apps_index(): - """apps.fuzzyworld.net/index.html from each deployed app's .fuzzyworld.json.""" + """apps.fuzzyworld.net/index.html from each deployed app's .fuzzyworld.json, newest deploy first.""" apps = [] for entry in sorted(os.listdir(APPS)): meta_path = os.path.join(APPS, entry, ".fuzzyworld.json") @@ -554,22 +1083,40 @@ def build_apps_index(): meta = json.load(f) except ValueError: continue - apps.append({"href": f"/{entry}/", "title": meta.get("title") or entry, - "description": meta.get("description", ""), "repo": meta.get("repo", entry)}) + apps.append({"app": entry, "href": f"/{entry}/", "title": meta.get("title") or entry, + "description": meta.get("description", ""), "repo": meta.get("repo", entry), + "commit": meta.get("commit", ""), "deployed": meta.get("deployed", "")}) + hist = app_history(apps) + for a in apps: + try: + a["time"] = datetime.datetime.fromisoformat(a["deployed"].replace("Z", "+00:00")).timestamp() + except ValueError: + a["time"] = 0 # Apps hosted elsewhere (e.g. engye on Cloudflare Pages), listed in server/site-apps/extra.json. + # They have no deploy record, so their repo's last commit stands in for the update time. extra = os.path.join(os.path.dirname(os.path.abspath(__file__)), "site-apps", "extra.json") if os.path.isfile(extra): with open(extra) as f: - apps += json.load(f) - apps.sort(key=lambda a: a["title"].lower()) + for a in json.load(f): + repo = os.path.join(GIT_ROOT, a.get("repo", "") + ".git") + last = git_text(repo, "log", "-1", "--format=%ct", "HEAD", check=False).strip() if a.get("repo") and os.path.isdir(repo) else "" + apps.append({**a, "time": int(last) if last.isdigit() else 0}) + apps.sort(key=lambda a: (-a["time"], a["title"].lower())) items = [] for a in apps: repo = os.path.join(GIT_ROOT, a.get("repo", "") + ".git") - src = (f' · <a href="{BASE_URL}/{urllib.parse.quote(a["repo"])}/">source</a>' - if a.get("repo") and os.path.isdir(repo) and is_public(repo) else "") + public = a.get("repo") and os.path.isdir(repo) and is_public(repo) + src = f' · <a href="{BASE_URL}/{urllib.parse.quote(a["repo"])}/">source</a>' if public else "" + when = (" · updated " + time.strftime("%Y-%m-%d", time.gmtime(a["time"]))) if a["time"] else "" desc = f'<p class="desc">{esc(a["description"])}</p>' if a.get("description") else "" - items.append(f'<li><a class="name" href="{esc(a["href"])}">{esc(a["title"])}</a>' - f'<span class="meta">{src}</span>{desc}</li>') + # Commit messages only for public repos: a private repo's history stays private. + changes = app_changes(repo, a["repo"], hist[a["app"]]) if public and a.get("app") in hist else "" + shots = app_shots(a["app"], a["repo"] if public else None) if a.get("app") else [] + thumb = (f'<a class="thumb" href="{esc(a["href"])}"><img src="{esc(shots[0][0])}" alt="" loading="lazy"></a>' + if shots else "") + cls = ' class="has-shot"' if shots else "" + items.append(f'<li{cls}>{thumb}<div><a class="name" href="{esc(a["href"])}">{esc(a["title"])}</a>' + f'<span class="meta">{when}{src}</span>{desc}{changes}</div></li>') body = (f"<h1>Pavel's apps</h1>" f'<p>Small web apps and games. No accounts, no sign-in. Source code is at ' f'<a href="{BASE_URL}/">code.fuzzyworld.net</a>.</p>' diff --git a/server/hooks/post-receive b/server/hooks/post-receive index 0633548..b6a632a 100755 --- a/server/hooks/post-receive +++ b/server/hooks/post-receive @@ -5,7 +5,23 @@ name=$(basename "$(cd "${GIT_DIR:-.}" && pwd)" .git) vis=$(git config --get fuzzyworld.visibility) log="$HOME/fw/logs/$name.log" mkdir -p "$HOME/fw/logs" -{ echo "== $(date '+%F %T') push"; cat; } >> "$log" # stdin: "<old> <new> <ref>" lines +updates=$(cat) # stdin: "<old> <new> <ref>" lines +{ echo "== $(date '+%F %T') push"; echo "$updates"; } >> "$log" + +# CI: queue a job for each pushed branch or tag whose commit has a .fw-ci.yml (or, not yet renamed, +# .gitlab-ci.yml) or a package.json. +# Runners (runner/fw-runner.py) take jobs from ~/fw/queue/pending/ through server/runner-api. +if [ "$(git config --get fuzzyworld.ci)" != off ]; then + while read -r old new ref; do + case "$ref" in refs/heads/*|refs/tags/*) ;; *) continue ;; esac + [ "$new" = 0000000000000000000000000000000000000000 ] && continue + sha=$(git rev-parse -q --verify "$new^{commit}") || continue + git cat-file -e "$sha:.fw-ci.yml" 2>/dev/null || git cat-file -e "$sha:.gitlab-ci.yml" 2>/dev/null || + git cat-file -e "$sha:package.json" 2>/dev/null || continue + "$HOME/bin/fw" ci-queue "$name" "$ref" "$sha" >> "$log" 2>&1 + done <<< "$updates" +fi + unset GIT_DIR GIT_QUARANTINE_PATH setsid nohup "$HOME/bin/fw" publish "$name" >> "$log" 2>&1 < /dev/null & if [ "$vis" = public ]; then diff --git a/server/install.sh b/server/install.sh index 4c73798..3617376 100755 --- a/server/install.sh +++ b/server/install.sh @@ -18,13 +18,17 @@ git config --global pack.threads 1 git config --global pack.windowMemory 64m git config --global pack.deltaCacheSize 16m git config --global core.bigFileThreshold 16m -mkdir -p ~/bin ~/fw/state ~/fw/logs +mkdir -p ~/bin ~/fw/state ~/fw/logs ~/fw/queue/pending ~/fw/queue/running ~/fw/ci +chmod +x ~/fw/tools/server/runner-api ~/fw/tools/server/mirror-only +# Pygments colors source code on the file pages (gen.py works without it, uncolored). +[ -d ~/fw/pylib/pygments ] || python3 -m pip install --quiet --target ~/fw/pylib "Pygments==2.21.0" ln -sfn ~/fw/tools/server/fw ~/bin/fw site=~/code.fuzzyworld.net cp ~/fw/tools/server/site/htaccess "$site/.htaccess" cp ~/fw/tools/server/site/style.css ~/fw/tools/server/site/robots.txt ~/fw/tools/server/site/404.html "$site/" mkdir -p "$site/why" && cp ~/fw/tools/server/site/why/index.html "$site/why/" mkdir -p "$site/contribute" && cp ~/fw/tools/server/site/contribute/index.html "$site/contribute/" +mkdir -p "$site/principles" && cp ~/fw/tools/server/site/principles/index.html "$site/principles/" apps=~/apps.fuzzyworld.net cp ~/fw/tools/server/site-apps/htaccess "$apps/.htaccess" cp ~/fw/tools/server/site/style.css "$apps/" diff --git a/server/runner-api b/server/runner-api new file mode 100755 index 0000000..27a3d5d --- /dev/null +++ b/server/runner-api @@ -0,0 +1,93 @@ +#!/bin/bash +# Forced command for CI runner keys in ~/.ssh/authorized_keys on DreamHost: +# +# restrict,command="/home/fuzzygamer/fw/tools/server/runner-api" ssh-ed25519 AAAA... truenas-runner +# +# A runner (runner/fw-runner.py) can then only: +# claim <runner> take the oldest queued job; prints its JSON, or nothing +# git-upload-pack '<repo>' clone/fetch ~/git/<name>.git (the job's source) +# put <repo> <sha> <file> store one result file (stdin) in ~/fw/ci/<repo>/<sha>/ +# done <repo> <sha> <job-id> the job is finished: remove it from the queue, refresh pages +# put-shot <app> <file> store one screenshot (stdin) in apps.fuzzyworld.net/_shots/<app>/ +# shots-done <app> <repo> refresh the apps index and the repo's page +# Anything else is refused. Every call is logged to ~/fw/logs/runner-api.log, except a claim that +# finds nothing (runners ask every few minutes). +set -u +cmd=${SSH_ORIGINAL_COMMAND:-} +QUEUE=$HOME/fw/queue +CI=$HOME/fw/ci +SHOTS=$HOME/apps.fuzzyworld.net/_shots +MAX_FILE=$((20 * 1024 * 1024)) +log="$HOME/fw/logs/runner-api.log" +mkdir -p "$(dirname "$log")" "$QUEUE/pending" "$QUEUE/running" +note() { echo "$(date -u '+%F %T') ${SSH_CLIENT%% *} $*" >> "$log"; } +[[ "$cmd" == claim* ]] || note "$cmd" + +die() { echo "runner-api: $*" >&2; exit 1; } +valid_name() { [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] && [[ "$1" != *.git ]]; } +valid_repo() { valid_name "$1" && [ -d "$HOME/git/$1.git" ]; } +valid_sha() { [[ "$1" =~ ^[0-9a-f]{40}$ ]]; } +valid_job() { [[ "$1" =~ ^[0-9]+-[A-Za-z0-9._-]+-[0-9a-f]{12}$ ]]; } + +# stdin to a file, refusing anything over MAX_FILE; written beside the target, then renamed. +store() { + local dest=$1 tmp="$1.part" + head -c $((MAX_FILE + 1)) > "$tmp" + if [ "$(stat -c %s "$tmp")" -gt "$MAX_FILE" ]; then rm -f "$tmp"; die "file too large"; fi + mv "$tmp" "$dest" +} + +read -r -a w <<< "$cmd" +case "${w[0]:-}" in + claim) + runner=${w[1]:-} + valid_name "$runner" || die "usage: claim <runner>" + # Jobs a runner took more than 2 hours ago (it crashed or lost the connection) go back in the queue. + find "$QUEUE/running" -name '*.json' -mmin +120 -exec mv -t "$QUEUE/pending/" {} + 2>/dev/null + for f in $(ls "$QUEUE/pending" 2>/dev/null | sort); do + # rename(2) is atomic: if two runners race for the same job, only one mv succeeds. + if mv "$QUEUE/pending/$f" "$QUEUE/running/$f" 2>/dev/null; then + touch "$QUEUE/running/$f" + note "$cmd: $f" + cat "$QUEUE/running/$f" + exit 0 + fi + done ;; + git-upload-pack) + repo=${cmd#git-upload-pack \'}; repo=${repo%\'}; repo=${repo#/home/fuzzygamer/}; repo=${repo#\~/} + [[ "$repo" =~ ^git/([A-Za-z0-9][A-Za-z0-9._-]*)\.git$ ]] && valid_repo "${BASH_REMATCH[1]}" || die "no such repo" + exec git-upload-pack "$HOME/$repo" ;; + put) + repo=${w[1]:-} sha=${w[2]:-} file=${w[3]:-} + valid_repo "$repo" && valid_sha "$sha" && [[ "$file" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*\.(log|json)$ ]] || die "usage: put <repo> <sha> <file.log|file.json>" + mkdir -p "$CI/$repo/$sha" + store "$CI/$repo/$sha/$file" + # A pipeline.json means the status changed: refresh the pages (in the background; publishing is locked per repo). + if [ "$file" = pipeline.json ]; then + setsid nohup "$HOME/bin/fw" publish "$repo" >> "$HOME/fw/logs/$repo.log" 2>&1 < /dev/null & + fi ;; + done) + repo=${w[1]:-} sha=${w[2]:-} job=${w[3]:-} + valid_repo "$repo" && valid_sha "$sha" && valid_job "$job" || die "usage: done <repo> <sha> <job-id>" + rm -f "$QUEUE/running/$job.json" ;; + put-shot) + app=${w[1]:-} file=${w[2]:-} + valid_name "$app" && [[ "$file" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*\.(png|json)$ ]] || die "usage: put-shot <app> <file.png|shots.json>" + mkdir -p "$SHOTS/$app" + store "$SHOTS/$app/$file" ;; + shots-done) + app=${w[1]:-} repo=${w[2]:-} + valid_name "$app" && valid_repo "$repo" || die "usage: shots-done <app> <repo>" + # Screenshots from earlier runs that shots.json no longer lists are removed. + python3 - "$SHOTS/$app" <<'PY' +import json, os, sys +d = sys.argv[1] +keep = {s["file"] for s in json.load(open(os.path.join(d, "shots.json"))).get("shots", [])} +for f in os.listdir(d): + if f.endswith(".png") and f not in keep: + os.remove(os.path.join(d, f)) +PY + setsid nohup "$HOME/bin/fw" app-deployed "$app" "$repo" >> "$HOME/fw/logs/$repo.log" 2>&1 < /dev/null & ;; + *) + die "this key may only run CI jobs (refused: $cmd)" ;; +esac diff --git a/server/site-apps/htaccess b/server/site-apps/htaccess index 9b84e43..b4f4572 100644 --- a/server/site-apps/htaccess +++ b/server/site-apps/htaccess @@ -28,6 +28,10 @@ RemoveType .php .php3 .php4 .php5 .php7 .php8 .phtml .phar <FilesMatch "(\.html|sw\.js|service-worker\.js|\.webmanifest|manifest\.json)$"> Header set Cache-Control "no-cache" </FilesMatch> + # Screenshots from CI runners keep their names when they are retaken. + <If "%{REQUEST_URI} =~ m#^/_shots/#"> + Header set Cache-Control "no-cache" + </If> <If "%{REQUEST_URI} =~ m#^/[^/]+/assets/#"> Header set Cache-Control "public, max-age=31536000, immutable" </If> diff --git a/server/site/contribute/index.html b/server/site/contribute/index.html index 1d605a9..cecb6d2 100644 --- a/server/site/contribute/index.html +++ b/server/site/contribute/index.html @@ -46,7 +46,7 @@ <p><a href="/why/">Why there are no merge requests here</a></p> </main> - <footer class="site"><a href="/">All repositories</a> · <a href="https://apps.fuzzyworld.net/">Apps</a></footer> + <footer class="site"><a href="/">All repositories</a> · <a href="https://apps.fuzzyworld.net/">Apps</a> · <a href="/principles/">Principles</a></footer> </body> </html> diff --git a/server/site/how/index.html b/server/site/how/index.html index b9f9c83..357384f 100644 --- a/server/site/how/index.html +++ b/server/site/how/index.html @@ -89,9 +89,17 @@ account to read my code. The one real loss is easy contributions: there are no merge requests here yet.</p> + <h2>Principles</h2> + <p>Things made with AI should be shared freely with the world. The code and apps here were made with + AI, so they are MIT-licensed: use them, change them, share them.</p> + <p>My conversations with the AI are different. My words and my way of working are mine. I publish a + conversation only when I choose to, I can take that permission back, and nobody may copy my words or + my style of work without my express permission.</p> + <p>More on the <a href="/principles/">principles</a> page.</p> + <p><a href="/why/">← Why I left GitLab</a></p> </main> - <footer class="site"><a href="/">All repositories</a> · <a href="https://apps.fuzzyworld.net/">Apps</a></footer> + <footer class="site"><a href="/">All repositories</a> · <a href="https://apps.fuzzyworld.net/">Apps</a> · <a href="/principles/">Principles</a></footer> </body> </html> diff --git a/server/site/htaccess b/server/site/htaccess index 2e6d563..19d3669 100644 --- a/server/site/htaccess +++ b/server/site/htaccess @@ -16,6 +16,13 @@ RemoveType .php .php3 .php4 .php5 .php7 .php8 .phtml .phar Require all denied </FilesMatch> +# objects/<sha>: file contents for VS Code in the browser (apps.fuzzyworld.net/vscode/). Always sent +# as downloads, so a repo's .html or .svg file can never render as a page on this site. +<If "%{REQUEST_URI} =~ m#^/[^/]+/objects/[0-9a-f]{40}$#"> + ForceType application/octet-stream + Header set Content-Disposition attachment +</If> + # Public clone copies: serve only what dumb-HTTP git needs. RedirectMatch 404 ^/[^/]+\.git/(config|description|hooks/|logs/|info/exclude|FETCH_HEAD|ORIG_HEAD) @@ -27,6 +34,17 @@ RedirectMatch 404 ^/[^/]+\.git/(config|description|hooks/|logs/|info/exclude|FET <FilesMatch "\.(html|css)$"> Header set Cache-Control "no-cache" </FilesMatch> + # VS Code in the browser reads files.json and objects/ from apps.fuzzyworld.net. + <If "%{REQUEST_URI} =~ m#^/[^/]+/(branch/[^/]+/)?files\.json$|^/[^/]+/objects/#"> + Header set Access-Control-Allow-Origin "*" + </If> + <If "%{REQUEST_URI} =~ m#^/[^/]+/(branch/[^/]+/)?files\.json$#"> + Header set Cache-Control "no-cache" + </If> + # objects/<sha> is named by its contents, so it never changes. + <If "%{REQUEST_URI} =~ m#^/[^/]+/objects/[0-9a-f]{40}$#"> + Header set Cache-Control "public, max-age=31536000, immutable" + </If> # Git objects and packs never change; refs do. <If "%{REQUEST_URI} =~ m#^/[^/]+\.git/objects/(pack/|[0-9a-f]{2}/)#"> Header set Cache-Control "public, max-age=31536000, immutable" diff --git a/server/site/principles/index.html b/server/site/principles/index.html new file mode 100644 index 0000000..bd2e956 --- /dev/null +++ b/server/site/principles/index.html @@ -0,0 +1,30 @@ +<!doctype html> +<html lang="en"> + +<head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Principles · code.fuzzyworld.net</title> + <link rel="stylesheet" href="/style.css"> +</head> + +<body> + <header class="site"><a class="home" href="/">code.fuzzyworld.net</a></header> + <main> + <h1>Principles</h1> + <h2>What's made with AI is shared</h2> + <p>Things made with AI should be shared freely with the world. The code and apps here were made with + AI, so they are MIT-licensed: use them, change them, share them.</p> + <p>My conversations with the AI are different. My words and my way of working are mine. I publish a + conversation only when I choose to, I can take that permission back, and nobody may copy my words or + my style of work without my express permission.</p> + <p class="meta">Copyright protects my words. It generally doesn't protect a style or a way of working, so + the last part is a statement of my terms and wishes, not something a license can enforce.</p> + <h2>No sign-in, no limits</h2> + <p>Everything here is static files: no accounts, no rate limits, nothing that runs code from a + repository. See <a href="/why/">why this site exists</a>.</p> + </main> + <footer class="site"><a href="/">All repositories</a> · <a href="https://apps.fuzzyworld.net/">Apps</a> · <a href="/why/">Why this site exists</a></footer> +</body> + +</html> diff --git a/server/site/style.css b/server/site/style.css index aa8b678..c3ddf21 100644 --- a/server/site/style.css +++ b/server/site/style.css @@ -52,7 +52,32 @@ ul.repos li { padding: 0.6rem 0; border-bottom: 1px solid var(--line); } ul.repos .name { font-weight: 600; overflow-wrap: anywhere; } ul.repos .meta { font-size: 0.9rem; white-space: nowrap; } ul.repos .desc { margin: 0.2rem 0 0; } +ul.repos ul.changes { margin: 0.3rem 0 0; padding-left: 1.2rem; font-size: 0.9rem; color: var(--muted); } +ul.repos ul.changes li { padding: 0; border: 0; overflow-wrap: anywhere; } details summary { cursor: pointer; color: var(--muted); margin: 1rem 0 0.5rem; } +.branch-note { background: var(--soft); border: 1px solid var(--line); border-radius: 6px; padding: 0.4rem 0.75rem; overflow-wrap: anywhere; } +/* Blame: who last changed each run of lines. */ +.source.blamed td.blame { white-space: nowrap; color: var(--muted); font-size: 12px; border-right: 1px solid var(--line); min-width: 18rem; } +.source.blamed tr.run td { border-top: 1px solid var(--line); } +.source.blamed .who { display: inline-block; max-width: 9rem; overflow: hidden; text-overflow: ellipsis; vertical-align: bottom; } +/* CI */ +.ci { font-weight: 700; text-decoration: none; } +.ci-success { color: var(--add-fg); } .ci-failed { color: var(--del-fg); } .ci-running { color: var(--hl-m); } .ci-error { color: var(--muted); } +pre.ci-log { max-height: 40rem; overflow: auto; white-space: pre-wrap; overflow-wrap: anywhere; } +/* Syntax highlighting (class names from server/gen.py token_class). */ +:root { --hl-c: #6a737d; --hl-k: #cf222e; --hl-s: #0a3069; --hl-m: #0550ae; --hl-nf: #8250df; --hl-nc: #953800; --hl-nt: #116329; } +@media (prefers-color-scheme: dark) { + :root { --hl-c: #8b949e; --hl-k: #ff7b72; --hl-s: #a5d6ff; --hl-m: #79c0ff; --hl-nf: #d2a8ff; --hl-nc: #ffa657; --hl-nt: #7ee787; } +} +.code .c { color: var(--hl-c); font-style: italic; } .code .cp, .code .k { color: var(--hl-k); } +.code .s { color: var(--hl-s); } .code .m, .code .nb, .code .na { color: var(--hl-m); } +.code .nf, .code .nd { color: var(--hl-nf); } .code .nc { color: var(--hl-nc); } .code .nt { color: var(--hl-nt); } +.code .gi { color: var(--add-fg); } .code .gd { color: var(--del-fg); } .code .gh { font-weight: 700; } @media (max-width: 40rem) { td.author, .log td.date { display: none; } } +/* Screenshots: thumbnails on the apps index, a row on a repo's summary page. */ +ul.repos li.has-shot { display: grid; grid-template-columns: 4.5rem 1fr; gap: 0.75rem; align-items: start; } +ul.repos .thumb img { display: block; width: 4.5rem; height: 6rem; object-fit: cover; object-position: top; border: 1px solid var(--line); border-radius: 6px; } +.shots { display: flex; gap: 0.75rem; overflow-x: auto; } +.shots img { display: block; height: 16rem; width: auto; border: 1px solid var(--line); border-radius: 6px; } diff --git a/server/site/why/index.html b/server/site/why/index.html index ae8c4bb..205f2f4 100644 --- a/server/site/why/index.html +++ b/server/site/why/index.html @@ -29,8 +29,16 @@ <p></p> <p>I'm currently trying to formalize the process of the move from GitLab to Dreamhost, and eventually my learnings are going to be at /how/ on this website.</p> + + <h2>Principles</h2> + <p>Things made with AI should be shared freely with the world. The code and apps here were made with + AI, so they are MIT-licensed: use them, change them, share them.</p> + <p>My conversations with the AI are different. My words and my way of working are mine. I publish a + conversation only when I choose to, I can take that permission back, and nobody may copy my words or + my style of work without my express permission.</p> + <p>More on the <a href="/principles/">principles</a> page.</p> </main> - <footer class="site"><a href="/">All repositories</a> · <a href="https://apps.fuzzyworld.net/">Apps</a></footer> + <footer class="site"><a href="/">All repositories</a> · <a href="https://apps.fuzzyworld.net/">Apps</a> · <a href="/principles/">Principles</a></footer> </body> </html> \ No newline at end of file diff --git a/vscode-web/fw-fs/extension.js b/vscode-web/fw-fs/extension.js new file mode 100644 index 0000000..9b853ac --- /dev/null +++ b/vscode-web/fw-fs/extension.js @@ -0,0 +1,184 @@ +// Read-only file system for VS Code in the browser: fuzzyworld:/<repo>[@<branch>]/<path>. +// +// code.fuzzyworld.net publishes, for each browsable branch, files.json ({path: [blob sha, size]}) +// and, for each file up to `maxBytes`, its contents at /<repo>/objects/<sha>. This extension +// turns those into folders and files, plus quick open (Ctrl+P) and search across files. +// Nothing can be written: the repositories change only by pushing to them. +const vscode = require("vscode"); + +const CODE = "https://code.fuzzyworld.net"; +const SEARCH_MAX_FILE = 256 * 1024; // larger files are skipped by text search +const FETCHES = 8; // parallel requests during text search + +const manifests = new Map(); // "<repo>[@<branch>]" -> Promise<{files, dirs, maxBytes}> +const contents = new Map(); // blob sha -> Promise<Uint8Array> + +function parse(uri) { + const seg = uri.path.split("/").filter(Boolean); + const root = seg[0] || ""; + const at = root.indexOf("@"); + return { + root, + repo: at < 0 ? root : root.slice(0, at), + branch: at < 0 ? "" : root.slice(at + 1), + rel: seg.slice(1).join("/"), + }; +} + +function manifest(uri) { + const { root, repo, branch } = parse(uri); + if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(repo)) throw vscode.FileSystemError.FileNotFound(uri); + if (!manifests.has(root)) { + const base = `${CODE}/${encodeURIComponent(repo)}` + + (branch ? `/branch/${branch.split("/").map(encodeURIComponent).join("~")}` : ""); + manifests.set(root, fetch(`${base}/files.json`).then(async (r) => { + if (!r.ok) { + manifests.delete(root); + throw vscode.FileSystemError.FileNotFound(`${repo}${branch ? " (branch " + branch + ")" : ""} is not on code.fuzzyworld.net`); + } + const m = await r.json(); + const files = new Map(Object.entries(m.files)); + const dirs = new Map([["", new Map()]]); + for (const p of files.keys()) { + const parts = p.split("/"); + for (let i = 0; i < parts.length; i++) { + const dir = parts.slice(0, i).join("/"); + if (!dirs.has(dir)) dirs.set(dir, new Map()); + dirs.get(dir).set(parts[i], i === parts.length - 1 ? vscode.FileType.File : vscode.FileType.Directory); + } + } + return { repo, files, dirs, maxBytes: m.maxBytes }; + })); + } + return manifests.get(root); +} + +async function read(m, rel, uri) { + const entry = m.files.get(rel); + if (!entry) throw vscode.FileSystemError.FileNotFound(uri); + const [sha, size] = entry; + if (size < 0) throw vscode.FileSystemError.Unavailable(`${rel} is a git submodule`); + if (size > m.maxBytes) throw vscode.FileSystemError.Unavailable(`${rel} is too large to open here; clone the repository to get it`); + if (!contents.has(sha)) { + contents.set(sha, fetch(`${CODE}/${encodeURIComponent(m.repo)}/objects/${sha}`).then(async (r) => { + if (!r.ok) { + contents.delete(sha); + throw vscode.FileSystemError.Unavailable(uri); + } + return new Uint8Array(await r.arrayBuffer()); + })); + } + return contents.get(sha); +} + +const readOnly = () => { throw vscode.FileSystemError.NoPermissions("code.fuzzyworld.net is read-only: clone the repository to change it"); }; + +const fileSystem = { + onDidChangeFile: new vscode.EventEmitter().event, + watch: () => new vscode.Disposable(() => {}), + async stat(uri) { + const m = await manifest(uri); + const { rel } = parse(uri); + if (m.dirs.has(rel)) return { type: vscode.FileType.Directory, ctime: 0, mtime: 0, size: 0 }; + const entry = m.files.get(rel); + if (!entry) throw vscode.FileSystemError.FileNotFound(uri); + return { type: vscode.FileType.File, ctime: 0, mtime: 0, size: Math.max(entry[1], 0), permissions: vscode.FilePermission?.Readonly }; + }, + async readDirectory(uri) { + const m = await manifest(uri); + const dir = m.dirs.get(parse(uri).rel); + if (!dir) throw vscode.FileSystemError.FileNotDirectory(uri); + return [...dir.entries()]; + }, + async readFile(uri) { + return read(await manifest(uri), parse(uri).rel, uri); + }, + createDirectory: readOnly, writeFile: readOnly, delete: readOnly, rename: readOnly, copy: readOnly, +}; + +// Quick open: files whose path contains the typed characters in order. +const fileSearch = { + async provideFileSearchResults(query, options) { + const m = await manifest(options.folder); + const want = query.pattern.toLowerCase().replace(/\s+/g, ""); + const out = []; + for (const p of m.files.keys()) { + const lower = p.toLowerCase(); + let i = 0; + for (const ch of lower) if (ch === want[i]) i++; + if (i === want.length) { + out.push(vscode.Uri.joinPath(options.folder, p)); + if (options.maxResults && out.length >= options.maxResults) break; + } + } + return out; + }, +}; + +// Search across files: fetches each text file (up to SEARCH_MAX_FILE) once, then matches line by line. +const textSearch = { + async provideTextSearchResults(query, options, progress, token) { + const m = await manifest(options.folder); + let source = query.isRegExp ? query.pattern : query.pattern.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + if (query.isWordMatch) source = `\\b${source}\\b`; + const re = new RegExp(source, query.isCaseSensitive ? "g" : "gi"); + const paths = [...m.files.entries()].filter(([, [, size]]) => size >= 0 && size <= SEARCH_MAX_FILE).map(([p]) => p); + const max = options.maxResults || 2000; + let found = 0; + let next = 0; + const decoder = new TextDecoder(); + async function worker() { + while (next < paths.length && found < max && !token.isCancellationRequested) { + const p = paths[next++]; + const uri = vscode.Uri.joinPath(options.folder, p); + let bytes; + try { bytes = await read(m, p, uri); } catch { continue; } + if (bytes.subarray(0, 8000).includes(0)) continue; // binary + const lines = decoder.decode(bytes).split("\n"); + for (let n = 0; n < lines.length && found < max; n++) { + re.lastIndex = 0; + let hit; + while ((hit = re.exec(lines[n])) && found < max) { + const end = hit.index + Math.max(hit[0].length, 1); + progress.report({ + uri, + ranges: new vscode.Range(n, hit.index, n, end), + preview: { text: lines[n], matches: new vscode.Range(0, hit.index, 0, end) }, + }); + found++; + if (!hit[0].length) re.lastIndex++; + } + } + } + } + await Promise.all(Array.from({ length: FETCHES }, worker)); + return { limitHit: found >= max }; + }, +}; + +exports.activate = function (context) { + context.subscriptions.push( + vscode.workspace.registerFileSystemProvider("fuzzyworld", fileSystem, { isCaseSensitive: true, isReadonly: true })); + // Proposed APIs (enabled for this extension in index.html's product configuration). + if (vscode.workspace.registerFileSearchProvider) { + context.subscriptions.push(vscode.workspace.registerFileSearchProvider("fuzzyworld", fileSearch)); + } + if (vscode.workspace.registerTextSearchProvider) { + context.subscriptions.push(vscode.workspace.registerTextSearchProvider("fuzzyworld", textSearch)); + } + openReadme(); +}; + +// Start with the README as text. (Markdown preview is a webview, and webviews need their own +// domain per view, which this static site doesn't have yet; so it isn't opened by default.) +async function openReadme() { + const folder = vscode.workspace.workspaceFolders && vscode.workspace.workspaceFolders[0]; + if (!folder || folder.uri.scheme !== "fuzzyworld" || vscode.window.visibleTextEditors.length) return; + try { + const m = await manifest(folder.uri); + const readme = [...m.dirs.get("").keys()].find((n) => /^readme(\.md|\.markdown|\.txt)?$/i.test(n)); + if (readme) await vscode.window.showTextDocument(vscode.Uri.joinPath(folder.uri, readme), { preview: true }); + } catch { + // no manifest: the explorer shows the error + } +} diff --git a/vscode-web/fw-fs/package.json b/vscode-web/fw-fs/package.json new file mode 100644 index 0000000..28767d4 --- /dev/null +++ b/vscode-web/fw-fs/package.json @@ -0,0 +1,21 @@ +{ + "name": "fuzzyworld-fs", + "displayName": "code.fuzzyworld.net", + "description": "Read-only access to the public repositories on code.fuzzyworld.net.", + "publisher": "fuzzyworld", + "version": "1.0.0", + "license": "MIT", + "engines": { "vscode": "^1.80.0" }, + "browser": "./extension.js", + "activationEvents": ["onFileSystem:fuzzyworld", "onSearch:fuzzyworld"], + "enabledApiProposals": ["fileSearchProvider", "textSearchProvider"], + "contributes": { + "configurationDefaults": { + "workbench.startupEditor": "none", + "typescript.tsserver.web.projectWideIntellisense.enabled": false, + "security.workspace.trust.enabled": false, + "extensions.autoCheckUpdates": false, + "telemetry.telemetryLevel": "off" + } + } +} diff --git a/vscode-web/fw-fs/package.nls.json b/vscode-web/fw-fs/package.nls.json new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/vscode-web/fw-fs/package.nls.json @@ -0,0 +1 @@ +{} diff --git a/vscode-web/index.html b/vscode-web/index.html new file mode 100644 index 0000000..53dbd31 --- /dev/null +++ b/vscode-web/index.html @@ -0,0 +1,69 @@ +<!doctype html> +<!-- VS Code in the browser for code.fuzzyworld.net: apps.fuzzyworld.net/vscode/?repo=<name>[&branch=<branch>]. + Installed by vscode-web/install.sh, beside dist/ (the vscode-web build) and fw-fs/ (our read-only file system). --> +<html lang="en"> +<head> +<meta charset="utf-8"> +<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0, user-scalable=no"> +<meta name="mobile-web-app-capable" content="yes"> +<title>VS Code · code.fuzzyworld.net</title> +<link rel="icon" href="dist/favicon.ico" type="image/x-icon"> +<link data-name="vs/workbench/workbench.web.main" rel="stylesheet" href="dist/out/vs/workbench/workbench.web.main.css"> +<style> + .fw-missing { font: 16px/1.5 system-ui, sans-serif; max-width: 40rem; margin: 3rem auto; padding: 0 16px; } + @media (prefers-color-scheme: dark) { body:has(.fw-missing) { background: #111113; color: #e8e8ea; } .fw-missing a { color: #79b8ff; } } +</style> +</head> +<body aria-label=""> +<script> + // Which repository: ?repo=<name>, optionally &branch=<branch> (the default branch otherwise). + (function () { + var q = new URLSearchParams(location.search); + var repo = q.get("repo") || "", branch = q.get("branch") || ""; + var okName = /^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(repo); + var okBranch = !branch || (!/[\s~^:?*\[\\@]|\.\.|\/\/|^\/|\/$/.test(branch)); + if (!okName || !okBranch) { + document.body.innerHTML = '<div class="fw-missing"><h1>Which repository?</h1><p>Open a repository from its page on ' + + '<a href="https://code.fuzzyworld.net/">code.fuzzyworld.net</a> ("Open in VS Code in the browser"), or add ' + + '<code>?repo=<name></code> to this address.</p></div>'; + window.fwStop = true; + return; + } + document.title = repo + (branch ? " (" + branch + ")" : "") + " · VS Code · code.fuzzyworld.net"; + var app = new URL(".", location.href); + window.product = { + productConfiguration: { + nameShort: "VS Code · code.fuzzyworld.net", + nameLong: "VS Code · code.fuzzyworld.net", + applicationName: "code-fuzzyworld", + dataFolderName: ".vscode-fuzzyworld", + version: "1.91.1", + extensionEnabledApiProposals: { "fuzzyworld.fuzzyworld-fs": ["fileSearchProvider", "textSearchProvider"] }, + // Webviews (e.g. Markdown preview) load from this site, not from Microsoft's CDN. + webviewContentExternalBaseUrlTemplate: new URL("dist/out/vs/workbench/contrib/webview/browser/pre/", app).href, + }, + folderUri: { scheme: "fuzzyworld", path: "/" + repo + (branch ? "@" + branch : "") }, + additionalBuiltinExtensions: [{ scheme: app.protocol.slice(0, -1), authority: app.host, path: app.pathname + "fw-fs" }], + }; + })(); +</script> +<script>if (!window.fwStop) document.write('<script src="dist/out/vs/loader.js"><\/script><script src="dist/out/vs/webPackagePaths.js"><\/script>');</script> +<script> + if (!window.fwStop) { + var dist = new URL("dist/", location.href).href; + Object.keys(self.webPackagePaths).forEach(function (key) { + self.webPackagePaths[key] = dist + "node_modules/" + key + "/" + self.webPackagePaths[key]; + }); + require.config({ + baseUrl: dist + "out", + recordStats: false, + trustedTypesPolicy: window.trustedTypes && window.trustedTypes.createPolicy("amdLoader", { createScriptURL: function (v) { return v; } }), + paths: self.webPackagePaths, + }); + document.write('<script src="dist/out/vs/workbench/workbench.web.main.nls.js"><\/script>' + + '<script src="dist/out/vs/workbench/workbench.web.main.js"><\/script>' + + '<script src="dist/out/vs/code/browser/workbench/workbench.js"><\/script>'); + } +</script> +</body> +</html> diff --git a/vscode-web/install.sh b/vscode-web/install.sh new file mode 100755 index 0000000..d771362 --- /dev/null +++ b/vscode-web/install.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# Install VS Code in the browser at apps.fuzzyworld.net/vscode/. Run from the laptop: +# +# vscode-web/install.sh build, then upload to dh:apps.fuzzyworld.net/vscode/ +# vscode-web/install.sh --out DIR only build into DIR (for trying it locally) +# +# The editor is the vscode-web npm package (VS Code's open-source web build, MIT), pinned below; +# npm checks the download against the registry's integrity hash. Beside it go index.html and +# fw-fs/, our read-only file system for code.fuzzyworld.net. The app has no .fuzzyworld.json, so +# it isn't listed on the apps index; repo pages link to it. +set -euo pipefail +cd "$(dirname "$0")" +VERSION=1.91.1 +HOST=${FW_HOST:-dh} +out="" +[ "${1:-}" = --out ] && out=${2:?--out needs a directory} + +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT +( cd "$tmp" && npm pack --silent "vscode-web@$VERSION" >/dev/null && tar xzf "vscode-web-$VERSION.tgz" ) +build=${out:-$tmp/build} +mkdir -p "$build" +rsync -a --delete "$tmp/package/dist/" "$build/dist/" +rsync -a --delete fw-fs/ "$build/fw-fs/" +cp index.html "$build/index.html" +echo "built VS Code web $VERSION in $build" + +if [ -z "$out" ]; then + ssh "$HOST" 'mkdir -p apps.fuzzyworld.net/vscode' + rsync -az --delete "$build/" "$HOST:apps.fuzzyworld.net/vscode/" + echo "installed: https://apps.fuzzyworld.net/vscode/?repo=<name>" +fi