fuzzyworld-tools / commit
Add fuzzyworld-tools: static code browser and repo tooling
.gitignore | 3 + README.md | 81 +++++++ bin/fw | 17 ++ server/fw | 164 ++++++++++++++ server/gen.py | 517 +++++++++++++++++++++++++++++++++++++++++++++ server/hooks/post-receive | 13 ++ server/install.sh | 23 ++ server/site/404.html | 17 ++ server/site/htaccess | 37 ++++ server/site/robots.txt | 4 + server/site/style.css | 58 +++++ server/site/why/index.html | 25 +++ 12 files changed, 959 insertions(+) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b908d4c --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +__pycache__/ +*.pyc +.DS_Store diff --git a/README.md b/README.md new file mode 100644 index 0000000..7c3fc55 --- /dev/null +++ b/README.md @@ -0,0 +1,81 @@ +# fuzzyworld-tools + +Tooling for Pavel's self-hosted code and apps on DreamHost: + +- **code.fuzzyworld.net**: static browse pages and read-only `git clone` for public repos +- **apps.fuzzyworld.net**: deployed web apps (`fw deploy`, Phase 4 of the migration; not built yet) + +There is no app server. Pages are generated when you push; Apache only serves files. + +## Commands + +On the laptop, `bin/fw` forwards repo commands to DreamHost (`ssh dh ~/bin/fw ...`): + +| Command | What it does | +|---|---| +| `fw list` | every repo, its visibility and last commit | +| `fw new <name> [--public] [--description TEXT]` | create `~/git/<name>.git` on DreamHost; **private unless `--public`** | +| `fw public <name>` | publish the source on code.fuzzyworld.net right away | +| `fw private <name>` | unpublish right away: pages, clone URL and index entry are removed | +| `fw describe <name> TEXT` | set the description shown on the index and summary page | +| `fw publish <name>` | refresh one repo's public copy and pages (the push hook does this) | +| `fw publish-all` | refresh every repo, then the index | + +Push to a repo with `git push dh:git/<name>.git` (`dh` is the SSH alias in `~/.ssh/config`). + +## Visibility + +Each bare repo stores its own setting: `git config fuzzyworld.visibility public|private`. +**Anything other than exactly `public` is private**, including a missing value, so a repo is never published by accident. +A private repo has no pages, no clone URL and no index entry. Pushing to it publishes nothing. + +Other settings in the repo config: `fuzzyworld.description`, plus `fuzzyworld.gitlab-id` / `fuzzyworld.gitlab-path` for repos migrated from GitLab. + +## Layout on DreamHost + +| Path | Contents | +|---|---| +| `~/git/<name>.git` | canonical bare repos (push here; not web-visible) | +| `~/code.fuzzyworld.net/<name>/` | generated pages for public repos | +| `~/code.fuzzyworld.net/<name>.git` | public clone copy (dumb HTTP), refreshed on push | +| `~/fw/tools/server/` | this repo's `server/` directory, installed by `server/install.sh` | +| `~/bin/fw` | symlink to `~/fw/tools/server/fw` | +| `~/fw/state/` | per-repo page manifests and locks | +| `~/fw/logs/<name>.log` | output of each push's publish run | +| `~/fw/pylib/` | Python-Markdown 3.7 (`pip install --target`), used for README rendering | + +Every repo uses the shared hook via `core.hooksPath=~/fw/tools/server/hooks`. `post-receive` starts `fw publish <name>` in the background, so a push returns immediately; pages update about 1–5 seconds later. + +## Page URLs + +| URL | Page | +|---|---| +| `/` | index of public repos, newest first | +| `/<name>/` | summary: description, clone command, recent commits, README | +| `/<name>/tree/<dir>/` | directory listing (default branch) | +| `/<name>/blob/<path>.html` | file view with line numbers; `.md` files rendered | +| `/<name>/raw/<path>` | raw copies of images only (png, jpg, gif, webp, ico, avif, bmp, svg; ≤ 2 MB) | +| `/<name>/log/` | last 100 commits | +| `/<name>/commit/<sha>.html` | one commit with its diff (truncated at 300 KB) | +| `/<name>/refs/` | branches and tags | +| `/why/` | why the site exists | + +Only the default branch can be browsed. Every branch and tag can be cloned. Author email addresses are not shown. + +## Security + +The site never executes anything from a repo: + +- pages are pre-generated HTML with all repo content escaped +- raw copies are written only for image extensions; `.htaccess` also denies `.php`, `.shtml`, `.cgi`, `.pl`, `.py` and `.sh` and disables SSI/CGI (DreamHost runs `.php` files and SSI otherwise; tested 2026-10-26) +- `Content-Security-Policy: script-src 'none'` on every response (this also covers SVG images) +- inside public clone copies, `config`, `description`, `hooks/` and `logs/` return 404 + +## Install / update + +```bash +server/install.sh # copy server/ to dh, install site files and hooks +server/install.sh --publish # ...and republish every repo +``` + +Requirements on DreamHost: git, Python 3.12, `flock`, `setsid`. diff --git a/bin/fw b/bin/fw new file mode 100755 index 0000000..ff837b2 --- /dev/null +++ b/bin/fw @@ -0,0 +1,17 @@ +#!/bin/bash +# fw (laptop side): runs repo commands on DreamHost over ssh. +# fw list | new <name> [--public] [--description TEXT] | public <name> | private <name> +# fw describe <name> TEXT | publish <name> | publish-all +# `fw deploy` and app scaffolding in `fw new` arrive in Phase 4. +set -euo pipefail +HOST=${FW_HOST:-dh} + +case "${1:-help}" in + list|new|public|private|describe|publish|publish-all|help|-h|--help) + # printf %q keeps arguments (e.g. a description with spaces) intact through ssh. + exec ssh "$HOST" "~/bin/fw $(printf '%q ' "$@")" ;; + deploy) + echo "fw deploy: not available yet (Phase 4)" >&2; exit 2 ;; + *) + echo "fw: unknown command '$1' (try: fw help)" >&2; exit 2 ;; +esac diff --git a/server/fw b/server/fw new file mode 100755 index 0000000..1ed6227 --- /dev/null +++ b/server/fw @@ -0,0 +1,164 @@ +#!/bin/bash +# fw: manage fuzzyworld git repos on DreamHost. +# +# fw list every repo with its visibility and last commit +# fw new <name> [--public] [--description TEXT] +# create ~/git/<name>.git (private unless --public) +# fw public <name> publish source on code.fuzzyworld.net +# fw private <name> unpublish: remove public clone copy and pages +# fw describe <name> TEXT set the description shown on the index +# fw publish <name> refresh public copy + pages (the push hook runs this) +# fw publish-all publish every public repo, then the index +# fw install-hooks point every repo at the shared hooks +# +# Visibility is `git config fuzzyworld.visibility` inside each repo. Anything other +# than exactly "public" is treated as private. +set -euo pipefail + +GIT_ROOT=${FW_GIT_ROOT:-$HOME/git} +SITE=${FW_CODE_SITE:-$HOME/code.fuzzyworld.net} +TOOLS=${FW_TOOLS:-$HOME/fw/tools} +STATE=${FW_STATE:-$HOME/fw/state} +LOGS=${FW_LOGS:-$HOME/fw/logs} +HOOKS=$TOOLS/server/hooks +GEN=(python3 "$TOOLS/server/gen.py") + +die() { echo "fw: $*" >&2; exit 1; } + +valid_name() { + [[ "$1" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] && [[ "$1" != *.git ]] || die "invalid repo name: '$1'" +} + +repo_dir() { valid_name "$1"; echo "$GIT_ROOT/$1.git"; } + +need_repo() { [ -d "$(repo_dir "$1")" ] || die "no such repo: $1"; } + +visibility() { + local v; v=$(git -C "$(repo_dir "$1")" config --get fuzzyworld.visibility || true) + [ "$v" = public ] && echo public || echo private +} + +# Serialise work on one repo (a push and a manual publish could overlap). +with_lock() { + local name=$1; shift + mkdir -p "$STATE/locks" + exec 9>"$STATE/locks/$name.lock" + flock -w 600 9 || die "timed out waiting for lock on $name" + "$@" +} + +unpublish() { + local name=$1 + valid_name "$name" + rm -rf -- "${SITE:?}/${name:?}" "${SITE:?}/${name:?}.git" + rm -f -- "${STATE:?}/${name:?}.json" +} + +publish_one() { + local name=$1 src pub + src=$(repo_dir "$name"); pub="$SITE/$name.git" + if [ "$(visibility "$name")" != public ]; then + unpublish "$name" + echo "$name: private, not published" + return + fi + # Public clone copy for dumb-HTTP `git clone https://code.fuzzyworld.net/<name>.git`. + # Objects are hardlinked on first copy; later fetches add only new packs. + if [ ! -d "$pub" ]; then + git clone --quiet --mirror --local "$src" "$pub" + else + git -C "$pub" fetch --quiet --prune origin '+refs/*:refs/*' + fi + git -C "$pub" symbolic-ref HEAD "$(git -C "$src" symbolic-ref HEAD)" + git -C "$pub" update-server-info + "${GEN[@]}" repo "$name" +} + +cmd_list() { + local r name + for r in "$GIT_ROOT"/*.git; do + [ -d "$r" ] || continue + name=$(basename "$r" .git) + printf '%-32s %-8s %s\n' "$name" "$(visibility "$name")" \ + "$(git -C "$r" log -1 --format='%cs %s' HEAD 2>/dev/null | cut -c1-60 || echo '(empty)')" + done +} + +cmd_new() { + local name="" vis=private desc="" + while [ $# -gt 0 ]; do + case "$1" in + --public) vis=public ;; + --description) desc=${2:-}; shift ;; + -*) die "unknown option $1" ;; + *) [ -z "$name" ] && name=$1 || die "unexpected argument $1" ;; + esac + shift + done + [ -n "$name" ] || die "usage: fw new <name> [--public] [--description TEXT]" + local r; r=$(repo_dir "$name") + [ -e "$r" ] && die "$name already exists" + git init --quiet --bare --initial-branch=main "$r" + git -C "$r" config fuzzyworld.visibility "$vis" + git -C "$r" config core.hooksPath "$HOOKS" + [ -n "$desc" ] && git -C "$r" config fuzzyworld.description "$desc" + echo "created $name ($vis)" + echo " push to: dh:git/$name.git" + [ "$vis" = public ] && echo " browse: https://code.fuzzyworld.net/$name/ (after the first push)" + return 0 +} + +cmd_set_visibility() { + local vis=$1 name=${2:-} + [ -n "$name" ] || die "usage: fw $vis <name>" + need_repo "$name" + git -C "$(repo_dir "$name")" config fuzzyworld.visibility "$vis" + with_lock "$name" publish_one "$name" + "${GEN[@]}" index +} + +cmd_describe() { + local name=${1:-}; shift || true + [ -n "$name" ] && [ $# -gt 0 ] || die "usage: fw describe <name> TEXT" + need_repo "$name" + git -C "$(repo_dir "$name")" config fuzzyworld.description "$*" + printf '%s\n' "$*" > "$(repo_dir "$name")/description" + with_lock "$name" publish_one "$name" + "${GEN[@]}" index +} + +cmd_publish() { + local name=${1:-} + [ -n "$name" ] || die "usage: fw publish <name>" + need_repo "$name" + with_lock "$name" publish_one "$name" + "${GEN[@]}" index +} + +cmd_publish_all() { + local r name + for r in "$GIT_ROOT"/*.git; do + name=$(basename "$r" .git) + ( with_lock "$name" publish_one "$name" ) || echo "fw: publish failed for $name" >&2 + done + "${GEN[@]}" index +} + +cmd_install_hooks() { + local r + for r in "$GIT_ROOT"/*.git; do git -C "$r" config core.hooksPath "$HOOKS"; done + echo "hooks installed on $(ls -d "$GIT_ROOT"/*.git | wc -l) repos" +} + +mkdir -p "$STATE" "$LOGS" +case "${1:-}" in + list) cmd_list ;; + new) shift; cmd_new "$@" ;; + public|private) cmd_set_visibility "$1" "${2:-}" ;; + describe) shift; cmd_describe "$@" ;; + publish) cmd_publish "${2:-}" ;; + publish-all) cmd_publish_all ;; + install-hooks) cmd_install_hooks ;; + ""|-h|--help|help) sed -n '2,15p' "$0" | sed 's/^# \{0,1\}//' ;; + *) die "unknown command: $1 (try: fw help)" ;; +esac diff --git a/server/gen.py b/server/gen.py new file mode 100755 index 0000000..7e06d66 --- /dev/null +++ b/server/gen.py @@ -0,0 +1,517 @@ +#!/usr/bin/env python3 +"""Static browse pages for code.fuzzyworld.net. + + gen.py repo <name> build or update the pages of one public repo + gen.py index rebuild the site index (public repos only) + +Only the git CLI and the standard library are required. Python-Markdown is used +for README/.md rendering when importable (from ~/fw/pylib); otherwise those files +are shown as plain text. + +Pages are incremental: a manifest per repo (~/fw/state/<name>.json) records the +blob SHA behind every file page, so a push rewrites only what changed. +""" + +import html +import json +import os +import posixpath +import re +import subprocess +import sys +import time +import urllib.parse + +HOME = os.path.expanduser("~") +GIT_ROOT = os.environ.get("FW_GIT_ROOT", os.path.join(HOME, "git")) +SITE = os.environ.get("FW_CODE_SITE", os.path.join(HOME, "code.fuzzyworld.net")) +APPS = os.environ.get("FW_APPS_SITE", os.path.join(HOME, "apps.fuzzyworld.net")) +STATE = os.environ.get("FW_STATE", os.path.join(HOME, "fw", "state")) +PYLIB = os.environ.get("FW_PYLIB", os.path.join(HOME, "fw", "pylib")) +BASE_URL = "https://code.fuzzyworld.net" +APPS_URL = "https://apps.fuzzyworld.net" + +# Bump when page templates change, so every repo gets a full rebuild once. +GEN_VERSION = 4 + +LOG_LIMIT = 100 +MAX_TEXT_BYTES = 1024 * 1024 +MAX_DIFF_BYTES = 300 * 1024 +MAX_RAW_BYTES = 2 * 1024 * 1024 +# Raw copies are written only for these: other extensions (.php, .shtml, ...) could be +# executed by the web server, and the browse pages already show text files. +# SVG is safe here because the site's CSP header (script-src 'none') covers SVG documents too. +RAW_EXTENSIONS = {".png", ".jpg", ".jpeg", ".gif", ".webp", ".ico", ".avif", ".bmp", ".svg"} +README_NAMES = ["readme.md", "readme.markdown", "readme", "readme.txt", "readme.rst"] + +sys.path.insert(0, PYLIB) +try: + import markdown # type: ignore +except ImportError: # pragma: no cover + markdown = None + + +# ---------------------------------------------------------------- git helpers + +def git(repo, *args, check=True): + r = subprocess.run(["git", "-C", repo, *args], capture_output=True, check=False) + if check and r.returncode != 0: + raise RuntimeError(f"git {' '.join(args)}: {r.stderr.decode(errors='replace').strip()}") + return r.stdout + + +def git_text(repo, *args, check=True): + return git(repo, *args, check=check).decode("utf-8", errors="replace") + + +class BlobReader: + """Streams blob contents through one `git cat-file --batch` process.""" + + def __init__(self, repo): + self.p = subprocess.Popen(["git", "-C", repo, "cat-file", "--batch"], + stdin=subprocess.PIPE, stdout=subprocess.PIPE) + + def read(self, sha): + self.p.stdin.write(sha.encode() + b"\n") + self.p.stdin.flush() + header = self.p.stdout.readline().split() + if len(header) < 3 or header[1] == b"missing": + return b"" + data = self.p.stdout.read(int(header[2])) + self.p.stdout.read(1) # trailing newline + return data + + def close(self): + self.p.stdin.close() + self.p.wait() + + +def repo_config(repo, key, default=""): + v = git_text(repo, "config", "--get", key, check=False).strip() + return v or default + + +def is_public(repo): + # Anything but the exact value "public" counts as private. + return repo_config(repo, "fuzzyworld.visibility") == "public" + + +# ---------------------------------------------------------------- html helpers + +esc = html.escape + + +def url(*parts): + """Site-absolute URL from path parts, each part quoted per segment.""" + path = "/".join(p.strip("/") for p in parts if p) + return "/" + "/".join(urllib.parse.quote(s) for s in path.split("/")) + + +def page(title, body, crumbs=None, repo=None): + nav = "" + if repo: + nav = (f'<nav class="repo-nav"><a href="{url(repo)}/">Summary</a>' + f'<a href="{url(repo, "tree")}/">Files</a>' + f'<a href="{url(repo, "log")}/">Log</a>' + f'<a href="{url(repo, "refs")}/">Branches & tags</a></nav>') + crumb_html = "" + if crumbs: + crumb_html = '<p class="crumbs">' + " / ".join(crumbs) + "</p>" + return f"""<!doctype html> +<html lang="en"> +<head> +<meta charset="utf-8"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<title>{esc(title)} · code.fuzzyworld.net</title> +<link rel="stylesheet" href="/style.css"> +</head> +<body> +<header class="site"><a class="home" href="/">code.fuzzyworld.net</a></header> +<main> +{nav} +{crumb_html} +{body} +</main> +<footer class="site"><a href="/">All repositories</a> · <a href="{APPS_URL}/">Apps</a> · <a href="/why/">Why this site exists</a></footer> +</body> +</html> +""" + + +def write(path, content): + os.makedirs(os.path.dirname(path), exist_ok=True) + tmp = path + ".tmp" + mode = "wb" if isinstance(content, bytes) else "w" + with open(tmp, mode, **({} if mode == "wb" else {"encoding": "utf-8"})) as f: + f.write(content) + os.replace(tmp, path) + + +def remove(path): + try: + os.remove(path) + except FileNotFoundError: + pass + + +def human_size(n): + for unit in ("B", "KB", "MB", "GB"): + if n < 1024 or unit == "GB": + return f"{n:.0f} {unit}" if unit == "B" else f"{n:.1f} {unit}" + n /= 1024 + + +def is_binary(data): + return b"\0" in data[:8000] + + +def safe_path(p): + parts = p.split("/") + return all(part not in ("", ".", "..") for part in parts) + + +# ---------------------------------------------------------------- markdown + +class LinkContext: + """Rewrites relative links in a repo's markdown to this site's pages.""" + + def __init__(self, name, base_dir, files, dirs, raw): + self.name, self.base_dir, self.files, self.dirs, self.raw = name, base_dir, files, dirs, raw + + def rewrite(self, href, image): + parsed = urllib.parse.urlsplit(href) + if parsed.scheme or parsed.netloc or href.startswith(("#", "/", "mailto:")): + return href + target = posixpath.normpath(posixpath.join(self.base_dir, urllib.parse.unquote(parsed.path))) + frag = ("#" + parsed.fragment) if parsed.fragment else "" + if image and target in self.raw: + return url(self.name, "raw", target) + if target in self.files: + return url(self.name, "blob", target) + ".html" + frag + if target in self.dirs or target == ".": + return url(self.name, "tree", "" if target == "." else target) + "/" + frag + return href + + +_ATTR = re.compile(r'(<(?:img|source)\b[^>]*?\bsrc|<a\b[^>]*?\bhref)=(["\'])(.*?)\2', re.IGNORECASE | re.DOTALL) + + +def render_markdown(text, ctx): + if not markdown: + return f"<pre>{esc(text)}</pre>" + out = markdown.Markdown(extensions=["tables", "fenced_code", "sane_lists"], output_format="html").convert(text) + # Rewrite links in the finished HTML, so raw <img>/<a> tags in the markdown are covered too. + def fix(m): + is_img = not m.group(1).lower().startswith("<a") + new = ctx.rewrite(html.unescape(m.group(3)), image=is_img) + return f"{m.group(1)}={m.group(2)}{esc(new, quote=True)}{m.group(2)}" + return _ATTR.sub(fix, out) + + +# ---------------------------------------------------------------- repo pages + +def tree_entries(repo): + """Default-branch files as {path: (mode, sha, size)}; submodules get size -1.""" + out = {} + data = git(repo, "ls-tree", "-r", "-l", "-z", "HEAD") + for rec in data.split(b"\0"): + if not rec: + continue + meta, path = rec.split(b"\t", 1) + mode, typ, sha, size = meta.decode().split() + p = path.decode("utf-8", errors="replace") + if not safe_path(p): + continue + out[p] = (mode, sha, -1 if typ == "commit" else int(size)) + return out + + +def all_dirs(files): + dirs = set() + for p in files: + d = posixpath.dirname(p) + while d: + dirs.add(d) + d = posixpath.dirname(d) + return dirs + + +def crumbs_for(name, path, last_is_file): + crumbs = [f'<a href="{url(name)}/">{esc(name)}</a>', f'<a href="{url(name, "tree")}/">files</a>'] + parts = path.split("/") if path else [] + for i, part in enumerate(parts): + sub = "/".join(parts[: i + 1]) + if i == len(parts) - 1: + crumbs.append(f"<strong>{esc(part)}</strong>") + else: + crumbs.append(f'<a href="{url(name, "tree", sub)}/">{esc(part)}</a>') + return crumbs + + +def build_tree_page(out, name, d, files, dirs): + prefix = d + "/" if d else "" + subdirs = sorted({p[len(prefix):].split("/")[0] for p in dirs if p.startswith(prefix) and p != d and "/" not in p[len(prefix):]}) + here = sorted(p for p in files if posixpath.dirname(p) == d) + rows = [] + if d: + parent = posixpath.dirname(d) + rows.append(f'<tr><td><a href="{url(name, "tree", parent)}/">..</a></td><td></td></tr>') + for s in subdirs: + rows.append(f'<tr><td class="dir"><a href="{url(name, "tree", prefix + s)}/">{esc(s)}/</a></td><td></td></tr>') + for p in here: + size = files[p][2] + label = "submodule" if size < 0 else human_size(size) + rows.append(f'<tr><td><a href="{url(name, "blob", p)}.html">{esc(posixpath.basename(p))}</a></td><td class="num">{label}</td></tr>') + body = f'<h1>{esc(d or name)}</h1>\n<table class="files">{"".join(rows)}</table>' + write(os.path.join(out, "tree", d, "index.html"), + page(f"{name}/{d}" if d else f"{name} files", body, crumbs_for(name, d, False) if d else None, repo=name)) + + +def build_blob_page(out, name, path, entry, blobs, ctx, raw_set): + mode, sha, size = entry + title = f"{name}/{path}" + crumbs = crumbs_for(name, path, True) + if size < 0: + body = f"<h1>{esc(posixpath.basename(path))}</h1><p>Git submodule at commit <code>{esc(sha)}</code>.</p>" + else: + data = blobs.read(sha) + raw_link = f' · <a href="{url(name, "raw", path)}">raw</a>' if path in raw_set else "" + meta = f'<p class="meta">{human_size(size)}{raw_link}</p>' + ext = posixpath.splitext(path)[1].lower() + if path in raw_set: + content = f'<p><img class="blob" src="{url(name, "raw", path)}" alt="{esc(posixpath.basename(path))}"></p>' + elif is_binary(data): + content = "<p>Binary file, not shown. Clone the repository to get it.</p>" + elif size > MAX_TEXT_BYTES: + content = "<p>File too large to show here. Clone the repository to get it.</p>" + else: + text = data.decode("utf-8", errors="replace") + lines = text.split("\n") + if lines and lines[-1] == "": + lines.pop() + numbered = "".join(f'<tr id="L{i}"><td class="ln"><a href="#L{i}">{i}</a></td><td class="code">{esc(l) or " "}</td></tr>' + for i, l in enumerate(lines, 1)) + source = f'<table class="source">{numbered}</table>' + if ext in (".md", ".markdown") and markdown: + ctx.base_dir = posixpath.dirname(path) + content = (f'<article class="markdown">{render_markdown(text, ctx)}</article>' + f"<details><summary>Source</summary>{source}</details>") + else: + content = source + body = f"<h1>{esc(posixpath.basename(path))}</h1>{meta}{content}" + write(os.path.join(out, "blob", path + ".html"), page(title, body, crumbs, repo=name)) + + +def commit_list(repo, limit): + fmt = "%H%x1f%h%x1f%an%x1f%ad%x1f%s" + raw = git_text(repo, "log", f"-n{limit}", "--date=short", f"--format={fmt}", "HEAD", check=False) + return [line.split("\x1f") for line in raw.splitlines() if line.count("\x1f") == 4] + + +def commits_table(name, commits): + rows = "".join(f'<tr><td class="date">{esc(d)}</td><td><a href="{url(name, "commit", h)}.html"><code>{esc(s)}</code></a></td>' + f"<td>{esc(subj)}</td><td class=\"author\">{esc(a)}</td></tr>" for h, s, a, d, subj in commits) + return f'<table class="log">{rows}</table>' + + +def build_commit_page(repo, out, name, sha): + path = os.path.join(out, "commit", sha + ".html") + if os.path.exists(path): + return # commits never change + head = git_text(repo, "show", "-s", "--format=%H%n%an <%ae>%n%ad%n%P%n%B", "--date=iso", sha) + full, author, date, parents, *msg = head.split("\n") + author = author.split(" <")[0] # don't publish email addresses + diff = git(repo, "show", "--format=", "--stat", "--patch", "-M", "--diff-merges=first-parent", sha) + truncated = len(diff) > MAX_DIFF_BYTES + text = diff[:MAX_DIFF_BYTES].decode("utf-8", errors="replace") + lines = [] + for l in text.split("\n"): + cls = "" + if l.startswith("+") and not l.startswith("+++"): + cls = "add" + elif l.startswith("-") and not l.startswith("---"): + cls = "del" + elif l.startswith("@@"): + cls = "hunk" + elif l.startswith("diff --git"): + cls = "file" + lines.append(f'<span class="{cls}">{esc(l)}</span>' if cls else esc(l)) + parent_links = " ".join(f'<a href="{url(name, "commit", p)}.html"><code>{p[:12]}</code></a>' for p in parents.split()) + body = (f"<h1>{esc(msg[0] if msg else full)}</h1>" + f'<p class="meta"><code>{full}</code> · {esc(author)} · {esc(date)}' + f'{" · parents " + parent_links if parent_links else ""}</p>' + f'<pre class="message">{esc(chr(10).join(msg).strip())}</pre>' + f'<pre class="diff">{chr(10).join(lines)}</pre>' + + ("<p>Diff truncated. Clone the repository to see all of it.</p>" if truncated else "")) + write(path, page(f"{name} {full[:12]}", body, [f'<a href="{url(name)}/">{esc(name)}</a>', "commit"], repo=name)) + + +def refs_rows(repo, pattern, datefield): + fmt = f"%(refname:short)%1f%({datefield}:short)%1f%(subject)" + raw = git_text(repo, "for-each-ref", f"--sort=-{datefield}", f"--format={fmt}", pattern, check=False) + return [l.split("\x1f") for l in raw.splitlines() if l.count("\x1f") == 2] + + +def build_refs_page(repo, out, name, default_branch): + branches = refs_rows(repo, "refs/heads", "committerdate") + tags = refs_rows(repo, "refs/tags", "creatordate") + def table(rows, mark_default): + if not rows: + return "<p>None.</p>" + return '<table class="log">' + "".join( + f'<tr><td><code>{esc(n)}</code>{" <em>(default)</em>" if mark_default and n == default_branch else ""}</td>' + f'<td class="date">{esc(d)}</td><td>{esc(s)}</td></tr>' for n, d, s in rows) + "</table>" + body = (f"<h1>Branches</h1>{table(branches, True)}<h1>Tags</h1>{table(tags, False)}" + f"<p>Only the default branch is browsable here. Clone the repository for the others.</p>") + write(os.path.join(out, "refs", "index.html"), page(f"{name} branches & tags", body, repo=name)) + + +def app_url(name): + return f"{APPS_URL}/{name}/" if os.path.exists(os.path.join(APPS, name, "index.html")) else None + + +def build_summary(repo, out, name, desc, files, ctx, commits, empty): + clone = f"git clone {BASE_URL}/{name}.git" + app = app_url(name) + links = f'<p class="meta">App: <a href="{app}">{esc(app)}</a></p>' if app else "" + readme_html = "" + if not empty: + lower = {p.lower(): p for p in files if "/" not in p} + for cand in README_NAMES: + if cand in lower: + p = lower[cand] + blobs = BlobReader(repo) + text = blobs.read(files[p][1]).decode("utf-8", errors="replace") + blobs.close() + ctx.base_dir = "" + rendered = render_markdown(text, ctx) if p.lower().endswith((".md", ".markdown")) else f"<pre>{esc(text)}</pre>" + readme_html = f'<article class="markdown readme">{rendered}</article>' + break + recent = commits_table(name, commits[:5]) if commits else "" + body = (f"<h1>{esc(name)}</h1>" + + (f'<p class="desc">{esc(desc)}</p>' if desc else "") + + links + + f'<pre class="clone">{esc(clone)}</pre>' + + ("<p>This repository is empty.</p>" if empty else f"<h2>Recent commits</h2>{recent}<p><a href=\"{url(name, 'log')}/\">Full log</a></p>") + + readme_html) + write(os.path.join(out, "index.html"), page(name, body, repo=None if empty else name)) + + +def build_repo(name): + repo = os.path.join(GIT_ROOT, name + ".git") + if not os.path.isdir(repo): + raise SystemExit(f"no such repo: {name}") + if not is_public(repo): + raise SystemExit(f"{name} is not public; refusing to publish") + out = os.path.join(SITE, name) + os.makedirs(STATE, exist_ok=True) + state_path = os.path.join(STATE, name + ".json") + try: + with open(state_path) as f: + state = json.load(f) + except (FileNotFoundError, ValueError): + state = {} + if state.get("version") != GEN_VERSION: + state = {} + + desc = repo_config(repo, "fuzzyworld.description") + empty = not git(repo, "rev-parse", "-q", "--verify", "HEAD", check=False).strip() + files = {} if empty else tree_entries(repo) + dirs = all_dirs(files) + raw_set = {p for p, (m, s, size) in files.items() + if 0 <= size <= MAX_RAW_BYTES and posixpath.splitext(p)[1].lower() in RAW_EXTENSIONS} + ctx = LinkContext(name, "", set(files), dirs, raw_set) + commits = [] if empty else commit_list(repo, LOG_LIMIT) + + old_files = state.get("files", {}) + old_dirs = set(state.get("dirs", [])) + old_commits = set(state.get("commits", [])) + head_changed = state.get("head") != (commits[0][0] if commits else None) + + # Remove pages for files, directories and commits that are gone. + for p in set(old_files) - set(files): + remove(os.path.join(out, "blob", p + ".html")) + remove(os.path.join(out, "raw", p)) + for d in old_dirs - dirs: + remove(os.path.join(out, "tree", d, "index.html")) + for h in old_commits - {c[0] for c in commits}: + remove(os.path.join(out, "commit", h + ".html")) + + # File pages and raw copies: only where the blob changed. Markdown pages are + # rebuilt whenever HEAD moves, because their links depend on the whole tree. + blobs = BlobReader(repo) + written = 0 + for p, entry in files.items(): + is_md = p.lower().endswith((".md", ".markdown")) + if old_files.get(p) == entry[1] and not (is_md and head_changed): + continue + try: + build_blob_page(out, name, p, entry, blobs, ctx, raw_set) + if p in raw_set: + write(os.path.join(out, "raw", p), blobs.read(entry[1])) + else: + remove(os.path.join(out, "raw", p)) + written += 1 + except OSError as e: # e.g. a path component too long for the filesystem + print(f"skip {p}: {e}", file=sys.stderr) + blobs.close() + + if not empty: + build_tree_page(out, name, "", files, dirs) + for d in dirs: + build_tree_page(out, name, d, files, dirs) + for c in commits: + build_commit_page(repo, out, name, c[0]) + default_branch = git_text(repo, "symbolic-ref", "--short", "HEAD", check=False).strip() + build_refs_page(repo, out, name, default_branch) + write(os.path.join(out, "log", "index.html"), + page(f"{name} log", f"<h1>Last {len(commits)} commits</h1>{commits_table(name, commits)}", repo=name)) + build_summary(repo, out, name, desc, files, ctx, commits, empty) + + state = {"version": GEN_VERSION, "head": commits[0][0] if commits else None, + "files": {p: e[1] for p, e in files.items()}, "dirs": sorted(dirs), + "commits": [c[0] for c in commits], "built": int(time.time())} + write(state_path, json.dumps(state)) + print(f"{name}: {len(files)} files, {written} file pages written, {len(commits)} commits") + + +def build_index(): + rows = [] + for entry in sorted(os.listdir(GIT_ROOT)): + if not entry.endswith(".git"): + continue + repo = os.path.join(GIT_ROOT, entry) + if not is_public(repo): + continue + name = entry[:-4] + if not os.path.exists(os.path.join(SITE, name, "index.html")): + continue # not published yet + last = git_text(repo, "log", "-1", "--format=%cs", "HEAD", check=False).strip() + rows.append((last, name, repo_config(repo, "fuzzyworld.description"), app_url(name))) + rows.sort(key=lambda r: r[0], reverse=True) + items = [] + for last, n, d, a in rows: + app = f' · <a href="{esc(a)}">app</a>' if a else "" + desc = f'<p class="desc">{esc(d)}</p>' if d else "" + items.append(f'<li><a class="name" href="{url(n)}/">{esc(n)}</a>' + f'<span class="meta"> · updated {esc(last or "never (empty)")}{app}</span>{desc}</li>') + body = (f"<h1>Pavel's code</h1>" + f"<p>Browse, or clone with <code>git clone {BASE_URL}/<repo>.git</code>. No account or sign-in needed. " + f'Small web apps are at <a href="{APPS_URL}/">apps.fuzzyworld.net</a>. ' + f'<a href="/why/">Why this site exists.</a></p>' + f'<ul class="repos">{"".join(items)}</ul>') + write(os.path.join(SITE, "index.html"), page("Repositories", body)) + print(f"index: {len(rows)} public repos") + + +def main(argv): + if len(argv) == 3 and argv[1] == "repo": + build_repo(argv[2]) + elif len(argv) == 2 and argv[1] == "index": + build_index() + else: + raise SystemExit(__doc__) + + +if __name__ == "__main__": + main(sys.argv) diff --git a/server/hooks/post-receive b/server/hooks/post-receive new file mode 100755 index 0000000..0633548 --- /dev/null +++ b/server/hooks/post-receive @@ -0,0 +1,13 @@ +#!/bin/bash +# Shared post-receive hook for every ~/git/<name>.git (via core.hooksPath). +# Publishing runs detached, so the push returns at once; output goes to ~/fw/logs/<name>.log. +name=$(basename "$(cd "${GIT_DIR:-.}" && pwd)" .git) +vis=$(git config --get fuzzyworld.visibility) +log="$HOME/fw/logs/$name.log" +mkdir -p "$HOME/fw/logs" +{ echo "== $(date '+%F %T') push"; cat; } >> "$log" # stdin: "<old> <new> <ref>" lines +unset GIT_DIR GIT_QUARANTINE_PATH +setsid nohup "$HOME/bin/fw" publish "$name" >> "$log" 2>&1 < /dev/null & +if [ "$vis" = public ]; then + echo "fw: updating https://code.fuzzyworld.net/$name/ in the background (log: ~/fw/logs/$name.log)" +fi diff --git a/server/install.sh b/server/install.sh new file mode 100755 index 0000000..833cbdc --- /dev/null +++ b/server/install.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# Install or update the DreamHost side of fuzzyworld-tools. Run from the laptop: +# server/install.sh copy tools, site files and hooks to `dh` +# server/install.sh --publish ...then republish every public repo +set -euo pipefail +cd "$(dirname "$0")/.." +HOST=${FW_HOST:-dh} + +ssh "$HOST" 'mkdir -p ~/fw/tools' +rsync -az --delete --exclude '__pycache__' server/ "$HOST:fw/tools/server/" +ssh "$HOST" bash -s <<'EOF' +set -euo pipefail +chmod +x ~/fw/tools/server/fw ~/fw/tools/server/hooks/* +mkdir -p ~/bin ~/fw/state ~/fw/logs +ln -sfn ~/fw/tools/server/fw ~/bin/fw +site=~/code.fuzzyworld.net +cp ~/fw/tools/server/site/htaccess "$site/.htaccess" +cp ~/fw/tools/server/site/style.css ~/fw/tools/server/site/robots.txt ~/fw/tools/server/site/404.html "$site/" +mkdir -p "$site/why" && cp ~/fw/tools/server/site/why/index.html "$site/why/" +~/bin/fw install-hooks +EOF +if [ "${1:-}" = --publish ]; then ssh "$HOST" '~/bin/fw publish-all'; fi +echo "installed on $HOST" diff --git a/server/site/404.html b/server/site/404.html new file mode 100644 index 0000000..610f289 --- /dev/null +++ b/server/site/404.html @@ -0,0 +1,17 @@ +<!doctype html> +<html lang="en"> +<head> +<meta charset="utf-8"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<title>Not found · code.fuzzyworld.net</title> +<link rel="stylesheet" href="/style.css"> +</head> +<body> +<header class="site"><a class="home" href="/">code.fuzzyworld.net</a></header> +<main> +<h1>Not found</h1> +<p>There's nothing at this address. Private repositories aren't listed here, and only the default branch can be browsed.</p> +<p><a href="/">All repositories</a></p> +</main> +</body> +</html> diff --git a/server/site/htaccess b/server/site/htaccess new file mode 100644 index 0000000..2e6d563 --- /dev/null +++ b/server/site/htaccess @@ -0,0 +1,37 @@ +# code.fuzzyworld.net. Installed as .htaccess by fuzzyworld-tools (server/install.sh). +Options -Indexes -Includes -ExecCGI +DirectoryIndex index.html +ErrorDocument 404 /404.html + +RewriteEngine On +# www.<site> -> <site>, keeping path and query. +RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC] +RewriteRule ^ https://%1%{REQUEST_URI} [R=301,L] + +# Nothing in this tree is ever executed: pages are pre-generated HTML, raw copies +# are images only, and public clone copies are plain git data. +RemoveHandler .php .php3 .php4 .php5 .php7 .php8 .phtml .phar .shtml .cgi .pl .py +RemoveType .php .php3 .php4 .php5 .php7 .php8 .phtml .phar +<FilesMatch "\.(php\d?|phtml|phar|shtml|cgi|pl|py|sh)$"> + Require all denied +</FilesMatch> + +# Public clone copies: serve only what dumb-HTTP git needs. +RedirectMatch 404 ^/[^/]+\.git/(config|description|hooks/|logs/|info/exclude|FETCH_HEAD|ORIG_HEAD) + +<IfModule mod_headers.c> + Header always set Content-Security-Policy "default-src 'self'; img-src 'self' https: data:; style-src 'self' 'unsafe-inline'; script-src 'none'; frame-ancestors 'none'" + Header always set X-Content-Type-Options nosniff + Header always set Referrer-Policy strict-origin-when-cross-origin + # Pages change on every push: browsers must revalidate (cheap 304 when unchanged). + <FilesMatch "\.(html|css)$"> + Header set Cache-Control "no-cache" + </FilesMatch> + # Git objects and packs never change; refs do. + <If "%{REQUEST_URI} =~ m#^/[^/]+\.git/objects/(pack/|[0-9a-f]{2}/)#"> + Header set Cache-Control "public, max-age=31536000, immutable" + </If> + <If "%{REQUEST_URI} =~ m#^/[^/]+\.git/(HEAD|info/refs|objects/info/packs)$#"> + Header set Cache-Control "no-cache" + </If> +</IfModule> diff --git a/server/site/robots.txt b/server/site/robots.txt new file mode 100644 index 0000000..365905e --- /dev/null +++ b/server/site/robots.txt @@ -0,0 +1,4 @@ +# Browse pages are static; crawl freely. Git data is for `git clone`, not for indexing. +User-agent: * +Disallow: /*.git/ +Allow: / diff --git a/server/site/style.css b/server/site/style.css new file mode 100644 index 0000000..aa8b678 --- /dev/null +++ b/server/site/style.css @@ -0,0 +1,58 @@ +:root { + --bg: #ffffff; --fg: #1d1d1f; --muted: #6b6b70; --line: #e4e4e8; --soft: #f5f5f7; + --link: #0a58ca; --add: #e6f4ea; --add-fg: #116329; --del: #fdecea; --del-fg: #a40e26; --hunk: #6f42c1; +} +@media (prefers-color-scheme: dark) { + :root { + --bg: #111113; --fg: #e8e8ea; --muted: #9a9aa2; --line: #2a2a2f; --soft: #1a1a1e; + --link: #79b8ff; --add: #12261a; --add-fg: #7ee2a8; --del: #2d1517; --del-fg: #ff9492; --hunk: #c5a5ff; + } +} +:root { color-scheme: light dark; } +* { box-sizing: border-box; } +html { -webkit-text-size-adjust: 100%; } +body { margin: 0; background: var(--bg); color: var(--fg); font: 16px/1.55 system-ui, -apple-system, "Segoe UI", sans-serif; } +a { color: var(--link); text-decoration: none; } +a:hover { text-decoration: underline; } +main { max-width: 72rem; margin: 0 auto; padding: 0 16px 3rem; } +header.site, footer.site { max-width: 72rem; margin: 0 auto; padding: 1rem 16px; color: var(--muted); } +header.site .home { font-weight: 600; color: var(--fg); } +footer.site { border-top: 1px solid var(--line); font-size: 0.9rem; } +h1 { font-size: 1.6rem; margin: 1rem 0 0.5rem; overflow-wrap: anywhere; } +h2 { font-size: 1.2rem; margin: 1.5rem 0 0.5rem; } +code, pre, .source, .diff { font: 13px/1.5 ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; tab-size: 4; } +pre { background: var(--soft); border: 1px solid var(--line); border-radius: 6px; padding: 0.75rem; overflow-x: auto; } +.repo-nav { display: flex; flex-wrap: wrap; gap: 0.25rem 1rem; padding: 0.5rem 0; border-bottom: 1px solid var(--line); } +.crumbs { color: var(--muted); overflow-wrap: anywhere; } +.meta, .desc { color: var(--muted); overflow-wrap: anywhere; } +pre.message { white-space: pre-wrap; overflow-wrap: anywhere; } +.clone { user-select: all; } +table { border-collapse: collapse; width: 100%; } +td, th { padding: 0.3rem 0.5rem; border-bottom: 1px solid var(--line); text-align: left; vertical-align: top; } +th { color: var(--muted); font-weight: 500; } +td.num, td.date { white-space: nowrap; color: var(--muted); } +td.date { width: 7rem; } +td.author { color: var(--muted); white-space: nowrap; } +.files td.dir a { font-weight: 600; } +.table-wrap, .source, .log, .index { display: block; overflow-x: auto; } +.source { border: 1px solid var(--line); border-radius: 6px; } +.source td { border: 0; padding: 0 0.5rem; white-space: pre; } +.source td.ln { text-align: right; color: var(--muted); user-select: none; width: 1%; } +.source td.ln a { color: inherit; } +.source tr:target { background: var(--soft); } +.diff .add { background: var(--add); color: var(--add-fg); display: inline-block; min-width: 100%; } +.diff .del { background: var(--del); color: var(--del-fg); display: inline-block; min-width: 100%; } +.diff .hunk { color: var(--hunk); } +.diff .file { font-weight: 700; } +.markdown img, img.blob { max-width: 100%; height: auto; } +.markdown table { display: block; overflow-x: auto; } +.readme { border-top: 1px solid var(--line); margin-top: 1.5rem; } +ul.repos { list-style: none; padding: 0; margin: 1rem 0; } +ul.repos li { padding: 0.6rem 0; border-bottom: 1px solid var(--line); } +ul.repos .name { font-weight: 600; overflow-wrap: anywhere; } +ul.repos .meta { font-size: 0.9rem; white-space: nowrap; } +ul.repos .desc { margin: 0.2rem 0 0; } +details summary { cursor: pointer; color: var(--muted); margin: 1rem 0 0.5rem; } +@media (max-width: 40rem) { + td.author, .log td.date { display: none; } +} diff --git a/server/site/why/index.html b/server/site/why/index.html new file mode 100644 index 0000000..2bf00f8 --- /dev/null +++ b/server/site/why/index.html @@ -0,0 +1,25 @@ +<!doctype html> +<html lang="en"> +<head> +<meta charset="utf-8"> +<meta name="viewport" content="width=device-width, initial-scale=1"> +<title>Why this site exists · code.fuzzyworld.net</title> +<link rel="stylesheet" href="/style.css"> +</head> +<body> +<header class="site"><a class="home" href="/">code.fuzzyworld.net</a></header> +<main> +<h1>Why my projects moved off GitLab</h1> +<p>My code and small web apps used to live on GitLab.com. In October 2026, GitLab started limiting anyone who isn't signed in to 60 requests an hour per IP address. That covers browsing pages, not only automated tools. On an office or campus network, everyone shares an IP, so a few people looking around can lock everyone out within minutes. I hit it myself from my work laptop.</p> +<p>I understand why: AI crawlers and coding agents are pushing huge amounts of traffic at code hosts. But the fix GitLab chose asks every reader to make an account and sign in just to read public code, and I don't want to require that of anyone.</p> +<p>So everything now lives on my own domain:</p> +<ul> +<li>code: <a href="https://code.fuzzyworld.net/">https://code.fuzzyworld.net/</a> (browse, or <code>git clone https://code.fuzzyworld.net/<repo>.git</code>)</li> +<li>apps: <a href="https://apps.fuzzyworld.net/">https://apps.fuzzyworld.net/</a></li> +</ul> +<p>No accounts, no sign-in, no limits beyond what a small static site can serve. If something you used is missing or broken, email me.</p> +<p>(Saved data from the old gitlab.io versions of the apps doesn't carry over, since browsers treat it as a different site. If you need yours, tell me.)</p> +</main> +<footer class="site"><a href="/">All repositories</a> · <a href="https://apps.fuzzyworld.net/">Apps</a></footer> +</body> +</html>