fuzzyworld

fuzzyworld-tools / files / README.md

README.md

10.9 KB · history · blame

fuzzyworld-tools

Tooling for Pavel's self-hosted code and apps on DreamHost. It replaces GitLab (code, CI, Pages) and WordPress (the blog) with static files plus a few helper machines.

The one rule: DreamHost only serves files. There is no app server. Pages are generated when you push, and Apache just sends them. Work that needs a real machine (CI, screenshots) runs on a runner (a home TrueNAS box, or the laptop when it is idle), which pulls jobs from DreamHost over ssh. No runner is a normal state: jobs wait.

The roadmap and backlog are in PLAN.md.

What it provides

Site What it is Details
code.fuzzyworld.net Browse pages and read-only git clone for public repos: every unmerged branch, syntax highlighting, history, blame, CI results, bug-report issues, AI-usage charts docs/code-site.md
apps.fuzzyworld.net Deployed web apps, one per path (/bubbles/, /slide/, ...), published with fw deploy, with CI screenshots docs/apps.md
writing.fuzzyworld.net The writing blog: a static site built from Markdown by fw blog, with comments by email docs/blog.md
VS Code in the browser Any public repo, read-only, at apps.fuzzyworld.net/vscode/?repo=<name> docs/operations.md
CI runners Run each push's pipeline (.fw-ci.yml) in Docker, like a GitLab runner docs/ci.md
Admin panel fw admin: a local web page for bug reports, blog comments and deploy state docs/admin.md

Philosophy

Things made with AI should be shared freely with the world. The code and apps here were made with AI, so they are MIT-licensed: use them, change them, share them.

My conversations with the AI are different. My words and my way of working are mine. I publish a conversation only when I choose to.

The same text is on /why-i-left-gitlab/.

Hosting tiers

Each feature needs a certain kind of host. A tier includes the ones above it, so you can pick what your hosting allows.

Tier The host must offer Features
T0, files only (ftp only) serving static files the built blog, Atom feeds, the landing page, the generated code and CI pages, VS Code in the browser
T1, files + jobs (ftp + python) ssh, cron, git hooks, a mailbox reachable by IMAP (or on the same host); nothing runs when a page is requested page generation on push, the CI queue and runner API, cleanup jobs, comments and bug reports by email, the fw commands
T2, files + an endpoint (ftp + php/cgi) a script that answers web requests the comment form (post.php): written, not enabled, not installed
T3, always-on process a long-running server nothing; out of scope

T2 is new attack surface (code that runs on a web request, as the same user as everything else on the account), so comments are accepted by email only. To use the form you would host server/comments/post.php on its own subdomain and set comments_form in the blog config.

Quick start

First create the repo on DreamHost (private unless --public):

fw new my-game --public --description "One line."

Then make it an app (details and every option: docs/apps.md):

// package.json of the app repo
"scripts": { "deploy": "fw deploy" },
"fuzzyworld": { "app": "my-game", "title": "My Game", "description": "One line." },
"devDependencies": { "fuzzyworld-tools": "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0" }
git push dh:git/my-game.git   # `dh` is the SSH alias in ~/.ssh/config
npm run deploy                # clone main, install, check, build, upload, print the URL

Use fw outside an app repo (for example on a second laptop):

npm install -g "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0"

Commands

bin/fw runs some commands on the laptop (deploy, repoint, usage, admin, blog) and forwards the rest to DreamHost (ssh dh ~/bin/fw ...).

Repos

Command What it does
fw list every repo, its visibility and last commit
fw new <name> [--public] [--description TEXT] create ~/git/<name>.git on DreamHost; private unless --public
fw public <name> / fw private <name> publish the source on code.fuzzyworld.net right away / unpublish it (pages, clone URL and index entry are removed)
fw describe <name> TEXT set the description shown on the index and summary page
fw publish <name> refresh one repo's public copy and pages (the push hook does this)
fw publish-all refresh every repo, printing [n/total] name (seconds), then the index and the CI dashboard
fw repoint [DIR] [--https] [--dry-run] for each repo in DIR whose origin is on gitlab.com: rename it gitlab, add origin on DreamHost (or the read-only https URL with --https, public repos only), fetch, retarget branch tracking. Changes only remotes, and checks that uncommitted work, HEAD and stashes are unchanged

Apps and deploys

Command What it does
fw deploy [--ref B] [--as NAME] [--here] [--dry-run] build and publish this repo's app (docs/apps.md)
fw deploys [--json] what is deployed where: the deploy ledger and each deployed app against its repo's default branch. Known gap: bin/fw does not forward it yet (see Known gaps); until then run ssh dh '~/bin/fw deploys'.

CI

Command What it does
fw ci <name> [branch] queue a CI run of a branch's latest commit (default: the default branch)
fw ci-list queued and running CI jobs
fw ci-watch follow CI in the terminal (private repos too): running jobs with each job's state, the queue, the last 5 results; ends when nothing is queued or running

Bug reports and comments

Command What it does
fw issues <repo> public\|reviewed\|private set who sees the repo's bug-report pages (default private) and republish them
fw issues list\|show\|record\|approve\|hide\|close\|reopen\|publish\|publish-all\|ingest\|imap-check moderate bug reports (docs/issues.md)
fw comments list\|show\|approve\|reject\|remove\|untrust\|publish\|export\|ingest\|imap-check moderate blog comments (docs/blog.md)

Laptop-side tools

Command What it does
fw admin [--port N] [--no-open] [--code-dir DIR] the local admin web page (docs/admin.md)
fw blog list\|build\|preview\|deploy <config> the static blog (docs/blog.md)
fw blog watch <config> [--once] [--dry-run], fw blog watch install\|uninstall <config> mirror the blog source into git and auto-deploy safe edits (one machine only)
fw blog import-wp <config> --wp URL, fw blog check <config> one-time WordPress import / compare the built site with the WordPress text
fw usage [--all\|--now\|--trailers [--session ID]] Claude Code token use per session; --trailers prints commit-message trailers (docs/usage.md)

npm run scripts in this repo: dev (preview the pages locally), deploy and deploy-status (deploy this repo's own server by default; -- all or a target name adds the runner and VS Code web; docs/operations.md), list-repos (fw list) and prepare-usage-for-pr (fw usage --trailers).

Visibility

Each bare repo stores its own setting: git config fuzzyworld.visibility public|private. Anything other than exactly public is private, including a missing value, so a repo is never published by accident. A private repo has no pages, no clone URL and no index entry, and pushing to it publishes nothing.

Other settings in the repo config:

Key Meaning
fuzzyworld.issues public, reviewed, or anything else = private: who sees bug-report pages (docs/issues.md)
fuzzyworld.description the one-line description
fuzzyworld.ci off: pushes queue no CI jobs
fuzzyworld.gitlab-id, fuzzyworld.gitlab-path for repos migrated from GitLab

Repository layout

Folder Contents
bin/ fw (laptop side) and deploy-tools
lib/ laptop-side programs: deploy.mjs, admin.py, usage.py, wp_import.py, repoint.sh
server/ everything installed on DreamHost: fw, gen.py (page generator), issues.py, comments.py, blog.py, blog_watch.py, deploys.py, runner-api, hooks, jobs, site templates
runner/ the CI runner (fw-runner.py), its installers and the screenshot script
vite/ the vite plugin for apps (fuzzyworld-tools/vite)
vscode-web/ VS Code in the browser: installer and the fw-fs extension
tests/ the test suite

Where things live on DreamHost: docs/operations.md.

Security in one paragraph

The site never executes anything from a repo: pages are pre-generated HTML with all repo content escaped, only image files are copied raw, scripts are denied by .htaccess, and the code site sends Content-Security-Policy: script-src 'none'. The full list is in docs/code-site.md.

Documentation

File Covers
docs/code-site.md page URLs, branches, licences, local preview, security
docs/apps.md package.json keys, fw deploy, the vite plugin, the apps index, screenshots
docs/ci.md queue, runners, .fw-ci.yml, secrets, watching CI
docs/issues.md bug reports by email: modes, privacy rules, run records, replay links
docs/admin.md fw admin: security model and pages
docs/blog.md fw blog, comments by email, the watcher, the WordPress import
docs/usage.md Claude Code usage trailers, cost estimates, usage charts
docs/operations.md DreamHost layout, scheduled jobs, installing and updating, VS Code in the browser, requirements
PLAN.md roadmap, decisions, backlog

Tests

python3 -m unittest discover tests     # the whole suite: pages, blog, comments, issues, runner, admin, deploys
node --test tests/deploy.test.mjs      # fw deploy's config, rsync filters and replay-template checks

Known gaps

Other problems found are tracked in the global queue, not here.

Source
1# fuzzyworld-tools
2
3Tooling for Pavel's self-hosted code and apps on DreamHost. It replaces GitLab (code, CI, Pages) and WordPress
4(the blog) with static files plus a few helper machines.
5
6**The one rule: DreamHost only serves files.** There is no app server. Pages are generated when you push, and
7Apache just sends them. Work that needs a real machine (CI, screenshots) runs on a *runner* (a home TrueNAS box, or
8the laptop when it is idle), which pulls jobs from DreamHost over ssh. No runner is a normal state: jobs wait.
9
10The roadmap and backlog are in [PLAN.md](PLAN.md).
11
12## What it provides
13
14| Site | What it is | Details |
15|---|---|---|
16| **code.fuzzyworld.net** | Browse pages and read-only `git clone` for public repos: every unmerged branch, syntax highlighting, history, blame, CI results, bug-report issues, AI-usage charts | [docs/code-site.md](docs/code-site.md) |
17| **apps.fuzzyworld.net** | Deployed web apps, one per path (`/bubbles/`, `/slide/`, ...), published with `fw deploy`, with CI screenshots | [docs/apps.md](docs/apps.md) |
18| **writing.fuzzyworld.net** | The writing blog: a static site built from Markdown by `fw blog`, with comments by email | [docs/blog.md](docs/blog.md) |
19| **VS Code in the browser** | Any public repo, read-only, at `apps.fuzzyworld.net/vscode/?repo=<name>` | [docs/operations.md](docs/operations.md#vs-code-in-the-browser) |
20| **CI runners** | Run each push's pipeline (`.fw-ci.yml`) in Docker, like a GitLab runner | [docs/ci.md](docs/ci.md) |
21| **Admin panel** | `fw admin`: a local web page for bug reports, blog comments and deploy state | [docs/admin.md](docs/admin.md) |
22
23## Philosophy
24
25Things made with AI should be shared freely with the world. The code and apps here were made with AI, so they
26are MIT-licensed: use them, change them, share them.
27
28My conversations with the AI are different. My words and my way of working are mine. I publish a conversation
29only when I choose to.
30
31The same text is on [/why-i-left-gitlab/](https://code.fuzzyworld.net/why-i-left-gitlab/).
32
33## Hosting tiers
34
35Each feature needs a certain kind of host. A tier includes the ones above it, so you can pick what your hosting allows.
36
37| Tier | The host must offer | Features |
38|---|---|---|
39| **T0, files only** (ftp only) | serving static files | the built blog, Atom feeds, the landing page, the generated code and CI pages, VS Code in the browser |
40| **T1, files + jobs** (ftp + python) | ssh, cron, git hooks, a mailbox reachable by IMAP (or on the same host); nothing runs when a page is requested | page generation on push, the CI queue and runner API, cleanup jobs, comments and bug reports by email, the `fw` commands |
41| **T2, files + an endpoint** (ftp + php/cgi) | a script that answers web requests | the comment form (`post.php`): written, **not enabled, not installed** |
42| **T3, always-on process** | a long-running server | nothing; out of scope |
43
44T2 is new attack surface (code that runs on a web request, as the same user as everything else on the account),
45so comments are accepted by email only. To use the form you would host `server/comments/post.php` on its own
46subdomain and set `comments_form` in the blog config.
47
48## Quick start
49
50First create the repo on DreamHost (private unless `--public`):
51
52```bash
53fw new my-game --public --description "One line."
54```
55
56Then make it an app (details and every option: [docs/apps.md](docs/apps.md)):
57
58```jsonc
59// package.json of the app repo
60"scripts": { "deploy": "fw deploy" },
61"fuzzyworld": { "app": "my-game", "title": "My Game", "description": "One line." },
62"devDependencies": { "fuzzyworld-tools": "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0" }
63```
64
65```bash
66git push dh:git/my-game.git # `dh` is the SSH alias in ~/.ssh/config
67npm run deploy # clone main, install, check, build, upload, print the URL
68```
69
70Use `fw` outside an app repo (for example on a second laptop):
71
72```bash
73npm install -g "git+https://code.fuzzyworld.net/fuzzyworld-tools.git#semver:^1.0.0"
74```
75
76## Commands
77
78`bin/fw` runs some commands on the laptop (`deploy`, `repoint`, `usage`, `admin`, `blog`) and forwards the rest to
79DreamHost (`ssh dh ~/bin/fw ...`).
80
81### Repos
82
83| Command | What it does |
84|---|---|
85| `fw list` | every repo, its visibility and last commit |
86| `fw new <name> [--public] [--description TEXT]` | create `~/git/<name>.git` on DreamHost; **private unless `--public`** |
87| `fw public <name>` / `fw private <name>` | publish the source on code.fuzzyworld.net right away / unpublish it (pages, clone URL and index entry are removed) |
88| `fw describe <name> TEXT` | set the description shown on the index and summary page |
89| `fw publish <name>` | refresh one repo's public copy and pages (the push hook does this) |
90| `fw publish-all` | refresh every repo, printing `[n/total] name (seconds)`, then the index and the CI dashboard |
91| `fw repoint [DIR] [--https] [--dry-run]` | for each repo in DIR whose `origin` is on gitlab.com: rename it `gitlab`, add `origin` on DreamHost (or the read-only https URL with `--https`, public repos only), fetch, retarget branch tracking. Changes only remotes, and checks that uncommitted work, HEAD and stashes are unchanged |
92
93### Apps and deploys
94
95| Command | What it does |
96|---|---|
97| `fw deploy [--ref B] [--as NAME] [--here] [--dry-run]` | build and publish this repo's app ([docs/apps.md](docs/apps.md)) |
98| `fw deploys [--json]` | what is deployed where: the deploy ledger and each deployed app against its repo's default branch. **Known gap: `bin/fw` does not forward it yet (see [Known gaps](#known-gaps)); until then run `ssh dh '~/bin/fw deploys'`.** |
99
100### CI
101
102| Command | What it does |
103|---|---|
104| `fw ci <name> [branch]` | queue a CI run of a branch's latest commit (default: the default branch) |
105| `fw ci-list` | queued and running CI jobs |
106| `fw ci-watch` | follow CI in the terminal (private repos too): running jobs with each job's state, the queue, the last 5 results; ends when nothing is queued or running |
107
108### Bug reports and comments
109
110| Command | What it does |
111|---|---|
112| `fw issues <repo> public\|reviewed\|private` | set who sees the repo's bug-report pages (default private) and republish them |
113| `fw issues list\|show\|record\|approve\|hide\|close\|reopen\|publish\|publish-all\|ingest\|imap-check` | moderate bug reports ([docs/issues.md](docs/issues.md)) |
114| `fw comments list\|show\|approve\|reject\|remove\|untrust\|publish\|export\|ingest\|imap-check` | moderate blog comments ([docs/blog.md](docs/blog.md#comments-by-email)) |
115
116### Laptop-side tools
117
118| Command | What it does |
119|---|---|
120| `fw admin [--port N] [--no-open] [--code-dir DIR]` | the local admin web page ([docs/admin.md](docs/admin.md)) |
121| `fw blog list\|build\|preview\|deploy <config>` | the static blog ([docs/blog.md](docs/blog.md)) |
122| `fw blog watch <config> [--once] [--dry-run]`, `fw blog watch install\|uninstall <config>` | mirror the blog source into git and auto-deploy safe edits (one machine only) |
123| `fw blog import-wp <config> --wp URL`, `fw blog check <config>` | one-time WordPress import / compare the built site with the WordPress text |
124| `fw usage [--all\|--now\|--trailers [--session ID]]` | Claude Code token use per session; `--trailers` prints commit-message trailers ([docs/usage.md](docs/usage.md)) |
125
126`npm run` scripts in this repo: `dev` (preview the pages locally), `deploy` and `deploy-status` (deploy this repo's own
127server by default; `-- all` or a target name adds the runner and VS Code web; [docs/operations.md](docs/operations.md#install-and-update)), `list-repos`
128(`fw list`) and `prepare-usage-for-pr` (`fw usage --trailers`).
129
130## Visibility
131
132Each bare repo stores its own setting: `git config fuzzyworld.visibility public|private`.
133**Anything other than exactly `public` is private**, including a missing value, so a repo is never published by
134accident. A private repo has no pages, no clone URL and no index entry, and pushing to it publishes nothing.
135
136Other settings in the repo config:
137
138| Key | Meaning |
139|---|---|
140| `fuzzyworld.issues` | `public`, `reviewed`, or anything else = private: who sees bug-report pages ([docs/issues.md](docs/issues.md)) |
141| `fuzzyworld.description` | the one-line description |
142| `fuzzyworld.ci` | `off`: pushes queue no CI jobs |
143| `fuzzyworld.gitlab-id`, `fuzzyworld.gitlab-path` | for repos migrated from GitLab |
144
145## Repository layout
146
147| Folder | Contents |
148|---|---|
149| `bin/` | `fw` (laptop side) and `deploy-tools` |
150| `lib/` | laptop-side programs: `deploy.mjs`, `admin.py`, `usage.py`, `wp_import.py`, `repoint.sh` |
151| `server/` | everything installed on DreamHost: `fw`, `gen.py` (page generator), `issues.py`, `comments.py`, `blog.py`, `blog_watch.py`, `deploys.py`, `runner-api`, hooks, jobs, site templates |
152| `runner/` | the CI runner (`fw-runner.py`), its installers and the screenshot script |
153| `vite/` | the vite plugin for apps (`fuzzyworld-tools/vite`) |
154| `vscode-web/` | VS Code in the browser: installer and the `fw-fs` extension |
155| `tests/` | the test suite |
156
157Where things live on DreamHost: [docs/operations.md](docs/operations.md#layout-on-dreamhost).
158
159## Security in one paragraph
160
161The site never executes anything from a repo: pages are pre-generated HTML with all repo content escaped, only
162image files are copied raw, scripts are denied by `.htaccess`, and the code site sends
163`Content-Security-Policy: script-src 'none'`. The full list is in [docs/code-site.md](docs/code-site.md#security).
164
165## Documentation
166
167| File | Covers |
168|---|---|
169| [docs/code-site.md](docs/code-site.md) | page URLs, branches, licences, local preview, security |
170| [docs/apps.md](docs/apps.md) | `package.json` keys, `fw deploy`, the vite plugin, the apps index, screenshots |
171| [docs/ci.md](docs/ci.md) | queue, runners, `.fw-ci.yml`, secrets, watching CI |
172| [docs/issues.md](docs/issues.md) | bug reports by email: modes, privacy rules, run records, replay links |
173| [docs/admin.md](docs/admin.md) | `fw admin`: security model and pages |
174| [docs/blog.md](docs/blog.md) | `fw blog`, comments by email, the watcher, the WordPress import |
175| [docs/usage.md](docs/usage.md) | Claude Code usage trailers, cost estimates, usage charts |
176| [docs/operations.md](docs/operations.md) | DreamHost layout, scheduled jobs, installing and updating, VS Code in the browser, requirements |
177| [PLAN.md](PLAN.md) | roadmap, decisions, backlog |
178
179## Tests
180
181```bash
182python3 -m unittest discover tests # the whole suite: pages, blog, comments, issues, runner, admin, deploys
183node --test tests/deploy.test.mjs # fw deploy's config, rsync filters and replay-template checks
184```
185
186## Known gaps
187
188- `fw deploys` is documented above but `bin/fw` on the laptop rejects it ("unknown command"). `fw admin` is not affected.
189- The npm package does not ship `server/` (`files` in `package.json`), but `fw blog` (including `import-wp` and `check`) needs it, and `fw usage --trailers` reads `server/prices.json` (without it the trailers lose the `@over100k` split). So these work fully only from a checkout of this repo.
190
191Other problems found are tracked in the global queue, not here.