fuzzyworld-tools / commit
Usage: per-model trailers, cost per commit, weekly output-token chart (Q107 stage 1)
PLAN.md | 9 + README.md | 22 +- lib/usage.py | 158 ++++++++--- package.json | 3 +- server/gen.py | 290 +++++++++++++++++++-- server/prices.json | 122 +++++++++ server/site/style.css | 38 +++ tests/fixtures/usage/sess1.jsonl | 10 + tests/fixtures/usage/sess1/subagents/agent-x.jsonl | 2 + tests/test_gen.py | 226 ++++++++++++++++ tests/test_usage.py | 103 ++++++++ 11 files changed, 906 insertions(+), 77 deletions(-) diff --git a/PLAN.md b/PLAN.md index 58cd48d..f5d10d7 100644 --- a/PLAN.md +++ b/PLAN.md @@ -210,6 +210,15 @@ he wants; see the tiers in the README, form = T2, email = T1). The form stays in how to price tokens (a per-model price table kept in the repo, stated as an estimate); what a "line" is (added lines in non-generated, non-vendored files); sessions that span several repos; commits without trailers (the ChatGPT era) shown as unknown, never as zero. Data stays derived from git; no transcripts are published. + **Q107 stage 1 (done 2026-11-04):** trailers that make cost computable (`FW-Tokens-Method: 2`, per-model + `FW-Model-Tokens`, written by `fw usage --trailers`; each message id counted once, subagents included); per-commit + estimated cost from `server/prices.json` (list prices, `as_of`; unknown stays unknown); one weekly "output tokens" + SVG chart on the Usage tab with a hatched strip for commits without data. Old method-1 trailers are left alone + (they ran about 1.5-2x high). + Remaining stages: (2) more per-repo charts: cost per week, cached reads per week, cost per commit (weekly average), + AI-marked commits per week (stacked), and an est. cost column in the Sessions table; (3) a site-wide `/usage/` + roll-up in `build_index` over public published repos, with a per-repo table; (4) optional: lines added/removed per + week and usage per added line. - File finder and code search (one shipped script plus a prebuilt index per repo) - code-server on TrueNAS, for editing in the browser (private, over Tailscale) - Provenance stats and transcript publishing (see below) diff --git a/README.md b/README.md index 0acf4d0..a93eaaf 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,7 @@ On the laptop, `bin/fw` runs `deploy` locally and forwards repo commands to Drea |---|---| | `fw deploy [--ref B] [--as NAME] [--here] [--dry-run]` | build and publish this repo's app (above). `--as NAME` publishes it at `apps.fuzzyworld.net/NAME/` instead, a second copy of the same repo (e.g. `explodorama-next` built from `main` while `/explodorama/` is deployed from a tag with `--ref`); the build gets `FW_APP=NAME`, which the vite plugin uses for `base`, and the index title gets the suffix | | `fw repoint [DIR] [--https] [--dry-run]` | for each repo in DIR whose `origin` is on gitlab.com: rename it `gitlab`, add `origin` → `dh:git/<repo>.git` (or read-only `https://code.fuzzyworld.net/<repo>.git` with `--https`, public repos only), fetch, and retarget branch tracking. Changes only remotes; checks that uncommitted work, HEAD and stashes are unchanged | -| `fw usage [--all\|--now]` | Claude Code token use per session from `~/.claude/projects` (runs locally): messages, current context size, cache-read, output (cache-read and output include subagents). `--now` prints one line for the newest session, to decide when to put it down | +| `fw usage [--all\|--now\|--trailers [--session ID]]` | Claude Code token use per session from `~/.claude/projects` (runs locally): messages, current context size, cache-read, output (cache-read and output include subagents). `--now` prints one line for the newest session, to decide when to put it down. `--trailers` prints the commit-message trailers (exact integers, one `FW-Model-Tokens` line per model, `FW-Tokens-Method: 2`) for the newest session here or the one named; it warns on stderr when that transcript is over 10 minutes old | | `fw admin [--port N] [--no-open] [--code-dir DIR]` | a local web page for reports and blog comments, over the commands below (runs locally; see [Admin panel](#admin-panel)) | | `fw comments list\|show\|approve\|reject\|remove\|untrust\|publish\|export\|ingest\|imap-check` | moderate blog comments received by email (see [Blog](#blog)); runs on the server | | `fw issues <repo> public\|reviewed\|private` | set who sees the repo's bug-report issue pages (`git config fuzzyworld.issues`; default private) and republish them | @@ -170,12 +170,20 @@ adds nothing new to read and would duplicate every page, so the branches page li commit. Every branch and tag can be cloned. Author email addresses are not shown. **Usage trailers:** a commit message can carry a Claude Code session's running token tally as git trailers: -`FW-Tokens-In`, `FW-Tokens-Out`, `FW-Tokens-Cached` (exact integers), plus `FW-Session` (the session id, for -provenance only) and `FW-Rtk-Saved`/`FW-Rtk-Cmds` (rtk's lifetime savings, not read by the site). Older commits -have one rounded line instead, `Usage: 10.6K in / 101.3K out / 5.1M cached (session tally)`, which is still read. -A commit's cost is the step from the previous tally (a tally that drops starts a new session). The log shows a -bar of output tokens per commit, highlighted at 2× the median or more; each commit page shows its step and the -session tally; `/<name>/usage/` ranks commits and totals sessions. Steps between old-style lines are only known +`FW-Tokens-In`, `FW-Tokens-Out`, `FW-Tokens-Cached` (exact integers), `FW-Tokens-Method: 2`, one +`FW-Model-Tokens: <model> in=… out=… read=… write5m=… write1h=…` line per model (so a cost can be computed; +requests with a prompt over 100K tokens on a model with a higher tier are filed under `<model>@over100k`; a cache write without a 5m/1h split is counted as 5m), plus +`FW-Session` (the session id, for provenance only) and `FW-Rtk-Saved`/`FW-Rtk-Cmds` (rtk's lifetime savings, not +read by the site). `npm run prepare-usage-for-pr` (same as `fw usage --trailers`) writes them. Commits without `FW-Tokens-Method` are method 1: that jq +count double counted repeated transcript lines (roughly 1.5-2× too high) and missed subagents; they are left as +written, shown with tokens but with the cost unknown. Even older commits have one rounded line, +`Usage: 10.6K in / 101.3K out / 5.1M cached (session tally)`, which is still read. +A commit's cost is the step from the previous tally; a new session starts when a tally (or any model's counter) +falls or the method changes. Dollar figures are estimates: `server/prices.json` holds the list prices per model +(with an `as_of` date) and the page says "estimated API list-price equivalent, not what was paid". The log shows a +bar of output tokens per commit, highlighted at 2× the median or more; each commit page shows its step, estimated +cost and the session tally; `/<name>/usage/` ranks commits, totals sessions and charts output tokens per week +(hatched = commits without usage data, unknown rather than zero). Steps between old-style lines are only known to within the rounding (about two tenths of K/M/G), so small ones show as ≈0; commits without a trailer show no bar. **License:** the summary page names the repo's license, read from a root `LICENSE`/`LICENCE`/`COPYING` file diff --git a/lib/usage.py b/lib/usage.py index d9a7831..20d308b 100644 --- a/lib/usage.py +++ b/lib/usage.py @@ -5,60 +5,134 @@ fw usage --all every project fw usage --now one line about the newest session here: messages, current context size, model share (use it to decide on "put down") + fw usage --trailers [--session ID] + the commit-message trailers for the newest session here (or the + one named): exact integers, plus one FW-Model-Tokens line per model + (in, out, cache read, 5-minute and 1-hour cache writes) so the + site can price a commit. Subagents are included, each message is + counted once. Warns on stderr when the transcript is over 10 + minutes old (it may be the wrong session). FW-Tokens-Method: 2 + marks this counting; older trailers double counted transcript lines. """ -import glob, json, os, re, sys, time +import glob, json, os, re, subprocess, sys, time root = os.path.expanduser('~/.claude/projects') +def _load_over100k(): + """{model: True} for models whose price table has an above_100k block.""" + try: + with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'server', 'prices.json')) as f: + return {k: True for k, v in json.load(f).get('models', {}).items() if 'above_100k' in v} + except (OSError, ValueError): + print('warning: server/prices.json is missing or unreadable; long-prompt (over 100K) requests ' + 'will not be filed separately and may be underpriced later', file=sys.stderr) + return {} + +OVER_100K = _load_over100k() + def scan(path): """Totals include the session's subagent transcripts (<session>/subagents/*.jsonl); - the context size and message count are the main conversation's.""" - msgs, tot, ctx, models = scan_one(path) + the context size and message count are the main conversation's. per_model is + {model: [in, out, read, write5m, write1h]} over main and subagents.""" + msgs, tot, ctx, models, per = scan_one(path) for sub in glob.glob(path[:-6] + '/subagents/*.jsonl'): - _, t, _, _ = scan_one(sub) + _, t, _, _, p = scan_one(sub) tot = [a + b for a, b in zip(tot, t)] - return msgs, tot, ctx, models + for k, v in p.items(): + per[k] = [a + b for a, b in zip(per.get(k, [0] * 5), v)] + return msgs, tot, ctx, models, per def scan_one(path): - seen, msgs, tot, ctx, models = set(), 0, [0, 0, 0, 0], 0, {} - for line in open(path, errors='ignore'): - try: d = json.loads(line) - except ValueError: continue - m = d.get('message') - u = m.get('usage') if isinstance(m, dict) else None - if not u or d.get('type') != 'assistant': continue - k = m.get('id') or d.get('uuid') - if k in seen: continue - seen.add(k) - v = [u.get('input_tokens', 0), u.get('output_tokens', 0), - u.get('cache_read_input_tokens', 0), u.get('cache_creation_input_tokens', 0)] - tot = [a + b for a, b in zip(tot, v)] - ctx = v[0] + v[2] + v[3] - msgs += 1 - models[m.get('model', '?')] = models.get(m.get('model', '?'), 0) + 1 - return msgs, tot, ctx, models + """Each message id counts once: Claude Code writes one line per content block, each repeating the usage. + per_model keys are the model id, or '<model>@over100k' for a request whose prompt (input + cache read + + cache write) is over 100,000 tokens when the model's price table has a higher tier.""" + seen, msgs, tot, ctx, models, per = set(), 0, [0, 0, 0, 0], 0, {}, {} + with open(path, errors='ignore') as fh: + for line in fh: + try: d = json.loads(line) + except ValueError: continue + m = d.get('message') + u = m.get('usage') if isinstance(m, dict) else None + if not u or d.get('type') != 'assistant': continue + k = m.get('id') or d.get('uuid') + if k in seen: continue + seen.add(k) + v = [u.get('input_tokens', 0), u.get('output_tokens', 0), + u.get('cache_read_input_tokens', 0), u.get('cache_creation_input_tokens', 0)] + tot = [a + b for a, b in zip(tot, v)] + ctx = v[0] + v[2] + v[3] + msgs += 1 + model = m.get('model', '?') + models[model] = models.get(model, 0) + 1 + cc = u.get('cache_creation') + if isinstance(cc, dict) and ('ephemeral_5m_input_tokens' in cc or 'ephemeral_1h_input_tokens' in cc): + w5, w1 = cc.get('ephemeral_5m_input_tokens', 0), cc.get('ephemeral_1h_input_tokens', 0) + else: + w5, w1 = v[3], 0 + key = model + '@over100k' if ctx > 100000 and model in OVER_100K else model + per[key] = [a + b for a, b in zip(per.get(key, [0] * 5), [v[0], v[1], v[2], w5, w1])] + return msgs, tot, ctx, models, per + +def trailers(path): + """The commit-message trailer lines for one session transcript.""" + _, _, _, _, per = scan(path) + per = {k: v for k, v in per.items() if k != '<synthetic>' and any(v)} + lines = [f'FW-Tokens-In: {sum(v[0] for v in per.values())}', + f'FW-Tokens-Out: {sum(v[1] for v in per.values())}', + f'FW-Tokens-Cached: {sum(v[2] for v in per.values())}', + 'FW-Tokens-Method: 2'] + for k in sorted(per): + v = per[k] + lines.append(f'FW-Model-Tokens: {k} in={v[0]} out={v[1]} read={v[2]} write5m={v[3]} write1h={v[4]}') + lines.append(f'FW-Session: {os.path.basename(path)[:-6]}') + try: + r = subprocess.run(['rtk', 'gain', '--format', 'json'], capture_output=True, text=True, timeout=30, check=True) + s = json.loads(r.stdout)['summary'] + lines += [f'FW-Rtk-Saved: {s["total_saved"]}', f'FW-Rtk-Cmds: {s["total_commands"]}'] + except (OSError, subprocess.SubprocessError, ValueError, KeyError): + pass + return lines def fmt(n): return f'{n:,}' -args = sys.argv[1:] -proj = os.path.join(root, re.sub(r'[^a-zA-Z0-9]', '-', os.getcwd())) -dirs = sorted(glob.glob(root + '/*')) if '--all' in args else [proj] -files = [f for d in dirs for f in glob.glob(d + '/*.jsonl')] -files.sort(key=os.path.getmtime, reverse=True) -if not files: - sys.exit('no sessions found for ' + proj) +def main(argv): + args = argv + proj = os.path.join(root, re.sub(r'[^a-zA-Z0-9]', '-', os.environ.get('INIT_CWD') or os.getcwd())) + dirs = sorted(glob.glob(root + '/*')) if '--all' in args else [proj] + files = [f for d in dirs for f in glob.glob(d + '/*.jsonl')] + files.sort(key=os.path.getmtime, reverse=True) + if not files: + sys.exit('no sessions found for ' + proj) + + if '--trailers' in args: + f = files[0] + if '--session' in args: + i = args.index('--session') + 1 + if i >= len(args): + sys.exit('usage: usage.py --trailers [--session ID]') + sid = args[i] + f = os.path.join(proj, sid + '.jsonl') + if not os.path.exists(f): + sys.exit('no such session: ' + f) + if time.time() - os.path.getmtime(f) > 600: + print(f'warning: {f} was last written over 10 minutes ago; is it the current session?', file=sys.stderr) + print('\n'.join(trailers(f))) + return + + if '--now' in args: + f = files[0] + msgs, tot, ctx, models, _ = scan(f) + share = ', '.join(f'{k.replace("claude-", "")} {v}' for k, v in sorted(models.items()) if k != '<synthetic>') + print(f'{os.path.basename(f)[:8]}: {msgs} msgs, context now {fmt(ctx)} tokens, ' + f'cache-read so far {fmt(tot[2])}, output {fmt(tot[1])}; models: {share}') + sys.exit() -if '--now' in args: - f = files[0] - msgs, tot, ctx, models = scan(f) - share = ', '.join(f'{k.replace("claude-", "")} {v}' for k, v in sorted(models.items()) if k != '<synthetic>') - print(f'{os.path.basename(f)[:8]}: {msgs} msgs, context now {fmt(ctx)} tokens, ' - f'cache-read so far {fmt(tot[2])}, output {fmt(tot[1])}; models: {share}') - sys.exit() + print(f'{"session":9}{"project":28}{"msgs":>6}{"context":>12}{"cache-read":>16}{"output":>12} last used') + for f in files: + msgs, tot, ctx, _, _ = scan(f) + p = os.path.basename(os.path.dirname(f)).replace('-Users-pavel-', '')[:27] + print(f'{os.path.basename(f)[:8]:9}{p:28}{msgs:>6}{fmt(ctx):>12}{fmt(tot[2]):>16}{fmt(tot[1]):>12} ' + + time.strftime('%Y-%m-%d %H:%M', time.localtime(os.path.getmtime(f)))) -print(f'{"session":9}{"project":28}{"msgs":>6}{"context":>12}{"cache-read":>16}{"output":>12} last used') -for f in files: - msgs, tot, ctx, _ = scan(f) - p = os.path.basename(os.path.dirname(f)).replace('-Users-pavel-', '')[:27] - print(f'{os.path.basename(f)[:8]:9}{p:28}{msgs:>6}{fmt(ctx):>12}{fmt(tot[2]):>16}{fmt(tot[1]):>12} ' - + time.strftime('%Y-%m-%d %H:%M', time.localtime(os.path.getmtime(f)))) +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/package.json b/package.json index 28d204d..604d771 100644 --- a/package.json +++ b/package.json @@ -11,7 +11,8 @@ "list-repos": "bin/fw list", "update-runner": "runner/install.sh", "update-laptop-runner": "runner/install-laptop.sh", - "update-vscode": "vscode-web/install.sh" + "update-vscode": "vscode-web/install.sh", + "prepare-usage-for-pr": "python3 lib/usage.py --trailers" }, "bin": { "fw": "bin/fw" diff --git a/server/gen.py b/server/gen.py index fd8f56f..a314c01 100755 --- a/server/gen.py +++ b/server/gen.py @@ -19,6 +19,7 @@ blob SHA behind every file page, so a push rewrites only what changed. """ import datetime +import hashlib import html import json import os @@ -43,7 +44,17 @@ BASE_URL = "https://code.fuzzyworld.net" APPS_URL = "https://apps.fuzzyworld.net" # Bump when page templates change, so every repo gets a full rebuild once. -GEN_VERSION = 9 +GEN_VERSION = 10 + + +def gen_key(): + """The version recorded in a repo's manifest: GEN_VERSION plus a hash of prices.json, so a rate-card + refresh rebuilds the commit pages (they show estimated costs).""" + try: + with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "prices.json"), "rb") as f: + return f"{GEN_VERSION}-{hashlib.sha1(f.read()).hexdigest()[:8]}" + except OSError: + return f"{GEN_VERSION}-noprices" COMMIT_LIMIT = 2000 # commit pages per repo, across all branches; older commits: clone HISTORY_LIMIT = 200 # commits listed on a file's history page @@ -901,14 +912,22 @@ def file_histories(repo, ref, paths): # FW-Tokens-In: 10638 # FW-Tokens-Out: 101300 # FW-Tokens-Cached: 5110043 +# FW-Tokens-Method: 2 (counting method; absent = 1, which counted duplicate transcript lines and +# missed subagents, so those tallies run roughly 1.5-2x high) +# FW-Model-Tokens: claude-opus-5-5 in=40 out=20311 read=4100233 write5m=0 write1h=88120 (one per model) # FW-Session: <session id> (provenance only: a resumed session may change it while the tally goes on) # Older commits have one rounded line instead: "Usage: 10.6K in / 101.3K out / 5.1M cached (session tally)". # A commit's own cost is the step from the previous commit's tally. The tally falls when a new -# session starts, and that commit's cost is its whole tally. +# session starts, and that commit's cost is its whole tally. Dollar cost needs the per-model lines and +# prices.json; without them the cost is unknown, never zero. USAGE_KEYS = ("FW-Tokens-In", "FW-Tokens-Out", "FW-Tokens-Cached") USAGE_RE = re.compile(r"^Usage:\s*([\d.]+)([KMG]?) in / ([\d.]+)([KMG]?) out / ([\d.]+)([KMG]?) cached", re.M) +MODEL_TOKENS_RE = re.compile(r"^FW-Model-Tokens:\s*(\S+)\s+in=(\d+)\s+out=(\d+)\s+read=(\d+)\s+write5m=(\d+)\s+write1h=(\d+)\s*$", re.M) +AI_COAUTHOR_RE = re.compile(r"^Co-Authored-By:.*(noreply@anthropic\.com|\bClaude\b|Copilot|ChatGPT|openai)", re.M | re.I) USAGE_UNITS = {"": 1, "K": 10**3, "M": 10**6, "G": 10**9} USAGE_HOT = 2 # a commit with this many times the median output is highlighted +OVER_100K = "@over100k" # model-key suffix for requests whose prompt exceeded 100,000 tokens +MAX_WEEKS = 104 # a longer history is charted by month def parse_usage(message): @@ -927,6 +946,62 @@ def parse_usage(message): return [float(m.group(i)) * u for i, u in zip((1, 3, 5), units)], [u / 20 for u in units] +def parse_model_usage(message): + """{model: [in, out, read, write5m, write1h]} from FW-Model-Tokens trailers, or None (older commits).""" + found = MODEL_TOKENS_RE.findall(message) + if not found: + return None + models = {} + for m in found: + cur = models.setdefault(m[0], [0] * 5) + models[m[0]] = [a + int(b) for a, b in zip(cur, m[1:])] + return models + + +def usage_method(message): + """The FW-Tokens-Method trailer as an int; commits without it used method 1.""" + m = re.search(r"^FW-Tokens-Method:\s*(\d+)\s*$", message, re.M) + return int(m.group(1)) if m else 1 + + +_prices = None + + +def load_prices(): + """server/prices.json beside this file (USD per 1M tokens), read once; {} if missing or broken.""" + global _prices + if _prices is None: + try: + with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "prices.json")) as f: + _prices = json.load(f) + except (OSError, ValueError): + _prices = {} + return _prices + + +def commit_cost(model_delta, prices): + """(dollars, unknown_models): the list-price estimate of a per-model delta, or (None, [...]) when it is + unknown: no model data, or a model with tokens that the price table lacks. Keys ending '@over100k' + use that model's above_100k rates.""" + if not model_delta: + return None, [] + models = prices.get("models", {}) + total, unknown = 0.0, [] + for key, (inp, out, read, w5, w1) in model_delta.items(): + if not any((inp, out, read, w5, w1)): + continue + base = key[:-len(OVER_100K)] if key.endswith(OVER_100K) else key + rates = models.get(base, {}) + if key.endswith(OVER_100K): + rates = rates.get("above_100k") + if not rates: + unknown.append(key) + continue + total += (inp * rates["input"] + out * rates["output"] + read * rates["cache_read"] + + w5 * rates["cache_write_5m"] + w1 * rates["cache_write_1h"]) / 1e6 + return (None, unknown) if unknown else (total, []) + + def fmt_tokens(n): """10638 -> '10.6K': the units the trailer is written in.""" for unit, suffix in ((10**9, "G"), (10**6, "M"), (10**3, "K")): @@ -940,46 +1015,185 @@ def approx_tokens(n, slack): def usage_costs(repo, ref): - """{sha: {"row", "tally", "delta", "slack", "session", "new"}} for each commit on `ref`'s first-parent - chain that has a Usage trailer, oldest first. `row` is the commit as commit_list returns it.""" + """(costs, total, commits) for `ref`'s first-parent chain: see compute_costs.""" raw = git_text(repo, "log", "--first-parent", "--reverse", "--date=short", f"--format={COMMIT_FORMAT}%x1f%B%x1e", ref, check=False) - costs, prev, session, total = {}, None, 0, 0 + parsed = [] for chunk in raw.split("\x1e"): fields = chunk.strip("\n").split("\x1f", 5) - if len(fields) != 6: - continue - total += 1 - parsed = parse_usage(fields[5]) + if len(fields) == 6: + parsed.append((fields[:5], fields[5])) + return compute_costs(parsed) + + +def models_restarted(prev, cur): + """True when per-model tallies cannot continue one session: a counter fell, a model with tokens vanished + (counters never disappear within a session), or model data appears where the previous commit had none.""" + if prev is None: + return cur is not None + if cur is None: + return False + for m, v in prev.items(): + now = cur.get(m) + if now is None: + if any(v): + return True + elif any(n < o for n, o in zip(now, v)): + return True + return False + + +def compute_costs(commits, prices=None): + """commits: [(fields, message)] oldest first, fields = COMMIT_FORMAT's five values (sha, short, author, + date, subject). Returns (costs, total, allc): + costs {sha: {"row", "tally", "delta", "slack", "session", "new", "method", "models", "cost", "unknown"}} + for each commit with a usage trailer. `models` is the per-model delta (None for old commits), `cost` + its list-price estimate in dollars (None = unknown), `unknown` the models that could not be priced. + total the number of commits + allc [{"sha", "date", "ai", "has_tokens"}] for every commit, so trailer-less ones can be bucketed. + A new session starts (delta = whole tally) at the first trailer commit, when the counting method + changes, when the output or cached tally falls, when any per-model counter falls or a model with tokens + vanishes, or when model data appears after commits that had none.""" + prices = load_prices() if prices is None else prices + costs, allc, prev, session = {}, [], None, 0 + for fields, message in commits: + parsed = parse_usage(message) + allc.append({"sha": fields[0], "date": fields[3], "has_tokens": bool(parsed), + "ai": bool(parsed) or bool(AI_COAUTHOR_RE.search(message))}) if not parsed: continue tally, slack = parsed - new = prev is None or tally[1] < prev[0][1] or tally[2] < prev[0][2] + method, tmodels = usage_method(message), parse_model_usage(message) + new = (prev is None or method != prev["method"] or tally[1] < prev["tally"][1] or tally[2] < prev["tally"][2] + or models_restarted(prev["models"], tmodels)) if new: session += 1 delta, delta_slack = tally, slack + mdelta = tmodels else: - delta = [t - p for t, p in zip(tally, prev[0])] - delta_slack = [s + p for s, p in zip(slack, prev[1])] - costs[fields[0]] = {"row": fields[:5], "tally": tally, "delta": delta, "slack": delta_slack, - "session": session, "new": new} - prev = (tally, slack) - return costs, total + delta = [t - p for t, p in zip(tally, prev["tally"])] + delta_slack = [s + p for s, p in zip(slack, prev["slack"])] + mdelta = ({m: [a - b for a, b in zip(v, (prev["models"] or {}).get(m, [0] * 5))] for m, v in tmodels.items()} + if tmodels is not None else None) + cost, unknown = commit_cost(mdelta, prices) + costs[fields[0]] = {"row": fields, "tally": tally, "delta": delta, "slack": delta_slack, + "session": session, "new": new, "method": method, "models": mdelta, + "cost": cost, "unknown": unknown} + prev = {"tally": tally, "slack": slack, "method": method, "models": tmodels} + return costs, len(allc), allc + + +def week_buckets(allc, costs): + """Commits grouped by Monday-started week of their author date, empty weeks included; by month + when the history is longer than MAX_WEEKS weeks. Per bucket: label, commits, ai, with_tokens, + with_cost, out, read, cost (sum over commits whose cost is known), nodata (commits without usage data).""" + if not allc: + return [] + dates = [datetime.date.fromisoformat(c["date"]) for c in allc] + first, last = min(dates), max(dates) + monday = lambda d: d - datetime.timedelta(days=d.weekday()) + by_month = (monday(last) - monday(first)).days // 7 + 1 > MAX_WEEKS + if by_month: + key = lambda d: (d.year, d.month) + keys, y, m = [], first.year, first.month + while (y, m) <= key(last): + keys.append((y, m)) + y, m = (y + 1, 1) if m == 12 else (y, m + 1) + labels = [f"month of {y}-{m:02d}" for y, m in keys] + else: + key = monday + keys = [monday(first) + datetime.timedelta(weeks=i) for i in range((monday(last) - monday(first)).days // 7 + 1)] + labels = [f"week of {k.isoformat()}" for k in keys] + buckets = [{"label": lb, "commits": 0, "ai": 0, "with_tokens": 0, "with_cost": 0, "out": 0, "read": 0, + "cost": 0.0, "nodata": 0} for lb in labels] + index = {k: i for i, k in enumerate(keys)} + for c, d in zip(allc, dates): + b = buckets[index[key(d)]] + b["commits"] += 1 + b["ai"] += c["ai"] + e = costs.get(c["sha"]) + if not e: + b["nodata"] += 1 + continue + b["with_tokens"] += 1 + b["out"] += e["delta"][1] + b["read"] += e["delta"][2] + if e["cost"] is not None: + b["with_cost"] += 1 + b["cost"] += e["cost"] + return buckets + + +def bucket_unit(buckets): + return "month" if buckets and buckets[0]["label"].startswith("month") else "week" + + +def bar_chart_svg(labels, values, *, fmt, unknown, highlight=None, stacked=None, aria, axis_fmt=fmt_tokens, pattern_id="hatch"): + """An inline SVG bar chart without script. labels: one text per bucket, ending in its date (the axis shows + the first and last); values: bar heights; fmt(value) -> text for the tooltip; unknown: per bucket + (commits lacking data, commits), drawn as a hatched strip under the bar since unknown is not zero; + highlight: bucket indexes drawn with the hot colour; stacked: optional per-bucket part of the value + drawn in a second colour at the bottom of the bar; axis_fmt formats the top axis label; pattern_id keeps the hatch pattern's id unique when a page has several charts. Colours come from CSS classes only.""" + left, right, top, base = 44, 716, 12, 140 + n = len(values) + vmax = max(values, default=0) or 1 + step = (right - left) / max(n, 1) + bw = max(step - 1, 1) + parts = [] + for i, v in enumerate(values): + x = left + i * step + bad, total = unknown[i] + tip = f"{labels[i]}: {fmt(v)}" + (f", {total - bad} of {total} commits with data" if total else ", no commits") + h = (base - top) * v / vmax + cls = "bar hot" if highlight and i in highlight else "bar" + inner = f"<title>{esc(tip)}</title>" + parts.append(f'<rect class="hit" x="{x:.1f}" y="{top}" width="{bw:.1f}" height="{base - top}">{inner}</rect>') + parts.append(f'<rect class="{cls}" x="{x:.1f}" y="{base - h:.1f}" width="{bw:.1f}" height="{h:.1f}">{inner}</rect>') + if stacked is not None and stacked[i]: + h2 = (base - top) * min(stacked[i], v) / vmax + parts.append(f'<rect class="bar2" x="{x:.1f}" y="{base - h2:.1f}" width="{bw:.1f}" height="{h2:.1f}">{inner}</rect>') + if bad: + parts.append(f'<rect class="unk" x="{x:.1f}" y="{base + 4}" width="{bw:.1f}" height="6" fill="url(#{pattern_id})">{inner}</rect>') + first, last = labels[0].rsplit(" ", 1)[-1], labels[-1].rsplit(" ", 1)[-1] + return (f'<svg class="chart" viewBox="0 0 720 200" role="img" aria-label="{esc(aria)}">' + f'<defs><pattern id="{pattern_id}" width="6" height="6" patternUnits="userSpaceOnUse">' + '<path class="hatchline" d="M0 6L6 0"/></pattern></defs>' + f'<line class="axis" x1="{left}" y1="{base}" x2="{right}" y2="{base}"/>' + + "".join(parts) + + f'<text x="{left - 4}" y="{base}" text-anchor="end">0</text>' + f'<text x="{left - 4}" y="{top + 8}" text-anchor="end">{esc(axis_fmt(vmax))}</text>' + f'<text x="{left}" y="{base + 28}" text-anchor="start">{esc(first)}</text>' + f'<text x="{right}" y="{base + 28}" text-anchor="end">{esc(last)}</text>' + f'<text x="{left}" y="194" text-anchor="start">hatched = commits without usage data (unknown, not zero)</text>' + '</svg>') _usage_cache = {} def repo_usage(name): - """{"costs", "commits", "median", "max"} for a repo's default branch (output tokens per commit).""" + """{"costs", "commits", "all", "median", "max", "dollars", "priced"} for a repo's default branch: + output tokens per commit, and the summed list-price estimate over the `priced` commits whose cost is known.""" if name not in _usage_cache: - costs, total = usage_costs(os.path.join(GIT_ROOT, name + ".git"), "HEAD") + costs, total, allc = usage_costs(os.path.join(GIT_ROOT, name + ".git"), "HEAD") outs = sorted(c["delta"][1] for c in costs.values()) - _usage_cache[name] = {"costs": costs, "commits": total, "median": outs[len(outs) // 2] if outs else 0, - "max": outs[-1] if outs else 0} + known = [c["cost"] for c in costs.values() if c["cost"] is not None] + _usage_cache[name] = {"costs": costs, "commits": total, "all": allc, + "median": outs[len(outs) // 2] if outs else 0, "max": outs[-1] if outs else 0, + "dollars": sum(known), "priced": len(known)} return _usage_cache[name] +def cost_text(c): + """The estimated dollar cost with its full label, or why the cost is unknown.""" + if c["cost"] is not None: + return (f"est. ${c['cost']:.2f} (API list-price equivalent at {load_prices().get('as_of', '?')} rates, " + "not what was paid)") + if c["unknown"]: + return f"cost unknown (no price for {', '.join(c['unknown'])})" + return "cost unknown (no per-model trailer)" + + def usage_cell(usage, sha): c = usage["costs"].get(sha) if not c: @@ -990,6 +1204,7 @@ def usage_cell(usage, sha): hot = real and median and out >= USAGE_HOT * median title = (f"{approx_tokens(out, slack)} output tokens, {approx_tokens(c['delta'][2], c['slack'][2])} cached reads" + (f" ({out / median:.1f}× the median commit)" if real and median else "") + + f" · {cost_text(c)}" + (" · first commit of a session: counts everything since it began" if c["new"] else "")) width = 100 * out / usage["max"] if usage["max"] else 0 return (f'<td class="cost" title="{esc(title)}"><span class="track"><span class="bar{" hot" if hot else ""}" ' @@ -1002,9 +1217,11 @@ def usage_panel(name, sha): if not c: return "" d, s, t = c["delta"], c["slack"], c["tally"] + cost = f" · {esc(cost_text(c))}" return ('<p class="meta usage">Claude Code usage: this commit ' f'{approx_tokens(d[1], s[1])} out / {approx_tokens(d[2], s[2])} cached' + (" (first commit of its session: everything since it began)" if c["new"] else "") + + cost + f' · session {c["session"]} tally {fmt_tokens(t[0])} in / {fmt_tokens(t[1])} out / {fmt_tokens(t[2])} cached' f' · <a href="{url(name, "usage")}/">usage by commit and session</a></p>') @@ -1044,20 +1261,39 @@ def build_usage_page(view, commit_set): f'<td class="num">{100 * sum(c["delta"][1] for c in cs) / total_out:.0f}%</td></tr>' for n, cs in sessions.items()) ranked = sorted(costs.values(), key=lambda c: -c["delta"][1]) - body = (f'<h1>Claude Code usage</h1><p class="meta">From the <code>Usage:</code> trailers in commit messages · ' + buckets = week_buckets(usage["all"], costs) + unit = bucket_unit(buckets) + chart = bar_chart_svg( + [b["label"] for b in buckets], [b["out"] for b in buckets], + fmt=lambda v: f"{fmt_tokens(v)} output tokens", unknown=[(b["nodata"], b["commits"]) for b in buckets], + aria=f"Output tokens per {unit}, from commits with usage trailers; hatched marks commits without usage data") + as_of = load_prices().get("as_of", "?") + cost_line = (f' · est. ${usage["dollars"]:,.2f} over the {usage["priced"]} commits with a known cost ' + f'(estimated API list-price equivalent at prices of {esc(as_of)}, not what was paid)' + if usage["priced"] else " · cost unknown (no commit has per-model trailers)") + body = (f'<h1>Claude Code usage</h1><p class="meta">From the <code>FW-Tokens-*</code> (or older <code>Usage:</code>) ' + f'trailers in commit messages · ' f'{len(costs)} of {usage["commits"]} commits have one · {len(sessions)} sessions · ' - f'~{fmt_tokens(total_out)} output tokens, ~{fmt_tokens(total_cached)} cached reads</p>' + f'~{fmt_tokens(total_out)} output tokens, ~{fmt_tokens(total_cached)} cached reads{cost_line}</p>' + f'<h2>Output tokens per {unit}</h2>{chart}' f'<h2>Commits by output tokens</h2>' f'{commits_table(name, [c["row"] for c in ranked[:30]], commit_set, usage)}' f'<h2>Sessions</h2><table class="log"><tr><th>#</th><th>Dates</th><th>Commits</th><th>Output</th>' f'<th>Cached</th><th>Share of output</th></tr>{session_rows}</table>' '<h2>How this is measured</h2><ul>' '<li>A trailer is the running token tally of one Claude Code session. A commit\'s cost is the step ' - 'from the previous commit\'s tally. A tally that drops starts a new session, and its first commit ' - 'gets the whole tally.</li>' + 'from the previous commit\'s tally. A new session starts, and its first commit gets the whole tally, ' + 'at the first trailer, when the counting method changes, when the output or cached total falls, ' + 'when any model\'s counter falls, when a model that had tokens disappears, or when per-model ' + 'lines appear after commits that had none.</li>' '<li>Bars and rankings use output tokens. Cached reads grow with how long the context is and how many ' 'turns ran, so they track session length more than the work; input tokens are negligible. These are ' - 'tokens, not dollars: the price depends on the model.</li>' + 'tokens; dollar figures are estimated API list-price equivalents, not what was paid.</li>' + '<li>Costs need the per-model trailers (<code>FW-Model-Tokens</code>) and are estimates at list prices ' + f'as of {esc(as_of)}. Commits without them show cost unknown, and unknown is not zero: it is left out of ' + 'the cost total and drawn hatched in the chart.</li>' + '<li>Trailers without <code>FW-Tokens-Method: 2</code> counted duplicate transcript lines ' + '(roughly 1.5-2× too high) and missed subagents; they are shown as written.</li>' '<li>Tallies are rounded to a tenth of K, M or G, so a step is only known to within about two ' 'such tenths. Steps at or under that show as ≈0.</li>' '<li>A commit with no trailer shows no bar, and the work before it is counted in the next commit ' @@ -1360,7 +1596,7 @@ def build_repo(name): state = json.load(f) except (FileNotFoundError, ValueError): state = {} - stale = state.get("version") != GEN_VERSION + stale = state.get("version") != gen_key() if stale: state = {} @@ -1425,7 +1661,7 @@ def build_repo(name): build_ci_pages(default_view) build_summary(repo, default_view, desc, d.get("_files", {}), d.get("_ctx"), d.get("_commits", []), empty, commit_set) - state = {"version": GEN_VERSION, "views": new_views, "commits": commit_shas, + state = {"version": gen_key(), "views": new_views, "commits": commit_shas, "aliases": sorted(aliases), "built": int(time.time())} write(state_path, json.dumps(state)) print(f"{name}: {len(views)} branches, {len(commit_shas)} commits") diff --git a/server/prices.json b/server/prices.json new file mode 100644 index 0000000..aaf1e6b --- /dev/null +++ b/server/prices.json @@ -0,0 +1,122 @@ +{ + "as_of": "2026-10-22", + "source": "Anthropic rate card PDF, prices effective 2026-10-22 (Claude API, Standard tier, Global scope, USD per 1M tokens)", + "notes": [ + "Claude Opus 4.5 and Haiku 4.5 list context window up to 200K only.", + "Costs computed from these are estimates: refresh from the current rate card when it changes." + ], + "models": { + "claude-fable-5-1": { + "input": 10, + "output": 50, + "cache_write_5m": 12.5, + "cache_write_1h": 20, + "cache_read": 0.25 + }, + "claude-mythos-5-1": { + "input": 10, + "output": 50, + "cache_write_5m": 12.5, + "cache_write_1h": 20, + "cache_read": 0.25 + }, + "claude-fable-5": { + "input": 10, + "output": 50, + "cache_write_5m": 12.5, + "cache_write_1h": 20, + "cache_read": 1.0 + }, + "claude-mythos-5": { + "input": 10, + "output": 50, + "cache_write_5m": 12.5, + "cache_write_1h": 20, + "cache_read": 1.0 + }, + "claude-opus-5-5": { + "input": 4, + "output": 20, + "cache_write_5m": 5, + "cache_write_1h": 8, + "cache_read": 0.2 + }, + "claude-opus-5": { + "input": 5, + "output": 25, + "cache_write_5m": 6.25, + "cache_write_1h": 10, + "cache_read": 0.5 + }, + "claude-opus-4-8": { + "input": 5, + "output": 25, + "cache_write_5m": 6.25, + "cache_write_1h": 10, + "cache_read": 0.5 + }, + "claude-opus-4-7": { + "input": 5, + "output": 25, + "cache_write_5m": 6.25, + "cache_write_1h": 10, + "cache_read": 0.5 + }, + "claude-opus-4-6": { + "input": 5, + "output": 25, + "cache_write_5m": 6.25, + "cache_write_1h": 10, + "cache_read": 0.5 + }, + "claude-opus-4-5": { + "input": 5, + "output": 25, + "cache_write_5m": 6.25, + "cache_write_1h": 10, + "cache_read": 0.5 + }, + "claude-sonnet-5-5": { + "input": 2, + "output": 10, + "cache_write_5m": 2.5, + "cache_write_1h": 4, + "cache_read": 0.1 + }, + "claude-sonnet-5": { + "input": 2, + "output": 10, + "cache_write_5m": 2.5, + "cache_write_1h": 4, + "cache_read": 0.2 + }, + "claude-sonnet-4-6": { + "input": 3, + "output": 15, + "cache_write_5m": 3.75, + "cache_write_1h": 6, + "cache_read": 0.3 + }, + "claude-haiku-5-5": { + "input": 0.1, + "output": 0.5, + "cache_write_5m": 0.125, + "cache_write_1h": 0.2, + "cache_read": 0.01, + "above_100k": { + "input": 0.5, + "output": 2.5, + "cache_write_5m": 0.625, + "cache_write_1h": 1.0, + "cache_read": 0.05 + } + }, + "claude-haiku-4-5": { + "input": 1, + "output": 5, + "cache_write_5m": 1.25, + "cache_write_1h": 2, + "cache_read": 0.1 + } + } +} diff --git a/server/site/style.css b/server/site/style.css index d28556c..0483f72 100644 --- a/server/site/style.css +++ b/server/site/style.css @@ -375,6 +375,44 @@ td.cost .bar.hot { background: var(--site-hunk); } +/* Inline SVG bar charts (the usage page): colours only here, so they follow dark mode */ +svg.chart { + display: block; + width: 100%; + height: auto; + margin: var(--spacing-md) 0; +} + +.chart .bar { + fill: var(--site-muted); +} + +.chart .bar.hot { + fill: var(--site-hunk); +} + +.chart .bar2 { + fill: var(--site-link); +} + +.chart .hit { + fill: transparent; +} + +.chart .axis { + stroke: var(--site-line); +} + +.chart .hatchline { + stroke: var(--site-muted); + stroke-width: 1; +} + +.chart text { + fill: var(--site-muted); + font-size: 16px; +} + /* File listing table */ .files td.dir a { font-weight: var(--font-weight-semibold); diff --git a/tests/fixtures/usage/sess1.jsonl b/tests/fixtures/usage/sess1.jsonl new file mode 100644 index 0000000..cdeb3b6 --- /dev/null +++ b/tests/fixtures/usage/sess1.jsonl @@ -0,0 +1,10 @@ +{"type": "assistant", "uuid": "u1a", "message": {"id": "m1", "model": "claude-opus-5-5", "content": [{"type": "thinking"}], "usage": {"input_tokens": 10, "output_tokens": 100, "cache_read_input_tokens": 1000, "cache_creation_input_tokens": 2000, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 2000}}}} +{"type": "assistant", "uuid": "u1b", "message": {"id": "m1", "model": "claude-opus-5-5", "content": [{"type": "text"}], "usage": {"input_tokens": 10, "output_tokens": 100, "cache_read_input_tokens": 1000, "cache_creation_input_tokens": 2000, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 2000}}}} +{"type": "assistant", "uuid": "u2a", "message": {"id": "m2", "model": "claude-sonnet-5-5", "content": [{"type": "text"}], "usage": {"input_tokens": 5, "output_tokens": 50, "cache_read_input_tokens": 500, "cache_creation_input_tokens": 300}}} +{"type": "assistant", "uuid": "u2b", "message": {"id": "m2", "model": "claude-sonnet-5-5", "content": [{"type": "text"}], "usage": {"input_tokens": 5, "output_tokens": 50, "cache_read_input_tokens": 500, "cache_creation_input_tokens": 300}}} +{"type": "assistant", "uuid": "u2c", "message": {"id": "m2", "model": "claude-sonnet-5-5", "content": [{"type": "text"}], "usage": {"input_tokens": 5, "output_tokens": 50, "cache_read_input_tokens": 500, "cache_creation_input_tokens": 300}}} +{"type": "assistant", "uuid": "u3", "message": {"id": "m3", "model": "claude-sonnet-5-5", "content": [{"type": "text"}], "usage": {"input_tokens": 1, "output_tokens": 10, "cache_read_input_tokens": 100, "cache_creation_input_tokens": 600, "cache_creation": {"ephemeral_5m_input_tokens": 200, "ephemeral_1h_input_tokens": 400}}}} +{"type": "assistant", "uuid": "u4", "message": {"id": "m4", "model": "claude-haiku-5-5", "content": [{"type": "text"}], "usage": {"input_tokens": 50, "output_tokens": 20, "cache_read_input_tokens": 99000, "cache_creation_input_tokens": 2000, "cache_creation": {"ephemeral_5m_input_tokens": 2000, "ephemeral_1h_input_tokens": 0}}}} +{"type": "assistant", "uuid": "u5", "message": {"id": "m5", "model": "claude-haiku-5-5", "content": [{"type": "text"}], "usage": {"input_tokens": 7, "output_tokens": 3, "cache_read_input_tokens": 70, "cache_creation_input_tokens": 0, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}}}} +{"type": "assistant", "uuid": "u6", "message": {"id": "m6", "model": "<synthetic>", "content": [{"type": "text"}], "usage": {"input_tokens": 0, "output_tokens": 0, "cache_read_input_tokens": 0, "cache_creation_input_tokens": 0, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}}}} +{"type": "user", "message": {"role": "user", "content": "hi"}} diff --git a/tests/fixtures/usage/sess1/subagents/agent-x.jsonl b/tests/fixtures/usage/sess1/subagents/agent-x.jsonl new file mode 100644 index 0000000..aa143b1 --- /dev/null +++ b/tests/fixtures/usage/sess1/subagents/agent-x.jsonl @@ -0,0 +1,2 @@ +{"type": "assistant", "uuid": "v1", "message": {"id": "s1", "model": "claude-opus-5-5", "content": [{"type": "text"}], "usage": {"input_tokens": 3, "output_tokens": 30, "cache_read_input_tokens": 300, "cache_creation_input_tokens": 50, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 50}}}} +{"type": "assistant", "uuid": "v1b", "message": {"id": "s1", "model": "claude-opus-5-5", "content": [{"type": "text"}], "usage": {"input_tokens": 3, "output_tokens": 30, "cache_read_input_tokens": 300, "cache_creation_input_tokens": 50, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 50}}}} diff --git a/tests/test_gen.py b/tests/test_gen.py index 5ffdd8d..0d4044c 100644 --- a/tests/test_gen.py +++ b/tests/test_gen.py @@ -2,6 +2,7 @@ import os import sys import unittest +from unittest import mock sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "server")) import gen # noqa: E402 @@ -28,5 +29,230 @@ class ReloadTimer(unittest.TestCase): self.assertEqual(gen.reload_timer(0), "") +PRICES = {"as_of": "2026-10-22", "models": { + "m-big": {"input": 10, "output": 50, "cache_write_5m": 12, "cache_write_1h": 20, "cache_read": 1}, + "m-small": {"input": 1, "output": 5, "cache_write_5m": 1, "cache_write_1h": 2, "cache_read": 0.1, + "above_100k": {"input": 2, "output": 10, "cache_write_5m": 2, "cache_write_1h": 4, "cache_read": 0.2}}}} + + +def commit(sha, date, message, author="a"): + return ([sha, sha[:3], author, date, "subject"], message) + + +def trailers(i, o, c, models=None, method=None): + t = f"x\n\nFW-Tokens-In: {i}\nFW-Tokens-Out: {o}\nFW-Tokens-Cached: {c}\n" + if method: + t += f"FW-Tokens-Method: {method}\n" + for m, v in (models or {}).items(): + t += f"FW-Model-Tokens: {m} in={v[0]} out={v[1]} read={v[2]} write5m={v[3]} write1h={v[4]}\n" + return t + + +class UsageCosts(unittest.TestCase): + def test_parse_model_usage(self): + msg = trailers(1, 2, 3, {"m-big": [1, 2, 3, 4, 5], "m-small": [0, 0, 0, 0, 0]}, 2) + self.assertEqual(gen.parse_model_usage(msg), {"m-big": [1, 2, 3, 4, 5], "m-small": [0, 0, 0, 0, 0]}) + old = "x\n\nFW-Tokens-In: 1\nFW-Tokens-Out: 2\nFW-Tokens-Cached: 3\n" + self.assertIsNone(gen.parse_model_usage(old)) + self.assertIsNone(gen.parse_model_usage("Usage: 1.0K in / 2.0K out / 3.0K cached (session tally)")) + self.assertEqual(gen.parse_usage(old), ([1.0, 2.0, 3.0], [0, 0, 0])) # unchanged + self.assertEqual(gen.usage_method(old), 1) + self.assertEqual(gen.usage_method(msg), 2) + + def test_commit_cost_arithmetic(self): + cost, unknown = gen.commit_cost({"m-big": [1_000_000, 100_000, 2_000_000, 10_000, 5_000]}, PRICES) + self.assertAlmostEqual(cost, 10 + 5 + 2 + 0.12 + 0.1) + self.assertEqual(unknown, []) + cost, _ = gen.commit_cost({"m-small@over100k": [1_000_000, 0, 0, 0, 0], "m-small": [0, 0, 1_000_000, 0, 0]}, PRICES) + self.assertAlmostEqual(cost, 2 + 0.1) + + def test_unknown_models_and_no_data(self): + self.assertEqual(gen.commit_cost(None, PRICES), (None, [])) + self.assertEqual(gen.commit_cost({"m-big": [1, 1, 1, 1, 1], "m-new": [1, 0, 0, 0, 0]}, PRICES), (None, ["m-new"])) + self.assertEqual(gen.commit_cost({"m-new": [0, 0, 0, 0, 0]}, PRICES), (0.0, [])) # zero tokens need no price + + def test_compute_costs_per_model_deltas(self): + costs, total, allc = gen.compute_costs([ + commit("aaa", "2026-10-26", trailers(10, 100, 1000, {"m-big": [10, 100, 1000, 0, 0]}, 2)), + commit("bbb", "2026-10-27", "plain"), + commit("ccc", "2026-10-28", trailers(30, 300, 3000, {"m-big": [20, 250, 2500, 0, 0], "m-small": [10, 50, 500, 0, 0]}, 2)), + ], PRICES) + self.assertEqual(total, 3) + self.assertTrue(costs["aaa"]["new"] and not costs["ccc"]["new"]) + self.assertEqual(costs["ccc"]["models"], {"m-big": [10, 150, 1500, 0, 0], "m-small": [10, 50, 500, 0, 0]}) + self.assertEqual(costs["ccc"]["delta"], [20, 200, 2000]) + self.assertAlmostEqual(costs["ccc"]["cost"], (100 + 7500 + 1500 + 10 + 250 + 50) / 1e6) + self.assertEqual([c["has_tokens"] for c in allc], [True, False, True]) + + def test_method_change_and_fallen_counter_start_new_session(self): + m = {"m-big": [10, 100, 1000, 0, 0]} + costs, _, _ = gen.compute_costs([ + commit("aaa", "2026-10-26", trailers(10, 100, 1000)), # method 1 + commit("bbb", "2026-10-27", trailers(20, 200, 2000, m, 2)), # method 2: new session + commit("ccc", "2026-10-28", trailers(25, 300, 3000, {"m-big": [10, 90, 1200, 0, 0]}, 2)), # model out fell + ], PRICES) + self.assertEqual([costs[k]["new"] for k in ("aaa", "bbb", "ccc")], [True, True, True]) + self.assertEqual([costs[k]["session"] for k in ("aaa", "bbb", "ccc")], [1, 2, 3]) + self.assertEqual(costs["bbb"]["delta"], [20, 200, 2000]) + self.assertIsNone(costs["aaa"]["cost"]) # old commit: tokens shown, cost unknown + + def test_unknown_model_gives_unknown_cost(self): + costs, _, _ = gen.compute_costs( + [commit("aaa", "2026-10-26", trailers(1, 1, 1, {"m-new": [1, 1, 1, 0, 0]}, 2))], PRICES) + self.assertIsNone(costs["aaa"]["cost"]) + self.assertEqual(costs["aaa"]["unknown"], ["m-new"]) + self.assertIn("m-new", gen.cost_text(costs["aaa"])) + + def test_ai_marker(self): + for line in ("Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>", "co-authored-by: X <x@openai.com>", + "Co-authored-by: Copilot <c@github.com>"): + self.assertTrue(gen.AI_COAUTHOR_RE.search("msg\n\n" + line), line) + self.assertFalse(gen.AI_COAUTHOR_RE.search("Co-Authored-By: Bob <bob@example.com>")) + self.assertFalse(gen.AI_COAUTHOR_RE.search("Claude helped with this")) + _, _, allc = gen.compute_costs([commit("aaa", "2026-10-26", "x\n\nCo-Authored-By: Claude <noreply@anthropic.com>"), + commit("bbb", "2026-10-26", "x")], PRICES) + self.assertEqual([c["ai"] for c in allc], [True, False]) + + +class Charts(unittest.TestCase): + def buckets(self): + costs, _, allc = gen.compute_costs([ + commit("aaa", "2026-10-26", trailers(10, 100, 1000, {"m-big": [10, 100, 1000, 0, 0]}, 2)), # Monday + commit("bbb", "2026-10-28", "plain"), + commit("ccc", "2026-11-12", trailers(20, 300, 2000, {"m-big": [20, 300, 2000, 0, 0]}, 2)), + ], PRICES) + return gen.week_buckets(allc, costs) + + def test_week_buckets_keep_empty_weeks_and_flag_unknown(self): + b = self.buckets() + self.assertEqual([x["label"] for x in b], ["week of 2026-10-26", "week of 2026-11-02", "week of 2026-11-09"]) + self.assertEqual([x["commits"] for x in b], [2, 0, 1]) + self.assertEqual([x["nodata"] for x in b], [1, 0, 0]) + self.assertEqual([x["with_tokens"] for x in b], [1, 0, 1]) + self.assertEqual(b[0]["out"], 100) + self.assertEqual(b[0]["with_cost"], 1) + + def test_long_history_buckets_by_month(self): + _, _, allc = gen.compute_costs([commit("aaa", "2020-01-15", "x"), commit("bbb", "2026-10-26", "x")], PRICES) + b = gen.week_buckets(allc, {}) + self.assertEqual(b[0]["label"], "month of 2020-01") + self.assertEqual(len(b), 82) + + def test_bar_chart_svg(self): + b = self.buckets() + svg = gen.bar_chart_svg([x["label"] for x in b], [x["out"] for x in b], + fmt=lambda v: f"{gen.fmt_tokens(v)} output tokens", + unknown=[(x["nodata"], x["commits"]) for x in b], aria="Output per week") + self.assertNotIn("<script", svg) + self.assertIn('viewBox="0 0 720 200"', svg) + self.assertIn("<title>week of 2026-10-26: 100 output tokens, 1 of 2 commits with data</title>", svg) + self.assertIn("<title>week of 2026-11-02: 0 output tokens, no commits</title>", svg) + self.assertEqual(svg.count('class="unk"'), 1) # only the first week has commits without data + self.assertIn("<pattern", svg) + self.assertIn("hatched = commits without usage data (unknown, not zero)", svg) + self.assertIn(">2026-10-26<", svg) + self.assertIn(">2026-11-09<", svg) + + +class SessionBreaks(unittest.TestCase): + def test_vanished_model_starts_a_new_session(self): + costs, _, _ = gen.compute_costs([ + commit("aaa", "2026-10-26", trailers(10, 100, 1000, {"m-big": [10, 100, 1000, 0, 0]}, 2)), + commit("bbb", "2026-10-27", trailers(20, 250, 2000, {"m-small": [20, 150, 2000, 0, 0]}, 2)), + ], PRICES) + self.assertTrue(costs["bbb"]["new"]) + self.assertEqual(costs["bbb"]["models"], {"m-small": [20, 150, 2000, 0, 0]}) + self.assertEqual(costs["bbb"]["delta"], [20, 250, 2000]) + + def test_model_data_after_commits_without_it_starts_a_new_session(self): + costs, _, _ = gen.compute_costs([ + commit("aaa", "2026-10-26", trailers(10, 100, 1000, None, 2)), + commit("bbb", "2026-10-27", trailers(20, 250, 2000, {"m-big": [20, 250, 2000, 0, 0]}, 2)), + ], PRICES) + self.assertTrue(costs["bbb"]["new"]) + self.assertEqual(costs["bbb"]["delta"], [20, 250, 2000]) + self.assertEqual(costs["bbb"]["models"], {"m-big": [20, 250, 2000, 0, 0]}) + + +class UsageRendering(unittest.TestCase): + def setUp(self): + self._cache = dict(gen._usage_cache) + self.addCleanup(lambda: (gen._usage_cache.clear(), gen._usage_cache.update(self._cache))) + + def usage(self, commits, name="r"): + costs, total, allc = gen.compute_costs(commits, PRICES) + known = [c["cost"] for c in costs.values() if c["cost"] is not None] + gen._usage_cache[name] = {"costs": costs, "commits": total, "all": allc, "median": 1, "max": 1000, + "dollars": sum(known), "priced": len(known)} + return gen._usage_cache[name] + + def page(self, commits): + self.usage(commits) + written = {} + view = gen.View("r", "main", True) + with mock.patch.object(gen, "write", lambda path, content: written.update({path: content})), \ + mock.patch.object(gen, "load_prices", return_value=PRICES), \ + mock.patch.object(gen, "ci_pipelines", return_value={}): + gen.build_usage_page(view, None) + return next(iter(written.values())) + + def test_cell_and_panel_label_cost_and_never_show_zero_for_unknown(self): + u = self.usage([commit("a" * 40, "2026-10-26", trailers(1, 1000, 5, {"m-big": [1, 1000, 5, 0, 0]}, 2)), + commit("b" * 40, "2026-10-27", "x\n\nFW-Tokens-In: 2\nFW-Tokens-Out: 2000\nFW-Tokens-Cached: 9\n")]) + with mock.patch.object(gen, "load_prices", return_value=PRICES): + cell = gen.usage_cell(u, "a" * 40) + self.assertIn("est. $0.05", cell) + self.assertIn("not what was paid", cell) + self.assertIn("2026-10-22", cell) + panel = gen.usage_panel("r", "a" * 40) + self.assertIn("not what was paid", panel) + old_cell, old_panel = gen.usage_cell(u, "b" * 40), gen.usage_panel("r", "b" * 40) + for html_ in (old_cell, old_panel): + self.assertIn("cost unknown", html_) + self.assertNotIn("$0", html_) + self.assertNotIn("est. $", html_) + + def test_usage_page_labels_and_hostile_model_name(self): + html_ = self.page([commit("a" * 40, "2026-10-26", trailers(1, 1000, 5, {"m-big": [1, 1000, 5, 0, 0]}, 2)), + commit("b" * 40, "2026-10-27", trailers(2, 2000, 9, {"<b>x</b>": [1, 1, 1, 0, 0]}, 2))]) + self.assertIn("not what was paid", html_) + self.assertIn("Output tokens per week", html_) + self.assertNotIn("<b>x</b>", html_) + self.assertIn("<b>x</b>", html_) # in the tooltip of the unpriced commit + self.assertIn("session starts", html_) + self.assertNotIn("<script", html_) + + def test_usage_page_old_commits_only_has_unknown_cost(self): + html_ = self.page([commit("a" * 40, "2026-10-26", "x\n\nFW-Tokens-In: 2\nFW-Tokens-Out: 2000\nFW-Tokens-Cached: 9\n")]) + self.assertIn("cost unknown", html_) + self.assertNotIn("est. $", html_) + + def test_usage_page_monthly_heading_for_long_history(self): + html_ = self.page([commit("a" * 40, "2020-01-06", trailers(1, 1000, 5, {"m-big": [1, 1000, 5, 0, 0]}, 2)), + commit("b" * 40, "2026-10-27", "plain")]) + self.assertIn("Output tokens per month", html_) + self.assertNotIn("per week", html_) + + def test_week_104_vs_105_boundary(self): + def labels(last): + _, _, allc = gen.compute_costs([commit("a" * 40, "2024-01-01", "x"), commit("b" * 40, last, "x")], PRICES) + return gen.week_buckets(allc, {}) + b104 = labels("2025-12-22") # Monday 103 weeks later: 104 weeks + b105 = labels("2025-12-29") # 105 weeks + self.assertEqual(len(b104), 104) + self.assertTrue(b104[0]["label"].startswith("week of")) + self.assertTrue(b105[0]["label"].startswith("month of")) + self.assertEqual(gen.bucket_unit(b104), "week") + self.assertEqual(gen.bucket_unit(b105), "month") + + def test_chart_hover_target_and_unique_pattern(self): + svg = gen.bar_chart_svg(["week of 2026-10-26", "week of 2026-11-02"], [0, 5], fmt=str, unknown=[(1, 1), (0, 1)], + aria="a", pattern_id="p2") + self.assertIn('id="p2"', svg) + self.assertIn("url(#p2)", svg) + self.assertEqual(svg.count('class="hit"'), 2) + self.assertTrue(svg.index('class="hit"') < svg.index('class="bar"')) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_usage.py b/tests/test_usage.py new file mode 100644 index 0000000..9cf97ab --- /dev/null +++ b/tests/test_usage.py @@ -0,0 +1,103 @@ +"""Tests for lib/usage.py --trailers. Run: python3 -m unittest discover tests""" +import io +import os +import shutil +import subprocess +import sys +import tempfile +import unittest +from unittest import mock + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(HERE, "..", "lib")) +import usage # noqa: E402 + +FIXTURE = os.path.join(HERE, "fixtures", "usage", "sess1.jsonl") + + +class Trailers(unittest.TestCase): + def lines(self): + with mock.patch.object(usage.subprocess, "run", side_effect=FileNotFoundError): # no rtk: its lines are omitted + return usage.trailers(FIXTURE) + + def test_exact_output(self): + # Duplicate lines per message id count once; <synthetic> is skipped; the subagent file is included; + # a cache write without the 5m/1h split counts as 5m; models sorted by name. + self.assertEqual(self.lines(), [ + "FW-Tokens-In: 76", + "FW-Tokens-Out: 213", + "FW-Tokens-Cached: 100970", + "FW-Tokens-Method: 2", + "FW-Model-Tokens: claude-haiku-5-5 in=7 out=3 read=70 write5m=0 write1h=0", + "FW-Model-Tokens: claude-haiku-5-5@over100k in=50 out=20 read=99000 write5m=2000 write1h=0", + "FW-Model-Tokens: claude-opus-5-5 in=13 out=130 read=1300 write5m=0 write1h=2050", + "FW-Model-Tokens: claude-sonnet-5-5 in=6 out=60 read=600 write5m=500 write1h=400", + "FW-Session: sess1", + ]) + + def test_over_100k_prompt_is_filed_separately(self): + _, _, _, _, per = usage.scan(FIXTURE) + self.assertEqual(per["claude-haiku-5-5@over100k"], [50, 20, 99000, 2000, 0]) + self.assertEqual(per["claude-haiku-5-5"], [7, 3, 70, 0, 0]) + self.assertNotIn("<synthetic>", [k for k, v in per.items() if any(v)]) + + def test_rtk_lines_when_rtk_works(self): + out = '{"summary": {"total_saved": 12, "total_commands": 3}}' + done = subprocess.CompletedProcess([], 0, stdout=out, stderr="") + with mock.patch.object(usage.subprocess, "run", return_value=done): + lines = usage.trailers(FIXTURE) + self.assertEqual(lines[-2:], ["FW-Rtk-Saved: 12", "FW-Rtk-Cmds: 3"]) + + def test_cli_names_session_and_warns_about_old_file(self): + with tempfile.TemporaryDirectory() as root: + proj = os.path.join(root, "-p") # the project dir name for cwd /p + os.makedirs(proj) + shutil.copy(FIXTURE, os.path.join(proj, "sess1.jsonl")) + os.utime(os.path.join(proj, "sess1.jsonl"), (0, 0)) + out, err = io.StringIO(), io.StringIO() + with mock.patch.object(usage, "root", root), mock.patch.object(usage.os, "getcwd", return_value="/p"), \ + mock.patch.object(usage, "trailers", return_value=["FW-Tokens-Method: 2"]), \ + mock.patch("sys.stdout", out), mock.patch("sys.stderr", err): + usage.main(["--trailers", "--session", "sess1"]) + self.assertEqual(out.getvalue(), "FW-Tokens-Method: 2\n") + self.assertIn("sess1.jsonl", err.getvalue()) + +class DefaultAndNow(unittest.TestCase): + """The main() refactor must not change the older outputs.""" + def run_main(self, args): + with tempfile.TemporaryDirectory() as root: + proj = os.path.join(root, "-p") + os.makedirs(proj) + shutil.copy(FIXTURE, os.path.join(proj, "sess1.jsonl")) + shutil.copytree(os.path.join(os.path.dirname(FIXTURE), "sess1"), os.path.join(proj, "sess1")) + out = io.StringIO() + with mock.patch.object(usage, "root", root), mock.patch.object(usage.os, "getcwd", return_value="/p"), \ + mock.patch("sys.stdout", out): + try: + usage.main(args) + except SystemExit: + pass + return out.getvalue() + + def test_now_line(self): + self.assertEqual(self.run_main(["--now"]), + "sess1.js: 6 msgs, context now 0 tokens, cache-read so far 100,970, output 213; " + "models: haiku-5-5 2, opus-5-5 1, sonnet-5-5 2\n") + + def test_default_table(self): + lines = self.run_main([]).splitlines() + self.assertTrue(lines[0].startswith("session project")) + self.assertRegex(lines[1], r"^sess1\.js +-p\s+6\s+0\s+100,970\s+213 \d{4}-\d\d-\d\d \d\d:\d\d$") + + def test_session_without_id_is_a_usage_message(self): + out = self.run_main(["--trailers", "--session"]) + self.assertEqual(out, "") # exits with the usage message, no traceback + with mock.patch.object(usage.sys, "exit", side_effect=SystemExit("usage: x")), \ + mock.patch.object(usage.glob, "glob", return_value=[FIXTURE]): + with self.assertRaises(SystemExit) as e: + usage.main(["--trailers", "--session"]) + self.assertIn("usage", str(e.exception)) + + +if __name__ == "__main__": + unittest.main()