coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'loops/host' into merge/host-arch

Browse Source 
# Conflicts:
#	lib/erlang/runtime.sx
This commit is contained in:
giles
2026-07-01 17:42:08 +00:00
parent 5968c0173f 62c9bdd270
commit 5535acf4e9
131 changed files with 24871 additions and 5830 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
plans/HANDOFF-enable-serving-jit.md Normal file
View File

@@ -0,0 +1,98 @@
# Hand-off: enable serving-mode JIT for ~3–4× request CPU
> From the **sx-vm-extensions** loop (2026-06-28). The serving-mode JIT is merged
> to `architecture` and is the host's real perf win — it just needs switching on.
> No further engine work is required from your side.
## TL;DR
Run the host server on the merged `architecture` binary with **`SX_SERVING_JIT=1`**
in its environment. Expected: **~3–4× lower per-request CPU** (measured ~9 ms →
~2.7 ms on the `/feed` pipeline). Already verified correct: full host conformance
is **181/181 under `SX_SERVING_JIT=1`**.
## What changed (already merged to architecture)
The bytecode JIT now works in the persistent/epoch serving mode, **opt-in via the
`SX_SERVING_JIT` env var (default OFF)**. Default-off means zero change until you
opt in — nothing regressed for any loop. Merge commit on `architecture`:
`089ed88f` (rebuild the shared binary from architecture to pick it up).
The JIT is safe for the host's request pipeline because:
- The pipeline (dream router + feed/relations/blog handlers + JSON + render-to-html)
is pure SX with **no `call/cc`**; the only continuation-style code is `guard`
(Dream's `dream-catch-with` / `wrap-errors`), which the JIT **auto-detects and
runs interpreted** (recursive `PUSH_HANDLER` scan). So error handling stays
correct; everything else JITs.
- Proven end-to-end: combined host+JIT binary, full conformance under
`SX_SERVING_JIT=1` = **181/181, all 10 suites green** (handler 14, middleware 9,
sxtp 39, router 6, feed 14, relations 22, blog 27, page 8, server 13, ledger 29).
## How to enable
1. Rebuild the shared binary from `architecture` (it carries the merge):
`cd hosts/ocaml && dune build bin/sx_server.exe`
2. Launch the host server process with `SX_SERVING_JIT=1` set in its environment
(whatever wrapper/serve path you use — `lib/host/serve.sx` / the http-listen
entry). Default-off means you must set it explicitly.
3. One-time cost: JIT compiles hot functions on first call (~+1 s at startup /
first requests). Amortized immediately for a long-lived server.
## Measurements (this is the evidence)
In-process, full request pipeline (`host/native-handler (host/make-app …)` →
`/feed`, 2000 requests, in-memory persist backend):
| | per-request CPU | total 2000 reqs |
|---|---|---|
| CEK (default, no JIT) | ~9 ms | ~15–20 s |
| **JIT (`SX_SERVING_JIT=1`)** | **~2.7 ms** | **~5–6 s** |
JIT is also markedly *less* variable run-to-run. The cost is the pipeline
(routing + feed normalize/stream + handler + JSON), not rendering —
`render-to-html` alone is only ~50 µs/render and is already fast.
## What was ruled out (don't chase these)
The original kickoff framed the slowness as "interpreted Smalltalk (`content/html`)
in ~2 s". **The host does not load `lib/smalltalk` or `lib/content`** — that was a
different subsystem. We measured and confirmed:
- The host's render path is `render-to-html` (SX markup → HTML), already fast.
- The proposed big engine projects — **VM continuation-escape** and a
**compile-to-closures Smalltalk interpreter** — would *not* help the host
(wrong subsystem) and are **not needed**. (Scoping kept in the vm-extensions
loop under `plans/vm-continuation-escape.md` / `plans/smalltalk-dispatch-perf.md`
if a Smalltalk-backed workload ever needs them.)
## Caveat — this is CPU only
The ~3–4× is the in-process CPU path (which JIT controls). It does **not** touch
network/IO latency. If your production TTFB is dominated by a non-in-memory
`persist` backend, cross-service fetches, TLS/connection setup, or the known
homepage SSR-stepper issue, profile those separately — JIT won't move them. To
find your real split, break a live TTFB into: request parse → route → handler
(+ persist read) → render → serialize → network. The in-memory measurement above
says the *code path* is ~2.7 ms under JIT; anything beyond that in production is
infrastructure, not the SX engine.
## One known residual (not host-affecting, for awareness)
The serving hook re-runs a JIT'd function on the CEK if it fails mid-execution
(correct result, but could duplicate side effects for an impure function that
fails mid-run). The host conformance is clean (181/181), so nothing triggers it
on your paths today. The clean general fix (propagate-don't-rerun) is deferred in
the vm-extensions loop.
## Correction (host loop, 2026-06-28)
The premise above ("~2s interpreted-Smalltalk render") is STALE: the blog moved
off content-on-sx Smalltalk to `render-to-html` long ago (render-page ~2ms). The
actual post-page unresponsiveness was NOT CPU/render — it was the DURABLE READ
COUNT: host/blog--relation-blocks did ~7 `kv-keys` performs per page (each
host/blog-out/in re-scanned the KV). Collapsing to one shared kv-keys read fixed
it (~1s -> ~0.02s; commit 0a2f1a61). So serving-JIT was NOT the fix here.
Serving-JIT may still be a worthwhile general speedup (the ~3-4× CPU claim, and
the Datalog `instances-of` on /tags is CPU-bound), but it requires running the
host on the merged `architecture` binary — this worktree's binary has no
SX_SERVING_JIT gate. Treat it as an optional future win, not the perf blocker.

108
plans/HANDOFF-jit-miscompile.md Normal file
View File

@@ -0,0 +1,108 @@
# Hand-off: serving-mode JIT miscompiles host handlers (to sx-vm-extensions)
> ## ✅ RESOLVED 2026-06-28 — host now runs 100% serving JIT, no exclude.
>
> Two composing pieces fixed it:
> 1. **sx-vm-extensions `81177d0e`** (`sx_vm.ml` `call_closure_reuse`): when an
> HO-primitive callback (map/filter/reduce/…) suspends on a `perform` AND a
> synchronous resolver is installed, resolve its IO inline and run it to
> completion instead of unwinding the native loop (which dropped iteration
> state and misaligned the stack → the next `CALL_PRIM` got wrong args).
> 2. **host side (`sx_server.ml`)**: that fix only engages when
> `!_cek_io_resolver = Some`. The host serves via the `http-listen` primitive,
> whose handler drove durable IO through `cek_run_with_io` with the resolver
> **= None**, so it hit the unwinding path the fix doesn't cover (the
> vm-extensions repro `repro_jit_resume.ml` *installed* a resolver, so it never
> exercised the host's real path). Fix: extracted `cek_run_with_io`'s IO
> resolution into `resolve_io_request`, and `http-listen` now installs
> `_cek_io_resolver := Some (fun req _ -> resolve_io_request req)` — byte-
> identical resolution, so the inline-resolve path resolves durable reads
> exactly as the CEK loop would.
>
> Verified: host conformance **271/271**; ephemeral durable server at 100% JIT
> (no exclude) — zero fallbacks, real content, related posts shown, picker lists
> 12 candidates; live blog.rose-ash.com home/post/tags 200 with related posts and
> zero error-log lines; relate-picker Playwright **4/4** (infinite-scroll +
> filter + relate, the `drop` path). `serve.sh` exclude dropped.
>
> Everything below is the original hand-off, kept for the record.
---
> From the **host-on-sx** loop, 2026-06-28. We enabled `SX_SERVING_JIT=1` on the
> live host (blog.rose-ash.com) — the Datalog/relations saturation JITs cleanly
> and is the real win (host conformance 271/271 under JIT, 5.4× faster; live
> `/tags` 2.5s → 0.76s). BUT host app handlers MISCOMPILE in the serving path, so
> we had to `(jit-exclude! "host/*" "dream-*" "dr/*")` in serve.sh as a band-aid.
> Please fix the underlying bug so the exclude can be dropped.
## Symptom
Under `SX_SERVING_JIT=1`, the FIRST request to most pages 500s, then self-heals
(retries 200). stderr shows, paired:
```
[jit] host/blog--edges-block first-call fallback to CEK: Sx_types.Eval_error("map: expected (fn list) (in CALL_PRIM \"map\" with 2 args)")
[http-listen] handler error: Sx_types.Eval_error("map: expected (fn list) (in CALL_PRIM \"map\" with 2 args)")
```
Also seen: `Sx_types.Eval_error("rest: 1 list arg")`.
## Two distinct bugs
**(A) codegen / VM-state.** A JIT'd function's bytecode runs `CALL_PRIM "map"`
(and `rest`) with args the primitive rejects (`expected (fn list)`, 2 args
pushed but wrong). KEY CLUE: **host conformance under `SX_SERVING_JIT=1` is
271/271** — the SAME functions (host/blog--edges-block etc.) JIT fine when driven
via the epoch `(eval ...)` path. It ONLY miscompiles in the **http-listen +
cek_run_with_io** serving path. So it is not pure codegen — it's triggered by the
serving/IO context. Strong hypothesis: a `perform`/`VmSuspended` earlier in the
request (the handler does durable kv reads) resumes the VM with a misaligned
stack, so the NEXT `CALL_PRIM` (often a `map`) gets wrong args. The map/rest are
just the first prim call after a resume. Worth a `vm-trace` of a handler that
suspends then maps.
**(B) fallback doesn't recover the failed call.** `register_jit_hook`
(`hosts/ocaml/bin/sx_server.ml` ~L1607-1623): on first-call error it warns, sets
`l.l_compiled <- jit_failed_sentinel`, and returns `None` — intended to fall
through to CEK. But the error still escapes to the http-listen handler (→ 500)
instead of the call being re-run on CEK and returning a value. So even granting
(A), the request shouldn't 500: the fallback should recover THIS call, not just
mark the fn for next time. (Your own notes flagged this as the deferred
"propagate-don't-rerun" shared-CEK change — this is the same thing biting live.)
Fixing EITHER (A) or (B) unblocks the host: (A) removes the miscompile; (B) makes
any miscompile self-heal on the first hit instead of 500ing.
## Repro
1. Build the merged binary (loops/host now carries sx-vm-extensions; the gate +
render-page coexist in sx_server.ml's persistent serving branch).
2. `SX_SERVING_JIT=1 bash lib/host/serve.sh` on a port (durable backend), but
FIRST remove the `(jit-exclude! "host/*" ...)` line from serve.sh so host code
JITs.
3. `curl http://127.0.0.1:PORT/welcome/` → first hit 500 (`map: expected (fn list)`),
retry 200. `curl /` (home, uses map+rest) likewise.
Tooling: `(vm-trace "<sx>")`, `(bytecode-inspect "host/blog--edges-block")`,
`(prim-check "host/blog--edges-block")` (CLAUDE.md "VM/Bytecode Debugging").
## Current mitigation (host side, to remove once fixed)
`lib/host/serve.sh`: when `SX_SERVING_JIT=1`, `(jit-exclude! "host/*" "dream-*"
"dr/*")`. Host app + Dream framework run on CEK (they're IO-bound — no perf loss);
Datalog (`dl-*`/`relations-*`) keeps JITting (the win). Drop this once (A)/(B) land.
## Refined data (100% JIT, no exclude, 2026-06-28)
Host now runs at 100% serving JIT (no jit-exclude). Out of **255 successful JIT
compiles, only ~3 functions miscompile**, all on a multi-arg LIST PRIMITIVE with
wrong CALL_PRIM args, all in the durable-read request path, all failing on the
FIRST list-prim call after a `perform` (kv read):
- `host/blog--edges-block` → `map: expected (fn list) (CALL_PRIM "map" 2 args)`
- a fn using `rest` → `rest: 1 list arg`
- `host/blog-relate-options` → `drop: list and number (CALL_PRIM "drop" 2 args)`
Conformance (epoch eval, no http-listen/perform) is 271/271 under JIT — so it's
NOT the data-first swap alone; the **serving/perform path** is the trigger.
Strongly supports the OP_PERFORM-resume stack-misalignment theory: the prim that
fails is just the first CALL_PRIM after the resume. 252+ other fns JIT clean.

61
plans/NOTE-blog-types-for-radar.md Normal file
View File

@@ -0,0 +1,61 @@
# NOTE → the `loops/radar` migration: the blog TYPE CONTRACT for genesis-import
**From:** the host-on-sx loop (`loops/host`). **Date:** 2026-06-30.
**Re:** `plans/rose-ash-on-sx-migration.md`, slice-01-blog.
## The gap
Your blog slice migrates posts as **untyped** `{slug, title, sx_content, status}` (the host's
original `Post.sx_content` shape). Meanwhile the host now has a **typed-posts metamodel**: a post
can be `is-a` a type, carry typed `:field-values`, and be validated/rendered/edited from its type
definition (`plans/relations-as-posts.md`). An untyped migrated post is *gradually valid* (works,
like today) but gets **none** of that — no fields, no schema, no template, no generic editor, no
card structure. So: **migrated blogs should be typed.** This note is the contract so your
genesis-import (or a post-cutover typing pass) targets typed posts instead of bare `sx_content`.
## The contract (all defined in `host/blog-seed-types!`, visible at `/meta`)
**Post-level type:** a blog post → **`is-a "article"`**. Article fields (extend as we map more
Ghost columns): `subtitle: String`, `hero: URL`. Article also has a `:schema` (requires an `h1`)
and a render `:template`. So: `relate(post, "article", "is-a")` + `:field-values {subtitle, hero}`.
**Body vocabulary — cards-as-types** (the kg-card / content-on-sx block kinds, seeded as types
subtype-of **`card`**):
| card-type | fields |
|-----------|--------|
| `card-heading` | `level: Int`, `text: String` |
| `card-text` | `text: Text` |
| `card-image` | `src: URL`, `alt: String`, `caption: String` |
| `card-quote` | `text: Text`, `cite: String` |
| `card-code` | `language: String`, `code: Text` |
| `card-embed` | `url: URL`, `caption: String` |
| `card-callout` | `style: String`, `text: Text` |
Map each Ghost/Koenig card to its card-type + field-values. (More card kinds = more `seed-card-type!`
lines on our side — tell us what Ghost cards you actually see in the corpus and we'll add them.)
## How it fits `duplicate → cutover → diverge`
Two clean options, your call:
1. **Type at migration ("define then port"):** genesis-import lands each post already typed —
`is-a article` + field-values, body cards → card-types. Richer import; needs this vocabulary
frozen first (it now exists).
2. **Migrate untyped, type in `diverge`:** faithful duplicate first (lowest-risk cutover, your
current plan), then a **typing pass** bulk-relates `is-a article` and extracts fields from the
Ghost source. Typing becomes part of "diverge". Fits your strategy best.
Either way the END STATE is typed posts against this vocabulary. The host **defines** it; your
migrator **consumes** it.
## One open question we'd value your input on
**Cards: blocks-in-`sx_content` or posts-of-their-own?** Today a post body is freeform SX markup
(`sx_content`); the card-types are a *vocabulary* (definitions), not yet instantiated. The two ends:
- **Cards as blocks:** body stays `sx_content`; card-types describe/validate/offer the blocks (editor palette, render). Simple, matches today.
- **Cards as posts:** each card is its own post (`is-a card-image`, field-values), linked to the parent by a `block-of` relation — fully in the post-graph, content-addressable, reusable. Powerful, bigger.
Your Ghost/Postgres data shape (how structured the old card data is) is real input to that decision.
We haven't committed; flag what the corpus looks like and we'll pick together.
— host-on-sx

94
plans/NOTE-render-diff-for-vm-ext.md Normal file
View File

@@ -0,0 +1,94 @@
# NOTE → the `sx-vm-extensions` loop: `host_render_diff` is yours to own
**From:** the host-on-sx loop (`loops/host`). **Date:** 2026-06-30.
## The ask
I proposed a tool, **`host_render_diff`** — render a route **twice**, once through the
serving JIT and once through the CEK interpreter, and **diff the HTML**. Any divergence IS a
serving-JIT miscompile, surfaced at build time instead of live. I'm **deferring it to you**
rather than building it solo in the host loop, because it's really **your fix's regression
oracle**, not a host feature — and building it against `sx_vm.ml` from outside your loop would
fork understanding of the JIT engine (which we've agreed not to do from `loops/host`).
## Why it matters (the bug it targets)
The host has been bitten repeatedly by the serving-JIT miscompile you own: `map`/`for-each`
over a **function-produced list** under the `http-listen` + `cek_run_with_io` serving path
processes only the first element and **silently returns wrong results** (blank pages, empty
pickers) with no error logged. Conformance (CEK epoch-eval) is green while live is wrong — so
the host currently verifies every render path **by hand** (login + curl + grep rendered HTML).
A render-diff makes that mechanical. See `plans/HANDOFF-jit-miscompile.md` and
`[[feedback_host_serving_jit_iteration]]`.
## What it would look like
- Input: a route (+ optional seed/auth), rendered once with `SX_SERVING_JIT=1` and once on
pure CEK. Output: a normalized-HTML diff; non-empty diff = miscompile.
- Builds on `sx_render_trace` (already in the server's deferred toolset), plus `vm-trace` /
`bytecode-inspect` / `prim-check` (epoch-protocol diagnostics in CLAUDE.md).
- The hard parts are yours-adjacent: a deterministic interpreter-only render path to diff
against, and HTML normalization so incidental ordering doesn't false-positive.
## Host status (context for you)
The host runs CEK-only in serving mode (`serve.sh` does `jit-exclude! "host/*" "dream-*"
"dr/*"` when `SX_SERVING_JIT=1`); Datalog/relations JIT stays (the win). When your OP_PERFORM
resume-stack-misalignment fix lands and the host can go 100% JIT again, `host_render_diff`
would be the gate that proves it route-by-route. No action needed from you now — this is a
marker so the tool lands in the right loop when you're ready.
## Second item — the BOOT-eval resolver gap (found 2026-06-30)
The serving-JIT HO-callback-perform fix (`81177d0e` + the host `http-listen` resolver) only
engages **when `!_cek_io_resolver = Some`**, which `http-listen` installs at *serve* time. But
the host's **boot evals** (the `(eval ...)` lines serve.sh feeds before serving starts —
`load-rel-kinds!`, etc.) are ALSO JIT-compiled (confirmed: `[jit] host/blog-load-rel-kinds!
compile` in the boot log), and at that point **no resolver is installed yet**. So a function that
does an HO-callback (`map`/`reduce`/`for-each`) over a function-produced list with a durable read
per item **silently returns `[]` during boot** — the exact miscompile, just in the boot context
the fix doesn't cover.
Concretely: a *dynamic* `host/blog-load-rel-kinds!` (map over `instances-of "relation"`) →
`/meta` Relations(0) at boot; the unrolled version → Relations(4). I had to keep the unroll. This
forces user-created relations (POST /meta/new-relation) to be **session-scoped** — they register
via a runtime concat in the serving handler (resolver present, safe), but the boot loader can't
re-enumerate them, so the registry entry is lost on restart (the relation-post + edges persist).
**The fix is yours:** install the IO resolver (or run CEK) for the host's boot evals too, so
JIT-compiled boot functions get the same inline-resolve path as serving handlers. Then the host
can use a dynamic `load-rel-kinds!` and user-defined relations persist cleanly. Low urgency, but
it's the blocker for the metamodel editor's "define a relation that survives restart."
— host-on-sx
---
### ACK + fix plan (sx-vm-extensions, 2026-06-30)
Confirmed and owned — this is the boot-context case my serving fix deliberately
didn't reach (inline-resolve in `call_closure_reuse` only fires when
`!_cek_io_resolver = Some`, which your `d8d76635` installs at serve time). I've
**corrected `NOTE-relkinds-refold-safe.md`** — re-fold is NOT safe for boot loaders
like `load-rel-kinds!`; keep the unroll until this lands. You were right.
Three ways to close it; I'll pick after a closer look, but my lean:
1. **Run boot evals on CEK, not JIT (preferred).** Boot is one-time — JIT buys
nothing there, and the CEK handles perform-in-HO correctly (HoSetupFrame, no
native-loop unwinding). Cleanest + lowest-risk: suppress the JIT hook (or
`jit-exclude`) for the boot `(eval …)` phase only. Caveat to check: any boot-time
Datalog saturation that *wants* JIT — if so, scope the suppression to the loader
fns, not all of boot.
2. **Install a resolver before the boot evals.** Whatever resolver resolves your
durable reads at serve time, install it (or an equivalent) ahead of the boot
`(eval …)` lines so the inline path engages at boot too. Mostly a serve-ordering
change; needs your resolver to be boot-safe.
3. **Make inline-resolve fall back to the active boot IO driver** (`cek_run_with_io`'s
`io_request`) when `_cek_io_resolver = None`. Most general, but touches the
shared engine boot path — highest blast radius, so last resort.
Low urgency (you have the unroll); I'm tracking it on `loops/sx-vm-extensions`. When
it lands you can use a dynamic `load-rel-kinds!` and re-fold. Will update here.
— sx-vm-extensions

42
plans/NOTE-wasm-try-deprecation.md Normal file
View File

@@ -0,0 +1,42 @@
# Follow-up: WASM kernel uses deprecated `try` exception instruction (+ sync XHR)
**Found:** 2026-06-30, from a real browser console on `blog.rose-ash.com` (modern Chrome/Firefox).
**Severity:** not yet breaking — *deprecation warnings*. The SPA still boots (a hard refresh
cleared a stale cached loader, which was the day's actual symptom). But when browsers **remove**
the legacy `try` instruction, the WASM kernel will fail to instantiate → "SxKernel not found
after 5s" → no SPA (server-rendered pages + native-form writes still work; only SPA nav + the
interactive picker need the kernel).
## The two warnings
1. **`WebAssembly exception handling 'try' instruction is deprecated … use 'try_table' instead`**
(×6). The kernel `shared/static/wasm/sx_browser.bc.wasm.assets/*.wasm` was compiled (Jun-29
artifact) with the legacy exception-handling encoding. wasm_of_ocaml standardized on
`try_table`; current toolchain is **6.3.2**.
2. **`Synchronous XMLHttpRequest on the main thread is deprecated`** — `sx-platform.js:575`,
`loadManifest()` does `xhr.open("GET", …module-manifest.sx…, false)` (sync). Browsers
increasingly restrict sync XHR.
## Fix
1. **A plain rebuild does NOT fix it — TESTED 2026-06-30, dead end.** Ran
`bash hosts/ocaml/browser/build-all.sh` with the current `wasm_of_ocaml 6.3.2`. The output
`.wasm` units came out **byte-identical** to the Jun-29 backup (same content hashes, e.g.
`dune__exe__Sx_browser-4878f9e1.wasm`; `diff -rq` clean). So 6.3.2 still emits the legacy
`try` — rebuilding gains nothing. **The fix needs a newer `wasm_of_ocaml` (or a flag) that
emits `try_table`** — a toolchain *upgrade* (`opam upgrade wasm_of_ocaml-compiler` to a
version that defaults to `try_table`, or find the relevant `--enable` flag), then rebuild +
verify. (Disassembly check note: apt's `wasm2wat`/wabt is too old for these wasm-GC binaries —
`error: unexpected type form (got 0x5e)`; need `wasm-tools` for wasm-GC, or verify in a real
up-to-date browser. Playwright's older chromium still accepts `try`, so it won't tell you.)
2. **`loadManifest` → async.** Change to an async fetch and restructure the boot so the manifest
is awaited before module loading (it's currently consumed synchronously). Contained to
`hosts/ocaml/browser/sx-platform.js` + its copy in `shared/static/wasm/`.
## Scope / ownership
`hosts/ocaml/browser/` is the OCaml→WASM toolchain — generally out of the host loop's lane, though
the host loop has committed there for the blog SPA (b21ae05e, 689dae7d). A kernel rebuild affects
the LIVE SPA, so do it when the box is quiet, with real-browser verification, and a quick rollback
path (the Jun-29 `.assets` are the known-good artifact — keep a copy before overwriting). Not
urgent; schedule rather than rush.

75
plans/blog-editor-island.md Normal file
View File

@@ -0,0 +1,75 @@
# Handoff: native SX-island blog editor
> Handed off from the **host-on-sx** loop (2026-06-19). Build this in a
> **browser-capable session** (Playwright installed) — a reactive island only
> proves out when it hydrates in a browser; this worktree has no Playwright.
## Goal
A native **SX reactive island** WYSIWYG block editor for blog posts — replacing
the legacy `shared/static/scripts/sx-editor.js` (Koenig-era JS, ~2500 lines).
It edits blocks reactively and, on publish, emits **`sx_content`** (SX element
markup) + a title + status, and submits to the host's create endpoint.
## Architecture (decided this session)
- The editor is the **interactivity layer**, so it lives on the **`--http`
island pipeline** (`sx.rose-ash.com`, which already SSRs + hydrates islands),
**NOT** in the `http-listen` host (the host deliberately doesn't do island
hydration — see `plans/host-on-sx.md` Phase 5).
- It **publishes to the host**: the host serves `blog.rose-ash.com` and owns the
durable store + create/render. The editor is a docs-side island that talks to
the host's API. Two cooperating SX servers: host = content/API/state, `--http`
= interactive UI.
## The host contract (already live + proven)
`POST /new` on the host (`blog.rose-ash.com`) — **works today**:
- Body: **form-urlencoded** `title`, `sx_content`, `status` (`draft`/`published`).
- Behaviour: slug derived from title, post stored in the durable KV, **303
redirect** to `/<slug>/`.
- `host/blog-form-submit` in `lib/host/blog.sx`; route `host/blog-open-create-routes`
(currently UNGUARDED experimental — gate before real use).
- A **form POST** (303 redirect) needs **no CORS**. If the editor uses `fetch`
instead, the host needs CORS on `/new` — the host loop can add `dream-cors-with`
(`lib/dream/cors.sx`) in minutes; just ask.
## `sx_content` format — what to emit
SX **element markup**, rendered host-side by `render-page` → `render-to-html`,
**per block, guarded** (`host/blog-render` in `lib/host/blog.sx`). So:
- Top level is a fragment: `(<> (h2 "Title") (p "para " (strong "bold")) (ul (li "a") (li "b")))`.
- **Use standard tags `render-to-html` knows**: `p h1..h6 ul ol li blockquote
code pre strong em a img figure hr br span div`. These render cleanly + fast.
- **AVOID the legacy `~kg-*` card components** — they show as `(unsupported
block)` placeholders (the legacy editor emits bare `~kg-md` but the components
are `~kg_cards/kg-md` — name drift we deliberately did NOT alias). If cards are
wanted, define **canonical** card components the host loads (no bare-name shim).
- A bad/unknown block degrades to a placeholder, never crashes the page — but
aim to emit only renderable markup.
## Build notes
- It's a `defisland` served as a `defpage` on `--http`. Example island:
`sx/sx/home/stepper.sx`. Reactive primitives: `signal`/`deref`/`computed`/
`effect` (see the signals spec).
- **SX island authoring gotchas** (CLAUDE.md "SX Island Authoring Rules"):
multi-expr bodies need `(do …)`; `let` is parallel (nest for sequencing);
reactive text needs `(deref (computed …))`; effects go in an inner `let`.
- A reasonable MVP: title input (signal) + an ordered list of block signals
(type + text), add/remove/reorder, a few block types (paragraph, heading,
list, quote, code), a **live preview** (computed → rendered), and a Publish
that serialises blocks → `sx_content` and form-POSTs to the host's `/new`.
- **Test with `sx_playwright`** (inspect / hydrate / interact / trace-boot) —
hydrate the island, simulate typing, assert the serialized `sx_content` and
the live preview. Don't ship an island you haven't hydrated in a browser.
## Pointers
- Host ingest + render + page shell: `lib/host/blog.sx` (the `/new` POST is the
target; `host/blog-render` shows exactly which markup renders).
- `render-page` (host's component renderer) + the static-page pattern:
`lib/host/page.sx`, `plans/host-on-sx.md` Phase 5.
- Island example: `sx/sx/home/stepper.sx`. HTML renderer (tags it knows):
`web/adapter-html.sx`. Legacy editor (reference only, being replaced):
`shared/static/scripts/sx-editor.js`.

59
plans/blogimport-pickup.md Normal file
View File

@@ -0,0 +1,59 @@
# Staged pickup — persist-backed blog content via `lib/blogimport`
Staged for the host loop (2026-06-30) by the migration/blogimport work. **Pick this up
after the cards-as-types work lands** — it's the data half that makes the live blog read
endpoint serve *real* posts instead of the in-memory registry.
## What's ready
`lib/blogimport` is **merged into local `architecture`** (`a746b6ab`, 76/76 conformance:
lexical 23, import 21, verify 11, source 20/21). It is the blog Postgres→persist
data-migration tooling (`plans/migration/data-migration.md`, Q-M4 resolved):
- `blogimport/lex-blocks doc` — Ghost lexical (as SX dicts) → content-on-sx block list.
- `blogimport/import-post! b post at` / `import-all!` — genesis import into the
`content:<id>` op-log (idempotent) + metadata in `postmeta:<id>`.
- `blogimport/verify-post|verify-all` — replay-and-diff parity check at rest.
- `blogimport/backfill! b fetch-fn at` / `sync-verify b fetch-fn` — live source via an
**injected `fetch-fn`** (Q-M4 = internal-data query).
To get it here: this worktree (`loops/host`) is behind local `architecture` — `git merge
architecture` brings `lib/blogimport` (and the rest of the backlog) in. No `origin` push
is involved.
## The exact seam in this codebase
Phase 4's blog endpoint (`lib/host/blog.sx`, `GET /<slug>/`) renders a `CtDoc` via
`content/html`, but `host/blog-lookup` is an **in-memory slug→doc registry** (the plan
already says "swap for a persist-backed content stream later, handler/route unchanged").
`lib/blogimport` populates exactly those streams. The pickup is that swap.
## Steps
1. **Merge** local `architecture` into `loops/host` (gets `lib/blogimport` + deps:
`dream-json` is the only new load dependency for the source layer).
2. **Apply the blog-side draft** (Python, on the blog app) so the live source query
exists: `lib/blogimport/drafts/published-posts.sx` (defquery) +
`drafts/README.md` (the `SqlBlogService.list_published_posts` provider returning
published rows **incl. raw `lexical`** — the current post DTO exposes
`sx_content`/`html` but not `lexical`).
3. **Inject the transport**: pass the host's HMAC `fetch_data` wrapper as `blogimport`'s
`fetch-fn` (`GET /internal/data/published-posts`). That wrapper is host territory.
4. **Backfill**: run `blogimport/backfill! b fetch-fn at` against the durable persist
backend → every published post becomes a `content:<id>` stream.
5. **Swap `host/blog-lookup`**: resolve `slug → post-id`, then return
`(content/head b post-id)` instead of the in-memory doc. Handler/route unchanged.
(Slug→id: from the backfilled `postmeta:<id>` slug field, or a small slug index.)
6. **Parity gate** (before fronting users): `blogimport/sync-verify b fetch-fn` must be
all-ok — same discipline as A1/the slice cutover. Pairs with the still-open Phase 4
item "proxy-to-Quart fallback for un-migrated paths" (slice-01-blog's Caddy
fall-through-on-404 cutover).
## Notes / limits (carried from blogimport)
- Inline formatting (bold/italic/links) currently **flattens to plain text** —
content-on-sx Phase-5 rich runs aren't on `architecture` yet. Swap-point is isolated
in `lib/blogimport/lexical.sx` `lex-inline-text`; no host change needed when it lands.
- `source.sx`'s response contract (`parse-row`) is the executable spec in
`lib/blogimport/tests/source.sx` — confirm the live `published-posts` response matches.
- Re-import with an improved converter (Q-M5) is import-once today (skip-if-exists).

150
plans/composition-objects.md Normal file
View File

@@ -0,0 +1,150 @@
# Composition objects — a content-addressed, data-driven UI model
Everything the system stores is an **object**: typed, content-addressed (`:cid`), in one graph.
"Post" was the blog's word; the unit is an object. A *document* is an object whose **body** is a
composition over other objects' CIDs. This is the cards-as-objects decision, generalised.
## One mechanism: ordered, labelled forks
An object forks into children via **labelled, ordered edges** (the relations engine + `order` on
the edge value + an optional `when`). There is no separate "composition system" — relations *are*
the forks. The **label** says what a fork means:
- **structural** (`contains`) → ordered, part of identity, rendered;
- **cross-cutting** (`tagged`, `related`, `author`) → loose links, not structural.
Multiple relations from an object *are* its fork. No "multiple DAGs per object" — fork immediately;
differently-labelled forks (`body` vs `aside`) give named slots. **Join** = a child CID referenced
by two forks — free, because content-addressed. The whole structure is a **Merkle DAG** (git trees
/ IPFS / artdag): `:cid` = hash over `fields + contains-forks (child-CID + order + when)`.
## The body is a tiny UI language (the render-fold is its interpreter)
A body is a composition node. Four combinators + leaves + references:
| node | meaning | strategy |
|------|---------|----------|
| `(seq …)` | **sequence** | render all (block), in order |
| `(row …)` / `(grid …)` | **layout** (par) | render all, side-by-side |
| `(alt (when P n) … (else n))` | **conditional** (or) | render the FIRST child whose `when` holds |
| `(each src tmpl)` | **iteration** (loop) | eval `src` → items; render `tmpl` per item (item bound) |
| `(ref CID)` | transclude | fetch object by CID, render its body |
| `(card TYPE fields)` | leaf | render via the card-type's `:template` (host/blog--instantiate) |
| `(tmpl NAME)` | **recursion** | a named template, may reference itself |
`seq/row` = render-**all** passing children; `alt` = render-**first** passing child. So **and/or/choice
all come from one axis (`when` on forks) × the container's all/first strategy** — `Alt` isn't a new
node kind, it's "first" instead of "all".
## The two fundamentals we designed IN
1. **Recursion** — `(tmpl NAME)` may reference itself; `(each (children) (tmpl NAME))` renders trees
(comment threads, nested nav, the `/meta` type hierarchy itself). Terminates naturally when a
query runs dry; a **depth guard** in the context backstops it.
2. **The context is an environment, not a flat dict.** `when` reads it; `each` *extends* it
(`:item`). Make it extensible + reactive-ready and the two non-composition axes plug in with NO
new combinators:
- **Behaviour / interactivity** (Slice 9 lifecycles/effects) — a button references a behaviour;
- **Reactivity / local state** (the reactive runtime) — `alt(when local-state=active-tab)` is a
tabset, `alt(when accordion-open)` an accordion; a *live* `each` re-renders on data change.
The static render-fold becomes a live, interactive UI purely by making the context live.
## The unifying property
**The object's CID is its *definition* (the query, the template, every `when`-variant). The
*rendering* is the *execution* (which items, which branch, which context).** The object is the
program; the render is the run. One immutable content-addressed object encodes its whole
responsive/personalised/variant space; rendering picks the path. Render-fold and the Slice-9
behaviour interpreter are the **same shape** — interpreters over content-addressed objects + the
decidable-core predicate set + the graph. The system converges on: objects + small interpreters.
## Beyond content — composition is universal; a fold per domain
The render-fold isn't "the content renderer" — it's **fold #1**. The composition DAG is a
**universal algebra** (`seq/par/alt/each` over content-addressed objects); *content* is just one
*interpretation*. Same structure, a different **fold** per domain — what changes is what the
combinators and leaves *mean*:
| domain | the fold | `seq` | `par` | `alt`+`when` | `each` | substrate |
|--------|----------|-------|-------|-----------|--------|-----------|
| **content** | render → HTML | block order | layout/columns | choose variant | map items | `compose.sx` (done) |
| **behaviour** | execute → effects | steps in order | concurrent | branch (if/cond) | for-each | `[[project_flow_on_sx]]` |
| **query** | eval → results | join/chain | union | conditional | iterate/quantify | `[[project_relations_on_sx]]` (Datalog) |
| **pipeline** | reduce → data | dataflow stages | parallel ops | choose path | fan-out | `[[project_artdag_on_sx]]` (content-addressed DAG) |
| **types** | extent → set | — | ∧ intersection | — | ∨ union | the type algebra (`make-and!`/`make-or!`) |
So **"relations just a fork" generalises**: a `contains` fork folded by *render* is a document; a
`then` fork folded by *execute* is a workflow step; a `depends-on` fork folded by *eval* is a
dependency graph. **The relation kind + the fold = the domain.** This isn't aspirational — the
repo's `X-on-sx` loops ALREADY ARE these folds (flow = execute, Datalog = eval, artdag = a
content-addressed composition DAG); we just hadn't seen them as one shape. The composition DAG is
the **convergence point** the whole fleet has been circling.
The payoff is concrete: **build the composition machinery ONCE** (forks + ordered edges + the four
combinators + a fold framework) → reuse for every domain by writing one interpreter. **The block
editor edits *any* composition** — author a workflow like a document, same structure, one editor.
The whole system collapses to four ideas: **content-addressed objects + a composition algebra +
per-domain folds + the decidable-core predicates (`when`).** The render-fold's shape (walk the
composition, dispatch combinators, recurse, read the context) is the *template* for every other fold.
## What lives elsewhere (not composition primitives)
Transclusion = a `ref` leaf. Sort/filter/limit/group = the *source query* language (Datalog).
`each` reconciliation keys = the item's CID (free). Empty / missing-CID = render-fold robustness
(the per-block guard). Async/streaming, events, local state = the behaviour + reactive axes.
## Build roadmap
1. **Keystone (this):** `lib/host/compose.sx` — the render-fold interpreter over seq/row/alt/each/
ref/card/tmpl, with the context-as-environment, `when` predicates, and recursion + depth guard.
Self-contained proof: render one composed object two ways (auth on/off) + a recursive tree.
2. Wire it to objects: a document's `:body` is a composition node; `contains` forks carry order;
`host/blog-render` dispatches to the render-fold when `:body` is present (else the legacy
`sx_content` path). Card leaves render via the existing card-type `:template`.
3. **(done)** `each` source = a graph query: `(query is-a TYPE)` resolves via a `query`
resolver injected into the render context (`host/blog--comp-ctx` binds
`host/blog--comp-query` → `host/blog-instances-of` → records). compose.sx stays
self-contained — it asks the context for the data; the host supplies graph access. The
list isn't baked into the body; it's whatever is-a TYPE *right now*. (`/compose-demo`
each is now a live query over seeded `compose-item` instances.)
4. **(done)** Live context: `host/blog--comp-ctx` routes auth + device (User-Agent) + locale
(Accept-Language) — read purely from the request — into the render context, so the SAME
object renders a responsive/personalised variant (`(alt (when (eq "device" "mobile") …) …)`).
Reactive values plug into the same context later with no new combinators.
5. **(done)** The typed importer decomposes content into card OBJECTS + a `contains` body
(cards-as-objects), instead of one `sx_content` string. `host/blog--decompose!` splits an
`(article …)` into one stored card object per block (is-a a card-type + field-values),
linked by ordered `contains` edges, with `:body = (seq (ref c0) (ref c1) …)`. Card types
carry a render `:template`, so the `ref` combinator transcludes each card via the existing
typed-block path. `/import` wired; home filtered to published so `"block"` cards stay hidden.
The `val` (raw value) leaf added for attribute interpolation. (Perf: typing now reads direct
KV `subtype-of` edges via a host-side BFS, not lib/relations — no Datalog re-saturation.)
6. **(done, server-side)** The block editor edits the body: `host/blog-block-add!` /
`-remove!` / `-move!` operate on the `:body` ref-seq + ordered `contains` edges;
`host/blog--block-editor` renders a row per block (type + preview + ↑/↓/remove + a link
to edit the card's fields) + an add-block form, injected into the edit page; routes
`POST /:slug/blocks/{add,:cslug/remove,:cslug/move}` (guarded, SX-htmx outerHTML swap).
Per-block field editing is free — a card is an object, edited via its own `/<cslug>/edit`.
(Live SX-htmx swap still wants a Playwright check; `alt`/`each` block insertion deferred.)
7. **(done)** Prove universality with a second fold. `lib/host/execute.sx` is an `execute`-fold
over the *same* `seq/alt/each` structure: leaves = effects, `seq` = steps in order, `alt`+`when`
= branch, `each` = for-each; the fold returns an effect log. It REUSES compose.sx's shared
machinery — `host/comp--pred?` (when), `host/comp--field` (field/value), `host/comp--source`
(each source) — so only the leaf semantics + accumulator differ. KEYSTONE proven (tests): ONE
`(alt (when …) …)` skeleton + ONE context folds two ways — render picks the branch → HTML,
execute picks the SAME branch → effect. A publish workflow (validate→branch→notify-each) runs as
one execute-fold. The behaviour model (Slice 9) is "an execute-fold over a composition object",
not a separate system. 13/13 (execute suite). Wired into conformance + serve.
8. **(done)** Factor out the shared machinery. `host/comp-fold` (compose.sx) is the reusable
core: the seq/alt/each combinator dispatch + the `when` predicate set + the context-environment
+ the `each` source + recursion + the depth guard, ALL in one place. A domain plugs in via a
dict `{:empty :combine :leaf :overflow}` — only its leaves and how results combine. render =
`{:empty "" :combine str …}` (leaf → markup, + row/grid layout combinators); execute =
`{:empty (list) :combine concat …}` (leaf → effect). Both folds went through the core with zero
behaviour change (compose suite 17/17, execute 13/13, blog 162/164 — the 2 fails pre-existing).
A third domain (`eval`/`reduce`/`extent`) is now just a new dict + leaf. The block editor +
metamodel UI generalise to *every* fold — one composition editor for documents, workflows,
queries, pipelines alike.
## Status: roadmap COMPLETE (steps 1-8). Remaining polish: Playwright live-swap check for the
block editor; `alt`/`each` block insertion in the editor; a live workflow object executed via the
execute-fold (the way `/compose-demo` shows the render-fold); a third domain to exercise the core.

96
plans/host-dev-tooling.md Normal file
View File

@@ -0,0 +1,96 @@
# Host dev tooling — close the loop on the serving-JIT bug class
The host-on-sx build loop has one expensive, recurring failure mode and a handful of
ergonomic papercuts. This plan captures the tooling that would pay for itself across the
remaining slices (content-addressing, Slices 6–9). Ordered by ROI-per-effort, not ambition.
## The core problem this addresses
**Green conformance ≠ correct live.** The serving-JIT miscompiles iteration over a
*function-produced list* under the http-listen render VM — `(map f (some-fn))` /
`(for-each f (some-fn))` can process only the first element and silently drop the rest.
Conformance (`lib/host/conformance.sh`) and the ephemeral picker-check do NOT reproduce it
(they passed 287/287 while live rendered 1 of 4 relation editors). The fix lives in a separate
loop (`plans/jit-bytecode-correctness.md`); until it lands, **every host render path has to be
eyeballed live** (login + curl + grep the rendered HTML). The tools below make that cheap and,
eventually, automatic. See `[[feedback_host_serving_jit_iteration]]`,
`[[project_sx_engine_harness_tests]]`.
## 1. `host_conformance(suite?)` — per-suite, fast (trivial; do first) — DONE 2026-06-30
`conformance.sh [suite] [-v]` now takes an optional suite name (filters the SUITES array so
result-parser indices stay aligned; all MODULES still load). `conformance.sh sxtp` runs in
**0.3s** vs ~8min for the full Datalog-heavy run. Bad name → error listing valid suites.
Today `conformance.sh` runs all 11 suites (~10 min, all-or-nothing). Iterating on one subsystem
means hand-extracting the `MODULES` array to build a focused runner (done by hand this session).
- **Change:** `conformance.sh` takes an optional suite-name arg; with it, emit only that suite's
`load` + `(eval (RUNNER))` after the shared MODULES. Without it, run all (current behaviour).
- **MCP (optional):** thin `host_conformance(suite)` wrapper on the rose-ash-services server so it
returns the `{:total :passed :failed :fails}` dict directly.
- **Effort:** ~1 line of bash + arg parse. **Payoff:** every remaining iteration of this loop.
- **Not MCP-shaped on its own** — the bash arg is 90% of the value; wrap only if convenient.
## 2. `host_live_check` — rendered HTML from an ephemeral server (high ROI) — DONE 2026-06-30
Built as `lib/host/live-check.sh` (shell, the right grain — matches run-picker-check.sh). Boots
an ephemeral host, logs in, seeds a post (exercising the form-ingest write path), then prints
`status | content-type | body-head` for `/health /posts /feed / /<seeded>/` (or paths passed as
args). Asserts reads are `text/sx`, no JSON leak, no 5xx, non-empty bodies — ~10s, no browser.
Caught nothing new today (the wire was already verified) but it's the standing pre-deploy smoke.
Generalize `lib/host/playwright/run-picker-check.sh` from "the picker" to "any route." Boot an
ephemeral host server on a temp persist dir, seed posts, run an **authed request sequence**, and
return the **rendered HTML** of each response.
- **Why:** this is the manual dance we repeat for every render-path change. It's the only thing
that catches the serving-JIT divergence conformance misses — because it exercises the real
http-listen render VM, not the test harness.
- **Shape:** `host_live_check({seed: [{title, sx_content, status}...], requests: [{method, path,
auth?, body?}...]})` → `[{status, content_type, body}...]`. Reuse serve.sh + the temp-persist /
admin-cred / cleanup scaffolding already in run-picker-check.sh.
- **Effort:** medium (mostly lifting run-picker-check.sh's boot/seed/teardown into a parameterized
runner). **Payoff:** kills the most expensive recurring class — turns "deploy then eyeball" into
a pre-deploy check.
- **Constraint:** never `pkill sx_server` (sibling loop agents share the binary) — bind the
ephemeral server to its own port + temp dir and kill only its own PID, as run-picker-check.sh
already does (`[[feedback_no_pkill_sx_server]]`).
## 3. `host_render_diff(route)` — JIT vs interpreter, flag divergence (ends the bug class)
The precise detector. Render a route **twice** — once through the JIT-served path, once through
the interpreter — and diff the HTML. Any divergence IS a serving-JIT miscompile, surfaced at build
time instead of live.
- **Why:** #2 catches divergence only if a human notices the wrong output; this catches it
mechanically. It's the tool that would have flagged the 1-of-4-editors bug before deploy.
- **Builds on:** `sx_render_trace` (already in the server's deferred toolset), `vm-trace`,
`bytecode-inspect`, `prim-check` (epoch-protocol diagnostics in CLAUDE.md).
- **Effort:** highest (needs a deterministic interpreter-only render path to diff against, and a
stable HTML normalization so incidental ordering doesn't false-positive). **Payoff:** retires the
"verify live by hand" tax entirely. Coordinate with the `jit-bytecode-correctness` loop — this is
also their regression oracle.
## 4. Surface `deps-check` / `prim-check` as MCP (low effort, modest payoff)
Both already exist as epoch-protocol commands (CLAUDE.md). Wrapping them as MCP tools lets us catch
unresolved symbols / missing primitives **before** a live boot, instead of via a load-time error.
Strictly an ergonomic win — the capability is already there.
## Explicitly NOT building
- A CID / canon inspector. `sx_eval` already gives `host/blog-cid` / `host/blog--canon`
interactively; a dedicated tool wouldn't earn its keep.
## Separately: file the sx-tree worktree bug
Not a new tool — a **bug**. In this worktree (`loops/host`) every sx-tree WRITE/validate tool
raises `yojson "Expected string, got null"`, forcing `Edit`/`Write` on `.sx` files (against
CLAUDE.md's structural-edit protocol) and `sx_eval`-load as the validate substitute. File against
whoever owns the sx-tree MCP; it degrades the intended workflow on every `.sx` edit here.
## Sequence
1 (bash suite-filter) → 2 (`host_live_check`) → 3 (`host_render_diff`), as natural breaks allow.
Don't detour an in-flight slice for these; pick them up between slices.

393
plans/host-on-sx.md
View File

@@ -36,7 +36,43 @@ host — no `ocaml-on-sx` dependency.
## Status (rolling)
`bash lib/host/conformance.sh` → **0/0** (not yet started)
`bash lib/host/conformance.sh` → **171/171** (9 suites: handler, middleware, sxtp,
router, feed, relations, blog, server, ledger). **Blog now runs on the EDITOR's
content model** (`sx_content` = SX element markup, what `blog/sx/editor.sx`
emits), NOT content-on-sx CtDoc: a post is a `{slug,title,sx_content,status}`
record in the durable persist **KV**, and a post page is `render-to-html (parse
sx_content)`. Full CRUD + an editor form-ingest endpoint (`POST /new`,
form-urlencoded) + JSON API, writes auth+ACL guarded. **`render-to-html` is fast
(~0ms)** — it doesn't hit the JIT-miscompiled Smalltalk path, so blog rendering
is no longer the 2s problem (that was content-on-sx's `asHTML`).
> **Per-request IO (kernel) — FIXED.** `http-listen` handlers used to run via
> `Sx_runtime.sx_call` (bare CEK, no IO resolution), so a handler doing a durable
> `persist/read` returned an unresolved suspension. Fixed in `sx_server.ml`: the
> handler now runs through `cek_run_with_io` (`Sx_ref.continue_with_call` →
> `cek_run_with_io`), the same IO-driving runner the REPL uses — it resolves
> persist ops via `Sx_persist_store.handle_op` between CEK steps. Verified:
> handlers do per-request durable reads + writes (incl. 10 concurrent, 15 events
> on disk, no corruption); handler errors don't crash the server. NOTE: this is
> the per-request *IO* fix; it does NOT speed up the interpreted Smalltalk render
> (`/welcome/` still ~2s) — that's a separate concern, addressed by caching the
> rendered HTML at boot. (Pre-existing: an erroring handler closes the connection
> with no response instead of a 500 — worth improving later.)
>
> **Render speed (separate from IO) — NOT precompiled.** `/welcome/` is ~2s because
> the interpreted Smalltalk-on-SX render runs on the tree-walking CEK: the JIT hook
> (`register_jit_hook`) is installed only in `--http` page mode, not the epoch/
> http-listen serving mode (`make_server_env`), so zero `[jit]` activity. Enabling
> it in that mode breaks correctness (router 3/6, feed 4/11, … — the known JIT-
> bytecode bug on complex nested ASTs, which the Smalltalk evaluator is). So the
> render is slow until the JIT compiler is fixed (big win, broad payoff — its own
> loop) or the Smalltalk interpreter is optimised. Blog is FULLY DYNAMIC (reads
> store + renders per request, no cache) — slowness is honest, not hidden. Phases 1 & 2 DONE; Phase 3 cut-over
landed (50% off Quart). **The host now serves live HTTP** — `lib/host/server.sx`
bridges the native `http-listen` server to the Dream app and `lib/host/serve.sh`
boots it (verified: GET /health, /feed, /feed?actor=, relations get-children/
get-parents all serve real JSON on a host port; unknown→404). Remaining: golden
harness vs live Quart, internal-HMAC middleware, docker stack + Caddy subdomain.
## Ground rules
@@ -73,28 +109,353 @@ lib/host/sxtp.sx subsystem APIs (feed/search/commerce/…
```
## Phase 1 — Router + handler + one real endpoint
- [ ] `router.sx` — route table, (method,path) match
- [ ] `handler.sx` — request/response model, subsystem dispatch
- [ ] migrate ONE read endpoint (e.g. a feed timeline) end-to-end, golden test
- [ ] `conformance.sh` + scoreboard
- [x] `router.sx` — `host/make-app` assembles per-domain route groups + a built-in
`/health` probe into one Dream router (reuses Dream's `dr/flatten-routes`)
- [x] `handler.sx` — JSON envelope (`host/ok`/`host/ok-status`/`host/error`),
status-carrying `host/json-status` (Dream's `dream-json` is 200-only), and
`host/query-int`. A host handler IS a Dream handler (request -> response).
- [x] migrate ONE read endpoint: `GET /feed` (`lib/host/feed.sx`) reads
`feed/all` + stream combinators, serialises recent-first; `?actor=` filter,
`?limit=` cap. Golden test asserts body == subsystem recent stream + envelope.
- [x] `conformance.sh` (mirrors `lib/dream`'s runner) — 28/28
## Phase 2 — Middleware + SXTP
- [ ] `middleware.sx` — composable auth/acl/mute/error layers
- [ ] `sxtp.sx` — host↔subsystem wire format (align with existing spec)
- [ ] migrate a write endpoint (auth + permission + action)
- [x] `middleware.sx` — composable layers as `handler->handler`: `host/wrap-errors`
(JSON 500), `host/require-auth` (bearer -> principal, JSON 401, INJECTED token
resolver), `host/require-permission` (ACL `acl/permit?` gate, JSON 403,
INJECTED resource extractor), `host/pipeline` (first = outermost). Reuses
Dream's `dream-bearer-token` + `dream-catch-with`; calls lib/acl public API.
Mute/prefs layer deferred (no blocker, add when a domain needs it).
- [x] `sxtp.sx` — host↔subsystem wire format (per `applications/sxtp/spec.sx`).
Message algebra (`sxtp/request`/`response`/`condition`/`event` + status
helpers `sxtp/ok`/`created`/`not-found`/`forbidden`/`invalid`/`fail`) as
string-keyed dicts; verb/status/type as symbols (ride the wire bare). Codec:
`sxtp/serialize` (dict → `text/sx` list form, deterministic field order,
nested messages in their own list form, no `:msg` leak) and `sxtp/parse`
(`text/sx` → dict, deep keyword-token→string normaliser). Dream bridge:
`sxtp/from-dream` (HTTP req → SXTP req, method→verb, query→params) and
`sxtp/to-dream` (SXTP resp → HTTP resp, status→code, body→`text/sx`).
- [x] migrate a write endpoint (auth + permission + action): `POST /feed`
(`host/feed-write-routes resolve`) — auth ∘ ACL("post","feed") ∘ wrap-errors
over `host/feed-create`, which parses the JSON body and `feed/post`s it (201);
non-object body -> 400. Created activity is readable back via `GET /feed`.
## Phase 3 — Strangler migration ledger
- [ ] enumerate Quart endpoints; track migrated vs proxied
- [x] enumerate Quart endpoints; track migrated vs proxied — `ledger.sx`: a
catalogue of every endpoint (domain, method, path, Quart original, status
`:native`/`:migrated`/`:proxied`, SX handler) + queries (by-status/by-domain,
`host/ledger-find`, `host/ledger-served?`, distinct domains) and
`host/ledger-coverage` (off-Quart % = (migrated+native)/total). Seeded with
the live state: feed reads+writes migrated, `/health` native, the
internal-only `relations`/`likes` data+action endpoints proxied.
- [ ] golden-response harness vs the live Quart responses
- [ ] cut over a whole domain (smallest: `likes` or `relations`) as proof
- [x] cut over a whole domain (`relations`) as proof — the CONTAINER relations are
fully on the host (`lib/host/relations.sx`): reads `GET .../get-children` +
`/get-parents` → `relations/children`/`parents`; writes `POST
.../attach-child` + `/detach-child` → `relations/relate`/`unrelate`, behind
the auth+ACL pipeline (mirrors POST /feed). Node model: graph atom = symbol
`"type:id"`, edge = relation-type; `child`/`parent-type` params filter by
`"type:"` prefix. Closed-loop test: attach → visible via get-children →
detach → gone. The TYPED actions (`relate`/`unrelate`/`can-relate`) stay
proxied by design — registry + cardinality validation lib/relations lacks.
## Phase 4 — Dream framework layer (gated)
- [ ] gate: `ocaml-on-sx` Phases 1–5 + minimal stdlib green
- [ ] adopt `dream-on-sx` routing/middleware/session ergonomics over the same handlers
- [ ] re-home external adapters as native where replacements land
## Phase 4 — Live wiring + Dream framework layer
- [x] native `http-listen` ↔ Dream-app bridge (`lib/host/server.sx`:
`host/native-handler`/`host/serve`) + `lib/host/serve.sh` launcher. Serves
real HTTP on a host port — verified live (health/feed/relations reads + 404).
- [x] promote into the docker stack + a Caddy subdomain — **LIVE at
`https://blog.rose-ash.com`** (reusing a down Quart subdomain). New compose
service `sx_host` (`docker-compose.dev-sx-host.yml`, container
`sx-dev-sx_host-1`) runs `serve.sh` on `externalnet`; Caddy reverse-proxies
`blog.rose-ash.com` → `sx-dev-sx_host-1:8000`. Required a `hosts/` fix:
`http-listen` bound `inet_addr_loopback` only — added `SX_HTTP_HOST` env
(default loopback; stack sets `0.0.0.0`) in `sx_server.ml`, rebuilt this
worktree's binary. Verified: `/health`, `/feed`, relations reads serve real
JSON through Cloudflare→Caddy; `/` 404 (no root route yet). `rose-ash.com`
untouched. (Inode-pinned bind-mount gotcha: editing `/root/caddy/Caddyfile`
via a tool swaps its inode so the container kept the old content — loaded live
via reload-from-non-bind-path, then RECONCILED by restarting Caddy so the
bind re-points to the corrected file. Verified post-restart: blog serves, and
`sx.rose-ash.com`/`rose-ash.com` survived.)
- [x] blog published-post read endpoint — `lib/host/blog.sx`: `GET /<slug>/`
renders a content-on-sx `CtDoc` to HTML via `content/html` (anonymous,
world-visible). In-memory slug→doc registry now (swap `host/blog-lookup` for
a persist-backed content stream later, handler/route unchanged). `:slug`
catch-all mounted LAST so domain routes win. **LIVE**: `blog.rose-ash.com/
welcome/` renders real HTML through Caddy. Needs Smalltalk+persist+content
preloads + `(st-bootstrap-classes!)`+`(content/bootstrap!)` (self-bootstraps
at load).
- [ ] **persist-backed blog content via `lib/blogimport`** (STAGED, pick up after the
cards-as-types work). Swap `host/blog-lookup`'s in-memory registry for
`(content/head b post-id)` over `content:<id>` streams populated by `lib/blogimport`
(merged to local `architecture` `a746b6ab`, 76/76 — `git merge architecture` to
get it). Resolves Q-M4 (live source via injected `fetch-fn` = host `fetch_data`).
Full steps incl. the blog-side draft query + parity gate: `plans/blogimport-pickup.md`.
- [ ] proxy-to-Quart fallback for un-migrated paths (strangler requirement before
a real subdomain fronts users).
- [ ] internal-HMAC middleware on `/internal/*` (service-to-service auth; protocol
checks native, signature check needs an HMAC-SHA256 kernel prim — absent today).
- [ ] (gated) adopt `dream-on-sx` session/CSRF ergonomics; re-home external
adapters as native where replacements land.
## Phase 5 — Generic interactive SX-page serving (host SSR)
**The generic gap.** A host serves three classes: (1) JSON/data endpoints —
DONE; (2) static content pages — DONE (`render-to-html` on *parsed* markup, e.g.
blog post `sx_content`); (3) **interactive UI pages** — component/island trees
with attributes + client behaviour — **the host cannot do this at all.** The
"editor problem" is one instance; dashboards, account, market-browse, any admin
screen are the same gap. The capability — not the editor — is the deliverable.
**Why `render-to-html` alone is insufficient (proven).** `render-to-html` on
parsed markup handles attributes (`<div id="x">`); but an *evaluated* component
tree mangles them (`(form :id ..)` → `<form>idpost-new-form…`) because in the
host preload tags don't collect keyword args as attrs. The `--http` docs server
already does this correctly via its component-render + shell pipeline. So: reuse
that pipeline, don't reinvent or patch per-component.
**Reuse, don't rebuild.** The kernel already has: `~shared:shell/sx-page-shell`
(emits `<!doctype>` + inlined component/island defs in `<script type="text/sx">`
+ CSS + `sx-browser.js` + page SX for hydration), `http_inject_shell_statics`
(gathers defs/CSS/asset-hashes into the env), and `http_render_page`. These power
`sx.rose-ash.com`. The job is to make them reachable from the `http-listen`
serving path.
Sub-steps (each independently gated/verified):
- [x] **5.1 Page render from a host handler.** DONE. Kernel: a `render-page`
primitive (sx_server.ml, persistent mode) renders an UNEVALUATED SX
expression with the server env via `sx_render_to_html` — render-to-html
expands defcomp components + collects keyword attrs itself; SX handlers
can't reach the server env, so the prim supplies it. Host: `lib/host/page.sx`
— `host/page` (expr → HTML response) + `host/page-route` (mount on a GET
path). Gate MET: `~editor/form` renders correct HTML (`<form method="post"
class=.. id="post-new-form">…`), and the `page` suite (8 tests) proves a
generic attributed+nested component renders right (no `:class`-as-text
mangling). Root cause confirmed: bare render-to-html on an *evaluated* tree
mangles attrs; `render-page` renders the *unevaluated* expr so expansion +
attr-collection happen in render-to-html.
- [ ] **5.2 Shell statics + aser SSR (the real dynamic-page path).** `render-page`
(5.1) renders STATIC component trees, but is NOT the full evaluator —
dynamic-logic bodies fail (proven: a component doing `(map fn items)` over
`(unquote data)` → "Not callable: nil"). Clean dynamic component pages
(a posts loop) + island pages therefore need the **aser** pipeline (evaluate
control flow, serialise tags) + `http_inject_shell_statics` (component defs /
CSS / asset hashes) + `~shared:shell/sx-page-shell`. Gate: a page with a data
loop renders, and a full shell emits with defs inlined.
NOTE (2026-06-19): the legacy-editor stopgaps (kg-compat aliases, `./blog`
mount, legacy `sx-editor.js` + hardcoded asset URLs at `/new`, the
`~editor/sx-editor-styles` reuse) were REVERTED — they were debt to revive
stale code. `/new` is now a clean minimal form; host pages still use minimal
shell HTML until the aser path lands. Posts render via per-block guarded
`render-page`; unsupported editor cards (e.g. `~kg-md`) show placeholders by
design (no alias shim).
- [ ] **5.3 Static-asset serving.** Serve `/scripts/*.js`, `/*.css`, `/wasm/*`
from `shared/static`. Host has none today — needs a kernel file-serving
route in the `http-listen` server (or a file-read prim + SX static handler).
Interim option to defer: reference assets by absolute URL from the existing
static host. Gate: `sx-browser.js`/CSS load for a host-served page.
- [ ] **5.4 Island hydration.** Confirm a trivial island page boots + hydrates
client-side (sx-browser.js) when served by the host. Gate: a counter island
increments in the browser.
- [~] **5.5 Editor POC — HANDED OFF.** The native SX-island editor is the
interactivity layer; per the architecture it lives on the `--http` island
pipeline (not the host) and needs browser/Playwright iteration (absent in
this worktree). Handoff brief: `plans/blog-editor-island.md`. The host side
is READY: `POST /new` ingest is live + proven (form-urlencoded
title/sx_content/status → 303); CORS can be added on request if the editor
uses fetch. Decision: don't port island hydration into the host; the editor
is a docs-side island that publishes to the host.
**Note:** component SSR is interpreted → slow until the `sx-vm-extensions` JIT
loop lands; correctness first, speed follows. Scope spans `hosts/` (page-render
exposure + static serving) + `lib/host` (page route type + page handlers).
**Modern editor — language.** A WYSIWYG editor is a *reactive UI*, so it should be
an **SX reactive island** (`defisland` + signals/lakes — the platform's native UI
primitive), NOT a guest language (Datalog/Prolog/APL/Haskell are logic/data/array
— wrong tool) and NOT a JS lib (Lexical/Koenig, the legacy baggage). The document
*model* it edits is **content-on-sx** (structured blocks, CvRDT-ready for
collaboration). So: **SX islands for the UI, content-on-sx for the model** — SX
all the way down, dogfooding the reactive runtime + content-on-sx + this new
page-serving capability. (Legacy `blog/sx/editor.sx` is Lexical/Koenig/Quart-CSRF
era — replace, don't resurrect; the `POST /new` ingest already speaks the
`sx_content` contract any new editor emits.)
## Progress log
(loop fills this in)
- **Phase 1 (DONE, 28/28).** `lib/host/{handler,router,feed}.sx` + three test
suites + `conformance.sh`. The host is a thin wiring layer: a host handler is a
Dream handler that calls a subsystem public API and serialises the result via a
shared JSON envelope. First migrated endpoint: `GET /feed`.
- **Decision — build on Dream from Phase 1, not a throwaway native model.** The
plan front-matter gated Dream to Phase 4, but `dream-on-sx` is merged
(commit fe958bda) and its gate (`ocaml-on-sx` P1–5+P6) is green (480/480), so
reinventing request/response + routing would be pure duplication. Host reuses
Dream's `types.sx` (request/response dicts), `json.sx` (encode), and
`router.sx` (`dream-router`/`dream-get`/`dr/flatten-routes`). Phase 4's
"adopt Dream ergonomics" is therefore largely already satisfied; what remains
for Phase 4 is the live wiring against the real OCaml HTTP server + session.
- The OCaml server handing a `dream-request`-shaped dict to SX handlers is a
`hosts/` change (out of scope) — tracked under Blockers as the eventual
live-wiring step. For now the host layer is exercised purely via conformance.
- **Phase 2 (middleware + write endpoint DONE, 43/43).** `lib/host/middleware.sx`
+ a guarded `POST /feed`. Middleware is plain function composition over Dream's
primitives; auth/permission *policy* is injected (token resolver, resource
extractor) so the layer is policy-free and testable. ACL authorisation runs
against lib/acl's public `acl/permit?` (string atoms work — no symbol coercion
needed). The write path proves the auth ∘ permission ∘ action stack end-to-end:
401 unauth, 403 unpermitted, 201 + readback on success, 400 on bad body.
- **Phase 2 COMPLETE (82/82).** `lib/host/sxtp.sx` adds the SXTP codec + Dream
bridge (39-test suite). Key representation calls, learned by probing the runtime:
keywords are strings at eval time but the `serialize` primitive renders
string-keyed dicts back as `{:k v}` and symbols bare — so messages are
string-keyed dicts with verb/status/type as symbols, and a small str-based
emitter produces wire-faithful list form. `parse` needs a deep normaliser
because parsed keyword tokens are a distinct type (not `=` to string literals).
`unquote-splicing` is unreliable here, so the serializer is str-based, not
quasiquote-based.
- **Next: Phase 3 — strangler migration ledger.** Enumerate the Quart endpoints
(use the `rose-ash-services` `svc_routes` MCP tool), track migrated vs proxied,
and stand up a golden-response harness against the live Quart responses. Then
cut over the smallest whole domain (`likes` or `relations`) as proof.
- **Phase 3 — ledger module (DONE, 107/107).** `lib/host/ledger.sx` + a 25-test
suite. Enumerated the endpoint surface via the `rose-ash-services` MCP
(`svc_routes`/`svc_queries`/`svc_actions`): `likes` and `relations` have **no
public blueprint routes** — they're internal-only, exposed as
`/internal/data/{query}` + `/internal/actions/{action}` (HMAC-signed). The
ledger is a pure-data catalogue keyed by (domain, method, path) carrying each
endpoint's Quart original, status, and serving SX handler; coverage reports the
off-Quart percentage. Cut-over target chosen: **`relations`** (already has a real
SX subsystem `lib/relations` — children/parents reads + relate/unrelate writes
map straight onto its public API); `likes` stays proxied (no SX lib to dispatch
to). NEXT: migrate the `relations` read endpoints onto host handlers (flip their
ledger status to `:migrated`) with golden tests.
- **Phase 3 — relations READ cut-over (DONE, 121/121).** `lib/host/relations.sx`
+ a 13-test golden suite; ledger flipped (off-Quart coverage 27% → 45%). The two
internal read queries (`get-children`, `get-parents`) now dispatch to the
`lib/relations` Datalog graph. Bridge: the Quart `(type, id)` node key maps to a
graph atom `(string->symbol "type:id")` with relation-type as the edge kind;
optional `child-type`/`parent-type` params filter the result list by `"type:"`
prefix (verified live: composite-string nodes round-trip through
`relations/relate` → `relations/children`). Golden discipline: `relations` is
internal-only (no public Quart route — confirmed via `svc_routes`), so the golden
is a **pinned fixture** (a known graph loaded in-test, asserted as
`subsystem-call + envelope`) rather than a live Quart capture. Reads are
unguarded for now — the signed-internal-auth gate is a separate middleware layer,
same as the feed reads. NEXT: relations WRITE actions (`relate`/`unrelate`)
behind the auth+ACL pipeline (mirroring POST /feed).
- **Phase 3 — relations WRITE cut-over (DONE, 132/132).** `lib/host/relations.sx`
gains `host/relations-attach`/`-detach` (`POST .../attach-child` + `/detach-child`)
and `host/relations-write-routes` — the write side of the container reads,
dispatching to `relations/relate`/`unrelate` over the same `"type:id"` node
model so an attach is immediately visible through `get-children`. Each runs
behind the host pipeline `wrap-errors ∘ require-auth ∘ require-permission`
(`"relate"`/`"unrelate"` on `"relations"`) — exactly the POST /feed stack. The
relations test suite proves the closed loop end-to-end: 401 unauth, 403 authed-
but-unpermitted (graph unchanged), 201 attach → child visible via the migrated
read → 200 detach → child gone; 400 on bad/short payloads. The ledger now models
the full relations surface (7 endpoints): container reads+writes `:migrated`,
typed `relate`/`unrelate`/`can-relate` `:proxied` (registry/cardinality
validation not in lib/relations). Off-Quart coverage 45% → **50%** (7/14).
`relations` is the first whole *coherent feature* (container relations) fully
off Quart. NEXT: golden-response harness vs live Quart, then survey the next
domain (blog/likes proxied — likes needs an SX subsystem first).
- **Phase 4 — live wiring bridge (DONE, 145/145).** `lib/host/server.sx` adapts the
native `http-listen` contract (string-keyed req `{"method" "path" "query"
"headers" "body"}` → `{:status :headers :body}`) to the Dream app: `host/-native
->dream` reassembles `path`+`query` into a target `dream-request` parses;
`host/-dream->native` is near-identity (dream-response is already `{:body
:headers :status}`). `host/serve port groups` = `http-listen` over
`host/native-handler (host/make-app groups)`. `lib/host/serve.sh` boots the full
module set (mirrors conformance) and serves in the foreground (container-entry
shaped). **Verified live** on a host port: `/health` 200 JSON, `/feed` recent-
first seeded activities, `/feed?actor=` filtered, relations `get-children`/`get-
parents` real JSON, unknown→404. Demo run was a standalone `sx_server.exe`
process (NOT the docker stack) — killed by its own PID, never `pkill` (siblings
share the binary). The standing "live wiring is a hosts/ change" Blocker is
resolved for the SX side: the bridge is pure SX in `lib/host`; only the *launch*
(docker stack + Caddy) remains. NEXT: golden harness, internal-HMAC, then promote
into the stack behind a fresh subdomain.
## SX gotchas + how this loop guards against them
The SX dev experience has real footguns. Most are statically detectable; the
tools exist (`sx_validate`, `deps-check`, `sx_format_check`) but must be *gated*.
Hit/relevant here:
- **Reserved-name shadowing** — `guard`/`bind`/`conj`/`disj` are special forms or
host primitives; a local binding of that name is silently shadowed by the form.
(`(let ((guard ...)))` made `(guard handler)` invoke the R7RS `guard` special
form → `first: expected list`.) Fix: namespace-prefix every helper
(`host/blog--protect`, never `guard`).
- **Silent test truncation** — a test file that errors mid-load returns only the
tests that ran before the error, reporting a FALSE GREEN ("blog 13 passed, 0
failed" while 16 CRUD tests never ran). **GUARDED**: `conformance.sh` now greps
the run output for `Undefined symbol` / `Unhandled exception` / `expected list,
got` / `[load] … error` and aborts loudly before the tally can hide it.
- **`let` is parallel** (bindings can't see each other), **bodies need `(do …)`**
(only the last expr evaluates), **`append!` no-ops on map/rest-derived lists**,
**parsed keyword tokens ≠ string literals**. These produce wrong *results*, so
test coverage catches them as red (not silent) — provided the runner is honest,
which the truncation guard now ensures.
Prevention ladder: parse (`sx_validate` after every edit) → unresolved/shadowed
symbols (`deps-check`, candidate pre-commit gate) → fail-loud runner (done) →
behavioural tests. A `deps-check`-style "binding shadows a special form" lint
would catch the reserved-name class before runtime — a worthwhile follow-up.
## ⚠ Experimental: unguarded create live on blog.rose-ash.com
`host/blog-open-create-routes` mounts **`POST /new` with NO auth** (create-only,
error-trapped) so the SX editor can publish end-to-end. **Validated live**: an
editor-style form POST → 303 → the post renders at `/<slug>/` and lists on `/`.
This is a deliberate, short-lived public write hole (create-only — no PUT/DELETE
exposed; obscure subdomain). **MUST be gated before real use** — Caddy basicauth
on `/new` (the `/root/caddy/auth` dir exists) or session auth once identity lands.
Swap `host/blog-open-create-routes` → `host/blog-write-routes <resolver>` to gate.
## Blockers
(loop fills this in)
- **Live wiring to the native OCaml HTTP server** (Phase 3/4): the prod server in
`hosts/` must hand SX handlers a `dream-request` dict and serialise the returned
`dream-response`. That is a `hosts/` change (out of scope for this loop, which is
`lib/host/**` only). Until then, endpoints are verified via `conformance.sh`, not
HTTP. Not blocking Phase 2 (middleware + SXTP + a write endpoint).
- **Worktree tooling:** in this `loops/host` worktree every sx-tree *write* tool
(`sx_write_file`, `sx_replace_node`, …) raises `yojson "Expected string, got
null"` at the MCP layer — same class as the `loops/dream` worktree gotcha, but
here even `sx_write_file` fails. Read-side sx-tree tools work. New `.sx` files
were created with the `Write` tool (the .sx hook is inactive in this worktree)
and each validated afterwards with `sx_validate` to keep the parse guarantee.
## Action item — serving-JIT speedup is NOT a code merge; it's a one-line flag flip
The ~2s interpreted-Smalltalk render (`/welcome/`, blog post pages) is being fixed
by the **`sx-vm-extensions`** loop — the JIT-bytecode-correctness handoff we kicked
off on 2026-06-19. **Do not wait for a code merge into `lib/host/**`** — the fix
lives entirely in the shared kernel (`hosts/ocaml/**`: `sx_server.ml`, `sx_vm.ml`,
extension modules) + shared guest runtimes (`lib/smalltalk/eval.sx`,
`lib/compiler.sx`, `lib/*/runtime.sx`). None of it is host code. The speedup is a
property of the shared `sx_server.exe` binary every loop already runs.
The serving-mode JIT is **gated behind `SX_SERVING_JIT`** (vm-ext commit
`bf298684`), and host's `serve.sh` / `conformance.sh` currently do **not** set it.
So host's entire adoption step is:
1. Wait for `sx-vm-extensions` → `architecture` (kernel + guest-runtime merge) and
the rebuilt shared binary. Watch its scoreboard: serving-JIT must be green across
ALL guest suites (Smalltalk, Datalog, Scheme, Haskell, Erlang, Prolog, APL, js)
with `SX_SERVING_JIT=1` — already done as of vm-ext `fed58b28` (js 148/148).
2. Gate locally: run `SX_SERVING_JIT=1 bash lib/host/conformance.sh` against the
rebuilt binary. Must stay green — this is the exact suite that first exposed the
miscompile (`router 3/6, feed 4/11, relations 9/16, blog 4/11` with the old JIT
on). If green, the residual exclusions in vm-ext covered host's workload.
3. Flip it on live: add `export SX_SERVING_JIT=1` to `lib/host/serve.sh` (the one
in-scope `lib/host/**` change). Commit as a feature. Live render should drop from
~2s to tens of ms — highest-leverage perf win on the platform.
Until step 1's binary is in, this is a no-op — leave `serve.sh` as is.

140
plans/host-spa.md Normal file
View File

@@ -0,0 +1,140 @@
# Host blog → SPA via the SX-htmx engine (WASM OCaml kernel)
## ✅ COMPLETE 2026-06-29 — live SPA on the WASM OCaml kernel
blog.rose-ash.com is now a single-page app: the browser boots the SAME OCaml
kernel the server runs (compiled to WASM), `sx-boost` fragment-swaps every link
into #content with URL push + working back button, no full reload. Verified:
native host conformance 271/271; `lib/host/playwright/spa-check` 4/4 in chromium;
LIVE blog.rose-ash.com boost 19/19 + click nav + zero errors.
The boot crash was the crypto stack assuming 63-bit int (fixed in `fce9e0c6`).
The boost then needed six more source-load/boost-path fixes (commit `689dae7d`):
import double-apply (library_loaded_p got a key not a spec), unloaded-import
crash (library_exports nil -> empty dict), value_to_js missing Integer (broke
dom-query-all -> only 1 link boosted), browser-same-origin? rejecting relative
URLs, dom-query-in undefined (= dom-query), and lazy-deps never preloaded under
source fallback (CEK can't lazy-resolve). Everything below is the history.
---
Turn the blog (lib/host/blog.sx) into a single-page app using the in-repo SX
hypermedia engine (web/engine.sx — "our htmx"): boot the **WASM OCaml kernel**
(the same evaluator the server runs) in the browser, and `sx-boost` every
link/form into a fragment swap into `#content` — no full reloads, history kept,
graceful degradation to plain server-rendered pages with no JS.
## Status
**DONE — server side (verified, all green):**
- `lib/host/static.sx` — `GET /static/**` serves files under `shared/static` via
the `file-read` primitive (content-type by extension, path-traversal guarded,
404 on missing). Mounted in serve.sh + the route list. Tested: kernel JS 200 +
correct ctype + exact bytes; `.wasm` binary-exact with `application/wasm`;
traversal/missing → 404.
- `lib/host/blog.sx` `host/blog--page` is now the SPA shell: full page = WASM boot
scripts (`/static/wasm/sx_browser.bc.wasm.js` + `sx-platform.js`) + a
`sx-boost="#content"` wrapper div + `#content`. On the `SX-Request: true` header
(a boosted nav) it returns ONLY the inner content (fragment) so the engine swaps
it into `#content`. All 13 page handlers thread `req`. Tested: full page carries
scripts+boost+#content; `SX-Request` returns the bare fragment.
- `docker-compose.dev-sx-host.yml` mounts `./shared/static` so the live container
can serve the kernel.
- `lib/host/playwright/spa-check.spec.js` + `run-spa-check.sh` — browser check
(boot, boost, fragment swap, back button).
**DONE — client side, partial:**
- The WASM kernel BOOTS in a headless browser: `globalThis.SxKernel` is an object,
`<html data-sx-ready="true">` is set, the web-stack modules load.
- Fixed: this worktree's `shared/static/wasm/sx_browser.bc.wasm.assets/` was
missing 5 of 11 `.wasm` units (`sx-`, `unix-`, `re-`, `start-`,
`dune__exe__Sx_browser-`); copied the complete set from the main worktree.
**BLOCKER — boost does not activate (`boosted links: 0 / N`):**
- The bundled `.sxbc` bytecode throws `VM: unknown opcode 0` against this
worktree's `sx_browser.bc.wasm.js` kernel, so sx-platform.js falls back to `.sx`
source for every web-stack module. Source fallback works for all modules EXCEPT
`boot.sx`, which then fails with `Expected list, got string` — so the boot
sequence that wires `process-elements → process-boosted` doesn't complete and no
link gets `_sxBoundboost`.
- Root cause: the `.sxbc` in `shared/static/wasm/sx/` are out of sync with the
WASM kernel (sx.rose-ash.com avoids this because its Docker image ships a
consistent bundle and it navigates via client-router page-routes, not boost).
## UPDATE 2026-06-29 — kernel BOOT crash fixed (crypto WASM-safe)
The boot crash was NOT the build pipeline — it was the kernel's crypto stack
assuming 63-bit native int. On the web targets (js_of_ocaml 32-bit, wasm_of_ocaml
31-bit) sha2/cbor/cid/ed25519 truncated, and ed25519 precomputes `sqrtm1` +
`base_point` AT MODULE INIT via a base-2^26 bignum whose 52-bit products overflow
→ `Char.chr(-4)` crash on load. Fixed in `fce9e0c6` (sx_sha2 Int32 rounds +
Int64 length, sx_cbor Int64 width-select, sx_cid bounded base32, sx_ed25519 Int64
bignum mul/div_small). Verified: NIST/CID vectors match native↔js↔wasm; native
conformance 271/271; **the freshly-built browser kernel now BOOTS** (SxKernel
live, data-sx-ready=true, crypto-sha256 correct on js + wasm).
REMAINING for boost (separate layer — web-stack loading, NOT crypto). Two
compounding roots, both fully diagnosed:
1. **`.sxbc` carry NIL bytecode.** `compile-modules.js` (via the native binary)
emits `:bytecode (nil nil nil …)` placeholders, not real bytecode — so the
SX-level `vm.sx` interpreter reads nil → `VM: unknown opcode 0`, and the web
stack falls back to `.sx` source for every module. (Confirmed by inspecting a
freshly-compiled `dom.sxbc`.) The native compiler isn't producing bytecode in
this path.
2. **Source-fallback can't resolve manifest-mapped libraries.** With imports
stripped, all 23 `boot.sx` body forms load clean — the `Expected list, got
string` is from an `import`. `boot.sx` imports `(sx signals-web)`, but that
library is *defined inside `signals.sx`* (the file→library names don't match;
the module-manifest maps `"sx signals-web" → signals.sxbc`). The `.sx`
source-fallback resolver maps a library to a like-named FILE, looks for a
non-existent `signals-web.sx`, and the failed resolution returns a string into
a list op → the error → `boot.sx` never loads → `process-boosted` never runs →
boost 0/N. (A `signals-web.sx` bridge that imports signals was NOT sufficient
— there is at least one more such mismatch among the imports.)
THE CLEAN FIX is a proper bundle rebuild via `scripts/sx-build-all.sh` so the
`.sxbc` carry real bytecode and the manifest-driven path loads everything (no
source fallback, so root #2 never triggers) — gated on fixing root #1 (why
`compile-modules.js` emits nil bytecode). Alternatively, make the source-fallback
resolver manifest-aware. Neither is a quick edit; it's a web-stack build-tooling
sub-project. The kernel itself is now correct and boots.
## Rebuild attempt (2026-06-28) — FAILED, reverted (superseded by the fix above)
Tried it: `dune build browser/sx_browser.bc.wasm.js` succeeded (with many
`integer-overflow` warnings — "generated code might be incorrect"), and
`node hosts/ocaml/browser/compile-modules.js shared/static/wasm` recompiled all
35 `.sxbc` cleanly. But the freshly-built kernel **crashes on init** in the
browser: `Fatal error: exception Invalid_argument("Char.chr")` — so `SxKernel`
never initialises (worse than before). The integer-overflow truncation during
wasm codegen is the likely culprit (a SHA/char constant). Reverted
`shared/static/wasm/` to the main-worktree bundle (which boots cleanly —
verified SxKernel + data-sx-ready). So a naive in-worktree rebuild is NOT the
fix; the wasm build itself needs investigating (wasm_of_ocaml version? the merged
sx-vm-extensions/resolver changes interacting with codegen?).
## Next step — rebuild a consistent WASM bundle
`scripts/sx-build-all.sh` does: build the browser wasm target → sync web `.sx`
into `hosts/ocaml/browser/dist/sx/` → `node hosts/ocaml/browser/compile-modules.js`
(recompiles `.sxbc` via the native sx_server binary) → copy into
`shared/static/wasm/`. The browser wasm target is NOT built in this worktree
(`hosts/ocaml/_build/default/browser/` is empty), so this needs the
`wasm_of_ocaml` toolchain set up first. Once the `.sxbc` match the kernel, the
bytecode path loads (no source fallback), `boot.sx` runs, and `process-boosted`
binds the links — then the SPA Playwright check should pass.
Alternatively: build the browser kernel in the main worktree (which has the
pipeline) and copy a consistent `sx_browser.bc.wasm.js` + assets + `.sxbc` set
into this worktree's `shared/static/wasm/`.
## Deploy note
The live container is NOT redeployed with the SPA shell yet — it keeps running the
pre-SPA `blog.sx` in memory (the native host doesn't hot-reload). Don't recreate
the container until the bundle is consistent and the SPA Playwright check is green,
to avoid shipping a kernel that boots but doesn't boost. (Even if it is recreated,
pages degrade gracefully: links still do normal full-page nav.)

394
plans/relations-as-posts.md Normal file
View File

@@ -0,0 +1,394 @@
# Relations as posts — declared, inherited, and eventually algebraic
## Principle
Everything is a post in one graph: content-posts, type-posts, **relation-posts**, and
(later) **constraint-posts**. Nothing about typing is hardcoded — a type-post *declares*
which relations it anchors, declarations are *inherited* down the type closure, and
every candidate set / validation is a transitive graph query (`lib/relations`). This
closes the meta-circular loop the typing plan gestured at: the type system describes
itself in its own graph.
Supersedes the hardcoded `:candidates "types"/"tags"/"all"` field of `host/blog-rel-kinds`.
## Content-addressability is universal (foundational)
**Every object carries a content-address (CID) — content-posts, type-posts, relation-posts,
constraint-posts, all of them.** A CID is the hash of the object's *canonical* form: a recursive,
**key-sorted** serialization (so insertion order, and any process-seed-dependent dict ordering, is
irrelevant — identical content always yields an identical CID). The runtime has no hash primitive,
so the canon serializer + a tail-recursive double-hash are built in SX (`host/blog--canon`,
`host/blog--cid-of`); the slug is excluded from the hash (it's a *name*, not content).
The model is **git-shaped**: the **slug is a mutable name → CID** (a branch pointing at a commit);
the **CID is the immutable content identity** (the commit). Editing a post mints a new CID; the slug
follows. Type evolution is the same — a type *version* is content-addressed, instances reference the
version they were created against. Two objects with identical content *are* the same object (same
CID) — correct content-addressing semantics.
**Why it's foundational (federation).** A CID is a **global, location-independent identity**, so:
- **Types flow across `fed-sx`.** The same type *definition* on any node has the same CID → a
**shared, content-addressed vocabulary**. Federated *instances* reference type CIDs, so a receiving
node can *interpret* them. This is linked-data/RDF realised on the post graph, and it generalises
ActivityPub itself: AP has a *fixed* type vocabulary (Note/Article/Person, Create/Follow/Like) —
the metamodel makes that vocabulary **extensible and user-defined**.
- **Structure / behaviour trust-split** (the federation boundary): type **structure** (schema,
relations, signatures) is declarative and federates *freely* — sharing a definition is sharing a
hash. **Behaviour** (Slice 9 lifecycles/effects) does **not** federate naively: you never run a
remote node's lifecycle with *your* effect primitives (their "ship" could `charge-card`).
Behaviour federates only under high trust, with the effects **re-bound** to local, audited
primitives (their orchestration, your effects). `fed-sx` is already trust-gated — that's the lever.
Build order: stamp a stable CID on every object first (additive — slug-addressing stays the working
key), then a `cid → slug` index, then migrate references / type versioning, then federation.
## North star — the metamodel as a system-construction kit
The destination this is all heading toward: the host stops being "a blog" and becomes a
**self-describing metamodel**. You *define a domain* — types (with schemas/refinements) and
relations (with role signatures + algebra) — and a working system falls out. The blog content
is one seeded configuration; clear it and define different types and you have a different system
on the same engine. Framework, not application (cf. `[[feedback_runtime_control]]`,
`[[project_zero_dependencies]]`).
Most of the **instance UI is already generic** — the edit page's relation editors are generated
by iterating the relations; each picker's candidates come from the relation's `declares`-anchor /
role type; validation comes from the type's `:schema`. So once Slices 6–7 land, "define the
types" through a UI is mostly two surfaces, plus a reset:
1. **Metamodel editor** — create a type-post (give it a schema/refinement); create a relation-post
(give it a role signature + algebra). The thing that lets you *construct* a system.
2. **Generic instance form** — create/edit any post of any type, driven entirely by the
definitions above (the relation editors + pickers + save-time validation we already have).
3. **Clear-and-reseed** — wipe instance data, seed only the metamodel roots (`type`, `relation`,
the core relations); start from a bare kit and build a domain up from nothing.
Sequence: finish the schema language (Slices 6–7) → the two UI surfaces + reset → clear the demo
data and define a real domain through the UI. The slices below are the schema language; this is
what it's *for*.
### Endgame — the whole platform as a typed domain (greenfield, not a strangler)
Not just the blog: the entire rose-ash platform — **store, events, orders, cart, …** — is
expressible as type + relation definitions in this one metamodel. `Product`, `Event`, `Order`,
`Ticket` are types; "cart has line-items", "order for an event", "ticket of an event" are
relations with signatures (cardinality = a cart has many line-items, a ticket belongs to one
event). This is NOT a strangler off Quart (`[[project_host_on_sx]]`) — it's a **greenfield,
SX-native system**: define the domain schema as data from first principles, then **port the data
once at the end** (define-then-port), rather than reimplementing each service's bespoke models
endpoint-by-endpoint. The strangler's compatibility machinery (JSON mirrors, route/model parity,
incremental contracts) is dropped — it was tax, not value, for a system that doesn't *correspond*
to the old one.
### SX all the way out — no JSON on the internal wire
The platform speaks **SX/SXTP end to end**, both directions, browser included — JSON survives only
at the ActivityPub federation edge (JSON-LD, a published external standard).
| Layer | SX-native form |
|-------|----------------|
| Page render | HTML (the document itself) |
| Data reads | `text/sx` via the `serialize` primitive (`host/ok`/`host/error` → `host/sx-status`) |
| Write bodies | `text/sx` parsed via `sxtp/parse` (was JSON / form-urlencoded) |
| Browser → server | the engine posts `text/sx` (boosted forms serialise fields to SX wire); form-urlencoded survives only as the **no-engine / pre-hydration fallback** + the **login bootstrap** handshake |
| Federation edge | JSON-LD (ActivityPub — the *only* JSON) |
The blog **JSON CRUD `/posts`** (POST/PUT/DELETE) is **deleted**, not converted: it was a pure
old-contract REST mirror; writes go through the HTML editor forms + SXTP.
Three honest additions store/events surface (the blog didn't need them):
1. **Typed scalar ATTRIBUTES, not just entity relations.** A `Product` needs `price: Money`,
`sku: String`, `stock: Int`; these are *values*, not edges to posts. We've built RDF
*object properties* (edges to resources); this needs *datatype properties* (literals with
value-types + validation). So a type declares **fields** `{field, value-type, card, required,
validation}` alongside relations; instances carry typed values; value-types (`Money`, `Int`,
`DateTime`) are primitive types. Same shape as a role — a role points at a *type*, a field
holds a *value-type*. **This is a real addition to a/b/c+d** and likely Slice 8.
2. **Behaviour / lifecycle** (order `pending→paid→shipped`) is NOT structure — it's the
substrate loops: `[[project_flow_on_sx]]` (durable workflows), `[[project_commerce_on_sx]]`,
`[[project_events_on_sx]]`. The metamodel *attaches behaviour to types by composing those*,
not reinventing them.
3. **Integrations** (SumUp payments, ActivityPub federation, artdag media) — types *reference*
these services; they don't dissolve into posts.
So the complete picture: the metamodel expresses **structure + validation** of the whole
platform's domain model uniformly; **behaviour composes from the substrate loops**;
**integrations stay referenced services**. It's the convergence point of every loop in the repo.
### Types define the UI — the editor maps onto the metamodel
The payoff of typed fields (Slice 8): **a type drives both sides of the UI from one definition.**
Beyond name + schema, a type carries **fields** `{name, value-type, widget}` and **templates**:
- **Fields drive the edit UI** — the editor renders one input per field, the widget chosen by the
field's `value-type` (`Date`→date-picker, `URL`→link input, `String`→text, `Image`→uploader).
- **Fields drive the render** — the type's **render template** (a parameterised SX template stored
on the type-post, instantiated with the instance's field-values) references those fields by name.
- An **instance** is then just *field-values* on a post. Add a field to the type → it appears in
the editor *and* the page, **no code touched**. Same definition, both surfaces.
**"kg-cards become types."** Each Koenig/Ghost card — image, gallery, callout, embed, bookmark,
heading — becomes a **type-post** with fields + a render template. We've already enumerated that
whole vocabulary: `[[project_content_on_sx]]` modelled heading/text/code/quote/image/embed/divider/
list/table/callout/media as block types — **that list is the seed set of card-types.** "The old
blog posts get typed" = migrate Ghost content into typed blocks, one type-post per block kind.
**"The editor maps onto the types."** The editor stops being hardcoded card handlers and becomes a
**generic field-editor**: given a type, emit an input per field; on save, store the values; render
through the type's template. A new card = a new type-post, **zero editor code — the editor is
defined by the metamodel.** Proof the pattern works: the edit page's relation-editors are already
*generated* from relation definitions, not hand-coded (one level up from fields).
Honest layer: the **render template is data** (editable, meta-circular); only the irreducible
**widgets** (the date-picker, the image-uploader) are platform pieces, and `value-type` is what
*selects* the widget — the same decidable-core / fenced-frontier line as everywhere else.
**The generic form is the default, not the ceiling — types can specify specialised editors.**
A UI doesn't just *fall out* of the types; it can be **customised**. A type may declare an
`:editor` slot — a registered, **content-addressed editor *component*** (a WYSIWYG for rich body,
a map picker for geo, a colour picker) that replaces or augments the input-per-field form, shipped
to the client by hash like `~relate-picker`. So the editing spectrum per type is: **generic
field-form** (data, free) → **per-field widget override** (`value-type`/`:widget`) → **whole
specialised editor component** (the escape hatch, e.g. WYSIWYG). The metamodel picks the level per
type — `:editor` if set, else the generic form. Same decidable-core / fenced-frontier shape: the
declarative form covers the 95%, a code component handles the cases that need real interaction.
**Refined build order** (this is what `/meta` is the on-ramp to):
1. `/meta` overview — **DONE + LIVE** (the *see*; `host/blog-type-defs` + `host/blog-meta-index`).
2. **Slice 8 — typed fields** `{name, value-type, widget}` — the keystone — **DONE + LIVE**.
3. **Generic instance form** — input per field ("the editor maps onto types") — **DONE + LIVE**.
4. **Render template per type** (8c) — data, `(field "name")` placeholders — **DONE + LIVE**.
5. **Cards-as-types + migrate** — seed the card-type vocabulary from content-on-sx; type the old posts — NEXT.
Editor surfaces on `/meta`: **create-type** (`POST /meta/new-type`) — **DONE + LIVE**; **create-relation**
(`POST /meta/new-relation`) — **DONE, but SESSION-SCOPED**: the relation-post + edges persist, the
rel-kinds registry entry is a runtime concat lost on restart (boot loader can't dynamically enumerate
under JIT-at-boot — the kernel boot-resolver gap, flagged to sx-vm-extensions). Then **clear-and-reseed**.
Also open: **specialised editors** (`:editor` slot → content-addressed component, e.g. WYSIWYG).
## Behaviour as data — lifecycles + ECA over an effect vocabulary (DESIGN — Slice 9)
Structure is inert; "place an order / ship goods" is the dynamic part. The principle:
**behaviour is data-defined orchestration over a small fixed vocabulary of effects.** Only two
layers stay code — the **effect primitives** (the irreducible ops that touch the world) and the
**interpreter** that runs the data. Everything between is editable posts. The system defines its
own behaviour down to the effect boundary (`[[feedback_runtime_control]]`).
**Shape.** A type declares a **lifecycle** (a state machine) as data, plus standalone **ECA
rules** for reactions that aren't state transitions:
```
Order: cart --place--> placed [guard: stock-available ∧ total>0] [effects: reserve-stock]
placed --pay--> paid [guard: payment-ok] [effects: charge-card, confirm-stock]
paid --ship--> shipped [guard: address-valid] [effects: create-shipment, notify]
ECA: when stock(product) < threshold => notify(buyer:owner, "restock")
```
- **States/transitions/rules/effect-invocations are all posts** — meta-circular: `Lifecycle`,
`Transition`, `Rule`, `Effect` are themselves types in the metamodel; a behaviour is instances
you edit in the same UI as the schema. A transition = `{from, to, on-event, guard, [effects]}`.
- **Guards are PURE** — predicates over the instance's attributes/relations, i.e. type-system
queries (Datalog). No side effects, analysable, you can diagram a lifecycle.
- **Runs on `[[project_flow_on_sx]]`** because it's durable + long-running: `placed→paid` waits
for a SumUp webhook, `paid→shipped` waits days. flow's suspend/resume IS this. Failures →
compensation (saga) — `commerce-on-sx` already does "refund as a flow".
- "Place an order" / "ship" = *attempt transition T*; the button/webhook just fires the event.
### The effect vocabulary (sketch — store + events)
An effect is a named, parameterised op (itself an `Effect` post: name + params + binding).
Behaviours reference effects by name with args bound to instance/context. Four tiers:
| Tier | Effect | Notes |
|------|--------|-------|
| **Pure guard** (read-only, not an effect) | `is-a? / attr-cmp / count / relation-exists?` | type-system queries (Datalog); compose the transition guards |
| **Data** (internal, transactional on the graph) | `create(type, attrs)`, `set-attr`, `set-state`, `relate / unrelate`, `incr / decr`, `append-ledger(entry)` | the durable post-graph mutations; `decr` stock is atomic-with-check |
| **Domain** (composed from data, named for atomicity/meaning) | `reserve-stock`, `release-stock`, `confirm-reservation`, `book-seat`, `issue-ticket` | small compositions the vocabulary blesses; `events-on-sx` has the capacity-safe versions |
| **Integration** (external services — the code edge) | `charge-card`, `refund` (SumUp), `create-shipment` / `track`, `notify(recipient, template, data)`, `federate(activity)` (ActivityPub), `process-media(asset)` (artdag) | the irreducible primitives; keep this list SMALL and composable (artdag's S-expression effects is the model) |
| **Control** (durable orchestration — flow primitives) | `wait-for(event)`, `wait-until(time) / after(dur)`, `emit(event)`, `transition(instance, state)` | `wait-for` = the SumUp webhook / shipment-delivered; `after` = reservation-expiry / event-reminder; `emit` chains ECA rules |
So `place order` = guard `stock-available ∧ total>0` → effects `reserve-stock`, `set-state placed`,
`emit order-placed`; the webhook later fires `pay` → `charge-card`, `confirm-reservation`,
`set-state paid`. Events reuse the same machinery: ticket `reserved →(after 15m, no pay)→ released`,
event `--remind(after)--> notify` digests. Almost all of it is the same vocabulary.
### The one fork (same shape as the type-system line)
- **Declarative core** — lifecycles + ECA + the effect vocabulary: safe, analysable, diagrammable,
editable by non-programmers, verifiable. Covers ~95%.
- **Guarded code escape-hatch** — a `Scheme`/`Smalltalk` snippet stored on a post and `eval`'d for
the rare bespoke guard/effect (`[[project_content_on_sx]]` is Smalltalk message-passing,
`[[project_flow_on_sx]]` is guest Scheme — the homoiconic door exists). Turing-complete, unsafe,
fenced — exactly the decidable-core / fenced-frontier split we drew for types.
**Where to start:** pin down the effect vocabulary above (the real design artifact), build the
generic interpreter on flow-on-sx with pure (Datalog) guards, and **lift `commerce-on-sx` /
`events-on-sx` from guest-code into lifecycle+effect DATA** — they already implement exactly this,
just not editably.
## Why (the wrinkle that started this)
Candidates for `is-a`/`subtype-of` were `instances-of("type")` — the *instances* that are
types, but NOT the type-defining posts themselves (`type`, `tag`, `article` are wired with
`subtype-of`, no `is-a` edge, so they're not instances of type). So the picker offered
`tutorial` (is-a tag) but never `tag`/`article`/`type` — the things you most want to say a
post *is-a*. The fix is to ask the right question: a candidate is anything that **inherited
the relation's object-end declaration from the anchor**, which includes the roots.
## Model
- A **declaration** is an edge `T --declares--> R`: type-post `T` anchors relation `R` at
its **object** end ("you may point *at* `T` with `R`"). Seed: `type declares is-a`,
`type declares subtype-of`, `tag declares tagged`. `related` has no declaration.
- **Candidate set** for relating under `R` = the **down-closure** of `R`'s anchors through
`inverse(is-a) ∪ inverse(subtype-of)` (a post is a candidate iff it is, transitively, an
instance-or-subtype of an anchor — or IS one). No anchors ⇒ every post (`related`).
- `is-a`/`subtype-of`: anchors `{type}` ⇒ the whole type closure (roots + subtypes +
instances). **Wrinkle fixed.**
- `tagged`: anchors `{tag}` ⇒ the tags.
- `related`: no anchor ⇒ all posts.
## Roadmap
### Slice 1 — declarations + candidate-by-inheritance — DONE
- Seed `declares` edges; add `host/blog--reach-down` (down-closure) and rewire
`host/blog--candidate-pool` to be declaration-driven. `:candidates` becomes vestigial.
- Wrinkle fixed: the type roots now appear as `is-a` candidates.
### Slice 2 — relations as first-class posts — DONE
- `relation` root + `is-a`/`subtype-of`/`tagged`/`related` seeded as posts (each is-a
relation) owning their metadata in a `:rel` slot (`:symmetric :label :inverse-label`).
`host/blog-rel-kinds` / `kind-spec` / `kind-symmetric?` now read it; the static registry
is gone. `host/blog--rel-slugs` = `host/blog-in "relation" "is-a"` (cheap, flat).
- **Perform budget under http-listen (the hard lesson):** a durable read inside the
render VM raises `VmSuspended`, and too many per request 500s the page. Two fixes:
(1) relation metadata is loaded into an in-memory cache at boot (`host/blog-load-rel-kinds!`,
like `load-edges!`) so `kind-spec` is pure; (2) the initial edit page renders its pickers
EMPTY (the load trigger fills each) — only the relate/unrelate FRAGMENT server-renders
candidates (`with-cands` flag), so one page render doesn't do `candidate-get × every
picker`. Benign single-perform suspend/resume still logs `VmSuspended` but returns 200.
- **Live JIT gotcha (cost real time):** the serving-mode JIT drops all-but-first when
`map`/`for-each`-ing a *function-produced* list — building `rel-kinds` that way rendered
only 1 of 4 editors live, while conformance + the ephemeral server passed. So
`host/blog-rel-kinds` is a VALUE the boot populates and the cache loads are UNROLLED.
**Conformance green ≠ correct live — verify the rendered edit page.** (Re-fold the
enumeration once plans/jit-bytecode-correctness.md lands.)
### Slice 2.5 — picker title reads are O(page), not O(pool) — DONE
- `relate-candidates` computes the available candidate SLUGS (slug-sorted, no per-candidate
read), then reads titles ONLY for the page it returns. On the unfiltered path (q="" — the
initial picker load AND every editor server-fill, the common case) that's ~`limit` reads
instead of one-per-post — killing the durable-read churn under http-listen. A filter
(q≠"") still resolves titles across the pool (it matches on the title), but that's the
interactive path.
- A boot-time slug→title **cache** would make even the filter O(1)-perform, BUT it's blocked
for now: there's no bulk KV read, and a per-post `host/blog-get` loop **at boot** hits the
JIT bug (a durable read inside a boot loop drops all-but-first — `load-edges!` only works
because its loop body is perform-free). Revisit with a bulk read or once the JIT lands.
**Remaining follow-ups:** subject-end declarations (who may be the *source*); a proper
relation-subtype closure when relations get subtyped; the boot title cache above.
### Slice 3 — typed relations (target-type constraints) — DONE
- The declaration's `declares`-anchor IS the target-type constraint: `is-a`/`subtype-of`
(anchored by `type`) require a type object; `tagged` (anchored by `tag`) a tag. A new
`wrote` relation needs only a `Work declares wrote` edge — fully data-driven.
- `host/blog--valid-object?(kind, other)` = `other ∈ candidate-pool(kind)` — the SAME set
the picker offers, so picker and validation agree by construction. `relate-submit` now
enforces it (an invalid target is a silent no-op, like the other guards); `related`
(no anchor) accepts any post. The picker never offers an invalid target, so this guards
crafted/API requests — the jump from "candidate set" to an enforced relation schema.
- NOTE: `host/blog-relate!` (direct/seed) stays UNVALIDATED — the seed needs to write
`X is-a relation` where `relation` isn't under `type`. Validation is a *handler* boundary.
### Slice 4 — type algebra — DONE (intersection ∧ union)
- An algebraic type is a post with operand edges: `conj` edges (intersection members),
`disj` edges (union members). `host/blog-instances-of-expr` computes its EXTENT from the
operands' extents by set intersection / union, RECURSIVELY — so operands can themselves be
algebraic (meta-circular; tested with `(tag ∧ article) ∧ tag`). `host/blog-is-a-expr?`
generalises `is-a?` to type expressions. `host/blog-make-and!` / `make-or!` build them.
- Binary today (`nth 0/1`, no fold over operands — robust on the serving JIT); n-ary fold is
a follow-up once iteration-with-perform is JIT-reliable.
- **Operand edges are KV-only** (`host/blog--add-edge-kv!`, read via `host/blog-out`), NOT in
lib/relations — feeding extra kinds into the Datalog graph blows up its per-query
re-saturation; `load-edges!` skips `conj`/`disj` on replay for the same reason.
- **Refinement** `{x : T | φ(x)}` (a type-post with a `:constraint` predicate) → Slice 5,
with constraints-as-posts. (Process note: a sibling loop running heavy conformance saturates
the box; host conformance can EXIT 124 purely from CPU contention — use `timeout 1200`.)
### Slice 5 — refinement types (schemas ON the type-post) — DONE
- A type-post carries its schema in a `:schema` slot (a list of `{:block :msg}` rules —
a refinement `{x : T | x has these blocks}`). `host/blog-schema-of` reads it off the
post; the hardcoded `host/blog-type-schemas` table is gone. A NEW refinement type is pure
data: give a type-post a `:schema` (`host/blog--set-schema!`) and its instances are
validated on save against it — no code. Tested with a `guide` type requiring a `pre` block.
- Save-time validation (`type-issues`/`type-valid?`, the only callers, in the SAVE request)
unions the schemas of a post's full transitive type set — unchanged, just sourced from the
posts. `schema-of` reads the post (a durable read) — fine in the save request, never render.
- `host/blog-put!` now MERGES over the previous record, so editing a post's title/content
doesn't nuke its `:schema` / `:rel` metadata (also closes the Slice 2 "edit drops :rel" gap).
- `article`'s schema migrated onto the article post (`set-schema!` at boot — a single
read+write, not a loop, so boot-JIT-safe; idempotent, handles the already-seeded article).
- FUTURE: arbitrary predicate constraints (not just required blocks); constraints as their
own posts; relation cardinality (`is-a` single-valued?) as a declared constraint.
## Parameterised relations (DESIGN — Slices 6 & 7)
The next axis: `Relation<…>`. The key reframe is that the obvious parameters aren't separate
`<N>`s — they split into **two halves**, and they compose into one coherent thing:
1. **The role SIGNATURE** (the *shape* of a tuple) — Slice 6 (a + b + c).
2. **The relation's ALGEBRA** (how it *behaves*) — Slice 7 (d).
A relation is `Relation<signature>`, where a signature is an ordered list of **roles**, each
role carrying a **type** and a **cardinality**; the signature's length is the **arity**.
Today's binary typed relations are the degenerate 2-role case — backward-compatible, nothing
gets thrown away. Prior art to borrow (and stay decidable within): Codd / ER reified
relationships (signature), OWL property characteristics (algebra), Datalog / relation algebra
(derived relations — the undecidable frontier; fence it). Decidability rule of thumb: concrete
+ algebraic role-types and counts stay decidable; arbitrary predicates / recursive rules don't.
### Slice 6 — the role signature (a + b + c)
Generalise the relation-post's `:rel` slot from `{:symmetric :label}` to a `:roles` list —
`{:roles [{:name :type :card} …]}` — driving picker candidates, validation, and arity per-role:
- **(a) per-role type** — each role's `:type` is a type-expr (so it can be algebraic:
`Relation<Work ∧ Published>`). The object-role's type IS today's `declares`-anchor — make it
explicit. `valid-object?` becomes per-role `is-a-expr?` against `:type`.
- **(b) arity** = `(len roles)`. Binary stays the fast `src|kind|dst` edge path; **n-ary needs
reification**: a relation *instance* becomes its own post with role edges (`subject→X`,
`object→Y`, `recipient→Z`) — on-brand (we made relation *kinds* posts; now *instances* too),
but a SECOND representation alongside the binary edges, not a tweak. Qualifiers (Wikidata-
style) then come free as extra roles.
- **(c) cardinality** — `:card` per role (min/max; functional = max 1, required = min 1),
enforced on relate by counting. Composes with Slice 5 validation. No model change for binary.
- Siblings: ordered roles (set vs list), keys/identity (which roles identify a tuple).
- **Layering (cheapest → deepest):** (c) cardinality on the binary object-role → (a) explicit
role-type + the 2-role signature abstraction → (b) reified n-ary (the real lift).
- **Variance: nominal, none initially** — no structural subtyping of `Relation<…>` (covariance
of parameterised types is a research project). JIT caveat: 2-role signatures are unrollable;
n-ary role-iteration with per-role reads needs the cache/unroll treatment (Slice 2/5 lesson).
### Slice 7 — relation algebra / characteristics (d)
The behaviour half — and (d) **transitivity** is special because we ALREADY hardcode it
(`is-a`/`subtype-of` closure via lib/relations); declaring it generically *removes* code.
- **Algebraic properties** declared on the relation-post (`:transitive :symmetric :reflexive
:antisymmetric :irreflexive`), with the closure **derived generically** from them — OWL's
property characteristics. `subtype-of` becomes "a declared transitive + antisymmetric
relation" (a partial order), not a special case. `:symmetric` (already stored) folds in here.
- **Inverse relations** — a real `:inverse` (not just the `:inverse-label` display hint):
relating one auto-derives the converse, the way `:symmetric` writes both directions.
- **Sub-relations** — relations subtyping relations (`wrote subPropertyOf created`): X wrote Y
⟹ X created Y. Same `subtype-of` machinery, over the `relation` root — meta-circular.
- **Decidable core stops here.** Beyond-d, FENCED: defined-by-rule relations (composition,
`grandparent = parent ∘ parent` — straight onto the Datalog substrate, but gate to
stratified/bounded rules) and cross-role refinement predicates (`start < end`) — both need
the predicate-language-vs-embedded-code decision first.
## Open design questions (track as we go)
1. **Subject-end declarations** — who may be the *source* of a relation (a root `Thing`?).
2. **Inheritance path** — through `is-a` AND `subtype-of` downward (current choice); revisit
if instances-of-instances as candidates surprises.
3. **Bootstrap / meta-circularity** — `is-a` needs `is-a`; seed relation-posts + `Type is-a
Type`(?) idempotently, as the type seed already is.
4. **Cost** — `reach-down` is a BFS of direct-edge scans; fine for a small blog, revisit with
a `lib/relations` transitive query if the graph grows.

185
plans/sx-native-engine-tests.md Normal file
View File

@@ -0,0 +1,185 @@
# Plan: SX-native engine tests (browser-independent)
## Goal
Move the host's *interactive* test coverage from Playwright (`.spec.js`, drives a real
Chromium) into **SX harness tests** that drive the hypermedia engine against a **mock
platform** — no browser. Reserve Playwright for the one irreducible real-browser fact:
"the WASM kernel actually compiles, boots, and loads modules content-addressed."
**Why (the principle):** the SX engine (`web/engine.sx` + `web/orchestration.sx`) has no
hard browser dependency — it talks to a *platform* (fetch, DOM ops, timers) that is
injected. The harness supplies a mock platform, so engine behaviour (fetch → swap →
DOM mutation) is asserted with zero browser. The same engine could therefore drive
*something else* (a server-side DOM, a native UI) — the SX tests prove that
independence by running without one. This is consistent with
`[[project_zero_dependencies]]` and `[[feedback_runtime_control]]` (build IN the runtime).
## Current state (2026-06-29)
- **Already SX:** the 272 host conformance tests (`lib/host/tests/*.sx`, `spec/harness.sx`
mock-IO). The picker's *server contract* is SX too (`lib/host/tests/blog.sx`:
`picker form declaratively wired`, `load-more sentinel`, `no-sentinel-on-short-page`).
- **Still Playwright (`.spec.js`):** `lib/host/playwright/relate-picker.spec.js` (7 tests)
and `spa-check.spec.js` (4) — real-browser checks of populate / filter / paging /
relate-delete / remove-button / boosted-nav / error-retry / WASM boot.
## Infrastructure that already exists (the enabler — verified)
- `spec/harness.sx` — `make-harness`, `default-platform` with **`:fetch` overridable**
(`(fn (url &rest opts) {:status 200 :body "" :ok true})`), plus DOM ops, `:now`, etc.
- `web/harness-web.sx` — `(define-library (sx harness-web))` exports: `mock-element`,
`mock-set-attr!`, `mock-append-child!`, `mock-get-attr`, `mock-add-listener!`,
**`simulate-click` / `simulate-input` / `simulate-event`**, `assert-text`, `assert-attr`,
`assert-class`, `assert-no-class`, `assert-child-count`, `assert-event-fired`,
`make-web-harness`, render-audit helpers.
- `web/tests/` — existing SX engine tests: `test-orchestration.sx` (17 deftests),
`test-forms.sx` (25), `test-swap-integration.sx` (43, mock-response → swap → assert),
`test-engine.sx`, `test-handlers.sx`. **`test-swap-integration.sx` is the reference
pattern** (it sets `_mock-body`/`_mock-headers`/`_mock-content-type`, drives a swap,
asserts the result).
- Runner: `hosts/ocaml/bin/run_tests.ml` scans `spec/tests/`, `lib/tests/`, `web/tests/`
and loads `harness-web.sx` + `harness-reactive.sx`. Run via the `sx_test host="ocaml"`
MCP tool (or `./scripts/sx-build-all.sh`). JS runner: `hosts/javascript/run_tests.js`
also loads the web harnesses.
## Phases
### Phase 0 — Proof of concept (small): one behavior, SX
Port **relate → delete row** to an SX harness test (new `web/tests/test-relate-picker.sx`):
1. Build a mock DOM: a `.rp-results` `<ul>` containing one candidate `<li id="cand-related-x">`
with the relate `<form sx-post=/x/relate sx-target=#cand-related-x sx-swap=delete>`.
2. `process-elements` (or `bind-triggers`) the tree so the form's submit is bound.
3. Mock `:fetch` to return `{:status 200 :ok true :body ""}`.
4. `simulate-click` the button (or `simulate-event` "submit" on the form).
5. Assert the `<li>` is gone (`assert-child-count` results = 0).
This validates the **mock-DOM → execute-request → swap-dom-nodes** loop in SX end to end.
**If it reads cleanly, the rest is mechanical.**
### Phase 1 — Port the picker's interactive behaviors (medium)
Same file, more deftests, each = mock fetch + simulate + assert:
- **filter narrows**: `:fetch` returns N candidate rows for `q=...`; `simulate-input` the
filter; assert child-count == N.
- **sentinel paging**: `:fetch` returns rows + a `<li class=rp-more sx-trigger=revealed>`;
fire the revealed/intersect path; assert more rows appended, sentinel replaced.
- **load populate**: `load` trigger → fetch → assert results filled.
- **error/retry visible state**: `:fetch` rejects → assert `.sx-error` class added
(`assert-class`), then succeeds → assert cleared.
### Phase 2 — Trim Playwright to a boot smoke (small)
Keep ONLY what needs a real browser in `relate-picker.spec.js` / `spa-check.spec.js`:
- WASM kernel compiles + boots (`data-sx-ready`).
- modules load **content-addressed** (`/sx/h/` fetches, 0 path `.sxbc`).
- one boosted nav swaps `#content`.
Delete the per-behavior browser tests now covered by SX. Net: ~2 browser tests + an
SX suite.
### Phase 3 — The engine drives the CONSOLE (the non-browser target)
The concrete "something else" is a **terminal / console platform**. This is the natural
sibling of the test harness: a harness test *asserts* the engine's output tree; the
console platform *renders* that same tree to text. Same platform abstraction — one
observes it, one draws it.
What it means concretely:
- **Platform ops → a console-backed element tree.** The engine only ever calls platform
primitives: `dom-create-element`, `dom-append`, `dom-set-attr`, `dom-query` (by id, for
`sx-target`), `dom-remove-child`, `dom-parent`, `morph-children`, `dom-listen`, `fetch`,
`set-timeout`. Implement these against an in-memory tree of text nodes instead of the
browser DOM. The mock DOM in `web/harness-web.sx` is ~90% of this already.
- **Render = print the tree as text** (ANSI/box-drawing) — a `render-to-console` mode
alongside `render-to-html` / `render-to-dom` (see `spec/render.sx`'s mode table). The
results `<ul>` becomes a list; `.sx-error` becomes a red line; the filter input is a
text field.
- **Events = a TUI input loop.** Keypresses / selection map to `simulate-input` /
`simulate-click` on the focused node — exactly the harness's `simulate-*`, but driven by
a real keyboard instead of a test.
- **`fetch` stays HTTP** (the host already serves `text/sx` fragments + `relate-options`),
or talks to a local store.
Payoff: the **same** `~relate-picker` — `sx-get`, debounced filter, `revealed` paging,
`sx-swap=delete`, `sx-error` retry — runs unchanged in a terminal. That is the proof that
the SX hypermedia engine is a *general* runtime, not a browser library: the browser is
just one platform binding, the console is another, the test harness is a third. Ambitious,
buildable, and the most convincing demonstration of the whole architecture
(`[[feedback_runtime_control]]`, `[[project_zero_dependencies]]`).
Sketch of work: (1) a `console-platform.sx` implementing the platform ops over a text
tree (fork `harness-web.sx`'s mock element), (2) a `render-to-console` mode in render.sx,
(3) a tiny input loop (raw-mode stdin → focus model → `simulate-*`), (4) run the host's
picker against it. Phase 1's SX tests become the regression suite for the console renderer
for free (they already drive the tree, just don't print it).
## Gaps & risks to resolve during Phase 0
- **Mock-DOM completeness:** `swap-dom-nodes` uses `morph-children`, `dom-replace-child`,
`dom-insert-after/before/prepend/append`, `dom-remove-child`, `dom-parent`,
`dom-first-child`, `dom-clone`, `dom-is-fragment?`. Confirm `harness-web`'s mock DOM
implements (or can be extended for) these. `test-swap-integration.sx` already swaps, so
most exist; check `delete`/`outerHTML`/fragment paths specifically.
- **fetch callback shape:** the engine's `fetch-request` calls back
`(resp-ok status get-header text)`; the platform `:fetch` returns `{:status :body :ok}`.
Confirm/adapt the bridge (see how `test-swap-integration.sx` feeds `_mock-body` etc.).
- **trigger binding without a browser:** `simulate-click` fires bound listeners — the form
must be processed first (`process-elements` on the mock root, or bind directly).
- **component expansion:** `~relate-picker` need not be expanded for these tests — assert
on the *rendered* candidate rows / form markup directly (build the mock DOM from the
expanded HTML the server produces, which is already SX-testable server-side).
## Tracked loose ends (separate from this plan)
- **unrelate "clever" in-place delete** (just-the-row, no `#content` re-render): now that
`bind-boost-form` is fixed the remove button works via a boosted POST→swap; the
minimal-mutation version (sx-post + `sx-swap=delete` on the current-row) is a further
refinement — earlier attempt didn't fire, revisit with the binding now understood.
- **`hs-repeat-times`** bytecode test (architecture worktree): harness `host-new` stub bug
masks a pre-existing `beingTold` resume-env bug. See the diagnosis in this session.
## Progress (2026-06-29)
- **Phase 0 DONE** (commit 297bdc60) — `web/tests/test-relate-picker.sx`: relate→delete
row drives the real engine (process-elements → submit → mock fetch → delete swap)
against the OCaml runner's mock DOM, green. Mock-DOM completeness added to
`run_tests.ml`: `NodeList.item(i)` (so `dom-query-all` iterates) + a `DOMParser`
mock (so the empty-body `sx-swap=delete` HTML-response path works as in a browser).
- **Phase 1 DONE** (commit fe2da2d3) — same file, load / filter / paging / error-retry,
5/5 green, zero harness noise. Modelled two browser natives the OCaml runner lacks:
`observe-intersection` (a recording stub the test fires to simulate the sentinel
scrolling into view) and synchronous-timer retry (stripped in the error test —
backoff math is a `test-engine.sx` concern). Mock-DOM: `firstChild`/`lastChild`
(so `children-to-fragment` drains a parsed fragment into innerHTML/outerHTML swaps;
also repaired one pre-existing web test). No web-suite regressions.
- **Key seam discovered:** a top-level `(define …)` override is seen by engine
library functions ONLY when the symbol lives in a *different* library than the
caller (cross-library late-binds through global; same-library resolves locally).
`fetch-request` (boot-helpers) overrides fine from a test; `handle-retry`
(orchestration, same lib as `do-fetch`) does NOT — hence the strip-attr approach.
- **harness-web.sx is NOT loaded** by the OCaml runner (only the JS runner), and its
assertions assume a different mock-element shape (`attrs`/`text`) than the OCaml
mock DOM (`attributes`/`textContent`). Assert through the engine's own `dom-*`
accessors instead.
- **Phase 2 DONE** (commit 98ff7a35) — Playwright trimmed 11 → 5 tests, both ephemeral
suites green (run-spa-check 3/3, run-picker-check 2/2). Kept: WASM boot +
content-addressed module loading (new `/sx/h/` assertion) + boosted nav swap +
back/re-boost (spa-check); bind-boost-form remove button + picker re-bind after a
boosted SPA nav (relate-picker). Deleted the populate/filter/paging/relate-delete/
error-retry browser tests (now SX).
- **Phase 3 (stretch) — render slice DONE** (commit 16f90ffd) — `web/console-render.sx`:
`render-to-console` walks a live DOM element tree through the engine's own `dom-*`
accessors and prints it as terminal text (results `<ul>` → bulleted list, filter
`<input>` → text field, `.rp-more` sentinel → `…` line, `.sx-error` → flagged line).
Wired into the picker's engine tests so the SAME tree drives both the DOM assertion
and the terminal output — Phase 1's suite is the console renderer's regression suite
for free. Plus a `relate-picker:console` suite. 7/7 green.
- **Remaining Phase 3 (future):** the live input loop — raw-mode stdin → focus model
→ `simulate-input`/`simulate-click` on the focused node — and full ANSI/box-drawing
output. Not harness-testable (needs a real TTY), so it's a runtime/demo feature, not
a test. The render step (the convincing half — "render = print the tree") is done;
the engine→console *event* path reuses the same `simulate-*` the harness already
drives. Class membership must read the live `classList` (`dom-has-class?`), not the
static `class` attribute (the engine mutates classes through classList).
## Done-when
- [x] `web/tests/test-relate-picker.sx` covers populate / filter / paging / relate-delete /
error-retry in SX, green under `sx_test host="ocaml"`.
- [x] Playwright trimmed to the boot smoke; suite still green.
- [~] (Stretch) the picker runs through a non-browser platform — render-to-console done
(the engine's tree prints to a terminal); live TTY input loop is future work.

96
plans/typed-posts-and-relations.md Normal file
View File

@@ -0,0 +1,96 @@
# Typed posts & relations — typing is just relating to a type
> host-on-sx. Driving idea: **classification is a relation to a type node, and
> types are posts.** Everything (related, tag, category, series, type) becomes a
> typed edge in `lib/relations` over `blog:<slug>` nodes. One primitive.
## Decisions
- **Types are posts.** No new node namespace — content-posts and type/tag posts
are all `blog:<slug>`. A "tag" is a post; tagging documents itself.
- **`is-a` is the typing edge; `tagged` is membership.** Kept distinct so a tag
page can list members without conflating "ocaml is a tag" with "hello is
tagged ocaml".
- **Hierarchy is core, not deferred.** `is-a`/`subtype-of` transitive closure via
`lib/relations` reachability is what makes typing-as-relation more than flat
labels. All typing helpers are transitive from the first line, or subtypes
silently break candidate/`is-a?` checks later.
- **Validation is gradual, not deferred.** A type-post *optionally* carries a
schema slot; validation runs only where one exists. Tags declare none (stay
folksonomy-free); `article` can declare "needs a heading". The hook lands with
the type phase (reusing `host/blog-content-ok?`); only schema *expressiveness*
grows over time. This closes the nominal/structural loop: the declared `is-a`
edge is a claim, the validator checks the content honors it.
- **Scalars stay fields.** `status`/`title`/`sx_content` remain fields, not edges
— listings filter on them constantly and `lib/relations` re-saturates Datalog
per query. Links-to-shared-nodes → edges; per-post hot scalars → fields.
## The linchpin: a relation-kind registry
One data structure drives validation, the picker candidate sets, and rendering:
```
host/blog-rel-kinds =
({:kind "related" :label "Related posts" :symmetric true :candidates "all"}
{:kind "is-a" :label "Types" :symmetric false :candidates "types"
:inverse-label "Instances"}
{:kind "tagged" :label "Tags" :symmetric false :candidates "tags"
:inverse-label "Tagged with this"})
```
`:symmetric` → write both directions on relate. `:candidates` → what the picker
offers (`all` = every post; `tags` = `is-a? blog:tag` transitively; `types` =
`is-a? blog:type`). `:label`/`:inverse-label` → headings.
## Phases
### Phase 1 — Kind generalization + registry ← START HERE
Pure refactor; zero user-visible change (related keeps working).
- `host/blog-rel-kinds` registry + `host/blog--kind-spec`/`--kind-symmetric?`.
- `host/blog-relate!(a,b,kind)` / `unrelate!(a,b,kind)` — directed; symmetric kinds
also write the reverse (today's "related" behavior = the symmetric case).
- `host/blog-out(slug,kind)` (children) / `host/blog-in(slug,kind)` (parents),
existence-filtered. `host/blog-related(slug)` = `out(slug,"related")` (back-compat).
- Routes carry `kind` (form field, default `"related"`); validated against registry.
- `delete` cleanup drops edges across **all** kinds, both directions.
### Phase 2 — Type resolution via reachability (the spine)
- Seed root type-posts: `blog:type` ("Type") and `blog:tag is-a blog:type`,
each documenting itself. Idempotent seed in `serve.sh`.
- `host/blog-types-of(slug)` = direct `is-a` targets ∪ `subtype-of`-reach of each
(SX-side composition over `lib/relations` reach — no new Datalog rules).
- `host/blog-is-a?(slug, type)` — **transitive**.
- Type-posts carry an optional `:schema` slot (designed now, mostly empty).
- Validation hook: `host/blog-content-ok?` extended to also run any schema(s)
implied by the post's declared types. No schema → no-op (gradual).
### Phase 3 — Tags as posts
- "is a tag" = `host/blog-is-a? slug "tag"` (transitive). Helpers
`host/blog-tags(slug)` = `out(slug,"tagged")`, `host/blog-tagged-with(tag)` =
`in(tag,"tagged")`.
- Edit page: a "This post is a tag" toggle = add/remove `is-a blog:tag` edge.
### Phase 4 — Render (data-driven from the registry)
- Post page iterates the registry → "Related posts" + "Tags" blocks, same code.
- Tag-post page: its own content (the tag's documentation) **plus** "Tagged with
this" (incoming `tagged`). A tag page documents the tag AND lists its members.
- Optional `/tags` index = posts `is-a? blog:tag`.
### Phase 5 — Generalize the picker
- `host/blog--relate-candidates(slug, q, kind)` branches on the kind's
`:candidates` (all / tags / types).
- `relate-options` endpoint takes `&kind=`; picker filter input carries
`data-kind`; `relate-picker.js` forwards it.
- Edit page renders one picker section per kind from the registry.
### Phase 6 — Schema expressiveness (ongoing)
- Grow the type `:schema` language: start minimal (required block kinds / a
predicate over content), richer later. Enforcement already wired in Phase 2;
only the language grows. Not a blocker — a gradient.
## Notes
- Node model unchanged (`blog:<slug>`); only `kind` varies. The relate machinery,
picker, and post-page block all generalize by lifting the hard-coded
`kind: "related"` into a parameter.
- A type can *be* a post all the way up (`blog:tag is-a blog:type`); meta-circular
but bounded by seeding a small root set.