coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge loops/erlang into architecture: Phases 7-10 (hot reload, FFI BIFs, BIF registry, VM opcode extension + erlang_ext); fixes cyclic-env identity hang

Browse Source 
# Conflicts:
#	hosts/ocaml/bin/run_tests.ml
#	plans/sx-vm-opcode-extension.md
This commit is contained in:
giles
2026-05-18 20:46:04 +00:00
parent a341041627 c352d94cc6
commit 77f17cc796
17 changed files with 5912 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files

128
plans/erlang-on-sx.md
View File

@@ -10,7 +10,9 @@ End-state goal: spawn a million processes, run the classic **ring benchmark**, p
- **Conformance:** not BEAM-compat. "Looks like Erlang, runs like Erlang, not byte-compatible." We care about semantics, not BEAM bug-for-bug.
- **Test corpus:** custom — ring, ping-pong, fibonacci-server, bank-account-server, echo-server, plus ~100 hand-written tests for patterns/guards/BIFs. No ISO Common Test.
- **Binaries:** basic bytes-lists only; full binary pattern matching deferred.
- **Hot code reload, distribution, NIFs:** out of scope entirely.
- **Distribution, NIFs:** out of scope entirely.
- **Hot code reload (Phase 7):** in scope — driven by [fed-sx](../plans/fed-sx-design.md) (section 17.5) which needs federated modules to be re-loaded without restarting the scheduler.
- **FFI BIFs (Phase 8):** in scope — Erlang code needs `crypto:hash`, `cid:from_bytes`, `file:read_file`, `httpc:request`, `sqlite:exec` to participate in fed-sx. A general FFI BIF registry replaces today's hard-coded BIF dispatch.
## Ground rules
@@ -95,10 +97,126 @@ Core mapping:
- [x] ETS-lite (in-memory tables via SX dicts) — **13 new eval tests**; `ets:new/2`, `insert/2`, `lookup/2`, `delete/1-2`, `tab2list/1`, `info/2` (size); set semantics with full Erlang-term keys
- [x] More BIFs — target 200+ test corpus green — **40 new eval tests**; 530/530 total. New: `abs/1`, `min/2`, `max/2`, `tuple_to_list/1`, `list_to_tuple/1`, `integer_to_list/1`, `list_to_integer/1`, `is_function/1-2`, `lists:seq/2-3`, `lists:sum/1`, `lists:nth/2`, `lists:last/1`, `lists:member/2`, `lists:append/2`, `lists:filter/2`, `lists:any/2`, `lists:all/2`, `lists:duplicate/2`
### Phase 7 — hot code reload
Driven by **fed-sx** (see `plans/fed-sx-design.md` §17.5): federated modules must be replaceable at runtime without bouncing the scheduler. Classic OTP behaviour: two versions per module ("current" and "old"), local calls stick to the version the process started with, cross-module (`M:F(...)`) calls always resolve to the current version, and `purge` kills any process still running old code.
- [x] Module version slot: `er-modules` entry becomes `{:current MOD-ENV :old MOD-ENV-or-nil :version INT}`; bump version on each load — **13 new runtime tests** (543/543 total)
- [x] `code:load_binary/3` (the canonical reload BIF) — re-parses module source, swaps `:current` → `:old`, installs new env as `:current`; returns `{module, Name}` or `{error, Reason}` (badarg / badfile / module_name_mismatch). **+8 eval tests** (551/551 total). `code:load_file/1` is a thin filesystem wrapper around this and lands once `file:read_file/1` is in (Phase 8).
- [x] `code:purge/1` + `code:soft_purge/1` — purge clears `:old` slot and kills any process whose `:initial-fun` env identity matches the old env (returns `true` if there was old code, `false` if there wasn't). soft_purge: refuses (returns `false`, leaves `:old` intact) if any process is still pinned to the old env; otherwise clears and returns `true`. **+10 eval tests** (561/561 total). Caveat: a true "lingering on old code" test needs `spawn/3` (still stubbed) or `fun M:F/A` syntax (not parsed) — anonymous `fun () -> M:F() end` closures capture the caller's env, not the module's, and cross-module calls always resolve to `:current`. Current tests therefore exercise the return-value matrix but not the kill path.
- [x] `code:which/1`, `code:is_loaded/1`, `code:all_loaded/0` — introspection. **+10 eval tests** (571/571 total). Return-value contract: `which` → `loaded` / `non_existing` (since we have no filesystem path); `is_loaded` → `{file, loaded}` / `false`; `all_loaded` → list of `{Module, loaded}` tuples. Non-atom Mod raises `error:badarg`.
- [x] Cross-module call `M:F(...)` dispatches to `:current`; local calls inside a module body keep using the env they closed over so a running process finishes its current function with the version it started with — **+6 eval tests** verifying the property end-to-end (577/577 total). No implementation change: `er-apply-user-module` already routes through `er-module-current-env`, and `er-mk-fun` captures its env by reference so closures created under v1 retain v1's `mod-env` even after the slot bumps to v2.
- [x] Tests: load v1 → spawn → load v2 → cross-module call hits v2 → local call inside v1 process keeps v1 semantics until function returns → purge kills v1 procs → soft_purge refuses while v1 procs alive — **+5 capstone eval tests** (582/582 total). Required extending `er-procs-on-env` from raw identity match to `er-env-derived-from?` (an env "comes from" mod-env if it IS mod-env or contains a value that's a fun closed over mod-env), because `er-apply-fun-clauses` does `er-env-copy closure-env` before binding params — so the spawned-from-inside-module fun's `:env` is a fresh dict, not mod-env. Test ladder runs as one single `erlang-eval-ast` program (every call to `ev` resets the scheduler via `er-sched-init!`, so Pid handles must live within one program).
### Phase 8 — FFI BIF mechanism + standard libs
Replace today's hardcoded BIF dispatch (`er-apply-bif`/`er-apply-remote-bif` in `transpile.sx`) with a runtime-extensible **BIF registry**. Each registry entry is `{:module :name :arity :fn :pure?}`. Standard libs are then registered at boot, and fed-sx can register new BIFs from `.sx` files. Includes the marshalling layer (Erlang term ↔ SX value) so wrappers stay one-liners.
- [x] BIF registry: `er-bif-registry` global dict keyed by `"Module/Name/Arity"`, with `er-register-bif!`/`er-register-pure-bif!`/`er-lookup-bif`/`er-list-bifs`/`er-bif-registry-reset!` helpers — **+18 runtime tests** (600/600 total). Entries are `{:module :name :arity :fn :pure?}`. Arity is part of the key so `m:f/1` and `m:f/2` are independent. Re-registering the same key replaces the previous entry; reset clears.
- [x] Migrate existing local + remote BIFs (length/hd/tl/lists:*/io:format/ets:*/etc.) onto the registry; delete the giant `cond` dispatch in `er-apply-bif`/`er-apply-remote-bif`. Conformance held at **600/600** after migration (baseline was 600, not the plan-text's 530 — the text was authored before Phase 7 work added rows). 67 builtin registrations across `erlang`/`lists`/`io`/`ets`/`code` modules; multi-arity BIFs (`is_function`, `spawn`, `exit`, `io:format`, `lists:seq`, `ets:delete`) register once per arity, all pointing at the same impl which dispatches on `(len vs)` internally. The four per-module cond dispatchers (`er-apply-lists-bif`, `er-apply-io-bif`, `er-apply-ets-bif`, `er-apply-code-bif`) are deleted. `er-apply-bif` and `er-apply-remote-bif` are now ~5-line registry lookups; user modules still win precedence over the registry.
- [x] Term-marshalling helpers: `er-of-sx` (SX → Erlang) and `er-to-sx` (Erlang → SX). atom ↔ symbol, nil ↔ `()`, cons → list, tuple → list (one-way; tuples flatten), binary ↔ SX string, integer / float / boolean passthrough. **+23 runtime tests** (623/623 total). Erlang maps (`dict ↔ map`) deferred — Erlang map term not implemented in this port; will land when `#{}` syntax does. Pids, refs, funs pass through unchanged. SX strings on the way back become Erlang binaries (most useful FFI return shape).
- [ ] `crypto:hash/2` — **BLOCKED** (no `sha256`/`sha512`/`blake3` primitive in this SX runtime). See Blockers.
- [ ] `cid:from_bytes/1`, `cid:to_string/1` — **BLOCKED** (needs `crypto:hash/2`). See Blockers.
- [x] `file:read_file/1`, `file:write_file/2`, `file:delete/1` — **+10 eval tests** (633/633 total). Returns `{ok, Binary}` / `ok` / `{error, Reason}` where Reason is `enoent`/`eacces`/`enotdir`/`eisdir`/`posix_error` (classified from the SX `file-read`/`-write`/`-delete` exception string). Path accepts SX string, Erlang binary, or Erlang char-code list. `file:list_dir/1` deferred — no directory-listing primitive in this SX runtime; see Blockers.
- [ ] `httpc:request/4` — **BLOCKED** (no HTTP client primitive). See Blockers.
- [ ] `sqlite:open/1`, `sqlite:close/1`, `sqlite:exec/2`, `sqlite:query/2` — **BLOCKED** (no SQLite primitive). See Blockers.
- [x] Tests: 1 round-trip per BIF; suite name `ffi`; conformance scoreboard auto-picks it up — **+14 ffi tests** at 637/637 total. Suite covers the 3 implemented file BIFs (9 tests: write-ok, read-ok-tag, payload-is-binary, byte_size content, missing-enoent, bad-path-enoent, binary-payload round-trip, delete-ok, read-after-delete-enoent) plus 5 negative asserts (one per blocked BIF — `crypto:hash`/`cid:from_bytes`/`file:list_dir`/`httpc:request`/`sqlite:exec`) so this suite fails fast if a future iteration adds a wrapper without registering proper tests. Target "+40 ffi tests" was relative to the original 5-BIF-family plan; with 5 of those families blocked on host primitives, the achievable count is 14 — the suite scaffolding is what matters and is ready to accept the remaining tests when the primitives land.
### Phase 9 — specialized opcodes (the BEAM analog)
**Driver:** Erlang-on-SX going through the general-purpose CEK machine has architectural perf ceilings (call/cc per receive, env-copy per call, mailbox rebuild on delete). The fix is specialized bytecode opcodes that bypass the general machinery for hot Erlang operations. Targets: 100k+ message hops/sec, 1M-process spawn in under 30sec. Layered perf strategy: Layer 1 (this) = specialized opcodes; Layer 2 (Phase 10, deferred) = multi-core scheduler.
**Architectural note:** opcodes get developed in `lib/erlang/vm/` (in scope). The **opcode extension mechanism in `hosts/ocaml/`** (Phase 9a) is **out of scope** for this loop — log as Blocker until a session that owns `hosts/` lands it. Sub-phases 9b-9g design and test opcodes against a stub dispatcher in the meantime; integrate when 9a is available.
**Shared-opcode discipline:** opcodes that another language port could plausibly use (pattern match, perform/handle, record access) get prepared for **chiselling out to `lib/guest/vm/`** when a second use materialises. Same lib/guest pattern, applied at the bytecode layer. Don't pre-extract; do annotate candidates in commit messages.
- [x] **9a — Opcode extension mechanism** — **INTEGRATED** (scope widened by user 2026-05-15: hosts/ in scope, merging back). Cherry-picked the 5 vm-ext commits (phases A-E: dispatch fallthrough for opcodes ≥200, `Sx_vm_extension` interface, `Sx_vm_extensions` registry, `extension-opcode-id` SX primitive, JIT skip path) onto loops/erlang. Force-linked `Sx_vm_extensions` into `bin/sx_server.ml` so its module-init runs (was dead-code-eliminated — only `run_tests` referenced it). `extension-opcode-id` is now live in the runtime: returns the registered opcode id, or nil for unknown names. Built clean; conformance held at **709/709** on the freshly built binary. Design: `plans/sx-vm-opcode-extension.md`.
- [x] **9b — `OP_PATTERN_TUPLE` / `OP_PATTERN_LIST` / `OP_PATTERN_BINARY`** — **+19 vm tests** (656/656 total). Stub dispatcher in `lib/erlang/vm/dispatcher.sx` mirrors the OCaml extension shape from `plans/sx-vm-opcode-extension.md`: `er-vm-register-opcode!`/`er-vm-lookup-opcode-by-id`/`er-vm-lookup-opcode-by-name`/`er-vm-dispatch`. Opcode IDs 128 (TUPLE), 129 (LIST), 130 (BINARY) per the guest-tier partition (128-199). Handlers are thin wrappers over the existing `er-match-tuple`/`er-match-cons`/`er-match-binary` for now; the real specialization (skip AST walk, register-machine operands) lands when 9a integrates. Conformance must remain unchanged — **656/656** preserved. Candidate for chiselling to `lib/guest/vm/match.sx` once a second port (Prolog? miniKanren?) wants the same opcodes.
- [x] **9c — `OP_PERFORM` / `OP_HANDLE`** — **+9 vm tests** (665/665 total). Stubs in `lib/erlang/vm/dispatcher.sx`: `OP_PERFORM` (id 131) raises `{:tag "vm-effect" :effect <name> :args <args>}`; `OP_HANDLE` (id 132) wraps a thunk in `guard`, catches matching effects (by `:effect` name), passes args to the handler, returns the handler's result. Non-matching effects rethrow to outer handlers (verified by a nested-handle test). Pure Erlang `receive` interface unchanged; this is the substrate for the eventual call/cc-free implementation when 9a integrates. Candidate for chiselling (Scheme call/cc, OCaml 5 effects, miniKanren all want the same shape).
- [x] **9d — `OP_RECEIVE_SCAN`** — **+10 vm tests** (675/675 total). Stub at id 133 in `lib/erlang/vm/dispatcher.sx`. Operand contract: `(clauses mbox-list env)` where each clause is `{:pattern :guards :body}`, mbox-list is a plain SX list (not a queue — caller does queue→list before invoking and queue-delete after). Walks mbox in arrival order; tries each clause per message; first match returns `{:matched true :index N :body B}` (env mutated with bindings, body NOT evaluated — caller chooses when); no match returns `{:matched false}`. Pure pattern scan; suspension is the caller's job (compose with OP_PERFORM "receive-suspend" once 9a integrates). The real opcode will skip the AST walk by JIT-compiling each clause's match expr; this stub re-uses `er-match!` for correctness.
- [x] **9e — `OP_SPAWN` / `OP_SEND` + lightweight scheduler** — **+16 vm tests** (691/691 total). Stubs at ids 134 (SPAWN) and 135 (SEND) in `lib/erlang/vm/dispatcher.sx`, plus the VM-process registry: `er-vm-procs` (dict pid → proc record), `er-vm-next-pid`, `er-vm-procs-reset!`, `er-vm-proc-new!`/`get`/`send!`/`mailbox`/`state`/`count`. Process record shape is the register-machine layout the real scheduler will use: `{:id :registers (list of 8 nil slots) :mailbox (SX list) :state ("runnable"/"waiting"/"dead") :initial-fn :initial-args}`. OP_SPAWN returns a numeric pid and allocates a fresh record; OP_SEND appends to the target's mailbox, flipping `:state` from "waiting" → "runnable" if needed (returns true on success, false on unknown pid — no crash). Sits parallel to `er-scheduler` (the language-level scheduler from Phase 3); the real VM scheduler will take over once 9a integrates and Erlang programs compile to bytecode. Perf targets in the bullet (spawn <50µs, send <5µs) defer to the integration step.
- [x] **9f — BIF dispatch table** — **+18 vm tests** (709/709 total). 10 hot BIFs get their own opcode IDs (136-145) in `lib/erlang/vm/dispatcher.sx`: `OP_BIF_LENGTH`, `OP_BIF_HD`, `OP_BIF_TL`, `OP_BIF_ELEMENT`, `OP_BIF_TUPLE_SIZE`, `OP_BIF_LISTS_REVERSE`, `OP_BIF_IS_INTEGER`, `OP_BIF_IS_ATOM`, `OP_BIF_IS_LIST`, `OP_BIF_IS_TUPLE`. Each opcode's handler IS the underlying `er-bif-*` impl directly (no registry-string-lookup), so cost is opcode-id → handler one-hop. Cold BIFs continue through `er-apply-bif` / `er-lookup-bif` as before. IDs 136-159 reserved for future hot-BIF additions.
- [x] **9h — `erlang_ext.ml`** — OCaml extension at `hosts/ocaml/lib/extensions/erlang_ext.ml` registering the 18-opcode Erlang namespace (ids **222-239**, names `erlang.OP_*` mirroring the SX stub dispatcher). Registered at sx_server startup via `Erlang_ext.register ()` (guarded against double-register Failure). `extension-opcode-id "erlang.OP_PATTERN_TUPLE"` → 222 … `OP_BIF_IS_TUPLE` → 239, unknown → nil. Handlers raise a descriptive not-wired `Eval_error` (bytecode emission is a later phase; SX stub dispatcher remains the working specialization path) — keeps the extension honest rather than silently corrupting the VM stack. id range 222+ dodges test_reg (210/211) + test_ext (220/221) so all three coexist in run_tests. **+5 OCaml ext tests** (run_tests `Suite: extensions/erlang_ext`); Erlang conformance held **709/709**.
- [x] **9i — wire SX dispatcher to real ids** — `lib/erlang/vm/dispatcher.sx` gains `er-vm-host-opcode-id` (thin `extension-opcode-id` wrapper) and `er-vm-effective-opcode-id name stub-id` (host id when non-nil, else stub-id). `extension-opcode-id` resolves lazily at call time so loading the file is safe even on a binary lacking the primitive; only invoking the resolver there would raise (documented prereq — the loop builds + runs against the binary that has it). **+6 vm tests** (715/715): OP_PATTERN_TUPLE→222, OP_BIF_IS_TUPLE→239, unknown→nil, effective prefers host (OP_BIF_LENGTH→230), effective falls back to stub on nil (999), and a sweep asserting the whole 18-name namespace maps contiguously to 222..239. Stub-local ids (128-145) registration untouched so the prior 72 vm tests stay green.
- [x] **9g — Conformance + perf bench** — Ran `lib/erlang/bench_ring.sh 10 100 500 1000` on the integrated binary (9a+9h+9i built in): 11/36/35/31 hops/s — **unchanged from the pre-integration baseline**, which is the correct expected result and doubles as a no-regression proof (the full extension wiring added zero per-hop cost). Conformance **715/715** on the same binary. Numbers recorded in `lib/erlang/bench_ring_results.md` with the rationale. The ~3000×/~1000× targets are gated on Phase 10 (bytecode emission) — the compiler doesn't emit `erlang.OP_*` yet, so every hop still takes the general CEK path. 9g's deliverable (honest measurement on the integrated binary) is complete.
### Phase 10 — bytecode emission (unlock the speedup)
The Phase 9 opcodes are registered, tested, and bridged SX↔OCaml, but inert: nothing emits them. Phase 10 makes the speedup real.
- [ ] **10a — compiler emits `erlang.OP_*` at hot sites** — **BLOCKED on `lib/compiler.sx` ownership (out of this loop's scope).** Architecture fully mapped (2026-05-15, see Blockers + design below). The correct implementation site is `lib/compiler.sx`'s `compile-call` — it must recognize calls to the Erlang runtime-helper functions that have a registered `erlang.OP_*` opcode and emit that opcode (via the already-live `extension-opcode-id` primitive) instead of a generic CALL. This is **generic shared compiler infrastructure** (any guest port — Prolog, Lua — would use the same intrinsic mechanism), explicitly excluded by the ground rules ("Don't edit lib/ root"; not in the widened hosts/-only scope). Concrete sub-steps for the owning session:
- **10a.1** Add an *intrinsic registry* to `lib/compiler.sx`: a dict `callee-name → extension-opcode-name`, populated by guests at load (e.g. Erlang registers `er-bif-length → "erlang.OP_BIF_LENGTH"`, `er-match-tuple → "erlang.OP_PATTERN_TUPLE"`, …).
- **10a.2** In `compile-call`: if the resolved callee is in the intrinsic registry AND `(extension-opcode-id name)` is non-nil, compile the args normally (push left→right) then emit the single opcode byte instead of `CALL`. Fall back to generic CALL when the opcode is absent (graceful on binaries without the extension).
- **10a.3** Define the operand/stack contract per opcode class and make `erlang_ext.ml`'s control handlers (222-229) match it (pattern opcodes need the pattern AST as a constant-pool operand + the scrutinee on the stack; perform/handle/receive/spawn/send need OCaml↔SX runtime-state access — see 10b-control note).
- **10a.4** Conformance must stay green; add bytecode-emission tests (compile an Erlang fn, disassemble, assert the opcode appears at the hot site).
Until a session owning `lib/compiler.sx` lands 10a.1-10a.2, the speedup cannot be realized from this loop. The BIF half of 10b (operand-less stack ops) is fully done and *would* light up immediately once emission exists.
- [~] **10b — real `erlang_ext.ml` handlers** — **10 of 18 real** (ALL BIF opcodes done: 230-239). Latest: `OP_BIF_ELEMENT` (233, pops Tuple-then-Index, 1-indexed, range-checked) and `OP_BIF_LISTS_REVERSE` (235, builds a fresh reversed cons chain in OCaml). Re-scoping correction: ELEMENT/REVERSE were earlier mislabelled "gated on 10a" — they're pure stack transforms (no bytecode operands; element/2 just pops 2), so they landed now. **21 e2e run_tests** total. Remaining 8 stubs are the genuine control/structural opcodes that DO need compiler-defined operands + runtime state: `OP_PATTERN_TUPLE/LIST/BINARY` (222-224), `OP_PERFORM/HANDLE` (225-226), `OP_RECEIVE_SCAN` (227), `OP_SPAWN/SEND` (228-229). not-wired guard repointed to 222. 715/715 unaffected. — earlier note: 8 of 18 real (all hot-BIFs done). Real register-machine handlers: `OP_BIF_LENGTH` (230, cons-walk), `OP_BIF_HD` (231), `OP_BIF_TL` (232), `OP_BIF_TUPLE_SIZE` (234, handles List + ListRef `:elements`), `OP_BIF_IS_INTEGER` (236, `Integer _`), `OP_BIF_IS_ATOM` (237), `OP_BIF_IS_LIST` (238, cons|nil), `OP_BIF_IS_TUPLE` (239) — all operate on the tagged-Dict value repr, push Erlang bool atoms via a `mk_atom` helper, raise on type errors. **15 end-to-end run_tests tests** (build real bytecode `[CONST i; op; RETURN]` with list/tuple/atom constants, assert via `Sx_vm.execute_module`). Still `not_wired`: the 8 control opcodes — `OP_PATTERN_TUPLE/LIST/BINARY` (222-224), `OP_PERFORM/HANDLE` (225-226), `OP_RECEIVE_SCAN` (227), `OP_SPAWN/SEND` (228-229) — plus `OP_BIF_ELEMENT` (233, needs 2 operands) and `OP_BIF_LISTS_REVERSE` (235). not-wired guard repointed to 233. 715/715 conformance unaffected (VM-bytecode path only; interpreter untouched). Remaining 10b: the 10 control/structural handlers.
- [ ] **10c — perf validation**: re-run `bench_ring.sh`; target 100k+ hops/sec at N=1000, 1M-process spawn < 30s; record in `bench_ring_results.md`. Conformance must stay green.
**Acceptance:** ring benchmark hits the 100k hops/sec target. All prior phase tests pass. Two opcodes chiselled to `lib/guest/vm/` (or annotated as candidates with a written rationale).
## Progress log
_Newest first._
- **2026-05-18 FIXED merge-blocking regression: cyclic-env hang in `er-env-derived-from?`** — A trial merge of loops/erlang → architecture regressed Erlang **715/715 → 0/0** on the architecture binary. Bisected: not loader semantics, not a uniform slowdown — pinpointed to the *single* Phase 7 capstone test (eval.sx lines 1314-1346; prefix-1313 was byte-identical speed on both binaries, 27s, prefix-1346 was 28s on loops vs >5min/hung on architecture). Isolated further: spawn+reload alone 0.6s, reload+purge alone 0.3s, but spawn+reload+**purge over forever-blocked procs** hung. Root cause: `er-env-derived-from?` (transpile.sx, used by `code:purge`/`soft_purge` via `er-procs-on-env`) compared closure envs with `(= env target-env)`. loops/erlang's evaluator implements dict `=` as **object identity**; architecture's 131-commit-newer evaluator changed it to **structural deep equality**. Erlang closure envs are large and **cyclic** (a module fun's `:env` transitively references the fun), so structural `=` over them never terminates. Fix: use `identical?` (pointer-identity predicate, present + consistent `(true false)` on *both* binaries) — the actually-intended semantics and host-independent. Verified: full eval.sx on the architecture binary >200s/hung → **59s**; full 10-suite conformance on the architecture binary now **715/715** (eval 385/385, vm 78/78, ffi 14/14, all process suites green). loops/erlang behaviour unchanged (`identical?` ≡ its old `=`-identity). One-file change (`lib/erlang/transpile.sx`, +7/-2). The merge can now be re-attempted; this was the sole blocker.
- **2026-05-15 Phase 10a — architecture traced, scoped, blocked on `lib/compiler.sx`** — Investigation-only iteration (correctly: faking compiler emission within scope is impossible and would be dishonest). Traced the full JIT path: `sx_vm.ml`'s `jit_compile_lambda` (the ref set at line 1206) invokes the SX-level `compile` from `lib/compiler.sx` via the CEK machine — that is the only SX→bytecode producer. Erlang's hot helpers are ordinary SX functions in `transpile.sx` that get JIT-compiled through exactly this path, so emitting `erlang.OP_*` means teaching `compiler.sx`'s `compile-call` to recognize them as intrinsics and emit the extension opcode (the file's own docstring already anticipates this — "Compilers call `extension-opcode-id` to emit extension opcodes" — designed but unimplemented; grep confirms zero `extension-opcode-id` uses in `compiler.sx`). `lib/compiler.sx` is lib-root: excluded by ground rules and the widened scope (editing it changes every guest's JIT — must be a shared-compiler session, not this loop). Recorded a precise Blockers entry + decomposed 10a into four numbered sub-steps (10a.1 intrinsic registry, 10a.2 `compile-call` emission with graceful CALL fallback, 10a.3 operand/stack contract for control opcodes, 10a.4 bytecode-emission tests) so the owning session can execute directly. Key payoff documented: all 10 BIF handlers (230-239) are already real, so they light up the instant 10a.1-10a.2 land — zero further Erlang-side work for the BIF speedup. No code changed; conformance unverified-but-untouched at **715/715** (no source touched). Phase 10's loop-reachable work (10b BIF half) is complete; the rest is correctly blocked and fully actionable elsewhere.
- **2026-05-15 Phase 10b — ELEMENT + LISTS_REVERSE real; all 10 BIF opcodes done** — Re-examined the earlier "gated on 10a" claim for ELEMENT/REVERSE and found it wrong: both are pure stack transforms with no need for bytecode operands (`element/2` just pops Tuple then Index off the VM stack; `lists:reverse/1` pops one list). Implemented both as real handlers in `erlang_ext.ml`. `OP_BIF_ELEMENT` (233): pops Tuple (TOS) then Index, handles List/ListRef `:elements`, 1-indexed, raises on out-of-range or wrong arg types. `OP_BIF_LISTS_REVERSE` (235): walks the cons chain building a fresh reversed one via local `mk_cons`/`mk_nil`, raises on improper list. Defined the calling convention for arity-2 ELEMENT: args pushed left→right so stack is `[Index Tuple]`, Tuple on top. 6 new e2e run_tests: element(2/1,{1,2,3}), element out-of-range raises, reverse-then-HD=9, reverse-then-TL-HD=8, reverse-then-LENGTH=3 (composes 3 real opcodes in one bytecode sequence). erlang_ext suite 15→21 PASS, dispatch_count 22. not-wired guard repointed 233→222 (OP_PATTERN_TUPLE — a genuine control opcode still stubbed). **All 10 BIF opcodes (230-239) now real**; the 8 remaining stubs are the true control/structural opcodes (pattern match, perform/handle, receive-scan, spawn/send) which genuinely need 10a's compiler-defined operand encoding + runtime-state access. Erlang conformance **715/715** (interpreter path untouched). 10b is now BIF-complete; the control-opcode half is the real remaining Phase 10 work and is correctly gated on 10a.
- **2026-05-15 Phase 10b — all 8 hot-BIF handlers real** — Built on the vertical slice: added 7 more real register-machine handlers in `erlang_ext.ml` (HD 231, TL 232, TUPLE_SIZE 234, IS_INTEGER 236, IS_ATOM 237, IS_LIST 238, IS_TUPLE 239), joining LENGTH 230. Shared helpers added: `mk_atom` (builds the Erlang bool atom `{tag→atom, name→true|false}`), `er_bool`, `is_tag` (Dict tag predicate). TUPLE_SIZE handles both `List` and `ListRef` `:elements` (Erlang tuples may be built mutably). IS_INTEGER keys off `Sx_types.Integer`. All raise descriptive `Eval_error` on type mismatch. The `op N "name"` stub helper now only covers the 10 remaining control/structural opcodes. 9 new end-to-end run_tests assertions added (HD, TL∘HD, TUPLE_SIZE, IS_INTEGER pos+neg, IS_ATOM, IS_LIST nil-true + tuple-false, IS_TUPLE) — each builds real bytecode with a list/tuple/atom constant and executes via `Sx_vm.execute_module`. erlang_ext suite 6→15 PASS; dispatch_count 12. not-wired guard repointed 231→233 (OP_BIF_ELEMENT, still stubbed — it needs two operands so it's a later sub-step). Erlang conformance **715/715** (the interpreter path is untouched; only the VM-bytecode dispatch gained real handlers). Remaining 10b: pattern tuple/list/binary, perform/handle, receive-scan, spawn/send, element, lists:reverse (10 opcodes).
- **2026-05-15 Phase 10b vertical slice — first real opcode handler, end-to-end VM proof** — Investigation first: confirmed Erlang runs as a pure tree-walking interpreter (`er-eval-expr` over CEK) — there is **no** Erlang→bytecode compiler, so full 10a (compiler emits opcodes) is a multi-week standalone effort, not one iteration. Rather than fake it, de-risked the whole Phase 9/10 architecture with a vertical slice: replaced the `not_wired` raise for `erlang.OP_BIF_LENGTH` (id 230) with a genuine register-machine handler in `erlang_ext.ml` — pops a value, walks the Erlang cons-list representation (`Dict` with `"tag"`→`"cons"`/`"nil"`, `"head"`, `"tail"`), pushes `Integer` length, raises on improper lists. Added an end-to-end run_tests test that builds real bytecode `[| 1; 0; 0; 230; 50 |]` (CONST idx 0 → OP_BIF_LENGTH → RETURN) with an Erlang `[1,2,3]` in `vc_constants`, executes via `Sx_vm.execute_module`, asserts `Integer 3`. This proves the complete path works: `extension-opcode-id` → bytecode → `Sx_vm` ≥200 dispatch fallthrough → `erlang_ext` handler → correct VM stack result — the load-bearing proof that Phase 9's wiring isn't just stubs. The other 17 opcodes still honestly raise `not_wired`; the prior not-wired guard test was repointed from 230 to 231 (OP_BIF_HD) so it still verifies the honest-failure path. erlang_ext suite 5→6 tests, dispatch_count now 2. Erlang conformance **715/715** unaffected (the new path is VM-bytecode-only; the interpreter path is untouched). 10b marked in-progress `[~]`; remaining: real handlers for the other 17 opcodes + 10a compiler emission. Builds clean via `dune build bin/run_tests.exe bin/sx_server.exe`.
- **2026-05-15 Phase 9g — perf bench recorded on integrated binary; Phase 10 scoped** — Built the fresh `sx_server.exe` (9a+9h+9i wired in), ran `lib/erlang/bench_ring.sh 10 100 500 1000`: 11/36/35/31 hops/s — statistically identical to the pre-9a baseline (11/24/26/29/34). This is the *expected* outcome and the iteration's actual deliverable: it proves the entire extension stack (vm-ext A-E cherry-pick + `Sx_vm_extensions` force-link + `erlang_ext.ml` + SX dispatcher bridge) added **zero per-hop overhead** — a clean no-regression result — while honestly showing the speedup hasn't arrived because the bytecode compiler still doesn't emit `erlang.OP_*` (every hop takes the general CEK path). Updated `bench_ring_results.md` with a "Phase 9g" section: the table + the rationale that unchanged numbers = correct + no-regression. Conformance **715/715** on the integrated binary. Added **Phase 10 — bytecode emission** to the roadmap (10a compiler emits opcodes at hot sites, 10b real register-machine `erlang_ext.ml` handlers replacing the not-wired raises, 10c perf validation against the 100k-hops/1M-spawn targets). Phase 9 is now fully ticked (9a-9i); the actual speedup is honestly deferred to Phase 10 rather than faked. No code change this iteration — measurement + documentation + roadmap.
- **2026-05-15 Phase 9i — SX dispatcher consults host opcode ids** — `lib/erlang/vm/dispatcher.sx` now bridges SX↔OCaml opcode ids. Two new functions: `er-vm-host-opcode-id` (wraps `extension-opcode-id`) and `er-vm-effective-opcode-id name stub-id` (host id if the OCaml `erlang_ext` registered it, else the stub-local id). Key SX-runtime fact established this iteration: symbol resolution is **lazy/call-time** — `(define f (fn () (extension-opcode-id "x")))` does NOT raise at load even when the primitive is absent; only calling `f` does. Combined with the earlier findings (guard can't catch undefined-symbol; no symbol-existence reflection), this means graceful in-SX degradation is impossible — so the design instead documents the binary prerequisite and relies on the loop building+running the freshly-built `hosts/ocaml/_build/default/bin/sx_server.exe` (conformance.sh's default, which has the vm-ext mechanism + erlang_ext). Stub-local registration (128-145) deliberately left intact so the 72 pre-existing vm tests don't move. 6 new vm tests: 222/239 lookups, unknown→nil, effective-prefers-host (230), effective-fallback (999), and a contiguity sweep over all 18 `erlang.OP_*` names asserting they map to 222..239 in order. vm suite 72→78. Total **715/715** on the fresh binary. Next: 9g — re-run ring bench, record numbers (note: stubs still wrap existing impls 1-to-1 so numbers won't move until the compiler emits these opcodes — a later phase).
- **2026-05-15 Phase 9h — erlang_ext.ml registered, opcode namespace live** — New `hosts/ocaml/lib/extensions/erlang_ext.ml` modelled on `test_ext.ml`: an `EXTENSION` module `name="erlang"`, per-instance `ErlangExtState` (dispatch counter), 18 opcodes ids 222-239 named `erlang.OP_*` exactly mirroring the SX stub dispatcher. Registered at sx_server startup with a second guarded line in `bin/sx_server.ml` (`try Erlang_ext.register () with Failure _ -> ()` — survives a re-entered server). `include_subdirs unqualified` in `lib/dune` already pulls `lib/extensions/*.ml` into the `sx` lib, so no dune edit needed. Handlers deliberately raise a descriptive `Eval_error` ("bytecode emission not yet wired (Phase 9j) — Erlang runs via CEK; specialization path is the SX stub dispatcher") rather than fake stack ops — the compiler doesn't emit these yet, so an honest loud failure beats silent corruption. Hit and fixed an opcode-id collision: the original 200-217 range clashed with run_tests' inline test_reg (210/211); relocated to 222-239 (clears test_reg + test_ext 220/221, all coexist; production sx_server only registers erlang). 5 new OCaml tests in run_tests `Suite: extensions/erlang_ext`: opcode-id 222 + 239 resolve, unknown→nil, dispatch raises not-wired (substring check, no Str dep since run_tests doesn't link str), dispatch_count state ≥1. Built via `eval $(opam env --switch=5.2.0); dune build bin/run_tests.exe bin/sx_server.exe`. Erlang conformance **709/709** on the rebuilt binary (the broad run_tests 1110 failures are loops/erlang's pre-existing months-old divergence from architecture — run_tests was never built on this branch before; my changes are isolated additive). Next: 9i — wire the SX stub dispatcher to consult `extension-opcode-id`.
- **2026-05-15 Phase 9a integrated — scope widened to hosts/** — User lifted the hosts/ scope restriction ("we are going to merge this back anyhow"). Cherry-picked the 5 `vm-ext` commits (phases A-E) from `loops/sx-vm-extensions` onto `loops/erlang` — only conflict was `plans/sx-vm-opcode-extension.md` (already had architecture's final copy from an earlier iteration; resolved `-X ours`, OCaml files auto-merged clean since loops/erlang never touched hosts/). Discovered `extension-opcode-id` was still "Undefined symbol" even on a fresh build: `Sx_vm_extensions`'s module-init (`install_dispatch` + primitive registration) only runs if the module is linked, and `sx_server.ml` never referenced it (only `run_tests.ml` did), so OCaml dead-code-eliminated it. Fix: added `let () = ignore (Sx_vm_extensions.id_of_name "")` force-link reference near the top of `bin/sx_server.ml`. Rebuilt with `dune build` (opam switch 5.2.0; `dune` not on PATH by default — `eval $(opam env --switch=5.2.0)` first). `extension-opcode-id` now live: returns nil for unregistered names, will return real ids once an extension registers. Conformance **709/709** on the freshly built binary (cherry-picked sx_vm.ml dispatch changes + force-link, zero regressions). 9a checkbox flipped from BLOCKED to INTEGRATED; Blockers entry resolved; added 9h (erlang_ext.ml) + 9i (wire SX dispatcher to real ids) as ordinary in-scope checkboxes, reordered 9g after them. Next: write `hosts/ocaml/lib/extensions/erlang_ext.ml`.
- **2026-05-14 Phase 9g logged as partially BLOCKED — perf bench waits on 9a** — Conformance half satisfied: 709/709 with all Phase 9 stub infrastructure loaded (10 opcode IDs registered, 72 vm-suite tests passing, zero regressions in tokenize/parse/eval/runtime/ring/ping-pong/bank/echo/fib/ffi suites). Perf-bench half can't move forward in this worktree because the stub handlers wrap the existing `er-bif-*` / `er-match-*` / scheduler impls 1-to-1; a ring benchmark with the new opcodes "active" would measure the same 34 hops/s already documented in `bench_ring_results.md`. Updated `bench_ring_results.md` with a Phase 9 status section explaining the pre-integration state (stubs ready, real measurement gated on 9a's bytecode compiler emitting these IDs at hot sites). Blockers entry added pairing 9g with the existing 9a Blocker. No code change; total **709/709** unchanged. Phase 9 stub work (9b-9f) is complete from this loop's vantage point — 9a and 9g remain BLOCKED on a `hosts/ocaml/` iteration.
- **2026-05-14 Phase 9f — hot-BIF opcode table green** — Ten hot BIFs get direct opcode IDs in `lib/erlang/vm/dispatcher.sx` so the bytecode compiler can emit them at hot call sites without paying the registry string-key hash: `OP_BIF_LENGTH (136)`, `OP_BIF_HD (137)`, `OP_BIF_TL (138)`, `OP_BIF_ELEMENT (139)`, `OP_BIF_TUPLE_SIZE (140)`, `OP_BIF_LISTS_REVERSE (141)`, `OP_BIF_IS_INTEGER (142)`, `OP_BIF_IS_ATOM (143)`, `OP_BIF_IS_LIST (144)`, `OP_BIF_IS_TUPLE (145)`. Implementation is one line per opcode: the handler IS the existing `er-bif-*` function directly — same `(vs)` signature as the dispatcher's `(operands)`, so the registration is `(er-vm-register-opcode! ID "NAME" er-bif-FOO)`. IDs 136-159 reserved for future hot-BIF additions; cold BIFs continue through `er-apply-bif`/`er-lookup-bif`. 18 new tests in `tests/vm.sx`: opcode-by-id verification (LENGTH), one positive test per BIF (length on 3-cons, hd, tl-is-cons, element index 2, tuple_size 4, lists:reverse preserves length AND actually reverses [head check], is_integer pos+neg, is_atom pos+neg, is_list pos+nil pos+tuple neg, is_tuple pos+neg), opcode-list-grew-to-16+. vm suite 54 → 72. Total **709/709** (+18 vm). Real perf benefit lands when 9a integrates and the compiler emits these IDs at hot sites.
- **2026-05-14 Phase 9e — OP_SPAWN / OP_SEND + VM-process registry green** — `lib/erlang/vm/dispatcher.sx` gains a parallel mini-runtime distinct from the language-level `er-scheduler`: `er-vm-procs` (dict pid → proc record), `er-vm-next-pid` (counter cell), `er-vm-procs-reset!`, plus six accessors (`er-vm-proc-new!`/`get`/`send!`/`mailbox`/`state`/`count`). Process record shape is the register-machine layout the real bytecode scheduler will use: `{:id :registers (8 nil slots) :mailbox :state :initial-fn :initial-args}` — fixed register width so cells don't grow during execution. Opcode 134 `OP_SPAWN` calls `er-vm-proc-new!` and returns the new pid; 135 `OP_SEND` appends to the target's mailbox and flips a waiting proc back to runnable, returns false for unknown pid (graceful, doesn't crash). 16 new tests in `tests/vm.sx`: opcode-by-id for both, spawn returns 0 / 1 / count=2 / state=runnable / mailbox empty / 8 registers, send returns true, 3-sends preserve arrival order (first + last verified), send to unknown pid returns false, isolation (p1's msgs don't leak into p2), reset clears procs + resets pid counter. vm suite 38 → 54. One gotcha during impl: SX `fn` bodies evaluate ONLY the last expression — `er-vm-procs-reset!` had two `set-nth!` calls back-to-back which silently dropped the first; wrapped in `(do ...)` to fix. Total **691/691** (+16 vm). Real scheduler with per-process scheduling latency and runnable queue is post-9a.
- **2026-05-14 Phase 9d — OP_RECEIVE_SCAN stub green** — Selective-receive primitive at opcode id 133 in `lib/erlang/vm/dispatcher.sx`. Operand contract: `(clauses mbox-list env)` — clauses are AST dicts (`{:pattern :guards :body}`), mbox-list is a plain SX list (queue → list is the caller's job), env is the binding target. Internal helpers `er-vm-receive-try-clauses` (per-message clause walker with env snapshot/restore on failure) and `er-vm-receive-scan-loop` (mailbox walker, arrival order). Match returns `{:matched true :index N :body B}` so the caller can queue-delete at N and then evaluate B in the now-mutated env; miss returns `{:matched false}` so the caller can suspend via OP_PERFORM "receive-suspend". Mirrors the existing `er-try-receive-loop` in `transpile.sx` but doesn't reach into the scheduler — purely VM-level. 10 new tests in `tests/vm.sx`: opcode registered, scan finds match at correct index, scan binds var, body left unevaluated, no-match leaves env untouched, empty mailbox, first-match wins (arrival order — verified by two `{ok, _}` msgs and binding the FIRST value). vm suite 28 → 38. Total **675/675** (+10 vm). When 9a integrates and the real OP_RECEIVE_SCAN compiles clauses into a register-machine match, the existing `er-eval-receive-loop` becomes a one-line dispatch wrapper.
- **2026-05-14 Phase 9c — OP_PERFORM / OP_HANDLE stubs green** — Two new opcodes in `lib/erlang/vm/dispatcher.sx`: id 131 `OP_PERFORM` raises `{:tag "vm-effect" :effect <name> :args <args>}`; id 132 `OP_HANDLE` wraps a thunk in SX `guard`, catches matching effects by `:effect` name, passes the `:args` list to the handler fn, returns the handler's result. New helper `er-vm-effect-marker?` predicates on the dict shape. Non-matching effects rethrow via a small box+rethrow dance (caught with `:else` first, decision deferred to a post-guard cond — re-raise outside the guard's scope so it propagates to outer handlers cleanly). 9 new tests in `tests/vm.sx`: opcode registered for each id; OP_PERFORM raises with correct tag/effect/args; OP_HANDLE catches matching effect; OP_HANDLE returns thunk result when no effect performed; OP_HANDLE rethrows non-matching effect to outer; nested OP_HANDLE blocks separate by effect name (inner handles "a", outer handles "b", performing "b" bypasses inner). vm suite grew 19 → 28 tests. Total **665/665** (+9 vm). Underlying call/cc + raise/guard machinery used by Erlang `receive` is unchanged; this is the shape for the eventual specialization when 9a integrates. Candidate for chiselling to `lib/guest/vm/effects.sx` — Scheme call/cc, OCaml 5 effects, miniKanren all want the same shape.
- **2026-05-14 Phase 9b — stub VM dispatcher + 3 pattern opcodes green** — New `lib/erlang/vm/dispatcher.sx` defines the stub opcode registry mirroring the OCaml `EXTENSION` shape from `plans/sx-vm-opcode-extension.md`: opcodes registered as `{:id :name :handler}` keyed by string-id, looked up by id OR by name, dispatched via `er-vm-dispatch`. Opcode IDs follow the guest-tier partition (128-199 reserved for guest extensions like erlang/lua). Three opcodes registered at load time via `er-vm-register-erlang-opcodes!`: 128 `OP_PATTERN_TUPLE` → `er-match-tuple`, 129 `OP_PATTERN_LIST` → `er-match-cons`, 130 `OP_PATTERN_BINARY` → `er-match-binary`. Operand contract: `(pattern-ast value env)` returning `true`/`false` and mutating env on success — same as the underlying match functions. New `lib/erlang/tests/vm.sx` suite with 19 tests: 7 dispatcher core (registered, lookup by id+name for all three, two miss cases, list-has-3+); 4 OP_PATTERN_TUPLE (match success + var bind, no-match, arity mismatch); 4 OP_PATTERN_LIST (match, head bind, tail-is-cons, no-match on nil); 3 OP_PATTERN_BINARY (match, segment bind, size mismatch); 1 dispatch error (unknown opcode raises). `conformance.sh` updated: added `vm` to SUITES, added `(load "lib/erlang/vm/dispatcher.sx")` before tests and `(load "lib/erlang/tests/vm.sx")` after ffi, added epoch 110 evaluator. AST shape gotcha: er-match! reads `:type` not `:tag`; binary segment `:size` must be an AST node `{:type "integer" :value "8"}` because `er-eval-expr` runs on it. Total **656/656** (+19 vm). 9b complete; 9c (OP_PERFORM/OP_HANDLE) is next.
- **2026-05-14 Phase 9a logged as Blocker — sub-phase 9b is next** — 9a (the opcode extension mechanism in `hosts/ocaml/evaluator/`) is explicitly out-of-scope for this loop per the plan itself (briefing scope rule + 9a's own text). Logged a Blockers entry citing `plans/sx-vm-opcode-extension.md` as the design doc and pointing at the fix path (a `hosts/` session lands the registration shape, then a follow-up here wires the stub dispatcher to the real one). Ticked 9a as DONE because its contract was "Log as Blocker" — that's complete. Sub-phases 9b–9g (PATTERN/PERFORM/RECEIVE/SPAWN_SEND/BIF/conformance) now in queue against a stub dispatcher in `lib/erlang/vm/`. No code change this iteration. Total **637/637** unchanged.
- **2026-05-14 Phase 9 scoped + supporting plan files synced** — Copied three plan files from `/root/rose-ash/plans/` (architecture branch) that this worktree was missing: `fed-sx-design.md` (124KB, the substrate design referenced from Phase 7/8 drivers), `fed-sx-milestone-1.md` (33KB, first concrete implementation milestone), `sx-vm-opcode-extension.md` (19KB, the prerequisite for Phase 9a — designs how `lib/<lang>/vm/` registers opcodes against the OCaml SX VM core). Then appended **Phase 9 — specialized opcodes (the BEAM analog)** to `plans/erlang-on-sx.md` covering sub-phases 9a-9g: 9a (opcode extension mechanism in `hosts/ocaml/`) is out-of-scope for this loop (will be logged as a Blocker when the next iteration tries to start it); 9b-9g (PATTERN_TUPLE/LIST/BINARY, PERFORM/HANDLE, RECEIVE_SCAN, SPAWN/SEND + lightweight scheduler, BIF dispatch table, conformance + perf bench) can be designed and tested against a stub dispatcher in the meantime. Targets: ring benchmark 100k+ hops/sec at N=1000 (~3000× speedup), 1M-process spawn under 30sec (~1000× speedup). Plan framing intact for Phase 7/8 — those reflect the actual implementation done in this loop; the architecture-branch framing diverges in language but the work is equivalent. No code touched this iteration. Total **637/637** unchanged.
- **2026-05-14 ffi test suite extracted, conformance scoreboard auto-picks it up** — New `lib/erlang/tests/ffi.sx` with its own counter trio (`er-ffi-test-count`/`-pass`/`-fails`) and `er-ffi-test` helper following the same pattern as runtime/eval/ring tests. The 10 file BIF eval tests from the previous iteration moved out of `eval.sx` (eval dropped from 395 to 385 tests) and into the new suite where they're now 9 tests (consolidated the two write+read tests). `conformance.sh` updated: added `ffi` to `SUITES` array with `er-ffi-test-pass`/`-count` symbols, added `(load "lib/erlang/tests/ffi.sx")` after `fib_server.sx`, added `(epoch 109) (eval "(list er-ffi-test-pass er-ffi-test-count)")`. Scoreboard markdown auto-updated to include the row. Suite also asserts that the 5 blocked BIFs (`crypto:hash`, `cid:from_bytes`, `file:list_dir`, `httpc:request`, `sqlite:exec`) are NOT yet registered — turns a future "added the wrapper but forgot to extend ffi tests" into a hard failure. One eval-comparison gotcha en route: SX's `=` does identity equality on dicts so comparing two separately-constructed `(er-mk-atom "true")` values is false; the existing eval suite has an `eev-deep=` helper that handles this, but the simpler fix in ffi was to extract `:name` via `ffi-nm` and compare strings. Total **637/637** (+14 ffi). Phase 8 fully ticked aside from the BLOCKED bullets — those remain unchecked with explicit Blockers references.
- **2026-05-14 file BIFs landed; crypto/cid/list_dir/http/sqlite blocked on missing host primitives** — Three new FFI BIFs registered in `runtime.sx`: `file:read_file/1`, `file:write_file/2`, `file:delete/1`. Each wraps the SX-host primitive (`file-read`, `file-write`, `file-delete`) inside a `guard` that converts thrown exception strings into Erlang `{error, Reason}` tuples. New helper `er-classify-file-error` does loose pattern-matching on the error message using `string-contains?` to map to standard POSIX-style reasons: `"No such"` → `enoent`, `"Permission denied"` → `eacces`, `"Not a directory"` → `enotdir`, `"Is a directory"` → `eisdir`, fallback `posix_error`. Filenames coerce through `er-source-to-string` so SX strings, Erlang binaries, and Erlang char-code lists all work. Read returns `{ok, Binary}` (bytes via `(map char->integer (string->list ...))` then `er-mk-binary`); write returns bare `ok`; delete returns bare `ok`. Bootstrap registrations added at the bottom of `er-register-builtin-bifs!` under `"file"`. 10 new eval tests: write-then-read round-trip, ok-tag, payload is binary, byte_size content, missing-file `enoent`, delete-ok, read-after-delete `enoent`, write to non-existent dir `enoent`, binary payload (5 raw bytes) round-trip preserving byte count. Blockers entry added covering five Phase 8 BIFs whose host primitives don't exist in this SX runtime: `crypto:hash/2`, `cid:from_bytes/1`/`to_string/1`, `file:list_dir/1`, `httpc:request/4`, `sqlite:open/exec/query/close`. Fix path documented inline (architecture-branch iteration to register OCaml-side primitives). Total **633/633** (+10 eval).
- **2026-05-14 term-marshalling helpers landed** — `er-to-sx` (Erlang term → SX-native) and `er-of-sx` (SX-native → Erlang term) plus internal helper `er-cons-to-sx-list` (recursive cons-chain walker). All three live in `runtime.sx` next to the BIF registry. Conversion table: atom ↔ symbol via `make-symbol`/`er-mk-atom`; nil ↔ `()`; cons-chain → SX list (recursive marshal of each head); tuple → SX list (one-way — tuples flatten and can't be reconstructed without a tag); binary ↔ SX string (bytes ↔ char codes via `char->integer`/`integer->char`); integer / float / boolean passthrough; opaque types (pid, ref, fun) passthrough. SX strings on the way back become Erlang binaries — the natural FFI return shape. Empty SX list (`type-of` `"nil"`) marshals back to `er-mk-nil`. Edit gotchas during implementation: SX has no `while`, `string-ref`, or `string-length` primitive — used `(map char->integer (string->list s))` for byte extraction and a recursive helper for cons-walking. 23 new runtime tests in `tests/runtime.sx`: 10 covering `er-to-sx` (atom/atom-is-symbol, nil, int / float / bool passthrough, binary→string, cons→list, tuple→list, nested), 8 covering `er-of-sx` (symbol→atom, atom-tag, string→binary, byte content, int passthrough, empty-list→nil, list→cons length, head field), 4 round-trips (int, atom, binary bytes, list length), 1 negative documenting that tuple round-trip flattens to cons. Total **623/623** (+23 runtime).
- **2026-05-14 BIF registry migration complete — cond chains gone** — `er-register-builtin-bifs!` at the end of `runtime.sx` populates the registry with all 67 built-in BIFs in five module namespaces. Pure ops (`length`, `hd`, `tl`, `element`, predicates, arithmetic, list/atom/integer conversions, all of `lists`) registered via `er-register-pure-bif!`; side-effecting ops (`spawn`, `self`, `exit`, `link`/`monitor`/`register`, `process_flag`, `make_ref`, `throw`/`error`, `io:format`, all of `ets`, all of `code`) via `er-register-bif!`. Multi-arity entries: `is_function/1`/`/2`, `spawn/1`/`/3`, `exit/1`/`/2`, `io:format/1`/`/2`, `lists:seq/2`/`/3`, `ets:delete/1`/`/2` — six pairs, twelve registrations, all pointing at the existing arity-dispatching impl. `throw` and `error` are registered with a tiny inline `(fn (vs) (raise ...))` lambda because the original code chained directly through `raise` inside the cond instead of an `er-bif-*` helper. `er-apply-bif` shrinks from a 44-line cond chain to a 5-line registry lookup. `er-apply-remote-bif` becomes a 7-line dispatcher (user-modules-first → registry → error). All four per-module dispatchers (`er-apply-lists-bif`, `er-apply-io-bif`, `er-apply-ets-bif`, `er-apply-code-bif`) deleted — net reduction ~110 lines of cond machinery. One subtle wrinkle: `tests/runtime.sx` calls `er-bif-registry-reset!` near the end of its BIF-registry tests, which would have left subsequent test files (ring, ping-pong, etc.) unable to call `length`/`spawn`/etc. Fix: re-call `er-register-builtin-bifs!` at the bottom of `tests/runtime.sx` to repopulate. Total **600/600** unchanged.
- **2026-05-14 Phase 8 BIF registry foundation** — `lib/erlang/runtime.sx` gains `er-bif-registry` (a `(list {})` mutable cell, same shape as `er-modules`) and five helpers: `er-bif-registry-get`/`er-bif-registry-reset!` (access + reset), `er-bif-key` (format `"Module/Name/Arity"`), `er-register-bif!` and `er-register-pure-bif!` (both upsert; differ only in the `:pure?` flag — pure ones are safe to inline, side-effecting ones go through normal IO), `er-lookup-bif` (returns the entry dict or nil), `er-list-bifs` (registered keys). Entries are `{:module :name :arity :fn :pure?}`. Lookup miss → nil; arity is part of the key so `m:f/1` and `m:f/2` are distinct; re-registering the same key replaces in-place (count stays the same); reset clears. Registry sits alongside `er-modules` in runtime.sx so any other piece of the system can register BIFs without touching the dispatcher — the migration onto this registry (the next checkbox) will rip out the giant cond chains in `er-apply-bif`/`er-apply-remote-bif`. 18 new runtime tests in `tests/runtime.sx`: empty-state, lookup-miss, register-grows-count, lookup-hit-fields (module/name/arity/pure?), fn-invocable, re-register-replaces, pure-flag-true, arity-disambiguation (3 entries for `fake:echo/1`, `fake:echo/2`, `fake:pure/2`), reset-clears, reset-lookup-nil. Total **600/600** (+18 runtime).
- **2026-05-14 Phase 7 capstone green — full hot-reload ladder works end-to-end** — Wires everything from the previous five iterations into one test program: load cap v1 with `start/0` (spawn-from-inside-module) + `loop/0` + `tag/0` → spawn Pid1 (running v1) → load cap v2 → assert `cap:tag()` returns v2 (cross-module dispatch hits `:current`) → spawn Pid2 (running v2) → `code:soft_purge(cap)` returns `false` (refuses while Pid1 is alive on v1's env) → `code:purge(cap)` returns `true` (kills Pid1, clears `:old`) → `code:soft_purge(cap)` returns `true` (clean — no `:old` left). To make this work, `er-procs-on-env` was extended with a new helper `er-env-derived-from?`: a process counts as "running on" mod-env if its `:initial-fun`'s `:env` IS mod-env directly OR contains at least one binding whose value is a fun closed over mod-env. Reason: `er-apply-fun-clauses` always `er-env-copy`s the closure-env before binding params, so a fun created inside a module body has a `:env` that's a *copy* of mod-env, not mod-env itself — the copy still contains the module's other functions as values, each pointing back to the canonical mod-env. The whole ladder runs as a single `erlang-eval-ast` invocation because each call to `ev` resets the scheduler via `er-sched-init!`, wiping any cross-call Pids. 5 capstone tests: v1 tag, v2 tag (cross-mod after reload), soft_purge-refuses, hard purge, soft_purge-clean-after-hard. Total **582/582** (+5 eval). Phase 7 fully ticked.
- **2026-05-14 hot-reload call-dispatch semantics verified** — Tests-only iteration: no implementation change, just six new eval tests that nail down the Erlang semantics already implicit in the current code. (1) `M:F()` after reload returns v2's value (cross-module call hits `:current`). (2) Inside a freshly-loaded body, a bare local call resolves through the new mod-env so a chain `a() -> b()` reflects v2's `b/0`. (3) Calling a fun captured BEFORE reload, whose body uses a local call, returns the v1 value (closure pinned to old mod-env via `er-mk-fun`'s `:env` reference). (4) Calling a fun captured BEFORE reload, whose body uses a cross-module call `M:b()`, returns v2's value (cross-module always wins over closed-over env). (5) Two captured funs from two distinct vintages stay independent — F1() + F2() = 10 + 20 = 30. (6) The slot version counter still bumps even while old captured funs are alive, demonstrating the closure-pinning doesn't block reloads. The "running process finishes its current function with the version it started with" property falls out of fun-as-closure semantics for free — there's no special bookkeeping. Total **577/577** (+6 eval).
- **2026-05-14 code introspection BIFs green** — `code:which/1`, `code:is_loaded/1`, `code:all_loaded/0` added to `er-apply-code-bif` dispatch with three small implementations in `transpile.sx`. `which` and `is_loaded` are dict-lookups on the module registry returning the loaded-marker (atom `loaded`) or the missing-marker (atom `non_existing` for which, atom `false` for is_loaded). Since we don't have a filesystem path representation, the standard `{file, Path}` shape for `is_loaded` becomes `{file, loaded}` — same tuple arity so destructuring code stays portable. `all_loaded` iterates `(keys (er-modules-get))` in reverse (so the result list preserves insertion order after the cons-prepend loop), wrapping each name in a `{Module, loaded}` tuple. **10 new eval tests**: non_existing for absent / loaded after load for which; missing / file-tag / loaded-value for is_loaded; empty / count-after-2-loads / first-entry-tag for all_loaded; badarg for both single-arg BIFs. Two of the all_loaded tests needed an explicit `(er-modules-reset!)` before the measurement because prior tests in the suite leave modules registered (the registry is process-global across the whole epoch session). Total **571/571** (+10 eval).
- **2026-05-14 code:purge/1 + code:soft_purge/1 green** — Two new BIFs in `transpile.sx`: `er-bif-code-purge` and `er-bif-code-soft-purge`, both dispatched through the existing `er-apply-code-bif` cond chain. Shared helper `er-procs-on-env` walks `(er-sched-processes)` and collects pids whose `:initial-fun` is a fun whose `:env` is identical (dict-identity, not structural) to a given env, filtering out already-dead procs. `er-bif-code-purge` looks up the module slot, returns `false` if either the module isn't registered or `:old` is nil; otherwise calls `er-cascade-exit!` on every matching pid with reason `killed`, replaces the slot with a fresh `er-mk-module-slot` that has `:old nil` (current + version preserved), returns `true`. `er-bif-code-soft-purge` returns `true` (treating "no module" / "no old version" as already-purged), else checks for lingering procs and returns `false` (leaving the slot untouched) if any, else clears `:old` and returns `true`. Non-atom Mod raises `error:badarg` from both. **10 new eval tests**: unknown / no-old / after-reload / idempotent for purge; unknown / no-old / clean for soft_purge; badarg for both; one "purge after spawn" test verifying return value (does NOT exercise the kill path — see caveat in plan). Total **561/561** (+10 eval). Implementation cost: 1 dispatch entry, 3 small BIFs, no scheduler changes.
- **2026-05-14 code:load_binary/3 green** — Canonical hot-reload entry point. Adds a `"code"` module branch to `er-apply-remote-bif`'s dispatch; new helpers `er-source-walk-bytes!` and `er-source-to-string` coerce any of {SX string, Erlang binary `<<...>>`, Erlang char-code cons list} to an SX source string before parsing. `er-bif-code-load-binary` is the BIF itself: validates `Mod` is an atom (`{error, badarg}` else), coerces source (`{error, badarg}` on unrecognised shape), wraps `erlang-load-module` in `guard` to convert parse failures into `{error, badfile}`, checks the parsed `-module(Name).` matches the BIF's first arg (`{error, module_name_mismatch}` else), returns `{module, Mod}`. Reload reuses the Phase-7 slot logic from the previous iteration so calling `code:load_binary(m, _, v2_source)` after `code:load_binary(m, _, v1_source)` bumps the slot to version 2 with v1 sitting in `:old`. 8 new eval tests: ok-tag/ok-name on first load, immediate cross-module call hits new env, reload-and-call returns v2 result, name-mismatch errors with both tag and reason, garbage source yields badfile, non-atom Mod is badarg. Total **551/551** (+8 eval). `code:load_file/1` deferred until `file:read_file/1` lands in Phase 8 (it's just a wrapper that reads bytes from disk then calls `load_binary`).
- **2026-05-14 Phase 7 module-version slot landed** — `er-modules` entries are now `{:current MOD-ENV :old MOD-ENV-or-nil :version INT :tag "module"}` instead of bare mod-env dicts. New helpers in `runtime.sx`: `er-mk-module-slot`, `er-module-current-env`, `er-module-old-env`, `er-module-version`. `erlang-load-module` updated: first load creates a slot with `:version 1` and `:old nil`; subsequent loads of the same module name copy `:current` into `:old` and increment `:version` (bump-and-shift, single-old-version retention as per OTP semantics). `er-apply-user-module` now reads via `er-module-current-env` so cross-module calls always hit the latest version. 13 new runtime tests (mostly in `tests/runtime.sx`): slot constructor + accessors, registry-after-first-load (v1, old nil), registry-after-second-load (v2, old = previous current env identity, current = new env), v3 on triple-load, registry-reset clears. Total **543/543** (was 530/530). Note: sx-tree path-based MCP tools (`sx_replace_node`, `sx_read_subtree`) are broken in this worktree's `mcp_tree.exe` (every path returns/replaces form 0); edits applied via a Python script then `sx_validate`d. Pattern-based tools (`sx_find_all`, `sx_rename_symbol`) still work fine.
- **2026-05-14 Phase 7 + Phase 8 scoped** — Plan extended with two new phases driven by fed-sx (see `plans/fed-sx-design.md` §17.5). Phase 7 brings hot code reload back in scope (was previously listed as out-of-scope): module versioning slot, `code:load_file/1`/`purge/1`/`soft_purge/1`/`which/1`/`is_loaded/1`, cross-module calls hitting current, local calls keeping start-time semantics until function returns. Phase 8 introduces a runtime-extensible **FFI BIF registry** that replaces today's hardcoded `er-apply-bif`/`er-apply-remote-bif` cond chains, plus a term-marshalling layer and concrete BIFs for `crypto:hash`, `cid:from_bytes`/`to_string`, `file:read_file`/`write_file`/`list_dir`/`delete`, `httpc:request`, `sqlite:open`/`exec`/`query`. Scope decisions header updated accordingly. Baseline 530/530 unchanged; no code touched this iteration.
- **2026-04-25 BIF round-out — Phase 6 complete, full plan ticked** — Added 18 standard BIFs in `lib/erlang/transpile.sx`. **erlang module:** `abs/1` (negates negative numbers), `min/2`/`max/2` (use `er-lt?` so cross-type comparisons follow Erlang term order), `tuple_to_list/1`/`list_to_tuple/1` (proper conversions), `integer_to_list/1` (returns SX string per the char-list shim), `list_to_integer/1` (uses `parse-number`, raises badarg on failure), `is_function/1` and `is_function/2` (arity-2 form scans the fun's clause patterns). **lists module:** `seq/2`/`seq/3` (right-fold builder with step), `sum/1`, `nth/2` (1-indexed, raises badarg out of range), `last/1`, `member/2`, `append/2` (alias for `++`), `filter/2`, `any/2`, `all/2`, `duplicate/2`. 40 new eval tests with positive + negative cases, plus a few that compose existing BIFs (e.g. `lists:sum(lists:seq(1, 100)) = 5050`). Total suite **530/530** — every checkbox in `plans/erlang-on-sx.md` is now ticked.
- **2026-04-25 ETS-lite green** — Scheduler state gains `:ets` (table-name → mutable list of tuples). New `er-apply-ets-bif` dispatches `ets:new/2` (registers table by atom name; rejects duplicate name with `{badarg, Name}`), `insert/2` (set semantics — replaces existing entry with the same first-element key, else appends), `lookup/2` (returns Erlang list — `[Tuple]` if found else `[]`), `delete/1` (drop table), `delete/2` (drop key; rebuilds entry list), `tab2list/1` (full list view), `info/2` with `size` only. Keys are full Erlang terms compared via `er-equal?`. 13 new eval tests: new return value, insert true, lookup hit + miss, set replace, info size after insert/delete, tab2list length, table delete, lookup-after-delete raises badarg, multi-key aggregate sum, tuple-key insert + lookup, two independent tables. Total suite 490/490.
- **2026-04-25 binary pattern matching green** — Parser additions: `<<...>>` literal/pattern in `er-parse-primary`, segment grammar `Value [: Size] [/ Spec]` (Spec defaults to `integer`, supports `binary` for tail). Critical fix: segment value uses `er-parse-primary` (not `er-parse-expr-prec`) so the trailing `:Size` doesn't get eaten by the postfix `Mod:Fun` remote-call handler. Runtime value: `{:tag "binary" :bytes (list of int 0-255)}`. Construction: integer segments emit big-endian bytes (size in bits, must be multiple of 8); binary-spec segments concatenate. Pattern matching consumes bytes from a cursor at the front, decoding integer segments big-endian, capturing `Rest/binary` tail at the end. Whole-binary length must consume exactly. New BIFs: `is_binary/1`, `byte_size/1`. Binaries participate in `er-equal?` (byte-wise) and format as `<<b1,b2,...>>`. 21 new eval tests: tag/predicate, byte_size for 8/16/32-bit segments, single + multi segment match, three 8-bit, tail rest size + content, badmatch on size mismatch, `=:=` equality, var-driven construction. Total suite 477/477.
@@ -131,4 +249,10 @@ _Newest first._
## Blockers
- _(none yet)_
- **Phase 10a — opcode emission requires `lib/compiler.sx` (out of scope)** (2026-05-15). Architecture fully traced this iteration: the OCaml JIT (`sx_vm.ml` `jit_compile_lambda`, ref-set at line 1206) invokes the SX-level `compile` from **`lib/compiler.sx`** via the CEK machine; that is the sole SX→bytecode producer. Erlang's hot helpers (`er-match-tuple`, `er-bif-*`, …) are SX functions in `transpile.sx` that get JIT-compiled through this path. To emit `erlang.OP_*` they must be recognized as intrinsics inside `compiler.sx`'s `compile-call` (the file's own docstring already anticipates this: "Compilers call `extension-opcode-id` to emit extension opcodes" — designed, not yet implemented). `lib/compiler.sx` is **lib-root**, excluded by the ground rules ("Don't edit lib/ root") and absent from the widened `lib/erlang/** + hosts/ocaml/** (extension only)` scope — editing it changes every guest language's JIT, so it must be owned by a shared-compiler session, not this loop. **Fix path:** that session implements 10a.1 (intrinsic registry in `compiler.sx`) + 10a.2 (`compile-call` emits the opcode when registered & `extension-opcode-id` non-nil, else generic CALL). Erlang's BIF handlers (10b, ids 230-239, all real) light up the instant emission exists — zero further work here. The control opcodes (222-229) additionally need 10a.3 (operand contract) + OCaml↔SX runtime-state bridging (Erlang scheduler/mailbox live in `lib/erlang/runtime.sx`, not OCaml).
- **Phase 9g — Perf bench gated on 9a** (2026-05-14). The conformance half of 9g (709/709 with stub VM loaded) is satisfied; the perf-bench half requires 9a's bytecode compiler to actually emit the new opcodes at hot call sites. Until then a benchmark would measure today's `er-bif-*` / `er-match-*` numbers unchanged (since the stub handlers wrap them 1-to-1). Re-fire 9g after 9a lands.
- **Phase 9a — Opcode extension mechanism** — **RESOLVED 2026-05-15.** User widened scope to include hosts/ (merging back anyhow). Cherry-picked vm-ext phases A-E + force-linked `Sx_vm_extensions` into sx_server.exe. `extension-opcode-id` live; conformance 709/709. Remaining integration work (erlang_ext.ml + wiring the SX stub dispatcher to consult real ids) tracked as ordinary in-scope checkboxes now, not blockers.
- **SX runtime lacks platform primitives for crypto / dir-listing / HTTP / SQLite** (2026-05-14). Probed in `mcp_tree.exe`'s embedded `sx_server.exe`: `(sha256 "x")`, `(blake3 "x")`, `(hash "sha256" "x")`, `(file-list-dir "plans")`, `(http-get "url")`, `(fetch "url")` all return `Undefined symbol`. Only file-byte-level primitives exist: `file-read` ✓, `file-write` ✓, `file-delete` ✓, `file-exists?` ✓. Out-of-scope to add these (they live in `hosts/` per ground rules). Blocked Phase 8 BIFs: `crypto:hash/2`, `cid:from_bytes/1`, `cid:to_string/1`, `file:list_dir/1`, `httpc:request/4`, `sqlite:open/exec/query/close`. **Fix path:** a future iteration on the architecture branch can register host primitives (e.g. expose OCaml's `Digestif` for hashes, `Sys.readdir` for list_dir, `cohttp` for httpc); the BIF wrappers here will then become one-line registrations against `er-bif-registry`.

2638
plans/fed-sx-design.md Normal file
View File

File diff suppressed because it is too large Load Diff

922
plans/fed-sx-milestone-1.md Normal file
View File

@@ -0,0 +1,922 @@
# fed-sx Milestone 1 — Kernel + Registries + Pin Smoke Test
Concrete implementation plan for the smallest fed-sx that proves the architecture
works end-to-end. Reference: `plans/fed-sx-design.md`. Prerequisite: Erlang-on-SX
Phases 7 (hot reload) + 8 (FFI BIFs).
## Goal
Ship a single-instance, single-actor fed-sx server that:
1. Boots from a verified genesis bundle.
2. Accepts and durably appends signed activities via `POST /activity`.
3. Folds them into projections in real time.
4. Serves AP-standard endpoints (actor, outbox, artifacts, capabilities).
5. Demonstrates **two extensibility proof-points** end-to-end with zero kernel
code changes between definition and use:
- **Verb extensibility** (§5 meta-level): publish `DefineActivity{Pin}` +
`DefineProjection{pin-state}`, then publish a `Pin` activity, observe it
validated and projected.
- **Reactive application extensibility** (§§18-19): publish
`DefineSubscription{Topic}` + `Subscribe{topic: smoketest}` +
`DefineTrigger{when: that subscription, then: publish TestEcho}`, then
publish a tagged Note, observe the subscription match, the trigger fire,
and the derived activity appear in the outbox.
Federation, multi-actor, advanced verbs, IPFS, browser UI, operator dashboard
are **explicitly v2**.
## Non-goals (what milestone 1 deliberately does NOT do)
- **Federation.** No `POST /inbox` from peers, no `Follow`, no delivery queue, no
webfinger discovery flow. Single instance only.
- **Multi-actor.** Single domain actor (`acct:next@next.rose-ash.com`).
- **IPFS / S3 storage backends.** Files on disk only.
- **Advanced verbs.** No `Endorse`, `Supersede`, `Test`, `Build`, `Compose`,
`Note`, `Announce`. Only the four bootstrap verbs (`Create`, `Update`, `Delete`)
plus a defined-from-the-log `Pin` for the smoke test. (`Announce` deferred —
no use case until federation exists.)
- **Browser UI.** Curl-shaped API only.
- **Operator dashboard, quarantine UX.** Logs only.
- **Performance work.** Functional correctness first; perf when measured.
- **Cross-host conformance test corpus.** Only the OCaml/Erlang-on-SX host runs
fed-sx in v1; conformance suite for other hosts is v2.
## Architecture summary
```
POST /activity
│
▼
┌──────────────────────────┐
│ HTTP server (Erlang-on-SX)│
└─────────────┬─────────────┘
│
┌─────────────▼──────────────┐
│ Validation pipeline driver │
│ (envelope→sig→schema→...) │
└─────────────┬──────────────┘
│
┌─────────────▼──────────────┐
│ Log append (JSONL segment) │ ← canonical
└─────────────┬──────────────┘
│
┌─────────────▼──────────────┐
│ Projection workers │ ← gen_server per
│ (fold scheduler) │ projection
└─────────────────────────────┘
│
▼
Projection state
(queryable via HTTP)
Native primitives (Erlang-on-SX BIFs from Phase 8):
crypto:* cid:* fs:* http:* sqlite:*
Genesis bundle (binary-embedded SX):
activity-types object-types projections
validators codecs sig-suites
```
## Build order
Eight steps in dependency order. Each step has concrete deliverables, testable
in isolation, and a clear acceptance check.
| Step | Title | Depends on |
|------|-------|------------|
| **1** | Repo skeleton + canonical CID computation | Phase 8 (cid BIFs) |
| **2** | Activity envelope + signature verify | Phase 8 (crypto BIFs) |
| **3** | JSONL log + sequence numbers | Phase 8 (fs BIFs) |
| **4** | Genesis bundle (SX sources + bundling + CID verification) | Step 1 |
| **5** | Registry mechanism + bootstrap-projection dispatch | Steps 2, 4 |
| **6** | Validation pipeline driver + `POST /activity` | Steps 2, 3, 5 |
| **7** | Projection scheduler (gen_server per projection) | Steps 5, 6 |
| **8** | HTTP server, AP endpoints, projection queries | Steps 6, 7 |
| **9** | Smoke tests (Pin verb + reactive application) | Steps 1-8 |
---
## Step 1 — Repo skeleton + canonical CID
**Deliverables:**
```
next/
├── README.md # what this is
├── kernel/ # Erlang-on-SX
│ └── (empty for now)
├── genesis/ # core SX bootstrap definitions
│ └── (empty for now)
├── tests/ # smoke test scripts
│ └── (empty for now)
└── data/ # gitignored runtime state
├── log/
├── objects/
├── snapshots/
├── indexes/
└── keys/
```
Plus one Erlang-on-SX module:
```erlang
% next/kernel/cid.erl
-module(cid).
-export([from_sx/1, to_string/1, from_string/1, equals/2]).
from_sx(SxValue) ->
Cbor = cid:cbor_encode(canonicalize_sx(SxValue)),
Hash = crypto:sha2_256(Cbor),
cid:from_bytes(<<"raw">>, Hash). % defaults to dag-cbor codec
canonicalize_sx(V) -> ... % sorts dict keys, normalizes strings
```
**Tests:**
- Same SX value → same CID across multiple invocations.
- Different SX values → different CIDs.
- Whitespace/comment differences in source → identical CIDs (parsed AST identical).
- Reordered dict keys → identical CIDs (sorted-key canonicalization).
- Cross-host parity (just OCaml host for v1, but write the test so adding hosts is mechanical).
**Acceptance:** `bash next/tests/cid.sh` passes 10+ cases.
---
## Step 2 — Activity envelope + signature verify
**Deliverables:**
```erlang
% next/kernel/envelope.erl
-module(envelope).
-export([validate_shape/1, canonical_bytes/1, verify_signature/2]).
% Envelope shape per design §3.1:
% #{id, type, actor, published, to, cc, audience_extras,
% object | target | origin | result,
% capabilities_required, proofs, signature}
validate_shape(Activity) -> ok | {error, Reason}.
canonical_bytes(Activity) ->
% Strip signature, canonicalize via dag-cbor, return bytes for sig coverage
Stripped = maps:remove(signature, Activity),
cid:cbor_encode(canonicalize_for_sig(Stripped)).
verify_signature(Activity, ActorState) ->
% Time-aware: find key with id == sig.key_id that was active at published
% Per design §9.6
...
```
**Tests:**
- Envelope shape: required fields present (id, type, actor, published, signature)
- Envelope shape: type is a known activity-type or unknown-but-string
- Envelope shape: signature has key_id, algorithm, value
- Sig verify: valid RSA-SHA256 signature against published key → ok
- Sig verify: valid Ed25519 signature → ok
- Sig verify: tampered envelope → fail
- Sig verify: key superseded before activity timestamp → fail
- Sig verify: key superseded after activity timestamp → ok (historical valid)
**Acceptance:** `bash next/tests/envelope.sh` passes 15+ cases.
---
## Step 3 — JSONL log + sequence numbers
**Deliverables:**
```erlang
% next/kernel/log.erl
-module(log).
-export([open/1, append/2, read_segment/2, tip/1, replay/3]).
% Per design §15.2: per-actor outbox, segments cap ~64MB,
% format = JSONL (one canonical JSON-LD activity per line)
open(ActorId) ->
BasePath = log_path_for_actor(ActorId),
fs:mkdir_p(BasePath),
{ok, #{base => BasePath, current => current_segment(BasePath), seq => next_seq(BasePath)}}.
append(LogState, Activity) ->
Json = jsonld:encode(Activity),
Path = current_segment_path(LogState),
Line = <<Json/binary, "\n">>,
fs:append_file(Path, Line),
NewSeq = LogState#{seq := LogState.seq + 1},
rotate_if_needed(NewSeq).
% replay/3 calls Fun(Activity, Acc) for every activity in chronological order
replay(LogState, InitAcc, Fun) -> ...
```
**Tests:**
- Append + read back gives identical activity (round-trip).
- Sequence numbers monotonic and gap-free per actor.
- Segment rotation at size threshold.
- Replay visits all activities in append order across multiple segments.
- Restart preserves tip pointer (seq number resumes correctly).
- Concurrent appends (using gen_server-mediated access) are serialized correctly.
**Acceptance:** `bash next/tests/log.sh` passes 10+ cases.
---
## Step 4 — Genesis bundle
**Deliverables:**
Genesis bundle SX sources (per design §12.2). Each is a small SX file authored
by hand for the bootstrap set:
```
next/genesis/
├── manifest.sx # bundle root: lists all definitions
├── activity-types/
│ ├── create.sx # DefineActivity{name: "Create", ...}
│ ├── update.sx
│ └── delete.sx
├── object-types/
│ ├── sx-artifact.sx
│ ├── note.sx
│ ├── tombstone.sx
│ ├── define-activity.sx # DefineObject for the Define* meta types
│ ├── define-object.sx
│ ├── define-projection.sx
│ ├── define-validator.sx
│ ├── define-codec.sx
│ ├── define-sig-suite.sx
│ └── snapshot.sx
├── projections/
│ ├── activity-log.sx # identity projection
│ ├── by-type.sx
│ ├── by-actor.sx
│ ├── by-object.sx
│ ├── actor-state.sx
│ ├── define-registry.sx # the chicken-and-egg projection
│ └── audience-graph.sx
├── validators/
│ ├── envelope-shape.sx
│ ├── signature.sx
│ └── type-schema.sx
├── codecs/
│ ├── dag-cbor.sx # delegates to cid:cbor_encode/decode BIFs
│ ├── raw.sx
│ └── dag-json.sx
├── sig-suites/
│ ├── rsa-sha256-2018.sx
│ └── ed25519-2020.sx
└── audience/
├── public.sx
├── followers.sx
└── direct.sx
```
Plus a build-time bundler:
```erlang
% next/kernel/bootstrap.erl
-module(bootstrap).
-export([build_genesis/1, verify_genesis/1, load_genesis/1]).
build_genesis(SourceDir) ->
% Walk SourceDir, parse each .sx file, build a single dag-cbor bundle,
% compute its CID, write bundle.cbor + CID to data/genesis/
...
verify_genesis(BundlePath) ->
% Compute CID of the bundle as loaded; compare to expected (hardcoded
% in the kernel binary). Mismatch → halt.
...
load_genesis(BundlePath) ->
% Parse the bundle, register all definitions in the in-memory registry
...
```
**Tests:**
- All genesis SX files parse cleanly.
- Bundle CID is deterministic (rebuild same sources → same CID).
- Bundle reload reproduces the exact same registry state.
- Tampered bundle → `verify_genesis` returns `{error, cid_mismatch}`.
**Acceptance:** `bash next/tests/bootstrap.sh` passes; `next/data/genesis/bundle.cbor`
created with a known stable CID.
---
## Step 5 — Registry mechanism + bootstrap dispatch
**Deliverables:**
Registries are gen_servers, one per kind, each holding the active version map:
```erlang
% next/kernel/registry.erl
-module(registry).
-behaviour(gen_server).
-export([start_link/0, lookup/2, register/3, list/1]).
% Internal state:
% #{activity_types => #{Name => #{cid, schema_fn, semantics_fn, supersedes}},
% object_types => ...,
% projections => ...,
% validators => ...,
% codecs => ...,
% sig_suites => ...,
% ...}
lookup(Kind, Name) -> {ok, Entry} | {error, not_found}.
register(Kind, Name, Entry) -> ok | {error, Reason}.
list(Kind) -> [#{name, cid}].
```
The `define-registry` projection's fold updates this gen_server's state when
new `Define*` activities arrive. (Bootstrapping circle resolved: at startup,
`bootstrap:load_genesis/1` populates the registry directly; from then on, the
projection fold maintains it.)
**Tests:**
- After genesis load, `registry:list(activity_types)` returns Create/Update/Delete.
- `registry:lookup(activity_types, "Create")` returns the schema and semantics.
- A new `DefineActivity{name: "Pin"}` activity (synthesised, hand-signed for the
test) routes through the projection fold, ends up in the registry.
- Lookup never caches across activities (verified by introducing a new definition
mid-test and confirming the next lookup sees it).
**Acceptance:** `bash next/tests/registry.sh` passes 10+ cases.
---
## Step 6 — Validation pipeline + POST /activity
**Deliverables:**
```erlang
% next/kernel/pipeline.erl
-module(pipeline).
-export([validate_inbound/1, validate_outbound/1]).
% Per design §14, run stages in order, halt on first failure.
validate_inbound(Activity) ->
Stages = [
fun stage_envelope/1,
fun stage_signature/1,
fun stage_replay/1,
fun stage_audience/1,
fun stage_activity_schema/1,
fun stage_object_schema/1,
fun stage_content_validators/1,
fun stage_capabilities/1,
fun stage_trust/1
],
run_stages(Activity, Stages).
validate_outbound(Activity) ->
% Subset of inbound stages (no replay, no trust check; auth done at HTTP layer)
...
```
```erlang
% next/kernel/outbox.erl
-module(outbox).
-export([publish/2]).
publish(ActorId, ActivityRequest) ->
Activity = construct_envelope(ActorId, ActivityRequest),
Signed = sig:sign(Activity, ActorId),
case pipeline:validate_outbound(Signed) of
ok ->
log:append(actor_log(ActorId), Signed),
projection:async_fold(Signed),
{ok, #{cid => cid:from_sx(Signed),
ap_id => maps:get(id, Signed)}};
{error, Reason} ->
{error, Reason}
end.
```
**Tests:**
- Valid activity through full pipeline → appended to log.
- Bad envelope → 400, not in log.
- Bad signature → 401, not in log.
- Replayed activity → 200 duplicate, not re-appended.
- Schema violation (e.g. Create with no object) → 422.
- Activity logged before projection completes (async).
**Acceptance:** `bash next/tests/pipeline.sh` passes 15+ cases.
---
## Step 7 — Projection scheduler
**Deliverables:**
```erlang
% next/kernel/projection.erl
-module(projection).
-export([start_link/1, async_fold/1, query/2, snapshot/1]).
-behaviour(gen_server).
% One gen_server per active projection. State:
% #{cid, name, fold_fn, current_state, log_tip,
% snapshot_dir, last_snapshot_at}
% async_fold/1 broadcasts a new activity to every projection gen_server;
% each folds it into its own state. Failures (gas, sandbox violation)
% tag the activity but don't affect log durability.
% query/2 returns current state (or state-as-of)
% snapshot/1 forces a snapshot now (also runs periodically)
```
```erlang
% next/kernel/sandbox.erl
-module(sandbox).
-export([eval_pure/2, eval_crypto/2, eval_effectful/3]).
% eval_pure runs an SX function in pure mode: no IO platform, gas budget,
% deterministic. Used by projection folds, validators, audience predicates.
% Wrapper over the SX runtime evaluator with a stripped platform.
```
**Tests:**
- New activity → all projections fold it concurrently.
- Projection fold completes within gas budget.
- Gas-exhausting fold → activity tagged, projection state unchanged, no kernel crash.
- Sandbox violation (fold tries IO) → same handling.
- Snapshot create + reload → state matches.
- Snapshot CID stable across kernel restarts.
**Acceptance:** `bash next/tests/projection.sh` passes 15+ cases.
---
## Step 8 — HTTP server + endpoints
**Deliverables:**
Core endpoints (per design §16.1):
```
GET /actors/<id> # actor doc
GET /actors/<id>/outbox # OrderedCollection
GET /actors/<id>/outbox?page=true # OrderedCollectionPage
POST /activity # publish (auth: bearer token)
GET /artifacts/<cid> # CID-addressed artifact
GET /artifacts/<cid>/raw
GET /projections # list of projections
GET /projections/<name> # full state
GET /projections/<name>?at=<ts> # time-travel
GET /projections/<name>/<key> # indexed lookup
GET /define-registry
GET /.well-known/sx-capabilities
GET /.well-known/webfinger
```
```erlang
% next/kernel/http_server.erl
-module(http_server).
-export([start/1, route/1]).
start(Port) ->
http:listen(Port, fun ?MODULE:route/1).
route(Request) -> {Status, Headers, Body}.
```
Content negotiation per `Accept`:
- `application/activity+json` (default)
- `application/cbor` (dag-cbor)
- `application/json` (compact, no @context expansion)
- `application/sx`
Auth on `POST /activity`: bearer token from env var `NEXT_PUBLISH_TOKEN`.
**Tests:**
- Each endpoint returns expected shape for known artifact.
- Content negotiation: same artifact in 4 representations.
- 404 for unknown artifact CID.
- 401 for `POST /activity` without token.
- Pagination: outbox with > 50 activities returns OrderedCollectionPage.
**Acceptance:** `bash next/tests/http.sh` passes 20+ cases.
---
## Step 9 — Smoke tests
**The proof points.** Two end-to-end smoke tests demonstrate, between them, that
fed-sx is genuinely a substrate for distributed reactive applications expressed
as data — not a system you extend by writing kernel code.
- **9a — Pin smoke test (`next/tests/smoke_pin.sh`)** — verb extensibility:
defining a new activity type and projection at runtime via `Define*`
artifacts. Verifies the meta-level (§5).
- **9b — Reactive application smoke test (`next/tests/smoke_app.sh`)** —
application extensibility: defining a new subscription type, subscribing,
registering a trigger, and observing the full reactive loop fire end-to-end
without kernel code changes. Verifies §§18-19.
Both must pass for milestone 1 acceptance.
### Step 9a — Pin smoke test
**Test script:** `next/tests/smoke_pin.sh`
```bash
#!/usr/bin/env bash
set -euo pipefail
# 0. Start a fresh fed-sx kernel (background)
./next/scripts/start.sh fresh
sleep 2
TOKEN=$(cat next/data/keys/publish.token)
# 1. Verify actor exists
curl -s http://localhost:9999/actors/next | jq -e '.type == "Person"'
# 2. Verify outbox has actor's first Create{Person}
curl -s http://localhost:9999/actors/next/outbox?page=true \
| jq -e '.orderedItems | length == 1 and .[0].type == "Create"'
# 3. Verify Pin is NOT a known activity type
curl -s http://localhost:9999/define-registry?kind=activity_types \
| jq -e '.[] | select(.name == "Pin") | length == 0' || exit 1
# 4. Publish DefineActivity{name: "Pin", schema: ..., semantics: ...}
PIN_DEF=$(cat <<'JSON'
{
"type": "Create",
"object": {
"type": "DefineActivity",
"name": "Pin",
"schema": "(fn (act) (and (string? (-> act :object :path)) (cid? (-> act :object :cid))))",
"semantics": "(fn (state act) (assoc-in state [:pins (-> act :object :path)] (-> act :object :cid)))"
}
}
JSON
)
curl -s -X POST http://localhost:9999/activity \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/activity+json" \
-d "$PIN_DEF" | jq -e '.cid' > /dev/null
# 5. Verify Pin IS now a known activity type
curl -s http://localhost:9999/define-registry?kind=activity_types \
| jq -e '.[] | select(.name == "Pin") | length == 1'
# 6. Also publish a DefineProjection{name: "pin-state"} that folds Pin into state
PIN_PROJ=$(cat <<'JSON'
{
"type": "Create",
"object": {
"type": "DefineProjection",
"name": "pin-state",
"initial-state": "{}",
"fold": "(fn (state act) (if (= (:type act) \"Pin\") (assoc state (-> act :object :path) (-> act :object :cid)) state))"
}
}
JSON
)
curl -s -X POST http://localhost:9999/activity \
-H "Authorization: Bearer $TOKEN" \
-d "$PIN_PROJ" | jq -e '.cid'
# 7. Now publish a Pin activity
PIN=$(cat <<'JSON'
{
"type": "Pin",
"object": {
"type": "PinSpec",
"path": "/docs/intro",
"cid": "bafyreigh2akiscaildc3xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxxe"
}
}
JSON
)
curl -s -X POST http://localhost:9999/activity \
-H "Authorization: Bearer $TOKEN" \
-d "$PIN" | jq -e '.cid'
# 8. Verify Pin appears in outbox
curl -s http://localhost:9999/actors/next/outbox?page=true \
| jq -e '.orderedItems | map(select(.type == "Pin")) | length == 1'
# 9. Verify pin-state projection has the entry
sleep 1 # allow async projection
curl -s http://localhost:9999/projections/pin-state \
| jq -e '."/docs/intro" == "bafyreigh2akiscaildc3xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxxe"'
# 10. Negative test: publish a malformed Pin (missing path) → expect 422
BAD_PIN='{"type": "Pin", "object": {"cid": "bafy..."}}'
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST http://localhost:9999/activity \
-H "Authorization: Bearer $TOKEN" -d "$BAD_PIN")
[[ "$HTTP_STATUS" == "422" ]] || { echo "expected 422, got $HTTP_STATUS"; exit 1; }
# 11. Restart kernel; verify state recovers
./next/scripts/stop.sh
./next/scripts/start.sh
sleep 2
curl -s http://localhost:9999/projections/pin-state \
| jq -e '."/docs/intro" == "bafyreigh2akiscaildc3xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxxe"'
echo "✓ Pin smoke test passed — verb extensibility demonstrated end-to-end"
```
**Acceptance for 9a:** smoke test exits 0. The whole flow happens with **zero
fed-sx kernel code changes** between defining the verb and using it.
### Step 9b — Reactive application smoke test
**The bigger proof point.** Demonstrates that fed-sx supports distributed
reactive applications composed of `DefineSubscription` + `DefineTrigger` +
`DefineProjection` — the application model from §§18-19.
The test runs on a single instance (federation is v2), so the "subscriber" and
"publisher" are the same actor. That's intentional — milestone 1 proves the
mechanism; milestone 2 spreads it across instances.
**Test script:** `next/tests/smoke_app.sh`
```bash
#!/usr/bin/env bash
set -euo pipefail
# Assumes 9a has already run (fresh kernel optional; can run alongside).
TOKEN=$(cat next/data/keys/publish.token)
BASE=http://localhost:9999
# 1. Verify "Topic" subscription type and "Subscribe" verb are NOT yet defined.
curl -s "$BASE/define-registry?kind=subscription_types" \
| jq -e 'map(select(.name == "Topic")) | length == 0'
# 2. Publish DefineSubscription{name: "Topic", ...}
TOPIC_DEF=$(cat <<'JSON'
{
"type": "Create",
"object": {
"type": "DefineSubscription",
"name": "Topic",
"schema": "(fn (sub) (string? (-> sub :tag)))",
"match": "(fn (sub act) (and (= (:type act) \"Note\") (member? (-> sub :tag) (or (-> act :object :tags) (list)))))",
"delivery": "{:default :push :modes (list :push :pull)}"
}
}
JSON
)
curl -s -X POST "$BASE/activity" \
-H "Authorization: Bearer $TOKEN" -d "$TOPIC_DEF" | jq -e '.cid'
# 3. Verify Topic IS now a known subscription type.
curl -s "$BASE/define-registry?kind=subscription_types" \
| jq -e 'map(select(.name == "Topic")) | length == 1'
# 4. Subscribe to the "smoketest" topic.
SUBSCRIBE=$(cat <<'JSON'
{
"type": "Subscribe",
"object": {"type": "Topic", "tag": "smoketest"}
}
JSON
)
SUB_CID=$(curl -s -X POST "$BASE/activity" \
-H "Authorization: Bearer $TOKEN" -d "$SUBSCRIBE" | jq -r '.cid')
# 5. Verify subscriptions projection has the new entry.
sleep 1
curl -s "$BASE/projections/subscriptions" \
| jq -e '.["https://next.rose-ash.com/actors/next"] | map(select(.type == "Topic")) | length == 1'
# 6. Define a projection that records matched activities (per-application
# namespace would happen via DefineApplication in v1.x; for v1 the
# projection is global to the actor).
TOPIC_PROJ=$(cat <<'JSON'
{
"type": "Create",
"object": {
"type": "DefineProjection",
"name": "topic-events",
"initial-state": "{}",
"fold": "(fn (state act) (if (and (= (:type act) \"Note\") (member? \"smoketest\" (or (-> act :object :tags) (list)))) (assoc-in state [(:cid act)] act) state))"
}
}
JSON
)
curl -s -X POST "$BASE/activity" \
-H "Authorization: Bearer $TOKEN" -d "$TOPIC_PROJ" | jq -e '.cid'
# 7. Define a trigger: when a Topic{smoketest} subscription matches, publish
# a TestEcho activity. We need an "Echo" activity type first.
ECHO_DEF=$(cat <<'JSON'
{
"type": "Create",
"object": {
"type": "DefineActivity",
"name": "TestEcho",
"schema": "(fn (act) (cid? (-> act :object :echoes)))",
"semantics": "(fn (state act) state)"
}
}
JSON
)
curl -s -X POST "$BASE/activity" \
-H "Authorization: Bearer $TOKEN" -d "$ECHO_DEF" | jq -e '.cid'
TRIGGER=$(cat <<JSON
{
"type": "Create",
"object": {
"type": "DefineTrigger",
"name": "echo-on-smoketest",
"when-subscription": "$SUB_CID",
"cascade-limit": 1,
"then": "(fn (act sub env) {:publish (list {:type \"TestEcho\" :object {:echoes (:cid act)}})})"
}
}
JSON
)
curl -s -X POST "$BASE/activity" \
-H "Authorization: Bearer $TOKEN" -d "$TRIGGER" | jq -e '.cid'
# 8. Capture outbox length so we can detect new entries.
BEFORE=$(curl -s "$BASE/actors/next/outbox?page=true" \
| jq -r '.orderedItems | length')
# 9. Publish a Note tagged "smoketest" — should match subscription, fire trigger,
# cause TestEcho to be published.
NOTE=$(cat <<'JSON'
{
"type": "Create",
"object": {
"type": "Note",
"content": "hello reactive world",
"tags": ["smoketest"]
}
}
JSON
)
NOTE_CID=$(curl -s -X POST "$BASE/activity" \
-H "Authorization: Bearer $TOKEN" -d "$NOTE" | jq -r '.cid')
# 10. Wait for projection + trigger.
sleep 2
# 11. Verify topic-events projection captured the Note.
curl -s "$BASE/projections/topic-events" \
| jq -e ". | to_entries | length == 1"
# 12. Verify outbox grew by exactly TWO activities (the Note + the trigger's TestEcho).
AFTER=$(curl -s "$BASE/actors/next/outbox?page=true" \
| jq -r '.orderedItems | length')
[[ $((AFTER - BEFORE)) == 2 ]] || { echo "expected +2 activities, got $((AFTER - BEFORE))"; exit 1; }
# 13. Verify the latest activity is a TestEcho referencing the original Note's CID.
curl -s "$BASE/actors/next/outbox?page=true" \
| jq -e ".orderedItems[0] | .type == \"TestEcho\" and .object.echoes == \"$NOTE_CID\""
# 14. Negative case: publish a Note WITHOUT the "smoketest" tag — must NOT
# trigger, must NOT echo.
BEFORE2=$(curl -s "$BASE/actors/next/outbox?page=true" | jq -r '.orderedItems | length')
NOTE_OTHER=$(cat <<'JSON'
{"type": "Create", "object": {"type": "Note", "content": "no match", "tags": ["other"]}}
JSON
)
curl -s -X POST "$BASE/activity" \
-H "Authorization: Bearer $TOKEN" -d "$NOTE_OTHER" | jq -e '.cid'
sleep 2
AFTER2=$(curl -s "$BASE/actors/next/outbox?page=true" | jq -r '.orderedItems | length')
[[ $((AFTER2 - BEFORE2)) == 1 ]] || { echo "expected +1 activity (no echo), got $((AFTER2 - BEFORE2))"; exit 1; }
# 15. Cascade limit check: prove the trigger doesn't recursively echo TestEcho.
# The TestEcho activity itself should NOT match the Topic{smoketest}
# subscription (it's not a Note), so no cascade, but verify cascade-depth
# was set to 1 on the echo so a future trigger on TestEcho would refuse.
LATEST_ECHO=$(curl -s "$BASE/actors/next/outbox?page=true" \
| jq -r '.orderedItems | map(select(.type == "TestEcho")) | .[0]')
echo "$LATEST_ECHO" | jq -e '."cascade-depth" == 1'
# 16. Restart kernel; verify subscription, trigger, projection all survive.
./next/scripts/stop.sh
./next/scripts/start.sh
sleep 2
curl -s "$BASE/projections/subscriptions" \
| jq -e '.["https://next.rose-ash.com/actors/next"] | map(select(.type == "Topic")) | length == 1'
curl -s "$BASE/projections/topic-events" | jq -e ". | to_entries | length >= 1"
curl -s "$BASE/define-registry?kind=triggers" \
| jq -e 'map(select(.name == "echo-on-smoketest")) | length == 1'
echo "✓ Reactive application smoke test passed — Subscribe + Trigger + Projection demonstrated end-to-end"
```
**What this proves (and what it doesn't):**
Proves:
- `DefineSubscription` + `Subscribe` mechanism works end-to-end.
- Subscription's `match-fn` evaluates correctly in pure mode against inbound
activities.
- `DefineTrigger` fires on subscription matches.
- Trigger's `then-sx` can publish derived activities (the `:publish` result).
- Cascade-depth metadata propagates correctly.
- Subscription state, trigger registration, and projection state all survive
kernel restart (snapshot + log replay).
- The full reactive application loop works without any kernel code changes
between defining the components and exercising them.
Does NOT prove (deferred to milestone 2+):
- Cross-instance subscriptions (federation).
- Trigger `:effect` results calling effectful primitives.
- `DefineApplication` bundle install/update/fork.
- Per-application namespace isolation.
- Cascade prevention against malicious cascading from peer instances.
**Acceptance for 9b:** smoke test exits 0. Like 9a, **zero fed-sx kernel code
changes** between defining the application components and observing them
operate.
---
## Acceptance criteria for milestone 1
All of:
1. **Each step's test suite passes** (`bash next/tests/<step>.sh`).
2. **Both smoke tests pass** (`bash next/tests/smoke_pin.sh` and
`bash next/tests/smoke_app.sh`).
3. **Erlang-on-SX baseline preserved** — adding fed-sx kernel modules in
`next/kernel/*.erl` doesn't break Phase 1-8 conformance.
4. **Restart durability** — kill the kernel mid-write, restart, projections
resume from snapshot, no log corruption.
5. **Manual Mastodon poke** — point a Mastodon account at
`https://next.rose-ash.com/actors/next` and verify the actor doc fetches and
webfinger discovery works (read-only AP interop, no follow).
## What lands when
This is the work-order an agent (or human) follows. Steps 1-3 can be done in
parallel after the Erlang Phase 8 BIFs land. Steps 4-7 are sequential. Step 8
can start in parallel with step 7. Step 9 is the integration test.
```
Phase 7+8 (loops/erlang) ───┐
│
▼
┌─── Step 1 ──┬─── Step 2 ──┬─── Step 3
│ │ │
└─────────────┼─── Step 4 ──┴────┐
│ │
└─── Step 5 ───────┤
│
Step 6 ─────┤
│
Step 7 ─────┤
│
Step 8 ─────┤
│
Step 9 ─────┘
```
Estimated effort if done by a focused agent loop, one feature per iteration:
~30-50 commits across all 9 steps. Could plausibly be a `loops/fed-sx` workstream
once Phase 7+8 are done.
## What's deferred to milestone 2
- **Federation** (the second-biggest piece). `POST /inbox`, Follow lifecycle,
delivery queue, backfill, capability negotiation between peers. Whole of
design §13.
- **Multi-actor** with per-user OAuth and capability tokens. Design §9.5.
- **IPFS storage backend** as a `DefineStorage` entry. Design §15.3.
- **Browser client + operator dashboard** (probably in Elm-on-SX or similar).
- **Rich verbs**: `Endorse`, `Supersede`, `Test`, `Build`, `Compose`, `Note`,
`Announce`. All defined as `DefineActivity` artifacts, federated.
- **Cross-host conformance** — Python/JS/Haskell hosts running fed-sx. Design
§11.8.
- **OpenTimestamps proofs** as a `DefineProof` entry.
- **Performance work** — JIT-compiled folds, snapshot acceleration, federation
batching.
Milestone 2 unlocks "real federation between two fed-sx instances." Milestone 3
is the rose-ash port (blog, market, events, federation, account, orders) as
fed-sx applications.
---
## Appendix A: open questions for milestone 1
A few things still under-specified; resolve as work begins.
1. **HTTP server library.** Does the Phase 8 `http:listen/2` BIF wrap an
existing OCaml HTTP server (the sx.rose-ash.com one) or something simpler?
Implementation choice deferred to Phase 8.
2. **JSON-LD library.** AP wire format requires JSON-LD canonicalization for
signature coverage. Either pull a library or write a minimal subset for the
shapes we actually use. Probably the latter — our envelope is well-defined.
3. **Bearer token rotation.** v1 uses a single env-var token. Token rotation
without restart needs registry-style mgmt; can wait.
4. **Snapshot rate limits.** Default in design is "every 1000 activities or
60 seconds." Tunable per-projection later; v1 uses the default.
5. **Genesis bundle format.** Dag-cbor map per §12.2; concrete schema needs
one round of refinement once we author the actual definitions in step 4.