erlang: sync fed-sx + opcode-ext plans; add Phase 9 (specialized opcodes)
This commit is contained in:
@@ -122,10 +122,30 @@ Replace today's hardcoded BIF dispatch (`er-apply-bif`/`er-apply-remote-bif` in
|
||||
- [ ] `sqlite:open/1`, `sqlite:close/1`, `sqlite:exec/2`, `sqlite:query/2` — **BLOCKED** (no SQLite primitive). See Blockers.
|
||||
- [x] Tests: 1 round-trip per BIF; suite name `ffi`; conformance scoreboard auto-picks it up — **+14 ffi tests** at 637/637 total. Suite covers the 3 implemented file BIFs (9 tests: write-ok, read-ok-tag, payload-is-binary, byte_size content, missing-enoent, bad-path-enoent, binary-payload round-trip, delete-ok, read-after-delete-enoent) plus 5 negative asserts (one per blocked BIF — `crypto:hash`/`cid:from_bytes`/`file:list_dir`/`httpc:request`/`sqlite:exec`) so this suite fails fast if a future iteration adds a wrapper without registering proper tests. Target "+40 ffi tests" was relative to the original 5-BIF-family plan; with 5 of those families blocked on host primitives, the achievable count is 14 — the suite scaffolding is what matters and is ready to accept the remaining tests when the primitives land.
|
||||
|
||||
### Phase 9 — specialized opcodes (the BEAM analog)
|
||||
|
||||
**Driver:** Erlang-on-SX going through the general-purpose CEK machine has architectural perf ceilings (call/cc per receive, env-copy per call, mailbox rebuild on delete). The fix is specialized bytecode opcodes that bypass the general machinery for hot Erlang operations. Targets: 100k+ message hops/sec, 1M-process spawn in under 30sec. Layered perf strategy: Layer 1 (this) = specialized opcodes; Layer 2 (Phase 10, deferred) = multi-core scheduler.
|
||||
|
||||
**Architectural note:** opcodes get developed in `lib/erlang/vm/` (in scope). The **opcode extension mechanism in `hosts/ocaml/`** (Phase 9a) is **out of scope** for this loop — log as Blocker until a session that owns `hosts/` lands it. Sub-phases 9b-9g design and test opcodes against a stub dispatcher in the meantime; integrate when 9a is available.
|
||||
|
||||
**Shared-opcode discipline:** opcodes that another language port could plausibly use (pattern match, perform/handle, record access) get prepared for **chiselling out to `lib/guest/vm/`** when a second use materialises. Same lib/guest pattern, applied at the bytecode layer. Don't pre-extract; do annotate candidates in commit messages.
|
||||
|
||||
- [ ] **9a — Opcode extension mechanism** (in `hosts/ocaml/evaluator/`) — **OUT OF SCOPE for this loop**. Log as Blocker. Lets `lib/<lang>/vm/` register opcodes without modifying SX VM core. Design lives in `plans/sx-vm-opcode-extension.md`.
|
||||
- [ ] **9b — `OP_PATTERN_TUPLE` / `OP_PATTERN_LIST` / `OP_PATTERN_BINARY`**: specialized pattern-match opcodes for Erlang's bread-and-butter `case` clauses. Replace SX-`case` dispatch on the hot path. Tests: every pattern shape, including nested. Conformance must remain 637/637 + all prior. Candidate for chiselling to `lib/guest/vm/match.sx`.
|
||||
- [ ] **9c — `OP_PERFORM` / `OP_HANDLE`** (algebraic effects style): replace the call/cc + raise/guard machinery used for `receive` suspension. Pure Erlang interface unchanged; underlying mechanism specialized. Candidate for chiselling (Scheme call/cc, OCaml 5 effects, miniKanren all want the same thing).
|
||||
- [ ] **9d — `OP_RECEIVE_SCAN`**: built on 9c. Specialized opcode for selective receive — scans mailbox in pattern order, suspends + binds on match. Should give 10-100× speedup on receive-heavy workloads (ring benchmark, bank, fib_server).
|
||||
- [ ] **9e — `OP_SPAWN` / `OP_SEND` + lightweight scheduler**: per-process register/heap layout, scheduler that runs Erlang bytecode units rather than going through general SX evaluator each time. Process record fields become VM register slots. Target: spawn cost under 50µs, send cost under 5µs.
|
||||
- [ ] **9f — BIF dispatch table**: `OP_BIF_<name>` for hot BIFs (`length/1`, `hd/1`, `tl/1`, `element/2`, `lists:reverse/1`, etc.) — direct dispatch, no registry lookup. Cold BIFs continue through the general dispatch path.
|
||||
- [ ] **9g — Conformance + perf bench**: full Phase 1-8 conformance must pass on the new VM. Ring benchmark target: **100k+ hops/sec at N=1000** (current ~30/sec → ~3000× speedup target). 1M-process spawn target: **under 30 seconds** (current ~9h extrapolation → ~1000× speedup target). Document achieved numbers in `lib/erlang/bench_ring_results.md`.
|
||||
|
||||
**Acceptance:** ring benchmark hits the 100k hops/sec target. All prior phase tests pass. Two opcodes chiselled to `lib/guest/vm/` (or annotated as candidates with a written rationale).
|
||||
|
||||
## Progress log
|
||||
|
||||
_Newest first._
|
||||
|
||||
- **2026-05-14 Phase 9 scoped + supporting plan files synced** — Copied three plan files from `/root/rose-ash/plans/` (architecture branch) that this worktree was missing: `fed-sx-design.md` (124KB, the substrate design referenced from Phase 7/8 drivers), `fed-sx-milestone-1.md` (33KB, first concrete implementation milestone), `sx-vm-opcode-extension.md` (19KB, the prerequisite for Phase 9a — designs how `lib/<lang>/vm/` registers opcodes against the OCaml SX VM core). Then appended **Phase 9 — specialized opcodes (the BEAM analog)** to `plans/erlang-on-sx.md` covering sub-phases 9a-9g: 9a (opcode extension mechanism in `hosts/ocaml/`) is out-of-scope for this loop (will be logged as a Blocker when the next iteration tries to start it); 9b-9g (PATTERN_TUPLE/LIST/BINARY, PERFORM/HANDLE, RECEIVE_SCAN, SPAWN/SEND + lightweight scheduler, BIF dispatch table, conformance + perf bench) can be designed and tested against a stub dispatcher in the meantime. Targets: ring benchmark 100k+ hops/sec at N=1000 (~3000× speedup), 1M-process spawn under 30sec (~1000× speedup). Plan framing intact for Phase 7/8 — those reflect the actual implementation done in this loop; the architecture-branch framing diverges in language but the work is equivalent. No code touched this iteration. Total **637/637** unchanged.
|
||||
|
||||
- **2026-05-14 ffi test suite extracted, conformance scoreboard auto-picks it up** — New `lib/erlang/tests/ffi.sx` with its own counter trio (`er-ffi-test-count`/`-pass`/`-fails`) and `er-ffi-test` helper following the same pattern as runtime/eval/ring tests. The 10 file BIF eval tests from the previous iteration moved out of `eval.sx` (eval dropped from 395 to 385 tests) and into the new suite where they're now 9 tests (consolidated the two write+read tests). `conformance.sh` updated: added `ffi` to `SUITES` array with `er-ffi-test-pass`/`-count` symbols, added `(load "lib/erlang/tests/ffi.sx")` after `fib_server.sx`, added `(epoch 109) (eval "(list er-ffi-test-pass er-ffi-test-count)")`. Scoreboard markdown auto-updated to include the row. Suite also asserts that the 5 blocked BIFs (`crypto:hash`, `cid:from_bytes`, `file:list_dir`, `httpc:request`, `sqlite:exec`) are NOT yet registered — turns a future "added the wrapper but forgot to extend ffi tests" into a hard failure. One eval-comparison gotcha en route: SX's `=` does identity equality on dicts so comparing two separately-constructed `(er-mk-atom "true")` values is false; the existing eval suite has an `eev-deep=` helper that handles this, but the simpler fix in ffi was to extract `:name` via `ffi-nm` and compare strings. Total **637/637** (+14 ffi). Phase 8 fully ticked aside from the BLOCKED bullets — those remain unchecked with explicit Blockers references.
|
||||
|
||||
- **2026-05-14 file BIFs landed; crypto/cid/list_dir/http/sqlite blocked on missing host primitives** — Three new FFI BIFs registered in `runtime.sx`: `file:read_file/1`, `file:write_file/2`, `file:delete/1`. Each wraps the SX-host primitive (`file-read`, `file-write`, `file-delete`) inside a `guard` that converts thrown exception strings into Erlang `{error, Reason}` tuples. New helper `er-classify-file-error` does loose pattern-matching on the error message using `string-contains?` to map to standard POSIX-style reasons: `"No such"` → `enoent`, `"Permission denied"` → `eacces`, `"Not a directory"` → `enotdir`, `"Is a directory"` → `eisdir`, fallback `posix_error`. Filenames coerce through `er-source-to-string` so SX strings, Erlang binaries, and Erlang char-code lists all work. Read returns `{ok, Binary}` (bytes via `(map char->integer (string->list ...))` then `er-mk-binary`); write returns bare `ok`; delete returns bare `ok`. Bootstrap registrations added at the bottom of `er-register-builtin-bifs!` under `"file"`. 10 new eval tests: write-then-read round-trip, ok-tag, payload is binary, byte_size content, missing-file `enoent`, delete-ok, read-after-delete `enoent`, write to non-existent dir `enoent`, binary payload (5 raw bytes) round-trip preserving byte count. Blockers entry added covering five Phase 8 BIFs whose host primitives don't exist in this SX runtime: `crypto:hash/2`, `cid:from_bytes/1`/`to_string/1`, `file:list_dir/1`, `httpc:request/4`, `sqlite:open/exec/query/close`. Fix path documented inline (architecture-branch iteration to register OCaml-side primitives). Total **633/633** (+10 eval).
|
||||
|
||||
2638
plans/fed-sx-design.md
Normal file
2638
plans/fed-sx-design.md
Normal file
File diff suppressed because it is too large
Load Diff
922
plans/fed-sx-milestone-1.md
Normal file
922
plans/fed-sx-milestone-1.md
Normal file
@@ -0,0 +1,922 @@
|
||||
# fed-sx Milestone 1 — Kernel + Registries + Pin Smoke Test
|
||||
|
||||
Concrete implementation plan for the smallest fed-sx that proves the architecture
|
||||
works end-to-end. Reference: `plans/fed-sx-design.md`. Prerequisite: Erlang-on-SX
|
||||
Phases 7 (hot reload) + 8 (FFI BIFs).
|
||||
|
||||
## Goal
|
||||
|
||||
Ship a single-instance, single-actor fed-sx server that:
|
||||
|
||||
1. Boots from a verified genesis bundle.
|
||||
2. Accepts and durably appends signed activities via `POST /activity`.
|
||||
3. Folds them into projections in real time.
|
||||
4. Serves AP-standard endpoints (actor, outbox, artifacts, capabilities).
|
||||
5. Demonstrates **two extensibility proof-points** end-to-end with zero kernel
|
||||
code changes between definition and use:
|
||||
- **Verb extensibility** (§5 meta-level): publish `DefineActivity{Pin}` +
|
||||
`DefineProjection{pin-state}`, then publish a `Pin` activity, observe it
|
||||
validated and projected.
|
||||
- **Reactive application extensibility** (§§18-19): publish
|
||||
`DefineSubscription{Topic}` + `Subscribe{topic: smoketest}` +
|
||||
`DefineTrigger{when: that subscription, then: publish TestEcho}`, then
|
||||
publish a tagged Note, observe the subscription match, the trigger fire,
|
||||
and the derived activity appear in the outbox.
|
||||
|
||||
Federation, multi-actor, advanced verbs, IPFS, browser UI, operator dashboard
|
||||
are **explicitly v2**.
|
||||
|
||||
## Non-goals (what milestone 1 deliberately does NOT do)
|
||||
|
||||
- **Federation.** No `POST /inbox` from peers, no `Follow`, no delivery queue, no
|
||||
webfinger discovery flow. Single instance only.
|
||||
- **Multi-actor.** Single domain actor (`acct:next@next.rose-ash.com`).
|
||||
- **IPFS / S3 storage backends.** Files on disk only.
|
||||
- **Advanced verbs.** No `Endorse`, `Supersede`, `Test`, `Build`, `Compose`,
|
||||
`Note`, `Announce`. Only the four bootstrap verbs (`Create`, `Update`, `Delete`)
|
||||
plus a defined-from-the-log `Pin` for the smoke test. (`Announce` deferred —
|
||||
no use case until federation exists.)
|
||||
- **Browser UI.** Curl-shaped API only.
|
||||
- **Operator dashboard, quarantine UX.** Logs only.
|
||||
- **Performance work.** Functional correctness first; perf when measured.
|
||||
- **Cross-host conformance test corpus.** Only the OCaml/Erlang-on-SX host runs
|
||||
fed-sx in v1; conformance suite for other hosts is v2.
|
||||
|
||||
## Architecture summary
|
||||
|
||||
```
|
||||
POST /activity
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────┐
|
||||
│ HTTP server (Erlang-on-SX)│
|
||||
└─────────────┬─────────────┘
|
||||
│
|
||||
┌─────────────▼──────────────┐
|
||||
│ Validation pipeline driver │
|
||||
│ (envelope→sig→schema→...) │
|
||||
└─────────────┬──────────────┘
|
||||
│
|
||||
┌─────────────▼──────────────┐
|
||||
│ Log append (JSONL segment) │ ← canonical
|
||||
└─────────────┬──────────────┘
|
||||
│
|
||||
┌─────────────▼──────────────┐
|
||||
│ Projection workers │ ← gen_server per
|
||||
│ (fold scheduler) │ projection
|
||||
└─────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Projection state
|
||||
(queryable via HTTP)
|
||||
|
||||
Native primitives (Erlang-on-SX BIFs from Phase 8):
|
||||
crypto:* cid:* fs:* http:* sqlite:*
|
||||
|
||||
Genesis bundle (binary-embedded SX):
|
||||
activity-types object-types projections
|
||||
validators codecs sig-suites
|
||||
```
|
||||
|
||||
## Build order
|
||||
|
||||
Eight steps in dependency order. Each step has concrete deliverables, testable
|
||||
in isolation, and a clear acceptance check.
|
||||
|
||||
| Step | Title | Depends on |
|
||||
|------|-------|------------|
|
||||
| **1** | Repo skeleton + canonical CID computation | Phase 8 (cid BIFs) |
|
||||
| **2** | Activity envelope + signature verify | Phase 8 (crypto BIFs) |
|
||||
| **3** | JSONL log + sequence numbers | Phase 8 (fs BIFs) |
|
||||
| **4** | Genesis bundle (SX sources + bundling + CID verification) | Step 1 |
|
||||
| **5** | Registry mechanism + bootstrap-projection dispatch | Steps 2, 4 |
|
||||
| **6** | Validation pipeline driver + `POST /activity` | Steps 2, 3, 5 |
|
||||
| **7** | Projection scheduler (gen_server per projection) | Steps 5, 6 |
|
||||
| **8** | HTTP server, AP endpoints, projection queries | Steps 6, 7 |
|
||||
| **9** | Smoke tests (Pin verb + reactive application) | Steps 1-8 |
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Repo skeleton + canonical CID
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
```
|
||||
next/
|
||||
├── README.md # what this is
|
||||
├── kernel/ # Erlang-on-SX
|
||||
│ └── (empty for now)
|
||||
├── genesis/ # core SX bootstrap definitions
|
||||
│ └── (empty for now)
|
||||
├── tests/ # smoke test scripts
|
||||
│ └── (empty for now)
|
||||
└── data/ # gitignored runtime state
|
||||
├── log/
|
||||
├── objects/
|
||||
├── snapshots/
|
||||
├── indexes/
|
||||
└── keys/
|
||||
```
|
||||
|
||||
Plus one Erlang-on-SX module:
|
||||
|
||||
```erlang
|
||||
% next/kernel/cid.erl
|
||||
-module(cid).
|
||||
-export([from_sx/1, to_string/1, from_string/1, equals/2]).
|
||||
|
||||
from_sx(SxValue) ->
|
||||
Cbor = cid:cbor_encode(canonicalize_sx(SxValue)),
|
||||
Hash = crypto:sha2_256(Cbor),
|
||||
cid:from_bytes(<<"raw">>, Hash). % defaults to dag-cbor codec
|
||||
|
||||
canonicalize_sx(V) -> ... % sorts dict keys, normalizes strings
|
||||
```
|
||||
|
||||
**Tests:**
|
||||
- Same SX value → same CID across multiple invocations.
|
||||
- Different SX values → different CIDs.
|
||||
- Whitespace/comment differences in source → identical CIDs (parsed AST identical).
|
||||
- Reordered dict keys → identical CIDs (sorted-key canonicalization).
|
||||
- Cross-host parity (just OCaml host for v1, but write the test so adding hosts is mechanical).
|
||||
|
||||
**Acceptance:** `bash next/tests/cid.sh` passes 10+ cases.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Activity envelope + signature verify
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
```erlang
|
||||
% next/kernel/envelope.erl
|
||||
-module(envelope).
|
||||
-export([validate_shape/1, canonical_bytes/1, verify_signature/2]).
|
||||
|
||||
% Envelope shape per design §3.1:
|
||||
% #{id, type, actor, published, to, cc, audience_extras,
|
||||
% object | target | origin | result,
|
||||
% capabilities_required, proofs, signature}
|
||||
validate_shape(Activity) -> ok | {error, Reason}.
|
||||
|
||||
canonical_bytes(Activity) ->
|
||||
% Strip signature, canonicalize via dag-cbor, return bytes for sig coverage
|
||||
Stripped = maps:remove(signature, Activity),
|
||||
cid:cbor_encode(canonicalize_for_sig(Stripped)).
|
||||
|
||||
verify_signature(Activity, ActorState) ->
|
||||
% Time-aware: find key with id == sig.key_id that was active at published
|
||||
% Per design §9.6
|
||||
...
|
||||
```
|
||||
|
||||
**Tests:**
|
||||
- Envelope shape: required fields present (id, type, actor, published, signature)
|
||||
- Envelope shape: type is a known activity-type or unknown-but-string
|
||||
- Envelope shape: signature has key_id, algorithm, value
|
||||
- Sig verify: valid RSA-SHA256 signature against published key → ok
|
||||
- Sig verify: valid Ed25519 signature → ok
|
||||
- Sig verify: tampered envelope → fail
|
||||
- Sig verify: key superseded before activity timestamp → fail
|
||||
- Sig verify: key superseded after activity timestamp → ok (historical valid)
|
||||
|
||||
**Acceptance:** `bash next/tests/envelope.sh` passes 15+ cases.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — JSONL log + sequence numbers
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
```erlang
|
||||
% next/kernel/log.erl
|
||||
-module(log).
|
||||
-export([open/1, append/2, read_segment/2, tip/1, replay/3]).
|
||||
|
||||
% Per design §15.2: per-actor outbox, segments cap ~64MB,
|
||||
% format = JSONL (one canonical JSON-LD activity per line)
|
||||
|
||||
open(ActorId) ->
|
||||
BasePath = log_path_for_actor(ActorId),
|
||||
fs:mkdir_p(BasePath),
|
||||
{ok, #{base => BasePath, current => current_segment(BasePath), seq => next_seq(BasePath)}}.
|
||||
|
||||
append(LogState, Activity) ->
|
||||
Json = jsonld:encode(Activity),
|
||||
Path = current_segment_path(LogState),
|
||||
Line = <<Json/binary, "\n">>,
|
||||
fs:append_file(Path, Line),
|
||||
NewSeq = LogState#{seq := LogState.seq + 1},
|
||||
rotate_if_needed(NewSeq).
|
||||
|
||||
% replay/3 calls Fun(Activity, Acc) for every activity in chronological order
|
||||
replay(LogState, InitAcc, Fun) -> ...
|
||||
```
|
||||
|
||||
**Tests:**
|
||||
- Append + read back gives identical activity (round-trip).
|
||||
- Sequence numbers monotonic and gap-free per actor.
|
||||
- Segment rotation at size threshold.
|
||||
- Replay visits all activities in append order across multiple segments.
|
||||
- Restart preserves tip pointer (seq number resumes correctly).
|
||||
- Concurrent appends (using gen_server-mediated access) are serialized correctly.
|
||||
|
||||
**Acceptance:** `bash next/tests/log.sh` passes 10+ cases.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Genesis bundle
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
Genesis bundle SX sources (per design §12.2). Each is a small SX file authored
|
||||
by hand for the bootstrap set:
|
||||
|
||||
```
|
||||
next/genesis/
|
||||
├── manifest.sx # bundle root: lists all definitions
|
||||
├── activity-types/
|
||||
│ ├── create.sx # DefineActivity{name: "Create", ...}
|
||||
│ ├── update.sx
|
||||
│ └── delete.sx
|
||||
├── object-types/
|
||||
│ ├── sx-artifact.sx
|
||||
│ ├── note.sx
|
||||
│ ├── tombstone.sx
|
||||
│ ├── define-activity.sx # DefineObject for the Define* meta types
|
||||
│ ├── define-object.sx
|
||||
│ ├── define-projection.sx
|
||||
│ ├── define-validator.sx
|
||||
│ ├── define-codec.sx
|
||||
│ ├── define-sig-suite.sx
|
||||
│ └── snapshot.sx
|
||||
├── projections/
|
||||
│ ├── activity-log.sx # identity projection
|
||||
│ ├── by-type.sx
|
||||
│ ├── by-actor.sx
|
||||
│ ├── by-object.sx
|
||||
│ ├── actor-state.sx
|
||||
│ ├── define-registry.sx # the chicken-and-egg projection
|
||||
│ └── audience-graph.sx
|
||||
├── validators/
|
||||
│ ├── envelope-shape.sx
|
||||
│ ├── signature.sx
|
||||
│ └── type-schema.sx
|
||||
├── codecs/
|
||||
│ ├── dag-cbor.sx # delegates to cid:cbor_encode/decode BIFs
|
||||
│ ├── raw.sx
|
||||
│ └── dag-json.sx
|
||||
├── sig-suites/
|
||||
│ ├── rsa-sha256-2018.sx
|
||||
│ └── ed25519-2020.sx
|
||||
└── audience/
|
||||
├── public.sx
|
||||
├── followers.sx
|
||||
└── direct.sx
|
||||
```
|
||||
|
||||
Plus a build-time bundler:
|
||||
|
||||
```erlang
|
||||
% next/kernel/bootstrap.erl
|
||||
-module(bootstrap).
|
||||
-export([build_genesis/1, verify_genesis/1, load_genesis/1]).
|
||||
|
||||
build_genesis(SourceDir) ->
|
||||
% Walk SourceDir, parse each .sx file, build a single dag-cbor bundle,
|
||||
% compute its CID, write bundle.cbor + CID to data/genesis/
|
||||
...
|
||||
|
||||
verify_genesis(BundlePath) ->
|
||||
% Compute CID of the bundle as loaded; compare to expected (hardcoded
|
||||
% in the kernel binary). Mismatch → halt.
|
||||
...
|
||||
|
||||
load_genesis(BundlePath) ->
|
||||
% Parse the bundle, register all definitions in the in-memory registry
|
||||
...
|
||||
```
|
||||
|
||||
**Tests:**
|
||||
- All genesis SX files parse cleanly.
|
||||
- Bundle CID is deterministic (rebuild same sources → same CID).
|
||||
- Bundle reload reproduces the exact same registry state.
|
||||
- Tampered bundle → `verify_genesis` returns `{error, cid_mismatch}`.
|
||||
|
||||
**Acceptance:** `bash next/tests/bootstrap.sh` passes; `next/data/genesis/bundle.cbor`
|
||||
created with a known stable CID.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Registry mechanism + bootstrap dispatch
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
Registries are gen_servers, one per kind, each holding the active version map:
|
||||
|
||||
```erlang
|
||||
% next/kernel/registry.erl
|
||||
-module(registry).
|
||||
-behaviour(gen_server).
|
||||
-export([start_link/0, lookup/2, register/3, list/1]).
|
||||
% Internal state:
|
||||
% #{activity_types => #{Name => #{cid, schema_fn, semantics_fn, supersedes}},
|
||||
% object_types => ...,
|
||||
% projections => ...,
|
||||
% validators => ...,
|
||||
% codecs => ...,
|
||||
% sig_suites => ...,
|
||||
% ...}
|
||||
|
||||
lookup(Kind, Name) -> {ok, Entry} | {error, not_found}.
|
||||
register(Kind, Name, Entry) -> ok | {error, Reason}.
|
||||
list(Kind) -> [#{name, cid}].
|
||||
```
|
||||
|
||||
The `define-registry` projection's fold updates this gen_server's state when
|
||||
new `Define*` activities arrive. (Bootstrapping circle resolved: at startup,
|
||||
`bootstrap:load_genesis/1` populates the registry directly; from then on, the
|
||||
projection fold maintains it.)
|
||||
|
||||
**Tests:**
|
||||
- After genesis load, `registry:list(activity_types)` returns Create/Update/Delete.
|
||||
- `registry:lookup(activity_types, "Create")` returns the schema and semantics.
|
||||
- A new `DefineActivity{name: "Pin"}` activity (synthesised, hand-signed for the
|
||||
test) routes through the projection fold, ends up in the registry.
|
||||
- Lookup never caches across activities (verified by introducing a new definition
|
||||
mid-test and confirming the next lookup sees it).
|
||||
|
||||
**Acceptance:** `bash next/tests/registry.sh` passes 10+ cases.
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Validation pipeline + POST /activity
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
```erlang
|
||||
% next/kernel/pipeline.erl
|
||||
-module(pipeline).
|
||||
-export([validate_inbound/1, validate_outbound/1]).
|
||||
|
||||
% Per design §14, run stages in order, halt on first failure.
|
||||
validate_inbound(Activity) ->
|
||||
Stages = [
|
||||
fun stage_envelope/1,
|
||||
fun stage_signature/1,
|
||||
fun stage_replay/1,
|
||||
fun stage_audience/1,
|
||||
fun stage_activity_schema/1,
|
||||
fun stage_object_schema/1,
|
||||
fun stage_content_validators/1,
|
||||
fun stage_capabilities/1,
|
||||
fun stage_trust/1
|
||||
],
|
||||
run_stages(Activity, Stages).
|
||||
|
||||
validate_outbound(Activity) ->
|
||||
% Subset of inbound stages (no replay, no trust check; auth done at HTTP layer)
|
||||
...
|
||||
```
|
||||
|
||||
```erlang
|
||||
% next/kernel/outbox.erl
|
||||
-module(outbox).
|
||||
-export([publish/2]).
|
||||
|
||||
publish(ActorId, ActivityRequest) ->
|
||||
Activity = construct_envelope(ActorId, ActivityRequest),
|
||||
Signed = sig:sign(Activity, ActorId),
|
||||
case pipeline:validate_outbound(Signed) of
|
||||
ok ->
|
||||
log:append(actor_log(ActorId), Signed),
|
||||
projection:async_fold(Signed),
|
||||
{ok, #{cid => cid:from_sx(Signed),
|
||||
ap_id => maps:get(id, Signed)}};
|
||||
{error, Reason} ->
|
||||
{error, Reason}
|
||||
end.
|
||||
```
|
||||
|
||||
**Tests:**
|
||||
- Valid activity through full pipeline → appended to log.
|
||||
- Bad envelope → 400, not in log.
|
||||
- Bad signature → 401, not in log.
|
||||
- Replayed activity → 200 duplicate, not re-appended.
|
||||
- Schema violation (e.g. Create with no object) → 422.
|
||||
- Activity logged before projection completes (async).
|
||||
|
||||
**Acceptance:** `bash next/tests/pipeline.sh` passes 15+ cases.
|
||||
|
||||
---
|
||||
|
||||
## Step 7 — Projection scheduler
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
```erlang
|
||||
% next/kernel/projection.erl
|
||||
-module(projection).
|
||||
-export([start_link/1, async_fold/1, query/2, snapshot/1]).
|
||||
-behaviour(gen_server).
|
||||
|
||||
% One gen_server per active projection. State:
|
||||
% #{cid, name, fold_fn, current_state, log_tip,
|
||||
% snapshot_dir, last_snapshot_at}
|
||||
|
||||
% async_fold/1 broadcasts a new activity to every projection gen_server;
|
||||
% each folds it into its own state. Failures (gas, sandbox violation)
|
||||
% tag the activity but don't affect log durability.
|
||||
|
||||
% query/2 returns current state (or state-as-of)
|
||||
% snapshot/1 forces a snapshot now (also runs periodically)
|
||||
```
|
||||
|
||||
```erlang
|
||||
% next/kernel/sandbox.erl
|
||||
-module(sandbox).
|
||||
-export([eval_pure/2, eval_crypto/2, eval_effectful/3]).
|
||||
|
||||
% eval_pure runs an SX function in pure mode: no IO platform, gas budget,
|
||||
% deterministic. Used by projection folds, validators, audience predicates.
|
||||
% Wrapper over the SX runtime evaluator with a stripped platform.
|
||||
```
|
||||
|
||||
**Tests:**
|
||||
- New activity → all projections fold it concurrently.
|
||||
- Projection fold completes within gas budget.
|
||||
- Gas-exhausting fold → activity tagged, projection state unchanged, no kernel crash.
|
||||
- Sandbox violation (fold tries IO) → same handling.
|
||||
- Snapshot create + reload → state matches.
|
||||
- Snapshot CID stable across kernel restarts.
|
||||
|
||||
**Acceptance:** `bash next/tests/projection.sh` passes 15+ cases.
|
||||
|
||||
---
|
||||
|
||||
## Step 8 — HTTP server + endpoints
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
Core endpoints (per design §16.1):
|
||||
|
||||
```
|
||||
GET /actors/<id> # actor doc
|
||||
GET /actors/<id>/outbox # OrderedCollection
|
||||
GET /actors/<id>/outbox?page=true # OrderedCollectionPage
|
||||
POST /activity # publish (auth: bearer token)
|
||||
GET /artifacts/<cid> # CID-addressed artifact
|
||||
GET /artifacts/<cid>/raw
|
||||
GET /projections # list of projections
|
||||
GET /projections/<name> # full state
|
||||
GET /projections/<name>?at=<ts> # time-travel
|
||||
GET /projections/<name>/<key> # indexed lookup
|
||||
GET /define-registry
|
||||
GET /.well-known/sx-capabilities
|
||||
GET /.well-known/webfinger
|
||||
```
|
||||
|
||||
```erlang
|
||||
% next/kernel/http_server.erl
|
||||
-module(http_server).
|
||||
-export([start/1, route/1]).
|
||||
|
||||
start(Port) ->
|
||||
http:listen(Port, fun ?MODULE:route/1).
|
||||
|
||||
route(Request) -> {Status, Headers, Body}.
|
||||
```
|
||||
|
||||
Content negotiation per `Accept`:
|
||||
- `application/activity+json` (default)
|
||||
- `application/cbor` (dag-cbor)
|
||||
- `application/json` (compact, no @context expansion)
|
||||
- `application/sx`
|
||||
|
||||
Auth on `POST /activity`: bearer token from env var `NEXT_PUBLISH_TOKEN`.
|
||||
|
||||
**Tests:**
|
||||
- Each endpoint returns expected shape for known artifact.
|
||||
- Content negotiation: same artifact in 4 representations.
|
||||
- 404 for unknown artifact CID.
|
||||
- 401 for `POST /activity` without token.
|
||||
- Pagination: outbox with > 50 activities returns OrderedCollectionPage.
|
||||
|
||||
**Acceptance:** `bash next/tests/http.sh` passes 20+ cases.
|
||||
|
||||
---
|
||||
|
||||
## Step 9 — Smoke tests
|
||||
|
||||
**The proof points.** Two end-to-end smoke tests demonstrate, between them, that
|
||||
fed-sx is genuinely a substrate for distributed reactive applications expressed
|
||||
as data — not a system you extend by writing kernel code.
|
||||
|
||||
- **9a — Pin smoke test (`next/tests/smoke_pin.sh`)** — verb extensibility:
|
||||
defining a new activity type and projection at runtime via `Define*`
|
||||
artifacts. Verifies the meta-level (§5).
|
||||
- **9b — Reactive application smoke test (`next/tests/smoke_app.sh`)** —
|
||||
application extensibility: defining a new subscription type, subscribing,
|
||||
registering a trigger, and observing the full reactive loop fire end-to-end
|
||||
without kernel code changes. Verifies §§18-19.
|
||||
|
||||
Both must pass for milestone 1 acceptance.
|
||||
|
||||
### Step 9a — Pin smoke test
|
||||
|
||||
**Test script:** `next/tests/smoke_pin.sh`
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# 0. Start a fresh fed-sx kernel (background)
|
||||
./next/scripts/start.sh fresh
|
||||
sleep 2
|
||||
TOKEN=$(cat next/data/keys/publish.token)
|
||||
|
||||
# 1. Verify actor exists
|
||||
curl -s http://localhost:9999/actors/next | jq -e '.type == "Person"'
|
||||
|
||||
# 2. Verify outbox has actor's first Create{Person}
|
||||
curl -s http://localhost:9999/actors/next/outbox?page=true \
|
||||
| jq -e '.orderedItems | length == 1 and .[0].type == "Create"'
|
||||
|
||||
# 3. Verify Pin is NOT a known activity type
|
||||
curl -s http://localhost:9999/define-registry?kind=activity_types \
|
||||
| jq -e '.[] | select(.name == "Pin") | length == 0' || exit 1
|
||||
|
||||
# 4. Publish DefineActivity{name: "Pin", schema: ..., semantics: ...}
|
||||
PIN_DEF=$(cat <<'JSON'
|
||||
{
|
||||
"type": "Create",
|
||||
"object": {
|
||||
"type": "DefineActivity",
|
||||
"name": "Pin",
|
||||
"schema": "(fn (act) (and (string? (-> act :object :path)) (cid? (-> act :object :cid))))",
|
||||
"semantics": "(fn (state act) (assoc-in state [:pins (-> act :object :path)] (-> act :object :cid)))"
|
||||
}
|
||||
}
|
||||
JSON
|
||||
)
|
||||
curl -s -X POST http://localhost:9999/activity \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/activity+json" \
|
||||
-d "$PIN_DEF" | jq -e '.cid' > /dev/null
|
||||
|
||||
# 5. Verify Pin IS now a known activity type
|
||||
curl -s http://localhost:9999/define-registry?kind=activity_types \
|
||||
| jq -e '.[] | select(.name == "Pin") | length == 1'
|
||||
|
||||
# 6. Also publish a DefineProjection{name: "pin-state"} that folds Pin into state
|
||||
PIN_PROJ=$(cat <<'JSON'
|
||||
{
|
||||
"type": "Create",
|
||||
"object": {
|
||||
"type": "DefineProjection",
|
||||
"name": "pin-state",
|
||||
"initial-state": "{}",
|
||||
"fold": "(fn (state act) (if (= (:type act) \"Pin\") (assoc state (-> act :object :path) (-> act :object :cid)) state))"
|
||||
}
|
||||
}
|
||||
JSON
|
||||
)
|
||||
curl -s -X POST http://localhost:9999/activity \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-d "$PIN_PROJ" | jq -e '.cid'
|
||||
|
||||
# 7. Now publish a Pin activity
|
||||
PIN=$(cat <<'JSON'
|
||||
{
|
||||
"type": "Pin",
|
||||
"object": {
|
||||
"type": "PinSpec",
|
||||
"path": "/docs/intro",
|
||||
"cid": "bafyreigh2akiscaildc3xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxxe"
|
||||
}
|
||||
}
|
||||
JSON
|
||||
)
|
||||
curl -s -X POST http://localhost:9999/activity \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-d "$PIN" | jq -e '.cid'
|
||||
|
||||
# 8. Verify Pin appears in outbox
|
||||
curl -s http://localhost:9999/actors/next/outbox?page=true \
|
||||
| jq -e '.orderedItems | map(select(.type == "Pin")) | length == 1'
|
||||
|
||||
# 9. Verify pin-state projection has the entry
|
||||
sleep 1 # allow async projection
|
||||
curl -s http://localhost:9999/projections/pin-state \
|
||||
| jq -e '."/docs/intro" == "bafyreigh2akiscaildc3xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxxe"'
|
||||
|
||||
# 10. Negative test: publish a malformed Pin (missing path) → expect 422
|
||||
BAD_PIN='{"type": "Pin", "object": {"cid": "bafy..."}}'
|
||||
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST http://localhost:9999/activity \
|
||||
-H "Authorization: Bearer $TOKEN" -d "$BAD_PIN")
|
||||
[[ "$HTTP_STATUS" == "422" ]] || { echo "expected 422, got $HTTP_STATUS"; exit 1; }
|
||||
|
||||
# 11. Restart kernel; verify state recovers
|
||||
./next/scripts/stop.sh
|
||||
./next/scripts/start.sh
|
||||
sleep 2
|
||||
curl -s http://localhost:9999/projections/pin-state \
|
||||
| jq -e '."/docs/intro" == "bafyreigh2akiscaildc3xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxxe"'
|
||||
|
||||
echo "✓ Pin smoke test passed — verb extensibility demonstrated end-to-end"
|
||||
```
|
||||
|
||||
**Acceptance for 9a:** smoke test exits 0. The whole flow happens with **zero
|
||||
fed-sx kernel code changes** between defining the verb and using it.
|
||||
|
||||
### Step 9b — Reactive application smoke test
|
||||
|
||||
**The bigger proof point.** Demonstrates that fed-sx supports distributed
|
||||
reactive applications composed of `DefineSubscription` + `DefineTrigger` +
|
||||
`DefineProjection` — the application model from §§18-19.
|
||||
|
||||
The test runs on a single instance (federation is v2), so the "subscriber" and
|
||||
"publisher" are the same actor. That's intentional — milestone 1 proves the
|
||||
mechanism; milestone 2 spreads it across instances.
|
||||
|
||||
**Test script:** `next/tests/smoke_app.sh`
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Assumes 9a has already run (fresh kernel optional; can run alongside).
|
||||
TOKEN=$(cat next/data/keys/publish.token)
|
||||
BASE=http://localhost:9999
|
||||
|
||||
# 1. Verify "Topic" subscription type and "Subscribe" verb are NOT yet defined.
|
||||
curl -s "$BASE/define-registry?kind=subscription_types" \
|
||||
| jq -e 'map(select(.name == "Topic")) | length == 0'
|
||||
|
||||
# 2. Publish DefineSubscription{name: "Topic", ...}
|
||||
TOPIC_DEF=$(cat <<'JSON'
|
||||
{
|
||||
"type": "Create",
|
||||
"object": {
|
||||
"type": "DefineSubscription",
|
||||
"name": "Topic",
|
||||
"schema": "(fn (sub) (string? (-> sub :tag)))",
|
||||
"match": "(fn (sub act) (and (= (:type act) \"Note\") (member? (-> sub :tag) (or (-> act :object :tags) (list)))))",
|
||||
"delivery": "{:default :push :modes (list :push :pull)}"
|
||||
}
|
||||
}
|
||||
JSON
|
||||
)
|
||||
curl -s -X POST "$BASE/activity" \
|
||||
-H "Authorization: Bearer $TOKEN" -d "$TOPIC_DEF" | jq -e '.cid'
|
||||
|
||||
# 3. Verify Topic IS now a known subscription type.
|
||||
curl -s "$BASE/define-registry?kind=subscription_types" \
|
||||
| jq -e 'map(select(.name == "Topic")) | length == 1'
|
||||
|
||||
# 4. Subscribe to the "smoketest" topic.
|
||||
SUBSCRIBE=$(cat <<'JSON'
|
||||
{
|
||||
"type": "Subscribe",
|
||||
"object": {"type": "Topic", "tag": "smoketest"}
|
||||
}
|
||||
JSON
|
||||
)
|
||||
SUB_CID=$(curl -s -X POST "$BASE/activity" \
|
||||
-H "Authorization: Bearer $TOKEN" -d "$SUBSCRIBE" | jq -r '.cid')
|
||||
|
||||
# 5. Verify subscriptions projection has the new entry.
|
||||
sleep 1
|
||||
curl -s "$BASE/projections/subscriptions" \
|
||||
| jq -e '.["https://next.rose-ash.com/actors/next"] | map(select(.type == "Topic")) | length == 1'
|
||||
|
||||
# 6. Define a projection that records matched activities (per-application
|
||||
# namespace would happen via DefineApplication in v1.x; for v1 the
|
||||
# projection is global to the actor).
|
||||
TOPIC_PROJ=$(cat <<'JSON'
|
||||
{
|
||||
"type": "Create",
|
||||
"object": {
|
||||
"type": "DefineProjection",
|
||||
"name": "topic-events",
|
||||
"initial-state": "{}",
|
||||
"fold": "(fn (state act) (if (and (= (:type act) \"Note\") (member? \"smoketest\" (or (-> act :object :tags) (list)))) (assoc-in state [(:cid act)] act) state))"
|
||||
}
|
||||
}
|
||||
JSON
|
||||
)
|
||||
curl -s -X POST "$BASE/activity" \
|
||||
-H "Authorization: Bearer $TOKEN" -d "$TOPIC_PROJ" | jq -e '.cid'
|
||||
|
||||
# 7. Define a trigger: when a Topic{smoketest} subscription matches, publish
|
||||
# a TestEcho activity. We need an "Echo" activity type first.
|
||||
ECHO_DEF=$(cat <<'JSON'
|
||||
{
|
||||
"type": "Create",
|
||||
"object": {
|
||||
"type": "DefineActivity",
|
||||
"name": "TestEcho",
|
||||
"schema": "(fn (act) (cid? (-> act :object :echoes)))",
|
||||
"semantics": "(fn (state act) state)"
|
||||
}
|
||||
}
|
||||
JSON
|
||||
)
|
||||
curl -s -X POST "$BASE/activity" \
|
||||
-H "Authorization: Bearer $TOKEN" -d "$ECHO_DEF" | jq -e '.cid'
|
||||
|
||||
TRIGGER=$(cat <<JSON
|
||||
{
|
||||
"type": "Create",
|
||||
"object": {
|
||||
"type": "DefineTrigger",
|
||||
"name": "echo-on-smoketest",
|
||||
"when-subscription": "$SUB_CID",
|
||||
"cascade-limit": 1,
|
||||
"then": "(fn (act sub env) {:publish (list {:type \"TestEcho\" :object {:echoes (:cid act)}})})"
|
||||
}
|
||||
}
|
||||
JSON
|
||||
)
|
||||
curl -s -X POST "$BASE/activity" \
|
||||
-H "Authorization: Bearer $TOKEN" -d "$TRIGGER" | jq -e '.cid'
|
||||
|
||||
# 8. Capture outbox length so we can detect new entries.
|
||||
BEFORE=$(curl -s "$BASE/actors/next/outbox?page=true" \
|
||||
| jq -r '.orderedItems | length')
|
||||
|
||||
# 9. Publish a Note tagged "smoketest" — should match subscription, fire trigger,
|
||||
# cause TestEcho to be published.
|
||||
NOTE=$(cat <<'JSON'
|
||||
{
|
||||
"type": "Create",
|
||||
"object": {
|
||||
"type": "Note",
|
||||
"content": "hello reactive world",
|
||||
"tags": ["smoketest"]
|
||||
}
|
||||
}
|
||||
JSON
|
||||
)
|
||||
NOTE_CID=$(curl -s -X POST "$BASE/activity" \
|
||||
-H "Authorization: Bearer $TOKEN" -d "$NOTE" | jq -r '.cid')
|
||||
|
||||
# 10. Wait for projection + trigger.
|
||||
sleep 2
|
||||
|
||||
# 11. Verify topic-events projection captured the Note.
|
||||
curl -s "$BASE/projections/topic-events" \
|
||||
| jq -e ". | to_entries | length == 1"
|
||||
|
||||
# 12. Verify outbox grew by exactly TWO activities (the Note + the trigger's TestEcho).
|
||||
AFTER=$(curl -s "$BASE/actors/next/outbox?page=true" \
|
||||
| jq -r '.orderedItems | length')
|
||||
[[ $((AFTER - BEFORE)) == 2 ]] || { echo "expected +2 activities, got $((AFTER - BEFORE))"; exit 1; }
|
||||
|
||||
# 13. Verify the latest activity is a TestEcho referencing the original Note's CID.
|
||||
curl -s "$BASE/actors/next/outbox?page=true" \
|
||||
| jq -e ".orderedItems[0] | .type == \"TestEcho\" and .object.echoes == \"$NOTE_CID\""
|
||||
|
||||
# 14. Negative case: publish a Note WITHOUT the "smoketest" tag — must NOT
|
||||
# trigger, must NOT echo.
|
||||
BEFORE2=$(curl -s "$BASE/actors/next/outbox?page=true" | jq -r '.orderedItems | length')
|
||||
NOTE_OTHER=$(cat <<'JSON'
|
||||
{"type": "Create", "object": {"type": "Note", "content": "no match", "tags": ["other"]}}
|
||||
JSON
|
||||
)
|
||||
curl -s -X POST "$BASE/activity" \
|
||||
-H "Authorization: Bearer $TOKEN" -d "$NOTE_OTHER" | jq -e '.cid'
|
||||
sleep 2
|
||||
AFTER2=$(curl -s "$BASE/actors/next/outbox?page=true" | jq -r '.orderedItems | length')
|
||||
[[ $((AFTER2 - BEFORE2)) == 1 ]] || { echo "expected +1 activity (no echo), got $((AFTER2 - BEFORE2))"; exit 1; }
|
||||
|
||||
# 15. Cascade limit check: prove the trigger doesn't recursively echo TestEcho.
|
||||
# The TestEcho activity itself should NOT match the Topic{smoketest}
|
||||
# subscription (it's not a Note), so no cascade, but verify cascade-depth
|
||||
# was set to 1 on the echo so a future trigger on TestEcho would refuse.
|
||||
LATEST_ECHO=$(curl -s "$BASE/actors/next/outbox?page=true" \
|
||||
| jq -r '.orderedItems | map(select(.type == "TestEcho")) | .[0]')
|
||||
echo "$LATEST_ECHO" | jq -e '."cascade-depth" == 1'
|
||||
|
||||
# 16. Restart kernel; verify subscription, trigger, projection all survive.
|
||||
./next/scripts/stop.sh
|
||||
./next/scripts/start.sh
|
||||
sleep 2
|
||||
curl -s "$BASE/projections/subscriptions" \
|
||||
| jq -e '.["https://next.rose-ash.com/actors/next"] | map(select(.type == "Topic")) | length == 1'
|
||||
curl -s "$BASE/projections/topic-events" | jq -e ". | to_entries | length >= 1"
|
||||
curl -s "$BASE/define-registry?kind=triggers" \
|
||||
| jq -e 'map(select(.name == "echo-on-smoketest")) | length == 1'
|
||||
|
||||
echo "✓ Reactive application smoke test passed — Subscribe + Trigger + Projection demonstrated end-to-end"
|
||||
```
|
||||
|
||||
**What this proves (and what it doesn't):**
|
||||
|
||||
Proves:
|
||||
- `DefineSubscription` + `Subscribe` mechanism works end-to-end.
|
||||
- Subscription's `match-fn` evaluates correctly in pure mode against inbound
|
||||
activities.
|
||||
- `DefineTrigger` fires on subscription matches.
|
||||
- Trigger's `then-sx` can publish derived activities (the `:publish` result).
|
||||
- Cascade-depth metadata propagates correctly.
|
||||
- Subscription state, trigger registration, and projection state all survive
|
||||
kernel restart (snapshot + log replay).
|
||||
- The full reactive application loop works without any kernel code changes
|
||||
between defining the components and exercising them.
|
||||
|
||||
Does NOT prove (deferred to milestone 2+):
|
||||
- Cross-instance subscriptions (federation).
|
||||
- Trigger `:effect` results calling effectful primitives.
|
||||
- `DefineApplication` bundle install/update/fork.
|
||||
- Per-application namespace isolation.
|
||||
- Cascade prevention against malicious cascading from peer instances.
|
||||
|
||||
**Acceptance for 9b:** smoke test exits 0. Like 9a, **zero fed-sx kernel code
|
||||
changes** between defining the application components and observing them
|
||||
operate.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance criteria for milestone 1
|
||||
|
||||
All of:
|
||||
|
||||
1. **Each step's test suite passes** (`bash next/tests/<step>.sh`).
|
||||
2. **Both smoke tests pass** (`bash next/tests/smoke_pin.sh` and
|
||||
`bash next/tests/smoke_app.sh`).
|
||||
3. **Erlang-on-SX baseline preserved** — adding fed-sx kernel modules in
|
||||
`next/kernel/*.erl` doesn't break Phase 1-8 conformance.
|
||||
4. **Restart durability** — kill the kernel mid-write, restart, projections
|
||||
resume from snapshot, no log corruption.
|
||||
5. **Manual Mastodon poke** — point a Mastodon account at
|
||||
`https://next.rose-ash.com/actors/next` and verify the actor doc fetches and
|
||||
webfinger discovery works (read-only AP interop, no follow).
|
||||
|
||||
## What lands when
|
||||
|
||||
This is the work-order an agent (or human) follows. Steps 1-3 can be done in
|
||||
parallel after the Erlang Phase 8 BIFs land. Steps 4-7 are sequential. Step 8
|
||||
can start in parallel with step 7. Step 9 is the integration test.
|
||||
|
||||
```
|
||||
Phase 7+8 (loops/erlang) ───┐
|
||||
│
|
||||
▼
|
||||
┌─── Step 1 ──┬─── Step 2 ──┬─── Step 3
|
||||
│ │ │
|
||||
└─────────────┼─── Step 4 ──┴────┐
|
||||
│ │
|
||||
└─── Step 5 ───────┤
|
||||
│
|
||||
Step 6 ─────┤
|
||||
│
|
||||
Step 7 ─────┤
|
||||
│
|
||||
Step 8 ─────┤
|
||||
│
|
||||
Step 9 ─────┘
|
||||
```
|
||||
|
||||
Estimated effort if done by a focused agent loop, one feature per iteration:
|
||||
~30-50 commits across all 9 steps. Could plausibly be a `loops/fed-sx` workstream
|
||||
once Phase 7+8 are done.
|
||||
|
||||
## What's deferred to milestone 2
|
||||
|
||||
- **Federation** (the second-biggest piece). `POST /inbox`, Follow lifecycle,
|
||||
delivery queue, backfill, capability negotiation between peers. Whole of
|
||||
design §13.
|
||||
- **Multi-actor** with per-user OAuth and capability tokens. Design §9.5.
|
||||
- **IPFS storage backend** as a `DefineStorage` entry. Design §15.3.
|
||||
- **Browser client + operator dashboard** (probably in Elm-on-SX or similar).
|
||||
- **Rich verbs**: `Endorse`, `Supersede`, `Test`, `Build`, `Compose`, `Note`,
|
||||
`Announce`. All defined as `DefineActivity` artifacts, federated.
|
||||
- **Cross-host conformance** — Python/JS/Haskell hosts running fed-sx. Design
|
||||
§11.8.
|
||||
- **OpenTimestamps proofs** as a `DefineProof` entry.
|
||||
- **Performance work** — JIT-compiled folds, snapshot acceleration, federation
|
||||
batching.
|
||||
|
||||
Milestone 2 unlocks "real federation between two fed-sx instances." Milestone 3
|
||||
is the rose-ash port (blog, market, events, federation, account, orders) as
|
||||
fed-sx applications.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: open questions for milestone 1
|
||||
|
||||
A few things still under-specified; resolve as work begins.
|
||||
|
||||
1. **HTTP server library.** Does the Phase 8 `http:listen/2` BIF wrap an
|
||||
existing OCaml HTTP server (the sx.rose-ash.com one) or something simpler?
|
||||
Implementation choice deferred to Phase 8.
|
||||
2. **JSON-LD library.** AP wire format requires JSON-LD canonicalization for
|
||||
signature coverage. Either pull a library or write a minimal subset for the
|
||||
shapes we actually use. Probably the latter — our envelope is well-defined.
|
||||
3. **Bearer token rotation.** v1 uses a single env-var token. Token rotation
|
||||
without restart needs registry-style mgmt; can wait.
|
||||
4. **Snapshot rate limits.** Default in design is "every 1000 activities or
|
||||
60 seconds." Tunable per-projection later; v1 uses the default.
|
||||
5. **Genesis bundle format.** Dag-cbor map per §12.2; concrete schema needs
|
||||
one round of refinement once we author the actual definitions in step 4.
|
||||
430
plans/sx-vm-opcode-extension.md
Normal file
430
plans/sx-vm-opcode-extension.md
Normal file
@@ -0,0 +1,430 @@
|
||||
# SX VM Opcode Extension Mechanism
|
||||
|
||||
Mechanism in `hosts/ocaml/evaluator/` that lets language ports register
|
||||
specialized bytecode opcodes without modifying the SX VM core. Direct
|
||||
prerequisite for **erlang-on-sx Phase 9** (the BEAM analog) and a structural
|
||||
enabler for any future language port that wants performance-critical opcodes.
|
||||
|
||||
Reference: `plans/erlang-on-sx.md` Phase 9, `plans/fed-sx-design.md` §17.5,
|
||||
`hosts/ocaml/lib/sx_vm.ml` (current VM).
|
||||
|
||||
Status: **design** — implementation pending. Sister workstream to the
|
||||
`loops/erlang` loop, but lives in `hosts/`, not `lib/erlang/`.
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Allow language ports to register custom bytecode opcodes in the SX VM, with:
|
||||
|
||||
- **Zero overhead for core opcodes.** Existing 37 opcodes (per `sx_vm.ml`)
|
||||
must dispatch identically. No regression for any existing language port or
|
||||
the core SX runtime.
|
||||
- **One additional dispatch step for extension opcodes.** Acceptable cost; the
|
||||
win comes from avoiding the general CEK machinery.
|
||||
- **Per-extension state slot.** Erlang's process scheduler, Haskell's thunk
|
||||
cache, etc. need somewhere to hang state alongside the VM.
|
||||
- **Compiler awareness.** The bytecode compiler (`lib/compiler.sx`) must be
|
||||
able to emit extension opcodes by name, looked up against the registered
|
||||
set.
|
||||
- **JIT compatibility.** Existing JIT (lazy lambda compilation) continues to
|
||||
work for code paths using only core opcodes. Extension opcodes are
|
||||
interpreted in v1; JITing them is a follow-up.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **Hot opcode reload.** Adding/replacing opcodes mid-runtime is not in
|
||||
scope. Extensions are compile-time additions to the OCaml binary. (If
|
||||
needed, that's a separate project.)
|
||||
- **Per-instance opcode sets.** All running instances of the SX VM share
|
||||
the same opcode set determined at build time. Selective opcode loading
|
||||
per instance is out of scope.
|
||||
- **Opcode hot-swap or supersession.** Once registered, opcodes are stable
|
||||
for the lifetime of the binary.
|
||||
- **Language-port isolation at the dispatch layer.** Two language ports can
|
||||
see each other's opcodes (they share the dispatch table). Isolation is a
|
||||
build-time concern — don't compile in extensions you don't trust.
|
||||
|
||||
---
|
||||
|
||||
## Why now
|
||||
|
||||
The Erlang-on-SX Phase 9 work needs this. Without it, Phase 9b-9g (the actual
|
||||
opcode implementations) have nowhere to plug in. The Erlang loop will hit
|
||||
this dependency as a Blocker; this design is what unblocks it.
|
||||
|
||||
It also enables the **shared opcode pattern** discussed in `plans/fed-sx-
|
||||
design.md` §17.5: opcodes Erlang Phase 9 produces that other ports could
|
||||
plausibly use (pattern match, perform/handle, record access) get chiselled
|
||||
out to `lib/guest/vm/` when a second port has an actual second use. Without
|
||||
the extension mechanism, each port would have to fork the SX VM core or
|
||||
modify shared dispatch — neither acceptable.
|
||||
|
||||
---
|
||||
|
||||
## Architectural overview
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ SX VM core (hosts/ocaml/lib/sx_vm.ml) │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────┐ │
|
||||
│ │ Bytecode dispatch loop │ │
|
||||
│ │ │ │
|
||||
│ │ match op with │ │
|
||||
│ │ | 1 (OP_CONST) -> ... │ │
|
||||
│ │ | 2 (OP_NIL) -> ... │ │
|
||||
│ │ | ... │ │
|
||||
│ │ | 199 -> ... (last core opcode) │ │
|
||||
│ │ | op when op >= 200 -> │ │
|
||||
│ │ Extensions.dispatch op vm │ │ ◄── new
|
||||
│ │ frame │ │
|
||||
│ └────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────┐ │
|
||||
│ │ Extension registry │ │
|
||||
│ │ opcode_id -> handler │ │ ◄── new
|
||||
│ │ opcode_name -> opcode_id │ │
|
||||
│ │ extension_state per extension │ │
|
||||
│ └────────────────────────────────────┘ │
|
||||
└──────────────────────────────────────────┘
|
||||
▲
|
||||
│ register at startup
|
||||
┌──────────────────┴──────────────────────┐
|
||||
│ Extension modules │
|
||||
│ hosts/ocaml/extensions/erlang.ml │
|
||||
│ hosts/ocaml/extensions/haskell.ml │
|
||||
│ hosts/ocaml/extensions/datalog.ml │
|
||||
│ hosts/ocaml/extensions/guest_vm.ml │ ◄── shared opcodes
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Opcode ID space partition
|
||||
|
||||
Current SX VM uses opcode IDs in roughly the range 1-162 (per inspection of
|
||||
`sx_vm.ml`). We partition the 0-255 space:
|
||||
|
||||
| Range | Use |
|
||||
|-------|-----|
|
||||
| 0 | reserved / NOP |
|
||||
| 1-127 | **core opcodes** — owned by the SX VM, locked schema |
|
||||
| 128-199 | **`lib/guest/vm/` shared opcodes** — chiselled-out shared opcodes |
|
||||
| 200-247 | **language-port opcodes** — registered by extensions |
|
||||
| 248-255 | reserved for future expansion / multi-byte opcodes |
|
||||
|
||||
This gives ~50 slots for shared opcodes (Phase 1-2 of `lib/guest/vm/` will
|
||||
not exhaust this; we can renegotiate if it does), ~50 for any single language
|
||||
port's specialized opcodes, and clean separation that makes it obvious which
|
||||
opcodes are stable (core), shared (guest), or port-specific (extension).
|
||||
|
||||
If we need more than 256 opcodes total, multi-byte opcodes (a leading 248-255
|
||||
byte plus a second byte) extend the space without breaking the schema.
|
||||
|
||||
### Extension module signature
|
||||
|
||||
```ocaml
|
||||
(* hosts/ocaml/lib/sx_vm_extension.ml *)
|
||||
|
||||
(** A handler for an extension opcode. Reads operands from bytecode,
|
||||
manipulates the VM stack, updates the frame's instruction pointer.
|
||||
May raise exceptions (which propagate via the existing VM error path). *)
|
||||
type handler = vm -> frame -> unit
|
||||
|
||||
(** State an extension carries alongside the VM. Opaque to the VM core;
|
||||
extensions cast as needed. *)
|
||||
type extension_state = ..
|
||||
|
||||
module type EXTENSION = sig
|
||||
(** Stable name for this extension (e.g. "erlang", "guest_vm"). *)
|
||||
val name : string
|
||||
|
||||
(** Initialize per-instance state. Called once when the VM starts and the
|
||||
extension is loaded. *)
|
||||
val init : unit -> extension_state
|
||||
|
||||
(** Opcodes this extension provides. Each is (opcode_id, opcode_name, handler).
|
||||
opcode_id must be in the range allowed for this extension's tier
|
||||
(128-199 for guest, 200-247 for ports). Conflicts cause startup failure. *)
|
||||
val opcodes : extension_state -> (int * string * handler) list
|
||||
end
|
||||
```
|
||||
|
||||
### Registration and dispatch
|
||||
|
||||
```ocaml
|
||||
(* hosts/ocaml/lib/sx_vm_extensions.ml *)
|
||||
|
||||
let extensions : (module EXTENSION) list ref = ref []
|
||||
let states : (string, extension_state) Hashtbl.t = Hashtbl.create 8
|
||||
let by_id : (int, handler) Hashtbl.t = Hashtbl.create 64
|
||||
let by_name : (string, int) Hashtbl.t = Hashtbl.create 64
|
||||
|
||||
let register (m : (module EXTENSION)) =
|
||||
let module M = (val m) in
|
||||
let st = M.init () in
|
||||
Hashtbl.add states M.name st;
|
||||
List.iter (fun (id, name, h) ->
|
||||
if Hashtbl.mem by_id id then
|
||||
failwith (Printf.sprintf "Opcode %d (%s) already registered" id name);
|
||||
Hashtbl.add by_id id h;
|
||||
Hashtbl.add by_name name id
|
||||
) (M.opcodes st);
|
||||
extensions := m :: !extensions
|
||||
|
||||
let dispatch op vm frame =
|
||||
match Hashtbl.find_opt by_id op with
|
||||
| Some handler -> handler vm frame
|
||||
| None -> raise (Invalid_opcode op)
|
||||
|
||||
let id_of_name name = Hashtbl.find_opt by_name name
|
||||
let state_of_extension name = Hashtbl.find_opt states name
|
||||
```
|
||||
|
||||
The dispatch path adds **one hashtable lookup per extension opcode**.
|
||||
Acceptable cost — and Erlang's specialized opcodes win >100× over going
|
||||
through the general CEK machine, so the overhead is negligible by comparison.
|
||||
|
||||
### Bytecode compiler integration
|
||||
|
||||
The compiler (`lib/compiler.sx`) needs to know extension opcode IDs to emit
|
||||
them. New SX primitive exposed to the compiler:
|
||||
|
||||
```sx
|
||||
(extension-opcode-id "erlang.OP_PATTERN_TUPLE_2") ; → 200, or nil if not loaded
|
||||
```
|
||||
|
||||
When the compiler wants to emit a specialized opcode, it queries by name. If
|
||||
the extension isn't loaded, the compiler falls back to the general path
|
||||
(emit a `CALL_PRIM` or general SX `case`). This means a language port's
|
||||
optimization is opt-in per build, and missing extensions degrade to slower
|
||||
correct execution rather than failure.
|
||||
|
||||
Naming convention: `<extension-name>.OP_<NAME>`. So `erlang.OP_PATTERN_TUPLE_2`,
|
||||
`guest_vm.OP_PERFORM`, etc.
|
||||
|
||||
### Per-extension state access
|
||||
|
||||
Some opcodes need state beyond the VM stack (Erlang's scheduler, mailbox
|
||||
state, etc.). Extensions store state in their `init`-returned value, accessed
|
||||
via `state_of_extension`:
|
||||
|
||||
```ocaml
|
||||
let op_spawn vm frame =
|
||||
let st = Sx_vm_extensions.state_of_extension "erlang"
|
||||
|> Option.get
|
||||
|> Obj.magic in (* extension casts to its known type *)
|
||||
let body = pop vm in
|
||||
let pid = Erlang_scheduler.spawn st body in
|
||||
push vm (pid_value pid);
|
||||
frame.ip <- frame.ip + 1
|
||||
```
|
||||
|
||||
Shared scheduler state lives in the Erlang extension's state value. Other
|
||||
extensions don't see it.
|
||||
|
||||
---
|
||||
|
||||
## Phase plan
|
||||
|
||||
Five sub-phases in dependency order. Each is testable in isolation.
|
||||
|
||||
### Phase A — Opcode ID partition + dispatch fallthrough
|
||||
|
||||
Smallest viable change to `sx_vm.ml`:
|
||||
|
||||
- Add the `| op when op >= 128 -> Sx_vm_extensions.dispatch op vm frame`
|
||||
fallthrough case.
|
||||
- Document the partition in a comment at the top of the opcode list.
|
||||
|
||||
**Tests:**
|
||||
- All existing SX VM tests pass unchanged (zero regression for core).
|
||||
- Calling `dispatch 200 ...` with no extension registered raises
|
||||
`Invalid_opcode 200`.
|
||||
|
||||
**Effort:** small. ~50 lines + tests.
|
||||
|
||||
### Phase B — Extension registry module
|
||||
|
||||
`hosts/ocaml/lib/sx_vm_extensions.ml` per the sketch above. Pure plumbing, no
|
||||
opcodes yet.
|
||||
|
||||
**Tests:**
|
||||
- Register a test extension with one opcode; dispatch finds it.
|
||||
- Duplicate opcode-id registration fails at startup.
|
||||
- `id_of_name` and `state_of_extension` lookups work.
|
||||
|
||||
**Effort:** small. ~150 lines + tests.
|
||||
|
||||
### Phase C — Compiler-side opcode lookup primitive
|
||||
|
||||
Expose `extension-opcode-id` as an SX primitive in `hosts/ocaml/lib/`. The
|
||||
compiler in `lib/compiler.sx` can call it to emit extension opcodes by name.
|
||||
|
||||
Does not require any extension to actually exist — the primitive returns
|
||||
`nil` for unknown names, and the compiler falls back.
|
||||
|
||||
**Tests:**
|
||||
- Primitive returns nil for unknown name.
|
||||
- After registering a test extension, primitive returns the registered ID.
|
||||
|
||||
**Effort:** small. Single primitive registration + compiler-side use docs.
|
||||
|
||||
### Phase D — Test extension demonstrating end-to-end flow
|
||||
|
||||
A dummy extension at `hosts/ocaml/extensions/test_ext.ml` registering one or
|
||||
two trivial opcodes (e.g. `OP_TEST_PUSH_42`, `OP_TEST_DOUBLE_TOS`). Wired
|
||||
into the build, available when running tests.
|
||||
|
||||
Compiler test: write SX that triggers the test compiler-extension to emit
|
||||
`OP_TEST_PUSH_42`, then verify the VM executes it correctly via
|
||||
`bytecode-inspect` and `vm-trace`.
|
||||
|
||||
**Tests:**
|
||||
- Bytecode emission via name lookup produces the right ID.
|
||||
- Execution produces the expected stack effect.
|
||||
- `bytecode-inspect` shows the opcode by name.
|
||||
- `vm-trace` correctly reports the extension opcode.
|
||||
|
||||
**Effort:** small. ~100 lines including build wiring.
|
||||
|
||||
### Phase E — JIT awareness (interpreted-only for v1)
|
||||
|
||||
The JIT (lazy lambda compilation) currently compiles based on opcode ranges.
|
||||
Extension opcodes (≥128) should fall through to interpretation, not be
|
||||
JIT-compiled in v1.
|
||||
|
||||
- Mark extension opcodes as "interpret only" in the JIT pre-analysis.
|
||||
- A lambda containing only core opcodes JIT-compiles as before.
|
||||
- A lambda containing any extension opcode runs interpreted.
|
||||
|
||||
JITing extension opcodes is a follow-up project; v1 keeps the JIT scope
|
||||
unchanged and just makes it correctly route mixed bytecode.
|
||||
|
||||
**Tests:**
|
||||
- Lambda with only core opcodes: JIT-compiled, fast path.
|
||||
- Lambda with extension opcode: interpreted, correct result.
|
||||
- Mixed lambda: interpreted, correct result.
|
||||
|
||||
**Effort:** small-medium. Requires understanding the JIT's pre-analysis
|
||||
(per `project_jit_compilation.md` memory: "Lazy JIT implemented: lambda
|
||||
bodies compiled on first VM call, cached, failures sentinel-marked").
|
||||
Extension-opcode detection becomes another reason to mark a lambda
|
||||
"interpret-only."
|
||||
|
||||
---
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
1. **Phase A-D pass their test suites.**
|
||||
2. **Zero regression on existing SX VM tests.** All language-port test
|
||||
suites currently passing on the architecture branch (Erlang 530+, Haskell
|
||||
285+, Datalog 276+, Smalltalk 625+, the SX core test suite, etc.) still
|
||||
pass.
|
||||
3. **Test extension demonstrates the flow end-to-end.** SX source compiles
|
||||
via the compiler with a registered extension opcode, executes through the
|
||||
VM via the dispatch fallthrough, returns correct result.
|
||||
4. **Documentation:** README in `hosts/ocaml/extensions/` explaining the
|
||||
pattern, with a worked example (the test extension is the canonical one).
|
||||
|
||||
After acceptance, the Erlang-on-SX Phase 9 work in `lib/erlang/vm/` can use
|
||||
this mechanism. The Erlang loop's Blocker for 9a is resolved.
|
||||
|
||||
---
|
||||
|
||||
## Risk and mitigation
|
||||
|
||||
**Risk: regression in core opcode dispatch.** A misplaced `match` arm could
|
||||
break something. *Mitigation:* run every existing language-port test suite
|
||||
before merging. The cost of this verification is real — probably an hour of
|
||||
machine time — but cheaper than discovering it after the fact.
|
||||
|
||||
**Risk: opcode ID conflicts as more extensions land.** If Erlang Phase 9
|
||||
claims IDs 200-220 and Haskell wants 215-235, we have a problem.
|
||||
*Mitigation:* maintain a registry document at `hosts/ocaml/extensions/
|
||||
README.md` listing claimed ID ranges per extension. Convention: each
|
||||
extension claims a contiguous block at first registration; collisions caught
|
||||
at startup with a clear error.
|
||||
|
||||
**Risk: extension state types leak through `Obj.magic`.** The extension state
|
||||
is type-erased in the registry. *Mitigation:* extensions cast in their own
|
||||
opcode handlers, never expose state to other extensions or the VM core.
|
||||
First-class modules / GADTs could add more type safety; deferred unless
|
||||
this becomes a concrete pain point.
|
||||
|
||||
**Risk: extensions become a back door for kernel mutation.** An extension
|
||||
opcode handler has full access to the VM. *Mitigation:* extensions are
|
||||
build-time additions, not runtime; they're as trusted as the rest of the
|
||||
binary. Operators audit at build time, not runtime. Same trust model as
|
||||
any other compiled-in code.
|
||||
|
||||
**Risk: shared `lib/guest/vm/` opcodes evolve under different language
|
||||
ports' needs.** *Mitigation:* the chiselling discipline (move to guest only
|
||||
on second use) ensures the shared opcodes are tested against at least two
|
||||
ports' actual usage before being considered stable.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
To be resolved during implementation, not blocking design approval:
|
||||
|
||||
1. **Multi-byte opcode encoding.** If we need >256 opcodes total, the
|
||||
leading-byte 248-255 schema accommodates it. Do we need multi-byte at
|
||||
v1? Probably not — 200+ opcodes per port is more than any port should
|
||||
reasonably want.
|
||||
2. **Extension ordering matters?** If two extensions register opcodes that
|
||||
read the same VM state, ordering of registration could matter for
|
||||
initialization. Probably not in practice; flag if it bites.
|
||||
3. **Hot-reload of extensions.** Out of scope for v1 (per non-goals). If
|
||||
wanted later, the registry would need teardown + re-registration; the
|
||||
`gen_server` `code_change/3` model from Erlang Phase 7 is a precedent.
|
||||
4. **Cross-extension opcode composition.** Can `guest_vm.OP_PERFORM` invoke
|
||||
`erlang.OP_RECEIVE_SCAN`? In principle yes — handlers can do anything.
|
||||
The interface is clean; the question is whether we want any conventions
|
||||
to keep ergonomics tractable. Defer until composition appears in
|
||||
practice.
|
||||
|
||||
---
|
||||
|
||||
## Implementation roadmap and sequencing
|
||||
|
||||
This is a sister workstream to `loops/erlang`. Probably best as a single
|
||||
focused session (not a continuous loop — the work is bounded, ~1-2 weeks
|
||||
of focused effort, not iterative).
|
||||
|
||||
Recommended sequencing:
|
||||
|
||||
1. **A + B + C land together** as a single PR — they're tightly coupled and
|
||||
easier to test as a unit. Branch: `loops/sx-vm-extensions` or similar.
|
||||
2. **D follows** in a second PR; demonstrates the end-to-end flow without
|
||||
committing to any real language port's opcode design.
|
||||
3. **E (JIT integration)** as a third PR, once the basic mechanism is
|
||||
battle-tested.
|
||||
4. **Extension scope check:** verify Erlang's Phase 9 sub-phases 9b-9g can
|
||||
actually use this mechanism. If gaps surface, they're addressable
|
||||
incrementally.
|
||||
5. **`hosts/ocaml/extensions/erlang.ml`** then becomes the *first real
|
||||
consumer* — written by whoever takes over from the Erlang loop's stub
|
||||
dispatcher. That's the integration moment that closes the loop.
|
||||
|
||||
Estimated total effort: 1-2 weeks for one focused engineer with OCaml SX VM
|
||||
familiarity. Much less if the implementer already knows `sx_vm.ml`.
|
||||
|
||||
---
|
||||
|
||||
## Relationship to other plans
|
||||
|
||||
- **`plans/erlang-on-sx.md` Phase 9:** unblocked by this work. Erlang loop
|
||||
develops opcodes against a stub dispatcher in `lib/erlang/vm/`; once this
|
||||
mechanism lands, swap stub for real registration via
|
||||
`hosts/ocaml/extensions/erlang.ml`.
|
||||
- **`plans/fed-sx-design.md` §17.5:** documents this as Layer-1 prerequisite.
|
||||
The shared-opcode discipline (lib/guest/vm/) is designed on top of this
|
||||
mechanism's `lib/guest/vm/` namespace allocation.
|
||||
- **Future language ports (Haskell, Datalog, Smalltalk perf phases):** will
|
||||
use the same mechanism. Each adds an extension module, claims an opcode
|
||||
range, registers handlers. The `lib/guest/vm/` opcodes get
|
||||
cross-referenced when the second port's needs justify chiselling.
|
||||
- **JIT roadmap (per `project_jit_architecture.md` memory):** extension
|
||||
opcodes are interpreted in v1. JITing them is a logical follow-up but
|
||||
a separate project.
|
||||
Reference in New Issue
Block a user