rose-ash/next/README.md

next — fed-sx Milestone 1 kernel

Single-instance, single-actor fed-sx server built as Erlang-on-SX modules. See plans/fed-sx-design.md for the architecture and plans/fed-sx-milestone-1.md for the build plan + per-step progress log.

Status

Both Step 9 smoke proof points are functional in-process:

• 9a-pure (verb extensibility) — Create{DefineActivity{Pin}} registers Pin at runtime; subsequent Pin{path, cid} activities fold into a pin-state projection. Zero kernel code between definition and use. See next/tests/smoke_pin_pure.sh.
• 9b-pure (reactive application) — A trigger projection matches Notes tagged smoketest and derives a TestEcho carrying the source CID. See next/tests/smoke_app_pure.sh.

The remaining 9a-tcp / 9b-tcp deliverables layer TCP transport on top — see Substrate gaps below.

Layout

next/
├── kernel/      Erlang-on-SX kernel modules (.erl)
├── genesis/     SX source files for the bootstrap bundle
├── tests/       Bash test scripts driving sx_server.exe via the epoch protocol
└── data/        Runtime state — gitignored

Module map

Module	Role
nx_cid.erl	Canonical CID wrapper around the host cid:to_string BIF
envelope.erl	Activity envelope shape, canonical bytes, time-aware sig verify
log.erl	Per-actor in-memory append log (open / append / tip / replay / entries)
registry.erl	Pure-functional + gen_server-wrapped registry keyed by Kind
pipeline.erl	Validation driver + stage_envelope/signature/replay/schema
projection.erl	Pure projection driver + gen_server-per-projection wrapper
outbox.erl	Envelope construct + sign + publish orchestrator + broadcast
bootstrap.erl	Genesis read/build/verify/load + one-call start/3 kernel bring-up
define_registry.erl	Meta-projection fold for Create{Define*} → registry
sandbox.erl	eval_pure/2,3 try/catch envelope for projection folds
nx_kernel.erl	Long-lived runtime orchestrator (state + gen_server)
http_server.erl	route/1,2 + format-aware GET + POST + Accept header content negotiation

Genesis bundle

next/genesis/ contains 31 SX files across 7 sections, all consumed as data (read + serialised by bootstrap:populate_registry, not eval'd):

• 3 activity-types — Create, Update, Delete
• 10 object-types — SXArtifact, Note, Tombstone, 6 Define* meta-types, Snapshot
• 7 projections — activity-log, by-type, by-actor, by-object, actor-state, define-registry, audience-graph
• 3 validators — envelope-shape, signature, type-schema
• 3 codecs — dag-cbor, raw, dag-json
• 2 sig-suites — rsa-sha256-2018, ed25519-2020
• 3 audience predicates — Public, Followers, Direct

manifest.sx is the bundle root, listed in dependency-friendly order.

Tests

43 test suites, ~560+ assertions. Each script drives sx_server.exe via the epoch protocol — loads the Erlang substrate, loads relevant kernel modules via code:load_binary / erlang-load-module, then exercises behaviour through erlang-eval-ast.

Conventions:

• Scripts marked _pure.sh exercise pure-functional state.
• Scripts marked _server.sh (or no suffix) exercise gen_server APIs and must inline start_link with operations — the Erlang-on-SX scheduler doesn't preserve spawned processes across separate erlang-eval-ast invocations.
• smoke_*_pure.sh are end-to-end smoke tests demonstrating the §Step 9 proof points without TCP / curl / JSON.

The Erlang-on-SX conformance gate (bash lib/erlang/conformance.sh, 729 / 729) is the no-regression contract — every commit on loops/fed-sx-m1 preserves it.

Substrate

Each .erl source file is hot-loaded at boot via code:load_binary(Mod, Filename, SourceString) (Phase 7 BIF). Tests drive the runtime via the epoch protocol:

printf '(epoch 1)\n(load "lib/erlang/runtime.sx")\n(epoch 2)\n<test-expr>\n' \
  | hosts/ocaml/_build/default/bin/sx_server.exe

The kernel calls into these host primitives: crypto:hash/2, cid:from_bytes/1, cid:to_string/1, file:read_file/1, file:write_file/2, file:delete/1, file:list_dir/1, code:load_binary/3, plus http:listen/2 (the briefing's allowed scope exception, added to lib/erlang/runtime.sx).

Substrate gaps (parked work)

These three gaps block the remaining unchecked deliverables:

1. Term codec (3b/3c) — all three substrate fixes done 2026-06-05: erlang:binary_to_list/1 and erlang:list_to_binary/1 registered in lib/erlang/runtime.sx (iolist-aware); the tokenizer's $X branch emits the decimal char code; atom_to_list/1 and integer_to_list/1 now return Erlang charlists (standard Erlang semantics) with list_to_atom/ list_to_integer accepting both charlists and SX strings for back-compat. 759/759 conformance. The full term-codec primitive set is in place — Step 3b on-disk segment writer can encode arbitrary Erlang activity terms (atoms, ints, binaries, tuples, lists) into byte sequences using only Erlang-native primitives.

2. SX-source eval bridge — There's no BIF that lets Erlang call into the SX evaluator on a parsed source string. Blocks evaluating the :schema / :fold / :predicate / :verify bodies from the genesis bundle. Erlang-fun stand-ins (pipeline:stage_schema, define_registry:fold, etc.) prove the API shapes; the bridge would let bundle bodies dispatch through them unchanged.

3. Dict ↔ proplist marshalling for http:listen/2 — done 2026-06-05. er-bif-http-listen now marshals the native server's request dict ({:method :path :query :headers :body}) into the proplist shape [{method, Bin}, {path, Bin}, {query, Bin}, {headers, [{Name, Value}]}, {body, Bin}] that http_server:route/2 consumes, and converts the handler's response proplist back to {:status :headers :body} for the native server to serialise. Helpers (er-http-req-of-sx, er-http-resp-to-sx, er-http-headers-of-sx, er-http-headers-to-sx, er-mk-proplist, er-proplist-get, er-binary->string, string->er-binary) live alongside the BIF wrapper in lib/erlang/runtime.sx. Verified by next/tests/http_listen_bridge.sh (20 cases) including a http_server:route/1 round-trip. Unblocks Step 8b-start (TCP listener spawn) and the curl-driven 9a-tcp / 9b-tcp smoke tests.

Bringing up the kernel

For tests, bootstrap:start/3(ActorId, KeySpec, ActorState) is the one-call boot:

KM = <<1,2,3,4>>,
KS = [{key_id, k1}, {algorithm, ed25519}, {value, KM}],
AS = [{public_keys, [[{id, k1}, {created, 0}, {value, KM}]]}],
Pid = bootstrap:start(alice, KS, AS),
%% nx_kernel + registry populated; you now have a kernel.

The HTTP layer (http_server) and nx_kernel:publish/1 flow through the same in-process gen_servers; http_publish_fold.sh is the end-to-end proof the chain works.

What's next (when work resumes)

In priority order:

1. 8b-start — http_server:start/1 spawns a process hosting http:listen/2. (8b-bridge done — see Substrate gap #3.)
2. 9a-tcp / 9b-tcp — replace the in-process smoke scripts with curl-driven versions hitting the running server.
3. Term codec / on-disk log — needs either a new BIF or a temp-file workaround; current in-memory log keeps everything functional otherwise.
4. SX-source eval bridge — unlocks real :schema / :fold body evaluation from the genesis bundle.