rose-ash/next/README.md

# next — fed-sx Milestone 1 kernel

Single-instance, single-actor fed-sx server built as Erlang-on-SX modules.
See `plans/fed-sx-design.md` for the architecture and
`plans/fed-sx-milestone-1.md` for the build plan + per-step progress log.

## Status

Both Step 9 smoke proof points are functional **in-process**:

- **9a-pure (verb extensibility)** — `Create{DefineActivity{Pin}}` registers Pin
  at runtime; subsequent `Pin{path, cid}` activities fold into a pin-state
  projection. Zero kernel code between definition and use.
  See `next/tests/smoke_pin_pure.sh`.
- **9b-pure (reactive application)** — A trigger projection matches Notes
  tagged `smoketest` and derives a `TestEcho` carrying the source CID.
  See `next/tests/smoke_app_pure.sh`.

The remaining `9a-tcp` / `9b-tcp` deliverables layer TCP transport on top — see
*Substrate gaps* below.

## Layout

```
next/
├── kernel/      Erlang-on-SX kernel modules (.erl)
├── genesis/     SX source files for the bootstrap bundle
├── tests/       Bash test scripts driving sx_server.exe via the epoch protocol
└── data/        Runtime state — gitignored
```

## Module map

| Module                | Role                                                                   |
|-----------------------|------------------------------------------------------------------------|
| `nx_cid.erl`          | Canonical CID wrapper around the host `cid:to_string` BIF              |
| `envelope.erl`        | Activity envelope shape, canonical bytes, time-aware sig verify        |
| `log.erl`             | Per-actor in-memory append log (open / append / tip / replay / entries) |
| `registry.erl`        | Pure-functional + gen_server-wrapped registry keyed by Kind             |
| `pipeline.erl`        | Validation driver + stage_envelope/signature/replay/schema             |
| `projection.erl`      | Pure projection driver + gen_server-per-projection wrapper             |
| `outbox.erl`          | Envelope construct + sign + publish orchestrator + broadcast            |
| `bootstrap.erl`       | Genesis read/build/verify/load + one-call `start/3` kernel bring-up    |
| `define_registry.erl` | Meta-projection fold for `Create{Define*}` → registry                  |
| `sandbox.erl`         | `eval_pure/2,3` try/catch envelope for projection folds                |
| `nx_kernel.erl`       | Long-lived runtime orchestrator; per-actor bucketed state (m2 Step 1a)  |
| `http_server.erl`     | route/1,2 + format-aware GET + POST + Accept header content negotiation |

## Genesis bundle

`next/genesis/` contains 31 SX files across 7 sections, all consumed as data
(read + serialised by `bootstrap:populate_registry`, not eval'd):

- 3 activity-types — Create, Update, Delete
- 10 object-types — SXArtifact, Note, Tombstone, 6 Define* meta-types, Snapshot
- 7 projections — activity-log, by-type, by-actor, by-object, actor-state,
  define-registry, audience-graph
- 3 validators — envelope-shape, signature, type-schema
- 3 codecs — dag-cbor, raw, dag-json
- 2 sig-suites — rsa-sha256-2018, ed25519-2020
- 3 audience predicates — Public, Followers, Direct

`manifest.sx` is the bundle root, listed in dependency-friendly order.

## Tests

43 test suites, ~560+ assertions. Each script drives `sx_server.exe` via the
epoch protocol — loads the Erlang substrate, loads relevant kernel modules
via `code:load_binary` / `erlang-load-module`, then exercises behaviour
through `erlang-eval-ast`.

Conventions:

- Scripts marked `_pure.sh` exercise pure-functional state.
- Scripts marked `_server.sh` (or no suffix) exercise gen_server APIs and
  must inline `start_link` with operations — the Erlang-on-SX scheduler
  doesn't preserve spawned processes across separate `erlang-eval-ast`
  invocations.
- `smoke_*_pure.sh` are end-to-end smoke tests demonstrating the §Step 9
  proof points without TCP / curl / JSON.

The Erlang-on-SX conformance gate (`bash lib/erlang/conformance.sh`, **729 /
729**) is the no-regression contract — every commit on `loops/fed-sx-m1`
preserves it.

## Substrate

Each `.erl` source file is hot-loaded at boot via
`code:load_binary(Mod, Filename, SourceString)` (Phase 7 BIF). Tests drive
the runtime via the epoch protocol:

```bash
printf '(epoch 1)\n(load "lib/erlang/runtime.sx")\n(epoch 2)\n<test-expr>\n' \
  | hosts/ocaml/_build/default/bin/sx_server.exe
```

The kernel calls into these host primitives: `crypto:hash/2`,
`cid:from_bytes/1`, `cid:to_string/1`, `file:read_file/1`, `file:write_file/2`,
`file:delete/1`, `file:list_dir/1`, `code:load_binary/3`, plus `http:listen/2`
(the briefing's allowed scope exception, added to `lib/erlang/runtime.sx`).

### Substrate gaps (parked work)

These three gaps block the remaining unchecked deliverables:

1. **Term codec** (`3b`/`3c`) — **all three substrate fixes done 2026-06-05:**
   `erlang:binary_to_list/1` and `erlang:list_to_binary/1` registered in
   `lib/erlang/runtime.sx` (iolist-aware); the tokenizer's `$X` branch
   emits the decimal char code; `atom_to_list/1` and `integer_to_list/1`
   now return Erlang charlists (standard Erlang semantics) with `list_to_atom`/
   `list_to_integer` accepting both charlists and SX strings for back-compat.
   759/759 conformance. The full term-codec primitive set is in place —
   Step 3b on-disk segment writer can encode arbitrary Erlang activity
   terms (atoms, ints, binaries, tuples, lists) into byte sequences using
   only Erlang-native primitives.

2. **SX-source eval bridge** — There's no BIF that lets Erlang call into the
   SX evaluator on a parsed source string. Blocks evaluating the `:schema` /
   `:fold` / `:predicate` / `:verify` bodies from the genesis bundle. Erlang-fun
   stand-ins (`pipeline:stage_schema`, `define_registry:fold`, etc.) prove the
   API shapes; the bridge would let bundle bodies dispatch through them
   unchanged.

3. **Dict ↔ proplist marshalling for `http:listen/2`** — **done 2026-06-05.**
   `er-bif-http-listen` marshals the native server's request dict
   (`{:method :path :query :headers :body}`) into the proplist shape
   `[{method, Bin}, {path, Bin}, {query, Bin}, {headers, [{Name, Value}]},
   {body, Bin}]` that `http_server:route/2` consumes, and converts the
   handler's response proplist back to `{:status :headers :body}` for the
   native server to serialise. Helpers (`er-request-dict-to-proplist`,
   `er-proplist-to-dict`, `er-of-sx-deep`, `er-to-sx-deep`,
   `er-dict-to-header-proplist`, `er-proplist-fill!`) live alongside the
   BIF wrapper in `lib/erlang/runtime.sx`. The BIF also spawns the handler
   into a real Erlang process via `er-spawn-fun` + `er-sched-run-all!`
   so `self()` / `gen_server:call` work inside route handlers (the kernel
   and projection gen_servers reach the handler this way). Verified by
   `next/tests/http_marshal.sh` and the live TCP smoke
   `next/tests/http_server_tcp.sh` / `http_server_start.sh`. Unblocks
   `Step 8b-start` (TCP listener spawn) and the curl-driven 9a-tcp / 9b-tcp
   smoke tests.

### Bringing up the kernel

For tests, `bootstrap:start/3(ActorId, KeySpec, ActorState)` is the
one-call boot:

```erlang
KM = <<1,2,3,4>>,
KS = [{key_id, k1}, {algorithm, ed25519}, {value, KM}],
AS = [{public_keys, [[{id, k1}, {created, 0}, {value, KM}]]}],
Pid = bootstrap:start(alice, KS, AS),
%% nx_kernel + registry populated; you now have a kernel.
```

The HTTP layer (`http_server`) and `nx_kernel:publish/1` flow through the
same in-process gen_servers; `http_publish_fold.sh` is the end-to-end proof
the chain works.

## What's next (when work resumes)

In priority order:

1. **8b-start** — `http_server:start/1` spawns a process hosting `http:listen/2`.
   (8b-bridge done — see Substrate gap #3.)
2. **9a-tcp / 9b-tcp** — replace the in-process smoke scripts with curl-driven
   versions hitting the running server.
3. **Term codec / on-disk log** — needs either a new BIF or a temp-file
   workaround; current in-memory log keeps everything functional otherwise.
4. **SX-source eval bridge** — unlocks real `:schema` / `:fold` body
   evaluation from the genesis bundle.