giles
24763c5199
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 28s
156 lines
7.0 KiB
Markdown
156 lines
7.0 KiB
Markdown
# next — fed-sx Milestone 1 kernel
|
|
|
|
Single-instance, single-actor fed-sx server built as Erlang-on-SX modules.
|
|
See `plans/fed-sx-design.md` for the architecture and
|
|
`plans/fed-sx-milestone-1.md` for the build plan + per-step progress log.
|
|
|
|
## Status
|
|
|
|
Both Step 9 smoke proof points are functional **in-process**:
|
|
|
|
- **9a-pure (verb extensibility)** — `Create{DefineActivity{Pin}}` registers Pin
|
|
at runtime; subsequent `Pin{path, cid}` activities fold into a pin-state
|
|
projection. Zero kernel code between definition and use.
|
|
See `next/tests/smoke_pin_pure.sh`.
|
|
- **9b-pure (reactive application)** — A trigger projection matches Notes
|
|
tagged `smoketest` and derives a `TestEcho` carrying the source CID.
|
|
See `next/tests/smoke_app_pure.sh`.
|
|
|
|
The remaining `9a-tcp` / `9b-tcp` deliverables layer TCP transport on top — see
|
|
*Substrate gaps* below.
|
|
|
|
## Layout
|
|
|
|
```
|
|
next/
|
|
├── kernel/ Erlang-on-SX kernel modules (.erl)
|
|
├── genesis/ SX source files for the bootstrap bundle
|
|
├── tests/ Bash test scripts driving sx_server.exe via the epoch protocol
|
|
└── data/ Runtime state — gitignored
|
|
```
|
|
|
|
## Module map
|
|
|
|
| Module | Role |
|
|
|-----------------------|------------------------------------------------------------------------|
|
|
| `nx_cid.erl` | Canonical CID wrapper around the host `cid:to_string` BIF |
|
|
| `envelope.erl` | Activity envelope shape, canonical bytes, time-aware sig verify |
|
|
| `log.erl` | Per-actor in-memory append log (open / append / tip / replay / entries) |
|
|
| `registry.erl` | Pure-functional + gen_server-wrapped registry keyed by Kind |
|
|
| `pipeline.erl` | Validation driver + stage_envelope/signature/replay/schema |
|
|
| `projection.erl` | Pure projection driver + gen_server-per-projection wrapper |
|
|
| `outbox.erl` | Envelope construct + sign + publish orchestrator + broadcast |
|
|
| `bootstrap.erl` | Genesis read/build/verify/load + one-call `start/3` kernel bring-up |
|
|
| `define_registry.erl` | Meta-projection fold for `Create{Define*}` → registry |
|
|
| `sandbox.erl` | `eval_pure/2,3` try/catch envelope for projection folds |
|
|
| `nx_kernel.erl` | Long-lived runtime orchestrator (state + gen_server) |
|
|
| `http_server.erl` | route/1,2 + format-aware GET + POST + Accept header content negotiation |
|
|
|
|
## Genesis bundle
|
|
|
|
`next/genesis/` contains 31 SX files across 7 sections, all consumed as data
|
|
(read + serialised by `bootstrap:populate_registry`, not eval'd):
|
|
|
|
- 3 activity-types — Create, Update, Delete
|
|
- 10 object-types — SXArtifact, Note, Tombstone, 6 Define* meta-types, Snapshot
|
|
- 7 projections — activity-log, by-type, by-actor, by-object, actor-state,
|
|
define-registry, audience-graph
|
|
- 3 validators — envelope-shape, signature, type-schema
|
|
- 3 codecs — dag-cbor, raw, dag-json
|
|
- 2 sig-suites — rsa-sha256-2018, ed25519-2020
|
|
- 3 audience predicates — Public, Followers, Direct
|
|
|
|
`manifest.sx` is the bundle root, listed in dependency-friendly order.
|
|
|
|
## Tests
|
|
|
|
43 test suites, ~560+ assertions. Each script drives `sx_server.exe` via the
|
|
epoch protocol — loads the Erlang substrate, loads relevant kernel modules
|
|
via `code:load_binary` / `erlang-load-module`, then exercises behaviour
|
|
through `erlang-eval-ast`.
|
|
|
|
Conventions:
|
|
|
|
- Scripts marked `_pure.sh` exercise pure-functional state.
|
|
- Scripts marked `_server.sh` (or no suffix) exercise gen_server APIs and
|
|
must inline `start_link` with operations — the Erlang-on-SX scheduler
|
|
doesn't preserve spawned processes across separate `erlang-eval-ast`
|
|
invocations.
|
|
- `smoke_*_pure.sh` are end-to-end smoke tests demonstrating the §Step 9
|
|
proof points without TCP / curl / JSON.
|
|
|
|
The Erlang-on-SX conformance gate (`bash lib/erlang/conformance.sh`, **729 /
|
|
729**) is the no-regression contract — every commit on `loops/fed-sx-m1`
|
|
preserves it.
|
|
|
|
## Substrate
|
|
|
|
Each `.erl` source file is hot-loaded at boot via
|
|
`code:load_binary(Mod, Filename, SourceString)` (Phase 7 BIF). Tests drive
|
|
the runtime via the epoch protocol:
|
|
|
|
```bash
|
|
printf '(epoch 1)\n(load "lib/erlang/runtime.sx")\n(epoch 2)\n<test-expr>\n' \
|
|
| hosts/ocaml/_build/default/bin/sx_server.exe
|
|
```
|
|
|
|
The kernel calls into these host primitives: `crypto:hash/2`,
|
|
`cid:from_bytes/1`, `cid:to_string/1`, `file:read_file/1`, `file:write_file/2`,
|
|
`file:delete/1`, `file:list_dir/1`, `code:load_binary/3`, plus `http:listen/2`
|
|
(the briefing's allowed scope exception, added to `lib/erlang/runtime.sx`).
|
|
|
|
### Substrate gaps (parked work)
|
|
|
|
These three gaps block the remaining unchecked deliverables:
|
|
|
|
1. **Term codec** (`3b`/`3c`) — `atom_to_list`/`integer_to_list` return
|
|
SX-strings (an opaque OCaml-string type), not Erlang charlists;
|
|
`binary_to_list`/`list_to_binary` are unregistered; `$X` char literals
|
|
decode to `nil` in `parse-number`. Net effect: no in-Erlang term ↔ binary
|
|
round-trip path. Blocks on-disk log persistence.
|
|
|
|
2. **SX-source eval bridge** — There's no BIF that lets Erlang call into the
|
|
SX evaluator on a parsed source string. Blocks evaluating the `:schema` /
|
|
`:fold` / `:predicate` / `:verify` bodies from the genesis bundle. Erlang-fun
|
|
stand-ins (`pipeline:stage_schema`, `define_registry:fold`, etc.) prove the
|
|
API shapes; the bridge would let bundle bodies dispatch through them
|
|
unchanged.
|
|
|
|
3. **Dict ↔ proplist marshalling for `http:listen/2`** — The native
|
|
`http-listen` primitive calls the handler with an SX dict; the BIF
|
|
wrapper's bridge would need to marshal that to / from an Erlang proplist.
|
|
Blocks `Step 8b-start` (actual TCP listening with working route dispatch).
|
|
The briefing allowed the BIF *wrapper* as a single scope exception; further
|
|
in-place modifications need agent approval.
|
|
|
|
### Bringing up the kernel
|
|
|
|
For tests, `bootstrap:start/3(ActorId, KeySpec, ActorState)` is the
|
|
one-call boot:
|
|
|
|
```erlang
|
|
KM = <<1,2,3,4>>,
|
|
KS = [{key_id, k1}, {algorithm, ed25519}, {value, KM}],
|
|
AS = [{public_keys, [[{id, k1}, {created, 0}, {value, KM}]]}],
|
|
Pid = bootstrap:start(alice, KS, AS),
|
|
%% nx_kernel + registry populated; you now have a kernel.
|
|
```
|
|
|
|
The HTTP layer (`http_server`) and `nx_kernel:publish/1` flow through the
|
|
same in-process gen_servers; `http_publish_fold.sh` is the end-to-end proof
|
|
the chain works.
|
|
|
|
## What's next (when work resumes)
|
|
|
|
In priority order:
|
|
|
|
1. **8b-bridge** — extend `er-bif-http-listen` with dict ↔ proplist marshalling
|
|
so requests reach `route/1` shaped correctly.
|
|
2. **8b-start** — `http_server:start/1` spawns a process hosting `http:listen/2`.
|
|
3. **9a-tcp / 9b-tcp** — replace the in-process smoke scripts with curl-driven
|
|
versions hitting the running server.
|
|
4. **Term codec / on-disk log** — needs either a new BIF or a temp-file
|
|
workaround; current in-memory log keeps everything functional otherwise.
|
|
5. **SX-source eval bridge** — unlocks real `:schema` / `:fold` body
|
|
evaluation from the genesis bundle.
|