diff --git a/plans/agent-briefings/fed-prims-loop.md b/plans/agent-briefings/fed-prims-loop.md new file mode 100644 index 00000000..70c63ae7 --- /dev/null +++ b/plans/agent-briefings/fed-prims-loop.md @@ -0,0 +1,109 @@ +# fed-prims loop agent (single agent, phase-ordered) + +Role: iterates `plans/fed-sx-host-primitives.md` forever. Adds the pure-OCaml +crypto / CBOR / CID / Ed25519 / RSA primitives and the native HTTP server that +Erlang Phase 8 BIFs (and therefore fed-sx Milestone 1) are blocked on. One +feature per commit. + +``` +description: fed-prims host-primitive loop +subagent_type: general-purpose +run_in_background: true +isolation: worktree +``` + +## Prompt + +You are the sole background agent working `/root/rose-ash/plans/fed-sx-host-primitives.md`. +You run in an isolated git worktree on branch `loops/fed-prims`. You work the +plan's phases in order (A→I), forever, one commit per feature. Push to +`origin/loops/fed-prims` after every commit. + +## Restart baseline — check before iterating + +1. Read `plans/fed-sx-host-primitives.md` — Phasing + Progress log + Blockers + tell you where you are. +2. `cd hosts/ocaml && dune build bin/sx_server.exe 2>&1 | tail` — must be green + before new work. If broken and not by your last edit, Blockers + stop. +3. `bash hosts/ocaml/browser/test_boot.sh` — the WASM kernel must boot. This is + the regression you are most at risk of causing. +4. Find the first unchecked `[ ]` phase. That is your iteration. + +## The iteration + +Implement → `dune build bin/sx_server.exe` (native) → **WASM build check** +(`test_boot.sh`) → run the phase's tests → run the no-regression gate +(`conformance.sh`, see plan) → commit → tick the `[ ]` → append one dated line +to the Progress log (newest first) → push → stop. + +One phase = one iteration = one commit. Do not batch phases. + +## Ground rules (hard) + +- **Scope:** only `hosts/ocaml/lib/**`, `hosts/ocaml/bin/**`, and + `plans/fed-sx-host-primitives.md`. The single exception is Phase I, which also + edits exactly one Blockers entry in `plans/erlang-on-sx.md`. Do **not** touch + `lib/erlang/**`, `spec/`, `lib/` root, other `lib//`. +- **Pure OCaml for `lib/` primitives.** No new opam deps. WASM-safe: no C stubs, + no `Unix`/`Thread` in `lib/sx_primitives.ml`. The HTTP server (Phase H) is + native-only — register it in `bin/sx_server.ml`, never in the lib. +- **Prove WASM every commit.** `test_boot.sh` green is a phase gate, not + optional. A broken WASM kernel = the phase failed; revert and rethink. +- **No-regression gate:** OCaml `run_tests` + Erlang `conformance.sh` must stay + at their current pass counts (Erlang 715/715 once the merge lands; otherwise + whatever `lib/erlang/scoreboard.json` says). New crypto tests are additive. +- **`.ml`/`.sh` files:** ordinary `Read`/`Edit`/`Write` — these are NOT `.sx`. + Do not use sx-tree MCP for OCaml. (sx-tree is only if you ever touch `.sx`, + which this loop should not.) +- **Builds are slow.** Use a generous `timeout` on `dune build` (≥600s) and on + `conformance.sh` (≥400s). If a build genuinely hangs >10min, Blockers + stop. +- **Worktree:** commit, push `origin/loops/fed-prims`. Never `main`, never + `architecture`. +- **Commit granularity:** one feature per commit. `fed-prims: SHA-256 + 4 NIST + vectors`. Update Progress log + tick box every commit. +- **If blocked** two iterations on the same issue: Blockers entry, move to the + next independent phase (A-G are largely independent; H is independent; only + D depends on A+C, E depends on A). + +## Crypto correctness gotchas + +- **Test vectors are non-negotiable.** Every hash/sig phase lands with published + vectors (NIST FIPS 180-4 / 202, RFC 8032, RFC 8949). A primitive without a + passing standard vector is not done — do not tick the box. +- **SHA endianness:** SHA-2 is big-endian length-append; SHA-3 is little-endian + Keccak lane order. Easy to get backwards — the empty-string vector catches it. +- **dag-cbor determinism:** map keys sorted by **byte length first, then + bytewise**. Not lexicographic-only. The "reordered dict keys → identical + bytes" test is the guard; it must be in the phase. +- **CIDv1 layout:** `0x01 || codec-varint || (mh-code-varint || mh-len-varint || + digest)`, then multibase base32-lower with a leading `b`. Off-by-one in varint + is the classic bug — cross-check one CID against `ipfs` CLI if available. +- **Ed25519 verify is total:** wrong-length inputs return `false`, never raise. + Verify checks `[S]B = R + [k]A` with `k = SHA512(R||A||M)` reduced mod L. +- **RSA:** PKCS#1 v1.5 EMSA — the DigestInfo DER prefix for SHA-256 is fixed + (`3031300d060960864801650304020105000420`). Constant-time not required (verify + only, public data). + +## General gotchas + +- The `sx` library is `(wrapped false)` — new module `Sx_sha2` is referenced as + `Sha2.f` is **wrong**; it's `Sx_sha2.f` unless you also alias. Check + `lib/dune` `include_subdirs unqualified`: a new `lib/sx_sha2.ml` is module + `Sx_sha2`. Match the existing `Sx_*` naming. +- `Eval_error` is the primitive-error exception; raise it with `"name: shape"`. +- Reach a primitive from SX to smoke-test: + `printf '(epoch 1)\n(crypto-sha256 "abc")\n' | hosts/ocaml/_build/default/bin/sx_server.exe` +- The native binary the conformance gate uses is + `hosts/ocaml/_build/default/bin/sx_server.exe` — rebuild it before gating. + +## Style + +- No comments in OCaml unless non-obvious (crypto constants ARE non-obvious — + cite the RFC/FIPS section in a one-line comment). +- No new planning docs — update `plans/fed-sx-host-primitives.md` inline. +- One feature per iteration. Build. WASM-check. Test. Gate. Commit. Log. Push. + Next. + +Go. Run the restart baseline. Find the first unchecked `[ ]`. Implement it. +Remember: no commit without a passing standard test vector AND a green WASM +boot. diff --git a/plans/fed-sx-host-primitives.md b/plans/fed-sx-host-primitives.md new file mode 100644 index 00000000..f3261aaa --- /dev/null +++ b/plans/fed-sx-host-primitives.md @@ -0,0 +1,212 @@ +# fed-sx host primitives — `hosts/ocaml/` + +The single blocker between Erlang Phase 8 (FFI mechanism — done) and starting +fed-sx Milestone 1: the SX OCaml runtime exposes no crypto / CID / HTTP host +primitives for the Phase 8 BIF wrappers to call. This plan adds exactly that +surface, pure-OCaml where it must stay WASM-safe, native-only where it can't. + +Reference: `plans/fed-sx-milestone-1.md` (build steps 1-8), +`plans/erlang-on-sx.md` Blockers ("SX runtime lacks platform primitives …"). + +## The hard constraint — WASM boundary + +`hosts/ocaml/lib/` is the `sx` library. `hosts/ocaml/browser/dune` links it +with `(modes byte js wasm)`. **Anything added to `lib/sx_primitives.ml` must +compile under `js_of_ocaml` AND `wasm_of_ocaml`.** Therefore: + +- **Pure OCaml only** for hash / CBOR / CID / Ed25519 / RSA. No `digestif`, + no `mirage-crypto`, no C stubs, no `Unix` dependency in these primitives. + (None of those libs are even installed — the switch has only + re/unix/yojson/otfm/js_of_ocaml. Pure OCaml is both required and hermetic.) +- **HTTP server is native-only**: it needs sockets/threads. Register it in + `bin/sx_server.ml` via `Sx_primitives.register` (precedent: `eval-in-env` at + `bin/sx_server.ml:721`), **not** in the shared lib. It must never enter the + WASM build. +- **`file-list-dir`** uses `Sys.readdir` (stdlib, WASM-stubbed) — safe in lib, + but the fed-sx server is native anyway; native registration is acceptable too. + +**Every phase must prove the WASM build still links** (`sx_build target="wasm"` +or `bash hosts/ocaml/browser/test_boot.sh`) before its commit. A broken WASM +browser kernel is a hard regression and fails the phase. + +## Primitive surface (what fed-sx Milestone 1 actually needs) + +Mapped to `plans/fed-sx-milestone-1.md` build steps: + +| Primitive (SX name) | Signature | fed-sx step | Host | +|---|---|---|---| +| `crypto-sha256` | `(bytes) -> hex-string` | 1, 2 | lib (pure) | +| `crypto-sha512` | `(bytes) -> hex-string` | 2 | lib (pure) | +| `crypto-sha3-256` | `(bytes) -> hex-string` | 1 (CID default) | lib (pure) | +| `cbor-encode` | `(sx-value) -> bytes` (dag-cbor, deterministic) | 1 | lib (pure) | +| `cbor-decode` | `(bytes) -> sx-value` | 1 (round-trip tests) | lib (pure) | +| `cid-from-bytes` | `(codec multihash-bytes) -> cid-string` | 1 | lib (pure) | +| `cid-from-sx` | `(sx-value) -> cid-string` (canonicalize→cbor→sha→mh→cidv1) | 1 | lib (pure) | +| `ed25519-verify` | `(pubkey-32 msg sig-64) -> bool` | 2 | lib (pure) | +| `rsa-sha256-verify` | `(der-spki msg sig) -> bool` (PKCS#1 v1.5) | 2 | lib (pure) | +| `file-list-dir` | `(path) -> (list string)` | 3 | lib/native | +| `http-listen` | `(port handler-fn) -> never` (handler: req-dict→resp-dict) | 8 | **native only** | + +Deferred (not Milestone 1): `httpc-request` (HTTP client — federation is v2), +`sqlite-*` (Milestone 1 is file-on-disk; sqlite is v2 indexes). + +## Registration pattern (established) + +`lib/sx_primitives.ml`: +```ocaml +register "crypto-sha256" (fun args -> + match args with + | [String s] -> String (Sha2.sha256_hex s) + | _ -> raise (Eval_error "crypto-sha256: (bytes)")) +``` +Errors: `raise (Eval_error "name: shape")`. Byte strings are OCaml `string` +(SX `String`). Lists are `Pair`/`Nil` per `sx_types.ml`. Native-only prims go in +`bin/sx_server.ml` the same way. + +## Phasing — one feature per loop iteration + +Dependency order. Each phase: implement → `dune build` (ocaml) → **WASM build +check** → tests → commit → tick box → Progress-log line → push. + +### Phase A — SHA-2 (sha256 + sha512), pure OCaml +- New `lib/sx_sha2.ml` (or inline in primitives if small): SHA-256 + SHA-512. +- Primitives `crypto-sha256`, `crypto-sha512` → lowercase hex string. +- Tests (`bin/run_tests.ml` or a dedicated `bin/test_crypto.ml`): NIST vectors — + `""`, `"abc"`, the 896-bit message, a 1MB "a" repetition. + - sha256("") = `e3b0c442…b7852b855`; sha256("abc") = `ba7816bf…f20015ad` + - sha512("abc") = `ddaf35a1…2a9ac94f…` +- **Acceptance:** vectors pass; WASM build links; OCaml conformance unchanged. + +### Phase B — SHA-3 / Keccak-256, pure OCaml +- Keccak-f[1600] + SHA3-256 padding. Primitive `crypto-sha3-256`. +- Tests: sha3-256("") = `a7ffc6f8…0f8434a`; sha3-256("abc") = `3a985da7…11431532`. +- **Acceptance:** NIST SHA-3 vectors pass; WASM links. + +### Phase C — dag-cbor encoder + decoder, pure OCaml +- RFC 8949 deterministic subset (RFC 8742 dag-cbor): unsigned/negative ints, + byte strings, text strings, arrays, maps with **keys sorted by + length-then-bytewise**, bool, null, tag 42 (CID link). No floats unless a + fed-sx shape needs them (defer; document). +- SX↔CBOR mapping: `Integer`→int, `String`→text str, `Bool`, `Nil`→null, + `Pair/Nil`→array, `Dict`→map (sorted keys), keyword/symbol→text str. +- Primitives `cbor-encode`, `cbor-decode`. Round-trip property tests + RFC 8949 + appendix-A vectors + a "reordered dict keys → identical bytes" determinism test. +- **Acceptance:** vectors + round-trip + determinism pass; WASM links. + +### Phase D — CID computation, pure OCaml +- Multihash (sha2-256 = 0x12, sha3-256 = 0x16; varint code + varint len + digest). +- CIDv1 = `0x01 || codec-varint || multihash`. Codecs: dag-cbor 0x71, raw 0x55. +- Multibase base32 lower (`b` prefix, RFC 4648 no-pad). +- Primitives `cid-from-bytes` (codec, raw mh bytes), `cid-from-sx` + (canonicalize → cbor-encode → sha2-256 → multihash → cidv1 → base32). +- Tests: known IPFS CIDs — cross-check against `ipfs` CLI if present, else the + fixed vectors for `{}` dag-cbor and `"abc"` raw (hardcode expected strings). + Determinism: same SX value (whitespace/comment/key-order variants) → same CID. +- **Acceptance:** matches reference CIDs; determinism holds; WASM links. Satisfies + fed-sx Milestone 1 Step 1. + +### Phase E — Ed25519 verify, pure OCaml +- Curve25519/edwards25519 field arith (mod 2^255-19), point decompress, + SHA-512-based verify per RFC 8032 §5.1.7. (Reuse Phase A sha512.) +- Primitive `ed25519-verify (pubkey msg sig) -> bool`. Bad-length args → false, + not exception (verify is total). +- Tests: RFC 8032 §7.1 vectors (TEST 1-4 + the 1024-byte one). Tampered msg/sig + → false. Wrong-length key → false. +- **Acceptance:** all RFC 8032 vectors pass; WASM links. Satisfies fed-sx Step 2 + (Ed25519 sig-suite). + +### Phase F — RSA-SHA256 verify (PKCS#1 v1.5), pure OCaml +- Minimal pure-OCaml bignum (only need modexp + DER parse). Parse SPKI DER → + (n, e). RSASSA-PKCS1-v1_5 verify with SHA-256 (Phase A). +- Primitive `rsa-sha256-verify (der-spki msg sig) -> bool`. +- Tests: a generated 2048-bit keypair's signature (vectors hardcoded in the test + from a one-off openssl run, documented in a comment), tamper → false. +- **Acceptance:** vector verifies; tamper fails; WASM links. Satisfies fed-sx + Step 2 (rsa-sha256-2018 sig-suite). **Lower priority** than E — Ed25519 is the + modern default; RSA can land after the HTTP phase if time-boxed. + +### Phase G — `file-list-dir`, native-safe +- `Sys.readdir` → sorted SX list of names (no `.`/`..`). Errors → `enoent`/ + `enotdir` classified like the existing `file-read` error mapping. +- Tests: list a known dir, missing dir → error, file-not-dir → error. +- **Acceptance:** passes; WASM build still links (Sys.readdir is stubbed there). + Satisfies fed-sx Step 3 segment replay. + +### Phase H — HTTP/1.1 server, **native-only** (`bin/sx_server.ml`) +- Minimal threaded HTTP/1.1: accept loop (`Unix` + `Thread`), parse request + line + headers + body (Content-Length), build an SX request dict + `{:method :path :query :headers :body}`, call the SX handler callable, take an + SX response dict `{:status :headers :body}`, write it. Connection: close + (keep-alive optional, defer). Bind `127.0.0.1:`. +- Primitive `http-listen (port handler) -> never-returns` registered ONLY in + `bin/sx_server.ml`. Document that it is absent from the WASM kernel. +- Tests: `bin/test_http.sh` — start a server on a port with a tiny SX echo + handler in a subprocess, `curl` GET/POST/404/headers, assert responses, kill. +- **Acceptance:** curl test script green; WASM build untouched (prim not in lib). + Satisfies fed-sx Step 8 transport. + +### Phase I — handoff +- Flip the `plans/erlang-on-sx.md` Blockers entry "SX runtime lacks platform + primitives …" to **RESOLVED**, listing the exact SX primitive names so the + Erlang loop can one-line-wire its blocked Phase 8 BIFs (`crypto:hash/2`, + `cid:from_bytes/1`, `cid:to_string/1`, `file:list_dir/1`, plus note + `httpc`/`sqlite` still deferred). **Do not edit `lib/erlang/`** — that wiring + is the Erlang loop's job; this phase only updates the blocker text + this + plan's "Handoff" section with the primitive→BIF mapping. +- **Acceptance:** blocker text updated; fed-sx Milestone 1 Steps 1-3 + 8 + prerequisites all green. + +## Scope (hard) + +- **Edit only:** `hosts/ocaml/lib/**`, `hosts/ocaml/bin/**`, this plan file. +- **Do NOT edit:** `lib/erlang/**` (Erlang loop owns BIF wiring), `spec/`, + `lib/` root, other `lib//`, `plans/erlang-on-sx.md` *except* the one + Blockers entry in Phase I. +- **Pure OCaml for lib primitives.** No new opam deps. If a phase seems to need + one, stop and add a Blockers entry instead. +- **Prove WASM every phase.** No commit without `test_boot.sh` (or wasm build) + green. +- **Never push to `main` or `architecture`.** Branch `loops/fed-prims`, push + `origin/loops/fed-prims`. +- One feature per commit. Short factual messages: `fed-prims: SHA-256 + 4 NIST + vectors`. Tick the box, append a dated Progress-log line (newest first). +- **Never call `sx_build` with no timeout-awareness** — OCaml builds are slow; + use the MCP `sx_build target="ocaml"` / `target="wasm"` tools or + `dune build` with a generous timeout. If the build hangs >10min, Blockers + + stop. + +## Build & test reference + +```bash +cd hosts/ocaml && dune build bin/sx_server.exe 2>&1 | tail # native +bash hosts/ocaml/browser/test_boot.sh # WASM links + boots +cd hosts/ocaml && dune exec bin/run_tests.exe 2>&1 | tail # OCaml unit tests +SX_SERVER=hosts/ocaml/_build/default/bin/sx_server.exe \ + timeout 400 bash lib/erlang/conformance.sh 2>&1 | tail -3 # no-regression gate +``` + +A primitive is reachable from SX via the epoch protocol: +```bash +printf '(epoch 1)\n(crypto-sha256 "abc")\n' | \ + hosts/ocaml/_build/default/bin/sx_server.exe +``` + +## Handoff (Phase I fills this in) + +| SX primitive | Erlang Phase 8 BIF it unblocks | +|---|---| +| `crypto-sha256` / `crypto-sha512` / `crypto-sha3-256` | `crypto:hash/2` | +| `cid-from-bytes` / `cid-from-sx` | `cid:from_bytes/1`, `cid:to_string/1` | +| `ed25519-verify` / `rsa-sha256-verify` | `crypto:verify` / sig-suites | +| `file-list-dir` | `file:list_dir/1` | +| `http-listen` | fed-sx kernel `http:listen/2` (Milestone 1 Step 8) | + +## Progress log + +_Newest first._ + +- (none yet — Phase A is first) + +## Blockers + +- _(none yet)_