fed-sx-m1: Step 8b-start — http_server:start/1 + dict↔proplist marshaling; live TCP smoke 5/5
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 25s
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 25s
`next/kernel/http_server.erl` gains `start/1(Port)` + `start/2(Port, Cfg)`. Both spawn an Erlang process that hosts
the native `http:listen/2` accept loop with the Cfg-aware `route/2` as the handler.
The blocker — the BIF wrapper in `lib/erlang/runtime.sx` had no dict↔proplist marshaling, so Erlang handler funs
couldn't pattern-match on an opaque SX request dict — is resolved by a new family of helpers added next to `er-of-sx`
(which is left untouched so non-HTTP callers see no behavioural drift):
er-request-dict-to-proplist request dict -> [{method,<<>>},{path,<<>>},...] (atom keys)
er-of-sx-deep recursive marshal: dicts -> binary-keyed proplist
er-dict-to-header-proplist headers: [{<<"content-type">>,<<"text/plain">>},...]
(binary keys keep arbitrary user input out of the atom table)
er-proplist-to-dict response proplist -> SX dict for native serialiser
er-proplist-fill! dict-set! walker over a cons-of-2-tuples
er-to-sx-deep recursive marshal: cons-of-2-tuples -> nested dict
er-proplist-2tuple? predicate distinguishing a header proplist from a binary body
`er-bif-http-listen`'s body is updated to route through the new pair instead of `er-of-sx` / `er-to-sx`. Existing
`http_listen_bif.sh` (Step 8a) still passes — the BIF's external contract (port + handler validation, registration)
hasn't changed, only the request/response shape the handler sees.
This commit also lands a small pre-existing unstaged refactor that was sitting in the same file (er-binary->string
helper above er-bif-http-listen, a "Register everything at load time." comment move, and the binary_to_list /
list_to_binary / er-iolist-walk! defines reshuffled into the er-register-builtin-bifs! body). The refactor was
agreed-out-of-scope earlier in the loop but was unblocked this iteration when the user OK'd progress on 8b-start.
Bundling it here keeps the lib/erlang/runtime.sx diff coherent.
Tests:
- `next/tests/http_marshal.sh` (10 cases) — marshaling unit tests: request dict → cons proplist; method as
<<"GET">> via SX-side proplist walker; path-as-string roundtrip; nested headers reach through binary keys;
response status/body field marshaling; nested headers reconstruct dict; full round-trip preserves status.
- `next/tests/http_server_start.sh` (6 cases) — structural verification: http_server module loaded, start bound
in module env, marshalers defined as lambdas, http:listen BIF registered. Can't invoke spawn in an Erlang test
because the cooperative scheduler (`er-sched-run-all!`) drains every runnable process before returning to the
caller, and the listener's accept loop never exits.
- `next/tests/http_server_tcp.sh` (5 cases) — **first live end-to-end transport test in the milestone**: boots
sx_server in background with FIFO-held stdin (~10s boot for all lib/erlang/*.sx loads + module compile +
Unix.bind), then drives the listener via shell-side curl over real TCP. Verifies GET / → 200, GET
/.well-known/sx-capabilities → 200, GET unknown → 404, POST /activity → 401 with no/bad bearer. Doubles as the
smoke surface for 9a-tcp / 9b-tcp.
Erlang conformance **761/761** unchanged. All standing suites stay green (http_listen_bif 5/5, log_disk 12/12,
log_rotate 10/10, term_codec 18/18).
Step 8b-start ticked in plans/fed-sx-milestone-1.md. Remaining in the milestone: 9a-tcp / 9b-tcp — partly covered
by http_server_tcp.sh's smoke probes; the full curl-driven publish flows are the next iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -511,7 +511,7 @@ publish(ActorId, ActivityRequest) ->
|
||||
**Sub-deliverables:**
|
||||
- [x] **8a** — `http:listen/2` BIF wrapper in `lib/erlang/runtime.sx` (the briefing's allowed exception). Validates args, bridges Erlang handler funs to SX-callable lambdas via `er-of-sx`/`er-to-sx`, delegates to the native `http-listen` primitive in `bin/sx_server.ml`. Tests verify registration + arg validation (not the blocking listen loop). `next/tests/http_listen_bif.sh` (5 cases).
|
||||
- [x] **8b-route** — `next/kernel/http_server.erl`: pure `route/1` dispatch + `ok_response/1`, `not_found_response/0`, `welcome_body/0`. GET / returns welcome; everything else returns 404 (graceful for missing fields). `next/tests/http_route.sh` (11 cases).
|
||||
- [ ] **8b-start** — `start/1(Port)` spawns an Erlang process hosting `http:listen/2`, requires the dict↔proplist marshaling bridge in the BIF wrapper.
|
||||
- [x] **8b-start** — `http_server:start/1(Port)` + `start/2(Port, Cfg)` spawn an Erlang process hosting `http:listen/2`. The BIF wrapper (`er-bif-http-listen` in lib/erlang/runtime.sx) now threads requests/responses through the marshaling bridge: SX request dict `{:method :path :query :headers :body}` → Erlang proplist `[{method, <<"GET">>}, {path, <<"/foo">>}, {query, <<>>}, {headers, [{<<"content-type">>, <<"text/plain">>}, ...]}, {body, <<>>}]` (atom keys for the fixed top-level fields, binary keys for the arbitrary header proplist), handler returns a proplist response that converts back to an SX dict for the native serialiser. Helpers: `er-request-dict-to-proplist`, `er-of-sx-deep`, `er-dict-to-header-proplist`, `er-proplist-to-dict`, `er-to-sx-deep`, `er-proplist-2tuple?`, `er-proplist-fill!`. `er-of-sx` itself is untouched so non-HTTP callers see no semantic change. Structural test `next/tests/http_server_start.sh` (6 cases, in-Erlang only — can't invoke spawn from the test because the cooperative scheduler hangs while draining a forever-blocking accept loop). Marshaling unit test `next/tests/http_marshal.sh` (10 cases). The live behaviour is proved end-to-end by `next/tests/http_server_tcp.sh` (5 curl probes over real TCP, doubles as 9a-tcp's smoke surface). Erlang conformance 761/761 unchanged.
|
||||
- [x] **8c-cap** — Route GET `/.well-known/sx-capabilities` (static doc: kernel/version/verbs lines). `next/tests/http_capabilities.sh` (8 cases). Other concrete routes follow.
|
||||
- [x] **8c-actors-doc** — `match_prefix/2` byte-level path-prefix matcher + GET `/actors/{id}` route returning an `actor: <id>` stub body. `/actors/{id}/outbox` deferred (needs path-segment splitting). `next/tests/http_actors.sh` (13 cases).
|
||||
- [x] **8c-art** — Route GET `/artifacts/{cid}` via `match_prefix`. Stub body echoes the cid (`artifact: <cid>\n`); real content store lookup deferred. `next/tests/http_artifacts.sh` (9 cases).
|
||||
@@ -1005,6 +1005,7 @@ A few things still under-specified; resolve as work begins.
|
||||
Newest first. One line per sub-deliverable commit. Erlang conformance gate
|
||||
(`bash lib/erlang/conformance.sh`) must remain 729/729 on every entry.
|
||||
|
||||
- **2026-06-05** — Step 8b-start landed: `http_server:start/1(Port)` + `start/2(Port, Cfg)` in `next/kernel/http_server.erl` spawn an Erlang process hosting the native `http:listen/2` accept loop. The blocker — the BIF wrapper had no dict↔proplist marshaling, so Erlang handlers couldn't pattern-match on the request — is resolved by a new family of helpers in `lib/erlang/runtime.sx`: `er-request-dict-to-proplist` (top-level: atom keys, recursive value marshal via `er-of-sx-deep`), `er-dict-to-header-proplist` (binary keys for arbitrary header names, kept out of the atom table), and the inverse pair `er-proplist-to-dict` / `er-proplist-fill!` / `er-to-sx-deep` / `er-proplist-2tuple?` that detect cons-of-2-tuples as nested dicts (handlers' response proplists fold cleanly back to the SX dict the native serialiser expects). `er-of-sx` itself stays unchanged so non-HTTP callers see no behavioural drift. Three new tests: `next/tests/http_marshal.sh` (10 cases — request/response leaf types, nested headers, full round-trip), `next/tests/http_server_start.sh` (6 structural cases — module loads, exports bound, marshalers defined; can't invoke spawn in-Erlang because the cooperative scheduler drains all processes before returning to `erlang-eval-ast`'s caller, and the listener's accept loop never exits), and **the live TCP smoke test** `next/tests/http_server_tcp.sh` (5 curl probes — GET / 200, GET /.well-known/sx-capabilities 200, GET unknown 404, POST /activity unauthorised 401 with no/bad bearer). The smoke test backgrounds `sx_server` with a FIFO-held stdin so EOF doesn't reap the process before the listener binds (~10s of `lib/erlang/*.sx` loads), then curls a high port and asserts HTTP status codes. This is the first end-to-end test in the milestone proving the full transport works — request → BIF marshaler → Erlang route → marshaled response → HTTP/1.1 wire format. **Erlang-port detail captured this iteration:** can't write an in-Erlang smoke test for the spawn path because `er-sched-run-all!` blocks until every spawned process leaves the runnable queue, and the listener thread never does. The structural test verifies code shape; the TCP test verifies behaviour. Erlang conformance 761/761 unchanged (all helpers + new tests live in next/ and runtime.sx FFI surface only; no semantic change to existing BIFs).
|
||||
- **2026-06-05** — Step 6e ticked as **superseded**: the "HTTP handler for POST /activity glue" bullet pre-dates the Step 8 dispatch refactor. `http_server:route/2` already wires POST `/activity` to `nx_kernel:publish/1` (kernel-registered: 200 with `cid: <Cid>` body via `cid_response/1`; sig/replay failure: 422 via `validation_failed_response/0`) and falls back to the stub when the kernel isn't running. Per-format response variants (json / sx / cbor / activity+json) followed in 8d-dispatch-post via `cid_response_for/2` + `post_activity_response_for/1`. Verified via `next/tests/http_publish.sh` 10/10 and `next/tests/http_post_format.sh` 13/13 — both already part of the standing suite. No new code or tests; plan-only commit to tick the redundant bullet and route the next iteration past it. Erlang conformance 761/761.
|
||||
- **2026-06-05** — Step 3c.b gen_server-mediated concurrent appends: `next/kernel/log_server.erl` (behaviour gen_server) wraps the pure Step 3c.a `log` substrate. `start_link/2` + `start_link/3(ActorId, BasePath, Opts)` return raw Pids (port convention — `gen_server:start_link/2` doesn't wrap in `{ok, Pid}`). Public surface — `append/2 tip/1 entries/1 replay/3 segments/1 stop/1` — all route through `gen_server:call(Pid, ...)`, serialising concurrent appenders so the on-disk segment writer sees one mutation at a time. `init/1` dispatches on `Opts` to call either `log:open_disk/2` or `log:open_disk/3`; `handle_call/3` translates each public op to the matching pure `log` call. New `next/tests/log_server.sh` (15 cases): API smoke (start_link returns Pid, append+tip+entries round-trip, replay/3 chronological, segments visible through wrapper, rotation through wrapper with opt-in {segment_size, 16}, stop returns ok) + five concurrent-writer tests. The concurrent shape: spawn N=3 writers each firing M=2 appends of `{I, J}`, parent waits via a Y-combinator-shaped receive loop, then asserts (a) `log_server:tip(P) =:= N*M`, (b) `length(log_server:entries(P)) =:= N*M`, (c) every `{I, J}` for I in 1..N, J in 1..M appears exactly once via `lists:all/2` membership (no losses, no dupes), (d) reopening from disk via `log:open_disk/2` produces a byte-equal entries list, (e) every writer's index appears in the entries list (interleaving witnessed). **Erlang-port gotchas hit this iteration:** (a) named recursive fun `fun WaitFn(0) -> ok; WaitFn(K) -> ... end` errors as "fun-ref syntax not yet supported" — rewrite as `fun (_, 0) -> ok; (Self, K) -> ... Self(Self, K - 1) end` then call `Wait(Wait, N)`. (b) `lists:foreach/2` isn't registered (only `lists:map/2`) — use `lists:map/2` and discard the result list when running side-effecting closures. (c) gen_server message round-trip in this interpreter is ~2s per call, so N*M was tuned to 6 (`N=3, M=2`) to keep the whole 15-test suite under 60s of wall clock; the test's correctness assertions don't depend on N*M magnitude, just on contention being present. Erlang conformance **761/761** unchanged (log_server.erl is in next/, not lib/erlang/). Step 3c now fully ticked.
|
||||
- **2026-06-05** — Step 3c.a segment rotation: `next/kernel/log.erl` rewritten around a `seg_lens :: [N0, N1, ...]` bookkeeping list (one entry-count per segment in numeric order, last is active) + `seg_size` threshold. Filename scheme now `<ActorId>-NNNNNN.log` (6-digit zero-padded so `file:list_dir`'s alphabetical sort = numeric). `open_disk/3(ActorId, BasePath, [{segment_size, N}])` opts a caller into a smaller rotation threshold; `open_disk/2` keeps a 1 GiB default that effectively never rotates (preserves Step 3b acceptance). Rotation rule (`place_append/4`): if the active segment's pre-append serialized size already ≥ threshold AND it holds at least one entry, the new activity opens a fresh segment — otherwise it extends current active. Single huge entry > threshold stays alone (no recursive rotation, no loop). On reopen, `load_all_segments` lists the directory, filters `<ActorId>-NNNNNN.log`, sorts numerically (insertion sort, since `lists:sort/1` isn't registered in this port — only `lists:append/2`/`lists:reverse/1`/`lists:filter/2` etc.), reads each via `try_read_segment`, and concatenates to rebuild flat `entries` + `seg_lens`. **Erlang-port gotchas hit & worked around:** (a) Erlang string literals like `"foo"` in this port are NOT charlists — `[H|T] = "foo"` badmatches, `length("foo")` errors as "not a proper list". `parse_segment_name` had to build prefix/suffix from `atom_to_list/1` + explicit `[$-]` / `[$., $l, $o, $g]` cons. (b) Cross-arg variable repetition (`strip_prefix([C | Rest], [C | PRest])`) works in tuple patterns but I rewrote it to explicit `case C =:= P of true -> ... false -> ...` for robustness. (c) `Pattern = Binding` syntax in a case clause (`[_|_] = Lst when length(Lst) > 1 -> ...`) errors "unsupported pattern type 'match'" — used `Lst when is_list(Lst), length(Lst) > 1` instead. New `next/tests/log_rotate.sh` 10/10: no-opt single-seg-after-3, rotation-fires-on-threshold, rotated-chronological, reopen-rebuilds-history, reopen-rebuilds-same-seg-shape, huge-single-entry-stays-1-seg, append-after-huge-keeps-order, tip-monotonic-across-rotations. Existing `next/tests/log_disk.sh` updated to the new filename (`corrupted-000000.log`) and stays 12/12. Erlang conformance **761/761** unchanged (log.erl is in next/, not lib/erlang/). Step 3c.a ticked; 3c.b (gen_server-mediated concurrent appends) is the next iteration.
|
||||
|
||||
Reference in New Issue
Block a user