From bd2c61367d74f5aba6bd2ebdfed20a7387ac3fbe Mon Sep 17 00:00:00 2001 From: giles Date: Sun, 7 Jun 2026 10:44:25 +0000 Subject: [PATCH] =?UTF-8?q?fed-sx-m2:=20Step=208e=20=E2=80=94=20httpc:requ?= =?UTF-8?q?est/4=20BIF=20wrapper=20(+=2010=20tests)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the BIF half of Step 8. Native http-request primitive landed in architecture via the fed-prims merge (the m2 plan's Blocker #2), so the briefing-allowed-exception wrapper in lib/erlang/runtime.sx can finally be wired. Marshalling at the BIF boundary: Url : Erlang binary -> SX string (byte-list -> integer->char). Method : Erlang atom upcased ('get -> "GET") for HTTP-wire convention, or Erlang binary passes through verbatim. Headers : Erlang proplist -> SX dict via er-proplist-to-dict. Body : Erlang binary -> SX string. Result {:status :headers :body} marshalled back to Erlang {ok, Status::integer, Headers::proplist (binary-keyed via er-of-sx-deep), Body::binary (char->integer over the SX string)}. Bad arg shapes (non-binary URL or body) raise error:badarg; native DNS / connect / bad-URL failures surface as Erlang error markers that the caller can catch. Test: next/tests/httpc_request.sh 10/10 - registration under httpc/request/4 - BIF marked non-pure - wrong-arity (/1) absent from registry - badarg on non-binary URL - badarg on non-binary body - live GET against `python3 -m http.server` -> Status 200 - body bytes match "hello from python\n" - headers come back as proplist (is_list/1 = true) - 404 path -> {ok, 404, ...} (not an error tuple) - method passed as binary works URLs spelled out as byte-list <<104,116,116,p,...>> binaries since the parser truncates <<"..."> string-literal binaries (same workaround backfill_drain.sh uses for inbox paths). Plan: 8e ticked; Blocker #2 marked RESOLVED with the merge that unblocked it referenced. Step 8f (live HTTP dispatch through delivery_worker) and Step 10c (peer-actor doc fetch) are now unblocked. No-regression gates green: Erlang conformance 761/761, http_multi_actor 44/44, follower_graph 18/18, follow_lifecycle 9/9, backfill 20/20, backfill_drain 6/6, http_listen_bif 5/5. Co-Authored-By: Claude Opus 4.7 (1M context) --- lib/erlang/runtime.sx | 58 ++++++++++++++ next/tests/httpc_request.sh | 153 ++++++++++++++++++++++++++++++++++++ plans/fed-sx-milestone-2.md | 72 +++++++++++++---- 3 files changed, 266 insertions(+), 17 deletions(-) create mode 100755 next/tests/httpc_request.sh diff --git a/lib/erlang/runtime.sx b/lib/erlang/runtime.sx index 32ce6e56..47d746c3 100644 --- a/lib/erlang/runtime.sx +++ b/lib/erlang/runtime.sx @@ -1593,6 +1593,63 @@ ((sx-handler (fn (req-dict) (er-http-resp-to-sx (er-apply-fun handler (list (er-http-req-of-sx req-dict))))))) (http-listen port sx-handler)))))) +;; httpc:request/4(Url, Method, Headers, Body) - BRIEFING-EXCEPTION: +;; the m2 briefing's one allowed scope exception for Step 8e, mirroring +;; M1 Step 8a's http:listen wrapper on the client side. +;; +;; Url is an Erlang binary (must start with http://). +;; Method is an Erlang atom or binary; passed through to the native +;; verbatim, so callers should supply 'get / 'post or <<"GET">> as +;; appropriate (the native compares uppercase). +;; Headers is an Erlang proplist [{Name, Value}, ...]; names and +;; values are binaries or atoms (er-proplist-to-dict handles both). +;; Body is an Erlang binary (use <<>> for empty). +;; +;; Returns a 4-tuple {ok, StatusInt, HeadersProplist, BodyBinary}. +;; The native primitive raises Eval_error on DNS / connect / bad URL; +;; that surfaces as an Erlang error marker the caller can catch via +;; Erlang try/catch. +(define + er-bif-httpc-request + (fn + (vs) + (let + ((url (nth vs 0)) + (method (nth vs 1)) + (headers (nth vs 2)) + (body (nth vs 3))) + (let + ((url-str + (cond + (er-binary? url) (list->string (map integer->char (get url :bytes))) + :else (raise (er-mk-error-marker (er-mk-atom "badarg"))))) + (method-str + (cond + ;; Erlang convention is lowercase atoms (get/post/put/...); + ;; the HTTP wire wants uppercase. Binaries pass through so + ;; callers can override with mixed-case verbs if needed. + (er-atom? method) (upcase (get method :name)) + (er-binary? method) (list->string (map integer->char (get method :bytes))) + :else (raise (er-mk-error-marker (er-mk-atom "badarg"))))) + (headers-dict + (cond + (er-nil? headers) (dict) + (er-cons? headers) (er-proplist-to-dict headers) + :else (raise (er-mk-error-marker (er-mk-atom "badarg"))))) + (body-str + (cond + (er-binary? body) (list->string (map integer->char (get body :bytes))) + (er-nil? body) "" + :else (raise (er-mk-error-marker (er-mk-atom "badarg")))))) + (let + ((resp (http-request method-str url-str headers-dict body-str))) + (er-mk-tuple + (list + (er-mk-atom "ok") + (get resp :status) + (er-of-sx-deep (get resp :headers)) + (er-mk-binary (map char->integer (string->list (get resp :body))))))))))) + ;; Register everything at load time. (define er-register-builtin-bifs! @@ -1796,5 +1853,6 @@ (er-mk-atom "ok"))) (er-register-bif! "http" "listen" 2 er-bif-http-listen) +(er-register-bif! "httpc" "request" 4 er-bif-httpc-request) (er-register-builtin-bifs!) diff --git a/next/tests/httpc_request.sh b/next/tests/httpc_request.sh new file mode 100755 index 00000000..a230a012 --- /dev/null +++ b/next/tests/httpc_request.sh @@ -0,0 +1,153 @@ +#!/usr/bin/env bash +# next/tests/httpc_request.sh — m2 Step 8e acceptance test. +# +# Verifies the httpc:request/4 BIF wrapper is registered, validates +# its arguments, and successfully roundtrips a real HTTP GET against +# a local server. Mirrors http_listen_bif.sh for the +# registration/validation half; the live half uses a background +# `python3 -m http.server` so we don't depend on a blocking SX-side +# http:listen process (Step 8f's concern). +# +# This BIF is the briefing's allowed-exception scope addition to +# lib/erlang/runtime.sx — the dispatch_fn that Step 8f will plumb +# into delivery_worker and Step 10c into peer_actors. + +set -uo pipefail +cd "$(git rev-parse --show-toplevel)" + +SX_SERVER="${SX_SERVER:-hosts/ocaml/_build/default/bin/sx_server.exe}" +if [ ! -x "$SX_SERVER" ]; then + SX_SERVER="/root/rose-ash/hosts/ocaml/_build/default/bin/sx_server.exe" +fi +if [ ! -x "$SX_SERVER" ]; then + echo "ERROR: sx_server.exe not found." >&2 + exit 1 +fi + +VERBOSE="${1:-}" +PASS=0; FAIL=0; ERRORS="" + +# ── live server (Python's stdlib, no extra deps) ───────────── +PORT=$(python3 -c 'import socket;s=socket.socket();s.bind(("127.0.0.1",0));print(s.getsockname()[1]);s.close()') +SRVROOT=$(mktemp -d) +echo "hello from python" > "$SRVROOT/hello.txt" +( cd "$SRVROOT" && python3 -m http.server "$PORT" >/dev/null 2>&1 ) & +SRV_PID=$! +TMPFILE=$(mktemp) +trap "rm -rf $SRVROOT $TMPFILE; kill $SRV_PID 2>/dev/null || true" EXIT +# wait for it to come up (up to ~3s) +for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do + if curl -fsS "http://127.0.0.1:$PORT/hello.txt" >/dev/null 2>&1; then + break + fi + sleep 0.2 +done + +# Spell URLs as Erlang byte-list binaries — <<"...">> string-literal +# binaries truncate to one byte in this parser (see backfill_drain.sh +# for the same workaround on inbox paths). +bytes_of() { python3 -c "import sys; print(','.join(str(b) for b in sys.argv[1].encode()))" "$1"; } +URL_HELLO_BYTES=$(bytes_of "http://127.0.0.1:$PORT/hello.txt") +URL_404_BYTES=$(bytes_of "http://127.0.0.1:$PORT/not_there.txt") +URL_BADBODY_BYTES=$(bytes_of "http://x/") +BODY_HELLO_BYTES=$(bytes_of "hello from python") +GET_METHOD_BYTES=$(bytes_of "GET") + +# Write a quoted heredoc so the SX escapes survive, then sed-replace +# the port number — keeps the SX source clean while still letting us +# bind to a free ephemeral port. +cat > "$TMPFILE" <<'EPOCHS' +(epoch 1) +(load "lib/erlang/tokenizer.sx") +(load "lib/erlang/parser.sx") +(load "lib/erlang/parser-core.sx") +(load "lib/erlang/parser-expr.sx") +(load "lib/erlang/parser-module.sx") +(load "lib/erlang/transpile.sx") +(load "lib/erlang/runtime.sx") +(load "lib/erlang/vm/dispatcher.sx") + +;; BIF registered under httpc/request/4 +(epoch 10) +(eval "(not (= (er-lookup-bif \"httpc\" \"request\" 4) nil))") + +;; BIF marked non-pure (network side effect) +(epoch 11) +(eval "(get (er-lookup-bif \"httpc\" \"request\" 4) :pure?)") + +;; Wrong arity not registered (httpc/request/1 should be nil) +(epoch 12) +(eval "(= (er-lookup-bif \"httpc\" \"request\" 1) nil)") + +;; Non-binary URL -> badarg +(epoch 13) +(eval "(get (erlang-eval-ast \"try httpc:request(not_a_binary, get, [], <<>>) catch error:badarg -> ok end\") :name)") + +;; Non-binary body -> badarg +(epoch 14) +(eval "(get (erlang-eval-ast \"try httpc:request(<<__URL_BAD__>>, get, [], not_a_binary) catch error:badarg -> ok end\") :name)") + +;; ── Live roundtrip: GET against python http.server ────────── +;; Returns 4-tuple {ok, Status, Headers, Body}; Status = 200, +;; Body binary equals "hello from python\n". +(epoch 20) +(eval "(get (erlang-eval-ast \"{ok, Status, _H, _B} = httpc:request(<<__URL_HELLO__>>, get, [], <<>>), case Status of 200 -> true; _ -> false end\") :name)") + +(epoch 21) +(eval "(get (erlang-eval-ast \"{ok, _S, _H, Body} = httpc:request(<<__URL_HELLO__>>, get, [], <<>>), case Body of <<__BODY_HELLO__,10>> -> true; _ -> false end\") :name)") + +;; Headers come back as Erlang proplist (i.e. a cons) +(epoch 22) +(eval "(get (erlang-eval-ast \"{ok, _S, Headers, _B} = httpc:request(<<__URL_HELLO__>>, get, [], <<>>), is_list(Headers)\") :name)") + +;; 404 for unknown path -> Status 404 (not an error tuple) +(epoch 23) +(eval "(get (erlang-eval-ast \"{ok, Status, _H, _B} = httpc:request(<<__URL_404__>>, get, [], <<>>), case Status of 404 -> true; _ -> false end\") :name)") + +;; Method passed as binary works too +(epoch 24) +(eval "(get (erlang-eval-ast \"{ok, Status, _H, _B} = httpc:request(<<__URL_HELLO__>>, <<__GET__>>, [], <<>>), case Status of 200 -> true; _ -> false end\") :name)") +EPOCHS + +sed -i "s|__URL_HELLO__|${URL_HELLO_BYTES}|g; s|__URL_404__|${URL_404_BYTES}|g; s|__URL_BAD__|${URL_BADBODY_BYTES}|g; s|__BODY_HELLO__|${BODY_HELLO_BYTES}|g; s|__GET__|${GET_METHOD_BYTES}|g" "$TMPFILE" + +OUTPUT=$(timeout 120 "$SX_SERVER" < "$TMPFILE" 2>/dev/null) + +check() { + local epoch="$1" desc="$2" expected="$3" + local actual + actual=$(echo "$OUTPUT" | awk -v e="$epoch" ' + $0 ~ "^\\(ok-len " e " " { getline; print; exit } + $0 ~ "^\\(ok " e " " { print; exit } + $0 ~ "^\\(error " e " " { print; exit } + ') + [ -z "$actual" ] && actual="" + if echo "$actual" | grep -qF -- "$expected"; then + PASS=$((PASS+1)) + [ "$VERBOSE" = "-v" ] && echo " ok $desc" + else + FAIL=$((FAIL+1)) + ERRORS+=" FAIL [$desc] (epoch $epoch) expected: $expected | actual: $actual +" + fi +} + +check 10 "BIF registered under httpc/request/4" "true" +check 11 "BIF marked non-pure" "false" +check 12 "no /1 arity registered" "true" +check 13 "non-binary URL -> badarg" "ok" +check 14 "non-binary body -> badarg" "ok" +check 20 "live GET returns Status 200" "true" +check 21 "live GET Body is hello text" "true" +check 22 "Headers come back as proplist" "true" +check 23 "404 surfaces as {ok, 404, ...}" "true" +check 24 "method passed as binary works" "true" + +TOTAL=$((PASS+FAIL)) +if [ $FAIL -eq 0 ]; then + echo "ok $PASS/$TOTAL next/tests/httpc_request.sh passed" +else + echo "FAIL $PASS/$TOTAL passed, $FAIL failed:" + echo "$ERRORS" +fi +[ $FAIL -eq 0 ] diff --git a/plans/fed-sx-milestone-2.md b/plans/fed-sx-milestone-2.md index 2d48ea8a..bde65cef 100644 --- a/plans/fed-sx-milestone-2.md +++ b/plans/fed-sx-milestone-2.md @@ -594,12 +594,18 @@ a dead-letter list visible via `/admin/dead-letter`. in `delivery_dispatch.sh` covering single-peer enqueue, two-peer fan-out, missing-worker skip, no-flag no-op, FIFO append across two publishes, empty delivery_set no-op. -- [ ] **8e** — `httpc:request/4` BIF wrapper. **Blocker:** the - briefing assumed a native `http-request` primitive existed in - `bin/sx_server.ml`; on inspection there's only `http-listen`. - The native http-CLIENT primitive belongs to `loops/fed-prims` - (host primitives loop). Blockers entry below. m2 work - continues with the in-process flow until the native lands. +- [x] **8e** — `httpc:request/4` BIF wrapper. ~~Blocker~~ resolved: + loops/fed-prims merged into architecture, native `http-request` + primitive available. Wrapper at `lib/erlang/runtime.sx` + (briefing-allowed-exception scope) marshals Erlang + `(Url::binary, Method::atom|binary, Headers::proplist, Body::binary)` + → SX `(http-request method url headers body)` → Erlang + `{ok, Status::integer, Headers::proplist, Body::binary}`. + Atom methods are upcased (`get` → `"GET"`) for HTTP-wire convention; + binaries pass through verbatim. Test: `next/tests/httpc_request.sh` + 10/10 pass — registration, badarg validation, live GET 200, + body bytes match, headers proplist shape, 404 surfaces as ok-tuple, + binary method works. - [ ] **8f** — Real HTTP dispatch through the BIF + content-type wiring. dispatch_fn for live use becomes a closure over the peer URL that calls `httpc:request/4` with the signed envelope @@ -1026,17 +1032,16 @@ proceed. re-running on the unmodified m1 closeout HEAD. 2. **Native `http-request` (HTTP client) primitive missing** — - discovered during Step 8e prep. The fed-sx-m2 briefing - ("Substrate available to you" §) claimed: "Native HTTP client - primitive (registered in `bin/sx_server.ml`): `http-request` — - exposed at the SX layer, currently native-only." On inspection - `bin/sx_server.ml` only registers `http-listen`; there is no - `http-request` registration. The HTTP client primitive belongs - to `loops/fed-prims` (host primitives loop) per the - one-primitive-loop-per-substrate convention. m2's Step 8e - wrapper (`httpc:request/4` BIF in `lib/erlang/runtime.sx`) - can land in a 1-line follow-up once the native exists; m2 - work continues with 8b-pure / 8c / 8d in the in-process flow. + ~~discovered during Step 8e prep~~ **RESOLVED 2026-06-07** by + the user-authorized `loops/fed-prims` → `architecture` merge. + The primitive now registers at `bin/sx_server.ml:868+` with + signature `(http-request meth url headers body)` returning a + `{:status :headers :body}` dict and raising `Eval_error` on + DNS / connect / bad URL. Step 8e wired the Erlang-side BIF + wrapper around it (`httpc:request/4`); see Progress log + entry for marshalling details. Step 8f (live HTTP dispatch + through `delivery_worker`) and Step 10c (peer-actor doc + fetch in `peer_actors`) are now unblocked. 3. **`erlang:send_after`-style timer primitive** — discovered during Step 8b prep. The retry loop needs a way for the @@ -1055,6 +1060,39 @@ proceed. Newest first. +- **2026-06-07** — Step 8e (closes the BIF half of Step 8; + live HTTP dispatch in 8f next): `httpc:request/4` BIF wrapper + landed in `lib/erlang/runtime.sx` (briefing-allowed-exception + scope). Marshalling: Erlang URL binary → SX string via + `(list->string (map integer->char (get url :bytes)))`; Erlang + atom method → upcased name (`get` → `"GET"`) for HTTP wire + convention; binary method passes through verbatim; headers + proplist → SX dict via existing `er-proplist-to-dict`; body + binary → SX string. Result `{:status :headers :body}` marshalled + back to Erlang `{ok, Status, Headers::proplist, Body::binary}` + via `er-of-sx-deep` on headers (which produces the binary-keyed + proplist `er-dict-to-header-proplist` shape) and + `(er-mk-binary (map char->integer (string->list body)))` for + body. Non-binary URL / body raise `error:badarg`; the native + primitive raises `Eval_error` on DNS / connect / bad URL which + surfaces as an Erlang error marker the caller can catch. + Blockers #2 (native http-request primitive) entry updated: + RESOLVED by the loops/fed-prims → architecture merge that the + user authorized. Test: `next/tests/httpc_request.sh` 10/10 — + 5 registration / validation cases (registration under + `httpc/request/4`, non-pure flag, no /1 arity, badarg on + non-binary URL, badarg on non-binary body) plus 5 live + roundtrip cases against a background `python3 -m http.server` + (Status 200, body bytes match `hello from python\n`, headers + proplist shape, 404 surfaces as `{ok, 404, ...}` not as an + error tuple, method passed as binary works). Adjacent gates: + Erlang conformance 761/761, http_multi_actor 44/44, follower_ + graph 18/18, follow_lifecycle 9/9, backfill 20/20, + backfill_drain 6/6, http_listen_bif 5/5 — all green; pre- + existing cold-startup timeout sensitivity on http_get_format + (120s internal) and nx_kernel_pure (240s internal) confirmed + with git stash to NOT be caused by this change. + - **2026-06-07** — Step 9c (closes Step 9): Follow → Accept → backfill drain (in-process). `maybe_auto_accept/3` now calls `maybe_backfill/3` after the Accept publish: when