Step 5d added ~150 lines to http_server.erl bringing it to ~1180
lines. erlang-load-module on this port scales superlinearly with
function count, so three more http_*.sh tests' internal sx_server
timeout (M1 default 240s) was no longer enough.
Bumped to 600s — matches the headroom the other eight http_*.sh
tests got in the Step 5d commit. Background-gate verification
flagged these three (no behaviour change; just budget).
http_publish 10/10, http_post_format 13/13, http_multi_actor 41/41
all green at 600s.
POST /actors/<id>/inbox is now special-cased in route/2 (next to
POST /activity) so the body + Cfg reach the new handle_inbox_post/3
handler.
Wire format: body = term_codec:encode(SignedActivity); the receiver
decodes into the activity proplist and runs the chain.
handle_inbox_post/3 orchestration:
1. kernel_has_actor(field(kernel, Cfg), TargetId) -> 404 if missing
2. decode_activity(Body) -> 422 on bad shape
3. envelope:get_field(actor, Activity) -> 422 if no peer id
4. resolve_peer_as(PeerId, Cfg) -> 401 if unknown
5. nx_kernel:inbox_state_for(TargetAtom) -> 404 belt-and-braces
6. pipeline:validate_inbound(Activity, PeerAS, InboxLog)
ok -> nx_kernel:append_inbox + 202
{error, bad_signature} -> 401
{error, no_signature} -> 401
{error, _} -> 422
resolve_peer_as/2 supports three Cfg paths in priority order:
{peer_as, [{PeerId, AS}, ...]} pure-fn pre-populated map
{peer_actors, AtomName} peer_actors gen_server cache
{peer_fetch_fn, fun/1} fallback on srv cache miss
Empty Cfg returns {error, no_peer_resolver} -> 401.
v1 actor_post/1 4a stub deleted; M1 actor_inbox_post_response/0
kept for response composition.
Projection broadcast on inbox success intentionally deferred to a
follow-up sub-deliverable.
inbox.sh 11/11 (acceptance suite for the basic chain):
- happy path -> 202
- inbox tip advances; outbox tip unchanged (per-actor bucket
independence carried through from Step 5a)
- empty / garbage body -> 422
- unknown peer -> 401
- bad peer-AS keys -> 401
- replay (same activity twice) -> 422 on second
- unknown target actor -> 404
- two distinct activities -> tip = 2
inbox_peer_resolution.sh 6/6 (Cfg resolution variants):
- peer_actors gen_server hit -> 202
- FetchFn fallback -> 202
- FetchFn error -> 401
- FetchFn caches into peer_actors (peers_srv shows [bob] after)
- No resolver -> 401
Tests split into two files because each epoch's kernel start_link
+ outbox construct + term_codec encode is expensive and a single
suite hits the wall-clock budget.
http_server.erl is now 1181 lines. erlang-load-module on this port
scales superlinearly with function count, so eight http_*.sh tests'
internal sx_server timeout bumped 60s -> 360s (http_route,
http_actors, http_accept, http_capabilities, http_capabilities_format,
http_content_type, http_artifacts, http_projections).
Conformance 761/761.
New next/kernel/peer_actors.erl is the federation-side cache for
{PeerActorId, PeerActorState} entries. PeerAS is exactly the shape
envelope:verify_signature/2 reads (proplist with :public_keys), so
the inbox handler can pipe the cache hit straight into
pipeline:validate_inbound/3 from Step 5b.
Pure-functional API:
new/0
lookup/2(PeerId, State) -> {ok, PeerAS} | not_found
store/3(PeerId, PeerAS, State) -> NewState
evict/2(PeerId, State) -> NewState
peers/1(State) -> [PeerId]
lookup_or_fetch/3(PeerId, FetchFn, State)
-> {ok, PeerAS, NewState} cache hit returns unchanged State,
miss stores FetchFn result.
| {error, Reason, State} FetchFn failure preserves cache.
| {error, {bad_fetch_return, X}, State}
FetchFn contract: (PeerId) -> {ok, PeerAS} | {error, Reason}.
Failed fetches do NOT poison the cache so callers can retry on
transient HTTP failures.
gen_server wrapper (registered name peer_actors):
start_link/0,1 start_link/1 accepts initial proplist for fixtures
stop/0
lookup_srv/1
store_srv/2
lookup_or_fetch_srv/2
peers_srv/0
evict_srv/1
handle_call dispatches mirror the pure-fn paths exactly.
The actual HTTP-GET fetch implementation (peer's actor doc -> peer
AS proplist) is Step 5d's responsibility — for 5c, FetchFn is just
the contract callers fill in.
19/19 in next/tests/peer_actors.sh:
- new/0 -> []
- lookup miss -> not_found
- store + lookup round-trip
- peers/1 in insertion order
- evict + evict-unknown no-op
- lookup_or_fetch miss invokes FetchFn, hits cache after
- lookup_or_fetch hit skips FetchFn (verified by tombstone fn)
- fetch error preserves cache state
- bad fetch return shape captured
- gen_server start_link + miss/hit/fetch/evict round-trips
- start_link/1 pre-populates cache from initial state
Conformance 761/761. 139/139 across 9 Step-5-adjacent suites
(inbox_pipeline, inbox_bucket, pipeline_signature, registry_server,
projection_server, nx_kernel_multi, bootstrap_start, http_publish,
smoke_app_pure, plus the new peer_actors).
New federation inbound pipeline that runs envelope-shape -> peer
signature -> replay against the receiving actor's inbox log.
pipeline.erl additions:
validate_inbound/3(Activity, PeerActorState, InboxLog)
runs inbound_stages(PeerAS, InboxLog) and halts on first
failure (existing run_stages/2 driver). Returns ok |
{error, Reason}.
inbound_stages/2(PeerAS, InboxLog)
[stage_envelope, stage_signature(PeerAS), stage_replay(InboxLog)]
M1's validate_inbound/1 and the static inbound_stages/0 (envelope-
only) are preserved — outbox-side callers don't have to re-key on
a peer-AS they don't have.
Signature verification routes through the peer's actor-state
:public_keys (NOT the local kernel's actor-state). Peer-AS
resolution is the caller's responsibility for 5b; Step 5c wires
the peer-actors cache lookup.
14 cases in next/tests/inbox_pipeline.sh:
- happy path: valid signed activity + correct peer AS + empty
inbox -> ok
- bad envelope shape -> {error, _} (stage_envelope rejects)
- unsigned activity -> stage_envelope rejects on
{missing_field, signature} before sig runs
- wrong peer AS (peer's claimed key bytes differ from real) ->
{error, bad_signature}
- replay: inbox already contains the same activity -> {error, replay}
- inbox with a different activity doesn't trigger replay
- inbound_stages/2 returns exactly 3 stages
- inbound_stages/0 still returns 1 stage
- validate_inbound/1 still works
- shape failure short-circuits before sig
- sig failure short-circuits before replay
- two distinct activities both verify against empty inbox
- inbox-of-one doesn't replay the other
Conformance 761/761. 130/130 across 10 Step-5-adjacent suites
(pipeline_envelope, pipeline_signature, pipeline_replay,
pipeline_driver, inbox_pipeline, inbox_bucket, nx_kernel_multi,
bootstrap_start, http_publish, outbox_publish, smoke_app_pure).
Adds the receiving-side log bucket every actor needs. add_actor/4
now opens a fresh in-memory log via log:open(ActorId, inbox_base_stub())
and stores it on the bucket as {actor_inbox, LogState} alongside
the outbox {log, _}. Two distinct base stubs ensure the in-memory
log module returns separate states even when the same ActorId is
the actor.
Pure-functional exports:
actor_inbox_state/2(ActorId, State) -> {ok, LogState} | {error, _}
actor_inbox_tip/2(ActorId, State) -> integer | nil
append_to_actor_inbox/3(ActorId, Activity, State)
-> {ok, NewTip, NewState} | {error, no_actor, State}
gen_server exports (mirror the outbox shape):
inbox_tip_for/1(ActorId) -> integer | nil
inbox_state_for/1(ActorId) -> {ok, LogState} | {error, _}
append_inbox/2(ActorId, Activity) -> {ok, NewTip} | {error, _}
handle_call dispatch added for all three.
Inbox and outbox tips are completely independent — appending to one
doesn't touch the other. This is the storage primitive 5b will
build the inbound validation pipeline on top of.
log:append/2 signature noted in code + progress log: it takes
(LogState, Activity) and returns {ok, NewState, Seq} — not
{ok, NewState} as I originally guessed.
next/tests/inbox_bucket.sh 14/14:
- fresh inbox tip = 0 (pure)
- actor_inbox_state {ok, _} (pure)
- append_to_actor_inbox/3 -> {ok, 1, _}
- tip advances after append
- unknown actor -> {error, no_actor, _}
- outbox + inbox tips fully independent
- two actors maintain independent inbox state
- gen_server inbox_tip_for/1 starts at 0
- gen_server append_inbox/2 -> {ok, 1}
- gen_server inbox != outbox tip
- gen_server unknown -> {error, no_actor}
- gen_server inbox_state_for {ok, _}
- two appends -> tip = 2
Conformance 761/761. 125/125 across 7 Step-5-adjacent suites
(inbox_bucket, nx_kernel_multi, nx_kernel_server, bootstrap_start,
http_publish, http_multi_actor, actor_lifecycle, smoke_app_pure).
POST /actors/<id>/inbox stays the 4a 202 'accepted' stub through
all of 4a-4d. The real inbound pipeline (peer sig verify + inbox-
bucket append + projection broadcast) is Step 5's whole topic, so
4e is closed as a deliberate scope boundary — no code change.
Step 4 fully closed (4a per-actor sub-paths, 4b token map,
4c route/3 + kernel access, 4d outbox listing + pagination, 4e
inbox-stays-stub).
Per-actor GET /actors/<id>/outbox now reads the bucket's log via
new nx_kernel:log_state_for/1 gen_server export and renders the
paged CID list.
nx_kernel additions:
log_state_for/1 gen_server call returning {ok, LogState} for
the named actor (mirrors log_tip_for/1's shape).
http_server additions:
- with_request_query/2 bakes Req's :query binary into Cfg as
{request_query, Q} so sub-resource handlers can parse params
without taking the Req as another arg
- kernel_actor_log_data/2 -> {Tip, Entries} via
nx_kernel:log_tip_for + log_state_for + log:entries
- parse_page/1 reads ?page=N (default 1, non-digits -> 1)
- page_size/0 returns 5 (test-friendly; production picks 20+)
- page_slice/2 + drop_take/3 + take/2 for the page extraction
- entry_cids/1 maps entries to :id CID binaries via envelope
- actor_outbox_full_response_for/5 renders text / JSON / SX:
text: outbox: <id>\ntip: N\npage: P\nitem: <cid>\n...
json: {"outbox":"<id>","tip":N,"page":P,"items":[...]}
sx: (outbox "<id>" :tip N :page P :items (...))
Empty page degrades to actor_outbox_with_tip_response_for so
epochs 50-57 from Step 4c still pass — the prefix is preserved.
8 new cases in next/tests/http_multi_actor.sh (41/41 total):
- 1 publish -> body contains outbox/tip=1/page=1/item: prefix
- 3 publishes -> body contains tip=3/page=1/item: prefix
- page=2 with 3 items -> empty page degrades to tip-only body
- 6 publishes page=1 -> tip=6/page=1/item: prefix
- 6 publishes page=2 -> tip=6/page=2/item: prefix
- JSON body shape with items array (1 entry)
- SX body shape with :items list (1 entry)
- bad ?page=bad falls back to page 1
Conformance 761/761. 117/117 across 11 Step-4-adjacent suites
(http_multi_actor, http_route, http_publish, http_post_format,
http_marshal, http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, nx_kernel_server, bootstrap_start, actor_lifecycle).
Substrate gotcha logged: named recursive funs fun F(...) -> F(...)
end aren't supported by the parser ('fun-ref syntax not yet
supported'); binary:matches/2 and lists:foreach/2 aren't registered.
Tests prove behaviour via match_prefix substring checks rather than
counting occurrences.
http_server:route/3(Req, Cfg, Kernel) is the new extended entry
point: folds the kernel reference (typically the registered
nx_kernel atom) into Cfg as {kernel, Kernel}. route/2 is
unchanged and stays the M1 surface.
The dispatch chain gained Cfg threading all the way down:
dispatch/3 -> dispatch/4 (M, P, F, Cfg)
actor_get/2 -> actor_get/3 (Rest, F, Cfg)
actor_subresource_get/3 -> /4 (Id, Sub, F, Cfg)
actor_outbox_response_for/3 (new) reads :kernel from Cfg and,
when the kernel atom is registered AND the actor exists, renders
'tip: <N>' alongside the actor id in text / JSON / SX content-
negotiated bodies. Unknown actors or unregistered kernels fall
back to the 4a stub.
Inbox / followers / following handlers accept Cfg but ignore it
for now — they layer real state lookup in 4d/4e/Step 5+.
Substrate gotcha logged in the Progress log: try/of/catch around
gen_server:call(nx_kernel, _) deadlocks in this port's scheduler
(probably the catch frame's mask defers reply delivery). The
live kernel_log_tip/2 helper does a bare call + integer guard
instead. nx_kernel_multi.sh already proves bare gen_server:call
into the same kernel works correctly.
8 new cases in next/tests/http_multi_actor.sh (33/33 total):
- route/3 with registered kernel: outbox body includes tip=0
- tip advances after POST publish through route/3 + token map
- unknown actor (ghost) falls back to 4a stub (no tip:)
- unregistered kernel ref falls back to stub
- JSON Accept renders {"outbox":"alice","tip":0}
- SX Accept renders (outbox "alice" :tip 0)
- Bob's outbox tip stays 0 while Alice publishes (per-actor)
- route/2 path unchanged: no tip field in body
Conformance 761/761. 121/121 across 10 Step-4-adjacent suites
(http_multi_actor, http_route, http_publish, http_post_format,
http_marshal, http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, bootstrap_start, actor_lifecycle).
POST /activity now routes through nx_kernel:publish_to/2 when the
bearer token resolves to an explicit ActorId via Cfg's :tokens
proplist:
Cfg = [{tokens, [{<<"alice-token">>, alice},
{<<"bob-token">>, bob}]}]
resolve_token/2 returns {ok, ActorId} on a :tokens hit. On a miss
it falls back to the M1 :publish_token single-token field — match
returns {ok, legacy}, routing through nx_kernel:publish/1 (which
fans out to bucket 0) so every M1 test continues to pass.
handle_post_activity threads the resolved ActorRef to
publish_if_kernel/3 which dispatches publish_to/2 for explicit
actor ids and publish/1 for the legacy atom. The no-kernel
auth-only path (which preserves the post_activity_response_for stub
for unit-style tests of http_server alone) is unchanged.
Dead expected_token/1 helper removed (was only called by the old
check_bearer arm that resolve_token replaces).
8 new cases in next/tests/http_multi_actor.sh (25/25 total):
- two-actor Cfg, Alice token -> 200 with cid:
- Alice token publishes to alice (log_tip alice=1, bob=0)
- Bob token publishes to bob (log_tip alice=0, bob=1)
- interleaved Alice + Bob + Alice -> {2, 1}
- unknown token + no :publish_token -> 401
- legacy :publish_token still works (M1 back-compat)
- tokens map AND legacy :publish_token coexist (each resolves to
its own actor; legacy lands on alice bucket via publish/1)
- no kernel + valid :tokens entry -> auth-only stub 200
Conformance 761/761. 116/116 across 10 Step-4-adjacent suites
(http_multi_actor, http_route, http_publish, http_post_format,
http_marshal, http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, bootstrap_start, actor_lifecycle).
Per design §16.1 each actor has /outbox /inbox /followers /following
sub-paths. New split_first_slash/1 helper lets the GET /actors/...
dispatch arm fan out on the sub-segment:
GET /actors/<id> actor doc (M1 — unchanged)
GET /actors/<id>/outbox outbox stub (4a)
GET /actors/<id>/inbox inbox stub (4a)
GET /actors/<id>/followers follower stub (4a)
GET /actors/<id>/following following stub (4a)
POST /actors/<id>/inbox 202 Accepted stub (4a; Step 5 real)
Four new content-negotiated response functions mirror the existing
actor_doc_response_for/2 shape (text / json / activity_json / sx
variants):
actor_outbox_response_for/2
actor_inbox_get_response_for/2
actor_followers_response_for/2
actor_following_response_for/2
POST returns 202 via new accepted_response/1 +
actor_inbox_post_response/0.
Unknown sub-paths under /actors/<id>/ return 404. Bare /actors/<id>
preserves the M1 actor-doc arm so http_route + http_post_format
regression suites stay green.
4b-4e (token map, route/3 kernel access, per-actor outbox listing
from log entries, real inbox pipeline) layer on top of this dispatch
in subsequent iterations.
17/17 in next/tests/http_multi_actor.sh covering:
- split_first_slash sanity (no slash / id+sub / trailing slash)
- all four GET sub-paths return 200 with stub bodies
- POST inbox returns 202 + 'accepted'
- unknown sub-paths return 404 (GET and POST)
- empty /actors/ returns 404
- body carries the actor id
- content negotiation: outbox JSON, inbox SX, followers JSON
Conformance 761/761. 120/120 across 10 Step-4-adjacent suites
(http_route, http_publish, http_post_format, http_marshal,
http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, actor_state_pure, bootstrap_start).
actor_state.erl fold_update routes patches through apply_patch/3
which special-cases two rotation patch entries per design §9.6:
{add_publicKey, KeyProplist}
Append to :public_keys; default :created to activity's
:published if unset.
{supersede, OldKeyId}
Mark the matching key with :superseded_at = activity's
:published. Existing :superseded_at preserved (idempotent);
unknown :id no-op.
Other patch entries still last-write-wins per key (Step 2b semantics
preserved; verified by actor_state_pure 19/19 unchanged).
New exports:
key_history/1 — full :public_keys list (preserves superseded)
active_keys_at/2 — subset active at time T (mirrors envelope's
is_active_at; envelope keeps that predicate
private, so a local copy lives here)
find_key_by_id/2 — lookup by :id in the history
Rotation-purpose schema gating per §9.6 (rotation must be signed
by a key with :rotate-key purpose) is deferred to Step 5 (peer-side
stage_signature will plumb purpose through the pipeline).
16/16 in next/tests/key_rotation.sh covering:
- rotation arithmetic (add_publicKey + supersede combined)
- new key :created = rotation activity's :published
- supersede marks :superseded_at correctly
- key_history preserves all keys (superseded included)
- active_keys_at semantics at T=pre / T=rotation / T=post
- live envelope:verify_signature/2 round-trips:
pre-rotation activity signed with K1 -> ok
post-rotation activity signed with K2 -> ok
post-rotation activity signed with K1 -> {error, no_active_key}
- non-rotation Update patches preserve key history
- add_publicKey alone (no supersede) keeps old key active
- supersede alone empties active set
- supersede with unknown id is a no-op
- second supersede on superseded key is idempotent
Conformance 761/761. 132/132 across 9 Step-3-adjacent suites
(key_rotation, actor_state_pure, actor_lifecycle, envelope_sig,
envelope_shape, envelope_canonical, nx_kernel_multi, bootstrap_start,
smoke_app_pure).
New nx_kernel:bootstrap_actor/4(ActorId, Profile, KeySpec, State)
single-call entry that adds an actor bucket and immediately publishes
a Create{Person|Service|Group} envelope as the bucket's first activity:
- Profile carries :type, :name, :preferredUsername, :summary, :icon,
:public_keys. :type defaults to person if unset.
- Kernel AS proplist built from Profile's :public_keys (falls back
to []).
- Create object built from Profile fields (Step 2b actor_state
fold picks the same field set).
gen_server variant bootstrap_actor/3 for live-kernel use plus a new
handle_call branch.
15/15 in next/tests/actor_lifecycle.sh covering pure + gen_server +
actor_state projection capture for all three actor types:
- Pure: bootstrap_actor advances log_tip = 1, Create has
object.type = person
- Pure: two actors share a kernel with independent log tips
- Pure: duplicate bootstrap_actor -> already_present
- Pure: typeless profile defaults to person
- Pure: empty public_keys handled
- gen_server: bootstrap_actor/3 against a live registered kernel
- actor_state projection captures Person, Service, Group profiles
- profile carries :preferredUsername + :public_keys from the
Create object
Closes Step 2 (2a Person/Service/Group genesis files,
2b actor_state projection fold, 2c bootstrap_actor + integration).
Conformance 761/761. 146/146 across 10 Step-2-adjacent suites
(actor_lifecycle, actor_state_pure, nx_kernel_multi, nx_kernel_server,
bootstrap_start, smoke_app_pure, smoke_pin_pure, define_registry_pure,
projection_server, outbox_publish).
next/kernel/actor_state.erl mirrors define_registry's structure: a
2-arity fold_fn that plugs into projection:start_link/3, an
Erlang-fun stand-in for the genesis actor-state.sx projection body.
State shape:
[{ActorId, Profile}, ...]
Profile is a property list with :type, :name, :preferredUsername,
:summary, :icon, :public_keys, :moved_to, :created. Maps #{} aren't
registered in this substrate, so this matches the kernel bucket /
registry shape convention.
Folding rules per design §9.1-§9.4:
- Create{Person|Service|Group}: register profile, capturing object
fields + :published seq as :created. Duplicate Create no-overwrite.
- Update{Person|Service|Group, patch}: deep-merge :patch into
profile last-write-wins per key.
- Move: record :moved_to.
Other activity types and non-actor object Creates pass through.
Local find_keyed/has_keyed/set_keyed helpers (same gap as Step 1a:
no lists:keyfind/keymember in this substrate).
19/19 in next/tests/actor_state_pure.sh covering:
- new/0/has/2/lookup/2/actors/1 base cases
- Create for Person/Service/Group all three actor types
- Profile field capture (name, preferredUsername, public_keys, created)
- Duplicate Create no-overwrite
- Two independent actors
- Update field merge + per-key last-write-wins
- Update for unknown actor pass-through
- Move :moved_to
- Non-actor Creates pass through
- Activities without :actor pass through
- fold_fn/0 returns is_function(F, 2)
Conformance 761/761. Step-2-adjacent no-regression gate 106/106
across 6 suites (define_registry_pure, projection_pure,
projection_server, nx_kernel_multi, bootstrap_start, smoke_app_pure).
Three new DefineObject artefacts in next/genesis/object-types/ for
the canonical actor object-types per design §9.1:
- Person: human-controlled identity (display name + handle + bio)
- Service: automated / programmatic actor (bot, feed, organisation)
- Group: multi-controller actor (member-set managed via Add/Remove)
Each is a small SX form with :name / :doc / :schema, identical
shape to existing object-types (note.sx, sx-artifact.sx etc) so the
existing bootstrap:populate_registry walk picks them up without
code changes. Manifest extended (object-types: 10 -> 13, total
entries: 31 -> 34).
Tests:
- genesis_parse.sh +7 cases (head form, :name, manifest membership);
57/57.
- Hardcoded counts bumped in bootstrap_read.sh, bootstrap_load.sh,
bootstrap_populate.sh, bootstrap_start.sh.
- bootstrap_build.sh 12/12 (bundle CID computed dynamically).
Conformance 761/761 preserved. 211/211 across 12 Step-2-adjacent
suites.
New gen_server exports add_actor/3, publish_to/2, log_tip_for/1,
actors/0, state_for/1, bucket_for/1, with_projections_for/2 —
each is a thin gen_server:call delegating to 1a's pure-functional
bucket API via fresh handle_call branches. Existing single-actor
calls (publish/1, log_tip/0, with_projections/1) route through
bucket 0 unchanged.
Per-actor mailbox sharding (one gen_server per bucket so distinct-
actor publishes don't serialise on a single mailbox) is forward-
looking — deferred to Step 4 where the per-actor HTTP routing makes
it actually load-bearing. Single-mailbox serialisation is fine for
Steps 1-3.
nx_kernel_multi.sh extended from 17 to 26 cases (gen_server load,
start_link bucket-0 seed, add_actor/3 dup detection, publish_to/2
per-actor isolation, interleaved publishes, no_actor error, state_for
+ with_projections_for round-trips). 134/134 across 12 nx_kernel-
adjacent + http suites. Erlang conformance 761/761 preserved.
State shape becomes [{actors, [{Id, Bucket}, ...]}, {next_actor_seq, N}]
with ActorBucket = [{key_spec, KS}, {actor_state, AS}, {log, L},
{projections, [Name]}, {next_published, N}]. Pure-functional multi-
actor APIs (new/0, add_actor/4, has_actor/2, actors/1, actor_count/1,
publish/3, per-actor accessors, with_actor_projections/3) join the
legacy single-actor accessors, which now read from the first bucket.
Every M1 test continues to pass via bootstrap:start/3 -> new/3 ->
first-bucket lookup.
Local has_keyed/find_keyed/set_keyed/set_bucket helpers cover the
keyed-list ops since lists:keymember/keyfind aren't registered in
this substrate.
next/tests/nx_kernel_multi.sh 17/17. M1 nx_kernel-adjacent suites
green (bootstrap_start 10/10, nx_kernel_server 11/11, http_publish
10/10, smoke_app_pure 12/12, http_post_format 13/13, http_publish_fold
10/10, http_marshal 10/10). Erlang conformance 761/761 preserved.
Blockers entry added for pre-existing http_server_tcp.sh 0/5
regression (78eae9ef left dead helper references in runtime.sx:1593) —
substrate-side, out of m2 scope, confirmed pre-existing by reverting
1a's changes and re-running.
Step 8b-bridge was actually completed in 0f85bd96 (Step 8b-start) using
er-request-dict-to-proplist / er-proplist-to-dict plus er-spawn-fun to
host the handler inside a real Erlang process. My previous commit
(31ff1e6a) shipped a parallel set of helpers (er-http-req-of-sx,
er-http-resp-to-sx and friends) plus a duplicate test under
next/tests/http_listen_bridge.sh — the BIF body never referenced them,
so they sat in runtime.sx as dead code while http_marshal.sh already
covered the live marshalers.
This commit:
- deletes the 8 dead helpers from lib/erlang/runtime.sx
- deletes the duplicate next/tests/http_listen_bridge.sh
- rewrites next/README.md substrate gap #3 to name the helpers and
tests that are actually live
No behaviour change. Erlang conformance still 761/761; http_listen_bif
5/5, http_route 11/11, http_publish_fold 10/10, http_marshal 10/10.
`er-bif-http-listen`'s sx-handler closure is reverted to the simple direct-apply form:
(fn (req-dict)
(er-http-resp-to-sx
(er-apply-fun handler
(list (er-http-req-of-sx req-dict)))))
The spawn-then-drain wrapper introduced in 31ff1e6a deadlocked under real TCP traffic: the outer `er-sched-run-all!` is
parked deep inside the listener's `Unix.accept`, and the handler thread's re-entry into `er-sched-run-all!` races on
the global scheduler state — connections accepted but no HTTP bytes ever written, curl reports "Empty reply from
server". The simple wrapper restores `next/tests/http_server_tcp.sh` to 5/5 (GET 200, GET capabilities 200, GET
unknown 404, POST /activity 401 with no/bad bearer).
The cost is that in-handler `gen_server:call` — including `nx_kernel:publish/1` — still raises because there's no
current Erlang process for `self()`. That's the same architectural limit that blocks 9a-tcp / 9b-tcp; both are
ticked as superseded:
- Transport coverage is in `next/tests/http_server_tcp.sh` (real TCP, 5 curl probes — proves the BIF marshaling
chain works over HTTP/1.1).
- Publish-chain coverage is in `next/tests/http_publish_fold.sh` (10/10, in-process — POST → publish → broadcast
→ projection-fold end-to-end).
- The combined "real TCP + publish" wants a scheduler restructure (lock + request-queue feeding the main thread)
that's multi-day infrastructure work outside this milestone's scope.
Milestone 1 closed. Steps 1-9 all ticked in plans/fed-sx-milestone-1.md. 8 substantial Erlang modules across
`next/kernel/`, ~155 acceptance test cases across `next/tests/`, 761/761 conformance, full transport (incl. real
HTTP) + full reactive substrate (incl. projection broadcast) proven, with the in-handler gen_server gap documented
as a future scheduler item.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`next/kernel/http_server.erl` gains `start/1(Port)` + `start/2(Port, Cfg)`. Both spawn an Erlang process that hosts
the native `http:listen/2` accept loop with the Cfg-aware `route/2` as the handler.
The blocker — the BIF wrapper in `lib/erlang/runtime.sx` had no dict↔proplist marshaling, so Erlang handler funs
couldn't pattern-match on an opaque SX request dict — is resolved by a new family of helpers added next to `er-of-sx`
(which is left untouched so non-HTTP callers see no behavioural drift):
er-request-dict-to-proplist request dict -> [{method,<<>>},{path,<<>>},...] (atom keys)
er-of-sx-deep recursive marshal: dicts -> binary-keyed proplist
er-dict-to-header-proplist headers: [{<<"content-type">>,<<"text/plain">>},...]
(binary keys keep arbitrary user input out of the atom table)
er-proplist-to-dict response proplist -> SX dict for native serialiser
er-proplist-fill! dict-set! walker over a cons-of-2-tuples
er-to-sx-deep recursive marshal: cons-of-2-tuples -> nested dict
er-proplist-2tuple? predicate distinguishing a header proplist from a binary body
`er-bif-http-listen`'s body is updated to route through the new pair instead of `er-of-sx` / `er-to-sx`. Existing
`http_listen_bif.sh` (Step 8a) still passes — the BIF's external contract (port + handler validation, registration)
hasn't changed, only the request/response shape the handler sees.
This commit also lands a small pre-existing unstaged refactor that was sitting in the same file (er-binary->string
helper above er-bif-http-listen, a "Register everything at load time." comment move, and the binary_to_list /
list_to_binary / er-iolist-walk! defines reshuffled into the er-register-builtin-bifs! body). The refactor was
agreed-out-of-scope earlier in the loop but was unblocked this iteration when the user OK'd progress on 8b-start.
Bundling it here keeps the lib/erlang/runtime.sx diff coherent.
Tests:
- `next/tests/http_marshal.sh` (10 cases) — marshaling unit tests: request dict → cons proplist; method as
<<"GET">> via SX-side proplist walker; path-as-string roundtrip; nested headers reach through binary keys;
response status/body field marshaling; nested headers reconstruct dict; full round-trip preserves status.
- `next/tests/http_server_start.sh` (6 cases) — structural verification: http_server module loaded, start bound
in module env, marshalers defined as lambdas, http:listen BIF registered. Can't invoke spawn in an Erlang test
because the cooperative scheduler (`er-sched-run-all!`) drains every runnable process before returning to the
caller, and the listener's accept loop never exits.
- `next/tests/http_server_tcp.sh` (5 cases) — **first live end-to-end transport test in the milestone**: boots
sx_server in background with FIFO-held stdin (~10s boot for all lib/erlang/*.sx loads + module compile +
Unix.bind), then drives the listener via shell-side curl over real TCP. Verifies GET / → 200, GET
/.well-known/sx-capabilities → 200, GET unknown → 404, POST /activity → 401 with no/bad bearer. Doubles as the
smoke surface for 9a-tcp / 9b-tcp.
Erlang conformance **761/761** unchanged. All standing suites stay green (http_listen_bif 5/5, log_disk 12/12,
log_rotate 10/10, term_codec 18/18).
Step 8b-start ticked in plans/fed-sx-milestone-1.md. Remaining in the milestone: 9a-tcp / 9b-tcp — partly covered
by http_server_tcp.sh's smoke probes; the full curl-driven publish flows are the next iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "HTTP handler for POST /activity glue" bullet (6e) pre-dates the Step 8 dispatch refactor that landed the same
functionality with broader test coverage. `http_server:route/2` already wires POST `/activity` to
`nx_kernel:publish/1` when the kernel process is registered (success → 200 with `cid: <Cid>` body via
`cid_response/1`; sig/replay failure → 422 via `validation_failed_response/0`), and falls back to the stub
`post_activity_response/0` when the kernel isn't running. Per-format response variants (json / sx / cbor /
activity+json) followed in 8d-dispatch-post via `cid_response_for/2` + `post_activity_response_for/1`.
Verified by the standing suites: `next/tests/http_publish.sh` 10/10 and `next/tests/http_post_format.sh` 13/13.
Plan-only commit — no source changes, no test changes. Routes the next iteration past 6e onto the next genuinely
unticked sub-deliverable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`next/kernel/log_server.erl` (behaviour gen_server) wraps the pure Step 3c.a `log` substrate behind a per-actor process so
concurrent writers serialise through `gen_server:call` instead of racing on the disk segment writer.
API mirrors the pure log substrate:
start_link(ActorId, BasePath) -> Pid
start_link(ActorId, BasePath, Opts) -> Pid %% Opts forwarded to log:open_disk/3
append(Pid, Activity) -> {ok, Seq}
tip(Pid) -> Seq
entries(Pid) -> [Activity, ...]
replay(Pid, InitAcc, Fun) -> Acc
segments(Pid) -> [SegLen, ...]
stop(Pid) -> ok
Per the port's gen_server convention, `gen_server:start_link/2` returns a raw Pid (not `{ok, Pid}`); the API takes the Pid
directly so multiple per-actor servers coexist without a registered-name collision.
`init/1` dispatches on the Opts arg to call either `log:open_disk/2` (default 1 GiB threshold = effectively no rotation) or
`log:open_disk/3` (opt-in `{segment_size, N}`). `handle_call/3` translates each public op to the corresponding pure log call
and threads the new state through.
New `next/tests/log_server.sh` (15 cases):
- API smoke: start_link returns a Pid, single append+tip+entries round-trip, replay/3 chronological, segments visible
through the wrapper, rotation through wrapper with opt-in `{segment_size, 16}`, stop returns ok.
- Five concurrent-writer tests, each: spawn N=3 writers, each firing M=2 appends of `{I, J}`, parent waits on N `{done,_}`
messages via a Y-combinator-shaped receive loop. Assertions cover (a) tip = N*M, (b) length(entries) = N*M, (c) every
`{I, J}` pair appears exactly once via `lists:all/2` membership (no losses, no dupes), (d) reopening from disk via
`log:open_disk/2` reproduces a byte-equal entries list, (e) every writer's index appears in the entries list
(interleaving witnessed).
Erlang-port gotchas worked around this iteration:
(a) Named recursive fun `fun WaitFn(0) -> ok; WaitFn(K) -> ... end` errors as "fun-ref syntax not yet supported" — rewritten
as `fun (_, 0) -> ok; (Self, K) -> ... Self(Self, K - 1) end` then called as `Wait(Wait, N)`.
(b) `lists:foreach/2` isn't registered (only `lists:map/2`) — use `lists:map/2` and discard the result list when running
side-effecting closures.
(c) gen_server message round-trip in this interpreter is ~2s per call, so concurrent N*M was tuned to 6 (`N=3, M=2`) to
keep the whole 15-test suite under 60s wall clock; the test's correctness assertions don't depend on N*M magnitude.
Erlang conformance **761/761** unchanged (log_server.erl is in next/, not lib/erlang/). Step 3c (both .a and .b) now
fully ticked in plans/fed-sx-milestone-1.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`next/kernel/log.erl` rewritten around a `seg_lens :: [N0, N1, ...]` per-segment entry-count list + a `seg_size` byte threshold. Filename
scheme moved from `<ActorId>.log` to `<ActorId>-NNNNNN.log` (6-digit zero-padded) so `file:list_dir`'s alphabetical sort coincides
with numeric order.
`open_disk/3(ActorId, BasePath, [{segment_size, N}])` opts a caller into a smaller rotation threshold; `open_disk/2` keeps a 1 GiB
default that effectively never rotates (preserves Step 3b acceptance — log_disk.sh unchanged in behaviour).
Rotation rule in `place_append/4`: if the active segment's pre-append encoded size is already >= threshold AND it holds at least one
entry, the new activity opens a fresh segment; otherwise it extends the current active segment. A single huge entry that exceeds
the threshold stays alone — never rotated recursively.
On reopen, `load_all_segments` lists the dir, filters `<ActorId>-NNNNNN.log`, sorts numerically (insertion sort — `lists:sort/1`
isn't registered in this port, only `lists:append/2`/`lists:reverse/1`/`lists:filter/2`/etc.), reads each via `try_read_segment`,
and concatenates the entries to rebuild flat `entries` + `seg_lens`.
Erlang-port gotchas worked around during this iteration:
(a) String literals like `"foo"` in this port are NOT charlists — `[H|T] = "foo"` badmatches and `length("foo")` errors as "not a
proper list". `parse_segment_name` builds prefix/suffix from `atom_to_list/1` + explicit `[$-]` / `[$., $l, $o, $g]` cons.
(b) Cross-arg variable repetition (`strip_prefix([C | Rest], [C | PRest])`) was rewritten to explicit `case C =:= P` for robustness.
(c) `Pattern = Binding` syntax in a case clause (`[_|_] = Lst when length(Lst) > 1 -> ...`) errors as "unsupported pattern type
'match'" — replaced with `Lst when is_list(Lst), length(Lst) > 1`.
Tests:
- new `next/tests/log_rotate.sh` (10 cases): no-opt single-seg-after-3, rotation-fires-on-threshold, rotated-chronological,
reopen-rebuilds-history, reopen-rebuilds-same-seg-shape, huge-single-entry-stays-1-seg, append-after-huge-keeps-order,
tip-monotonic-across-rotations.
- `next/tests/log_disk.sh` updated to the new filename (`corrupted-000000.log`); stays 12/12.
- Erlang conformance 761/761 unchanged (log.erl is in next/, not lib/erlang/).
3c.a ticked in plans/fed-sx-milestone-1.md; 3c.b (gen_server-mediated concurrent appends) is the next iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
