New next/kernel/discovery.erl with the local-side webfinger
primitives per design §13.7:
parse_acct/1(Bin) -> {ok, User, Host} | {error, _}
Accepts <<acct:user@host>> (with prefix) or <<user@host>>
(bare). Host preserves an optional :port suffix. Rejects
empty user/host and missing @.
parse_resource/1 alias for the webfinger ?resource= shape
actor_url_for/2(User, Host)
Synthesises <<http://<host>/actors/<user>>>. TLS / https
is v3, gated on a TLS substrate Blocker.
webfinger_body/3(User, Host, ActorUrl)
Builds the RFC 7033 JSON body:
{"subject":"acct:<user>@<host>",
"links":[{"rel":"self",
"type":"application/activity+json",
"href":"<actor_url>"}]}
Hand-rolled byte concatenation — no JSON BIF on this port.
Substrate gotcha re-confirmed: <<"acct:">> string literals
truncate to one byte on this port. "acct:" is spelled as
<<97,99,99,116,58>> in the implementation.
12/12 in next/tests/discovery.sh covering:
- parse_acct prefixed + bare forms
- host with :port preserved
- reject empty user / missing @ / empty host
- parse_resource alias
- actor_url_for synthesis + port preservation
- webfinger_body prefix shape + byte_size sanity
Step 10b (http_server route GET /.well-known/webfinger) and
Step 10c (peer-actor fetch via Step 5's lookup_or_fetch slot)
layer on top. 10c gates on Blockers #2 (native http-request
primitive missing).
delivery_worker state shape gains :next_retry proplist alongside
the existing :attempts:
[{peer, _}, {pending, _}, {attempts, [{Cid, N}]},
{next_retry, [{Cid, NextRetryAt}]}, {dead_letter, _},
{dispatch_fn, _}]
New pure-functional exports:
record_failure_pure/3(Cid, Now, State)
Bumps :attempts for Cid. On the 6th failure
(backoff_for returns dead_letter) moves the matching
activity from :pending to :dead_letter and clears the
:next_retry entry. Otherwise sets next_retry to
Now + backoff_for(NewAttempts).
record_success_pure/2(Cid, State)
Clears both :attempts and :next_retry for Cid.
next_due_pure/2(Now, State)
Returns cids whose retry time has passed (insertion
order preserved so the worker drains in FIFO retry
order).
attempts_for/2, next_retry_at/2, dead_letter_list/1
Read-side accessors.
Internal helper move_to_dead_letter/2 + take_by_cid/4 walks
:pending to find the matching activity by cid.
11/11 in next/tests/delivery_retry.sh covering:
- fresh state: 0 attempts / undefined retry / [] dead_letter
- record_failure bumps to 1
- record_failure sets next_retry_at = Now + 30 (slot 1)
- second failure: attempts=2, NextRetryAt = Now + 300 (slot 2)
- record_success clears both
- next_due returns due cids
- next_due empty before due
- 6th failure -> dead-letter; activity out of :pending
- dead-lettered cid removed from :next_retry
- per-cid isolation: success on one doesn't disturb another
delivery_worker.sh 17/17 unchanged (new exports are additive).
Blockers added:
#2 — Native http-request primitive missing in bin/sx_server.ml
(briefing assumed it existed; only http-listen exists).
Belongs to loops/fed-prims. Step 8e wrapper waits for
the native.
#3 — erlang:send_after-style timer primitive missing. Needed
for the real retry loop. Belongs to loops/erlang. 8b-pure
captures the semantics so 8b-timer is a 1-shot wiring
when the primitive lands.
Conformance preserved at 761/761.
outbox:publish/2 now walks the computed delivery_set and enqueues
the signed activity onto each matching delivery_worker
(registered under the peer-id atom). Missing workers are silently
skipped — lazy worker creation belongs to the kernel manager
later in Step 8.
Gated by Context's {dispatch_deliveries, true} so every M1
outbox caller (and every M2 caller that doesn't yet care about
delivery) stays back-compat: default off.
New helpers in outbox.erl:
dispatch_deliveries/3(Activity, DeliverySet, Context)
gates on Context :dispatch_deliveries flag
enqueue_each/2(Activity, [PeerId | _])
whereis-guarded enqueue per peer
7/7 in next/tests/delivery_dispatch.sh:
- single peer enqueued
- two peers both enqueued (fan-out)
- missing worker silently skipped
- no :dispatch_deliveries flag -> no-op (back-compat)
- two publishes -> FIFO append on the queue
- empty delivery_set -> no-op
outbox_publish.sh 17/17 unchanged; delivery_worker.sh 17/17
unchanged. Conformance preserved at 761/761 from the Step 8a
baseline.
outbox:publish/2 now computes the audience-resolved delivery set
after sign + log and stashes it in the Result proplist as
{delivery_set, [ActorId, ...]}. Step 8's delivery-queue worker
reads it off the publish result.
New compute_delivery_set/3(Request, Signed, Context):
- Pulls :follower_graph from Context (defaults to empty graph)
- Calls recipients_envelope/2 to synthesise a minimal envelope
from Request's :to / :cc + Signed's :actor
- Routes through delivery:delivery_set/3 unchanged
The envelope construct/4 surface doesn't carry :to / :cc (only
type / actor / published / object), and changing that ripples
through every envelope shape test. recipients_envelope/2 keeps
the compute boundary local to outbox.
4 new cases in outbox_publish.sh (17/17 total):
- Result :delivery_set empty default
- explicit :to -> [bob] in set
- followers symbol expands via Context :follower_graph
- self-suppression (alice in :to drops to []bob])
Module loads rebumped: follower_graph + delivery added as
dependencies; outbox shifts from epoch 5 to epoch 7. Internal
sx_server timeout bumped 240s -> 480s to fit the larger module
set.
Step 7 fully closed (7a delivery module + 7b public expansion
+ 7c outbox integration). Federation now has the end-to-end
audience resolution: an outbound activity's :to / :cc plus any
follower_graph expansion becomes a deduped recipient list ready
for Step 8 to dispatch.
Conformance running + adjacent gate running.
delivery:expand_audience(public, Sender, Graph) now returns the
sender's followers (same as the followers symbol). Per design
§13.4 the practical Public fan-out semantics for an open social
network is 'every follower of the publishing actor'. The
explicit shared-inbox peer-instance model (Mastodon-style
per-instance broadcast) defers to v3 when there's a real
known-peer-instance registry to drive it.
19/19 in delivery_set.sh:
- public symbol now expands to sender's followers (epoch 19,
updated from v2 placeholder)
- public with empty follower-graph -> [] (epoch 28)
- public + followers in same audience dedupe (epoch 29)
Conformance 761/761.
New next/kernel/delivery.erl computes the audience-resolved
deduplicated recipient list for an outbound activity.
delivery_set/2(Activity, KernelState)
delivery_set/3(Activity, KernelState, FollowerGraph)
Returns a deduplicated list of ActorId atoms. Step 8 will
resolve each entry to {PeerInstanceUrl, ActorId} via the
peer-actors cache.
Sources unioned then deduped:
- :to field (single ActorId or list, atoms or audience symbols)
- :cc field (same shape)
- audience-symbol expansion:
followers -> sender's followers from follower_graph
public -> [] for v2 (Step 7b layers known-peer-instance set)
Self-delivery suppressed every time the sender's ActorId appears
in the set.
Module lives in its own file (not inside outbox.erl) so Step 8's
delivery-queue gen_server has a clean home alongside it.
17/17 in next/tests/delivery_set.sh covering:
- empty activity -> []
- single :to atom + list :to recipients
- :to + :cc unioned
- self-suppression
- duplicate / cross-field dedup
- followers symbol expands via follower_graph state
- empty follower-graph -> []
- public v2 placeholder -> []
- mixed explicit + followers
- collect_recipients raw flat
- suppress_self drops every match
- dedup preserves first-occurrence order
- expand_audience pass-through for plain ActorId
Conformance 761/761. 86/86 across 6 Step-7-adjacent suites
(follower_graph, follow_lifecycle, auto_accept, inbox,
nx_kernel_multi, outbox_publish).
Per design §13.2 the v2 Follow policy is open-world: every
successfully-ingested Follow triggers an Accept publish from the
target actor. Enabled per-Cfg via {auto_accept_follows, true} so
manual-moderation deployments can leave it off; default off.
http_server.erl run_inbox_pipeline gained maybe_auto_accept/3:
maybe_auto_accept(TargetAtom, Activity, Cfg) ->
case field(auto_accept_follows, Cfg) of
true ->
case envelope:get_field(type, Activity) of
{ok, follow} ->
Req = [{type, accept}, {object, Activity}],
nx_kernel:publish_to(TargetAtom, Req);
_ -> ok
end;
_ -> ok
end.
The publish routes through the full outbox pipeline (envelope
construct + HMAC sign + log append + outbox projection broadcast).
When the target's outbox :projections list shares the same
follower_graph projection that inbox broadcasts into, the bilateral
relationship fold-converges automatically — alice.followers = [bob]
and bob.following = [alice], both pending lists clear. No extra
test scaffolding needed because outbox:publish already runs the
broadcast hook from Step 7c.
Bad-sig and non-Follow ingestion short-circuit before the Accept
attempt (the validation pipeline rejects before run_inbox_pipeline's
ok branch fires).
9/9 in next/tests/auto_accept.sh:
- auto_accept on: alice's outbox tip advances to 1
- alice's outbox entry has :type = accept
- follower_graph converges to {alice.followers=[bob],
bob.following=[alice]}
- both sides' pending lists clear after the Accept fold
- auto_accept off (default): outbox stays empty; pending_inbound
still gets populated from the Step 6b inbox-projection path,
but alice.followers stays empty until human moderation acts
- non-Follow ingestion (Create{Note}) with auto_accept on: no
Accept published
- bad-sig Follow with auto_accept on: no Accept (sig short-circuit
in pipeline before maybe_auto_accept runs)
Step 6 fully closed (6a follower_graph projection, 6b inbox -> projection
broadcast wiring, 6c auto-Accept publish).
Conformance 761/761. 89/89 across 7 Step-6-adjacent suites
(inbox, inbox_peer_resolution, follower_graph, follow_lifecycle,
auto_accept, http_publish, nx_kernel_multi).
http_server.erl run_inbox_pipeline now calls
broadcast_to_inbox_projections/2 after a successful
nx_kernel:append_inbox. Cfg may carry {inbox_projections,
[Name, ...]} listing projection gen_servers that should see every
successfully-ingested inbound activity. Each gets the activity via
projection:async_fold/2 — fire-and-forget so the inbox handler
doesn't block on fold processing. Empty / absent
:inbox_projections is a no-op (back-compat with Step 5d callers).
v2 leaves the routing field global (every inbound activity goes
to every named projection); per-actor projection wiring is a
forward-looking follow-up.
9/9 in next/tests/follow_lifecycle.sh:
- Follow ingestion -> 202
- follower_graph state: alice.pending_inbound = [bob]
- follower_graph state: bob.pending_outbound = [alice]
- inbox tip advances to 1 (Step 5a invariant preserved)
- no inbox_projections Cfg -> projection state stays empty
- end-to-end: Follow + Accept fold converges to
alice.followers = [bob] and bob.following = [alice]
(Accept fed via projection:async_fold for v2 — auto-Accept
publish is Step 6c)
- bad-sig inbound short-circuits before broadcast
- two distinct peer Follows accumulate
bootstrap_start.sh internal sx_server timeout bumped 300s -> 600s
to match the cumulative cost trend other tests are seeing on this
port. (bootstrap_start doesn't load http_server but loads bootstrap
+ the full genesis bundle + 9 kernel modules — same cumulative
compile budget.)
Conformance 761/761.
New next/kernel/follower_graph.erl is the Erlang-fun stand-in for
the genesis follower-graph.sx projection body, mirroring the
shape of actor_state.erl and define_registry.erl.
State shape (substrate has no maps, so a proplist):
[{ActorId, [{following, [PeerId, ...]},
{followers, [PeerId, ...]},
{pending_outbound, [PeerId, ...]},
{pending_inbound, [PeerId, ...]}]}, ...]
Fold rules per design §13.2:
Follow{actor: A, object: B}
add B to A.pending_outbound
add A to B.pending_inbound
Accept{actor: B, object: Follow{A->B}}
A moves from B.pending_inbound -> B.followers
B moves from A.pending_outbound -> A.following
Reject{actor: B, object: Follow{A->B}}
clear A from B.pending_inbound, B from A.pending_outbound
Undo{actor: A, object: Follow{A->B}}
drop A<->B from every list on either side
only the Follow's original actor may Undo it
Edge cases handled:
- self-follow (alice -> alice) is a no-op
- duplicate Follow is idempotent (list sets)
- Accept/Reject/Undo whose :object isn't a Follow proplist
passes through
- Undo by the wrong actor (carol Undoing Follow{alice->bob})
is a no-op
Public API:
new/0, lookup/2, actors/1
following/2, followers/2,
pending_outbound/2, pending_inbound/2
is_following/3, has_follower/3,
is_pending_outbound/3, is_pending_inbound/3
fold/2, fold_fn/0
fold_fn/0 returns the standard 2-arity Erlang fun for
projection:start_link/3 (same plug shape as actor_state and
define_registry).
Local find_keyed/set_keyed/contains/remove_member helpers — no
lists:keyfind/keymember/member in this substrate (same gap as
Step 1a/2b/5a/5c).
18/18 in next/tests/follower_graph.sh covering all four verbs,
predicates, edge cases (self-follow, duplicate Follow, untyped
activity, non-Follow :object, wrong-actor Undo).
Step 6b wires this into the inbox handler so a peer Follow lands,
fires auto-Accept publish (open-world policy per §13.2; manual
moderation deferred to v3).
Conformance 761/761. 130/130 across 9 Step-6-adjacent suites
(inbox, inbox_bucket, inbox_pipeline, inbox_peer_resolution,
actor_state_pure, define_registry_pure, projection_pure,
nx_kernel_multi, smoke_app_pure).
POST /actors/<id>/inbox is now special-cased in route/2 (next to
POST /activity) so the body + Cfg reach the new handle_inbox_post/3
handler.
Wire format: body = term_codec:encode(SignedActivity); the receiver
decodes into the activity proplist and runs the chain.
handle_inbox_post/3 orchestration:
1. kernel_has_actor(field(kernel, Cfg), TargetId) -> 404 if missing
2. decode_activity(Body) -> 422 on bad shape
3. envelope:get_field(actor, Activity) -> 422 if no peer id
4. resolve_peer_as(PeerId, Cfg) -> 401 if unknown
5. nx_kernel:inbox_state_for(TargetAtom) -> 404 belt-and-braces
6. pipeline:validate_inbound(Activity, PeerAS, InboxLog)
ok -> nx_kernel:append_inbox + 202
{error, bad_signature} -> 401
{error, no_signature} -> 401
{error, _} -> 422
resolve_peer_as/2 supports three Cfg paths in priority order:
{peer_as, [{PeerId, AS}, ...]} pure-fn pre-populated map
{peer_actors, AtomName} peer_actors gen_server cache
{peer_fetch_fn, fun/1} fallback on srv cache miss
Empty Cfg returns {error, no_peer_resolver} -> 401.
v1 actor_post/1 4a stub deleted; M1 actor_inbox_post_response/0
kept for response composition.
Projection broadcast on inbox success intentionally deferred to a
follow-up sub-deliverable.
inbox.sh 11/11 (acceptance suite for the basic chain):
- happy path -> 202
- inbox tip advances; outbox tip unchanged (per-actor bucket
independence carried through from Step 5a)
- empty / garbage body -> 422
- unknown peer -> 401
- bad peer-AS keys -> 401
- replay (same activity twice) -> 422 on second
- unknown target actor -> 404
- two distinct activities -> tip = 2
inbox_peer_resolution.sh 6/6 (Cfg resolution variants):
- peer_actors gen_server hit -> 202
- FetchFn fallback -> 202
- FetchFn error -> 401
- FetchFn caches into peer_actors (peers_srv shows [bob] after)
- No resolver -> 401
Tests split into two files because each epoch's kernel start_link
+ outbox construct + term_codec encode is expensive and a single
suite hits the wall-clock budget.
http_server.erl is now 1181 lines. erlang-load-module on this port
scales superlinearly with function count, so eight http_*.sh tests'
internal sx_server timeout bumped 60s -> 360s (http_route,
http_actors, http_accept, http_capabilities, http_capabilities_format,
http_content_type, http_artifacts, http_projections).
Conformance 761/761.
New next/kernel/peer_actors.erl is the federation-side cache for
{PeerActorId, PeerActorState} entries. PeerAS is exactly the shape
envelope:verify_signature/2 reads (proplist with :public_keys), so
the inbox handler can pipe the cache hit straight into
pipeline:validate_inbound/3 from Step 5b.
Pure-functional API:
new/0
lookup/2(PeerId, State) -> {ok, PeerAS} | not_found
store/3(PeerId, PeerAS, State) -> NewState
evict/2(PeerId, State) -> NewState
peers/1(State) -> [PeerId]
lookup_or_fetch/3(PeerId, FetchFn, State)
-> {ok, PeerAS, NewState} cache hit returns unchanged State,
miss stores FetchFn result.
| {error, Reason, State} FetchFn failure preserves cache.
| {error, {bad_fetch_return, X}, State}
FetchFn contract: (PeerId) -> {ok, PeerAS} | {error, Reason}.
Failed fetches do NOT poison the cache so callers can retry on
transient HTTP failures.
gen_server wrapper (registered name peer_actors):
start_link/0,1 start_link/1 accepts initial proplist for fixtures
stop/0
lookup_srv/1
store_srv/2
lookup_or_fetch_srv/2
peers_srv/0
evict_srv/1
handle_call dispatches mirror the pure-fn paths exactly.
The actual HTTP-GET fetch implementation (peer's actor doc -> peer
AS proplist) is Step 5d's responsibility — for 5c, FetchFn is just
the contract callers fill in.
19/19 in next/tests/peer_actors.sh:
- new/0 -> []
- lookup miss -> not_found
- store + lookup round-trip
- peers/1 in insertion order
- evict + evict-unknown no-op
- lookup_or_fetch miss invokes FetchFn, hits cache after
- lookup_or_fetch hit skips FetchFn (verified by tombstone fn)
- fetch error preserves cache state
- bad fetch return shape captured
- gen_server start_link + miss/hit/fetch/evict round-trips
- start_link/1 pre-populates cache from initial state
Conformance 761/761. 139/139 across 9 Step-5-adjacent suites
(inbox_pipeline, inbox_bucket, pipeline_signature, registry_server,
projection_server, nx_kernel_multi, bootstrap_start, http_publish,
smoke_app_pure, plus the new peer_actors).
New federation inbound pipeline that runs envelope-shape -> peer
signature -> replay against the receiving actor's inbox log.
pipeline.erl additions:
validate_inbound/3(Activity, PeerActorState, InboxLog)
runs inbound_stages(PeerAS, InboxLog) and halts on first
failure (existing run_stages/2 driver). Returns ok |
{error, Reason}.
inbound_stages/2(PeerAS, InboxLog)
[stage_envelope, stage_signature(PeerAS), stage_replay(InboxLog)]
M1's validate_inbound/1 and the static inbound_stages/0 (envelope-
only) are preserved — outbox-side callers don't have to re-key on
a peer-AS they don't have.
Signature verification routes through the peer's actor-state
:public_keys (NOT the local kernel's actor-state). Peer-AS
resolution is the caller's responsibility for 5b; Step 5c wires
the peer-actors cache lookup.
14 cases in next/tests/inbox_pipeline.sh:
- happy path: valid signed activity + correct peer AS + empty
inbox -> ok
- bad envelope shape -> {error, _} (stage_envelope rejects)
- unsigned activity -> stage_envelope rejects on
{missing_field, signature} before sig runs
- wrong peer AS (peer's claimed key bytes differ from real) ->
{error, bad_signature}
- replay: inbox already contains the same activity -> {error, replay}
- inbox with a different activity doesn't trigger replay
- inbound_stages/2 returns exactly 3 stages
- inbound_stages/0 still returns 1 stage
- validate_inbound/1 still works
- shape failure short-circuits before sig
- sig failure short-circuits before replay
- two distinct activities both verify against empty inbox
- inbox-of-one doesn't replay the other
Conformance 761/761. 130/130 across 10 Step-5-adjacent suites
(pipeline_envelope, pipeline_signature, pipeline_replay,
pipeline_driver, inbox_pipeline, inbox_bucket, nx_kernel_multi,
bootstrap_start, http_publish, outbox_publish, smoke_app_pure).
Adds the receiving-side log bucket every actor needs. add_actor/4
now opens a fresh in-memory log via log:open(ActorId, inbox_base_stub())
and stores it on the bucket as {actor_inbox, LogState} alongside
the outbox {log, _}. Two distinct base stubs ensure the in-memory
log module returns separate states even when the same ActorId is
the actor.
Pure-functional exports:
actor_inbox_state/2(ActorId, State) -> {ok, LogState} | {error, _}
actor_inbox_tip/2(ActorId, State) -> integer | nil
append_to_actor_inbox/3(ActorId, Activity, State)
-> {ok, NewTip, NewState} | {error, no_actor, State}
gen_server exports (mirror the outbox shape):
inbox_tip_for/1(ActorId) -> integer | nil
inbox_state_for/1(ActorId) -> {ok, LogState} | {error, _}
append_inbox/2(ActorId, Activity) -> {ok, NewTip} | {error, _}
handle_call dispatch added for all three.
Inbox and outbox tips are completely independent — appending to one
doesn't touch the other. This is the storage primitive 5b will
build the inbound validation pipeline on top of.
log:append/2 signature noted in code + progress log: it takes
(LogState, Activity) and returns {ok, NewState, Seq} — not
{ok, NewState} as I originally guessed.
next/tests/inbox_bucket.sh 14/14:
- fresh inbox tip = 0 (pure)
- actor_inbox_state {ok, _} (pure)
- append_to_actor_inbox/3 -> {ok, 1, _}
- tip advances after append
- unknown actor -> {error, no_actor, _}
- outbox + inbox tips fully independent
- two actors maintain independent inbox state
- gen_server inbox_tip_for/1 starts at 0
- gen_server append_inbox/2 -> {ok, 1}
- gen_server inbox != outbox tip
- gen_server unknown -> {error, no_actor}
- gen_server inbox_state_for {ok, _}
- two appends -> tip = 2
Conformance 761/761. 125/125 across 7 Step-5-adjacent suites
(inbox_bucket, nx_kernel_multi, nx_kernel_server, bootstrap_start,
http_publish, http_multi_actor, actor_lifecycle, smoke_app_pure).
Per-actor GET /actors/<id>/outbox now reads the bucket's log via
new nx_kernel:log_state_for/1 gen_server export and renders the
paged CID list.
nx_kernel additions:
log_state_for/1 gen_server call returning {ok, LogState} for
the named actor (mirrors log_tip_for/1's shape).
http_server additions:
- with_request_query/2 bakes Req's :query binary into Cfg as
{request_query, Q} so sub-resource handlers can parse params
without taking the Req as another arg
- kernel_actor_log_data/2 -> {Tip, Entries} via
nx_kernel:log_tip_for + log_state_for + log:entries
- parse_page/1 reads ?page=N (default 1, non-digits -> 1)
- page_size/0 returns 5 (test-friendly; production picks 20+)
- page_slice/2 + drop_take/3 + take/2 for the page extraction
- entry_cids/1 maps entries to :id CID binaries via envelope
- actor_outbox_full_response_for/5 renders text / JSON / SX:
text: outbox: <id>\ntip: N\npage: P\nitem: <cid>\n...
json: {"outbox":"<id>","tip":N,"page":P,"items":[...]}
sx: (outbox "<id>" :tip N :page P :items (...))
Empty page degrades to actor_outbox_with_tip_response_for so
epochs 50-57 from Step 4c still pass — the prefix is preserved.
8 new cases in next/tests/http_multi_actor.sh (41/41 total):
- 1 publish -> body contains outbox/tip=1/page=1/item: prefix
- 3 publishes -> body contains tip=3/page=1/item: prefix
- page=2 with 3 items -> empty page degrades to tip-only body
- 6 publishes page=1 -> tip=6/page=1/item: prefix
- 6 publishes page=2 -> tip=6/page=2/item: prefix
- JSON body shape with items array (1 entry)
- SX body shape with :items list (1 entry)
- bad ?page=bad falls back to page 1
Conformance 761/761. 117/117 across 11 Step-4-adjacent suites
(http_multi_actor, http_route, http_publish, http_post_format,
http_marshal, http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, nx_kernel_server, bootstrap_start, actor_lifecycle).
Substrate gotcha logged: named recursive funs fun F(...) -> F(...)
end aren't supported by the parser ('fun-ref syntax not yet
supported'); binary:matches/2 and lists:foreach/2 aren't registered.
Tests prove behaviour via match_prefix substring checks rather than
counting occurrences.
http_server:route/3(Req, Cfg, Kernel) is the new extended entry
point: folds the kernel reference (typically the registered
nx_kernel atom) into Cfg as {kernel, Kernel}. route/2 is
unchanged and stays the M1 surface.
The dispatch chain gained Cfg threading all the way down:
dispatch/3 -> dispatch/4 (M, P, F, Cfg)
actor_get/2 -> actor_get/3 (Rest, F, Cfg)
actor_subresource_get/3 -> /4 (Id, Sub, F, Cfg)
actor_outbox_response_for/3 (new) reads :kernel from Cfg and,
when the kernel atom is registered AND the actor exists, renders
'tip: <N>' alongside the actor id in text / JSON / SX content-
negotiated bodies. Unknown actors or unregistered kernels fall
back to the 4a stub.
Inbox / followers / following handlers accept Cfg but ignore it
for now — they layer real state lookup in 4d/4e/Step 5+.
Substrate gotcha logged in the Progress log: try/of/catch around
gen_server:call(nx_kernel, _) deadlocks in this port's scheduler
(probably the catch frame's mask defers reply delivery). The
live kernel_log_tip/2 helper does a bare call + integer guard
instead. nx_kernel_multi.sh already proves bare gen_server:call
into the same kernel works correctly.
8 new cases in next/tests/http_multi_actor.sh (33/33 total):
- route/3 with registered kernel: outbox body includes tip=0
- tip advances after POST publish through route/3 + token map
- unknown actor (ghost) falls back to 4a stub (no tip:)
- unregistered kernel ref falls back to stub
- JSON Accept renders {"outbox":"alice","tip":0}
- SX Accept renders (outbox "alice" :tip 0)
- Bob's outbox tip stays 0 while Alice publishes (per-actor)
- route/2 path unchanged: no tip field in body
Conformance 761/761. 121/121 across 10 Step-4-adjacent suites
(http_multi_actor, http_route, http_publish, http_post_format,
http_marshal, http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, bootstrap_start, actor_lifecycle).
POST /activity now routes through nx_kernel:publish_to/2 when the
bearer token resolves to an explicit ActorId via Cfg's :tokens
proplist:
Cfg = [{tokens, [{<<"alice-token">>, alice},
{<<"bob-token">>, bob}]}]
resolve_token/2 returns {ok, ActorId} on a :tokens hit. On a miss
it falls back to the M1 :publish_token single-token field — match
returns {ok, legacy}, routing through nx_kernel:publish/1 (which
fans out to bucket 0) so every M1 test continues to pass.
handle_post_activity threads the resolved ActorRef to
publish_if_kernel/3 which dispatches publish_to/2 for explicit
actor ids and publish/1 for the legacy atom. The no-kernel
auth-only path (which preserves the post_activity_response_for stub
for unit-style tests of http_server alone) is unchanged.
Dead expected_token/1 helper removed (was only called by the old
check_bearer arm that resolve_token replaces).
8 new cases in next/tests/http_multi_actor.sh (25/25 total):
- two-actor Cfg, Alice token -> 200 with cid:
- Alice token publishes to alice (log_tip alice=1, bob=0)
- Bob token publishes to bob (log_tip alice=0, bob=1)
- interleaved Alice + Bob + Alice -> {2, 1}
- unknown token + no :publish_token -> 401
- legacy :publish_token still works (M1 back-compat)
- tokens map AND legacy :publish_token coexist (each resolves to
its own actor; legacy lands on alice bucket via publish/1)
- no kernel + valid :tokens entry -> auth-only stub 200
Conformance 761/761. 116/116 across 10 Step-4-adjacent suites
(http_multi_actor, http_route, http_publish, http_post_format,
http_marshal, http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, bootstrap_start, actor_lifecycle).
Per design §16.1 each actor has /outbox /inbox /followers /following
sub-paths. New split_first_slash/1 helper lets the GET /actors/...
dispatch arm fan out on the sub-segment:
GET /actors/<id> actor doc (M1 — unchanged)
GET /actors/<id>/outbox outbox stub (4a)
GET /actors/<id>/inbox inbox stub (4a)
GET /actors/<id>/followers follower stub (4a)
GET /actors/<id>/following following stub (4a)
POST /actors/<id>/inbox 202 Accepted stub (4a; Step 5 real)
Four new content-negotiated response functions mirror the existing
actor_doc_response_for/2 shape (text / json / activity_json / sx
variants):
actor_outbox_response_for/2
actor_inbox_get_response_for/2
actor_followers_response_for/2
actor_following_response_for/2
POST returns 202 via new accepted_response/1 +
actor_inbox_post_response/0.
Unknown sub-paths under /actors/<id>/ return 404. Bare /actors/<id>
preserves the M1 actor-doc arm so http_route + http_post_format
regression suites stay green.
4b-4e (token map, route/3 kernel access, per-actor outbox listing
from log entries, real inbox pipeline) layer on top of this dispatch
in subsequent iterations.
17/17 in next/tests/http_multi_actor.sh covering:
- split_first_slash sanity (no slash / id+sub / trailing slash)
- all four GET sub-paths return 200 with stub bodies
- POST inbox returns 202 + 'accepted'
- unknown sub-paths return 404 (GET and POST)
- empty /actors/ returns 404
- body carries the actor id
- content negotiation: outbox JSON, inbox SX, followers JSON
Conformance 761/761. 120/120 across 10 Step-4-adjacent suites
(http_route, http_publish, http_post_format, http_marshal,
http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, actor_state_pure, bootstrap_start).
actor_state.erl fold_update routes patches through apply_patch/3
which special-cases two rotation patch entries per design §9.6:
{add_publicKey, KeyProplist}
Append to :public_keys; default :created to activity's
:published if unset.
{supersede, OldKeyId}
Mark the matching key with :superseded_at = activity's
:published. Existing :superseded_at preserved (idempotent);
unknown :id no-op.
Other patch entries still last-write-wins per key (Step 2b semantics
preserved; verified by actor_state_pure 19/19 unchanged).
New exports:
key_history/1 — full :public_keys list (preserves superseded)
active_keys_at/2 — subset active at time T (mirrors envelope's
is_active_at; envelope keeps that predicate
private, so a local copy lives here)
find_key_by_id/2 — lookup by :id in the history
Rotation-purpose schema gating per §9.6 (rotation must be signed
by a key with :rotate-key purpose) is deferred to Step 5 (peer-side
stage_signature will plumb purpose through the pipeline).
16/16 in next/tests/key_rotation.sh covering:
- rotation arithmetic (add_publicKey + supersede combined)
- new key :created = rotation activity's :published
- supersede marks :superseded_at correctly
- key_history preserves all keys (superseded included)
- active_keys_at semantics at T=pre / T=rotation / T=post
- live envelope:verify_signature/2 round-trips:
pre-rotation activity signed with K1 -> ok
post-rotation activity signed with K2 -> ok
post-rotation activity signed with K1 -> {error, no_active_key}
- non-rotation Update patches preserve key history
- add_publicKey alone (no supersede) keeps old key active
- supersede alone empties active set
- supersede with unknown id is a no-op
- second supersede on superseded key is idempotent
Conformance 761/761. 132/132 across 9 Step-3-adjacent suites
(key_rotation, actor_state_pure, actor_lifecycle, envelope_sig,
envelope_shape, envelope_canonical, nx_kernel_multi, bootstrap_start,
smoke_app_pure).
New nx_kernel:bootstrap_actor/4(ActorId, Profile, KeySpec, State)
single-call entry that adds an actor bucket and immediately publishes
a Create{Person|Service|Group} envelope as the bucket's first activity:
- Profile carries :type, :name, :preferredUsername, :summary, :icon,
:public_keys. :type defaults to person if unset.
- Kernel AS proplist built from Profile's :public_keys (falls back
to []).
- Create object built from Profile fields (Step 2b actor_state
fold picks the same field set).
gen_server variant bootstrap_actor/3 for live-kernel use plus a new
handle_call branch.
15/15 in next/tests/actor_lifecycle.sh covering pure + gen_server +
actor_state projection capture for all three actor types:
- Pure: bootstrap_actor advances log_tip = 1, Create has
object.type = person
- Pure: two actors share a kernel with independent log tips
- Pure: duplicate bootstrap_actor -> already_present
- Pure: typeless profile defaults to person
- Pure: empty public_keys handled
- gen_server: bootstrap_actor/3 against a live registered kernel
- actor_state projection captures Person, Service, Group profiles
- profile carries :preferredUsername + :public_keys from the
Create object
Closes Step 2 (2a Person/Service/Group genesis files,
2b actor_state projection fold, 2c bootstrap_actor + integration).
Conformance 761/761. 146/146 across 10 Step-2-adjacent suites
(actor_lifecycle, actor_state_pure, nx_kernel_multi, nx_kernel_server,
bootstrap_start, smoke_app_pure, smoke_pin_pure, define_registry_pure,
projection_server, outbox_publish).
next/kernel/actor_state.erl mirrors define_registry's structure: a
2-arity fold_fn that plugs into projection:start_link/3, an
Erlang-fun stand-in for the genesis actor-state.sx projection body.
State shape:
[{ActorId, Profile}, ...]
Profile is a property list with :type, :name, :preferredUsername,
:summary, :icon, :public_keys, :moved_to, :created. Maps #{} aren't
registered in this substrate, so this matches the kernel bucket /
registry shape convention.
Folding rules per design §9.1-§9.4:
- Create{Person|Service|Group}: register profile, capturing object
fields + :published seq as :created. Duplicate Create no-overwrite.
- Update{Person|Service|Group, patch}: deep-merge :patch into
profile last-write-wins per key.
- Move: record :moved_to.
Other activity types and non-actor object Creates pass through.
Local find_keyed/has_keyed/set_keyed helpers (same gap as Step 1a:
no lists:keyfind/keymember in this substrate).
19/19 in next/tests/actor_state_pure.sh covering:
- new/0/has/2/lookup/2/actors/1 base cases
- Create for Person/Service/Group all three actor types
- Profile field capture (name, preferredUsername, public_keys, created)
- Duplicate Create no-overwrite
- Two independent actors
- Update field merge + per-key last-write-wins
- Update for unknown actor pass-through
- Move :moved_to
- Non-actor Creates pass through
- Activities without :actor pass through
- fold_fn/0 returns is_function(F, 2)
Conformance 761/761. Step-2-adjacent no-regression gate 106/106
across 6 suites (define_registry_pure, projection_pure,
projection_server, nx_kernel_multi, bootstrap_start, smoke_app_pure).
New gen_server exports add_actor/3, publish_to/2, log_tip_for/1,
actors/0, state_for/1, bucket_for/1, with_projections_for/2 —
each is a thin gen_server:call delegating to 1a's pure-functional
bucket API via fresh handle_call branches. Existing single-actor
calls (publish/1, log_tip/0, with_projections/1) route through
bucket 0 unchanged.
Per-actor mailbox sharding (one gen_server per bucket so distinct-
actor publishes don't serialise on a single mailbox) is forward-
looking — deferred to Step 4 where the per-actor HTTP routing makes
it actually load-bearing. Single-mailbox serialisation is fine for
Steps 1-3.
nx_kernel_multi.sh extended from 17 to 26 cases (gen_server load,
start_link bucket-0 seed, add_actor/3 dup detection, publish_to/2
per-actor isolation, interleaved publishes, no_actor error, state_for
+ with_projections_for round-trips). 134/134 across 12 nx_kernel-
adjacent + http suites. Erlang conformance 761/761 preserved.
State shape becomes [{actors, [{Id, Bucket}, ...]}, {next_actor_seq, N}]
with ActorBucket = [{key_spec, KS}, {actor_state, AS}, {log, L},
{projections, [Name]}, {next_published, N}]. Pure-functional multi-
actor APIs (new/0, add_actor/4, has_actor/2, actors/1, actor_count/1,
publish/3, per-actor accessors, with_actor_projections/3) join the
legacy single-actor accessors, which now read from the first bucket.
Every M1 test continues to pass via bootstrap:start/3 -> new/3 ->
first-bucket lookup.
Local has_keyed/find_keyed/set_keyed/set_bucket helpers cover the
keyed-list ops since lists:keymember/keyfind aren't registered in
this substrate.
next/tests/nx_kernel_multi.sh 17/17. M1 nx_kernel-adjacent suites
green (bootstrap_start 10/10, nx_kernel_server 11/11, http_publish
10/10, smoke_app_pure 12/12, http_post_format 13/13, http_publish_fold
10/10, http_marshal 10/10). Erlang conformance 761/761 preserved.
Blockers entry added for pre-existing http_server_tcp.sh 0/5
regression (78eae9ef left dead helper references in runtime.sx:1593) —
substrate-side, out of m2 scope, confirmed pre-existing by reverting
1a's changes and re-running.
`next/kernel/http_server.erl` gains `start/1(Port)` + `start/2(Port, Cfg)`. Both spawn an Erlang process that hosts
the native `http:listen/2` accept loop with the Cfg-aware `route/2` as the handler.
The blocker — the BIF wrapper in `lib/erlang/runtime.sx` had no dict↔proplist marshaling, so Erlang handler funs
couldn't pattern-match on an opaque SX request dict — is resolved by a new family of helpers added next to `er-of-sx`
(which is left untouched so non-HTTP callers see no behavioural drift):
er-request-dict-to-proplist request dict -> [{method,<<>>},{path,<<>>},...] (atom keys)
er-of-sx-deep recursive marshal: dicts -> binary-keyed proplist
er-dict-to-header-proplist headers: [{<<"content-type">>,<<"text/plain">>},...]
(binary keys keep arbitrary user input out of the atom table)
er-proplist-to-dict response proplist -> SX dict for native serialiser
er-proplist-fill! dict-set! walker over a cons-of-2-tuples
er-to-sx-deep recursive marshal: cons-of-2-tuples -> nested dict
er-proplist-2tuple? predicate distinguishing a header proplist from a binary body
`er-bif-http-listen`'s body is updated to route through the new pair instead of `er-of-sx` / `er-to-sx`. Existing
`http_listen_bif.sh` (Step 8a) still passes — the BIF's external contract (port + handler validation, registration)
hasn't changed, only the request/response shape the handler sees.
This commit also lands a small pre-existing unstaged refactor that was sitting in the same file (er-binary->string
helper above er-bif-http-listen, a "Register everything at load time." comment move, and the binary_to_list /
list_to_binary / er-iolist-walk! defines reshuffled into the er-register-builtin-bifs! body). The refactor was
agreed-out-of-scope earlier in the loop but was unblocked this iteration when the user OK'd progress on 8b-start.
Bundling it here keeps the lib/erlang/runtime.sx diff coherent.
Tests:
- `next/tests/http_marshal.sh` (10 cases) — marshaling unit tests: request dict → cons proplist; method as
<<"GET">> via SX-side proplist walker; path-as-string roundtrip; nested headers reach through binary keys;
response status/body field marshaling; nested headers reconstruct dict; full round-trip preserves status.
- `next/tests/http_server_start.sh` (6 cases) — structural verification: http_server module loaded, start bound
in module env, marshalers defined as lambdas, http:listen BIF registered. Can't invoke spawn in an Erlang test
because the cooperative scheduler (`er-sched-run-all!`) drains every runnable process before returning to the
caller, and the listener's accept loop never exits.
- `next/tests/http_server_tcp.sh` (5 cases) — **first live end-to-end transport test in the milestone**: boots
sx_server in background with FIFO-held stdin (~10s boot for all lib/erlang/*.sx loads + module compile +
Unix.bind), then drives the listener via shell-side curl over real TCP. Verifies GET / → 200, GET
/.well-known/sx-capabilities → 200, GET unknown → 404, POST /activity → 401 with no/bad bearer. Doubles as the
smoke surface for 9a-tcp / 9b-tcp.
Erlang conformance **761/761** unchanged. All standing suites stay green (http_listen_bif 5/5, log_disk 12/12,
log_rotate 10/10, term_codec 18/18).
Step 8b-start ticked in plans/fed-sx-milestone-1.md. Remaining in the milestone: 9a-tcp / 9b-tcp — partly covered
by http_server_tcp.sh's smoke probes; the full curl-driven publish flows are the next iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`next/kernel/log_server.erl` (behaviour gen_server) wraps the pure Step 3c.a `log` substrate behind a per-actor process so
concurrent writers serialise through `gen_server:call` instead of racing on the disk segment writer.
API mirrors the pure log substrate:
start_link(ActorId, BasePath) -> Pid
start_link(ActorId, BasePath, Opts) -> Pid %% Opts forwarded to log:open_disk/3
append(Pid, Activity) -> {ok, Seq}
tip(Pid) -> Seq
entries(Pid) -> [Activity, ...]
replay(Pid, InitAcc, Fun) -> Acc
segments(Pid) -> [SegLen, ...]
stop(Pid) -> ok
Per the port's gen_server convention, `gen_server:start_link/2` returns a raw Pid (not `{ok, Pid}`); the API takes the Pid
directly so multiple per-actor servers coexist without a registered-name collision.
`init/1` dispatches on the Opts arg to call either `log:open_disk/2` (default 1 GiB threshold = effectively no rotation) or
`log:open_disk/3` (opt-in `{segment_size, N}`). `handle_call/3` translates each public op to the corresponding pure log call
and threads the new state through.
New `next/tests/log_server.sh` (15 cases):
- API smoke: start_link returns a Pid, single append+tip+entries round-trip, replay/3 chronological, segments visible
through the wrapper, rotation through wrapper with opt-in `{segment_size, 16}`, stop returns ok.
- Five concurrent-writer tests, each: spawn N=3 writers, each firing M=2 appends of `{I, J}`, parent waits on N `{done,_}`
messages via a Y-combinator-shaped receive loop. Assertions cover (a) tip = N*M, (b) length(entries) = N*M, (c) every
`{I, J}` pair appears exactly once via `lists:all/2` membership (no losses, no dupes), (d) reopening from disk via
`log:open_disk/2` reproduces a byte-equal entries list, (e) every writer's index appears in the entries list
(interleaving witnessed).
Erlang-port gotchas worked around this iteration:
(a) Named recursive fun `fun WaitFn(0) -> ok; WaitFn(K) -> ... end` errors as "fun-ref syntax not yet supported" — rewritten
as `fun (_, 0) -> ok; (Self, K) -> ... Self(Self, K - 1) end` then called as `Wait(Wait, N)`.
(b) `lists:foreach/2` isn't registered (only `lists:map/2`) — use `lists:map/2` and discard the result list when running
side-effecting closures.
(c) gen_server message round-trip in this interpreter is ~2s per call, so concurrent N*M was tuned to 6 (`N=3, M=2`) to
keep the whole 15-test suite under 60s wall clock; the test's correctness assertions don't depend on N*M magnitude.
Erlang conformance **761/761** unchanged (log_server.erl is in next/, not lib/erlang/). Step 3c (both .a and .b) now
fully ticked in plans/fed-sx-milestone-1.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`next/kernel/log.erl` rewritten around a `seg_lens :: [N0, N1, ...]` per-segment entry-count list + a `seg_size` byte threshold. Filename
scheme moved from `<ActorId>.log` to `<ActorId>-NNNNNN.log` (6-digit zero-padded) so `file:list_dir`'s alphabetical sort coincides
with numeric order.
`open_disk/3(ActorId, BasePath, [{segment_size, N}])` opts a caller into a smaller rotation threshold; `open_disk/2` keeps a 1 GiB
default that effectively never rotates (preserves Step 3b acceptance — log_disk.sh unchanged in behaviour).
Rotation rule in `place_append/4`: if the active segment's pre-append encoded size is already >= threshold AND it holds at least one
entry, the new activity opens a fresh segment; otherwise it extends the current active segment. A single huge entry that exceeds
the threshold stays alone — never rotated recursively.
On reopen, `load_all_segments` lists the dir, filters `<ActorId>-NNNNNN.log`, sorts numerically (insertion sort — `lists:sort/1`
isn't registered in this port, only `lists:append/2`/`lists:reverse/1`/`lists:filter/2`/etc.), reads each via `try_read_segment`,
and concatenates the entries to rebuild flat `entries` + `seg_lens`.
Erlang-port gotchas worked around during this iteration:
(a) String literals like `"foo"` in this port are NOT charlists — `[H|T] = "foo"` badmatches and `length("foo")` errors as "not a
proper list". `parse_segment_name` builds prefix/suffix from `atom_to_list/1` + explicit `[$-]` / `[$., $l, $o, $g]` cons.
(b) Cross-arg variable repetition (`strip_prefix([C | Rest], [C | PRest])`) was rewritten to explicit `case C =:= P` for robustness.
(c) `Pattern = Binding` syntax in a case clause (`[_|_] = Lst when length(Lst) > 1 -> ...`) errors as "unsupported pattern type
'match'" — replaced with `Lst when is_list(Lst), length(Lst) > 1`.
Tests:
- new `next/tests/log_rotate.sh` (10 cases): no-opt single-seg-after-3, rotation-fires-on-threshold, rotated-chronological,
reopen-rebuilds-history, reopen-rebuilds-same-seg-shape, huge-single-entry-stays-1-seg, append-after-huge-keeps-order,
tip-monotonic-across-rotations.
- `next/tests/log_disk.sh` updated to the new filename (`corrupted-000000.log`); stays 12/12.
- Erlang conformance 761/761 unchanged (log.erl is in next/, not lib/erlang/).
3c.a ticked in plans/fed-sx-milestone-1.md; 3c.b (gen_server-mediated concurrent appends) is the next iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
