Closes Step 10 (10a discovery + 10b webfinger + 10c fetch). New
next/kernel/discovery_fetch.erl produces a 1-arity FetchFn closure
suitable for peer_actors:lookup_or_fetch_srv/2, completing the
discovery half that Step 5c's peer_actors cache stubbed out.
discovery_fetch API:
make_fetch_fn(Cfg) -> fun((PeerId) -> {ok, AS} | {error, _})
fetch(Url, Cfg) -> {ok, AS} | {error, _}
actor_doc_url(BaseUrl, PeerAtom) -> <Base>/actors/<peer>
accept_header/0 -> <<"application/vnd.fed-sx.actor-doc">>
decode_body(Body) -> {ok, AS} | {error, bad_actor_doc}
Closure GETs <base>/actors/<peer> via the Step 8e BIF with
Accept = application/vnd.fed-sx.actor-doc, decodes the response
body via term_codec:decode/1, returns the peer-actor-state
proplist (currently [{public_keys, [...]}]) in the shape
envelope:verify_signature consumes.
Cfg reuses dispatch_http's :peer_url / :peer_url_fn resolution so
a single Cfg threads through both delivery (8f) and discovery (10c).
Server side: http_server.erl extended to serve the same MIME.
- accept_format/1 matches application/vnd.fed-sx.actor-doc first
via the new actor_doc_prefix/0 — content negotiation atom is
`actor_doc`.
- content_type_for(actor_doc) emits the MIME on outbound.
- actor_doc_response_for/3 kernel-aware arm: with kernel + actor
-> 200 + term_codec:encode of nx_kernel:state_for/1 result.
Unknown actor -> not_found_response/0. Other formats fall
through to the existing /2 stub variants.
- actor_get/3 route dispatch threads Cfg to the /3 arm.
Port quirks documented:
* This Erlang doesn't support Mod:Fun(X) dispatch on a variable
module — kernel_actor_state/2 hardcodes nx_kernel; the Cfg
:kernel field is just a "no kernel wired" -> nil flag.
* nx_kernel:actor_state/1 is the LEGACY single-bucket accessor
that takes State (not ActorId); the server-side variant we
want is state_for/1 (gen_server:call wrapper). Easy mismatch,
documented in the comment.
Outcome mapping:
2xx + decodable body -> {ok, AS}
2xx + bad body -> {error, bad_actor_doc}
non-2xx -> {error, {status, N}}
resolver miss -> {error, no_peer_url}
transport -> {error, Reason} (BIF re-raises)
Test: next/tests/discovery_fetch.sh 11/11
Server side (in-process via http_server:actor_doc_response_for):
- Accept negotiation
- kernel + actor -> 200 + decodable body w/ :public_keys
- unknown actor -> 404
Closure side (live HTTP against background python stub returning
hand-crafted term_codec bytes):
- URL construction <base>/actors/X
- fetch live -> {ok, AS}
- make_fetch_fn closure -> {ok, AS} via static :peer_url map
- missing peer -> {error, no_peer_url}
- 404 path -> {error, {status, 404}}
- peer_actors:lookup_or_fetch/3 caches the result
Test setup note: Python term_codec encoder uses ELEMENT COUNT
(not byte length) for l/t headers — see encode/1 in term_codec.erl
which does integer_to_list(length(T)). Easy bug, documented in the
test's python source.
No-regression gates green: Erlang conformance 761/761,
httpc_request 10/10, dispatch_http 10/10, http_listen_bif 5/5,
peer_actors 19/19, discovery 12/12, http_accept 13/13,
http_actors 13/13.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
actor_outbox_response_for/3 in http_server.erl now reads ?since=
from the query string before paging:
Q = field(request_query, Cfg),
Filtered = case parse_since(Q) of
nil -> Entries;
SinceCid -> backfill:since_cid_entries(SinceCid, Entries)
end,
Slice = page_slice(Filtered, Page),
...
New helpers:
parse_since/1 — scan query for since=<Cid>, value is the
binary up to next & or end-of-binary. nil
when absent.
scan_param/2,3 — generic 'find Name=Value anywhere in &-sep
query'. Used for since= today; could be
factored over parse_page=.
skip_to_amp/1 — walk past the next & for the iteration step.
Order-independent: ?since=X&page=2 and ?page=2&since=X both
work. Unknown cid -> backfill:since_cid_entries returns []
-> empty page -> body degrades to tip-only shape (Step 4d
back-compat).
Three new cases in http_multi_actor.sh (44/44 total):
- ?since=<first cid> filters out the first publish, leaving
2 of 3 items in the paged response
- ?since=<unknown cid> -> empty page; body has tip but no
item: lines (tip-only degrade)
- ?since=<cid> + ?page=1 combined — pagination still applies
to the filtered list
Latent issue surfaced + fixed in passing: http_multi_actor.sh
was missing follower_graph + delivery + backfill module loads
(outbox has depended on follower_graph + delivery since Step 7c
and now backfill from 9a). Added all three with epoch 100/101/
102 to match the c6b49200 fix-up pattern. 41 existing tests now
also exercise the live path through outbox:publish without
crashing on missing module deps.
GET /.well-known/webfinger?resource=acct:user@host lands in
http_server.erl next to the existing /.well-known/sx-capabilities
arm.
Dispatch chain:
route/2 -> dispatch/4 (matches webfinger path) -> handle_webfinger/1
-> webfinger_for_query/2
-> parse_resource_param/1 (matches "resource=" + collect via
take_until_amp/1)
-> discovery:parse_acct/1
-> webfinger_lookup/3 — host check + kernel actor lookup
-> 200 + discovery:webfinger_body/3 (application/activity+json)
-> 404 on any miss
Cfg surface:
{webfinger_host, Binary} optional; when set the acct's @host
must match exactly. Missing -> any.
{kernel, Atom} optional; when set, the user must be
a known actor in the registered kernel.
Missing -> every user is 'known' (pure
route tests).
route/2 already threads the Req's :query into Cfg as
:request_query (Step 4d), so the handler doesn't need to take
the Req directly.
10/10 in next/tests/webfinger_route.sh:
- GET happy path (no kernel cfg'd) -> 200
- body has subject prefix
- body has href substring
- missing ?resource= -> 404
- garbage 'resource=garbage' -> 404
- kernel cfg: alice 200, ghost 404
- :webfinger_host matches @host -> 200
- :webfinger_host mismatch -> 404
- POST -> 404 (only GET handled)
discovery.sh 12/12 unchanged, http_route.sh 11/11 unchanged.
Per design §13.2 the v2 Follow policy is open-world: every
successfully-ingested Follow triggers an Accept publish from the
target actor. Enabled per-Cfg via {auto_accept_follows, true} so
manual-moderation deployments can leave it off; default off.
http_server.erl run_inbox_pipeline gained maybe_auto_accept/3:
maybe_auto_accept(TargetAtom, Activity, Cfg) ->
case field(auto_accept_follows, Cfg) of
true ->
case envelope:get_field(type, Activity) of
{ok, follow} ->
Req = [{type, accept}, {object, Activity}],
nx_kernel:publish_to(TargetAtom, Req);
_ -> ok
end;
_ -> ok
end.
The publish routes through the full outbox pipeline (envelope
construct + HMAC sign + log append + outbox projection broadcast).
When the target's outbox :projections list shares the same
follower_graph projection that inbox broadcasts into, the bilateral
relationship fold-converges automatically — alice.followers = [bob]
and bob.following = [alice], both pending lists clear. No extra
test scaffolding needed because outbox:publish already runs the
broadcast hook from Step 7c.
Bad-sig and non-Follow ingestion short-circuit before the Accept
attempt (the validation pipeline rejects before run_inbox_pipeline's
ok branch fires).
9/9 in next/tests/auto_accept.sh:
- auto_accept on: alice's outbox tip advances to 1
- alice's outbox entry has :type = accept
- follower_graph converges to {alice.followers=[bob],
bob.following=[alice]}
- both sides' pending lists clear after the Accept fold
- auto_accept off (default): outbox stays empty; pending_inbound
still gets populated from the Step 6b inbox-projection path,
but alice.followers stays empty until human moderation acts
- non-Follow ingestion (Create{Note}) with auto_accept on: no
Accept published
- bad-sig Follow with auto_accept on: no Accept (sig short-circuit
in pipeline before maybe_auto_accept runs)
Step 6 fully closed (6a follower_graph projection, 6b inbox -> projection
broadcast wiring, 6c auto-Accept publish).
Conformance 761/761. 89/89 across 7 Step-6-adjacent suites
(inbox, inbox_peer_resolution, follower_graph, follow_lifecycle,
auto_accept, http_publish, nx_kernel_multi).
http_server.erl run_inbox_pipeline now calls
broadcast_to_inbox_projections/2 after a successful
nx_kernel:append_inbox. Cfg may carry {inbox_projections,
[Name, ...]} listing projection gen_servers that should see every
successfully-ingested inbound activity. Each gets the activity via
projection:async_fold/2 — fire-and-forget so the inbox handler
doesn't block on fold processing. Empty / absent
:inbox_projections is a no-op (back-compat with Step 5d callers).
v2 leaves the routing field global (every inbound activity goes
to every named projection); per-actor projection wiring is a
forward-looking follow-up.
9/9 in next/tests/follow_lifecycle.sh:
- Follow ingestion -> 202
- follower_graph state: alice.pending_inbound = [bob]
- follower_graph state: bob.pending_outbound = [alice]
- inbox tip advances to 1 (Step 5a invariant preserved)
- no inbox_projections Cfg -> projection state stays empty
- end-to-end: Follow + Accept fold converges to
alice.followers = [bob] and bob.following = [alice]
(Accept fed via projection:async_fold for v2 — auto-Accept
publish is Step 6c)
- bad-sig inbound short-circuits before broadcast
- two distinct peer Follows accumulate
bootstrap_start.sh internal sx_server timeout bumped 300s -> 600s
to match the cumulative cost trend other tests are seeing on this
port. (bootstrap_start doesn't load http_server but loads bootstrap
+ the full genesis bundle + 9 kernel modules — same cumulative
compile budget.)
Conformance 761/761.
POST /actors/<id>/inbox is now special-cased in route/2 (next to
POST /activity) so the body + Cfg reach the new handle_inbox_post/3
handler.
Wire format: body = term_codec:encode(SignedActivity); the receiver
decodes into the activity proplist and runs the chain.
handle_inbox_post/3 orchestration:
1. kernel_has_actor(field(kernel, Cfg), TargetId) -> 404 if missing
2. decode_activity(Body) -> 422 on bad shape
3. envelope:get_field(actor, Activity) -> 422 if no peer id
4. resolve_peer_as(PeerId, Cfg) -> 401 if unknown
5. nx_kernel:inbox_state_for(TargetAtom) -> 404 belt-and-braces
6. pipeline:validate_inbound(Activity, PeerAS, InboxLog)
ok -> nx_kernel:append_inbox + 202
{error, bad_signature} -> 401
{error, no_signature} -> 401
{error, _} -> 422
resolve_peer_as/2 supports three Cfg paths in priority order:
{peer_as, [{PeerId, AS}, ...]} pure-fn pre-populated map
{peer_actors, AtomName} peer_actors gen_server cache
{peer_fetch_fn, fun/1} fallback on srv cache miss
Empty Cfg returns {error, no_peer_resolver} -> 401.
v1 actor_post/1 4a stub deleted; M1 actor_inbox_post_response/0
kept for response composition.
Projection broadcast on inbox success intentionally deferred to a
follow-up sub-deliverable.
inbox.sh 11/11 (acceptance suite for the basic chain):
- happy path -> 202
- inbox tip advances; outbox tip unchanged (per-actor bucket
independence carried through from Step 5a)
- empty / garbage body -> 422
- unknown peer -> 401
- bad peer-AS keys -> 401
- replay (same activity twice) -> 422 on second
- unknown target actor -> 404
- two distinct activities -> tip = 2
inbox_peer_resolution.sh 6/6 (Cfg resolution variants):
- peer_actors gen_server hit -> 202
- FetchFn fallback -> 202
- FetchFn error -> 401
- FetchFn caches into peer_actors (peers_srv shows [bob] after)
- No resolver -> 401
Tests split into two files because each epoch's kernel start_link
+ outbox construct + term_codec encode is expensive and a single
suite hits the wall-clock budget.
http_server.erl is now 1181 lines. erlang-load-module on this port
scales superlinearly with function count, so eight http_*.sh tests'
internal sx_server timeout bumped 60s -> 360s (http_route,
http_actors, http_accept, http_capabilities, http_capabilities_format,
http_content_type, http_artifacts, http_projections).
Conformance 761/761.
Per-actor GET /actors/<id>/outbox now reads the bucket's log via
new nx_kernel:log_state_for/1 gen_server export and renders the
paged CID list.
nx_kernel additions:
log_state_for/1 gen_server call returning {ok, LogState} for
the named actor (mirrors log_tip_for/1's shape).
http_server additions:
- with_request_query/2 bakes Req's :query binary into Cfg as
{request_query, Q} so sub-resource handlers can parse params
without taking the Req as another arg
- kernel_actor_log_data/2 -> {Tip, Entries} via
nx_kernel:log_tip_for + log_state_for + log:entries
- parse_page/1 reads ?page=N (default 1, non-digits -> 1)
- page_size/0 returns 5 (test-friendly; production picks 20+)
- page_slice/2 + drop_take/3 + take/2 for the page extraction
- entry_cids/1 maps entries to :id CID binaries via envelope
- actor_outbox_full_response_for/5 renders text / JSON / SX:
text: outbox: <id>\ntip: N\npage: P\nitem: <cid>\n...
json: {"outbox":"<id>","tip":N,"page":P,"items":[...]}
sx: (outbox "<id>" :tip N :page P :items (...))
Empty page degrades to actor_outbox_with_tip_response_for so
epochs 50-57 from Step 4c still pass — the prefix is preserved.
8 new cases in next/tests/http_multi_actor.sh (41/41 total):
- 1 publish -> body contains outbox/tip=1/page=1/item: prefix
- 3 publishes -> body contains tip=3/page=1/item: prefix
- page=2 with 3 items -> empty page degrades to tip-only body
- 6 publishes page=1 -> tip=6/page=1/item: prefix
- 6 publishes page=2 -> tip=6/page=2/item: prefix
- JSON body shape with items array (1 entry)
- SX body shape with :items list (1 entry)
- bad ?page=bad falls back to page 1
Conformance 761/761. 117/117 across 11 Step-4-adjacent suites
(http_multi_actor, http_route, http_publish, http_post_format,
http_marshal, http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, nx_kernel_server, bootstrap_start, actor_lifecycle).
Substrate gotcha logged: named recursive funs fun F(...) -> F(...)
end aren't supported by the parser ('fun-ref syntax not yet
supported'); binary:matches/2 and lists:foreach/2 aren't registered.
Tests prove behaviour via match_prefix substring checks rather than
counting occurrences.
http_server:route/3(Req, Cfg, Kernel) is the new extended entry
point: folds the kernel reference (typically the registered
nx_kernel atom) into Cfg as {kernel, Kernel}. route/2 is
unchanged and stays the M1 surface.
The dispatch chain gained Cfg threading all the way down:
dispatch/3 -> dispatch/4 (M, P, F, Cfg)
actor_get/2 -> actor_get/3 (Rest, F, Cfg)
actor_subresource_get/3 -> /4 (Id, Sub, F, Cfg)
actor_outbox_response_for/3 (new) reads :kernel from Cfg and,
when the kernel atom is registered AND the actor exists, renders
'tip: <N>' alongside the actor id in text / JSON / SX content-
negotiated bodies. Unknown actors or unregistered kernels fall
back to the 4a stub.
Inbox / followers / following handlers accept Cfg but ignore it
for now — they layer real state lookup in 4d/4e/Step 5+.
Substrate gotcha logged in the Progress log: try/of/catch around
gen_server:call(nx_kernel, _) deadlocks in this port's scheduler
(probably the catch frame's mask defers reply delivery). The
live kernel_log_tip/2 helper does a bare call + integer guard
instead. nx_kernel_multi.sh already proves bare gen_server:call
into the same kernel works correctly.
8 new cases in next/tests/http_multi_actor.sh (33/33 total):
- route/3 with registered kernel: outbox body includes tip=0
- tip advances after POST publish through route/3 + token map
- unknown actor (ghost) falls back to 4a stub (no tip:)
- unregistered kernel ref falls back to stub
- JSON Accept renders {"outbox":"alice","tip":0}
- SX Accept renders (outbox "alice" :tip 0)
- Bob's outbox tip stays 0 while Alice publishes (per-actor)
- route/2 path unchanged: no tip field in body
Conformance 761/761. 121/121 across 10 Step-4-adjacent suites
(http_multi_actor, http_route, http_publish, http_post_format,
http_marshal, http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, bootstrap_start, actor_lifecycle).
POST /activity now routes through nx_kernel:publish_to/2 when the
bearer token resolves to an explicit ActorId via Cfg's :tokens
proplist:
Cfg = [{tokens, [{<<"alice-token">>, alice},
{<<"bob-token">>, bob}]}]
resolve_token/2 returns {ok, ActorId} on a :tokens hit. On a miss
it falls back to the M1 :publish_token single-token field — match
returns {ok, legacy}, routing through nx_kernel:publish/1 (which
fans out to bucket 0) so every M1 test continues to pass.
handle_post_activity threads the resolved ActorRef to
publish_if_kernel/3 which dispatches publish_to/2 for explicit
actor ids and publish/1 for the legacy atom. The no-kernel
auth-only path (which preserves the post_activity_response_for stub
for unit-style tests of http_server alone) is unchanged.
Dead expected_token/1 helper removed (was only called by the old
check_bearer arm that resolve_token replaces).
8 new cases in next/tests/http_multi_actor.sh (25/25 total):
- two-actor Cfg, Alice token -> 200 with cid:
- Alice token publishes to alice (log_tip alice=1, bob=0)
- Bob token publishes to bob (log_tip alice=0, bob=1)
- interleaved Alice + Bob + Alice -> {2, 1}
- unknown token + no :publish_token -> 401
- legacy :publish_token still works (M1 back-compat)
- tokens map AND legacy :publish_token coexist (each resolves to
its own actor; legacy lands on alice bucket via publish/1)
- no kernel + valid :tokens entry -> auth-only stub 200
Conformance 761/761. 116/116 across 10 Step-4-adjacent suites
(http_multi_actor, http_route, http_publish, http_post_format,
http_marshal, http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, bootstrap_start, actor_lifecycle).
Per design §16.1 each actor has /outbox /inbox /followers /following
sub-paths. New split_first_slash/1 helper lets the GET /actors/...
dispatch arm fan out on the sub-segment:
GET /actors/<id> actor doc (M1 — unchanged)
GET /actors/<id>/outbox outbox stub (4a)
GET /actors/<id>/inbox inbox stub (4a)
GET /actors/<id>/followers follower stub (4a)
GET /actors/<id>/following following stub (4a)
POST /actors/<id>/inbox 202 Accepted stub (4a; Step 5 real)
Four new content-negotiated response functions mirror the existing
actor_doc_response_for/2 shape (text / json / activity_json / sx
variants):
actor_outbox_response_for/2
actor_inbox_get_response_for/2
actor_followers_response_for/2
actor_following_response_for/2
POST returns 202 via new accepted_response/1 +
actor_inbox_post_response/0.
Unknown sub-paths under /actors/<id>/ return 404. Bare /actors/<id>
preserves the M1 actor-doc arm so http_route + http_post_format
regression suites stay green.
4b-4e (token map, route/3 kernel access, per-actor outbox listing
from log entries, real inbox pipeline) layer on top of this dispatch
in subsequent iterations.
17/17 in next/tests/http_multi_actor.sh covering:
- split_first_slash sanity (no slash / id+sub / trailing slash)
- all four GET sub-paths return 200 with stub bodies
- POST inbox returns 202 + 'accepted'
- unknown sub-paths return 404 (GET and POST)
- empty /actors/ returns 404
- body carries the actor id
- content negotiation: outbox JSON, inbox SX, followers JSON
Conformance 761/761. 120/120 across 10 Step-4-adjacent suites
(http_route, http_publish, http_post_format, http_marshal,
http_publish_fold, http_listen_bif, http_server_start,
nx_kernel_multi, actor_state_pure, bootstrap_start).
`next/kernel/http_server.erl` gains `start/1(Port)` + `start/2(Port, Cfg)`. Both spawn an Erlang process that hosts
the native `http:listen/2` accept loop with the Cfg-aware `route/2` as the handler.
The blocker — the BIF wrapper in `lib/erlang/runtime.sx` had no dict↔proplist marshaling, so Erlang handler funs
couldn't pattern-match on an opaque SX request dict — is resolved by a new family of helpers added next to `er-of-sx`
(which is left untouched so non-HTTP callers see no behavioural drift):
er-request-dict-to-proplist request dict -> [{method,<<>>},{path,<<>>},...] (atom keys)
er-of-sx-deep recursive marshal: dicts -> binary-keyed proplist
er-dict-to-header-proplist headers: [{<<"content-type">>,<<"text/plain">>},...]
(binary keys keep arbitrary user input out of the atom table)
er-proplist-to-dict response proplist -> SX dict for native serialiser
er-proplist-fill! dict-set! walker over a cons-of-2-tuples
er-to-sx-deep recursive marshal: cons-of-2-tuples -> nested dict
er-proplist-2tuple? predicate distinguishing a header proplist from a binary body
`er-bif-http-listen`'s body is updated to route through the new pair instead of `er-of-sx` / `er-to-sx`. Existing
`http_listen_bif.sh` (Step 8a) still passes — the BIF's external contract (port + handler validation, registration)
hasn't changed, only the request/response shape the handler sees.
This commit also lands a small pre-existing unstaged refactor that was sitting in the same file (er-binary->string
helper above er-bif-http-listen, a "Register everything at load time." comment move, and the binary_to_list /
list_to_binary / er-iolist-walk! defines reshuffled into the er-register-builtin-bifs! body). The refactor was
agreed-out-of-scope earlier in the loop but was unblocked this iteration when the user OK'd progress on 8b-start.
Bundling it here keeps the lib/erlang/runtime.sx diff coherent.
Tests:
- `next/tests/http_marshal.sh` (10 cases) — marshaling unit tests: request dict → cons proplist; method as
<<"GET">> via SX-side proplist walker; path-as-string roundtrip; nested headers reach through binary keys;
response status/body field marshaling; nested headers reconstruct dict; full round-trip preserves status.
- `next/tests/http_server_start.sh` (6 cases) — structural verification: http_server module loaded, start bound
in module env, marshalers defined as lambdas, http:listen BIF registered. Can't invoke spawn in an Erlang test
because the cooperative scheduler (`er-sched-run-all!`) drains every runnable process before returning to the
caller, and the listener's accept loop never exits.
- `next/tests/http_server_tcp.sh` (5 cases) — **first live end-to-end transport test in the milestone**: boots
sx_server in background with FIFO-held stdin (~10s boot for all lib/erlang/*.sx loads + module compile +
Unix.bind), then drives the listener via shell-side curl over real TCP. Verifies GET / → 200, GET
/.well-known/sx-capabilities → 200, GET unknown → 404, POST /activity → 401 with no/bad bearer. Doubles as the
smoke surface for 9a-tcp / 9b-tcp.
Erlang conformance **761/761** unchanged. All standing suites stay green (http_listen_bif 5/5, log_disk 12/12,
log_rotate 10/10, term_codec 18/18).
Step 8b-start ticked in plans/fed-sx-milestone-1.md. Remaining in the milestone: 9a-tcp / 9b-tcp — partly covered
by http_server_tcp.sh's smoke probes; the full curl-driven publish flows are the next iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
