giles
bd2c61367d
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 37s
Closes the BIF half of Step 8. Native http-request primitive landed in architecture via the fed-prims merge (the m2 plan's Blocker #2), so the briefing-allowed-exception wrapper in lib/erlang/runtime.sx can finally be wired. Marshalling at the BIF boundary: Url : Erlang binary -> SX string (byte-list -> integer->char). Method : Erlang atom upcased ('get -> "GET") for HTTP-wire convention, or Erlang binary passes through verbatim. Headers : Erlang proplist -> SX dict via er-proplist-to-dict. Body : Erlang binary -> SX string. Result {:status :headers :body} marshalled back to Erlang {ok, Status::integer, Headers::proplist (binary-keyed via er-of-sx-deep), Body::binary (char->integer over the SX string)}. Bad arg shapes (non-binary URL or body) raise error:badarg; native DNS / connect / bad-URL failures surface as Erlang error markers that the caller can catch. Test: next/tests/httpc_request.sh 10/10 - registration under httpc/request/4 - BIF marked non-pure - wrong-arity (/1) absent from registry - badarg on non-binary URL - badarg on non-binary body - live GET against `python3 -m http.server` -> Status 200 - body bytes match "hello from python\n" - headers come back as proplist (is_list/1 = true) - 404 path -> {ok, 404, ...} (not an error tuple) - method passed as binary works URLs spelled out as byte-list <<104,116,116,p,...>> binaries since the parser truncates <<"..."> string-literal binaries (same workaround backfill_drain.sh uses for inbox paths). Plan: 8e ticked; Blocker #2 marked RESOLVED with the merge that unblocked it referenced. Step 8f (live HTTP dispatch through delivery_worker) and Step 10c (peer-actor doc fetch) are now unblocked. No-regression gates green: Erlang conformance 761/761, http_multi_actor 44/44, follower_graph 18/18, follow_lifecycle 9/9, backfill 20/20, backfill_drain 6/6, http_listen_bif 5/5. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1544 lines
78 KiB
Markdown
1544 lines
78 KiB
Markdown
# fed-sx Milestone 2 — Multi-actor + Federation
|
|
|
|
Real federation between two fed-sx instances. Per-actor state, signed
|
|
inbox delivery, Follow lifecycle, audience-resolving outbound queue, and
|
|
the rich verbs (Note, Announce, Endorse) needed for federated propagation.
|
|
Reference: `plans/fed-sx-design.md` (especially §9 identity, §13 federation,
|
|
§16 HTTP endpoints). Builds on Milestone 1 (see `plans/fed-sx-milestone-1.md`).
|
|
|
|
## Goal
|
|
|
|
Two cooperating fed-sx instances `A` and `B`, each hosting one or more
|
|
actors, can:
|
|
|
|
1. **Discover** each other's actors via webfinger + actor docs.
|
|
2. **Follow** across instances (`Follow` → `Accept` → state).
|
|
3. **Publish** a `Note` on `B` and have it land in every follower's
|
|
`actor-state` projection on `A` via signed inbox delivery.
|
|
4. **Announce** a peer's activity, propagating it to followers of the
|
|
announcer.
|
|
5. **Rotate keys** on either side without breaking historical sig
|
|
verification (per §9.6).
|
|
|
|
Acceptance: the §11 smoke test (`smoke_federate.sh`) drives all of the
|
|
above against two locally-running kernel instances on distinct ports, no
|
|
human-in-the-loop, and exits 0.
|
|
|
|
## Non-goals (what milestone 2 deliberately does NOT do)
|
|
|
|
- **Real WAN federation.** Both instances run on `localhost:PortA` and
|
|
`localhost:PortB`. Cross-instance HTTP is unencrypted plaintext.
|
|
TLS, NAT traversal, and signed HTTP-message headers (per RFC 9421)
|
|
are v3.
|
|
- **ActivityPub Mastodon interop.** No HTTP-signatures-2018 compat layer,
|
|
no Linked-Data-Signatures, no JSON-LD canonicalisation. Cross-fed-sx
|
|
only.
|
|
- **IPFS / S3 storage backends.** Still local files only.
|
|
- **Browser client + operator dashboard.** Curl-shaped API only.
|
|
- **Capability tokens / delegation.** Multi-actor means multi-user, not
|
|
multi-device for a single actor. Capability tokens (per §9.5) defer.
|
|
- **Cross-host conformance.** Only OCaml/Erlang-on-SX host runs fed-sx
|
|
in v2.
|
|
- **Performance work.** Functional correctness first.
|
|
- **Spam/abuse infrastructure.** Per §13.6 the layers are designed; v2
|
|
implements signature verification + replay defense; reputation,
|
|
rate-limiting, instance allowlists / blocklists are v3.
|
|
- **Operator quarantine UX.** Logs only.
|
|
|
|
## Architecture summary
|
|
|
|
```
|
|
Instance A Instance B
|
|
(port 9999) (port 9998)
|
|
|
|
Outbox ┌─────────────────┐ ┌─────────────────┐
|
|
────────▶ │ HTTP server │ │ HTTP server │
|
|
│ POST /activity │ │ POST /activity │
|
|
│ POST /inbox │ │ POST /inbox │
|
|
│ GET /actors/.. │ │ GET /actors/.. │
|
|
│ GET /.well- │ │ GET /.well- │
|
|
│ known/* │ │ known/* │
|
|
└────────┬─────────┘ └────────┬─────────┘
|
|
│ │
|
|
┌────────▼─────────┐ ┌────────▼─────────┐
|
|
│ nx_kernel │ ◀ HTTPS ▶ │ nx_kernel │
|
|
│ multi-actor │ deliveries │ multi-actor │
|
|
│ bucket map │ (signed) │ bucket map │
|
|
│ ActorA -> {…} │ │ ActorB -> {…} │
|
|
│ ActorC -> {…} │ │ │
|
|
└────────┬─────────┘ └────────┬─────────┘
|
|
│ │
|
|
┌────────▼─────────┐ ┌────────▼─────────┐
|
|
│ Delivery queue │ │ Delivery queue │
|
|
│ (one worker per │ │ (one worker per │
|
|
│ peer instance) │ │ peer instance) │
|
|
└──────────────────┘ └──────────────────┘
|
|
│
|
|
│ HTTP POST /inbox to peer
|
|
▼
|
|
(peer instance)
|
|
```
|
|
|
|
The federation transport is plain HTTP POST of canonical-bytes-signed
|
|
activities to each follower's actor inbox. Delivery is push (§13.1); pull
|
|
+ relay deferred to v3.
|
|
|
|
## Build order
|
|
|
|
Twelve steps in dependency order.
|
|
|
|
| Step | Title | Depends on |
|
|
|------|----------------------------------------------------|-----------------------|
|
|
| **1** | Per-actor state buckets in nx_kernel | M1 closeout |
|
|
| **2** | Actor lifecycle activities (Person/Service/Group) | Step 1 |
|
|
| **3** | Key rotation via Update + actor-state projection | Steps 2, M1 §9.6 |
|
|
| **4** | Multi-actor HTTP routing (per-actor outbox/inbox) | Steps 1, M1 8b-start |
|
|
| **5** | POST /inbox: peer signature verify + ingestion | Steps 3, 4 |
|
|
| **6** | Follow lifecycle (Follow / Accept / Reject / Undo) | Step 5 |
|
|
| **7** | Audience-resolving delivery set computation | Step 6 |
|
|
| **8** | Outbound delivery queue + retry/backoff | Step 7 |
|
|
| **9** | Backfill modes on Follow accept | Steps 6, 8 |
|
|
| **10** | Discovery: webfinger + actor doc fetch | Step 4 |
|
|
| **11** | Rich verbs as runtime artifacts (Note, Announce, Endorse) | Step 8 |
|
|
| **12** | Two-instance smoke test (`smoke_federate.sh`) | Steps 1-11 |
|
|
|
|
Steps 1-3 are the multi-actor foundation. Steps 4-10 are the federation
|
|
core. Steps 11-12 close the proof points.
|
|
|
|
---
|
|
|
|
## Step 1 — Per-actor state buckets
|
|
|
|
Today `nx_kernel` holds one actor's state at the top of its property list.
|
|
Make it bucketed by ActorId so a single kernel can host any number of
|
|
actors.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **1a** — Pure-functional bucket APIs. State shape becomes
|
|
`[{actors, [{ActorId, ActorBucket}, ...]}, {next_actor_seq, N}]`
|
|
with `ActorBucket = [{key_spec, KS}, {actor_state, AS}, {log, L},
|
|
{projections, [Name]}, {next_published, N}]`. New exports: `new/0`,
|
|
`add_actor/4`, `has_actor/2`, `actors/1`, `actor_count/1`,
|
|
`next_actor_seq/1`, `actor_bucket/2`, `publish/3`, per-actor
|
|
accessors (`actor_log_state/2`, `actor_log_tip/2`, `actor_key_spec/2`,
|
|
`actor_state/2`, `actor_projections/2`, `actor_next_published/2`),
|
|
`with_actor_projections/3`. Legacy single-actor accessors
|
|
(`actor_id/1`, `key_spec/1`, `actor_state/1`, `log_state/1`,
|
|
`log_tip/1`, `projections/1`, `next_published/1`,
|
|
`with_projections/2`, legacy `publish/2`) continue to read from the
|
|
first bucket — every M1 test passes via `bootstrap:start/3` →
|
|
`new/3` → first-bucket lookup. `lists:keymember`/`keyfind` not in
|
|
the substrate; local `has_keyed`/`find_keyed`/`set_keyed`/
|
|
`set_bucket` helpers handle the keyed-list ops.
|
|
`next/tests/nx_kernel_multi.sh` 17/17.
|
|
- [x] **1b** — Multi-actor gen_server. `start_link/3` still seeds
|
|
bucket 0; new exports `add_actor/3`, `publish_to/2(ActorId,
|
|
Request)`, `log_tip_for/1`, `actors/0`, `state_for/1`,
|
|
`bucket_for/1`, `with_projections_for/2` delegate to the pure-
|
|
functional bucket APIs via fresh `handle_call` branches. Existing
|
|
`publish/1`/`log_tip/0`/`with_projections/1` route through bucket
|
|
0 unchanged. Per-actor mailbox concurrency (one gen_server per
|
|
bucket so distinct-actor publishes don't serialise) is forward-
|
|
looking — deferred to Step 4 (multi-actor HTTP routing) where it
|
|
actually pays off. `nx_kernel_multi.sh` extended with 9 gen_server
|
|
cases (26 total), every M1 nx_kernel-adjacent + http suite still
|
|
green (134 / 134 across 12 suites).
|
|
|
|
**Acceptance:** `bash next/tests/nx_kernel_multi.sh` passes 12+ cases.
|
|
|
|
---
|
|
|
|
## Step 2 — Actor lifecycle activities
|
|
|
|
Per design §9.1, an actor is a Person, Service, or Group object,
|
|
created by `Create{Person{...}}`. The kernel needs to fold this into
|
|
an actor-state projection that downstream code can read for keys,
|
|
publicKey rotation history, profile fields, follower counts, etc.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **2a** — Genesis additions: `DefineObject{Person}` /
|
|
`DefineObject{Service}` / `DefineObject{Group}` — three new SX
|
|
files in `next/genesis/object-types/` plus manifest entries (now
|
|
13 object-types total, 34 total genesis entries). Each defines
|
|
`:name`, `:doc`, `:schema (fn (obj) (string? (-> obj :name)))`.
|
|
`next/tests/genesis_parse.sh` extended +7 cases (head form +
|
|
:name + manifest membership), now 57/57. Bootstrap suite
|
|
count assertions bumped (`bootstrap_read.sh` 15/15,
|
|
`bootstrap_load.sh` 15/15, `bootstrap_populate.sh` 14/14,
|
|
`bootstrap_start.sh` 10/10). `bootstrap_build.sh` 12/12 picks
|
|
up the new bundle CID dynamically.
|
|
- [x] **2b** — Actor-state projection fold (Erlang-fun stand-in,
|
|
mirrors Step 5d-pure's `define_registry`). `next/kernel/actor_state.erl`
|
|
with state shape `[{ActorId, Profile}, ...]` where `Profile` is a
|
|
proplist with `:type / :name / :preferredUsername / :summary /
|
|
:icon / :public_keys / :moved_to / :created`. Maps `#{}` aren't
|
|
registered in the substrate, so the profile is a property list
|
|
(same shape choice as the kernel's bucket / registry state).
|
|
Folding rules:
|
|
- `Create{Person|Service|Group}` (from a known `:actor`):
|
|
captures profile fields + `:created` (=`:published` seq).
|
|
Duplicate Creates are no-overwrite.
|
|
- `Update{Person|Service|Group, patch}`: merges `:patch` into the
|
|
profile last-write-wins per key.
|
|
- `Move`: records `:moved_to` on the profile.
|
|
Other activity types and non-actor object Creates pass through.
|
|
`fold_fn/0` plugs into `projection:start_link/3`. Local
|
|
`find_keyed/has_keyed/set_keyed` helpers (same gap as 1a — no
|
|
`lists:keyfind`/`keymember` in the substrate). 19 cases in
|
|
`actor_state_pure.sh`.
|
|
- [x] **2c** — `nx_kernel:bootstrap_actor/4(ActorId, Profile,
|
|
KeySpec, State)` — adds an actor bucket and publishes
|
|
`Create{Person|Service|Group}` as the bucket's first activity in
|
|
one call. Profile carries `:type` (defaults to `person`), `:name`,
|
|
`:preferredUsername`, `:summary`, `:icon`, `:public_keys`; the
|
|
function builds the Create's `:object` from the profile and the
|
|
kernel-side AS from `:public_keys`. gen_server variant
|
|
`bootstrap_actor/3` for live-kernel use; integration test in
|
|
`actor_lifecycle.sh` ties 2a artefacts, 2b projection, and 2c
|
|
bootstrap together end-to-end (pure + gen_server + projection
|
|
capture for all three actor types). 15/15.
|
|
|
|
**Acceptance:** `bash next/tests/actor_lifecycle.sh` passes 10+ cases.
|
|
|
|
---
|
|
|
|
## Step 3 — Key rotation via Update + actor-state
|
|
|
|
Per §9.2: rotation is itself an activity. The actor-state projection
|
|
keeps the full key history (with `created` / `superseded_at`) so
|
|
`envelope:verify_signature/2` continues to find historical keys when
|
|
verifying activities published before the rotation.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **3** — `actor_state.erl` `fold_update` now routes patches
|
|
through `apply_patch/3`, which special-cases two rotation patch
|
|
entries:
|
|
- `{add_publicKey, KeyProplist}` appends the key to `:public_keys`,
|
|
defaulting `:created` to the activity's `:published` if unset.
|
|
- `{supersede, OldKeyId}` marks the matching key with
|
|
`:superseded_at` = activity's `:published` (idempotent: existing
|
|
`:superseded_at` is preserved; unknown ids are no-ops).
|
|
Other patch entries fall through to last-write-wins per key
|
|
(preserving Step 2b semantics; verified by extra
|
|
`actor_state_pure.sh` cases).
|
|
New exports `key_history/1` (full list incl. superseded entries),
|
|
`active_keys_at/2` (subset active at time T, mirroring envelope's
|
|
`is_active_at` semantics — envelope keeps its predicate private,
|
|
so a local copy lives here), and `find_key_by_id/2`.
|
|
Rotation-purpose schema gating per §9.6 ("rotation activity must
|
|
itself be signed by an active key with `rotate-key` purpose") is
|
|
deferred to Step 5 (peer-side `stage_signature` will plumb the
|
|
purpose check through pipeline). 16 cases in `key_rotation.sh`
|
|
cover rotation arithmetic, `key_history` preservation, and live
|
|
`envelope:verify_signature/2` round-trips for pre / post / mid
|
|
rotation activities — including the negative case (post-rotation
|
|
K1-signed activity returns `{error, no_active_key}`).
|
|
|
|
**Acceptance:** `bash next/tests/key_rotation.sh` passes 12+ cases.
|
|
|
|
---
|
|
|
|
## Step 4 — Multi-actor HTTP routing
|
|
|
|
Per-actor URLs per design §16.1:
|
|
|
|
```
|
|
GET /actors/<id> # actor doc
|
|
GET /actors/<id>/outbox # OrderedCollection
|
|
GET /actors/<id>/outbox?page=N # page
|
|
POST /actors/<id>/inbox # peer delivery to this actor
|
|
GET /actors/<id>/followers # follower list
|
|
GET /actors/<id>/following # following list
|
|
POST /activity # authenticated publisher API (existing)
|
|
```
|
|
|
|
`POST /activity` still picks the publishing actor from the bearer
|
|
token; the token now maps to an `:actor_id` rather than a fixed `alice`.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **4a** — Per-actor URL routing. New `split_first_slash/1`
|
|
helper splits the `/actors/` suffix into `{Id, SubPath}`.
|
|
GET dispatch routes `outbox` / `inbox` / `followers` / `following`
|
|
sub-paths to four new content-negotiated response functions
|
|
(`actor_outbox_response_for/2`, `actor_inbox_get_response_for/2`,
|
|
`actor_followers_response_for/2`,
|
|
`actor_following_response_for/2`) — text / json / activity_json /
|
|
sx variants per existing format pattern. POST dispatch routes
|
|
`inbox` to a 202 Accepted stub (`actor_inbox_post_response/0` +
|
|
`accepted_response/1`). Unknown sub-paths under `/actors/<id>/`
|
|
return 404. Bare `/actors/<id>` keeps the M1 `actor_doc_response_for`
|
|
arm. 17 cases in `http_multi_actor.sh`.
|
|
- [x] **4b** — Token → ActorId map. New `resolve_token/2` reads
|
|
`:tokens` from Cfg (proplist `[{Token, ActorId}, ...]`) and
|
|
returns `{ok, ActorId}` on match. Falls back to the M1
|
|
`:publish_token` single-token field on miss (returns
|
|
`{ok, legacy}`, route through `nx_kernel:publish/1` to bucket 0
|
|
unchanged). Cfg with both fields: `:tokens` wins for matched
|
|
tokens; `:publish_token` only consulted on `:tokens` miss.
|
|
`handle_post_activity` now threads the resolved `ActorRef` to
|
|
`publish_if_kernel/3` which dispatches `publish_to/2` for
|
|
explicit actor ids and `publish/1` for the `legacy` atom.
|
|
No-kernel auth-only path unchanged. The dead M1
|
|
`expected_token/1` helper is gone. 8 new cases in
|
|
`http_multi_actor.sh` (25/25 total).
|
|
- [x] **4c** — `http_server:route/3(Req, Cfg, Kernel)` is sugar
|
|
that folds the Kernel reference (typically the registered
|
|
`nx_kernel` atom) into Cfg as `{kernel, Kernel}`. The dispatch
|
|
chain gained a Cfg arg threaded all the way to per-actor
|
|
sub-resource handlers (`dispatch/3` → `dispatch/4`, `actor_get/2`
|
|
→ `actor_get/3`, `actor_subresource_get/3` → /4). The outbox
|
|
sub-resource handler now reads `:kernel` and, when the actor
|
|
exists in the kernel, renders `tip: <N>` in text / JSON / SX
|
|
variants — proving the plumbing works end-to-end. Unknown
|
|
actors or unregistered kernels fall back to the 4a stub.
|
|
`try`/`of`/`catch` around `gen_server:call` deadlocks in this
|
|
port's scheduler (probably the catch-frame mask defers reply
|
|
delivery); the live handler does a bare `nx_kernel:log_tip_for/1`
|
|
+ integer guard instead. 8 new cases in `http_multi_actor.sh`
|
|
(33/33 total).
|
|
- [x] **4d** — Per-actor outbox listing reads from the named
|
|
bucket's log entries via new `nx_kernel:log_state_for/1`
|
|
gen_server export. `actor_outbox_full_response_for/5` renders
|
|
text / JSON / SX bodies with `:tip`, `:page`, and the page's
|
|
`:items` CID list. Empty pages degrade to the 4c tip-only body
|
|
to preserve back-compat with epochs 50-57. `?page=N` pagination
|
|
parsed at `route/2` time and threaded via Cfg as
|
|
`{request_query, Q}`; `page_size/0` returns 5 (proof of concept
|
|
— production picks 20+). 8 new cases in `http_multi_actor.sh`
|
|
(41/41 total). Substrate gotcha: named recursive funs
|
|
`fun F(...) -> ... F(...) end` not supported; `binary:matches/2`
|
|
and `lists:foreach/2` not registered — tests prove behaviour
|
|
via `match_prefix` substring checks rather than counting.
|
|
- [x] **4e** — POST /actors/<id>/inbox stays the 4a 202 stub
|
|
through 4a-4d; the real ingestion pipeline (sig verify + inbox-
|
|
bucket append + projection broadcast) is Step 5's whole topic.
|
|
No code change for this checkbox — it's a deliberate scope
|
|
boundary so 4d's listing semantics land cleanly before
|
|
inbound traffic shapes the same per-actor URLs.
|
|
|
|
**Acceptance:** `bash next/tests/http_multi_actor.sh` passes 14+ cases.
|
|
|
|
---
|
|
|
|
## Step 5 — POST /inbox: signature verify + ingestion
|
|
|
|
The receiving side of federation. A peer instance POSTs a signed activity
|
|
to `/actors/<id>/inbox`; the kernel verifies the signature, runs the
|
|
inbound validation pipeline, appends to the receiving actor's log
|
|
(separate from outbox — the inbox is its own log for activities the
|
|
actor *received*), and broadcasts to projections.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **5a** — Per-actor `:actor_inbox` log bucket in nx_kernel.
|
|
`add_actor/4` now opens a fresh inbox log (distinct base stub) for
|
|
each new actor; the bucket carries `[..., {actor_inbox, LogState}, ...]`
|
|
alongside the existing `:log` outbox field. Pure-functional
|
|
exports: `actor_inbox_state/2`, `actor_inbox_tip/2`,
|
|
`append_to_actor_inbox/3`. gen_server exports: `inbox_tip_for/1`,
|
|
`inbox_state_for/1`, `append_inbox/2`. Inbox and outbox tips are
|
|
fully independent (appending to one doesn't touch the other).
|
|
`next/tests/inbox_bucket.sh` 14/14. Signature verification +
|
|
pipeline gating live in 5b.
|
|
- [x] **5b** — Inbound validation pipeline. New
|
|
`pipeline:validate_inbound/3(Activity, PeerActorState, InboxLog)`
|
|
runs the federation inbound stage list — `stage_envelope` →
|
|
`stage_signature(PeerAS)` → `stage_replay(InboxLog)` — halting
|
|
on the first failure. New helper `inbound_stages/2(PeerAS, InboxLog)`
|
|
exposes the stage list for callers that want to splice extra
|
|
stages. Existing `validate_inbound/1` and the static
|
|
`inbound_stages/0` (envelope-only) stay untouched so outbox-side
|
|
callers don't have to re-key on a peer-AS they don't have. Sig
|
|
verification uses the peer's actor-state `:public_keys`, NOT the
|
|
local kernel's; peer-AS resolution is the caller's responsibility
|
|
(Step 5c wires the cache lookup). 14 cases in
|
|
`inbox_pipeline.sh`: happy path, bad shape, missing :signature
|
|
(rejected by stage_envelope before sig runs), wrong peer AS
|
|
(bad_signature), replay against inbox, distinct activities both
|
|
verify, stage short-circuit ordering verified.
|
|
- [x] **5c** — Peer-actors cache (`peer_actors.erl`). State shape
|
|
`[{PeerActorId, PeerActorState}, ...]` keyed by atom; PeerAS is
|
|
exactly the shape `envelope:verify_signature/2` reads (proplist
|
|
with `:public_keys`). Pure exports: `new/0`, `lookup/2`,
|
|
`store/3`, `evict/2`, `peers/1`, and the load-bearing
|
|
`lookup_or_fetch/3(PeerId, FetchFn, State)` that calls the
|
|
caller-supplied `FetchFn :: (PeerId) -> {ok, PeerAS} | {error, _}`
|
|
on miss and stores the successful result. Failed fetches do NOT
|
|
poison the cache so callers can retry on transient errors.
|
|
gen_server wrapper: `start_link/0,1`, `lookup_srv/1`,
|
|
`store_srv/2`, `lookup_or_fetch_srv/2`, `peers_srv/0`,
|
|
`evict_srv/1`. `start_link/1` accepts an initial state proplist
|
|
for tests / fixtures. 19/19 in `peer_actors.sh`. The actual
|
|
fetch implementation (HTTP GET of the peer's actor doc) is
|
|
Step 5d's responsibility — for 5c, FetchFn is just a contract.
|
|
- [x] **5d** — http_server inbox handler wires the chain. POST
|
|
/actors/<id>/inbox is now special-cased in `route/2` (next to
|
|
POST /activity) so the body + full Cfg reach the handler. New
|
|
`handle_inbox_post/3` orchestrates: `kernel_has_actor` →
|
|
`decode_activity` (term_codec wire format) → `resolve_peer_as`
|
|
(Cfg `:peer_as` map > `:peer_actors` srv > `:peer_fetch_fn`
|
|
fallback) → `pipeline:validate_inbound/3` → `nx_kernel:append_inbox`.
|
|
Status codes:
|
|
- 202 Accepted on pipeline ok + inbox append
|
|
- 401 Unauthorized on bad_signature / no_signature / unknown
|
|
peer / fetch error
|
|
- 404 Not Found on unknown target actor
|
|
- 422 Unprocessable on shape / decode / replay failure
|
|
v1 stub `actor_post/1` removed; the route/2 special case
|
|
supersedes it. M1 `actor_inbox_post_response/0` kept for
|
|
callers that need to compose the response shape.
|
|
Projection broadcast on success is intentionally deferred —
|
|
the same TODO covers outbox broadcast invariance and lands in
|
|
a follow-up sub-deliverable. `inbox.sh` 11/11 covers happy
|
|
path / shape / sig / replay / unknown-target / multi-message;
|
|
`inbox_peer_resolution.sh` 6/6 covers the four peer-AS
|
|
resolution paths. Tests split into two files because the
|
|
cumulative cost of one kernel start_link per epoch pushed a
|
|
single suite past the wall-clock budget.
|
|
|
|
**Acceptance:** `bash next/tests/inbox.sh` passes 16+ cases.
|
|
|
|
---
|
|
|
|
## Step 6 — Follow lifecycle
|
|
|
|
Per §13.2:
|
|
|
|
```sx
|
|
(activity 'Follow ;; from A → B
|
|
:object actor-id-B
|
|
:to (list actor-id-B))
|
|
```
|
|
|
|
B responds with `Accept` (or `Reject`); A's follower-graph projection
|
|
tracks the state. `Undo{Follow}` reverses it.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **6a** — `follower_graph.erl` Erlang-fun stand-in for the
|
|
genesis `follower-graph.sx` projection body. State shape is a
|
|
property-list keyed by ActorId (maps `#{}` not in substrate),
|
|
each entry carries `{following, followers, pending_outbound,
|
|
pending_inbound}` lists. Fold rules:
|
|
- `Follow{actor: A, object: B}` — A → pending_outbound(B);
|
|
B → pending_inbound(A).
|
|
- `Accept{actor: B, object: F=Follow{A→B}}` — A → following(B)
|
|
on A's bucket; B → followers(A) on B's bucket; pendings cleared.
|
|
- `Reject{actor: B, object: F}` — pendings cleared, no promote.
|
|
- `Undo{actor: A, object: F}` — drops A↔B from every list; only
|
|
F's original actor can Undo (carol can't Undo F{A→B}).
|
|
Self-follows are no-ops; duplicate Follows are idempotent;
|
|
Accept/Reject/Undo of non-Follow `:object`s pass through.
|
|
18 cases in `follower_graph.sh`. The `fold_fn/0` 2-arity fun
|
|
plugs into `projection:start_link/3` exactly like
|
|
`define_registry:fold_fn/0` and `actor_state:fold_fn/0`.
|
|
- [x] **6b** — Wire follower-graph fold to the inbox handler.
|
|
`http_server.erl` `run_inbox_pipeline` now calls
|
|
`broadcast_to_inbox_projections/2` after a successful
|
|
`nx_kernel:append_inbox`. Cfg may carry `{inbox_projections,
|
|
[Name, ...]}` listing projection gen_servers; each gets the
|
|
activity via `projection:async_fold/2` (fire-and-forget so the
|
|
handler doesn't block on fold processing). Field absent =
|
|
no-op. v2 leaves the routing field global; per-actor
|
|
projection wiring is a forward-looking follow-up. 9/9 in
|
|
`follow_lifecycle.sh` covering 202 ingestion, follower_graph
|
|
pending-state mutation on both sides, no-inbox_projections
|
|
no-op path, bad-sig short-circuit (projection stays clean),
|
|
multi-peer accumulation, end-to-end Follow+Accept projection
|
|
convergence (Accept fed in via projection:async_fold for v2).
|
|
- [x] **6c** — Auto-Accept publish. New `maybe_auto_accept/3` in
|
|
`http_server.erl` fires after a successful inbox ingestion if
|
|
Cfg carries `{auto_accept_follows, true}` AND the activity's
|
|
`:type` is `follow`. The handler constructs an
|
|
`Accept{actor: target, object: OriginalFollow}` request and
|
|
routes it through `nx_kernel:publish_to/2`, which goes through
|
|
the full outbox pipeline (envelope construct + HMAC sign + log
|
|
append + outbox projection broadcast). When the target's
|
|
outbox `:projections` list includes the same follower_graph
|
|
projection the inbox uses, the Accept fold-converges the
|
|
bilateral relationship — `alice.followers = [bob]` and
|
|
`bob.following = [alice]` — without any test scaffolding.
|
|
Default is off; manual-moderation deployments leave the flag
|
|
unset. Bad-sig / non-Follow ingestion short-circuits before
|
|
the Accept attempt. 9/9 in `auto_accept.sh`.
|
|
|
|
**Acceptance:** `bash next/tests/follow_lifecycle.sh` passes 14+ cases.
|
|
|
|
---
|
|
|
|
## Step 7 — Audience-resolving delivery set
|
|
|
|
For each outbound activity, compute the set of inbox URLs to POST to.
|
|
Sources: explicit `:to` + `:cc` recipients, plus `Public` / `Followers`
|
|
expansion via the audience predicates from M1's genesis bundle.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **7a** — `delivery:delivery_set/2,3` returns the
|
|
audience-resolved deduplicated list of ActorId atoms for an
|
|
outbound activity. Sources: explicit `:to` and `:cc` fields
|
|
(atom or list of atoms / audience symbols), plus expansion of
|
|
`followers` (via follower_graph) and `public` (v2 placeholder
|
|
— Step 7c). Self-delivery is suppressed every time the
|
|
sender's ActorId appears in the set. Returns are ActorId
|
|
atoms for now; Step 8 will resolve each entry to
|
|
`{PeerInstanceUrl, ActorId}` via the peer-actors cache. 17
|
|
cases in `delivery_set.sh` covering empty / single / list /
|
|
cc-union / self-suppress / dedup / followers-expand /
|
|
public-empty / mixed audience / collect_recipients +
|
|
suppress_self + dedup helpers + expand_audience pass-through.
|
|
Module lives in `next/kernel/delivery.erl` (separate from
|
|
outbox so Step 8's delivery-queue gen_server has a clean home).
|
|
- [x] **7b** — Public audience expansion. v2 default: `public`
|
|
expands to the sender's followers (same as `followers`) per
|
|
design §13.4 — the practical fan-out for an open social
|
|
network is "every follower of the publishing actor". The
|
|
explicit shared-inbox peer-instance model (Mastodon-style
|
|
per-instance broadcast) defers to v3 when there's a real
|
|
known-peer-instance registry to drive it. `public + followers`
|
|
in the same audience deduplicates because both symbols
|
|
expand identically. 19/19 in `delivery_set.sh` (2 new cases
|
|
+ 1 case updated from the v2 placeholder behavior).
|
|
- [x] **7c** — Outbox-side integration. `outbox:publish/2`
|
|
now computes the delivery set after sign + log and stashes it
|
|
in the Result proplist as `{delivery_set, [ActorId, ...]}`.
|
|
Context's optional `:follower_graph` field carries a
|
|
follower_graph state for `public` / `followers` audience
|
|
expansion; absent -> empty graph (explicit `:to`/`:cc`
|
|
recipients still resolve). New helper
|
|
`compute_delivery_set/3(Request, Signed, Context)` and
|
|
`recipients_envelope/2` synthesise a minimal recipient
|
|
envelope from Request's `:to`/`:cc` + Signed's `:actor` so
|
|
`delivery:delivery_set/3` can process it unchanged
|
|
(outbox:construct/4 doesn't carry `:to`/`:cc` through the
|
|
envelope shape, and changing that surface would ripple to
|
|
every existing envelope test). Step 8's delivery-queue
|
|
worker will read `{delivery_set, [ActorId, ...]}` off the
|
|
publish result. 17/17 in `outbox_publish.sh` (+4 new cases:
|
|
empty-default, explicit-:to, followers-symbol-via-graph,
|
|
self-suppression). Module load chain rebumped from epoch 5
|
|
to epoch 7 (adds follower_graph + delivery as dependencies)
|
|
and the test's internal sx_server timeout bumped 240s →
|
|
480s to fit the larger module set.
|
|
|
|
---
|
|
|
|
## Step 8 — Outbound delivery queue
|
|
|
|
Per §13.4: every queued delivery has retry semantics. v2 uses one
|
|
gen_server-per-peer-instance worker holding a small queue. Failures
|
|
back off exponentially; permanent failures (HTTP 410, bad TLS) move to
|
|
a dead-letter list visible via `/admin/dead-letter`.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **8a** — `delivery_worker.erl` skeleton: pure-functional
|
|
state shape `[{peer, _}, {pending, [_]}, {attempts, [{Cid, N}]},
|
|
{dead_letter, [_]}, {dispatch_fn, _}]` plus
|
|
`enqueue_pure/3`, `drain_pure/1`, `deliver_one_pure/2` and the
|
|
backoff schedule (`backoff_for/1`, `schedule_for/1`) matching
|
|
§13.4 (30s / 5m / 30m / 6h / 24h then dead-letter).
|
|
gen_server wrapper with `start_link/1,2`, `enqueue/2`, `flush/1`,
|
|
`pending_srv/1`, `set_dispatch_fn/2`. dispatch_fn is a
|
|
caller-supplied 1-arity fun so tests can stub the HTTP POST;
|
|
Step 8f plugs in the live httpc call without touching the
|
|
queue logic. No actual HTTP yet; no retry timer wiring yet.
|
|
17/17 in `delivery_worker.sh`.
|
|
- [x] **8b-pure** — Retry-time bookkeeping (pure-functional).
|
|
State shape gains `{next_retry, [{Cid, NextRetryAt}]}` alongside
|
|
the existing `:attempts`. New exports:
|
|
`record_failure_pure/3(Cid, Now, State)`,
|
|
`record_success_pure/2(Cid, State)`,
|
|
`next_due_pure/2(Now, State)`, `attempts_for/2`,
|
|
`next_retry_at/2`, `dead_letter_list/1`.
|
|
`record_failure_pure` bumps the attempt counter and computes
|
|
`Now + backoff_for(NewAttempts)` as the next retry; on the 6th
|
|
failure (`backoff_for` returns `dead_letter`) the matching
|
|
activity moves from `:pending` to `:dead_letter` and the cid
|
|
is cleared from `:next_retry`. `record_success_pure` clears
|
|
both. `next_due_pure` returns cids whose retry time has
|
|
passed. 11 cases in `delivery_retry.sh`.
|
|
- [ ] **8b-timer** — Erlang-side timer wiring (`erlang:send_after`
|
|
self-cast or equivalent). Needs the same substrate primitive
|
|
that `gen_server` uses for `timeout` returns. Defer behind
|
|
substrate gap discovery for now — see Blockers.
|
|
- [x] **8c** — Delivery-state projection
|
|
(`next/kernel/delivery_state.erl`). Folds delivery events into
|
|
per-peer worker-shaped snapshots so the outbound queue survives
|
|
kernel restart. Event shapes:
|
|
`[{type, enqueued|delivered|failed|dead_lettered}, {peer, _},
|
|
{activity, _} | {cid, _}, {now, _}?]`. State shape
|
|
`[{PeerId, WorkerProplist}, ...]` mirrors `delivery_worker:new/1`'s
|
|
output so a fresh gen_server can be hydrated on restart. Public
|
|
API: `new/0`, `fold/2`, `fold_fn/0`, `peer_state/2`, `peers/1`,
|
|
per-field accessors (`pending`, `attempts`, `next_retry`,
|
|
`dead_letter`). Uses `delivery_worker:backoff_for/1` to decide
|
|
dead-letter promotion on the 6th failure, so the projection
|
|
and the live worker stay in lockstep. 14/14 in
|
|
`delivery_state.sh`. The restart-hydration helper
|
|
(`delivery_worker:state_from_proj/2` or similar) lands when
|
|
8b-timer wires the live retry loop.
|
|
- [x] **8d** — `outbox:publish/2` dispatches each delivery-set
|
|
entry to the matching worker. New `dispatch_deliveries/3` +
|
|
`enqueue_each/2` in `outbox.erl` walk the computed
|
|
`delivery_set` and call `delivery_worker:enqueue(PeerId,
|
|
Activity)` for each registered peer atom. Missing workers
|
|
(no `whereis`) are silently skipped — lazy worker creation
|
|
belongs to the kernel manager (Step 8d-mgr or later).
|
|
Gated by `Context` field `{dispatch_deliveries, true}` so
|
|
every M1 outbox caller stays back-compat (default off). 7/7
|
|
in `delivery_dispatch.sh` covering single-peer enqueue,
|
|
two-peer fan-out, missing-worker skip, no-flag no-op,
|
|
FIFO append across two publishes, empty delivery_set no-op.
|
|
- [x] **8e** — `httpc:request/4` BIF wrapper. ~~Blocker~~ resolved:
|
|
loops/fed-prims merged into architecture, native `http-request`
|
|
primitive available. Wrapper at `lib/erlang/runtime.sx`
|
|
(briefing-allowed-exception scope) marshals Erlang
|
|
`(Url::binary, Method::atom|binary, Headers::proplist, Body::binary)`
|
|
→ SX `(http-request method url headers body)` → Erlang
|
|
`{ok, Status::integer, Headers::proplist, Body::binary}`.
|
|
Atom methods are upcased (`get` → `"GET"`) for HTTP-wire convention;
|
|
binaries pass through verbatim. Test: `next/tests/httpc_request.sh`
|
|
10/10 pass — registration, badarg validation, live GET 200,
|
|
body bytes match, headers proplist shape, 404 surfaces as ok-tuple,
|
|
binary method works.
|
|
- [ ] **8f** — Real HTTP dispatch through the BIF + content-type
|
|
wiring. dispatch_fn for live use becomes a closure over the
|
|
peer URL that calls `httpc:request/4` with the signed envelope
|
|
bytes as the body.
|
|
|
|
**Tests:**
|
|
|
|
- Successful delivery → worker queue empties.
|
|
- Failed delivery → backoff schedule respected.
|
|
- Dead-letter after max attempts.
|
|
- Cross-restart: queue restored from delivery-state projection.
|
|
- Concurrent deliveries to multiple peers don't serialise.
|
|
|
|
**Acceptance:** `bash next/tests/delivery_queue.sh` passes 16+ cases.
|
|
|
|
---
|
|
|
|
## Step 9 — Backfill on Follow accept
|
|
|
|
Per §13.3: A wants B's history when A first follows B. Four modes:
|
|
|
|
| Mode | Behavior |
|
|
|-----------|---------------------------------------------|
|
|
| `none` | New follower sees only forward-going content |
|
|
| `last-N` | Backfill last N activities |
|
|
| `last-T` | Backfill last T duration of activities |
|
|
| `full` | Backfill entire outbox |
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **9a** — Pure-functional backfill slicing in
|
|
`next/kernel/backfill.erl`:
|
|
- `slice/2,3(Mode, LogState[, Wrap])` returns the entry list
|
|
for a given mode. Wrap=true marks each entry
|
|
`{backfilled, true}` so receiving projections can decide
|
|
whether to re-fold or skip (per §13.3, wrapped bodies
|
|
preserve `:id` so replay defence still catches duplicates).
|
|
- Modes: `none`, `full`, `{last_n, N}`, `{last_t, T, NowFn}`,
|
|
`{since_cid, Cid}`. NowFn is a 0-arity fun so tests can
|
|
fake-time it.
|
|
- `parse_mode/1` lifts the Follow activity's `:backfill`
|
|
value (atom or proplist) into the internal mode tuple;
|
|
unknown shapes degrade to `none` (open-world default).
|
|
Substrate gotchas re-confirmed:
|
|
`lists:nthtail/2` not in this port (rolled `drop_n/2`);
|
|
pattern-alias `Pat = Var` not supported (rewrote
|
|
`parse_mode/1` clauses with explicit deconstruction).
|
|
20/20 in `backfill.sh` covering all 5 modes (with edge
|
|
cases: N=0, N>length, T=0, since_cid hit/miss/unknown),
|
|
wrap_backfill, parse_mode atoms / tuples / proplists /
|
|
unknown.
|
|
- [x] **9b** — `GET /actors/<id>/outbox?since=Cid` pagination
|
|
route. The Step 4d outbox handler in `http_server.erl`
|
|
(`actor_outbox_response_for/3`) now reads `?since=` from the
|
|
query string via new `parse_since/1` + `scan_param/2,3` +
|
|
`skip_to_amp/1` (handles `since=X&page=2` and `page=2&since=X`
|
|
identically), pre-filters entries via
|
|
`backfill:since_cid_entries/2`, then runs the existing page
|
|
slice on the filtered list. `?since=unknown` → empty page →
|
|
body degrades to the tip-only shape (Step 4d back-compat).
|
|
3 new cases in `http_multi_actor.sh` (44/44 total) — exercise
|
|
filtering, unknown-cid, combined `?since= + ?page=`. Also
|
|
added `follower_graph` + `delivery` + `backfill` module loads
|
|
to `http_multi_actor.sh` (downstream dependency since Step
|
|
7c/9a — must have been latently broken; the existing 41
|
|
passes + 3 new = 44 now all green).
|
|
- [x] **9c** — Follow → Accept → backfill drain (in-process).
|
|
`maybe_auto_accept/3` in `http_server.erl` now calls a new
|
|
`maybe_backfill/3` after the Accept publish: when Cfg carries
|
|
`{backfill_enabled, true}` AND the Follow envelope carries a
|
|
`:backfill` field, the receiver parses the mode via
|
|
`backfill:parse_mode/1`, slices its outbox via
|
|
`backfill:slice/3` (Wrap=true so each entry gets
|
|
`{backfilled, true}`), and enqueues every slice entry onto
|
|
the peer's delivery_worker if registered (silently skipped
|
|
otherwise — kernel manager lazy creation belongs upstream).
|
|
6/6 in `backfill_drain.sh` covering full path + entry marker
|
|
+ flag-off no-op + missing-backfill-field no-op + missing-
|
|
worker silent skip. The live HTTP dispatch of those queued
|
|
entries still gates on Blockers #2 (httpc).
|
|
|
|
**Tests:**
|
|
|
|
- `last-N` mode delivers exactly N most-recent activities.
|
|
- `last-T` mode delivers everything published since `now - T`.
|
|
- `full` mode delivers everything, page by page.
|
|
- `none` mode delivers nothing.
|
|
- Backfilled activities preserve original `:id` (CID).
|
|
|
|
**Acceptance:** `bash next/tests/backfill.sh` passes 12+ cases.
|
|
|
|
---
|
|
|
|
## Step 10 — Discovery
|
|
|
|
Per §13.7: webfinger plus actor doc fetch.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **10a** — Local-side discovery primitives in
|
|
`next/kernel/discovery.erl`:
|
|
- `parse_acct/1(<<"acct:user@host">>)` and
|
|
`parse_acct/1(<<"user@host">>)` (prefix optional) return
|
|
`{ok, User, Host}` or `{error, _}`. Reject empty user/host
|
|
and missing `@`. Host preserves an optional `:port` suffix.
|
|
- `parse_resource/1` is an alias for the webfinger query
|
|
parameter shape.
|
|
- `actor_url_for/2(User, Host)` synthesises
|
|
`http://<host>/actors/<user>` (TLS / https is v3, gated by
|
|
a TLS substrate Blocker).
|
|
- `webfinger_body/3(User, Host, ActorUrl)` builds the RFC 7033
|
|
JSON body with `:subject` + `:links[]` carrying
|
|
`rel: self / type: application/activity+json / href`.
|
|
Hand-rolled byte concatenation — no JSON BIF on this port.
|
|
`<<"...">>` string-literal segments truncate to one byte on
|
|
this port (briefing gotcha re-confirmed), so `"acct:"` is
|
|
spelled as `<<97,99,99,116,58>>`. 12/12 in `discovery.sh`.
|
|
- [x] **10b** — http_server route
|
|
`GET /.well-known/webfinger?resource=acct:user@host`. New
|
|
dispatch arm next to `/.well-known/sx-capabilities` calls
|
|
`handle_webfinger/1(Cfg)`, which reads `:request_query` from
|
|
Cfg (threaded by route/2 from the Req's `:query` field per
|
|
Step 4d), parses the `resource=` param via
|
|
`parse_resource_param/1` + `take_until_amp/1`, hands off to
|
|
`discovery:parse_acct/1`, then to `webfinger_lookup/3`:
|
|
- Optional Cfg `:webfinger_host` (binary) — when set, the
|
|
acct's `@host` must match exactly; missing accepts any.
|
|
- Optional Cfg `:kernel` (atom, per Step 4c) — uses
|
|
`kernel_has_actor/2` to verify the actor exists. When no
|
|
kernel cfg'd (pure route tests), every user is "known".
|
|
- Match → 200 + `discovery:webfinger_body/3` rendered as
|
|
`application/activity+json`; miss → 404.
|
|
10/10 in `webfinger_route.sh` covering happy paths
|
|
(no-kernel, with-kernel, host-match), 404 paths
|
|
(missing-resource, bad-acct, unknown-actor, host-mismatch,
|
|
wrong-method).
|
|
- [ ] **10c** — Peer-actor fetch + cache write. Gates on
|
|
Blockers #2 (native `http-request` primitive missing).
|
|
Step 5's peer_actors cache already exposes the
|
|
`lookup_or_fetch` shape; this Step 10c plugs the discovery
|
|
HTTP fetch into the FetchFn slot.
|
|
|
|
**Tests:**
|
|
|
|
- Webfinger for known actor → 200 with `links[].href`.
|
|
- Webfinger for unknown → 404.
|
|
- Cross-instance: A resolves an acct on B → fetch succeeds.
|
|
- Actor-doc fetch caches the result.
|
|
- Cache invalidation on key rotation (v3 — for now, no TTL).
|
|
|
|
**Acceptance:** `bash next/tests/discovery.sh` passes 12+ cases.
|
|
|
|
---
|
|
|
|
## Step 11 — Rich verbs as runtime artifacts
|
|
|
|
Per the verb-extensibility proof point (M1 §9a), new verbs land as
|
|
`DefineActivity` artifacts published into the genesis-equivalent boot
|
|
log, not as kernel code changes. v2 adds:
|
|
|
|
| Verb | Object shape | Use case |
|
|
|---------|---------------------------------------|---------------------------------------|
|
|
| `Note` | `{content, tags?}` | Short authored message |
|
|
| `Announce` | `{object: <ActivityCid>}` | Propagate a peer's activity to followers |
|
|
| `Endorse` | `{object: <Cid>, kind: like|share}` | Cross-actor signaling |
|
|
|
|
Announce is the critical one for federation — it lets one actor
|
|
re-broadcast another actor's content to their own followers.
|
|
|
|
**Deliverables:**
|
|
|
|
- [x] **11b** — Projection folds for the new verbs.
|
|
- `next/kernel/announce_state.erl`: tracks per-Cid announcer
|
|
set. Public API `new/0`, `fold/2`, `fold_fn/0`,
|
|
`announcers_for/2`, `announce_count/2`,
|
|
`announced_cids/1`, `has_announced/3`. Set semantics
|
|
(duplicate Announce by same actor is a no-op).
|
|
- `next/kernel/endorsement_state.erl`: tracks per-Cid +
|
|
per-kind + per-actor endorsement counters. Public API
|
|
`new/0`, `fold/2`, `fold_fn/0`, `counters_for/2`,
|
|
`total_for/2`, `kinds_for/2`, `endorsers_for/3`,
|
|
`has_endorsed/4`. Additive semantics (re-endorse by same
|
|
actor under same kind bumps the counter; Undo{Endorse}
|
|
semantics defer to a follow-up).
|
|
Both `fold_fn/0`s plug into `projection:start_link/3`. 19/19
|
|
in `rich_verbs.sh` covering happy paths + predicates + non-
|
|
matching-activity pass-through.
|
|
- [x] **11a** — Announce + Endorse genesis activity-types
|
|
(Note already exists as an object-type from M1 — Create{Note}
|
|
is the publish path). Two new `DefineActivity` SX files in
|
|
`next/genesis/activity-types/` with `:name`, `:doc`,
|
|
`:schema` (Announce: `:object` must be a string CID; Endorse:
|
|
`:object` and `:kind` must both be strings). Manifest updated
|
|
to 5 activity-types / 36 total entries. Hardcoded count
|
|
assertions bumped in `bootstrap_read.sh`, `bootstrap_load.sh`,
|
|
`bootstrap_populate.sh`, `bootstrap_start.sh`. `genesis_parse.sh`
|
|
+4 cases for the two new files (head form + name).
|
|
- Each is shipped to a fresh instance via a bootstrap manifest entry
|
|
*or* published as the first activity on the actor's outbox; either
|
|
works because of the verb-extensibility mechanism.
|
|
- Announce-specific delivery: the announced activity's CID is included
|
|
in the Announce; followers can re-fetch the referenced activity from
|
|
the original instance if their projection wants to fold the body.
|
|
|
|
**Tests:**
|
|
|
|
- Define + publish Note works end-to-end.
|
|
- Define + publish Announce wraps another activity by CID.
|
|
- Announce delivery: A announces B's Note; A's followers see the
|
|
Announce; their `feed` projection optionally fetches the wrapped Note.
|
|
- Endorse increments an endorsement counter on the target Activity.
|
|
- Verb registration is observable in the `define-registry` projection.
|
|
|
|
**Acceptance:** `bash next/tests/rich_verbs.sh` passes 14+ cases.
|
|
|
|
---
|
|
|
|
## Step 12 — Two-instance smoke test
|
|
|
|
**The proof point.** `next/tests/smoke_federate.sh` spins up two kernel
|
|
instances on distinct ports, walks them through the full federation
|
|
flow, and exits 0.
|
|
|
|
**Test outline:**
|
|
|
|
```bash
|
|
# 0. Start two instances: A on 9999, B on 9998
|
|
./next/scripts/start_pair.sh
|
|
|
|
# 1. Bootstrap two actors: alice@A, bob@B
|
|
curl -X POST :9999/activity \
|
|
-H "Authorization: Bearer $TOKEN_A" \
|
|
-d '{"type":"Create","object":{"type":"Person","name":"alice"}}'
|
|
|
|
curl -X POST :9998/activity \
|
|
-H "Authorization: Bearer $TOKEN_B" \
|
|
-d '{"type":"Create","object":{"type":"Person","name":"bob"}}'
|
|
|
|
# 2. alice@A discovers bob@B via webfinger
|
|
curl :9999/.well-known/webfinger?resource=acct:bob@localhost:9998
|
|
|
|
# 3. alice follows bob
|
|
curl -X POST :9999/activity \
|
|
-d '{"type":"Follow","object":"http://localhost:9998/actors/bob"}'
|
|
|
|
# 4. Expect alice's follower-graph: pending_outbound includes bob
|
|
curl :9999/actors/alice/following | jq -e '.[] | select(.id == "bob")'
|
|
|
|
# 5. Expect bob auto-accepts; alice's pending_outbound clears
|
|
sleep 1
|
|
curl :9999/actors/alice/following | jq -e '.[] | select(.id == "bob")'
|
|
|
|
# 6. bob publishes a Note
|
|
curl -X POST :9998/activity -d '{"type":"Create","object":{"type":"Note","content":"hi"}}'
|
|
|
|
# 7. alice's inbox receives the Note
|
|
sleep 1
|
|
curl :9999/actors/alice/inbox?page=true | jq -e '.orderedItems[] | .type == "Create" and .object.type == "Note"'
|
|
|
|
# 8. alice's actor-state projection has the new Note
|
|
curl :9999/projections/feed | jq -e ". | length > 0"
|
|
|
|
# 9. Key rotation: bob rotates keys
|
|
curl -X POST :9998/activity -d '{"type":"Update","object":"bob","patch":{...}}'
|
|
|
|
# 10. alice still verifies older Notes against the old key
|
|
# (via actor-state's key history)
|
|
|
|
# 11. Announce: alice announces bob's Note
|
|
curl -X POST :9999/activity -d '{"type":"Announce","object":"<bob-note-cid>"}'
|
|
|
|
# 12. Verify Announce delivers to alice's followers (zero in v1 but
|
|
# the activity should be in alice's outbox)
|
|
|
|
# 13. Shutdown both instances; restart; verify state survives
|
|
./next/scripts/stop_pair.sh
|
|
./next/scripts/start_pair.sh
|
|
curl :9999/actors/alice/following | jq -e '.[] | select(.id == "bob")'
|
|
```
|
|
|
|
**Acceptance for Step 12:** `smoke_federate.sh` exits 0. The full flow
|
|
runs without any human-in-the-loop coordination, both instances'
|
|
projections converge, and a restart preserves all federation state.
|
|
|
|
---
|
|
|
|
## Acceptance criteria for milestone 2
|
|
|
|
All of:
|
|
|
|
1. **Each step's test suite passes** (`bash next/tests/<step>.sh`).
|
|
2. **The federation smoke test passes** (`bash next/tests/smoke_federate.sh`).
|
|
3. **Milestone 1 baseline preserved** — the entire M1 test suite still
|
|
passes (~560 assertions across 50 suites).
|
|
4. **Erlang-on-SX conformance** — adding multi-actor + federation kernel
|
|
code in `next/kernel/*.erl` doesn't break Phase 1-8 conformance
|
|
(currently 761/761).
|
|
5. **Restart durability** — kill both instances mid-delivery, restart,
|
|
queues resume, projections converge, no log corruption.
|
|
6. **Manual real Mastodon poke** — point a Mastodon account at
|
|
`https://next-A.rose-ash.com/actors/alice` and verify the actor
|
|
doc fetches. (Read-only AP interop only — Mastodon Follow is v3
|
|
gating on HTTP-Signatures-2018 compat.)
|
|
|
|
## What lands when
|
|
|
|
Steps 1-3 are sequential (multi-actor foundation). Steps 4-10 are
|
|
mostly sequential within the federation core but some can parallelise:
|
|
4-6 are sequential; 7-9 can interleave after 6 lands.
|
|
|
|
```
|
|
M1 closeout (HEAD) ──┐
|
|
│
|
|
▼
|
|
┌─── Step 1 ──┬─── Step 2 ──┬─── Step 3
|
|
│ │ │
|
|
└─────────────┼─── Step 4 ──┘
|
|
│
|
|
└─── Step 5 ────┐
|
|
│
|
|
Step 6 ───┤
|
|
│
|
|
Step 7 ───┤
|
|
Step 8 ───┤
|
|
Step 9 ───┤
|
|
│
|
|
Step 10 ──┤
|
|
│
|
|
Step 11 ──┤
|
|
│
|
|
Step 12 ──┘
|
|
```
|
|
|
|
Estimated effort: ~40-60 commits across all 12 steps. A focused agent
|
|
loop (`loops/fed-sx-m2`) should be able to land this with the same
|
|
discipline as M1.
|
|
|
|
## What's deferred to milestone 3
|
|
|
|
- **rose-ash port** (the headline of M3). Blog, market, events,
|
|
federation hub, account, orders — all delivered as fed-sx
|
|
applications. Each existing rose-ash domain becomes
|
|
`DefineApplication{...}` artifacts.
|
|
- **TLS / HTTP-Signatures-2018 / RFC 9421**. Real Mastodon interop.
|
|
- **Multi-instance over real WAN.** Cross-instance over TLS, NAT
|
|
traversal, peer instance allowlists.
|
|
- **IPFS / S3 storage backends** as `DefineStorage` entries.
|
|
- **Browser client + operator dashboard.** Probably Elm-on-SX.
|
|
- **Cross-host conformance** — Python / JS / Haskell hosts running
|
|
fed-sx with the same conformance corpus.
|
|
- **OpenTimestamps proofs** as `DefineProof` entries.
|
|
- **Reputation, allowlists, rate-limiting** — full §13.6 abuse
|
|
posture.
|
|
- **Performance work** — JIT-compiled folds, snapshot acceleration,
|
|
federation batching, mailbox prioritisation.
|
|
- **Capability tokens / delegation** — multi-device for a single
|
|
actor.
|
|
|
|
---
|
|
|
|
## Appendix A: open questions for milestone 2
|
|
|
|
Things still under-specified; resolve as work begins.
|
|
|
|
1. **Inbox-side stage_signature key fetching.** When A receives a
|
|
POST /inbox from peer instance B for the first time, A needs B's
|
|
actor doc to verify the signature. Synchronous fetch vs. queue-
|
|
and-retry? Synchronous is simpler but blocks the inbox handler;
|
|
queue-and-retry needs deferred validation state. Probably
|
|
synchronous with a 5s timeout for v2.
|
|
|
|
2. **Backfill granularity for `last-N`.** N counts forward (oldest
|
|
first) or backward (newest first)? Forward matches projection-fold
|
|
semantics; backward matches user expectation. Probably forward
|
|
for v2, document the choice.
|
|
|
|
3. **Auto-Accept policy on Follow.** v2 ships open-world: every
|
|
Follow is auto-accepted. Manual moderation (held in a `pending`
|
|
list, accepted via /admin/) is v3 with the operator dashboard.
|
|
|
|
4. **Delivery worker per peer instance vs. per peer actor.** Per
|
|
instance is simpler (one HTTPS connection pool) but throttles
|
|
inter-actor bandwidth on busy peers. v2 starts with per-instance;
|
|
per-actor sharding is a perf tweak in §15.
|
|
|
|
5. **Two-instance test harness.** How do we start a pair of kernels
|
|
in one bash test? Probably `bootstrap:start/3` twice with different
|
|
ActorIds + ports + base paths. Need to confirm `nx_kernel` can be
|
|
started under different registered atoms (`nx_kernel_a`, `nx_kernel_b`)
|
|
for the test. Process registration in this port supports arbitrary
|
|
atom names (verified in M1).
|
|
|
|
6. **Multi-host conformance.** Adding cross-host tests for federation
|
|
requires Python/JS hosts to implement the v2 spec corpus too.
|
|
Deferred to v3; v2 conformance is one-host only.
|
|
|
|
7. **Storage of received activities.** When A receives a Note from B
|
|
via /inbox, does A keep B's signed envelope verbatim (for re-broadcast
|
|
on Announce), or does A re-construct + re-sign with A's own key?
|
|
AP-canon: keep verbatim. Confirm at Step 5.
|
|
|
|
---
|
|
|
|
## Blockers
|
|
|
|
Pre-existing regressions inherited from the M1 closeout. Out of m2
|
|
scope (substrate, not `next/**`), tracked here so iteration can
|
|
proceed.
|
|
|
|
1. **`next/tests/http_server_tcp.sh` 0/5** — pre-existing regression
|
|
introduced by `78eae9ef` (`fed-sx-m1: 8b-bridge cleanup`).
|
|
`lib/erlang/runtime.sx:1593` still references `er-http-resp-to-sx`
|
|
and `er-http-req-of-sx` in `er-bif-http-listen`'s sx-handler body,
|
|
but the cleanup commit removed both helpers without rewriting the
|
|
BIF. Listener binds (TCP socket accepts), but every request handler
|
|
crashes on first call to the undefined helpers — curl gets 000 /
|
|
empty body. Fix needs to rewrite the sx-handler body around the
|
|
live `er-request-dict-to-proplist` / `er-proplist-to-dict`
|
|
helpers (which the cleanup commit's message claimed are already
|
|
in use, but which the BIF body never picked up). Substrate work,
|
|
belongs on `loops/erlang`. m2 work continues against the in-process
|
|
HTTP layer (`http_marshal.sh` 10/10, `http_publish_fold.sh` 10/10)
|
|
until resolved. Confirmed pre-existing by stashing 1a's changes and
|
|
re-running on the unmodified m1 closeout HEAD.
|
|
|
|
2. **Native `http-request` (HTTP client) primitive missing** —
|
|
~~discovered during Step 8e prep~~ **RESOLVED 2026-06-07** by
|
|
the user-authorized `loops/fed-prims` → `architecture` merge.
|
|
The primitive now registers at `bin/sx_server.ml:868+` with
|
|
signature `(http-request meth url headers body)` returning a
|
|
`{:status :headers :body}` dict and raising `Eval_error` on
|
|
DNS / connect / bad URL. Step 8e wired the Erlang-side BIF
|
|
wrapper around it (`httpc:request/4`); see Progress log
|
|
entry for marshalling details. Step 8f (live HTTP dispatch
|
|
through `delivery_worker`) and Step 10c (peer-actor doc
|
|
fetch in `peer_actors`) are now unblocked.
|
|
|
|
3. **`erlang:send_after`-style timer primitive** — discovered
|
|
during Step 8b prep. The retry loop needs a way for the
|
|
delivery_worker to wake itself up after `backoff_for(N)`
|
|
seconds. Erlang's `erlang:send_after/3` is the standard
|
|
primitive; this port doesn't seem to register it (looked at
|
|
how `gen_server` handles `timeout` returns — it's a
|
|
message-loop self-cast that needs a delayed send). Belongs
|
|
to `loops/erlang` (Erlang runtime substrate). m2 captures the
|
|
retry semantics pure-functionally in 8b-pure so 8b-timer
|
|
becomes a 1-shot wiring when the primitive lands.
|
|
|
|
---
|
|
|
|
## Progress log
|
|
|
|
Newest first.
|
|
|
|
- **2026-06-07** — Step 8e (closes the BIF half of Step 8;
|
|
live HTTP dispatch in 8f next): `httpc:request/4` BIF wrapper
|
|
landed in `lib/erlang/runtime.sx` (briefing-allowed-exception
|
|
scope). Marshalling: Erlang URL binary → SX string via
|
|
`(list->string (map integer->char (get url :bytes)))`; Erlang
|
|
atom method → upcased name (`get` → `"GET"`) for HTTP wire
|
|
convention; binary method passes through verbatim; headers
|
|
proplist → SX dict via existing `er-proplist-to-dict`; body
|
|
binary → SX string. Result `{:status :headers :body}` marshalled
|
|
back to Erlang `{ok, Status, Headers::proplist, Body::binary}`
|
|
via `er-of-sx-deep` on headers (which produces the binary-keyed
|
|
proplist `er-dict-to-header-proplist` shape) and
|
|
`(er-mk-binary (map char->integer (string->list body)))` for
|
|
body. Non-binary URL / body raise `error:badarg`; the native
|
|
primitive raises `Eval_error` on DNS / connect / bad URL which
|
|
surfaces as an Erlang error marker the caller can catch.
|
|
Blockers #2 (native http-request primitive) entry updated:
|
|
RESOLVED by the loops/fed-prims → architecture merge that the
|
|
user authorized. Test: `next/tests/httpc_request.sh` 10/10 —
|
|
5 registration / validation cases (registration under
|
|
`httpc/request/4`, non-pure flag, no /1 arity, badarg on
|
|
non-binary URL, badarg on non-binary body) plus 5 live
|
|
roundtrip cases against a background `python3 -m http.server`
|
|
(Status 200, body bytes match `hello from python\n`, headers
|
|
proplist shape, 404 surfaces as `{ok, 404, ...}` not as an
|
|
error tuple, method passed as binary works). Adjacent gates:
|
|
Erlang conformance 761/761, http_multi_actor 44/44, follower_
|
|
graph 18/18, follow_lifecycle 9/9, backfill 20/20,
|
|
backfill_drain 6/6, http_listen_bif 5/5 — all green; pre-
|
|
existing cold-startup timeout sensitivity on http_get_format
|
|
(120s internal) and nx_kernel_pure (240s internal) confirmed
|
|
with git stash to NOT be caused by this change.
|
|
|
|
- **2026-06-07** — Step 9c (closes Step 9): Follow → Accept →
|
|
backfill drain (in-process). `maybe_auto_accept/3` now calls
|
|
`maybe_backfill/3` after the Accept publish: when
|
|
`:backfill_enabled` is true and the Follow envelope carries a
|
|
`:backfill` field, the receiver parses the mode, slices its
|
|
outbox via `backfill:slice/3` (Wrap=true), and enqueues every
|
|
entry onto the peer's delivery_worker. Silent skip when the
|
|
worker isn't registered (kernel manager lazy creation
|
|
upstream). 6/6 in `backfill_drain.sh`. Step 9 fully closed
|
|
(9a slicing + 9b ?since route + 9c Accept-drain). Live HTTP
|
|
dispatch of queued entries still gates on Blockers #2
|
|
(httpc).
|
|
|
|
- **2026-06-07** — Step 9b: outbox `?since=Cid` pagination.
|
|
`actor_outbox_response_for/3` in `http_server.erl` now reads
|
|
`?since=` from the query string via new `parse_since/1` +
|
|
`scan_param/2,3` + `skip_to_amp/1` (works whether the param
|
|
is first or after `&`), pre-filters entries through
|
|
`backfill:since_cid_entries/2`, then runs the existing page
|
|
slice on the filtered list. Unknown cid -> empty page -> tip-
|
|
only degrade. Three new cases in `http_multi_actor.sh` (44/44
|
|
total) cover filter, unknown-cid, combined since+page.
|
|
Latent issue surfaced + fixed in passing: the test was missing
|
|
`follower_graph` + `delivery` + `backfill` module loads
|
|
(since Step 7c made outbox depend on them); added all three.
|
|
|
|
- **2026-06-07** — Step 9a: pure-functional backfill slicing.
|
|
`next/kernel/backfill.erl` with `slice/2,3(Mode, LogState
|
|
[, Wrap])` returning the appropriate activity list. Modes
|
|
`none / full / {last_n, N} / {last_t, T, NowFn} /
|
|
{since_cid, Cid}` cover the §13.3 grammar; `wrap_backfill/1`
|
|
marks each entry `{backfilled, true}` (id preserved so the
|
|
receiver's replay defence still works). `parse_mode/1` lifts
|
|
the Follow activity's `:backfill` value (atom or proplist)
|
|
into the internal mode tuple; unknown shapes -> none. 20/20
|
|
in `backfill.sh`. Substrate gotchas re-confirmed:
|
|
`lists:nthtail/2` not registered (rolled `drop_n/2`); pattern-
|
|
alias `Pat = Var` not supported in this port (rewrote
|
|
`parse_mode/1` clauses with explicit deconstruction).
|
|
Conformance preserved at 761/761.
|
|
|
|
- **2026-06-07** — Step 11b: projection folds for the new verbs.
|
|
Two new modules in `next/kernel/`:
|
|
`announce_state.erl` (per-Cid announcer-set fold, set
|
|
semantics) and `endorsement_state.erl` (per-Cid + per-kind
|
|
+ per-actor counter, additive semantics). Both follow the
|
|
same plug shape as `actor_state` / `follower_graph` /
|
|
`delivery_state`: `fold_fn/0` returns a 2-arity Erlang fun
|
|
for `projection:start_link/3`. Non-matching activities pass
|
|
through unchanged. Read-side accessors cover both
|
|
enumeration (announcers_for / endorsers_for) and predicates
|
|
(has_announced / has_endorsed) so the feed/timeline layer
|
|
doesn't have to re-implement that logic. 19/19 in
|
|
`rich_verbs.sh`. Conformance preserved at 761/761.
|
|
|
|
- **2026-06-07** — Step 11a: Announce + Endorse genesis
|
|
activity-types. Two new DefineActivity SX files in
|
|
`next/genesis/activity-types/`: announce.sx (`:object` is a
|
|
CID string — the referenced activity to re-broadcast),
|
|
endorse.sx (`:object` is a CID, `:kind` is a string variant
|
|
like 'like' or 'share'). Manifest extended to 5 activity-types /
|
|
36 total entries. Bootstrap suite count assertions bumped
|
|
(`bootstrap_read`, `bootstrap_load`, `bootstrap_populate`,
|
|
`bootstrap_start`). `genesis_parse.sh` +4 cases. M1's Note
|
|
object-type is unchanged — Create{Note{...}} is still the
|
|
publish path. The runtime-publish demo (verb extensibility
|
|
via `Create{DefineActivity{...}}` at runtime) from M1 §9a
|
|
still works; these files are the genesis pre-shipped
|
|
variants for v2 baseline.
|
|
|
|
- **2026-06-07** — Step 10b: webfinger HTTP route.
|
|
`GET /.well-known/webfinger?resource=acct:user@host` lands in
|
|
`http_server.erl` next to the existing
|
|
`/.well-known/sx-capabilities` arm. New `handle_webfinger/1`
|
|
reads `:request_query` from Cfg (threaded via route/2 since
|
|
Step 4d), parses `resource=` + the acct: URI via
|
|
`discovery:parse_acct/1`, optionally matches against Cfg's
|
|
`:webfinger_host`, checks actor existence via the kernel atom
|
|
(when cfg'd), and renders the body via
|
|
`discovery:webfinger_body/3`. 10/10 in `webfinger_route.sh`.
|
|
Conformance + adjacent tests (`http_route` 11/11, `discovery`
|
|
12/12) preserved.
|
|
|
|
- **2026-06-07** — Step 10a: discovery primitives. New
|
|
`next/kernel/discovery.erl` parses acct: URIs
|
|
(prefix optional), synthesises `http://<host>/actors/<user>`,
|
|
and builds RFC 7033 webfinger JSON bodies. Hand-rolled byte
|
|
concatenation since this port has no JSON BIF and `<<"...">>`
|
|
string literals truncate to one byte (substrate gotcha). 12/12
|
|
in `discovery.sh`. The route wiring (10b) and peer-actor
|
|
fetch (10c) layer on top — 10c gates on Blockers #2.
|
|
|
|
- **2026-06-07** — Step 8c: delivery-state projection. New
|
|
`next/kernel/delivery_state.erl` folds enqueue / delivered /
|
|
failed / dead_lettered events into a per-peer worker-shaped
|
|
snapshot. State shape mirrors `delivery_worker:new/1`'s output
|
|
so a fresh gen_server can be hydrated from the projection on
|
|
kernel restart. The fail branch calls
|
|
`delivery_worker:backoff_for/1` directly, so the projection and
|
|
the live worker compute identical retry slots / dead-letter
|
|
thresholds. `fold_fn/0` plugs into `projection:start_link/3`
|
|
just like `actor_state` and `follower_graph`. 14/14 in
|
|
`delivery_state.sh`; delivery_worker.sh 17/17 + delivery_retry.sh
|
|
11/11 unchanged. Conformance preserved at 761/761. The
|
|
hydration helper that loads a worker's pure state from the
|
|
projection lands once 8b-timer can wire the live retry loop
|
|
(Blockers #3 still open).
|
|
|
|
- **2026-06-07** — Step 8b-pure: retry-time bookkeeping.
|
|
`delivery_worker` state shape gains `:next_retry` proplist
|
|
alongside `:attempts`. `record_failure_pure/3(Cid, Now, State)`
|
|
bumps the per-cid counter and computes the next retry as
|
|
`Now + backoff_for(NewAttempts)`. On the 6th failure
|
|
(`backoff_for` returns `dead_letter`) the matching activity
|
|
moves from `:pending` to `:dead_letter`. `record_success_pure/2`
|
|
clears both `:attempts` and `:next_retry` for the cid.
|
|
`next_due_pure/2(Now, State)` returns the cids whose retry
|
|
time has passed (insertion order preserved). 11/11 in
|
|
`delivery_retry.sh`. 8b-timer (real timer wiring via
|
|
`erlang:send_after`-style primitive) and 8e
|
|
(`httpc:request/4` BIF) hit substrate gaps — Blockers entries
|
|
added pointing to loops/erlang + loops/fed-prims. Conformance
|
|
preserved at 761/761.
|
|
|
|
- **2026-06-07** — Step 8d: outbox dispatches delivery_set to
|
|
workers. `outbox:publish/2` gained `dispatch_deliveries/3` and
|
|
`enqueue_each/2`: after `log:append` + projection broadcast,
|
|
the resolved `delivery_set` is walked and each registered
|
|
peer-id atom's `delivery_worker:enqueue(PeerId, Activity)` is
|
|
called. Missing workers (no `erlang:whereis`) are silently
|
|
skipped. Gated by Context's `{dispatch_deliveries, true}` —
|
|
default off so every M1 outbox caller stays back-compat. 7/7
|
|
in `delivery_dispatch.sh`; `outbox_publish.sh` +
|
|
`delivery_worker.sh` both still 17/17. Conformance preserved
|
|
at 761/761 from the Step 8a baseline.
|
|
|
|
- **2026-06-07** — Step 8a: delivery_worker skeleton.
|
|
`next/kernel/delivery_worker.erl` with pure-functional state +
|
|
enqueue / drain / deliver_one + backoff schedule (30s / 5m /
|
|
30m / 6h / 24h then dead-letter, per design §13.4). gen_server
|
|
wrapper exposes the same APIs under the peer-id atom. dispatch
|
|
is a caller-supplied `:dispatch_fn` fun — Step 8b layers the
|
|
retry timer, Step 8c persists the queue, Step 8d wires
|
|
`outbox:publish/2` to dispatch, Step 8e brings the
|
|
`httpc:request/4` BIF (substrate exception per briefing), Step
|
|
8f closes with live HTTP. 17/17 in `delivery_worker.sh`.
|
|
Conformance 761/761.
|
|
|
|
- **2026-06-07** — Step 7c (closes Step 7): outbox-side
|
|
delivery_set integration. `outbox:publish/2` computes the
|
|
audience-resolved delivery set after sign + log and stashes
|
|
it in the Result proplist as `{delivery_set, [ActorId, ...]}`.
|
|
New `compute_delivery_set/3(Request, Signed, Context)`
|
|
threads `:follower_graph` from Context through to
|
|
`delivery:delivery_set/3`. `recipients_envelope/2` synthesises
|
|
a minimal envelope from the Request's `:to`/`:cc` + Signed's
|
|
`:actor` so the existing delivery API works unchanged
|
|
(envelope construct/4 doesn't carry the audience fields
|
|
through). 17/17 in `outbox_publish.sh` (+4 new: empty-default,
|
|
explicit-:to, followers-symbol-via-graph, self-suppression).
|
|
Module load order shifted from epoch 5 to epoch 7 to make
|
|
room for follower_graph + delivery; internal sx_server
|
|
timeout bumped 240s → 480s. Step 7 fully closed (7a delivery
|
|
module + 7b public expansion + 7c outbox integration).
|
|
|
|
- **2026-06-06** — Step 7b: public audience expansion.
|
|
`delivery:expand_audience(public, Sender, Graph)` now returns
|
|
the sender's followers (same as `followers`) — per design
|
|
§13.4 that's the practical fan-out semantics for an open
|
|
social network. The explicit shared-inbox peer-instance model
|
|
defers to v3. 19/19 in `delivery_set.sh` (+2 new cases:
|
|
public-with-empty-graph, public+followers-dedupe; +1 case
|
|
updated from the v2 placeholder). Conformance 761/761
|
|
preserved.
|
|
|
|
- **2026-06-06** — Step 7a: audience-resolving delivery set.
|
|
New `next/kernel/delivery.erl`: `delivery_set/2,3(Activity,
|
|
KernelState[, FollowerGraph])` returns a deduplicated list of
|
|
ActorId atoms — the targets an outbound activity needs to be
|
|
POSTed to. Sources: `:to` and `:cc` fields (single atom or
|
|
list, atoms or audience symbols), plus expansion of `followers`
|
|
via the supplied follower_graph state. `public` placeholder
|
|
returns `[]` for v2; Step 7b will populate via a known-
|
|
peer-instance set. Self-delivery suppressed. ActorIds for now —
|
|
Step 8 resolves each entry to `{PeerInstanceUrl, ActorId}` via
|
|
peer-actors cache. 17/17 in `delivery_set.sh`. Conformance
|
|
761/761. Lives in its own module (not inside `outbox`) so the
|
|
Step 8 delivery-queue gen_server has a clean home.
|
|
|
|
- **2026-06-06** — Step 6c (closes Step 6): auto-Accept publish on
|
|
Follow ingestion. New `maybe_auto_accept/3` in `http_server.erl`
|
|
fires after successful inbox append + projection broadcast:
|
|
if Cfg has `{auto_accept_follows, true}` and the activity is a
|
|
`Follow`, construct `[{type, accept}, {object, OriginalFollow}]`
|
|
and route through `nx_kernel:publish_to/2`. The publish goes
|
|
through the full outbox pipeline (construct + sign + log +
|
|
projection broadcast), so when the target's outbox `:projections`
|
|
share the same follower_graph projection that inbox broadcasts
|
|
into, the bilateral relationship fold-converges automatically
|
|
(`alice.followers = [bob]`, `bob.following = [alice]`, both
|
|
pending lists clear). Default off; bad-sig / non-Follow
|
|
ingestion short-circuits before the Accept attempt. 9/9 in
|
|
`auto_accept.sh`. Conformance 761/761. Step 6 fully closed
|
|
(6a + 6b + 6c).
|
|
|
|
- **2026-06-06** — Step 6b: wire follower_graph fold to the
|
|
inbox handler. New `broadcast_to_inbox_projections/2` in
|
|
`http_server.erl` casts every successfully-ingested activity
|
|
into each `:inbox_projections` Cfg entry via
|
|
`projection:async_fold/2`. Fire-and-forget so the inbox
|
|
handler doesn't block on fold processing. Empty / absent
|
|
`:inbox_projections` is a no-op (back-compat with Steps 5d
|
|
callers). 9/9 in `follow_lifecycle.sh` covering 202 + bilateral
|
|
pending-state mutation + bad-sig short-circuit + multi-peer
|
|
+ end-to-end projection convergence on Follow+Accept. Conformance
|
|
761/761. Auto-Accept publish (the receiving kernel responds
|
|
with a signed Accept) is Step 6c.
|
|
|
|
- **2026-06-06** — Step 6a: follower-graph projection
|
|
(`follower_graph.erl`). Pure-functional fold over Follow /
|
|
Accept / Reject / Undo activities per design §13.2. State is a
|
|
proplist keyed by ActorId carrying `{following, followers,
|
|
pending_outbound, pending_inbound}` lists. Follow pushes onto
|
|
pendings; Accept moves both sides from pendings into the
|
|
permanent lists; Reject just clears pendings; Undo drops the
|
|
pair everywhere (and only the Follow's original actor can Undo).
|
|
Self-follow is a no-op; duplicate Follow is idempotent;
|
|
Accept/Reject/Undo of a non-Follow `:object` passes through.
|
|
`fold_fn/0` is the standard 2-arity fun for
|
|
`projection:start_link/3` (same shape as `actor_state` and
|
|
`define_registry`). 18/18 in `follower_graph.sh`. Conformance
|
|
761/761.
|
|
|
|
- **2026-06-06** — Step 5d: POST /actors/<id>/inbox real ingestion.
|
|
`route/2` now special-cases POST `/actors/<id>/inbox` next to POST
|
|
`/activity` so the body + full Cfg reach the new
|
|
`handle_inbox_post/3` handler. Flow:
|
|
`kernel_has_actor` -> `decode_activity` (term_codec wire format)
|
|
-> `resolve_peer_as` (Cfg `:peer_as` map > `:peer_actors` srv >
|
|
`:peer_fetch_fn` fallback) -> `pipeline:validate_inbound/3` ->
|
|
`nx_kernel:append_inbox`. Status codes 202 / 401 / 404 / 422
|
|
per design §16.1. v1 stub `actor_post/1` removed; M1
|
|
`actor_inbox_post_response/0` kept for response shape composition.
|
|
Projection broadcast on inbox success intentionally deferred to a
|
|
follow-up. `inbox.sh` 11/11 (basic ingestion: happy path / shape
|
|
/ sig / replay / unknown-target / multi-message);
|
|
`inbox_peer_resolution.sh` 6/6 (peer-AS resolution variants).
|
|
Split into two files because cumulative per-epoch kernel
|
|
start_link + outbox construct + term_codec encode pushed a
|
|
single suite past the wall-clock budget. http_server.erl now
|
|
1181 lines — load time on this Erlang port scales superlinearly
|
|
with function count, so eight http_*.sh tests' internal sx_server
|
|
timeout bumped 60s → 360s. Conformance 761/761.
|
|
|
|
- **2026-06-06** — Step 5c: peer-actors cache (`peer_actors.erl`).
|
|
Pure-functional cache of `{PeerActorId, PeerAS}` entries with
|
|
the load-bearing `lookup_or_fetch/3(PeerId, FetchFn, State)`
|
|
entry: cache hit returns stored PeerAS unchanged; miss calls
|
|
`FetchFn(PeerId)`, stores success, returns `{ok, PeerAS,
|
|
NewState}`. Fetch errors don't poison the cache so callers can
|
|
retry on transient HTTP failures. gen_server wrapper exposes
|
|
the same shape under registered name `peer_actors`;
|
|
`start_link/1` accepts an initial proplist for tests.
|
|
Per-design v2 fetches are synchronous over plaintext HTTP; the
|
|
actual http-client call lands in Step 5d. 19/19 in
|
|
`peer_actors.sh`. Conformance 761/761. 139/139 across 9
|
|
Step-5-adjacent suites.
|
|
|
|
- **2026-06-06** — Step 5b: federation inbound pipeline.
|
|
`pipeline:validate_inbound/3(Activity, PeerAS, InboxLog)` runs
|
|
`stage_envelope` → `stage_signature(PeerAS)` → `stage_replay(InboxLog)`
|
|
in order, halting on first failure. New `inbound_stages/2`
|
|
helper returns the 3-stage list. M1's `validate_inbound/1` +
|
|
static `inbound_stages/0` (envelope-only) preserved for outbox-
|
|
side callers. 14/14 in `inbox_pipeline.sh` covering happy path,
|
|
bad shape, missing :signature, wrong peer AS, replay against
|
|
inbox, distinct activities both verify, stage short-circuit
|
|
ordering. Sig verification routes through the peer's AS (not the
|
|
local kernel's) — Step 5c will wire the cache lookup. Conformance
|
|
761/761. 130/130 across 10 Step-5-adjacent suites
|
|
(pipeline_envelope, pipeline_signature, pipeline_replay,
|
|
pipeline_driver, inbox_pipeline, inbox_bucket, nx_kernel_multi,
|
|
bootstrap_start, http_publish, outbox_publish, smoke_app_pure).
|
|
|
|
- **2026-06-06** — Step 5a: per-actor :actor_inbox log bucket.
|
|
`nx_kernel.erl` `add_actor/4` now opens a fresh log via
|
|
`log:open/2` with a distinct `inbox_base_stub()` for each new
|
|
bucket and stores it as `{actor_inbox, LogState}` alongside the
|
|
existing outbox `:log`. Pure exports `actor_inbox_state/2`,
|
|
`actor_inbox_tip/2`, `append_to_actor_inbox/3` + gen_server
|
|
exports `inbox_tip_for/1`, `inbox_state_for/1`, `append_inbox/2`.
|
|
`log:append/2` is `(LogState, Activity) -> {ok, NewState, Seq}` —
|
|
noted for future iterations. Inbox / outbox tips are fully
|
|
independent. `next/tests/inbox_bucket.sh` 14/14. Conformance
|
|
761/761. 125/125 across 7 Step-5-adjacent suites
|
|
(inbox_bucket, nx_kernel_multi, nx_kernel_server,
|
|
bootstrap_start, http_publish, http_multi_actor, actor_lifecycle,
|
|
smoke_app_pure).
|
|
|
|
- **2026-06-06** — Step 4d: per-actor outbox listing + pagination.
|
|
New `nx_kernel:log_state_for/1` gen_server export returns
|
|
`{ok, LogState}` for an actor. `actor_outbox_response_for/3`
|
|
now extracts `{Tip, Entries}` via `kernel_actor_log_data/2`,
|
|
parses `?page=N` from the Req's `:query` field (threaded
|
|
through Cfg as `{request_query, Q}`), and renders a paged
|
|
body. Text body adds `page: N\nitem: <cid>\n...`; JSON adds
|
|
`"page":N,"items":[...]`; SX adds `:page N :items (...)`.
|
|
Empty pages (out-of-range or actor-with-no-publishes) degrade
|
|
back to the 4c tip-only shape, preserving epochs 50-57.
|
|
`page_size/0` is 5 for tests (production picks 20+). 8 new
|
|
cases in `http_multi_actor.sh` (41/41 total). Conformance
|
|
761/761. 117/117 across 11 Step-4-adjacent suites. **Gotcha**
|
|
noted: named recursive funs `fun F(...) -> ... F(...) end`
|
|
fail with "fun-ref syntax not yet supported"; `binary:matches/2`
|
|
and `lists:foreach/2` aren't registered in this substrate.
|
|
|
|
- **2026-06-06** — Step 4c: route/3 with kernel access.
|
|
`http_server:route/3(Req, Cfg, Kernel)` folds the kernel
|
|
reference into Cfg as `{kernel, _}`. Dispatch chain refactored
|
|
to thread Cfg through to per-actor sub-resource handlers.
|
|
Outbox handler reads `:kernel` and renders `tip: <N>` (in
|
|
text / JSON / SX content-negotiated variants) when the actor
|
|
exists; falls back to the 4a stub otherwise. Substrate quirk
|
|
found: `try`/`of`/`catch` around `gen_server:call` deadlocks
|
|
in this port's scheduler — bare call + integer guard works.
|
|
Inbox / followers / following handlers accept Cfg but ignore
|
|
it; real state lookup lands in 4d/4e/Step 5+. 8 new cases in
|
|
`http_multi_actor.sh` (33/33 total). Conformance 761/761.
|
|
121/121 across 10 Step-4-adjacent suites. **Gotcha** noted
|
|
for future iterations: avoid try/catch wrapping gen_server
|
|
calls in this substrate.
|
|
|
|
- **2026-06-06** — Step 4b: token -> ActorId map. Cfg's `:tokens`
|
|
proplist (`[{Token, ActorId}, ...]`) maps bearer tokens to
|
|
per-actor publishers. `handle_post_activity` threads the
|
|
resolved `ActorRef` to `publish_if_kernel/3` which calls
|
|
`nx_kernel:publish_to/2` for explicit actor ids and `publish/1`
|
|
for the back-compat `legacy` atom (M1's `:publish_token`
|
|
single-token field still works as-is). When both fields are
|
|
present, `:tokens` takes precedence; `:publish_token` is the
|
|
fallback on miss. Dead `expected_token/1` helper removed. 8
|
|
new cases in `http_multi_actor.sh` (25/25 total) covering
|
|
two-actor token routing, log-tip isolation, interleaved
|
|
publishes, bad-token 401, back-compat coexistence, no-kernel
|
|
stub path. Conformance 761/761 preserved. 116/116 across 10
|
|
Step-4-adjacent suites.
|
|
|
|
- **2026-06-06** — Step 4a: per-actor HTTP sub-paths. New
|
|
`split_first_slash/1` helper lets GET / POST `/actors/<id>/...`
|
|
paths route on the sub-segment (`outbox`, `inbox`, `followers`,
|
|
`following`). Four new content-negotiated response stubs
|
|
(`actor_outbox_response_for/2`, `actor_inbox_get_response_for/2`,
|
|
`actor_followers_response_for/2`, `actor_following_response_for/2`)
|
|
with text / json / activity_json / sx variants, mirroring the
|
|
existing `actor_doc_response_for/2` shape. POST
|
|
`/actors/<id>/inbox` returns a 202 Accepted stub
|
|
(`actor_inbox_post_response/0` + `accepted_response/1`); real
|
|
ingestion pipeline lands in Step 5. Unknown sub-paths return
|
|
404. Bare `/actors/<id>` keeps the M1 actor-doc arm intact —
|
|
`http_route` and `http_post_format` regression suites unchanged
|
|
(10/10 each). 17/17 in `http_multi_actor.sh`. Conformance
|
|
761/761 preserved. 120/120 across 10 Step-4-adjacent suites.
|
|
|
|
- **2026-06-06** — Step 3 (closes Step 3): key rotation via Update.
|
|
`actor_state.erl` `fold_update` routes patches through
|
|
`apply_patch/3` which special-cases `{add_publicKey, KeyProplist}`
|
|
(append + default `:created` to activity's `:published`) and
|
|
`{supersede, OldKeyId}` (mark `:superseded_at`, idempotent).
|
|
Other patch entries still last-write-wins per key. New exports
|
|
`key_history/1`, `active_keys_at/2`, `find_key_by_id/2` give the
|
|
projection-driven view that `envelope:verify_signature/2`
|
|
consumes for time-aware lookup. Rotation-purpose schema gating
|
|
(`rotate-key` purpose check on the rotation activity itself)
|
|
deferred to Step 5 (peer-side stage_signature). `key_rotation.sh`
|
|
16/16 covers rotation arithmetic, key_history preservation,
|
|
active_keys_at at T=pre, T=rotation, T=post, and live
|
|
`envelope:verify_signature/2` round-trips for pre / post / cross
|
|
scenarios including the negative-case post-rotation K1 sig.
|
|
Conformance 761/761 preserved. 132/132 across 9 Step-3-adjacent
|
|
suites (key_rotation, actor_state_pure, actor_lifecycle,
|
|
envelope_sig, envelope_shape, envelope_canonical, nx_kernel_multi,
|
|
bootstrap_start, smoke_app_pure).
|
|
|
|
- **2026-06-06** — Step 2c (closes Step 2): `bootstrap_actor/4` +
|
|
end-to-end `actor_lifecycle.sh`. New pure-functional export
|
|
`nx_kernel:bootstrap_actor/4(ActorId, Profile, KeySpec, State)`
|
|
adds an actor bucket via `add_actor/4`, derives the kernel AS
|
|
proplist from `Profile`'s `:public_keys`, builds a Create
|
|
envelope wrapping the profile's `:type` (defaults `person`) +
|
|
field set, and calls `publish/3`. gen_server variant
|
|
`bootstrap_actor/3` for live-kernel use plus a corresponding
|
|
`handle_call` branch. `actor_lifecycle.sh` 15/15 covers pure
|
|
bootstrap (log_tip advances, Create-shape, dup detection),
|
|
two-actor independence, gen_server bootstrap, and
|
|
`actor_state` projection capture for Person + Service + Group.
|
|
Step 2 fully closed (2a + 2b + 2c). Conformance 761/761.
|
|
146/146 across 10 Step-2-adjacent suites.
|
|
|
|
- **2026-06-06** — Step 2b: actor-state projection Erlang module.
|
|
New `next/kernel/actor_state.erl` with `fold/2` over Create / Update
|
|
/ Move activities. Profile is a property list of `:type / :name /
|
|
:preferredUsername / :summary / :icon / :public_keys / :moved_to /
|
|
:created`. Create captures fields and `:published` as `:created`;
|
|
duplicate Create is no-overwrite; non-actor Creates and `:actor`-
|
|
less envelopes pass through. Update last-write-wins per patch key.
|
|
Move records `:moved_to`. `fold_fn/0` is a 2-arity Erlang fun for
|
|
`projection:start_link/3` (structural twin of `define_registry`).
|
|
`next/tests/actor_state_pure.sh` 19/19. Conformance 761/761.
|
|
Step-2-adjacent no-regression gate 106/106 across 6 suites.
|
|
|
|
- **2026-06-06** — Step 2a: genesis Person/Service/Group object-
|
|
types. Three new SX files in `next/genesis/object-types/` with
|
|
the same shape as `note.sx` / `sx-artifact.sx` (`:name`, `:doc`,
|
|
`:schema` checking `(string? (-> obj :name))`). Manifest extended
|
|
to 13 object-types / 34 total entries. `genesis_parse.sh` +7
|
|
cases (57/57). Hardcoded counts bumped in `bootstrap_read.sh`,
|
|
`bootstrap_load.sh`, `bootstrap_populate.sh`, `bootstrap_start.sh`
|
|
(66/66 across those four). `bootstrap_build.sh` 12/12 (bundle CID
|
|
computed dynamically). Conformance 761/761 preserved. 211 / 211
|
|
across 12 Step-2-adjacent suites.
|
|
|
|
- **2026-06-06** — Step 1b: gen_server multi-actor calls.
|
|
`nx_kernel` exports `add_actor/3`, `publish_to/2`, `log_tip_for/1`,
|
|
`actors/0`, `state_for/1`, `bucket_for/1`,
|
|
`with_projections_for/2` — each is a `gen_server:call` delegating
|
|
to the pure-functional bucket API from 1a. Existing single-actor
|
|
calls untouched. `nx_kernel_multi.sh` extended with 9 gen_server
|
|
cases (26 total); 134 / 134 across 12 nx_kernel-adjacent + http
|
|
suites. Conformance 761/761 preserved. Per-actor mailbox sharding
|
|
noted as forward-looking — current single gen_server serialises
|
|
publishes across actors, which is fine for Steps 1-3 (single-actor
|
|
HTTP endpoints) and is naturally untangled by Step 4's per-actor
|
|
routing.
|
|
|
|
- **2026-06-06** — Step 1a: per-actor bucket refactor of `nx_kernel`.
|
|
State shape now `[{actors, [{Id, Bucket}, …]}, {next_actor_seq, N}]`;
|
|
added pure-functional multi-actor APIs (`new/0`, `add_actor/4`,
|
|
`has_actor/2`, `actors/1`, `publish/3`, per-actor accessors,
|
|
`with_actor_projections/3`). Legacy single-actor accessors
|
|
preserved as bucket-0 lookups so every M1 test continues to
|
|
pass via `bootstrap:start/3` → `new/3` → first-bucket read.
|
|
Local `has_keyed`/`find_keyed`/`set_keyed`/`set_bucket` helpers
|
|
cover the keyed-list ops since `lists:keymember`/`keyfind` aren't
|
|
registered in this substrate. New test suite
|
|
`next/tests/nx_kernel_multi.sh` 17/17; all M1 nx_kernel-adjacent
|
|
suites green (`bootstrap_start`, `nx_kernel_server`, `http_publish`,
|
|
`smoke_app_pure`, `http_post_format`, `http_publish_fold`,
|
|
`http_marshal`). Erlang conformance 761/761 preserved.
|