923 lines
33 KiB
Markdown
923 lines
33 KiB
Markdown
# fed-sx Milestone 1 — Kernel + Registries + Pin Smoke Test
|
|
|
|
Concrete implementation plan for the smallest fed-sx that proves the architecture
|
|
works end-to-end. Reference: `plans/fed-sx-design.md`. Prerequisite: Erlang-on-SX
|
|
Phases 7 (hot reload) + 8 (FFI BIFs).
|
|
|
|
## Goal
|
|
|
|
Ship a single-instance, single-actor fed-sx server that:
|
|
|
|
1. Boots from a verified genesis bundle.
|
|
2. Accepts and durably appends signed activities via `POST /activity`.
|
|
3. Folds them into projections in real time.
|
|
4. Serves AP-standard endpoints (actor, outbox, artifacts, capabilities).
|
|
5. Demonstrates **two extensibility proof-points** end-to-end with zero kernel
|
|
code changes between definition and use:
|
|
- **Verb extensibility** (§5 meta-level): publish `DefineActivity{Pin}` +
|
|
`DefineProjection{pin-state}`, then publish a `Pin` activity, observe it
|
|
validated and projected.
|
|
- **Reactive application extensibility** (§§18-19): publish
|
|
`DefineSubscription{Topic}` + `Subscribe{topic: smoketest}` +
|
|
`DefineTrigger{when: that subscription, then: publish TestEcho}`, then
|
|
publish a tagged Note, observe the subscription match, the trigger fire,
|
|
and the derived activity appear in the outbox.
|
|
|
|
Federation, multi-actor, advanced verbs, IPFS, browser UI, operator dashboard
|
|
are **explicitly v2**.
|
|
|
|
## Non-goals (what milestone 1 deliberately does NOT do)
|
|
|
|
- **Federation.** No `POST /inbox` from peers, no `Follow`, no delivery queue, no
|
|
webfinger discovery flow. Single instance only.
|
|
- **Multi-actor.** Single domain actor (`acct:next@next.rose-ash.com`).
|
|
- **IPFS / S3 storage backends.** Files on disk only.
|
|
- **Advanced verbs.** No `Endorse`, `Supersede`, `Test`, `Build`, `Compose`,
|
|
`Note`, `Announce`. Only the four bootstrap verbs (`Create`, `Update`, `Delete`)
|
|
plus a defined-from-the-log `Pin` for the smoke test. (`Announce` deferred —
|
|
no use case until federation exists.)
|
|
- **Browser UI.** Curl-shaped API only.
|
|
- **Operator dashboard, quarantine UX.** Logs only.
|
|
- **Performance work.** Functional correctness first; perf when measured.
|
|
- **Cross-host conformance test corpus.** Only the OCaml/Erlang-on-SX host runs
|
|
fed-sx in v1; conformance suite for other hosts is v2.
|
|
|
|
## Architecture summary
|
|
|
|
```
|
|
POST /activity
|
|
│
|
|
▼
|
|
┌──────────────────────────┐
|
|
│ HTTP server (Erlang-on-SX)│
|
|
└─────────────┬─────────────┘
|
|
│
|
|
┌─────────────▼──────────────┐
|
|
│ Validation pipeline driver │
|
|
│ (envelope→sig→schema→...) │
|
|
└─────────────┬──────────────┘
|
|
│
|
|
┌─────────────▼──────────────┐
|
|
│ Log append (JSONL segment) │ ← canonical
|
|
└─────────────┬──────────────┘
|
|
│
|
|
┌─────────────▼──────────────┐
|
|
│ Projection workers │ ← gen_server per
|
|
│ (fold scheduler) │ projection
|
|
└─────────────────────────────┘
|
|
│
|
|
▼
|
|
Projection state
|
|
(queryable via HTTP)
|
|
|
|
Native primitives (Erlang-on-SX BIFs from Phase 8):
|
|
crypto:* cid:* fs:* http:* sqlite:*
|
|
|
|
Genesis bundle (binary-embedded SX):
|
|
activity-types object-types projections
|
|
validators codecs sig-suites
|
|
```
|
|
|
|
## Build order
|
|
|
|
Eight steps in dependency order. Each step has concrete deliverables, testable
|
|
in isolation, and a clear acceptance check.
|
|
|
|
| Step | Title | Depends on |
|
|
|------|-------|------------|
|
|
| **1** | Repo skeleton + canonical CID computation | Phase 8 (cid BIFs) |
|
|
| **2** | Activity envelope + signature verify | Phase 8 (crypto BIFs) |
|
|
| **3** | JSONL log + sequence numbers | Phase 8 (fs BIFs) |
|
|
| **4** | Genesis bundle (SX sources + bundling + CID verification) | Step 1 |
|
|
| **5** | Registry mechanism + bootstrap-projection dispatch | Steps 2, 4 |
|
|
| **6** | Validation pipeline driver + `POST /activity` | Steps 2, 3, 5 |
|
|
| **7** | Projection scheduler (gen_server per projection) | Steps 5, 6 |
|
|
| **8** | HTTP server, AP endpoints, projection queries | Steps 6, 7 |
|
|
| **9** | Smoke tests (Pin verb + reactive application) | Steps 1-8 |
|
|
|
|
---
|
|
|
|
## Step 1 — Repo skeleton + canonical CID
|
|
|
|
**Deliverables:**
|
|
|
|
```
|
|
next/
|
|
├── README.md # what this is
|
|
├── kernel/ # Erlang-on-SX
|
|
│ └── (empty for now)
|
|
├── genesis/ # core SX bootstrap definitions
|
|
│ └── (empty for now)
|
|
├── tests/ # smoke test scripts
|
|
│ └── (empty for now)
|
|
└── data/ # gitignored runtime state
|
|
├── log/
|
|
├── objects/
|
|
├── snapshots/
|
|
├── indexes/
|
|
└── keys/
|
|
```
|
|
|
|
Plus one Erlang-on-SX module:
|
|
|
|
```erlang
|
|
% next/kernel/cid.erl
|
|
-module(cid).
|
|
-export([from_sx/1, to_string/1, from_string/1, equals/2]).
|
|
|
|
from_sx(SxValue) ->
|
|
Cbor = cid:cbor_encode(canonicalize_sx(SxValue)),
|
|
Hash = crypto:sha2_256(Cbor),
|
|
cid:from_bytes(<<"raw">>, Hash). % defaults to dag-cbor codec
|
|
|
|
canonicalize_sx(V) -> ... % sorts dict keys, normalizes strings
|
|
```
|
|
|
|
**Tests:**
|
|
- Same SX value → same CID across multiple invocations.
|
|
- Different SX values → different CIDs.
|
|
- Whitespace/comment differences in source → identical CIDs (parsed AST identical).
|
|
- Reordered dict keys → identical CIDs (sorted-key canonicalization).
|
|
- Cross-host parity (just OCaml host for v1, but write the test so adding hosts is mechanical).
|
|
|
|
**Acceptance:** `bash next/tests/cid.sh` passes 10+ cases.
|
|
|
|
---
|
|
|
|
## Step 2 — Activity envelope + signature verify
|
|
|
|
**Deliverables:**
|
|
|
|
```erlang
|
|
% next/kernel/envelope.erl
|
|
-module(envelope).
|
|
-export([validate_shape/1, canonical_bytes/1, verify_signature/2]).
|
|
|
|
% Envelope shape per design §3.1:
|
|
% #{id, type, actor, published, to, cc, audience_extras,
|
|
% object | target | origin | result,
|
|
% capabilities_required, proofs, signature}
|
|
validate_shape(Activity) -> ok | {error, Reason}.
|
|
|
|
canonical_bytes(Activity) ->
|
|
% Strip signature, canonicalize via dag-cbor, return bytes for sig coverage
|
|
Stripped = maps:remove(signature, Activity),
|
|
cid:cbor_encode(canonicalize_for_sig(Stripped)).
|
|
|
|
verify_signature(Activity, ActorState) ->
|
|
% Time-aware: find key with id == sig.key_id that was active at published
|
|
% Per design §9.6
|
|
...
|
|
```
|
|
|
|
**Tests:**
|
|
- Envelope shape: required fields present (id, type, actor, published, signature)
|
|
- Envelope shape: type is a known activity-type or unknown-but-string
|
|
- Envelope shape: signature has key_id, algorithm, value
|
|
- Sig verify: valid RSA-SHA256 signature against published key → ok
|
|
- Sig verify: valid Ed25519 signature → ok
|
|
- Sig verify: tampered envelope → fail
|
|
- Sig verify: key superseded before activity timestamp → fail
|
|
- Sig verify: key superseded after activity timestamp → ok (historical valid)
|
|
|
|
**Acceptance:** `bash next/tests/envelope.sh` passes 15+ cases.
|
|
|
|
---
|
|
|
|
## Step 3 — JSONL log + sequence numbers
|
|
|
|
**Deliverables:**
|
|
|
|
```erlang
|
|
% next/kernel/log.erl
|
|
-module(log).
|
|
-export([open/1, append/2, read_segment/2, tip/1, replay/3]).
|
|
|
|
% Per design §15.2: per-actor outbox, segments cap ~64MB,
|
|
% format = JSONL (one canonical JSON-LD activity per line)
|
|
|
|
open(ActorId) ->
|
|
BasePath = log_path_for_actor(ActorId),
|
|
fs:mkdir_p(BasePath),
|
|
{ok, #{base => BasePath, current => current_segment(BasePath), seq => next_seq(BasePath)}}.
|
|
|
|
append(LogState, Activity) ->
|
|
Json = jsonld:encode(Activity),
|
|
Path = current_segment_path(LogState),
|
|
Line = <<Json/binary, "\n">>,
|
|
fs:append_file(Path, Line),
|
|
NewSeq = LogState#{seq := LogState.seq + 1},
|
|
rotate_if_needed(NewSeq).
|
|
|
|
% replay/3 calls Fun(Activity, Acc) for every activity in chronological order
|
|
replay(LogState, InitAcc, Fun) -> ...
|
|
```
|
|
|
|
**Tests:**
|
|
- Append + read back gives identical activity (round-trip).
|
|
- Sequence numbers monotonic and gap-free per actor.
|
|
- Segment rotation at size threshold.
|
|
- Replay visits all activities in append order across multiple segments.
|
|
- Restart preserves tip pointer (seq number resumes correctly).
|
|
- Concurrent appends (using gen_server-mediated access) are serialized correctly.
|
|
|
|
**Acceptance:** `bash next/tests/log.sh` passes 10+ cases.
|
|
|
|
---
|
|
|
|
## Step 4 — Genesis bundle
|
|
|
|
**Deliverables:**
|
|
|
|
Genesis bundle SX sources (per design §12.2). Each is a small SX file authored
|
|
by hand for the bootstrap set:
|
|
|
|
```
|
|
next/genesis/
|
|
├── manifest.sx # bundle root: lists all definitions
|
|
├── activity-types/
|
|
│ ├── create.sx # DefineActivity{name: "Create", ...}
|
|
│ ├── update.sx
|
|
│ └── delete.sx
|
|
├── object-types/
|
|
│ ├── sx-artifact.sx
|
|
│ ├── note.sx
|
|
│ ├── tombstone.sx
|
|
│ ├── define-activity.sx # DefineObject for the Define* meta types
|
|
│ ├── define-object.sx
|
|
│ ├── define-projection.sx
|
|
│ ├── define-validator.sx
|
|
│ ├── define-codec.sx
|
|
│ ├── define-sig-suite.sx
|
|
│ └── snapshot.sx
|
|
├── projections/
|
|
│ ├── activity-log.sx # identity projection
|
|
│ ├── by-type.sx
|
|
│ ├── by-actor.sx
|
|
│ ├── by-object.sx
|
|
│ ├── actor-state.sx
|
|
│ ├── define-registry.sx # the chicken-and-egg projection
|
|
│ └── audience-graph.sx
|
|
├── validators/
|
|
│ ├── envelope-shape.sx
|
|
│ ├── signature.sx
|
|
│ └── type-schema.sx
|
|
├── codecs/
|
|
│ ├── dag-cbor.sx # delegates to cid:cbor_encode/decode BIFs
|
|
│ ├── raw.sx
|
|
│ └── dag-json.sx
|
|
├── sig-suites/
|
|
│ ├── rsa-sha256-2018.sx
|
|
│ └── ed25519-2020.sx
|
|
└── audience/
|
|
├── public.sx
|
|
├── followers.sx
|
|
└── direct.sx
|
|
```
|
|
|
|
Plus a build-time bundler:
|
|
|
|
```erlang
|
|
% next/kernel/bootstrap.erl
|
|
-module(bootstrap).
|
|
-export([build_genesis/1, verify_genesis/1, load_genesis/1]).
|
|
|
|
build_genesis(SourceDir) ->
|
|
% Walk SourceDir, parse each .sx file, build a single dag-cbor bundle,
|
|
% compute its CID, write bundle.cbor + CID to data/genesis/
|
|
...
|
|
|
|
verify_genesis(BundlePath) ->
|
|
% Compute CID of the bundle as loaded; compare to expected (hardcoded
|
|
% in the kernel binary). Mismatch → halt.
|
|
...
|
|
|
|
load_genesis(BundlePath) ->
|
|
% Parse the bundle, register all definitions in the in-memory registry
|
|
...
|
|
```
|
|
|
|
**Tests:**
|
|
- All genesis SX files parse cleanly.
|
|
- Bundle CID is deterministic (rebuild same sources → same CID).
|
|
- Bundle reload reproduces the exact same registry state.
|
|
- Tampered bundle → `verify_genesis` returns `{error, cid_mismatch}`.
|
|
|
|
**Acceptance:** `bash next/tests/bootstrap.sh` passes; `next/data/genesis/bundle.cbor`
|
|
created with a known stable CID.
|
|
|
|
---
|
|
|
|
## Step 5 — Registry mechanism + bootstrap dispatch
|
|
|
|
**Deliverables:**
|
|
|
|
Registries are gen_servers, one per kind, each holding the active version map:
|
|
|
|
```erlang
|
|
% next/kernel/registry.erl
|
|
-module(registry).
|
|
-behaviour(gen_server).
|
|
-export([start_link/0, lookup/2, register/3, list/1]).
|
|
% Internal state:
|
|
% #{activity_types => #{Name => #{cid, schema_fn, semantics_fn, supersedes}},
|
|
% object_types => ...,
|
|
% projections => ...,
|
|
% validators => ...,
|
|
% codecs => ...,
|
|
% sig_suites => ...,
|
|
% ...}
|
|
|
|
lookup(Kind, Name) -> {ok, Entry} | {error, not_found}.
|
|
register(Kind, Name, Entry) -> ok | {error, Reason}.
|
|
list(Kind) -> [#{name, cid}].
|
|
```
|
|
|
|
The `define-registry` projection's fold updates this gen_server's state when
|
|
new `Define*` activities arrive. (Bootstrapping circle resolved: at startup,
|
|
`bootstrap:load_genesis/1` populates the registry directly; from then on, the
|
|
projection fold maintains it.)
|
|
|
|
**Tests:**
|
|
- After genesis load, `registry:list(activity_types)` returns Create/Update/Delete.
|
|
- `registry:lookup(activity_types, "Create")` returns the schema and semantics.
|
|
- A new `DefineActivity{name: "Pin"}` activity (synthesised, hand-signed for the
|
|
test) routes through the projection fold, ends up in the registry.
|
|
- Lookup never caches across activities (verified by introducing a new definition
|
|
mid-test and confirming the next lookup sees it).
|
|
|
|
**Acceptance:** `bash next/tests/registry.sh` passes 10+ cases.
|
|
|
|
---
|
|
|
|
## Step 6 — Validation pipeline + POST /activity
|
|
|
|
**Deliverables:**
|
|
|
|
```erlang
|
|
% next/kernel/pipeline.erl
|
|
-module(pipeline).
|
|
-export([validate_inbound/1, validate_outbound/1]).
|
|
|
|
% Per design §14, run stages in order, halt on first failure.
|
|
validate_inbound(Activity) ->
|
|
Stages = [
|
|
fun stage_envelope/1,
|
|
fun stage_signature/1,
|
|
fun stage_replay/1,
|
|
fun stage_audience/1,
|
|
fun stage_activity_schema/1,
|
|
fun stage_object_schema/1,
|
|
fun stage_content_validators/1,
|
|
fun stage_capabilities/1,
|
|
fun stage_trust/1
|
|
],
|
|
run_stages(Activity, Stages).
|
|
|
|
validate_outbound(Activity) ->
|
|
% Subset of inbound stages (no replay, no trust check; auth done at HTTP layer)
|
|
...
|
|
```
|
|
|
|
```erlang
|
|
% next/kernel/outbox.erl
|
|
-module(outbox).
|
|
-export([publish/2]).
|
|
|
|
publish(ActorId, ActivityRequest) ->
|
|
Activity = construct_envelope(ActorId, ActivityRequest),
|
|
Signed = sig:sign(Activity, ActorId),
|
|
case pipeline:validate_outbound(Signed) of
|
|
ok ->
|
|
log:append(actor_log(ActorId), Signed),
|
|
projection:async_fold(Signed),
|
|
{ok, #{cid => cid:from_sx(Signed),
|
|
ap_id => maps:get(id, Signed)}};
|
|
{error, Reason} ->
|
|
{error, Reason}
|
|
end.
|
|
```
|
|
|
|
**Tests:**
|
|
- Valid activity through full pipeline → appended to log.
|
|
- Bad envelope → 400, not in log.
|
|
- Bad signature → 401, not in log.
|
|
- Replayed activity → 200 duplicate, not re-appended.
|
|
- Schema violation (e.g. Create with no object) → 422.
|
|
- Activity logged before projection completes (async).
|
|
|
|
**Acceptance:** `bash next/tests/pipeline.sh` passes 15+ cases.
|
|
|
|
---
|
|
|
|
## Step 7 — Projection scheduler
|
|
|
|
**Deliverables:**
|
|
|
|
```erlang
|
|
% next/kernel/projection.erl
|
|
-module(projection).
|
|
-export([start_link/1, async_fold/1, query/2, snapshot/1]).
|
|
-behaviour(gen_server).
|
|
|
|
% One gen_server per active projection. State:
|
|
% #{cid, name, fold_fn, current_state, log_tip,
|
|
% snapshot_dir, last_snapshot_at}
|
|
|
|
% async_fold/1 broadcasts a new activity to every projection gen_server;
|
|
% each folds it into its own state. Failures (gas, sandbox violation)
|
|
% tag the activity but don't affect log durability.
|
|
|
|
% query/2 returns current state (or state-as-of)
|
|
% snapshot/1 forces a snapshot now (also runs periodically)
|
|
```
|
|
|
|
```erlang
|
|
% next/kernel/sandbox.erl
|
|
-module(sandbox).
|
|
-export([eval_pure/2, eval_crypto/2, eval_effectful/3]).
|
|
|
|
% eval_pure runs an SX function in pure mode: no IO platform, gas budget,
|
|
% deterministic. Used by projection folds, validators, audience predicates.
|
|
% Wrapper over the SX runtime evaluator with a stripped platform.
|
|
```
|
|
|
|
**Tests:**
|
|
- New activity → all projections fold it concurrently.
|
|
- Projection fold completes within gas budget.
|
|
- Gas-exhausting fold → activity tagged, projection state unchanged, no kernel crash.
|
|
- Sandbox violation (fold tries IO) → same handling.
|
|
- Snapshot create + reload → state matches.
|
|
- Snapshot CID stable across kernel restarts.
|
|
|
|
**Acceptance:** `bash next/tests/projection.sh` passes 15+ cases.
|
|
|
|
---
|
|
|
|
## Step 8 — HTTP server + endpoints
|
|
|
|
**Deliverables:**
|
|
|
|
Core endpoints (per design §16.1):
|
|
|
|
```
|
|
GET /actors/<id> # actor doc
|
|
GET /actors/<id>/outbox # OrderedCollection
|
|
GET /actors/<id>/outbox?page=true # OrderedCollectionPage
|
|
POST /activity # publish (auth: bearer token)
|
|
GET /artifacts/<cid> # CID-addressed artifact
|
|
GET /artifacts/<cid>/raw
|
|
GET /projections # list of projections
|
|
GET /projections/<name> # full state
|
|
GET /projections/<name>?at=<ts> # time-travel
|
|
GET /projections/<name>/<key> # indexed lookup
|
|
GET /define-registry
|
|
GET /.well-known/sx-capabilities
|
|
GET /.well-known/webfinger
|
|
```
|
|
|
|
```erlang
|
|
% next/kernel/http_server.erl
|
|
-module(http_server).
|
|
-export([start/1, route/1]).
|
|
|
|
start(Port) ->
|
|
http:listen(Port, fun ?MODULE:route/1).
|
|
|
|
route(Request) -> {Status, Headers, Body}.
|
|
```
|
|
|
|
Content negotiation per `Accept`:
|
|
- `application/activity+json` (default)
|
|
- `application/cbor` (dag-cbor)
|
|
- `application/json` (compact, no @context expansion)
|
|
- `application/sx`
|
|
|
|
Auth on `POST /activity`: bearer token from env var `NEXT_PUBLISH_TOKEN`.
|
|
|
|
**Tests:**
|
|
- Each endpoint returns expected shape for known artifact.
|
|
- Content negotiation: same artifact in 4 representations.
|
|
- 404 for unknown artifact CID.
|
|
- 401 for `POST /activity` without token.
|
|
- Pagination: outbox with > 50 activities returns OrderedCollectionPage.
|
|
|
|
**Acceptance:** `bash next/tests/http.sh` passes 20+ cases.
|
|
|
|
---
|
|
|
|
## Step 9 — Smoke tests
|
|
|
|
**The proof points.** Two end-to-end smoke tests demonstrate, between them, that
|
|
fed-sx is genuinely a substrate for distributed reactive applications expressed
|
|
as data — not a system you extend by writing kernel code.
|
|
|
|
- **9a — Pin smoke test (`next/tests/smoke_pin.sh`)** — verb extensibility:
|
|
defining a new activity type and projection at runtime via `Define*`
|
|
artifacts. Verifies the meta-level (§5).
|
|
- **9b — Reactive application smoke test (`next/tests/smoke_app.sh`)** —
|
|
application extensibility: defining a new subscription type, subscribing,
|
|
registering a trigger, and observing the full reactive loop fire end-to-end
|
|
without kernel code changes. Verifies §§18-19.
|
|
|
|
Both must pass for milestone 1 acceptance.
|
|
|
|
### Step 9a — Pin smoke test
|
|
|
|
**Test script:** `next/tests/smoke_pin.sh`
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
# 0. Start a fresh fed-sx kernel (background)
|
|
./next/scripts/start.sh fresh
|
|
sleep 2
|
|
TOKEN=$(cat next/data/keys/publish.token)
|
|
|
|
# 1. Verify actor exists
|
|
curl -s http://localhost:9999/actors/next | jq -e '.type == "Person"'
|
|
|
|
# 2. Verify outbox has actor's first Create{Person}
|
|
curl -s http://localhost:9999/actors/next/outbox?page=true \
|
|
| jq -e '.orderedItems | length == 1 and .[0].type == "Create"'
|
|
|
|
# 3. Verify Pin is NOT a known activity type
|
|
curl -s http://localhost:9999/define-registry?kind=activity_types \
|
|
| jq -e '.[] | select(.name == "Pin") | length == 0' || exit 1
|
|
|
|
# 4. Publish DefineActivity{name: "Pin", schema: ..., semantics: ...}
|
|
PIN_DEF=$(cat <<'JSON'
|
|
{
|
|
"type": "Create",
|
|
"object": {
|
|
"type": "DefineActivity",
|
|
"name": "Pin",
|
|
"schema": "(fn (act) (and (string? (-> act :object :path)) (cid? (-> act :object :cid))))",
|
|
"semantics": "(fn (state act) (assoc-in state [:pins (-> act :object :path)] (-> act :object :cid)))"
|
|
}
|
|
}
|
|
JSON
|
|
)
|
|
curl -s -X POST http://localhost:9999/activity \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-H "Content-Type: application/activity+json" \
|
|
-d "$PIN_DEF" | jq -e '.cid' > /dev/null
|
|
|
|
# 5. Verify Pin IS now a known activity type
|
|
curl -s http://localhost:9999/define-registry?kind=activity_types \
|
|
| jq -e '.[] | select(.name == "Pin") | length == 1'
|
|
|
|
# 6. Also publish a DefineProjection{name: "pin-state"} that folds Pin into state
|
|
PIN_PROJ=$(cat <<'JSON'
|
|
{
|
|
"type": "Create",
|
|
"object": {
|
|
"type": "DefineProjection",
|
|
"name": "pin-state",
|
|
"initial-state": "{}",
|
|
"fold": "(fn (state act) (if (= (:type act) \"Pin\") (assoc state (-> act :object :path) (-> act :object :cid)) state))"
|
|
}
|
|
}
|
|
JSON
|
|
)
|
|
curl -s -X POST http://localhost:9999/activity \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-d "$PIN_PROJ" | jq -e '.cid'
|
|
|
|
# 7. Now publish a Pin activity
|
|
PIN=$(cat <<'JSON'
|
|
{
|
|
"type": "Pin",
|
|
"object": {
|
|
"type": "PinSpec",
|
|
"path": "/docs/intro",
|
|
"cid": "bafyreigh2akiscaildc3xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxxe"
|
|
}
|
|
}
|
|
JSON
|
|
)
|
|
curl -s -X POST http://localhost:9999/activity \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-d "$PIN" | jq -e '.cid'
|
|
|
|
# 8. Verify Pin appears in outbox
|
|
curl -s http://localhost:9999/actors/next/outbox?page=true \
|
|
| jq -e '.orderedItems | map(select(.type == "Pin")) | length == 1'
|
|
|
|
# 9. Verify pin-state projection has the entry
|
|
sleep 1 # allow async projection
|
|
curl -s http://localhost:9999/projections/pin-state \
|
|
| jq -e '."/docs/intro" == "bafyreigh2akiscaildc3xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxxe"'
|
|
|
|
# 10. Negative test: publish a malformed Pin (missing path) → expect 422
|
|
BAD_PIN='{"type": "Pin", "object": {"cid": "bafy..."}}'
|
|
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X POST http://localhost:9999/activity \
|
|
-H "Authorization: Bearer $TOKEN" -d "$BAD_PIN")
|
|
[[ "$HTTP_STATUS" == "422" ]] || { echo "expected 422, got $HTTP_STATUS"; exit 1; }
|
|
|
|
# 11. Restart kernel; verify state recovers
|
|
./next/scripts/stop.sh
|
|
./next/scripts/start.sh
|
|
sleep 2
|
|
curl -s http://localhost:9999/projections/pin-state \
|
|
| jq -e '."/docs/intro" == "bafyreigh2akiscaildc3xqxx4xqxx4xqxx4xqxx4xqxx4xqxx4xqxxe"'
|
|
|
|
echo "✓ Pin smoke test passed — verb extensibility demonstrated end-to-end"
|
|
```
|
|
|
|
**Acceptance for 9a:** smoke test exits 0. The whole flow happens with **zero
|
|
fed-sx kernel code changes** between defining the verb and using it.
|
|
|
|
### Step 9b — Reactive application smoke test
|
|
|
|
**The bigger proof point.** Demonstrates that fed-sx supports distributed
|
|
reactive applications composed of `DefineSubscription` + `DefineTrigger` +
|
|
`DefineProjection` — the application model from §§18-19.
|
|
|
|
The test runs on a single instance (federation is v2), so the "subscriber" and
|
|
"publisher" are the same actor. That's intentional — milestone 1 proves the
|
|
mechanism; milestone 2 spreads it across instances.
|
|
|
|
**Test script:** `next/tests/smoke_app.sh`
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
# Assumes 9a has already run (fresh kernel optional; can run alongside).
|
|
TOKEN=$(cat next/data/keys/publish.token)
|
|
BASE=http://localhost:9999
|
|
|
|
# 1. Verify "Topic" subscription type and "Subscribe" verb are NOT yet defined.
|
|
curl -s "$BASE/define-registry?kind=subscription_types" \
|
|
| jq -e 'map(select(.name == "Topic")) | length == 0'
|
|
|
|
# 2. Publish DefineSubscription{name: "Topic", ...}
|
|
TOPIC_DEF=$(cat <<'JSON'
|
|
{
|
|
"type": "Create",
|
|
"object": {
|
|
"type": "DefineSubscription",
|
|
"name": "Topic",
|
|
"schema": "(fn (sub) (string? (-> sub :tag)))",
|
|
"match": "(fn (sub act) (and (= (:type act) \"Note\") (member? (-> sub :tag) (or (-> act :object :tags) (list)))))",
|
|
"delivery": "{:default :push :modes (list :push :pull)}"
|
|
}
|
|
}
|
|
JSON
|
|
)
|
|
curl -s -X POST "$BASE/activity" \
|
|
-H "Authorization: Bearer $TOKEN" -d "$TOPIC_DEF" | jq -e '.cid'
|
|
|
|
# 3. Verify Topic IS now a known subscription type.
|
|
curl -s "$BASE/define-registry?kind=subscription_types" \
|
|
| jq -e 'map(select(.name == "Topic")) | length == 1'
|
|
|
|
# 4. Subscribe to the "smoketest" topic.
|
|
SUBSCRIBE=$(cat <<'JSON'
|
|
{
|
|
"type": "Subscribe",
|
|
"object": {"type": "Topic", "tag": "smoketest"}
|
|
}
|
|
JSON
|
|
)
|
|
SUB_CID=$(curl -s -X POST "$BASE/activity" \
|
|
-H "Authorization: Bearer $TOKEN" -d "$SUBSCRIBE" | jq -r '.cid')
|
|
|
|
# 5. Verify subscriptions projection has the new entry.
|
|
sleep 1
|
|
curl -s "$BASE/projections/subscriptions" \
|
|
| jq -e '.["https://next.rose-ash.com/actors/next"] | map(select(.type == "Topic")) | length == 1'
|
|
|
|
# 6. Define a projection that records matched activities (per-application
|
|
# namespace would happen via DefineApplication in v1.x; for v1 the
|
|
# projection is global to the actor).
|
|
TOPIC_PROJ=$(cat <<'JSON'
|
|
{
|
|
"type": "Create",
|
|
"object": {
|
|
"type": "DefineProjection",
|
|
"name": "topic-events",
|
|
"initial-state": "{}",
|
|
"fold": "(fn (state act) (if (and (= (:type act) \"Note\") (member? \"smoketest\" (or (-> act :object :tags) (list)))) (assoc-in state [(:cid act)] act) state))"
|
|
}
|
|
}
|
|
JSON
|
|
)
|
|
curl -s -X POST "$BASE/activity" \
|
|
-H "Authorization: Bearer $TOKEN" -d "$TOPIC_PROJ" | jq -e '.cid'
|
|
|
|
# 7. Define a trigger: when a Topic{smoketest} subscription matches, publish
|
|
# a TestEcho activity. We need an "Echo" activity type first.
|
|
ECHO_DEF=$(cat <<'JSON'
|
|
{
|
|
"type": "Create",
|
|
"object": {
|
|
"type": "DefineActivity",
|
|
"name": "TestEcho",
|
|
"schema": "(fn (act) (cid? (-> act :object :echoes)))",
|
|
"semantics": "(fn (state act) state)"
|
|
}
|
|
}
|
|
JSON
|
|
)
|
|
curl -s -X POST "$BASE/activity" \
|
|
-H "Authorization: Bearer $TOKEN" -d "$ECHO_DEF" | jq -e '.cid'
|
|
|
|
TRIGGER=$(cat <<JSON
|
|
{
|
|
"type": "Create",
|
|
"object": {
|
|
"type": "DefineTrigger",
|
|
"name": "echo-on-smoketest",
|
|
"when-subscription": "$SUB_CID",
|
|
"cascade-limit": 1,
|
|
"then": "(fn (act sub env) {:publish (list {:type \"TestEcho\" :object {:echoes (:cid act)}})})"
|
|
}
|
|
}
|
|
JSON
|
|
)
|
|
curl -s -X POST "$BASE/activity" \
|
|
-H "Authorization: Bearer $TOKEN" -d "$TRIGGER" | jq -e '.cid'
|
|
|
|
# 8. Capture outbox length so we can detect new entries.
|
|
BEFORE=$(curl -s "$BASE/actors/next/outbox?page=true" \
|
|
| jq -r '.orderedItems | length')
|
|
|
|
# 9. Publish a Note tagged "smoketest" — should match subscription, fire trigger,
|
|
# cause TestEcho to be published.
|
|
NOTE=$(cat <<'JSON'
|
|
{
|
|
"type": "Create",
|
|
"object": {
|
|
"type": "Note",
|
|
"content": "hello reactive world",
|
|
"tags": ["smoketest"]
|
|
}
|
|
}
|
|
JSON
|
|
)
|
|
NOTE_CID=$(curl -s -X POST "$BASE/activity" \
|
|
-H "Authorization: Bearer $TOKEN" -d "$NOTE" | jq -r '.cid')
|
|
|
|
# 10. Wait for projection + trigger.
|
|
sleep 2
|
|
|
|
# 11. Verify topic-events projection captured the Note.
|
|
curl -s "$BASE/projections/topic-events" \
|
|
| jq -e ". | to_entries | length == 1"
|
|
|
|
# 12. Verify outbox grew by exactly TWO activities (the Note + the trigger's TestEcho).
|
|
AFTER=$(curl -s "$BASE/actors/next/outbox?page=true" \
|
|
| jq -r '.orderedItems | length')
|
|
[[ $((AFTER - BEFORE)) == 2 ]] || { echo "expected +2 activities, got $((AFTER - BEFORE))"; exit 1; }
|
|
|
|
# 13. Verify the latest activity is a TestEcho referencing the original Note's CID.
|
|
curl -s "$BASE/actors/next/outbox?page=true" \
|
|
| jq -e ".orderedItems[0] | .type == \"TestEcho\" and .object.echoes == \"$NOTE_CID\""
|
|
|
|
# 14. Negative case: publish a Note WITHOUT the "smoketest" tag — must NOT
|
|
# trigger, must NOT echo.
|
|
BEFORE2=$(curl -s "$BASE/actors/next/outbox?page=true" | jq -r '.orderedItems | length')
|
|
NOTE_OTHER=$(cat <<'JSON'
|
|
{"type": "Create", "object": {"type": "Note", "content": "no match", "tags": ["other"]}}
|
|
JSON
|
|
)
|
|
curl -s -X POST "$BASE/activity" \
|
|
-H "Authorization: Bearer $TOKEN" -d "$NOTE_OTHER" | jq -e '.cid'
|
|
sleep 2
|
|
AFTER2=$(curl -s "$BASE/actors/next/outbox?page=true" | jq -r '.orderedItems | length')
|
|
[[ $((AFTER2 - BEFORE2)) == 1 ]] || { echo "expected +1 activity (no echo), got $((AFTER2 - BEFORE2))"; exit 1; }
|
|
|
|
# 15. Cascade limit check: prove the trigger doesn't recursively echo TestEcho.
|
|
# The TestEcho activity itself should NOT match the Topic{smoketest}
|
|
# subscription (it's not a Note), so no cascade, but verify cascade-depth
|
|
# was set to 1 on the echo so a future trigger on TestEcho would refuse.
|
|
LATEST_ECHO=$(curl -s "$BASE/actors/next/outbox?page=true" \
|
|
| jq -r '.orderedItems | map(select(.type == "TestEcho")) | .[0]')
|
|
echo "$LATEST_ECHO" | jq -e '."cascade-depth" == 1'
|
|
|
|
# 16. Restart kernel; verify subscription, trigger, projection all survive.
|
|
./next/scripts/stop.sh
|
|
./next/scripts/start.sh
|
|
sleep 2
|
|
curl -s "$BASE/projections/subscriptions" \
|
|
| jq -e '.["https://next.rose-ash.com/actors/next"] | map(select(.type == "Topic")) | length == 1'
|
|
curl -s "$BASE/projections/topic-events" | jq -e ". | to_entries | length >= 1"
|
|
curl -s "$BASE/define-registry?kind=triggers" \
|
|
| jq -e 'map(select(.name == "echo-on-smoketest")) | length == 1'
|
|
|
|
echo "✓ Reactive application smoke test passed — Subscribe + Trigger + Projection demonstrated end-to-end"
|
|
```
|
|
|
|
**What this proves (and what it doesn't):**
|
|
|
|
Proves:
|
|
- `DefineSubscription` + `Subscribe` mechanism works end-to-end.
|
|
- Subscription's `match-fn` evaluates correctly in pure mode against inbound
|
|
activities.
|
|
- `DefineTrigger` fires on subscription matches.
|
|
- Trigger's `then-sx` can publish derived activities (the `:publish` result).
|
|
- Cascade-depth metadata propagates correctly.
|
|
- Subscription state, trigger registration, and projection state all survive
|
|
kernel restart (snapshot + log replay).
|
|
- The full reactive application loop works without any kernel code changes
|
|
between defining the components and exercising them.
|
|
|
|
Does NOT prove (deferred to milestone 2+):
|
|
- Cross-instance subscriptions (federation).
|
|
- Trigger `:effect` results calling effectful primitives.
|
|
- `DefineApplication` bundle install/update/fork.
|
|
- Per-application namespace isolation.
|
|
- Cascade prevention against malicious cascading from peer instances.
|
|
|
|
**Acceptance for 9b:** smoke test exits 0. Like 9a, **zero fed-sx kernel code
|
|
changes** between defining the application components and observing them
|
|
operate.
|
|
|
|
---
|
|
|
|
## Acceptance criteria for milestone 1
|
|
|
|
All of:
|
|
|
|
1. **Each step's test suite passes** (`bash next/tests/<step>.sh`).
|
|
2. **Both smoke tests pass** (`bash next/tests/smoke_pin.sh` and
|
|
`bash next/tests/smoke_app.sh`).
|
|
3. **Erlang-on-SX baseline preserved** — adding fed-sx kernel modules in
|
|
`next/kernel/*.erl` doesn't break Phase 1-8 conformance.
|
|
4. **Restart durability** — kill the kernel mid-write, restart, projections
|
|
resume from snapshot, no log corruption.
|
|
5. **Manual Mastodon poke** — point a Mastodon account at
|
|
`https://next.rose-ash.com/actors/next` and verify the actor doc fetches and
|
|
webfinger discovery works (read-only AP interop, no follow).
|
|
|
|
## What lands when
|
|
|
|
This is the work-order an agent (or human) follows. Steps 1-3 can be done in
|
|
parallel after the Erlang Phase 8 BIFs land. Steps 4-7 are sequential. Step 8
|
|
can start in parallel with step 7. Step 9 is the integration test.
|
|
|
|
```
|
|
Phase 7+8 (loops/erlang) ───┐
|
|
│
|
|
▼
|
|
┌─── Step 1 ──┬─── Step 2 ──┬─── Step 3
|
|
│ │ │
|
|
└─────────────┼─── Step 4 ──┴────┐
|
|
│ │
|
|
└─── Step 5 ───────┤
|
|
│
|
|
Step 6 ─────┤
|
|
│
|
|
Step 7 ─────┤
|
|
│
|
|
Step 8 ─────┤
|
|
│
|
|
Step 9 ─────┘
|
|
```
|
|
|
|
Estimated effort if done by a focused agent loop, one feature per iteration:
|
|
~30-50 commits across all 9 steps. Could plausibly be a `loops/fed-sx` workstream
|
|
once Phase 7+8 are done.
|
|
|
|
## What's deferred to milestone 2
|
|
|
|
- **Federation** (the second-biggest piece). `POST /inbox`, Follow lifecycle,
|
|
delivery queue, backfill, capability negotiation between peers. Whole of
|
|
design §13.
|
|
- **Multi-actor** with per-user OAuth and capability tokens. Design §9.5.
|
|
- **IPFS storage backend** as a `DefineStorage` entry. Design §15.3.
|
|
- **Browser client + operator dashboard** (probably in Elm-on-SX or similar).
|
|
- **Rich verbs**: `Endorse`, `Supersede`, `Test`, `Build`, `Compose`, `Note`,
|
|
`Announce`. All defined as `DefineActivity` artifacts, federated.
|
|
- **Cross-host conformance** — Python/JS/Haskell hosts running fed-sx. Design
|
|
§11.8.
|
|
- **OpenTimestamps proofs** as a `DefineProof` entry.
|
|
- **Performance work** — JIT-compiled folds, snapshot acceleration, federation
|
|
batching.
|
|
|
|
Milestone 2 unlocks "real federation between two fed-sx instances." Milestone 3
|
|
is the rose-ash port (blog, market, events, federation, account, orders) as
|
|
fed-sx applications.
|
|
|
|
---
|
|
|
|
## Appendix A: open questions for milestone 1
|
|
|
|
A few things still under-specified; resolve as work begins.
|
|
|
|
1. **HTTP server library.** Does the Phase 8 `http:listen/2` BIF wrap an
|
|
existing OCaml HTTP server (the sx.rose-ash.com one) or something simpler?
|
|
Implementation choice deferred to Phase 8.
|
|
2. **JSON-LD library.** AP wire format requires JSON-LD canonicalization for
|
|
signature coverage. Either pull a library or write a minimal subset for the
|
|
shapes we actually use. Probably the latter — our envelope is well-defined.
|
|
3. **Bearer token rotation.** v1 uses a single env-var token. Token rotation
|
|
without restart needs registry-style mgmt; can wait.
|
|
4. **Snapshot rate limits.** Default in design is "every 1000 activities or
|
|
60 seconds." Tunable per-projection later; v1 uses the default.
|
|
5. **Genesis bundle format.** Dag-cbor map per §12.2; concrete schema needs
|
|
one round of refinement once we author the actual definitions in step 4.
|