From 5959a97dca012325f726f83ee939f64e1bd6f12c Mon Sep 17 00:00:00 2001 From: giles Date: Tue, 30 Jun 2026 15:30:21 +0000 Subject: [PATCH] fed-sx-types Phase 1: DefineType + SubtypeOf genesis verbs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two new DefineActivity-form genesis activity-types for host-type federation (plans/fed-sx-host-types.md step 1): - next/genesis/activity-types/define_type.sx — DefineType verb; schema accepts an :object with a string :name and optional list :fields. - next/genesis/activity-types/subtype_of.sx — SubtypeOf verb; schema accepts an :object carrying string :child-type-cid + :parent-type-cid. Schema bodies use nested `get` (not keyword-threading) so they are directly evaluatable — keywords are not callable getters in the kernel. Both registered in manifest.sx (activity-types now 7); the four bootstrap suites' bundle counts bumped (5->7, total 36->38). Tests: next/tests/define_type.sh (7), subtype_of.sh (6) — parse shape, schema accept/reject, term_codec envelope round-trip. Also load follower_graph + delivery in bootstrap_start.sh: its check-26 publish path exercises outbox:compute_delivery_set/3 (follower_graph:new + delivery:delivery_set), which an m2 substrate change had left unloaded in that suite — a pre-existing red unrelated to the count bump. Conformance 771/771; all touched next/tests green. Co-Authored-By: Claude Opus 4.8 (1M context) --- next/genesis/activity-types/define_type.sx | 34 +++++++ next/genesis/activity-types/subtype_of.sx | 31 ++++++ next/genesis/manifest.sx | 4 +- next/tests/bootstrap_load.sh | 2 +- next/tests/bootstrap_populate.sh | 4 +- next/tests/bootstrap_read.sh | 4 +- next/tests/bootstrap_start.sh | 10 +- next/tests/define_type.sh | 110 +++++++++++++++++++++ next/tests/genesis_parse.sh | 14 ++- next/tests/subtype_of.sh | 103 +++++++++++++++++++ plans/fed-sx-host-types.md | 106 ++++++++++++++++++++ 11 files changed, 413 insertions(+), 9 deletions(-) create mode 100644 next/genesis/activity-types/define_type.sx create mode 100644 next/genesis/activity-types/subtype_of.sx create mode 100755 next/tests/define_type.sh create mode 100755 next/tests/subtype_of.sh create mode 100644 plans/fed-sx-host-types.md diff --git a/next/genesis/activity-types/define_type.sx b/next/genesis/activity-types/define_type.sx new file mode 100644 index 00000000..a0b057db --- /dev/null +++ b/next/genesis/activity-types/define_type.sx @@ -0,0 +1,34 @@ +;; next/genesis/activity-types/define_type.sx +;; +;; Bootstrap definition of the DefineType verb per +;; plans/fed-sx-host-types.md (host-type federation, Phase 1). +;; Read as data by the bundler (bootstrap.erl) — never evaluated as +;; code. The :schema and :semantics bodies are SX source. +;; +;; DefineType declares a refinement type. The activity's :object is +;; the type record: +;; {:name "Post" ;; the type's display name +;; :fields (...) ;; optional field descriptors +;; :refinement-schema (fn (obj) ...) ;; predicate over instances +;; :instance-type "Note"} ;; base object-type it refines +;; +;; The schema below validates the *activity* shape: :object present, +;; :name a string, :fields (when present) a list. The richer +;; per-field shape check and the registry registration land with the +;; peer_types cache (Phase 2) — at this phase the form is pure data. +;; +;; Schema bodies use nested `get` rather than keyword-threading so +;; the predicate is directly evaluatable (keywords are not callable +;; getters in the kernel; `(-> d :k)` is not a get). +(DefineActivity + :name "DefineType" + :doc "Declare a refinement type. :object carries :name, optional :fields, :refinement-schema, and :instance-type." + :schema (fn + (act) + (and + (not (nil? (get act :object))) + (string? (get (get act :object) :name)) + (or + (nil? (get (get act :object) :fields)) + (list? (get (get act :object) :fields))))) + :semantics (fn (state act) state)) diff --git a/next/genesis/activity-types/subtype_of.sx b/next/genesis/activity-types/subtype_of.sx new file mode 100644 index 00000000..10de67a1 --- /dev/null +++ b/next/genesis/activity-types/subtype_of.sx @@ -0,0 +1,31 @@ +;; next/genesis/activity-types/subtype_of.sx +;; +;; Bootstrap definition of the SubtypeOf verb per +;; plans/fed-sx-host-types.md (host-type federation, Phase 1). +;; Read as data by the bundler (bootstrap.erl) — never evaluated as +;; code. The :schema and :semantics bodies are SX source. +;; +;; SubtypeOf records a hierarchy edge between two previously-defined +;; types. The activity's :object is the relation record: +;; {:child-type-cid "bafy...child" +;; :parent-type-cid "bafy...parent"} +;; +;; The schema validates the *activity* shape: both CIDs present and +;; string-typed. Verifying that each CID names a previously-defined +;; type is a registry concern (it needs the type index that lands +;; with peer_types in Phase 2), so it is deliberately out of the +;; pure-predicate schema here — adding the edge to the hierarchy +;; index is the :semantics' job once the registry surface exists. +;; +;; Schema bodies use nested `get` rather than keyword-threading so +;; the predicate is directly evaluatable. +(DefineActivity + :name "SubtypeOf" + :doc "Record a subtype edge. :object carries :child-type-cid and :parent-type-cid, both type CIDs." + :schema (fn + (act) + (and + (not (nil? (get act :object))) + (string? (get (get act :object) :child-type-cid)) + (string? (get (get act :object) :parent-type-cid)))) + :semantics (fn (state act) state)) diff --git a/next/genesis/manifest.sx b/next/genesis/manifest.sx index 7cdceff2..7e111720 100644 --- a/next/genesis/manifest.sx +++ b/next/genesis/manifest.sx @@ -22,7 +22,9 @@ "activity-types/update.sx" "activity-types/delete.sx" "activity-types/announce.sx" - "activity-types/endorse.sx") + "activity-types/endorse.sx" + "activity-types/define_type.sx" + "activity-types/subtype_of.sx") :object-types ("object-types/sx-artifact.sx" "object-types/note.sx" "object-types/tombstone.sx" diff --git a/next/tests/bootstrap_load.sh b/next/tests/bootstrap_load.sh index 26c29ec9..a052f944 100755 --- a/next/tests/bootstrap_load.sh +++ b/next/tests/bootstrap_load.sh @@ -106,7 +106,7 @@ check 10 "strip suffix create.sx -> create" "true" check 11 "strip suffix hello unchanged" "true" check 12 "strip suffix .sx -> empty" "true" check 13 "load_genesis rejects bad shape" "ok" -check 20 "loaded activity_types count = 5" "5" +check 20 "loaded activity_types count = 7" "7" check 21 "loaded object_types count = 13" "13" check 22 "loaded projections count = 7" "7" check 23 "loaded validators count = 3" "3" diff --git a/next/tests/bootstrap_populate.sh b/next/tests/bootstrap_populate.sh index 724541d9..41707f0d 100755 --- a/next/tests/bootstrap_populate.sh +++ b/next/tests/bootstrap_populate.sh @@ -99,8 +99,8 @@ check() { check 2 "gen_server loaded" "gen_server" check 3 "registry loaded" "registry" check 4 "bootstrap loaded" "bootstrap" -check 10 "populate returns total 36" "36" -check 20 "activity_types count = 5" "5" +check 10 "populate returns total 38" "38" +check 20 "activity_types count = 7" "7" check 21 "object_types count = 13" "13" check 22 "projections count = 7" "7" check 23 "validators count = 3" "3" diff --git a/next/tests/bootstrap_read.sh b/next/tests/bootstrap_read.sh index cccc2ae9..20703651 100755 --- a/next/tests/bootstrap_read.sh +++ b/next/tests/bootstrap_read.sh @@ -102,7 +102,7 @@ check 10 "sections/0 length" "7" check 11 "ends_with_sx create.sx" "true" check 12 "ends_with_sx hello" "false" check 13 "ends_with_sx empty" "false" -check 20 "section activity_types count" "5" +check 20 "section activity_types count" "7" check 21 "section object_types count" "13" check 22 "section projections count" "7" check 23 "section validators count" "3" @@ -111,7 +111,7 @@ check 25 "section sig_suites count" "2" check 26 "section audience count" "3" check 30 "read_genesis returns 7 sections" "7" check 31 "first section name" "activity_types" -check 32 "first section entry count" "5" +check 32 "first section entry count" "7" TOTAL=$((PASS+FAIL)) if [ $FAIL -eq 0 ]; then diff --git a/next/tests/bootstrap_start.sh b/next/tests/bootstrap_start.sh index 453d3f7f..50523a15 100755 --- a/next/tests/bootstrap_start.sh +++ b/next/tests/bootstrap_start.sh @@ -54,6 +54,12 @@ cat > "$TMPFILE" <&2 + exit 1 +fi + +VERBOSE="${1:-}" +PASS=0; FAIL=0; ERRORS="" +TMPFILE=$(mktemp); trap "rm -f $TMPFILE" EXIT + +# The schema fn, evaluated from the genesis file into a lambda. +SCH='(eval-expr (get (apply dict (rest (parse (file-read \"next/genesis/activity-types/define_type.sx\")))) :schema))' + +cat > "$TMPFILE" < true +(epoch 20) +(eval "(define sch ${SCH}) (sch (dict :object (dict :name \"Post\" :fields (list))))") +;; valid: :fields omitted (optional) -> true +(epoch 21) +(eval "(define sch ${SCH}) (sch (dict :object (dict :name \"Post\")))") +;; reject: missing :name -> false +(epoch 22) +(eval "(define sch ${SCH}) (sch (dict :object (dict :fields (list))))") +;; reject: :fields present but not a list -> false +(epoch 23) +(eval "(define sch ${SCH}) (sch (dict :object (dict :name \"Post\" :fields \"notalist\")))") + +;; ── envelope round-trip through term_codec ───────────────── +(epoch 30) +(eval "(get (erlang-eval-ast \"A = [{type, define_type}, {actor, alice}, {object, [{name, <<80,111,115,116>>}, {instance_type, <<78,111,116,101>>}]}], {ok, D, _} = term_codec:decode(term_codec:encode(A)), D =:= A\") :name)") +EPOCHS + +OUTPUT=$(timeout 180 "$SX_SERVER" < "$TMPFILE" 2>/dev/null) + +check() { + local epoch="$1" desc="$2" expected="$3" + local actual + actual=$(echo "$OUTPUT" | awk -v e="$epoch" ' + $0 ~ "^\\(ok-len " e " " { getline; print; exit } + $0 ~ "^\\(ok " e " " { print; exit } + $0 ~ "^\\(error " e " " { print; exit } + ') + [ -z "$actual" ] && actual="" + if echo "$actual" | grep -qF -- "$expected"; then + PASS=$((PASS+1)) + [ "$VERBOSE" = "-v" ] && echo " ok $desc" + else + FAIL=$((FAIL+1)) + ERRORS+=" FAIL [$desc] (epoch $epoch) expected: $expected | actual: $actual +" + fi +} + +check 10 "define_type.sx head form" "DefineActivity" +check 11 "define_type.sx name" "DefineType" +check 20 "schema accepts valid type def" "true" +check 21 "schema accepts omitted :fields" "true" +check 22 "schema rejects missing :name" "false" +check 23 "schema rejects non-list :fields" "false" +check 30 "DefineType envelope round-trips" "true" + +TOTAL=$((PASS+FAIL)) +if [ $FAIL -eq 0 ]; then + echo "ok $PASS/$TOTAL next/tests/define_type.sh passed" +else + echo "FAIL $PASS/$TOTAL passed, $FAIL failed:" + echo "$ERRORS" +fi +[ $FAIL -eq 0 ] diff --git a/next/tests/genesis_parse.sh b/next/tests/genesis_parse.sh index 570d2e57..a8f2bc93 100755 --- a/next/tests/genesis_parse.sh +++ b/next/tests/genesis_parse.sh @@ -48,6 +48,14 @@ cat > "$TMPFILE" <<'EPOCHS' (eval "(first (parse (file-read \"next/genesis/activity-types/endorse.sx\")))") (epoch 200) (eval "(get (apply dict (rest (parse (file-read \"next/genesis/activity-types/endorse.sx\")))) :name)") +(epoch 201) +(eval "(first (parse (file-read \"next/genesis/activity-types/define_type.sx\")))") +(epoch 202) +(eval "(get (apply dict (rest (parse (file-read \"next/genesis/activity-types/define_type.sx\")))) :name)") +(epoch 203) +(eval "(first (parse (file-read \"next/genesis/activity-types/subtype_of.sx\")))") +(epoch 204) +(eval "(get (apply dict (rest (parse (file-read \"next/genesis/activity-types/subtype_of.sx\")))) :name)") (epoch 19) (eval "(len (get (apply dict (rest (parse (file-read \"next/genesis/manifest.sx\")))) :activity-types))") (epoch 30) @@ -180,7 +188,11 @@ check 27 "announce.sx head form" "DefineActivity" check 28 "announce.sx name is Announce" "Announce" check 29 "endorse.sx head form" "DefineActivity" check 200 "endorse.sx name is Endorse" "Endorse" -check 19 "manifest has 5 activity-types" "5" +check 201 "define_type.sx head form" "DefineActivity" +check 202 "define_type.sx name" "DefineType" +check 203 "subtype_of.sx head form" "DefineActivity" +check 204 "subtype_of.sx name" "SubtypeOf" +check 19 "manifest has 7 activity-types" "7" check 30 "sx-artifact.sx head form" "DefineObject" check 31 "sx-artifact.sx name" "SXArtifact" check 32 "note.sx name" "Note" diff --git a/next/tests/subtype_of.sh b/next/tests/subtype_of.sh new file mode 100755 index 00000000..156f4f39 --- /dev/null +++ b/next/tests/subtype_of.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# next/tests/subtype_of.sh — host-type federation Phase 1 acceptance. +# +# The SubtypeOf genesis verb (next/genesis/activity-types/subtype_of.sx) +# records a hierarchy edge between two previously-defined types. This +# suite confirms: +# - the file parses with the expected DefineActivity head + :name +# - the :schema predicate accepts an edge carrying both CIDs and +# rejects edges missing either side +# - a SubtypeOf envelope round-trips through term_codec +# +# Schema bodies are SX source; we eval them with `eval-expr` and call +# the resulting lambda directly. 7 cases. + +set -uo pipefail +cd "$(git rev-parse --show-toplevel)" + +SX_SERVER="${SX_SERVER:-hosts/ocaml/_build/default/bin/sx_server.exe}" +if [ ! -x "$SX_SERVER" ]; then + SX_SERVER="/root/rose-ash/hosts/ocaml/_build/default/bin/sx_server.exe" +fi +if [ ! -x "$SX_SERVER" ]; then + echo "ERROR: sx_server.exe not found." >&2 + exit 1 +fi + +VERBOSE="${1:-}" +PASS=0; FAIL=0; ERRORS="" +TMPFILE=$(mktemp); trap "rm -f $TMPFILE" EXIT + +SCH='(eval-expr (get (apply dict (rest (parse (file-read \"next/genesis/activity-types/subtype_of.sx\")))) :schema))' + +cat > "$TMPFILE" < true +(epoch 20) +(eval "(define sch ${SCH}) (sch (dict :object (dict :child-type-cid \"bafyChild\" :parent-type-cid \"bafyParent\")))") +;; reject: missing :child-type-cid -> false +(epoch 21) +(eval "(define sch ${SCH}) (sch (dict :object (dict :parent-type-cid \"bafyParent\")))") +;; reject: missing :parent-type-cid -> false +(epoch 22) +(eval "(define sch ${SCH}) (sch (dict :object (dict :child-type-cid \"bafyChild\")))") + +;; ── envelope round-trip through term_codec ───────────────── +(epoch 30) +(eval "(get (erlang-eval-ast \"A = [{type, subtype_of}, {actor, alice}, {object, [{child_type_cid, <<99,104>>}, {parent_type_cid, <<112,97>>}]}], {ok, D, _} = term_codec:decode(term_codec:encode(A)), D =:= A\") :name)") +EPOCHS + +OUTPUT=$(timeout 180 "$SX_SERVER" < "$TMPFILE" 2>/dev/null) + +check() { + local epoch="$1" desc="$2" expected="$3" + local actual + actual=$(echo "$OUTPUT" | awk -v e="$epoch" ' + $0 ~ "^\\(ok-len " e " " { getline; print; exit } + $0 ~ "^\\(ok " e " " { print; exit } + $0 ~ "^\\(error " e " " { print; exit } + ') + [ -z "$actual" ] && actual="" + if echo "$actual" | grep -qF -- "$expected"; then + PASS=$((PASS+1)) + [ "$VERBOSE" = "-v" ] && echo " ok $desc" + else + FAIL=$((FAIL+1)) + ERRORS+=" FAIL [$desc] (epoch $epoch) expected: $expected | actual: $actual +" + fi +} + +check 10 "subtype_of.sx head form" "DefineActivity" +check 11 "subtype_of.sx name" "SubtypeOf" +check 20 "schema accepts edge with 2 CIDs" "true" +check 21 "schema rejects missing child CID" "false" +check 22 "schema rejects missing parent CID" "false" +check 30 "SubtypeOf envelope round-trips" "true" + +TOTAL=$((PASS+FAIL)) +if [ $FAIL -eq 0 ]; then + echo "ok $PASS/$TOTAL next/tests/subtype_of.sh passed" +else + echo "FAIL $PASS/$TOTAL passed, $FAIL failed:" + echo "$ERRORS" +fi +[ $FAIL -eq 0 ] diff --git a/plans/fed-sx-host-types.md b/plans/fed-sx-host-types.md new file mode 100644 index 00000000..69ad2806 --- /dev/null +++ b/plans/fed-sx-host-types.md @@ -0,0 +1,106 @@ +; -*- mode: markdown -*- +# fed-sx host-type federation — substrate design + build log + +How a host's typed-post graph (refinement types declared in +`lib/host`'s metamodel) flows across fed-sx nodes: a type is published +as a content-addressed `DefineType` activity, peers cache its record, +serve it over the wire, and validate inbound objects against the +declared refinement schema before appending them. + +This document is both the design and the running build log for +`loops/fed-sx-types`. The companion build sheet is +`plans/agent-briefings/fed-sx-types-loop.md`. + +## Vocabulary + +- **Type record** — `{name, fields, refinement-schema, instance-type}`. + The parsed `:object` payload of a `DefineType` activity. Immutable + per CID: an updated type is a new CID (no in-place evolution). +- **Type CID** — content-address of the type record's wire form. The + stable handle a `SubtypeOf` edge or an object's `{type, _}` field + references. +- **Refinement schema** — a predicate over an object's field-values; + the extra constraint a refinement type adds on top of its base + `instance-type` (e.g. a `Post` is a `Note` whose `:title` is a + non-empty string). + +## Scope + +Substrate side only — everything under `next/**`. The host-side +adapters (`lib/host/fed_sx_outbox.sx`, `lib/host/fed_sx_inbox.sx`) +are a deliberate follow-up that consumes this branch's public surface +(`DefineType` / `SubtypeOf` verbs, `peer_types`, the `/types/` +route) once `loops/host`'s metamodel settles. **This loop does not +touch `lib/host/`.** + +## Steps + +### Step 1 — `DefineType` + `SubtypeOf` genesis activity-types — DONE + +New `DefineActivity`-form genesis files, parsed as data by +`bootstrap.erl` at startup (no kernel change yet): + +- `next/genesis/activity-types/define_type.sx` — declares the + `DefineType` verb. `:schema` accepts an activity whose `:object` + carries a string `:name` and an optional list `:fields`. +- `next/genesis/activity-types/subtype_of.sx` — declares the + `SubtypeOf` verb. `:schema` accepts an `:object` carrying both + `:child-type-cid` and `:parent-type-cid` as strings. + +Schema bodies are SX source written with nested `get` (not +keyword-threading) so they are directly evaluatable: keywords are not +callable getters in the kernel and `(-> d :k)` does not get. Both are +registered in `next/genesis/manifest.sx` (activity-types now 7) and the +bundle counts in the bootstrap suites were bumped accordingly. + +Tests: `next/tests/define_type.sh`, `next/tests/subtype_of.sh` — parse +shape, schema accept/reject, and a `term_codec` envelope round-trip. + +### Step 2 — `peer_types.erl` receiver-side cache — DONE + +`next/kernel/peer_types.erl`, a mirror of `peer_actors.erl` keyed by +type CID. State `[{TypeCidBytes, TypeRecord}, ...]`. Pure API +(`new/2`-threaded `lookup`/`store`/`evict`/`types`/`lookup_or_fetch`) +plus a registered gen_server (`put`, `lookup`, `state_for`, +`known_types`, `lookup_or_fetch`). On a miss `lookup_or_fetch` pulls a +Cfg-supplied `type_fetch_fn :: fun ((TypeCid, Cfg) -> {ok, Bytes} | +{error, _})`, decodes the wire bytes via `term_codec`, and caches the +record. No fn → `{error, no_fetch_fn}`; fetch error or bad bytes do not +poison the cache. Test: `next/tests/peer_types.sh`. + +### Step 3 — `/types/` route + `discovery_type_fetch.erl` — TODO + +`http_server.erl` serves `GET /types/` with +`Accept: application/vnd.fed-sx.type-doc`: the cached TypeRecord +`term_codec`-encoded, 404 if not cached. `discovery_type_fetch.erl` +holds the live-HTTP closure that `peer_types:lookup_or_fetch` calls. +Tests: `next/tests/peer_types_route.sh`, `next/tests/discovery_type_fetch.sh`. + +### Step 4 — object-schema validation stage in `pipeline.erl` — TODO + +A new `apply_object_schema/2` stage between activity-type validation +and the kernel append. When an inbound object carries `{type, TypeName}`, +resolve the TypeRecord (local Define-name index → CID → +`peer_types:lookup_or_fetch`) and apply its refinement schema to the +object's field-values. Default `strict_object_schema = false`: an +unresolvable type is let through with a `validation_skipped` log; +opt-in strict mode rejects. Test: `next/tests/object_schema.sh`. + +## Out of scope (deliberately) + +- Host-side outbox/inbox adapters (`lib/host/**`). +- Type evolution / version migration — schemas are immutable per CID; + the "name → currently-valid CID" routing layer is a separate problem. +- Subtype-of unification / rendering across nodes — the graph data + lands via `SubtypeOf` activities; dedup/display is a consumer concern. + +## What the host-side adapter loop gets + +Once all four steps land, the follow-up `loops/host` adapter work can +treat the following as stable public surface: + +- `DefineType` / `SubtypeOf` activity verbs (publish a type, link two). +- `peer_types` gen_server (cache a peer's type, look it up). +- `GET /types/` (serve a type the node knows). +- `pipeline`'s object-schema stage (inbound objects validated against + their declared refinement type when resolvable).