Add sexp ActivityPub extension plan with implementation phases

Defines a backwards-compatible AP extension using s-expressions as the wire format: content negotiation, component discovery protocol, WebSocket streaming, and a path to publishing as a FEP. Includes bidirectional JSON-LD bridging for Mastodon/Pleroma compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 13:40:15 +00:00
parent da8a766e3f
commit 5c6d83f474
1 changed files with 353 additions and 0 deletions
--- a/docs/sexpr-activitypub-extension.md
+++ b/docs/sexpr-activitypub-extension.md
@@ -0,0 +1,353 @@
+# S-expression Wire Format for ActivityPub
+
+**An AP extension that replaces JSON-LD with s-expressions as the native wire format.**
+
+---
+
+## Why JSON-LD Is the Wrong Format for ActivityPub
+
+JSON-LD was chosen for AP because of its semantic web lineage, but in practice nobody uses the linked data features — servers just pattern-match on `type` fields and ignore the `@context`. The format is:
+
+- **Verbose** — deeply nested objects with string keys, quoted values, commas, colons
+- **Ambiguous** — compaction/expansion rules mean the same activity can have many valid JSON representations
+- **Lossy** — post content is flattened to an HTML string inside a JSON string; structure is destroyed
+- **Hostile to signing** — JSON-LD canonicalization is fragile and implementation-dependent
+- **Hostile to AI** — agents must parse JSON, then parse embedded HTML, then reason about structure
+
+---
+
+## Sexp AP: The Format
+
+### Activities as Expressions
+
+Every AP activity type is a symbol — the verb is the head of the expression:
+
+```scheme
+(Create :actor alice
+  (Note :id post-123
+    (p "content here")))
+
+(Announce :actor bob
+  (Create :actor alice
+    (Note :id post-123)))
+
+(Undo :actor alice
+  (Like :actor alice :object post-123))
+```
+
+Read top to bottom: who did what to what. No `"type"` field to find, no `"object"` wrapper to unwrap.
+
+### Full Activity Example
+
+```scheme
+(Create
+  :id "https://rose-ash.com/ap/activities/456"
+  :actor "https://rose-ash.com/ap/users/alice"
+  :published "2026-02-28T14:00:00Z"
+  :to ("https://www.w3.org/ns/activitystreams#Public")
+  :cc ("https://rose-ash.com/ap/users/alice/followers")
+
+  (Note
+    :id "https://rose-ash.com/ap/posts/123"
+    :attributed-to "https://rose-ash.com/ap/users/alice"
+    :in-reply-to "https://remote.social/posts/789"
+    :published "2026-02-28T14:00:00Z"
+    :summary "Weekend market update"
+
+    ;; Content is sexp — structure preserved, not flattened to HTML string
+    (section
+      (p "Three new vendors joining this Saturday.")
+      (use "vendor-list" :vendors vendors)
+      (use "calendar-widget" :calendar-id 42))
+
+    ;; Attachments are typed expressions, not opaque JSON blobs
+    (attachment
+      (Image :url "https://rose-ash.com/media/market.jpg"
+        :media-type "image/jpeg"
+        :width 1200 :height 800
+        :name "Saturday market stalls"))
+
+    ;; Tags as data
+    (tag
+      (Hashtag :name "#markets" :href "https://rose-ash.com/tags/markets")
+      (Mention :name "@bob@remote.social" :href "https://remote.social/users/bob"))))
+```
+
+### JSON-LD Equivalent (for comparison)
+
+```json
+{
+  "@context": "https://www.w3.org/ns/activitystreams",
+  "type": "Create",
+  "id": "https://rose-ash.com/ap/activities/456",
+  "actor": "https://rose-ash.com/ap/users/alice",
+  "published": "2026-02-28T14:00:00Z",
+  "to": ["https://www.w3.org/ns/activitystreams#Public"],
+  "cc": ["https://rose-ash.com/ap/users/alice/followers"],
+  "object": {
+    "type": "Note",
+    "id": "https://rose-ash.com/ap/posts/123",
+    "attributedTo": "https://rose-ash.com/ap/users/alice",
+    "inReplyTo": "https://remote.social/posts/789",
+    "published": "2026-02-28T14:00:00Z",
+    "summary": "Weekend market update",
+    "content": "<p>Three new vendors joining this Saturday.</p>",
+    "attachment": [
+      {
+        "type": "Image",
+        "url": "https://rose-ash.com/media/market.jpg",
+        "mediaType": "image/jpeg",
+        "width": 1200,
+        "height": 800,
+        "name": "Saturday market stalls"
+      }
+    ],
+    "tag": [
+      {"type": "Hashtag", "name": "#markets", "href": "https://rose-ash.com/tags/markets"},
+      {"type": "Mention", "name": "@bob@remote.social", "href": "https://remote.social/users/bob"}
+    ]
+  }
+}
+```
+
+The JSON-LD is twice the size. The actual content — the post body — is a flat HTML string inside a JSON string. Structure lost.
+
+### Collections as Lists
+
+An outbox page:
+
+```scheme
+(OrderedCollectionPage
+  :id "https://rose-ash.com/ap/users/alice/outbox?page=1"
+  :part-of "https://rose-ash.com/ap/users/alice/outbox"
+  :next "https://rose-ash.com/ap/users/alice/outbox?page=2"
+  :total-items 142
+
+  (Create :actor alice :published "2026-02-28T14:00:00Z"
+    (Note :id post-3 (p "Latest post")))
+  (Announce :actor alice :published "2026-02-27T09:00:00Z"
+    (Note :id post-2 :attributed-to bob))
+  (Create :actor alice :published "2026-02-26T18:00:00Z"
+    (Note :id post-1 (p "Earlier post"))))
+```
+
+Items are children of the collection expression. No `"orderedItems": [...]` wrapper.
+
+### Actor Profiles
+
+```scheme
+(Person
+  :id "https://rose-ash.com/ap/users/alice"
+  :preferred-username "alice"
+  :name "Alice"
+  :summary (p "Co-op member, market organiser")
+  :inbox "https://rose-ash.com/ap/users/alice/inbox"
+  :outbox "https://rose-ash.com/ap/users/alice/outbox"
+  :followers "https://rose-ash.com/ap/users/alice/followers"
+  :following "https://rose-ash.com/ap/users/alice/following"
+  :public-key (:id "https://rose-ash.com/ap/users/alice#main-key"
+               :owner "https://rose-ash.com/ap/users/alice"
+               :pem "-----BEGIN PUBLIC KEY-----\n..."))
+```
+
+---
+
+## Content Negotiation
+
+Servers advertise sexp support via HTTP headers:
+
+```
+Accept: application/x-sexpr+ap, application/activity+json;q=0.9
+Content-Type: application/x-sexpr+ap; charset=utf-8
+```
+
+- Servers that don't understand sexp get standard `application/activity+json` (JSON-LD)
+- Servers that do get `application/x-sexpr+ap` — compact, structured, renderable
+- Graceful degradation built in from day one
+
+### Dual-Content Bridging
+
+For JSON-LD peers, include sexp as an extension field:
+
+```json
+{
+  "@context": ["https://www.w3.org/ns/activitystreams", "https://rose-ash.com/ns/sexpr"],
+  "type": "Create",
+  "object": {
+    "type": "Note",
+    "content": "<p>Hello world</p>",
+    "rose:sexpr": "(p \"Hello world\")"
+  }
+}
+```
+
+Remote Mastodon/Pleroma servers see standard `content` HTML. Rose-ash instances see `rose:sexpr` and render natively — richer, smaller, with embedded components.
+
+---
+
+## Advantages
+
+### Deterministic Signatures
+
+Sexp serialization has a canonical form: sorted keywords, no trailing whitespace, single-space separators. HTTP Signatures and content hashes are reproducible without JSON-LD canonicalization — which is notoriously fragile and implementation-dependent.
+
+### Content Is Structure
+
+In JSON-LD AP, post content is `"content": "<p>Hello</p>"` — an HTML string embedded in JSON. Clients must parse JSON, extract the string, then parse the HTML. Two parse steps, structure lost at each.
+
+In sexp AP, the content *is* the tree. `(p "Hello")` is simultaneously the wire format, the parse result, and the renderable document. One parse step, structure preserved.
+
+### Components Cross the Federation
+
+A rose-ash instance sending `(use "product-card" :name "Sourdough" :price "3.50")` to another rose-ash instance sends a component invocation. If both instances share the same component definition (identified by content hash), the receiving instance renders a full product card — not just flat text.
+
+This is federated UI, not just federated data. The artdag content-addressing model applies: components are content-addressed artifacts discoverable across the federation.
+
+### AI-Native
+
+An AI agent consuming a sexp AP feed gets structured trees it can parse, reason about, and respond to — not HTML soup embedded in JSON. It can generate replies as sexp, which the server translates to AP activities. The agent's tool vocabulary (components) is the same as the federation's content vocabulary.
+
+### Size
+
+Typical `Create{Note}` activity: ~800 bytes JSON-LD vs ~400 bytes sexp. With component references instead of inline content: even smaller. Compresses better with gzip/brotli due to repetitive keywords.
+
+---
+
+## Implementation Plan
+
+### Phase 1: Sexp ↔ AP Serialization Layer
+
+Build bidirectional translation between sexp and JSON-LD AP:
+
+**sexp → JSON-LD** (outbound to non-sexp peers):
+- Activity type symbol → `"type"` field
+- Keyword attributes → JSON object properties
+- Child expressions → nested `"object"`, `"attachment"`, `"tag"` arrays
+- Content sexp children → rendered to HTML string for `"content"` field
+- Deterministic: same sexp always produces same JSON-LD
+
+**JSON-LD → sexp** (inbound from any peer):
+- `"type"` → head symbol
+- Object properties → keyword attributes
+- `"content"` HTML → parsed to sexp (best-effort; HTML→sexp parser)
+- Nested objects → child expressions
+
+**Files:**
+- `shared/infrastructure/ap_sexpr.py` — serializer/deserializer
+- `shared/sexp/tests/test_ap_sexpr.py` — round-trip tests
+
+### Phase 2: Content Negotiation in Federation Service
+
+Add `Accept` header handling to the federation service inbox/outbox:
+
+- Outbound: check if target server accepts `application/x-sexpr+ap`
+  - If yes: send sexp directly
+  - If no: translate to JSON-LD, include `rose:sexpr` extension field
+- Inbound: accept both `application/x-sexpr+ap` and `application/activity+json`
+  - If sexp: parse directly
+  - If JSON-LD: translate to sexp internally
+- WebFinger: advertise sexp support in actor metadata
+
+**Files:**
+- `federation/bp/inbox/routes.py` — add content-type handling
+- `federation/bp/outbox/routes.py` — add accept-header negotiation
+- `federation/services/delivery.py` — per-peer format selection
+
+### Phase 3: Sexp Content in Activities
+
+Replace HTML content with sexp in outbound activities:
+
+- When publishing a post, the sexp body becomes the canonical AP content
+- For sexp peers: content is sent as-is (with component references)
+- For JSON-LD peers: sexp is rendered to HTML for the `"content"` field
+- Inbound HTML content from remote peers: parsed to sexp for storage/display
+
+**Depends on:** Ghost removal (Phase 1-5 of ghost-removal-plan.md) — posts must be sexp before they can be federated as sexp.
+
+**Files:**
+- `federation/services/publishing.py` — sexp content in Create/Update activities
+- `shared/sexp/html_to_sexp.py` — HTML → sexp parser for inbound content
+
+### Phase 4: Component Discovery Protocol
+
+Enable sexp-speaking peers to discover and cache shared components:
+
+- Actor metadata includes a `components` endpoint URL
+- `GET /ap/components/` returns a manifest: component name → content hash
+- `GET /ap/components/{hash}` returns the component definition (sexp)
+- Receiving instance caches component definitions by hash
+- Component invocations in federated content (`use "product-card" ...`) resolve from cache
+
+**Files:**
+- `federation/bp/components/routes.py` — component manifest + fetch endpoints
+- `shared/sexp/component_manifest.py` — hash generation, manifest building
+
+### Phase 5: Real-Time Federation via WebSocket
+
+For sexp-speaking peers, offer a persistent WebSocket connection instead of HTTP POST delivery:
+
+- Peers negotiate WS upgrade during Follow/Accept handshake
+- Activities stream as sexp over the WebSocket
+- Includes mutation commands for live UI updates:
+  ```scheme
+  (swap! "#federation-feed" :prepend
+    (use "federation-post"
+      :actor "alice@remote.social"
+      :content (p "Just posted!")
+      :published "2026-02-28T14:00:00Z"))
+  ```
+- Fallback: HTTP POST delivery for peers that don't support WS
+
+**Files:**
+- `federation/services/ws_delivery.py` — WebSocket connection manager
+- `federation/bp/ws/routes.py` — WebSocket endpoint
+
+### Phase 6: Specification Document
+
+Write a formal extension specification:
+
+- MIME type: `application/x-sexpr+ap`
+- Canonical serialization rules (for signatures)
+- Activity type → symbol mapping (all standard AP types)
+- Content negotiation protocol
+- Component discovery protocol
+- WebSocket streaming protocol
+- JSON-LD bridging rules (`rose:sexpr` extension field)
+- Security considerations (no eval, escaping rules, signature verification)
+
+Publish as a FEP (Fediverse Enhancement Proposal).
+
+---
+
+## Backwards Compatibility
+
+The extension is fully backwards-compatible:
+
+| Peer Type | Outbound Behaviour | Inbound Behaviour |
+|---|---|---|
+| Standard AP (Mastodon, Pleroma) | JSON-LD with `rose:sexpr` extension field | Parse JSON-LD, translate to sexp internally |
+| Sexp-aware AP (other rose-ash instances) | Native sexp, no translation | Parse sexp directly |
+| Sexp-aware with shared components | Sexp with component references | Resolve components from cache, render natively |
+
+No existing AP implementation breaks. Sexp-unaware servers ignore the `rose:sexpr` field and use `content` HTML as always. Sexp-aware servers get a richer, faster, more structured exchange.
+
+---
+
+## Relationship to Other Plans
+
+- **sexpr-js-runtime-plan.md** — the client runtime that renders federated sexp content in the browser
+- **ghost-removal-plan.md** — posts must be sexp before they can be federated as sexp (Phase 3 dependency)
+- **sexpr-ai-integration.md** — AI agents consuming/producing federated content benefit from sexp structure
+- **artdag** — content-addressed component definitions use the same CID model as artdag artifacts
+
+---
+
+## Rose-Ash Specific Benefits
+
+Rose-ash's virtual actor projections (blog, market, events services sharing a keypair) currently emit JSON-LD activities. With sexp AP:
+
+- A market actor's `Create{Product}` activity includes `(use "product-card" ...)` — followers on other rose-ash instances see a full product card, not just a text description
+- A calendar actor's `Create{Event}` includes `(use "calendar-widget" ...)` — followers see an interactive calendar entry with ticket availability
+- Cross-instance federation becomes rich UI exchange, not just text+image syndication
+
+The component registry becomes a shared vocabulary across the cooperative network. Instances that share components render each other's content natively. The fediverse becomes a distributed application platform, not just a social network.