coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Unify sexp protocol and ActivityPub extension into single spec
All checks were successful
Build and Deploy / build-and-deploy (push) Successful in 2m30s
Details

Browse Source 
Merges sexpr-activitypub-extension.md and sexpr-protocol-and-tiered-clients.md
into sexpr-unified-protocol.md — recognising that browsing, federation, and
real-time updates are all the same thing: peers exchanging s-expressions on
a bidirectional stream. One format, one connection, one parser.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
giles
2026-02-28 14:44:39 +00:00
parent a70d3648ec
commit 3e29c2a334
3 changed files with 731 additions and 807 deletions
Download Patch File Download Diff File Expand all files Collapse all files

353
docs/sexpr-activitypub-extension.md
View File

@@ -1,353 +0,0 @@
# S-expression Wire Format for ActivityPub
**An AP extension that replaces JSON-LD with s-expressions as the native wire format.**
---
## Why JSON-LD Is the Wrong Format for ActivityPub
JSON-LD was chosen for AP because of its semantic web lineage, but in practice nobody uses the linked data features — servers just pattern-match on `type` fields and ignore the `@context`. The format is:
- **Verbose** — deeply nested objects with string keys, quoted values, commas, colons
- **Ambiguous** — compaction/expansion rules mean the same activity can have many valid JSON representations
- **Lossy** — post content is flattened to an HTML string inside a JSON string; structure is destroyed
- **Hostile to signing** — JSON-LD canonicalization is fragile and implementation-dependent
- **Hostile to AI** — agents must parse JSON, then parse embedded HTML, then reason about structure
---
## Sexp AP: The Format
### Activities as Expressions
Every AP activity type is a symbol — the verb is the head of the expression:
```scheme
(Create :actor alice
(Note :id post-123
(p "content here")))
(Announce :actor bob
(Create :actor alice
(Note :id post-123)))
(Undo :actor alice
(Like :actor alice :object post-123))
```
Read top to bottom: who did what to what. No `"type"` field to find, no `"object"` wrapper to unwrap.
### Full Activity Example
```scheme
(Create
:id "https://rose-ash.com/ap/activities/456"
:actor "https://rose-ash.com/ap/users/alice"
:published "2026-02-28T14:00:00Z"
:to ("https://www.w3.org/ns/activitystreams#Public")
:cc ("https://rose-ash.com/ap/users/alice/followers")
(Note
:id "https://rose-ash.com/ap/posts/123"
:attributed-to "https://rose-ash.com/ap/users/alice"
:in-reply-to "https://remote.social/posts/789"
:published "2026-02-28T14:00:00Z"
:summary "Weekend market update"
;; Content is sexp — structure preserved, not flattened to HTML string
(section
(p "Three new vendors joining this Saturday.")
(use "vendor-list" :vendors vendors)
(use "calendar-widget" :calendar-id 42))
;; Attachments are typed expressions, not opaque JSON blobs
(attachment
(Image :url "https://rose-ash.com/media/market.jpg"
:media-type "image/jpeg"
:width 1200 :height 800
:name "Saturday market stalls"))
;; Tags as data
(tag
(Hashtag :name "#markets" :href "https://rose-ash.com/tags/markets")
(Mention :name "@bob@remote.social" :href "https://remote.social/users/bob"))))
```
### JSON-LD Equivalent (for comparison)
```json
{
"@context": "https://www.w3.org/ns/activitystreams",
"type": "Create",
"id": "https://rose-ash.com/ap/activities/456",
"actor": "https://rose-ash.com/ap/users/alice",
"published": "2026-02-28T14:00:00Z",
"to": ["https://www.w3.org/ns/activitystreams#Public"],
"cc": ["https://rose-ash.com/ap/users/alice/followers"],
"object": {
"type": "Note",
"id": "https://rose-ash.com/ap/posts/123",
"attributedTo": "https://rose-ash.com/ap/users/alice",
"inReplyTo": "https://remote.social/posts/789",
"published": "2026-02-28T14:00:00Z",
"summary": "Weekend market update",
"content": "<p>Three new vendors joining this Saturday.</p>",
"attachment": [
{
"type": "Image",
"url": "https://rose-ash.com/media/market.jpg",
"mediaType": "image/jpeg",
"width": 1200,
"height": 800,
"name": "Saturday market stalls"
}
],
"tag": [
{"type": "Hashtag", "name": "#markets", "href": "https://rose-ash.com/tags/markets"},
{"type": "Mention", "name": "@bob@remote.social", "href": "https://remote.social/users/bob"}
]
}
}
```
The JSON-LD is twice the size. The actual content — the post body — is a flat HTML string inside a JSON string. Structure lost.
### Collections as Lists
An outbox page:
```scheme
(OrderedCollectionPage
:id "https://rose-ash.com/ap/users/alice/outbox?page=1"
:part-of "https://rose-ash.com/ap/users/alice/outbox"
:next "https://rose-ash.com/ap/users/alice/outbox?page=2"
:total-items 142
(Create :actor alice :published "2026-02-28T14:00:00Z"
(Note :id post-3 (p "Latest post")))
(Announce :actor alice :published "2026-02-27T09:00:00Z"
(Note :id post-2 :attributed-to bob))
(Create :actor alice :published "2026-02-26T18:00:00Z"
(Note :id post-1 (p "Earlier post"))))
```
Items are children of the collection expression. No `"orderedItems": [...]` wrapper.
### Actor Profiles
```scheme
(Person
:id "https://rose-ash.com/ap/users/alice"
:preferred-username "alice"
:name "Alice"
:summary (p "Co-op member, market organiser")
:inbox "https://rose-ash.com/ap/users/alice/inbox"
:outbox "https://rose-ash.com/ap/users/alice/outbox"
:followers "https://rose-ash.com/ap/users/alice/followers"
:following "https://rose-ash.com/ap/users/alice/following"
:public-key (:id "https://rose-ash.com/ap/users/alice#main-key"
:owner "https://rose-ash.com/ap/users/alice"
:pem "-----BEGIN PUBLIC KEY-----\n..."))
```
---
## Content Negotiation
Servers advertise sexp support via HTTP headers:
```
Accept: application/x-sexpr+ap, application/activity+json;q=0.9
Content-Type: application/x-sexpr+ap; charset=utf-8
```
- Servers that don't understand sexp get standard `application/activity+json` (JSON-LD)
- Servers that do get `application/x-sexpr+ap` — compact, structured, renderable
- Graceful degradation built in from day one
### Dual-Content Bridging
For JSON-LD peers, include sexp as an extension field:
```json
{
"@context": ["https://www.w3.org/ns/activitystreams", "https://rose-ash.com/ns/sexpr"],
"type": "Create",
"object": {
"type": "Note",
"content": "<p>Hello world</p>",
"rose:sexpr": "(p \"Hello world\")"
}
}
```
Remote Mastodon/Pleroma servers see standard `content` HTML. Rose-ash instances see `rose:sexpr` and render natively — richer, smaller, with embedded components.
---
## Advantages
### Deterministic Signatures
Sexp serialization has a canonical form: sorted keywords, no trailing whitespace, single-space separators. HTTP Signatures and content hashes are reproducible without JSON-LD canonicalization — which is notoriously fragile and implementation-dependent.
### Content Is Structure
In JSON-LD AP, post content is `"content": "<p>Hello</p>"` — an HTML string embedded in JSON. Clients must parse JSON, extract the string, then parse the HTML. Two parse steps, structure lost at each.
In sexp AP, the content *is* the tree. `(p "Hello")` is simultaneously the wire format, the parse result, and the renderable document. One parse step, structure preserved.
### Components Cross the Federation
A rose-ash instance sending `(use "product-card" :name "Sourdough" :price "3.50")` to another rose-ash instance sends a component invocation. If both instances share the same component definition (identified by content hash), the receiving instance renders a full product card — not just flat text.
This is federated UI, not just federated data. The artdag content-addressing model applies: components are content-addressed artifacts discoverable across the federation.
### AI-Native
An AI agent consuming a sexp AP feed gets structured trees it can parse, reason about, and respond to — not HTML soup embedded in JSON. It can generate replies as sexp, which the server translates to AP activities. The agent's tool vocabulary (components) is the same as the federation's content vocabulary.
### Size
Typical `Create{Note}` activity: ~800 bytes JSON-LD vs ~400 bytes sexp. With component references instead of inline content: even smaller. Compresses better with gzip/brotli due to repetitive keywords.
---
## Implementation Plan
### Phase 1: Sexp ↔ AP Serialization Layer
Build bidirectional translation between sexp and JSON-LD AP:
**sexp → JSON-LD** (outbound to non-sexp peers):
- Activity type symbol → `"type"` field
- Keyword attributes → JSON object properties
- Child expressions → nested `"object"`, `"attachment"`, `"tag"` arrays
- Content sexp children → rendered to HTML string for `"content"` field
- Deterministic: same sexp always produces same JSON-LD
**JSON-LD → sexp** (inbound from any peer):
- `"type"` → head symbol
- Object properties → keyword attributes
- `"content"` HTML → parsed to sexp (best-effort; HTML→sexp parser)
- Nested objects → child expressions
**Files:**
- `shared/infrastructure/ap_sexpr.py` — serializer/deserializer
- `shared/sexp/tests/test_ap_sexpr.py` — round-trip tests
### Phase 2: Content Negotiation in Federation Service
Add `Accept` header handling to the federation service inbox/outbox:
- Outbound: check if target server accepts `application/x-sexpr+ap`
- If yes: send sexp directly
- If no: translate to JSON-LD, include `rose:sexpr` extension field
- Inbound: accept both `application/x-sexpr+ap` and `application/activity+json`
- If sexp: parse directly
- If JSON-LD: translate to sexp internally
- WebFinger: advertise sexp support in actor metadata
**Files:**
- `federation/bp/inbox/routes.py` — add content-type handling
- `federation/bp/outbox/routes.py` — add accept-header negotiation
- `federation/services/delivery.py` — per-peer format selection
### Phase 3: Sexp Content in Activities
Replace HTML content with sexp in outbound activities:
- When publishing a post, the sexp body becomes the canonical AP content
- For sexp peers: content is sent as-is (with component references)
- For JSON-LD peers: sexp is rendered to HTML for the `"content"` field
- Inbound HTML content from remote peers: parsed to sexp for storage/display
**Depends on:** Ghost removal (Phase 1-5 of ghost-removal-plan.md) — posts must be sexp before they can be federated as sexp.
**Files:**
- `federation/services/publishing.py` — sexp content in Create/Update activities
- `shared/sexp/html_to_sexp.py` — HTML → sexp parser for inbound content
### Phase 4: Component Discovery Protocol
Enable sexp-speaking peers to discover and cache shared components:
- Actor metadata includes a `components` endpoint URL
- `GET /ap/components/` returns a manifest: component name → content hash
- `GET /ap/components/{hash}` returns the component definition (sexp)
- Receiving instance caches component definitions by hash
- Component invocations in federated content (`use "product-card" ...`) resolve from cache
**Files:**
- `federation/bp/components/routes.py` — component manifest + fetch endpoints
- `shared/sexp/component_manifest.py` — hash generation, manifest building
### Phase 5: Real-Time Federation via WebSocket
For sexp-speaking peers, offer a persistent WebSocket connection instead of HTTP POST delivery:
- Peers negotiate WS upgrade during Follow/Accept handshake
- Activities stream as sexp over the WebSocket
- Includes mutation commands for live UI updates:
```scheme
(swap! "#federation-feed" :prepend
(use "federation-post"
:actor "alice@remote.social"
:content (p "Just posted!")
:published "2026-02-28T14:00:00Z"))
```
- Fallback: HTTP POST delivery for peers that don't support WS
**Files:**
- `federation/services/ws_delivery.py` — WebSocket connection manager
- `federation/bp/ws/routes.py` — WebSocket endpoint
### Phase 6: Specification Document
Write a formal extension specification:
- MIME type: `application/x-sexpr+ap`
- Canonical serialization rules (for signatures)
- Activity type → symbol mapping (all standard AP types)
- Content negotiation protocol
- Component discovery protocol
- WebSocket streaming protocol
- JSON-LD bridging rules (`rose:sexpr` extension field)
- Security considerations (no eval, escaping rules, signature verification)
Publish as a FEP (Fediverse Enhancement Proposal).
---
## Backwards Compatibility
The extension is fully backwards-compatible:
| Peer Type | Outbound Behaviour | Inbound Behaviour |
|---|---|---|
| Standard AP (Mastodon, Pleroma) | JSON-LD with `rose:sexpr` extension field | Parse JSON-LD, translate to sexp internally |
| Sexp-aware AP (other rose-ash instances) | Native sexp, no translation | Parse sexp directly |
| Sexp-aware with shared components | Sexp with component references | Resolve components from cache, render natively |
No existing AP implementation breaks. Sexp-unaware servers ignore the `rose:sexpr` field and use `content` HTML as always. Sexp-aware servers get a richer, faster, more structured exchange.
---
## Relationship to Other Plans
- **sexpr-js-runtime-plan.md** — the client runtime that renders federated sexp content in the browser
- **ghost-removal-plan.md** — posts must be sexp before they can be federated as sexp (Phase 3 dependency)
- **sexpr-ai-integration.md** — AI agents consuming/producing federated content benefit from sexp structure
- **artdag** — content-addressed component definitions use the same CID model as artdag artifacts
---
## Rose-Ash Specific Benefits
Rose-ash's virtual actor projections (blog, market, events services sharing a keypair) currently emit JSON-LD activities. With sexp AP:
- A market actor's `Create{Product}` activity includes `(use "product-card" ...)` — followers on other rose-ash instances see a full product card, not just a text description
- A calendar actor's `Create{Event}` includes `(use "calendar-widget" ...)` — followers see an interactive calendar entry with ticket availability
- Cross-instance federation becomes rich UI exchange, not just text+image syndication
The component registry becomes a shared vocabulary across the cooperative network. Instances that share components render each other's content natively. The fediverse becomes a distributed application platform, not just a social network.

454
docs/sexpr-protocol-and-tiered-clients.md
View File

@@ -1,454 +0,0 @@
# Sexp Protocol and Tiered Client Architecture
**One server, three clients, progressive enhancement from HTML to native.**
---
## Overview
The same rose-ash application serves three tiers of client from the same route handlers and component trees:
```
┌─────────────────────────────┐
│ rose-ash server (Quart) │
│ │
│ Same sexp component tree │
│ Same data, same logic │
│ │
├──────────┬──────────┬────────┤
│ HTTPS │ HTTPS │ SEXPR │
│ HTML out │ sexp out │ native │
└────┬─────┴────┬─────┴───┬────┘
│ │ │
┌──────────▼──┐ ┌─────▼──────┐ ┌▼──────────────┐
│ Browser │ │ Browser + │ │ Rust client │
│ (vanilla) │ │ extension │ │ (native) │
│ │ │ │ │ │
│ HTML + HTMX │ │ sexpr.js │ │ sexp protocol │
│ Full CSS │ │ over HTTPS │ │ over QUIC │
│ ~200ms load │ │ ~80ms load │ │ ~20ms load │
└─────────────┘ └────────────┘ └────────────────┘
Tier 0 Tier 1 Tier 2
```
---
## Content Negotiation
The server does format selection on a single endpoint:
```python
@bp.get("/markets/")
async def markets():
# Same data, same component tree, always
data = await get_markets(g.s)
tree = sexp('(page :title "Markets" (each markets ...))', markets=data)
accept = request.headers.get("Accept", "")
if "application/x-sexpr" in accept:
# Tier 1 (extension) or Tier 2 (native): raw sexp
return Response(serialize(tree), content_type="application/x-sexpr")
# Tier 0: render to HTML for vanilla browsers
html = render_to_html(tree)
return Response(html, content_type="text/html")
```
One route. One component tree. The output format is the only thing that changes.
---
## Tier Comparison
| | Tier 0: Browser | Tier 1: Extension | Tier 2: Rust Client |
|---|---|---|---|
| URL | `https://rose-ash.com` | `https://rose-ash.com` | `sexpr://rose-ash.com` |
| Protocol | HTTPS | HTTPS | sexpr:// over QUIC |
| Wire format | HTML | sexp over HTTP | sexp native |
| Rendering | Browser DOM | sexpr.js → DOM | Rust → GPU |
| Component cache | Browser cache (URL-keyed) | IndexedDB (hash-keyed) | Disk (hash-keyed, pre-parsed AST) |
| Real-time | HTMX polling / SSE | WebSocket sexp mutations | Native bidirectional stream |
| Bundle size | HTMX 14KB + CSS | sexpr.js ~8KB | 5MB binary, zero runtime deps |
| Page load | ~200ms | ~80ms | ~20ms |
| Memory | ~200MB per tab | ~150MB per tab | ~20MB |
| Offline | Service worker (if built) | Component cache + manifest | Full local cache |
| AI agent | Parse HTML (painful) | Parse sexp (easy) | Native sexp (trivial) |
| Federation | N/A | AP via fetch | Native AP sexp stream |
---
## The Sexp Protocol (Tier 2)
### Why Not Just HTTP
HTTP is a request-response protocol with text headers and a body. Strip away the historical baggage and what you actually need is:
```
Client sends: method + path + metadata + optional body
Server sends: status + metadata + body
```
That's just an s-expression:
```scheme
;; Request
(GET "/markets/"
:accept "application/x-sexpr"
:auth "bearer tok_abc123"
:if-none-match "sha3-a1b2c3")
;; Response
(200
:content-hash "sha3-d4e5f6"
:cache :immutable
:components ("sha3-aaa" "sha3-bbb")
(page :title "Markets" :layout "main"
(section (h1 "Markets")
(each markets (lambda (m)
(use "vendor-card" :name (get m "name")))))))
```
No header/body split. No chunked transfer encoding. No `Content-Length` because the parser knows when the expression ends (balanced parens). No `Content-Type` because everything is sexp. No MIME, no multipart, no `Transfer-Encoding`, no `Connection: keep-alive` negotiation.
### What HTTP Gets Wrong That Sexp Fixes
#### 1. Headers are a bad key-value format
HTTP headers are case-insensitive, can be duplicated, have weird continuation rules, and are parsed separately from the body. In sexp, metadata is just keywords in the expression — same parser, same format, no special case.
#### 2. Request/response is too rigid
HTTP is strictly one request → one response. To get around this, we've bolted on WebSocket (separate protocol, upgrade handshake), Server-Sent Events (hack using chunked encoding), HTTP/2 server push (failed, being removed), and HTTP/3 QUIC streams (complex, still one-request-one-response per stream).
A sexp protocol is **bidirectional from the start**:
```scheme
;; Client opens connection, sends request
(GET "/feed/" :stream #t)
;; Server responds with initial content
(200 :stream #t
(page :title "Feed"
(div :id "feed" (p "Loading..."))))
;; Server pushes updates as they happen (same connection)
(push! (swap! "#feed" :prepend
(use "post-card" :title "New post" :author "alice")))
;; Client sends an action (same connection)
(POST "/like/" :body (:post-id 123))
;; Server responds with mutation
(push! (swap! "#like-count-123" :inner "43"))
;; Client navigates (same connection, no new handshake)
(GET "/markets/")
;; Server responds
(200 (page :title "Markets" ...))
```
One persistent connection. Requests, responses, and pushes are all sexp expressions on the same stream. No protocol upgrade, no separate WebSocket connection, no polling.
#### 3. Caching is overcomplicated
HTTP caching: `Cache-Control`, `ETag`, `If-None-Match`, `If-Modified-Since`, `Vary`, `Age`, `Expires`, `Last-Modified`, `s-maxage`, `stale-while-revalidate`... a dozen headers with complex interaction rules.
With content-addressed sexp:
```scheme
;; Request with known hash
(GET "/markets/" :have "sha3-a1b2c3")
;; If unchanged:
(304)
;; If changed:
(200 :hash "sha3-d4e5f6" (page ...))
```
One field. The hash *is* the cache key, the ETag, and the content address.
Components take this further — the client sends what it already has:
```scheme
(GET "/markets/"
:have-components ("sha3-aaa" "sha3-bbb" "sha3-ccc"))
;; Server only sends components the client is missing
(200
:new-components (
(component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
(page ...))
```
#### 4. Status codes are arbitrary numbers
```scheme
(ok (page ...))
(redirect :to "/new-location/" :permanent #t)
(not-found :message "Page does not exist")
(error :code "auth-required" :message "Please log in"
:login-url "sexpr://rose-ash.com/login/")
```
The status *is* the expression. Machines pattern-match on the head symbol. Humans read it. AI agents understand it without a lookup table.
#### 5. Forms and file uploads are a mess
HTTP: `application/x-www-form-urlencoded` or `multipart/form-data` with boundary strings, MIME parts, content-disposition headers.
Sexp:
```scheme
(POST "/submit/"
:body (
:username "alice"
:email "alice@example.com"
:avatar (file :name "photo.jpg" :type "image/jpeg" :data <binary>)))
```
Structured data with inline binary. One format.
#### 6. The protocol is self-describing
HTTP has no introspection. You need OpenAPI/Swagger specs bolted on separately.
```scheme
(GET "/__schema/")
;; Response: the API describes itself
(schema
(endpoint "/" :method GET
:returns (page)
:params (:stream bool))
(endpoint "/markets/" :method GET
:returns (page :contains (list (use "vendor-card"))))
(endpoint "/like/" :method POST
:params (:post-id int)
:returns (mutation)))
```
An AI agent hitting `/__schema/` learns the entire API surface as parseable sexp. No separate OpenAPI doc. The schema *is* the API.
### Protocol Stack
```
┌─────────────────────────────────────────┐
│ Application: sexp documents, mutations │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ Session: bidirectional sexp stream │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ Transport: QUIC (or TCP+TLS) │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ Network: IP │
└─────────────────────────────────────────┘
```
QUIC is ideal — multiplexed streams, built-in TLS, connection migration. Each sexp expression gets its own QUIC stream. Requests and pushes multiplex without head-of-line blocking. Connection survives network changes (mobile → wifi).
---
## Rust Client Architecture
```
┌─────────────────────────────────────┐
│ sexpr-client (Rust) │
│ │
│ ┌──────────┐ ┌─────────────────┐ │
│ │ Parser │ │ Component │ │
│ │ (zero- │ │ Cache │ │
│ │ copy) │ │ (SHA3 → AST) │ │
│ └──────────┘ └─────────────────┘ │
│ ┌──────────┐ ┌─────────────────┐ │
│ │ Layout │ │ Network │ │
│ │ Engine │ │ (tokio + QUIC) │ │
│ └──────────┘ └─────────────────┘ │
│ ┌──────────────────────────────┐ │
│ │ Renderer (wgpu / iced) │ │
│ └──────────────────────────────┘ │
└─────────────────────────────────────┘
```
**Why Rust makes this fast:**
- **Parser**: sexp parsing is trivial recursive descent. Nanoseconds per node. 100-1000x faster than JS.
- **Renderer**: Skip the DOM entirely. Render directly to GPU-backed surface via `wgpu` or `iced`.
- **Networking**: `reqwest` + `tokio` with connection pooling. Component manifests fetched in parallel.
- **Component cache**: Pre-parsed ASTs on disk, content-addressed. No parse step on cache hit.
- **Memory**: No GC pauses. 10-50MB where a browser tab uses 200-500MB.
**What you skip by not being a browser:**
- No HTML parser (5-10ms per page)
- No CSS cascade resolution (the most expensive part of browser rendering)
- No DOM construction (sexp AST *is* the tree)
- No JavaScript engine (logic in the sexp evaluator, compiled Rust)
- No security sandbox overhead (no arbitrary JS execution)
- No 2000+ web platform APIs you don't use
Page load: network (~50ms) → parse (~0.1ms) → layout (~2ms) → paint (~3ms) = **under 60ms**.
---
## Fallback Gateway
For users without the extension or native client, a gateway translates:
```
sexpr://rose-ash.com/markets/
→ Gateway fetches sexp from server
→ Renders to HTML
→ Serves to browser at https://gateway.rose-ash.com/markets/
```
The regular web always works. The sexp protocol is an acceleration layer, not a requirement.
---
## Implementation Plan
### Phase 1: Content Negotiation (Quart)
Add `Accept` header handling to existing Quart routes.
- Check for `application/x-sexpr` in `Accept` header
- If present: serialize the sexp tree and return directly (skip HTML render)
- If absent: render to HTML as today
- Add `Vary: Accept` header to responses
**Files:**
- `shared/infrastructure/factory.py` — middleware for content negotiation
- `shared/sexp/serialize.py` — canonical sexp serializer (deterministic output for caching/signing)
**Result:** Existing routes serve sexp to any client that asks. Zero changes to route logic.
### Phase 2: sexpr.js Client Library
Build the browser-side JS runtime (see `sexpr-js-runtime-plan.md` for full details).
- Parser + renderer: `parse()` → AST → `renderToDOM()`
- Mutation engine: `swap!`, `batch!`, `class!`, `request!`
- Component registry with localStorage cache (content-addressed)
- `<script src="sexpr.js">` drop-in for any page
**Result:** Any page can opt into client-side sexp rendering.
### Phase 3: Browser Extension
Package sexpr.js as a WebExtension that intercepts `sexpr://` URLs.
- Register `sexpr://` protocol handler
- Intercept navigation → fetch via HTTPS with `Accept: application/x-sexpr`
- Parse response → render to DOM in current tab
- Component cache in extension storage (IndexedDB, content-addressed)
- Address bar shows `sexpr://rose-ash.com/markets/`
**Tech:** WebExtension API (Firefox + Chrome), JS/TS.
**Result:** Users install extension, navigate to `sexpr://` URLs, get sexp-rendered pages.
### Phase 4: Sexp Protocol Specification
Define the native protocol:
- **Framing**: length-prefixed sexp expressions over QUIC streams
- **Request format**: `(METHOD path :keyword value ... body)`
- **Response format**: `(status :keyword value ... body)`
- **Push format**: `(push! mutation)` — server-initiated, any time
- **Stream lifecycle**: open on first request, multiplex subsequent requests
- **Caching**: content-hash based (`:have "sha3-..."`, `:have-components (...)`)
- **Authentication**: token in request keywords (`:auth "bearer ..."`)
- **Canonical serialization**: deterministic output rules for signing
- **Schema introspection**: `GET /__schema/` returns API description as sexp
Publish as a specification document.
### Phase 5: Rust Protocol Server
Build a QUIC server that speaks the sexp protocol alongside Hypercorn:
- Listen on a separate port (e.g., 4433 for QUIC)
- Parse sexp requests, route to the same handler logic as Quart
- Bidirectional stream: pushes, requests, responses on one connection
- Component manifest endpoint: serve component hashes + definitions
- Connection pooling, TLS via rustls
**Tech:** Rust, `quinn` (QUIC), `tokio`, sexp parser crate.
**Architecture option:** The Rust server could proxy to Quart for data/logic and handle only the protocol layer. Or it could run the full application logic natively (long-term, after porting route handlers to Rust).
**Files:**
- `sexpr-server/` — new Rust crate alongside the Python services
- `sexpr-server/src/protocol.rs` — framing, parsing, routing
- `sexpr-server/src/quic.rs` — QUIC listener + stream management
### Phase 6: Rust Native Client
Build the standalone sexp document viewer:
- QUIC client connecting to sexpr:// servers
- Sexp parser (zero-copy, arena-allocated AST)
- Component cache on disk (SQLite or filesystem, SHA3-keyed)
- Layout engine (flexbox subset — enough for document layout)
- GPU renderer via `wgpu` or `iced`
- Text rendering via `cosmic-text` or `fontdue`
- Input handling: keyboard, mouse, touch
- Bidirectional stream: real-time mutations, navigation without reconnection
**Tech:** Rust, `quinn`, `wgpu`/`iced`, `cosmic-text`, `tokio`.
**Files:**
- `sexpr-client/` — new Rust crate
- `sexpr-client/src/parser.rs` — zero-copy sexp parser
- `sexpr-client/src/cache.rs` — content-addressed component cache
- `sexpr-client/src/layout.rs` — layout engine
- `sexpr-client/src/render.rs` — GPU renderer
### Phase 7: Fallback Gateway
HTTP proxy that translates for browsers without extension/client:
- Accepts regular HTTPS requests
- Fetches sexp from the server (internal)
- Renders to HTML
- Serves to browser
- Adds `<link rel="alternate" type="application/x-sexpr" href="sexpr://...">` for discovery
**Result:** `https://rose-ash.com` always works. `sexpr://rose-ash.com` is an acceleration layer.
---
## Migration Path
Each tier builds on the last. No tier breaks the others:
1. **Today**: Tier 0 works. Ship HTML. Done.
2. **Add `Accept` header check** (Phase 1): Same routes now serve sexp to clients that ask. Tier 0 unchanged.
3. **Build sexpr.js** (Phase 2): Browser extension or `<script>` tag. Tier 1 works.
4. **Build extension** (Phase 3): `sexpr://` URLs work in browser. Tier 1 complete.
5. **Build Rust server** (Phase 5): Native protocol alongside HTTPS. Tier 2 infrastructure ready.
6. **Build Rust client** (Phase 6): `sexpr://` URLs work natively. Tier 2 complete.
A user with the Rust client visits the same URL as someone with Firefox. The server serves both from the same handler. If the Rust client is offline, the user opens `https://rose-ash.com` in a browser and gets the HTML version. Same content, same components, same data.
---
## Cooperative Angle
- Members who install the Rust client get the fast native experience
- Visitors browsing the public site get standard HTML — no barrier to entry
- Federated peers negotiate the best format they support
- AI agents get structured sexp via any tier
- The 5MB Rust binary replaces a 500MB browser for accessing the cooperative platform
- Auto-updates via content-addressed components — no app store gatekeeping
---
## Relationship to Other Plans
- **sexpr-js-runtime-plan.md** — Phase 2 of this plan; the JS library that powers Tier 1
- **ghost-removal-plan.md** — posts must be sexp before content negotiation adds value
- **sexpr-ai-integration.md** — AI agents benefit from all three tiers
- **sexpr-activitypub-extension.md** — federation over the native protocol (Tier 2 peers speak AP natively)

731
docs/sexpr-unified-protocol.md Normal file
View File

@@ -0,0 +1,731 @@
# The Sexp Protocol
**A unified protocol for documents, applications, and federation.**
**One format. Every boundary.**
---
## Core Insight
The sexp protocol replaces HTTP, HTML, JSON-LD, WebSocket, and ActivityPub's inbox model with a single concept: **peers exchange s-expressions on a bidirectional stream.**
There is no distinction between:
- A client requesting a page (what HTTP does)
- A server pushing a real-time update (what WebSocket does)
- A peer delivering a federated activity (what AP inbox POST does)
- A server sending a DOM mutation (what HTMX does)
They are all sexp expressions sent between two peers:
```scheme
;; Browsing (request → response)
(GET "/markets/")
(ok (page :title "Markets" ...))
;; Real-time push (server → client, unsolicited)
(push! (swap! "#feed" :prepend (use "post-card" :title "New post")))
;; Federation delivery (peer → peer)
(Create :actor "alice@rose-ash.com"
(Note :id post-123 (p "Hello from the fediverse!")))
;; Client action (client → server)
(POST "/like/" :body (:post-id 123))
;; Mutation response (server → client)
(push! (swap! "#like-count-123" :inner "43"))
```
Same parser. Same stream. Same connection. The "type" of interaction is determined by the head symbol of the expression, not by the protocol layer.
---
## Why Replace HTTP and JSON-LD
### HTTP's Problems
HTTP is a request-response protocol with text headers and a body. Strip away the historical baggage and what you actually need is:
```
Peer A sends: verb + target + metadata + optional body
Peer B sends: status + metadata + body
```
That's just an s-expression. But HTTP adds:
- **Header/body split** — two parsing phases, different formats
- **Rigid request-response** — to work around this, we bolted on WebSocket (separate protocol, upgrade handshake), SSE (chunked encoding hack), HTTP/2 push (failed, removed), HTTP/3 QUIC streams (complex, still one-request-one-response per stream)
- **Overcomplicated caching** — `Cache-Control`, `ETag`, `If-None-Match`, `If-Modified-Since`, `Vary`, `Age`, `Expires`, `Last-Modified`, `s-maxage`, `stale-while-revalidate` — a dozen headers with complex interaction rules
- **Arbitrary status codes** — memorised numbers with no semantic meaning in the format
- **Form encoding mess** — `application/x-www-form-urlencoded` or `multipart/form-data` with boundary strings, MIME parts, content-disposition headers
### JSON-LD's Problems
JSON-LD was chosen for ActivityPub because of its semantic web lineage, but in practice nobody uses the linked data features — servers just pattern-match on `type` fields and ignore the `@context`. The format is:
- **Verbose** — deeply nested objects with string keys, quoted values, commas, colons
- **Ambiguous** — compaction/expansion rules mean the same activity can have many valid JSON representations
- **Lossy** — post content is flattened to an HTML string inside a JSON string; structure destroyed
- **Hostile to signing** — JSON-LD canonicalization is fragile and implementation-dependent
- **Hostile to AI** — agents must parse JSON, then parse embedded HTML, then reason about structure
### The Sexp Solution
One format that is:
- **Compact** — half the size of equivalent JSON-LD, no closing tags like HTML
- **Unambiguous** — one canonical serialization, deterministic for signing
- **Structural** — content *is* the tree, not a string embedded in a string
- **Parseable** — trivial recursive descent, no backtracking, nanoseconds per node
- **Bidirectional** — requests, responses, pushes, and activities all use the same syntax
- **AI-native** — agents parse, generate, and reason about sexp as naturally as tool calls
---
## Protocol Specification
### Transport
```
┌─────────────────────────────────────────┐
│ Application: sexp expressions │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ Session: bidirectional sexp stream │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ Transport: QUIC (or TCP+TLS fallback) │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ Network: IP │
└─────────────────────────────────────────┘
```
QUIC is ideal — multiplexed streams, built-in TLS, connection migration. Each sexp expression gets its own QUIC stream. Requests and pushes multiplex without head-of-line blocking. Connection survives network changes (mobile → wifi).
**Framing:** Length-prefixed sexp expressions over QUIC streams. The parser knows when an expression ends (balanced parens), but length-prefixing enables efficient buffer allocation.
**URL scheme:** `sexpr://host:port/path`
### Expression Types
Every expression on the wire is one of these forms:
#### Requests (client → server)
```scheme
(GET "/markets/"
:auth "bearer tok_abc123"
:have "sha3-a1b2c3"
:have-components ("sha3-aaa" "sha3-bbb"))
(POST "/like/"
:auth "bearer tok_abc123"
:body (:post-id 123))
(POST "/submit/"
:body (
:username "alice"
:email "alice@example.com"
:avatar (file :name "photo.jpg" :type "image/jpeg" :data <binary>)))
;; Streaming request — keep connection open for pushes
(GET "/feed/" :stream #t)
```
#### Responses (server → client)
```scheme
;; Success with content
(ok :hash "sha3-d4e5f6"
(page :title "Markets" :layout "main"
(section (h1 "Markets")
(each markets (lambda (m)
(use "vendor-card" :name (get m "name")))))))
;; Not modified (client already has current version)
(not-modified)
;; Redirect
(redirect :to "/new-location/" :permanent #t)
;; Not found
(not-found :message "Page does not exist")
;; Error
(error :code "auth-required" :message "Please log in"
:login-url "sexpr://rose-ash.com/login/")
;; Response with new components the client is missing
(ok
:new-components (
(component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
(page ...))
```
#### Pushes (server → client, unsolicited)
```scheme
;; DOM mutation
(push! (swap! "#feed" :prepend
(use "post-card" :title "New post" :author "alice")))
;; Batch mutation
(push! (batch!
(swap! "#notifications" :append (div :class "toast" "Saved!"))
(class! "#save-btn" :remove "loading")))
;; Component update (new version available)
(push! (component-update "vendor-card" :hash "sha3-eee"))
```
#### Activities (peer → peer, federation)
```scheme
;; Create
(Create
:id "https://rose-ash.com/ap/activities/456"
:actor "https://rose-ash.com/ap/users/alice"
:published "2026-02-28T14:00:00Z"
:to (:public)
:cc ("https://rose-ash.com/ap/users/alice/followers")
(Note
:id "https://rose-ash.com/ap/posts/123"
:attributed-to "https://rose-ash.com/ap/users/alice"
:in-reply-to "https://remote.social/posts/789"
:summary "Weekend market update"
(section
(p "Three new vendors joining this Saturday.")
(use "vendor-list" :vendors vendors)
(use "calendar-widget" :calendar-id 42))
(attachment
(Image :url "https://rose-ash.com/media/market.jpg"
:media-type "image/jpeg"
:width 1200 :height 800
:name "Saturday market stalls"))
(tag
(Hashtag :name "#markets" :href "https://rose-ash.com/tags/markets")
(Mention :name "@bob@remote.social" :href "https://remote.social/users/bob"))))
;; Follow
(Follow :actor "https://rose-ash.com/ap/users/alice"
:object "https://remote.social/users/bob")
;; Accept (response to Follow — on the same connection)
(Accept :actor "https://remote.social/users/bob"
(Follow :actor "https://rose-ash.com/ap/users/alice"
:object "https://remote.social/users/bob"))
;; Like, Announce, Undo — all just expressions
(Like :actor alice :object post-123)
(Announce :actor bob (Note :id post-123))
(Undo :actor alice (Like :actor alice :object post-123))
```
**Key insight:** Activities are not "posted to an inbox" — they are expressions sent on the bidirectional stream. The peer-to-peer connection *is* the inbox. When you follow someone, their activities stream to you on the same connection you used to follow them. No inbox endpoint, no HTTP POST delivery, no polling.
#### Collections (queryable, paginated)
```scheme
;; Request an actor's outbox
(GET "/ap/users/alice/outbox" :page 1)
;; Response
(ok
(OrderedCollectionPage
:part-of "https://rose-ash.com/ap/users/alice/outbox"
:next "/ap/users/alice/outbox?page=2"
:total-items 142
(Create :actor alice :published "2026-02-28T14:00:00Z"
(Note :id post-3 (p "Latest post")))
(Announce :actor alice :published "2026-02-27T09:00:00Z"
(Note :id post-2 :attributed-to bob))
(Create :actor alice :published "2026-02-26T18:00:00Z"
(Note :id post-1 (p "Earlier post")))))
```
#### Actor Profiles
```scheme
(Person
:id "https://rose-ash.com/ap/users/alice"
:preferred-username "alice"
:name "Alice"
:summary (p "Co-op member, market organiser")
:inbox "sexpr://rose-ash.com/ap/users/alice/inbox"
:outbox "sexpr://rose-ash.com/ap/users/alice/outbox"
:followers "sexpr://rose-ash.com/ap/users/alice/followers"
:following "sexpr://rose-ash.com/ap/users/alice/following"
:components "sexpr://rose-ash.com/ap/components/"
:public-key (:id "https://rose-ash.com/ap/users/alice#main-key"
:owner "https://rose-ash.com/ap/users/alice"
:pem "-----BEGIN PUBLIC KEY-----\n..."))
```
#### Schema Introspection
```scheme
(GET "/__schema/")
(ok
(schema
(endpoint "/" :method GET
:returns (page)
:params (:stream bool))
(endpoint "/markets/" :method GET
:returns (page :contains (list (use "vendor-card"))))
(endpoint "/like/" :method POST
:params (:post-id int)
:returns (mutation))
(activity Create
:object (Note)
:delivers-to :followers)
(activity Follow
:object (Person)
:expects (Accept))))
```
An AI agent hitting `/__schema/` learns the entire surface — pages, actions, federation activities — as parseable sexp. No separate OpenAPI doc. The schema *is* the API.
### Caching
Content-addressed, hash-based. No expiry headers, no revalidation dance.
```scheme
;; Client has a cached version
(GET "/markets/" :have "sha3-a1b2c3")
;; Unchanged
(not-modified)
;; Changed — new hash included
(ok :hash "sha3-d4e5f6" (page ...))
```
Component-level caching:
```scheme
;; Client reports which components it has
(GET "/markets/" :have-components ("sha3-aaa" "sha3-bbb"))
;; Server sends only missing components alongside the page
(ok
:new-components (
(component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
(page ...))
```
The hash *is* the cache key, the ETag, and the content address. One concept replaces twelve HTTP headers.
### Component Discovery
Actors advertise a `:components` endpoint in their profile. Peers fetch and cache component definitions by content hash:
```scheme
;; Fetch component manifest
(GET "/ap/components/")
(ok
(component-manifest
("cart-mini" :hash "sha3-aaa")
("vendor-card" :hash "sha3-bbb")
("calendar-widget" :hash "sha3-ccc")
("post-card" :hash "sha3-ddd")))
;; Fetch a specific component by hash
(GET "/ap/components/sha3-bbb")
(ok
(component "vendor-card" :hash "sha3-bbb"
(params name stall image)
(div :class "vendor-card"
(img :src image :alt name)
(h3 name)
(p :class "stall" stall))))
```
When a federated activity includes `(use "vendor-card" :name "Sourdough" :stall "A12")`, the receiving peer resolves the component from its cache (by hash) or fetches it from the sender's manifest. Federated UI, not just federated data.
### Signatures
Sexp serialization has a canonical form:
- Keywords sorted alphabetically
- Single space between atoms
- No trailing whitespace
- UTF-8 encoding
This makes signatures deterministic without JSON-LD canonicalization:
```scheme
(signed :sig "base64..." :key-id "https://rose-ash.com/ap/users/alice#main-key"
(Create :actor "https://rose-ash.com/ap/users/alice"
(Note :id post-123 (p "Hello"))))
```
Sign the canonical serialization of the inner expression. Verify by re-serializing and checking. No compaction/expansion ambiguity.
---
## Bidirectional Stream: The Unified Model
A connection between two sexp-speaking peers carries everything on one stream:
```scheme
;; === Connection opened ===
;; Client browses
(GET "/")
(ok (page :title "Home" ...))
;; Client navigates (same connection)
(GET "/markets/")
(ok (page :title "Markets" ...))
;; Client follows a user (federation)
(Follow :actor alice :object bob)
;; Server confirms
(Accept :actor bob (Follow :actor alice :object bob))
;; Bob posts something — server pushes activity
(Create :actor bob :published "2026-02-28T15:00:00Z"
(Note :id post-456 (p "New vendor announcement!")))
;; Client sees it rendered in real-time via mutation
(push! (swap! "#feed" :prepend
(use "post-card" :actor "bob" :content (p "New vendor announcement!"))))
;; Client likes the post
(POST "/like/" :body (:post-id 456))
;; Server pushes mutation + delivers Like activity to bob
(push! (swap! "#like-count-456" :inner "12"))
;; Client opens a streaming feed
(GET "/feed/" :stream #t)
(ok :stream #t (page :title "Feed" ...))
;; More activities arrive over time...
(Create :actor charlie :published "2026-02-28T15:30:00Z"
(Note :id post-789 (p "Market day tomorrow!")))
(push! (swap! "#feed" :prepend
(use "post-card" :actor "charlie" :content (p "Market day tomorrow!"))))
;; === Connection persists ===
```
Browsing, federation, real-time updates, and user actions — all on one bidirectional stream. The distinction between "web server", "AP inbox", and "WebSocket" disappears. They were always the same thing — peers exchanging structured expressions.
---
## Backwards Compatibility
### With HTTP (Tier 0 and Tier 1)
The sexp protocol runs alongside HTTPS. The same server handles both:
```python
@bp.get("/markets/")
async def markets():
data = await get_markets(g.s)
tree = sexp('(page :title "Markets" ...)', markets=data)
accept = request.headers.get("Accept", "")
if "application/x-sexpr" in accept:
return Response(serialize(tree), content_type="application/x-sexpr")
html = render_to_html(tree)
return Response(html, content_type="text/html")
```
### With ActivityPub (JSON-LD peers)
For Mastodon, Pleroma, and other JSON-LD AP servers:
| Peer Type | Outbound | Inbound |
|---|---|---|
| Standard AP (Mastodon, Pleroma) | Translate sexp → JSON-LD, include `rose:sexpr` extension field | Parse JSON-LD, translate to sexp internally |
| Sexp-aware AP (other rose-ash instances) | Native sexp on bidirectional stream | Parse sexp directly |
| Sexp-aware with shared components | Sexp with component references | Resolve from cache, render natively |
JSON-LD bridging for non-sexp peers:
```json
{
"@context": ["https://www.w3.org/ns/activitystreams", "https://rose-ash.com/ns/sexpr"],
"type": "Create",
"object": {
"type": "Note",
"content": "<p>Hello world</p>",
"rose:sexpr": "(p \"Hello world\")"
}
}
```
No existing AP implementation breaks. The `rose:sexpr` field is ignored by servers that don't understand it.
### Fallback Gateway
For browsers without extension or native client:
```
sexpr://rose-ash.com/markets/
→ Gateway fetches sexp from server
→ Renders to HTML
→ Serves to browser at https://rose-ash.com/markets/
```
---
## Three Client Tiers
```
┌─────────────────────────────┐
│ rose-ash server (Quart) │
│ │
│ Same sexp component tree │
│ Same data, same logic │
│ │
├──────────┬──────────┬────────┤
│ HTTPS │ HTTPS │ SEXPR │
│ HTML out │ sexp out │ native │
└────┬─────┴────┬─────┴───┬────┘
│ │ │
┌──────────▼──┐ ┌─────▼──────┐ ┌▼──────────────┐
│ Browser │ │ Browser + │ │ Rust client │
│ (vanilla) │ │ extension │ │ (native) │
│ │ │ │ │ │
│ HTML + HTMX │ │ sexpr.js │ │ sexp protocol │
│ Full CSS │ │ over HTTPS │ │ over QUIC │
│ ~200ms load │ │ ~80ms load │ │ ~20ms load │
└─────────────┘ └────────────┘ └────────────────┘
Tier 0 Tier 1 Tier 2
```
| | Tier 0: Browser | Tier 1: Extension | Tier 2: Rust Client |
|---|---|---|---|
| URL | `https://rose-ash.com` | `https://rose-ash.com` | `sexpr://rose-ash.com` |
| Protocol | HTTPS | HTTPS | sexpr:// over QUIC |
| Wire format | HTML | sexp over HTTP | sexp native stream |
| Rendering | Browser DOM | sexpr.js → DOM | Rust → GPU |
| Component cache | Browser cache (URL-keyed) | IndexedDB (hash-keyed) | Disk (hash-keyed, pre-parsed AST) |
| Real-time | HTMX polling / SSE | WebSocket sexp mutations | Native bidirectional stream |
| Federation | N/A | AP via fetch | Native sexp stream |
| Bundle size | HTMX 14KB + CSS | sexpr.js ~8KB | 5MB binary, zero runtime deps |
| Page load | ~200ms | ~80ms | ~20ms |
| Memory per page | ~200MB | ~150MB | ~20MB |
| AI integration | Parse HTML (painful) | Parse sexp (easy) | Native sexp (trivial) |
### Rust Client Architecture
```
┌─────────────────────────────────────┐
│ sexpr-client (Rust) │
│ │
│ ┌──────────┐ ┌─────────────────┐ │
│ │ Parser │ │ Component │ │
│ │ (zero- │ │ Cache │ │
│ │ copy) │ │ (SHA3 → AST) │ │
│ └──────────┘ └─────────────────┘ │
│ ┌──────────┐ ┌─────────────────┐ │
│ │ Layout │ │ Network │ │
│ │ Engine │ │ (tokio + QUIC) │ │
│ └──────────┘ └─────────────────┘ │
│ ┌──────────────────────────────┐ │
│ │ Renderer (wgpu / iced) │ │
│ └──────────────────────────────┘ │
└─────────────────────────────────────┘
```
**Why it's fast:**
- Parser: nanoseconds per node (trivial recursive descent in Rust)
- No HTML parser, no CSS cascade, no DOM construction, no JS engine
- Render directly to GPU surface — skip the entire browser rendering pipeline
- Pre-parsed ASTs from disk cache — zero parse time on cache hit
- 10-50MB memory vs 200-500MB per browser tab
Page load: network (~50ms) → parse (~0.1ms) → layout (~2ms) → paint (~3ms) = **under 60ms**.
---
## Implementation Plan
### Phase 1: Content Negotiation (Quart)
Add `Accept` header handling to existing Quart routes.
- Check for `application/x-sexpr` in `Accept` header
- If present: serialize sexp tree, return directly
- If absent: render to HTML as today
- Add `Vary: Accept` header
**Files:**
- `shared/infrastructure/factory.py` — content negotiation middleware
- `shared/sexp/serialize.py` — canonical sexp serializer (deterministic for signing/caching)
**Result:** Existing routes serve sexp to any client that asks. Zero changes to route logic.
### Phase 2: Sexp ↔ JSON-LD Bridge
Bidirectional translation for AP federation with non-sexp peers:
**sexp → JSON-LD** (outbound):
- Activity type symbol → `"type"` field
- Keyword attributes → JSON object properties
- Content sexp → rendered to HTML for `"content"` field
- Include `rose:sexpr` extension field
**JSON-LD → sexp** (inbound):
- `"type"` → head symbol
- `"content"` HTML → parsed to sexp (best-effort)
- Nested objects → child expressions
**Files:**
- `shared/infrastructure/ap_sexpr.py` — serializer/deserializer
- `shared/sexp/html_to_sexp.py` — HTML → sexp parser for inbound content
- `shared/sexp/tests/test_ap_sexpr.py` — round-trip tests
### Phase 3: sexpr.js Client Library
Browser-side JS runtime (see `sexpr-js-runtime-plan.md`):
- Parser + renderer: `parse()` → AST → `renderToDOM()`
- Mutation engine: `swap!`, `batch!`, `class!`, `request!`
- Component registry with localStorage cache (content-addressed)
- WebSocket connection for real-time pushes
### Phase 4: Browser Extension
Package sexpr.js as a WebExtension:
- Register `sexpr://` protocol handler
- Intercept navigation → fetch via HTTPS with `Accept: application/x-sexpr`
- Parse → render to DOM
- Component cache in IndexedDB (content-addressed)
**Tech:** WebExtension API (Firefox + Chrome), JS/TS.
### Phase 5: Component Discovery
Enable peers to discover and cache shared components:
- Actor profiles include `:components` endpoint URL
- `GET /ap/components/` → manifest (name → hash)
- `GET /ap/components/{hash}` → component definition (sexp)
- Content-addressed cache on client (hash-keyed)
- Federated content with `(use "component" ...)` resolves from cache
**Files:**
- `federation/bp/components/routes.py` — manifest + fetch endpoints
- `shared/sexp/component_manifest.py` — hash generation
### Phase 6: Protocol Specification
Formal specification document:
- **Framing**: length-prefixed sexp over QUIC streams
- **Expression types**: requests, responses, pushes, activities, collections
- **Canonical serialization**: deterministic rules for signing
- **Caching**: content-hash based
- **Authentication**: token in request keywords
- **Component discovery**: manifest protocol
- **Bidirectional streaming**: lifecycle, multiplexing
- **Schema introspection**: `/__schema/` endpoint
- **JSON-LD bridging**: rules for non-sexp AP peers
- **Security**: no eval, escaping, signature verification
Publish as a FEP (Fediverse Enhancement Proposal) and standalone specification.
### Phase 7: Rust Protocol Server
QUIC server alongside Hypercorn:
- Listen on separate port (e.g., 4433)
- Parse sexp requests, route to same handler logic
- Bidirectional stream: pushes, requests, responses, activities
- Component manifest endpoint
- Federation delivery on persistent connections (replaces HTTP POST to inbox)
**Tech:** Rust, `quinn` (QUIC), `tokio`, `rustls`.
**Files:**
- `sexpr-server/src/protocol.rs` — framing, parsing, routing
- `sexpr-server/src/quic.rs` — QUIC listener + stream management
- `sexpr-server/src/federation.rs` — peer connection manager
### Phase 8: Rust Native Client
Standalone sexp document viewer:
- QUIC client for `sexpr://` URLs
- Zero-copy sexp parser (arena-allocated AST)
- Component cache on disk (SHA3-keyed)
- Layout engine (flexbox subset)
- GPU renderer via `wgpu` or `iced`
- Text rendering via `cosmic-text`
- Bidirectional stream: browsing + federation + real-time on one connection
**Tech:** Rust, `quinn`, `wgpu`/`iced`, `cosmic-text`, `tokio`.
**Files:**
- `sexpr-client/src/parser.rs` — zero-copy parser
- `sexpr-client/src/cache.rs` — content-addressed component cache
- `sexpr-client/src/layout.rs` — layout engine
- `sexpr-client/src/render.rs` — GPU renderer
- `sexpr-client/src/stream.rs` — bidirectional connection manager
### Phase 9: Fallback Gateway
HTTP proxy for browsers without extension/client:
- Accept HTTPS requests at `https://rose-ash.com`
- Fetch sexp from server internally
- Render to HTML, serve to browser
- Add `<link rel="alternate" type="application/x-sexpr" href="sexpr://...">` for discovery
---
## Migration Path
Each phase builds on the last. No phase breaks existing functionality:
```
Phase 1 (content negotiation) ── Tier 0 unchanged, sexp available
Phase 2 (JSON-LD bridge) ── federation works with all AP peers
Phase 3 (sexpr.js) ── Tier 1 client-side rendering
Phase 4 (extension) ── sexpr:// URLs in browser
Phase 5 (components) ── federated UI exchange
Phase 6 (spec) ── formal protocol document
Phase 7 (Rust server) ── native protocol alongside HTTPS
Phase 8 (Rust client) ── Tier 2 native experience
Phase 9 (gateway) ── sexpr:// accessible from any browser
```
Depends on:
- **Ghost removal** (see `ghost-removal-plan.md`) — posts must be sexp before Phases 2-3 add real value
- **sexpr.js runtime** (see `sexpr-js-runtime-plan.md`) — the JS library that powers Phases 3-4
---
## Cooperative Angle
- Members install the Rust client → fast native experience, 5MB binary, no app store
- Visitors browse `https://rose-ash.com` → standard HTML, no barrier
- Federated co-ops connect via persistent sexp streams → rich UI exchange, not just text syndication
- AI agents speak the protocol natively → components as tool calls, mutations as actions
- Auto-updates via content-addressed components → no gatekeeping
- The component registry is a shared vocabulary across the cooperative network
---
## Relationship to Other Plans
| Document | Role |
|---|---|
| `sexpr-js-runtime-plan.md` | The JS library powering Tier 1 (Phases 3-4) |
| `ghost-removal-plan.md` | Posts must be sexp before federation/client rendering adds value |
| `sexpr-ai-integration.md` | AI agents benefit from all tiers and the self-describing schema |
---
*The document is the program. The program is the document. The protocol is both.*