rose-ash/docs/sexpr-unified-protocol.md

# The Sexp Protocol

**A unified protocol for documents, applications, and federation.**
**One format. Every boundary.**

---

## Core Insight

The sexp protocol replaces HTTP, HTML, JSON-LD, WebSocket, and ActivityPub's inbox model with a single concept: **peers exchange s-expressions on a bidirectional stream.**

There is no distinction between:
- A client requesting a page (what HTTP does)
- A server pushing a real-time update (what WebSocket does)
- A peer delivering a federated activity (what AP inbox POST does)
- A server sending a DOM mutation (what HTMX does)

They are all sexp expressions sent between two peers:

```scheme
;; Browsing (request → response)
(GET "/markets/")
(ok (page :title "Markets" ...))

;; Real-time push (server → client, unsolicited)
(push! (swap! "#feed" :prepend (use "post-card" :title "New post")))

;; Federation delivery (peer → peer)
(Create :actor "alice@rose-ash.com"
  (Note :id post-123 (p "Hello from the fediverse!")))

;; Client action (client → server)
(POST "/like/" :body (:post-id 123))

;; Mutation response (server → client)
(push! (swap! "#like-count-123" :inner "43"))
```

Same parser. Same stream. Same connection. The "type" of interaction is determined by the head symbol of the expression, not by the protocol layer.

---

## Why Replace HTTP and JSON-LD

### HTTP's Problems

HTTP is a request-response protocol with text headers and a body. Strip away the historical baggage and what you actually need is:

```
Peer A sends: verb + target + metadata + optional body
Peer B sends: status + metadata + body
```

That's just an s-expression. But HTTP adds:
- **Header/body split** — two parsing phases, different formats
- **Rigid request-response** — to work around this, we bolted on WebSocket (separate protocol, upgrade handshake), SSE (chunked encoding hack), HTTP/2 push (failed, removed), HTTP/3 QUIC streams (complex, still one-request-one-response per stream)
- **Overcomplicated caching** — `Cache-Control`, `ETag`, `If-None-Match`, `If-Modified-Since`, `Vary`, `Age`, `Expires`, `Last-Modified`, `s-maxage`, `stale-while-revalidate` — a dozen headers with complex interaction rules
- **Arbitrary status codes** — memorised numbers with no semantic meaning in the format
- **Form encoding mess** — `application/x-www-form-urlencoded` or `multipart/form-data` with boundary strings, MIME parts, content-disposition headers

### JSON-LD's Problems

JSON-LD was chosen for ActivityPub because of its semantic web lineage, but in practice nobody uses the linked data features — servers just pattern-match on `type` fields and ignore the `@context`. The format is:

- **Verbose** — deeply nested objects with string keys, quoted values, commas, colons
- **Ambiguous** — compaction/expansion rules mean the same activity can have many valid JSON representations
- **Lossy** — post content is flattened to an HTML string inside a JSON string; structure destroyed
- **Hostile to signing** — JSON-LD canonicalization is fragile and implementation-dependent
- **Hostile to AI** — agents must parse JSON, then parse embedded HTML, then reason about structure

### The Sexp Solution

One format that is:
- **Compact** — half the size of equivalent JSON-LD, no closing tags like HTML
- **Unambiguous** — one canonical serialization, deterministic for signing
- **Structural** — content *is* the tree, not a string embedded in a string
- **Parseable** — trivial recursive descent, no backtracking, nanoseconds per node
- **Bidirectional** — requests, responses, pushes, and activities all use the same syntax
- **AI-native** — agents parse, generate, and reason about sexp as naturally as tool calls

---

## Protocol Specification

### Transport

```
┌─────────────────────────────────────────┐
│  Application: sexp expressions          │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│  Session: bidirectional sexp stream     │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│  Transport: QUIC (or TCP+TLS fallback)  │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│  Network: IP                            │
└─────────────────────────────────────────┘
```

QUIC is ideal — multiplexed streams, built-in TLS, connection migration. Each sexp expression gets its own QUIC stream. Requests and pushes multiplex without head-of-line blocking. Connection survives network changes (mobile → wifi).

**Framing:** Length-prefixed sexp expressions over QUIC streams. The parser knows when an expression ends (balanced parens), but length-prefixing enables efficient buffer allocation.

**URL scheme:** `sexpr://host:port/path`

### Expression Types

Every expression on the wire is one of these forms:

#### Requests (client → server)

The head symbol is the verb. **Any symbol is a valid verb.** There is no fixed set like HTTP's seven methods. The verb describes the intent; the path is the noun:

```scheme
;; Reads
(GET "/markets/")
(GET "/markets/" :have "sha3-a1b2c3" :have-components ("sha3-aaa" "sha3-bbb"))
(GET "/feed/" :stream #t)

;; Writes (HTTP-style, but you're not limited to these)
(POST "/users/" :body (:name "alice" :email "alice@example.com"))
(DELETE "/posts/draft-123/")

;; Domain verbs — describe what's actually happening
(publish "/posts/draft-123/")
(reserve "/events/saturday-market/" :tickets 2)
(subscribe "/newsletters/weekly/")
(unsubscribe "/newsletters/weekly/")
(pin "/posts/important-announcement/")
(refund "/orders/ord-789/" :reason "damaged")
(schedule "/calendar/saturday/" :slot 3 :name "Pottery Workshop")
(rsvp "/events/annual-meeting/" :attending #t :guests 2)
(transfer "/inventory/sourdough/" :from "stall-a" :to "stall-b" :quantity 5)
(bid "/auctions/vintage-table/" :amount 45.00)

;; Cooperative governance
(propose "/governance/" :title "New market hours"
  :body (p "I suggest we open at 8am..."))
(second "/governance/proposals/42/")
(vote "/governance/proposals/42/" :choice :approve)
(ratify "/governance/proposals/42/")

;; Compute mesh
(render :recipe "bafyrecipe..." :input "bafyinput..." :requirements (:min-vram 8))
(transcode :input "bafyvideo..." :format "h265" :quality 28)

;; File uploads — structured, not multipart MIME
(POST "/upload/"
  :body (
    :username "alice"
    :avatar (file :name "photo.jpg" :type "image/jpeg" :data <binary>)))
```

`(reserve "/events/saturday-market/" :tickets 2)` reads as English. An AI agent doesn't need API documentation to understand what that does. The URL is the noun, the verb is the verb — no more `POST /api/users/123/send-password-reset-email` where the action is buried in the URL because HTTP doesn't have the right verb.

**Verb behaviour is declared in the schema**, not assumed by convention:

```scheme
(GET "/__schema/")

(ok
  (schema
    (verb GET :idempotent #t :auth :optional
      :description "Retrieve a resource")
    (verb reserve :idempotent #f :auth :required
      :params (:tickets int)
      :returns (reservation)
      :description "Reserve tickets for an event")
    (verb cancel-reservation :idempotent #t :auth :required
      :params (:reservation-id str)
      :returns (ok)
      :description "Cancel a ticket reservation")
    (verb publish :idempotent #t :auth :admin
      :description "Publish a draft post")
    (verb vote :idempotent #t :auth :member
      :params (:choice (enum :approve :reject :abstain))
      :returns (ok)
      :description "Vote on a governance proposal")))
```

No more arguing about whether `PATCH` should be idempotent. The schema says what each verb does, what it takes, what it returns, and who can call it. AI agents read the schema and know the full API surface — including domain-specific verbs they've never seen before.

#### Responses (server → client)

```scheme
;; Success with content
(ok :hash "sha3-d4e5f6"
  (page :title "Markets" :layout "main"
    (section (h1 "Markets")
      (each markets (lambda (m)
        (use "vendor-card" :name (get m "name")))))))

;; Not modified (client already has current version)
(not-modified)

;; Redirect
(redirect :to "/new-location/" :permanent #t)

;; Not found
(not-found :message "Page does not exist")

;; Error
(error :code "auth-required" :message "Please log in"
  :login-url "sexpr://rose-ash.com/login/")

;; Response with new components the client is missing
(ok
  :new-components (
    (component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
  (page ...))
```

#### Pushes (server → client, unsolicited)

```scheme
;; DOM mutation
(push! (swap! "#feed" :prepend
  (use "post-card" :title "New post" :author "alice")))

;; Batch mutation
(push! (batch!
  (swap! "#notifications" :append (div :class "toast" "Saved!"))
  (class! "#save-btn" :remove "loading")))

;; Component update (new version available)
(push! (component-update "vendor-card" :hash "sha3-eee"))
```

#### Activities (peer → peer, federation)

```scheme
;; Create
(Create
  :id "https://rose-ash.com/ap/activities/456"
  :actor "https://rose-ash.com/ap/users/alice"
  :published "2026-02-28T14:00:00Z"
  :to (:public)
  :cc ("https://rose-ash.com/ap/users/alice/followers")

  (Note
    :id "https://rose-ash.com/ap/posts/123"
    :attributed-to "https://rose-ash.com/ap/users/alice"
    :in-reply-to "https://remote.social/posts/789"
    :summary "Weekend market update"

    (section
      (p "Three new vendors joining this Saturday.")
      (use "vendor-list" :vendors vendors)
      (use "calendar-widget" :calendar-id 42))

    (attachment
      (Image :url "https://rose-ash.com/media/market.jpg"
        :media-type "image/jpeg"
        :width 1200 :height 800
        :name "Saturday market stalls"))

    (tag
      (Hashtag :name "#markets" :href "https://rose-ash.com/tags/markets")
      (Mention :name "@bob@remote.social" :href "https://remote.social/users/bob"))))

;; Follow
(Follow :actor "https://rose-ash.com/ap/users/alice"
  :object "https://remote.social/users/bob")

;; Accept (response to Follow — on the same connection)
(Accept :actor "https://remote.social/users/bob"
  (Follow :actor "https://rose-ash.com/ap/users/alice"
    :object "https://remote.social/users/bob"))

;; Like, Announce, Undo — all just expressions
(Like :actor alice :object post-123)
(Announce :actor bob (Note :id post-123))
(Undo :actor alice (Like :actor alice :object post-123))
```

**Key insight:** Activities are not "posted to an inbox" — they are expressions sent on the bidirectional stream. The peer-to-peer connection *is* the inbox. When you follow someone, their activities stream to you on the same connection you used to follow them. No inbox endpoint, no HTTP POST delivery, no polling.

#### Collections (queryable, paginated)

```scheme
;; Request an actor's outbox
(GET "/ap/users/alice/outbox" :page 1)

;; Response
(ok
  (OrderedCollectionPage
    :part-of "https://rose-ash.com/ap/users/alice/outbox"
    :next "/ap/users/alice/outbox?page=2"
    :total-items 142

    (Create :actor alice :published "2026-02-28T14:00:00Z"
      (Note :id post-3 (p "Latest post")))
    (Announce :actor alice :published "2026-02-27T09:00:00Z"
      (Note :id post-2 :attributed-to bob))
    (Create :actor alice :published "2026-02-26T18:00:00Z"
      (Note :id post-1 (p "Earlier post")))))
```

#### Actor Profiles

```scheme
(Person
  :id "https://rose-ash.com/ap/users/alice"
  :preferred-username "alice"
  :name "Alice"
  :summary (p "Co-op member, market organiser")
  :inbox "sexpr://rose-ash.com/ap/users/alice/inbox"
  :outbox "sexpr://rose-ash.com/ap/users/alice/outbox"
  :followers "sexpr://rose-ash.com/ap/users/alice/followers"
  :following "sexpr://rose-ash.com/ap/users/alice/following"
  :components "sexpr://rose-ash.com/ap/components/"
  :public-key (:id "https://rose-ash.com/ap/users/alice#main-key"
               :owner "https://rose-ash.com/ap/users/alice"
               :pem "-----BEGIN PUBLIC KEY-----\n..."))
```

#### Schema Introspection

```scheme
(GET "/__schema/")

(ok
  (schema
    (endpoint "/" :method GET
      :returns (page)
      :params (:stream bool))
    (endpoint "/markets/" :method GET
      :returns (page :contains (list (use "vendor-card"))))
    (endpoint "/like/" :method POST
      :params (:post-id int)
      :returns (mutation))
    (activity Create
      :object (Note)
      :delivers-to :followers)
    (activity Follow
      :object (Person)
      :expects (Accept))))
```

An AI agent hitting `/__schema/` learns the entire surface — pages, actions, federation activities — as parseable sexp. No separate OpenAPI doc. The schema *is* the API.

### Caching

Content-addressed, hash-based. No expiry headers, no revalidation dance.

```scheme
;; Client has a cached version
(GET "/markets/" :have "sha3-a1b2c3")

;; Unchanged
(not-modified)

;; Changed — new hash included
(ok :hash "sha3-d4e5f6" (page ...))
```

Component-level caching:

```scheme
;; Client reports which components it has
(GET "/markets/" :have-components ("sha3-aaa" "sha3-bbb"))

;; Server sends only missing components alongside the page
(ok
  :new-components (
    (component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
  (page ...))
```

The hash *is* the cache key, the ETag, and the content address. One concept replaces twelve HTTP headers.

### Component Discovery

Actors advertise a `:components` endpoint in their profile. Peers fetch and cache component definitions by content hash:

```scheme
;; Fetch component manifest
(GET "/ap/components/")

(ok
  (component-manifest
    ("cart-mini" :hash "sha3-aaa")
    ("vendor-card" :hash "sha3-bbb")
    ("calendar-widget" :hash "sha3-ccc")
    ("post-card" :hash "sha3-ddd")))

;; Fetch a specific component by hash
(GET "/ap/components/sha3-bbb")

(ok
  (component "vendor-card" :hash "sha3-bbb"
    (params name stall image)
    (div :class "vendor-card"
      (img :src image :alt name)
      (h3 name)
      (p :class "stall" stall))))
```

When a federated activity includes `(use "vendor-card" :name "Sourdough" :stall "A12")`, the receiving peer resolves the component from its cache (by hash) or fetches it from the sender's manifest. Federated UI, not just federated data.

### Signatures

Sexp serialization has a canonical form:
- Keywords sorted alphabetically
- Single space between atoms
- No trailing whitespace
- UTF-8 encoding

This makes signatures deterministic without JSON-LD canonicalization:

```scheme
(signed :sig "base64..." :key-id "https://rose-ash.com/ap/users/alice#main-key"
  (Create :actor "https://rose-ash.com/ap/users/alice"
    (Note :id post-123 (p "Hello"))))
```

Sign the canonical serialization of the inner expression. Verify by re-serializing and checking. No compaction/expansion ambiguity.

---

## Bidirectional Stream: The Unified Model

A connection between two sexp-speaking peers carries everything on one stream:

```scheme
;; === Connection opened ===

;; Client browses
(GET "/")
(ok (page :title "Home" ...))

;; Client navigates (same connection)
(GET "/markets/")
(ok (page :title "Markets" ...))

;; Client follows a user (federation)
(Follow :actor alice :object bob)

;; Server confirms
(Accept :actor bob (Follow :actor alice :object bob))

;; Bob posts something — server pushes activity
(Create :actor bob :published "2026-02-28T15:00:00Z"
  (Note :id post-456 (p "New vendor announcement!")))

;; Client sees it rendered in real-time via mutation
(push! (swap! "#feed" :prepend
  (use "post-card" :actor "bob" :content (p "New vendor announcement!"))))

;; Client likes the post
(POST "/like/" :body (:post-id 456))

;; Server pushes mutation + delivers Like activity to bob
(push! (swap! "#like-count-456" :inner "12"))

;; Client opens a streaming feed
(GET "/feed/" :stream #t)
(ok :stream #t (page :title "Feed" ...))

;; More activities arrive over time...
(Create :actor charlie :published "2026-02-28T15:30:00Z"
  (Note :id post-789 (p "Market day tomorrow!")))
(push! (swap! "#feed" :prepend
  (use "post-card" :actor "charlie" :content (p "Market day tomorrow!"))))

;; === Connection persists ===
```

Browsing, federation, real-time updates, and user actions — all on one bidirectional stream. The distinction between "web server", "AP inbox", and "WebSocket" disappears. They were always the same thing — peers exchanging structured expressions.

---

## Backwards Compatibility

### With HTTP (Tier 0 and Tier 1)

The sexp protocol runs alongside HTTPS. The same server handles both:

```python
@bp.get("/markets/")
async def markets():
    data = await get_markets(g.s)
    tree = sexp('(page :title "Markets" ...)', markets=data)

    accept = request.headers.get("Accept", "")
    if "application/x-sexpr" in accept:
        return Response(serialize(tree), content_type="application/x-sexpr")

    html = render_to_html(tree)
    return Response(html, content_type="text/html")
```

### With ActivityPub (JSON-LD peers)

For Mastodon, Pleroma, and other JSON-LD AP servers:

| Peer Type | Outbound | Inbound |
|---|---|---|
| Standard AP (Mastodon, Pleroma) | Translate sexp → JSON-LD, include `rose:sexpr` extension field | Parse JSON-LD, translate to sexp internally |
| Sexp-aware AP (other rose-ash instances) | Native sexp on bidirectional stream | Parse sexp directly |
| Sexp-aware with shared components | Sexp with component references | Resolve from cache, render natively |

JSON-LD bridging for non-sexp peers:

```json
{
  "@context": ["https://www.w3.org/ns/activitystreams", "https://rose-ash.com/ns/sexpr"],
  "type": "Create",
  "object": {
    "type": "Note",
    "content": "<p>Hello world</p>",
    "rose:sexpr": "(p \"Hello world\")"
  }
}
```

No existing AP implementation breaks. The `rose:sexpr` field is ignored by servers that don't understand it.

### Fallback Gateway

For browsers without extension or native client:

```
sexpr://rose-ash.com/markets/
  → Gateway fetches sexp from server
  → Renders to HTML
  → Serves to browser at https://rose-ash.com/markets/
```

---

## Three Client Tiers

```
                    ┌─────────────────────────────┐
                    │     rose-ash server (Quart)  │
                    │                              │
                    │  Same sexp component tree    │
                    │  Same data, same logic       │
                    │                              │
                    ├──────────┬──────────┬────────┤
                    │ HTTPS    │ HTTPS    │ SEXPR  │
                    │ HTML out │ sexp out │ native │
                    └────┬─────┴────┬─────┴───┬────┘
                         │          │         │
              ┌──────────▼──┐ ┌─────▼──────┐ ┌▼──────────────┐
              │   Browser   │ │  Browser + │ │  Rust client   │
              │   (vanilla) │ │  extension │ │  (native)      │
              │             │ │            │ │                │
              │ HTML + HTMX │ │ sexpr.js   │ │ sexp protocol  │
              │ Full CSS    │ │ over HTTPS │ │ over QUIC      │
              │ ~200ms load │ │ ~80ms load │ │ ~20ms load     │
              └─────────────┘ └────────────┘ └────────────────┘
                 Tier 0           Tier 1          Tier 2
```

| | Tier 0: Browser | Tier 1: Extension | Tier 2: Rust Client |
|---|---|---|---|
| URL | `https://rose-ash.com` | `https://rose-ash.com` | `sexpr://rose-ash.com` |
| Protocol | HTTPS | HTTPS | sexpr:// over QUIC |
| Wire format | HTML | sexp over HTTP | sexp native stream |
| Rendering | Browser DOM | sexpr.js → DOM | Rust → GPU |
| Component cache | Browser cache (URL-keyed) | IndexedDB (hash-keyed) | Disk (hash-keyed, pre-parsed AST) |
| Real-time | HTMX polling / SSE | WebSocket sexp mutations | Native bidirectional stream |
| Federation | N/A | AP via fetch | Native sexp stream |
| Bundle size | HTMX 14KB + CSS | sexpr.js ~8KB | 5MB binary, zero runtime deps |
| Page load | ~200ms | ~80ms | ~20ms |
| Memory per page | ~200MB | ~150MB | ~20MB |
| AI integration | Parse HTML (painful) | Parse sexp (easy) | Native sexp (trivial) |

### Rust Client Architecture

```
┌─────────────────────────────────────┐
│        sexpr-client (Rust)          │
│                                     │
│  ┌──────────┐  ┌─────────────────┐  │
│  │  Parser   │  │  Component      │  │
│  │  (zero-   │  │  Cache          │  │
│  │   copy)   │  │  (SHA3 → AST)  │  │
│  └──────────┘  └─────────────────┘  │
│  ┌──────────┐  ┌─────────────────┐  │
│  │  Layout   │  │  Network        │  │
│  │  Engine   │  │  (tokio + QUIC) │  │
│  └──────────┘  └─────────────────┘  │
│  ┌──────────────────────────────┐   │
│  │  Renderer (wgpu / iced)      │   │
│  └──────────────────────────────┘   │
└─────────────────────────────────────┘
```

**Why it's fast:**
- Parser: nanoseconds per node (trivial recursive descent in Rust)
- No HTML parser, no CSS cascade, no DOM construction, no JS engine
- Render directly to GPU surface — skip the entire browser rendering pipeline
- Pre-parsed ASTs from disk cache — zero parse time on cache hit
- 10-50MB memory vs 200-500MB per browser tab

Page load: network (~50ms) → parse (~0.1ms) → layout (~2ms) → paint (~3ms) = **under 60ms**.

---

## Implementation Plan

### Phase 1: Content Negotiation (Quart)

Add `Accept` header handling to existing Quart routes.

- Check for `application/x-sexpr` in `Accept` header
- If present: serialize sexp tree, return directly
- If absent: render to HTML as today
- Add `Vary: Accept` header

**Files:**
- `shared/infrastructure/factory.py` — content negotiation middleware
- `shared/sexp/serialize.py` — canonical sexp serializer (deterministic for signing/caching)

**Result:** Existing routes serve sexp to any client that asks. Zero changes to route logic.

### Phase 2: Sexp ↔ JSON-LD Bridge

Bidirectional translation for AP federation with non-sexp peers:

**sexp → JSON-LD** (outbound):
- Activity type symbol → `"type"` field
- Keyword attributes → JSON object properties
- Content sexp → rendered to HTML for `"content"` field
- Include `rose:sexpr` extension field

**JSON-LD → sexp** (inbound):
- `"type"` → head symbol
- `"content"` HTML → parsed to sexp (best-effort)
- Nested objects → child expressions

**Files:**
- `shared/infrastructure/ap_sexpr.py` — serializer/deserializer
- `shared/sexp/html_to_sexp.py` — HTML → sexp parser for inbound content
- `shared/sexp/tests/test_ap_sexpr.py` — round-trip tests

### Phase 3: sexpr.js Client Library

Browser-side JS runtime (see `sexpr-js-runtime-plan.md`):

- Parser + renderer: `parse()` → AST → `renderToDOM()`
- Mutation engine: `swap!`, `batch!`, `class!`, `request!`
- Component registry with localStorage cache (content-addressed)
- WebSocket connection for real-time pushes

### Phase 4: Browser Extension

Package sexpr.js as a WebExtension:

- Register `sexpr://` protocol handler
- Intercept navigation → fetch via HTTPS with `Accept: application/x-sexpr`
- Parse → render to DOM
- Component cache in IndexedDB (content-addressed)

**Tech:** WebExtension API (Firefox + Chrome), JS/TS.

### Phase 5: Component Discovery

Enable peers to discover and cache shared components:

- Actor profiles include `:components` endpoint URL
- `GET /ap/components/` → manifest (name → hash)
- `GET /ap/components/{hash}` → component definition (sexp)
- Content-addressed cache on client (hash-keyed)
- Federated content with `(use "component" ...)` resolves from cache

**Files:**
- `federation/bp/components/routes.py` — manifest + fetch endpoints
- `shared/sexp/component_manifest.py` — hash generation

### Phase 6: Protocol Specification

Formal specification document:

- **Framing**: length-prefixed sexp over QUIC streams
- **Expression types**: requests, responses, pushes, activities, collections
- **Canonical serialization**: deterministic rules for signing
- **Caching**: content-hash based
- **Authentication**: token in request keywords
- **Component discovery**: manifest protocol
- **Bidirectional streaming**: lifecycle, multiplexing
- **Schema introspection**: `/__schema/` endpoint
- **JSON-LD bridging**: rules for non-sexp AP peers
- **Security**: no eval, escaping, signature verification

Publish as a FEP (Fediverse Enhancement Proposal) and standalone specification.

### Phase 7: Rust Protocol Server

QUIC server alongside Hypercorn:

- Listen on separate port (e.g., 4433)
- Parse sexp requests, route to same handler logic
- Bidirectional stream: pushes, requests, responses, activities
- Component manifest endpoint
- Federation delivery on persistent connections (replaces HTTP POST to inbox)

**Tech:** Rust, `quinn` (QUIC), `tokio`, `rustls`.

**Files:**
- `sexpr-server/src/protocol.rs` — framing, parsing, routing
- `sexpr-server/src/quic.rs` — QUIC listener + stream management
- `sexpr-server/src/federation.rs` — peer connection manager

### Phase 8: Rust Native Client

Standalone sexp document viewer:

- QUIC client for `sexpr://` URLs
- Zero-copy sexp parser (arena-allocated AST)
- Component cache on disk (SHA3-keyed)
- Layout engine (flexbox subset)
- GPU renderer via `wgpu` or `iced`
- Text rendering via `cosmic-text`
- Bidirectional stream: browsing + federation + real-time on one connection

**Tech:** Rust, `quinn`, `wgpu`/`iced`, `cosmic-text`, `tokio`.

**Files:**
- `sexpr-client/src/parser.rs` — zero-copy parser
- `sexpr-client/src/cache.rs` — content-addressed component cache
- `sexpr-client/src/layout.rs` — layout engine
- `sexpr-client/src/render.rs` — GPU renderer
- `sexpr-client/src/stream.rs` — bidirectional connection manager

### Phase 9: Fallback Gateway

HTTP proxy for browsers without extension/client:

- Accept HTTPS requests at `https://rose-ash.com`
- Fetch sexp from server internally
- Render to HTML, serve to browser
- Add `<link rel="alternate" type="application/x-sexpr" href="sexpr://...">` for discovery

---

## Migration Path

Each phase builds on the last. No phase breaks existing functionality:

```
Phase 1 (content negotiation) ── Tier 0 unchanged, sexp available
Phase 2 (JSON-LD bridge)      ── federation works with all AP peers
Phase 3 (sexpr.js)            ── Tier 1 client-side rendering
Phase 4 (extension)           ── sexpr:// URLs in browser
Phase 5 (components)          ── federated UI exchange
Phase 6 (spec)                ── formal protocol document
Phase 7 (Rust server)         ── native protocol alongside HTTPS
Phase 8 (Rust client)         ── Tier 2 native experience
Phase 9 (gateway)             ── sexpr:// accessible from any browser
```

Depends on:
- **Ghost removal** (see `ghost-removal-plan.md`) — posts must be sexp before Phases 2-3 add real value
- **sexpr.js runtime** (see `sexpr-js-runtime-plan.md`) — the JS library that powers Phases 3-4

---

## Client as Node: Cooperative Compute Mesh

### Everyone Has a Server

The original web was peer-to-peer — everyone ran a server on their workstation. Then we centralised everything into data centres because HTTP was stateless and browsers were passive. The sexp protocol with client-as-node reverses that.

Each member's Rust client is not just a viewer — it's a full peer node:
- An **ActivityPub instance** (keypair, identity, inbox/outbox)
- An **IPFS node** (storing and serving content-addressed data)
- An **artdag worker** (local GPU for media processing)
- A **sexp peer** (bidirectional streams to relay and other peers)

```
┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│ Alice        │  │ Bob          │  │ Charlie      │
│ RTX 4070     │  │ M2 MacBook   │  │ RX 7900      │
│ 12GB VRAM    │  │ 16GB unified │  │ 20GB VRAM    │
│              │  │              │  │              │
│ artdag node  │  │ artdag node  │  │ artdag node  │
│ IPFS node    │  │ IPFS node    │  │ IPFS node    │
│ sexp peer    │  │ sexp peer    │  │ sexp peer    │
└──────┬───────┘  └──────┬───────┘  └──────┬───────┘
       │                 │                 │
       └────────┬────────┴────────┬────────┘
                │                 │
       ┌────────▼─────────────────▼────────┐
       │  rose-ash relay                   │
       │                                   │
       │  • Message queue (offline inbox)  │
       │  • Capability registry            │
       │  • IPFS pinning service           │
       │  • HTTPS gateway (Tier 0)         │
       │  • Peer directory                 │
       │  • Federation bridge (JSON-LD)    │
       └───────────────────────────────────┘
```

### Offline Persistence

When a member's client goes offline, their content persists on IPFS. The relay provides two services:

**IPFS pinning** — members' CIDs are pinned by the cooperative's pinning node, ensuring content stays available even when the author's client is off. This is cheap — just disk storage, no compute.

**Message queuing** — activities addressed to an offline member are held by the relay and drained when they reconnect:

```scheme
;; Alice is offline. Bob sends her a message.
;; The relay holds it.

;; Alice's client comes online, connects to relay
(hello :actor "alice@rose-ash.com" :last-seen "2026-02-28T12:00:00Z")

;; Relay drains the queue
(queued :count 3 :since "2026-02-28T12:00:00Z"
  (Create :actor bob :published "2026-02-28T16:00:00Z"
    (Note (p "See you at the market Saturday!")))
  (Like :actor charlie :object alice-post-42)
  (Follow :actor dave :object alice))

;; Alice's client processes them, sends acknowledgment
(ack :through "2026-02-28T16:00:00Z")

;; Relay clears the queue. Now alice is live —
;; subsequent activities stream directly peer-to-peer.
```

### Cooperative GPU Sharing

Members contribute idle GPU cycles to the cooperative. The relay acts as a job matchmaker:

```scheme
;; Alice uploads a video. Her laptop has integrated graphics — too slow.
(submit-job
  :type "artdag/render"
  :recipe "bafyrecipe..."
  :input "bafyinput..."
  :requirements (:min-vram 8 :gpu #t))

;; Relay knows Charlie's RTX 7900 is online and idle.
;; Job routes to Charlie's client.
(job :id "job-789" :assigned-to charlie
  :type "artdag/render"
  :recipe "bafyrecipe..."
  :input "bafyinput...")

;; Charlie's client runs the job, pins result to IPFS
(job-complete :id "job-789"
  :output "bafyoutput..."
  :duration-ms 4200
  :worker charlie)

;; Alice gets notified
(push! (swap! "#render-status" :inner
  (use "render-complete" :cid "bafyoutput...")))
```

This is already how artdag works conceptually. The L1 server is a Celery worker that picks up rendering tasks. Replace "Celery worker on a cloud server" with "Celery worker on a member's desktop" and the architecture barely changes. The task queue just has different workers.

### Economics

| | Centralised (current) | Cooperative mesh |
|---|---|---|
| Image/video processing | Cloud GPU ($2-5/hr) | Member's local GPU (free) |
| Content storage | Server disk + S3 | IPFS (distributed) + pinning |
| Content serving | Server bandwidth | Peer-to-peer + IPFS |
| Server cost | GPU instances + storage + bandwidth | Cheap relay (CPU + disk only) |
| Scaling | More users = more cost | More members = more capacity |

The co-op's infrastructure cost drops to: **one small VPS + IPFS pinning storage.** That's it. All compute — rendering, processing, serving content — is distributed across members' machines.

More members joining makes the network faster and more capable, not more expensive. Like BitTorrent seeding, but for an entire application platform.

### The Relay Server's Role

The relay is minimal — a matchmaker and persistence layer, not a compute provider:

- **Peer directory**: who's online, their QUIC address, their GPU capabilities
- **Message queue**: hold activities for offline members
- **IPFS pinning**: persist content when authors are offline
- **HTTPS gateway**: serve HTML to Tier 0 browsers (visitors, search engines)
- **Federation bridge**: translate sexp ↔ JSON-LD for Mastodon/Pleroma peers
- **Job queue**: match GPU-intensive tasks to available peers
- **Capability registry**: what each peer can do (GPU model, VRAM, storage)

The relay does no rendering, no media processing, no content generation. Its cost stays flat regardless of member count.

### Content Flow

```
Author creates post:
  1. Edit in Rust client (local)
  2. Render media with local GPU (artdag)
  3. Pin content + media to IPFS (local node)
  4. Publish CIDs to relay (for pinning + discovery)
  5. Stream activity to connected followers (peer-to-peer)
  6. Relay queues activity for offline followers

Reader views post:
  1. Fetch sexp from author's client (if online, peer-to-peer)
  2. Or fetch from IPFS by CID (if author offline)
  3. Or fetch from relay gateway as HTML (if Tier 0 browser)
  4. Components resolved from local cache (content-addressed)
  5. Render locally (Rust GPU or sexpr.js in browser)
```

No server rendered anything. No server stored anything permanently. No server paid for GPU time. The cooperative's members are the infrastructure.

---

## Cooperative Angle

- Members install the Rust client → fast native experience, 5MB binary, no app store
- Visitors browse `https://rose-ash.com` → standard HTML, no barrier
- Federated co-ops connect via persistent sexp streams → rich UI exchange, not just text syndication
- AI agents speak the protocol natively → components as tool calls, mutations as actions
- Auto-updates via content-addressed components → no gatekeeping
- The component registry is a shared vocabulary across the cooperative network
- Members' desktops are the cloud — contributing GPU, storage, and bandwidth
- The relay server stays cheap and flat-cost regardless of growth
- The original vision of the web: everyone has a server

---

## Relationship to Other Plans

| Document | Role |
|---|---|
| `sexpr-js-runtime-plan.md` | The JS library powering Tier 1 (Phases 3-4) |
| `ghost-removal-plan.md` | Posts must be sexp before federation/client rendering adds value |
| `sexpr-ai-integration.md` | AI agents benefit from all tiers and the self-describing schema |
| artdag (`artdag/`) | The media processing engine that runs on member GPUs |

---

*The document is the program. The program is the document. The protocol is both. The network is its members.*