giles
19240c6ca3
Members' Rust clients become full peer nodes — AP instances, IPFS nodes, and artdag GPU workers. The relay server becomes a lightweight matchmaker (message queue, pinning, peer directory) while all compute, rendering, and content serving is distributed across members' own hardware. Back to the original vision of the web: everyone has a server. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
885 lines
33 KiB
Markdown
885 lines
33 KiB
Markdown
# The Sexp Protocol
|
|
|
|
**A unified protocol for documents, applications, and federation.**
|
|
**One format. Every boundary.**
|
|
|
|
---
|
|
|
|
## Core Insight
|
|
|
|
The sexp protocol replaces HTTP, HTML, JSON-LD, WebSocket, and ActivityPub's inbox model with a single concept: **peers exchange s-expressions on a bidirectional stream.**
|
|
|
|
There is no distinction between:
|
|
- A client requesting a page (what HTTP does)
|
|
- A server pushing a real-time update (what WebSocket does)
|
|
- A peer delivering a federated activity (what AP inbox POST does)
|
|
- A server sending a DOM mutation (what HTMX does)
|
|
|
|
They are all sexp expressions sent between two peers:
|
|
|
|
```scheme
|
|
;; Browsing (request → response)
|
|
(GET "/markets/")
|
|
(ok (page :title "Markets" ...))
|
|
|
|
;; Real-time push (server → client, unsolicited)
|
|
(push! (swap! "#feed" :prepend (use "post-card" :title "New post")))
|
|
|
|
;; Federation delivery (peer → peer)
|
|
(Create :actor "alice@rose-ash.com"
|
|
(Note :id post-123 (p "Hello from the fediverse!")))
|
|
|
|
;; Client action (client → server)
|
|
(POST "/like/" :body (:post-id 123))
|
|
|
|
;; Mutation response (server → client)
|
|
(push! (swap! "#like-count-123" :inner "43"))
|
|
```
|
|
|
|
Same parser. Same stream. Same connection. The "type" of interaction is determined by the head symbol of the expression, not by the protocol layer.
|
|
|
|
---
|
|
|
|
## Why Replace HTTP and JSON-LD
|
|
|
|
### HTTP's Problems
|
|
|
|
HTTP is a request-response protocol with text headers and a body. Strip away the historical baggage and what you actually need is:
|
|
|
|
```
|
|
Peer A sends: verb + target + metadata + optional body
|
|
Peer B sends: status + metadata + body
|
|
```
|
|
|
|
That's just an s-expression. But HTTP adds:
|
|
- **Header/body split** — two parsing phases, different formats
|
|
- **Rigid request-response** — to work around this, we bolted on WebSocket (separate protocol, upgrade handshake), SSE (chunked encoding hack), HTTP/2 push (failed, removed), HTTP/3 QUIC streams (complex, still one-request-one-response per stream)
|
|
- **Overcomplicated caching** — `Cache-Control`, `ETag`, `If-None-Match`, `If-Modified-Since`, `Vary`, `Age`, `Expires`, `Last-Modified`, `s-maxage`, `stale-while-revalidate` — a dozen headers with complex interaction rules
|
|
- **Arbitrary status codes** — memorised numbers with no semantic meaning in the format
|
|
- **Form encoding mess** — `application/x-www-form-urlencoded` or `multipart/form-data` with boundary strings, MIME parts, content-disposition headers
|
|
|
|
### JSON-LD's Problems
|
|
|
|
JSON-LD was chosen for ActivityPub because of its semantic web lineage, but in practice nobody uses the linked data features — servers just pattern-match on `type` fields and ignore the `@context`. The format is:
|
|
|
|
- **Verbose** — deeply nested objects with string keys, quoted values, commas, colons
|
|
- **Ambiguous** — compaction/expansion rules mean the same activity can have many valid JSON representations
|
|
- **Lossy** — post content is flattened to an HTML string inside a JSON string; structure destroyed
|
|
- **Hostile to signing** — JSON-LD canonicalization is fragile and implementation-dependent
|
|
- **Hostile to AI** — agents must parse JSON, then parse embedded HTML, then reason about structure
|
|
|
|
### The Sexp Solution
|
|
|
|
One format that is:
|
|
- **Compact** — half the size of equivalent JSON-LD, no closing tags like HTML
|
|
- **Unambiguous** — one canonical serialization, deterministic for signing
|
|
- **Structural** — content *is* the tree, not a string embedded in a string
|
|
- **Parseable** — trivial recursive descent, no backtracking, nanoseconds per node
|
|
- **Bidirectional** — requests, responses, pushes, and activities all use the same syntax
|
|
- **AI-native** — agents parse, generate, and reason about sexp as naturally as tool calls
|
|
|
|
---
|
|
|
|
## Protocol Specification
|
|
|
|
### Transport
|
|
|
|
```
|
|
┌─────────────────────────────────────────┐
|
|
│ Application: sexp expressions │
|
|
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
|
|
│ Session: bidirectional sexp stream │
|
|
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
|
|
│ Transport: QUIC (or TCP+TLS fallback) │
|
|
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
|
|
│ Network: IP │
|
|
└─────────────────────────────────────────┘
|
|
```
|
|
|
|
QUIC is ideal — multiplexed streams, built-in TLS, connection migration. Each sexp expression gets its own QUIC stream. Requests and pushes multiplex without head-of-line blocking. Connection survives network changes (mobile → wifi).
|
|
|
|
**Framing:** Length-prefixed sexp expressions over QUIC streams. The parser knows when an expression ends (balanced parens), but length-prefixing enables efficient buffer allocation.
|
|
|
|
**URL scheme:** `sexpr://host:port/path`
|
|
|
|
### Expression Types
|
|
|
|
Every expression on the wire is one of these forms:
|
|
|
|
#### Requests (client → server)
|
|
|
|
```scheme
|
|
(GET "/markets/"
|
|
:auth "bearer tok_abc123"
|
|
:have "sha3-a1b2c3"
|
|
:have-components ("sha3-aaa" "sha3-bbb"))
|
|
|
|
(POST "/like/"
|
|
:auth "bearer tok_abc123"
|
|
:body (:post-id 123))
|
|
|
|
(POST "/submit/"
|
|
:body (
|
|
:username "alice"
|
|
:email "alice@example.com"
|
|
:avatar (file :name "photo.jpg" :type "image/jpeg" :data <binary>)))
|
|
|
|
;; Streaming request — keep connection open for pushes
|
|
(GET "/feed/" :stream #t)
|
|
```
|
|
|
|
#### Responses (server → client)
|
|
|
|
```scheme
|
|
;; Success with content
|
|
(ok :hash "sha3-d4e5f6"
|
|
(page :title "Markets" :layout "main"
|
|
(section (h1 "Markets")
|
|
(each markets (lambda (m)
|
|
(use "vendor-card" :name (get m "name")))))))
|
|
|
|
;; Not modified (client already has current version)
|
|
(not-modified)
|
|
|
|
;; Redirect
|
|
(redirect :to "/new-location/" :permanent #t)
|
|
|
|
;; Not found
|
|
(not-found :message "Page does not exist")
|
|
|
|
;; Error
|
|
(error :code "auth-required" :message "Please log in"
|
|
:login-url "sexpr://rose-ash.com/login/")
|
|
|
|
;; Response with new components the client is missing
|
|
(ok
|
|
:new-components (
|
|
(component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
|
|
(page ...))
|
|
```
|
|
|
|
#### Pushes (server → client, unsolicited)
|
|
|
|
```scheme
|
|
;; DOM mutation
|
|
(push! (swap! "#feed" :prepend
|
|
(use "post-card" :title "New post" :author "alice")))
|
|
|
|
;; Batch mutation
|
|
(push! (batch!
|
|
(swap! "#notifications" :append (div :class "toast" "Saved!"))
|
|
(class! "#save-btn" :remove "loading")))
|
|
|
|
;; Component update (new version available)
|
|
(push! (component-update "vendor-card" :hash "sha3-eee"))
|
|
```
|
|
|
|
#### Activities (peer → peer, federation)
|
|
|
|
```scheme
|
|
;; Create
|
|
(Create
|
|
:id "https://rose-ash.com/ap/activities/456"
|
|
:actor "https://rose-ash.com/ap/users/alice"
|
|
:published "2026-02-28T14:00:00Z"
|
|
:to (:public)
|
|
:cc ("https://rose-ash.com/ap/users/alice/followers")
|
|
|
|
(Note
|
|
:id "https://rose-ash.com/ap/posts/123"
|
|
:attributed-to "https://rose-ash.com/ap/users/alice"
|
|
:in-reply-to "https://remote.social/posts/789"
|
|
:summary "Weekend market update"
|
|
|
|
(section
|
|
(p "Three new vendors joining this Saturday.")
|
|
(use "vendor-list" :vendors vendors)
|
|
(use "calendar-widget" :calendar-id 42))
|
|
|
|
(attachment
|
|
(Image :url "https://rose-ash.com/media/market.jpg"
|
|
:media-type "image/jpeg"
|
|
:width 1200 :height 800
|
|
:name "Saturday market stalls"))
|
|
|
|
(tag
|
|
(Hashtag :name "#markets" :href "https://rose-ash.com/tags/markets")
|
|
(Mention :name "@bob@remote.social" :href "https://remote.social/users/bob"))))
|
|
|
|
;; Follow
|
|
(Follow :actor "https://rose-ash.com/ap/users/alice"
|
|
:object "https://remote.social/users/bob")
|
|
|
|
;; Accept (response to Follow — on the same connection)
|
|
(Accept :actor "https://remote.social/users/bob"
|
|
(Follow :actor "https://rose-ash.com/ap/users/alice"
|
|
:object "https://remote.social/users/bob"))
|
|
|
|
;; Like, Announce, Undo — all just expressions
|
|
(Like :actor alice :object post-123)
|
|
(Announce :actor bob (Note :id post-123))
|
|
(Undo :actor alice (Like :actor alice :object post-123))
|
|
```
|
|
|
|
**Key insight:** Activities are not "posted to an inbox" — they are expressions sent on the bidirectional stream. The peer-to-peer connection *is* the inbox. When you follow someone, their activities stream to you on the same connection you used to follow them. No inbox endpoint, no HTTP POST delivery, no polling.
|
|
|
|
#### Collections (queryable, paginated)
|
|
|
|
```scheme
|
|
;; Request an actor's outbox
|
|
(GET "/ap/users/alice/outbox" :page 1)
|
|
|
|
;; Response
|
|
(ok
|
|
(OrderedCollectionPage
|
|
:part-of "https://rose-ash.com/ap/users/alice/outbox"
|
|
:next "/ap/users/alice/outbox?page=2"
|
|
:total-items 142
|
|
|
|
(Create :actor alice :published "2026-02-28T14:00:00Z"
|
|
(Note :id post-3 (p "Latest post")))
|
|
(Announce :actor alice :published "2026-02-27T09:00:00Z"
|
|
(Note :id post-2 :attributed-to bob))
|
|
(Create :actor alice :published "2026-02-26T18:00:00Z"
|
|
(Note :id post-1 (p "Earlier post")))))
|
|
```
|
|
|
|
#### Actor Profiles
|
|
|
|
```scheme
|
|
(Person
|
|
:id "https://rose-ash.com/ap/users/alice"
|
|
:preferred-username "alice"
|
|
:name "Alice"
|
|
:summary (p "Co-op member, market organiser")
|
|
:inbox "sexpr://rose-ash.com/ap/users/alice/inbox"
|
|
:outbox "sexpr://rose-ash.com/ap/users/alice/outbox"
|
|
:followers "sexpr://rose-ash.com/ap/users/alice/followers"
|
|
:following "sexpr://rose-ash.com/ap/users/alice/following"
|
|
:components "sexpr://rose-ash.com/ap/components/"
|
|
:public-key (:id "https://rose-ash.com/ap/users/alice#main-key"
|
|
:owner "https://rose-ash.com/ap/users/alice"
|
|
:pem "-----BEGIN PUBLIC KEY-----\n..."))
|
|
```
|
|
|
|
#### Schema Introspection
|
|
|
|
```scheme
|
|
(GET "/__schema/")
|
|
|
|
(ok
|
|
(schema
|
|
(endpoint "/" :method GET
|
|
:returns (page)
|
|
:params (:stream bool))
|
|
(endpoint "/markets/" :method GET
|
|
:returns (page :contains (list (use "vendor-card"))))
|
|
(endpoint "/like/" :method POST
|
|
:params (:post-id int)
|
|
:returns (mutation))
|
|
(activity Create
|
|
:object (Note)
|
|
:delivers-to :followers)
|
|
(activity Follow
|
|
:object (Person)
|
|
:expects (Accept))))
|
|
```
|
|
|
|
An AI agent hitting `/__schema/` learns the entire surface — pages, actions, federation activities — as parseable sexp. No separate OpenAPI doc. The schema *is* the API.
|
|
|
|
### Caching
|
|
|
|
Content-addressed, hash-based. No expiry headers, no revalidation dance.
|
|
|
|
```scheme
|
|
;; Client has a cached version
|
|
(GET "/markets/" :have "sha3-a1b2c3")
|
|
|
|
;; Unchanged
|
|
(not-modified)
|
|
|
|
;; Changed — new hash included
|
|
(ok :hash "sha3-d4e5f6" (page ...))
|
|
```
|
|
|
|
Component-level caching:
|
|
|
|
```scheme
|
|
;; Client reports which components it has
|
|
(GET "/markets/" :have-components ("sha3-aaa" "sha3-bbb"))
|
|
|
|
;; Server sends only missing components alongside the page
|
|
(ok
|
|
:new-components (
|
|
(component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
|
|
(page ...))
|
|
```
|
|
|
|
The hash *is* the cache key, the ETag, and the content address. One concept replaces twelve HTTP headers.
|
|
|
|
### Component Discovery
|
|
|
|
Actors advertise a `:components` endpoint in their profile. Peers fetch and cache component definitions by content hash:
|
|
|
|
```scheme
|
|
;; Fetch component manifest
|
|
(GET "/ap/components/")
|
|
|
|
(ok
|
|
(component-manifest
|
|
("cart-mini" :hash "sha3-aaa")
|
|
("vendor-card" :hash "sha3-bbb")
|
|
("calendar-widget" :hash "sha3-ccc")
|
|
("post-card" :hash "sha3-ddd")))
|
|
|
|
;; Fetch a specific component by hash
|
|
(GET "/ap/components/sha3-bbb")
|
|
|
|
(ok
|
|
(component "vendor-card" :hash "sha3-bbb"
|
|
(params name stall image)
|
|
(div :class "vendor-card"
|
|
(img :src image :alt name)
|
|
(h3 name)
|
|
(p :class "stall" stall))))
|
|
```
|
|
|
|
When a federated activity includes `(use "vendor-card" :name "Sourdough" :stall "A12")`, the receiving peer resolves the component from its cache (by hash) or fetches it from the sender's manifest. Federated UI, not just federated data.
|
|
|
|
### Signatures
|
|
|
|
Sexp serialization has a canonical form:
|
|
- Keywords sorted alphabetically
|
|
- Single space between atoms
|
|
- No trailing whitespace
|
|
- UTF-8 encoding
|
|
|
|
This makes signatures deterministic without JSON-LD canonicalization:
|
|
|
|
```scheme
|
|
(signed :sig "base64..." :key-id "https://rose-ash.com/ap/users/alice#main-key"
|
|
(Create :actor "https://rose-ash.com/ap/users/alice"
|
|
(Note :id post-123 (p "Hello"))))
|
|
```
|
|
|
|
Sign the canonical serialization of the inner expression. Verify by re-serializing and checking. No compaction/expansion ambiguity.
|
|
|
|
---
|
|
|
|
## Bidirectional Stream: The Unified Model
|
|
|
|
A connection between two sexp-speaking peers carries everything on one stream:
|
|
|
|
```scheme
|
|
;; === Connection opened ===
|
|
|
|
;; Client browses
|
|
(GET "/")
|
|
(ok (page :title "Home" ...))
|
|
|
|
;; Client navigates (same connection)
|
|
(GET "/markets/")
|
|
(ok (page :title "Markets" ...))
|
|
|
|
;; Client follows a user (federation)
|
|
(Follow :actor alice :object bob)
|
|
|
|
;; Server confirms
|
|
(Accept :actor bob (Follow :actor alice :object bob))
|
|
|
|
;; Bob posts something — server pushes activity
|
|
(Create :actor bob :published "2026-02-28T15:00:00Z"
|
|
(Note :id post-456 (p "New vendor announcement!")))
|
|
|
|
;; Client sees it rendered in real-time via mutation
|
|
(push! (swap! "#feed" :prepend
|
|
(use "post-card" :actor "bob" :content (p "New vendor announcement!"))))
|
|
|
|
;; Client likes the post
|
|
(POST "/like/" :body (:post-id 456))
|
|
|
|
;; Server pushes mutation + delivers Like activity to bob
|
|
(push! (swap! "#like-count-456" :inner "12"))
|
|
|
|
;; Client opens a streaming feed
|
|
(GET "/feed/" :stream #t)
|
|
(ok :stream #t (page :title "Feed" ...))
|
|
|
|
;; More activities arrive over time...
|
|
(Create :actor charlie :published "2026-02-28T15:30:00Z"
|
|
(Note :id post-789 (p "Market day tomorrow!")))
|
|
(push! (swap! "#feed" :prepend
|
|
(use "post-card" :actor "charlie" :content (p "Market day tomorrow!"))))
|
|
|
|
;; === Connection persists ===
|
|
```
|
|
|
|
Browsing, federation, real-time updates, and user actions — all on one bidirectional stream. The distinction between "web server", "AP inbox", and "WebSocket" disappears. They were always the same thing — peers exchanging structured expressions.
|
|
|
|
---
|
|
|
|
## Backwards Compatibility
|
|
|
|
### With HTTP (Tier 0 and Tier 1)
|
|
|
|
The sexp protocol runs alongside HTTPS. The same server handles both:
|
|
|
|
```python
|
|
@bp.get("/markets/")
|
|
async def markets():
|
|
data = await get_markets(g.s)
|
|
tree = sexp('(page :title "Markets" ...)', markets=data)
|
|
|
|
accept = request.headers.get("Accept", "")
|
|
if "application/x-sexpr" in accept:
|
|
return Response(serialize(tree), content_type="application/x-sexpr")
|
|
|
|
html = render_to_html(tree)
|
|
return Response(html, content_type="text/html")
|
|
```
|
|
|
|
### With ActivityPub (JSON-LD peers)
|
|
|
|
For Mastodon, Pleroma, and other JSON-LD AP servers:
|
|
|
|
| Peer Type | Outbound | Inbound |
|
|
|---|---|---|
|
|
| Standard AP (Mastodon, Pleroma) | Translate sexp → JSON-LD, include `rose:sexpr` extension field | Parse JSON-LD, translate to sexp internally |
|
|
| Sexp-aware AP (other rose-ash instances) | Native sexp on bidirectional stream | Parse sexp directly |
|
|
| Sexp-aware with shared components | Sexp with component references | Resolve from cache, render natively |
|
|
|
|
JSON-LD bridging for non-sexp peers:
|
|
|
|
```json
|
|
{
|
|
"@context": ["https://www.w3.org/ns/activitystreams", "https://rose-ash.com/ns/sexpr"],
|
|
"type": "Create",
|
|
"object": {
|
|
"type": "Note",
|
|
"content": "<p>Hello world</p>",
|
|
"rose:sexpr": "(p \"Hello world\")"
|
|
}
|
|
}
|
|
```
|
|
|
|
No existing AP implementation breaks. The `rose:sexpr` field is ignored by servers that don't understand it.
|
|
|
|
### Fallback Gateway
|
|
|
|
For browsers without extension or native client:
|
|
|
|
```
|
|
sexpr://rose-ash.com/markets/
|
|
→ Gateway fetches sexp from server
|
|
→ Renders to HTML
|
|
→ Serves to browser at https://rose-ash.com/markets/
|
|
```
|
|
|
|
---
|
|
|
|
## Three Client Tiers
|
|
|
|
```
|
|
┌─────────────────────────────┐
|
|
│ rose-ash server (Quart) │
|
|
│ │
|
|
│ Same sexp component tree │
|
|
│ Same data, same logic │
|
|
│ │
|
|
├──────────┬──────────┬────────┤
|
|
│ HTTPS │ HTTPS │ SEXPR │
|
|
│ HTML out │ sexp out │ native │
|
|
└────┬─────┴────┬─────┴───┬────┘
|
|
│ │ │
|
|
┌──────────▼──┐ ┌─────▼──────┐ ┌▼──────────────┐
|
|
│ Browser │ │ Browser + │ │ Rust client │
|
|
│ (vanilla) │ │ extension │ │ (native) │
|
|
│ │ │ │ │ │
|
|
│ HTML + HTMX │ │ sexpr.js │ │ sexp protocol │
|
|
│ Full CSS │ │ over HTTPS │ │ over QUIC │
|
|
│ ~200ms load │ │ ~80ms load │ │ ~20ms load │
|
|
└─────────────┘ └────────────┘ └────────────────┘
|
|
Tier 0 Tier 1 Tier 2
|
|
```
|
|
|
|
| | Tier 0: Browser | Tier 1: Extension | Tier 2: Rust Client |
|
|
|---|---|---|---|
|
|
| URL | `https://rose-ash.com` | `https://rose-ash.com` | `sexpr://rose-ash.com` |
|
|
| Protocol | HTTPS | HTTPS | sexpr:// over QUIC |
|
|
| Wire format | HTML | sexp over HTTP | sexp native stream |
|
|
| Rendering | Browser DOM | sexpr.js → DOM | Rust → GPU |
|
|
| Component cache | Browser cache (URL-keyed) | IndexedDB (hash-keyed) | Disk (hash-keyed, pre-parsed AST) |
|
|
| Real-time | HTMX polling / SSE | WebSocket sexp mutations | Native bidirectional stream |
|
|
| Federation | N/A | AP via fetch | Native sexp stream |
|
|
| Bundle size | HTMX 14KB + CSS | sexpr.js ~8KB | 5MB binary, zero runtime deps |
|
|
| Page load | ~200ms | ~80ms | ~20ms |
|
|
| Memory per page | ~200MB | ~150MB | ~20MB |
|
|
| AI integration | Parse HTML (painful) | Parse sexp (easy) | Native sexp (trivial) |
|
|
|
|
### Rust Client Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────┐
|
|
│ sexpr-client (Rust) │
|
|
│ │
|
|
│ ┌──────────┐ ┌─────────────────┐ │
|
|
│ │ Parser │ │ Component │ │
|
|
│ │ (zero- │ │ Cache │ │
|
|
│ │ copy) │ │ (SHA3 → AST) │ │
|
|
│ └──────────┘ └─────────────────┘ │
|
|
│ ┌──────────┐ ┌─────────────────┐ │
|
|
│ │ Layout │ │ Network │ │
|
|
│ │ Engine │ │ (tokio + QUIC) │ │
|
|
│ └──────────┘ └─────────────────┘ │
|
|
│ ┌──────────────────────────────┐ │
|
|
│ │ Renderer (wgpu / iced) │ │
|
|
│ └──────────────────────────────┘ │
|
|
└─────────────────────────────────────┘
|
|
```
|
|
|
|
**Why it's fast:**
|
|
- Parser: nanoseconds per node (trivial recursive descent in Rust)
|
|
- No HTML parser, no CSS cascade, no DOM construction, no JS engine
|
|
- Render directly to GPU surface — skip the entire browser rendering pipeline
|
|
- Pre-parsed ASTs from disk cache — zero parse time on cache hit
|
|
- 10-50MB memory vs 200-500MB per browser tab
|
|
|
|
Page load: network (~50ms) → parse (~0.1ms) → layout (~2ms) → paint (~3ms) = **under 60ms**.
|
|
|
|
---
|
|
|
|
## Implementation Plan
|
|
|
|
### Phase 1: Content Negotiation (Quart)
|
|
|
|
Add `Accept` header handling to existing Quart routes.
|
|
|
|
- Check for `application/x-sexpr` in `Accept` header
|
|
- If present: serialize sexp tree, return directly
|
|
- If absent: render to HTML as today
|
|
- Add `Vary: Accept` header
|
|
|
|
**Files:**
|
|
- `shared/infrastructure/factory.py` — content negotiation middleware
|
|
- `shared/sexp/serialize.py` — canonical sexp serializer (deterministic for signing/caching)
|
|
|
|
**Result:** Existing routes serve sexp to any client that asks. Zero changes to route logic.
|
|
|
|
### Phase 2: Sexp ↔ JSON-LD Bridge
|
|
|
|
Bidirectional translation for AP federation with non-sexp peers:
|
|
|
|
**sexp → JSON-LD** (outbound):
|
|
- Activity type symbol → `"type"` field
|
|
- Keyword attributes → JSON object properties
|
|
- Content sexp → rendered to HTML for `"content"` field
|
|
- Include `rose:sexpr` extension field
|
|
|
|
**JSON-LD → sexp** (inbound):
|
|
- `"type"` → head symbol
|
|
- `"content"` HTML → parsed to sexp (best-effort)
|
|
- Nested objects → child expressions
|
|
|
|
**Files:**
|
|
- `shared/infrastructure/ap_sexpr.py` — serializer/deserializer
|
|
- `shared/sexp/html_to_sexp.py` — HTML → sexp parser for inbound content
|
|
- `shared/sexp/tests/test_ap_sexpr.py` — round-trip tests
|
|
|
|
### Phase 3: sexpr.js Client Library
|
|
|
|
Browser-side JS runtime (see `sexpr-js-runtime-plan.md`):
|
|
|
|
- Parser + renderer: `parse()` → AST → `renderToDOM()`
|
|
- Mutation engine: `swap!`, `batch!`, `class!`, `request!`
|
|
- Component registry with localStorage cache (content-addressed)
|
|
- WebSocket connection for real-time pushes
|
|
|
|
### Phase 4: Browser Extension
|
|
|
|
Package sexpr.js as a WebExtension:
|
|
|
|
- Register `sexpr://` protocol handler
|
|
- Intercept navigation → fetch via HTTPS with `Accept: application/x-sexpr`
|
|
- Parse → render to DOM
|
|
- Component cache in IndexedDB (content-addressed)
|
|
|
|
**Tech:** WebExtension API (Firefox + Chrome), JS/TS.
|
|
|
|
### Phase 5: Component Discovery
|
|
|
|
Enable peers to discover and cache shared components:
|
|
|
|
- Actor profiles include `:components` endpoint URL
|
|
- `GET /ap/components/` → manifest (name → hash)
|
|
- `GET /ap/components/{hash}` → component definition (sexp)
|
|
- Content-addressed cache on client (hash-keyed)
|
|
- Federated content with `(use "component" ...)` resolves from cache
|
|
|
|
**Files:**
|
|
- `federation/bp/components/routes.py` — manifest + fetch endpoints
|
|
- `shared/sexp/component_manifest.py` — hash generation
|
|
|
|
### Phase 6: Protocol Specification
|
|
|
|
Formal specification document:
|
|
|
|
- **Framing**: length-prefixed sexp over QUIC streams
|
|
- **Expression types**: requests, responses, pushes, activities, collections
|
|
- **Canonical serialization**: deterministic rules for signing
|
|
- **Caching**: content-hash based
|
|
- **Authentication**: token in request keywords
|
|
- **Component discovery**: manifest protocol
|
|
- **Bidirectional streaming**: lifecycle, multiplexing
|
|
- **Schema introspection**: `/__schema/` endpoint
|
|
- **JSON-LD bridging**: rules for non-sexp AP peers
|
|
- **Security**: no eval, escaping, signature verification
|
|
|
|
Publish as a FEP (Fediverse Enhancement Proposal) and standalone specification.
|
|
|
|
### Phase 7: Rust Protocol Server
|
|
|
|
QUIC server alongside Hypercorn:
|
|
|
|
- Listen on separate port (e.g., 4433)
|
|
- Parse sexp requests, route to same handler logic
|
|
- Bidirectional stream: pushes, requests, responses, activities
|
|
- Component manifest endpoint
|
|
- Federation delivery on persistent connections (replaces HTTP POST to inbox)
|
|
|
|
**Tech:** Rust, `quinn` (QUIC), `tokio`, `rustls`.
|
|
|
|
**Files:**
|
|
- `sexpr-server/src/protocol.rs` — framing, parsing, routing
|
|
- `sexpr-server/src/quic.rs` — QUIC listener + stream management
|
|
- `sexpr-server/src/federation.rs` — peer connection manager
|
|
|
|
### Phase 8: Rust Native Client
|
|
|
|
Standalone sexp document viewer:
|
|
|
|
- QUIC client for `sexpr://` URLs
|
|
- Zero-copy sexp parser (arena-allocated AST)
|
|
- Component cache on disk (SHA3-keyed)
|
|
- Layout engine (flexbox subset)
|
|
- GPU renderer via `wgpu` or `iced`
|
|
- Text rendering via `cosmic-text`
|
|
- Bidirectional stream: browsing + federation + real-time on one connection
|
|
|
|
**Tech:** Rust, `quinn`, `wgpu`/`iced`, `cosmic-text`, `tokio`.
|
|
|
|
**Files:**
|
|
- `sexpr-client/src/parser.rs` — zero-copy parser
|
|
- `sexpr-client/src/cache.rs` — content-addressed component cache
|
|
- `sexpr-client/src/layout.rs` — layout engine
|
|
- `sexpr-client/src/render.rs` — GPU renderer
|
|
- `sexpr-client/src/stream.rs` — bidirectional connection manager
|
|
|
|
### Phase 9: Fallback Gateway
|
|
|
|
HTTP proxy for browsers without extension/client:
|
|
|
|
- Accept HTTPS requests at `https://rose-ash.com`
|
|
- Fetch sexp from server internally
|
|
- Render to HTML, serve to browser
|
|
- Add `<link rel="alternate" type="application/x-sexpr" href="sexpr://...">` for discovery
|
|
|
|
---
|
|
|
|
## Migration Path
|
|
|
|
Each phase builds on the last. No phase breaks existing functionality:
|
|
|
|
```
|
|
Phase 1 (content negotiation) ── Tier 0 unchanged, sexp available
|
|
Phase 2 (JSON-LD bridge) ── federation works with all AP peers
|
|
Phase 3 (sexpr.js) ── Tier 1 client-side rendering
|
|
Phase 4 (extension) ── sexpr:// URLs in browser
|
|
Phase 5 (components) ── federated UI exchange
|
|
Phase 6 (spec) ── formal protocol document
|
|
Phase 7 (Rust server) ── native protocol alongside HTTPS
|
|
Phase 8 (Rust client) ── Tier 2 native experience
|
|
Phase 9 (gateway) ── sexpr:// accessible from any browser
|
|
```
|
|
|
|
Depends on:
|
|
- **Ghost removal** (see `ghost-removal-plan.md`) — posts must be sexp before Phases 2-3 add real value
|
|
- **sexpr.js runtime** (see `sexpr-js-runtime-plan.md`) — the JS library that powers Phases 3-4
|
|
|
|
---
|
|
|
|
## Client as Node: Cooperative Compute Mesh
|
|
|
|
### Everyone Has a Server
|
|
|
|
The original web was peer-to-peer — everyone ran a server on their workstation. Then we centralised everything into data centres because HTTP was stateless and browsers were passive. The sexp protocol with client-as-node reverses that.
|
|
|
|
Each member's Rust client is not just a viewer — it's a full peer node:
|
|
- An **ActivityPub instance** (keypair, identity, inbox/outbox)
|
|
- An **IPFS node** (storing and serving content-addressed data)
|
|
- An **artdag worker** (local GPU for media processing)
|
|
- A **sexp peer** (bidirectional streams to relay and other peers)
|
|
|
|
```
|
|
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
|
│ Alice │ │ Bob │ │ Charlie │
|
|
│ RTX 4070 │ │ M2 MacBook │ │ RX 7900 │
|
|
│ 12GB VRAM │ │ 16GB unified │ │ 20GB VRAM │
|
|
│ │ │ │ │ │
|
|
│ artdag node │ │ artdag node │ │ artdag node │
|
|
│ IPFS node │ │ IPFS node │ │ IPFS node │
|
|
│ sexp peer │ │ sexp peer │ │ sexp peer │
|
|
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
|
|
│ │ │
|
|
└────────┬────────┴────────┬────────┘
|
|
│ │
|
|
┌────────▼─────────────────▼────────┐
|
|
│ rose-ash relay │
|
|
│ │
|
|
│ • Message queue (offline inbox) │
|
|
│ • Capability registry │
|
|
│ • IPFS pinning service │
|
|
│ • HTTPS gateway (Tier 0) │
|
|
│ • Peer directory │
|
|
│ • Federation bridge (JSON-LD) │
|
|
└───────────────────────────────────┘
|
|
```
|
|
|
|
### Offline Persistence
|
|
|
|
When a member's client goes offline, their content persists on IPFS. The relay provides two services:
|
|
|
|
**IPFS pinning** — members' CIDs are pinned by the cooperative's pinning node, ensuring content stays available even when the author's client is off. This is cheap — just disk storage, no compute.
|
|
|
|
**Message queuing** — activities addressed to an offline member are held by the relay and drained when they reconnect:
|
|
|
|
```scheme
|
|
;; Alice is offline. Bob sends her a message.
|
|
;; The relay holds it.
|
|
|
|
;; Alice's client comes online, connects to relay
|
|
(hello :actor "alice@rose-ash.com" :last-seen "2026-02-28T12:00:00Z")
|
|
|
|
;; Relay drains the queue
|
|
(queued :count 3 :since "2026-02-28T12:00:00Z"
|
|
(Create :actor bob :published "2026-02-28T16:00:00Z"
|
|
(Note (p "See you at the market Saturday!")))
|
|
(Like :actor charlie :object alice-post-42)
|
|
(Follow :actor dave :object alice))
|
|
|
|
;; Alice's client processes them, sends acknowledgment
|
|
(ack :through "2026-02-28T16:00:00Z")
|
|
|
|
;; Relay clears the queue. Now alice is live —
|
|
;; subsequent activities stream directly peer-to-peer.
|
|
```
|
|
|
|
### Cooperative GPU Sharing
|
|
|
|
Members contribute idle GPU cycles to the cooperative. The relay acts as a job matchmaker:
|
|
|
|
```scheme
|
|
;; Alice uploads a video. Her laptop has integrated graphics — too slow.
|
|
(submit-job
|
|
:type "artdag/render"
|
|
:recipe "bafyrecipe..."
|
|
:input "bafyinput..."
|
|
:requirements (:min-vram 8 :gpu #t))
|
|
|
|
;; Relay knows Charlie's RTX 7900 is online and idle.
|
|
;; Job routes to Charlie's client.
|
|
(job :id "job-789" :assigned-to charlie
|
|
:type "artdag/render"
|
|
:recipe "bafyrecipe..."
|
|
:input "bafyinput...")
|
|
|
|
;; Charlie's client runs the job, pins result to IPFS
|
|
(job-complete :id "job-789"
|
|
:output "bafyoutput..."
|
|
:duration-ms 4200
|
|
:worker charlie)
|
|
|
|
;; Alice gets notified
|
|
(push! (swap! "#render-status" :inner
|
|
(use "render-complete" :cid "bafyoutput...")))
|
|
```
|
|
|
|
This is already how artdag works conceptually. The L1 server is a Celery worker that picks up rendering tasks. Replace "Celery worker on a cloud server" with "Celery worker on a member's desktop" and the architecture barely changes. The task queue just has different workers.
|
|
|
|
### Economics
|
|
|
|
| | Centralised (current) | Cooperative mesh |
|
|
|---|---|---|
|
|
| Image/video processing | Cloud GPU ($2-5/hr) | Member's local GPU (free) |
|
|
| Content storage | Server disk + S3 | IPFS (distributed) + pinning |
|
|
| Content serving | Server bandwidth | Peer-to-peer + IPFS |
|
|
| Server cost | GPU instances + storage + bandwidth | Cheap relay (CPU + disk only) |
|
|
| Scaling | More users = more cost | More members = more capacity |
|
|
|
|
The co-op's infrastructure cost drops to: **one small VPS + IPFS pinning storage.** That's it. All compute — rendering, processing, serving content — is distributed across members' machines.
|
|
|
|
More members joining makes the network faster and more capable, not more expensive. Like BitTorrent seeding, but for an entire application platform.
|
|
|
|
### The Relay Server's Role
|
|
|
|
The relay is minimal — a matchmaker and persistence layer, not a compute provider:
|
|
|
|
- **Peer directory**: who's online, their QUIC address, their GPU capabilities
|
|
- **Message queue**: hold activities for offline members
|
|
- **IPFS pinning**: persist content when authors are offline
|
|
- **HTTPS gateway**: serve HTML to Tier 0 browsers (visitors, search engines)
|
|
- **Federation bridge**: translate sexp ↔ JSON-LD for Mastodon/Pleroma peers
|
|
- **Job queue**: match GPU-intensive tasks to available peers
|
|
- **Capability registry**: what each peer can do (GPU model, VRAM, storage)
|
|
|
|
The relay does no rendering, no media processing, no content generation. Its cost stays flat regardless of member count.
|
|
|
|
### Content Flow
|
|
|
|
```
|
|
Author creates post:
|
|
1. Edit in Rust client (local)
|
|
2. Render media with local GPU (artdag)
|
|
3. Pin content + media to IPFS (local node)
|
|
4. Publish CIDs to relay (for pinning + discovery)
|
|
5. Stream activity to connected followers (peer-to-peer)
|
|
6. Relay queues activity for offline followers
|
|
|
|
Reader views post:
|
|
1. Fetch sexp from author's client (if online, peer-to-peer)
|
|
2. Or fetch from IPFS by CID (if author offline)
|
|
3. Or fetch from relay gateway as HTML (if Tier 0 browser)
|
|
4. Components resolved from local cache (content-addressed)
|
|
5. Render locally (Rust GPU or sexpr.js in browser)
|
|
```
|
|
|
|
No server rendered anything. No server stored anything permanently. No server paid for GPU time. The cooperative's members are the infrastructure.
|
|
|
|
---
|
|
|
|
## Cooperative Angle
|
|
|
|
- Members install the Rust client → fast native experience, 5MB binary, no app store
|
|
- Visitors browse `https://rose-ash.com` → standard HTML, no barrier
|
|
- Federated co-ops connect via persistent sexp streams → rich UI exchange, not just text syndication
|
|
- AI agents speak the protocol natively → components as tool calls, mutations as actions
|
|
- Auto-updates via content-addressed components → no gatekeeping
|
|
- The component registry is a shared vocabulary across the cooperative network
|
|
- Members' desktops are the cloud — contributing GPU, storage, and bandwidth
|
|
- The relay server stays cheap and flat-cost regardless of growth
|
|
- The original vision of the web: everyone has a server
|
|
|
|
---
|
|
|
|
## Relationship to Other Plans
|
|
|
|
| Document | Role |
|
|
|---|---|
|
|
| `sexpr-js-runtime-plan.md` | The JS library powering Tier 1 (Phases 3-4) |
|
|
| `ghost-removal-plan.md` | Posts must be sexp before federation/client rendering adds value |
|
|
| `sexpr-ai-integration.md` | AI agents benefit from all tiers and the self-describing schema |
|
|
| artdag (`artdag/`) | The media processing engine that runs on member GPUs |
|
|
|
|
---
|
|
|
|
*The document is the program. The program is the document. The protocol is both. The network is its members.*
|