mono/docs/sexpr-unified-protocol.md

The Sexp Protocol

A unified protocol for documents, applications, and federation. One format. Every boundary.

---

Core Insight

The sexp protocol replaces HTTP, HTML, JSON-LD, WebSocket, and ActivityPub's inbox model with a single concept: peers exchange s-expressions on a bidirectional stream.

There is no distinction between:

• A client requesting a page (what HTTP does)
• A server pushing a real-time update (what WebSocket does)
• A peer delivering a federated activity (what AP inbox POST does)
• A server sending a DOM mutation (what HTMX does)

They are all sexp expressions sent between two peers:

;; Browsing (request → response)
(GET "/markets/")
(ok (page :title "Markets" ...))

;; Real-time push (server → client, unsolicited)
(push! (swap! "#feed" :prepend (use "post-card" :title "New post")))

;; Federation delivery (peer → peer)
(Create :actor "alice@rose-ash.com"
  (Note :id post-123 (p "Hello from the fediverse!")))

;; Client action (client → server)
(POST "/like/" :body (:post-id 123))

;; Mutation response (server → client)
(push! (swap! "#like-count-123" :inner "43"))

Same parser. Same stream. Same connection. The "type" of interaction is determined by the head symbol of the expression, not by the protocol layer.

---

Why Replace HTTP and JSON-LD

HTTP's Problems

HTTP is a request-response protocol with text headers and a body. Strip away the historical baggage and what you actually need is:

Peer A sends: verb + target + metadata + optional body
Peer B sends: status + metadata + body

That's just an s-expression. But HTTP adds:

• Header/body split — two parsing phases, different formats
• Rigid request-response — to work around this, we bolted on WebSocket (separate protocol, upgrade handshake), SSE (chunked encoding hack), HTTP/2 push (failed, removed), HTTP/3 QUIC streams (complex, still one-request-one-response per stream)
• Overcomplicated caching — Cache-Control, ETag, If-None-Match, If-Modified-Since, Vary, Age, Expires, Last-Modified, s-maxage, stale-while-revalidate — a dozen headers with complex interaction rules
• Arbitrary status codes — memorised numbers with no semantic meaning in the format
• Form encoding mess — application/x-www-form-urlencoded or multipart/form-data with boundary strings, MIME parts, content-disposition headers

JSON-LD's Problems

JSON-LD was chosen for ActivityPub because of its semantic web lineage, but in practice nobody uses the linked data features — servers just pattern-match on type fields and ignore the @context. The format is:

• Verbose — deeply nested objects with string keys, quoted values, commas, colons
• Ambiguous — compaction/expansion rules mean the same activity can have many valid JSON representations
• Lossy — post content is flattened to an HTML string inside a JSON string; structure destroyed
• Hostile to signing — JSON-LD canonicalization is fragile and implementation-dependent
• Hostile to AI — agents must parse JSON, then parse embedded HTML, then reason about structure

The Sexp Solution

One format that is:

• Compact — half the size of equivalent JSON-LD, no closing tags like HTML
• Unambiguous — one canonical serialization, deterministic for signing
• Structural — content is the tree, not a string embedded in a string
• Parseable — trivial recursive descent, no backtracking, nanoseconds per node
• Bidirectional — requests, responses, pushes, and activities all use the same syntax
• AI-native — agents parse, generate, and reason about sexp as naturally as tool calls

---

Protocol Specification

Transport

┌─────────────────────────────────────────┐
│  Application: sexp expressions          │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│  Session: bidirectional sexp stream     │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│  Transport: QUIC (or TCP+TLS fallback)  │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│  Network: IP                            │
└─────────────────────────────────────────┘

QUIC is ideal — multiplexed streams, built-in TLS, connection migration. Each sexp expression gets its own QUIC stream. Requests and pushes multiplex without head-of-line blocking. Connection survives network changes (mobile → wifi).

Framing: Length-prefixed sexp expressions over QUIC streams. The parser knows when an expression ends (balanced parens), but length-prefixing enables efficient buffer allocation.

URL scheme: sexpr://host:port/path

Expression Types

Every expression on the wire is one of these forms:

Requests (client → server)

(GET "/markets/"
  :auth "bearer tok_abc123"
  :have "sha3-a1b2c3"
  :have-components ("sha3-aaa" "sha3-bbb"))

(POST "/like/"
  :auth "bearer tok_abc123"
  :body (:post-id 123))

(POST "/submit/"
  :body (
    :username "alice"
    :email "alice@example.com"
    :avatar (file :name "photo.jpg" :type "image/jpeg" :data <binary>)))

;; Streaming request — keep connection open for pushes
(GET "/feed/" :stream #t)

Responses (server → client)

;; Success with content
(ok :hash "sha3-d4e5f6"
  (page :title "Markets" :layout "main"
    (section (h1 "Markets")
      (each markets (lambda (m)
        (use "vendor-card" :name (get m "name")))))))

;; Not modified (client already has current version)
(not-modified)

;; Redirect
(redirect :to "/new-location/" :permanent #t)

;; Not found
(not-found :message "Page does not exist")

;; Error
(error :code "auth-required" :message "Please log in"
  :login-url "sexpr://rose-ash.com/login/")

;; Response with new components the client is missing
(ok
  :new-components (
    (component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
  (page ...))

Pushes (server → client, unsolicited)

;; DOM mutation
(push! (swap! "#feed" :prepend
  (use "post-card" :title "New post" :author "alice")))

;; Batch mutation
(push! (batch!
  (swap! "#notifications" :append (div :class "toast" "Saved!"))
  (class! "#save-btn" :remove "loading")))

;; Component update (new version available)
(push! (component-update "vendor-card" :hash "sha3-eee"))

Activities (peer → peer, federation)

;; Create
(Create
  :id "https://rose-ash.com/ap/activities/456"
  :actor "https://rose-ash.com/ap/users/alice"
  :published "2026-02-28T14:00:00Z"
  :to (:public)
  :cc ("https://rose-ash.com/ap/users/alice/followers")

  (Note
    :id "https://rose-ash.com/ap/posts/123"
    :attributed-to "https://rose-ash.com/ap/users/alice"
    :in-reply-to "https://remote.social/posts/789"
    :summary "Weekend market update"

    (section
      (p "Three new vendors joining this Saturday.")
      (use "vendor-list" :vendors vendors)
      (use "calendar-widget" :calendar-id 42))

    (attachment
      (Image :url "https://rose-ash.com/media/market.jpg"
        :media-type "image/jpeg"
        :width 1200 :height 800
        :name "Saturday market stalls"))

    (tag
      (Hashtag :name "#markets" :href "https://rose-ash.com/tags/markets")
      (Mention :name "@bob@remote.social" :href "https://remote.social/users/bob"))))

;; Follow
(Follow :actor "https://rose-ash.com/ap/users/alice"
  :object "https://remote.social/users/bob")

;; Accept (response to Follow — on the same connection)
(Accept :actor "https://remote.social/users/bob"
  (Follow :actor "https://rose-ash.com/ap/users/alice"
    :object "https://remote.social/users/bob"))

;; Like, Announce, Undo — all just expressions
(Like :actor alice :object post-123)
(Announce :actor bob (Note :id post-123))
(Undo :actor alice (Like :actor alice :object post-123))

Key insight: Activities are not "posted to an inbox" — they are expressions sent on the bidirectional stream. The peer-to-peer connection is the inbox. When you follow someone, their activities stream to you on the same connection you used to follow them. No inbox endpoint, no HTTP POST delivery, no polling.

Collections (queryable, paginated)

;; Request an actor's outbox
(GET "/ap/users/alice/outbox" :page 1)

;; Response
(ok
  (OrderedCollectionPage
    :part-of "https://rose-ash.com/ap/users/alice/outbox"
    :next "/ap/users/alice/outbox?page=2"
    :total-items 142

    (Create :actor alice :published "2026-02-28T14:00:00Z"
      (Note :id post-3 (p "Latest post")))
    (Announce :actor alice :published "2026-02-27T09:00:00Z"
      (Note :id post-2 :attributed-to bob))
    (Create :actor alice :published "2026-02-26T18:00:00Z"
      (Note :id post-1 (p "Earlier post")))))

Actor Profiles

(Person
  :id "https://rose-ash.com/ap/users/alice"
  :preferred-username "alice"
  :name "Alice"
  :summary (p "Co-op member, market organiser")
  :inbox "sexpr://rose-ash.com/ap/users/alice/inbox"
  :outbox "sexpr://rose-ash.com/ap/users/alice/outbox"
  :followers "sexpr://rose-ash.com/ap/users/alice/followers"
  :following "sexpr://rose-ash.com/ap/users/alice/following"
  :components "sexpr://rose-ash.com/ap/components/"
  :public-key (:id "https://rose-ash.com/ap/users/alice#main-key"
               :owner "https://rose-ash.com/ap/users/alice"
               :pem "-----BEGIN PUBLIC KEY-----\n..."))

Schema Introspection

(GET "/__schema/")

(ok
  (schema
    (endpoint "/" :method GET
      :returns (page)
      :params (:stream bool))
    (endpoint "/markets/" :method GET
      :returns (page :contains (list (use "vendor-card"))))
    (endpoint "/like/" :method POST
      :params (:post-id int)
      :returns (mutation))
    (activity Create
      :object (Note)
      :delivers-to :followers)
    (activity Follow
      :object (Person)
      :expects (Accept))))

An AI agent hitting /__schema/ learns the entire surface — pages, actions, federation activities — as parseable sexp. No separate OpenAPI doc. The schema is the API.

Caching

Content-addressed, hash-based. No expiry headers, no revalidation dance.

;; Client has a cached version
(GET "/markets/" :have "sha3-a1b2c3")

;; Unchanged
(not-modified)

;; Changed — new hash included
(ok :hash "sha3-d4e5f6" (page ...))

Component-level caching:

;; Client reports which components it has
(GET "/markets/" :have-components ("sha3-aaa" "sha3-bbb"))

;; Server sends only missing components alongside the page
(ok
  :new-components (
    (component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
  (page ...))

The hash is the cache key, the ETag, and the content address. One concept replaces twelve HTTP headers.

Component Discovery

Actors advertise a :components endpoint in their profile. Peers fetch and cache component definitions by content hash:

;; Fetch component manifest
(GET "/ap/components/")

(ok
  (component-manifest
    ("cart-mini" :hash "sha3-aaa")
    ("vendor-card" :hash "sha3-bbb")
    ("calendar-widget" :hash "sha3-ccc")
    ("post-card" :hash "sha3-ddd")))

;; Fetch a specific component by hash
(GET "/ap/components/sha3-bbb")

(ok
  (component "vendor-card" :hash "sha3-bbb"
    (params name stall image)
    (div :class "vendor-card"
      (img :src image :alt name)
      (h3 name)
      (p :class "stall" stall))))

When a federated activity includes (use "vendor-card" :name "Sourdough" :stall "A12"), the receiving peer resolves the component from its cache (by hash) or fetches it from the sender's manifest. Federated UI, not just federated data.

Signatures

Sexp serialization has a canonical form:

• Keywords sorted alphabetically
• Single space between atoms
• No trailing whitespace
• UTF-8 encoding

This makes signatures deterministic without JSON-LD canonicalization:

(signed :sig "base64..." :key-id "https://rose-ash.com/ap/users/alice#main-key"
  (Create :actor "https://rose-ash.com/ap/users/alice"
    (Note :id post-123 (p "Hello"))))

Sign the canonical serialization of the inner expression. Verify by re-serializing and checking. No compaction/expansion ambiguity.

---

Bidirectional Stream: The Unified Model

A connection between two sexp-speaking peers carries everything on one stream:

;; === Connection opened ===

;; Client browses
(GET "/")
(ok (page :title "Home" ...))

;; Client navigates (same connection)
(GET "/markets/")
(ok (page :title "Markets" ...))

;; Client follows a user (federation)
(Follow :actor alice :object bob)

;; Server confirms
(Accept :actor bob (Follow :actor alice :object bob))

;; Bob posts something — server pushes activity
(Create :actor bob :published "2026-02-28T15:00:00Z"
  (Note :id post-456 (p "New vendor announcement!")))

;; Client sees it rendered in real-time via mutation
(push! (swap! "#feed" :prepend
  (use "post-card" :actor "bob" :content (p "New vendor announcement!"))))

;; Client likes the post
(POST "/like/" :body (:post-id 456))

;; Server pushes mutation + delivers Like activity to bob
(push! (swap! "#like-count-456" :inner "12"))

;; Client opens a streaming feed
(GET "/feed/" :stream #t)
(ok :stream #t (page :title "Feed" ...))

;; More activities arrive over time...
(Create :actor charlie :published "2026-02-28T15:30:00Z"
  (Note :id post-789 (p "Market day tomorrow!")))
(push! (swap! "#feed" :prepend
  (use "post-card" :actor "charlie" :content (p "Market day tomorrow!"))))

;; === Connection persists ===

Browsing, federation, real-time updates, and user actions — all on one bidirectional stream. The distinction between "web server", "AP inbox", and "WebSocket" disappears. They were always the same thing — peers exchanging structured expressions.

---

Backwards Compatibility

With HTTP (Tier 0 and Tier 1)

The sexp protocol runs alongside HTTPS. The same server handles both:

@bp.get("/markets/")
async def markets():
    data = await get_markets(g.s)
    tree = sexp('(page :title "Markets" ...)', markets=data)

    accept = request.headers.get("Accept", "")
    if "application/x-sexpr" in accept:
        return Response(serialize(tree), content_type="application/x-sexpr")

    html = render_to_html(tree)
    return Response(html, content_type="text/html")

With ActivityPub (JSON-LD peers)

For Mastodon, Pleroma, and other JSON-LD AP servers:

Peer Type	Outbound	Inbound
Standard AP (Mastodon, Pleroma)	Translate sexp → JSON-LD, include rose:sexpr extension field	Parse JSON-LD, translate to sexp internally
Sexp-aware AP (other rose-ash instances)	Native sexp on bidirectional stream	Parse sexp directly
Sexp-aware with shared components	Sexp with component references	Resolve from cache, render natively

JSON-LD bridging for non-sexp peers:

{
  "@context": ["https://www.w3.org/ns/activitystreams", "https://rose-ash.com/ns/sexpr"],
  "type": "Create",
  "object": {
    "type": "Note",
    "content": "<p>Hello world</p>",
    "rose:sexpr": "(p \"Hello world\")"
  }
}

No existing AP implementation breaks. The rose:sexpr field is ignored by servers that don't understand it.

Fallback Gateway

For browsers without extension or native client:

sexpr://rose-ash.com/markets/
  → Gateway fetches sexp from server
  → Renders to HTML
  → Serves to browser at https://rose-ash.com/markets/

---

Three Client Tiers

                    ┌─────────────────────────────┐
                    │     rose-ash server (Quart)  │
                    │                              │
                    │  Same sexp component tree    │
                    │  Same data, same logic       │
                    │                              │
                    ├──────────┬──────────┬────────┤
                    │ HTTPS    │ HTTPS    │ SEXPR  │
                    │ HTML out │ sexp out │ native │
                    └────┬─────┴────┬─────┴───┬────┘
                         │          │         │
              ┌──────────▼──┐ ┌─────▼──────┐ ┌▼──────────────┐
              │   Browser   │ │  Browser + │ │  Rust client   │
              │   (vanilla) │ │  extension │ │  (native)      │
              │             │ │            │ │                │
              │ HTML + HTMX │ │ sexpr.js   │ │ sexp protocol  │
              │ Full CSS    │ │ over HTTPS │ │ over QUIC      │
              │ ~200ms load │ │ ~80ms load │ │ ~20ms load     │
              └─────────────┘ └────────────┘ └────────────────┘
                 Tier 0           Tier 1          Tier 2

	Tier 0: Browser	Tier 1: Extension	Tier 2: Rust Client
URL	https://rose-ash.com	https://rose-ash.com	sexpr://rose-ash.com
Protocol	HTTPS	HTTPS	sexpr:// over QUIC
Wire format	HTML	sexp over HTTP	sexp native stream
Rendering	Browser DOM	sexpr.js → DOM	Rust → GPU
Component cache	Browser cache (URL-keyed)	IndexedDB (hash-keyed)	Disk (hash-keyed, pre-parsed AST)
Real-time	HTMX polling / SSE	WebSocket sexp mutations	Native bidirectional stream
Federation	N/A	AP via fetch	Native sexp stream
Bundle size	HTMX 14KB + CSS	sexpr.js ~8KB	5MB binary, zero runtime deps
Page load	~200ms	~80ms	~20ms
Memory per page	~200MB	~150MB	~20MB
AI integration	Parse HTML (painful)	Parse sexp (easy)	Native sexp (trivial)

Rust Client Architecture

┌─────────────────────────────────────┐
│        sexpr-client (Rust)          │
│                                     │
│  ┌──────────┐  ┌─────────────────┐  │
│  │  Parser   │  │  Component      │  │
│  │  (zero-   │  │  Cache          │  │
│  │   copy)   │  │  (SHA3 → AST)  │  │
│  └──────────┘  └─────────────────┘  │
│  ┌──────────┐  ┌─────────────────┐  │
│  │  Layout   │  │  Network        │  │
│  │  Engine   │  │  (tokio + QUIC) │  │
│  └──────────┘  └─────────────────┘  │
│  ┌──────────────────────────────┐   │
│  │  Renderer (wgpu / iced)      │   │
│  └──────────────────────────────┘   │
└─────────────────────────────────────┘

Why it's fast:

• Parser: nanoseconds per node (trivial recursive descent in Rust)
• No HTML parser, no CSS cascade, no DOM construction, no JS engine
• Render directly to GPU surface — skip the entire browser rendering pipeline
• Pre-parsed ASTs from disk cache — zero parse time on cache hit
• 10-50MB memory vs 200-500MB per browser tab

Page load: network (~50ms) → parse (~0.1ms) → layout (~2ms) → paint (~3ms) = under 60ms.

---

Implementation Plan

Phase 1: Content Negotiation (Quart)

Add Accept header handling to existing Quart routes.

• Check for application/x-sexpr in Accept header
• If present: serialize sexp tree, return directly
• If absent: render to HTML as today
• Add Vary: Accept header

Files:

• shared/infrastructure/factory.py — content negotiation middleware
• shared/sexp/serialize.py — canonical sexp serializer (deterministic for signing/caching)

Result: Existing routes serve sexp to any client that asks. Zero changes to route logic.

Phase 2: Sexp ↔ JSON-LD Bridge

Bidirectional translation for AP federation with non-sexp peers:

sexp → JSON-LD (outbound):

• Activity type symbol → "type" field
• Keyword attributes → JSON object properties
• Content sexp → rendered to HTML for "content" field
• Include rose:sexpr extension field

JSON-LD → sexp (inbound):

• "type" → head symbol
• "content" HTML → parsed to sexp (best-effort)
• Nested objects → child expressions

Files:

• shared/infrastructure/ap_sexpr.py — serializer/deserializer
• shared/sexp/html_to_sexp.py — HTML → sexp parser for inbound content
• shared/sexp/tests/test_ap_sexpr.py — round-trip tests

Phase 3: sexpr.js Client Library

Browser-side JS runtime (see sexpr-js-runtime-plan.md):

• Parser + renderer: parse() → AST → renderToDOM()
• Mutation engine: swap!, batch!, class!, request!
• Component registry with localStorage cache (content-addressed)
• WebSocket connection for real-time pushes

Phase 4: Browser Extension

Package sexpr.js as a WebExtension:

• Register sexpr:// protocol handler
• Intercept navigation → fetch via HTTPS with Accept: application/x-sexpr
• Parse → render to DOM
• Component cache in IndexedDB (content-addressed)

Tech: WebExtension API (Firefox + Chrome), JS/TS.

Phase 5: Component Discovery

Enable peers to discover and cache shared components:

• Actor profiles include :components endpoint URL
• GET /ap/components/ → manifest (name → hash)
• GET /ap/components/{hash} → component definition (sexp)
• Content-addressed cache on client (hash-keyed)
• Federated content with (use "component" ...) resolves from cache

Files:

• federation/bp/components/routes.py — manifest + fetch endpoints
• shared/sexp/component_manifest.py — hash generation

Phase 6: Protocol Specification

Formal specification document:

• Framing: length-prefixed sexp over QUIC streams
• Expression types: requests, responses, pushes, activities, collections
• Canonical serialization: deterministic rules for signing
• Caching: content-hash based
• Authentication: token in request keywords
• Component discovery: manifest protocol
• Bidirectional streaming: lifecycle, multiplexing
• Schema introspection: /__schema/ endpoint
• JSON-LD bridging: rules for non-sexp AP peers
• Security: no eval, escaping, signature verification

Publish as a FEP (Fediverse Enhancement Proposal) and standalone specification.

Phase 7: Rust Protocol Server

QUIC server alongside Hypercorn:

• Listen on separate port (e.g., 4433)
• Parse sexp requests, route to same handler logic
• Bidirectional stream: pushes, requests, responses, activities
• Component manifest endpoint
• Federation delivery on persistent connections (replaces HTTP POST to inbox)

Tech: Rust, quinn (QUIC), tokio, rustls.

Files:

• sexpr-server/src/protocol.rs — framing, parsing, routing
• sexpr-server/src/quic.rs — QUIC listener + stream management
• sexpr-server/src/federation.rs — peer connection manager

Phase 8: Rust Native Client

Standalone sexp document viewer:

• QUIC client for sexpr:// URLs
• Zero-copy sexp parser (arena-allocated AST)
• Component cache on disk (SHA3-keyed)
• Layout engine (flexbox subset)
• GPU renderer via wgpu or iced
• Text rendering via cosmic-text
• Bidirectional stream: browsing + federation + real-time on one connection

Tech: Rust, quinn, wgpu/iced, cosmic-text, tokio.

Files:

• sexpr-client/src/parser.rs — zero-copy parser
• sexpr-client/src/cache.rs — content-addressed component cache
• sexpr-client/src/layout.rs — layout engine
• sexpr-client/src/render.rs — GPU renderer
• sexpr-client/src/stream.rs — bidirectional connection manager

Phase 9: Fallback Gateway

HTTP proxy for browsers without extension/client:

• Accept HTTPS requests at https://rose-ash.com
• Fetch sexp from server internally
• Render to HTML, serve to browser
• Add <link rel="alternate" type="application/x-sexpr" href="sexpr://..."> for discovery

---

Migration Path

Each phase builds on the last. No phase breaks existing functionality:

Phase 1 (content negotiation) ── Tier 0 unchanged, sexp available
Phase 2 (JSON-LD bridge)      ── federation works with all AP peers
Phase 3 (sexpr.js)            ── Tier 1 client-side rendering
Phase 4 (extension)           ── sexpr:// URLs in browser
Phase 5 (components)          ── federated UI exchange
Phase 6 (spec)                ── formal protocol document
Phase 7 (Rust server)         ── native protocol alongside HTTPS
Phase 8 (Rust client)         ── Tier 2 native experience
Phase 9 (gateway)             ── sexpr:// accessible from any browser

Depends on:

• Ghost removal (see ghost-removal-plan.md) — posts must be sexp before Phases 2-3 add real value
• sexpr.js runtime (see sexpr-js-runtime-plan.md) — the JS library that powers Phases 3-4

---

Cooperative Angle

• Members install the Rust client → fast native experience, 5MB binary, no app store
• Visitors browse https://rose-ash.com → standard HTML, no barrier
• Federated co-ops connect via persistent sexp streams → rich UI exchange, not just text syndication
• AI agents speak the protocol natively → components as tool calls, mutations as actions
• Auto-updates via content-addressed components → no gatekeeping
• The component registry is a shared vocabulary across the cooperative network

---

Relationship to Other Plans

Document	Role
sexpr-js-runtime-plan.md	The JS library powering Tier 1 (Phases 3-4)
ghost-removal-plan.md	Posts must be sexp before federation/client rendering adds value
sexpr-ai-integration.md	AI agents benefit from all tiers and the self-describing schema

---

The document is the program. The program is the document. The protocol is both.