# Sexp Protocol and Tiered Client Architecture

**One server, three clients, progressive enhancement from HTML to native.**

---

## Overview

The same rose-ash application serves three tiers of client from the same route handlers and component trees:

```
                    ┌─────────────────────────────┐
                    │     rose-ash server (Quart)  │
                    │                              │
                    │  Same sexp component tree    │
                    │  Same data, same logic       │
                    │                              │
                    ├──────────┬──────────┬────────┤
                    │ HTTPS    │ HTTPS    │ SEXPR  │
                    │ HTML out │ sexp out │ native │
                    └────┬─────┴────┬─────┴───┬────┘
                         │          │         │
              ┌──────────▼──┐ ┌─────▼──────┐ ┌▼──────────────┐
              │   Browser   │ │  Browser + │ │  Rust client   │
              │   (vanilla) │ │  extension │ │  (native)      │
              │             │ │            │ │                │
              │ HTML + HTMX │ │ sexpr.js   │ │ sexp protocol  │
              │ Full CSS    │ │ over HTTPS │ │ over QUIC      │
              │ ~200ms load │ │ ~80ms load │ │ ~20ms load     │
              └─────────────┘ └────────────┘ └────────────────┘
                 Tier 0           Tier 1          Tier 2
```

---

## Content Negotiation

The server does format selection on a single endpoint:

```python
@bp.get("/markets/")
async def markets():
    # Same data, same component tree, always
    data = await get_markets(g.s)
    tree = sexp('(page :title "Markets" (each markets ...))', markets=data)

    accept = request.headers.get("Accept", "")

    if "application/x-sexpr" in accept:
        # Tier 1 (extension) or Tier 2 (native): raw sexp
        return Response(serialize(tree), content_type="application/x-sexpr")

    # Tier 0: render to HTML for vanilla browsers
    html = render_to_html(tree)
    return Response(html, content_type="text/html")
```

One route. One component tree. The output format is the only thing that changes.

---

## Tier Comparison

| | Tier 0: Browser | Tier 1: Extension | Tier 2: Rust Client |
|---|---|---|---|
| URL | `https://rose-ash.com` | `https://rose-ash.com` | `sexpr://rose-ash.com` |
| Protocol | HTTPS | HTTPS | sexpr:// over QUIC |
| Wire format | HTML | sexp over HTTP | sexp native |
| Rendering | Browser DOM | sexpr.js → DOM | Rust → GPU |
| Component cache | Browser cache (URL-keyed) | IndexedDB (hash-keyed) | Disk (hash-keyed, pre-parsed AST) |
| Real-time | HTMX polling / SSE | WebSocket sexp mutations | Native bidirectional stream |
| Bundle size | HTMX 14KB + CSS | sexpr.js ~8KB | 5MB binary, zero runtime deps |
| Page load | ~200ms | ~80ms | ~20ms |
| Memory | ~200MB per tab | ~150MB per tab | ~20MB |
| Offline | Service worker (if built) | Component cache + manifest | Full local cache |
| AI agent | Parse HTML (painful) | Parse sexp (easy) | Native sexp (trivial) |
| Federation | N/A | AP via fetch | Native AP sexp stream |

---

## The Sexp Protocol (Tier 2)

### Why Not Just HTTP

HTTP is a request-response protocol with text headers and a body. Strip away the historical baggage and what you actually need is:

```
Client sends: method + path + metadata + optional body
Server sends: status + metadata + body
```

That's just an s-expression:

```scheme
;; Request
(GET "/markets/"
  :accept "application/x-sexpr"
  :auth "bearer tok_abc123"
  :if-none-match "sha3-a1b2c3")

;; Response
(200
  :content-hash "sha3-d4e5f6"
  :cache :immutable
  :components ("sha3-aaa" "sha3-bbb")

  (page :title "Markets" :layout "main"
    (section (h1 "Markets")
      (each markets (lambda (m)
        (use "vendor-card" :name (get m "name")))))))
```

No header/body split. No chunked transfer encoding. No `Content-Length` because the parser knows when the expression ends (balanced parens). No `Content-Type` because everything is sexp. No MIME, no multipart, no `Transfer-Encoding`, no `Connection: keep-alive` negotiation.

### What HTTP Gets Wrong That Sexp Fixes

#### 1. Headers are a bad key-value format

HTTP headers are case-insensitive, can be duplicated, have weird continuation rules, and are parsed separately from the body. In sexp, metadata is just keywords in the expression — same parser, same format, no special case.

#### 2. Request/response is too rigid

HTTP is strictly one request → one response. To get around this, we've bolted on WebSocket (separate protocol, upgrade handshake), Server-Sent Events (hack using chunked encoding), HTTP/2 server push (failed, being removed), and HTTP/3 QUIC streams (complex, still one-request-one-response per stream).

A sexp protocol is **bidirectional from the start**:

```scheme
;; Client opens connection, sends request
(GET "/feed/" :stream #t)

;; Server responds with initial content
(200 :stream #t
  (page :title "Feed"
    (div :id "feed" (p "Loading..."))))

;; Server pushes updates as they happen (same connection)
(push! (swap! "#feed" :prepend
  (use "post-card" :title "New post" :author "alice")))

;; Client sends an action (same connection)
(POST "/like/" :body (:post-id 123))

;; Server responds with mutation
(push! (swap! "#like-count-123" :inner "43"))

;; Client navigates (same connection, no new handshake)
(GET "/markets/")

;; Server responds
(200 (page :title "Markets" ...))
```

One persistent connection. Requests, responses, and pushes are all sexp expressions on the same stream. No protocol upgrade, no separate WebSocket connection, no polling.

#### 3. Caching is overcomplicated

HTTP caching: `Cache-Control`, `ETag`, `If-None-Match`, `If-Modified-Since`, `Vary`, `Age`, `Expires`, `Last-Modified`, `s-maxage`, `stale-while-revalidate`... a dozen headers with complex interaction rules.

With content-addressed sexp:

```scheme
;; Request with known hash
(GET "/markets/" :have "sha3-a1b2c3")

;; If unchanged:
(304)

;; If changed:
(200 :hash "sha3-d4e5f6" (page ...))
```

One field. The hash *is* the cache key, the ETag, and the content address.

Components take this further — the client sends what it already has:

```scheme
(GET "/markets/"
  :have-components ("sha3-aaa" "sha3-bbb" "sha3-ccc"))

;; Server only sends components the client is missing
(200
  :new-components (
    (component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
  (page ...))
```

#### 4. Status codes are arbitrary numbers

```scheme
(ok (page ...))
(redirect :to "/new-location/" :permanent #t)
(not-found :message "Page does not exist")
(error :code "auth-required" :message "Please log in"
  :login-url "sexpr://rose-ash.com/login/")
```

The status *is* the expression. Machines pattern-match on the head symbol. Humans read it. AI agents understand it without a lookup table.

#### 5. Forms and file uploads are a mess

HTTP: `application/x-www-form-urlencoded` or `multipart/form-data` with boundary strings, MIME parts, content-disposition headers.

Sexp:

```scheme
(POST "/submit/"
  :body (
    :username "alice"
    :email "alice@example.com"
    :avatar (file :name "photo.jpg" :type "image/jpeg" :data <binary>)))
```

Structured data with inline binary. One format.

#### 6. The protocol is self-describing

HTTP has no introspection. You need OpenAPI/Swagger specs bolted on separately.

```scheme
(GET "/__schema/")

;; Response: the API describes itself
(schema
  (endpoint "/" :method GET
    :returns (page)
    :params (:stream bool))
  (endpoint "/markets/" :method GET
    :returns (page :contains (list (use "vendor-card"))))
  (endpoint "/like/" :method POST
    :params (:post-id int)
    :returns (mutation)))
```

An AI agent hitting `/__schema/` learns the entire API surface as parseable sexp. No separate OpenAPI doc. The schema *is* the API.

### Protocol Stack

```
┌─────────────────────────────────────────┐
│  Application: sexp documents, mutations │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│  Session: bidirectional sexp stream     │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│  Transport: QUIC (or TCP+TLS)           │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│  Network: IP                            │
└─────────────────────────────────────────┘
```

QUIC is ideal — multiplexed streams, built-in TLS, connection migration. Each sexp expression gets its own QUIC stream. Requests and pushes multiplex without head-of-line blocking. Connection survives network changes (mobile → wifi).

---

## Rust Client Architecture

```
┌─────────────────────────────────────┐
│        sexpr-client (Rust)          │
│                                     │
│  ┌──────────┐  ┌─────────────────┐  │
│  │  Parser   │  │  Component      │  │
│  │  (zero-   │  │  Cache          │  │
│  │   copy)   │  │  (SHA3 → AST)  │  │
│  └──────────┘  └─────────────────┘  │
│  ┌──────────┐  ┌─────────────────┐  │
│  │  Layout   │  │  Network        │  │
│  │  Engine   │  │  (tokio + QUIC) │  │
│  └──────────┘  └─────────────────┘  │
│  ┌──────────────────────────────┐   │
│  │  Renderer (wgpu / iced)      │   │
│  └──────────────────────────────┘   │
└─────────────────────────────────────┘
```

**Why Rust makes this fast:**

- **Parser**: sexp parsing is trivial recursive descent. Nanoseconds per node. 100-1000x faster than JS.
- **Renderer**: Skip the DOM entirely. Render directly to GPU-backed surface via `wgpu` or `iced`.
- **Networking**: `reqwest` + `tokio` with connection pooling. Component manifests fetched in parallel.
- **Component cache**: Pre-parsed ASTs on disk, content-addressed. No parse step on cache hit.
- **Memory**: No GC pauses. 10-50MB where a browser tab uses 200-500MB.

**What you skip by not being a browser:**
- No HTML parser (5-10ms per page)
- No CSS cascade resolution (the most expensive part of browser rendering)
- No DOM construction (sexp AST *is* the tree)
- No JavaScript engine (logic in the sexp evaluator, compiled Rust)
- No security sandbox overhead (no arbitrary JS execution)
- No 2000+ web platform APIs you don't use

Page load: network (~50ms) → parse (~0.1ms) → layout (~2ms) → paint (~3ms) = **under 60ms**.

---

## Fallback Gateway

For users without the extension or native client, a gateway translates:

```
sexpr://rose-ash.com/markets/
  → Gateway fetches sexp from server
  → Renders to HTML
  → Serves to browser at https://gateway.rose-ash.com/markets/
```

The regular web always works. The sexp protocol is an acceleration layer, not a requirement.

---

## Implementation Plan

### Phase 1: Content Negotiation (Quart)

Add `Accept` header handling to existing Quart routes.

- Check for `application/x-sexpr` in `Accept` header
- If present: serialize the sexp tree and return directly (skip HTML render)
- If absent: render to HTML as today
- Add `Vary: Accept` header to responses

**Files:**
- `shared/infrastructure/factory.py` — middleware for content negotiation
- `shared/sexp/serialize.py` — canonical sexp serializer (deterministic output for caching/signing)

**Result:** Existing routes serve sexp to any client that asks. Zero changes to route logic.

### Phase 2: sexpr.js Client Library

Build the browser-side JS runtime (see `sexpr-js-runtime-plan.md` for full details).

- Parser + renderer: `parse()` → AST → `renderToDOM()`
- Mutation engine: `swap!`, `batch!`, `class!`, `request!`
- Component registry with localStorage cache (content-addressed)
- `<script src="sexpr.js">` drop-in for any page

**Result:** Any page can opt into client-side sexp rendering.

### Phase 3: Browser Extension

Package sexpr.js as a WebExtension that intercepts `sexpr://` URLs.

- Register `sexpr://` protocol handler
- Intercept navigation → fetch via HTTPS with `Accept: application/x-sexpr`
- Parse response → render to DOM in current tab
- Component cache in extension storage (IndexedDB, content-addressed)
- Address bar shows `sexpr://rose-ash.com/markets/`

**Tech:** WebExtension API (Firefox + Chrome), JS/TS.

**Result:** Users install extension, navigate to `sexpr://` URLs, get sexp-rendered pages.

### Phase 4: Sexp Protocol Specification

Define the native protocol:

- **Framing**: length-prefixed sexp expressions over QUIC streams
- **Request format**: `(METHOD path :keyword value ... body)`
- **Response format**: `(status :keyword value ... body)`
- **Push format**: `(push! mutation)` — server-initiated, any time
- **Stream lifecycle**: open on first request, multiplex subsequent requests
- **Caching**: content-hash based (`:have "sha3-..."`, `:have-components (...)`)
- **Authentication**: token in request keywords (`:auth "bearer ..."`)
- **Canonical serialization**: deterministic output rules for signing
- **Schema introspection**: `GET /__schema/` returns API description as sexp

Publish as a specification document.

### Phase 5: Rust Protocol Server

Build a QUIC server that speaks the sexp protocol alongside Hypercorn:

- Listen on a separate port (e.g., 4433 for QUIC)
- Parse sexp requests, route to the same handler logic as Quart
- Bidirectional stream: pushes, requests, responses on one connection
- Component manifest endpoint: serve component hashes + definitions
- Connection pooling, TLS via rustls

**Tech:** Rust, `quinn` (QUIC), `tokio`, sexp parser crate.

**Architecture option:** The Rust server could proxy to Quart for data/logic and handle only the protocol layer. Or it could run the full application logic natively (long-term, after porting route handlers to Rust).

**Files:**
- `sexpr-server/` — new Rust crate alongside the Python services
- `sexpr-server/src/protocol.rs` — framing, parsing, routing
- `sexpr-server/src/quic.rs` — QUIC listener + stream management

### Phase 6: Rust Native Client

Build the standalone sexp document viewer:

- QUIC client connecting to sexpr:// servers
- Sexp parser (zero-copy, arena-allocated AST)
- Component cache on disk (SQLite or filesystem, SHA3-keyed)
- Layout engine (flexbox subset — enough for document layout)
- GPU renderer via `wgpu` or `iced`
- Text rendering via `cosmic-text` or `fontdue`
- Input handling: keyboard, mouse, touch
- Bidirectional stream: real-time mutations, navigation without reconnection

**Tech:** Rust, `quinn`, `wgpu`/`iced`, `cosmic-text`, `tokio`.

**Files:**
- `sexpr-client/` — new Rust crate
- `sexpr-client/src/parser.rs` — zero-copy sexp parser
- `sexpr-client/src/cache.rs` — content-addressed component cache
- `sexpr-client/src/layout.rs` — layout engine
- `sexpr-client/src/render.rs` — GPU renderer

### Phase 7: Fallback Gateway

HTTP proxy that translates for browsers without extension/client:

- Accepts regular HTTPS requests
- Fetches sexp from the server (internal)
- Renders to HTML
- Serves to browser
- Adds `<link rel="alternate" type="application/x-sexpr" href="sexpr://...">` for discovery

**Result:** `https://rose-ash.com` always works. `sexpr://rose-ash.com` is an acceleration layer.

---

## Migration Path

Each tier builds on the last. No tier breaks the others:

1. **Today**: Tier 0 works. Ship HTML. Done.
2. **Add `Accept` header check** (Phase 1): Same routes now serve sexp to clients that ask. Tier 0 unchanged.
3. **Build sexpr.js** (Phase 2): Browser extension or `<script>` tag. Tier 1 works.
4. **Build extension** (Phase 3): `sexpr://` URLs work in browser. Tier 1 complete.
5. **Build Rust server** (Phase 5): Native protocol alongside HTTPS. Tier 2 infrastructure ready.
6. **Build Rust client** (Phase 6): `sexpr://` URLs work natively. Tier 2 complete.

A user with the Rust client visits the same URL as someone with Firefox. The server serves both from the same handler. If the Rust client is offline, the user opens `https://rose-ash.com` in a browser and gets the HTML version. Same content, same components, same data.

---

## Cooperative Angle

- Members who install the Rust client get the fast native experience
- Visitors browsing the public site get standard HTML — no barrier to entry
- Federated peers negotiate the best format they support
- AI agents get structured sexp via any tier
- The 5MB Rust binary replaces a 500MB browser for accessing the cooperative platform
- Auto-updates via content-addressed components — no app store gatekeeping

---

## Relationship to Other Plans

- **sexpr-js-runtime-plan.md** — Phase 2 of this plan; the JS library that powers Tier 1
- **ghost-removal-plan.md** — posts must be sexp before content negotiation adds value
- **sexpr-ai-integration.md** — AI agents benefit from all three tiers
- **sexpr-activitypub-extension.md** — federation over the native protocol (Tier 2 peers speak AP natively)