Add sexp protocol spec and tiered client architecture plan
Defines three client tiers (browser HTML, browser extension with sexpr.js, Rust native client) served from the same route handlers via content negotiation. Includes native sexp:// protocol design over QUIC, content-addressed caching, bidirectional streaming, self-describing schema, and implementation plan from Phase 1 (Quart content negotiation) through Phase 7 (fallback gateway). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
454
docs/sexpr-protocol-and-tiered-clients.md
Normal file
454
docs/sexpr-protocol-and-tiered-clients.md
Normal file
@@ -0,0 +1,454 @@
|
||||
# Sexp Protocol and Tiered Client Architecture
|
||||
|
||||
**One server, three clients, progressive enhancement from HTML to native.**
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The same rose-ash application serves three tiers of client from the same route handlers and component trees:
|
||||
|
||||
```
|
||||
┌─────────────────────────────┐
|
||||
│ rose-ash server (Quart) │
|
||||
│ │
|
||||
│ Same sexp component tree │
|
||||
│ Same data, same logic │
|
||||
│ │
|
||||
├──────────┬──────────┬────────┤
|
||||
│ HTTPS │ HTTPS │ SEXPR │
|
||||
│ HTML out │ sexp out │ native │
|
||||
└────┬─────┴────┬─────┴───┬────┘
|
||||
│ │ │
|
||||
┌──────────▼──┐ ┌─────▼──────┐ ┌▼──────────────┐
|
||||
│ Browser │ │ Browser + │ │ Rust client │
|
||||
│ (vanilla) │ │ extension │ │ (native) │
|
||||
│ │ │ │ │ │
|
||||
│ HTML + HTMX │ │ sexpr.js │ │ sexp protocol │
|
||||
│ Full CSS │ │ over HTTPS │ │ over QUIC │
|
||||
│ ~200ms load │ │ ~80ms load │ │ ~20ms load │
|
||||
└─────────────┘ └────────────┘ └────────────────┘
|
||||
Tier 0 Tier 1 Tier 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Content Negotiation
|
||||
|
||||
The server does format selection on a single endpoint:
|
||||
|
||||
```python
|
||||
@bp.get("/markets/")
|
||||
async def markets():
|
||||
# Same data, same component tree, always
|
||||
data = await get_markets(g.s)
|
||||
tree = sexp('(page :title "Markets" (each markets ...))', markets=data)
|
||||
|
||||
accept = request.headers.get("Accept", "")
|
||||
|
||||
if "application/x-sexpr" in accept:
|
||||
# Tier 1 (extension) or Tier 2 (native): raw sexp
|
||||
return Response(serialize(tree), content_type="application/x-sexpr")
|
||||
|
||||
# Tier 0: render to HTML for vanilla browsers
|
||||
html = render_to_html(tree)
|
||||
return Response(html, content_type="text/html")
|
||||
```
|
||||
|
||||
One route. One component tree. The output format is the only thing that changes.
|
||||
|
||||
---
|
||||
|
||||
## Tier Comparison
|
||||
|
||||
| | Tier 0: Browser | Tier 1: Extension | Tier 2: Rust Client |
|
||||
|---|---|---|---|
|
||||
| URL | `https://rose-ash.com` | `https://rose-ash.com` | `sexpr://rose-ash.com` |
|
||||
| Protocol | HTTPS | HTTPS | sexpr:// over QUIC |
|
||||
| Wire format | HTML | sexp over HTTP | sexp native |
|
||||
| Rendering | Browser DOM | sexpr.js → DOM | Rust → GPU |
|
||||
| Component cache | Browser cache (URL-keyed) | IndexedDB (hash-keyed) | Disk (hash-keyed, pre-parsed AST) |
|
||||
| Real-time | HTMX polling / SSE | WebSocket sexp mutations | Native bidirectional stream |
|
||||
| Bundle size | HTMX 14KB + CSS | sexpr.js ~8KB | 5MB binary, zero runtime deps |
|
||||
| Page load | ~200ms | ~80ms | ~20ms |
|
||||
| Memory | ~200MB per tab | ~150MB per tab | ~20MB |
|
||||
| Offline | Service worker (if built) | Component cache + manifest | Full local cache |
|
||||
| AI agent | Parse HTML (painful) | Parse sexp (easy) | Native sexp (trivial) |
|
||||
| Federation | N/A | AP via fetch | Native AP sexp stream |
|
||||
|
||||
---
|
||||
|
||||
## The Sexp Protocol (Tier 2)
|
||||
|
||||
### Why Not Just HTTP
|
||||
|
||||
HTTP is a request-response protocol with text headers and a body. Strip away the historical baggage and what you actually need is:
|
||||
|
||||
```
|
||||
Client sends: method + path + metadata + optional body
|
||||
Server sends: status + metadata + body
|
||||
```
|
||||
|
||||
That's just an s-expression:
|
||||
|
||||
```scheme
|
||||
;; Request
|
||||
(GET "/markets/"
|
||||
:accept "application/x-sexpr"
|
||||
:auth "bearer tok_abc123"
|
||||
:if-none-match "sha3-a1b2c3")
|
||||
|
||||
;; Response
|
||||
(200
|
||||
:content-hash "sha3-d4e5f6"
|
||||
:cache :immutable
|
||||
:components ("sha3-aaa" "sha3-bbb")
|
||||
|
||||
(page :title "Markets" :layout "main"
|
||||
(section (h1 "Markets")
|
||||
(each markets (lambda (m)
|
||||
(use "vendor-card" :name (get m "name")))))))
|
||||
```
|
||||
|
||||
No header/body split. No chunked transfer encoding. No `Content-Length` because the parser knows when the expression ends (balanced parens). No `Content-Type` because everything is sexp. No MIME, no multipart, no `Transfer-Encoding`, no `Connection: keep-alive` negotiation.
|
||||
|
||||
### What HTTP Gets Wrong That Sexp Fixes
|
||||
|
||||
#### 1. Headers are a bad key-value format
|
||||
|
||||
HTTP headers are case-insensitive, can be duplicated, have weird continuation rules, and are parsed separately from the body. In sexp, metadata is just keywords in the expression — same parser, same format, no special case.
|
||||
|
||||
#### 2. Request/response is too rigid
|
||||
|
||||
HTTP is strictly one request → one response. To get around this, we've bolted on WebSocket (separate protocol, upgrade handshake), Server-Sent Events (hack using chunked encoding), HTTP/2 server push (failed, being removed), and HTTP/3 QUIC streams (complex, still one-request-one-response per stream).
|
||||
|
||||
A sexp protocol is **bidirectional from the start**:
|
||||
|
||||
```scheme
|
||||
;; Client opens connection, sends request
|
||||
(GET "/feed/" :stream #t)
|
||||
|
||||
;; Server responds with initial content
|
||||
(200 :stream #t
|
||||
(page :title "Feed"
|
||||
(div :id "feed" (p "Loading..."))))
|
||||
|
||||
;; Server pushes updates as they happen (same connection)
|
||||
(push! (swap! "#feed" :prepend
|
||||
(use "post-card" :title "New post" :author "alice")))
|
||||
|
||||
;; Client sends an action (same connection)
|
||||
(POST "/like/" :body (:post-id 123))
|
||||
|
||||
;; Server responds with mutation
|
||||
(push! (swap! "#like-count-123" :inner "43"))
|
||||
|
||||
;; Client navigates (same connection, no new handshake)
|
||||
(GET "/markets/")
|
||||
|
||||
;; Server responds
|
||||
(200 (page :title "Markets" ...))
|
||||
```
|
||||
|
||||
One persistent connection. Requests, responses, and pushes are all sexp expressions on the same stream. No protocol upgrade, no separate WebSocket connection, no polling.
|
||||
|
||||
#### 3. Caching is overcomplicated
|
||||
|
||||
HTTP caching: `Cache-Control`, `ETag`, `If-None-Match`, `If-Modified-Since`, `Vary`, `Age`, `Expires`, `Last-Modified`, `s-maxage`, `stale-while-revalidate`... a dozen headers with complex interaction rules.
|
||||
|
||||
With content-addressed sexp:
|
||||
|
||||
```scheme
|
||||
;; Request with known hash
|
||||
(GET "/markets/" :have "sha3-a1b2c3")
|
||||
|
||||
;; If unchanged:
|
||||
(304)
|
||||
|
||||
;; If changed:
|
||||
(200 :hash "sha3-d4e5f6" (page ...))
|
||||
```
|
||||
|
||||
One field. The hash *is* the cache key, the ETag, and the content address.
|
||||
|
||||
Components take this further — the client sends what it already has:
|
||||
|
||||
```scheme
|
||||
(GET "/markets/"
|
||||
:have-components ("sha3-aaa" "sha3-bbb" "sha3-ccc"))
|
||||
|
||||
;; Server only sends components the client is missing
|
||||
(200
|
||||
:new-components (
|
||||
(component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
|
||||
(page ...))
|
||||
```
|
||||
|
||||
#### 4. Status codes are arbitrary numbers
|
||||
|
||||
```scheme
|
||||
(ok (page ...))
|
||||
(redirect :to "/new-location/" :permanent #t)
|
||||
(not-found :message "Page does not exist")
|
||||
(error :code "auth-required" :message "Please log in"
|
||||
:login-url "sexpr://rose-ash.com/login/")
|
||||
```
|
||||
|
||||
The status *is* the expression. Machines pattern-match on the head symbol. Humans read it. AI agents understand it without a lookup table.
|
||||
|
||||
#### 5. Forms and file uploads are a mess
|
||||
|
||||
HTTP: `application/x-www-form-urlencoded` or `multipart/form-data` with boundary strings, MIME parts, content-disposition headers.
|
||||
|
||||
Sexp:
|
||||
|
||||
```scheme
|
||||
(POST "/submit/"
|
||||
:body (
|
||||
:username "alice"
|
||||
:email "alice@example.com"
|
||||
:avatar (file :name "photo.jpg" :type "image/jpeg" :data <binary>)))
|
||||
```
|
||||
|
||||
Structured data with inline binary. One format.
|
||||
|
||||
#### 6. The protocol is self-describing
|
||||
|
||||
HTTP has no introspection. You need OpenAPI/Swagger specs bolted on separately.
|
||||
|
||||
```scheme
|
||||
(GET "/__schema/")
|
||||
|
||||
;; Response: the API describes itself
|
||||
(schema
|
||||
(endpoint "/" :method GET
|
||||
:returns (page)
|
||||
:params (:stream bool))
|
||||
(endpoint "/markets/" :method GET
|
||||
:returns (page :contains (list (use "vendor-card"))))
|
||||
(endpoint "/like/" :method POST
|
||||
:params (:post-id int)
|
||||
:returns (mutation)))
|
||||
```
|
||||
|
||||
An AI agent hitting `/__schema/` learns the entire API surface as parseable sexp. No separate OpenAPI doc. The schema *is* the API.
|
||||
|
||||
### Protocol Stack
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Application: sexp documents, mutations │
|
||||
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
|
||||
│ Session: bidirectional sexp stream │
|
||||
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
|
||||
│ Transport: QUIC (or TCP+TLS) │
|
||||
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
|
||||
│ Network: IP │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
QUIC is ideal — multiplexed streams, built-in TLS, connection migration. Each sexp expression gets its own QUIC stream. Requests and pushes multiplex without head-of-line blocking. Connection survives network changes (mobile → wifi).
|
||||
|
||||
---
|
||||
|
||||
## Rust Client Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ sexpr-client (Rust) │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌─────────────────┐ │
|
||||
│ │ Parser │ │ Component │ │
|
||||
│ │ (zero- │ │ Cache │ │
|
||||
│ │ copy) │ │ (SHA3 → AST) │ │
|
||||
│ └──────────┘ └─────────────────┘ │
|
||||
│ ┌──────────┐ ┌─────────────────┐ │
|
||||
│ │ Layout │ │ Network │ │
|
||||
│ │ Engine │ │ (tokio + QUIC) │ │
|
||||
│ └──────────┘ └─────────────────┘ │
|
||||
│ ┌──────────────────────────────┐ │
|
||||
│ │ Renderer (wgpu / iced) │ │
|
||||
│ └──────────────────────────────┘ │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Why Rust makes this fast:**
|
||||
|
||||
- **Parser**: sexp parsing is trivial recursive descent. Nanoseconds per node. 100-1000x faster than JS.
|
||||
- **Renderer**: Skip the DOM entirely. Render directly to GPU-backed surface via `wgpu` or `iced`.
|
||||
- **Networking**: `reqwest` + `tokio` with connection pooling. Component manifests fetched in parallel.
|
||||
- **Component cache**: Pre-parsed ASTs on disk, content-addressed. No parse step on cache hit.
|
||||
- **Memory**: No GC pauses. 10-50MB where a browser tab uses 200-500MB.
|
||||
|
||||
**What you skip by not being a browser:**
|
||||
- No HTML parser (5-10ms per page)
|
||||
- No CSS cascade resolution (the most expensive part of browser rendering)
|
||||
- No DOM construction (sexp AST *is* the tree)
|
||||
- No JavaScript engine (logic in the sexp evaluator, compiled Rust)
|
||||
- No security sandbox overhead (no arbitrary JS execution)
|
||||
- No 2000+ web platform APIs you don't use
|
||||
|
||||
Page load: network (~50ms) → parse (~0.1ms) → layout (~2ms) → paint (~3ms) = **under 60ms**.
|
||||
|
||||
---
|
||||
|
||||
## Fallback Gateway
|
||||
|
||||
For users without the extension or native client, a gateway translates:
|
||||
|
||||
```
|
||||
sexpr://rose-ash.com/markets/
|
||||
→ Gateway fetches sexp from server
|
||||
→ Renders to HTML
|
||||
→ Serves to browser at https://gateway.rose-ash.com/markets/
|
||||
```
|
||||
|
||||
The regular web always works. The sexp protocol is an acceleration layer, not a requirement.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Content Negotiation (Quart)
|
||||
|
||||
Add `Accept` header handling to existing Quart routes.
|
||||
|
||||
- Check for `application/x-sexpr` in `Accept` header
|
||||
- If present: serialize the sexp tree and return directly (skip HTML render)
|
||||
- If absent: render to HTML as today
|
||||
- Add `Vary: Accept` header to responses
|
||||
|
||||
**Files:**
|
||||
- `shared/infrastructure/factory.py` — middleware for content negotiation
|
||||
- `shared/sexp/serialize.py` — canonical sexp serializer (deterministic output for caching/signing)
|
||||
|
||||
**Result:** Existing routes serve sexp to any client that asks. Zero changes to route logic.
|
||||
|
||||
### Phase 2: sexpr.js Client Library
|
||||
|
||||
Build the browser-side JS runtime (see `sexpr-js-runtime-plan.md` for full details).
|
||||
|
||||
- Parser + renderer: `parse()` → AST → `renderToDOM()`
|
||||
- Mutation engine: `swap!`, `batch!`, `class!`, `request!`
|
||||
- Component registry with localStorage cache (content-addressed)
|
||||
- `<script src="sexpr.js">` drop-in for any page
|
||||
|
||||
**Result:** Any page can opt into client-side sexp rendering.
|
||||
|
||||
### Phase 3: Browser Extension
|
||||
|
||||
Package sexpr.js as a WebExtension that intercepts `sexpr://` URLs.
|
||||
|
||||
- Register `sexpr://` protocol handler
|
||||
- Intercept navigation → fetch via HTTPS with `Accept: application/x-sexpr`
|
||||
- Parse response → render to DOM in current tab
|
||||
- Component cache in extension storage (IndexedDB, content-addressed)
|
||||
- Address bar shows `sexpr://rose-ash.com/markets/`
|
||||
|
||||
**Tech:** WebExtension API (Firefox + Chrome), JS/TS.
|
||||
|
||||
**Result:** Users install extension, navigate to `sexpr://` URLs, get sexp-rendered pages.
|
||||
|
||||
### Phase 4: Sexp Protocol Specification
|
||||
|
||||
Define the native protocol:
|
||||
|
||||
- **Framing**: length-prefixed sexp expressions over QUIC streams
|
||||
- **Request format**: `(METHOD path :keyword value ... body)`
|
||||
- **Response format**: `(status :keyword value ... body)`
|
||||
- **Push format**: `(push! mutation)` — server-initiated, any time
|
||||
- **Stream lifecycle**: open on first request, multiplex subsequent requests
|
||||
- **Caching**: content-hash based (`:have "sha3-..."`, `:have-components (...)`)
|
||||
- **Authentication**: token in request keywords (`:auth "bearer ..."`)
|
||||
- **Canonical serialization**: deterministic output rules for signing
|
||||
- **Schema introspection**: `GET /__schema/` returns API description as sexp
|
||||
|
||||
Publish as a specification document.
|
||||
|
||||
### Phase 5: Rust Protocol Server
|
||||
|
||||
Build a QUIC server that speaks the sexp protocol alongside Hypercorn:
|
||||
|
||||
- Listen on a separate port (e.g., 4433 for QUIC)
|
||||
- Parse sexp requests, route to the same handler logic as Quart
|
||||
- Bidirectional stream: pushes, requests, responses on one connection
|
||||
- Component manifest endpoint: serve component hashes + definitions
|
||||
- Connection pooling, TLS via rustls
|
||||
|
||||
**Tech:** Rust, `quinn` (QUIC), `tokio`, sexp parser crate.
|
||||
|
||||
**Architecture option:** The Rust server could proxy to Quart for data/logic and handle only the protocol layer. Or it could run the full application logic natively (long-term, after porting route handlers to Rust).
|
||||
|
||||
**Files:**
|
||||
- `sexpr-server/` — new Rust crate alongside the Python services
|
||||
- `sexpr-server/src/protocol.rs` — framing, parsing, routing
|
||||
- `sexpr-server/src/quic.rs` — QUIC listener + stream management
|
||||
|
||||
### Phase 6: Rust Native Client
|
||||
|
||||
Build the standalone sexp document viewer:
|
||||
|
||||
- QUIC client connecting to sexpr:// servers
|
||||
- Sexp parser (zero-copy, arena-allocated AST)
|
||||
- Component cache on disk (SQLite or filesystem, SHA3-keyed)
|
||||
- Layout engine (flexbox subset — enough for document layout)
|
||||
- GPU renderer via `wgpu` or `iced`
|
||||
- Text rendering via `cosmic-text` or `fontdue`
|
||||
- Input handling: keyboard, mouse, touch
|
||||
- Bidirectional stream: real-time mutations, navigation without reconnection
|
||||
|
||||
**Tech:** Rust, `quinn`, `wgpu`/`iced`, `cosmic-text`, `tokio`.
|
||||
|
||||
**Files:**
|
||||
- `sexpr-client/` — new Rust crate
|
||||
- `sexpr-client/src/parser.rs` — zero-copy sexp parser
|
||||
- `sexpr-client/src/cache.rs` — content-addressed component cache
|
||||
- `sexpr-client/src/layout.rs` — layout engine
|
||||
- `sexpr-client/src/render.rs` — GPU renderer
|
||||
|
||||
### Phase 7: Fallback Gateway
|
||||
|
||||
HTTP proxy that translates for browsers without extension/client:
|
||||
|
||||
- Accepts regular HTTPS requests
|
||||
- Fetches sexp from the server (internal)
|
||||
- Renders to HTML
|
||||
- Serves to browser
|
||||
- Adds `<link rel="alternate" type="application/x-sexpr" href="sexpr://...">` for discovery
|
||||
|
||||
**Result:** `https://rose-ash.com` always works. `sexpr://rose-ash.com` is an acceleration layer.
|
||||
|
||||
---
|
||||
|
||||
## Migration Path
|
||||
|
||||
Each tier builds on the last. No tier breaks the others:
|
||||
|
||||
1. **Today**: Tier 0 works. Ship HTML. Done.
|
||||
2. **Add `Accept` header check** (Phase 1): Same routes now serve sexp to clients that ask. Tier 0 unchanged.
|
||||
3. **Build sexpr.js** (Phase 2): Browser extension or `<script>` tag. Tier 1 works.
|
||||
4. **Build extension** (Phase 3): `sexpr://` URLs work in browser. Tier 1 complete.
|
||||
5. **Build Rust server** (Phase 5): Native protocol alongside HTTPS. Tier 2 infrastructure ready.
|
||||
6. **Build Rust client** (Phase 6): `sexpr://` URLs work natively. Tier 2 complete.
|
||||
|
||||
A user with the Rust client visits the same URL as someone with Firefox. The server serves both from the same handler. If the Rust client is offline, the user opens `https://rose-ash.com` in a browser and gets the HTML version. Same content, same components, same data.
|
||||
|
||||
---
|
||||
|
||||
## Cooperative Angle
|
||||
|
||||
- Members who install the Rust client get the fast native experience
|
||||
- Visitors browsing the public site get standard HTML — no barrier to entry
|
||||
- Federated peers negotiate the best format they support
|
||||
- AI agents get structured sexp via any tier
|
||||
- The 5MB Rust binary replaces a 500MB browser for accessing the cooperative platform
|
||||
- Auto-updates via content-addressed components — no app store gatekeeping
|
||||
|
||||
---
|
||||
|
||||
## Relationship to Other Plans
|
||||
|
||||
- **sexpr-js-runtime-plan.md** — Phase 2 of this plan; the JS library that powers Tier 1
|
||||
- **ghost-removal-plan.md** — posts must be sexp before content negotiation adds value
|
||||
- **sexpr-ai-integration.md** — AI agents benefit from all three tiers
|
||||
- **sexpr-activitypub-extension.md** — federation over the native protocol (Tier 2 peers speak AP natively)
|
||||
Reference in New Issue
Block a user