Add sexp protocol spec and tiered client architecture plan
All checks were successful
Build and Deploy / build-and-deploy (push) Successful in 1m16s
All checks were successful
Build and Deploy / build-and-deploy (push) Successful in 1m16s
Defines three client tiers (browser HTML, browser extension with sexpr.js, Rust native client) served from the same route handlers via content negotiation. Includes native sexp:// protocol design over QUIC, content-addressed caching, bidirectional streaming, self-describing schema, and implementation plan from Phase 1 (Quart content negotiation) through Phase 7 (fallback gateway). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
454
docs/sexpr-protocol-and-tiered-clients.md
Normal file
454
docs/sexpr-protocol-and-tiered-clients.md
Normal file
@@ -0,0 +1,454 @@
|
||||
# Sexp Protocol and Tiered Client Architecture
|
||||
|
||||
**One server, three clients, progressive enhancement from HTML to native.**
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The same rose-ash application serves three tiers of client from the same route handlers and component trees:
|
||||
|
||||
```
|
||||
┌─────────────────────────────┐
|
||||
│ rose-ash server (Quart) │
|
||||
│ │
|
||||
│ Same sexp component tree │
|
||||
│ Same data, same logic │
|
||||
│ │
|
||||
├──────────┬──────────┬────────┤
|
||||
│ HTTPS │ HTTPS │ SEXPR │
|
||||
│ HTML out │ sexp out │ native │
|
||||
└────┬─────┴────┬─────┴───┬────┘
|
||||
│ │ │
|
||||
┌──────────▼──┐ ┌─────▼──────┐ ┌▼──────────────┐
|
||||
│ Browser │ │ Browser + │ │ Rust client │
|
||||
│ (vanilla) │ │ extension │ │ (native) │
|
||||
│ │ │ │ │ │
|
||||
│ HTML + HTMX │ │ sexpr.js │ │ sexp protocol │
|
||||
│ Full CSS │ │ over HTTPS │ │ over QUIC │
|
||||
│ ~200ms load │ │ ~80ms load │ │ ~20ms load │
|
||||
└─────────────┘ └────────────┘ └────────────────┘
|
||||
Tier 0 Tier 1 Tier 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Content Negotiation
|
||||
|
||||
The server does format selection on a single endpoint:
|
||||
|
||||
```python
|
||||
@bp.get("/markets/")
|
||||
async def markets():
|
||||
# Same data, same component tree, always
|
||||
data = await get_markets(g.s)
|
||||
tree = sexp('(page :title "Markets" (each markets ...))', markets=data)
|
||||
|
||||
accept = request.headers.get("Accept", "")
|
||||
|
||||
if "application/x-sexpr" in accept:
|
||||
# Tier 1 (extension) or Tier 2 (native): raw sexp
|
||||
return Response(serialize(tree), content_type="application/x-sexpr")
|
||||
|
||||
# Tier 0: render to HTML for vanilla browsers
|
||||
html = render_to_html(tree)
|
||||
return Response(html, content_type="text/html")
|
||||
```
|
||||
|
||||
One route. One component tree. The output format is the only thing that changes.
|
||||
|
||||
---
|
||||
|
||||
## Tier Comparison
|
||||
|
||||
| | Tier 0: Browser | Tier 1: Extension | Tier 2: Rust Client |
|
||||
|---|---|---|---|
|
||||
| URL | `https://rose-ash.com` | `https://rose-ash.com` | `sexpr://rose-ash.com` |
|
||||
| Protocol | HTTPS | HTTPS | sexpr:// over QUIC |
|
||||
| Wire format | HTML | sexp over HTTP | sexp native |
|
||||
| Rendering | Browser DOM | sexpr.js → DOM | Rust → GPU |
|
||||
| Component cache | Browser cache (URL-keyed) | IndexedDB (hash-keyed) | Disk (hash-keyed, pre-parsed AST) |
|
||||
| Real-time | HTMX polling / SSE | WebSocket sexp mutations | Native bidirectional stream |
|
||||
| Bundle size | HTMX 14KB + CSS | sexpr.js ~8KB | 5MB binary, zero runtime deps |
|
||||
| Page load | ~200ms | ~80ms | ~20ms |
|
||||
| Memory | ~200MB per tab | ~150MB per tab | ~20MB |
|
||||
| Offline | Service worker (if built) | Component cache + manifest | Full local cache |
|
||||
| AI agent | Parse HTML (painful) | Parse sexp (easy) | Native sexp (trivial) |
|
||||
| Federation | N/A | AP via fetch | Native AP sexp stream |
|
||||
|
||||
---
|
||||
|
||||
## The Sexp Protocol (Tier 2)
|
||||
|
||||
### Why Not Just HTTP
|
||||
|
||||
HTTP is a request-response protocol with text headers and a body. Strip away the historical baggage and what you actually need is:
|
||||
|
||||
```
|
||||
Client sends: method + path + metadata + optional body
|
||||
Server sends: status + metadata + body
|
||||
```
|
||||
|
||||
That's just an s-expression:
|
||||
|
||||
```scheme
|
||||
;; Request
|
||||
(GET "/markets/"
|
||||
:accept "application/x-sexpr"
|
||||
:auth "bearer tok_abc123"
|
||||
:if-none-match "sha3-a1b2c3")
|
||||
|
||||
;; Response
|
||||
(200
|
||||
:content-hash "sha3-d4e5f6"
|
||||
:cache :immutable
|
||||
:components ("sha3-aaa" "sha3-bbb")
|
||||
|
||||
(page :title "Markets" :layout "main"
|
||||
(section (h1 "Markets")
|
||||
(each markets (lambda (m)
|
||||
(use "vendor-card" :name (get m "name")))))))
|
||||
```
|
||||
|
||||
No header/body split. No chunked transfer encoding. No `Content-Length` because the parser knows when the expression ends (balanced parens). No `Content-Type` because everything is sexp. No MIME, no multipart, no `Transfer-Encoding`, no `Connection: keep-alive` negotiation.
|
||||
|
||||
### What HTTP Gets Wrong That Sexp Fixes
|
||||
|
||||
#### 1. Headers are a bad key-value format
|
||||
|
||||
HTTP headers are case-insensitive, can be duplicated, have weird continuation rules, and are parsed separately from the body. In sexp, metadata is just keywords in the expression — same parser, same format, no special case.
|
||||
|
||||
#### 2. Request/response is too rigid
|
||||
|
||||
HTTP is strictly one request → one response. To get around this, we've bolted on WebSocket (separate protocol, upgrade handshake), Server-Sent Events (hack using chunked encoding), HTTP/2 server push (failed, being removed), and HTTP/3 QUIC streams (complex, still one-request-one-response per stream).
|
||||
|
||||
A sexp protocol is **bidirectional from the start**:
|
||||
|
||||
```scheme
|
||||
;; Client opens connection, sends request
|
||||
(GET "/feed/" :stream #t)
|
||||
|
||||
;; Server responds with initial content
|
||||
(200 :stream #t
|
||||
(page :title "Feed"
|
||||
(div :id "feed" (p "Loading..."))))
|
||||
|
||||
;; Server pushes updates as they happen (same connection)
|
||||
(push! (swap! "#feed" :prepend
|
||||
(use "post-card" :title "New post" :author "alice")))
|
||||
|
||||
;; Client sends an action (same connection)
|
||||
(POST "/like/" :body (:post-id 123))
|
||||
|
||||
;; Server responds with mutation
|
||||
(push! (swap! "#like-count-123" :inner "43"))
|
||||
|
||||
;; Client navigates (same connection, no new handshake)
|
||||
(GET "/markets/")
|
||||
|
||||
;; Server responds
|
||||
(200 (page :title "Markets" ...))
|
||||
```
|
||||
|
||||
One persistent connection. Requests, responses, and pushes are all sexp expressions on the same stream. No protocol upgrade, no separate WebSocket connection, no polling.
|
||||
|
||||
#### 3. Caching is overcomplicated
|
||||
|
||||
HTTP caching: `Cache-Control`, `ETag`, `If-None-Match`, `If-Modified-Since`, `Vary`, `Age`, `Expires`, `Last-Modified`, `s-maxage`, `stale-while-revalidate`... a dozen headers with complex interaction rules.
|
||||
|
||||
With content-addressed sexp:
|
||||
|
||||
```scheme
|
||||
;; Request with known hash
|
||||
(GET "/markets/" :have "sha3-a1b2c3")
|
||||
|
||||
;; If unchanged:
|
||||
(304)
|
||||
|
||||
;; If changed:
|
||||
(200 :hash "sha3-d4e5f6" (page ...))
|
||||
```
|
||||
|
||||
One field. The hash *is* the cache key, the ETag, and the content address.
|
||||
|
||||
Components take this further — the client sends what it already has:
|
||||
|
||||
```scheme
|
||||
(GET "/markets/"
|
||||
:have-components ("sha3-aaa" "sha3-bbb" "sha3-ccc"))
|
||||
|
||||
;; Server only sends components the client is missing
|
||||
(200
|
||||
:new-components (
|
||||
(component "vendor-card" :hash "sha3-ddd" (params ...) (template ...)))
|
||||
(page ...))
|
||||
```
|
||||
|
||||
#### 4. Status codes are arbitrary numbers
|
||||
|
||||
```scheme
|
||||
(ok (page ...))
|
||||
(redirect :to "/new-location/" :permanent #t)
|
||||
(not-found :message "Page does not exist")
|
||||
(error :code "auth-required" :message "Please log in"
|
||||
:login-url "sexpr://rose-ash.com/login/")
|
||||
```
|
||||
|
||||
The status *is* the expression. Machines pattern-match on the head symbol. Humans read it. AI agents understand it without a lookup table.
|
||||
|
||||
#### 5. Forms and file uploads are a mess
|
||||
|
||||
HTTP: `application/x-www-form-urlencoded` or `multipart/form-data` with boundary strings, MIME parts, content-disposition headers.
|
||||
|
||||
Sexp:
|
||||
|
||||
```scheme
|
||||
(POST "/submit/"
|
||||
:body (
|
||||
:username "alice"
|
||||
:email "alice@example.com"
|
||||
:avatar (file :name "photo.jpg" :type "image/jpeg" :data <binary>)))
|
||||
```
|
||||
|
||||
Structured data with inline binary. One format.
|
||||
|
||||
#### 6. The protocol is self-describing
|
||||
|
||||
HTTP has no introspection. You need OpenAPI/Swagger specs bolted on separately.
|
||||
|
||||
```scheme
|
||||
(GET "/__schema/")
|
||||
|
||||
;; Response: the API describes itself
|
||||
(schema
|
||||
(endpoint "/" :method GET
|
||||
:returns (page)
|
||||
:params (:stream bool))
|
||||
(endpoint "/markets/" :method GET
|
||||
:returns (page :contains (list (use "vendor-card"))))
|
||||
(endpoint "/like/" :method POST
|
||||
:params (:post-id int)
|
||||
:returns (mutation)))
|
||||
```
|
||||
|
||||
An AI agent hitting `/__schema/` learns the entire API surface as parseable sexp. No separate OpenAPI doc. The schema *is* the API.
|
||||
|
||||
### Protocol Stack
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Application: sexp documents, mutations │
|
||||
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
|
||||
│ Session: bidirectional sexp stream │
|
||||
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
|
||||
│ Transport: QUIC (or TCP+TLS) │
|
||||
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
|
||||
│ Network: IP │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
QUIC is ideal — multiplexed streams, built-in TLS, connection migration. Each sexp expression gets its own QUIC stream. Requests and pushes multiplex without head-of-line blocking. Connection survives network changes (mobile → wifi).
|
||||
|
||||
---
|
||||
|
||||
## Rust Client Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ sexpr-client (Rust) │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌─────────────────┐ │
|
||||
│ │ Parser │ │ Component │ │
|
||||
│ │ (zero- │ │ Cache │ │
|
||||
│ │ copy) │ │ (SHA3 → AST) │ │
|
||||
│ └──────────┘ └─────────────────┘ │
|
||||
│ ┌──────────┐ ┌─────────────────┐ │
|
||||
│ │ Layout │ │ Network │ │
|
||||
│ │ Engine │ │ (tokio + QUIC) │ │
|
||||
│ └──────────┘ └─────────────────┘ │
|
||||
│ ┌──────────────────────────────┐ │
|
||||
│ │ Renderer (wgpu / iced) │ │
|
||||
│ └──────────────────────────────┘ │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Why Rust makes this fast:**
|
||||
|
||||
- **Parser**: sexp parsing is trivial recursive descent. Nanoseconds per node. 100-1000x faster than JS.
|
||||
- **Renderer**: Skip the DOM entirely. Render directly to GPU-backed surface via `wgpu` or `iced`.
|
||||
- **Networking**: `reqwest` + `tokio` with connection pooling. Component manifests fetched in parallel.
|
||||
- **Component cache**: Pre-parsed ASTs on disk, content-addressed. No parse step on cache hit.
|
||||
- **Memory**: No GC pauses. 10-50MB where a browser tab uses 200-500MB.
|
||||
|
||||
**What you skip by not being a browser:**
|
||||
- No HTML parser (5-10ms per page)
|
||||
- No CSS cascade resolution (the most expensive part of browser rendering)
|
||||
- No DOM construction (sexp AST *is* the tree)
|
||||
- No JavaScript engine (logic in the sexp evaluator, compiled Rust)
|
||||
- No security sandbox overhead (no arbitrary JS execution)
|
||||
- No 2000+ web platform APIs you don't use
|
||||
|
||||
Page load: network (~50ms) → parse (~0.1ms) → layout (~2ms) → paint (~3ms) = **under 60ms**.
|
||||
|
||||
---
|
||||
|
||||
## Fallback Gateway
|
||||
|
||||
For users without the extension or native client, a gateway translates:
|
||||
|
||||
```
|
||||
sexpr://rose-ash.com/markets/
|
||||
→ Gateway fetches sexp from server
|
||||
→ Renders to HTML
|
||||
→ Serves to browser at https://gateway.rose-ash.com/markets/
|
||||
```
|
||||
|
||||
The regular web always works. The sexp protocol is an acceleration layer, not a requirement.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Content Negotiation (Quart)
|
||||
|
||||
Add `Accept` header handling to existing Quart routes.
|
||||
|
||||
- Check for `application/x-sexpr` in `Accept` header
|
||||
- If present: serialize the sexp tree and return directly (skip HTML render)
|
||||
- If absent: render to HTML as today
|
||||
- Add `Vary: Accept` header to responses
|
||||
|
||||
**Files:**
|
||||
- `shared/infrastructure/factory.py` — middleware for content negotiation
|
||||
- `shared/sexp/serialize.py` — canonical sexp serializer (deterministic output for caching/signing)
|
||||
|
||||
**Result:** Existing routes serve sexp to any client that asks. Zero changes to route logic.
|
||||
|
||||
### Phase 2: sexpr.js Client Library
|
||||
|
||||
Build the browser-side JS runtime (see `sexpr-js-runtime-plan.md` for full details).
|
||||
|
||||
- Parser + renderer: `parse()` → AST → `renderToDOM()`
|
||||
- Mutation engine: `swap!`, `batch!`, `class!`, `request!`
|
||||
- Component registry with localStorage cache (content-addressed)
|
||||
- `<script src="sexpr.js">` drop-in for any page
|
||||
|
||||
**Result:** Any page can opt into client-side sexp rendering.
|
||||
|
||||
### Phase 3: Browser Extension
|
||||
|
||||
Package sexpr.js as a WebExtension that intercepts `sexpr://` URLs.
|
||||
|
||||
- Register `sexpr://` protocol handler
|
||||
- Intercept navigation → fetch via HTTPS with `Accept: application/x-sexpr`
|
||||
- Parse response → render to DOM in current tab
|
||||
- Component cache in extension storage (IndexedDB, content-addressed)
|
||||
- Address bar shows `sexpr://rose-ash.com/markets/`
|
||||
|
||||
**Tech:** WebExtension API (Firefox + Chrome), JS/TS.
|
||||
|
||||
**Result:** Users install extension, navigate to `sexpr://` URLs, get sexp-rendered pages.
|
||||
|
||||
### Phase 4: Sexp Protocol Specification
|
||||
|
||||
Define the native protocol:
|
||||
|
||||
- **Framing**: length-prefixed sexp expressions over QUIC streams
|
||||
- **Request format**: `(METHOD path :keyword value ... body)`
|
||||
- **Response format**: `(status :keyword value ... body)`
|
||||
- **Push format**: `(push! mutation)` — server-initiated, any time
|
||||
- **Stream lifecycle**: open on first request, multiplex subsequent requests
|
||||
- **Caching**: content-hash based (`:have "sha3-..."`, `:have-components (...)`)
|
||||
- **Authentication**: token in request keywords (`:auth "bearer ..."`)
|
||||
- **Canonical serialization**: deterministic output rules for signing
|
||||
- **Schema introspection**: `GET /__schema/` returns API description as sexp
|
||||
|
||||
Publish as a specification document.
|
||||
|
||||
### Phase 5: Rust Protocol Server
|
||||
|
||||
Build a QUIC server that speaks the sexp protocol alongside Hypercorn:
|
||||
|
||||
- Listen on a separate port (e.g., 4433 for QUIC)
|
||||
- Parse sexp requests, route to the same handler logic as Quart
|
||||
- Bidirectional stream: pushes, requests, responses on one connection
|
||||
- Component manifest endpoint: serve component hashes + definitions
|
||||
- Connection pooling, TLS via rustls
|
||||
|
||||
**Tech:** Rust, `quinn` (QUIC), `tokio`, sexp parser crate.
|
||||
|
||||
**Architecture option:** The Rust server could proxy to Quart for data/logic and handle only the protocol layer. Or it could run the full application logic natively (long-term, after porting route handlers to Rust).
|
||||
|
||||
**Files:**
|
||||
- `sexpr-server/` — new Rust crate alongside the Python services
|
||||
- `sexpr-server/src/protocol.rs` — framing, parsing, routing
|
||||
- `sexpr-server/src/quic.rs` — QUIC listener + stream management
|
||||
|
||||
### Phase 6: Rust Native Client
|
||||
|
||||
Build the standalone sexp document viewer:
|
||||
|
||||
- QUIC client connecting to sexpr:// servers
|
||||
- Sexp parser (zero-copy, arena-allocated AST)
|
||||
- Component cache on disk (SQLite or filesystem, SHA3-keyed)
|
||||
- Layout engine (flexbox subset — enough for document layout)
|
||||
- GPU renderer via `wgpu` or `iced`
|
||||
- Text rendering via `cosmic-text` or `fontdue`
|
||||
- Input handling: keyboard, mouse, touch
|
||||
- Bidirectional stream: real-time mutations, navigation without reconnection
|
||||
|
||||
**Tech:** Rust, `quinn`, `wgpu`/`iced`, `cosmic-text`, `tokio`.
|
||||
|
||||
**Files:**
|
||||
- `sexpr-client/` — new Rust crate
|
||||
- `sexpr-client/src/parser.rs` — zero-copy sexp parser
|
||||
- `sexpr-client/src/cache.rs` — content-addressed component cache
|
||||
- `sexpr-client/src/layout.rs` — layout engine
|
||||
- `sexpr-client/src/render.rs` — GPU renderer
|
||||
|
||||
### Phase 7: Fallback Gateway
|
||||
|
||||
HTTP proxy that translates for browsers without extension/client:
|
||||
|
||||
- Accepts regular HTTPS requests
|
||||
- Fetches sexp from the server (internal)
|
||||
- Renders to HTML
|
||||
- Serves to browser
|
||||
- Adds `<link rel="alternate" type="application/x-sexpr" href="sexpr://...">` for discovery
|
||||
|
||||
**Result:** `https://rose-ash.com` always works. `sexpr://rose-ash.com` is an acceleration layer.
|
||||
|
||||
---
|
||||
|
||||
## Migration Path
|
||||
|
||||
Each tier builds on the last. No tier breaks the others:
|
||||
|
||||
1. **Today**: Tier 0 works. Ship HTML. Done.
|
||||
2. **Add `Accept` header check** (Phase 1): Same routes now serve sexp to clients that ask. Tier 0 unchanged.
|
||||
3. **Build sexpr.js** (Phase 2): Browser extension or `<script>` tag. Tier 1 works.
|
||||
4. **Build extension** (Phase 3): `sexpr://` URLs work in browser. Tier 1 complete.
|
||||
5. **Build Rust server** (Phase 5): Native protocol alongside HTTPS. Tier 2 infrastructure ready.
|
||||
6. **Build Rust client** (Phase 6): `sexpr://` URLs work natively. Tier 2 complete.
|
||||
|
||||
A user with the Rust client visits the same URL as someone with Firefox. The server serves both from the same handler. If the Rust client is offline, the user opens `https://rose-ash.com` in a browser and gets the HTML version. Same content, same components, same data.
|
||||
|
||||
---
|
||||
|
||||
## Cooperative Angle
|
||||
|
||||
- Members who install the Rust client get the fast native experience
|
||||
- Visitors browsing the public site get standard HTML — no barrier to entry
|
||||
- Federated peers negotiate the best format they support
|
||||
- AI agents get structured sexp via any tier
|
||||
- The 5MB Rust binary replaces a 500MB browser for accessing the cooperative platform
|
||||
- Auto-updates via content-addressed components — no app store gatekeeping
|
||||
|
||||
---
|
||||
|
||||
## Relationship to Other Plans
|
||||
|
||||
- **sexpr-js-runtime-plan.md** — Phase 2 of this plan; the JS library that powers Tier 1
|
||||
- **ghost-removal-plan.md** — posts must be sexp before content negotiation adds value
|
||||
- **sexpr-ai-integration.md** — AI agents benefit from all three tiers
|
||||
- **sexpr-activitypub-extension.md** — federation over the native protocol (Tier 2 peers speak AP natively)
|
||||
Reference in New Issue
Block a user