diff --git a/docs/sexpr-protocol-and-tiered-clients.md b/docs/sexpr-protocol-and-tiered-clients.md new file mode 100644 index 0000000..520009a --- /dev/null +++ b/docs/sexpr-protocol-and-tiered-clients.md @@ -0,0 +1,454 @@ +# Sexp Protocol and Tiered Client Architecture + +**One server, three clients, progressive enhancement from HTML to native.** + +--- + +## Overview + +The same rose-ash application serves three tiers of client from the same route handlers and component trees: + +``` + ┌─────────────────────────────┐ + │ rose-ash server (Quart) │ + │ │ + │ Same sexp component tree │ + │ Same data, same logic │ + │ │ + ├──────────┬──────────┬────────┤ + │ HTTPS │ HTTPS │ SEXPR │ + │ HTML out │ sexp out │ native │ + └────┬─────┴────┬─────┴───┬────┘ + │ │ │ + ┌──────────▼──┐ ┌─────▼──────┐ ┌▼──────────────┐ + │ Browser │ │ Browser + │ │ Rust client │ + │ (vanilla) │ │ extension │ │ (native) │ + │ │ │ │ │ │ + │ HTML + HTMX │ │ sexpr.js │ │ sexp protocol │ + │ Full CSS │ │ over HTTPS │ │ over QUIC │ + │ ~200ms load │ │ ~80ms load │ │ ~20ms load │ + └─────────────┘ └────────────┘ └────────────────┘ + Tier 0 Tier 1 Tier 2 +``` + +--- + +## Content Negotiation + +The server does format selection on a single endpoint: + +```python +@bp.get("/markets/") +async def markets(): + # Same data, same component tree, always + data = await get_markets(g.s) + tree = sexp('(page :title "Markets" (each markets ...))', markets=data) + + accept = request.headers.get("Accept", "") + + if "application/x-sexpr" in accept: + # Tier 1 (extension) or Tier 2 (native): raw sexp + return Response(serialize(tree), content_type="application/x-sexpr") + + # Tier 0: render to HTML for vanilla browsers + html = render_to_html(tree) + return Response(html, content_type="text/html") +``` + +One route. One component tree. The output format is the only thing that changes. + +--- + +## Tier Comparison + +| | Tier 0: Browser | Tier 1: Extension | Tier 2: Rust Client | +|---|---|---|---| +| URL | `https://rose-ash.com` | `https://rose-ash.com` | `sexpr://rose-ash.com` | +| Protocol | HTTPS | HTTPS | sexpr:// over QUIC | +| Wire format | HTML | sexp over HTTP | sexp native | +| Rendering | Browser DOM | sexpr.js → DOM | Rust → GPU | +| Component cache | Browser cache (URL-keyed) | IndexedDB (hash-keyed) | Disk (hash-keyed, pre-parsed AST) | +| Real-time | HTMX polling / SSE | WebSocket sexp mutations | Native bidirectional stream | +| Bundle size | HTMX 14KB + CSS | sexpr.js ~8KB | 5MB binary, zero runtime deps | +| Page load | ~200ms | ~80ms | ~20ms | +| Memory | ~200MB per tab | ~150MB per tab | ~20MB | +| Offline | Service worker (if built) | Component cache + manifest | Full local cache | +| AI agent | Parse HTML (painful) | Parse sexp (easy) | Native sexp (trivial) | +| Federation | N/A | AP via fetch | Native AP sexp stream | + +--- + +## The Sexp Protocol (Tier 2) + +### Why Not Just HTTP + +HTTP is a request-response protocol with text headers and a body. Strip away the historical baggage and what you actually need is: + +``` +Client sends: method + path + metadata + optional body +Server sends: status + metadata + body +``` + +That's just an s-expression: + +```scheme +;; Request +(GET "/markets/" + :accept "application/x-sexpr" + :auth "bearer tok_abc123" + :if-none-match "sha3-a1b2c3") + +;; Response +(200 + :content-hash "sha3-d4e5f6" + :cache :immutable + :components ("sha3-aaa" "sha3-bbb") + + (page :title "Markets" :layout "main" + (section (h1 "Markets") + (each markets (lambda (m) + (use "vendor-card" :name (get m "name"))))))) +``` + +No header/body split. No chunked transfer encoding. No `Content-Length` because the parser knows when the expression ends (balanced parens). No `Content-Type` because everything is sexp. No MIME, no multipart, no `Transfer-Encoding`, no `Connection: keep-alive` negotiation. + +### What HTTP Gets Wrong That Sexp Fixes + +#### 1. Headers are a bad key-value format + +HTTP headers are case-insensitive, can be duplicated, have weird continuation rules, and are parsed separately from the body. In sexp, metadata is just keywords in the expression — same parser, same format, no special case. + +#### 2. Request/response is too rigid + +HTTP is strictly one request → one response. To get around this, we've bolted on WebSocket (separate protocol, upgrade handshake), Server-Sent Events (hack using chunked encoding), HTTP/2 server push (failed, being removed), and HTTP/3 QUIC streams (complex, still one-request-one-response per stream). + +A sexp protocol is **bidirectional from the start**: + +```scheme +;; Client opens connection, sends request +(GET "/feed/" :stream #t) + +;; Server responds with initial content +(200 :stream #t + (page :title "Feed" + (div :id "feed" (p "Loading...")))) + +;; Server pushes updates as they happen (same connection) +(push! (swap! "#feed" :prepend + (use "post-card" :title "New post" :author "alice"))) + +;; Client sends an action (same connection) +(POST "/like/" :body (:post-id 123)) + +;; Server responds with mutation +(push! (swap! "#like-count-123" :inner "43")) + +;; Client navigates (same connection, no new handshake) +(GET "/markets/") + +;; Server responds +(200 (page :title "Markets" ...)) +``` + +One persistent connection. Requests, responses, and pushes are all sexp expressions on the same stream. No protocol upgrade, no separate WebSocket connection, no polling. + +#### 3. Caching is overcomplicated + +HTTP caching: `Cache-Control`, `ETag`, `If-None-Match`, `If-Modified-Since`, `Vary`, `Age`, `Expires`, `Last-Modified`, `s-maxage`, `stale-while-revalidate`... a dozen headers with complex interaction rules. + +With content-addressed sexp: + +```scheme +;; Request with known hash +(GET "/markets/" :have "sha3-a1b2c3") + +;; If unchanged: +(304) + +;; If changed: +(200 :hash "sha3-d4e5f6" (page ...)) +``` + +One field. The hash *is* the cache key, the ETag, and the content address. + +Components take this further — the client sends what it already has: + +```scheme +(GET "/markets/" + :have-components ("sha3-aaa" "sha3-bbb" "sha3-ccc")) + +;; Server only sends components the client is missing +(200 + :new-components ( + (component "vendor-card" :hash "sha3-ddd" (params ...) (template ...))) + (page ...)) +``` + +#### 4. Status codes are arbitrary numbers + +```scheme +(ok (page ...)) +(redirect :to "/new-location/" :permanent #t) +(not-found :message "Page does not exist") +(error :code "auth-required" :message "Please log in" + :login-url "sexpr://rose-ash.com/login/") +``` + +The status *is* the expression. Machines pattern-match on the head symbol. Humans read it. AI agents understand it without a lookup table. + +#### 5. Forms and file uploads are a mess + +HTTP: `application/x-www-form-urlencoded` or `multipart/form-data` with boundary strings, MIME parts, content-disposition headers. + +Sexp: + +```scheme +(POST "/submit/" + :body ( + :username "alice" + :email "alice@example.com" + :avatar (file :name "photo.jpg" :type "image/jpeg" :data ))) +``` + +Structured data with inline binary. One format. + +#### 6. The protocol is self-describing + +HTTP has no introspection. You need OpenAPI/Swagger specs bolted on separately. + +```scheme +(GET "/__schema/") + +;; Response: the API describes itself +(schema + (endpoint "/" :method GET + :returns (page) + :params (:stream bool)) + (endpoint "/markets/" :method GET + :returns (page :contains (list (use "vendor-card")))) + (endpoint "/like/" :method POST + :params (:post-id int) + :returns (mutation))) +``` + +An AI agent hitting `/__schema/` learns the entire API surface as parseable sexp. No separate OpenAPI doc. The schema *is* the API. + +### Protocol Stack + +``` +┌─────────────────────────────────────────┐ +│ Application: sexp documents, mutations │ +│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ +│ Session: bidirectional sexp stream │ +│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ +│ Transport: QUIC (or TCP+TLS) │ +│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ +│ Network: IP │ +└─────────────────────────────────────────┘ +``` + +QUIC is ideal — multiplexed streams, built-in TLS, connection migration. Each sexp expression gets its own QUIC stream. Requests and pushes multiplex without head-of-line blocking. Connection survives network changes (mobile → wifi). + +--- + +## Rust Client Architecture + +``` +┌─────────────────────────────────────┐ +│ sexpr-client (Rust) │ +│ │ +│ ┌──────────┐ ┌─────────────────┐ │ +│ │ Parser │ │ Component │ │ +│ │ (zero- │ │ Cache │ │ +│ │ copy) │ │ (SHA3 → AST) │ │ +│ └──────────┘ └─────────────────┘ │ +│ ┌──────────┐ ┌─────────────────┐ │ +│ │ Layout │ │ Network │ │ +│ │ Engine │ │ (tokio + QUIC) │ │ +│ └──────────┘ └─────────────────┘ │ +│ ┌──────────────────────────────┐ │ +│ │ Renderer (wgpu / iced) │ │ +│ └──────────────────────────────┘ │ +└─────────────────────────────────────┘ +``` + +**Why Rust makes this fast:** + +- **Parser**: sexp parsing is trivial recursive descent. Nanoseconds per node. 100-1000x faster than JS. +- **Renderer**: Skip the DOM entirely. Render directly to GPU-backed surface via `wgpu` or `iced`. +- **Networking**: `reqwest` + `tokio` with connection pooling. Component manifests fetched in parallel. +- **Component cache**: Pre-parsed ASTs on disk, content-addressed. No parse step on cache hit. +- **Memory**: No GC pauses. 10-50MB where a browser tab uses 200-500MB. + +**What you skip by not being a browser:** +- No HTML parser (5-10ms per page) +- No CSS cascade resolution (the most expensive part of browser rendering) +- No DOM construction (sexp AST *is* the tree) +- No JavaScript engine (logic in the sexp evaluator, compiled Rust) +- No security sandbox overhead (no arbitrary JS execution) +- No 2000+ web platform APIs you don't use + +Page load: network (~50ms) → parse (~0.1ms) → layout (~2ms) → paint (~3ms) = **under 60ms**. + +--- + +## Fallback Gateway + +For users without the extension or native client, a gateway translates: + +``` +sexpr://rose-ash.com/markets/ + → Gateway fetches sexp from server + → Renders to HTML + → Serves to browser at https://gateway.rose-ash.com/markets/ +``` + +The regular web always works. The sexp protocol is an acceleration layer, not a requirement. + +--- + +## Implementation Plan + +### Phase 1: Content Negotiation (Quart) + +Add `Accept` header handling to existing Quart routes. + +- Check for `application/x-sexpr` in `Accept` header +- If present: serialize the sexp tree and return directly (skip HTML render) +- If absent: render to HTML as today +- Add `Vary: Accept` header to responses + +**Files:** +- `shared/infrastructure/factory.py` — middleware for content negotiation +- `shared/sexp/serialize.py` — canonical sexp serializer (deterministic output for caching/signing) + +**Result:** Existing routes serve sexp to any client that asks. Zero changes to route logic. + +### Phase 2: sexpr.js Client Library + +Build the browser-side JS runtime (see `sexpr-js-runtime-plan.md` for full details). + +- Parser + renderer: `parse()` → AST → `renderToDOM()` +- Mutation engine: `swap!`, `batch!`, `class!`, `request!` +- Component registry with localStorage cache (content-addressed) +- `