rose-ash/docs/isomorphic-sx-plan.md

# SX Isomorphic Architecture Roadmap

## Context

SX has a working server-client pipeline: server evaluates pages with IO (DB, fragments), serializes as SX wire format, client parses and renders to DOM. The language and primitives are already isomorphic — same spec, same semantics, both sides. What's missing is the **plumbing** that makes the boundary between server and client a sliding window rather than a fixed wall.

The key insight: **s-expressions can partially unfold on the server after IO, then finish unfolding on the client.** The system should be clever enough to know which downstream components have data fetches, resolve those server-side, and send the rest as pure SX for client rendering. Eventually, the client can also do IO (mapping server DB queries to REST calls), handle routing (SPA), and even work offline with cached data.

## Current State (what's solid)

- **Primitive parity:** 100%. ~80 pure primitives, same names/semantics, JS and Python.
- **eval/parse/render:** Complete both sides. sx-ref.js has eval, parse, render-to-html, render-to-dom, aser.
- **Engine:** engine.sx (morph, swaps, triggers, history), orchestration.sx (fetch, events), boot.sx (hydration) — all transpiled.
- **Wire format:** Server `_aser` → SX source → client parses → renders to DOM. Boundary is clean.
- **Component caching:** Hash-based localStorage for component definitions and style dictionaries.
- **CSS on-demand:** CSSX resolves keywords to CSS rules, injects only used rules.
- **Boundary enforcement:** `boundary.sx` + `SX_BOUNDARY_STRICT=1` validates all primitives/IO/helpers at registration.

## Architecture Phases

---

### Phase 1: Component Distribution & Dependency Analysis — DONE

**What it enables:** Per-page component bundles instead of sending every definition to every page. Smaller payloads, faster boot, better cache hit rates.

**Implemented:**

1. **Transitive closure analyzer** — `shared/sx/deps.py` (now `shared/sx/ref/deps.sx`, spec-level)
   - Walk component body AST, collect all `~name` refs
   - Recursively follow into their bodies
   - Handle control forms (`if`/`when`/`cond`/`case`) — include ALL branches
   - `components_needed(source, env) -> set[str]`

2. **IO reference analysis** — `deps.sx` also tracks IO primitive usage
   - `scan-io-refs` / `transitive-io-refs` / `component-pure?`
   - Used by Phase 2 for automatic server/client boundary

3. **Per-page component block** — `_build_pages_sx()` in `helpers.py`
   - Each page entry includes `:deps` list of required components
   - Client page registry carries dep info for prefetching

4. **SX partial responses** — `components_for_request()` diffs against `SX-Components` header, sends only missing components

**Files:** `shared/sx/ref/deps.sx`, `shared/sx/deps.py`, `shared/sx/helpers.py`, `shared/sx/jinja_bridge.py`

---

### Phase 2: Smart Server/Client Boundary — DONE

**What it enables:** Formalized partial evaluation model. Server evaluates IO, serializes pure subtrees. The system automatically knows "this component needs server data" vs "this component is pure and can render anywhere."

**Implemented:**

1. **Automatic IO detection** — `deps.sx` walks component bodies for IO primitive refs
   - `compute-all-io-refs` computes transitive IO analysis for all components
   - `component-pure?` returns true if no IO refs transitively

2. **Selective expansion** — `_aser` expands known components server-side via `_aser_component`
   - IO-dependent components expand server-side (IO must resolve)
   - Unknown components serialize for client rendering
   - `_expand_components` context var controls override

3. **Component metadata** — computed at registration, cached on Component objects

**Files:** `shared/sx/ref/deps.sx`, `shared/sx/async_eval.py`, `shared/sx/jinja_bridge.py`

---

### Phase 3: Client-Side Routing (SPA Mode) — DONE

**What it enables:** After initial page load, client resolves routes locally using cached components. Only hits server for fresh data or unknown routes.

**Implemented:**

1. **Client-side page registry** — `_build_pages_sx()` serializes defpage routing info
   - `<script type="text/sx-pages">` with name, path, auth, content, deps, closure, has-data
   - Processed by `boot.sx` → `_page-routes` list

2. **Client route matcher** — `shared/sx/ref/router.sx`
   - `parse-route-pattern` converts Flask-style `/docs/<slug>` to matchers
   - `find-matching-route` matches URL against registered routes
   - `match-route-segments` handles literal and param segments

3. **Client-side route intercept** — `orchestration.sx`
   - `try-client-route` — match URL, eval content locally, swap DOM
   - `bind-client-route-link` — intercept boost link clicks
   - Pure pages render immediately, no server roundtrip
   - Falls through to server fetch on miss

4. **Integration with engine** — boost link clicks try client route first, fall back to standard fetch

**Files:** `shared/sx/ref/router.sx`, `shared/sx/ref/boot.sx`, `shared/sx/ref/orchestration.sx`, `shared/sx/helpers.py`, `shared/sx/pages.py`

---

### Phase 4: Client Async & IO Bridge — DONE

**What it enables:** Client fetches server-evaluated data and renders `:data` pages locally. Data cached to avoid redundant fetches on back/forward navigation.

**The approach:** Separate IO from rendering. Server evaluates `:data` expression (async, with DB/service access), serializes result as SX wire format. Client fetches this pre-evaluated data, parses it, merges into env, renders pure `:content` client-side. No continuations needed — all IO happens server-side.

**Implemented:**

1. **Abstract `resolve-page-data`** — spec-level primitive in `orchestration.sx`
   - `(resolve-page-data name params callback)` — platform decides transport
   - Spec says "I need data for this page"; platform provides concrete implementation
   - Browser platform: HTTP fetch to `/sx/data/` endpoint

2. **Server data endpoint** — `pages.py`
   - `evaluate_page_data()` — evaluates `:data` expression, kebab-cases dict keys, serializes as SX
   - `auto_mount_page_data()` — mounts `GET /sx/data/<page_name>` endpoint
   - Per-page auth enforcement via `_check_page_auth()`
   - Response content type: `text/sx; charset=utf-8`

3. **Client-side data rendering** — `orchestration.sx`
   - `try-client-route` handles `:data` pages: fetch data → parse SX → merge into env → render content
   - Console log: `sx:route client+data <pathname>` confirms client-side rendering
   - Component deps computed for `:data` pages too (not just pure pages)

4. **Client data cache** — `orchestration.sx`
   - `_page-data-cache` dict keyed by `page-name:param=value`
   - 30s TTL (configurable via `_page-data-cache-ttl`)
   - Cache hit: `sx:route client+cache <pathname>` — renders instantly
   - Cache miss: fetches, caches, renders
   - Stale entries evicted on next access

5. **Test page** — `sx/sx/data-test.sx`
   - Exercises full data pipeline: server time, pipeline steps, phase/transport metadata
   - Navigate from another page → console shows `sx:route client+data`
   - Navigate back → console shows `sx:route client+cache`

6. **Unit tests** — `shared/sx/tests/test_page_data.py` (20 tests)
   - Serialize roundtrip for all data types
   - Kebab-case key conversion
   - Component deps for `:data` pages
   - Full pipeline simulation (serialize → parse → merge → eval)

**Files:**
- `shared/sx/ref/orchestration.sx` — `resolve-page-data` spec, data cache
- `shared/sx/ref/bootstrap_js.py` — platform `resolvePageData` implementation
- `shared/sx/pages.py` — `evaluate_page_data()`, `auto_mount_page_data()`
- `shared/sx/helpers.py` — deps for `:data` pages
- `sx/sx/data-test.sx` — test component
- `sx/sxc/pages/docs.sx` — test page defpage
- `sx/sxc/pages/helpers.py` — `data-test-data` helper
- `sx/sx/boundary.sx` — helper declaration
- `shared/sx/tests/test_page_data.py` — unit tests

---

### Phase 5: Async Continuations & Inline IO

**What it enables:** Components call IO primitives directly in their body (e.g. `(query ...)`). The evaluator suspends mid-evaluation, fetches data, resumes. Same component source works on both server (Python async/await) and client (continuation-based suspension).

**The problem:** The existing `shift/reset` continuations extension is synchronous (throw/catch). Client-side IO via `fetch()` returns a Promise, and you can't throw-catch across an async boundary. The evaluator needs Promise-aware continuations.

**Approach:**

1. **Async-aware shift/reset** — extend the continuations extension:
   - `sfShift` captures the continuation and returns a Promise
   - `sfReset` awaits Promise results in the trampoline
   - Continuation resume feeds the fetched value back into the evaluation

2. **IO primitive bridge** — register async IO primitives in client `PRIMITIVES`:
   - `query` → fetch to `/internal/data/`
   - `service` → fetch to target service internal endpoint
   - `frag` → fetch fragment HTML
   - `current-user` → cached from initial page load

3. **CPS transform option** — alternative to Promise-aware shift/reset:
   - Transform the evaluator to continuation-passing style
   - Every eval step takes a continuation argument
   - IO primitives call the continuation after fetch resolves
   - Architecturally cleaner but requires deeper changes

**Depends on:** Phase 4 (data endpoint infrastructure)

**Verification:**
- Component calling `(query ...)` on client fetches data and renders
- Same component source → identical output on server and client
- Suspension visible: placeholder → resolved content

---

### Phase 6: Streaming & Suspense

**What it enables:** Server streams partially-evaluated SX as IO resolves. Client renders available subtrees immediately, fills in suspended parts. Like React Suspense but built on delimited continuations.

**Approach:**

1. **Continuation-based suspension** — when `_aser` encounters IO during slot evaluation, emit a placeholder with a suspension ID, schedule async resolution:
   ```python
   yield SxExpr(f'(~suspense :id "{placeholder_id}" :fallback (div "Loading..."))')
   schedule_fill(placeholder_id, io_coroutine)
   ```

2. **Chunked transfer** — Quart async generator responses:
   - First chunk: HTML shell + synchronous content + placeholders
   - Subsequent chunks: `<script>` tags replacing placeholders with resolved content

3. **Client suspension rendering** — `~suspense` component renders fallback, listens for resolution via inline script or SSE (existing SSE infrastructure in orchestration.sx).

4. **Priority-based IO** — above-fold content resolves first. All IO starts concurrently (`asyncio.create_task`), results flushed in priority order.

**Files:**
- `shared/sx/async_eval.py` — streaming `_aser` variant
- `shared/sx/helpers.py` — chunked response builder
- New: `shared/sx/ref/suspense.sx` — client suspension rendering
- `shared/sx/ref/boot.sx` — handle resolution scripts

**Depends on:** Phase 5 (async continuations for filling suspended subtrees), Phase 2 (IO analysis for priority)

---

### Phase 7: Full Isomorphism

**What it enables:** Same SX code runs on either side. Runtime chooses optimal split. Offline-first with cached data + client eval.

**Approach:**

1. **Runtime boundary optimizer** — given component tree + IO dependency graph, decide per-component: server-expand, client-render, or stream. Planning step cached at registration, recomputed on component change.

2. **Affinity annotations** — optional developer hints:
   ```lisp
   (defcomp ~product-grid (&key products)
     :affinity :client  ;; interactive, prefer client
     ...)
   (defcomp ~auth-menu (&key user)
     :affinity :server  ;; auth-sensitive, always server
     ...)
   ```
   Default: auto (runtime decides from IO analysis).

3. **Optimistic data updates** — extend existing `apply-optimistic`/`revert-optimistic` in `engine.sx` from DOM-level to data-level:
   - Client updates cached data optimistically (e.g., like button increments count)
   - Sends mutation to server
   - If server confirms, keep; if rejects, revert cached data and re-render

4. **Offline data layer** — Service Worker intercepts `/internal/data/` requests, serves from IndexedDB when offline, syncs when back online.

5. **Isomorphic testing** — evaluate same component on Python and JS, compare output. Extends existing `test_sx_ref.py` cross-evaluator comparison.

6. **Universal page descriptor** — `defpage` is portable: server executes via `execute_page()`, client executes via route match → fetch data → eval content → render DOM. Same descriptor, different execution environment.

**Depends on:** All previous phases.

---

## Cross-Cutting Concerns

### Error Reporting (all phases)
- Phase 1: "Unknown component" includes which page expected it and what bundle was sent
- Phase 2: Server logs which components expanded server-side vs sent to client
- Phase 3: Client route failures include unmatched path and available routes
- Phase 4: Client IO errors include query name, params, server response
- Source location tracking in parser → propagate through eval → include in error messages

### Backward Compatibility (all phases)
- Pages without annotations behave as today
- `SX-Request` / `SX-Components` / `SX-Css` header protocol continues
- Existing `.sx` files require no changes
- `_expand_components` continues as override
- Each phase is opt-in: disable → identical to previous behavior

### Spec Integrity
All new behavior specified in `.sx` files under `shared/sx/ref/` before implementation. Bootstrappers transpile from spec. This ensures JS and Python stay in sync.

## Critical Files

| File | Role | Phases |
|------|------|--------|
| `shared/sx/async_eval.py` | Core evaluator, `_aser`, server/client boundary | 2, 6 |
| `shared/sx/helpers.py` | `sx_page()`, `sx_response()`, output pipeline | 1, 3, 4 |
| `shared/sx/jinja_bridge.py` | `_COMPONENT_ENV`, component registry | 1, 2 |
| `shared/sx/pages.py` | `defpage`, `execute_page()`, page lifecycle, data endpoint | 2, 3, 4 |
| `shared/sx/ref/boot.sx` | Client boot, component caching, page registry | 1, 3, 4 |
| `shared/sx/ref/orchestration.sx` | Client fetch/swap/morph, routing, data cache | 3, 4, 5 |
| `shared/sx/ref/eval.sx` | Evaluator spec | 5 |
| `shared/sx/ref/engine.sx` | Morph, swaps, triggers | 3 |
| `shared/sx/ref/deps.sx` | Dependency + IO analysis (spec) | 1, 2 |
| `shared/sx/ref/router.sx` | Client-side route matching | 3 |
| `shared/sx/ref/bootstrap_js.py` | JS bootstrapper, platform implementations | 4, 5 |
| New: `shared/sx/ref/suspense.sx` | Streaming/suspension | 6 |