rose-ash/docs/isomorphic-sx-plan.md

# SX Isomorphic Architecture Roadmap

## Context

SX has a working server-client pipeline: server evaluates pages with IO (DB, fragments), serializes as SX wire format, client parses and renders to DOM. The language and primitives are already isomorphic — same spec, same semantics, both sides. What's missing is the **plumbing** that makes the boundary between server and client a sliding window rather than a fixed wall.

The key insight: **s-expressions can partially unfold on the server after IO, then finish unfolding on the client.** The system should be clever enough to know which downstream components have data fetches, resolve those server-side, and send the rest as pure SX for client rendering. Eventually, the client can also do IO (mapping server DB queries to REST calls), handle routing (SPA), and even work offline with cached data.

## Current State (what's solid)

- **Primitive parity:** 100%. ~80 pure primitives, same names/semantics, JS and Python.
- **eval/parse/render:** Complete both sides. sx-ref.js has eval, parse, render-to-html, render-to-dom, aser.
- **Engine:** engine.sx (morph, swaps, triggers, history), orchestration.sx (fetch, events), boot.sx (hydration) — all transpiled.
- **Wire format:** Server `_aser` → SX source → client parses → renders to DOM. Boundary is clean.
- **Component caching:** Hash-based localStorage for component definitions and style dictionaries.
- **CSS on-demand:** CSSX resolves keywords to CSS rules, injects only used rules.
- **Boundary enforcement:** `boundary.sx` + `SX_BOUNDARY_STRICT=1` validates all primitives/IO/helpers at registration.

## Architecture Phases

---

### Phase 1: Component Distribution & Dependency Analysis

**What it enables:** Per-page component bundles instead of sending every definition to every page. Smaller payloads, faster boot, better cache hit rates.

**The problem:** `client_components_tag()` in `shared/sx/jinja_bridge.py` serializes ALL entries in `_COMPONENT_ENV`. The `sx_page()` template sends everything or nothing based on a single global hash. No mechanism determines which components a page actually needs.

**Approach:**

1. **Transitive closure analyzer** — new module `shared/sx/deps.py`
   - Walk `Component.body` AST, collect all `Symbol` refs starting with `~`
   - Recursively follow into their bodies
   - Handle control forms (`if`/`when`/`cond`/`case`) — include ALL branches
   - Handle macros — expand during walk using limited eval
   - Function: `transitive_deps(name: str, env: dict) -> set[str]`
   - Cache result on `Component` object (invalidate on hot-reload)

2. **Runtime component scanning** — after `_aser` serializes page content, scan the SX string for `(~name` patterns (parallel to existing `scan_classes_from_sx` for CSS). Then compute transitive closure to get sub-components.

3. **Per-page component block** in `sx_page()` — replace all-or-nothing with page-specific bundle. Hash changes per page, localStorage cache keyed by route pattern.

4. **SX partial responses** — `components_for_request()` already diffs against `SX-Components` header. Enhance with transitive closure so only truly needed missing components are sent.

**Files:**
- New: `shared/sx/deps.py` — dependency analysis
- `shared/sx/jinja_bridge.py` — per-page bundle generation, cache deps on Component
- `shared/sx/helpers.py` — modify `sx_page()` and `sx_response()` for page-specific bundles
- `shared/sx/types.py` — add `deps: set[str]` to Component
- `shared/sx/ref/boot.sx` — per-page component caching alongside global cache

**Verification:**
- Page using 5/50 components → `data-components` block contains only those 5 + transitive deps
- No "Unknown component" errors after bundle reduction
- Payload size reduction measurable

---

### Phase 2: Smart Server/Client Boundary

**What it enables:** Formalized partial evaluation model. Server evaluates IO, serializes pure subtrees. The system automatically knows "this component needs server data" vs "this component is pure and can render anywhere."

**Current mechanism:** `_aser` in `async_eval.py` already does partial evaluation — IO primitives are awaited and substituted, HTML tags and component calls serialize as SX. The `_expand_components` context var controls expansion. But this is a global toggle, not per-component.

**Approach:**

1. **Automatic IO detection** — extend Phase 1 AST walker to check for references to `IO_PRIMITIVES` names (`frag`, `query`, `service`, `current-user`, etc.)
   - `has_io_deps(name: str, env: dict) -> bool`
   - Computed at registration time, cached on Component

2. **Component metadata** — enrich Component with analysis results:
   ```python
   ComponentMeta:
     deps: set[str]        # transitive component deps (Phase 1)
     io_refs: set[str]     # IO primitive names referenced
     is_pure: bool         # True if io_refs empty (transitively)
   ```

3. **Selective expansion** — refine `_aser` (line ~1335): instead of checking a global `_expand_components` flag, check the component's `is_pure` metadata:
   - IO-dependent → expand server-side (IO must resolve)
   - Pure → serialize for client (let client render)
   - Explicit override: `:server true` on defcomp forces server expansion

4. **Data manifest** for pages — `PageDef` produces a declaration of what IO the page needs, enabling Phase 3 (client can prefetch data) and Phase 5 (streaming).

**Files:**
- `shared/sx/deps.py` — add IO analysis
- `shared/sx/types.py` — add metadata fields to Component
- `shared/sx/async_eval.py` — refine `_aser` component expansion logic
- `shared/sx/jinja_bridge.py` — compute IO metadata at registration
- `shared/sx/pages.py` — data manifest on PageDef

**Verification:**
- Components calling `(query ...)` classified IO-dependent; pure components classified pure
- Existing pages produce identical output (regression)

---

### Phase 3: Client-Side Routing (SPA Mode)

**What it enables:** After initial page load, client resolves routes locally using cached components + data. Only hits server for fresh data or unknown routes. Like Next.js client-side navigation.

**Current mechanism:** All routing is server-side via `defpage` → Quart routes. Client navigates via `sx-boost` links doing `sx-get` + morphing. Every navigation = server roundtrip.

**Approach:**

1. **Client-side page registry** — serialize defpage routing info to client as `<script type="text/sx-pages">`:
   ```json
   {"docs-page": {"path": "/docs/:slug", "auth": "public",
                   "content": "(case slug ...)", "data": null}}
   ```
   Pure pages (no `:data`) can be evaluated entirely client-side.

2. **Client route matcher** — new spec file `shared/sx/ref/router.sx`:
   - Convert `/docs/<slug>` patterns to matchers
   - On boost-link click: match URL → if found and pure, evaluate locally
   - If IO needed: fetch data from server, evaluate content locally
   - No match: fall through to standard fetch (existing behavior)

3. **Data endpoint** — `GET /internal/page-data/<page-name>?<params>` returns JSON with evaluated `:data` expression. Reuses `execute_page()` logic but stops after `:data` step.

4. **Layout caching** — layouts depend on auth/fragments, so cache current layout and reuse across navigations. `SX-Layout-Hash` header tracks staleness.

5. **Integration with orchestration.sx** — intercept `bind-boost-link` to try client-side resolution first.

**Files:**
- `shared/sx/pages.py` — `serialize_for_client()`, data-only execution path
- `shared/sx/helpers.py` — include page registry in `sx_page()`
- New: `shared/sx/ref/router.sx` — client-side route matching
- `shared/sx/ref/boot.sx` — process `<script type="text/sx-pages">`
- `shared/sx/ref/orchestration.sx` — client-side route intercept
- Service blueprints — `/internal/page-data/` endpoint

**Depends on:** Phase 1 (client knows which components each page needs), Phase 2 (which pages are pure vs IO)

**Verification:**
- Pure page navigation: zero server requests
- IO page navigation: exactly one data request (not full page fetch)
- Browser back/forward works with client-resolved routes
- Disabling client registry → identical behavior to current

---

### Phase 4: Client Async & IO Bridge

**What it enables:** Client evaluates IO primitives by mapping them to server REST calls. Same SX code, different transport. `(query "market" "products" :ids "1,2,3")` on server → DB; on client → `fetch("/internal/data/products?ids=1,2,3")`.

**Approach:**

1. **Async client evaluator** — two possible mechanisms:
   - **Promise-based:** `evalExpr` returns value or Promise; rendering awaits
   - **Continuation-based:** use existing `shift/reset` to suspend on IO, resume when data arrives (architecturally cleaner, leverages existing spec)

2. **IO primitive bridge** — register async IO primitives in client `PRIMITIVES`:
   - `query` → fetch to `/internal/data/`
   - `service` → fetch to target service internal endpoint
   - `frag` → fetch fragment HTML
   - `current-user` → cached from initial page load

3. **Client data cache** — keyed by `(service, query, params-hash)`, configurable TTL, server can invalidate via `SX-Invalidate` header.

4. **Optimistic updates** — extend existing `apply-optimistic`/`revert-optimistic` in `engine.sx` from DOM-level to data-level.

**Files:**
- `shared/sx/ref/eval.sx` — async dispatch path (or new `async-eval.sx`)
- New: `shared/sx/ref/io-bridge.sx` — client IO implementations
- `shared/sx/ref/boot.sx` — register IO bridge at init
- `shared/sx/ref/bootstrap_js.py` — emit async-aware code
- `/internal/data/` endpoints — ensure client-accessible (CORS, auth)

**Depends on:** Phase 2 (IO affinity), Phase 3 (routing for when to trigger IO)

**Verification:**
- Client `(query ...)` returns identical data to server-side
- Data cache prevents redundant fetches
- Same component source → identical output on either side

---

### Phase 5: Streaming & Suspense

**What it enables:** Server streams partially-evaluated SX as IO resolves. Client renders available subtrees immediately, fills in suspended parts. Like React Suspense but built on delimited continuations.

**Approach:**

1. **Continuation-based suspension** — when `_aser` encounters IO during slot evaluation, emit a placeholder with a suspension ID, schedule async resolution:
   ```python
   yield SxExpr(f'(~suspense :id "{placeholder_id}" :fallback (div "Loading..."))')
   schedule_fill(placeholder_id, io_coroutine)
   ```

2. **Chunked transfer** — Quart async generator responses:
   - First chunk: HTML shell + synchronous content + placeholders
   - Subsequent chunks: `<script>` tags replacing placeholders with resolved content

3. **Client suspension rendering** — `~suspense` component renders fallback, listens for resolution via inline script or SSE (existing SSE infrastructure in orchestration.sx).

4. **Priority-based IO** — above-fold content resolves first. All IO starts concurrently (`asyncio.create_task`), results flushed in priority order.

**Files:**
- `shared/sx/async_eval.py` — streaming `_aser` variant
- `shared/sx/helpers.py` — chunked response builder
- New: `shared/sx/ref/suspense.sx` — client suspension rendering
- `shared/sx/ref/boot.sx` — handle resolution scripts

**Depends on:** Phase 4 (client async for filling suspended subtrees), Phase 2 (IO analysis for priority)

---

### Phase 6: Full Isomorphism

**What it enables:** Same SX code runs on either side. Runtime chooses optimal split. Offline-first with cached data + client eval.

**Approach:**

1. **Runtime boundary optimizer** — given component tree + IO dependency graph, decide per-component: server-expand, client-render, or stream. Planning step cached at registration, recomputed on component change.

2. **Affinity annotations** — optional developer hints:
   ```lisp
   (defcomp ~product-grid (&key products)
     :affinity :client  ;; interactive, prefer client
     ...)
   (defcomp ~auth-menu (&key user)
     :affinity :server  ;; auth-sensitive, always server
     ...)
   ```
   Default: auto (runtime decides from IO analysis).

3. **Offline data layer** — Service Worker intercepts `/internal/data/` requests, serves from IndexedDB when offline, syncs when back online.

4. **Isomorphic testing** — evaluate same component on Python and JS, compare output. Extends existing `test_sx_ref.py` cross-evaluator comparison.

5. **Universal page descriptor** — `defpage` is portable: server executes via `execute_page()`, client executes via route match → fetch data → eval content → render DOM. Same descriptor, different execution environment.

**Depends on:** All previous phases.

---

## Cross-Cutting Concerns

### Error Reporting (all phases)
- Phase 1: "Unknown component" includes which page expected it and what bundle was sent
- Phase 2: Server logs which components expanded server-side vs sent to client
- Phase 3: Client route failures include unmatched path and available routes
- Phase 4: Client IO errors include query name, params, server response
- Source location tracking in parser → propagate through eval → include in error messages

### Backward Compatibility (all phases)
- Pages without annotations behave as today
- `SX-Request` / `SX-Components` / `SX-Css` header protocol continues
- Existing `.sx` files require no changes
- `_expand_components` continues as override
- Each phase is opt-in: disable → identical to previous behavior

### Spec Integrity
All new behavior specified in `.sx` files under `shared/sx/ref/` before implementation. Bootstrappers transpile from spec. This ensures JS and Python stay in sync.

## Critical Files

| File | Role | Phases |
|------|------|--------|
| `shared/sx/async_eval.py` | Core evaluator, `_aser`, server/client boundary | 2, 5 |
| `shared/sx/helpers.py` | `sx_page()`, `sx_response()`, output pipeline | 1, 3 |
| `shared/sx/jinja_bridge.py` | `_COMPONENT_ENV`, component registry | 1, 2 |
| `shared/sx/pages.py` | `defpage`, `execute_page()`, page lifecycle | 2, 3 |
| `shared/sx/ref/boot.sx` | Client boot, component caching | 1, 3, 4 |
| `shared/sx/ref/orchestration.sx` | Client fetch/swap/morph | 3, 4 |
| `shared/sx/ref/eval.sx` | Evaluator spec | 4 |
| `shared/sx/ref/engine.sx` | Morph, swaps, triggers | 3 |
| New: `shared/sx/deps.py` | Dependency analysis | 1, 2 |
| New: `shared/sx/ref/router.sx` | Client-side routing | 3 |
| New: `shared/sx/ref/io-bridge.sx` | Client IO primitives | 4 |
| New: `shared/sx/ref/suspense.sx` | Streaming/suspension | 5 |