rose-ash/docs/isomorphic-sx-plan.md

# Isomorphic SX Architecture Migration Plan

## Context

The sx layer already renders full pages client-side — `sx_page()` ships raw sx source + component definitions to the browser, `sx.js` evaluates and renders them. Components are cached in localStorage with a hash-based invalidation protocol (cookie `sx-comp-hash` → server skips sending defs if hash matches).

**Key insight from the user:** Pages/routes are just components. They belong in the same component registry, cached in localStorage alongside `defcomp` definitions. On navigation, if the client's component hash is current, the server doesn't need to send any s-expression source at all — just data. The client already has the page component cached and renders it locally with fresh data from the API.

### Target Architecture

```
First visit:
  Server → component defs (including page components) + page data → client caches defs in localStorage

Subsequent navigation (same session, hash valid):
  Client has page component cached → fetches only JSON data from /api/data/ → renders locally
  Server sends: { data: {...} } — zero sx source

SSR (bots, first paint):
  Server evaluates the same page component with direct DB queries → sends rendered HTML
  Client hydrates (binds SxEngine handlers, no re-render)
```

This is React-like data fetching with an s-expression view layer instead of JSX, and the component transport is a content-addressed cache rather than a JS bundle.

### Data Delivery Modes

The data side is not a single pattern — it's a spectrum that can be mixed per page and per fragment:

**Mode A: Server-bundled data** — Server evaluates the page's `:data` slot, resolves all queries (including cross-service `fetch_data` calls), returns one JSON blob. Fewest round-trips. Server aggregates.

**Mode B: Client-fetched data** — Client evaluates `:data` slot locally. Each `(query ...)` / `(service ...)` hits the relevant service's `/api/data/` endpoint independently. More round-trips but fully decoupled — each service handles its own data.

**Mode C: Hybrid** — Server bundles same-service data (direct DB). Client fetches cross-service data in parallel from other services' APIs. Mirrors current server pattern: own-domain = SQLAlchemy, cross-domain = `fetch_data()` HTTP.

The same spectrum applies to **fragments** (`frag` / `fetch_fragment`):

- **Server-composed:** Server calls `fetch_fragment()` during page evaluation, bakes result into data bundle or renders inline.
- **Client-composed:** Client's `(frag ...)` primitive fetches from the service's public fragment endpoint. Fragment returns sx source, client renders locally using cached component defs.
- **Mixed:** Stable fragments (nav, auth menu) server-composed; content-specific fragments client-fetched.

A `(query ...)` or `(frag ...)` call resolves differently depending on execution context (server vs client) but produces the same result. The choice of mode can be per-page, per-fragment, or even per-request.

## Delivery Order

```
Phase 1 (Primitive Parity) ──┐
                              ├── Phase 4 (Client Data Primitives) ──┐
Phase 3 (Public Data API) ───┘                                      ├── Phase 5 (Data-Only Navigation)
Phase 2 (Server-Side Rendering) ────────────────────────────────────┘
```

Phases 1-3 are independent. Recommended order: **3 → 1 → 2 → 4 → 5**

---

## Phase 1: Primitive Parity

Align JS and Python primitive sets so the same component source evaluates identically on both sides.

### 1a: Add missing pure primitives to sx.js

Add to `PRIMITIVES` in `shared/static/scripts/sx.js`:

| Primitive | JS implementation |
|-----------|-------------------|
| `clamp` | `Math.max(lo, Math.min(hi, x))` |
| `chunk-every` | partition list into n-size sublists |
| `zip-pairs` | `[[coll[0],coll[1]], [coll[2],coll[3]], ...]` |
| `dissoc` | shallow copy without specified keys |
| `into` | target-type-aware merge |
| `format-date` | minimal strftime translator covering `%Y %m %d %b %B %H %M %S` |
| `parse-int` | `parseInt` with NaN fallback to default |
| `assert` | throw if falsy |

Fix existing parity gaps: `round` needs optional `ndigits`; `min`/`max` need to accept a single list arg.

### 1b: Inject `window.__sxConfig` for server-context primitives

Modify `sx_page()` in `shared/sx/helpers.py` to inject before sx.js:

```js
window.__sxConfig = {
  appUrls: { blog: "https://blog.rose-ash.com", ... },
  assetUrl: "https://static...",
  config: { /* public subset */ },
  currentUser: { id, username, display_name, avatar } | null,
  relations: [ /* serialized RelationDef list */ ]
};
```

Sources: `ctx` has `blog_url`, `market_url`, etc. `g.user` has user info. `shared/infrastructure/urls.py` has the URL map.

Add JS primitives reading from `__sxConfig`: `app-url`, `asset-url`, `config`, `current-user`, `relations-from`.

`url-for` has no JS equivalent — isomorphic code uses `app-url` instead.

### 1c: Add `defpage` to sx.js evaluator

Add `defpage` to `SPECIAL_FORMS`. Parse the declaration, store it in `_componentEnv` under `"page:name"` (same registry as components). The page definition includes: name, path pattern, auth requirement, layout spec, and unevaluated AST for data/content/filter/aside/menu slots.

Since pages live in `_componentEnv`, they're automatically included in the component hash, cached in localStorage, and skipped when the hash matches. No separate `<script data-pages>` block needed — they ship with components.

**Files:** `shared/static/scripts/sx.js`, `shared/sx/helpers.py`

**Verify:** `(format-date "2024-03-15" "%d %b %Y")` produces same output in Python and JS.

---

## Phase 2: Server-Side Rendering (SSR)

Full-page HTML rendering on the server for SEO and first-paint.

### 2a: Add `render_mode` to `execute_page()`

In `shared/sx/pages.py`:

```python
async def execute_page(..., render_mode: str = "client") -> str:
```

When `render_mode="server"`:
- Evaluate all slots via `async_render()` (→ HTML) instead of `async_eval_to_sx()` (→ sx source)
- Layout headers also rendered to HTML
- Pass to new `ssr_page()` instead of `sx_page()`

### 2b: Create `ssr_page()` in helpers.py

Wraps pre-rendered HTML in a document shell:
- Same `<head>` (CSS, CSRF, meta)
- Rendered HTML inline in `<body>` — no `<script type="text/sx" data-mount>`
- Still ships component defs in `<script type="text/sx" data-components>` (client needs them for subsequent navigation)
- Still includes sx.js + body.js (for SPA takeover after first paint)
- Adds `<meta name="sx-ssr" content="true">`
- Injects `__sxConfig` (Phase 1b)

### 2c: SSR trigger

Utility `should_ssr(request)`:
- Bot UA patterns → SSR
- `?_render=server` → SSR (debug)
- `SX-Request: true` header → always client
- Per-page opt-in via `defpage :ssr true`
- Default → client (current behavior)

### 2d: Hydration in sx.js

When sx.js detects `<meta name="sx-ssr">`:
- Skip `Sx.mount()` — DOM already correct
- Run `SxEngine.process(document.body)` — bind sx-get/post handlers
- Run `Sx.hydrate()` — process `[data-sx]` elements
- Load component defs into registry (for subsequent navigations)

**Files:** `shared/sx/pages.py`, `shared/sx/helpers.py`, `shared/static/scripts/sx.js`

**Verify:** Googlebot UA → response has rendered HTML, no `<script data-mount>`. Normal UA → unchanged behavior.

---

## Phase 3: Public Data API

Expose browser-accessible JSON endpoints mirroring internal `/internal/data/` queries.

### 3a: Shared blueprint factory

New `shared/sx/api_data.py`:

```python
def create_public_data_blueprint(service_name: str) -> Blueprint:
    """Session-authed public data blueprint at /api/data/"""
```

Queries registered with auth level: `"public"`, `"login"`, `"admin"`. Validates session (not HMAC). Returns JSON.

### 3b: Extract and share handler implementations

Refactor `bp/data/routes.py` per service — separate query logic from HMAC auth. Same function serves both internal and public paths.

### 3c: Per-service public data blueprints

New `bp/api_data/routes.py` per service:

| Service | Public queries | Auth |
|---------|---------------|------|
| blog | `post-by-slug`, `post-by-id`, `search-posts` | public |
| market | `products-by-ids`, `marketplaces-for-container` | public |
| events | `visible-entries-for-period`, `calendars-for-container`, `entries-for-page` | public |
| cart | `cart-summary`, `cart-items` | login |
| likes | `is-liked`, `liked-slugs` | login |
| account | `newsletters` | public |

Admin queries and write-actions stay internal only.

### 3d: Public fragment endpoints

The existing internal fragment system (`/internal/fragments/<type>`, HMAC-signed) needs public equivalents. Each service already has `create_handler_blueprint()` mounting defhandler fragments. Add a parallel public endpoint:

`GET /api/fragments/<type>?params...` — session-authed, returns `text/sx` (same wire format the client already handles via SxEngine).

This can reuse the same `execute_handler()` machinery — the only difference is auth (session vs HMAC). The blueprint factory in `shared/sx/api_data.py` can handle both data and fragment registration:

```python
bp.register_fragment("container-cards", handler_fn, auth="public")
```

The client's `(frag ...)` primitive then fetches from these public endpoints instead of the HMAC-signed internal ones.

### 3e: Register in app factories

Each service's `app.py` registers the new blueprint.

**Files:** New `shared/sx/api_data.py`, new `{service}/bp/api_data/routes.py` per service, `{service}/app.py`

**Verify:** `curl /api/data/post-by-slug?slug=test` → JSON. `curl /api/fragments/container-cards?type=page&id=1` → sx source. Login-gated query without session → 401.

---

## Phase 4: Client Data Primitives

Async data-fetching in sx.js so I/O primitives work client-side via the public API.

### 4a: Async evaluator — `sxEvalAsync()`

New function in `sx.js` returning a `Promise`. Mirrors `async_eval.py`:
- Literals/symbols → `Promise.resolve(syncValue)`
- I/O primitives (`query`, `service`, `frag`, etc.) → `fetch()` calls to `/api/data/`
- Control flow → sequential async with short-circuit
- `map`/`filter` with I/O → `Promise.all`

### 4b: I/O primitive dispatch

```javascript
IO_PRIMITIVES = {
  "query":        (svc, name, kw) => fetch(__sxConfig.appUrls[svc] + "/api/data/" + name + "?" + params(kw), {credentials:"include"}).then(r=>r.json()),
  "service":      (method, kw)    => fetch("/api/data/" + method + "?" + params(kw), {credentials:"include"}).then(r=>r.json()),
  "frag":         (svc, type, kw) => fetch(__sxConfig.appUrls[svc] + "/api/fragments/" + type + "?" + params(kw), {credentials:"include"}).then(r=>r.text()),
  "current-user": ()              => Promise.resolve(__sxConfig.currentUser),
  "request-arg":  (name)          => Promise.resolve(new URLSearchParams(location.search).get(name)),
  "request-path": ()              => Promise.resolve(location.pathname),
  "nav-tree":     ()              => fetch("/api/data/nav-tree", {credentials:"include"}).then(r=>r.json()),
};
```

### 4c: Async DOM renderer — `renderDOMAsync()`

Two-pass (avoids restructuring sync renderer):
1. Walk AST, collect I/O call sites with placeholders
2. `Promise.all` to resolve all I/O in parallel
3. Substitute resolved values into AST
4. Call existing sync `renderDOM()` on resolved tree

### 4d: Wire into `Sx.mount()`

Detect I/O nodes. If present → async path. Otherwise → existing sync path (zero overhead for pure components).

**Files:** `shared/static/scripts/sx.js` (major addition)

**Verify:** Page with `(query "blog" "post-by-slug" :slug "test")` in sx source → client fetches `/api/data/post-by-slug?slug=test`, renders result.

---

## Phase 5: Data-Only Navigation

When the client already has page components cached, navigation requires only a data fetch — no sx source from the server.

### 5a: Page components in the registry

`defpage` definitions are already in `_componentEnv` (Phase 1c) and cached in localStorage with the component hash. On navigation, if the hash is valid, the client has all page definitions locally.

Build a `_pageRegistry` mapping URL path patterns → page definitions, populated when `defpage` forms are evaluated. Path patterns (`/posts/<slug>/`) converted to regex matchers for URL matching.

### 5b: Navigation intercept

Extend SxEngine's link click handler:

```
1. Extract URL path from clicked link
2. Match against _pageRegistry
3. If matched:
   a. Evaluate :data slot via sxEvalAsync() → parallel API fetches
   b. Render :content/:filter/:aside via renderDOMAsync()
   c. Morph into existing ~app-body (headers persist, slots update)
   d. Push history state
   e. Update document title
4. If not matched → existing server fetch (graceful fallback)
```

### 5c: Data delivery — flexible per page

Three modes available (see Context section). The page definition can declare its preference:

```scheme
(defpage blog-post
  :path "/posts/<slug>/"
  :data-mode :server    ; :server (bundled), :client (fetch individually), :hybrid
  :data (query "blog" "post-by-slug" :slug slug)
  :content (~post-detail post))
```

**Mode :server** — Client sends `SX-Page: blog-post` header on navigation. Server evaluates `:data` slot (all queries, including cross-service), returns single JSON blob:
```python
if request.headers.get("SX-Page"):
    data = await evaluate_data_slot(page_def, url_params)
    return jsonify(data)
```

**Mode :client** — Client evaluates `:data` slot locally via `sxEvalAsync()`. Each `(query ...)` hits `/api/data/` independently. Each `(frag ...)` hits `/api/fragments/`. No server data endpoint needed.

**Mode :hybrid** — Server bundles own-service data (direct DB). Client fetches cross-service data and fragments in parallel. The `:data` slot is split: server evaluates local queries, returns partial bundle + a manifest of remaining queries. Client resolves the rest.

Default mode can be `:server` (fewest round-trips, simplest). Pages opt into `:client` or `:hybrid` when they want more decoupling or when cross-service data is heavy and benefits from parallel client fetches.

### 5d: Popstate handling

On browser back/forward:
1. Check `_pageRegistry` for popped URL
2. If matched → client render (same as 5b)
3. If not → existing server fetch + morph

### 5e: Graceful fallback

Routes not in `_pageRegistry` fall through to server fetch. Partially migrated apps work — Python-only routes use server fetch, defpage routes get SPA behavior. No big-bang cutover.

**Files:** `shared/static/scripts/sx.js`, `shared/sx/helpers.py`, `shared/sx/pages.py`

**Verify:** Playwright: load page → click link to defpage route → assert no HTML response fetched (only JSON) → content correct → URL updated → back button works.

---

## Summary: The Full Lifecycle

```
1. App startup: Python loads .sx files → defcomp + defpage registered in _COMPONENT_ENV
                                        → hash computed

2. First visit:  Server sends HTML shell + component/page defs + __sxConfig + page sx source
                  Client evaluates, renders, caches defs in localStorage, sets cookie

3. Return visit: Cookie hash matches → server sends HTML shell with empty <script data-components>
                  Client loads defs from localStorage → renders page

4. SPA navigation: Client matches URL against _pageRegistry
                    → fetches data from /api/data/ (or server data-only endpoint)
                    → renders page component locally with fresh data
                    → morphs DOM, pushes history
                    → zero sx source transferred

5. Bot/SSR:      Server detects bot UA → evaluates page server-side with direct DB queries
                  → sends rendered HTML + component defs
                  → client hydrates (binds handlers, no re-render)
```

## Migration per Service

Each service migrates independently, no coordination needed:
1. Add public data blueprint (Phase 3) — immediate standalone value
2. Convert remaining Jinja routes to `defpage` — already in progress
3. Enable SSR for bots (Phase 2) — per-page opt-in
4. Client data primitives (Phase 4) — global once sx.js updated
5. Data-only navigation (Phase 5) — automatic for any `defpage` route

---

## Why: Architectural Rationale

The end state is: **sx.js is the only JavaScript in the browser.** All application code — components, pages, routing, event handling, data fetching — is expressed in sx, evaluated by the interpreter, with behavior mediated through bound primitives.

### Benefits

**Single language everywhere.** Components, pages, routing, event handling, data fetching — all sx. No context-switching between JS idioms and template syntax. One language for the entire frontend and the server rendering path.

**Portability.** The same source runs on any VM that implements the ~50-primitive interface. Today: Python + JS. Tomorrow: WASM, edge workers, native mobile, embedded devices. Coupled to a primitive contract, not to a specific runtime.

**Smaller wire transfer.** S-expressions are terser than equivalent JS. Combined with content-addressed caching (hash/localStorage), most navigations transfer zero code — just data.

**Inspectability.** The sx source is the running program — no build step, no source maps, no minification. View source shows exactly what executes. The AST is the structure the evaluator walks. Debugging is tracing a tree.

**Controlled surface area.** The only JS that runs is sx.js. Everything else is mediated through defined primitives. No npm supply chain. No third-party scripts with ambient DOM access. Components can only do what primitives allow — the capability surface is fully controlled.

**Hot-reloadable everything.** Components are data (cached AST). Swapping a definition is replacing a dict entry. No module system, no import graph, no HMR machinery. Already works for .sx file changes in dev mode — extends to behaviors too.

**AI-friendly.** S-expressions are trivially parseable and generatable. An LLM produces correct sx far more reliably than JS/JSX — fewer syntax edge cases, no semicolons/braces/arrow-function ambiguities. The codebase becomes more amenable to automated generation and transformation.

**Security boundary.** No `eval()`, no dynamic `<script>` injection, no prototype pollution. The sx evaluator is a sandbox — it only resolves symbols against the primitive table and component env. Auditing what any sx expression can do means auditing the primitive bindings.

### Performance and WASM

The tradeoff is interpreter overhead — a tree-walking interpreter is slower than native JS execution. For UI rendering (building DOM, handling events, fetching data), this is not the bottleneck — DOM operations dominate, and those are the same speed regardless of initiator.

If performance ever becomes a concern, WASM is the escape hatch at three levels:

1. **Evaluator in WASM.** Rewrite `sxEval` in Rust/Zig → WASM. The tight inner loop (symbol lookup, env traversal, function application) runs ~10-50x faster. DOM rendering stays in JS (it calls browser APIs regardless).

2. **Compile sx to WASM.** Ahead-of-time compiler: `.sx` → WASM modules. Each `defcomp` becomes a WASM function returning DOM instructions. Eliminates the interpreter entirely. The content-addressed cache stores compiled WASM blobs instead of sx source.

3. **Compute-heavy primitives in WASM.** Keep the sx interpreter in JS, bind specific primitives to WASM (image processing, crypto, data transformation). Most pragmatic and least disruptive — additive, no architecture change.

The primitive-binding model means the evaluator doesn't care what's behind a primitive. `(blur-image data radius)` could be a JS Canvas call today and a WASM JAX kernel tomorrow. The sx source doesn't change.

### Server-Driven by Default: The React Question

The sx system is architecturally aligned with HTMX/LiveView — server-driven UI — even though it does far more on the client (full s-expression evaluation, DOM rendering, morph reconciliation, component caching). The server is the single source of truth. Every UI state is a URL. Auth is enforced at render time. There are no state synchronization bugs because there is no client state to synchronize.

React's client-state model (`useState`, `useEffect`, Context, Suspense) exists because React was built for SPAs that need to feel like native apps — optimistic updates, offline capability, instant feedback without network latency. But it created an entire category of problems: state management libraries, hydration mismatches, cache invalidation, stale closures, memory leaks from forgotten cleanup, the `useEffect` footgun.

**The question is not "should sx have useState" — it's which specific interactions actually suffer from the server round-trip.**

For most of our apps, that's a very short list:
- Toggle a mobile nav panel
- Gallery image switching
- Quantity steppers
- Live search-as-you-type

These don't need a general-purpose reactive state system. They need **targeted client-side primitives** that handle those specific cases without abandoning the server-driven model.

**The dangerous path:** Add `useState` → need `useEffect` for cleanup → need Context to avoid prop drilling → need Suspense for async state → rebuild React inside sx → lose the simplicity that makes the server-driven model work.

**The careful path:** Keep server-driven as the default. Add explicit, targeted escape hatches for interactions that genuinely need client-side state. Make those escape hatches obviously different from the normal flow so they don't creep into everything.

#### What sx has vs React

| React feature | SX status | Verdict |
|---|---|---|
| Components + props | `defcomp` + `&key` | Done — cleaner than JSX |
| Fragments, conditionals, lists | `<>`, `if`/`when`/`cond`, `map` | Done — more expressive |
| Macros | `defmacro` | Done — React has nothing like this |
| OOB updates / portals | `sx-swap-oob` | Done — more powerful (server-driven) |
| DOM reconciliation | `_morphDOM` (id-keyed) | Done — works during SxEngine swaps |
| Reactive client state | None | **By design.** Server is source of truth. |
| Component lifecycle | None | Add targeted primitives if body.js behaviors move to sx |
| Context / providers | `_componentEnv` global | Sufficient for auth/theme; revisit if trees get deep |
| Suspense / loading | `sx-request` CSS class | Sufficient for server-driven; revisit for Phase 4 client data |
| Two-way data binding | None | Not needed — HTMX model (form POST → new HTML) works |
| Error boundaries | Global `sx:responseError` | Sufficient; per-component boundaries are a future nice-to-have |
| Keyed list reconciliation | id-based morph | Works; add `:key` prop support if list update bugs arise |

#### Targeted escape hatches (not a general state system)

For the few interactions that need client-side responsiveness, add **specific primitives** rather than a general framework:

- `(toggle! el "class")` — CSS class toggle, no server trip
- `(set-attr! el "attr" value)` — attribute manipulation
- `(on-event el "click" handler)` — declarative event binding within sx
- `(timer interval-ms handler)` — with automatic cleanup on DOM removal

These are imperative DOM operations exposed as primitives — not reactive state. They let components handle simple client-side interactions without importing React's entire mental model. The server-driven flow remains the default for anything involving data.