mono/docs/isomorphic-sx-plan.md

Isomorphic SX Architecture Migration Plan

Context

The sx layer already renders full pages client-side — sx_page() ships raw sx source + component definitions to the browser, sx.js evaluates and renders them. Components are cached in localStorage with a hash-based invalidation protocol (cookie sx-comp-hash → server skips sending defs if hash matches).

Key insight from the user: Pages/routes are just components. They belong in the same component registry, cached in localStorage alongside defcomp definitions. On navigation, if the client's component hash is current, the server doesn't need to send any s-expression source at all — just data. The client already has the page component cached and renders it locally with fresh data from the API.

Target Architecture

First visit:
  Server → component defs (including page components) + page data → client caches defs in localStorage

Subsequent navigation (same session, hash valid):
  Client has page component cached → fetches only JSON data from /api/data/ → renders locally
  Server sends: { data: {...} } — zero sx source

SSR (bots, first paint):
  Server evaluates the same page component with direct DB queries → sends rendered HTML
  Client hydrates (binds SxEngine handlers, no re-render)

This is React-like data fetching with an s-expression view layer instead of JSX, and the component transport is a content-addressed cache rather than a JS bundle.

Data Delivery Modes

The data side is not a single pattern — it's a spectrum that can be mixed per page and per fragment:

Mode A: Server-bundled data — Server evaluates the page's :data slot, resolves all queries (including cross-service fetch_data calls), returns one JSON blob. Fewest round-trips. Server aggregates.

Mode B: Client-fetched data — Client evaluates :data slot locally. Each (query ...) / (service ...) hits the relevant service's /api/data/ endpoint independently. More round-trips but fully decoupled — each service handles its own data.

Mode C: Hybrid — Server bundles same-service data (direct DB). Client fetches cross-service data in parallel from other services' APIs. Mirrors current server pattern: own-domain = SQLAlchemy, cross-domain = fetch_data() HTTP.

The same spectrum applies to fragments (frag / fetch_fragment):

• Server-composed: Server calls fetch_fragment() during page evaluation, bakes result into data bundle or renders inline.
• Client-composed: Client's (frag ...) primitive fetches from the service's public fragment endpoint. Fragment returns sx source, client renders locally using cached component defs.
• Mixed: Stable fragments (nav, auth menu) server-composed; content-specific fragments client-fetched.

A (query ...) or (frag ...) call resolves differently depending on execution context (server vs client) but produces the same result. The choice of mode can be per-page, per-fragment, or even per-request.

Delivery Order

Phase 1 (Primitive Parity) ──┐
                              ├── Phase 4 (Client Data Primitives) ──┐
Phase 3 (Public Data API) ───┘                                      ├── Phase 5 (Data-Only Navigation)
Phase 2 (Server-Side Rendering) ────────────────────────────────────┘

Phases 1-3 are independent. Recommended order: 3 → 1 → 2 → 4 → 5

---

Phase 1: Primitive Parity

Align JS and Python primitive sets so the same component source evaluates identically on both sides.

1a: Add missing pure primitives to sx.js

Add to PRIMITIVES in shared/static/scripts/sx.js:

Primitive	JS implementation
clamp	Math.max(lo, Math.min(hi, x))
chunk-every	partition list into n-size sublists
zip-pairs	[[coll[0],coll[1]], [coll[2],coll[3]], ...]
dissoc	shallow copy without specified keys
into	target-type-aware merge
format-date	minimal strftime translator covering %Y %m %d %b %B %H %M %S
parse-int	parseInt with NaN fallback to default
assert	throw if falsy

Fix existing parity gaps: round needs optional ndigits; min/max need to accept a single list arg.

1b: Inject window.__sxConfig for server-context primitives

Modify sx_page() in shared/sx/helpers.py to inject before sx.js:

window.__sxConfig = {
  appUrls: { blog: "https://blog.rose-ash.com", ... },
  assetUrl: "https://static...",
  config: { /* public subset */ },
  currentUser: { id, username, display_name, avatar } | null,
  relations: [ /* serialized RelationDef list */ ]
};

Sources: ctx has blog_url, market_url, etc. g.user has user info. shared/infrastructure/urls.py has the URL map.

Add JS primitives reading from __sxConfig: app-url, asset-url, config, current-user, relations-from.

url-for has no JS equivalent — isomorphic code uses app-url instead.

1c: Add defpage to sx.js evaluator

Add defpage to SPECIAL_FORMS. Parse the declaration, store it in _componentEnv under "page:name" (same registry as components). The page definition includes: name, path pattern, auth requirement, layout spec, and unevaluated AST for data/content/filter/aside/menu slots.

Since pages live in _componentEnv, they're automatically included in the component hash, cached in localStorage, and skipped when the hash matches. No separate <script data-pages> block needed — they ship with components.

Files: shared/static/scripts/sx.js, shared/sx/helpers.py

Verify: (format-date "2024-03-15" "%d %b %Y") produces same output in Python and JS.

---

Phase 2: Server-Side Rendering (SSR)

Full-page HTML rendering on the server for SEO and first-paint.

2a: Add render_mode to execute_page()

In shared/sx/pages.py:

async def execute_page(..., render_mode: str = "client") -> str:

When render_mode="server":

• Evaluate all slots via async_render() (→ HTML) instead of async_eval_to_sx() (→ sx source)
• Layout headers also rendered to HTML
• Pass to new ssr_page() instead of sx_page()

2b: Create ssr_page() in helpers.py

Wraps pre-rendered HTML in a document shell:

• Same <head> (CSS, CSRF, meta)
• Rendered HTML inline in <body> — no <script type="text/sx" data-mount>
• Still ships component defs in <script type="text/sx" data-components> (client needs them for subsequent navigation)
• Still includes sx.js + body.js (for SPA takeover after first paint)
• Adds <meta name="sx-ssr" content="true">
• Injects __sxConfig (Phase 1b)

2c: SSR trigger

Utility should_ssr(request):

• Bot UA patterns → SSR
• ?_render=server → SSR (debug)
• SX-Request: true header → always client
• Per-page opt-in via defpage :ssr true
• Default → client (current behavior)

2d: Hydration in sx.js

When sx.js detects <meta name="sx-ssr">:

• Skip Sx.mount() — DOM already correct
• Run SxEngine.process(document.body) — bind sx-get/post handlers
• Run Sx.hydrate() — process [data-sx] elements
• Load component defs into registry (for subsequent navigations)

Files: shared/sx/pages.py, shared/sx/helpers.py, shared/static/scripts/sx.js

Verify: Googlebot UA → response has rendered HTML, no <script data-mount>. Normal UA → unchanged behavior.

---

Phase 3: Public Data API

Expose browser-accessible JSON endpoints mirroring internal /internal/data/ queries.

3a: Shared blueprint factory

New shared/sx/api_data.py:

def create_public_data_blueprint(service_name: str) -> Blueprint:
    """Session-authed public data blueprint at /api/data/"""

Queries registered with auth level: "public", "login", "admin". Validates session (not HMAC). Returns JSON.

3b: Extract and share handler implementations

Refactor bp/data/routes.py per service — separate query logic from HMAC auth. Same function serves both internal and public paths.

3c: Per-service public data blueprints

New bp/api_data/routes.py per service:

Service	Public queries	Auth
blog	post-by-slug, post-by-id, search-posts	public
market	products-by-ids, marketplaces-for-container	public
events	visible-entries-for-period, calendars-for-container, entries-for-page	public
cart	cart-summary, cart-items	login
likes	is-liked, liked-slugs	login
account	newsletters	public

Admin queries and write-actions stay internal only.

3d: Public fragment endpoints

The existing internal fragment system (/internal/fragments/<type>, HMAC-signed) needs public equivalents. Each service already has create_handler_blueprint() mounting defhandler fragments. Add a parallel public endpoint:

GET /api/fragments/<type>?params... — session-authed, returns text/sx (same wire format the client already handles via SxEngine).

This can reuse the same execute_handler() machinery — the only difference is auth (session vs HMAC). The blueprint factory in shared/sx/api_data.py can handle both data and fragment registration:

bp.register_fragment("container-cards", handler_fn, auth="public")

The client's (frag ...) primitive then fetches from these public endpoints instead of the HMAC-signed internal ones.

3e: Register in app factories

Each service's app.py registers the new blueprint.

Files: New shared/sx/api_data.py, new {service}/bp/api_data/routes.py per service, {service}/app.py

Verify: curl /api/data/post-by-slug?slug=test → JSON. curl /api/fragments/container-cards?type=page&id=1 → sx source. Login-gated query without session → 401.

---

Phase 4: Client Data Primitives

Async data-fetching in sx.js so I/O primitives work client-side via the public API.

4a: Async evaluator — sxEvalAsync()

New function in sx.js returning a Promise. Mirrors async_eval.py:

• Literals/symbols → Promise.resolve(syncValue)
• I/O primitives (query, service, frag, etc.) → fetch() calls to /api/data/
• Control flow → sequential async with short-circuit
• map/filter with I/O → Promise.all

4b: I/O primitive dispatch

IO_PRIMITIVES = {
  "query":        (svc, name, kw) => fetch(__sxConfig.appUrls[svc] + "/api/data/" + name + "?" + params(kw), {credentials:"include"}).then(r=>r.json()),
  "service":      (method, kw)    => fetch("/api/data/" + method + "?" + params(kw), {credentials:"include"}).then(r=>r.json()),
  "frag":         (svc, type, kw) => fetch(__sxConfig.appUrls[svc] + "/api/fragments/" + type + "?" + params(kw), {credentials:"include"}).then(r=>r.text()),
  "current-user": ()              => Promise.resolve(__sxConfig.currentUser),
  "request-arg":  (name)          => Promise.resolve(new URLSearchParams(location.search).get(name)),
  "request-path": ()              => Promise.resolve(location.pathname),
  "nav-tree":     ()              => fetch("/api/data/nav-tree", {credentials:"include"}).then(r=>r.json()),
};

4c: Async DOM renderer — renderDOMAsync()

Two-pass (avoids restructuring sync renderer):

1. Walk AST, collect I/O call sites with placeholders
2. Promise.all to resolve all I/O in parallel
3. Substitute resolved values into AST
4. Call existing sync renderDOM() on resolved tree

4d: Wire into Sx.mount()

Detect I/O nodes. If present → async path. Otherwise → existing sync path (zero overhead for pure components).

Files: shared/static/scripts/sx.js (major addition)

Verify: Page with (query "blog" "post-by-slug" :slug "test") in sx source → client fetches /api/data/post-by-slug?slug=test, renders result.

---

Phase 5: Data-Only Navigation

When the client already has page components cached, navigation requires only a data fetch — no sx source from the server.

5a: Page components in the registry

defpage definitions are already in _componentEnv (Phase 1c) and cached in localStorage with the component hash. On navigation, if the hash is valid, the client has all page definitions locally.

Build a _pageRegistry mapping URL path patterns → page definitions, populated when defpage forms are evaluated. Path patterns (/posts/<slug>/) converted to regex matchers for URL matching.

5b: Navigation intercept

Extend SxEngine's link click handler:

1. Extract URL path from clicked link
2. Match against _pageRegistry
3. If matched:
   a. Evaluate :data slot via sxEvalAsync() → parallel API fetches
   b. Render :content/:filter/:aside via renderDOMAsync()
   c. Morph into existing ~app-body (headers persist, slots update)
   d. Push history state
   e. Update document title
4. If not matched → existing server fetch (graceful fallback)

5c: Data delivery — flexible per page

Three modes available (see Context section). The page definition can declare its preference:

(defpage blog-post
  :path "/posts/<slug>/"
  :data-mode :server    ; :server (bundled), :client (fetch individually), :hybrid
  :data (query "blog" "post-by-slug" :slug slug)
  :content (~post-detail post))

Mode :server — Client sends SX-Page: blog-post header on navigation. Server evaluates :data slot (all queries, including cross-service), returns single JSON blob:

if request.headers.get("SX-Page"):
    data = await evaluate_data_slot(page_def, url_params)
    return jsonify(data)

Mode :client — Client evaluates :data slot locally via sxEvalAsync(). Each (query ...) hits /api/data/ independently. Each (frag ...) hits /api/fragments/. No server data endpoint needed.

Mode :hybrid — Server bundles own-service data (direct DB). Client fetches cross-service data and fragments in parallel. The :data slot is split: server evaluates local queries, returns partial bundle + a manifest of remaining queries. Client resolves the rest.

Default mode can be :server (fewest round-trips, simplest). Pages opt into :client or :hybrid when they want more decoupling or when cross-service data is heavy and benefits from parallel client fetches.

5d: Popstate handling

On browser back/forward:

1. Check _pageRegistry for popped URL
2. If matched → client render (same as 5b)
3. If not → existing server fetch + morph

5e: Graceful fallback

Routes not in _pageRegistry fall through to server fetch. Partially migrated apps work — Python-only routes use server fetch, defpage routes get SPA behavior. No big-bang cutover.

Files: shared/static/scripts/sx.js, shared/sx/helpers.py, shared/sx/pages.py

Verify: Playwright: load page → click link to defpage route → assert no HTML response fetched (only JSON) → content correct → URL updated → back button works.

---

Summary: The Full Lifecycle

1. App startup: Python loads .sx files → defcomp + defpage registered in _COMPONENT_ENV
                                        → hash computed

2. First visit:  Server sends HTML shell + component/page defs + __sxConfig + page sx source
                  Client evaluates, renders, caches defs in localStorage, sets cookie

3. Return visit: Cookie hash matches → server sends HTML shell with empty <script data-components>
                  Client loads defs from localStorage → renders page

4. SPA navigation: Client matches URL against _pageRegistry
                    → fetches data from /api/data/ (or server data-only endpoint)
                    → renders page component locally with fresh data
                    → morphs DOM, pushes history
                    → zero sx source transferred

5. Bot/SSR:      Server detects bot UA → evaluates page server-side with direct DB queries
                  → sends rendered HTML + component defs
                  → client hydrates (binds handlers, no re-render)

Migration per Service

Each service migrates independently, no coordination needed:

1. Add public data blueprint (Phase 3) — immediate standalone value
2. Convert remaining Jinja routes to defpage — already in progress
3. Enable SSR for bots (Phase 2) — per-page opt-in
4. Client data primitives (Phase 4) — global once sx.js updated
5. Data-only navigation (Phase 5) — automatic for any defpage route

---

Why: Architectural Rationale

The end state is: sx.js is the only JavaScript in the browser. All application code — components, pages, routing, event handling, data fetching — is expressed in sx, evaluated by the interpreter, with behavior mediated through bound primitives.

Benefits

Single language everywhere. Components, pages, routing, event handling, data fetching — all sx. No context-switching between JS idioms and template syntax. One language for the entire frontend and the server rendering path.

Portability. The same source runs on any VM that implements the ~50-primitive interface. Today: Python + JS. Tomorrow: WASM, edge workers, native mobile, embedded devices. Coupled to a primitive contract, not to a specific runtime.

Smaller wire transfer. S-expressions are terser than equivalent JS. Combined with content-addressed caching (hash/localStorage), most navigations transfer zero code — just data.

Inspectability. The sx source is the running program — no build step, no source maps, no minification. View source shows exactly what executes. The AST is the structure the evaluator walks. Debugging is tracing a tree.

Controlled surface area. The only JS that runs is sx.js. Everything else is mediated through defined primitives. No npm supply chain. No third-party scripts with ambient DOM access. Components can only do what primitives allow — the capability surface is fully controlled.

Hot-reloadable everything. Components are data (cached AST). Swapping a definition is replacing a dict entry. No module system, no import graph, no HMR machinery. Already works for .sx file changes in dev mode — extends to behaviors too.

AI-friendly. S-expressions are trivially parseable and generatable. An LLM produces correct sx far more reliably than JS/JSX — fewer syntax edge cases, no semicolons/braces/arrow-function ambiguities. The codebase becomes more amenable to automated generation and transformation.

Security boundary. No eval(), no dynamic <script> injection, no prototype pollution. The sx evaluator is a sandbox — it only resolves symbols against the primitive table and component env. Auditing what any sx expression can do means auditing the primitive bindings.

Performance and WASM

The tradeoff is interpreter overhead — a tree-walking interpreter is slower than native JS execution. For UI rendering (building DOM, handling events, fetching data), this is not the bottleneck — DOM operations dominate, and those are the same speed regardless of initiator.

If performance ever becomes a concern, WASM is the escape hatch at three levels:

1. Evaluator in WASM. Rewrite sxEval in Rust/Zig → WASM. The tight inner loop (symbol lookup, env traversal, function application) runs ~10-50x faster. DOM rendering stays in JS (it calls browser APIs regardless).

2. Compile sx to WASM. Ahead-of-time compiler: .sx → WASM modules. Each defcomp becomes a WASM function returning DOM instructions. Eliminates the interpreter entirely. The content-addressed cache stores compiled WASM blobs instead of sx source.

3. Compute-heavy primitives in WASM. Keep the sx interpreter in JS, bind specific primitives to WASM (image processing, crypto, data transformation). Most pragmatic and least disruptive — additive, no architecture change.

The primitive-binding model means the evaluator doesn't care what's behind a primitive. (blur-image data radius) could be a JS Canvas call today and a WASM JAX kernel tomorrow. The sx source doesn't change.