rose-ash/docs/isomorphic-sx-plan.md

SX Isomorphic Architecture Roadmap

Context

SX has a working server-client pipeline: server evaluates pages with IO (DB, fragments), serializes as SX wire format, client parses and renders to DOM. The language and primitives are already isomorphic — same spec, same semantics, both sides. What's missing is the plumbing that makes the boundary between server and client a sliding window rather than a fixed wall.

The key insight: s-expressions can partially unfold on the server after IO, then finish unfolding on the client. The system should be clever enough to know which downstream components have data fetches, resolve those server-side, and send the rest as pure SX for client rendering. Eventually, the client can also do IO (mapping server DB queries to REST calls), handle routing (SPA), and even work offline with cached data.

Current State (what's solid)

• Primitive parity: 100%. ~80 pure primitives, same names/semantics, JS and Python.
• eval/parse/render: Complete both sides. sx-ref.js has eval, parse, render-to-html, render-to-dom, aser.
• Engine: engine.sx (morph, swaps, triggers, history), orchestration.sx (fetch, events), boot.sx (hydration) — all transpiled.
• Wire format: Server _aser → SX source → client parses → renders to DOM. Boundary is clean.
• Component caching: Hash-based localStorage for component definitions and style dictionaries.
• CSS on-demand: CSSX resolves keywords to CSS rules, injects only used rules.
• Boundary enforcement: boundary.sx + SX_BOUNDARY_STRICT=1 validates all primitives/IO/helpers at registration.

Architecture Phases

---

Phase 1: Component Distribution & Dependency Analysis — DONE

What it enables: Per-page component bundles instead of sending every definition to every page. Smaller payloads, faster boot, better cache hit rates.

Implemented:

1. Transitive closure analyzer — shared/sx/deps.py (now shared/sx/ref/deps.sx, spec-level)

   • Walk component body AST, collect all ~name refs
   • Recursively follow into their bodies
   • Handle control forms (if/when/cond/case) — include ALL branches
   • components_needed(source, env) -> set[str]

2. IO reference analysis — deps.sx also tracks IO primitive usage

   • scan-io-refs / transitive-io-refs / component-pure?
   • Used by Phase 2 for automatic server/client boundary

3. Per-page component block — _build_pages_sx() in helpers.py

   • Each page entry includes :deps list of required components
   • Client page registry carries dep info for prefetching

4. SX partial responses — components_for_request() diffs against SX-Components header, sends only missing components

Files: shared/sx/ref/deps.sx, shared/sx/deps.py, shared/sx/helpers.py, shared/sx/jinja_bridge.py

---

Phase 2: Smart Server/Client Boundary — DONE

What it enables: Formalized partial evaluation model. Server evaluates IO, serializes pure subtrees. The system automatically knows "this component needs server data" vs "this component is pure and can render anywhere."

Implemented:

1. Automatic IO detection — deps.sx walks component bodies for IO primitive refs

   • compute-all-io-refs computes transitive IO analysis for all components
   • component-pure? returns true if no IO refs transitively

2. Selective expansion — _aser expands known components server-side via _aser_component

   • IO-dependent components expand server-side (IO must resolve)
   • Unknown components serialize for client rendering
   • _expand_components context var controls override

3. Component metadata — computed at registration, cached on Component objects

Files: shared/sx/ref/deps.sx, shared/sx/async_eval.py, shared/sx/jinja_bridge.py

---

Phase 3: Client-Side Routing (SPA Mode) — DONE

What it enables: After initial page load, client resolves routes locally using cached components. Only hits server for fresh data or unknown routes.

Implemented:

1. Client-side page registry — _build_pages_sx() serializes defpage routing info

   • <script type="text/sx-pages"> with name, path, auth, content, deps, closure, has-data
   • Processed by boot.sx → _page-routes list

2. Client route matcher — shared/sx/ref/router.sx

   • parse-route-pattern converts Flask-style /docs/<slug> to matchers
   • find-matching-route matches URL against registered routes
   • match-route-segments handles literal and param segments

3. Client-side route intercept — orchestration.sx

   • try-client-route — match URL, eval content locally, swap DOM
   • bind-client-route-link — intercept boost link clicks
   • Pure pages render immediately, no server roundtrip
   • Falls through to server fetch on miss

4. Integration with engine — boost link clicks try client route first, fall back to standard fetch

Files: shared/sx/ref/router.sx, shared/sx/ref/boot.sx, shared/sx/ref/orchestration.sx, shared/sx/helpers.py, shared/sx/pages.py

---

Phase 4: Client Async & IO Bridge — DONE

What it enables: Client fetches server-evaluated data and renders :data pages locally. Data cached to avoid redundant fetches on back/forward navigation.

The approach: Separate IO from rendering. Server evaluates :data expression (async, with DB/service access), serializes result as SX wire format. Client fetches this pre-evaluated data, parses it, merges into env, renders pure :content client-side. No continuations needed — all IO happens server-side.

Implemented:

1. Abstract resolve-page-data — spec-level primitive in orchestration.sx

   • (resolve-page-data name params callback) — platform decides transport
   • Spec says "I need data for this page"; platform provides concrete implementation
   • Browser platform: HTTP fetch to /sx/data/ endpoint

2. Server data endpoint — pages.py

   • evaluate_page_data() — evaluates :data expression, kebab-cases dict keys, serializes as SX
   • auto_mount_page_data() — mounts GET /sx/data/<page_name> endpoint
   • Per-page auth enforcement via _check_page_auth()
   • Response content type: text/sx; charset=utf-8

3. Client-side data rendering — orchestration.sx

   • try-client-route handles :data pages: fetch data → parse SX → merge into env → render content
   • Console log: sx:route client+data <pathname> confirms client-side rendering
   • Component deps computed for :data pages too (not just pure pages)

4. Client data cache — orchestration.sx

   • _page-data-cache dict keyed by page-name:param=value
   • 30s TTL (configurable via _page-data-cache-ttl)
   • Cache hit: sx:route client+cache <pathname> — renders instantly
   • Cache miss: fetches, caches, renders
   • Stale entries evicted on next access

5. Test page — sx/sx/data-test.sx

   • Exercises full data pipeline: server time, pipeline steps, phase/transport metadata
   • Navigate from another page → console shows sx:route client+data
   • Navigate back → console shows sx:route client+cache

6. Unit tests — shared/sx/tests/test_page_data.py (20 tests)

   • Serialize roundtrip for all data types
   • Kebab-case key conversion
   • Component deps for :data pages
   • Full pipeline simulation (serialize → parse → merge → eval)

Files:

• shared/sx/ref/orchestration.sx — resolve-page-data spec, data cache
• shared/sx/ref/bootstrap_js.py — platform resolvePageData implementation
• shared/sx/pages.py — evaluate_page_data(), auto_mount_page_data()
• shared/sx/helpers.py — deps for :data pages
• sx/sx/data-test.sx — test component
• sx/sxc/pages/docs.sx — test page defpage
• sx/sxc/pages/helpers.py — data-test-data helper
• sx/sx/boundary.sx — helper declaration
• shared/sx/tests/test_page_data.py — unit tests

---

Phase 5: Async Continuations & Inline IO

What it enables: Components call IO primitives directly in their body (e.g. (query ...)). The evaluator suspends mid-evaluation, fetches data, resumes. Same component source works on both server (Python async/await) and client (continuation-based suspension).

The problem: The existing shift/reset continuations extension is synchronous (throw/catch). Client-side IO via fetch() returns a Promise, and you can't throw-catch across an async boundary. The evaluator needs Promise-aware continuations.

Approach:

1. Async-aware shift/reset — extend the continuations extension:

   • sfShift captures the continuation and returns a Promise
   • sfReset awaits Promise results in the trampoline
   • Continuation resume feeds the fetched value back into the evaluation

2. IO primitive bridge — register async IO primitives in client PRIMITIVES:

   • query → fetch to /internal/data/
   • service → fetch to target service internal endpoint
   • frag → fetch fragment HTML
   • current-user → cached from initial page load

3. CPS transform option — alternative to Promise-aware shift/reset:

   • Transform the evaluator to continuation-passing style
   • Every eval step takes a continuation argument
   • IO primitives call the continuation after fetch resolves
   • Architecturally cleaner but requires deeper changes

Depends on: Phase 4 (data endpoint infrastructure)

Verification:

• Component calling (query ...) on client fetches data and renders
• Same component source → identical output on server and client
• Suspension visible: placeholder → resolved content

---

Phase 6: Streaming & Suspense

What it enables: Server streams partially-evaluated SX as IO resolves. Client renders available subtrees immediately, fills in suspended parts. Like React Suspense but built on delimited continuations.

Approach:

1. Continuation-based suspension — when _aser encounters IO during slot evaluation, emit a placeholder with a suspension ID, schedule async resolution:

   yield SxExpr(f'(~suspense :id "{placeholder_id}" :fallback (div "Loading..."))')
   schedule_fill(placeholder_id, io_coroutine)
   

2. Chunked transfer — Quart async generator responses:

   • First chunk: HTML shell + synchronous content + placeholders
   • Subsequent chunks: <script> tags replacing placeholders with resolved content

3. Client suspension rendering — ~suspense component renders fallback, listens for resolution via inline script or SSE (existing SSE infrastructure in orchestration.sx).

4. Priority-based IO — above-fold content resolves first. All IO starts concurrently (asyncio.create_task), results flushed in priority order.

Files:

• shared/sx/async_eval.py — streaming _aser variant
• shared/sx/helpers.py — chunked response builder
• New: shared/sx/ref/suspense.sx — client suspension rendering
• shared/sx/ref/boot.sx — handle resolution scripts

Depends on: Phase 5 (async continuations for filling suspended subtrees), Phase 2 (IO analysis for priority)

---

Phase 7: Full Isomorphism

What it enables: Same SX code runs on either side. Runtime chooses optimal split. Offline-first with cached data + client eval.

Approach:

1. Runtime boundary optimizer — given component tree + IO dependency graph, decide per-component: server-expand, client-render, or stream. Planning step cached at registration, recomputed on component change.

2. Affinity annotations — optional developer hints:

   (defcomp ~product-grid (&key products)
     :affinity :client  ;; interactive, prefer client
     ...)
   (defcomp ~auth-menu (&key user)
     :affinity :server  ;; auth-sensitive, always server
     ...)
   

   Default: auto (runtime decides from IO analysis).

3. Optimistic data updates — extend existing apply-optimistic/revert-optimistic in engine.sx from DOM-level to data-level:

   • Client updates cached data optimistically (e.g., like button increments count)
   • Sends mutation to server
   • If server confirms, keep; if rejects, revert cached data and re-render

4. Offline data layer — Service Worker intercepts /internal/data/ requests, serves from IndexedDB when offline, syncs when back online.

5. Isomorphic testing — evaluate same component on Python and JS, compare output. Extends existing test_sx_ref.py cross-evaluator comparison.

6. Universal page descriptor — defpage is portable: server executes via execute_page(), client executes via route match → fetch data → eval content → render DOM. Same descriptor, different execution environment.

Depends on: All previous phases.

---

Cross-Cutting Concerns

Error Reporting (all phases)

• Phase 1: "Unknown component" includes which page expected it and what bundle was sent
• Phase 2: Server logs which components expanded server-side vs sent to client
• Phase 3: Client route failures include unmatched path and available routes
• Phase 4: Client IO errors include query name, params, server response
• Source location tracking in parser → propagate through eval → include in error messages

Backward Compatibility (all phases)

• Pages without annotations behave as today
• SX-Request / SX-Components / SX-Css header protocol continues
• Existing .sx files require no changes
• _expand_components continues as override
• Each phase is opt-in: disable → identical to previous behavior

Spec Integrity

All new behavior specified in .sx files under shared/sx/ref/ before implementation. Bootstrappers transpile from spec. This ensures JS and Python stay in sync.

Critical Files

File	Role	Phases
shared/sx/async_eval.py	Core evaluator, _aser, server/client boundary	2, 6
shared/sx/helpers.py	sx_page(), sx_response(), output pipeline	1, 3, 4
shared/sx/jinja_bridge.py	_COMPONENT_ENV, component registry	1, 2
shared/sx/pages.py	defpage, execute_page(), page lifecycle, data endpoint	2, 3, 4
shared/sx/ref/boot.sx	Client boot, component caching, page registry	1, 3, 4
shared/sx/ref/orchestration.sx	Client fetch/swap/morph, routing, data cache	3, 4, 5
shared/sx/ref/eval.sx	Evaluator spec	5
shared/sx/ref/engine.sx	Morph, swaps, triggers	3
shared/sx/ref/deps.sx	Dependency + IO analysis (spec)	1, 2
shared/sx/ref/router.sx	Client-side route matching	3
shared/sx/ref/bootstrap_js.py	JS bootstrapper, platform implementations	4, 5
New: shared/sx/ref/suspense.sx	Streaming/suspension	6