Add Plans section to SX docs with isomorphic architecture roadmap
All checks were successful
Build and Deploy / build-and-deploy (push) Successful in 2m38s
All checks were successful
Build and Deploy / build-and-deploy (push) Successful in 2m38s
New top-level nav section at /plans/ with the 6-phase isomorphic architecture plan: component distribution, smart boundary, SPA routing, client IO bridge, streaming suspense, and full isomorphism. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,446 +1,273 @@
|
||||
# Isomorphic SX Architecture Migration Plan
|
||||
# SX Isomorphic Architecture Roadmap
|
||||
|
||||
## Context
|
||||
|
||||
The sx layer already renders full pages client-side — `sx_page()` ships raw sx source + component definitions to the browser, `sx.js` evaluates and renders them. Components are cached in localStorage with a hash-based invalidation protocol (cookie `sx-comp-hash` → server skips sending defs if hash matches).
|
||||
SX has a working server-client pipeline: server evaluates pages with IO (DB, fragments), serializes as SX wire format, client parses and renders to DOM. The language and primitives are already isomorphic — same spec, same semantics, both sides. What's missing is the **plumbing** that makes the boundary between server and client a sliding window rather than a fixed wall.
|
||||
|
||||
**Key insight from the user:** Pages/routes are just components. They belong in the same component registry, cached in localStorage alongside `defcomp` definitions. On navigation, if the client's component hash is current, the server doesn't need to send any s-expression source at all — just data. The client already has the page component cached and renders it locally with fresh data from the API.
|
||||
The key insight: **s-expressions can partially unfold on the server after IO, then finish unfolding on the client.** The system should be clever enough to know which downstream components have data fetches, resolve those server-side, and send the rest as pure SX for client rendering. Eventually, the client can also do IO (mapping server DB queries to REST calls), handle routing (SPA), and even work offline with cached data.
|
||||
|
||||
### Target Architecture
|
||||
## Current State (what's solid)
|
||||
|
||||
```
|
||||
First visit:
|
||||
Server → component defs (including page components) + page data → client caches defs in localStorage
|
||||
- **Primitive parity:** 100%. ~80 pure primitives, same names/semantics, JS and Python.
|
||||
- **eval/parse/render:** Complete both sides. sx-ref.js has eval, parse, render-to-html, render-to-dom, aser.
|
||||
- **Engine:** engine.sx (morph, swaps, triggers, history), orchestration.sx (fetch, events), boot.sx (hydration) — all transpiled.
|
||||
- **Wire format:** Server `_aser` → SX source → client parses → renders to DOM. Boundary is clean.
|
||||
- **Component caching:** Hash-based localStorage for component definitions and style dictionaries.
|
||||
- **CSS on-demand:** CSSX resolves keywords to CSS rules, injects only used rules.
|
||||
- **Boundary enforcement:** `boundary.sx` + `SX_BOUNDARY_STRICT=1` validates all primitives/IO/helpers at registration.
|
||||
|
||||
Subsequent navigation (same session, hash valid):
|
||||
Client has page component cached → fetches only JSON data from /api/data/ → renders locally
|
||||
Server sends: { data: {...} } — zero sx source
|
||||
|
||||
SSR (bots, first paint):
|
||||
Server evaluates the same page component with direct DB queries → sends rendered HTML
|
||||
Client hydrates (binds SxEngine handlers, no re-render)
|
||||
```
|
||||
|
||||
This is React-like data fetching with an s-expression view layer instead of JSX, and the component transport is a content-addressed cache rather than a JS bundle.
|
||||
|
||||
### Data Delivery Modes
|
||||
|
||||
The data side is not a single pattern — it's a spectrum that can be mixed per page and per fragment:
|
||||
|
||||
**Mode A: Server-bundled data** — Server evaluates the page's `:data` slot, resolves all queries (including cross-service `fetch_data` calls), returns one JSON blob. Fewest round-trips. Server aggregates.
|
||||
|
||||
**Mode B: Client-fetched data** — Client evaluates `:data` slot locally. Each `(query ...)` / `(service ...)` hits the relevant service's `/api/data/` endpoint independently. More round-trips but fully decoupled — each service handles its own data.
|
||||
|
||||
**Mode C: Hybrid** — Server bundles same-service data (direct DB). Client fetches cross-service data in parallel from other services' APIs. Mirrors current server pattern: own-domain = SQLAlchemy, cross-domain = `fetch_data()` HTTP.
|
||||
|
||||
The same spectrum applies to **fragments** (`frag` / `fetch_fragment`):
|
||||
|
||||
- **Server-composed:** Server calls `fetch_fragment()` during page evaluation, bakes result into data bundle or renders inline.
|
||||
- **Client-composed:** Client's `(frag ...)` primitive fetches from the service's public fragment endpoint. Fragment returns sx source, client renders locally using cached component defs.
|
||||
- **Mixed:** Stable fragments (nav, auth menu) server-composed; content-specific fragments client-fetched.
|
||||
|
||||
A `(query ...)` or `(frag ...)` call resolves differently depending on execution context (server vs client) but produces the same result. The choice of mode can be per-page, per-fragment, or even per-request.
|
||||
|
||||
## Delivery Order
|
||||
|
||||
```
|
||||
Phase 1 (Primitive Parity) ──┐
|
||||
├── Phase 4 (Client Data Primitives) ──┐
|
||||
Phase 3 (Public Data API) ───┘ ├── Phase 5 (Data-Only Navigation)
|
||||
Phase 2 (Server-Side Rendering) ────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Phases 1-3 are independent. Recommended order: **3 → 1 → 2 → 4 → 5**
|
||||
## Architecture Phases
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Primitive Parity
|
||||
### Phase 1: Component Distribution & Dependency Analysis
|
||||
|
||||
Align JS and Python primitive sets so the same component source evaluates identically on both sides.
|
||||
**What it enables:** Per-page component bundles instead of sending every definition to every page. Smaller payloads, faster boot, better cache hit rates.
|
||||
|
||||
### 1a: Add missing pure primitives to sx.js
|
||||
**The problem:** `client_components_tag()` in `shared/sx/jinja_bridge.py` serializes ALL entries in `_COMPONENT_ENV`. The `sx_page()` template sends everything or nothing based on a single global hash. No mechanism determines which components a page actually needs.
|
||||
|
||||
Add to `PRIMITIVES` in `shared/static/scripts/sx.js`:
|
||||
**Approach:**
|
||||
|
||||
| Primitive | JS implementation |
|
||||
|-----------|-------------------|
|
||||
| `clamp` | `Math.max(lo, Math.min(hi, x))` |
|
||||
| `chunk-every` | partition list into n-size sublists |
|
||||
| `zip-pairs` | `[[coll[0],coll[1]], [coll[2],coll[3]], ...]` |
|
||||
| `dissoc` | shallow copy without specified keys |
|
||||
| `into` | target-type-aware merge |
|
||||
| `format-date` | minimal strftime translator covering `%Y %m %d %b %B %H %M %S` |
|
||||
| `parse-int` | `parseInt` with NaN fallback to default |
|
||||
| `assert` | throw if falsy |
|
||||
1. **Transitive closure analyzer** — new module `shared/sx/deps.py`
|
||||
- Walk `Component.body` AST, collect all `Symbol` refs starting with `~`
|
||||
- Recursively follow into their bodies
|
||||
- Handle control forms (`if`/`when`/`cond`/`case`) — include ALL branches
|
||||
- Handle macros — expand during walk using limited eval
|
||||
- Function: `transitive_deps(name: str, env: dict) -> set[str]`
|
||||
- Cache result on `Component` object (invalidate on hot-reload)
|
||||
|
||||
Fix existing parity gaps: `round` needs optional `ndigits`; `min`/`max` need to accept a single list arg.
|
||||
2. **Runtime component scanning** — after `_aser` serializes page content, scan the SX string for `(~name` patterns (parallel to existing `scan_classes_from_sx` for CSS). Then compute transitive closure to get sub-components.
|
||||
|
||||
### 1b: Inject `window.__sxConfig` for server-context primitives
|
||||
3. **Per-page component block** in `sx_page()` — replace all-or-nothing with page-specific bundle. Hash changes per page, localStorage cache keyed by route pattern.
|
||||
|
||||
Modify `sx_page()` in `shared/sx/helpers.py` to inject before sx.js:
|
||||
4. **SX partial responses** — `components_for_request()` already diffs against `SX-Components` header. Enhance with transitive closure so only truly needed missing components are sent.
|
||||
|
||||
```js
|
||||
window.__sxConfig = {
|
||||
appUrls: { blog: "https://blog.rose-ash.com", ... },
|
||||
assetUrl: "https://static...",
|
||||
config: { /* public subset */ },
|
||||
currentUser: { id, username, display_name, avatar } | null,
|
||||
relations: [ /* serialized RelationDef list */ ]
|
||||
};
|
||||
```
|
||||
**Files:**
|
||||
- New: `shared/sx/deps.py` — dependency analysis
|
||||
- `shared/sx/jinja_bridge.py` — per-page bundle generation, cache deps on Component
|
||||
- `shared/sx/helpers.py` — modify `sx_page()` and `sx_response()` for page-specific bundles
|
||||
- `shared/sx/types.py` — add `deps: set[str]` to Component
|
||||
- `shared/sx/ref/boot.sx` — per-page component caching alongside global cache
|
||||
|
||||
Sources: `ctx` has `blog_url`, `market_url`, etc. `g.user` has user info. `shared/infrastructure/urls.py` has the URL map.
|
||||
|
||||
Add JS primitives reading from `__sxConfig`: `app-url`, `asset-url`, `config`, `current-user`, `relations-from`.
|
||||
|
||||
`url-for` has no JS equivalent — isomorphic code uses `app-url` instead.
|
||||
|
||||
### 1c: Add `defpage` to sx.js evaluator
|
||||
|
||||
Add `defpage` to `SPECIAL_FORMS`. Parse the declaration, store it in `_componentEnv` under `"page:name"` (same registry as components). The page definition includes: name, path pattern, auth requirement, layout spec, and unevaluated AST for data/content/filter/aside/menu slots.
|
||||
|
||||
Since pages live in `_componentEnv`, they're automatically included in the component hash, cached in localStorage, and skipped when the hash matches. No separate `<script data-pages>` block needed — they ship with components.
|
||||
|
||||
**Files:** `shared/static/scripts/sx.js`, `shared/sx/helpers.py`
|
||||
|
||||
**Verify:** `(format-date "2024-03-15" "%d %b %Y")` produces same output in Python and JS.
|
||||
**Verification:**
|
||||
- Page using 5/50 components → `data-components` block contains only those 5 + transitive deps
|
||||
- No "Unknown component" errors after bundle reduction
|
||||
- Payload size reduction measurable
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Server-Side Rendering (SSR)
|
||||
### Phase 2: Smart Server/Client Boundary
|
||||
|
||||
Full-page HTML rendering on the server for SEO and first-paint.
|
||||
**What it enables:** Formalized partial evaluation model. Server evaluates IO, serializes pure subtrees. The system automatically knows "this component needs server data" vs "this component is pure and can render anywhere."
|
||||
|
||||
### 2a: Add `render_mode` to `execute_page()`
|
||||
**Current mechanism:** `_aser` in `async_eval.py` already does partial evaluation — IO primitives are awaited and substituted, HTML tags and component calls serialize as SX. The `_expand_components` context var controls expansion. But this is a global toggle, not per-component.
|
||||
|
||||
In `shared/sx/pages.py`:
|
||||
**Approach:**
|
||||
|
||||
```python
|
||||
async def execute_page(..., render_mode: str = "client") -> str:
|
||||
```
|
||||
1. **Automatic IO detection** — extend Phase 1 AST walker to check for references to `IO_PRIMITIVES` names (`frag`, `query`, `service`, `current-user`, etc.)
|
||||
- `has_io_deps(name: str, env: dict) -> bool`
|
||||
- Computed at registration time, cached on Component
|
||||
|
||||
When `render_mode="server"`:
|
||||
- Evaluate all slots via `async_render()` (→ HTML) instead of `async_eval_to_sx()` (→ sx source)
|
||||
- Layout headers also rendered to HTML
|
||||
- Pass to new `ssr_page()` instead of `sx_page()`
|
||||
2. **Component metadata** — enrich Component with analysis results:
|
||||
```python
|
||||
ComponentMeta:
|
||||
deps: set[str] # transitive component deps (Phase 1)
|
||||
io_refs: set[str] # IO primitive names referenced
|
||||
is_pure: bool # True if io_refs empty (transitively)
|
||||
```
|
||||
|
||||
### 2b: Create `ssr_page()` in helpers.py
|
||||
3. **Selective expansion** — refine `_aser` (line ~1335): instead of checking a global `_expand_components` flag, check the component's `is_pure` metadata:
|
||||
- IO-dependent → expand server-side (IO must resolve)
|
||||
- Pure → serialize for client (let client render)
|
||||
- Explicit override: `:server true` on defcomp forces server expansion
|
||||
|
||||
Wraps pre-rendered HTML in a document shell:
|
||||
- Same `<head>` (CSS, CSRF, meta)
|
||||
- Rendered HTML inline in `<body>` — no `<script type="text/sx" data-mount>`
|
||||
- Still ships component defs in `<script type="text/sx" data-components>` (client needs them for subsequent navigation)
|
||||
- Still includes sx.js + body.js (for SPA takeover after first paint)
|
||||
- Adds `<meta name="sx-ssr" content="true">`
|
||||
- Injects `__sxConfig` (Phase 1b)
|
||||
4. **Data manifest** for pages — `PageDef` produces a declaration of what IO the page needs, enabling Phase 3 (client can prefetch data) and Phase 5 (streaming).
|
||||
|
||||
### 2c: SSR trigger
|
||||
**Files:**
|
||||
- `shared/sx/deps.py` — add IO analysis
|
||||
- `shared/sx/types.py` — add metadata fields to Component
|
||||
- `shared/sx/async_eval.py` — refine `_aser` component expansion logic
|
||||
- `shared/sx/jinja_bridge.py` — compute IO metadata at registration
|
||||
- `shared/sx/pages.py` — data manifest on PageDef
|
||||
|
||||
Utility `should_ssr(request)`:
|
||||
- Bot UA patterns → SSR
|
||||
- `?_render=server` → SSR (debug)
|
||||
- `SX-Request: true` header → always client
|
||||
- Per-page opt-in via `defpage :ssr true`
|
||||
- Default → client (current behavior)
|
||||
|
||||
### 2d: Hydration in sx.js
|
||||
|
||||
When sx.js detects `<meta name="sx-ssr">`:
|
||||
- Skip `Sx.mount()` — DOM already correct
|
||||
- Run `SxEngine.process(document.body)` — bind sx-get/post handlers
|
||||
- Run `Sx.hydrate()` — process `[data-sx]` elements
|
||||
- Load component defs into registry (for subsequent navigations)
|
||||
|
||||
**Files:** `shared/sx/pages.py`, `shared/sx/helpers.py`, `shared/static/scripts/sx.js`
|
||||
|
||||
**Verify:** Googlebot UA → response has rendered HTML, no `<script data-mount>`. Normal UA → unchanged behavior.
|
||||
**Verification:**
|
||||
- Components calling `(query ...)` classified IO-dependent; pure components classified pure
|
||||
- Existing pages produce identical output (regression)
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Public Data API
|
||||
### Phase 3: Client-Side Routing (SPA Mode)
|
||||
|
||||
Expose browser-accessible JSON endpoints mirroring internal `/internal/data/` queries.
|
||||
**What it enables:** After initial page load, client resolves routes locally using cached components + data. Only hits server for fresh data or unknown routes. Like Next.js client-side navigation.
|
||||
|
||||
### 3a: Shared blueprint factory
|
||||
**Current mechanism:** All routing is server-side via `defpage` → Quart routes. Client navigates via `sx-boost` links doing `sx-get` + morphing. Every navigation = server roundtrip.
|
||||
|
||||
New `shared/sx/api_data.py`:
|
||||
**Approach:**
|
||||
|
||||
```python
|
||||
def create_public_data_blueprint(service_name: str) -> Blueprint:
|
||||
"""Session-authed public data blueprint at /api/data/"""
|
||||
```
|
||||
1. **Client-side page registry** — serialize defpage routing info to client as `<script type="text/sx-pages">`:
|
||||
```json
|
||||
{"docs-page": {"path": "/docs/:slug", "auth": "public",
|
||||
"content": "(case slug ...)", "data": null}}
|
||||
```
|
||||
Pure pages (no `:data`) can be evaluated entirely client-side.
|
||||
|
||||
Queries registered with auth level: `"public"`, `"login"`, `"admin"`. Validates session (not HMAC). Returns JSON.
|
||||
2. **Client route matcher** — new spec file `shared/sx/ref/router.sx`:
|
||||
- Convert `/docs/<slug>` patterns to matchers
|
||||
- On boost-link click: match URL → if found and pure, evaluate locally
|
||||
- If IO needed: fetch data from server, evaluate content locally
|
||||
- No match: fall through to standard fetch (existing behavior)
|
||||
|
||||
### 3b: Extract and share handler implementations
|
||||
3. **Data endpoint** — `GET /internal/page-data/<page-name>?<params>` returns JSON with evaluated `:data` expression. Reuses `execute_page()` logic but stops after `:data` step.
|
||||
|
||||
Refactor `bp/data/routes.py` per service — separate query logic from HMAC auth. Same function serves both internal and public paths.
|
||||
4. **Layout caching** — layouts depend on auth/fragments, so cache current layout and reuse across navigations. `SX-Layout-Hash` header tracks staleness.
|
||||
|
||||
### 3c: Per-service public data blueprints
|
||||
5. **Integration with orchestration.sx** — intercept `bind-boost-link` to try client-side resolution first.
|
||||
|
||||
New `bp/api_data/routes.py` per service:
|
||||
**Files:**
|
||||
- `shared/sx/pages.py` — `serialize_for_client()`, data-only execution path
|
||||
- `shared/sx/helpers.py` — include page registry in `sx_page()`
|
||||
- New: `shared/sx/ref/router.sx` — client-side route matching
|
||||
- `shared/sx/ref/boot.sx` — process `<script type="text/sx-pages">`
|
||||
- `shared/sx/ref/orchestration.sx` — client-side route intercept
|
||||
- Service blueprints — `/internal/page-data/` endpoint
|
||||
|
||||
| Service | Public queries | Auth |
|
||||
|---------|---------------|------|
|
||||
| blog | `post-by-slug`, `post-by-id`, `search-posts` | public |
|
||||
| market | `products-by-ids`, `marketplaces-for-container` | public |
|
||||
| events | `visible-entries-for-period`, `calendars-for-container`, `entries-for-page` | public |
|
||||
| cart | `cart-summary`, `cart-items` | login |
|
||||
| likes | `is-liked`, `liked-slugs` | login |
|
||||
| account | `newsletters` | public |
|
||||
**Depends on:** Phase 1 (client knows which components each page needs), Phase 2 (which pages are pure vs IO)
|
||||
|
||||
Admin queries and write-actions stay internal only.
|
||||
|
||||
### 3d: Public fragment endpoints
|
||||
|
||||
The existing internal fragment system (`/internal/fragments/<type>`, HMAC-signed) needs public equivalents. Each service already has `create_handler_blueprint()` mounting defhandler fragments. Add a parallel public endpoint:
|
||||
|
||||
`GET /api/fragments/<type>?params...` — session-authed, returns `text/sx` (same wire format the client already handles via SxEngine).
|
||||
|
||||
This can reuse the same `execute_handler()` machinery — the only difference is auth (session vs HMAC). The blueprint factory in `shared/sx/api_data.py` can handle both data and fragment registration:
|
||||
|
||||
```python
|
||||
bp.register_fragment("container-cards", handler_fn, auth="public")
|
||||
```
|
||||
|
||||
The client's `(frag ...)` primitive then fetches from these public endpoints instead of the HMAC-signed internal ones.
|
||||
|
||||
### 3e: Register in app factories
|
||||
|
||||
Each service's `app.py` registers the new blueprint.
|
||||
|
||||
**Files:** New `shared/sx/api_data.py`, new `{service}/bp/api_data/routes.py` per service, `{service}/app.py`
|
||||
|
||||
**Verify:** `curl /api/data/post-by-slug?slug=test` → JSON. `curl /api/fragments/container-cards?type=page&id=1` → sx source. Login-gated query without session → 401.
|
||||
**Verification:**
|
||||
- Pure page navigation: zero server requests
|
||||
- IO page navigation: exactly one data request (not full page fetch)
|
||||
- Browser back/forward works with client-resolved routes
|
||||
- Disabling client registry → identical behavior to current
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Client Data Primitives
|
||||
### Phase 4: Client Async & IO Bridge
|
||||
|
||||
Async data-fetching in sx.js so I/O primitives work client-side via the public API.
|
||||
**What it enables:** Client evaluates IO primitives by mapping them to server REST calls. Same SX code, different transport. `(query "market" "products" :ids "1,2,3")` on server → DB; on client → `fetch("/internal/data/products?ids=1,2,3")`.
|
||||
|
||||
### 4a: Async evaluator — `sxEvalAsync()`
|
||||
**Approach:**
|
||||
|
||||
New function in `sx.js` returning a `Promise`. Mirrors `async_eval.py`:
|
||||
- Literals/symbols → `Promise.resolve(syncValue)`
|
||||
- I/O primitives (`query`, `service`, `frag`, etc.) → `fetch()` calls to `/api/data/`
|
||||
- Control flow → sequential async with short-circuit
|
||||
- `map`/`filter` with I/O → `Promise.all`
|
||||
1. **Async client evaluator** — two possible mechanisms:
|
||||
- **Promise-based:** `evalExpr` returns value or Promise; rendering awaits
|
||||
- **Continuation-based:** use existing `shift/reset` to suspend on IO, resume when data arrives (architecturally cleaner, leverages existing spec)
|
||||
|
||||
### 4b: I/O primitive dispatch
|
||||
2. **IO primitive bridge** — register async IO primitives in client `PRIMITIVES`:
|
||||
- `query` → fetch to `/internal/data/`
|
||||
- `service` → fetch to target service internal endpoint
|
||||
- `frag` → fetch fragment HTML
|
||||
- `current-user` → cached from initial page load
|
||||
|
||||
```javascript
|
||||
IO_PRIMITIVES = {
|
||||
"query": (svc, name, kw) => fetch(__sxConfig.appUrls[svc] + "/api/data/" + name + "?" + params(kw), {credentials:"include"}).then(r=>r.json()),
|
||||
"service": (method, kw) => fetch("/api/data/" + method + "?" + params(kw), {credentials:"include"}).then(r=>r.json()),
|
||||
"frag": (svc, type, kw) => fetch(__sxConfig.appUrls[svc] + "/api/fragments/" + type + "?" + params(kw), {credentials:"include"}).then(r=>r.text()),
|
||||
"current-user": () => Promise.resolve(__sxConfig.currentUser),
|
||||
"request-arg": (name) => Promise.resolve(new URLSearchParams(location.search).get(name)),
|
||||
"request-path": () => Promise.resolve(location.pathname),
|
||||
"nav-tree": () => fetch("/api/data/nav-tree", {credentials:"include"}).then(r=>r.json()),
|
||||
};
|
||||
```
|
||||
3. **Client data cache** — keyed by `(service, query, params-hash)`, configurable TTL, server can invalidate via `SX-Invalidate` header.
|
||||
|
||||
### 4c: Async DOM renderer — `renderDOMAsync()`
|
||||
4. **Optimistic updates** — extend existing `apply-optimistic`/`revert-optimistic` in `engine.sx` from DOM-level to data-level.
|
||||
|
||||
Two-pass (avoids restructuring sync renderer):
|
||||
1. Walk AST, collect I/O call sites with placeholders
|
||||
2. `Promise.all` to resolve all I/O in parallel
|
||||
3. Substitute resolved values into AST
|
||||
4. Call existing sync `renderDOM()` on resolved tree
|
||||
**Files:**
|
||||
- `shared/sx/ref/eval.sx` — async dispatch path (or new `async-eval.sx`)
|
||||
- New: `shared/sx/ref/io-bridge.sx` — client IO implementations
|
||||
- `shared/sx/ref/boot.sx` — register IO bridge at init
|
||||
- `shared/sx/ref/bootstrap_js.py` — emit async-aware code
|
||||
- `/internal/data/` endpoints — ensure client-accessible (CORS, auth)
|
||||
|
||||
### 4d: Wire into `Sx.mount()`
|
||||
**Depends on:** Phase 2 (IO affinity), Phase 3 (routing for when to trigger IO)
|
||||
|
||||
Detect I/O nodes. If present → async path. Otherwise → existing sync path (zero overhead for pure components).
|
||||
|
||||
**Files:** `shared/static/scripts/sx.js` (major addition)
|
||||
|
||||
**Verify:** Page with `(query "blog" "post-by-slug" :slug "test")` in sx source → client fetches `/api/data/post-by-slug?slug=test`, renders result.
|
||||
**Verification:**
|
||||
- Client `(query ...)` returns identical data to server-side
|
||||
- Data cache prevents redundant fetches
|
||||
- Same component source → identical output on either side
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Data-Only Navigation
|
||||
### Phase 5: Streaming & Suspense
|
||||
|
||||
When the client already has page components cached, navigation requires only a data fetch — no sx source from the server.
|
||||
**What it enables:** Server streams partially-evaluated SX as IO resolves. Client renders available subtrees immediately, fills in suspended parts. Like React Suspense but built on delimited continuations.
|
||||
|
||||
### 5a: Page components in the registry
|
||||
**Approach:**
|
||||
|
||||
`defpage` definitions are already in `_componentEnv` (Phase 1c) and cached in localStorage with the component hash. On navigation, if the hash is valid, the client has all page definitions locally.
|
||||
1. **Continuation-based suspension** — when `_aser` encounters IO during slot evaluation, emit a placeholder with a suspension ID, schedule async resolution:
|
||||
```python
|
||||
yield SxExpr(f'(~suspense :id "{placeholder_id}" :fallback (div "Loading..."))')
|
||||
schedule_fill(placeholder_id, io_coroutine)
|
||||
```
|
||||
|
||||
Build a `_pageRegistry` mapping URL path patterns → page definitions, populated when `defpage` forms are evaluated. Path patterns (`/posts/<slug>/`) converted to regex matchers for URL matching.
|
||||
2. **Chunked transfer** — Quart async generator responses:
|
||||
- First chunk: HTML shell + synchronous content + placeholders
|
||||
- Subsequent chunks: `<script>` tags replacing placeholders with resolved content
|
||||
|
||||
### 5b: Navigation intercept
|
||||
3. **Client suspension rendering** — `~suspense` component renders fallback, listens for resolution via inline script or SSE (existing SSE infrastructure in orchestration.sx).
|
||||
|
||||
Extend SxEngine's link click handler:
|
||||
4. **Priority-based IO** — above-fold content resolves first. All IO starts concurrently (`asyncio.create_task`), results flushed in priority order.
|
||||
|
||||
```
|
||||
1. Extract URL path from clicked link
|
||||
2. Match against _pageRegistry
|
||||
3. If matched:
|
||||
a. Evaluate :data slot via sxEvalAsync() → parallel API fetches
|
||||
b. Render :content/:filter/:aside via renderDOMAsync()
|
||||
c. Morph into existing ~app-body (headers persist, slots update)
|
||||
d. Push history state
|
||||
e. Update document title
|
||||
4. If not matched → existing server fetch (graceful fallback)
|
||||
```
|
||||
**Files:**
|
||||
- `shared/sx/async_eval.py` — streaming `_aser` variant
|
||||
- `shared/sx/helpers.py` — chunked response builder
|
||||
- New: `shared/sx/ref/suspense.sx` — client suspension rendering
|
||||
- `shared/sx/ref/boot.sx` — handle resolution scripts
|
||||
|
||||
### 5c: Data delivery — flexible per page
|
||||
|
||||
Three modes available (see Context section). The page definition can declare its preference:
|
||||
|
||||
```scheme
|
||||
(defpage blog-post
|
||||
:path "/posts/<slug>/"
|
||||
:data-mode :server ; :server (bundled), :client (fetch individually), :hybrid
|
||||
:data (query "blog" "post-by-slug" :slug slug)
|
||||
:content (~post-detail post))
|
||||
```
|
||||
|
||||
**Mode :server** — Client sends `SX-Page: blog-post` header on navigation. Server evaluates `:data` slot (all queries, including cross-service), returns single JSON blob:
|
||||
```python
|
||||
if request.headers.get("SX-Page"):
|
||||
data = await evaluate_data_slot(page_def, url_params)
|
||||
return jsonify(data)
|
||||
```
|
||||
|
||||
**Mode :client** — Client evaluates `:data` slot locally via `sxEvalAsync()`. Each `(query ...)` hits `/api/data/` independently. Each `(frag ...)` hits `/api/fragments/`. No server data endpoint needed.
|
||||
|
||||
**Mode :hybrid** — Server bundles own-service data (direct DB). Client fetches cross-service data and fragments in parallel. The `:data` slot is split: server evaluates local queries, returns partial bundle + a manifest of remaining queries. Client resolves the rest.
|
||||
|
||||
Default mode can be `:server` (fewest round-trips, simplest). Pages opt into `:client` or `:hybrid` when they want more decoupling or when cross-service data is heavy and benefits from parallel client fetches.
|
||||
|
||||
### 5d: Popstate handling
|
||||
|
||||
On browser back/forward:
|
||||
1. Check `_pageRegistry` for popped URL
|
||||
2. If matched → client render (same as 5b)
|
||||
3. If not → existing server fetch + morph
|
||||
|
||||
### 5e: Graceful fallback
|
||||
|
||||
Routes not in `_pageRegistry` fall through to server fetch. Partially migrated apps work — Python-only routes use server fetch, defpage routes get SPA behavior. No big-bang cutover.
|
||||
|
||||
**Files:** `shared/static/scripts/sx.js`, `shared/sx/helpers.py`, `shared/sx/pages.py`
|
||||
|
||||
**Verify:** Playwright: load page → click link to defpage route → assert no HTML response fetched (only JSON) → content correct → URL updated → back button works.
|
||||
**Depends on:** Phase 4 (client async for filling suspended subtrees), Phase 2 (IO analysis for priority)
|
||||
|
||||
---
|
||||
|
||||
## Summary: The Full Lifecycle
|
||||
### Phase 6: Full Isomorphism
|
||||
|
||||
```
|
||||
1. App startup: Python loads .sx files → defcomp + defpage registered in _COMPONENT_ENV
|
||||
→ hash computed
|
||||
**What it enables:** Same SX code runs on either side. Runtime chooses optimal split. Offline-first with cached data + client eval.
|
||||
|
||||
2. First visit: Server sends HTML shell + component/page defs + __sxConfig + page sx source
|
||||
Client evaluates, renders, caches defs in localStorage, sets cookie
|
||||
**Approach:**
|
||||
|
||||
3. Return visit: Cookie hash matches → server sends HTML shell with empty <script data-components>
|
||||
Client loads defs from localStorage → renders page
|
||||
1. **Runtime boundary optimizer** — given component tree + IO dependency graph, decide per-component: server-expand, client-render, or stream. Planning step cached at registration, recomputed on component change.
|
||||
|
||||
4. SPA navigation: Client matches URL against _pageRegistry
|
||||
→ fetches data from /api/data/ (or server data-only endpoint)
|
||||
→ renders page component locally with fresh data
|
||||
→ morphs DOM, pushes history
|
||||
→ zero sx source transferred
|
||||
2. **Affinity annotations** — optional developer hints:
|
||||
```lisp
|
||||
(defcomp ~product-grid (&key products)
|
||||
:affinity :client ;; interactive, prefer client
|
||||
...)
|
||||
(defcomp ~auth-menu (&key user)
|
||||
:affinity :server ;; auth-sensitive, always server
|
||||
...)
|
||||
```
|
||||
Default: auto (runtime decides from IO analysis).
|
||||
|
||||
5. Bot/SSR: Server detects bot UA → evaluates page server-side with direct DB queries
|
||||
→ sends rendered HTML + component defs
|
||||
→ client hydrates (binds handlers, no re-render)
|
||||
```
|
||||
3. **Offline data layer** — Service Worker intercepts `/internal/data/` requests, serves from IndexedDB when offline, syncs when back online.
|
||||
|
||||
## Migration per Service
|
||||
4. **Isomorphic testing** — evaluate same component on Python and JS, compare output. Extends existing `test_sx_ref.py` cross-evaluator comparison.
|
||||
|
||||
Each service migrates independently, no coordination needed:
|
||||
1. Add public data blueprint (Phase 3) — immediate standalone value
|
||||
2. Convert remaining Jinja routes to `defpage` — already in progress
|
||||
3. Enable SSR for bots (Phase 2) — per-page opt-in
|
||||
4. Client data primitives (Phase 4) — global once sx.js updated
|
||||
5. Data-only navigation (Phase 5) — automatic for any `defpage` route
|
||||
5. **Universal page descriptor** — `defpage` is portable: server executes via `execute_page()`, client executes via route match → fetch data → eval content → render DOM. Same descriptor, different execution environment.
|
||||
|
||||
**Depends on:** All previous phases.
|
||||
|
||||
---
|
||||
|
||||
## Why: Architectural Rationale
|
||||
## Cross-Cutting Concerns
|
||||
|
||||
The end state is: **sx.js is the only JavaScript in the browser.** All application code — components, pages, routing, event handling, data fetching — is expressed in sx, evaluated by the interpreter, with behavior mediated through bound primitives.
|
||||
### Error Reporting (all phases)
|
||||
- Phase 1: "Unknown component" includes which page expected it and what bundle was sent
|
||||
- Phase 2: Server logs which components expanded server-side vs sent to client
|
||||
- Phase 3: Client route failures include unmatched path and available routes
|
||||
- Phase 4: Client IO errors include query name, params, server response
|
||||
- Source location tracking in parser → propagate through eval → include in error messages
|
||||
|
||||
### Benefits
|
||||
### Backward Compatibility (all phases)
|
||||
- Pages without annotations behave as today
|
||||
- `SX-Request` / `SX-Components` / `SX-Css` header protocol continues
|
||||
- Existing `.sx` files require no changes
|
||||
- `_expand_components` continues as override
|
||||
- Each phase is opt-in: disable → identical to previous behavior
|
||||
|
||||
**Single language everywhere.** Components, pages, routing, event handling, data fetching — all sx. No context-switching between JS idioms and template syntax. One language for the entire frontend and the server rendering path.
|
||||
### Spec Integrity
|
||||
All new behavior specified in `.sx` files under `shared/sx/ref/` before implementation. Bootstrappers transpile from spec. This ensures JS and Python stay in sync.
|
||||
|
||||
**Portability.** The same source runs on any VM that implements the ~50-primitive interface. Today: Python + JS. Tomorrow: WASM, edge workers, native mobile, embedded devices. Coupled to a primitive contract, not to a specific runtime.
|
||||
## Critical Files
|
||||
|
||||
**Smaller wire transfer.** S-expressions are terser than equivalent JS. Combined with content-addressed caching (hash/localStorage), most navigations transfer zero code — just data.
|
||||
|
||||
**Inspectability.** The sx source is the running program — no build step, no source maps, no minification. View source shows exactly what executes. The AST is the structure the evaluator walks. Debugging is tracing a tree.
|
||||
|
||||
**Controlled surface area.** The only JS that runs is sx.js. Everything else is mediated through defined primitives. No npm supply chain. No third-party scripts with ambient DOM access. Components can only do what primitives allow — the capability surface is fully controlled.
|
||||
|
||||
**Hot-reloadable everything.** Components are data (cached AST). Swapping a definition is replacing a dict entry. No module system, no import graph, no HMR machinery. Already works for .sx file changes in dev mode — extends to behaviors too.
|
||||
|
||||
**AI-friendly.** S-expressions are trivially parseable and generatable. An LLM produces correct sx far more reliably than JS/JSX — fewer syntax edge cases, no semicolons/braces/arrow-function ambiguities. The codebase becomes more amenable to automated generation and transformation.
|
||||
|
||||
**Security boundary.** No `eval()`, no dynamic `<script>` injection, no prototype pollution. The sx evaluator is a sandbox — it only resolves symbols against the primitive table and component env. Auditing what any sx expression can do means auditing the primitive bindings.
|
||||
|
||||
### Performance and WASM
|
||||
|
||||
The tradeoff is interpreter overhead — a tree-walking interpreter is slower than native JS execution. For UI rendering (building DOM, handling events, fetching data), this is not the bottleneck — DOM operations dominate, and those are the same speed regardless of initiator.
|
||||
|
||||
If performance ever becomes a concern, WASM is the escape hatch at three levels:
|
||||
|
||||
1. **Evaluator in WASM.** Rewrite `sxEval` in Rust/Zig → WASM. The tight inner loop (symbol lookup, env traversal, function application) runs ~10-50x faster. DOM rendering stays in JS (it calls browser APIs regardless).
|
||||
|
||||
2. **Compile sx to WASM.** Ahead-of-time compiler: `.sx` → WASM modules. Each `defcomp` becomes a WASM function returning DOM instructions. Eliminates the interpreter entirely. The content-addressed cache stores compiled WASM blobs instead of sx source.
|
||||
|
||||
3. **Compute-heavy primitives in WASM.** Keep the sx interpreter in JS, bind specific primitives to WASM (image processing, crypto, data transformation). Most pragmatic and least disruptive — additive, no architecture change.
|
||||
|
||||
The primitive-binding model means the evaluator doesn't care what's behind a primitive. `(blur-image data radius)` could be a JS Canvas call today and a WASM JAX kernel tomorrow. The sx source doesn't change.
|
||||
|
||||
### Server-Driven by Default: The React Question
|
||||
|
||||
The sx system is architecturally aligned with HTMX/LiveView — server-driven UI — even though it does far more on the client (full s-expression evaluation, DOM rendering, morph reconciliation, component caching). The server is the single source of truth. Every UI state is a URL. Auth is enforced at render time. There are no state synchronization bugs because there is no client state to synchronize.
|
||||
|
||||
React's client-state model (`useState`, `useEffect`, Context, Suspense) exists because React was built for SPAs that need to feel like native apps — optimistic updates, offline capability, instant feedback without network latency. But it created an entire category of problems: state management libraries, hydration mismatches, cache invalidation, stale closures, memory leaks from forgotten cleanup, the `useEffect` footgun.
|
||||
|
||||
**The question is not "should sx have useState" — it's which specific interactions actually suffer from the server round-trip.**
|
||||
|
||||
For most of our apps, that's a very short list:
|
||||
- Toggle a mobile nav panel
|
||||
- Gallery image switching
|
||||
- Quantity steppers
|
||||
- Live search-as-you-type
|
||||
|
||||
These don't need a general-purpose reactive state system. They need **targeted client-side primitives** that handle those specific cases without abandoning the server-driven model.
|
||||
|
||||
**The dangerous path:** Add `useState` → need `useEffect` for cleanup → need Context to avoid prop drilling → need Suspense for async state → rebuild React inside sx → lose the simplicity that makes the server-driven model work.
|
||||
|
||||
**The careful path:** Keep server-driven as the default. Add explicit, targeted escape hatches for interactions that genuinely need client-side state. Make those escape hatches obviously different from the normal flow so they don't creep into everything.
|
||||
|
||||
#### What sx has vs React
|
||||
|
||||
| React feature | SX status | Verdict |
|
||||
|---|---|---|
|
||||
| Components + props | `defcomp` + `&key` | Done — cleaner than JSX |
|
||||
| Fragments, conditionals, lists | `<>`, `if`/`when`/`cond`, `map` | Done — more expressive |
|
||||
| Macros | `defmacro` | Done — React has nothing like this |
|
||||
| OOB updates / portals | `sx-swap-oob` | Done — more powerful (server-driven) |
|
||||
| DOM reconciliation | `_morphDOM` (id-keyed) | Done — works during SxEngine swaps |
|
||||
| Reactive client state | None | **By design.** Server is source of truth. |
|
||||
| Component lifecycle | None | Add targeted primitives if body.js behaviors move to sx |
|
||||
| Context / providers | `_componentEnv` global | Sufficient for auth/theme; revisit if trees get deep |
|
||||
| Suspense / loading | `sx-request` CSS class | Sufficient for server-driven; revisit for Phase 4 client data |
|
||||
| Two-way data binding | None | Not needed — HTMX model (form POST → new HTML) works |
|
||||
| Error boundaries | Global `sx:responseError` | Sufficient; per-component boundaries are a future nice-to-have |
|
||||
| Keyed list reconciliation | id-based morph | Works; add `:key` prop support if list update bugs arise |
|
||||
|
||||
#### Targeted escape hatches (not a general state system)
|
||||
|
||||
For the few interactions that need client-side responsiveness, add **specific primitives** rather than a general framework:
|
||||
|
||||
- `(toggle! el "class")` — CSS class toggle, no server trip
|
||||
- `(set-attr! el "attr" value)` — attribute manipulation
|
||||
- `(on-event el "click" handler)` — declarative event binding within sx
|
||||
- `(timer interval-ms handler)` — with automatic cleanup on DOM removal
|
||||
|
||||
These are imperative DOM operations exposed as primitives — not reactive state. They let components handle simple client-side interactions without importing React's entire mental model. The server-driven flow remains the default for anything involving data.
|
||||
| File | Role | Phases |
|
||||
|------|------|--------|
|
||||
| `shared/sx/async_eval.py` | Core evaluator, `_aser`, server/client boundary | 2, 5 |
|
||||
| `shared/sx/helpers.py` | `sx_page()`, `sx_response()`, output pipeline | 1, 3 |
|
||||
| `shared/sx/jinja_bridge.py` | `_COMPONENT_ENV`, component registry | 1, 2 |
|
||||
| `shared/sx/pages.py` | `defpage`, `execute_page()`, page lifecycle | 2, 3 |
|
||||
| `shared/sx/ref/boot.sx` | Client boot, component caching | 1, 3, 4 |
|
||||
| `shared/sx/ref/orchestration.sx` | Client fetch/swap/morph | 3, 4 |
|
||||
| `shared/sx/ref/eval.sx` | Evaluator spec | 4 |
|
||||
| `shared/sx/ref/engine.sx` | Morph, swaps, triggers | 3 |
|
||||
| New: `shared/sx/deps.py` | Dependency analysis | 1, 2 |
|
||||
| New: `shared/sx/ref/router.sx` | Client-side routing | 3 |
|
||||
| New: `shared/sx/ref/io-bridge.sx` | Client IO primitives | 4 |
|
||||
| New: `shared/sx/ref/suspense.sx` | Streaming/suspension | 5 |
|
||||
|
||||
@@ -13,7 +13,8 @@
|
||||
(dict :label "Examples" :href "/examples/click-to-load")
|
||||
(dict :label "Essays" :href "/essays/")
|
||||
(dict :label "Specs" :href "/specs/")
|
||||
(dict :label "Bootstrappers" :href "/bootstrappers/"))))
|
||||
(dict :label "Bootstrappers" :href "/bootstrappers/")
|
||||
(dict :label "Plans" :href "/plans/"))))
|
||||
(<> (map (lambda (item)
|
||||
(~nav-link
|
||||
:href (get item "href")
|
||||
|
||||
@@ -101,6 +101,10 @@
|
||||
(dict :label "Continuations" :href "/specs/continuations")
|
||||
(dict :label "call/cc" :href "/specs/callcc")))
|
||||
|
||||
(define plans-nav-items (list
|
||||
(dict :label "Isomorphic Architecture" :href "/plans/isomorphic-architecture"
|
||||
:summary "Making the server/client boundary a sliding window — per-page bundles, smart expansion, SPA routing, client IO, streaming suspense.")))
|
||||
|
||||
(define bootstrappers-nav-items (list
|
||||
(dict :label "Overview" :href "/bootstrappers/")
|
||||
(dict :label "JavaScript" :href "/bootstrappers/javascript")
|
||||
|
||||
388
sx/sx/plans.sx
Normal file
388
sx/sx/plans.sx
Normal file
@@ -0,0 +1,388 @@
|
||||
;; Plans section — architecture roadmaps and implementation plans
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; Plans index page
|
||||
;; ---------------------------------------------------------------------------
|
||||
|
||||
(defcomp ~plans-index-content ()
|
||||
(~doc-page :title "Plans"
|
||||
(div :class "space-y-4"
|
||||
(p :class "text-lg text-stone-600 mb-4"
|
||||
"Architecture roadmaps and implementation plans for SX.")
|
||||
(div :class "space-y-3"
|
||||
(map (fn (item)
|
||||
(a :href (get item "href")
|
||||
:sx-get (get item "href") :sx-target "#main-panel" :sx-select "#main-panel"
|
||||
:sx-swap "outerHTML" :sx-push-url "true"
|
||||
:class "block rounded border border-stone-200 p-4 hover:border-violet-300 hover:bg-violet-50 transition-colors"
|
||||
(div :class "font-semibold text-stone-800" (get item "label"))
|
||||
(when (get item "summary")
|
||||
(p :class "text-sm text-stone-500 mt-1" (get item "summary")))))
|
||||
plans-nav-items)))))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; Isomorphic Architecture Roadmap
|
||||
;; ---------------------------------------------------------------------------
|
||||
|
||||
(defcomp ~plan-isomorphic-content ()
|
||||
(~doc-page :title "Isomorphic Architecture Roadmap"
|
||||
|
||||
(~doc-section :title "Context" :id "context"
|
||||
(p "SX has a working server-client pipeline: server evaluates pages with IO (DB, fragments), serializes as SX wire format, client parses and renders to DOM. The language and primitives are already isomorphic " (em "— same spec, same semantics, both sides.") " What's missing is the " (strong "plumbing") " that makes the boundary between server and client a sliding window rather than a fixed wall.")
|
||||
(p "The key insight: " (strong "s-expressions can partially unfold on the server after IO, then finish unfolding on the client.") " The system should be clever enough to know which downstream components have data fetches, resolve those server-side, and send the rest as pure SX for client rendering. Eventually, the client can also do IO (mapping server DB queries to REST calls), handle routing (SPA), and even work offline with cached data."))
|
||||
|
||||
(~doc-section :title "Current State" :id "current-state"
|
||||
(ul :class "space-y-2 text-stone-700 list-disc pl-5"
|
||||
(li (strong "Primitive parity: ") "100%. ~80 pure primitives, same names/semantics, JS and Python.")
|
||||
(li (strong "eval/parse/render: ") "Complete both sides. sx-ref.js has eval, parse, render-to-html, render-to-dom, aser.")
|
||||
(li (strong "Engine: ") "engine.sx (morph, swaps, triggers, history), orchestration.sx (fetch, events), boot.sx (hydration) — all transpiled.")
|
||||
(li (strong "Wire format: ") "Server _aser → SX source → client parses → renders to DOM. Boundary is clean.")
|
||||
(li (strong "Component caching: ") "Hash-based localStorage for component definitions and style dictionaries.")
|
||||
(li (strong "CSS on-demand: ") "CSSX resolves keywords to CSS rules, injects only used rules.")
|
||||
(li (strong "Boundary enforcement: ") "boundary.sx + SX_BOUNDARY_STRICT=1 validates all primitives/IO/helpers at registration.")))
|
||||
|
||||
;; -----------------------------------------------------------------------
|
||||
;; Phase 1
|
||||
;; -----------------------------------------------------------------------
|
||||
|
||||
(~doc-section :title "Phase 1: Component Distribution & Dependency Analysis" :id "phase-1"
|
||||
|
||||
(div :class "rounded border border-violet-200 bg-violet-50 p-4 mb-4"
|
||||
(p :class "text-violet-900 font-medium" "What it enables")
|
||||
(p :class "text-violet-800" "Per-page component bundles instead of sending every definition to every page. Smaller payloads, faster boot, better cache hit rates."))
|
||||
|
||||
(~doc-subsection :title "The Problem"
|
||||
(p "client_components_tag() in jinja_bridge.py serializes ALL entries in _COMPONENT_ENV. The sx_page() template sends everything or nothing based on a single global hash. No mechanism determines which components a page actually needs."))
|
||||
|
||||
(~doc-subsection :title "Approach"
|
||||
|
||||
(div :class "space-y-4"
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "1. Transitive closure analyzer")
|
||||
(p "New module shared/sx/deps.py:")
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li "Walk Component.body AST, collect all Symbol refs starting with ~")
|
||||
(li "Recursively follow into their bodies")
|
||||
(li "Handle control forms (if/when/cond/case) — include ALL branches")
|
||||
(li "Handle macros — expand during walk using limited eval"))
|
||||
(~doc-code :code (highlight "def transitive_deps(name: str, env: dict) -> set[str]:\n \"\"\"Compute transitive component dependencies.\"\"\"\n seen = set()\n def walk(n):\n if n in seen: return\n seen.add(n)\n comp = env.get(n)\n if comp:\n for dep in _scan_ast(comp.body):\n walk(dep)\n walk(name)\n return seen - {name}" "python")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "2. Runtime component scanning")
|
||||
(p "After _aser serializes page content, scan the SX string for (~name patterns (parallel to existing scan_classes_from_sx for CSS). Then compute transitive closure to get sub-components."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "3. Per-page component block")
|
||||
(p "In sx_page() — replace all-or-nothing with page-specific bundle. Hash changes per page, localStorage cache keyed by route pattern."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "4. SX partial responses")
|
||||
(p "components_for_request() already diffs against SX-Components header. Enhance with transitive closure so only truly needed missing components are sent."))))
|
||||
|
||||
(~doc-subsection :title "Files"
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm"
|
||||
(li "New: shared/sx/deps.py — dependency analysis")
|
||||
(li "shared/sx/jinja_bridge.py — per-page bundle generation")
|
||||
(li "shared/sx/helpers.py — modify sx_page() and sx_response()")
|
||||
(li "shared/sx/types.py — add deps: set[str] to Component")
|
||||
(li "shared/sx/ref/boot.sx — per-page component caching")))
|
||||
|
||||
(~doc-subsection :title "Verification"
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li "Page using 5/50 components → data-components block contains only those 5 + transitive deps")
|
||||
(li "No \"Unknown component\" errors after bundle reduction")
|
||||
(li "Payload size reduction measurable"))))
|
||||
|
||||
;; -----------------------------------------------------------------------
|
||||
;; Phase 2
|
||||
;; -----------------------------------------------------------------------
|
||||
|
||||
(~doc-section :title "Phase 2: Smart Server/Client Boundary" :id "phase-2"
|
||||
|
||||
(div :class "rounded border border-violet-200 bg-violet-50 p-4 mb-4"
|
||||
(p :class "text-violet-900 font-medium" "What it enables")
|
||||
(p :class "text-violet-800" "Formalized partial evaluation model. Server evaluates IO, serializes pure subtrees. The system automatically knows \"this component needs server data\" vs \"this component is pure and can render anywhere.\""))
|
||||
|
||||
(~doc-subsection :title "Current Mechanism"
|
||||
(p "_aser in async_eval.py already does partial evaluation — IO primitives are awaited and substituted, HTML tags and component calls serialize as SX. The _expand_components context var controls expansion. But this is a global toggle, not per-component."))
|
||||
|
||||
(~doc-subsection :title "Approach"
|
||||
|
||||
(div :class "space-y-4"
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "1. Automatic IO detection")
|
||||
(p "Extend Phase 1 AST walker to check for references to IO_PRIMITIVES names (frag, query, service, current-user, etc.).")
|
||||
(~doc-code :code (highlight "def has_io_deps(name: str, env: dict) -> bool:\n \"\"\"True if component transitively references any IO primitive.\"\"\"\n ..." "python")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "2. Component metadata")
|
||||
(~doc-code :code (highlight "ComponentMeta:\n deps: set[str] # transitive component deps (Phase 1)\n io_refs: set[str] # IO primitive names referenced\n is_pure: bool # True if io_refs empty (transitively)" "python")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "3. Selective expansion")
|
||||
(p "Refine _aser: instead of checking a global _expand_components flag, check the component's is_pure metadata:")
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li "IO-dependent → expand server-side (IO must resolve)")
|
||||
(li "Pure → serialize for client (let client render)")
|
||||
(li "Explicit override: :server true on defcomp forces server expansion")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "4. Data manifest for pages")
|
||||
(p "PageDef produces a declaration of what IO the page needs, enabling Phase 3 (client can prefetch data) and Phase 5 (streaming)."))))
|
||||
|
||||
(~doc-subsection :title "Verification"
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li "Components calling (query ...) classified IO-dependent; pure components classified pure")
|
||||
(li "Existing pages produce identical output (regression)"))))
|
||||
|
||||
;; -----------------------------------------------------------------------
|
||||
;; Phase 3
|
||||
;; -----------------------------------------------------------------------
|
||||
|
||||
(~doc-section :title "Phase 3: Client-Side Routing (SPA Mode)" :id "phase-3"
|
||||
|
||||
(div :class "rounded border border-violet-200 bg-violet-50 p-4 mb-4"
|
||||
(p :class "text-violet-900 font-medium" "What it enables")
|
||||
(p :class "text-violet-800" "After initial page load, client resolves routes locally using cached components + data. Only hits server for fresh data or unknown routes. Like Next.js client-side navigation."))
|
||||
|
||||
(~doc-subsection :title "Current Mechanism"
|
||||
(p "All routing is server-side via defpage → Quart routes. Client navigates via sx-boost links doing sx-get + morphing. Every navigation = server roundtrip."))
|
||||
|
||||
(~doc-subsection :title "Approach"
|
||||
|
||||
(div :class "space-y-4"
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "1. Client-side page registry")
|
||||
(p "Serialize defpage routing info to client:")
|
||||
(~doc-code :code (highlight "(script :type \"text/sx-pages\")\n;; {\"docs-page\": {\"path\": \"/docs/:slug\", \"auth\": \"public\",\n;; \"content\": \"(case slug ...)\", \"data\": null}}" "lisp")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "2. Client route matcher")
|
||||
(p "New spec file shared/sx/ref/router.sx — convert /docs/<slug> patterns to matchers. On boost-link click: match URL → if found and pure, evaluate locally. If IO needed: fetch data from server, evaluate content locally. No match: fall through to standard fetch."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "3. Data endpoint")
|
||||
(~doc-code :code (highlight "GET /internal/page-data/<page-name>?<params>\n# Returns JSON with evaluated :data expression\n# Reuses execute_page() logic, stops after :data step" "python")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "4. Layout caching")
|
||||
(p "Layouts depend on auth/fragments, so cache current layout and reuse across navigations. SX-Layout-Hash header tracks staleness."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "5. Integration with orchestration.sx")
|
||||
(p "Intercept bind-boost-link to try client-side resolution first."))))
|
||||
|
||||
(div :class "rounded border border-amber-200 bg-amber-50 p-3 mt-2"
|
||||
(p :class "text-amber-800 text-sm" (strong "Depends on: ") "Phase 1 (client knows which components each page needs), Phase 2 (which pages are pure vs IO)."))
|
||||
|
||||
(~doc-subsection :title "Verification"
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li "Pure page navigation: zero server requests")
|
||||
(li "IO page navigation: exactly one data request (not full page fetch)")
|
||||
(li "Browser back/forward works with client-resolved routes")
|
||||
(li "Disabling client registry → identical behavior to current"))))
|
||||
|
||||
;; -----------------------------------------------------------------------
|
||||
;; Phase 4
|
||||
;; -----------------------------------------------------------------------
|
||||
|
||||
(~doc-section :title "Phase 4: Client Async & IO Bridge" :id "phase-4"
|
||||
|
||||
(div :class "rounded border border-violet-200 bg-violet-50 p-4 mb-4"
|
||||
(p :class "text-violet-900 font-medium" "What it enables")
|
||||
(p :class "text-violet-800" "Client evaluates IO primitives by mapping them to server REST calls. Same SX code, different transport. (query \"market\" \"products\" :ids \"1,2,3\") on server → DB; on client → fetch(\"/internal/data/products?ids=1,2,3\")."))
|
||||
|
||||
(~doc-subsection :title "Approach"
|
||||
|
||||
(div :class "space-y-4"
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "1. Async client evaluator")
|
||||
(p "Two possible mechanisms:")
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li (strong "Promise-based: ") "evalExpr returns value or Promise; rendering awaits")
|
||||
(li (strong "Continuation-based: ") "use existing shift/reset to suspend on IO, resume when data arrives (architecturally cleaner, leverages existing spec)")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "2. IO primitive bridge")
|
||||
(p "Register async IO primitives in client PRIMITIVES:")
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm"
|
||||
(li "query → fetch to /internal/data/")
|
||||
(li "service → fetch to target service internal endpoint")
|
||||
(li "frag → fetch fragment HTML")
|
||||
(li "current-user → cached from initial page load")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "3. Client data cache")
|
||||
(p "Keyed by (service, query, params-hash), configurable TTL, server can invalidate via SX-Invalidate header."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "4. Optimistic updates")
|
||||
(p "Extend existing apply-optimistic/revert-optimistic in engine.sx from DOM-level to data-level."))))
|
||||
|
||||
(div :class "rounded border border-amber-200 bg-amber-50 p-3 mt-2"
|
||||
(p :class "text-amber-800 text-sm" (strong "Depends on: ") "Phase 2 (IO affinity), Phase 3 (routing for when to trigger IO)."))
|
||||
|
||||
(~doc-subsection :title "Verification"
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li "Client (query ...) returns identical data to server-side")
|
||||
(li "Data cache prevents redundant fetches")
|
||||
(li "Same component source → identical output on either side"))))
|
||||
|
||||
;; -----------------------------------------------------------------------
|
||||
;; Phase 5
|
||||
;; -----------------------------------------------------------------------
|
||||
|
||||
(~doc-section :title "Phase 5: Streaming & Suspense" :id "phase-5"
|
||||
|
||||
(div :class "rounded border border-violet-200 bg-violet-50 p-4 mb-4"
|
||||
(p :class "text-violet-900 font-medium" "What it enables")
|
||||
(p :class "text-violet-800" "Server streams partially-evaluated SX as IO resolves. Client renders available subtrees immediately, fills in suspended parts. Like React Suspense but built on delimited continuations."))
|
||||
|
||||
(~doc-subsection :title "Approach"
|
||||
|
||||
(div :class "space-y-4"
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "1. Continuation-based suspension")
|
||||
(p "When _aser encounters IO during slot evaluation, emit a placeholder with a suspension ID, schedule async resolution:")
|
||||
(~doc-code :code (highlight "(~suspense :id \"placeholder-123\"\n :fallback (div \"Loading...\"))" "lisp")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "2. Chunked transfer")
|
||||
(p "Quart async generator responses:")
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li "First chunk: HTML shell + synchronous content + placeholders")
|
||||
(li "Subsequent chunks: <script> tags replacing placeholders with resolved content")))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "3. Client suspension rendering")
|
||||
(p "~suspense component renders fallback, listens for resolution via inline script or SSE (existing SSE infrastructure in orchestration.sx)."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "4. Priority-based IO")
|
||||
(p "Above-fold content resolves first. All IO starts concurrently (asyncio.create_task), results flushed in priority order."))))
|
||||
|
||||
(div :class "rounded border border-amber-200 bg-amber-50 p-3 mt-2"
|
||||
(p :class "text-amber-800 text-sm" (strong "Depends on: ") "Phase 4 (client async for filling suspended subtrees), Phase 2 (IO analysis for priority).")))
|
||||
|
||||
;; -----------------------------------------------------------------------
|
||||
;; Phase 6
|
||||
;; -----------------------------------------------------------------------
|
||||
|
||||
(~doc-section :title "Phase 6: Full Isomorphism" :id "phase-6"
|
||||
|
||||
(div :class "rounded border border-violet-200 bg-violet-50 p-4 mb-4"
|
||||
(p :class "text-violet-900 font-medium" "What it enables")
|
||||
(p :class "text-violet-800" "Same SX code runs on either side. Runtime chooses optimal split. Offline-first with cached data + client eval."))
|
||||
|
||||
(~doc-subsection :title "Approach"
|
||||
|
||||
(div :class "space-y-4"
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "1. Runtime boundary optimizer")
|
||||
(p "Given component tree + IO dependency graph, decide per-component: server-expand, client-render, or stream. Planning step cached at registration, recomputed on component change."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "2. Affinity annotations")
|
||||
(~doc-code :code (highlight "(defcomp ~product-grid (&key products)\n :affinity :client ;; interactive, prefer client\n ...)\n\n(defcomp ~auth-menu (&key user)\n :affinity :server ;; auth-sensitive, always server\n ...)" "lisp"))
|
||||
(p "Default: auto (runtime decides from IO analysis)."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "3. Offline data layer")
|
||||
(p "Service Worker intercepts /internal/data/ requests, serves from IndexedDB when offline, syncs when back online."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "4. Isomorphic testing")
|
||||
(p "Evaluate same component on Python and JS, compare output. Extends existing test_sx_ref.py cross-evaluator comparison."))
|
||||
|
||||
(div
|
||||
(h4 :class "font-semibold text-stone-700" "5. Universal page descriptor")
|
||||
(p "defpage is portable: server executes via execute_page(), client executes via route match → fetch data → eval content → render DOM. Same descriptor, different execution environment."))))
|
||||
|
||||
(div :class "rounded border border-amber-200 bg-amber-50 p-3 mt-2"
|
||||
(p :class "text-amber-800 text-sm" (strong "Depends on: ") "All previous phases.")))
|
||||
|
||||
;; -----------------------------------------------------------------------
|
||||
;; Cross-Cutting Concerns
|
||||
;; -----------------------------------------------------------------------
|
||||
|
||||
(~doc-section :title "Cross-Cutting Concerns" :id "cross-cutting"
|
||||
|
||||
(~doc-subsection :title "Error Reporting"
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li "Phase 1: \"Unknown component\" includes which page expected it and what bundle was sent")
|
||||
(li "Phase 2: Server logs which components expanded server-side vs sent to client")
|
||||
(li "Phase 3: Client route failures include unmatched path and available routes")
|
||||
(li "Phase 4: Client IO errors include query name, params, server response")
|
||||
(li "Source location tracking in parser → propagate through eval → include in error messages")))
|
||||
|
||||
(~doc-subsection :title "Backward Compatibility"
|
||||
(ul :class "list-disc pl-5 text-stone-700 space-y-1"
|
||||
(li "Pages without annotations behave as today")
|
||||
(li "SX-Request / SX-Components / SX-Css header protocol continues")
|
||||
(li "Existing .sx files require no changes")
|
||||
(li "_expand_components continues as override")
|
||||
(li "Each phase is opt-in: disable → identical to previous behavior")))
|
||||
|
||||
(~doc-subsection :title "Spec Integrity"
|
||||
(p "All new behavior specified in .sx files under shared/sx/ref/ before implementation. Bootstrappers transpile from spec. This ensures JS and Python stay in sync.")))
|
||||
|
||||
;; -----------------------------------------------------------------------
|
||||
;; Critical Files
|
||||
;; -----------------------------------------------------------------------
|
||||
|
||||
(~doc-section :title "Critical Files" :id "critical-files"
|
||||
(div :class "overflow-x-auto rounded border border-stone-200"
|
||||
(table :class "w-full text-left text-sm"
|
||||
(thead (tr :class "border-b border-stone-200 bg-stone-100"
|
||||
(th :class "px-3 py-2 font-medium text-stone-600" "File")
|
||||
(th :class "px-3 py-2 font-medium text-stone-600" "Role")
|
||||
(th :class "px-3 py-2 font-medium text-stone-600" "Phases")))
|
||||
(tbody
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/async_eval.py")
|
||||
(td :class "px-3 py-2 text-stone-700" "Core evaluator, _aser, server/client boundary")
|
||||
(td :class "px-3 py-2 text-stone-600" "2, 5"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/helpers.py")
|
||||
(td :class "px-3 py-2 text-stone-700" "sx_page(), sx_response(), output pipeline")
|
||||
(td :class "px-3 py-2 text-stone-600" "1, 3"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/jinja_bridge.py")
|
||||
(td :class "px-3 py-2 text-stone-700" "_COMPONENT_ENV, component registry")
|
||||
(td :class "px-3 py-2 text-stone-600" "1, 2"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/pages.py")
|
||||
(td :class "px-3 py-2 text-stone-700" "defpage, execute_page(), page lifecycle")
|
||||
(td :class "px-3 py-2 text-stone-600" "2, 3"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ref/boot.sx")
|
||||
(td :class "px-3 py-2 text-stone-700" "Client boot, component caching")
|
||||
(td :class "px-3 py-2 text-stone-600" "1, 3, 4"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ref/orchestration.sx")
|
||||
(td :class "px-3 py-2 text-stone-700" "Client fetch/swap/morph")
|
||||
(td :class "px-3 py-2 text-stone-600" "3, 4"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ref/eval.sx")
|
||||
(td :class "px-3 py-2 text-stone-700" "Evaluator spec")
|
||||
(td :class "px-3 py-2 text-stone-600" "4"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ref/engine.sx")
|
||||
(td :class "px-3 py-2 text-stone-700" "Morph, swaps, triggers")
|
||||
(td :class "px-3 py-2 text-stone-600" "3"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/deps.py")
|
||||
(td :class "px-3 py-2 text-stone-700" "Dependency analysis (new)")
|
||||
(td :class "px-3 py-2 text-stone-600" "1, 2"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ref/router.sx")
|
||||
(td :class "px-3 py-2 text-stone-700" "Client-side routing (new)")
|
||||
(td :class "px-3 py-2 text-stone-600" "3"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ref/io-bridge.sx")
|
||||
(td :class "px-3 py-2 text-stone-700" "Client IO primitives (new)")
|
||||
(td :class "px-3 py-2 text-stone-600" "4"))
|
||||
(tr :class "border-b border-stone-100"
|
||||
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ref/suspense.sx")
|
||||
(td :class "px-3 py-2 text-stone-700" "Streaming/suspension (new)")
|
||||
(td :class "px-3 py-2 text-stone-600" "5"))))))))
|
||||
@@ -386,3 +386,32 @@
|
||||
(~bootstrapper-js-content
|
||||
:bootstrapper-source bootstrapper-source
|
||||
:bootstrapped-output bootstrapped-output))))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; Plans section
|
||||
;; ---------------------------------------------------------------------------
|
||||
|
||||
(defpage plans-index
|
||||
:path "/plans/"
|
||||
:auth :public
|
||||
:layout (:sx-section
|
||||
:section "Plans"
|
||||
:sub-label "Plans"
|
||||
:sub-href "/plans/"
|
||||
:sub-nav (~section-nav :items plans-nav-items :current "")
|
||||
:selected "")
|
||||
:content (~plans-index-content))
|
||||
|
||||
(defpage plan-page
|
||||
:path "/plans/<slug>"
|
||||
:auth :public
|
||||
:layout (:sx-section
|
||||
:section "Plans"
|
||||
:sub-label "Plans"
|
||||
:sub-href "/plans/"
|
||||
:sub-nav (~section-nav :items plans-nav-items
|
||||
:current (find-current plans-nav-items slug))
|
||||
:selected (or (find-current plans-nav-items slug) ""))
|
||||
:content (case slug
|
||||
"isomorphic-architecture" (~plan-isomorphic-content)
|
||||
:else (~plans-index-content)))
|
||||
|
||||
Reference in New Issue
Block a user