coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 4 complete: client data cache + plan update

Browse Source 
- Add page data cache in orchestration.sx (30s TTL, keyed by page-name+params)
- Cache hit path: sx:route client+cache (instant render, no fetch)
- Cache miss path: sx:route client+data (fetch, cache, render)
- Fix HTMX response dep computation to include :data pages
- Update isomorphic-sx-plan.md: Phases 1-4 marked done with details,
  reorder remaining phases (continuations→Phase 5, suspense→Phase 6,
  optimistic updates→Phase 7)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
giles
2026-03-07 00:06:22 +00:00
parent 732923a7ef
commit d3617ab7f3
4 changed files with 241 additions and 139 deletions
Download Patch File Download Diff File Expand all files Collapse all files

257
docs/isomorphic-sx-plan.md
View File

@@ -20,136 +20,147 @@ The key insight: **s-expressions can partially unfold on the server after IO, th
---
### Phase 1: Component Distribution & Dependency Analysis
### Phase 1: Component Distribution & Dependency Analysis — DONE
**What it enables:** Per-page component bundles instead of sending every definition to every page. Smaller payloads, faster boot, better cache hit rates.
**The problem:** `client_components_tag()` in `shared/sx/jinja_bridge.py` serializes ALL entries in `_COMPONENT_ENV`. The `sx_page()` template sends everything or nothing based on a single global hash. No mechanism determines which components a page actually needs.
**Implemented:**
**Approach:**
1. **Transitive closure analyzer** — new module `shared/sx/deps.py`
- Walk `Component.body` AST, collect all `Symbol` refs starting with `~`
1. **Transitive closure analyzer** — `shared/sx/deps.py` (now `shared/sx/ref/deps.sx`, spec-level)
- Walk component body AST, collect all `~name` refs
- Recursively follow into their bodies
- Handle control forms (`if`/`when`/`cond`/`case`) — include ALL branches
- Handle macros — expand during walk using limited eval
- Function: `transitive_deps(name: str, env: dict) -> set[str]`
- Cache result on `Component` object (invalidate on hot-reload)
- `components_needed(source, env) -> set[str]`
2. **Runtime component scanning** — after `_aser` serializes page content, scan the SX string for `(~name` patterns (parallel to existing `scan_classes_from_sx` for CSS). Then compute transitive closure to get sub-components.
2. **IO reference analysis** — `deps.sx` also tracks IO primitive usage
- `scan-io-refs` / `transitive-io-refs` / `component-pure?`
- Used by Phase 2 for automatic server/client boundary
3. **Per-page component block** in `sx_page()` — replace all-or-nothing with page-specific bundle. Hash changes per page, localStorage cache keyed by route pattern.
3. **Per-page component block** — `_build_pages_sx()` in `helpers.py`
- Each page entry includes `:deps` list of required components
- Client page registry carries dep info for prefetching
4. **SX partial responses** — `components_for_request()` already diffs against `SX-Components` header. Enhance with transitive closure so only truly needed missing components are sent.
4. **SX partial responses** — `components_for_request()` diffs against `SX-Components` header, sends only missing components
**Files:**
- New: `shared/sx/deps.py` — dependency analysis
- `shared/sx/jinja_bridge.py` — per-page bundle generation, cache deps on Component
- `shared/sx/helpers.py` — modify `sx_page()` and `sx_response()` for page-specific bundles
- `shared/sx/types.py` — add `deps: set[str]` to Component
- `shared/sx/ref/boot.sx` — per-page component caching alongside global cache
**Verification:**
- Page using 5/50 components → `data-components` block contains only those 5 + transitive deps
- No "Unknown component" errors after bundle reduction
- Payload size reduction measurable
**Files:** `shared/sx/ref/deps.sx`, `shared/sx/deps.py`, `shared/sx/helpers.py`, `shared/sx/jinja_bridge.py`
---
### Phase 2: Smart Server/Client Boundary
### Phase 2: Smart Server/Client Boundary — DONE
**What it enables:** Formalized partial evaluation model. Server evaluates IO, serializes pure subtrees. The system automatically knows "this component needs server data" vs "this component is pure and can render anywhere."
**Current mechanism:** `_aser` in `async_eval.py` already does partial evaluation — IO primitives are awaited and substituted, HTML tags and component calls serialize as SX. The `_expand_components` context var controls expansion. But this is a global toggle, not per-component.
**Implemented:**
**Approach:**
1. **Automatic IO detection** — `deps.sx` walks component bodies for IO primitive refs
- `compute-all-io-refs` computes transitive IO analysis for all components
- `component-pure?` returns true if no IO refs transitively
1. **Automatic IO detection** — extend Phase 1 AST walker to check for references to `IO_PRIMITIVES` names (`frag`, `query`, `service`, `current-user`, etc.)
- `has_io_deps(name: str, env: dict) -> bool`
- Computed at registration time, cached on Component
2. **Selective expansion** — `_aser` expands known components server-side via `_aser_component`
- IO-dependent components expand server-side (IO must resolve)
- Unknown components serialize for client rendering
- `_expand_components` context var controls override
2. **Component metadata** — enrich Component with analysis results:
```python
ComponentMeta:
deps: set[str] # transitive component deps (Phase 1)
io_refs: set[str] # IO primitive names referenced
is_pure: bool # True if io_refs empty (transitively)
```
3. **Component metadata** — computed at registration, cached on Component objects
3. **Selective expansion** — refine `_aser` (line ~1335): instead of checking a global `_expand_components` flag, check the component's `is_pure` metadata:
- IO-dependent → expand server-side (IO must resolve)
- Pure → serialize for client (let client render)
- Explicit override: `:server true` on defcomp forces server expansion
4. **Data manifest** for pages — `PageDef` produces a declaration of what IO the page needs, enabling Phase 3 (client can prefetch data) and Phase 5 (streaming).
**Files:**
- `shared/sx/deps.py` — add IO analysis
- `shared/sx/types.py` — add metadata fields to Component
- `shared/sx/async_eval.py` — refine `_aser` component expansion logic
- `shared/sx/jinja_bridge.py` — compute IO metadata at registration
- `shared/sx/pages.py` — data manifest on PageDef
**Verification:**
- Components calling `(query ...)` classified IO-dependent; pure components classified pure
- Existing pages produce identical output (regression)
**Files:** `shared/sx/ref/deps.sx`, `shared/sx/async_eval.py`, `shared/sx/jinja_bridge.py`
---
### Phase 3: Client-Side Routing (SPA Mode)
### Phase 3: Client-Side Routing (SPA Mode) — DONE
**What it enables:** After initial page load, client resolves routes locally using cached components + data. Only hits server for fresh data or unknown routes. Like Next.js client-side navigation.
**What it enables:** After initial page load, client resolves routes locally using cached components. Only hits server for fresh data or unknown routes.
**Current mechanism:** All routing is server-side via `defpage` → Quart routes. Client navigates via `sx-boost` links doing `sx-get` + morphing. Every navigation = server roundtrip.
**Implemented:**
**Approach:**
1. **Client-side page registry** — `_build_pages_sx()` serializes defpage routing info
- `<script type="text/sx-pages">` with name, path, auth, content, deps, closure, has-data
- Processed by `boot.sx` → `_page-routes` list
1. **Client-side page registry** — serialize defpage routing info to client as `<script type="text/sx-pages">`:
```json
{"docs-page": {"path": "/docs/:slug", "auth": "public",
"content": "(case slug ...)", "data": null}}
```
Pure pages (no `:data`) can be evaluated entirely client-side.
2. **Client route matcher** — `shared/sx/ref/router.sx`
- `parse-route-pattern` converts Flask-style `/docs/<slug>` to matchers
- `find-matching-route` matches URL against registered routes
- `match-route-segments` handles literal and param segments
2. **Client route matcher** — new spec file `shared/sx/ref/router.sx`:
- Convert `/docs/<slug>` patterns to matchers
- On boost-link click: match URL → if found and pure, evaluate locally
- If IO needed: fetch data from server, evaluate content locally
- No match: fall through to standard fetch (existing behavior)
3. **Client-side route intercept** — `orchestration.sx`
- `try-client-route` — match URL, eval content locally, swap DOM
- `bind-client-route-link` — intercept boost link clicks
- Pure pages render immediately, no server roundtrip
- Falls through to server fetch on miss
3. **Data endpoint** — `GET /internal/page-data/<page-name>?<params>` returns JSON with evaluated `:data` expression. Reuses `execute_page()` logic but stops after `:data` step.
4. **Integration with engine** — boost link clicks try client route first, fall back to standard fetch
4. **Layout caching** — layouts depend on auth/fragments, so cache current layout and reuse across navigations. `SX-Layout-Hash` header tracks staleness.
5. **Integration with orchestration.sx** — intercept `bind-boost-link` to try client-side resolution first.
**Files:**
- `shared/sx/pages.py` — `serialize_for_client()`, data-only execution path
- `shared/sx/helpers.py` — include page registry in `sx_page()`
- New: `shared/sx/ref/router.sx` — client-side route matching
- `shared/sx/ref/boot.sx` — process `<script type="text/sx-pages">`
- `shared/sx/ref/orchestration.sx` — client-side route intercept
- Service blueprints — `/internal/page-data/` endpoint
**Depends on:** Phase 1 (client knows which components each page needs), Phase 2 (which pages are pure vs IO)
**Verification:**
- Pure page navigation: zero server requests
- IO page navigation: exactly one data request (not full page fetch)
- Browser back/forward works with client-resolved routes
- Disabling client registry → identical behavior to current
**Files:** `shared/sx/ref/router.sx`, `shared/sx/ref/boot.sx`, `shared/sx/ref/orchestration.sx`, `shared/sx/helpers.py`, `shared/sx/pages.py`
---
### Phase 4: Client Async & IO Bridge
### Phase 4: Client Async & IO Bridge — DONE
**What it enables:** Client evaluates IO primitives by mapping them to server REST calls. Same SX code, different transport. `(query "market" "products" :ids "1,2,3")` on server → DB; on client → `fetch("/internal/data/products?ids=1,2,3")`.
**What it enables:** Client fetches server-evaluated data and renders `:data` pages locally. Data cached to avoid redundant fetches on back/forward navigation.
**The approach:** Separate IO from rendering. Server evaluates `:data` expression (async, with DB/service access), serializes result as SX wire format. Client fetches this pre-evaluated data, parses it, merges into env, renders pure `:content` client-side. No continuations needed — all IO happens server-side.
**Implemented:**
1. **Abstract `resolve-page-data`** — spec-level primitive in `orchestration.sx`
- `(resolve-page-data name params callback)` — platform decides transport
- Spec says "I need data for this page"; platform provides concrete implementation
- Browser platform: HTTP fetch to `/sx/data/` endpoint
2. **Server data endpoint** — `pages.py`
- `evaluate_page_data()` — evaluates `:data` expression, kebab-cases dict keys, serializes as SX
- `auto_mount_page_data()` — mounts `GET /sx/data/<page_name>` endpoint
- Per-page auth enforcement via `_check_page_auth()`
- Response content type: `text/sx; charset=utf-8`
3. **Client-side data rendering** — `orchestration.sx`
- `try-client-route` handles `:data` pages: fetch data → parse SX → merge into env → render content
- Console log: `sx:route client+data <pathname>` confirms client-side rendering
- Component deps computed for `:data` pages too (not just pure pages)
4. **Client data cache** — `orchestration.sx`
- `_page-data-cache` dict keyed by `page-name:param=value`
- 30s TTL (configurable via `_page-data-cache-ttl`)
- Cache hit: `sx:route client+cache <pathname>` — renders instantly
- Cache miss: fetches, caches, renders
- Stale entries evicted on next access
5. **Test page** — `sx/sx/data-test.sx`
- Exercises full data pipeline: server time, pipeline steps, phase/transport metadata
- Navigate from another page → console shows `sx:route client+data`
- Navigate back → console shows `sx:route client+cache`
6. **Unit tests** — `shared/sx/tests/test_page_data.py` (20 tests)
- Serialize roundtrip for all data types
- Kebab-case key conversion
- Component deps for `:data` pages
- Full pipeline simulation (serialize → parse → merge → eval)
**Files:**
- `shared/sx/ref/orchestration.sx` — `resolve-page-data` spec, data cache
- `shared/sx/ref/bootstrap_js.py` — platform `resolvePageData` implementation
- `shared/sx/pages.py` — `evaluate_page_data()`, `auto_mount_page_data()`
- `shared/sx/helpers.py` — deps for `:data` pages
- `sx/sx/data-test.sx` — test component
- `sx/sxc/pages/docs.sx` — test page defpage
- `sx/sxc/pages/helpers.py` — `data-test-data` helper
- `sx/sx/boundary.sx` — helper declaration
- `shared/sx/tests/test_page_data.py` — unit tests
---
### Phase 5: Async Continuations & Inline IO
**What it enables:** Components call IO primitives directly in their body (e.g. `(query ...)`). The evaluator suspends mid-evaluation, fetches data, resumes. Same component source works on both server (Python async/await) and client (continuation-based suspension).
**The problem:** The existing `shift/reset` continuations extension is synchronous (throw/catch). Client-side IO via `fetch()` returns a Promise, and you can't throw-catch across an async boundary. The evaluator needs Promise-aware continuations.
**Approach:**
1. **Async client evaluator** — two possible mechanisms:
- **Promise-based:** `evalExpr` returns value or Promise; rendering awaits
- **Continuation-based:** use existing `shift/reset` to suspend on IO, resume when data arrives (architecturally cleaner, leverages existing spec)
1. **Async-aware shift/reset** — extend the continuations extension:
- `sfShift` captures the continuation and returns a Promise
- `sfReset` awaits Promise results in the trampoline
- Continuation resume feeds the fetched value back into the evaluation
2. **IO primitive bridge** — register async IO primitives in client `PRIMITIVES`:
- `query` → fetch to `/internal/data/`
@@ -157,27 +168,22 @@ The key insight: **s-expressions can partially unfold on the server after IO, th
- `frag` → fetch fragment HTML
- `current-user` → cached from initial page load
3. **Client data cache** — keyed by `(service, query, params-hash)`, configurable TTL, server can invalidate via `SX-Invalidate` header.
3. **CPS transform option** — alternative to Promise-aware shift/reset:
- Transform the evaluator to continuation-passing style
- Every eval step takes a continuation argument
- IO primitives call the continuation after fetch resolves
- Architecturally cleaner but requires deeper changes
4. **Optimistic updates** — extend existing `apply-optimistic`/`revert-optimistic` in `engine.sx` from DOM-level to data-level.
**Files:**
- `shared/sx/ref/eval.sx` — async dispatch path (or new `async-eval.sx`)
- New: `shared/sx/ref/io-bridge.sx` — client IO implementations
- `shared/sx/ref/boot.sx` — register IO bridge at init
- `shared/sx/ref/bootstrap_js.py` — emit async-aware code
- `/internal/data/` endpoints — ensure client-accessible (CORS, auth)
**Depends on:** Phase 2 (IO affinity), Phase 3 (routing for when to trigger IO)
**Depends on:** Phase 4 (data endpoint infrastructure)
**Verification:**
- Client `(query ...)` returns identical data to server-side
- Data cache prevents redundant fetches
- Same component source → identical output on either side
- Component calling `(query ...)` on client fetches data and renders
- Same component source → identical output on server and client
- Suspension visible: placeholder → resolved content
---
### Phase 5: Streaming & Suspense
### Phase 6: Streaming & Suspense
**What it enables:** Server streams partially-evaluated SX as IO resolves. Client renders available subtrees immediately, fills in suspended parts. Like React Suspense but built on delimited continuations.
@@ -203,11 +209,11 @@ The key insight: **s-expressions can partially unfold on the server after IO, th
- New: `shared/sx/ref/suspense.sx` — client suspension rendering
- `shared/sx/ref/boot.sx` — handle resolution scripts
**Depends on:** Phase 4 (client async for filling suspended subtrees), Phase 2 (IO analysis for priority)
**Depends on:** Phase 5 (async continuations for filling suspended subtrees), Phase 2 (IO analysis for priority)
---
### Phase 6: Full Isomorphism
### Phase 7: Full Isomorphism
**What it enables:** Same SX code runs on either side. Runtime chooses optimal split. Offline-first with cached data + client eval.
@@ -226,11 +232,16 @@ The key insight: **s-expressions can partially unfold on the server after IO, th
```
Default: auto (runtime decides from IO analysis).
3. **Offline data layer** — Service Worker intercepts `/internal/data/` requests, serves from IndexedDB when offline, syncs when back online.
3. **Optimistic data updates** — extend existing `apply-optimistic`/`revert-optimistic` in `engine.sx` from DOM-level to data-level:
- Client updates cached data optimistically (e.g., like button increments count)
- Sends mutation to server
- If server confirms, keep; if rejects, revert cached data and re-render
4. **Isomorphic testing** — evaluate same component on Python and JS, compare output. Extends existing `test_sx_ref.py` cross-evaluator comparison.
4. **Offline data layer** — Service Worker intercepts `/internal/data/` requests, serves from IndexedDB when offline, syncs when back online.
5. **Universal page descriptor** — `defpage` is portable: server executes via `execute_page()`, client executes via route match → fetch data → eval content → render DOM. Same descriptor, different execution environment.
5. **Isomorphic testing** — evaluate same component on Python and JS, compare output. Extends existing `test_sx_ref.py` cross-evaluator comparison.
6. **Universal page descriptor** — `defpage` is portable: server executes via `execute_page()`, client executes via route match → fetch data → eval content → render DOM. Same descriptor, different execution environment.
**Depends on:** All previous phases.
@@ -259,15 +270,15 @@ All new behavior specified in `.sx` files under `shared/sx/ref/` before implemen
| File | Role | Phases |
|------|------|--------|
| `shared/sx/async_eval.py` | Core evaluator, `_aser`, server/client boundary | 2, 5 |
| `shared/sx/helpers.py` | `sx_page()`, `sx_response()`, output pipeline | 1, 3 |
| `shared/sx/async_eval.py` | Core evaluator, `_aser`, server/client boundary | 2, 6 |
| `shared/sx/helpers.py` | `sx_page()`, `sx_response()`, output pipeline | 1, 3, 4 |
| `shared/sx/jinja_bridge.py` | `_COMPONENT_ENV`, component registry | 1, 2 |
| `shared/sx/pages.py` | `defpage`, `execute_page()`, page lifecycle | 2, 3 |
| `shared/sx/ref/boot.sx` | Client boot, component caching | 1, 3, 4 |
| `shared/sx/ref/orchestration.sx` | Client fetch/swap/morph | 3, 4 |
| `shared/sx/ref/eval.sx` | Evaluator spec | 4 |
| `shared/sx/pages.py` | `defpage`, `execute_page()`, page lifecycle, data endpoint | 2, 3, 4 |
| `shared/sx/ref/boot.sx` | Client boot, component caching, page registry | 1, 3, 4 |
| `shared/sx/ref/orchestration.sx` | Client fetch/swap/morph, routing, data cache | 3, 4, 5 |
| `shared/sx/ref/eval.sx` | Evaluator spec | 5 |
| `shared/sx/ref/engine.sx` | Morph, swaps, triggers | 3 |
| New: `shared/sx/deps.py` | Dependency analysis | 1, 2 |
| New: `shared/sx/ref/router.sx` | Client-side routing | 3 |
| New: `shared/sx/ref/io-bridge.sx` | Client IO primitives | 4 |
| New: `shared/sx/ref/suspense.sx` | Streaming/suspension | 5 |
| `shared/sx/ref/deps.sx` | Dependency + IO analysis (spec) | 1, 2 |
| `shared/sx/ref/router.sx` | Client-side route matching | 3 |
| `shared/sx/ref/bootstrap_js.py` | JS bootstrapper, platform implementations | 4, 5 |
| New: `shared/sx/ref/suspense.sx` | Streaming/suspension | 6 |