Migrate all apps to defpage declarative page routes

Replace Python GET page handlers with declarative defpage definitions in .sx files across all 8 apps (sx docs, orders, account, market, cart, federation, events, blog). Each app now has sxc/pages/ with setup functions, layout registrations, page helpers, and .sx defpage declarations. Core infrastructure: add g I/O primitive, PageDef support for auth/layout/ data/content/filter/aside/menu slots, post_author auth level, and custom layout registration. Remove ~1400 lines of render_*_page/render_*_oob boilerplate. Update all endpoint references in routes, sx_components, and templates to defpage_* naming. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 14:52:34 +00:00
parent 5b4cacaf19
commit c243d17eeb
108 changed files with 3598 additions and 2851 deletions
--- a/docs/isomorphic-sx-plan.md
+++ b/docs/isomorphic-sx-plan.md
@@ -0,0 +1,360 @@
+# Isomorphic SX Architecture Migration Plan
+
+## Context
+
+The sx layer already renders full pages client-side — `sx_page()` ships raw sx source + component definitions to the browser, `sx.js` evaluates and renders them. Components are cached in localStorage with a hash-based invalidation protocol (cookie `sx-comp-hash` → server skips sending defs if hash matches).
+
+**Key insight from the user:** Pages/routes are just components. They belong in the same component registry, cached in localStorage alongside `defcomp` definitions. On navigation, if the client's component hash is current, the server doesn't need to send any s-expression source at all — just data. The client already has the page component cached and renders it locally with fresh data from the API.
+
+### Target Architecture
+
+```
+First visit:
+  Server → component defs (including page components) + page data → client caches defs in localStorage
+
+Subsequent navigation (same session, hash valid):
+  Client has page component cached → fetches only JSON data from /api/data/ → renders locally
+  Server sends: { data: {...} } — zero sx source
+
+SSR (bots, first paint):
+  Server evaluates the same page component with direct DB queries → sends rendered HTML
+  Client hydrates (binds SxEngine handlers, no re-render)
+```
+
+This is React-like data fetching with an s-expression view layer instead of JSX, and the component transport is a content-addressed cache rather than a JS bundle.
+
+### Data Delivery Modes
+
+The data side is not a single pattern — it's a spectrum that can be mixed per page and per fragment:
+
+**Mode A: Server-bundled data** — Server evaluates the page's `:data` slot, resolves all queries (including cross-service `fetch_data` calls), returns one JSON blob. Fewest round-trips. Server aggregates.
+
+**Mode B: Client-fetched data** — Client evaluates `:data` slot locally. Each `(query ...)` / `(service ...)` hits the relevant service's `/api/data/` endpoint independently. More round-trips but fully decoupled — each service handles its own data.
+
+**Mode C: Hybrid** — Server bundles same-service data (direct DB). Client fetches cross-service data in parallel from other services' APIs. Mirrors current server pattern: own-domain = SQLAlchemy, cross-domain = `fetch_data()` HTTP.
+
+The same spectrum applies to **fragments** (`frag` / `fetch_fragment`):
+
+- **Server-composed:** Server calls `fetch_fragment()` during page evaluation, bakes result into data bundle or renders inline.
+- **Client-composed:** Client's `(frag ...)` primitive fetches from the service's public fragment endpoint. Fragment returns sx source, client renders locally using cached component defs.
+- **Mixed:** Stable fragments (nav, auth menu) server-composed; content-specific fragments client-fetched.
+
+A `(query ...)` or `(frag ...)` call resolves differently depending on execution context (server vs client) but produces the same result. The choice of mode can be per-page, per-fragment, or even per-request.
+
+## Delivery Order
+
+```
+Phase 1 (Primitive Parity) ──┐
+                              ├── Phase 4 (Client Data Primitives) ──┐
+Phase 3 (Public Data API) ───┘                                      ├── Phase 5 (Data-Only Navigation)
+Phase 2 (Server-Side Rendering) ────────────────────────────────────┘
+```
+
+Phases 1-3 are independent. Recommended order: **3 → 1 → 2 → 4 → 5**
+
+---
+
+## Phase 1: Primitive Parity
+
+Align JS and Python primitive sets so the same component source evaluates identically on both sides.
+
+### 1a: Add missing pure primitives to sx.js
+
+Add to `PRIMITIVES` in `shared/static/scripts/sx.js`:
+
+| Primitive | JS implementation |
+|-----------|-------------------|
+| `clamp` | `Math.max(lo, Math.min(hi, x))` |
+| `chunk-every` | partition list into n-size sublists |
+| `zip-pairs` | `[[coll[0],coll[1]], [coll[2],coll[3]], ...]` |
+| `dissoc` | shallow copy without specified keys |
+| `into` | target-type-aware merge |
+| `format-date` | minimal strftime translator covering `%Y %m %d %b %B %H %M %S` |
+| `parse-int` | `parseInt` with NaN fallback to default |
+| `assert` | throw if falsy |
+
+Fix existing parity gaps: `round` needs optional `ndigits`; `min`/`max` need to accept a single list arg.
+
+### 1b: Inject `window.__sxConfig` for server-context primitives
+
+Modify `sx_page()` in `shared/sx/helpers.py` to inject before sx.js:
+
+```js
+window.__sxConfig = {
+  appUrls: { blog: "https://blog.rose-ash.com", ... },
+  assetUrl: "https://static...",
+  config: { /* public subset */ },
+  currentUser: { id, username, display_name, avatar } | null,
+  relations: [ /* serialized RelationDef list */ ]
+};
+```
+
+Sources: `ctx` has `blog_url`, `market_url`, etc. `g.user` has user info. `shared/infrastructure/urls.py` has the URL map.
+
+Add JS primitives reading from `__sxConfig`: `app-url`, `asset-url`, `config`, `current-user`, `relations-from`.
+
+`url-for` has no JS equivalent — isomorphic code uses `app-url` instead.
+
+### 1c: Add `defpage` to sx.js evaluator
+
+Add `defpage` to `SPECIAL_FORMS`. Parse the declaration, store it in `_componentEnv` under `"page:name"` (same registry as components). The page definition includes: name, path pattern, auth requirement, layout spec, and unevaluated AST for data/content/filter/aside/menu slots.
+
+Since pages live in `_componentEnv`, they're automatically included in the component hash, cached in localStorage, and skipped when the hash matches. No separate `<script data-pages>` block needed — they ship with components.
+
+**Files:** `shared/static/scripts/sx.js`, `shared/sx/helpers.py`
+
+**Verify:** `(format-date "2024-03-15" "%d %b %Y")` produces same output in Python and JS.
+
+---
+
+## Phase 2: Server-Side Rendering (SSR)
+
+Full-page HTML rendering on the server for SEO and first-paint.
+
+### 2a: Add `render_mode` to `execute_page()`
+
+In `shared/sx/pages.py`:
+
+```python
+async def execute_page(..., render_mode: str = "client") -> str:
+```
+
+When `render_mode="server"`:
+- Evaluate all slots via `async_render()` (→ HTML) instead of `async_eval_to_sx()` (→ sx source)
+- Layout headers also rendered to HTML
+- Pass to new `ssr_page()` instead of `sx_page()`
+
+### 2b: Create `ssr_page()` in helpers.py
+
+Wraps pre-rendered HTML in a document shell:
+- Same `<head>` (CSS, CSRF, meta)
+- Rendered HTML inline in `<body>` — no `<script type="text/sx" data-mount>`
+- Still ships component defs in `<script type="text/sx" data-components>` (client needs them for subsequent navigation)
+- Still includes sx.js + body.js (for SPA takeover after first paint)
+- Adds `<meta name="sx-ssr" content="true">`
+- Injects `__sxConfig` (Phase 1b)
+
+### 2c: SSR trigger
+
+Utility `should_ssr(request)`:
+- Bot UA patterns → SSR
+- `?_render=server` → SSR (debug)
+- `SX-Request: true` header → always client
+- Per-page opt-in via `defpage :ssr true`
+- Default → client (current behavior)
+
+### 2d: Hydration in sx.js
+
+When sx.js detects `<meta name="sx-ssr">`:
+- Skip `Sx.mount()` — DOM already correct
+- Run `SxEngine.process(document.body)` — bind sx-get/post handlers
+- Run `Sx.hydrate()` — process `[data-sx]` elements
+- Load component defs into registry (for subsequent navigations)
+
+**Files:** `shared/sx/pages.py`, `shared/sx/helpers.py`, `shared/static/scripts/sx.js`
+
+**Verify:** Googlebot UA → response has rendered HTML, no `<script data-mount>`. Normal UA → unchanged behavior.
+
+---
+
+## Phase 3: Public Data API
+
+Expose browser-accessible JSON endpoints mirroring internal `/internal/data/` queries.
+
+### 3a: Shared blueprint factory
+
+New `shared/sx/api_data.py`:
+
+```python
+def create_public_data_blueprint(service_name: str) -> Blueprint:
+    """Session-authed public data blueprint at /api/data/"""
+```
+
+Queries registered with auth level: `"public"`, `"login"`, `"admin"`. Validates session (not HMAC). Returns JSON.
+
+### 3b: Extract and share handler implementations
+
+Refactor `bp/data/routes.py` per service — separate query logic from HMAC auth. Same function serves both internal and public paths.
+
+### 3c: Per-service public data blueprints
+
+New `bp/api_data/routes.py` per service:
+
+| Service | Public queries | Auth |
+|---------|---------------|------|
+| blog | `post-by-slug`, `post-by-id`, `search-posts` | public |
+| market | `products-by-ids`, `marketplaces-for-container` | public |
+| events | `visible-entries-for-period`, `calendars-for-container`, `entries-for-page` | public |
+| cart | `cart-summary`, `cart-items` | login |
+| likes | `is-liked`, `liked-slugs` | login |
+| account | `newsletters` | public |
+
+Admin queries and write-actions stay internal only.
+
+### 3d: Public fragment endpoints
+
+The existing internal fragment system (`/internal/fragments/<type>`, HMAC-signed) needs public equivalents. Each service already has `create_handler_blueprint()` mounting defhandler fragments. Add a parallel public endpoint:
+
+`GET /api/fragments/<type>?params...` — session-authed, returns `text/sx` (same wire format the client already handles via SxEngine).
+
+This can reuse the same `execute_handler()` machinery — the only difference is auth (session vs HMAC). The blueprint factory in `shared/sx/api_data.py` can handle both data and fragment registration:
+
+```python
+bp.register_fragment("container-cards", handler_fn, auth="public")
+```
+
+The client's `(frag ...)` primitive then fetches from these public endpoints instead of the HMAC-signed internal ones.
+
+### 3e: Register in app factories
+
+Each service's `app.py` registers the new blueprint.
+
+**Files:** New `shared/sx/api_data.py`, new `{service}/bp/api_data/routes.py` per service, `{service}/app.py`
+
+**Verify:** `curl /api/data/post-by-slug?slug=test` → JSON. `curl /api/fragments/container-cards?type=page&id=1` → sx source. Login-gated query without session → 401.
+
+---
+
+## Phase 4: Client Data Primitives
+
+Async data-fetching in sx.js so I/O primitives work client-side via the public API.
+
+### 4a: Async evaluator — `sxEvalAsync()`
+
+New function in `sx.js` returning a `Promise`. Mirrors `async_eval.py`:
+- Literals/symbols → `Promise.resolve(syncValue)`
+- I/O primitives (`query`, `service`, `frag`, etc.) → `fetch()` calls to `/api/data/`
+- Control flow → sequential async with short-circuit
+- `map`/`filter` with I/O → `Promise.all`
+
+### 4b: I/O primitive dispatch
+
+```javascript
+IO_PRIMITIVES = {
+  "query":        (svc, name, kw) => fetch(__sxConfig.appUrls[svc] + "/api/data/" + name + "?" + params(kw), {credentials:"include"}).then(r=>r.json()),
+  "service":      (method, kw)    => fetch("/api/data/" + method + "?" + params(kw), {credentials:"include"}).then(r=>r.json()),
+  "frag":         (svc, type, kw) => fetch(__sxConfig.appUrls[svc] + "/api/fragments/" + type + "?" + params(kw), {credentials:"include"}).then(r=>r.text()),
+  "current-user": ()              => Promise.resolve(__sxConfig.currentUser),
+  "request-arg":  (name)          => Promise.resolve(new URLSearchParams(location.search).get(name)),
+  "request-path": ()              => Promise.resolve(location.pathname),
+  "nav-tree":     ()              => fetch("/api/data/nav-tree", {credentials:"include"}).then(r=>r.json()),
+};
+```
+
+### 4c: Async DOM renderer — `renderDOMAsync()`
+
+Two-pass (avoids restructuring sync renderer):
+1. Walk AST, collect I/O call sites with placeholders
+2. `Promise.all` to resolve all I/O in parallel
+3. Substitute resolved values into AST
+4. Call existing sync `renderDOM()` on resolved tree
+
+### 4d: Wire into `Sx.mount()`
+
+Detect I/O nodes. If present → async path. Otherwise → existing sync path (zero overhead for pure components).
+
+**Files:** `shared/static/scripts/sx.js` (major addition)
+
+**Verify:** Page with `(query "blog" "post-by-slug" :slug "test")` in sx source → client fetches `/api/data/post-by-slug?slug=test`, renders result.
+
+---
+
+## Phase 5: Data-Only Navigation
+
+When the client already has page components cached, navigation requires only a data fetch — no sx source from the server.
+
+### 5a: Page components in the registry
+
+`defpage` definitions are already in `_componentEnv` (Phase 1c) and cached in localStorage with the component hash. On navigation, if the hash is valid, the client has all page definitions locally.
+
+Build a `_pageRegistry` mapping URL path patterns → page definitions, populated when `defpage` forms are evaluated. Path patterns (`/posts/<slug>/`) converted to regex matchers for URL matching.
+
+### 5b: Navigation intercept
+
+Extend SxEngine's link click handler:
+
+```
+1. Extract URL path from clicked link
+2. Match against _pageRegistry
+3. If matched:
+   a. Evaluate :data slot via sxEvalAsync() → parallel API fetches
+   b. Render :content/:filter/:aside via renderDOMAsync()
+   c. Morph into existing ~app-body (headers persist, slots update)
+   d. Push history state
+   e. Update document title
+4. If not matched → existing server fetch (graceful fallback)
+```
+
+### 5c: Data delivery — flexible per page
+
+Three modes available (see Context section). The page definition can declare its preference:
+
+```scheme
+(defpage blog-post
+  :path "/posts/<slug>/"
+  :data-mode :server    ; :server (bundled), :client (fetch individually), :hybrid
+  :data (query "blog" "post-by-slug" :slug slug)
+  :content (~post-detail post))
+```
+
+**Mode :server** — Client sends `SX-Page: blog-post` header on navigation. Server evaluates `:data` slot (all queries, including cross-service), returns single JSON blob:
+```python
+if request.headers.get("SX-Page"):
+    data = await evaluate_data_slot(page_def, url_params)
+    return jsonify(data)
+```
+
+**Mode :client** — Client evaluates `:data` slot locally via `sxEvalAsync()`. Each `(query ...)` hits `/api/data/` independently. Each `(frag ...)` hits `/api/fragments/`. No server data endpoint needed.
+
+**Mode :hybrid** — Server bundles own-service data (direct DB). Client fetches cross-service data and fragments in parallel. The `:data` slot is split: server evaluates local queries, returns partial bundle + a manifest of remaining queries. Client resolves the rest.
+
+Default mode can be `:server` (fewest round-trips, simplest). Pages opt into `:client` or `:hybrid` when they want more decoupling or when cross-service data is heavy and benefits from parallel client fetches.
+
+### 5d: Popstate handling
+
+On browser back/forward:
+1. Check `_pageRegistry` for popped URL
+2. If matched → client render (same as 5b)
+3. If not → existing server fetch + morph
+
+### 5e: Graceful fallback
+
+Routes not in `_pageRegistry` fall through to server fetch. Partially migrated apps work — Python-only routes use server fetch, defpage routes get SPA behavior. No big-bang cutover.
+
+**Files:** `shared/static/scripts/sx.js`, `shared/sx/helpers.py`, `shared/sx/pages.py`
+
+**Verify:** Playwright: load page → click link to defpage route → assert no HTML response fetched (only JSON) → content correct → URL updated → back button works.
+
+---
+
+## Summary: The Full Lifecycle
+
+```
+1. App startup: Python loads .sx files → defcomp + defpage registered in _COMPONENT_ENV
+                                        → hash computed
+
+2. First visit:  Server sends HTML shell + component/page defs + __sxConfig + page sx source
+                  Client evaluates, renders, caches defs in localStorage, sets cookie
+
+3. Return visit: Cookie hash matches → server sends HTML shell with empty <script data-components>
+                  Client loads defs from localStorage → renders page
+
+4. SPA navigation: Client matches URL against _pageRegistry
+                    → fetches data from /api/data/ (or server data-only endpoint)
+                    → renders page component locally with fresh data
+                    → morphs DOM, pushes history
+                    → zero sx source transferred
+
+5. Bot/SSR:      Server detects bot UA → evaluates page server-side with direct DB queries
+                  → sends rendered HTML + component defs
+                  → client hydrates (binds handlers, no re-render)
+```
+
+## Migration per Service
+
+Each service migrates independently, no coordination needed:
+1. Add public data blueprint (Phase 3) — immediate standalone value
+2. Convert remaining Jinja routes to `defpage` — already in progress
+3. Enable SSR for bots (Phase 2) — per-page opt-in
+4. Client data primitives (Phase 4) — global once sx.js updated
+5. Data-only navigation (Phase 5) — automatic for any `defpage` route