coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HS parser: fix number+comparison keyword collision, eval-hs uses hs-compile

Browse Source 
Parser: skip unit suffix when next ident is a comparison keyword
(starts, ends, contains, matches, is, does, in, precedes, follows).
Fixes "123 starts with '12'" returning "123starts" instead of true.

eval-hs: use hs-compile directly instead of hs-to-sx-from-source with
"return " prefix, which was causing the parser to consume the comparison
as a string suffix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
giles
2026-04-15 11:29:01 +00:00
parent a93e5924df
commit 4f02f82f4e
377 changed files with 9517 additions and 8694 deletions
Download Patch File Download Diff File Expand all files Collapse all files

127
sx/sx/plans/art-dag-sx.sx
View File

@@ -1,127 +0,0 @@
;; ---------------------------------------------------------------------------
;; Art DAG on SX — SX endpoints as portals into media processing environments
;; ---------------------------------------------------------------------------
(defcomp ~plans/art-dag-sx/plan-art-dag-sx-content ()
(~docs/page :title "Art DAG on SX"
(p (~tw :tokens "text-stone-500 text-sm italic mb-8")
"An SX endpoint is a portal into an execution environment. The URL is the entry point; the boundary declaration determines what world you enter.")
;; =====================================================================
;; I. The endpoint as portal
;; =====================================================================
(~docs/section :title "The endpoint as portal" :id "endpoint-as-portal"
(p "An SX endpoint isn't a static route handler. It's a portal into an execution environment. A blog endpoint has " (code "query-posts") ", " (code "render-markdown") ". An art-dag endpoint has " (code "fetch-recipe") ", " (code "resolve-cid") ", " (code "gpu-exec") ", " (code "encode-stream") ", " (code "open-feed") ". Same evaluator. Different primitives. Different capabilities.")
(p "The URL is the entry point. The boundary declaration determines what world you enter. When you hit " (code "/art/render/Qm...") ", the evaluator boots with media-processing primitives. When you hit " (code "/blog/post/hello") ", the evaluator boots with content primitives. The SX code looks the same either way " (em "- ") "function calls, let bindings, conditionals. But the set of primitives available changes everything about what the program can do.")
(p "This is the same principle as the browser/server split. The browser has " (code "render-to-dom") " and " (code "signal") ". The server has " (code "query-db") " and " (code "render-to-html") ". Neither is a restricted version of the other " (em "- ") "they are different environments with different capabilities. The art-dag environment is a third world: " (code "gpu-exec") ", " (code "resolve-cid") ", " (code "encode-stream") ". Same language. Different universe."))
;; =====================================================================
;; II. Recipes as SX
;; =====================================================================
(~docs/section :title "Recipes as SX" :id "recipes-as-sx"
(p "Art DAG recipes are already s-expression effect chains. Currently executed by L1 Celery workers via Python. The SX version: recipes " (em "are") " SX programs. They evaluate in an environment that has media processing primitives. A recipe doesn't \"call\" a GPU function " (em "- ") "it evaluates in an environment where " (code "gpu-exec") " is a primitive.")
(~docs/code :src (highlight ";; A recipe is an SX program in a media-processing environment\n(define composite-layers\n (fn (base-cid overlay-cid blend)\n (let ((base (resolve-cid base-cid))\n (overlay (resolve-cid overlay-cid)))\n (gpu-exec :op \"composite\"\n :layers (list base overlay)\n :blend blend\n :output :stream))))" "lisp"))
(p "This isn't a DSL embedded in Python. It's SX, the same language that renders pages and defines components. The recipe author uses " (code "let") ", " (code "fn") ", " (code "map") ", " (code "if") " " (em "- ") "the full language. The only difference is what primitives are available. " (code "resolve-cid") " fetches content-addressed data. " (code "gpu-exec") " dispatches GPU operations. These are primitives, not library calls. They exist in the environment the same way " (code "+") " and " (code "str") " exist.")
(p "The recipe is data. It's an s-expression. You can parse it, analyze it, transform it, serialize it, hash it, store it, transmit it. You can inspect a recipe's dependency graph the same way " (code "deps.sx") " inspects component dependencies. You can type-check a recipe the same way " (code "typed-sx") " type-checks components. The tools already exist. They just need a new set of primitives to reason about."))
;; =====================================================================
;; III. Split execution
;; =====================================================================
(~docs/section :title "Split execution" :id "split-execution"
(p "Some recipe steps run against cached data (fast, local). Others need GPU. Others need live input. The evaluator doesn't dispatch " (em "- ") "the boundary declarations do. " (code "(with-boundary (gpu-compute) ...)") " migrates to a GPU-capable host. " (code "(with-boundary (live-ingest) ...)") " opens WebRTC feeds. The recipe author doesn't manage infrastructure " (em "- ") "they declare capabilities, and execution flows to where they exist.")
(~docs/code :src (highlight "(define live-composite\n (fn (recipe-cid camera-count)\n (let ((recipe (resolve-cid recipe-cid)))\n ;; Phase 1: cached data (local, fast)\n (let ((base-layers (map resolve-cid (get recipe \"layers\"))))\n ;; Phase 2: GPU processing\n (with-boundary (gpu-compute)\n (let ((composed (gpu-exec :op \"composite\"\n :layers base-layers\n :blend (get recipe \"blend\"))))\n ;; Phase 3: live feeds\n (with-boundary (live-ingest)\n (let ((feeds (map (fn (i)\n (open-feed :protocol \"webrtc\"\n :label (str \"camera-\" i)))\n (range 0 camera-count))))\n ;; Phase 4: encode and stream\n (with-boundary (encoding)\n (encode-stream\n :sources (concat (list composed) feeds)\n :codec \"h264\"\n :output :stream))))))))))" "lisp"))
(p "Four phases. Four capability requirements. The program reads linearly " (em "- ") "resolve cached layers, composite on GPU, open live feeds, encode output. But execution migrates across hosts as needed. The " (code "with-boundary") " blocks are the seams. Everything inside a boundary block runs on a host that provides those capabilities. Everything outside runs wherever the program started.")
(p "This is the same mechanism described in the generative SX plan's environment migration section. " (code "with-boundary") " serializes the environment (" (code "env-snapshot") "), ships the pending expression to a capable host, and execution continues there. The recipe author writes a linear program. The runtime makes it distributed."))
;; =====================================================================
;; IV. Content-addressed everything
;; =====================================================================
(~docs/section :title "Content-addressed everything" :id "content-addressed"
(p "Recipes are CIDs. Effects are CIDs. Intermediate frames are content-addressed. The execution DAG is a content-addressed graph " (em "- ") "every step can be verified, cached, or replayed.")
(p "A composite of three layers with a specific blend mode always produces the same CID. Caching is automatic: if the CID exists locally, skip the computation. This is the Art DAG's existing model " (em "- ") "SHA3-256 hashes identify everything. SX makes it explicit: the recipe source itself is content-addressed. Two users who write the same recipe get the same CID. They're running the same program.")
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4 my-4")
(p (~tw :tokens "text-stone-700 font-medium mb-2") "Content addressing as memoization")
(p (~tw :tokens "text-stone-600 text-sm") "Every function call with content-addressed inputs has a content-addressed output. " (code "(gpu-exec :op \"composite\" :layers (list CID-a CID-b) :blend \"multiply\")") " always produces the same result CID. The runtime can check: does this output CID exist? If yes, skip the computation. The entire execution DAG becomes a cache key. Rerunning a recipe that's already been computed is instantaneous " (em "- ") "every intermediate result already exists."))
(p "The execution trace is also content-addressed. You can inspect exactly what happened: which CIDs were resolved, which GPU operations ran, which feeds were opened, what the final output was. The trace is the recipe's proof of work. It's immutable, verifiable, and shareable."))
;; =====================================================================
;; V. Feed generation
;; =====================================================================
(~docs/section :title "Feed generation" :id "feed-generation"
(p "The executing program creates new endpoints as a side effect. " (code "(open-feed ...)") " doesn't return data " (em "- ") "it returns a connectable endpoint. WebRTC peers, SSE streams, WebSocket channels. These are generative acts: the endpoint didn't exist before the recipe ran. The program grew its own input surface.")
(p "When the recipe completes or the island disposes, feeds are cleaned up via the disposal mechanism. An " (code "effect") " in island scope that opens a feed returns a cleanup function. When the island unmounts, the cleanup runs, the feed closes, the endpoint disappears. The lifecycle is automatic.")
(~docs/code :src (highlight ";; A feed is a connectable endpoint, not raw data\n(define create-camera-mosaic\n (fn (camera-ids)\n ;; Each open-feed returns a connectable URL, not bytes\n (let ((feeds (map (fn (id)\n (open-feed :protocol \"webrtc\"\n :label (str \"cam-\" id)\n :quality \"720p\"))\n camera-ids)))\n ;; The mosaic recipe composes feeds as inputs\n (gpu-exec :op \"mosaic\"\n :inputs feeds\n :layout \"grid\"\n :output (open-feed :protocol \"sse\"\n :label \"mosaic-output\"\n :format \"mjpeg\")))))" "lisp"))
(p "The output is itself a feed. A client connects to the mosaic output URL and receives composed frames. The feeds are the program's I/O surface " (em "- ") "they exist because the program created them, and they die when the program stops. No static route configuration. No service mesh. The program declares what it needs and creates what it produces."))
;; =====================================================================
;; VI. The client boundary
;; =====================================================================
(~docs/section :title "The client boundary" :id "client-boundary"
(p "The browser is just another execution environment with its own primitive set: " (code "render-to-dom") ", " (code "signal") ", " (code "deref") ", " (code "open-feed") " (as WebRTC consumer). A streaming art-dag response arrives as SX wire format. The client evaluates it in island scope " (em "- ") "signals bind to stream frames, " (code "reactive-list") " renders feed thumbnails, " (code "computed") " derives overlay parameters. The server pushes frames; the client renders them reactively. No special video player " (em "- ") "just signals and DOM.")
(~docs/code :src (highlight "(defisland ~plans/art-dag-sx/live-canvas ()\n (let ((frames (signal nil))\n (feed-url (signal nil)))\n ;; Connect to stream when URL arrives\n (effect (fn ()\n (when (deref feed-url)\n (connect-stream (deref feed-url)\n :on-frame (fn (f) (reset! frames f))))))\n (div :class \"relative aspect-video bg-black rounded\"\n (when (deref frames)\n (canvas :width 1920 :height 1080\n :draw (fn (ctx)\n (draw-frame ctx (deref frames)))))\n (when (not (deref frames))\n (p :class \"absolute inset-0 flex items-center justify-center text-white/50\"\n \"Waiting for stream...\")))))" "lisp"))
(p "The island is reactive. When " (code "frames") " updates, only the canvas redraws. When " (code "feed-url") " updates, the effect reconnects. No polling loop. No WebSocket message handler parsing JSON. The stream is a signal source. The DOM is a signal consumer. The reactive graph connects them.")
(p "This is the same island architecture from the reactive islands plan " (em "- ") "signals, effects, computed, disposal. The only difference is the data source. Instead of an HTMX response mutating a signal, a WebRTC stream mutates a signal. The rendering pipeline doesn't know or care where the data comes from. It reacts."))
;; =====================================================================
;; VII. L1/L2 integration
;; =====================================================================
(~docs/section :title "L1/L2 integration" :id "l1-l2"
(p "L1 is the compute layer (Celery workers, GPU nodes). L2 is the registry (ActivityPub, recipe discovery). In SX terms: L1 hosts provide " (code "gpu-exec") ", " (code "encode-stream") ", " (code "resolve-cid-local") ". L2 hosts provide " (code "discover-recipe") ", " (code "publish-recipe") ", " (code "federate-activity") ".")
(p "An SX program that needs both crosses boundaries as needed " (em "- ") "fetch recipe metadata from L2, execute it on L1, publish results back to L2.")
(~docs/code :src (highlight ";; A full pipeline crossing L1 and L2 boundaries\n(define render-and-publish\n (fn (recipe-name output-label)\n ;; L2: discover the recipe\n (with-boundary (registry)\n (let ((recipe-cid (discover-recipe :name recipe-name\n :version \"latest\")))\n ;; L1: execute the recipe\n (with-boundary (gpu-compute)\n (let ((result-cid (gpu-exec :recipe (resolve-cid recipe-cid)\n :output :cid)))\n ;; L2: publish the result\n (with-boundary (registry)\n (publish-recipe\n :name output-label\n :input-cid recipe-cid\n :output-cid result-cid\n :activity \"Create\"))))))))" "lisp"))
(p "Three boundary crossings. L2 to find the recipe. L1 to execute it. L2 to publish the result. The program reads as a linear sequence of operations. The runtime handles the dispatch " (em "- ") "which host provides " (code "discover-recipe") ", which host provides " (code "gpu-exec") ", which host provides " (code "publish-recipe") ". The program author doesn't configure endpoints or manage connections. They declare capabilities."))
;; =====================================================================
;; VIII. The primitive sets
;; =====================================================================
(~docs/section :title "The primitive sets" :id "primitive-sets"
(p "Each execution environment provides its own set of primitives. The language is the same everywhere. The capabilities differ.")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Environment")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Primitives")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Runs on")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-medium text-stone-700") "Browser")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "render-to-dom, signal, deref, connect-stream")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Client"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-medium text-stone-700") "App server")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "query-db, render-to-html, fetch-fragment")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Quart service"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-medium text-stone-700") "L1 Worker")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "gpu-exec, resolve-cid, encode-stream, cache-put")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Celery + GPU"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-medium text-stone-700") "L2 Registry")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "discover-recipe, publish-recipe, federate")
(td (~tw :tokens "px-3 py-2 text-stone-600") "FastAPI"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-medium text-stone-700") "Live Ingest")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "open-feed, capture-frame, transcode")
(td (~tw :tokens "px-3 py-2 text-stone-600") "WebRTC gateway"))
(tr
(td (~tw :tokens "px-3 py-2 font-medium text-stone-700") "IPFS Node")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "pin-cid, resolve-cid, dag-put, dag-get")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Kubo")))))
(p "A pure SX program (no IO primitives) runs on all six. A program that calls " (code "gpu-exec") " runs on L1 workers. A program that calls " (code "render-to-dom") " runs in the browser. The boundary declaration is the type signature of the environment. It tells you where the program can execute.")
(p "Adding a new environment means declaring a new primitive set. A hypothetical audio-processing environment would provide " (code "mix-tracks") ", " (code "apply-effect") ", " (code "encode-audio") ". A program that uses those primitives runs wherever that environment is hosted. The language doesn't change. The evaluator doesn't change. Only the available primitives change.")
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 my-4")
(p (~tw :tokens "text-violet-900 font-medium") "One language, many worlds")
(p (~tw :tokens "text-violet-800 text-sm") "The art-dag integration isn't a new feature bolted onto SX. It's a demonstration of what SX already is: a language where the execution environment is parameterized by its primitive set. The browser, the app server, the GPU worker, and the IPFS node all run the same evaluator. They differ only in what primitives they provide. The art-dag is just another world you can enter through an endpoint.")))
))

123
sx/sx/plans/async-eval-convergence.sx
View File

@@ -1,123 +0,0 @@
;; ---------------------------------------------------------------------------
;; Async Evaluator Convergence — Bootstrap async_eval.py from Spec
;; ---------------------------------------------------------------------------
(defcomp ~plans/async-eval-convergence/plan-async-eval-convergence-content ()
(~docs/page :title "Async Evaluator Convergence"
(~docs/section :title "The Problem" :id "problem"
(p "There are currently " (strong "three") " lambda call implementations that must be kept in sync:")
(ol (~tw :tokens "list-decimal list-inside space-y-2 mt-2")
(li (code "shared/sx/ref/eval.sx") " — the canonical spec, bootstrapped to " (code "sx-ref.js") " and " (code "sx_ref.py"))
(li (code "shared/sx/evaluator.py") " — hand-written synchronous Python evaluator")
(li (code "shared/sx/async_eval.py") " — hand-written asynchronous Python evaluator (the production server path)"))
(p "Every semantic change to the evaluator — lenient lambda arity, new special forms, calling convention tweaks — must be applied to all three. The spec is authoritative but " (code "async_eval.py") " is what actually serves pages. This is a maintenance hazard and a source of subtle divergence bugs.")
(p "The lenient arity change (lambda params pad missing args with nil instead of erroring) exposed this: the spec and sync evaluator were updated, but " (code "async_eval.py") " still had strict arity checking, causing production crashes."))
(~docs/section :title "Why async_eval.py Exists" :id "why"
(p "The async evaluator exists because SX page rendering needs to:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Await IO primitives") " — page helpers like " (code "highlight") ", " (code "reference-data") ", " (code "component-source") " call Python async functions (DB queries, HTTP fetches). The spec evaluator is synchronous.")
(li (strong "Expand server-affinity components") " — " (code ":affinity :server") " components must be fully expanded server-side before serialising to SX wire format. This requires async eval of the component body.")
(li (strong "Handle the aser rendering mode") " — the " (code "_aser") " function evaluates control flow server-side but serialises HTML tags and component calls as SX source for the client. This hybrid eval/serialize mode isn't in the spec."))
(p "None of these require " (em "different") " evaluation semantics — they require the " (em "same") " semantics with async IO at the boundary."))
;; -----------------------------------------------------------------------
;; Architecture
;; -----------------------------------------------------------------------
(~docs/section :title "Target Architecture" :id "architecture"
(p "The goal is to " (strong "eliminate hand-written evaluator code entirely") ". All evaluation semantics come from the spec via bootstrapping. The host provides only:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Platform primitives") " — type constructors, env operations, DOM/HTML primitives")
(li (strong "Async IO bridge") " — a thin wrapper that makes the bootstrapped evaluator await-compatible")
(li (strong "Rendering modes") " — aser/render-to-html dispatch, already mostly specced in " (code "render.sx")))
(p "The bootstrapped " (code "sx_ref.py") " already has correct eval semantics. The question is how to make it async-aware without forking the spec."))
;; -----------------------------------------------------------------------
;; Approach: Async Adapter Layer
;; -----------------------------------------------------------------------
(~docs/section :title "Approach: Async Adapter Layer" :id "approach"
(p "Rather than making the spec itself async (which would pollute it with Python-specific concerns), introduce a thin adapter layer between the bootstrapped evaluator and the IO boundary:")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Phase 1 — Async call hook")
(p "The bootstrapped evaluator calls primitives via " (code "apply(fn, args)") ". In the Python host, " (code "apply") " is a platform primitive. Replace it with an async-aware version:")
(~docs/code :src (highlight "(define apply-fn\n (fn (f args)\n ;; Platform provides: if f returns a coroutine, await it\n (apply-maybe-async f args)))" "lisp"))
(p "The bootstrapper emits " (code "apply_maybe_async") " as a Python " (code "async def") " that checks if the result is a coroutine and awaits it if so. Pure functions return immediately. IO primitives return coroutines that get awaited. " (strong "Zero overhead for pure calls") " — just an " (code "isinstance") " check.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Phase 2 — Async trampoline")
(p "The spec's trampoline loop resolves thunks synchronously. The Python bootstrapper emits an " (code "async def trampoline") " variant that can await thunks whose bodies contain IO calls. The trampoline structure is identical — only the " (code "await") " keyword is added.")
(~docs/code :src (highlight "# Bootstrapper emits this for Python async target\nasync def trampoline(val):\n while isinstance(val, Thunk):\n val = await eval_expr(val.expr, val.env)\n return val" "python"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Phase 3 — Aser as spec module")
(p "The " (code "_aser") " rendering mode (evaluate control flow, serialize HTML/components as SX source) should be specced as a module in " (code "render.sx") " alongside " (code "render-to-html") " and " (code "render-to-dom") ". It's currently hand-written Python because it predates the spec, but its logic is pure SX: walk the AST, eval certain forms, serialize others.")
(p "Once aser is specced, the bootstrapper emits it with the same async adapter — IO calls within aser bodies get awaited transparently.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Phase 4 — Delete hand-written evaluators")
(p "With the async adapter + specced aser, " (code "evaluator.py") " and " (code "async_eval.py") " become dead code. Delete them. All evaluation flows through the bootstrapped " (code "sx_ref.py") " with async adapter."))
;; -----------------------------------------------------------------------
;; What changes in the bootstrapper
;; -----------------------------------------------------------------------
(~docs/section :title "Bootstrapper Changes" :id "bootstrapper"
(p "The Python bootstrapper (" (code "bootstrap_py.py") ") gains a new emit mode: " (code "--async") ". This emits:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (code "async def eval_expr") " instead of " (code "def eval_expr"))
(li (code "async def trampoline") " with " (code "await") " on thunk eval")
(li (code "apply_maybe_async") " that awaits coroutine results")
(li "All higher-order forms (" (code "map") ", " (code "filter") ", " (code "reduce") ", etc.) as " (code "async def") " with " (code "await") " on callback invocations"))
(p "The JS bootstrapper is unaffected — the browser evaluator is synchronous (IO is handled by the SxEngine request pipeline, not inline eval)."))
;; -----------------------------------------------------------------------
;; Migration path
;; -----------------------------------------------------------------------
(~docs/section :title "Migration Path" :id "migration"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Step")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "What")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Risk")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "1")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Add async emit mode to bootstrap_py.py. Generate async_sx_ref.py alongside sx_ref.py.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Low — new file, nothing changes yet"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "2")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Run async_sx_ref.py in parallel with async_eval.py, compare outputs on every page render.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Low — shadow mode, no user impact"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "3")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Spec aser in render.sx. Bootstrap it. Shadow-compare with hand-written _aser.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Medium — aser has edge cases around OOB, fragments"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "4")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Switch page rendering to async_sx_ref.py. Keep async_eval.py as fallback.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Medium — production path changes"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "5")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Delete evaluator.py and async_eval.py.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Low — once shadow confirms parity"))))))
;; -----------------------------------------------------------------------
;; Principles
;; -----------------------------------------------------------------------
(~docs/section :title "Principles" :id "principles"
(ul (~tw :tokens "list-disc list-inside space-y-2")
(li (strong "The spec is the single source of truth.") " All SX evaluation semantics live in .sx files. Host code implements platform primitives, not evaluation rules.")
(li (strong "Async is a host concern, not a language concern.") " The spec is synchronous. The Python bootstrapper emits async wrappers. The JS bootstrapper emits sync code. The spec doesn't know or care.")
(li (strong "Shadow-compare before switching.") " Every migration step runs both paths in parallel and asserts identical output. No big-bang cutover.")
(li (strong "Aser is just another render mode.") " It belongs in render.sx alongside render-to-html and render-to-dom. It's not special — it's the 'evaluate some, serialize the rest' mode.")))
(~docs/section :title "Outcome" :id "outcome"
(p "After convergence:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "One evaluator implementation (the spec), bootstrapped to every host")
(li "Semantic changes made once in .sx, automatically propagated")
(li "~2,000 lines of hand-written Python evaluator code deleted")
(li "The lenient-arity class of bug becomes impossible — it's fixed in the spec, done everywhere")))))

302
sx/sx/plans/cek-reactive.sx
View File

@@ -1,302 +0,0 @@
;; Deref as Shift — CEK-Based Reactive DOM Renderer
;; Phase B: replace explicit effects with implicit continuation capture.
(defcomp ~plans/cek-reactive/plan-cek-reactive-content ()
(~docs/page :title "Deref as Shift — CEK-Based Reactive DOM Renderer"
(p (~tw :tokens "text-stone-500 text-sm italic mb-8")
"Phase A collapsed signals to plain dicts with zero platform primitives. "
"Phase B replaces explicit effect wrapping in the reactive DOM renderer "
"with implicit continuation capture: when " (code "deref") " encounters a signal "
"inside a " (code "reactive-reset") " boundary, it performs " (code "shift") ", "
"capturing the rest of the expression as a continuation. "
"That continuation IS the subscriber.")
;; =====================================================================
;; The Insight
;; =====================================================================
(~docs/section :title "The Insight" :id "insight"
(p "Each reactive binding is a micro-computation:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "reactive-text") ": given signal value, set text node content to " (code "(str value)"))
(li (code "reactive-attr") ": given signal value, set attribute to " (code "(str value)")))
(p "Currently wrapped in explicit " (code "effect") " calls. With deref-as-shift, "
"the continuation captures this automatically:")
(~docs/code :src (highlight
";; User writes:\n(div :class (str \"count-\" (deref counter))\n (str \"Value: \" (deref counter)))\n\n;; Renderer internally wraps each expression:\n(div :class (reactive-reset update-attr-fn (str \"count-\" (deref counter)))\n (reactive-reset update-text-fn (str \"Value: \" (deref counter))))\n\n;; When (deref counter) hits a signal inside reactive-reset:\n;; 1. Shift: capture continuation (str \"count-\" [HOLE])\n;; 2. Register continuation as signal subscriber\n;; 3. Return current value for initial render\n;; When counter changes:\n;; Re-invoke continuation with new value → update-fn updates DOM"
"lisp")))
;; =====================================================================
;; Step 1: Bootstrap CEK to JavaScript
;; =====================================================================
(~docs/section :title "Step 1: Bootstrap CEK to JavaScript" :id "step-1"
(p "Add " (code "frames.sx") " + " (code "cek.sx") " to the JS build pipeline. "
"Currently CEK is Python-only.")
(~docs/subsection :title "1a. platform_js.py — SPEC_MODULES + platform code"
(p "Add to " (code "SPEC_MODULES") " dict:")
(~docs/code :src (highlight
"\"frames\": (\"frames.sx\", \"frames (CEK continuation frames)\"),\n\"cek\": (\"cek.sx\", \"cek (explicit CEK machine evaluator)\"),\n\n# Add ordering (new constant):\nSPEC_MODULE_ORDER = [\"deps\", \"frames\", \"page-helpers\", \"router\", \"signals\", \"cek\"]"
"python"))
(p "Add " (code "PLATFORM_CEK_JS") " constant (mirrors " (code "PLATFORM_CEK_PY") "):")
(~docs/code :src (highlight
"// Primitive aliases used by cek.sx\nvar inc = PRIMITIVES[\"inc\"];\nvar dec = PRIMITIVES[\"dec\"];\nvar zip_pairs = PRIMITIVES[\"zip-pairs\"];\n\nfunction makeCekContinuation(captured, restKont) {\n var c = new Continuation(function(v) { return v !== undefined ? v : NIL; });\n c._cek_data = {\"captured\": captured, \"rest-kont\": restKont};\n return c;\n}\nfunction continuationData(c) {\n return (c && c._cek_data) ? c._cek_data : {};\n}"
"javascript"))
(p "Add " (code "CEK_FIXUPS_JS") " — iterative " (code "cek-run") " override:")
(~docs/code :src (highlight
"cekRun = function(state) {\n while (!cekTerminal_p(state)) { state = cekStep(state); }\n return cekValue(state);\n};"
"javascript")))
(~docs/subsection :title "1b. run_js_sx.py — Update compile_ref_to_js"
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li "Auto-add " (code "\"frames\"") " when " (code "\"cek\"") " in spec_mod_set (mirror Python " (code "bootstrap_py.py") ")")
(li "Auto-add " (code "\"cek\"") " + " (code "\"frames\"") " when " (code "\"dom\"") " adapter included (CEK needed for reactive rendering)")
(li "Use " (code "SPEC_MODULE_ORDER") " for ordering instead of " (code "sorted()"))
(li "Add " (code "has_cek") " flag")
(li "Include " (code "PLATFORM_CEK_JS") " after transpiled code when " (code "has_cek"))
(li "Include " (code "CEK_FIXUPS_JS") " in fixups section when " (code "has_cek"))))
(~docs/subsection :title "1c. js.sx — RENAMES for predicate functions"
(p "Default mangling handles most names. Only add RENAMES where " (code "?")
" suffix needs clean JS names:")
(~docs/code :src (highlight
"\"cek-terminal?\" \"cekTerminalP\"\n\"kont-empty?\" \"kontEmptyP\"\n\"make-cek-continuation\" \"makeCekContinuation\"\n\"continuation-data\" \"continuationData\""
"lisp")))
(~docs/subsection :title "1d. bootstrap_py.py — RENAMES for CEK predicates"
(~docs/code :src (highlight
"\"cek-terminal?\": \"cek_terminal_p\",\n\"kont-empty?\": \"kont_empty_p\",\n\"make-cek-continuation\": \"make_cek_continuation\",\n\"continuation-data\": \"continuation_data\","
"python")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li "Rebootstrap JS: " (code "python3 bootstrap_js.py"))
(li "Check output contains frame constructors + CEK step functions")
(li "Run existing CEK Python tests: " (code "python3 run_cek_tests.py") " (should still pass)"))))
;; =====================================================================
;; Step 2: ReactiveResetFrame + DerefFrame
;; =====================================================================
(~docs/section :title "Step 2: ReactiveResetFrame + DerefFrame" :id "step-2"
(p "New frame types in " (code "frames.sx") " that enable deref-as-shift.")
(~docs/subsection :title "2a. New frame constructors"
(~docs/code :src (highlight
";; ReactiveResetFrame: delimiter for reactive deref-as-shift\n;; Carries an update-fn that gets called with new values on re-render.\n(define make-reactive-reset-frame\n (fn (env update-fn first-render?)\n {:type \"reactive-reset\" :env env :update-fn update-fn\n :first-render first-render?}))\n\n;; DerefFrame: awaiting evaluation of deref's argument\n(define make-deref-frame\n (fn (env)\n {:type \"deref\" :env env}))"
"lisp")))
(~docs/subsection :title "2b. Update kont-capture-to-reset"
(p "Must stop at EITHER " (code "\"reset\"") " OR " (code "\"reactive-reset\"") ":")
(~docs/code :src (highlight
"(define kont-capture-to-reset\n (fn (kont)\n (define scan\n (fn (k captured)\n (if (empty? k)\n (error \"shift without enclosing reset\")\n (let ((frame (first k)))\n (if (or (= (frame-type frame) \"reset\")\n (= (frame-type frame) \"reactive-reset\"))\n (list captured (rest k))\n (scan (rest k) (append captured (list frame))))))))\n (scan kont (list))))"
"lisp")))
(~docs/subsection :title "2c. Helpers to scan for ReactiveResetFrame"
(~docs/code :src (highlight
"(define has-reactive-reset-frame?\n (fn (kont)\n (if (empty? kont) false\n (if (= (frame-type (first kont)) \"reactive-reset\") true\n (has-reactive-reset-frame? (rest kont))))))\n\n;; Returns 3 values: (captured, frame, rest)\n(define kont-capture-to-reactive-reset\n (fn (kont)\n (define scan\n (fn (k captured)\n (if (empty? k)\n (error \"reactive deref without enclosing reactive-reset\")\n (let ((frame (first k)))\n (if (= (frame-type frame) \"reactive-reset\")\n (list captured frame (rest k))\n (scan (rest k) (append captured (list frame))))))))\n (scan kont (list))))"
"lisp"))))
;; =====================================================================
;; Step 3: Make deref a CEK Special Form
;; =====================================================================
(~docs/section :title "Step 3: Make deref a CEK Special Form" :id "step-3"
(p "When " (code "deref") " encounters a signal inside a " (code "reactive-reset")
", perform shift.")
(~docs/subsection :title "3a. Add to special form dispatch in cek.sx"
(p "In the dispatch table (around where " (code "reset") " and " (code "shift") " are):")
(~docs/code :src (highlight
"(= name \"deref\") (step-sf-deref args env kont)"
"lisp")))
(~docs/subsection :title "3b. step-sf-deref"
(p "Evaluates the argument first (push DerefFrame), then decides whether to shift:")
(~docs/code :src (highlight
"(define step-sf-deref\n (fn (args env kont)\n (make-cek-state\n (first args) env\n (kont-push (make-deref-frame env) kont))))"
"lisp")))
(~docs/subsection :title "3c. Handle DerefFrame in step-continue"
(p "When the deref argument is evaluated, decide: shift or return.")
(~docs/code :src (highlight
"(= ft \"deref\")\n (let ((val value)\n (fenv (get frame \"env\")))\n (if (not (signal? val))\n ;; Not a signal: pass through\n (make-cek-value val fenv rest-k)\n ;; Signal: check for ReactiveResetFrame\n (if (has-reactive-reset-frame? rest-k)\n ;; Perform reactive shift\n (reactive-shift-deref val fenv rest-k)\n ;; No reactive-reset: normal deref (scope-based tracking)\n (do\n (let ((ctx (context \"sx-reactive\" nil)))\n (when ctx\n (let ((dep-list (get ctx \"deps\"))\n (notify-fn (get ctx \"notify\")))\n (when (not (contains? dep-list val))\n (append! dep-list val)\n (signal-add-sub! val notify-fn)))))\n (make-cek-value (signal-value val) fenv rest-k)))))"
"lisp")))
(~docs/subsection :title "3d. reactive-shift-deref — the heart"
(~docs/code :src (highlight
"(define reactive-shift-deref\n (fn (sig env kont)\n (let ((scan-result (kont-capture-to-reactive-reset kont))\n (captured-frames (first scan-result))\n (reset-frame (nth scan-result 1))\n (remaining-kont (nth scan-result 2))\n (update-fn (get reset-frame \"update-fn\")))\n ;; Sub-scope for nested subscriber cleanup on re-invocation\n (let ((sub-disposers (list)))\n (let ((subscriber\n (fn ()\n ;; Dispose previous nested subscribers\n (for-each (fn (d) (invoke d)) sub-disposers)\n (set! sub-disposers (list))\n ;; Re-invoke: push fresh ReactiveResetFrame (first-render=false)\n (let ((new-reset (make-reactive-reset-frame env update-fn false))\n (new-kont (concat captured-frames\n (list new-reset)\n remaining-kont)))\n (with-island-scope\n (fn (d) (append! sub-disposers d))\n (fn ()\n (cek-run\n (make-cek-value (signal-value sig) env new-kont))))))))\n ;; Register subscriber\n (signal-add-sub! sig subscriber)\n ;; Register cleanup with island scope\n (register-in-scope\n (fn ()\n (signal-remove-sub! sig subscriber)\n (for-each (fn (d) (invoke d)) sub-disposers)))\n ;; Return current value for initial render\n (make-cek-value (signal-value sig) env remaining-kont))))))"
"lisp")))
(~docs/subsection :title "3e. Handle ReactiveResetFrame in step-continue"
(p "When expression completes normally (or after re-invocation):")
(~docs/code :src (highlight
"(= ft \"reactive-reset\")\n (let ((update-fn (get frame \"update-fn\"))\n (first? (get frame \"first-render\")))\n ;; On re-render (not first), call update-fn with new value\n (when (and update-fn (not first?))\n (invoke update-fn value))\n (make-cek-value value env rest-k))"
"lisp"))
(p (strong "Key:") " On first render, update-fn is NOT called — the value flows back to the caller "
"who inserts it into the DOM. On re-render (subscriber fires), update-fn IS called "
"to mutate the existing DOM.")))
;; =====================================================================
;; Step 4: Integrate into adapter-dom.sx
;; =====================================================================
(~docs/section :title "Step 4: Integrate into adapter-dom.sx" :id "step-4"
(p "Add CEK reactive path alongside existing effect-based path, controlled by opt-in flag.")
(~docs/subsection :title "4a. Opt-in flag"
(~docs/code :src (highlight
"(define *use-cek-reactive* false)\n(define enable-cek-reactive! (fn () (set! *use-cek-reactive* true)))"
"lisp")))
(~docs/subsection :title "4b. CEK reactive attribute binding"
(~docs/code :src (highlight
"(define cek-reactive-attr\n (fn (el attr-name expr env)\n (let ((update-fn (fn (val)\n (cond\n (or (nil? val) (= val false)) (dom-remove-attr el attr-name)\n (= val true) (dom-set-attr el attr-name \"\")\n :else (dom-set-attr el attr-name (str val))))))\n ;; Mark for morph protection\n (let ((existing (or (dom-get-attr el \"data-sx-reactive-attrs\") \"\"))\n (updated (if (empty? existing) attr-name (str existing \",\" attr-name))))\n (dom-set-attr el \"data-sx-reactive-attrs\" updated))\n ;; Initial render via CEK with ReactiveResetFrame\n (let ((initial (cek-run\n (make-cek-state expr env\n (list (make-reactive-reset-frame env update-fn true))))))\n (invoke update-fn initial)))))"
"lisp")))
(~docs/subsection :title "4c. Modify render-dom-element dispatch"
(p "In attribute processing, add conditional:")
(~docs/code :src (highlight
"(context \"sx-island-scope\" nil)\n (if *use-cek-reactive*\n (cek-reactive-attr el attr-name attr-expr env)\n (reactive-attr el attr-name\n (fn () (trampoline (eval-expr attr-expr env)))))"
"lisp"))
(p "Similarly for text positions and conditional rendering."))
(~docs/subsection :title "4d. CEK reactive text"
(~docs/code :src (highlight
"(define cek-reactive-text\n (fn (expr env)\n (let ((node (create-text-node \"\"))\n (update-fn (fn (val)\n (dom-set-text-content node (str val)))))\n (let ((initial (cek-run\n (make-cek-state expr env\n (list (make-reactive-reset-frame env update-fn true))))))\n (dom-set-text-content node (str initial))\n node))))"
"lisp")))
(~docs/subsection :title "4e. What stays unchanged"
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "reactive-list") " — keyed reconciliation is complex; keep effect-based for now")
(li (code "reactive-spread") " — spread tracking is complex; keep effect-based")
(li (code "effect") ", " (code "computed") " — still needed for non-rendering side effects")
(li "Existing " (code "reactive-*") " functions — remain as default path"))))
;; =====================================================================
;; Step 5: Tests
;; =====================================================================
(~docs/section :title "Step 5: Tests" :id "step-5"
(~docs/subsection :title "5a. test-cek-reactive.sx"
(p "Tests:")
(ol (~tw :tokens "list-decimal pl-6 mb-4 space-y-1")
(li (code "deref") " non-signal passes through (no shift)")
(li (code "deref") " signal without reactive-reset: returns value, no subscription")
(li (code "deref") " signal with reactive-reset: shifts, registers subscriber, update-fn called on change")
(li "Expression with deref: " (code "(str \"hello \" (deref sig))") " — continuation captures rest")
(li "Multi-deref: both signals create subscribers, both fire correctly")
(li "Disposal: removing island scope unsubscribes all continuations")
(li "Stale subscriber cleanup: re-invocation disposes nested subscribers")))
(~docs/subsection :title "5b. run_cek_reactive_tests.py"
(p "Mirrors " (code "run_cek_tests.py") ". "
"Loads frames.sx, cek.sx, signals.sx, runs test-cek-reactive.sx.")))
;; =====================================================================
;; Step 6: Browser Demo
;; =====================================================================
(~docs/section :title "Step 6: Browser Demo" :id "step-6"
(p "Demo showing:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li "Counter island with implicit reactivity (no explicit effects)")
(li (code "(deref counter)") " in text position auto-updates")
(li (code "(str \"count-\" (deref class-sig))") " in attr position auto-updates")
(li "Side-by-side comparison: effect-based vs continuation-based code")))
;; =====================================================================
;; Multi-Deref Handling
;; =====================================================================
(~docs/section :title "Multi-Deref Handling" :id "multi-deref"
(~docs/code :src (highlight "(str (deref first-name) \" \" (deref last-name))" "lisp"))
(ol (~tw :tokens "list-decimal pl-6 mb-6 space-y-3")
(li (strong "Initial render:") " First " (code "deref") " hits signal → shifts, captures "
(code "(str [HOLE] \" \" (deref last-name))") ". Subscriber registered for "
(code "first-name") ". Returns current value. Second " (code "deref")
" runs (no ReactiveResetFrame between it and the already-consumed one) — "
"falls through to normal scope-based tracking.")
(li (strong "first-name changes:") " Subscriber fires → re-pushes ReactiveResetFrame → "
"re-invokes continuation with new first-name value → second " (code "deref")
" hits ReactiveResetFrame again → shifts, creates NEW subscriber for "
(code "last-name") ". Old last-name subscriber cleaned up via sub-scope disposal.")
(li (strong "last-name changes:") " Its subscriber fires → re-invokes inner continuation → "
"update-fn called with new result."))
(p "This creates O(n) nested continuations for n derefs. Fine for small reactive expressions."))
;; =====================================================================
;; Commit Strategy
;; =====================================================================
(~docs/section :title "Commit Strategy" :id "commits"
(ol (~tw :tokens "list-decimal pl-6 mb-4 space-y-1")
(li (strong "Commit 1:") " Bootstrap CEK to JS (Step 1) — mechanical, independent")
(li (strong "Commit 2:") " ReactiveResetFrame + DerefFrame (Step 2) — new frame types")
(li (strong "Commit 3:") " Deref-as-shift + adapter integration + tests (Steps 3-5) — the core change")
(li (strong "Commit 4:") " Browser demo (Step 6)")))
;; =====================================================================
;; Files
;; =====================================================================
(~docs/section :title "Files" :id "files"
(div (~tw :tokens "overflow-x-auto mb-6")
(table (~tw :tokens "min-w-full text-sm")
(thead (tr
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "File")
(th (~tw :tokens "text-left pb-2 font-semibold") "Change")))
(tbody
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/platform_js.py")
(td "SPEC_MODULES entries, PLATFORM_CEK_JS, CEK_FIXUPS_JS, SPEC_MODULE_ORDER"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/run_js_sx.py")
(td "compile_ref_to_js: has_cek, auto-inclusion, ordering, platform code"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/js.sx")
(td "RENAMES for CEK predicate functions"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/bootstrap_py.py")
(td "RENAMES for CEK predicates"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/frames.sx")
(td "ReactiveResetFrame, DerefFrame, has-reactive-reset-frame?, kont-capture-to-reactive-reset"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/cek.sx")
(td "step-sf-deref, reactive-shift-deref, deref in dispatch, ReactiveResetFrame in step-continue"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/adapter-dom.sx")
(td "*use-cek-reactive* flag, cek-reactive-attr, cek-reactive-text, conditional dispatch"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/test-cek-reactive.sx")
(td (strong "New:") " continuation-based reactivity tests"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/run_cek_reactive_tests.py")
(td (strong "New:") " Python test runner"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/sx/ref/sx_ref.py")
(td "Rebootstrap (generated)"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono text-xs") "shared/static/scripts/sx-browser.js")
(td "Rebootstrap (generated)"))))))
;; =====================================================================
;; Risks
;; =====================================================================
(~docs/section :title "Risks" :id "risks"
(ol (~tw :tokens "list-decimal pl-6 mb-4 space-y-2")
(li (strong "Performance:") " CEK allocates a dict per step. Mitigated: opt-in flag, tree-walk remains default.")
(li (strong "Multi-deref stale subscribers:") " Mitigated: sub-scope disposal before re-invocation.")
(li (strong "Interaction with user shift/reset:") " " (code "kont-capture-to-reactive-reset")
" only scans for " (code "\"reactive-reset\"") ", not " (code "\"reset\"") ". Orthogonal.")
(li (strong "JS bootstrapper complexity:") " ~10 RENAMES for predicates. Default mangling handles the rest.")))))

419
sx/sx/plans/content-addressed-components.sx
View File

@@ -1,419 +0,0 @@
;; ---------------------------------------------------------------------------
;; Content-Addressed Components
;; ---------------------------------------------------------------------------
(defcomp ~plans/content-addressed-components/plan-content-addressed-components-content ()
(~docs/page :title "Content-Addressed Components"
(~docs/section :title "The Premise" :id "premise"
(p "SX components are pure functions. Boundary enforcement guarantees it — a component cannot call IO primitives, make network requests, access cookies, or touch the filesystem. " (code "Component.is_pure") " is a structural property, verified at registration time by scanning the transitive closure of IO references via " (code "deps.sx") ".")
(p "Pure functions have a remarkable property: " (strong "their identity is their content.") " Two components that produce the same serialized form are the same component, regardless of who wrote them or where they're hosted. This means we can content-address them — compute a cryptographic hash of the canonical serialized form, and that hash " (em "is") " the component's identity.")
(p "Content addressing turns components into shared infrastructure. Define " (code "~card") " once, pin it to IPFS, and every SX application on the planet can use it by CID. No package registry, no npm install, no version conflicts. The CID " (em "is") " the version. The hash " (em "is") " the trust. Boundary enforcement " (em "is") " the sandbox.")
(p "This plan details how to get from the current name-based, per-server component model to a content-addressed, globally-shared one."))
;; -----------------------------------------------------------------------
;; Current State
;; -----------------------------------------------------------------------
(~docs/section :title "Current State" :id "current-state"
(p "What already exists and what's missing.")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Capability")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Status")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Where")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Deterministic serialization")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Partial — " (code "serialize(body, pretty=True)") " from AST, but no canonical normalization")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "parser.py:296-427"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Component identity")
(td (~tw :tokens "px-3 py-2 text-stone-700") "By name (" (code "~card") ") — names are mutable, server-local")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "types.py:157-180"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Bundle hashing")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SHA256 of all defs concatenated — per-bundle, not per-component")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "jinja_bridge.py:60-86"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Purity verification")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-green-700 font-medium") "Complete") " — " (code "is_pure") " via transitive IO ref analysis")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "deps.sx, boundary.py"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Dependency graph")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-green-700 font-medium") "Complete") " — " (code "Component.deps") " transitive closure")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "deps.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "IPFS infrastructure")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-green-700 font-medium") "Exists") " — IPFSPin model, async upload tasks, CID tracking")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "models/federation.py, artdag/l1/tasks/"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client component caching")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Hash-based localStorage — but keyed by bundle hash, not individual CID")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "boot.sx, helpers.py"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Content-addressed components")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-red-700 font-medium") "Not yet") " — no per-component CID, no IPFS resolution")
(td (~tw :tokens "px-3 py-2 text-stone-600") "—"))))))
;; -----------------------------------------------------------------------
;; Canonical Serialization
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 1: Canonical Serialization" :id "canonical-serialization"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "The foundation")
(p (~tw :tokens "text-violet-800") "Same component must always produce the same bytes, regardless of original formatting, whitespace, or comment placement. Without this, content addressing is meaningless."))
(~docs/subsection :title "The Problem"
(p "Currently " (code "serialize(body, pretty=True)") " produces readable SX source from the parsed AST. But serialization isn't fully canonical — it depends on the internal representation order, and there's no normalization pass. Two semantically identical components formatted differently would produce different hashes.")
(p "We need a " (strong "canonical form") " that strips all variance:"))
(~docs/subsection :title "Canonical Form Rules"
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-2")
(li (strong "Strip comments.") " Comments are parsing artifacts, not part of the AST. The serializer already ignores them (it works from the parsed tree), but any future comment-preserving parser must not affect canonical output.")
(li (strong "Normalize whitespace.") " Single space between tokens, newline before each top-level form in a body. No trailing whitespace. No blank lines.")
(li (strong "Sort keyword arguments alphabetically.") " In component calls: " (code "(~card :class \"x\" :title \"y\")") " not " (code "(~card :title \"y\" :class \"x\")") ". In dict literals: " (code "{:a 1 :b 2}") " not " (code "{:b 2 :a 1}") ".")
(li (strong "Normalize string escapes.") " Use " (code "\\n") " not literal newlines in strings. Escape only what must be escaped.")
(li (strong "Normalize numbers.") " " (code "1.0") " not " (code "1.00") " or " (code "1.") ". " (code "42") " not " (code "042") ".")
(li (strong "Include the full definition form.") " Hash the complete " (code "(defcomp ~plans/content-addressed-components/name (params) body)") ", not just the body. The name and parameter signature are part of the component's identity.")))
(~docs/subsection :title "Implementation"
(p "New spec function in a " (code "canonical.sx") " module:")
(~docs/code :src (highlight "(define canonical-serialize\n (fn (node)\n ;; Produce a canonical s-expression string from an AST node.\n ;; Deterministic: same AST always produces same output.\n ;; Used for CID computation — NOT for human-readable output.\n (case (type-of node)\n \"list\"\n (str \"(\" (join \" \" (map canonical-serialize node)) \")\")\n \"dict\"\n (let ((sorted-keys (sort (keys node))))\n (str \"{\" (join \" \"\n (map (fn (k)\n (str \":\" k \" \" (canonical-serialize (get node k))))\n sorted-keys)) \"}\"))\n \"string\"\n (str '\"' (escape-canonical node) '\"')\n \"number\"\n (canonical-number node)\n \"symbol\"\n (symbol-name node)\n \"keyword\"\n (str \":\" (keyword-name node))\n \"boolean\"\n (if node \"true\" \"false\")\n \"nil\"\n \"nil\")))" "lisp"))
(p "This function must be bootstrapped to both Python and JS — the server computes CIDs at registration time, the client verifies them on fetch.")
(p "The canonical serializer is distinct from " (code "serialize()") " for display. " (code "serialize(pretty=True)") " remains for human-readable output. " (code "canonical-serialize") " is for hashing only.")))
;; -----------------------------------------------------------------------
;; CID Computation
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 2: CID Computation" :id "cid-computation"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "Every component gets a stable, unique content identifier. Same source → same CID, always. Different source → different CID, always."))
(~docs/subsection :title "CID Format"
(p "Use " (a :href "https://github.com/multiformats/cid" (~tw :tokens "text-violet-700 underline") "CIDv1") " with:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Hash function:") " SHA3-256 (already used by artdag for content addressing)")
(li (strong "Codec:") " raw (the content is the canonical SX source bytes, not a DAG-PB wrapper)")
(li (strong "Base encoding:") " base32lower for URL-safe representation (" (code "bafy...") " prefix)"))
(~docs/code :src (highlight ";; CID computation pipeline\n(define component-cid\n (fn (component)\n ;; 1. Reconstruct full defcomp form\n ;; 2. Canonical serialize\n ;; 3. SHA3-256 hash\n ;; 4. Wrap as CIDv1\n (let ((source (canonical-serialize\n (list 'defcomp\n (symbol (str \"~\" (component-name component)))\n (component-params-list component)\n (component-body component)))))\n (cid-v1 :sha3-256 :raw (encode-utf8 source)))))" "lisp")))
(~docs/subsection :title "Where CIDs Live"
(p "Each " (code "Component") " object gains a " (code "cid") " field, computed at registration time:")
(~docs/code :src (highlight ";; types.py extension\n@dataclass\nclass Component:\n name: str\n params: list[str]\n has_children: bool\n body: Any\n closure: dict[str, Any]\n css_classes: set[str]\n deps: set[str] # by name\n io_refs: set[str]\n cid: str | None = None # computed after registration\n dep_cids: dict[str, str] | None = None # name → CID" "python"))
(p "After " (code "compute_all_deps()") " runs, a new " (code "compute_all_cids()") " pass fills in CIDs for every component. Dependency CIDs are also recorded — when a component references " (code "~card") ", we store both the name and card's CID."))
(~docs/subsection :title "CID Stability"
(p "A component's CID changes when and only when its " (strong "semantics") " change:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Reformatting the " (code ".sx") " source file → same AST → same canonical form → " (strong "same CID"))
(li "Adding a comment → stripped by parser → same AST → " (strong "same CID"))
(li "Changing a class name in the body → different AST → " (strong "different CID"))
(li "Renaming the component → different defcomp form → " (strong "different CID") " (name is part of identity)"))
(p "This means CIDs are " (em "immutable versions") ". There's no " (code "~card@1.2.3") " — there's " (code "~card") " at CID " (code "bafy...abc") " and " (code "~card") " at CID " (code "bafy...def") ". The name is a human-friendly alias; the CID is the truth.")))
;; -----------------------------------------------------------------------
;; Component Manifest
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 3: Component Manifest" :id "manifest"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "Metadata that travels with a CID — what a component needs, what it provides, whether it's safe to run. Enough information to resolve, validate, and render without fetching the source first."))
(~docs/subsection :title "Manifest Structure"
(~docs/code :src (highlight ";; Component manifest — published alongside the source\n(SxComponent\n :name \"~product-card\"\n :cid \"bafy...productcard\"\n :source-bytes 847\n :params (:title :price :image-url)\n :has-children true\n :pure true\n :deps (\n {:name \"~card\" :cid \"bafy...card\"}\n {:name \"~price-tag\" :cid \"bafy...pricetag\"}\n {:name \"~lazy-image\" :cid \"bafy...lazyimg\"})\n :css-atoms (:border :rounded :p-4 :text-sm :font-bold\n :text-green-700 :line-through :text-stone-400)\n :author \"https://rose-ash.com/apps/market\"\n :published \"2026-03-06T14:30:00Z\")" "lisp"))
(p "Key fields:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code ":cid") " — content address of the canonical serialized source")
(li (code ":deps") " — dependency CIDs, not just names. A consumer can recursively resolve the entire tree by CID without name ambiguity")
(li (code ":pure") " — pre-computed purity flag. The consumer " (em "re-verifies") " this after fetching (never trust the manifest alone), but it enables fast rejection of IO-dependent components before downloading")
(li (code ":deps") " includes style component CIDs. No separate " (code ":css-atoms") " field needed — styling is just more components")
(li (code ":params") " — parameter signature for tooling, documentation, IDE support")
(li (code ":author") " — who published this. AP actor URL, verifiable via HTTP Signatures")))
(~docs/subsection :title "Manifest CID"
(p "The manifest itself is content-addressed. But the manifest CID is " (em "not") " the component CID — they're separate objects. The component CID is derived from the source alone (pure content). The manifest CID includes metadata that could change (author, publication date) without changing the component.")
(p "Resolution order: manifest CID → manifest → component CID → component source. Or shortcut: component CID → source directly, if you already know what you need.")))
;; -----------------------------------------------------------------------
;; IPFS Storage & Resolution
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 4: IPFS Storage & Resolution" :id "ipfs"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "Components live on IPFS. Any browser can fetch them by CID. No origin server needed. No CDN. No DNS. The content network IS the distribution network."))
(~docs/subsection :title "Server-Side: Publication"
(p "On component registration (startup or hot-reload), the server:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-1")
(li "Computes canonical form and CID")
(li "Checks " (code "IPFSPin") " — if CID already pinned, skip (content can't have changed)")
(li "Pins canonical source to IPFS (async Celery task, same pattern as artdag)")
(li "Creates/updates " (code "IPFSPin") " record with " (code "pin_type=\"component\""))
(li "Publishes manifest to IPFS (separate CID)")
(li "Optionally announces via AP outbox for federated discovery"))
(~docs/code :src (highlight ";; IPFSPin usage for components\nIPFSPin(\n content_hash=\"sha3-256:abcdef...\",\n ipfs_cid=\"bafy...productcard\",\n pin_type=\"component\",\n source_type=\"market\", # which service defined it\n metadata={\n \"name\": \"~product-card\",\n \"manifest_cid\": \"bafy...manifest\",\n \"deps\": [\"bafy...card\", \"bafy...pricetag\"],\n \"pure\": True\n }\n)" "python")))
(~docs/subsection :title "Client-Side: Resolution"
(p "New spec module " (code "resolve.sx") " — the client-side component resolution pipeline:")
(~docs/code :src (highlight "(define resolve-component-by-cid\n (fn (cid callback)\n ;; Resolution cascade:\n ;; 1. Check component env (already loaded?)\n ;; 2. Check localStorage (keyed by CID = cache-forever)\n ;; 3. Check origin server (/sx/components?cid=bafy...)\n ;; 4. Fetch from IPFS gateway\n ;; 5. Verify hash matches CID\n ;; 6. Parse, validate purity, register, callback\n (let ((cached (local-storage-get (str \"sx-cid:\" cid))))\n (if cached\n (do\n (register-component-source cached)\n (callback true))\n (fetch-component-by-cid cid\n (fn (source)\n (if (verify-cid cid source)\n (do\n (local-storage-set (str \"sx-cid:\" cid) source)\n (register-component-source source)\n (callback true))\n (do\n (log-warn (str \"sx:cid verification failed \" cid))\n (callback false)))))))))" "lisp"))
(p "The cache-forever semantics are the key insight: because CIDs are content-addressed, a cached component " (strong "can never be stale") ". If the source changes, it gets a new CID. Old CIDs remain valid forever. There is no cache invalidation problem."))
(~docs/subsection :title "Resolution Cascade"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Layer")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Lookup")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Latency")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "When")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "1. Component env")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "(env-has? env cid)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "0ms")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Already loaded this session"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "2. localStorage")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "localStorage[\"sx-cid:\" + cid]")
(td (~tw :tokens "px-3 py-2 text-stone-600") "<1ms")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Previously fetched, persists across sessions"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "3. Origin server")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "GET /sx/components?cid=bafy...")
(td (~tw :tokens "px-3 py-2 text-stone-600") "~20ms")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Same-origin component, not yet cached"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "4. IPFS gateway")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "GET https://gateway/ipfs/{cid}")
(td (~tw :tokens "px-3 py-2 text-stone-600") "~200ms")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Foreign component, federated content"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "5. Local IPFS node")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "ipfs cat {cid}")
(td (~tw :tokens "px-3 py-2 text-stone-600") "~5ms")
(td (~tw :tokens "px-3 py-2 text-stone-600") "User runs own IPFS node (power users)")))))
(p "Layer 5 is optional — checked between 2 and 3 if " (code "window.ipfs") " or a local gateway is detected. For most users, layers 1-4 cover all cases.")))
;; -----------------------------------------------------------------------
;; Security Model
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 5: Security Model" :id "security"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "The hard part")
(p (~tw :tokens "text-violet-800") "Loading code from the network is the web's original sin. Content-addressed components are safe because of three structural guarantees — not policies, not trust, not sandboxes that can be escaped."))
(~docs/subsection :title "Guarantee 1: Purity is Structural"
(p "SX boundary enforcement isn't a runtime sandbox — it's a registration-time structural check. When a component is loaded from IPFS and parsed, " (code "compute_all_io_refs()") " walks its entire AST and transitive dependencies. If " (em "any") " node references an IO primitive, the component is classified as IO-dependent and " (strong "rejected for untrusted registration."))
(p "This means the evaluator literally doesn't have IO primitives in scope when running an IPFS-loaded component. It's not that we catch IO calls — the names don't resolve. There's nothing to catch.")
(~docs/code :src (highlight "(define register-untrusted-component\n (fn (source origin)\n ;; Parse the defcomp from source\n ;; Run compute-all-io-refs on the parsed component\n ;; If io_refs is non-empty → REJECT\n ;; If pure → register in env with :origin metadata\n (let ((comp (parse-component source)))\n (if (not (component-pure? comp))\n (do\n (log-warn (str \"sx:reject IO component from \" origin))\n nil)\n (do\n (register-component comp)\n (log-info (str \"sx:registered \" (component-name comp)\n \" from \" origin))\n comp)))))" "lisp")))
(~docs/subsection :title "Guarantee 2: Content Verification"
(p "The CID IS the hash. When you fetch " (code "bafy...abc") " from any source — IPFS gateway, origin server, peer — you hash the response and compare. If it doesn't match, you reject it. No MITM attack can alter the content without changing the CID.")
(p "This is stronger than HTTPS. HTTPS trusts the certificate authority, the DNS resolver, and the server operator. Content addressing trusts " (em "mathematics") ". The hash either matches or it doesn't."))
(~docs/subsection :title "Guarantee 3: Evaluation Limits"
(p "Pure doesn't mean terminating. A component could contain an infinite loop or exponential recursion. SX evaluators enforce step limits:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Max eval steps:") " configurable per context. Untrusted components get a lower limit than local ones.")
(li (strong "Max recursion depth:") " prevents stack exhaustion.")
(li (strong "Max output size:") " prevents a component from producing gigabytes of DOM nodes."))
(p "Exceeding any limit halts evaluation and returns an error node. The worst case is wasted CPU — never data exfiltration, never unauthorized IO."))
(~docs/subsection :title "Trust Tiers"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Tier")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Source")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Allowed")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Eval limits")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Local")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Server's own " (code ".sx") " files")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Pure + IO primitives + page helpers")
(td (~tw :tokens "px-3 py-2 text-stone-600") "None (trusted)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Followed")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components from followed AP actors")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Pure only (IO rejected)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Standard limits"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Federated")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components from any IPFS source")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Pure only (IO rejected)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Strict limits"))))))
(~docs/subsection :title "What Can Go Wrong"
(p "Honest accounting of the attack surface:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Visual spoofing:") " A malicious component could render UI that looks like a login form. Mitigation: untrusted components render inside a visually distinct container with origin attribution.")
(li (strong "CSS abuse:") " A component's CSS atoms could interfere with page layout. Mitigation: scoped CSS — untrusted components' classes are namespaced.")
(li (strong "Resource exhaustion:") " A component could be expensive to evaluate. Mitigation: step limits, timeout, lazy rendering for off-screen components.")
(li (strong "Privacy leak via CSS:") " Background-image URLs could phone home. Mitigation: CSP restrictions on untrusted component rendering contexts.")
(li (strong "Dependency confusion:") " A malicious manifest could claim deps that are different components with the same name. Mitigation: deps are referenced by CID, not name. Name is informational only."))))
;; -----------------------------------------------------------------------
;; Wire Format & Prefetch Integration
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 6: Wire Format & Prefetch Integration" :id "wire-format"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "Pages and SX responses reference components by CID. The prefetch system resolves them from the most efficient source. Components become location-independent."))
(~docs/subsection :title "CID References in Page Registry"
(p "The page registry (shipped to the client as " (code "<script type=\"text/sx-pages\">") ") currently lists deps by name. Extend to include CIDs:")
(~docs/code :src (highlight "{:name \"docs-page\" :path \"/language/docs/<slug>\"\n :auth \"public\" :has-data false\n :deps ({:name \"~essay-foo\" :cid \"bafy...essay\"}\n {:name \"~doc-code\" :cid \"bafy...doccode\"})\n :content \"(case slug ...)\" :closure {}}" "lisp"))
(p "The " (a :href "/sx/(etc.(plan.predictive-prefetch))" (~tw :tokens "text-violet-700 underline") "predictive prefetch system") " uses these CIDs to fetch components from the resolution cascade rather than only from the origin server's " (code "/sx/components") " endpoint."))
(~docs/subsection :title "SX Response Component Headers"
(p "Currently, " (code "SX-Components") " header lists loaded component names. Extend to support CIDs:")
(~docs/code :src (highlight "Request:\nSX-Components: ~card:bafy...card,~plans/environment-images/nav:bafy...nav\n\nResponse:\nSX-Component-CIDs: ~plans/content-addressed-components/essay-foo:bafy...essay,~docs/code:bafy...doccode\n\n;; Response body only includes defs the client doesn't have\n(defcomp ~plans/content-addressed-components/essay-foo ...)" "http"))
(p "The client can then verify received components match their declared CIDs. If the origin server is compromised, CID verification catches the tampered response."))
(~docs/subsection :title "Federated Content"
(p "When an ActivityPub activity arrives with SX content, it declares component requirements by CID:")
(~docs/code :src (highlight "(Create\n :actor \"https://other-instance.com/users/bob\"\n :object (Note\n :content (~product-card :title \"Bob's Widget\" :price 29.99)\n :requires (list\n {:name \"~product-card\" :cid \"bafy...prodcard\"}\n {:name \"~price-tag\" :cid \"bafy...pricetag\"})))" "lisp"))
(p "The receiving browser resolves required components through the cascade. If Bob's instance is down, the components are still fetchable from IPFS. The content is self-describing and self-resolving.")))
;; -----------------------------------------------------------------------
;; Component Sharing & Discovery
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 7: Sharing & Discovery" :id "sharing"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "Servers publish component collections via AP. Other servers follow them. Like npm, but federated, content-addressed, and structurally safe."))
(~docs/subsection :title "Component Registry as AP Actor"
(p "Each server exposes a component registry actor:")
(~docs/code :src (highlight "(Service\n :id \"https://rose-ash.com/sx-registry\"\n :type \"SxComponentRegistry\"\n :name \"Rose Ash Components\"\n :outbox \"https://rose-ash.com/sx-registry/outbox\"\n :followers \"https://rose-ash.com/sx-registry/followers\")" "lisp"))
(p "Follow the registry to receive component updates. The outbox is a chronological feed of Create/Update/Delete activities for components. 'Update' means a new CID for the same name — consumers decide whether to adopt it."))
(~docs/subsection :title "Discovery Protocol"
(p "Webfinger-style lookup for components by name:")
(~docs/code :src (highlight "GET /.well-known/sx-component?name=~product-card\n\n{\n \"name\": \"~product-card\",\n \"cid\": \"bafy...prodcard\",\n \"manifest_cid\": \"bafy...manifest\",\n \"gateway\": \"https://rose-ash.com/ipfs/\",\n \"author\": \"https://rose-ash.com/apps/market\"\n}" "http"))
(p "This is an optional convenience — any consumer that knows the CID can skip discovery and fetch directly from IPFS. Discovery answers the question: " (em "\"what's the current version of ~product-card on rose-ash.com?\""))
)
(~docs/subsection :title "Name Resolution"
(p "Names are human-friendly aliases for CIDs. The same name on different servers can refer to different components (different CIDs). Conflict resolution is simple:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Local wins:") " If the server defines " (code "~card") ", that definition takes precedence over any federated " (code "~card") ".")
(li (strong "CID pinning:") " When referencing a federated component, pin the CID. " (code "(:name \"~card\" :cid \"bafy...abc\")") " — the name is informational, the CID is authoritative.")
(li (strong "No global namespace:") " There is no \"npm\" that owns " (code "~card") ". Names are scoped to the server that defines them. CIDs are global."))))
;; -----------------------------------------------------------------------
;; Spec modules
;; -----------------------------------------------------------------------
(~docs/section :title "Spec Modules" :id "spec-modules"
(p "Per the SX host architecture principle, all content-addressing logic is specced in " (code ".sx") " files and bootstrapped:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Spec module")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Functions")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Platform obligations")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "canonical.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "canonical-serialize") ", " (code "canonical-number") ", " (code "escape-canonical"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "None — pure string operations"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "cid.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "component-cid") ", " (code "verify-cid") ", " (code "cid-to-string") ", " (code "parse-cid"))
(td (~tw :tokens "px-3 py-2 text-stone-600") (code "sha3-256") ", " (code "encode-base32") ", " (code "encode-utf8")))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "resolve.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "resolve-component-by-cid") ", " (code "resolve-deps-recursive") ", " (code "register-untrusted-component"))
(td (~tw :tokens "px-3 py-2 text-stone-600") (code "local-storage-get/set") ", " (code "fetch-cid") ", " (code "register-component-source"))))))
;; -----------------------------------------------------------------------
;; Critical files
;; -----------------------------------------------------------------------
(~docs/section :title "Critical Files" :id "critical-files"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Role")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Phase")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/canonical.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Canonical serialization spec (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/cid.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "CID computation and verification spec (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/types.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Add " (code "cid") " and " (code "dep_cids") " to Component")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/jinja_bridge.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Add " (code "compute_all_cids()") " to registration lifecycle")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/models/federation.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "IPFSPin records for component CIDs")
(td (~tw :tokens "px-3 py-2 text-stone-600") "4"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/resolve.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client-side CID resolution cascade (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "4"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/helpers.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "CIDs in page registry, " (code "/sx/components?cid=") " endpoint")
(td (~tw :tokens "px-3 py-2 text-stone-600") "6"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/orchestration.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "CID-aware prefetch in resolution cascade")
(td (~tw :tokens "px-3 py-2 text-stone-600") "6"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/infrastructure/activitypub.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Component registry actor, Webfinger extension")
(td (~tw :tokens "px-3 py-2 text-stone-600") "7"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/boundary.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Trust tier enforcement for untrusted components")
(td (~tw :tokens "px-3 py-2 text-stone-600") "5"))))))
;; -----------------------------------------------------------------------
;; Relationship
;; -----------------------------------------------------------------------
(~docs/section :title "Relationships" :id "relationships"
(p "This plan is the foundation for several other plans and roadmaps:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (a :href "/sx/(etc.(plan.sx-activity))" (~tw :tokens "text-violet-700 underline") "SX-Activity") " Phase 2 (content-addressed components on IPFS) is a summary of this plan. This plan supersedes that section with full detail.")
(li (a :href "/sx/(etc.(plan.predictive-prefetch))" (~tw :tokens "text-violet-700 underline") "Predictive prefetching") " gains CID-based resolution — the " (code "/sx/components") " endpoint and IPFS gateway become alternative resolution paths in the prefetch cascade.")
(li (a :href "/sx/(etc.(plan.isomorphic-architecture))" (~tw :tokens "text-violet-700 underline") "Isomorphic architecture") " Phase 1 (component distribution) is enhanced — CIDs make per-page bundles verifiable and cross-server shareable.")
(li "The SX-Activity vision of " (strong "serverless applications on IPFS") " depends entirely on this plan. Without content-addressed components, applications can't be pinned to IPFS as self-contained artifacts."))
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-3 mt-2")
(p (~tw :tokens "text-amber-800 text-sm") (strong "Depends on: ") "deps.sx (complete), boundary enforcement (complete), IPFS infrastructure (exists in artdag, needs wiring to web platform)."))))))
;; ---------------------------------------------------------------------------
;; Predictive Component Prefetching
;; ---------------------------------------------------------------------------

304
sx/sx/plans/environment-images.sx
View File

@@ -1,304 +0,0 @@
;; ---------------------------------------------------------------------------
;; Content-Addressed Environment Images
;; ---------------------------------------------------------------------------
(defcomp ~plans/environment-images/plan-environment-images-content ()
(~docs/page :title "Content-Addressed Environment Images"
(~docs/section :title "The Idea" :id "idea"
(p "Every served SX endpoint should point back to its spec. The spec CIDs identify the exact evaluator, renderer, parser, and primitives that produced the output. This makes every endpoint " (strong "fully executable") " — anyone with the CIDs can independently reproduce the result.")
(p "But evaluating spec files from source on every cold start is wasteful. The specs are pure — same source always produces the same evaluated environment. So we can serialize the " (em "evaluated") " environment as a content-addressed image: all defcomps, defmacros, bound symbols, resolved closures frozen into a single artifact. The image CID is a function of its contents. Load the image, skip evaluation, get the same result.")
(p "The chain becomes:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-2")
(li (strong "Served page") " → CID of the spec that defines its semantics")
(li (strong "Spec CID") " → the evaluator, renderer, parser, primitives that any conforming host can execute")
(li (strong "Image CID") " → the pre-evaluated environment, a cache of (2) that any conforming host can deserialize"))
(p "The spec is the truth. The image is a verified cache. The bootstrapper that compiled the spec for a particular host is an implementation detail — irrelevant to the content address.")
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mt-4")
(p (~tw :tokens "text-violet-900 font-medium") "Prior art")
(p (~tw :tokens "text-violet-800") (a :href "https://github.com/KinaKnowledge/juno-lang" (~tw :tokens "underline") "Juno") " — a self-hosted Lisp-to-JS compiler — implements image persistence: serialize a running environment, restore it later, even bundle it as a standalone HTML document. Their Seedling IDE saves/restores entire development sessions as images. SX can do this more rigorously because our images are content-addressed (Juno's are not) and our components are boundary-enforced pure (Juno's are not).")))
;; -----------------------------------------------------------------------
;; What gets serialized
;; -----------------------------------------------------------------------
(~docs/section :title "What Gets Serialized" :id "what"
(p "An environment image is a snapshot of everything produced by evaluating the spec files. Not the source — the result.")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Category")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Contents")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Source")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components")
(td (~tw :tokens "px-3 py-2 text-stone-700") "All " (code "defcomp") " definitions — name, params, body AST, closure bindings, CID, deps, css_classes")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "Service .sx files + shared/sx/templates/"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Macros")
(td (~tw :tokens "px-3 py-2 text-stone-700") "All " (code "defmacro") " definitions — name, params, body AST, closure")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "Spec files"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Bindings")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Top-level " (code "define") " values — constants, lookup tables, configuration")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "Spec files + service .sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Primitives")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Registry of pure primitive names (not implementations — those are host-specific)")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "primitives.sx, boundary.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Spec provenance")
(td (~tw :tokens "px-3 py-2 text-stone-700") "CIDs of the spec files that produced this image")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "eval.sx, render.sx, parser.sx, ...")))))
(p "Notably " (strong "absent") " from the image:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "IO primitive implementations") " — these are host-specific. The image records their " (em "names") " (for boundary enforcement) but not their code.")
(li (strong "Page helpers") " — same reason. " (code "fetch-data") ", " (code "app-url") " etc. are registered by the host app at startup.")
(li (strong "Runtime state") " — no request context, no DB connections, no session data. The image is a pure function's result, not a running process snapshot.")))
;; -----------------------------------------------------------------------
;; Image format
;; -----------------------------------------------------------------------
(~docs/section :title "Image Format" :id "format"
(p "The image is itself an s-expression — the same format the spec is written in. This means the image can be parsed by the same parser, inspected by the same tools, and content-addressed by the same canonical serializer.")
(~docs/code :src (highlight "(sx-image\n :version 1\n :spec-cids {:eval \"bafy...eval\"\n :render \"bafy...render\"\n :parser \"bafy...parser\"\n :primitives \"bafy...prims\"\n :boundary \"bafy...boundary\"\n :signals \"bafy...signals\"}\n\n :components (\n (defcomp ~plans/environment-images/card (&key title subtitle &rest children)\n (div :class \"card\" (h2 title) (when subtitle (p subtitle)) children))\n (defcomp ~plans/environment-images/nav (&key items current)\n (nav :class \"nav\" (map (fn (item) ...) items)))\n ;; ... all registered components\n )\n\n :macros (\n (defmacro when (test &rest body)\n (list 'if test (cons 'begin body) nil))\n ;; ... all macros\n )\n\n :bindings (\n (define void-elements (list \"area\" \"base\" \"br\" \"col\" ...))\n (define boolean-attrs (list \"checked\" \"disabled\" ...))\n ;; ... all top-level defines\n )\n\n :primitive-names (\"str\" \"+\" \"-\" \"*\" \"/\" \"=\" \"<\" \">\" ...)\n :io-names (\"fetch-data\" \"call-action\" \"app-url\" ...))" "lisp"))
(p "The " (code ":spec-cids") " field is the key. It links this image back to the exact spec that produced it. Anyone can verify the image by:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-1")
(li "Fetch the spec files by CID")
(li "Evaluate them with a conforming evaluator")
(li "Serialize the resulting environment")
(li "Compare — it must produce the same image")))
;; -----------------------------------------------------------------------
;; Image CID
;; -----------------------------------------------------------------------
(~docs/section :title "Image CID" :id "image-cid"
(p "The image CID is computed by canonical-serializing the entire " (code "(sx-image ...)") " form and hashing it. Same process as component CIDs, just applied to the whole environment.")
(p "The relationship between spec CIDs and image CID is deterministic:")
(~docs/code :src (highlight ";; The image CID is a pure function of the spec CIDs\n;; (assuming a deterministic evaluator, which SX guarantees)\n(define image-cid-from-specs\n (fn (spec-cids)\n ;; 1. Fetch each spec file by CID\n ;; 2. Evaluate all specs in a fresh environment\n ;; 3. Extract components, macros, bindings\n ;; 4. Build (sx-image ...) form\n ;; 5. Canonical serialize\n ;; 6. Hash → CID\n ))" "lisp"))
(p "This means you can compute the expected image CID from the spec CIDs " (em "without") " having the image. If someone hands you an image claiming to be from spec " (code "bafy...eval") ", you can verify it by re-evaluating the spec and comparing CIDs. The image is a verifiable cache.")
(p "In practice, you'd only do this verification once per spec version. After that, the image CID is trusted by content-addressing — same bytes, same hash, forever."))
;; -----------------------------------------------------------------------
;; Endpoint provenance
;; -----------------------------------------------------------------------
(~docs/section :title "Endpoint Provenance" :id "provenance"
(p "Every served page gains a provenance header linking it to the spec that rendered it:")
(~docs/code :src (highlight "HTTP/1.1 200 OK\nContent-Type: text/html\nSX-Spec: bafy...eval,bafy...render,bafy...parser,bafy...prims\nSX-Image: bafy...image\nSX-Page-Components: ~plans/environment-images/card:bafy...card,~plans/environment-images/nav:bafy...nav" "http"))
(p "Three levels of verification:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Level")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "What you verify")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Trust assumption")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Component")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Fetch " (code "~plans/environment-images/card") " by CID, verify hash")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Trust the evaluator"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Image")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Fetch image by CID, deserialize, re-render page")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Trust the image producer"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Spec")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Fetch specs by CID, re-evaluate, compare image CID")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Trust only the hash function")))))
(p "Level 3 is the nuclear option — full independent verification from source. It's expensive but proves the entire chain. Most consumers will operate at level 1 (component verification) or level 2 (image verification)."))
;; -----------------------------------------------------------------------
;; Cold start optimization
;; -----------------------------------------------------------------------
(~docs/section :title "Cold Start: Images as Cache" :id "cold-start"
(p "The practical motivation: evaluating all spec files + service components on every server restart is slow. An image eliminates this.")
(~docs/subsection :title "Server Startup"
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-2")
(li "Check if a cached image exists for the current spec CIDs")
(li "If yes: deserialize the image (fast — parsing a single file, no evaluation)")
(li "If no: evaluate spec files from source, build image, cache it")
(li "Register IO primitives and page helpers (host-specific, not in image)")
(li "Ready to serve"))
(~docs/code :src (highlight "(define load-environment\n (fn (spec-cids image-cache-dir)\n (let ((expected-image-cid (image-cid-for-specs spec-cids))\n (cached-path (str image-cache-dir \"/\" expected-image-cid \".sx\")))\n (if (file-exists? cached-path)\n ;; Fast path: deserialize\n (let ((image (parse (read-file cached-path))))\n (if (= (verify-image-cid image) expected-image-cid)\n (deserialize-image image)\n ;; Cache corrupted — rebuild\n (build-and-cache-image spec-cids image-cache-dir)))\n ;; Cold path: evaluate from source\n (build-and-cache-image spec-cids image-cache-dir)))))" "lisp")))
(~docs/subsection :title "Client Boot"
(p "The client already caches component definitions in localStorage keyed by bundle hash. Images extend this: cache the entire evaluated environment, not just individual components.")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-1")
(li "Page ships " (code "SX-Image") " header with image CID")
(li "Client checks localStorage for " (code "sx-image:{cid}"))
(li "If hit: deserialize and boot (no component-by-component parsing)")
(li "If miss: fetch image from origin or IPFS, deserialize, cache")
(li "Same cache-forever semantics as component CIDs — content can't be stale"))
(p "First visit to any SX-powered site: one image fetch. Every subsequent visit: instant boot from cache. Cross-site: if two sites share the same spec CIDs, the image is shared too.")))
;; -----------------------------------------------------------------------
;; Standalone HTML
;; -----------------------------------------------------------------------
(~docs/section :title "Standalone HTML Bundles" :id "standalone"
(p "An image can be inlined into a single HTML document, producing a fully self-contained application with no server dependency:")
(~docs/code :src (highlight "<!doctype html>\n<html>\n<head>\n <script type=\"text/sx-image\">\n (sx-image\n :version 1\n :spec-cids {...}\n :components (...)\n :macros (...)\n :bindings (...))\n </script>\n <script type=\"text/sx-pages\">\n (defpage home :path \"/\" :content (~home-page))\n </script>\n <script src=\"sx-ref.js\"></script>\n</head>\n<body>\n <div id=\"app\"></div>\n <script>\n // Deserialize image, register components, render\n Sx.bootFromImage(document.querySelector('[type=\"text/sx-image\"]'))\n </script>\n</body>\n</html>" "html"))
(p "This document is its own CID. Pin it to IPFS and it's a permanent, executable, verifiable application. No origin server, no CDN, no DNS. The content network is the deployment target.")
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-4 mt-4")
(p (~tw :tokens "text-amber-900 font-medium") "Juno comparison")
(p (~tw :tokens "text-amber-800") "Juno's Seedling IDE already does this — export a running environment as a standalone HTML file. But their images are opaque JavaScript blobs serialized by the runtime. SX images are " (strong "s-expressions") " — parseable, inspectable, content-addressable. You can diff two SX images and see exactly what changed. You can extract a single component from an image by CID. You can merge images from different sources by composing their component lists. The format IS the tooling.")))
;; -----------------------------------------------------------------------
;; Namespace scoping
;; -----------------------------------------------------------------------
(~docs/section :title "Namespaced Environments" :id "namespaces"
(p "As the component library grows across services, a flat environment risks name collisions. Images provide a natural boundary for namespace scoping.")
(~docs/code :src (highlight "(sx-image\n :version 1\n :namespace \"market\"\n :spec-cids {...}\n :extends \"bafy...shared-image\" ;; inherits shared components\n :components (\n ;; market-specific components\n (defcomp ~plans/environment-images/product-card ...)\n (defcomp ~plans/environment-images/price-tag ...)\n ))" "lisp"))
(p "Resolution: " (code "market/~plans/environment-images/product-card") " → look in market image first, then fall through to the shared image (via " (code ":extends") "). Each service produces its own image, layered on top of the shared base.")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Image")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Contents")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Extends")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "bafy...shared")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~plans/environment-images/card, ~plans/environment-images/nav, ~nav-data/section-nav, ~docs/page, ~docs/code — shared components from " (code "shared/sx/templates/"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "None (root)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "bafy...blog")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~shared:cards/post-card, ~post-body, ~tag-list — blog-specific from " (code "blog/sx/"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "bafy...shared"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "bafy...market")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~plans/environment-images/product-card, ~plans/environment-images/price-tag, ~shared:fragments/cart-mini — market-specific from " (code "market/sx/"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "bafy...shared"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "bafy...sx-docs")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~docs/section, ~examples/source, plans, essays — sx docs from " (code "sx/sx/"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "bafy...shared")))))
(p "The " (code ":extends") " field is a CID, not a name. Image composition is content-addressed: changing the shared image produces a new shared CID, which invalidates all service images that extend it. Exactly the right cascading behavior."))
;; -----------------------------------------------------------------------
;; Spec → Image → Page chain
;; -----------------------------------------------------------------------
(~docs/section :title "The Verification Chain" :id "chain"
(p "The full provenance chain from served page back to source:")
(~docs/code :src (highlight ";; 1. Page served with provenance headers\n;;\n;; SX-Spec: bafy...eval,bafy...render,...\n;; SX-Image: bafy...market-image\n;; SX-Page: (defpage product :path \"/products/<slug>\" ...)\n;;\n;; 2. Verify image → spec\n;;\n;; Fetch specs by CID → evaluate → build image → compare CID\n;; If match: the image was correctly produced from these specs\n;;\n;; 3. Verify page → image\n;;\n;; Deserialize image → evaluate page defn → render\n;; If output matches served HTML: the page was correctly rendered\n;;\n;; 4. Trust chain terminates at the spec\n;;\n;; The spec is self-hosting (eval.sx evaluates itself)\n;; The spec's CID is its identity\n;; No external trust anchor needed beyond the hash function" "lisp"))
(p "This is stronger than code signing. Code signing says " (em "\"this entity vouches for this binary.\"") " Content addressing says " (em "\"this binary is the deterministic output of this source.\"") " No entity needed. No certificate authority. No revocation lists. Just math."))
;; -----------------------------------------------------------------------
;; Implementation phases
;; -----------------------------------------------------------------------
(~docs/section :title "Implementation" :id "implementation"
(~docs/subsection :title "Phase 1: Image Serialization"
(p "Spec module " (code "image.sx") " — serialize and deserialize evaluated environments.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "serialize-environment") " — walk the env, extract components/macros/bindings, produce " (code "(sx-image ...)") " form")
(li (code "deserialize-image") " — parse image, reconstitute components/macros/bindings into env")
(li (code "image-cid") " — canonical-serialize the image form, hash → CID")
(li "Must handle closure serialization — component closures reference other components by name, which must be re-linked on deserialization")))
(~docs/subsection :title "Phase 2: Spec Provenance"
(p "Compute CIDs for all spec files at startup. Attach to environment metadata.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Hash each spec file's canonical source at load time")
(li "Store in env metadata as " (code ":spec-cids") " dict")
(li "Include in image serialization")))
(~docs/subsection :title "Phase 3: Server-Side Caching"
(p "Cache images on disk keyed by spec CIDs. Skip evaluation on warm restart.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "On startup: compute spec CIDs → derive expected image CID → check cache")
(li "Cache hit: deserialize (parse only, no eval)")
(li "Cache miss: evaluate specs, serialize image, write cache")
(li "Any spec file change → new spec CID → new image CID → cache miss → rebuild")))
(~docs/subsection :title "Phase 4: Client Images"
(p "Ship image CID in response headers. Client caches full env in localStorage.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "SX-Image") " response header with image CID")
(li "Client boot checks localStorage for cached image")
(li "Cache hit: deserialize, skip per-component fetch/parse")
(li "Cache miss: fetch image (single request), deserialize, cache")))
(~docs/subsection :title "Phase 5: Standalone Export"
(p "Generate self-contained HTML with inlined image. Pin to IPFS.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Inline " (code "(sx-image ...)") " as " (code "<script type=\"text/sx-image\">"))
(li "Inline page definitions as " (code "<script type=\"text/sx-pages\">"))
(li "Include sx-ref.js (or link to its CID)")
(li "The resulting HTML is a complete application — pin its CID to IPFS")))
(~docs/subsection :title "Phase 6: Namespaced Images"
(p "Per-service images with " (code ":extends") " for layered composition.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Shared image: components from " (code "shared/sx/templates/"))
(li "Service images: extend shared, add service-specific components")
(li "Resolution: service image → shared image → primitives")
(li "Image merge: combine two images with conflict detection"))))
;; -----------------------------------------------------------------------
;; Dependencies
;; -----------------------------------------------------------------------
(~docs/section :title "Dependencies" :id "dependencies"
(p "What must exist before this plan can execute:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Dependency")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Status")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Plan")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Canonical serialization")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-red-700 font-medium") "Not started"))
(td (~tw :tokens "px-3 py-2") (a :href "/sx/(etc.(plan.content-addressed-components))" (~tw :tokens "text-violet-700 underline") "Content-Addressed Components") " Phase 1"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Component CIDs")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-red-700 font-medium") "Not started"))
(td (~tw :tokens "px-3 py-2") (a :href "/sx/(etc.(plan.content-addressed-components))" (~tw :tokens "text-violet-700 underline") "Content-Addressed Components") " Phase 2"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Purity verification")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-green-700 font-medium") "Complete"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "deps.sx + boundary.py"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Self-hosting spec")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-green-700 font-medium") "Complete"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "eval.sx, render.sx, parser.sx, ..."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Self-hosting bootstrappers")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-green-700 font-medium") "Complete"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "py.sx, js.sx — G0 == G1"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "IPFS infrastructure")
(td (~tw :tokens "px-3 py-2 text-stone-700") (span (~tw :tokens "text-green-700 font-medium") "Exists"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "artdag L1/L2, IPFSPin model")))))
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-3 mt-2")
(p (~tw :tokens "text-amber-800 text-sm") (strong "Builds on: ") (a :href "/sx/(etc.(plan.content-addressed-components))" (~tw :tokens "underline") "Content-Addressed Components") " (canonical serialization + CIDs), " (a :href "/sx/(etc.(plan.self-hosting-bootstrapper))" (~tw :tokens "underline") "self-hosting bootstrappers") " (spec-first architecture). " (strong "Enables: ") (a :href "/sx/(etc.(plan.sx-activity))" (~tw :tokens "underline") "SX-Activity") " (serverless applications on IPFS).")))))

818
sx/sx/plans/foundations.sx
View File

@@ -1,818 +0,0 @@
;; Foundations — The Computational Floor
;; From scoped effects to CEK: what's beneath algebraic effects and why it's the limit.
(defcomp ~plans/foundations/plan-foundations-content ()
(~docs/page :title "Foundations \u2014 The Computational Floor"
(p (~tw :tokens "text-stone-500 text-sm italic mb-8")
"Scoped effects unify everything SX does. But they are not the bottom. "
"Beneath them is a hierarchy of increasingly fundamental primitives, "
"terminating at three irreducible things: an expression, an environment, and a continuation. "
"This document traces that hierarchy, proves where the floor is, "
"and maps the path from where SX stands to the deepest layer.")
;; -----------------------------------------------------------------------
;; The hierarchy
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "The Hierarchy")
(p "Every layer is definable in terms of the one below. "
"No layer can be decomposed without the layer beneath it.")
(~docs/code :src
(str "Layer 0: CEK machine (expression + environment + continuation) \u2714 DONE\n"
"Layer 1: Continuations (shift / reset \u2014 delimited capture) \u2714 DONE\n"
"Layer 2: Algebraic effects (operations + handlers) \u2714 DONE\n"
"Layer 3: Scoped effects (+ region delimitation) \u2714 DONE\n"
"Layer 4: SX patterns (spread, provide, island, lake, signal) \u2714 DONE"))
(p "All five layers are implemented. The entire hierarchy from patterns down to raw CEK "
"is specced in " (code ".sx") " files and bootstrapped to Python and JavaScript. "
"No hand-written evaluation logic remains.")
;; -----------------------------------------------------------------------
;; What we built (status)
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "What We Built")
(div (~tw :tokens "overflow-x-auto mb-6")
(table (~tw :tokens "min-w-full text-sm")
(thead (tr
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Layer")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Spec files")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "What it provides")
(th (~tw :tokens "text-left pb-2 font-semibold") "Tests")))
(tbody
(tr (td (~tw :tokens "pr-4 py-1") "0 \u2014 CEK")
(td (~tw :tokens "pr-4 font-mono text-xs") "cek.sx, frames.sx")
(td (~tw :tokens "pr-4") "Explicit step function, 20+ frame types, cek-call dispatch, CEK-native HO forms")
(td "43 CEK + 26 reactive"))
(tr (td (~tw :tokens "pr-4 py-1") "1 \u2014 Continuations")
(td (~tw :tokens "pr-4 font-mono text-xs") "continuations.sx, callcc.sx")
(td (~tw :tokens "pr-4") "shift/reset (delimited), call/cc (full), ReactiveResetFrame + DerefFrame")
(td "Continuation tests"))
(tr (td (~tw :tokens "pr-4 py-1") "2 \u2014 Effect signatures")
(td (~tw :tokens "pr-4 font-mono text-xs") "boundary.sx, eval.sx")
(td (~tw :tokens "pr-4") ":effects annotations on define, boundary enforcement at startup")
(td "Boundary validation"))
(tr (td (~tw :tokens "pr-4 py-1") "3 \u2014 Scoped effects")
(td (~tw :tokens "pr-4 font-mono text-xs") "eval.sx, adapters")
(td (~tw :tokens "pr-4") "scope/provide/context/emit!/emitted, scope-push!/scope-pop!")
(td "Scope integration"))
(tr (td (~tw :tokens "pr-4 py-1") "4 \u2014 Patterns")
(td (~tw :tokens "pr-4 font-mono text-xs") "signals.sx, adapter-dom.sx, engine.sx")
(td (~tw :tokens "pr-4") "signal/deref/computed/effect/batch, island/lake, spread/collect")
(td "20 signal + 26 CEK reactive")))))
;; -----------------------------------------------------------------------
;; Layer 0: The CEK machine
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "Layer 0: The CEK Machine")
(p "The CEK machine (Felleisen & Friedman, 1986) is a small-step evaluator with three registers:")
(div (~tw :tokens "overflow-x-auto mb-6")
(table (~tw :tokens "min-w-full text-sm")
(thead (tr
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Register")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Name")
(th (~tw :tokens "text-left pb-2 font-semibold") "Meaning")))
(tbody
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "C") (td (~tw :tokens "pr-4") "Control") (td "The expression being evaluated"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "E") (td (~tw :tokens "pr-4") "Environment") (td "The bindings in scope \u2014 names to values"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "K") (td (~tw :tokens "pr-4") "Kontinuation") (td "What to do with the result")))))
(p "Every step of evaluation is a transition: take the current (C, E, K), "
"produce a new (C\u2032, E\u2032, K\u2032). That's it. That's all computation is.")
(p "Can you remove any register?")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (strong "Remove C") " \u2014 nothing to evaluate. No computation.")
(li (strong "Remove E") " \u2014 names are meaningless. Lose abstraction. "
"Every value must be literal, every function must be closed. No variables, no closures, no reuse.")
(li (strong "Remove K") " \u2014 no sequencing. The result of an expression goes nowhere. "
"Lose composition \u2014 you can evaluate one thing but never connect it to the next."))
(p "Three things, all necessary, none decomposable further.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "CEK in SX")
(p "The CEK machine is the default evaluator on both client (JS) and server (Python). "
"Every " (code "eval-expr") " call goes through " (code "cek-run") ". "
"The spec lives in two files:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "frames.sx") " \u2014 20+ frame types (IfFrame, ArgFrame, MapFrame, ReactiveResetFrame, ...)")
(li (code "cek.sx") " \u2014 step function, run loop, special form handlers, HO form handlers, cek-call"))
(p (code "cek-call") " is the universal function dispatch. "
"It replaces the old " (code "invoke") " shim \u2014 SX lambdas go through " (code "cek-run") ", "
"native callables through " (code "apply") ". One calling convention, bootstrapped identically to every host.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "CEK-native higher-order forms")
(p "All higher-order forms step element-by-element through the CEK machine:")
(~docs/code :src
(str ";; map pushes a MapFrame, calls f on each element via continue-with-call\n"
"(map (fn (x) (* 2 (deref counter))) items)\n"
"\n"
";; deref inside the callback goes through CEK's DerefFrame\n"
";; \u2192 reactive-shift-deref fires if inside a reactive-reset boundary\n"
";; \u2192 the continuation from deref to the MapFrame is captured as a subscriber"))
(p "This means " (code "deref") " works inside " (code "map") ", " (code "filter") ", "
(code "reduce") ", " (code "for-each") ", " (code "some") ", " (code "every?") " callbacks. "
"The HO forms don't escape to tree-walk \u2014 the CEK machine processes every step.")
;; -----------------------------------------------------------------------
;; Layer 1: Delimited continuations
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "Layer 1: Delimited Continuations")
(p "Delimited continuations (Felleisen 1988, Danvy & Filinski 1990) "
"expose the K register as a first-class value:")
(~docs/code :src
(str ";; reset marks a point in the continuation\n"
"(reset\n"
" (+ 1 (shift k ;; k = \"the rest up to reset\"\n"
" (k (k 10)))) ;; invoke it twice: 1 + (1 + 10) = 12\n"
")"))
(p (code "reset") " says: \"here's a boundary.\" "
(code "shift") " says: \"give me everything between here and that boundary as a callable function.\" "
"The captured continuation " (em "is") " a slice of the K register.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "Deref as shift")
(p "The reactive payoff. " (code "deref") " inside a " (code "reactive-reset") " boundary "
"is shift/reset applied to signals:")
(~docs/code :src
(str ";; User writes:\n"
"(div :class (str \"count-\" (deref counter))\n"
" (str \"Value: \" (deref counter)))\n"
"\n"
";; CEK sees (deref counter) \u2192 signal? \u2192 reactive-reset on stack?\n"
";; Yes: capture (str \"count-\" [HOLE]) as continuation\n"
";; Register as subscriber. Return current value.\n"
";; When counter changes: re-invoke continuation \u2192 update DOM."))
(p "No explicit " (code "effect()") " wrapping needed. "
"The continuation capture IS the subscription mechanism.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "The Filinski embedding")
(p "Filinski (1994) proved that " (code "shift/reset") " can encode "
(em "any") " monadic effect. State, exceptions, nondeterminism, I/O, "
"continuations themselves \u2014 all expressible as shift/reset patterns. "
"This means layer 1 is already computationally complete for effects. "
"Everything above is structure, not power.")
;; -----------------------------------------------------------------------
;; The floor proof
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "Why Layer 0 Is the Floor")
(p "Two independent arguments:")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "1. Church\u2013Turing")
(p "The Church\u2013Turing thesis: any effectively computable function can be computed by "
"a Turing machine (equivalently, by the lambda calculus, equivalently, by the CEK machine). "
"This has withstood 90 years of attempts to disprove it. "
"No formal system has ever been shown to compute more.")
(p "This means CEK captures " (em "all") " computation. Adding more primitives doesn't let you "
"compute anything new. It only changes what's " (em "convenient") " to express.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "2. Irreducibility of C, E, K")
(p "The three registers are independently necessary:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (strong "C without E") " = combinatory logic. Turing-complete but inhumane. "
"No named bindings \u2014 everything via S, K, I combinators.")
(li (strong "C without K") " = single expression evaluation. "
"You can compute one thing but can't compose it with anything.")
(li (strong "E without C") " = a phone book with no one to call.")
(li (strong "K without C") " = a to-do list with nothing on it."))
(p "You can " (em "encode") " any register into another (CPS eliminates K, "
"De Bruijn indices eliminate E), but encoding isn't elimination. "
"The information is still there, just hidden in a different representation.")
;; -----------------------------------------------------------------------
;; Digging deeper: what's next
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "Progress")
(div (~tw :tokens "overflow-x-auto mb-6")
(table (~tw :tokens "min-w-full text-sm")
(thead (tr
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Step")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "What")
(th (~tw :tokens "text-left pb-2 font-semibold") "Status")))
(tbody
(tr (td (~tw :tokens "pr-4 py-1") "1")
(td (~tw :tokens "pr-4") "Serializable CEK state")
(td (~tw :tokens "text-emerald-600 font-semibold") "\u2714 Done \u2014 freeze-scope, freeze-signal, freeze-to-sx, thaw-from-sx"))
(tr (td (~tw :tokens "pr-4 py-1") "2")
(td (~tw :tokens "pr-4") "CEK stepping debugger")
(td (~tw :tokens "text-emerald-600 font-semibold") "\u2714 Done \u2014 live island with reactive code view, DOM preview"))
(tr (td (~tw :tokens "pr-4 py-1") "3")
(td (~tw :tokens "pr-4") "Content-addressed computation")
(td (~tw :tokens "text-emerald-600 font-semibold") "\u2714 Done \u2014 content-hash, freeze-to-cid, thaw-from-cid"))
(tr (td (~tw :tokens "pr-4 py-1") "3.5")
(td (~tw :tokens "pr-4") "Data representations")
(td (~tw :tokens "text-stone-400") "Planned — byte buffers + typed structs"))
(tr (td (~tw :tokens "pr-4 py-1") "3.7")
(td (~tw :tokens "pr-4") "Verified components")
(td (~tw :tokens "text-stone-400") "Planned — content-addressed UI trust"))
(tr (td (~tw :tokens "pr-4 py-1") "4")
(td (~tw :tokens "pr-4") "Concurrent CEK")
(td (~tw :tokens "text-amber-600 font-semibold") "Spec complete — implementation next"))
(tr (td (~tw :tokens "pr-4 py-1") "5")
(td (~tw :tokens "pr-4") "Linear effects")
(td (~tw :tokens "text-stone-400") "Future")))))
;; -----------------------------------------------------------------------
;; Step 3.5: Data Representations
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "Step 3.5: Data Representations")
(p "Two primitives that sit below concurrency but above the raw CEK machine. "
"Both are about how values are represented — enabling work that's currently host-only.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "3.5a Byte Buffers")
(p "Fixed-size mutable byte arrays. A small primitive surface that unlocks "
"binary protocol parsing, image headers, wire formats, and efficient CID computation:")
(~docs/code :src
(str ";; Create and write\n"
"(let ((buf (make-buffer 256)))\n"
" (buffer-write-u8! buf 0 #xFF)\n"
" (buffer-write-u32! buf 1 magic-number)\n"
" (buffer-slice buf 0 5))\n"
"\n"
";; Read from network/file\n"
"(let ((header (buffer-slice packet 0 12)))\n"
" (case (buffer-read-u8 header 0)\n"
" (1 (parse-handshake header))\n"
" (2 (parse-data header))\n"
" (else (error \"unknown packet type\"))))"))
(p "Primitives:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "make-buffer") " — allocate N zero-filled bytes")
(li (code "buffer-read-u8/u16/u32") ", " (code "buffer-read-i8/i16/i32") " — typed reads at offset")
(li (code "buffer-write-u8!/u16!/u32!") " — typed writes at offset (big-endian default)")
(li (code "buffer-slice") " — zero-copy view of a region")
(li (code "buffer-length") " — byte count")
(li (code "buffer->list") ", " (code "list->buffer") " — conversion to/from byte lists")
(li (code "buffer->string") ", " (code "string->buffer") " — UTF-8 encode/decode"))
(p "Host mapping:")
(div (~tw :tokens "overflow-x-auto mb-6")
(table (~tw :tokens "min-w-full text-sm")
(thead (tr
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Primitive")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "JavaScript")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Python")
(th (~tw :tokens "text-left pb-2 font-semibold") "OCaml")))
(tbody
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "make-buffer")
(td (~tw :tokens "pr-4") "new ArrayBuffer(n)")
(td (~tw :tokens "pr-4") "bytearray(n)")
(td "Bytes.create n"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "buffer-read-u32")
(td (~tw :tokens "pr-4") "DataView.getUint32")
(td (~tw :tokens "pr-4") "struct.unpack_from")
(td "Bytes.get_int32_be"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "buffer-slice")
(td (~tw :tokens "pr-4") "new Uint8Array(buf, off, len)")
(td (~tw :tokens "pr-4") "memoryview(buf)[off:off+len]")
(td "Bytes.sub buf off len")))))
(p "Byte buffers connect directly to content addressing (Step 3). "
"Computing a CID currently requires host-side hashing of opaque SX text. "
"With buffers, the hash input can be a canonical binary representation — "
"deterministic, compact, and consistent across hosts.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "3.5b Typed Structs")
(p "Named product types with fixed fields. On interpreted hosts, structs are syntactic sugar over dicts. "
"On compiled hosts (OCaml, Rust), they compile to unboxed records — no dict overhead, "
"no hash lookups, no string key comparisons:")
(~docs/code :src
(str ";; Define a struct\n"
"(defstruct point (x : number) (y : number))\n"
"\n"
";; Construct\n"
"(let ((p (make-point 3 4)))\n"
" ;; Access — compiles to field offset on native hosts\n"
" (+ (point-x p) (point-y p))) ;; => 7\n"
"\n"
";; Pattern match\n"
"(case p\n"
" ((point x y) (sqrt (+ (* x x) (* y y))))) ;; => 5"))
(p "What " (code "defstruct") " generates:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "make-point") " — constructor (arity-checked)")
(li (code "point?") " — type predicate")
(li (code "point-x") ", " (code "point-y") " — field accessors")
(li (code "point") " — case pattern for destructuring"))
(p "The performance difference matters for compiled hosts:")
(div (~tw :tokens "overflow-x-auto mb-6")
(table (~tw :tokens "min-w-full text-sm")
(thead (tr
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Operation")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Dict (current)")
(th (~tw :tokens "text-left pb-2 font-semibold") "Struct (compiled)")))
(tbody
(tr (td (~tw :tokens "pr-4 py-1") "Field access")
(td (~tw :tokens "pr-4") "Hash lookup — O(1) amortized, cache-unfriendly")
(td "Offset load — O(1) actual, single instruction"))
(tr (td (~tw :tokens "pr-4 py-1") "Construction")
(td (~tw :tokens "pr-4") "Allocate hash table + insert N entries")
(td "Allocate N words, write sequentially"))
(tr (td (~tw :tokens "pr-4 py-1") "Pattern match")
(td (~tw :tokens "pr-4") "N string comparisons")
(td "Tag check + field projection")))))
(p "Typed structs connect to the gradual type system in " (code "types.sx") ". "
"A " (code "defstruct") " declaration is a type definition that the type checker can verify "
"and the compiler can exploit. On interpreted hosts, the same code runs — just slower.")
;; -----------------------------------------------------------------------
;; Step 3.7: Verified Components
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "Step 3.7: Verified Components")
(p "Content-addressed components become a trust mechanism. "
"HTTPS tells you the connection is authentic. "
"Verified components tell you the " (em "UI") " is authentic — "
"that the payment form in your browser is the exact component that was audited, "
"not a tampered copy injected by XSS, a rogue extension, or a compromised CDN.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "Why SX can do this")
(p "Most frameworks can't verify UI at the component level because there's no stable identity. "
"A React component is compiled, bundled, minified, tree-shaken — "
"the thing in the browser bears no relationship to the source. In SX:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (strong "Components are source") " — the " (code ".sx") " definition IS the component. No compilation step that could diverge.")
(li (strong "Components are pure functions") " — same inputs, same output. Deterministic.")
(li (strong "Content addressing is built in") " — " (code "freeze-to-cid") " gives every component a CID (Step 3).")
(li (strong "The evaluator runs in the browser") " — the client can independently compute the CID of any component it receives."))
(p "Because components are pure functions defined in source form, "
"verifying the definition IS verifying the behaviour. "
"There is no gap between \"what was audited\" and \"what runs.\" "
"That gap is where every UI supply chain attack lives.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "3.7a Transitive closure CID")
(p "A component's CID must cover its entire dependency tree. "
"If " (code "~bank/payment-form") " calls " (code "~bank/amount-input") " calls "
(code "~ui/text-field") ", all three definitions are part of the CID:")
(~docs/code :src
(str ";; Shallow CID — just this component's definition\n"
"(freeze-to-cid ~bank/payment-form) ;; => bafyrei..abc\n"
"\n"
";; Deep CID — component + all transitive dependencies\n"
"(freeze-to-cid-deep ~bank/payment-form) ;; => bafyrei..xyz\n"
"\n"
";; The deep CID changes if ANY dependency changes.\n"
";; A one-character change in ~ui/text-field\n"
";; produces a completely different deep CID."))
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "3.7b Canonical serialization")
(p "For CIDs to match across hosts, the serialized form must be identical. "
"Canonical SX: no comments, no redundant whitespace, deterministic key ordering in dicts, "
"normalized number representation:")
(~docs/code :src
(str ";; These must produce the same CID on JS, Python, and OCaml:\n"
"(canonical-sx '(div :class \"card\" (p \"hello\")))\n"
";; => \"(div :class \\\"card\\\" (p \\\"hello\\\"))\"\n"
"\n"
";; Dict key ordering is sorted:\n"
"(canonical-sx '{:b 2 :a 1}) ;; => \"{:a 1 :b 2}\""))
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "3.7c Browser verification")
(p "The client-side verification flow:")
(~docs/code :src
(str ";; Server sends component + CID via aser wire format\n"
";; Browser receives, independently computes CID, compares\n"
"\n"
";; Per-component verification\n"
"(component-verify ~bank/payment-form\n"
" :expected-cid \"bafyrei...\"\n"
" :on-mismatch :refuse) ;; or :warn, :log\n"
"\n"
";; Verify entire page component tree against published manifest\n"
"(page-verify\n"
" :manifest-url \"/.well-known/sx-manifest.json\"\n"
" :on-mismatch :refuse)\n"
"\n"
";; Query verification status (for UI indicators)\n"
"(verified? ~bank/payment-form) ;; => true/false"))
(p "Visual indicator — like the HTTPS lock icon, but for individual UI components. "
"The browser knows which components have verified CIDs and can surface this to the user.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "3.7d Manifest and discovery")
(p "Publishers declare expected CIDs via a well-known manifest:")
(~docs/code :src
(str ";; .well-known/sx-manifest.json\n"
"{\"version\": 1,\n"
" \"components\": {\n"
" \"~bank/payment-form\": {\n"
" \"cid\": \"bafyrei...abc\",\n"
" \"cid-deep\": \"bafyrei...xyz\",\n"
" \"audited\": \"2026-03-01\",\n"
" \"auditor\": \"security-firm.com\"\n"
" },\n"
" \"~bank/login\": {\n"
" \"cid\": \"bafyrei...def\",\n"
" \"cid-deep\": \"bafyrei...uvw\",\n"
" \"audited\": \"2026-02-15\"\n"
" }\n"
" },\n"
" \"signature\": \"...\"\n"
"}"))
(p "Alternative discovery mechanisms:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (strong "DNS TXT") " — " (code "_sx-verify.bank.com TXT \"payment-form=bafyrei...\""))
(li (strong "Certificate transparency") " — append-only log of component CIDs, publicly auditable")
(li (strong "IPFS") " — the CID is the address; fetching from IPFS is self-verifying")
(li (strong "Signed manifest") " — publisher signs the manifest with their TLS key; browser verifies signature"))
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "How this differs from SRI")
(p "Subresource Integrity (SRI) already does hash verification for " (code "<script>") " tags. "
"But SRI has three gaps that verified components close:")
(div (~tw :tokens "overflow-x-auto mb-6")
(table (~tw :tokens "min-w-full text-sm")
(thead (tr
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "SRI")
(th (~tw :tokens "text-left pb-2 font-semibold") "SX Verified Components")))
(tbody
(tr (td (~tw :tokens "pr-4 py-1 font-semibold") "Granularity")
(td (~tw :tokens "pr-4") "Whole files (a JS bundle)")
(td "Individual components"))
(tr (td (~tw :tokens "pr-4 py-1 font-semibold") "Who sets the hash?")
(td (~tw :tokens "pr-4") "The server — if compromised, serves matching hashes")
(td "Independent manifest — client verifies against external source"))
(tr (td (~tw :tokens "pr-4 py-1 font-semibold") "What's verified?")
(td (~tw :tokens "pr-4") "The file bytes — says nothing about runtime behaviour")
(td "The definition — and since components are pure functions, definition = behaviour")))))
;; -----------------------------------------------------------------------
;; Step 4: Concurrent CEK — deep spec
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "Step 4: Concurrent CEK")
(p "Multiple CEK machines running concurrently. The machines are independent \u2014 "
"each has its own C, E, K registers. Communication is via typed channels. "
"The host provides the concurrency mechanism (Web Workers, forkIO, tokio::spawn). "
"The spec defines the coordination primitives.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "4.1 Spawn")
(p "The fundamental primitive. Create a new CEK machine from a thunk, "
"run it concurrently, return a signal that resolves with the result:")
(~docs/code :src
(str "(let ((result (spawn (fn () (* 6 7)))))
"
" ;; result is a signal \u2014 nil until the computation completes
"
" (effect (fn ()
"
" (when (deref result)
"
" (log (str \"answer: \" (deref result))))))
"
" ;; => \"answer: 42\" (eventually)"))
(p "Implementation: the host freezes the thunk\u2019s CEK state, ships it to a worker, "
"the worker thaws and runs, posts back the result, the signal resolves. "
"The reactive system propagates the change \u2014 any island deref\u2019ing the signal re-renders.")
(p "The default implementation is synchronous (no actual concurrency). "
"The host overrides " (code "spawn-impl") " for real concurrency:")
(~docs/code :src
(str ";; Host-provided spawn (Web Worker)
"
"(set! spawn-impl
"
" (fn (thunk result-sig)
"
" (let ((frozen (freeze-to-sx \"spawn\")))
"
" (worker-post frozen)
"
" (worker-on-message
"
" (fn (val) (reset! result-sig val))))))"))
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "4.2 Channels")
(p "Typed communication pipes between concurrent computations. "
"Send is non-blocking. Receive suspends via " (code "shift") " until data arrives:")
(~docs/code :src
(str ";; Create a channel
"
"(make-channel \"results\")
"
"
"
";; Producer (in spawned computation)
"
"(spawn (fn ()
"
" (channel-send \"results\" (expensive-computation))))
"
"
"
";; Consumer (in main thread)
"
"(let ((val (channel-receive \"results\")))
"
" ;; val is a signal if no data yet, or the value if buffered
"
" (when (signal? val)
"
" ;; Reactive: re-renders when data arrives
"
" (div \"Waiting...\" (deref val))))"))
(p "Channels are named and registered globally. Multiple producers and consumers "
"can share a channel. The buffer stores values until a consumer reads them. "
"When a consumer arrives and the buffer is empty, a signal is returned "
"that resolves when a producer sends.")
(p "Channel semantics:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (strong "Unbuffered") " \u2014 send blocks until a receiver is ready (rendezvous)")
(li (strong "Buffered") " \u2014 send succeeds immediately, values queue (current default)")
(li (strong "Broadcast") " \u2014 every receiver gets every value (pub/sub)")
(li (strong "Select") " \u2014 wait on multiple channels, take the first that has data"))
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "4.3 Fork / Join")
(p "Spawn multiple computations, collect all results. "
"The spec primitive, not a library pattern:")
(~docs/code :src
(str ";; Fork: run N computations concurrently
"
"(let ((results (fork-join (list
"
" (fn () (fetch-user 1))
"
" (fn () (fetch-user 2))
"
" (fn () (fetch-user 3))))))
"
" ;; results is a list of signals
"
" ;; All resolve when all computations complete
"
" (div
"
" (map (fn (r) (div (deref r))) results)))"))
(p "Fork/join composes with the reactive system. Each result signal drives "
"a reactive DOM node. Results appear as they complete, not all-or-nothing.")
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "4.4 Scheduler")
(p "The scheduler decides which CEK machine steps next. "
"Different strategies for different use cases:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (strong "Round-robin") " \u2014 fair, each machine gets equal time")
(li (strong "Priority") " \u2014 UI-facing computations step first")
(li (strong "Work-stealing") " \u2014 idle workers steal from busy ones")
(li (strong "Cooperative") " \u2014 machines yield explicitly via " (code "yield!"))
(li (strong "DAG-ordered") " \u2014 step machines whose inputs are ready (Art DAG)"))
(~docs/code :src
(str ";; Cooperative scheduling
"
"(spawn (fn ()
"
" (for-each (fn (item)
"
" (process item)
"
" (yield!) ;; let other machines run
"
" ) items)))
"
"
"
";; DAG scheduling
"
"(dag-execute
"
" {:nodes {:a (fn () (load-image \"input.jpg\"))
"
" :b (fn (a) (resize a 512 512))
"
" :c (fn (a) (blur a 3))
"
" :d (fn (b c) (composite b c))}
"
" :edges {[:a] [:b :c]
"
" [:b :c] [:d]}})"))
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "4.5 Content-addressed concurrency")
(p "Concurrent computation + content addressing = verifiable parallel execution:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li "Each spawned computation\u2019s initial state has a CID")
(li "Each result has a CID")
(li "The mapping input-CID \u2192 output-CID is a proof of computation")
(li "Memoize: if input-CID was computed before, return cached output-CID")
(li "Distribute: send input-CID to whichever worker has the data")
(li "Verify: any machine can re-run from input-CID and check output-CID matches"))
(~docs/code :src
(str ";; Content-addressed spawn
"
"(let ((input-cid (freeze-to-cid \"job\")))
"
" ;; Check cache first
"
" (or (result-cache-get input-cid)
"
" ;; Not cached \u2014 compute and cache
"
" (let ((result (spawn (fn () (expensive-work)))))
"
" (effect (fn ()
"
" (when (deref result)
"
" (result-cache-put input-cid
"
" (freeze-to-cid \"result\")))))
"
" result)))"))
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "4.6 Host mapping")
(p "Each host implements concurrency differently. The spec is the same:")
(div (~tw :tokens "overflow-x-auto mb-6")
(table (~tw :tokens "min-w-full text-sm")
(thead (tr
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Primitive")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "JavaScript")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Python")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "OCaml")
(th (~tw :tokens "text-left pb-2 font-semibold") "Rust/WASM")))
(tbody
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "spawn")
(td (~tw :tokens "pr-4") "Web Worker")
(td (~tw :tokens "pr-4") "asyncio.create_task")
(td (~tw :tokens "pr-4") "Eio.Fiber.fork")
(td "tokio::spawn"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "channel")
(td (~tw :tokens "pr-4") "MessageChannel")
(td (~tw :tokens "pr-4") "asyncio.Queue")
(td (~tw :tokens "pr-4") "Eio.Stream.t")
(td "mpsc::channel"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "yield!")
(td (~tw :tokens "pr-4") "setTimeout(0)")
(td (~tw :tokens "pr-4") "await asyncio.sleep(0)")
(td (~tw :tokens "pr-4") "Eio.Fiber.yield")
(td "tokio::task::yield_now"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "freeze/thaw")
(td (~tw :tokens "pr-4") "postMessage + JSON")
(td (~tw :tokens "pr-4") "pickle / SX text")
(td (~tw :tokens "pr-4") "Marshal + SX text")
(td "serde + SX text"))
(tr (td (~tw :tokens "pr-4 py-1 font-mono") "select")
(td (~tw :tokens "pr-4") "Promise.race")
(td (~tw :tokens "pr-4") "asyncio.wait FIRST_COMPLETED")
(td (~tw :tokens "pr-4") "Eio.Fiber.any")
(td "tokio::select!")))))
(h3 (~tw :tokens "text-lg font-semibold mt-8 mb-3") "4.7 Roadmap")
(div (~tw :tokens "overflow-x-auto mb-6")
(table (~tw :tokens "min-w-full text-sm")
(thead (tr
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Phase")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "What")
(th (~tw :tokens "text-left pr-4 pb-2 font-semibold") "Host")
(th (~tw :tokens "text-left pb-2 font-semibold") "Validates")))
(tbody
(tr (td (~tw :tokens "pr-4 py-1") "4a")
(td (~tw :tokens "pr-4") "spawn + Web Worker")
(td (~tw :tokens "pr-4") "JavaScript")
(td "Freeze/thaw across execution contexts"))
(tr (td (~tw :tokens "pr-4 py-1") "4b")
(td (~tw :tokens "pr-4") "Channels (buffered)")
(td (~tw :tokens "pr-4") "JavaScript")
(td "Cross-worker communication"))
(tr (td (~tw :tokens "pr-4 py-1") "4c")
(td (~tw :tokens "pr-4") "Fork/join + DAG scheduler")
(td (~tw :tokens "pr-4") "JavaScript")
(td "Art DAG integration path"))
(tr (td (~tw :tokens "pr-4 py-1") "4d")
(td (~tw :tokens "pr-4") "OCaml bootstrapper → native compilation")
(td (~tw :tokens "pr-4") "OCaml")
(td "Native performance, direct CEK-to-ML mapping"))
(tr (td (~tw :tokens "pr-4 py-1") "4e")
(td (~tw :tokens "pr-4") "Rust/WASM bootstrapper")
(td (~tw :tokens "pr-4") "Rust")
(td "Browser performance, WASM target"))
(tr (td (~tw :tokens "pr-4 py-1") "4f")
(td (~tw :tokens "pr-4") "Content-addressed spawn")
(td (~tw :tokens "pr-4") "Any")
(td "Memoization, distribution, verification"))
(tr (td (~tw :tokens "pr-4 py-1") "4g")
(td (~tw :tokens "pr-4") "IPFS-backed content store")
(td (~tw :tokens "pr-4") "Any")
(td "Global content addressing, peer-to-peer"))
(tr (td (~tw :tokens "pr-4 py-1") "4h")
(td (~tw :tokens "pr-4") "Select + broadcast channels")
(td (~tw :tokens "pr-4") "Any")
(td "Complex coordination patterns"))
(tr (td (~tw :tokens "pr-4 py-1") "4i")
(td (~tw :tokens "pr-4") "Linear channels")
(td (~tw :tokens "pr-4") "Any")
(td "Resource safety, exactly-once delivery")))))
(p "Each phase is independently valuable. Phase 4a (Web Worker spawn) is the immediate next build. "
"Phase 4d (OCaml) proves the spec compiles to native code — the CEK machine maps directly to ML's runtime model. "
"Phase 4e (Rust/WASM) proves it can run in the browser at near-native speed. "
"Phase 4g (IPFS) makes it distributed.")
;; -----------------------------------------------------------------------
;; Step 5: Linear effects (future)
;; -----------------------------------------------------------------------
(h2 (~tw :tokens "text-xl font-bold mt-12 mb-4") "Step 5: Linear Effects")
(p "Constrain the continuation: a handler " (em "must") " resume exactly once. "
"No dropping (resource leak), no duplicating (nondeterminism). "
"This is the linearity axis from the three-axis model.")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (strong "Affine channels") " \u2014 send at most once. Prevents duplicate messages.")
(li (strong "Linear scopes") " \u2014 a freeze scope that must be thawed. Prevents orphaned state.")
(li (strong "Session types") " \u2014 a protocol that must complete. Prevents half-open connections.")
(li (strong "Resource handles") " \u2014 a file/DB handle that must be closed. Prevents leaks."))
(p "Linear effects connect SX to Rust\u2019s ownership model. "
"Affine types (use at most once) are the practical sweet spot. "
"Full linearity (use exactly once) is for protocols and resources.")
(p "This is the furthest horizon. The infrastructure \u2014 freeze scopes, channels, "
"content addressing \u2014 must be solid first. Linearity is a discipline on top, "
"not a prerequisite beneath.")
(p (~tw :tokens "text-stone-500 text-sm italic mt-12")
"The floor is thought itself. We can't go deeper, because there's no one left to dig.")))

51
sx/sx/plans/fragment-protocol.sx
View File

@@ -1,51 +0,0 @@
;; ---------------------------------------------------------------------------
;; Fragment Protocol
;; ---------------------------------------------------------------------------
(defcomp ~plans/fragment-protocol/plan-fragment-protocol-content ()
(~docs/page :title "Fragment Protocol"
(~docs/section :title "Context" :id "context"
(p "Fragment endpoints return raw sexp source (e.g., " (code "(~blog-nav-wrapper :items ...)") "). The consuming service embeds this in its page sexp, which the client evaluates. But service-specific components like " (code "~blog-nav-wrapper") " are only in that service's component env — not in the consumer's. So the consumer's " (code "client_components_tag()") " never sends them to the client, causing \"Unknown component\" errors.")
(p "The fix: transfer component definitions alongside fragments. Services tell the provider what they already have; the provider sends only what's missing."))
(~docs/section :title "What exists" :id "exists"
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Fragment GET infrastructure works (" (code "shared/infrastructure/fragments.py") ")")
(li (code "X-Fragment-Request") " header protocol for internal service calls")
(li "Content type negotiation for text/html and text/sx responses")
(li "Fragment caching and composition in page rendering"))))
(~docs/section :title "What remains" :id "remains"
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-4")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "POST sexp protocol: ") "Switch from GET to POST with structured sexp body containing " (code ":components") " list of what consumer already has")
(li (strong "Structured response: ") (code "(fragment-response :defs (...) :content (...))") " — provider sends only missing component defs")
(li (strong (code "fragment_response()") " builder: ") "New function in helpers.py that diffs provider's component env against consumer's list")
(li (strong "Register received defs: ") "Consumer parses " (code ":defs") " from response and registers into its " (code "_COMPONENT_ENV"))
(li (strong "Shared blueprint factory: ") (code "create_fragment_blueprint(handlers)") " to deduplicate the identical fragment endpoint pattern across 8 services"))))
(~docs/section :title "Files to modify" :id "files"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Change")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/infrastructure/fragments.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "POST sexp body, parse response, register defs"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/helpers.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "fragment_response() builder"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/infrastructure/fragment_endpoint.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "NEW — shared blueprint factory"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "*/bp/fragments/routes.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "All 8 services: use create_fragment_blueprint"))))))))
;; ---------------------------------------------------------------------------
;; Glue Decoupling
;; ---------------------------------------------------------------------------

337
sx/sx/plans/generative-sx.sx
View File

@@ -1,337 +0,0 @@
;; ---------------------------------------------------------------------------
;; Generative SX — programs that write themselves as they run
;; ---------------------------------------------------------------------------
(defcomp ~plans/generative-sx/plan-generative-sx-content ()
(~docs/page :title "Generative SX"
(p (~tw :tokens "text-stone-500 text-sm italic mb-8")
"In the browser, SX is a program that modifies itself in response to external stimuli. Outside the browser, it becomes a program that writes itself as it runs.")
;; =====================================================================
;; I. The observation
;; =====================================================================
(~docs/section :title "The observation" :id "observation"
(p "The marshes work made something visible. A server response arrives carrying " (code "(reset! (use-store \"price\") 14.99)") " inside a " (code "data-init") " script. The SX evaluator parses this string, evaluates it in its own environment, and mutates its own signal graph. The program accepted new source at runtime and changed itself.")
(p "This isn't metaprogramming. Macros expand at compile time — they transform source before evaluation. This is different: the program is " (em "already running") " when it receives new code, evaluates it, and continues with an extended state. The DOM is just the boundary. The signal graph is just the state. The mechanism is general:")
(ol (~tw :tokens "list-decimal list-inside space-y-2 text-stone-600")
(li "A running SX evaluator with an environment")
(li "New SX source arrives (from any external source)")
(li "The evaluator parses it and evaluates it " (em "in its own environment"))
(li "The environment grows — new definitions, new state, new capabilities")
(li "Evaluation continues with the extended environment"))
(p "In the browser, step 2 is an HTTP response. Step 5 is the next render cycle. But nothing about this mechanism requires a browser, a DOM, or HTTP. The evaluator doesn't know where the source came from. It just evaluates."))
;; =====================================================================
;; II. What already exists
;; =====================================================================
(~docs/section :title "What already exists" :id "exists"
(p "The pieces are already built. They just haven't been connected into the generative pattern.")
(~docs/subsection :title "Homoiconicity"
(p "SX code is SX data. " (code "parse") " takes a string and returns a list. " (code "aser") " takes a list and returns a string. These round-trip perfectly. The program can read its own source as naturally as it reads a config file, because they're the same format.")
(~docs/code :src (highlight ";; Code is data\n(define source \"(+ 1 2)\")\n(define ast (parse source)) ;; → (list '+ 1 2)\n(define result (eval-expr ast env)) ;; → 3\n\n;; Data is code\n(define spec '(define greet (fn (name) (str \"Hello, \" name \"!\"))))\n(eval-expr spec env)\n(greet \"world\") ;; → \"Hello, world!\"" "lisp")))
(~docs/subsection :title "Runtime eval"
(p (code "eval-expr") " is available at runtime, not just boot. " (code "data-init") " scripts already use it. Any SX string can become running code at any point in the program's execution. This is not " (code "eval()") " bolted onto a language that doesn't want it — it's the " (em "primary mechanism") " of the language."))
(~docs/subsection :title "The environment model"
(p (code "env-extend") " creates a child scope. " (code "env-set!") " adds to the current scope. " (code "define") " creates new bindings. New definitions don't replace old ones — they extend the environment. The program grows monotonically. Every previous state is still reachable through scope chains.")
(p "This is the critical property. A generative program doesn't destroy what it was — it " (em "becomes more") ". Each generation includes everything before it plus what it just wrote."))
(~docs/subsection :title "The bootstrapper"
(p "The bootstrapper reads " (code "eval.sx") " — the evaluator's definition of itself — and emits JavaScript or Python that " (em "is") " that evaluator. The spec writes itself into a host language. This is already a generative program, frozen at build time: read source → transform → emit target. Generative SX unfreezes this. The transformation happens " (em "while the program runs") ", not before."))
(~docs/subsection :title "Content-addressed identity"
(p "From the Art DAG: all data identified by SHA3-256 hashes. If a program fragment is identified by its hash, then \"writing yourself\" means producing new hashes. The history is immutable. You can always go back. A generative program isn't a mutating blob — it's a DAG of versioned states.")))
;; =====================================================================
;; III. The generative pattern
;; =====================================================================
(~docs/section :title "The generative pattern" :id "pattern"
(p "A generative SX program starts with a seed and grows by evaluating its own output.")
(~docs/code :src (highlight ";; The core loop\n(define run\n (fn (env source)\n (let ((ast (parse source))\n (result (eval-expr ast env)))\n ;; result might be:\n ;; a value → done, return it\n ;; a string → new SX source, evaluate it (grow)\n ;; a list of defs → new definitions to add to env\n ;; a dict → {source: \"...\", env-patch: {...}} (grow + configure)\n (cond\n (string? result)\n (run env result) ;; evaluate the output\n (and (dict? result) (has-key? result \"source\"))\n (let ((patched (env-merge env (get result \"env-patch\"))))\n (run patched (get result \"source\")))\n :else\n result))))" "lisp"))
(p "The program evaluates source. If the result is more source, it evaluates that too. Each iteration can extend the environment — add new functions, new macros, new primitives. The environment grows. The program becomes capable of things it couldn't do at the start.")
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 my-4")
(p (~tw :tokens "text-violet-900 font-medium") "This is not eval-in-a-loop")
(p (~tw :tokens "text-violet-800 text-sm") "A REPL evaluates user input in a persistent environment. That's interactive, not generative. The generative pattern is different: the program itself decides what to evaluate next. No user in the loop. The output of one evaluation becomes the input to the next. The program writes itself."))
(~docs/subsection :title "Three modes"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Mode")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Input")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Output")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Growth")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-medium text-stone-700") "Analytic")
(td (~tw :tokens "px-3 py-2 text-stone-600") "SX program + data")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Analysis results")
(td (~tw :tokens "px-3 py-2 text-stone-600") "None — pure function"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-medium text-stone-700") "Synthetic")
(td (~tw :tokens "px-3 py-2 text-stone-600") "SX program")
(td (~tw :tokens "px-3 py-2 text-stone-600") "New SX source")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Code generation — new defs emitted"))
(tr
(td (~tw :tokens "px-3 py-2 font-medium text-stone-700") "Generative")
(td (~tw :tokens "px-3 py-2 text-stone-600") "SX seed")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Running program")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Self-extending — output evaluated as input")))))))
;; =====================================================================
;; IV. Concrete manifestations
;; =====================================================================
(~docs/section :title "Concrete manifestations" :id "manifestations"
(~docs/subsection :title "1. The spec that compiles itself"
(p "Currently: " (code "bootstrap_js.py") " (Python) reads " (code "eval.sx") " (SX) and emits " (code "sx-browser.js") " (JavaScript). Three languages. Two of them are hosts, one is the spec.")
(p "Generative version: " (code "eval.sx") " evaluates itself with a code-generation adapter. The evaluator walks its own AST and emits the target language. No Python bootstrapper. No JavaScript template. The spec " (em "is") " the compiler.")
(~docs/code :src (highlight ";; bootstrap.sx — the spec compiles itself\n;;\n;; Load the codegen adapter for the target\n(define emit (load-adapter target)) ;; target = \"js\" | \"py\" | \"go\" | ...\n;;\n;; Read the spec files\n(define spec-source (read-file \"eval.sx\"))\n(define spec-ast (parse spec-source))\n;;\n;; Walk the AST and emit target code\n;; The walker IS the evaluator — eval.sx evaluating eval.sx\n;; with emit instead of execute\n(define target-code\n (eval-expr spec-ast\n (env-extend codegen-env\n ;; Override define, fn, if, etc. to emit instead of execute\n (codegen-special-forms emit))))\n;;\n(write-file (str \"sx-ref.\" (target-extension target)) target-code)" "lisp"))
(p "This is the bootstrapper rewritten as a generative program. The spec reads itself, walks itself, and writes the output language. Adding a new target means writing a new " (code "load-adapter") " — a set of emitters for the SX special forms. The walker doesn't change. The spec doesn't change. Only the output format changes.")
(p "The current bootstrappers (" (code "bootstrap_js.py") ", " (code "bootstrap_py.py") ") would become the first two adapters. Future targets (Go, Rust, WASM) are additional adapters, written in SX and bootstrapped like everything else."))
(~docs/subsection :title "2. The program that discovers its dependencies"
(p (code "deps.sx") " analyzes component dependency graphs. It walks component ASTs, finds " (code "~plans/content-addressed-components/name") " references, computes transitive closures. This is the analytic mode — SX analyzing SX.")
(p "The generative version: a program that discovers it needs a component, searches for its definition (local files, IPFS, a registry), loads it, evaluates it, and continues. The program grows its own capability set at runtime.")
(~docs/code :src (highlight ";; A program that discovers and loads what it needs\n(define render-page\n (fn (page-name)\n (let ((page-def (lookup-component page-name)))\n (when (nil? page-def)\n ;; Component not found locally — fetch from registry\n (let ((source (fetch-component-source page-name)))\n ;; Evaluate the definition — it joins the environment\n (eval-expr (parse source) env)\n ;; Now it exists\n (set! page-def (lookup-component page-name))))\n ;; Render with all dependencies resolved\n (render-to-html (list page-def)))))" "lisp"))
(p "This already happens in the browser. " (code "sx_response") " prepends missing component definitions as a " (code "data-components") " script block. The client evaluates them and they join the environment. The generative version makes this explicit: the program tells you what it needs, you give it source, it evaluates it, it grows."))
(~docs/subsection :title "3. The test suite that writes tests"
(p "Given a function's signature and a set of properties (" (code "prove.sx") " already has the property language), generate test cases that verify the properties. The program reads its own function definitions, generates SX expressions that test them, and evaluates those expressions.")
(~docs/code :src (highlight ";; Given: a function and properties about it\n(define-property string-reverse-involutory\n :forall ((s string?))\n :holds (= (reverse (reverse s)) s))\n\n;; Generate: test cases from the property\n;; The program reads the property, generates test source, evals it\n(define tests (generate-tests string-reverse-involutory))\n;; tests = list of (assert (= (reverse (reverse \"hello\")) \"hello\"))\n;; (assert (= (reverse (reverse \"\")) \"\"))\n;; (assert (= (reverse (reverse \"a\")) \"a\"))\n;; ... (random strings, edge cases)\n(for-each (fn (t) (eval-expr t env)) tests)" "lisp"))
(p "The program analyzed itself (read the property), generated new SX (the test cases), and evaluated it (ran the tests). Three modes — analytic, synthetic, generative — in sequence."))
(~docs/subsection :title "4. The server that extends its own API"
(p "An SX server receives a request it doesn't know how to handle. Instead of returning 404, it examines the request, generates a handler, evaluates it, and handles the request.")
(~docs/code :src (highlight ";; A route handler that generates new route handlers\n(define handle-unknown-route\n (fn (path params)\n ;; Analyze what was requested\n (let ((segments (split path \"/\"))\n (resource (nth segments 1))\n (action (nth segments 2)))\n ;; Check if a schema exists for this resource\n (let ((schema (lookup-schema resource)))\n (when schema\n ;; Generate a CRUD handler from the schema\n (let ((handler-source (generate-crud-handler resource action schema)))\n ;; Evaluate it — the handler now exists\n (eval-expr (parse handler-source) env)\n ;; Route future requests to the generated handler\n (register-route path (env-get env (str resource \"-\" action)))\n ;; Handle this request with the new handler\n ((env-get env (str resource \"-\" action)) params)))))))" "lisp"))
(p "This is not code generation in the Rails scaffolding sense — those generate files you then edit. This generates running code at runtime. The handler didn't exist. Now it does. The server grew."))
(~docs/subsection :title "5. The macro system that learns idioms"
(p "A generative macro system that detects repeated patterns in code and synthesizes macros to capture them. The program watches itself being written and abstracts its own patterns.")
(~docs/code :src (highlight ";; The program notices this pattern appearing repeatedly:\n;; (div :class \"card\" (h2 title) (p body) children...)\n;;\n;; It generates:\n(defmacro ~card (title body &rest children)\n (div :class \"card\"\n (h2 ,title)\n (p ,body)\n ,@children))\n;;\n;; And rewrites its own source to use the new macro.\n;; This is an SX program that:\n;; 1. Analyzed its own AST (found repeated subtrees)\n;; 2. Synthesized a macro (extracted the pattern)\n;; 3. Evaluated the macro definition (extended env)\n;; 4. Rewrote its own source (used the macro)\n;; Four generative steps." "lisp"))
(p "The connection to the Art DAG: each version of the source is content-addressed. The original (before macros) and the refactored (after macros) are both immutable nodes. The generative step is an edge in the DAG. You can always inspect what the program was before it rewrote itself.")))
;; =====================================================================
;; V. The seed
;; =====================================================================
(~docs/section :title "The seed" :id "seed"
(p "A generative SX program starts with a seed. The seed must contain enough to bootstrap the generative loop: a parser, an evaluator, and the " (code "run") " function. Everything else is grown.")
(~docs/code :src (highlight ";; seed.sx — the minimal generative program\n;;\n;; This file contains:\n;; - The SX parser (parse)\n;; - The SX evaluator (eval-expr)\n;; - The generative loop (run)\n;; - A source acquisition function (next-source)\n;;\n;; Everything else — primitives, rendering, networking, persistence —\n;; is loaded by the program as it discovers it needs them.\n\n(define run\n (fn (env)\n (let ((source (next-source env)))\n (when source\n (let ((result (eval-expr (parse source) env)))\n (run env))))))\n\n;; Start with a bare environment\n(run (env-extend (dict\n \"parse\" parse\n \"eval-expr\" eval-expr\n \"next-source\" initial-source-fn)))" "lisp"))
(p "The seed is a quine that doesn't just reproduce itself — it " (em "extends") " itself. Each call to " (code "next-source") " returns new SX that the seed evaluates in its own environment. The environment grows. The seed's capabilities grow. But the seed itself never changes — it's the fixed point of the generative process.")
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4 my-4")
(p (~tw :tokens "text-stone-700 font-medium mb-2") "The minimal seed is the spec")
(p (~tw :tokens "text-stone-600 text-sm") (code "eval.sx") " + " (code "parser.sx") " + " (code "primitives.sx") " = a complete SX evaluator defined in SX. This is already the seed. The bootstrappers compile it to JavaScript and Python. A generative runtime compiles it to " (em "itself") " — the seed evaluates the seed and obtains a running evaluator that can evaluate anything, including more of itself.")))
;; =====================================================================
;; VI. Growth constraints
;; =====================================================================
(~docs/section :title "Growth constraints" :id "constraints"
(p "Unconstrained self-modification is dangerous. A program that can rewrite any part of itself can rewrite its own safety checks. Generative SX needs growth constraints — rules about what the program can and cannot do to itself.")
(~docs/subsection :title "The boundary"
(p "The boundary system (" (code "boundary.sx") ") already enforces this. Pure primitives can't do IO. IO primitives can't escape their declared capabilities. Components are classified as pure or IO-dependent. The boundary is checked at registration time — " (code "SX_BOUNDARY_STRICT=1") " means violations crash at startup.")
(p "For generative programs, the boundary extends: generated code is subject to the same constraints as hand-written code. A generative program can't synthesize an IO primitive — it can only compose existing ones. It can't bypass the boundary by generating code that accesses raw platform APIs. The sandbox applies to generated code exactly as it applies to original code.")
(p "This is the key safety property: " (strong "generative SX is sandboxed generative SX") ". The generated code runs in the same evaluator with the same restrictions. No escape hatches."))
(~docs/subsection :title "Content addressing as audit trail"
(p "Every piece of generated code is content-addressed. The SHA3-256 hash of the generated source is its identity. You can trace any piece of running code back to the generation step that produced it, the input that triggered that step, and the state of the environment at that point.")
(p "This makes generative programs auditable. \"Where did this function come from?\" has a definite answer: it was generated by " (em "this") " code, from " (em "this") " input, at " (em "this") " point in the generative sequence. The DAG of content hashes is the program's autobiography."))
(~docs/subsection :title "Monotonic growth"
(p "The environment model is append-only. " (code "define") " creates new bindings; it doesn't destroy old ones (inner scopes shadow, but the outer binding persists). " (code "env-extend") " creates a child — the parent is immutable.")
(p "A generative program can extend its environment but cannot shrink it. It can add new functions but cannot delete existing ones. It can shadow a function with a new definition but cannot destroy the original. This means the program's history is preserved in its scope chain — you can always inspect what it was before any given generation step.")
(p "Destructive operations (" (code "set!") ") are confined to mutable cells explicitly created for that purpose. The generative loop itself operates on immutable environments extended with each step.")))
;; =====================================================================
;; VII. Host properties
;; =====================================================================
(~docs/section :title "Host properties" :id "host-properties"
(p "A generative SX program runs on a host — JavaScript, Python, Go, bare metal. The host must provide specific properties or the generative loop breaks. These aren't preferences. They're " (em "requirements") ". A host that violates any of them can't run generative SX correctly.")
(~docs/subsection :title "Lossless parse/serialize round-trip"
(p "The host must implement " (code "parse") " and " (code "aser") " such that " (code "(aser (parse source))") " produces semantically equivalent source. Generated code passes through " (code "parse → transform → serialize → parse") " cycles. If the round-trip is lossy — if whitespace, keyword order, or nested structure is corrupted — the generative loop silently degrades. After enough iterations, the program isn't what it thinks it is.")
(p "This is homoiconicity at the implementation level, not just the language level. The host's data structures must faithfully represent the full AST, and the serializer must faithfully reproduce it."))
(~docs/subsection :title "Runtime eval with first-class environments"
(p (code "eval-in") " requires evaluating arbitrary expressions in arbitrary environments at runtime. The host must support:")
(ul (~tw :tokens "list-disc pl-5 space-y-1 text-stone-600")
(li "Creating new environments (" (code "env-extend") ")")
(li "Adding bindings to existing environments (" (code "env-set!") ")")
(li "Inspecting environment contents (" (code "env-snapshot") ")")
(li "Passing environments as values — storing them in variables, returning them from functions"))
(p "Environments aren't implementation detail in a generative program. They're the " (em "state") ". The running environment at generation step N is the complete description of what the program knows. The host must treat environments as first-class values, not hidden interpreter internals."))
(~docs/subsection :title "Monotonic environment growth"
(p "A generative program that can " (code "undefine") " things becomes unpredictable. If generation step N+1 removes a function that step N defined, step N+2 might reference the missing function and fail — or worse, silently bind to a different function in an outer scope.")
(p "The host must enforce that environments grow monotonically. New bindings append. Existing bindings in a given scope are immutable once set (or explicitly versioned). " (code "env-extend") " creates children; it never mutates the parent. This makes the generative loop convergent — each step strictly increases the program's capabilities, never decreases them."))
(~docs/subsection :title "Content-addressed storage"
(p "Every generated fragment gets a SHA3-256 identity. The host needs native or near-native hashing and a content-addressed store — an in-memory dict at minimum, IPFS at scale. This provides the audit trail: you can always answer \"where did this code come from?\" by walking the hash chain back to the generation step that produced it.")
(p "Without content addressing, generative programs are opaque. You can't diff two versions of a generated function. You can't roll back to a previous generation. You can't verify that two nodes in a seed network generated the same code from the same input. Content addressing makes the generative process " (em "inspectable") "."))
(~docs/subsection :title "Boundary enforcement on generated code"
(p "Generated code must pass through the same boundary validation as hand-written code. If " (code "write-file") " is a Tier 2 IO primitive, a generated expression can't call it unless the evaluation context permits Tier 2.")
(p "The host must enforce this " (em "at eval time") ", not just at definition time — because generated code didn't exist at definition time. Every call to " (code "eval-in") " must check the boundary. Every primitive invoked by generated code must verify its tier. There is no \"trusted generated code\" — all code is untrusted until the boundary clears it."))
(~docs/subsection :title "Correct quotation and splicing"
(p "Quasiquote (" (code "`") "), unquote (" (code ",") "), and unquote-splicing (" (code ",@") ") must work correctly for programmatic code construction. The host needs these as first-class operations, not string concatenation.")
(p "A generative program builds code by template:")
(~docs/code :src (highlight ";; The generative program builds new definitions by template\n(define gen-handler\n (fn (name params body)\n `(define ,name\n (fn ,params\n ,@body))))\n\n;; gen-handler produces an AST, not a string\n;; The AST can be inspected, transformed, hashed, then evaluated\n(eval-in (gen-handler 'greet '(name) '((str \"Hello, \" name))) env)" "lisp"))
(p "String concatenation would work — " (code "(str \"(define \" name \" ...)\")") " — but it's fragile, unstructured, and can't be inspected before evaluation. Quasiquote produces an AST. The generative program works with " (em "structure") ", not text."))
(~docs/subsection :title "Tail-call optimization"
(p "The generative loop is inherently recursive: eval produces source, which is eval'd, which may produce more source. Without TCO, the loop blows the stack after enough iterations. The trampoline/thunk mechanism in the spec handles this, but the host must implement it efficiently.")
(p "This is not optional. A generative program that can only recurse a few thousand times before crashing is not a generative program — it's a demo. The self-compiling spec (Phase 1) alone requires walking every node of " (code "eval.sx") ", which is thousands of recursive calls."))
(~docs/subsection :title "Deterministic evaluation order"
(p "If two hosts evaluate the same generative program and get different results because of evaluation order, the content hashes diverge. The programs are no longer equivalent. They can't federate (Phase 5), can't verify each other's output, can't share generated code.")
(p "The host must guarantee: dict iteration order is deterministic (insertion order). Argument evaluation is left-to-right. Effect sequencing follows definition order. No observable nondeterminism in pure evaluation. This is what makes generative programs " (em "reproducible") " — same seed, same input, same output, regardless of host."))
(~docs/subsection :title "Serializable state"
(p "For Phase 4 (self-extending server) and Phase 5 (seed network), a generative program needs to pause, serialize its state, and resume elsewhere. The host needs the ability to serialize an environment + pending expression as data.")
(p "This doesn't require first-class continuations (though those work). It requires that everything in the environment is serializable: functions serialize as their source AST, signals as their current value, environments as nested dicts. The " (code "env-snapshot") " primitive provides this. The host must ensure nothing in the environment is opaque — no host-language closures that can't be serialized, no hidden mutable state that isn't captured by the snapshot."))
(~docs/subsection :title "IO isolation"
(p "The generative primitives (" (code "read-file") ", " (code "write-file") ", " (code "list-files") ") are the " (em "only") " way generated code touches the outside world. The host must be able to intercept, log, and deny all IO. There is no escape hatch through FFI or native calls.")
(p "This is what makes generative programs auditable. If the host allows generated code to call raw " (code "fs.writeFileSync") " or " (code "os.system") ", the boundary is meaningless. The host must virtualize all IO through the declared primitives. Generated code that tries to escape the sandbox hits the boundary, not the OS."))
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 my-4")
(p (~tw :tokens "text-violet-900 font-medium") "The acid test")
(p (~tw :tokens "text-violet-800 text-sm") "SX already has properties 1–7 in the spec. They fall out of the language design. Properties 8–10 (deterministic evaluation, serializable state, IO isolation) are constraints on the " (em "host implementation") ", not the language. A host that violates 1–7 can't run the spec correctly. A host that violates 8–10 can run it but can't be trusted in a generative context.")
(p (~tw :tokens "text-violet-800 text-sm mt-2") "Phase 1 — the self-compiling spec — tests properties 1–7. If a host can compile the spec from the spec, it necessarily has them. The remaining three are operational guarantees verified by the seed network (Phase 5), where multiple hosts must agree on the same generative output.")))
;; =====================================================================
;; VIII. Environment migration
;; =====================================================================
(~docs/section :title "Environment migration" :id "env-migration"
(p "SX endpoints tunnel into different execution environments with different primitive sets. A browser has " (code "render-to-dom") " but no " (code "gpu-exec") ". A render node has " (code "gpu-exec") " but no " (code "fetch-fragment") ". An ingest server has " (code "open-feed") " but neither. The boundary isn't just a restriction — it's a " (em "capability declaration") ". It tells you what an environment " (em "can do") ".")
(~docs/subsection :title "Boundary as capability declaration"
(p "Every environment declares its boundary: the set of primitives it provides. SX source is portable across any environment that satisfies its primitive requirements. If a program only uses pure primitives (Tier 0), it runs anywhere. If it calls " (code "gpu-exec") ", it needs an environment that provides " (code "gpu-exec") ". The boundary is a type signature on the environment itself — not \"what can this code do\" but \"what must the host provide.\"")
(p "This inverts the usual framing. The boundary doesn't " (em "forbid") " — it " (em "requires") ". A generated program that calls " (code "encode-stream") " is declaring a hardware dependency. The boundary system doesn't block the call — it routes the program to a host that can satisfy it."))
(~docs/subsection :title "with-boundary as migration point"
(p "Execution migrates to where the primitives are. When the evaluator hits a " (code "with-boundary") " block requiring primitives the current host doesn't have, it serializes state (" (code "env-snapshot") "), ships the pending expression plus environment to a host that has them, and execution continues there. The block is the unit of migration, not individual primitive calls.")
(~docs/code :src (highlight "(with-boundary (media-processing encoding)\n (let ((frames (gpu-exec recipe cached-layers)))\n (encode-stream frames :codec \"h264\"\n :on-input-needed (fn (slot)\n (with-boundary (live-ingest)\n (open-feed :protocol \"webrtc\" :slot slot))))))" "lisp"))
(p "This program starts wherever it starts. When it hits " (code "(with-boundary (media-processing encoding) ...)") ", the evaluator checks: does the current host provide " (code "gpu-exec") " and " (code "encode-stream") "? If yes, evaluate in place. If no, snapshot the environment, serialize the pending expression, and dispatch to a host that does. Inside the encoding block, " (code ":on-input-needed") " triggers a nested migration — the " (code "(with-boundary (live-ingest) ...)") " block dispatches to an ingest server that provides " (code "open-feed") ".")
(p "The program doesn't know where it runs. It declares what it needs. The runtime figures out " (em "where") "."))
(~docs/subsection :title "Declaration, not discovery"
(p "Boundary requirements are declared at scope boundaries, not discovered at call time. This is the critical constraint. A generative program that synthesizes a " (code "with-boundary") " block is declaring — at generation time — what the block will need. The declaration is inspectable before execution. You can analyze a generated program's boundary requirements without running it.")
(p "This gives constraint checking on generated code. A generative loop that produces a " (code "with-boundary") " block must produce a valid boundary declaration. If the generated block calls " (code "gpu-exec") " but doesn't declare " (code "media-processing") ", the boundary checker rejects it — at generation time, not at runtime. The program must say what it needs before it needs it."))
(~docs/subsection :title "Nested migration"
(p "Nested " (code "with-boundary") " blocks are nested migrations. The program walks the capability graph, carrying its state, accumulating content-addressed history. Each migration is an edge in the DAG — the source environment, the target environment, the serialized state, the pending expression. All content-addressed. All auditable.")
(p "A three-level nesting — browser to render node to ingest server — is three migrations. The browser evaluates the outer expression, hits a " (code "with-boundary") " requiring GPU, migrates to the render node. The render node evaluates until it hits a " (code "with-boundary") " requiring live ingest, migrates to the ingest server. Each migration carries the accumulated environment. Each return ships results back up the chain.")
(p "The nesting depth is bounded by the capability graph. If there are four distinct environment types, the maximum nesting is four. In practice, most programs need one or two migrations. The deep nesting is there for generative programs that discover capabilities as they run."))
(~docs/subsection :title "Environment chaining"
(p "Split execution — cached layers on one host, GPU rendering on another, encoding on a third — is just environment chaining. The evaluator runs in one environment until it hits a primitive requiring a different one. The primitive " (em "is") " the dispatch.")
(p "This collapses the distinction between \"local function call\" and \"remote service invocation.\" From the SX program's perspective, " (code "gpu-exec") " is a primitive. Whether it runs on the local GPU or a remote render farm is an environment configuration detail, not a language-level concern. The " (code "with-boundary") " block declares the requirement. The runtime satisfies it. The program doesn't care how.")
(p "Environment chaining also explains the Art DAG's three-phase execution pattern (analyze, plan, execute). Each phase runs in a different environment with different primitives. The analyze phase needs " (code "content-hash") " and " (code "list-files") ". The plan phase needs " (code "env-snapshot") " and scheduling primitives. The execute phase needs " (code "gpu-exec") " and storage primitives. Three " (code "with-boundary") " blocks. Three environments. One program.")))
;; =====================================================================
;; IX. Implementation phases
;; =====================================================================
(~docs/section :title "Implementation phases" :id "phases"
(~docs/subsection :title "Phase 0: Generative primitives"
(p "Add the minimal set of primitives needed for a generative loop. These are IO primitives — they cross the boundary.")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Primitive")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Signature")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Description")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "read-file")
(td (~tw :tokens "px-3 py-2 text-stone-600") "(path) → string")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Read file contents as string"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "write-file")
(td (~tw :tokens "px-3 py-2 text-stone-600") "(path content) → nil")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Write string to file"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "list-files")
(td (~tw :tokens "px-3 py-2 text-stone-600") "(path pattern) → list")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Glob-match files in directory"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "eval-in")
(td (~tw :tokens "px-3 py-2 text-stone-600") "(source env) → any")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Parse and evaluate SX source in given environment"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "env-snapshot")
(td (~tw :tokens "px-3 py-2 text-stone-600") "(env) → dict")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Serialize environment to inspectable dict"))
(tr
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "content-hash")
(td (~tw :tokens "px-3 py-2 text-stone-600") "(source) → string")
(td (~tw :tokens "px-3 py-2 text-stone-600") "SHA3-256 hash of source string")))))
(p "These are the building blocks. The generative loop composes them. The primitives themselves are minimal — no networking, no databases, no UI. Just: read, write, evaluate, inspect, hash."))
(~docs/subsection :title "Phase 1: Self-compiling spec"
(p "Rewrite " (code "bootstrap_js.py") " as " (code "bootstrap.sx") ". The bootstrapper becomes an SX program that reads the spec files and emits target code.")
(ol (~tw :tokens "list-decimal list-inside space-y-2 text-stone-600")
(li "Write " (code "codegen-js.sx") " — JavaScript code generation adapter (emit JS from SX AST)")
(li "Write " (code "codegen-py.sx") " — Python code generation adapter")
(li "Write " (code "bootstrap.sx") " — the generative loop that loads a spec, loads an adapter, and emits")
(li "Verify: " (code "(bootstrap \"js\")") " produces identical output to " (code "bootstrap_js.py"))
(li "Retire the Python bootstrappers"))
(p "This is the first real generative program: SX reading SX and writing JavaScript. The same program, with a different adapter, writes Python. Or Go. Or WASM. The spec doesn't change. Only the adapter changes."))
(~docs/subsection :title "Phase 2: Generative deps"
(p "Rewrite " (code "deps.sx") " as a generative program. Instead of computing a static dependency graph, it runs continuously: watch for new component definitions, update the graph, re-emit optimized bundles.")
(p "This is the deps analyzer turned inside out. Instead of \"analyze all components, output a graph,\" it's \"when a new component appears, update the running graph.\" The dependency analysis is an ongoing computation, not a one-shot pass."))
(~docs/subsection :title "Phase 3: Generative testing"
(p "Connect " (code "prove.sx") " to the generative loop. When a new function is defined, automatically generate property tests, run them, report failures. When a function changes, regenerate and rerun only the affected tests.")
(p "The test suite is not a separate artifact — it's a side effect of the generative process. Every function that enters the environment is tested. The tests are generated from properties, not hand-written. The program verifies itself as it grows."))
(~docs/subsection :title "Phase 4: The self-extending server"
(p "An SX server with a generative core. New routes, handlers, and middleware can be added at runtime by evaluating SX source. The server's API surface is a living environment that grows with use.")
(p "Not a scripting layer bolted onto a framework — the server " (em "is") " a generative SX program. Its routes are SX definitions. Its middleware is SX functions. Adding a new endpoint means evaluating a new " (code "defhandler") " in the running environment."))
(~docs/subsection :title "Phase 5: The seed network"
(p "Multiple generative SX programs exchanging source. Each node runs a seed. When node A discovers a capability it lacks, it requests the source from node B. Node B's generated code is content-addressed — A can verify it, evaluate it, and grow.")
(p "This is SX-Activity applied to generative programs. The wire format is SX. The content is SX. The evaluation is SX. The programs share source, not data. They grow together.")))
;; =====================================================================
;; X. The strange loop
;; =====================================================================
(~docs/section :title "The strange loop" :id "strange-loop"
(p "Hofstadter's strange loop: a hierarchy of levels where the top level reaches back down and affects the bottom level. In a generative SX program:")
(ul (~tw :tokens "list-disc pl-5 space-y-2 text-stone-600")
(li "The bottom level is the evaluator — it evaluates expressions")
(li "The middle level is the program — expressions that produce values")
(li "The top level is the generator — values that are new expressions")
(li "The loop closes: new expressions are evaluated by the evaluator"))
(p "The program that writes itself is not a metaphor. It's a literal description of what happens when an SX expression evaluates to an SX string that is then parsed and evaluated. The output of evaluation becomes the input to evaluation. The program is both the writer and the written.")
(p "The browser version constrains this to DOM mutations. The server version constrains it to request handling. The unconstrained version — a bare seed with " (code "next-source") " — is a strange loop in its purest form: an evaluator that evaluates what it generates, generating what it evaluates.")
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4 mt-6")
(p (~tw :tokens "text-stone-700 font-medium mb-2") "The practical consequence")
(p (~tw :tokens "text-stone-600 text-sm") "An SX development environment where the tools are written in SX, running inside SX, modifying SX. The editor understands the code because it " (em "is") " the code. The debugger inspects the environment because it " (em "shares") " the environment. The compiler reads the spec because the spec is in the same format as everything else. There is no impedance mismatch between any layer because there is only one layer.")))
))

49
sx/sx/plans/glue-decoupling.sx
View File

@@ -1,49 +0,0 @@
;; ---------------------------------------------------------------------------
;; Glue Decoupling
;; ---------------------------------------------------------------------------
(defcomp ~plans/glue-decoupling/plan-glue-decoupling-content ()
(~docs/page :title "Cross-App Decoupling via Glue Services"
(~docs/section :title "Context" :id "context"
(p "All cross-domain FK constraints have been dropped (with pragmatic exceptions for OrderItem.product_id and CartItem). Cross-domain writes go through internal HTTP and activity bus. However, " (strong "25+ cross-app model imports remain") " — apps still import from each other's models/ directories. This means every app needs every other app's code on disk to start.")
(p "The goal: eliminate all cross-app model imports. Every app only imports from its own models/, from shared/, and from a new glue/ service layer."))
(~docs/section :title "Current state" :id "current"
(p "Apps are partially decoupled via HTTP interfaces (fetch_data, call_action, send_internal_activity) and DTOs. The Cart microservice split (relations, likes, orders) is complete. But direct model imports persist in:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Cart") " — 9 files importing from market, events, blog")
(li (strong "Blog") " — 8 files importing from cart, events, market")
(li (strong "Events") " — 5 files importing from blog, market, cart")
(li (strong "Market") " — 1 file importing from blog")))
(~docs/section :title "What remains" :id "remains"
(div (~tw :tokens "space-y-3")
(div (~tw :tokens "rounded border border-stone-200 p-3")
(h4 (~tw :tokens "font-semibold text-stone-700") "1. glue/services/pages.py")
(p (~tw :tokens "text-sm text-stone-600") "Dict-based Post access for non-blog apps: get_page_by_slug, get_page_by_id, get_pages_by_ids, page_exists, search_posts."))
(div (~tw :tokens "rounded border border-stone-200 p-3")
(h4 (~tw :tokens "font-semibold text-stone-700") "2. glue/services/page_config.py")
(p (~tw :tokens "text-sm text-stone-600") "PageConfig CRUD: get_page_config, get_or_create_page_config, get_page_configs_by_ids."))
(div (~tw :tokens "rounded border border-stone-200 p-3")
(h4 (~tw :tokens "font-semibold text-stone-700") "3. glue/services/calendars.py")
(p (~tw :tokens "text-sm text-stone-600") "Calendar queries + entry associations (from blog): get_calendars_for_page, toggle_entry_association, get_associated_entries."))
(div (~tw :tokens "rounded border border-stone-200 p-3")
(h4 (~tw :tokens "font-semibold text-stone-700") "4. glue/services/marketplaces.py")
(p (~tw :tokens "text-sm text-stone-600") "MarketPlace CRUD (from blog+events): get_marketplaces_for_page, create_marketplace, soft_delete_marketplace."))
(div (~tw :tokens "rounded border border-stone-200 p-3")
(h4 (~tw :tokens "font-semibold text-stone-700") "5. glue/services/cart_items.py")
(p (~tw :tokens "text-sm text-stone-600") "CartItem/CalendarEntry queries for cart: get_cart_items, find_or_create_cart_item, clear_cart_for_order."))
(div (~tw :tokens "rounded border border-stone-200 p-3")
(h4 (~tw :tokens "font-semibold text-stone-700") "6. glue/services/products.py")
(p (~tw :tokens "text-sm text-stone-600") "Minimal Product access for cart orders: get_product."))
(div (~tw :tokens "rounded border border-stone-200 p-3")
(h4 (~tw :tokens "font-semibold text-stone-700") "7. Model registration + cleanup")
(p (~tw :tokens "text-sm text-stone-600") "register_models() in glue/setup.py, update all app.py files, delete moved service files."))))
(~docs/section :title "Docker consideration" :id "docker"
(p (~tw :tokens "text-stone-600") "For glue services to work in Docker (single app per container), model files from other apps must be importable. Recommended: try/except at import time — glue services that can't import a model raise ImportError at call time, which only happens if called from the wrong app."))))
;; ---------------------------------------------------------------------------
;; Social Sharing
;; ---------------------------------------------------------------------------

28
sx/sx/plans/index.sx
View File

@@ -1,28 +0,0 @@
(defcomp
~plans/index/plans-index-content
()
(~docs/page
:title "Plans"
(div
(~tw :tokens "space-y-4")
(p
(~tw :tokens "text-lg text-stone-600 mb-4")
"Architecture roadmaps and implementation plans for SX.")
(div
(~tw :tokens "space-y-3")
(map
(fn
(item)
(a
:href (get item "href")
:sx-get (get item "href")
:sx-target "#sx-content"
:sx-select "#sx-content"
:sx-swap "outerHTML"
:sx-push-url "true"
(~tw :tokens "block rounded border border-stone-200 p-4 hover:border-violet-300 hover:bg-violet-50 transition-colors")
(div (~tw :tokens "font-semibold text-stone-800") (get item "label"))
(when
(get item "summary")
(p (~tw :tokens "text-sm text-stone-500 mt-1") (get item "summary")))))
plans-nav-items)))))

333
sx/sx/plans/isolated-evaluator.sx
View File

@@ -1,333 +0,0 @@
;; ---------------------------------------------------------------------------
;; Isolated Evaluator — Shared platform layer, isolated JS, Rust WASM
;; ---------------------------------------------------------------------------
(defcomp ~plans/isolated-evaluator/plan-isolated-evaluator-content ()
(~docs/page :title "Isolated Evaluator"
(~docs/section :title "Context" :id "context"
(p "The SX spec is already split into three layers:")
(ul (~tw :tokens "list-disc list-inside space-y-1 mt-2")
(li (code "spec/") " \u2014 Core language (19 files): evaluator, parser, primitives, CEK, types, continuations. Host-independent.")
(li (code "web/") " \u2014 Web framework (20 files): signals, adapters, engine, orchestration, boot, router, deps. Built on core spec.")
(li (code "sx/") " \u2014 Application (sx-docs website). Built on web framework."))
(p "Bootstrappers search " (code "spec/ \u2192 web/ \u2192 shared/sx/ref/") " for " (code ".sx") " files. The separation is clean.")
(p "This plan takes the next step: " (strong "isolate the evaluator from the real world") ". The JS evaluator should run in the same sandbox as Rust/WASM \u2014 unable to touch DOM, fetch, timers, or storage directly. Both evaluators call into a shared " (code "sx-platform.js") " for all browser access.")
(p "This also involves sorting out the JavaScript: eliminating hand-coded JS that duplicates specced " (code ".sx") " logic, and moving web framework " (code ".sx") " from compiled-into-evaluator to runtime-evaluated."))
(~docs/section :title "Existing Architecture" :id "existing"
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Three-layer spec split (DONE)")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Layer")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Directory")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Files")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Content")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Core spec")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "spec/"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "19")
(td (~tw :tokens "px-3 py-2 text-stone-600") "eval, parser, primitives, render, types, CEK, continuations, boundary-core"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Web framework")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "web/"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "20")
(td (~tw :tokens "px-3 py-2 text-stone-600") "adapters, signals, engine, orchestration, boot, router, deps, forms, boundary-web"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "Application")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "sx/"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "\u2014")
(td (~tw :tokens "px-3 py-2 text-stone-600") "sx-docs website, page components, content")))))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Rust/WASM evaluator (DONE)")
(p (code "sx-rust/") " has a working parser + evaluator + render-to-html in WASM: 9,823 lines generated Rust, 75 real primitives, 154 stubs, 92 tests passing. Currently pure computation \u2014 no DOM interaction.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "What needs to change")
(p "Currently " (strong "everything") " (core + web framework) gets compiled into one monolithic " (code "sx-browser.js") ". The web framework " (code ".sx") " files (signals, engine, orchestration, boot, etc.) are baked into the evaluator output by the bootstrapper. They should instead be " (strong "evaluated at runtime") " by the core evaluator, like any other " (code ".sx") " code.")
(p "The JavaScript platform primitives (DOM, fetch, timers, storage) are also inlined into the bootstrapped output. They need to be extracted into a standalone " (code "sx-platform.js") " module that both JS and WASM evaluators share."))
;; -----------------------------------------------------------------------
;; What's core vs web (bootstrapped vs runtime)
;; -----------------------------------------------------------------------
(~docs/section :title "Bootstrapped vs Runtime-Evaluated" :id "bootstrap-vs-runtime"
(p "The key question: what MUST be compiled into the evaluator vs what can be loaded as " (code ".sx") " at runtime?")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Must be bootstrapped (core spec)")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Dir")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Why")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "eval.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "spec/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "IS the language \u2014 can't evaluate without it"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "parser.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "spec/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Can't read .sx source without a parser"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "primitives.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "spec/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "80+ built-in pure functions \u2014 must be native"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "render.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "spec/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "HTML_TAGS registry, parse-element-args"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "adapter-html.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "render-to-html \u2014 co-recursive with eval-expr"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "adapter-sx.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "aser (wire format) \u2014 co-recursive with eval-expr"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "adapter-dom.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "render-to-dom \u2014 co-recursive with eval-expr")))))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Runtime-evaluated (web framework)")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Dir")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Why it can be runtime")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "signals.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Pure computation \u2014 dicts with markers, no new types"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "engine.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Pure logic \u2014 trigger parsing, swap specs, morph"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "orchestration.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Event binding + fetch \u2014 calls platform primitives"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "boot.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Page lifecycle \u2014 calls platform primitives"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "router.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "URL pattern matching \u2014 pure computation"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "deps.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Component dependency analysis \u2014 pure AST walking"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "page-helpers.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Data transformation helpers"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "forms.sx"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "web/")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Server-only definition forms")))))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "adapter-dom.sx reactive split")
(p (code "adapter-dom.sx") " contains reactive-aware code (" (code "reactive-text") ", " (code "reactive-attr") ", " (code "render-dom-island") ", " (code "render-dom-lake") ") interleaved with core DOM rendering. These call " (code "signal?") " and " (code "deref") " from " (code "signals.sx") " via environment lookup \u2014 no compile-time dependency. Option: split reactive DOM functions into " (code "adapter-dom-reactive.sx") " (web/ layer), keeping base " (code "adapter-dom.sx") " purely about elements/text/fragments/components.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Hand-coded JS to clean up")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (code "CONTINUATIONS_JS") " in " (code "platform_js.py") " \u2014 hand-coded shift/reset. Should use specced " (code "continuations.sx") " or be eliminated if continuations are application-level.")
(li (code "ASYNC_IO_JS") " in " (code "platform_js.py") " \u2014 hand-coded async rendering dispatch. Already replaced by " (code "adapter-async.sx") " for Python. JS version should also be bootstrapped or eliminated.")
(li "Various wrapper functions in " (code "PLATFORM_BOOT_JS") " that duplicate logic from " (code "boot.sx") ".")))
;; -----------------------------------------------------------------------
;; Phase 1: Extract sx-platform.js
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 1: Extract sx-platform.js" :id "phase-1"
(p (strong "Goal:") " All real-world-touching JavaScript lives in one standalone module. The evaluator never directly accesses " (code "document") ", " (code "window") ", " (code "fetch") ", " (code "localStorage") ", " (code "history") ", etc.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Architecture")
(~docs/code :src (highlight " \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 sx-platform.js \u2502 \u2190 DOM, fetch, timers, storage\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 sx-evaluator.js \u2502 \u2502 sx-wasm-shim.js \u2502\n \u2502 (isolated JS) \u2502 \u2502 (WASM instance \u2502\n \u2502 \u2502 \u2502 + handle table) \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518" "text"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "What moves into sx-platform.js")
(p "Extracted from " (code "platform_js.py") " string constants:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Category")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Source")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "~Functions")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "DOM primitives")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "PLATFORM_DOM_JS"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "~50 (createElement, setAttribute, appendChild...)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Engine platform")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "PLATFORM_ENGINE_PURE_JS"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "~6 (locationHref, pushState, nowMs...)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Orchestration platform")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "PLATFORM_ORCHESTRATION_JS"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "~80 (fetch, abort, timers, SSE, scroll, media...)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Boot platform")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "PLATFORM_BOOT_JS"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "~20 (mount target, localStorage, cookies, logging...)"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "Parser helpers")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "PLATFORM_PARSER_JS"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "~4 (isIdentStart, parseNumber...)")))))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Isolation rule")
(p "After extraction, searching " (code "sx-evaluator.js") " for " (code "document") ", " (code "window") ", " (code "fetch") ", " (code "localStorage") ", " (code "history") ", " (code "setTimeout") ", " (code "console") " should find " (strong "zero") " direct references.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "The callSxFunction bridge")
(p "Platform code (event listeners, timers) needs to invoke SX lambdas. The evaluator provides a single " (code "callSxFunction(fn, args) \u2192 result") " bridge to the platform at registration time. This is the ONE evaluator-to-platform callback.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Implementation")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "Modify " (code "platform_js.py") " to emit platform functions as a separate output section")
(li "Create " (code "sx-platform.js") " as an IIFE that sets " (code "globalThis.SxPlatform = {...}"))
(li "The evaluator IIFE reads " (code "globalThis.SxPlatform") " at init, registers each function as a PRIMITIVE")
(li "Clean up " (code "CONTINUATIONS_JS") " and " (code "ASYNC_IO_JS") " \u2014 eliminate or bootstrap")
(li "Test that existing pages work identically")))
;; -----------------------------------------------------------------------
;; Phase 2: Isolate the JS Evaluator
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 2: Isolate the JS Evaluator" :id "phase-2"
(p (strong "Goal:") " " (code "sx-evaluator.js") " contains ONLY core spec + render adapters. Web framework " (code ".sx") " is evaluated at runtime.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Core-only build")
(p "The bootstrapper already supports selecting which modules to compile. A core-only build:")
(~docs/code :src (highlight "# In run_js_sx.py \u2014 core-only build\ncompile_ref_to_js(\n adapters=[\"parser\", \"html\", \"sx\", \"dom\"], # core spec + adapters\n modules=None, # no signals, engine, orchestration, boot\n extensions=None, # no continuations\n spec_modules=None # no deps, router, cek, frames, page-helpers\n)" "python"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Web framework loading")
(p "Web framework " (code ".sx") " files ship as " (code "<script type=\"text/sx-lib\">") " blocks. The platform boot shim evaluates them before component scripts:")
(~docs/code :src (highlight "<script src=\"/static/scripts/sx-platform.js\"></script>\n<script src=\"/static/scripts/sx-evaluator.js\"></script>\n<script type=\"text/sx-lib\">\n ;; concatenated web/ framework: signals, deps, router,\n ;; engine, orchestration, boot\n</script>\n<script type=\"text/sx\" data-components>\n ;; page component definitions\n</script>" "html"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Boot chicken-and-egg")
(p (code "boot.sx") " orchestrates the boot sequence but is itself web framework code. Solution: thin native boot shim (~30 lines) in " (code "sx-platform.js") ":")
(~docs/code :src (highlight "SxPlatform.boot = function(evaluator) {\n // 1. Evaluate web framework .sx libraries\n var libs = document.querySelectorAll('script[type=\"text/sx-lib\"]');\n for (var i = 0; i < libs.length; i++) {\n evaluator.evalSource(libs[i].textContent);\n }\n // 2. Call boot-init (defined in boot.sx)\n evaluator.callFunction('boot-init');\n};" "javascript"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Performance")
(p "Parsing + evaluating ~5,000 lines of web framework " (code ".sx") " at startup takes ~10\u201350ms. After " (code "define") ", functions are Lambda objects dispatched identically to compiled functions. " (strong "Zero ongoing performance difference.")))
;; -----------------------------------------------------------------------
;; Phase 3: Wire Up Rust/WASM
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 3: Wire Up Rust/WASM" :id "phase-3"
(p (strong "Goal:") " Rust evaluator calls " (code "sx-platform.js") " via wasm-bindgen imports. Handle table bridges DOM references.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Handle table (JS-side)")
(~docs/code :src (highlight "// In sx-wasm-shim.js\nconst handles = [null]; // index 0 = null handle\nfunction allocHandle(obj) { handles.push(obj); return handles.length - 1; }\nfunction getHandle(id) { return handles[id]; }\nfunction freeHandle(id) { handles[id] = null; }" "javascript"))
(p "DOM nodes are JS objects. The handle table maps " (code "u32") " IDs to JS objects. Rust stores " (code "Value::Handle(u32)") " and passes the " (code "u32") " to imported JS functions.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Value::Handle in Rust")
(~docs/code :src (highlight "// In platform.rs\npub enum Value {\n // ... existing variants ...\n Handle(u32), // opaque reference to JS-side object\n}" "rust"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "WASM imports from platform")
(~docs/code :src (highlight "#[wasm_bindgen(module = \"/sx-platform-wasm.js\")]\nextern \"C\" {\n fn platform_create_element(tag: &str) -> u32;\n fn platform_create_text_node(text: &str) -> u32;\n fn platform_set_attr(handle: u32, name: &str, value: &str);\n fn platform_append_child(parent: u32, child: u32);\n fn platform_add_event_listener(handle: u32, event: &str, callback_id: u32);\n // ... ~50 DOM primitives\n}" "rust"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Callback table for events")
(p "When Rust creates an event handler (a Lambda), it stores it in a callback table and gets a " (code "u32") " ID. JS " (code "addEventListener") " wraps it: when the event fires, JS calls into WASM with the callback ID. Rust looks up the Lambda and evaluates it.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "sx-wasm-shim.js")
(p "Thin glue (~100 lines):")
(ul (~tw :tokens "list-disc list-inside space-y-1 mt-2")
(li "Instantiate WASM module")
(li "Wire handle table")
(li "Delegate all platform calls to " (code "sx-platform.js"))
(li "Provide " (code "invoke_callback") " \u2192 Rust for event dispatch")))
;; -----------------------------------------------------------------------
;; Phase 4: Web Framework Loading
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 4: Web Framework Loading" :id "phase-4"
(p (strong "Goal:") " Both JS and WASM evaluators load the same web framework " (code ".sx") " files at runtime.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Boot sequence (identical for both evaluators)")
(ol (~tw :tokens "list-decimal list-inside space-y-2 mt-2")
(li "Load " (code "sx-platform.js") " + evaluator (" (code "sx-evaluator.js") " or " (code "sx-wasm-shim.js") ")")
(li "Platform registers primitives with evaluator")
(li "Platform boot shim evaluates " (code "<script type=\"text/sx-lib\">") " blocks")
(li "Dependency order: signals \u2192 deps \u2192 frames \u2192 router \u2192 page-helpers \u2192 engine \u2192 orchestration \u2192 boot")
(li (code "boot-init") " called \u2014 processes component scripts, hydrates, initializes engine")
(li "Page is interactive"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Library dependency order")
(~docs/code :src (highlight "signals.sx \u2190 no SX deps (uses only core primitives)\ndeps.sx \u2190 no SX deps\nframes.sx \u2190 no SX deps\nrouter.sx \u2190 no SX deps\npage-helpers.sx \u2190 no SX deps\nengine.sx \u2190 uses render.sx (core), adapter-dom.sx (core)\norchestration.sx \u2190 depends on engine.sx\nboot.sx \u2190 depends on orchestration.sx" "text")))
;; -----------------------------------------------------------------------
;; Phase 5: Verification + Rollout
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 5: Verification + Rollout" :id "phase-5"
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Shadow comparison")
(p "Run both JS and WASM evaluators on same input, compare outputs:")
(ol (~tw :tokens "list-decimal list-inside space-y-1 mt-2")
(li (strong "Parse") " \u2014 same AST (already testable with current WASM exports)")
(li (strong "Eval") " \u2014 same values (already testable)")
(li (strong "Render") " \u2014 same DOM structure (requires Phase 3)"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Feature flag")
(p "Server sets " (code "data-sx-runtime=\"wasm\"") " or " (code "\"js\"") " on root element. Boot shim loads appropriate evaluator. Progressive enhancement: try WASM, fall back to JS.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Test matrix")
(p "All " (code "test-*.sx") " files from both " (code "spec/") " and " (code "web/") " run on BOTH evaluators."))
;; -----------------------------------------------------------------------
;; Principles
;; -----------------------------------------------------------------------
(~docs/section :title "Principles" :id "principles"
(ul (~tw :tokens "list-disc list-inside space-y-2")
(li (strong "Core is what defines the language.") " Parser, evaluator, primitives, render modes. If you can't run SX without it, it's core (" (code "spec/") "). If you can write it " (em "in") " SX, it's web framework (" (code "web/") ").")
(li (strong "Web framework runs ON the evaluator.") " Signals, engine, orchestration, boot, router, deps \u2014 these are SX programs. They're evaluated at runtime, not compiled into the evaluator.")
(li (strong "Isolation is the boundary.") " The evaluator can't touch the real world. Platform primitives are the only bridge. JS and WASM evaluators have identical isolation.")
(li (strong "Shared platform, not duplicated.") " One implementation of every browser primitive in " (code "sx-platform.js") ". Both evaluators use it. Fix a bug once, both get the fix.")
(li (strong "Handle table is the WASM boundary.") " Rust holds " (code "u32") " IDs. JavaScript holds real objects. Swap the handle table for a different host and the Rust code doesn't change.")
(li (strong "Progressive, not all-or-nothing.") " WASM is an enhancement. JS remains the fallback. Feature-flagged per page. Gradual and reversible.")))
;; -----------------------------------------------------------------------
;; Interactions
;; -----------------------------------------------------------------------
(~docs/section :title "Interaction with Other Plans" :id "interactions"
(ul (~tw :tokens "list-disc list-inside space-y-2")
(li (strong (a :href "/sx/(etc.(plan.rust-wasm-host))" "Rust/WASM Host")) " \u2014 this plan supersedes it. That plan didn't distinguish core from web framework or propose evaluator isolation. The shared platform layer is the key architectural difference.")
(li (strong (a :href "/sx/(etc.(plan.wasm-bytecode-vm))" "WASM Bytecode VM")) " \u2014 complementary. This plan bootstraps the tree-walking evaluator. A future bytecode VM compiles SX to bytecodes. The platform layer and handle table are shared.")
(li (strong (a :href "/sx/(etc.(plan.runtime-slicing))" "Runtime Slicing")) " \u2014 simplified. With web framework as runtime-evaluated " (code ".sx") ", slicing becomes: ship core + only the framework files you need. L0 pages skip signals/engine entirely.")
(li (strong (a :href "/sx/(etc.(plan.reactive-runtime))" "Reactive Runtime")) " \u2014 signals are web framework code, confirming they layer on top of core without special treatment.")
(li (strong (a :href "/sx/(etc.(plan.foundations))" "Foundations")) " \u2014 the core/web split is the same principle: small kernel, everything else built on top.")))
;; -----------------------------------------------------------------------
;; Outcome
;; -----------------------------------------------------------------------
(~docs/section :title "Outcome" :id "outcome"
(p "After completion:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "Core evaluator is ~3,200 lines of bootstrapped spec (" (code "spec/") " + adapters from " (code "web/") ").")
(li "JS evaluator is isolated \u2014 can't touch the real world, same sandbox as WASM.")
(li "Shared " (code "sx-platform.js") " provides all browser primitives to both evaluators.")
(li "Rust/WASM evaluator runs in the browser with full DOM rendering via handle table.")
(li "Web framework " (code ".sx") " files (signals, engine, orchestration, boot) are runtime-evaluated by whichever evaluator is active.")
(li "Hand-coded JS (" (code "CONTINUATIONS_JS") ", " (code "ASYNC_IO_JS") ") eliminated or bootstrapped from spec.")
(li "The architecture proof is complete: one spec, isolated evaluator, shared platform, deployment-time target selection.")))))

682
sx/sx/plans/isomorphic.sx
View File

@@ -1,682 +0,0 @@
;; ---------------------------------------------------------------------------
;; Isomorphic Architecture Roadmap
;; ---------------------------------------------------------------------------
(defcomp ~plans/isomorphic/plan-isomorphic-content ()
(~docs/page :title "Isomorphic Architecture Roadmap"
(~docs/section :title "Context" :id "context"
(p "SX has a working server-client pipeline: server evaluates pages with IO (DB, fragments), serializes as SX wire format, client parses and renders to DOM. The language and primitives are already isomorphic " (em "— same spec, same semantics, both sides.") " What's missing is the " (strong "plumbing") " that makes the boundary between server and client a sliding window rather than a fixed wall.")
(p "The key insight: " (strong "s-expressions can partially unfold on the server after IO, then finish unfolding on the client.") " The system knows which components have data fetches (via IO detection in " (a :href "/sx/(language.(spec.deps))" (~tw :tokens "text-violet-700 underline") "deps.sx") "), resolves those server-side, and sends the rest as pure SX for client rendering. The boundary slides automatically based on what each component actually needs."))
(~docs/section :title "Current State" :id "current-state"
(ul (~tw :tokens "space-y-2 text-stone-700 list-disc pl-5")
(li (strong "Primitive parity: ") "100%. ~80 pure primitives, same names/semantics, JS and Python.")
(li (strong "eval/parse/render: ") "Complete both sides. sx-ref.js has eval, parse, render-to-html, render-to-dom, aser.")
(li (strong "Engine: ") "engine.sx (morph, swaps, triggers, history), orchestration.sx (fetch, events), boot.sx (hydration) — all transpiled.")
(li (strong "Wire format: ") "Server _aser → SX source → client parses → renders to DOM. Boundary is clean.")
(li (strong "Component caching: ") "Hash-based localStorage for component definitions.")
(li (strong "Boundary enforcement: ") "boundary.sx + SX_BOUNDARY_STRICT=1 validates all primitives/IO/helpers at registration.")
(li (strong "Dependency analysis: ") "deps.sx computes per-page component bundles — only definitions a page actually uses are sent.")
(li (strong "IO detection: ") "deps.sx classifies every component as pure or IO-dependent. Server expands IO components, serializes pure ones for client.")
(li (strong "Client-side routing: ") "router.sx matches URL patterns. Pure pages render instantly without server roundtrips. Pages with :data fall through to server transparently.")
(li (strong "Client IO proxy: ") "IO primitives registered on the client call back to the server via fetch. Components with IO deps can render client-side.")
(li (strong "Streaming/suspense: ") "defpage :stream true enables chunked HTML. ~shared:pages/suspense placeholders show loading skeletons; __sxResolve() fills in content as IO completes.")))
;; -----------------------------------------------------------------------
;; Phase 1
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 1: Component Distribution & Dependency Analysis" :id "phase-1"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-4 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(language.(spec.deps))" (~tw :tokens "text-green-700 underline text-sm font-medium") "View canonical spec: deps.sx")
(a :href "/sx/(geography.(isomorphism.bundle-analyzer))" (~tw :tokens "text-green-700 underline text-sm font-medium") "Live bundle analyzer"))
(p (~tw :tokens "text-green-900 font-medium") "What it enables")
(p (~tw :tokens "text-green-800") "Per-page component bundles instead of sending every definition to every page. Smaller payloads, faster boot, better cache hit rates."))
(~docs/subsection :title "The Problem"
(p "The page boot payload serializes every component definition in the environment. A page that uses 5 components still receives all 50+. No mechanism determines which components a page actually needs — the boundary between \"loaded\" and \"used\" is invisible."))
(~docs/subsection :title "Implementation"
(p "The dependency analysis algorithm is defined in "
(a :href "/sx/(language.(spec.deps))" (~tw :tokens "text-violet-700 underline") "deps.sx")
" — a spec module bootstrapped to every host. Each host loads it via " (code "--spec-modules deps") " and provides 6 platform functions. The spec is the single source of truth; hosts are interchangeable.")
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. Transitive closure (deps.sx)")
(p "9 functions that walk the component graph. The core:")
(~docs/code :src (highlight "(define (transitive-deps name env)\n (let ((key (if (starts-with? name \"~\") name\n (concat \"~\" name)))\n (seen (set-create)))\n (transitive-deps-walk key env seen)\n (set-remove seen key)))" "lisp"))
(p (code "scan-refs") " walks a component body AST collecting " (code "~") " symbols. "
(code "transitive-deps") " follows references recursively through the env. "
(code "compute-all-deps") " batch-computes and caches deps for every component. "
"Circular references terminate safely via a seen-set."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. Page scanning")
(~docs/code :src (highlight "(define (components-needed page-source env)\n (let ((direct (scan-components-from-source page-source))\n (all-needed (set-create)))\n (for-each (fn (name) ...\n (set-add! all-needed name)\n (set-union! all-needed (component-deps comp)))\n direct)\n all-needed))" "lisp"))
(p (code "scan-components-from-source") " finds " (code "(~plans/content-addressed-components/name") " patterns in serialized SX via regex. " (code "components-needed") " combines scanning with the cached transitive closure to produce the minimal component set for a page."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Per-page CSS scoping")
(p (code "page-css-classes") " unions the CSS classes from only the components in the page bundle. Pages that don't use a component never pay for its styles."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "4. Platform interface")
(p "The spec declares 6 functions each host implements natively — the only host-specific code:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "component-deps") " / " (code "component-set-deps!") " — read/write the cached deps set on a component object")
(li (code "component-css-classes") " — read pre-scanned CSS class names from a component")
(li (code "env-components") " — enumerate all component entries in an environment")
(li (code "regex-find-all") " / " (code "scan-css-classes") " — host-native regex and CSS scanning")))))
(~docs/subsection :title "Spec module"
(p "deps.sx is loaded as a " (strong "spec module") " — an optional extension to the core spec. The bootstrapper flag " (code "--spec-modules deps") " includes it in the generated output alongside the core evaluator, parser, and renderer. Phase 2 IO detection was added to the same module — same bootstrapping mechanism, no architecture changes needed.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "shared/sx/ref/deps.sx — canonical spec (14 functions, 8 platform declarations)")
(li "Bootstrapped to all host targets via --spec-modules deps")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "15 dedicated tests: scan, transitive closure, circular deps, compute-all, components-needed")
(li "Bootstrapped output verified on both host targets")
(li "Full test suite passes with zero regressions")
(li (a :href "/sx/(geography.(isomorphism.bundle-analyzer))" (~tw :tokens "text-violet-700 underline") "Live bundle analyzer") " shows real per-page savings on this app"))))
;; -----------------------------------------------------------------------
;; Phase 2
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 2: Smart Server/Client Boundary — IO Detection" :id "phase-2"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-4 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(language.(spec.deps))" (~tw :tokens "text-green-700 underline text-sm font-medium") "View canonical spec: deps.sx")
(a :href "/sx/(geography.(isomorphism.bundle-analyzer))" (~tw :tokens "text-green-700 underline text-sm font-medium") "Live bundle analyzer with IO"))
(p (~tw :tokens "text-green-900 font-medium") "What it enables")
(p (~tw :tokens "text-green-800") "Automatic IO detection and selective expansion. Server expands IO-dependent components, serializes pure ones for client. Per-component intelligence replaces global toggle."))
(~docs/subsection :title "IO Detection in the Spec"
(p "Five new functions in "
(a :href "/sx/(language.(spec.deps))" (~tw :tokens "text-violet-700 underline") "deps.sx")
" extend the Phase 1 walker to detect IO primitive references:")
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. IO scanning")
(p (code "scan-io-refs") " walks an AST node, collecting symbol names that match an IO name set. The IO set is provided by the host from boundary declarations (all three tiers: core IO, deployment IO, page helpers).")
(~docs/code :src (highlight "(define scan-io-refs\n (fn (node io-names)\n (let ((refs (list)))\n (scan-io-refs-walk node io-names refs)\n refs)))" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. Transitive IO closure")
(p (code "transitive-io-refs") " follows component deps recursively, unioning IO refs from all reachable components and macros. Cycle-safe via seen-set.")
(~docs/code :src (highlight "(define transitive-io-refs\n (fn (name env io-names)\n ;; Walk deps, scan each body for IO refs,\n ;; union all refs transitively.\n ...))" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Batch computation")
(p (code "compute-all-io-refs") " iterates the env, computes transitive IO refs for each component, and caches the result via " (code "component-set-io-refs!") ". Called after " (code "compute-all-deps") " at component registration time."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "4. Component metadata")
(p "Each component now carries " (code "io_refs") " (transitive IO primitive names) alongside " (code "deps") " and " (code "css_classes") ". The derived " (code "is_pure") " property is true when " (code "io_refs") " is empty — the component can render anywhere without server data."))))
(~docs/subsection :title "Selective Expansion"
(p "The partial evaluator " (code "_aser") " now uses per-component IO metadata instead of a global toggle:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "IO-dependent") " → expand server-side (IO must resolve)")
(li (strong "Pure") " → serialize for client (let client render)")
(li (strong "Layout slot context") " → all components still expand (backwards compat via " (code "_expand_components") " context var)"))
(p "A component calling " (code "(highlight ...)") " or " (code "(query ...)") " is IO-dependent. A component with only HTML tags and string ops is pure."))
(~docs/subsection :title "Platform interface additions"
(p "Two new platform functions each host implements:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "(component-io-refs c) → cached IO ref list")
(li "(component-set-io-refs! c refs) → cache IO refs on component")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Components calling (query ...) or (highlight ...) classified IO-dependent")
(li "Pure components (HTML-only) classified pure with empty io_refs")
(li "Transitive IO detection: component calling ~other where ~other calls (current-user) → IO-dependent")
(li "Bootstrapped to both hosts (sx_ref.py + sx-ref.js)")
(li (a :href "/sx/(geography.(isomorphism.bundle-analyzer))" (~tw :tokens "text-violet-700 underline") "Live bundle analyzer") " shows per-page IO classification"))))
;; -----------------------------------------------------------------------
;; Phase 3
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 3: Client-Side Routing (SPA Mode)" :id "phase-3"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-4 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(language.(spec.router))" (~tw :tokens "text-green-700 underline text-sm font-medium") "View canonical spec: router.sx")
(a :href "/sx/(geography.(isomorphism.routing-analyzer))" (~tw :tokens "text-green-700 underline text-sm font-medium") "Live routing analyzer"))
(p (~tw :tokens "text-green-900 font-medium") "What it enables")
(p (~tw :tokens "text-green-800") "After initial page load, pure pages render instantly without server roundtrips. Client matches routes locally, evaluates content expressions with cached components, and only falls back to server for pages with :data dependencies."))
(~docs/subsection :title "Architecture"
(p "Three-layer approach: spec defines pure route matching, page registry bridges server metadata to client, orchestration intercepts navigation for try-first/fallback.")
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. Route matching spec (router.sx)")
(p "New spec module with pure functions for Flask-style route pattern matching:")
(~docs/code :src (highlight "(define split-path-segments ;; \"/language/docs/hello\" → (\"docs\" \"hello\")\n(define parse-route-pattern ;; \"/language/docs/<slug>\" → segment descriptors\n(define match-route-segments ;; segments + pattern → params dict or nil\n(define find-matching-route ;; path + route table → first match" "lisp"))
(p "No platform interface needed — uses only pure string and list primitives. Bootstrapped to both hosts via " (code "--spec-modules deps,router") "."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. Page registry")
(p "Server serializes defpage metadata as SX dict literals inside " (code "<script type=\"text/sx-pages\">") ". Each entry carries name, path pattern, auth level, has-data flag, serialized content expression, and closure values.")
(~docs/code :src (highlight "{:name \"docs-page\" :path \"/language/docs/<slug>\"\n :auth \"public\" :has-data false\n :content \"(case slug ...)\" :closure {}}" "lisp"))
(p "boot.sx processes these at startup using the SX parser — the same " (code "parse") " function from parser.sx — building route entries with parsed patterns into the " (code "_page-routes") " table. No JSON dependency."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Client-side interception (orchestration.sx)")
(p (code "bind-client-route-link") " replaces " (code "bind-boost-link") " in boost processing. On click:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Extract pathname from href")
(li "Call " (code "find-matching-route") " against " (code "_page-routes"))
(li "If match found AND no :data: evaluate content expression locally with component env + URL params")
(li "If evaluation succeeds: swap into #main-panel, pushState, log " (code "\"sx:route client /path\""))
(li "If anything fails (no match, has data, eval error): transparent fallback to server fetch"))
(p (code "handle-popstate") " also tries client routing before server fetch on back/forward."))))
(~docs/subsection :title "What becomes client-routable"
(p "All pages with content expressions — most of this docs app. Pure pages render instantly; :data pages fetch data then render client-side (Phase 4):")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "/") ", " (code "/language/docs/") ", " (code "/language/docs/<slug>") " (most slugs), " (code "/applications/protocols/") ", " (code "/applications/protocols/<slug>"))
(li (code "/examples/") ", " (code "/examples/<slug>") ", " (code "/etc/essays/") ", " (code "/etc/essays/<slug>"))
(li (code "/etc/plans/") ", " (code "/etc/plans/<slug>") ", " (code "/geography/isomorphism/") ", " (code "/language/bootstrappers/")))
(p "Pages that fall through to server:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "/language/docs/primitives") " and " (code "/language/docs/special-forms") " (call " (code "primitives-data") " / " (code "special-forms-data") " helpers)")
(li (code "/reference/<slug>") " (has " (code ":data (reference-data slug)") ")")
(li (code "/language/bootstrappers/<slug>") " (has " (code ":data (bootstrapper-data slug)") ")")
(li (code "/geography/isomorphism/bundle-analyzer") " (has " (code ":data (bundle-analyzer-data)") ")")
(li (code "/geography/isomorphism/data-test") " (has " (code ":data (data-test-data)") " — " (a :href "/sx/(geography.(isomorphism.data-test))" (~tw :tokens "text-violet-700 underline") "Phase 4 demo") ")")))
(~docs/subsection :title "Try-first/fallback design"
(p "Client routing uses a try-first approach: attempt local evaluation in a try/catch, fall back to server fetch on any failure. This avoids needing perfect static analysis of content expressions — if a content expression calls a page helper the client doesn't have, the eval throws, and the server handles it transparently.")
(p "Console messages provide visibility: " (code "sx:route client /essays/why-sexps") " vs " (code "sx:route server /specs/eval") "."))
(~docs/subsection :title "Files"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "shared/sx/ref/router.sx — route pattern matching spec")
(li "shared/sx/ref/boot.sx — process page registry scripts")
(li "shared/sx/ref/orchestration.sx — client route interception")
(li "shared/sx/ref/bootstrap_js.py — router spec module + platform functions")
(li "shared/sx/ref/bootstrap_py.py — router spec module (parity)")
(li "shared/sx/helpers.py — page registry SX serialization")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Pure page navigation: zero server requests, console shows \"sx:route client\"")
(li "IO/data page fallback: falls through to server fetch transparently")
(li "Browser back/forward works with client-routed pages")
(li "Disabling page registry → identical behavior to before")
(li "Bootstrap parity: sx_ref.py and sx-ref.js both contain router functions"))))
;; -----------------------------------------------------------------------
;; Phase 4
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 4: Client Async & IO Bridge" :id "phase-4"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-4 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(geography.(isomorphism.data-test))" (~tw :tokens "text-green-700 underline text-sm font-medium") "Live data test page"))
(p (~tw :tokens "text-green-900 font-medium") "What it enables")
(p (~tw :tokens "text-green-800") "Client fetches server-evaluated data and renders :data pages locally. Data cached with TTL to avoid redundant fetches on back/forward navigation. All IO stays server-side — no continuations needed."))
(~docs/subsection :title "Architecture"
(p "Separates IO from rendering. Server evaluates :data expression (async, with DB/service access), serializes result as SX wire format. Client fetches pre-evaluated data, parses it, merges into env, renders pure :content client-side.")
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. Abstract resolve-page-data")
(p "Spec-level primitive in orchestration.sx. The spec says \"I need data for this page\" — platform provides transport:")
(~docs/code :src (highlight "(resolve-page-data page-name params\n (fn (data)\n ;; data is a dict — merge into env and render\n (let ((env (merge closure params data))\n (rendered (try-eval-content content-src env)))\n (swap-rendered-content target rendered pathname))))" "lisp"))
(p "Browser platform: HTTP fetch to " (code "/sx/data/<page-name>") ". Future platforms could use IPC, cache, WebSocket, etc."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. Server data endpoint")
(p (code "evaluate_page_data()") " evaluates the :data expression, kebab-cases dict keys (Python " (code "total_count") " → SX " (code "total-count") "), serializes as SX wire format.")
(p "Response content type: " (code "text/sx; charset=utf-8") ". Per-page auth enforcement via " (code "_check_page_auth()") "."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Client data cache")
(p "In-memory cache in orchestration.sx, keyed by " (code "page-name:param=value") ". 30-second TTL prevents redundant fetches on back/forward navigation:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Cache miss: " (code "sx:route client+data /path") " — fetches from server, caches, renders")
(li "Cache hit: " (code "sx:route client+cache /path") " — instant render from cached data")
(li "After TTL: stale entry evicted, fresh fetch on next visit"))
(p "Try it: navigate to the " (a :href "/sx/(geography.(isomorphism.data-test))" (~tw :tokens "text-violet-700 underline") "data test page") ", go back, return within 30s — the server-time stays the same (cached). Wait 30s+ and return — new time (fresh fetch)."))))
(~docs/subsection :title "Files"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "shared/sx/ref/orchestration.sx — resolve-page-data spec, data cache")
(li "shared/sx/ref/bootstrap_js.py — platform resolvePageData (HTTP fetch)")
(li "shared/sx/pages.py — evaluate_page_data(), auto_mount_page_data()")
(li "shared/sx/helpers.py — deps for :data pages in page registry")
(li "sx/sx/data-test.sx — test component")
(li "shared/sx/tests/test_page_data.py — 30 unit tests")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "30 unit tests: serialize roundtrip, kebab-case, deps, full pipeline simulation, cache TTL")
(li "Console: " (code "sx:route client+data") " on first visit, " (code "sx:route client+cache") " on return within 30s")
(li (a :href "/sx/(geography.(isomorphism.data-test))" (~tw :tokens "text-violet-700 underline") "Live data test page") " exercises the full pipeline with server time + pipeline steps")
(li "append! and dict-set! registered as proper primitives in spec + both hosts"))))
;; -----------------------------------------------------------------------
;; Phase 5
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 5: Client IO Proxy" :id "phase-5"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-4 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete"))
(p (~tw :tokens "text-green-900 font-medium") "What it enables")
(p (~tw :tokens "text-green-800") "Components with IO dependencies render client-side. IO primitives are proxied to the server — the client evaluator calls them like normal functions, the proxy fetches results via HTTP, the async DOM renderer awaits the promises and continues."))
(~docs/subsection :title "How it works"
(p "Instead of async-aware continuations (originally planned), Phase 5 was solved by combining three mechanisms that emerged from Phases 3-4:")
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. IO dependency detection (from Phase 2)")
(p "The component dep analyzer scans AST bodies for IO primitive names (highlight, asset-url, query, frag, etc.) and computes transitive IO refs. Pages include their IO dep list in the page registry."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. IO proxy registration")
(p (code "registerIoDeps(names)") " in orchestration.sx registers proxy functions for each IO primitive. When the client evaluator encounters " (code "(highlight code \"sx\")") ", the proxy sends an HTTP request to the server's IO endpoint and returns a Promise."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Async DOM renderer")
(p (code "asyncRenderToDom") " walks the expression tree and handles Promises transparently. When a subexpression returns a Promise (from an IO proxy call), the renderer awaits it and continues building the DOM tree. No continuations needed — JavaScript's native Promise mechanism provides the suspension."))))
(~docs/subsection :title "Why continuations weren't needed"
(p "The original Phase 5 plan called for async-aware shift/reset or a CPS transform of the evaluator. In practice, JavaScript's Promise mechanism provided the same capability: the async DOM renderer naturally suspends when it encounters a Promise and resumes when it resolves.")
(p "Delimited continuations remain valuable for Phase 6 (streaming/suspense on the " (em "server") " side, where Python doesn't have native Promise-based suspension in the evaluator). But for client-side IO, Promises + async render were sufficient."))
(~docs/subsection :title "Files"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "shared/sx/ref/orchestration.sx — registerIoDeps, IO proxy registration")
(li "shared/sx/ref/bootstrap_js.py — asyncRenderToDom, IO proxy HTTP transport")
(li "shared/sx/helpers.py — io_deps in page registry entries")
(li "shared/sx/deps.py — transitive IO ref computation")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Navigate to any page with IO deps (e.g. /testing/eval) — console shows IO proxy calls")
(li "Components using " (code "highlight") " render correctly via proxy")
(li "Pages with " (code "asset-url") " resolve script paths via proxy")
(li "Async render completes without blocking — partial results appear as promises resolve"))))
;; -----------------------------------------------------------------------
;; Phase 6
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 6: Streaming & Suspense" :id "phase-6"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-4 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(geography.(isomorphism.streaming))" (~tw :tokens "text-green-700 underline text-sm font-medium") "Live streaming demo"))
(p (~tw :tokens "text-green-900 font-medium") "What it enables")
(p (~tw :tokens "text-green-800") "Server streams partially-evaluated SX as IO resolves. Client renders available subtrees immediately with loading skeletons, fills in suspended parts as data arrives."))
(~docs/subsection :title "What was built"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "~shared:pages/suspense") " component — renders fallback content with a stable DOM ID, replaced when resolution arrives")
(li (code "defpage :stream true") " — opts a page into streaming response mode")
(li (code "defpage :fallback expr") " — custom loading skeleton for streaming pages")
(li (code "execute_page_streaming()") " — Quart async generator response that yields HTML chunks")
(li (code "sx_page_streaming_parts()") " — splits the HTML shell into streamable parts")
(li (code "Sx.resolveSuspense(id, sx)") " — client-side function to replace suspense placeholders")
(li (code "window.__sxResolve") " bootstrap — queues resolutions that arrive before sx.js loads")
(li "Concurrent IO: data eval + header eval run in parallel via " (code "asyncio.create_task"))
(li "Completion-order streaming: whichever IO finishes first gets sent first via " (code "asyncio.wait(FIRST_COMPLETED)"))))
(~docs/subsection :title "Architecture"
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. Suspense component")
(p "When streaming, the server renders the page with " (code "~shared:pages/suspense") " placeholders instead of awaiting IO:")
(~docs/code :src (highlight "(~app-body\n :header-rows (~shared:pages/suspense :id \"stream-headers\"\n :fallback (div :class \"h-12 bg-stone-200 animate-pulse\"))\n :content (~shared:pages/suspense :id \"stream-content\"\n :fallback (div :class \"p-8 animate-pulse\" ...)))" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. Chunked transfer")
(p "Quart async generator response yields chunks in order:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-1")
(li "HTML shell + CSS + component defs + page registry + suspense page SX + scripts (immediate)")
(li "Resolution " (code "<script>") " tags as each IO completes")
(li "Closing " (code "</body></html>"))))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Client resolution")
(p "Each resolution chunk is an inline script:")
(~docs/code :src (highlight "<script>\n window.__sxResolve(\"stream-content\",\n \"(~article :title \\\"Hello\\\")\")\n</script>" "html"))
(p "The client parses the SX, renders to DOM, and replaces the suspense placeholder's children."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "4. Concurrent IO")
(p "Data evaluation and header construction run in parallel. " (code "asyncio.wait(FIRST_COMPLETED)") " yields resolution chunks in whatever order IO completes — no artificial sequencing."))))
(~docs/subsection :title "Continuation foundation"
(p "Delimited continuations (" (code "reset") "/" (code "shift") ") are implemented in the Python evaluator (async_eval.py lines 586-624) and available as special forms. Phase 6 uses the simpler pattern of concurrent IO + completion-order streaming, but the continuation machinery is in place for Phase 7's more sophisticated evaluation-level suspension."))
(~docs/subsection :title "Files"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "shared/sx/templates/pages.sx — ~shared:pages/suspense component definition")
(li "shared/sx/types.py — PageDef.stream, PageDef.fallback_expr fields")
(li "shared/sx/evaluator.py — defpage :stream/:fallback parsing")
(li "shared/sx/pages.py — execute_page_streaming(), streaming route mounting")
(li "shared/sx/helpers.py — sx_page_streaming_parts(), sx_streaming_resolve_script()")
(li "shared/sx/ref/boot.sx — resolve-suspense spec (canonical)")
(li "shared/sx/ref/bootstrap_js.py — resolveSuspense on Sx object, __sxPending/Resolve init")
(li "shared/static/scripts/sx-browser.js — bootstrapped output (DO NOT EDIT)")
(li "shared/sx/async_eval.py — reset/shift special forms (continuation foundation)")
(li "sx/sx/streaming-demo.sx — demo content component")
(li "sx/sxc/pages/docs.sx — streaming-demo defpage")
(li "sx/sxc/pages/helpers.py — streaming-demo-data page helper")))
(~docs/subsection :title "Demonstration"
(p "The " (a :href "/sx/(geography.(isomorphism.streaming))" (~tw :tokens "text-violet-700 underline") "streaming demo page") " exercises the full pipeline:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-1")
(li "Navigate to " (a :href "/sx/(geography.(isomorphism.streaming))" (~tw :tokens "text-violet-700 underline") "/sx/(geography.(isomorphism.streaming))"))
(li "The page skeleton appears " (strong "instantly") " — animated loading skeletons fill the content area")
(li "After ~1.5 seconds, the real content replaces the skeletons (streamed from server)")
(li "Open the Network tab — observe " (code "Transfer-Encoding: chunked") " on the document response")
(li "The document response shows multiple chunks arriving over time: shell first, then resolution scripts")))
(~docs/subsection :title "What to verify"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Instant shell: ") "The page HTML arrives immediately — no waiting for the 1.5s data fetch")
(li (strong "Suspense placeholders: ") "The " (code "~shared:pages/suspense") " component renders a " (code "data-suspense") " wrapper with animated fallback content")
(li (strong "Resolution: ") "The " (code "__sxResolve()") " inline script replaces the placeholder with real rendered content")
(li (strong "Chunked encoding: ") "Network tab shows the document as a chunked response with multiple frames")
(li (strong "Concurrent IO: ") "Header and content resolve independently — whichever finishes first appears first")
(li (strong "HTMX fallback: ") "SX/HTMX requests bypass streaming and receive a standard response"))))
;; -----------------------------------------------------------------------
;; Phase 7
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 7: Full Isomorphism" :id "phase-7"
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete"))
(p (~tw :tokens "text-green-900 font-medium") "What it enables")
(p (~tw :tokens "text-green-800") "Same SX code runs on either side. Runtime chooses optimal split via affinity annotations and render plans. Client data cache managed via invalidation headers and server-driven updates. Cross-host isomorphism verified by 61 automated tests."))
(~docs/subsection :title "7a. Affinity Annotations & Render Target"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-3 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete"))
(p (~tw :tokens "text-green-800 text-sm") "Components declare where they prefer to render. The spec combines affinity with IO analysis to produce a per-component render target decision."))
(p "Affinity annotations let component authors express rendering preferences:")
(~docs/code :src (highlight "(defcomp ~plans/isomorphic/product-grid (&key products)\n :affinity :client ;; interactive, prefer client rendering\n (div ...))\n\n(defcomp ~plans/isomorphic/auth-menu (&key user)\n :affinity :server ;; auth-sensitive, always server\n (div ...))\n\n(defcomp ~plans/isomorphic/card (&key title)\n ;; no annotation = :affinity :auto (default)\n ;; runtime decides from IO analysis\n (div ...))" "lisp"))
(p "The " (code "render-target") " function in deps.sx combines affinity with IO analysis:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code ":affinity :server") " → always " (code "\"server\"") " (auth-sensitive, secrets, heavy IO)")
(li (code ":affinity :client") " → always " (code "\"client\"") " (interactive, IO proxied)")
(li (code ":affinity :auto") " (default) → " (code "\"server\"") " if IO-dependent, " (code "\"client\"") " if pure"))
(p "The server's partial evaluator (" (code "_aser") ") uses " (code "render_target") " instead of the previous " (code "is_pure") " check. Components with " (code ":affinity :client") " are serialized for client rendering even if they call IO primitives — the IO proxy (Phase 5) handles the calls.")
(~docs/subsection :title "Files"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "shared/sx/ref/eval.sx — defcomp annotation parsing, defcomp-kwarg helper")
(li "shared/sx/ref/deps.sx — render-target function, platform interface")
(li "shared/sx/types.py — Component.affinity field, render_target property")
(li "shared/sx/evaluator.py — _sf_defcomp annotation extraction")
(li "shared/sx/async_eval.py — _aser uses render_target")
(li "shared/sx/ref/bootstrap_js.py — Component.affinity, componentAffinity()")
(li "shared/sx/ref/bootstrap_py.py — component_affinity(), make_component()")
(li "shared/sx/ref/test-eval.sx — 4 new defcomp affinity tests")
(li "shared/sx/ref/test-deps.sx — 6 new render-target tests")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "269 spec tests pass (10 new: 4 eval + 6 deps)")
(li "79 Python unit tests pass")
(li "Bootstrapped to both hosts (sx_ref.py + sx-browser.js)")
(li "Backward compatible: existing defcomp without :affinity defaults to \"auto\""))))
(~docs/subsection :title "7b. Runtime Boundary Optimizer"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-3 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete"))
(p (~tw :tokens "text-green-800 text-sm") "Per-page render plans computed at registration time. Each page knows exactly which components render server-side vs client-side, cached on PageDef."))
(p "Given component tree + IO dependency graph + affinity annotations, decide per-component: server-expand, client-render, or stream. Planning step cached at registration, recomputed on component change.")
(p (code "page-render-plan") " in deps.sx computes per-page boundary decisions:")
(~docs/code :src (highlight "(page-render-plan page-source env io-names)\n;; Returns:\n;; {:components {~plans/content-addressed-components/name \"server\"|\"client\" ...}\n;; :server (list of server-expanded names)\n;; :client (list of client-rendered names)\n;; :io-deps (IO primitives needed by server components)}" "lisp"))
(~docs/subsection :title "Integration Points"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "shared/sx/ref/deps.sx") " — " (code "page-render-plan") " spec function")
(li (code "shared/sx/deps.py") " — Python wrapper, dispatches to bootstrapped code")
(li (code "shared/sx/pages.py") " — " (code "compute_page_render_plans()") " called at mount time, caches on PageDef")
(li (code "shared/sx/helpers.py") " — " (code "_build_pages_sx()") " includes " (code ":render-plan") " in client page registry")
(li (code "shared/sx/types.py") " — " (code "PageDef.render_plan") " field")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "5 new spec tests (page-render-plan suite)")
(li "Render plans visible on " (a :href "/sx/(geography.(isomorphism.affinity))" "affinity demo page"))
(li "Client page registry includes :render-plan for each page"))))
(~docs/subsection :title "7c. Cache Invalidation & Optimistic Data Updates"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-3 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete"))
(p (~tw :tokens "text-green-800 text-sm") "Client data cache management, optimistic predicted mutations with snapshot rollback, and server-driven cache updates."))
(p "The client-side page data cache (30-second TTL) now supports cache invalidation, server-driven updates, and optimistic mutations. The client predicts the result of a mutation, immediately re-renders with the predicted data, and confirms or reverts when the server responds.")
(~docs/subsection :title "Cache Invalidation"
(p "Component authors can declare cache invalidation on elements that trigger mutations:")
(~docs/code :src (highlight ";; Clear specific page's cache after successful action\n(form :sx-post \"/cart/remove\"\n :sx-cache-invalidate \"cart-page\"\n ...)\n\n;; Clear ALL page caches after action\n(button :sx-post \"/admin/reset\"\n :sx-cache-invalidate \"*\")" "lisp"))
(p "The server can also control client cache via response headers:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "SX-Cache-Invalidate: page-name") " — clear cache for a page")
(li (code "SX-Cache-Update: page-name") " — replace cache with the response body (SX-format data)")))
(~docs/subsection :title "Optimistic Mutations"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "optimistic-cache-update") " — applies a mutator function to cached data, saves a snapshot for rollback")
(li (strong "optimistic-cache-revert") " — restores the pre-mutation snapshot if the server rejects")
(li (strong "optimistic-cache-confirm") " — discards the snapshot after server confirmation")
(li (strong "submit-mutation") " — orchestration function: predict, submit, confirm/revert")
(li (strong "/sx/action/<name>") " — server endpoint for processing mutations (POST, returns SX wire format)")))
(~docs/subsection :title "Files"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "shared/sx/ref/orchestration.sx — cache management + optimistic cache functions + submit-mutation spec")
(li "shared/sx/ref/engine.sx — SX-Cache-Invalidate, SX-Cache-Update response headers")
(li "shared/sx/pages.py — mount_action_endpoint for /sx/action/<name>")
(li "sx/sx/optimistic-demo.sx — live demo component")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Live demo at " (a :href "/sx/(geography.(isomorphism.optimistic))" (~tw :tokens "text-violet-600 hover:underline") "/sx/(geography.(isomorphism.optimistic))"))
(li "Console log: " (code "sx:optimistic confirmed") " / " (code "sx:optimistic reverted")))))
(~docs/subsection :title "7d. Offline Data Layer"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-3 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete"))
(p (~tw :tokens "text-green-800 text-sm") "Service Worker with IndexedDB caching, connectivity tracking, and offline mutation queue with replay on reconnect."))
(p "A Service Worker registered at " (code "/sx-sw.js") " provides three-tier caching, plus an offline mutation queue that builds on Phase 7c's optimistic updates:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "/sx/data/* ") "— network-first with IndexedDB fallback. Page data cached on fetch, served from IndexedDB when offline.")
(li (strong "/sx/io/* ") "— network-first with IndexedDB fallback. IO proxy responses cached the same way.")
(li (strong "/static/* ") "— stale-while-revalidate via Cache API. Serves cached assets immediately, updates in background.")
(li (strong "Offline mutations") " — " (code "offline-aware-mutation") " routes to " (code "submit-mutation") " when online, " (code "offline-queue-mutation") " when offline. " (code "offline-sync") " replays the queue on reconnect."))
(~docs/subsection :title "How It Works"
(ol (~tw :tokens "list-decimal list-inside text-stone-700 space-y-2")
(li "On boot, " (code "sx-browser.js") " registers the SW at " (code "/sx-sw.js") " (root scope)")
(li "SW intercepts fetch events and routes by URL pattern")
(li "For data/IO: try network first, on failure serve from IndexedDB")
(li "For static assets: serve from Cache API, revalidate in background")
(li "Cache invalidation propagates: element attr / response header → in-memory cache → SW message → IndexedDB")
(li "Offline mutations queue locally, replay on reconnect via " (code "offline-sync"))))
(~docs/subsection :title "Files"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "shared/static/scripts/sx-sw.js — Service Worker (network-first + stale-while-revalidate)")
(li "shared/sx/ref/orchestration.sx — offline queue, sync, connectivity tracking, sw-post-message")
(li "shared/sx/pages.py — mount_service_worker() serves SW at /sx-sw.js")
(li "sx/sx/offline-demo.sx — live demo component")))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Live demo at " (a :href "/sx/(geography.(isomorphism.offline))" (~tw :tokens "text-violet-600 hover:underline") "/sx/(geography.(isomorphism.offline))"))
(li "Test with DevTools Network → Offline mode")
(li "Console log: " (code "sx:offline queued") ", " (code "sx:offline syncing") ", " (code "sx:offline synced")))))
(~docs/subsection :title "7e. Isomorphic Testing"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-3 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete"))
(p (~tw :tokens "text-green-800 text-sm") "Cross-host test suite: same SX expressions evaluated on Python (sx_ref.py) and JS (sx-browser.js via Node.js), HTML output compared."))
(p "61 isomorphic tests verify that Python and JS produce identical results:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "37 eval tests: arithmetic, comparison, strings, collections, logic, let/lambda, higher-order, dict, keywords, cond/case")
(li "24 render tests: elements, attributes, nesting, void elements, boolean attrs, conditionals, map, components, affinity, HTML escaping"))
(~docs/subsection :title "Files"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "shared/sx/tests/test_isomorphic.py — cross-host test suite")
(li "Run: " (code "python3 -m pytest shared/sx/tests/test_isomorphic.py -q")))))
(~docs/subsection :title "7f. Universal Page Descriptor"
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-3 mb-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete"))
(p (~tw :tokens "text-green-800 text-sm") "defpage is portable: same descriptor executes on server (execute_page) and client (tryClientRoute)."))
(p "The defpage descriptor is universal — the same definition works on both hosts:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Server: ") (code "execute_page()") " evaluates :data and :content slots, expands server components via " (code "_aser") ", returns SX wire format")
(li (strong "Client: ") (code "try-client-route") " matches route, evaluates content SX, renders to DOM. Data pages fetch via " (code "/sx/data/") ", IO proxied via " (code "/sx/io/"))
(li (strong "Render plan: ") "each page's " (code ":render-plan") " is included in the client page registry, showing which components render where")
(li (strong "Console visibility: ") "client logs " (code "sx:route plan pagename — N server, M client") " on each navigation")))
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-3 mt-2")
(p (~tw :tokens "text-amber-800 text-sm") (strong "Depends on: ") "All previous phases.")))
;; -----------------------------------------------------------------------
;; Cross-Cutting Concerns
;; -----------------------------------------------------------------------
(~docs/section :title "Cross-Cutting Concerns" :id "cross-cutting"
(~docs/subsection :title "Error Reporting"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Phase 1: \"Unknown component\" includes which page expected it and what bundle was sent")
(li "Phase 2: Server logs which components expanded server-side vs sent to client")
(li "Phase 3: Client route failures include unmatched path and available routes")
(li "Phase 4: Client data errors include page name, params, server response status")
(li "Source location tracking in parser → propagate through eval → include in error messages")))
(~docs/subsection :title "Backward Compatibility"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Pages without annotations behave as today")
(li "SX-Request / SX-Components / SX-Css header protocol continues")
(li "Existing .sx files require no changes")
(li "_expand_components continues as override")
(li "Each phase is opt-in: disable → identical to previous behavior")))
(~docs/subsection :title "Spec Integrity"
(p "All new behavior specified in .sx files under shared/sx/ref/ before implementation. Bootstrappers transpile from spec. This ensures JS and Python stay in sync.")))
;; -----------------------------------------------------------------------
;; Critical Files
;; -----------------------------------------------------------------------
(~docs/section :title "Critical Files" :id "critical-files"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Role")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Phases")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/async_eval.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Core evaluator, _aser, server/client boundary")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2, 5"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/helpers.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "sx_page(), sx_response(), output pipeline")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1, 3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/jinja_bridge.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "_COMPONENT_ENV, component registry")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1, 2"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/pages.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "defpage, execute_page(), page lifecycle")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2, 3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/boot.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client boot, component caching")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1, 3, 4"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/orchestration.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client fetch/swap/morph")
(td (~tw :tokens "px-3 py-2 text-stone-600") "3, 4"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/eval.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Evaluator spec")
(td (~tw :tokens "px-3 py-2 text-stone-600") "4"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/engine.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Morph, swaps, triggers")
(td (~tw :tokens "px-3 py-2 text-stone-600") "3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/deps.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Dependency analysis (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1, 2"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/router.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client-side routing (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/io-bridge.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client IO primitives (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "4"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/suspense.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Streaming/suspension (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "5"))))))))
;; ---------------------------------------------------------------------------
;; SX CI Pipeline
;; ---------------------------------------------------------------------------

511
sx/sx/plans/js-bootstrapper.sx
View File

@@ -1,511 +0,0 @@
;; ---------------------------------------------------------------------------
;; js.sx — Self-Hosting JavaScript Bootstrapper
;; ---------------------------------------------------------------------------
(defcomp ~plans/js-bootstrapper/plan-js-bootstrapper-content ()
(~docs/page :title "js.sx — JavaScript Bootstrapper"
;; -----------------------------------------------------------------------
;; Overview
;; -----------------------------------------------------------------------
(~docs/section :title "Overview" :id "overview"
(p (code "bootstrap_js.py") " is a 4,361-line Python program that reads the "
(code ".sx") " spec files and emits " (code "sx-ref.js") " — the entire "
"browser runtime. Parser, evaluator, three rendering adapters (HTML, SX wire, DOM), "
"the engine (fetch/swap/trigger), orchestration, boot, signals, router, component "
"dependency analysis — all transpiled from " (code ".sx") " spec files into JavaScript.")
(p (code "js.sx") " replaces that Python program with an SX program. "
"Like " (code "py.sx") " for Python, " (code "js.sx") " is an SX-to-JavaScript "
"translator written in SX. But it goes further.")
(p "Because the JS bootstrapper produces " (em "browser code") ", " (code "js.sx")
" can also compile " (em "any") " SX component tree into a standalone JavaScript "
"program. The server evaluates an " (code ".sx") " page definition, calls "
(code "#js") " on the result, and gets a self-contained JS bundle that renders "
"the same output in the browser — no SX runtime needed."))
;; -----------------------------------------------------------------------
;; Two Modes
;; -----------------------------------------------------------------------
(~docs/section :title "Two Compilation Modes" :id "modes"
(~docs/subsection :title "Mode 1: Spec Bootstrapper"
(p "Same job as " (code "bootstrap_js.py") ". Read spec " (code ".sx") " files, "
"emit " (code "sx-ref.js") ".")
(~docs/code :src (highlight ";; Translate eval.sx to JavaScript
(js-translate-file (parse-file \"eval.sx\"))
;; → \"function evalExpr(expr, env) { ... }\"
;; Full bootstrap: all spec modules → sx-ref.js
(js-bootstrap
:adapters (list \"html\" \"sx\" \"dom\" \"engine\" \"boot\")
:modules (list \"deps\" \"router\" \"signals\"))" "lisp"))
(p "The output is identical to " (code "python bootstrap_js.py") ". "
"Verification: " (code "diff <(python bootstrap_js.py) <(python run_js_sx.py)") "."))
(~docs/subsection :title "Mode 2: Component Compiler"
(p "Server-side SX evaluation + " (code "js.sx") " translation = static JS output. "
"Given a component tree that the server has already evaluated (data fetched, "
"conditionals resolved, loops expanded), " (code "js.sx") " compiles the "
"resulting DOM description into a JavaScript program that builds the same DOM.")
(~docs/code :src (highlight ";; Server evaluates the page (fetches data, expands components)
;; Result is a resolved SX tree: (div :class \"...\" (h1 \"Hello\") ...)
;; js.sx compiles that tree to standalone JS
(js-compile-component evaluated-tree)
;; → \"(function(){
;; var el = document.createElement('div');
;; el.className = '...';
;; var h1 = document.createElement('h1');
;; h1.textContent = 'Hello';
;; el.appendChild(h1);
;; return el;
;; })()\"" "lisp"))
(p "This is ahead-of-time compilation. The browser receives JavaScript, "
"not s-expressions. No parser, no evaluator, no runtime. "
"Just DOM construction code.")))
;; -----------------------------------------------------------------------
;; Architecture
;; -----------------------------------------------------------------------
(~docs/section :title "Architecture" :id "architecture"
(p "The JS bootstrapper has more moving parts than the Python one because "
"JavaScript is the " (em "client") " host. The browser runtime includes "
"things Python never needs:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 my-4")
(table (~tw :tokens "w-full text-sm")
(thead (~tw :tokens "bg-stone-50")
(tr
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Spec Module")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Purpose")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Python?")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Browser?")))
(tbody
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "eval.sx")
(td (~tw :tokens "px-4 py-2") "Core evaluator, special forms, TCO")
(td (~tw :tokens "px-4 py-2 text-center") "Yes") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "render.sx")
(td (~tw :tokens "px-4 py-2") "Tag registry, void elements, boolean attrs")
(td (~tw :tokens "px-4 py-2 text-center") "Yes") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "parser.sx")
(td (~tw :tokens "px-4 py-2") "Tokenizer, parser, serializer")
(td (~tw :tokens "px-4 py-2 text-center") "—") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "adapter-html.sx")
(td (~tw :tokens "px-4 py-2") "Render to HTML string")
(td (~tw :tokens "px-4 py-2 text-center") "Yes") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "adapter-sx.sx")
(td (~tw :tokens "px-4 py-2") "Serialize to SX wire format")
(td (~tw :tokens "px-4 py-2 text-center") "Yes") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100 bg-blue-50")
(td (~tw :tokens "px-4 py-2 font-mono") "adapter-dom.sx")
(td (~tw :tokens "px-4 py-2") "Render to live DOM nodes")
(td (~tw :tokens "px-4 py-2 text-center") "—") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100 bg-blue-50")
(td (~tw :tokens "px-4 py-2 font-mono") "engine.sx")
(td (~tw :tokens "px-4 py-2") "Fetch, swap, trigger, history")
(td (~tw :tokens "px-4 py-2 text-center") "—") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100 bg-blue-50")
(td (~tw :tokens "px-4 py-2 font-mono") "orchestration.sx")
(td (~tw :tokens "px-4 py-2") "Element scanning, attribute processing")
(td (~tw :tokens "px-4 py-2 text-center") "—") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100 bg-blue-50")
(td (~tw :tokens "px-4 py-2 font-mono") "boot.sx")
(td (~tw :tokens "px-4 py-2") "Script processing, mount, hydration")
(td (~tw :tokens "px-4 py-2 text-center") "—") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "signals.sx")
(td (~tw :tokens "px-4 py-2") "Reactive signal runtime")
(td (~tw :tokens "px-4 py-2 text-center") "Yes") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "deps.sx")
(td (~tw :tokens "px-4 py-2") "Component dependency analysis")
(td (~tw :tokens "px-4 py-2 text-center") "Yes") (td (~tw :tokens "px-4 py-2 text-center") "Yes"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "router.sx")
(td (~tw :tokens "px-4 py-2") "Client-side route matching")
(td (~tw :tokens "px-4 py-2 text-center") "Yes") (td (~tw :tokens "px-4 py-2 text-center") "Yes")))))
(p "Blue rows are browser-only modules. " (code "js.sx") " must handle all of them. "
"The platform interface is also larger: DOM primitives (" (code "dom-create-element")
", " (code "dom-append") ", " (code "dom-set-attr") ", ...), "
"browser APIs (" (code "fetch") ", " (code "history") ", " (code "localStorage")
"), and event handling."))
;; -----------------------------------------------------------------------
;; Translation Rules
;; -----------------------------------------------------------------------
(~docs/section :title "Translation Rules" :id "translation"
(p (code "js.sx") " shares the same pattern as " (code "py.sx") " — expression translator, "
"statement translator, name mangling — but with JavaScript-specific mappings:")
(~docs/subsection :title "Name Mangling"
(p "SX uses kebab-case. JavaScript uses camelCase.")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 my-4")
(table (~tw :tokens "w-full text-sm")
(thead (~tw :tokens "bg-stone-50")
(tr
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "SX")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "JavaScript")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Rule")))
(tbody
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "eval-expr")
(td (~tw :tokens "px-4 py-2 font-mono") "evalExpr")
(td (~tw :tokens "px-4 py-2") "kebab → camelCase"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "nil?")
(td (~tw :tokens "px-4 py-2 font-mono") "isNil")
(td (~tw :tokens "px-4 py-2") "predicate → is-prefix"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "empty?")
(td (~tw :tokens "px-4 py-2 font-mono") "isEmpty")
(td (~tw :tokens "px-4 py-2") "? → is-prefix (general)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "set!")
(td (~tw :tokens "px-4 py-2 font-mono") "—")
(td (~tw :tokens "px-4 py-2") "assignment (no rename)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "dom-create-element")
(td (~tw :tokens "px-4 py-2 font-mono") "domCreateElement")
(td (~tw :tokens "px-4 py-2") "platform function rename"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "delete")
(td (~tw :tokens "px-4 py-2 font-mono") "delete_")
(td (~tw :tokens "px-4 py-2") "JS reserved word escape"))))))
(~docs/subsection :title "Special Forms → JavaScript"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 my-4")
(table (~tw :tokens "w-full text-sm")
(thead (~tw :tokens "bg-stone-50")
(tr
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "SX")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "JavaScript")))
(tbody
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(if c t e)")
(td (~tw :tokens "px-4 py-2 font-mono") "(sxTruthy(c) ? t : e)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(when c body)")
(td (~tw :tokens "px-4 py-2 font-mono") "(sxTruthy(c) ? body : NIL)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(let ((a 1)) body)")
(td (~tw :tokens "px-4 py-2 font-mono") "(function(a) { return body; })(1)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(fn (x) body)")
(td (~tw :tokens "px-4 py-2 font-mono") "function(x) { return body; }"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(define name val)")
(td (~tw :tokens "px-4 py-2 font-mono") "var name = val;"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(and a b c)")
(td (~tw :tokens "px-4 py-2 font-mono") "(sxTruthy(a) ? (sxTruthy(b) ? c : b) : a)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(case x \"a\" 1 ...)")
(td (~tw :tokens "px-4 py-2 font-mono") "sxCase(x, [[\"a\", () => 1], ...])"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(str a b c)")
(td (~tw :tokens "px-4 py-2 font-mono") "sxStr(a, b, c)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "&rest args")
(td (~tw :tokens "px-4 py-2 font-mono") "...args (rest params)"))))))
(~docs/subsection :title "JavaScript Advantages"
(p "JavaScript is easier to target than Python in two key ways:")
(ul (~tw :tokens "list-disc pl-6 space-y-2 text-stone-700")
(li (strong "No mutation problem. ")
"JavaScript closures capture by reference, not by value. "
(code "set!") " from a nested function Just Works — no cell variable "
"hack needed. This eliminates the hardest part of " (code "py.sx") ".")
(li (strong "Expression-oriented. ")
"JavaScript's comma operator, ternary, and IIFEs make "
"almost everything expressible as an expression. "
"The statement/expression duality is less painful than Python."))))
;; -----------------------------------------------------------------------
;; Component Compilation
;; -----------------------------------------------------------------------
(~docs/section :title "Component Compilation" :id "component-compiler"
(p "Mode 2 is the interesting one. The server already evaluates SX page "
"definitions — it fetches data, resolves conditionals, expands components, "
"and produces a complete DOM description as an SX tree. Currently this tree "
"is either:")
(ul (~tw :tokens "list-disc pl-6 space-y-1 text-stone-700")
(li "Rendered to HTML server-side (" (code "render-to-html") ")")
(li "Serialized as SX wire format for the client to render (" (code "aser") ")"))
(p "A third option: " (strong "compile it to JavaScript") ". "
"The SX tree is already fully resolved — no data to fetch, no conditionals "
"to evaluate. It's just a description of DOM nodes. " (code "js.sx")
" walks this tree and emits imperative JavaScript that constructs the same DOM.")
(~docs/subsection :title "What Gets Compiled"
(p "A resolved SX tree like:")
(~docs/code :src (highlight "(div :class \"container\"
(h1 \"Hello\")
(ul (map (fn (item)
(li :class \"item\" (get item \"name\")))
items)))" "lisp"))
(p "After server-side evaluation (with " (code "items") " = "
(code "[{\"name\": \"Alice\"}, {\"name\": \"Bob\"}]") "):")
(~docs/code :src (highlight "(div :class \"container\"
(h1 \"Hello\")
(ul
(li :class \"item\" \"Alice\")
(li :class \"item\" \"Bob\")))" "lisp"))
(p "Compiles to:")
(~docs/code :src (highlight "var _0 = document.createElement('div');
_0.className = 'container';
var _1 = document.createElement('h1');
_1.textContent = 'Hello';
_0.appendChild(_1);
var _2 = document.createElement('ul');
var _3 = document.createElement('li');
_3.className = 'item';
_3.textContent = 'Alice';
_2.appendChild(_3);
var _4 = document.createElement('li');
_4.className = 'item';
_4.textContent = 'Bob';
_2.appendChild(_4);
_0.appendChild(_2);" "javascript")))
(~docs/subsection :title "Why Not Just Use HTML?"
(p "HTML already does this — " (code "innerHTML") " parses and builds DOM. "
"Why compile to JS instead?")
(ul (~tw :tokens "list-disc pl-6 space-y-2 text-stone-700")
(li (strong "Event handlers. ")
"HTML can't express " (code ":on-click") " or " (code ":sx-get")
" — those need JavaScript. The compiled JS can wire up event "
"listeners inline during construction.")
(li (strong "Reactive islands. ")
"Signal bindings (" (code "deref") "), reactive text nodes, and "
"reactive attributes need to register subscriptions during construction. "
"Compiled JS can create signals and wire subscriptions as it builds the DOM.")
(li (strong "No parse overhead. ")
"The browser doesn't need to parse HTML or SX source. "
"The JavaScript engine JIT-compiles the DOM construction code. "
"For large pages, this can be faster than " (code "innerHTML") ".")
(li (strong "Tree-shakeable. ")
"The compiled output only contains what the page uses. "
"No SX parser, no evaluator, no rendering runtime. "
"A static page compiles to pure DOM API calls — zero framework overhead.")
(li (strong "Portable. ")
"The output is a JavaScript module. It works in any JS environment: "
"browser, Node, Deno, Bun. Server-side rendered pages become "
"testable JavaScript programs.")))
(~docs/subsection :title "Hybrid Mode"
(p "Not every page is fully static. Some parts are server-rendered, "
"some are interactive. " (code "js.sx") " handles this with a hybrid approach:")
(ul (~tw :tokens "list-disc pl-6 space-y-2 text-stone-700")
(li (strong "Static subtrees") " → compiled to DOM construction code (no runtime)")
(li (strong "Reactive islands") " → compiled with signal creation + subscriptions "
"(needs signal runtime, ~2KB)")
(li (strong "Hypermedia attributes") " (" (code "sx-get") ", " (code "sx-post")
") → compiled with event listeners + fetch calls "
"(needs engine, ~5KB)")
(li (strong "Client-routed pages") " → full SX runtime included"))
(p "The compiler analyzes the tree and includes only the runtime slices needed. "
"A purely static page ships zero SX runtime. "
"A page with one reactive counter ships just the signal runtime.")))
;; -----------------------------------------------------------------------
;; The Bootstrap Chain
;; -----------------------------------------------------------------------
(~docs/section :title "The Bootstrap Chain" :id "chain"
(p "With both " (code "py.sx") " and " (code "js.sx") ", the full picture:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 my-4")
(table (~tw :tokens "w-full text-sm")
(thead (~tw :tokens "bg-stone-50")
(tr
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Translator")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Written in")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Outputs")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Replaces")))
(tbody
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "z3.sx")
(td (~tw :tokens "px-4 py-2") "SX")
(td (~tw :tokens "px-4 py-2") "SMT-LIB")
(td (~tw :tokens "px-4 py-2 text-stone-400 italic") "(none — new capability)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "prove.sx")
(td (~tw :tokens "px-4 py-2") "SX")
(td (~tw :tokens "px-4 py-2") "Constraint proofs")
(td (~tw :tokens "px-4 py-2 text-stone-400 italic") "(none — new capability)"))
(tr (~tw :tokens "border-t border-stone-100 bg-violet-50")
(td (~tw :tokens "px-4 py-2 font-mono text-violet-700") "py.sx")
(td (~tw :tokens "px-4 py-2") "SX")
(td (~tw :tokens "px-4 py-2") "Python")
(td (~tw :tokens "px-4 py-2 font-mono") "bootstrap_py.py"))
(tr (~tw :tokens "border-t border-stone-100 bg-blue-50")
(td (~tw :tokens "px-4 py-2 font-mono text-blue-700") "js.sx")
(td (~tw :tokens "px-4 py-2") "SX")
(td (~tw :tokens "px-4 py-2") "JavaScript")
(td (~tw :tokens "px-4 py-2 font-mono") "bootstrap_js.py"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono text-stone-400") "go.sx")
(td (~tw :tokens "px-4 py-2") "SX")
(td (~tw :tokens "px-4 py-2") "Go")
(td (~tw :tokens "px-4 py-2 text-stone-400 italic") "(future host)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono text-stone-400") "rs.sx")
(td (~tw :tokens "px-4 py-2") "SX")
(td (~tw :tokens "px-4 py-2") "Rust")
(td (~tw :tokens "px-4 py-2 text-stone-400 italic") "(future host)")))))
(p "Every translator is an SX program. The only Python left is the platform "
"interface (types, DOM primitives, runtime support functions) and the thin "
"runner script that loads " (code "py.sx") " or " (code "js.sx")
" and feeds it the spec files."))
;; -----------------------------------------------------------------------
;; Implementation Plan
;; -----------------------------------------------------------------------
(~docs/section :title "Implementation" :id "implementation"
(~docs/subsection :title "Phase 1: Expression Translator"
(p "Core SX-to-JavaScript expression translation.")
(ul (~tw :tokens "list-disc pl-6 space-y-1 text-stone-700")
(li (code "js-mangle") " — SX name → JavaScript identifier (RENAMES + kebab→camelCase)")
(li (code "js-literal") " — atoms: numbers, strings, booleans, nil, symbols, keywords")
(li (code "js-expr") " — recursive expression translator")
(li "Ternary: " (code "if") ", " (code "when") ", " (code "cond") ", " (code "and") ", " (code "or"))
(li (code "let") " → IIFE: " (code "(function(a) { return body; })(val)"))
(li (code "fn") " → " (code "function(x) { return body; }"))
(li (code "str") " → " (code "sxStr(...)"))
(li "Infix: " (code "+") ", " (code "-") ", " (code "*") ", " (code "/") ", "
(code "===") ", " (code "!==") ", " (code "%"))
(li (code "&rest") " → " (code "...args") " (rest parameters)")))
(~docs/subsection :title "Phase 2: Statement Translator"
(p "Top-level and function body statement emission.")
(ul (~tw :tokens "list-disc pl-6 space-y-1 text-stone-700")
(li (code "js-statement") " — emit as JavaScript statement")
(li (code "define") " → " (code "var name = expr;"))
(li (code "set!") " → direct assignment (closures capture by reference)")
(li (code "for-each") " → " (code "for (var i = 0; i < arr.length; i++)") " loop")
(li (code "do") "/" (code "begin") " → comma expression or block")
(li "Function bodies with multiple expressions → explicit " (code "return"))))
(~docs/subsection :title "Phase 3: Spec Bootstrapper"
(p "Process spec files identically to " (code "bootstrap_js.py") ".")
(ul (~tw :tokens "list-disc pl-6 space-y-1 text-stone-700")
(li (code "js-extract-defines") " — parse .sx source, collect top-level defines")
(li (code "js-translate-file") " — translate a list of define expressions")
(li "Adapter selection: parser, html, sx, dom, engine, orchestration, boot")
(li "Dependency resolution: engine requires dom, boot requires engine + parser")
(li "Static sections (IIFE wrapper, platform interface) stay as string templates")))
(~docs/subsection :title "Phase 4: Component Compiler"
(p "Ahead-of-time compilation of evaluated SX trees to JavaScript.")
(ul (~tw :tokens "list-disc pl-6 space-y-1 text-stone-700")
(li (code "js-compile-element") " — emit " (code "createElement") " + attribute setting")
(li (code "js-compile-text") " — emit " (code "textContent") " or " (code "createTextNode"))
(li (code "js-compile-component") " — inline-expand or emit component call")
(li (code "js-compile-island") " — emit signal creation + reactive DOM subscriptions")
(li (code "js-compile-fragment") " — emit " (code "DocumentFragment") " construction")
(li "Runtime slicing: analyze tree → include only necessary runtime modules")))
(~docs/subsection :title "Phase 5: Verification"
(~docs/code :src (highlight "# Mode 1: spec bootstrapper parity
python bootstrap_js.py > sx-ref-g0.js
python run_js_sx.py > sx-ref-g1.js
diff sx-ref-g0.js sx-ref-g1.js # must be empty
# Mode 2: component compilation correctness
# Server renders page → SX tree → compile to JS
# Compare DOM output: runtime-rendered vs compiled
python test_js_compile.py # renders both, diffs DOM" "bash")))
;; -----------------------------------------------------------------------
;; Comparison with py.sx
;; -----------------------------------------------------------------------
(~docs/section :title "Comparison with py.sx" :id "comparison"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 my-4")
(table (~tw :tokens "w-full text-sm")
(thead (~tw :tokens "bg-stone-50")
(tr
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Concern")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") (code "py.sx"))
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") (code "js.sx"))))
(tbody
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2") "Naming convention")
(td (~tw :tokens "px-4 py-2") "snake_case")
(td (~tw :tokens "px-4 py-2") "camelCase"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2") "Closures & mutation")
(td (~tw :tokens "px-4 py-2") "Cell variable hack")
(td (~tw :tokens "px-4 py-2") "Direct (reference capture)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2") "Spec modules")
(td (~tw :tokens "px-4 py-2") "eval, render, html, sx, deps, signals")
(td (~tw :tokens "px-4 py-2") "All 12 modules"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2") "Platform interface")
(td (~tw :tokens "px-4 py-2") "~300 lines")
(td (~tw :tokens "px-4 py-2") "~1500 lines (DOM, browser APIs)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2") "RENAMES table")
(td (~tw :tokens "px-4 py-2") "~200 entries")
(td (~tw :tokens "px-4 py-2") "~350 entries"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2") "Component compilation")
(td (~tw :tokens "px-4 py-2 text-stone-400") "N/A")
(td (~tw :tokens "px-4 py-2") "Ahead-of-time DOM compiler"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2") "Estimated size")
(td (~tw :tokens "px-4 py-2") "~800-1000 lines")
(td (~tw :tokens "px-4 py-2") "~1200-1500 lines"))))))
;; -----------------------------------------------------------------------
;; Implications
;; -----------------------------------------------------------------------
(~docs/section :title "Implications" :id "implications"
(~docs/subsection :title "Zero-Runtime Static Sites"
(p "A static page written in SX compiles to a JavaScript program with "
"no SX runtime dependency. The output is just DOM API calls — "
(code "createElement") ", " (code "appendChild") ", " (code "textContent")
". This gives SX a compilation target competitive with Svelte's "
"approach: components compile away, the framework disappears.")
(p "Combined with the " (a :href "/sx/(etc.(plan.content-addressed-components))" "content-addressed components")
" plan, a page's compiled JS could be stored on IPFS by its content hash. "
"The server returns a CID. The browser fetches and executes pre-compiled JavaScript. "
"No parser, no evaluator, no network round-trip for component definitions."))
(~docs/subsection :title "Progressive Enhancement Layers"
(p "The component compiler naturally supports progressive enhancement:")
(ol (~tw :tokens "list-decimal pl-6 space-y-1 text-stone-700")
(li (strong "HTML") " — server renders to HTML string. No JS needed. Works everywhere.")
(li (strong "Compiled JS") " — server compiles to DOM construction code. "
"Event handlers work. No SX runtime. Kilobytes, not megabytes.")
(li (strong "SX runtime") " — full evaluator + engine. Client-side routing, "
"component caching, reactive islands. The current architecture.")
(li (strong "SX + signals") " — full reactive islands. Fine-grained DOM updates."))
(p "Each layer adds capability and weight. The right layer depends on the page. "
"A blog post needs layer 1. An interactive form needs layer 2. "
"A single-page app needs layer 3. A real-time dashboard needs layer 4. "
(code "js.sx") " makes layer 2 possible — it didn't exist before."))
(~docs/subsection :title "The Bootstrap Completion"
(p "With " (code "py.sx") " and " (code "js.sx") " both written in SX:")
(ul (~tw :tokens "list-disc pl-6 space-y-2 text-stone-700")
(li "The " (em "spec") " defines SX semantics (" (code "eval.sx") ", " (code "render.sx") ", ...)")
(li "The " (em "translators") " convert the spec to host languages (" (code "py.sx") ", " (code "js.sx") ")")
(li "The " (em "prover") " verifies the spec's properties (" (code "z3.sx") ", " (code "prove.sx") ")")
(li "All four are written " (em "in") " SX, executable " (em "by") " any SX evaluator"))
(p "The language defines itself, verifies itself, and compiles itself. "
"The Python and JavaScript \"bootstrappers\" are not programs that produce SX — "
"they are SX programs that produce Python and JavaScript. "
"The arrow points the other way."))))))

110
sx/sx/plans/live-streaming.sx
View File

@@ -1,110 +0,0 @@
;; ---------------------------------------------------------------------------
;; Live Streaming — SSE & WebSocket
;; ---------------------------------------------------------------------------
(defcomp ~plans/live-streaming/plan-live-streaming-content ()
(~docs/page :title "Live Streaming"
(~docs/section :title "Context" :id "context"
(p "SX streaming currently uses chunked transfer encoding: the server sends an HTML shell with "
(code "~shared:pages/suspense") " placeholders, then resolves each one via inline "
(code "<script>__sxResolve(id, sx)</script>") " chunks as IO completes. "
"Once the response finishes, the connection closes. Each slot resolves exactly once.")
(p "This is powerful for initial page load but doesn't support live updates "
"— dashboard metrics, chat messages, collaborative editing, real-time notifications. "
"For that we need a persistent transport: " (strong "SSE") " (Server-Sent Events) or " (strong "WebSockets") ".")
(p "The key insight: the client already has " (code "Sx.resolveSuspense(id, sxSource)") " which replaces "
"DOM content by suspense ID. A persistent connection just needs to keep calling it."))
(~docs/section :title "Design" :id "design"
(~docs/subsection :title "Transport Hierarchy"
(p "Three tiers, progressively more capable:")
(ol (~tw :tokens "list-decimal list-inside space-y-2 text-stone-700 text-sm")
(li (strong "Chunked streaming") " (done) — single HTTP response, each suspense resolves once. "
"Best for: initial page load with slow IO.")
(li (strong "SSE") " — persistent one-way connection, server pushes resolve events. "
"Best for: dashboards, notifications, progress bars, any read-only live data.")
(li (strong "WebSocket") " — bidirectional, client can send events back. "
"Best for: chat, collaborative editing, interactive applications.")))
(~docs/subsection :title "SSE Protocol"
(p "A " (code "~live") " component declares a persistent connection to an SSE endpoint:")
(~docs/code :src (highlight "(~live :src \"/api/stream/dashboard\"\n (~shared:pages/suspense :id \"cpu\" :fallback (span \"Loading...\"))\n (~shared:pages/suspense :id \"memory\" :fallback (span \"Loading...\"))\n (~shared:pages/suspense :id \"requests\" :fallback (span \"Loading...\")))" "lisp"))
(p "The server SSE endpoint yields SX resolve events:")
(~docs/code :src (highlight "async def dashboard_stream():\n while True:\n stats = await get_system_stats()\n yield sx_sse_event(\"cpu\", f'(~stat-badge :value \"{stats.cpu}%\")')\n yield sx_sse_event(\"memory\", f'(~stat-badge :value \"{stats.mem}%\")')\n await asyncio.sleep(1)" "python"))
(p "SSE wire format — each event is a suspense resolve:")
(~docs/code :src (highlight "event: sx-resolve\ndata: {\"id\": \"cpu\", \"sx\": \"(~stat-badge :value \\\"42%\\\")\"}\n\nevent: sx-resolve\ndata: {\"id\": \"memory\", \"sx\": \"(~stat-badge :value \\\"68%\\\")\"}" "text")))
(~docs/subsection :title "WebSocket Protocol"
(p "A " (code "~ws") " component establishes a bidirectional channel:")
(~docs/code :src (highlight "(~ws :src \"/ws/chat\"\n :on-message handle-chat-message\n (~shared:pages/suspense :id \"messages\" :fallback (div \"Connecting...\"))\n (~shared:pages/suspense :id \"typing\" :fallback (span)))" "lisp"))
(p "Client can send SX expressions back:")
(~docs/code :src (highlight ";; Client sends:\n(sx-send ws-conn '(chat-message :text \"hello\" :user \"alice\"))\n\n;; Server receives, broadcasts to all connected clients:\n;; event: sx-resolve for \"messages\" suspense" "lisp")))
(~docs/subsection :title "Shared Resolution Mechanism"
(p "All three transports use the same client-side resolution:")
(ul (~tw :tokens "list-disc list-inside space-y-1 text-stone-600 text-sm")
(li (code "Sx.resolveSuspense(id, sxSource)") " — already exists, parses SX and renders to DOM")
(li "SSE: " (code "EventSource") " → " (code "onmessage") " → " (code "resolveSuspense()"))
(li "WS: " (code "WebSocket") " → " (code "onmessage") " → " (code "resolveSuspense()"))
(li "The component env (defs needed for rendering) can be sent once on connection open")
(li "Subsequent events only need the SX expression — lightweight wire format"))))
(~docs/section :title "Implementation" :id "implementation"
(~docs/subsection :title "Phase 1: SSE Infrastructure"
(ol (~tw :tokens "list-decimal list-inside space-y-2 text-stone-700 text-sm")
(li "Add " (code "~live") " component to " (code "shared/sx/templates/") " — renders child suspense placeholders, "
"emits " (code "data-sx-live") " attribute with SSE endpoint URL")
(li "Add " (code "sx-live.js") " client module — on boot, finds " (code "[data-sx-live]") " elements, "
"opens EventSource, routes events to " (code "resolveSuspense()"))
(li "Add " (code "sx_sse_event(id, sx)") " helper for Python SSE endpoints — formats SSE wire protocol")
(li "Add " (code "sse_stream()") " Quart helper — returns async generator Response with correct headers")))
(~docs/subsection :title "Phase 2: Defpage Integration"
(ol (~tw :tokens "list-decimal list-inside space-y-2 text-stone-700 text-sm")
(li "New " (code ":live") " defpage slot — declares SSE endpoint + suspense bindings")
(li "Auto-mount SSE endpoint alongside the page route")
(li "Component defs sent as first SSE event on connection open")
(li "Automatic reconnection with exponential backoff")))
(~docs/subsection :title "Phase 3: WebSocket"
(ol (~tw :tokens "list-decimal list-inside space-y-2 text-stone-700 text-sm")
(li "Add " (code "~ws") " component — bidirectional channel with send/receive")
(li "Add " (code "sx-ws.js") " client module — WebSocket management, message routing")
(li "Server-side: Quart WebSocket handlers that receive and broadcast SX events")
(li "Client-side: " (code "sx-send") " primitive for sending SX expressions to server")))
(~docs/subsection :title "Phase 4: Spec & Boundary"
(ol (~tw :tokens "list-decimal list-inside space-y-2 text-stone-700 text-sm")
(li "Spec " (code "~live") " and " (code "~ws") " in " (code "render.sx") " (how they render in each mode)")
(li "Add SSE/WS IO primitives to " (code "boundary.sx"))
(li "Bootstrap SSE/WS connection management into " (code "sx-ref.js"))
(li "Spec-level tests for resolve, reconnection, and message routing"))))
(~docs/section :title "Files" :id "files"
(table (~tw :tokens "w-full text-left border-collapse")
(thead
(tr (~tw :tokens "border-b border-stone-200")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Purpose")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/templates/live.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~live component definition"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/static/scripts/sx-live.js")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SSE client — EventSource → resolveSuspense"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/sse.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SSE helpers — event formatting, stream response"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/static/scripts/sx-ws.js")
(td (~tw :tokens "px-3 py-2 text-stone-700") "WebSocket client — bidirectional SX channel"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/render.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Spec: ~live and ~ws rendering in all modes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/boundary.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SSE/WS IO primitive declarations")))))))

650
sx/sx/plans/mother-language.sx
View File

@@ -1,650 +0,0 @@
;; ---------------------------------------------------------------------------
;; Mother Language — SX as its own compiler, OCaml as the substrate
;; ---------------------------------------------------------------------------
(defcomp ~plans/mother-language/plan-mother-language-content ()
(~docs/page :title "Mother Language"
(p (~tw :tokens "text-stone-500 text-sm italic mb-8")
"The ideal language for evaluating the SX core spec is SX itself. "
"The path: OCaml as the initial substrate (closest existing language to what CEK is), "
"Koka as an alternative (compile-time linearity), ultimately a self-hosting SX compiler "
"that emits machine code directly from the spec. One language. Every target.")
;; -----------------------------------------------------------------------
;; The argument
;; -----------------------------------------------------------------------
(~docs/section :title "The Argument" :id "argument"
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "What the evaluator actually does")
(p "The CEK machine is a " (code "state \u2192 state") " loop over sum types. "
"Each step pattern-matches on the Control register, consults the Environment, "
"and transforms the Kontinuation. Every SX expression, every component render, "
"every signal update goes through this loop. It's the hot path.")
(p "The ideal host language is one where this loop compiles to a tight jump table "
"with minimal allocation. That means: algebraic types, pattern matching, "
"persistent data structures, and a native effect system.")
(h4 (~tw :tokens "font-semibold mt-6 mb-2") "Why multiple hosts is the wrong goal")
(p "The current architecture bootstraps the spec to Python, JavaScript, and Rust. "
"Each host has impedance mismatches:")
(ul (~tw :tokens "list-disc list-inside space-y-1 mt-2")
(li (strong "Python") " \u2014 slow. Tree-walk overhead is 100\u20131000x vs native. "
"The async adapter is complex because Python's async model is cooperative, not effect-based.")
(li (strong "JavaScript") " \u2014 no sum types, prototype-based dispatch, GC pauses unpredictable. "
"The bootstrapper works around JS limitations rather than mapping naturally.")
(li (strong "Rust") " \u2014 ownership fights shared immutable trees. Every " (code "Rc<Vec<Value>>")
" is overhead. Closures capturing environments need " (code "Arc") " gymnastics."))
(p "Each host makes the evaluator work, but none make it " (em "natural") ". "
"The translation is structure-" (em "creating") ", not structure-" (em "preserving") ".")
(h4 (~tw :tokens "font-semibold mt-6 mb-2") "The Mother Language is SX")
(p "The spec defines the semantics. The CEK machine is the most explicit form of those semantics. "
"The ideal \"language\" is one that maps 1:1 onto CEK transitions and compiles them to "
"optimal machine code. That language is SX itself \u2014 compiled, not interpreted.")
(p "The hosts (Python, JS, Haskell, Rust) become " (em "platform layers") " \u2014 "
"they provide IO, DOM, database, GPU access. The evaluator itself is always the same "
"compiled SX core, embedded as a native library or WASM module."))
;; -----------------------------------------------------------------------
;; Why OCaml
;; -----------------------------------------------------------------------
(~docs/section :title "Why OCaml" :id "ocaml"
(p "OCaml is the closest existing language to what the CEK machine is. "
"The translation from " (code "cek.sx") " to OCaml is nearly mechanical.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Natural mapping")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "SX concept")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "OCaml primitive")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Notes")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Value (Nil | Num | Str | List | ...)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Algebraic variant type")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Direct mapping, no boxing"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Frame (IfFrame | ArgFrame | MapFrame | ...)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Algebraic variant type")
(td (~tw :tokens "px-3 py-2 text-stone-600") "20+ variants, pattern match dispatch"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Environment (persistent map)")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "Map.S"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "Built-in balanced tree, structural sharing"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Continuation (list of frames)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Immutable list")
(td (~tw :tokens "px-3 py-2 text-stone-600") "cons/match, O(1) push/pop"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "cek-step (pattern match on C)")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "match") " expression")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Compiles to jump table"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "shift/reset")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "perform") " / " (code "continue"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "Native in OCaml 5"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Concurrent CEK (fibers)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Domains + effect handlers")
(td (~tw :tokens "px-3 py-2 text-stone-600") "One fiber per CEK machine"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "Linear continuations")
(td (~tw :tokens "px-3 py-2 text-stone-700") "One-shot continuations (default)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Runtime-enforced, not compile-time")))))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Compilation targets")
(ul (~tw :tokens "list-disc list-inside space-y-1 mt-2")
(li (strong "Native") " \u2014 OCaml's native compiler produces fast binaries, small footprint. "
"Embed in Python via C ABI (ctypes/cffi). Embed in Node via N-API.")
(li (strong "WASM") " \u2014 " (code "wasm_of_ocaml") " is mature (used by Facebook's Flow/Reason). "
"Produces compact " (code ".wasm") " modules. Loaded via " (code "sx-platform.js") " like the Rust WASM plan.")
(li (strong "JavaScript") " \u2014 " (code "js_of_ocaml") " for legacy browser targets. "
"Falls back to JS when WASM isn't available."))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "What OCaml replaces")
(p "The Haskell and Rust evaluator implementations become unnecessary. "
"OCaml covers both server (native) and client (WASM) from one codebase. "
"The sx-haskell and sx-rust work proved the spec is host-independent \u2014 "
"OCaml is the convergence point.")
(p "Python and JavaScript evaluators remain as " (em "platform layers") " \u2014 "
"they provide IO primitives, not evaluation logic. The Python web framework calls "
"the OCaml evaluator via FFI for rendering. The browser loads the WASM evaluator "
"and connects it to " (code "sx-platform.js") " for DOM access."))
;; -----------------------------------------------------------------------
;; Koka as alternative
;; -----------------------------------------------------------------------
(~docs/section :title "Koka as Alternative" :id "koka"
(p "Koka (Daan Leijen, MSR) addresses OCaml's one weakness: "
(strong "compile-time linearity") ".")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Where Koka wins")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Perceus reference counting") " \u2014 the compiler tracks which values are used linearly. "
"Linear values are mutated in-place (zero allocation). "
"Non-linear values use reference counting (no GC at all).")
(li (strong "Effect types") " \u2014 effects are tracked in the type system. "
"A function's type says exactly which effects it can perform. "
"The type checker enforces that handlers exist for every effect.")
(li (strong "One-shot continuations by default") " \u2014 like OCaml 5, but " (em "statically enforced") ". "
"The type system prevents invoking a linear continuation twice. "
"No runtime check needed."))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Where Koka is weaker")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Maturity") " \u2014 research language. Smaller ecosystem, fewer FFI bindings, "
"less battle-tested than OCaml.")
(li (strong "WASM backend") " \u2014 compiles to C \u2192 WASM (via Emscripten). "
"Works but less optimized than " (code "wasm_of_ocaml") " or Rust's " (code "wasm-pack") ".")
(li (strong "Embeddability") " \u2014 C FFI works but less established for Python/Node embedding."))
(p "Koka is the right choice " (em "if") " compile-time linearity proves essential for correctness. "
"The decision point is Step 5 (Linear Effects) of the foundations plan. "
"If SX's own type system can enforce linearity at spec-validation time, "
"OCaml's runtime enforcement is sufficient. If not, Koka becomes the better substrate."))
;; -----------------------------------------------------------------------
;; The path to self-hosting
;; -----------------------------------------------------------------------
(~docs/section :title "The Path to Self-Hosting" :id "self-hosting"
(p "The end state: SX compiles itself. No intermediate language, no general-purpose host.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Phase 1: OCaml bootstrapper")
(p "Write " (code "bootstrap_ml.py") " \u2014 reads " (code "cek.sx") " + " (code "frames.sx")
" + " (code "primitives.sx") " + " (code "eval.sx") ", emits OCaml source. "
"Same pattern as the existing Rust/Python/JS bootstrappers.")
(p "The OCaml output is a standalone module:")
(~docs/code :src (highlight "type value =\n | Nil | Bool of bool | Num of float | Str of string\n | Sym of string | Kw of string\n | List of value list | Dict of (value * value) list\n | Lambda of params * value list * env\n | Component of string * params * value list * env\n | Handle of int (* opaque FFI reference *)\n\ntype frame =\n | IfFrame of value list * value list * env\n | ArgFrame of value list * value list * env\n | MapFrame of value * value list * value list * env\n | ReactiveResetFrame of value\n | DerefFrame of value\n (* ... 20+ frame types from frames.sx *)\n\ntype kont = frame list\ntype state = value * env * kont\n\nlet step ((ctrl, env, kont) : state) : state =\n match ctrl with\n | Lit v -> continue_val v kont\n | Var name -> continue_val (Env.find name env) kont\n | App (f, args) -> (f, env, ArgFrame(args, [], env) :: kont)\n | ..." "ocaml"))
(h4 (~tw :tokens "font-semibold mt-6 mb-2") "Phase 2: Native + WASM builds")
(p "Compile the OCaml output to:")
(ul (~tw :tokens "list-disc list-inside space-y-1 mt-2")
(li (code "sx_core.so") " / " (code "sx_core.dylib") " \u2014 native shared library, C ABI")
(li (code "sx_core.wasm") " \u2014 via " (code "wasm_of_ocaml") " for browser")
(li (code "sx_core.js") " \u2014 via " (code "js_of_ocaml") " as JS fallback"))
(p "Python web framework calls " (code "sx_core.so") " via cffi. "
"Browser loads " (code "sx_core.wasm") " via " (code "sx-platform.js") ".")
(h4 (~tw :tokens "font-semibold mt-6 mb-2") "Phase 3: SX evaluates web framework")
(p "The compiled core evaluator loads web framework " (code ".sx") " at runtime "
"(signals, engine, orchestration, boot). Same as the "
(a :href "/sx/(etc.(plan.isolated-evaluator))" "Isolated Evaluator") " plan, "
"but the evaluator is compiled OCaml/WASM instead of bootstrapped JS.")
(h4 (~tw :tokens "font-semibold mt-6 mb-2") "Phase 4: SX linearity checking")
(p "Extend " (code "types.sx") " with quantity annotations:")
(~docs/code :src (highlight ";; Quantity annotations on types\n(define-type (Signal a) :quantity :affine) ;; use at most once per scope\n(define-type (Channel a) :quantity :linear) ;; must be consumed exactly once\n\n;; Effect declarations with linearity\n(define-io-primitive \"send-message\"\n :params (channel message)\n :quantity :linear\n :effects [io]\n :doc \"Must be handled exactly once.\")\n\n;; The type checker (specced in .sx, compiled to OCaml) validates\n;; linearity at component registration time. Runtime enforcement\n;; by OCaml's one-shot continuations is the safety net." "lisp"))
(p "The type checker runs at spec-validation time. The compiled evaluator "
"executes already-verified code. SX's type system provides the linearity "
"guarantees, not the host language.")
(h4 (~tw :tokens "font-semibold mt-6 mb-2") "Phase 5: Self-hosting compiler")
(p "Write the compiler itself in SX:")
(ul (~tw :tokens "list-disc list-inside space-y-1 mt-2")
(li "Spec the CEK-to-native code generation in " (code ".sx") " files")
(li "The Phase 2 OCaml evaluator compiles the compiler spec")
(li "The compiled compiler can then compile itself")
(li "OCaml becomes the bootstrap language only \u2014 "
"needed once to get the self-hosting loop started, then never touched again"))
(~docs/code :src (highlight ";; The bootstrap chain\n\nStep 0: Python evaluator (existing)\n \u2193 evaluates bootstrap_ml.py\nStep 1: OCaml evaluator (compiled from spec by Python)\n \u2193 evaluates compiler.sx\nStep 2: SX compiler (compiled from .sx by OCaml evaluator)\n \u2193 compiles itself\nStep 3: SX compiler (compiled by itself)\n \u2193 compiles everything\n \u2193 emits native, WASM, JS from .sx spec\n \u2193 OCaml is no longer in the chain" "text"))
(p "At Step 3, the only language is SX. The compiler reads " (code ".sx") " and emits machine code. "
"OCaml was the scaffolding. The scaffolding comes down."))
;; -----------------------------------------------------------------------
;; Concurrent CEK on OCaml 5
;; -----------------------------------------------------------------------
(~docs/section :title "Concurrent CEK on OCaml 5" :id "concurrent-cek"
(p "OCaml 5's concurrency model maps directly onto the "
(a :href "/sx/(etc.(plan.foundations))" "Foundations") " plan's concurrent CEK spec.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Mapping")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "SX primitive")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "OCaml 5")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Characteristic")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "spawn"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "Fiber via " (code "perform Spawn"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "Lightweight, scheduled by effect handler"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "channel"))
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "Eio.Stream"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "Typed, bounded, backpressure"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "yield!"))
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "perform Yield"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "Cooperative, zero-cost"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "select"))
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "Eio.Fiber.any"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "First-to-complete"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "fork-join"))
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "Eio.Fiber.all"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "Structured concurrency"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "DAG scheduler")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Domains + fiber pool")
(td (~tw :tokens "px-3 py-2 text-stone-600") "True parallelism across cores")))))
(p "Each concurrent CEK machine is a fiber. The scheduler is an effect handler. "
"This isn't simulating concurrency \u2014 it's using native concurrency whose mechanism " (em "is") " effects.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "The Art DAG connection")
(p "Art DAG's 3-phase execution (analyze \u2192 plan \u2192 execute) maps onto "
"concurrent CEK + OCaml domains:")
(ul (~tw :tokens "list-disc list-inside space-y-1 mt-2")
(li "Analyze: single CEK machine walks the DAG graph (one fiber)")
(li "Plan: resolve dependencies, topological sort (pure computation)")
(li "Execute: spawn one fiber per independent node, fan out to domains (true parallelism)")
(li "GPU kernels: fiber performs " (code "Gpu_dispatch") " effect, handler calls into GPU via C FFI")))
;; -----------------------------------------------------------------------
;; Linear effects
;; -----------------------------------------------------------------------
(~docs/section :title "Linear Effects" :id "linear-effects"
(p "The linearity axis from foundations. Two enforcement layers:")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Layer 1: SX type system (primary)")
(p "Quantity annotations in " (code "types.sx") " checked at spec-validation time:")
(ul (~tw :tokens "list-disc list-inside space-y-1 mt-2")
(li (code ":linear") " (1) \u2014 must be used exactly once")
(li (code ":affine") " (\u22641) \u2014 may be used at most once (can drop)")
(li (code ":unrestricted") " (\u03c9) \u2014 may be used any number of times"))
(p "Linear effects guarantee: a " (code "send-message") " effect is handled exactly once. "
"A channel is consumed. A resource handle is closed. "
"The type checker proves this before the evaluator ever runs.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Layer 2: Host runtime (safety net)")
(p "OCaml 5's one-shot continuations enforce linearity at runtime. "
"A continuation can only be " (code "continue") "'d once \u2014 second invocation raises an exception. "
"This catches any bugs in the type checker itself.")
(p "If Koka replaces OCaml: compile-time enforcement replaces runtime enforcement. "
"The safety net becomes a proof. Same semantics, stronger guarantees.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Decision point")
(p "When Step 5 (Linear Effects) of the foundations plan is reached:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "If SX's type checker can enforce linearity reliably \u2192 "
(strong "stay on OCaml") ". Runtime one-shot is sufficient.")
(li "If linearity bugs keep slipping through \u2192 "
(strong "switch to Koka") ". Compile-time enforcement closes the gap.")
(li "If the self-hosting compiler (Phase 5) reaches maturity \u2192 "
(strong "it doesn't matter") ". SX compiles itself, the substrate is an implementation detail.")))
;; -----------------------------------------------------------------------
;; Compiled all the way down
;; -----------------------------------------------------------------------
(~docs/section :title "Compiled All the Way Down" :id "compiled"
(p "The self-hosting compiler doesn't just compile the evaluator. "
"It compiles " (em "everything") ". Component definitions, page layouts, "
"event handlers, signal computations \u2014 all compiled to native machine code. "
"SX is not an interpreted scripting language with a nice spec. "
"It's a compiled language whose compiler also runs in the browser.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "JIT in the browser")
(p "The server sends SX (component definitions, page content). "
"The client receives it and " (strong "compiles to WASM and executes") ". "
"Not interprets. Not dispatches bytecodes. Compiles.")
(~docs/code :src (highlight "Server sends: (defcomp ~card (&key title) (div :class \"card\" (h2 title)))\n\nClient does:\n 1. Parse SX source (fast \u2014 it's s-expressions)\n 2. Hash AST \u2192 CID\n 3. Cache hit? Call the already-compiled WASM function\n 4. Cache miss? Compile to WASM, cache by CID, call it\n\nStep 4 is the JIT. The compiler (itself WASM) emits a WASM\nfunction, instantiates it via WebAssembly.instantiate, caches\nthe module by CID. Next time: direct function call, zero overhead." "text"))
(p "This is what V8 does with JavaScript. What LuaJIT does with Lua. "
"The difference: SX's semantics are simpler (no prototype chains, no " (code "this")
" binding, no implicit coercion), so the compiler is simpler. "
"And content-addressing means compiled artifacts are cacheable by CID \u2014 "
"compile once, store forever.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "The compilation tiers")
(~docs/code :src (highlight "Tier 0: .sx source \u2192 tree-walking CEK (correct, slow \u2014 current)\nTier 1: .sx source \u2192 bytecodes \u2192 dispatch loop (correct, fast)\nTier 2: .sx source \u2192 WASM functions \u2192 execute (correct, fastest)\nTier 3: .sx source \u2192 native machine code (ahead-of-time, maximum)" "text"))
(p "Each tier is faster. Tier 1 (bytecodes) is the "
(a :href "/sx/(etc.(plan.wasm-bytecode-vm))" "WASM Bytecode VM")
" plan \u2014 compact wire format, no parse overhead, better cache locality. "
"Tier 2 is JIT \u2014 the compiler emitting WASM functions on the fly. "
"Tier 3 is AOT \u2014 the entire app precompiled. "
"All tiers use the same spec, same platform layer, same platform primitives.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Server-side precompilation")
(p "The server can compile too. Instead of sending SX source for the client to JIT, "
"send precompiled WASM:")
(~docs/code :src (highlight ";; Option A: send SX source, client JIT compiles\nContent-Type: text/sx\n\n(div :class \"card\" (h2 \"hello\"))\n\n;; Option B: send precompiled WASM, client instantiates directly\nContent-Type: application/wasm\nX-Sx-Cid: bafyrei...\n\n<binary WASM module>" "text"))
(p "Option B skips parsing and compilation entirely. The client instantiates "
"the WASM module and calls it. The server did all the work.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Content-addressed compilation cache")
(p "Every " (code ".sx") " expression has a CID. Every compiled artifact has a CID. "
"The mapping is deterministic \u2014 the compiler is a pure function:")
(~docs/code :src (highlight "source CID \u2192 compiled WASM CID\nbafyrei... \u2192 bafyrei...\n\nThis mapping is cacheable everywhere:\n\u2022 Browser cache \u2014 first visitor compiles, second visitor gets cached WASM\n\u2022 CDN \u2014 compiled artifacts served at the edge\n\u2022 IPFS \u2014 content-addressed by definition, globally deduplicated\n\u2022 Local disk \u2014 offline apps work from cached compiled components" "text"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Entire apps as machine code")
(p "The entire application can be ahead-of-time compiled to a WASM binary. "
"Component definitions, page layouts, event handlers, signal computations \u2014 "
"all compiled to native WASM functions. The \"app\" is a " (code ".wasm") " file. "
"The platform layer provides DOM and fetch. Everything in between is compiled machine code.")
(p "The only thing that stays JIT-compiled is truly dynamic content \u2014 "
"user-generated SX, REPL input, " (code "eval") "'d strings. "
"And even those get JIT'd on first use and cached by CID.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "The architecture")
(~docs/code :src (highlight "sx-platform.js \u2190 DOM, fetch, timers (the real world)\n \u2191 calls\nsx-compiler.wasm \u2190 the SX compiler (itself compiled to WASM)\n \u2191 compiles\n.sx source \u2190 received from server / cache / inline\n \u2193 emits\nnative WASM functions \u2190 cached by CID, instantiated on demand\n \u2193 executes\nactual DOM mutations via platform primitives" "text"))
(p "The compiler is WASM. The code it produces is WASM. "
"It's compiled code all the way down. "
"The only interpreter in the system is the CPU."))
;; -----------------------------------------------------------------------
;; Security model
;; -----------------------------------------------------------------------
(~docs/section :title "Security Model" :id "security"
(p "Compiled SX running as WASM is " (em "more secure") " than plain JavaScript, "
"not less. JS has ambient access to the full browser API. "
"WASM + the platform layer means compiled SX code has "
(strong "zero ambient capabilities") " \u2014 every capability is explicitly granted.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Five defence layers")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Layer")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Enforced by")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "What it prevents")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "1. WASM sandbox")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Browser")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Memory isolation, no system calls, no DOM access except via explicit imports. Validated before execution."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "2. Platform capabilities")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "sx-platform.js"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "Compiled code can only call functions you register. No fetch? Can't fetch. No localStorage? Can't read storage. The platform is a capability system."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "3. Content-addressed verification")
(td (~tw :tokens "px-3 py-2 text-stone-700") "CID determinism")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Compiler is deterministic: same source \u2192 same CID. Client can re-compile and verify. Tampered WASM produces wrong CID \u2192 reject."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "4. Per-component attenuation")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Platform scoping")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Different components get different capability subsets. User-generated content gets a locked-down platform \u2014 can render DOM but can't fetch or listen to events."))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "5. Source-first fallback")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client compiler")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Don't trust precompiled WASM? Compile from source locally. The client has the compiler. Precompilation is an optimisation, not a trust requirement.")))))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Content-addressed tamper detection")
(p "The server sends both SX source and precompiled WASM CID. The client can verify:")
(~docs/code :src (highlight ";; Server sends:\nContent-Type: application/wasm\nX-Sx-Source-Cid: bafyrei..source\nX-Sx-Compiled-Cid: bafyrei..compiled\n\n;; Client verifies (optional, configurable):\n1. Hash the WASM binary \u2192 matches X-Sx-Compiled-Cid?\n2. Compile source locally \u2192 produces same compiled CID?\n3. Check manifest of pinned CIDs \u2192 CID is expected?\n\n;; Any mismatch = tampered = reject" "text"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Capability attenuation per component")
(p "The platform scopes capabilities per evaluator instance. "
"App shell gets full access. Third-party or user-generated content gets the minimum:")
(~docs/code :src (highlight "// Full capabilities for the app shell\nplatform.registerAll(appShellCompiler);\n\n// Restricted for user-generated content\nplatform.registerSubset(userContentCompiler, {\n allow: [\"dom-create-element\", \"dom-set-attr\", \"dom-append\",\n \"dom-create-text-node\", \"dom-set-text\"],\n deny: [\"fetch\", \"localStorage\", \"dom-listen\",\n \"dom-set-inner-html\", \"eval\"]\n});\n\n// The restricted compiler's WASM module literally doesn't\n// have imports for the denied functions. Not just blocked\n// at runtime \u2014 absent from the binary." "javascript"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Component manifests")
(p "The app ships with a manifest of expected CIDs for its core components. "
"Like subresource integrity (SRI) but for compiled code:")
(~docs/code :src (highlight ";; Component manifest (shipped with the app, signed)\n{\n \"~card\": \"bafyrei..abc\"\n \"~header\": \"bafyrei..def\"\n \"~nav-item\": \"bafyrei..ghi\"\n}\n\n;; On navigation: server sends component update\n;; Client compiles \u2192 checks CID against manifest\n;; Match = trusted, execute\n;; Mismatch = tampered, reject and report" "text"))
(p "The security model is " (em "structural") ", not bolt-on. "
"WASM isolation, platform capabilities, content-addressed verification, "
"and per-component attenuation all arise naturally from the architecture. "
"The platform layer that enables Rust/OCaml interop is the same layer "
"that enforces security boundaries."))
;; -----------------------------------------------------------------------
;; Isomorphic rendering + SEO
;; -----------------------------------------------------------------------
(~docs/section :title "Isomorphic Rendering" :id "isomorphic"
(p "WASM is invisible to search engines. But SX is already isomorphic \u2014 "
"the same spec, the same components, rendered to HTML on the server "
"and to DOM on the client. Compiled WASM doesn't change this. "
"It makes the client side faster without affecting what crawlers see.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "The rendering pipeline")
(~docs/code :src (highlight "Crawler visits:\n GET /page\n \u2192 Server compiles SX (native OCaml)\n \u2192 render-to-html (adapter-html.sx)\n \u2192 Full static HTML with semantic markup\n \u2192 Google indexes it\n\nUser first visit:\n GET /page\n \u2192 Server renders HTML (same as crawler)\n \u2192 Browser displays immediately (no JS needed)\n \u2192 Client loads sx-compiler.wasm + sx-platform.js\n \u2192 Hydrates: attaches event handlers, activates islands\n \u2192 Page is interactive\n\nUser navigates (SPA):\n sx-get /next-page\n \u2192 Server sends SX wire format (aser)\n \u2192 Client compiles + renders via WASM\n \u2192 Morph engine patches the DOM" "text"))
(p "The server and client have the " (em "same compiler") " from the " (em "same spec") ". "
(code "adapter-html.sx") " produces HTML strings. "
(code "adapter-dom.sx") " produces DOM nodes. "
"Two rendering modes of one evaluator. The compiled WASM version "
"makes hydration and SPA navigation faster, but the initial HTML "
"is always server-rendered.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "What crawlers see")
(ul (~tw :tokens "list-disc list-inside space-y-1 mt-2")
(li "Fully rendered HTML \u2014 no \"loading...\" skeleton, no JS-dependent content")
(li "Semantic markup \u2014 " (code "<h1>") ", " (code "<nav>") ", " (code "<article>") ", " (code "<a href>")
" \u2014 all from the SX component tree")
(li "Meta tags, canonical URLs, structured data \u2014 rendered server-side by " (code "shell.sx"))
(li "No WASM, no JS required \u2014 the HTML is the page, complete on first byte"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Content-addressed prerendering")
(p "The server can prerender every page to static HTML, hash it, "
"and cache it at the CDN edge:")
(~docs/code :src (highlight "Page source CID \u2192 Rendered HTML CID\nbafyrei..source \u2192 bafyrei..html\n\n\u2022 Crawler hits CDN \u2192 instant HTML, no server round-trip\n\u2022 Page content changes \u2192 new source CID \u2192 new HTML CID \u2192 CDN invalidated\n\u2022 Same CID = same HTML forever \u2192 infinite cache, zero revalidation" "text"))
(p "This is the same content-addressed caching as compiled WASM, "
"applied to the HTML output. Both the compiled client code and "
"the server-rendered HTML are cached by CID. "
"The entire delivery pipeline is content-addressed.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Progressive enhancement")
(p "The page works at every level:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Client capability")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Experience")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "No JS (crawler, reader mode)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Full HTML. Links work. Forms submit. Content is complete."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "JS, no WASM")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Falls back to js_of_ocaml evaluator or interpreted JS. SPA navigation, islands, signals all work."))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "JS + WASM")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Full compiled pipeline. JIT compilation, cached WASM functions, near-native rendering speed."))))))
;; -----------------------------------------------------------------------
;; WASM responses — htmx with compiled components
;; -----------------------------------------------------------------------
(~docs/section :title "WASM Responses" :id "wasm-responses"
(p "The server doesn't have to send HTML fragments for HTMX-style swaps. "
"It can send compiled WASM functions that produce DOM directly, "
"or \u2014 when the client already has the components cached \u2014 "
"just the data as gzipped bytecode.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Three response tiers")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Client cache state")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Server sends")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Client does")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Nothing cached")
(td (~tw :tokens "px-3 py-2 text-stone-700") "HTML (fallback)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Parse HTML, morph DOM"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Compiler cached")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SX source")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Compile, execute, morph DOM"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components cached")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Bytecode (data only)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Call cached WASM function with args"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components cached + gzip")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Gzipped bytecode")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Decompress, call cached function")))))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Why bytecode is tiny")
(p "The bytecode doesn't repeat what the client already knows. "
"Component CID references are 4-byte pointers to already-cached compiled functions. "
"Tag names, class strings, attribute keys \u2014 all baked into the compiled component. "
"The only payload is the data that differs between instances.")
(~docs/code :src (highlight ";; HTML response: ~450 bytes\n<div class=\"card\"><h2>Hello</h2><p class=\"text-stone-600\">World</p></div>\n\n;; SX source response: ~60 bytes\n(~card :title \"Hello\" :body \"World\")\n\n;; Bytecode response: ~18 bytes\n[CALL_CID bafyrei..card] [STR \"Hello\"] [STR \"World\"]\n\n;; Gzipped bytecode: ~12 bytes" "text"))
(p "For a list of 50 cards:")
(~docs/code :src (highlight "HTML: 50\u00d7 <div class=\"card\"><h2>...</h2>...</div> ~22 KB\nSX source: 50\u00d7 (~card :title \"...\" :body \"...\") ~3 KB\nBytecode: [CALL_CID] + 50\u00d7 [STR, STR] ~800 bytes\nGzipped bytecode: ~400 bytes" "text"))
(p "The markup structure is in the compiled component. "
"The class strings are in the compiled component. "
"The only thing on the wire is the data that differs between instances. "
"Gzip crushes the repetitive framing to almost nothing.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Content negotiation")
(p "The client advertises what it has via request headers:")
(~docs/code :src (highlight ";; Client tells server what's cached\nAccept: application/sx-bytecode, text/sx, text/html\nX-Sx-Cached-Components: bafyrei..card, bafyrei..header, bafyrei..nav\n\n;; Server picks the smallest response:\n;; - Client has ~card cached? Send bytecode (data only)\n;; - Client has compiler but not ~card? Send SX source\n;; - Client has nothing? Send HTML" "text"))
(p "The server and client negotiate the optimal response format automatically. "
"First visit is HTML (full progressive enhancement). "
"Subsequent navigations are gzipped bytecode \u2014 "
"just the data, a few hundred bytes, instant render via cached compiled components.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "The server becomes a data API")
(p "At the bytecode tier, the server is no longer rendering views. "
"It's serving data \u2014 structured arguments for compiled client-side functions. "
"The rendering logic is cached on the client as compiled WASM. "
"This inverts the traditional server-rendered model without sacrificing "
"progressive enhancement \u2014 the HTML fallback is always available."))
;; -----------------------------------------------------------------------
;; How this changes existing plans
;; -----------------------------------------------------------------------
(~docs/section :title "Impact on Existing Plans" :id "impact"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Plan")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Impact")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700")
(a :href "/sx/(etc.(plan.rust-wasm-host))" "Rust/WASM Host"))
(td (~tw :tokens "px-3 py-2 text-stone-600")
"Superseded. OCaml/WASM replaces Rust/WASM. The handle table and platform layer design carry over."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700")
(a :href "/sx/(etc.(plan.isolated-evaluator))" "Isolated Evaluator"))
(td (~tw :tokens "px-3 py-2 text-stone-600")
"Architecture preserved. sx-platform.js and evaluator isolation apply to the OCaml evaluator too. The JS evaluator becomes a fallback."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700")
(a :href "/sx/(etc.(plan.foundations))" "Foundations"))
(td (~tw :tokens "px-3 py-2 text-stone-600")
"Accelerated. OCaml 5 has native effects/continuations/fibers. Steps 4 (Concurrent CEK) and 5 (Linear Effects) map directly."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700")
(a :href "/sx/(etc.(plan.wasm-bytecode-vm))" "WASM Bytecode VM"))
(td (~tw :tokens "px-3 py-2 text-stone-600")
"Subsumed as Tier 1. Bytecodes become an intermediate step on the path to native WASM compilation. The dispatch loop is Tier 1; JIT to WASM functions is Tier 2; AOT is Tier 3."))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700")
(a :href "/sx/(etc.(plan.self-hosting-bootstrapper))" "Self-Hosting Bootstrapper"))
(td (~tw :tokens "px-3 py-2 text-stone-600")
"Converges. js.sx and py.sx are self-hosting emitters. The self-hosting SX compiler (Phase 5 here) is the logical endpoint."))))))
;; -----------------------------------------------------------------------
;; Principles
;; -----------------------------------------------------------------------
(~docs/section :title "Principles" :id "principles"
(ul (~tw :tokens "list-disc list-inside space-y-2")
(li (strong "SX is the language.") " Not OCaml, not Rust, not Haskell. "
"Those are substrates. SX defines the semantics, SX checks the types, "
"SX will ultimately compile itself.")
(li (strong "OCaml is scaffolding.") " The closest existing language to CEK. "
"Used to bootstrap the self-hosting compiler. Comes down when the compiler is mature.")
(li (strong "The spec is the compiler's input.") " " (code "cek.sx") ", " (code "frames.sx")
", " (code "primitives.sx") " \u2014 the same files that define the language "
"become the compiler's source. One truth, one artifact.")
(li (strong "Platforms provide effects, not evaluation.") " Python provides database/HTTP. "
"JavaScript provides DOM. GPU provides tensor ops. "
"The evaluator is always compiled SX, embedded via FFI or WASM.")
(li (strong "Linearity belongs in the spec.") " SX's type system checks it. "
"The host provides runtime backup. If the type system is sound, the runtime check never fires.")
(li (strong "One language, every target.") " Not \"one spec, multiple implementations.\" "
"One " (em "compiled") " evaluator, deployed as native/.wasm/.js depending on context. "
"The evaluator binary is the same code everywhere. Only the platform layer changes.")))
;; -----------------------------------------------------------------------
;; Outcome
;; -----------------------------------------------------------------------
(~docs/section :title "Outcome" :id "outcome"
(p "After completion:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "SX core compiled from spec via OCaml \u2014 native + WASM from one codebase.")
(li "Python web framework calls native " (code "sx_core.so") " for rendering \u2014 "
"100\u20131000x faster than interpreted Python.")
(li "Browser loads " (code "sx_core.wasm") " \u2014 same compiler, same spec, near-native speed.")
(li "Server sends SX, client JIT-compiles to WASM functions and caches by CID.")
(li "Entire apps AOT-compiled to " (code ".wasm") " binaries. "
"Platform provides DOM/fetch. Everything else is machine code.")
(li "Content-addressed compilation cache: compile once, cache by CID, serve from CDN/IPFS forever.")
(li "Concurrent CEK runs on OCaml 5 fibers/domains \u2014 true parallelism for Art DAG.")
(li "Linear effects validated by SX type system, enforced by OCaml one-shot continuations.")
(li "Self-hosting compiler: SX compiles itself to machine code. OCaml scaffolding removed.")
(li "The only interpreter in the system is the CPU."))
(p (~tw :tokens "text-stone-500 text-sm italic mt-12")
"The Mother Language was always SX. We just needed to find the right scaffolding to stand it up."))))

245
sx/sx/plans/nav-redesign.sx
View File

@@ -1,245 +0,0 @@
;; ---------------------------------------------------------------------------
;; Navigation Redesign — SX Docs
;; ---------------------------------------------------------------------------
(defcomp ~plans/nav-redesign/plan-nav-redesign-content ()
(~docs/page :title "Navigation Redesign"
(~docs/section :title "The Problem" :id "problem"
(p "The current navigation is a horizontal menu bar system: root bar, sx bar, sub-section bar. 13 top-level sections crammed into a scrolling horizontal row. Hover to see dropdowns. Click a section, get a second bar underneath. Click a page, get a third bar. Three stacked bars eating vertical space on every page.")
(p "It's a conventional web pattern and it's bad for this site. SX docs has a deep hierarchy — sections contain subsections contain pages. Horizontal bars can't express depth. They flatten everything into one level and hide the rest behind hover states that don't work on mobile, that obscure content, that require spatial memory of where things are.")
(p "The new nav is vertical, hierarchical, and infinite. No dropdowns. No menu bars. Just a centered breadcrumb trail that expands downward as you drill in."))
;; -----------------------------------------------------------------------
;; Design
;; -----------------------------------------------------------------------
(~docs/section :title "Design" :id "design"
(~docs/subsection :title "Structure"
(p "One vertical column, centered. Each level is a row.")
(~docs/code :src (highlight ";; Home (nothing selected)\n;;\n;; [ sx ]\n;;\n;; Docs CSSX Reference Protocols Examples\n;; Essays Philosophy Specs Bootstrappers\n;; Testing Isomorphism Plans Reactive Islands\n\n\n;; Section selected (e.g. Plans)\n;;\n;; [ sx ]\n;;\n;; < Plans >\n;;\n;; Status Reader Macros Theorem Prover\n;; Self-Hosting JS Bootstrapper SX-Activity\n;; Predictive Prefetching Content-Addressed\n;; Environment Images Runtime Slicing Typed SX\n;; Fragment Protocol ...\n\n\n;; Page selected (e.g. Typed SX under Plans)\n;;\n;; [ sx ]\n;;\n;; < Plans >\n;;\n;; < Typed SX >\n;;\n;; [ page content here ]" "lisp")))
(~docs/subsection :title "Rules"
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-2")
(li (strong "Logo at top, centered.") " Always visible. Click = home. The only fixed element.")
(li (strong "Level 1: section list.") " Shown on home page as a wrapped, centered list of links. This is the full menu — no hiding, no hamburger.")
(li (strong "When a section is selected:") " Section name replaces the list. Left arrow and right arrow for sibling navigation (previous/next section). The section's children appear as a new list below.")
(li (strong "When a child is selected:") " Same pattern — child name replaces the list, arrows for siblings, sub-children appear below. Recurse ad infinitum.")
(li (strong "Breadcrumb trail.") " Each selected level stays visible as a single row above the current level. The trail is: logo → section → subsection → page. Each row has arrows. Click any ancestor to navigate up.")
(li (strong "No dropdowns.") " Never. Hover does nothing special. The hierarchy is always visible in the breadcrumb trail.")
(li (strong "No hamburger menu.") " The nav IS the page on the home/index views. On content pages, the breadcrumb trail is compact enough to show without hiding.")
(li (strong "Responsive by default.") " Vertical + centered + wrapped = works at any width. No breakpoint-specific layout needed."))))
;; -----------------------------------------------------------------------
;; Visual language
;; -----------------------------------------------------------------------
(~docs/section :title "Visual Language" :id "visual"
(~docs/subsection :title "Levels"
(p "Each level has decreasing visual weight:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Level")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Selected state")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "List state")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Logo")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Large, violet, always visible")
(td (~tw :tokens "px-3 py-2 text-stone-600") "—"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Section")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Medium text, violet-700, arrows")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Medium text, stone-600, wrapped inline"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Subsection")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Smaller text, violet-600, arrows")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Small text, stone-500, wrapped inline"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Level 3+")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Same as subsection")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Same as subsection"))))))
(~docs/subsection :title "Arrows"
(p "Left and right arrows are inline with the selected item name. They navigate to the previous/next sibling in the current list. Keyboard accessible: left/right arrow keys when the row is focused.")
(~docs/code :src (highlight ";; Arrow rendering\n;;\n;; < Plans >\n;;\n;; < is a link to /plans/content-addressed-components\n;; (the previous sibling in plans-nav-items)\n;; > is a link to /plans/fragment-protocol\n;; (the next sibling)\n;; \"Plans\" is a link to /plans/ (the section index)\n;;\n;; At the edges, the arrow wraps:\n;; first item: < wraps to last\n;; last item: > wraps to first" "lisp")))
(~docs/subsection :title "Transitions"
(p "Selecting an item: the list fades/collapses, the selected item moves to breadcrumb position, children appear below. This is an L0 morph — the server renders the new state, the client morphs. No JS animation library needed, just CSS transitions on the morph targets.")
(p "Going up: click an ancestor in the breadcrumb. Its children (the level below) expand back into a list. Reverse of the drill-down.")))
;; -----------------------------------------------------------------------
;; Data model
;; -----------------------------------------------------------------------
(~docs/section :title "Data Model" :id "data"
(p "The current nav data is flat — each section has its own " (code "define") ". The new model is a single tree:")
(~docs/code :src (highlight "(define sx-nav-tree\n {:label \"sx\"\n :href \"/\"\n :children (list\n {:label \"Docs\"\n :href \"/language/docs/introduction\"\n :children docs-nav-items}\n {:label \"CSSX\"\n :href \"/applications/cssx/\"\n :children cssx-nav-items}\n {:label \"Reference\"\n :href \"/reference/\"\n :children reference-nav-items}\n {:label \"Protocols\"\n :href \"/applications/protocols/wire-format\"\n :children protocols-nav-items}\n {:label \"Examples\"\n :href \"/examples/click-to-load\"\n :children examples-nav-items}\n {:label \"Essays\"\n :href \"/etc/essays/\"\n :children essays-nav-items}\n {:label \"Philosophy\"\n :href \"/etc/philosophy/sx-manifesto\"\n :children philosophy-nav-items}\n {:label \"Specs\"\n :href \"/language/specs/\"\n :children specs-nav-items}\n {:label \"Bootstrappers\"\n :href \"/language/bootstrappers/\"\n :children bootstrappers-nav-items}\n {:label \"Testing\"\n :href \"/language/testing/\"\n :children testing-nav-items}\n {:label \"Isomorphism\"\n :href \"/geography/isomorphism/\"\n :children isomorphism-nav-items}\n {:label \"Plans\"\n :href \"/etc/plans/\"\n :children plans-nav-items}\n {:label \"Reactive Islands\"\n :href \"/reactive-islands/\"\n :children reactive-islands-nav-items})})" "lisp"))
(p "The existing per-section lists (" (code "docs-nav-items") ", " (code "plans-nav-items") ", etc.) remain unchanged — they just become the " (code ":children") " of tree nodes. Sub-sections that have their own sub-items can nest further:")
(~docs/code :src (highlight ";; Future: deeper nesting\n{:label \"Plans\"\n :href \"/etc/plans/\"\n :children (list\n {:label \"Status\" :href \"/etc/plans/status\"}\n {:label \"Bootstrappers\" :href \"/etc/plans/self-hosting-bootstrapper\"\n :children (list\n {:label \"py.sx\" :href \"/etc/plans/self-hosting-bootstrapper\"}\n {:label \"js.sx\" :href \"/etc/plans/js-bootstrapper\"})}\n ;; ...\n )}" "lisp"))
(p "The tree depth is unlimited. The nav component recurses."))
;; -----------------------------------------------------------------------
;; Components
;; -----------------------------------------------------------------------
(~docs/section :title "Components" :id "components"
(p "Three new components replace the entire menu bar system:")
(~docs/subsection :title "~plans/nav-redesign/logo"
(~docs/code :src (highlight "(defcomp ~plans/nav-redesign/logo ()\n (a :href \"/\"\n :sx-get \"/\" :sx-target \"#main-panel\" :sx-select \"#main-panel\"\n :sx-swap \"outerHTML\" :sx-push-url \"true\"\n :class \"block text-center py-4\"\n (span :class \"text-2xl font-bold text-violet-700\" \"sx\")))" "lisp"))
(p "Always at the top. Always centered. The anchor."))
(~docs/subsection :title "~plans/nav-redesign/nav-breadcrumb"
(~docs/code :src (highlight "(defcomp ~plans/nav-redesign/nav-breadcrumb (&key path siblings level)\n ;; Renders one breadcrumb row: < Label >\n ;; path = the nav tree node for this level\n ;; siblings = list of sibling nodes (for arrow nav)\n ;; level = depth (controls text size/color)\n (let ((idx (find-index siblings path))\n (prev (nth siblings (mod (- idx 1) (len siblings))))\n (next (nth siblings (mod (+ idx 1) (len siblings)))))\n (div :class (str \"flex items-center justify-center gap-3 py-1\"\n (nav-level-classes level))\n (a :href (get prev \"href\")\n :sx-get (get prev \"href\") :sx-target \"#main-panel\"\n :sx-select \"#main-panel\" :sx-swap \"outerHTML\"\n :sx-push-url \"true\"\n :class \"text-stone-400 hover:text-violet-600\"\n :aria-label \"Previous\"\n \"<\")\n (a :href (get path \"href\")\n :sx-get (get path \"href\") :sx-target \"#main-panel\"\n :sx-select \"#main-panel\" :sx-swap \"outerHTML\"\n :sx-push-url \"true\"\n :class \"font-medium\"\n (get path \"label\"))\n (a :href (get next \"href\")\n :sx-get (get next \"href\") :sx-target \"#main-panel\"\n :sx-select \"#main-panel\" :sx-swap \"outerHTML\"\n :sx-push-url \"true\"\n :class \"text-stone-400 hover:text-violet-600\"\n :aria-label \"Next\"\n \">\"))))" "lisp"))
(p "One row per selected level. Shows the current node with left/right arrows to siblings."))
(~docs/subsection :title "~plans/nav-redesign/nav-list"
(~docs/code :src (highlight "(defcomp ~plans/nav-redesign/nav-list (&key items level)\n ;; Renders a wrapped list of links — the children of the current level\n (div :class (str \"flex flex-wrap justify-center gap-x-4 gap-y-2 py-2\"\n (nav-level-classes level))\n (map (fn (item)\n (a :href (get item \"href\")\n :sx-get (get item \"href\") :sx-target \"#main-panel\"\n :sx-select \"#main-panel\" :sx-swap \"outerHTML\"\n :sx-push-url \"true\"\n :class \"hover:text-violet-700 transition-colors\"\n (get item \"label\")))\n items)))" "lisp"))
(p "The children of the current level, rendered as a centered wrapped list of plain links."))
(~docs/subsection :title "~plans/nav-redesign/nav — the composition"
(~docs/code :src (highlight "(defcomp ~plans/nav-redesign/nav (&key trail children-items level)\n ;; trail = list of {node, siblings} from root to current\n ;; children-items = children of the deepest selected node\n ;; level = depth of children\n (div :class \"max-w-3xl mx-auto px-4\"\n ;; Logo\n (~plans/nav-redesign/logo)\n ;; Breadcrumb trail (one row per selected ancestor)\n (map-indexed (fn (i crumb)\n (~nav-breadcrumb\n :path (get crumb \"node\")\n :siblings (get crumb \"siblings\")\n :level (+ i 1)))\n trail)\n ;; Children of the deepest selected node\n (when children-items\n (~plans/nav-redesign/nav-list :items children-items :level level))))" "lisp"))
(p "That's the entire navigation. Three small components composed. No bars, no dropdowns, no mobile variants.")))
;; -----------------------------------------------------------------------
;; Path resolution
;; -----------------------------------------------------------------------
(~docs/section :title "Path Resolution" :id "resolution"
(p "Given a URL path, compute the breadcrumb trail and children. This is a tree walk:")
(~docs/code :src (highlight "(define resolve-nav-path\n (fn (tree current-href)\n ;; Walk sx-nav-tree, find the node matching current-href,\n ;; return the trail of ancestors + current children.\n ;;\n ;; Returns: {:trail (list of {:node N :siblings S})\n ;; :children (list) or nil\n ;; :depth number}\n ;;\n ;; Example: current-href = \"/etc/plans/typed-sx\"\n ;; → trail: [{:node Plans :siblings [Docs, CSSX, ...]}\n ;; {:node Typed-SX :siblings [Status, Reader-Macros, ...]}]\n ;; → children: nil (leaf node)\n ;; → depth: 2\n (let ((result (walk-nav-tree tree current-href (list))))\n result)))" "lisp"))
(p "This runs server-side (it's a pure function, no IO). The layout component calls it with the current URL and passes the result to " (code "~plans/nav-redesign/nav") ". Same pattern as the current " (code "find-current") " but produces a richer result.")
(p "For sx-get navigations (HTMX swaps), the server re-renders the nav with the new path. The morph diffs the old and new nav — breadcrumb rows appear/disappear, the list changes. CSS transitions handle the visual."))
;; -----------------------------------------------------------------------
;; What goes away
;; -----------------------------------------------------------------------
(~docs/section :title "What Goes Away" :id "removal"
(p "Significant deletion:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Component")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Why")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "~shared:layout/menu-row-sx")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/templates/layout.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Horizontal bar with colour levels — replaced by breadcrumb rows"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "~sx-header-row")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "sx/sx/layouts.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Top menu bar — replaced by logo + breadcrumb"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "~sx-sub-row")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "sx/sx/layouts.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Sub-section bar — replaced by second breadcrumb row"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "~sx-main-nav")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "sx/sx/layouts.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Horizontal nav list — replaced by ~plans/nav-redesign/nav-list"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "~nav-data/section-nav")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "sx/sx/nav-data.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Sub-nav builder — replaced by ~plans/nav-redesign/nav-list"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "~shared:layout/nav-link")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/templates/layout.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Complex link with aria-selected + submenu wrapper — replaced by plain a tags"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "~shared:layout/mobile-menu-section")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/templates/layout.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Separate mobile menu — new nav is inherently responsive"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "6 layout variants")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "sx/sx/layouts.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "full/oob/mobile × home/section — replaced by one layout with ~plans/nav-redesign/nav"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") ".nav-group CSS")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/templates/shell.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Hover submenu CSS — no submenus to hover")))))
(p "The layout variants collapse from 6 (full/oob/mobile × home/section) to 2 (full/oob). No mobile variant needed — the nav is one column, it works everywhere."))
;; -----------------------------------------------------------------------
;; Layout simplification
;; -----------------------------------------------------------------------
(~docs/section :title "Layout Simplification" :id "layout"
(p "The defpage layout declarations currently specify section, sub-label, sub-href, sub-nav, selected — five params to configure two menu bars. The new layout takes one param: the nav trail.")
(~docs/code :src (highlight ";; Current (verbose, configures two bars)\n(defpage plan-page\n :path \"/etc/plans/<slug>\"\n :layout (:sx-section\n :section \"Plans\"\n :sub-label \"Plans\"\n :sub-href \"/etc/plans/\"\n :sub-nav (~nav-data/section-nav :items plans-nav-items\n :current (find-current plans-nav-items slug))\n :selected (or (find-current plans-nav-items slug) \"\"))\n :content (...))\n\n;; New (one param, nav computed from URL)\n(defpage plan-page\n :path \"/etc/plans/<slug>\"\n :layout (:sx-docs :path (str \"/etc/plans/\" slug))\n :content (...))" "lisp"))
(p "The layout component computes the nav trail internally from the path and the nav tree. No more passing section names, sub-labels, or pre-built nav components through layout params.")
(~docs/code :src (highlight "(defcomp ~plans/nav-redesign/docs-layout-full (&key path)\n (let ((nav-state (resolve-nav-path sx-nav-tree path)))\n (<> (~root-header-auto)\n (~sx-nav\n :trail (get nav-state \"trail\")\n :children-items (get nav-state \"children\")\n :level (get nav-state \"depth\")))))\n\n(defcomp ~plans/nav-redesign/docs-layout-oob (&key path)\n (let ((nav-state (resolve-nav-path sx-nav-tree path)))\n (<> (~oob-nav\n :trail (get nav-state \"trail\")\n :children-items (get nav-state \"children\")\n :level (get nav-state \"depth\"))\n (~root-header-auto true))))" "lisp"))
(p "Two layout components instead of twelve. Every defpage in docs.sx simplifies from five layout params to one."))
;; -----------------------------------------------------------------------
;; Scope
;; -----------------------------------------------------------------------
(~docs/section :title "Scope" :id "scope"
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-4 mb-4")
(p (~tw :tokens "text-amber-900 font-medium") "SX docs only — for now")
(p (~tw :tokens "text-amber-800") "This redesign applies to the SX docs app (" (code "sx/") "). The other services (blog, market, events, etc.) keep their current navigation. If the pattern proves out, it can migrate to shared infrastructure and replace the root menu system too."))
(p "What changes:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "sx/sx/nav-data.sx") " — add " (code "sx-nav-tree") " (wraps existing lists, no content change)")
(li (code "sx/sx/layouts.sx") " — rewrite: delete 12 components, add 5 (logo, breadcrumb, list, nav, 2 layouts)")
(li (code "sx/sxc/pages/docs.sx") " — simplify every defpage's " (code ":layout") " declaration")
(li (code "sx/sxc/pages/layouts.py") " — register new layout names")
(li (code "shared/sx/templates/layout.sx") " — no changes needed (shared components untouched, only SX-docs-specific ones change)"))
(p "What doesn't change:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Page content — all " (code "~plan-*-content") ", " (code "~doc-*-content") ", etc. are untouched")
(li "Nav data — all " (code "*-nav-items") " lists are unchanged, just composed into a tree")
(li "Routing — all defpage paths stay the same")
(li "Other services — blog, market, etc. unaffected")))
;; -----------------------------------------------------------------------
;; Implementation
;; -----------------------------------------------------------------------
(~docs/section :title "Implementation" :id "implementation"
(~docs/subsection :title "Phase 1: Nav tree + resolution"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Add " (code "sx-nav-tree") " to " (code "nav-data.sx") " — compose existing " (code "*-nav-items") " lists into a tree")
(li "Write " (code "resolve-nav-path") " — pure function, tree walk, returns trail + children")
(li "Test: given a path, produces the correct breadcrumb trail and child list")))
(~docs/subsection :title "Phase 2: New components"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Write " (code "~plans/nav-redesign/logo") ", " (code "~plans/nav-redesign/nav-breadcrumb") ", " (code "~plans/nav-redesign/nav-list") ", " (code "~plans/nav-redesign/nav"))
(li "Write " (code "~plans/nav-redesign/docs-layout-full") " and " (code "~plans/nav-redesign/docs-layout-oob"))
(li "Register new layout in " (code "layouts.py"))
(li "Test with one defpage first — verify morph transitions work")))
(~docs/subsection :title "Phase 3: Migrate all defpages"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Update every defpage in " (code "docs.sx") " to use " (code ":layout (:sx-docs :path ...)"))
(li "This is mechanical — replace the 5-param layout block with 1-param")))
(~docs/subsection :title "Phase 4: Delete old components"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Delete " (code "~sx-main-nav") ", " (code "~sx-header-row") ", " (code "~sx-sub-row") ", " (code "~nav-data/section-nav"))
(li "Delete all 12 SX layout variants from " (code "layouts.sx"))
(li "Delete old layout registrations from " (code "layouts.py"))
(li "Remove " (code ".nav-group") " CSS if no other service uses it"))))))

257
sx/sx/plans/predictive-prefetch.sx
View File

@@ -1,257 +0,0 @@
;; ---------------------------------------------------------------------------
;; Predictive Component Prefetching
;; ---------------------------------------------------------------------------
(defcomp ~plans/predictive-prefetch/plan-predictive-prefetch-content ()
(~docs/page :title "Predictive Component Prefetching"
(~docs/section :title "Context" :id "context"
(p "Phase 3 of the isomorphic roadmap added client-side routing with component dependency checking. When a user clicks a link, " (code "try-client-route") " checks " (code "has-all-deps?") " — if the target page needs components not yet loaded, the client falls back to a server fetch. This works correctly but misses an opportunity: " (strong "we can prefetch those missing components before the click happens."))
(p "The page registry already carries " (code ":deps") " metadata for every page. The client already knows which components are loaded via " (code "loaded-component-names") ". The gap is a mechanism to " (em "proactively") " resolve the difference — fetching missing component definitions so that by the time the user clicks, client-side routing succeeds.")
(p "But this goes beyond just hover-to-prefetch. The full spectrum includes: bundling linked routes' components with the initial page load, batch-prefetching after idle, predicting mouse trajectory toward links, and even splitting the component/data fetch so that " (code ":data") " pages can prefetch their components and only fetch data on click. Each strategy trades bandwidth for latency, and pages should be able to declare which tradeoff they want."))
(~docs/section :title "Current State" :id "current-state"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Layer")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "What exists")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Where")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Page registry")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Each page carries " (code ":deps (\"~card\" \"~essay-foo\" ...)"))
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "helpers.py → <script type=\"text/sx-pages\">"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Dep check")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "has-all-deps?") " gates client routing")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "orchestration.sx:546-559"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Component bundle")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Per-page inline " (code "<script type=\"text/sx\" data-components>"))
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "helpers.py:715, jinja_bridge.py"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Incremental defs")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "components_for_request()") " sends only missing defs in SX responses")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "helpers.py:459-509"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Preload cache")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "sx-preload") " prefetches full responses on hover/mousedown")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "orchestration.sx:686-708"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Route matching")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "find-matching-route") " matches pathname to page entry")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "router.sx"))))))
;; -----------------------------------------------------------------------
;; Prefetch strategies
;; -----------------------------------------------------------------------
(~docs/section :title "Prefetch Strategies" :id "strategies"
(p "Prefetching is a spectrum from conservative to aggressive. The system should support all of these, configured declaratively per link or per page via " (code "defpage") " metadata and " (code "sx-prefetch") " attributes.")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Strategy")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Trigger")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "What prefetches")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Latency on click")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Eager bundle")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Initial page load")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components for linked routes included in " (code "<script data-components>"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "Zero — already in memory"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Idle timer")
(td (~tw :tokens "px-3 py-2 text-stone-700") "After page settles (requestIdleCallback or setTimeout)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components for visible nav links, batched in one request")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Zero if idle fetch completed"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Viewport")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Link scrolls into view (IntersectionObserver)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components for that link's route")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Zero if user scrolled before clicking"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Mouse approach")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Cursor moving toward link (trajectory prediction)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components for predicted target")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Near-zero — fetch starts ~200ms before hover"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Hover")
(td (~tw :tokens "px-3 py-2 text-stone-700") "mouseover (150ms debounce)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components for hovered link's route")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Low — typical hover-to-click is 300-500ms"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Mousedown")
(td (~tw :tokens "px-3 py-2 text-stone-700") "mousedown (0ms debounce)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components for clicked link's route")
(td (~tw :tokens "px-3 py-2 text-stone-600") "~80ms — mousedown-to-click gap"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "Components + data")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Any of the above")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Components " (em "and") " page data for " (code ":data") " pages")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Zero for components; data fetch may still be in flight")))))
(~docs/subsection :title "Eager Bundle"
(p "The server already computes per-page component bundles. For key navigation paths — the main nav bar, section nav — the server can include " (em "linked routes' components") " in the initial bundle, not just the current page's.")
(~docs/code :src (highlight ";; defpage metadata declares eager prefetch targets\n(defpage docs-page\n :path \"/language/docs/<slug>\"\n :auth :public\n :prefetch :eager ;; bundle deps for all linked pure routes\n :content (case slug ...))" "lisp"))
(p "Implementation: " (code "components_for_page()") " already scans the page SX for component refs. Extend it to also scan for " (code "href") " attributes, match them against the page registry, and include those pages' deps in the bundle. The cost is a larger initial payload; the benefit is zero-latency navigation within a section."))
(~docs/subsection :title "Idle Timer"
(p "After page load and initial render, use " (code "requestIdleCallback") " (or a fallback " (code "setTimeout") ") to scan visible nav links and batch-prefetch their missing components in a single request.")
(~docs/code :src (highlight "(define prefetch-visible-links-on-idle\n (fn ()\n (request-idle-callback\n (fn ()\n (let ((links (dom-query-all \"a[href][sx-get]\"))\n (all-missing (list)))\n (for-each\n (fn (link)\n (let ((missing (compute-missing-deps\n (url-pathname (dom-get-attr link \"href\")))))\n (when missing\n (for-each (fn (d) (append! all-missing d))\n missing))))\n links)\n (when (not (empty? all-missing))\n (prefetch-components (dedupe all-missing))))))))" "lisp"))
(p "Called once from " (code "boot-init") " after initial processing. Batches all missing deps into one network request. Low priority — browser handles it when idle."))
(~docs/subsection :title "Mouse Approach (Trajectory Prediction)"
(p "Don't wait for the cursor to reach the link — predict where it's heading. Track the last few " (code "mousemove") " events, extrapolate the trajectory, and if it points toward a link, start prefetching before the hover event fires.")
(~docs/code :src (highlight "(define bind-approach-prefetch\n (fn (container)\n ;; Track mouse trajectory within a nav container.\n ;; On each mousemove, extrapolate position ~200ms ahead.\n ;; If projected point intersects a link's bounding box,\n ;; prefetch that link's route deps.\n (let ((last-x 0) (last-y 0) (last-t 0)\n (prefetched (dict)))\n (dom-add-listener container \"mousemove\"\n (fn (e)\n (let ((now (timestamp))\n (dt (- now last-t)))\n (when (> dt 16) ;; ~60fps throttle\n (let ((vx (/ (- (event-x e) last-x) dt))\n (vy (/ (- (event-y e) last-y) dt))\n (px (+ (event-x e) (* vx 200)))\n (py (+ (event-y e) (* vy 200)))\n (target (dom-element-at-point px py)))\n (when (and target (dom-has-attr? target \"href\")\n (not (get prefetched\n (dom-get-attr target \"href\"))))\n (let ((href (dom-get-attr target \"href\")))\n (set! prefetched\n (merge prefetched {href true}))\n (prefetch-route-deps\n (url-pathname href)))))\n (set! last-x (event-x e))\n (set! last-y (event-y e))\n (set! last-t now))))))))" "lisp"))
(p "This is the most speculative strategy — best suited for dense navigation areas (section sidebars, nav bars) where the cursor trajectory is a strong predictor. The " (code "prefetched") " dict prevents duplicate fetches within the same container interaction."))
(~docs/subsection :title "Components + Data (Hybrid Prefetch)"
(p "The most interesting strategy. For pages with " (code ":data") " dependencies, current behavior is full server fallback. But the page's " (em "components") " are still pure and prefetchable. If we prefetch components ahead of time, the click only needs to fetch " (em "data") " — a much smaller, faster response.")
(p "This creates a new rendering path:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-1")
(li "Prefetch: hover/idle/viewport triggers " (code "prefetch-components") " for the target page")
(li "Click: client has components, but page has " (code ":data") " — fetch data from server")
(li "Server returns " (em "only data") " (JSON or SX bindings), not the full rendered page")
(li "Client evaluates the content expression with prefetched components + fetched data")
(li "Result: faster than full server render, no redundant component transfer"))
(~docs/code :src (highlight ";; Declarative: prefetch components, fetch data on click\n(defpage reference-page\n :path \"/reference/<slug>\"\n :auth :public\n :prefetch :components ;; prefetch components, data stays server-fetched\n :data (reference-data slug)\n :content (~reference/attrs-content :attrs attrs))\n\n;; On click, client-side flow:\n;; 1. Components already prefetched (from hover/idle)\n;; 2. GET /reference/attributes → server returns data bindings\n;; 3. Client evals (reference-data slug) result + content expr\n;; 4. Renders locally with cached components" "lisp"))
(p "This is a stepping stone toward full Phase 4 (client IO bridge) of the isomorphic roadmap — it achieves partial client rendering for data pages without needing a general-purpose client async evaluator. The server is a data service, the client is the renderer."))
(~docs/subsection :title "Declarative Configuration"
(p "All strategies configured via " (code "defpage") " metadata and " (code "sx-prefetch") " attributes on links/containers:")
(~docs/code :src (highlight ";; Page-level: what to prefetch for routes linking TO this page\n(defpage docs-page\n :path \"/language/docs/<slug>\"\n :prefetch :eager) ;; bundle with linking page\n\n(defpage reference-page\n :path \"/reference/<slug>\"\n :prefetch :components) ;; prefetch components, data on click\n\n;; Link-level: override per-link\n(a :href \"/language/docs/components\"\n :sx-prefetch \"idle\") ;; prefetch after page idle\n\n;; Container-level: approach prediction for nav areas\n(nav :sx-prefetch \"approach\"\n (a :href \"/language/docs/\") (a :href \"/reference/\") ...)" "lisp"))
(p "Priority cascade: explicit " (code "sx-prefetch") " on link > " (code ":prefetch") " on target defpage > default (hover). The system never prefetches the same components twice — " (code "_prefetch-pending") " and " (code "loaded-component-names") " handle dedup.")))
;; -----------------------------------------------------------------------
;; Design
;; -----------------------------------------------------------------------
(~docs/section :title "Implementation Design" :id "design"
(p "Per the SX host architecture principle: all SX-specific logic goes in " (code ".sx") " spec files and gets bootstrapped. The prefetch logic — scanning links, computing missing deps, managing the component cache — must be specced in " (code ".sx") ", not written directly in JS or Python.")
(~docs/subsection :title "Phase 1: Component Fetch Endpoint (Python)"
(p "A new " (strong "public") " endpoint (not " (code "/internal/") " — the client's browser calls it) that returns component definitions by name.")
(~docs/code :src (highlight "GET /<service-prefix>/sx/components?names=~plans/predictive-prefetch/card,~essay-foo\n\nResponse (text/sx):\n(defcomp ~plans/predictive-prefetch/card (&key title &rest children)\n (div :class \"border rounded p-4\" (h2 title) children))\n(defcomp ~plans/predictive-prefetch/essay-foo (&key id)\n (div (~plans/predictive-prefetch/card :title id)))" "http"))
(p "The server resolves transitive deps via " (code "deps.py") ", subtracts anything listed in the " (code "SX-Components") " request header (already loaded), serializes and returns. This is essentially " (code "components_for_request()") " driven by an explicit " (code "?names=") " param.")
(p "Cache-friendly: the response is a pure function of component hash + requested names. " (code "Cache-Control: public, max-age=3600") " with the component hash as ETag."))
(~docs/subsection :title "Phase 2: Client Prefetch Logic (SX spec)"
(p "New functions in " (code "orchestration.sx") " (or a new " (code "prefetch.sx") " if scope warrants):")
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. compute-missing-deps")
(p "Given a pathname, find the page, return dep names not in " (code "loaded-component-names") ". Returns nil if page not found or has data (can't client-route anyway).")
(~docs/code :src (highlight "(define compute-missing-deps\n (fn (pathname)\n (let ((match (find-matching-route pathname _page-routes)))\n (when (and match (not (get match \"has-data\")))\n (let ((deps (or (get match \"deps\") (list)))\n (loaded (loaded-component-names)))\n (filter (fn (d) (not (contains? loaded d))) deps))))))" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. prefetch-components")
(p "Fetch component definitions from the server for a list of names. Deduplicates in-flight requests. On success, parses and registers the returned definitions into the component env.")
(~docs/code :src (highlight "(define _prefetch-pending (dict))\n\n(define prefetch-components\n (fn (names)\n (let ((key (join \",\" (sort names))))\n (when (not (get _prefetch-pending key))\n (set! _prefetch-pending\n (merge _prefetch-pending {key true}))\n (fetch-components-from-server names\n (fn (sx-text)\n (sx-process-component-text sx-text)\n (dict-remove! _prefetch-pending key)))))))" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. prefetch-route-deps")
(p "High-level composition: compute missing deps for a route, fetch if any.")
(~docs/code :src (highlight "(define prefetch-route-deps\n (fn (pathname)\n (let ((missing (compute-missing-deps pathname)))\n (when (and missing (not (empty? missing)))\n (log-info (str \"sx:prefetch \"\n (len missing) \" components for \" pathname))\n (prefetch-components missing)))))" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "4. Trigger: link hover")
(p "On mouseover of a boosted link, prefetch its route's missing components. Debounced 150ms to avoid fetching on quick mouse-throughs.")
(~docs/code :src (highlight "(define bind-prefetch-on-hover\n (fn (link)\n (let ((timer nil))\n (dom-add-listener link \"mouseover\"\n (fn (e)\n (clear-timeout timer)\n (set! timer (set-timeout\n (fn () (prefetch-route-deps\n (url-pathname (dom-get-attr link \"href\"))))\n 150))))\n (dom-add-listener link \"mouseout\"\n (fn (e) (clear-timeout timer))))))" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "5. Trigger: viewport intersection (opt-in)")
(p "More aggressive strategy: when a link scrolls into view, prefetch its route's deps. Opt-in via " (code "sx-prefetch=\"visible\"") " attribute.")
(~docs/code :src (highlight "(define bind-prefetch-on-visible\n (fn (link)\n (observe-intersection link\n (fn () (prefetch-route-deps\n (url-pathname (dom-get-attr link \"href\"))))\n true 0)))" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "6. Integration into process-elements")
(p "During the existing hydration pass, for each boosted link:")
(~docs/code :src (highlight ";; In process-elements, after binding boost behavior:\n(when (and (should-boost-link? link)\n (dom-get-attr link \"href\"))\n (bind-prefetch-on-hover link))\n\n;; Explicit viewport prefetch:\n(when (dom-has-attr? link \"sx-prefetch\")\n (bind-prefetch-on-visible link))" "lisp")))))
(~docs/subsection :title "Phase 3: Boundary Declaration"
(p "Two new IO primitives in " (code "boundary.sx") " (browser-only):")
(~docs/code :src (highlight ";; IO primitives (browser-only)\n(io fetch-components-from-server (names callback) -> void)\n(io sx-process-component-text (sx-text) -> void)" "lisp"))
(p "These are thin wrappers around " (code "fetch()") " + the existing component script processing logic already in the boundary adapter."))
(~docs/subsection :title "Phase 4: Bootstrap"
(p (code "bootstrap_js.py") " picks up the new functions from the spec and emits them into " (code "sx-browser.js") ". The two new boundary IO functions get implemented in the JS boundary adapter — the hand-written glue code that the bootstrapper doesn't generate.")
(~docs/code :src (highlight "// fetch-components-from-server: calls the endpoint\nfunction fetchComponentsFromServer(names, callback) {\n const url = `${routePrefix}/sx/components?names=${names.join(\",\")}`;\n const headers = {\n \"SX-Components\": loadedComponentNames().join(\",\")\n };\n fetch(url, { headers })\n .then(r => r.ok ? r.text() : \"\")\n .then(text => callback(text))\n .catch(() => {}); // silent fail — prefetch is best-effort\n}\n\n// sx-process-component-text: parse defcomp/defmacro into env\nfunction sxProcessComponentText(sxText) {\n if (!sxText) return;\n const frag = document.createElement(\"div\");\n frag.innerHTML =\n `<script type=\"text/sx\" data-components>${sxText}<\\/script>`;\n Sx.processScripts(frag);\n}" "javascript"))))
;; -----------------------------------------------------------------------
;; Request flow
;; -----------------------------------------------------------------------
(~docs/section :title "Request Flow" :id "request-flow"
(p "End-to-end example: user hovers a link, components prefetch, click goes client-side.")
(~docs/code :src (highlight "User hovers link \"/language/docs/sx-manifesto\"\n |\n +-- bind-prefetch-on-hover fires (150ms debounce)\n |\n +-- compute-missing-deps(\"/language/docs/sx-manifesto\")\n | +-- find-matching-route -> page with deps:\n | | [\"~essay-sx-manifesto\", \"~doc-code\"]\n | +-- loaded-component-names -> [\"~nav\", \"~footer\", \"~doc-code\"]\n | +-- missing: [\"~essay-sx-manifesto\"]\n |\n +-- prefetch-components([\"~essay-sx-manifesto\"])\n | +-- GET /sx/components?names=~essay-sx-manifesto\n | | Headers: SX-Components: ~plans/environment-images/nav,~footer,~doc-code\n | +-- Server resolves transitive deps\n | | (also needs ~plans/predictive-prefetch/rich-text, subtracts already-loaded)\n | +-- Response:\n | (defcomp ~plans/predictive-prefetch/essay-sx-manifesto ...) \n | (defcomp ~plans/predictive-prefetch/rich-text ...)\n |\n +-- sx-process-component-text registers defcomps in env\n |\n +-- User clicks link\n +-- try-client-route(\"/language/docs/sx-manifesto\")\n +-- has-all-deps? -> true (prefetched!)\n +-- eval content -> DOM\n +-- Client-side render, no server roundtrip" "text")))
;; -----------------------------------------------------------------------
;; File changes
;; -----------------------------------------------------------------------
(~docs/section :title "File Changes" :id "file-changes"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Change")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Phase")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/helpers.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "New " (code "sx_components_endpoint()") " route handler")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/infrastructure/factory.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Register " (code "/sx/components") " route on all SX apps")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/orchestration.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Prefetch functions: compute-missing-deps, prefetch-components, prefetch-route-deps, bind-prefetch-on-hover, bind-prefetch-on-visible")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/boundary.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Declare " (code "fetch-components-from-server") ", " (code "sx-process-component-text"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/bootstrap_js.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Emit new spec functions, boundary adapter stubs")
(td (~tw :tokens "px-3 py-2 text-stone-600") "4"))))))
;; -----------------------------------------------------------------------
;; Non-goals & rollout
;; -----------------------------------------------------------------------
(~docs/section :title "Non-Goals (This Phase)" :id "non-goals"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Analytics-driven prediction") " — no ML models or click-frequency heuristics. Trajectory prediction uses geometry, not statistics.")
(li (strong "Cross-service prefetch") " — components are per-service. A link to a different service domain is always a server navigation.")
(li (strong "Service worker caching") " — could layer on later, but basic fetch + in-memory registration is sufficient.")
(li (strong "Full client-side data evaluation") " — the components+data strategy fetches data from the server, it doesn't replicate server IO on the client. That's Phase 4 of the isomorphic roadmap.")))
(~docs/section :title "Rollout" :id "rollout"
(p "Incremental, each step independently valuable:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-2")
(li (strong "Component endpoint") " — purely additive. Refactor " (code "components_for_request()") " to accept explicit " (code "?names=") " param.")
(li (strong "Core spec functions") " — " (code "compute-missing-deps") ", " (code "prefetch-components") ", " (code "prefetch-route-deps") " in orchestration.sx. Testable in isolation.")
(li (strong "Hover prefetch") " — wire " (code "bind-prefetch-on-hover") " into " (code "process-elements") ". All boosted links get it automatically. Console logs show activity.")
(li (strong "Idle batch prefetch") " — call " (code "prefetch-visible-links-on-idle") " from " (code "boot-init") ". One request prefetches all visible nav deps after page settles.")
(li (strong "Viewport + approach") " — opt-in via " (code "sx-prefetch") " attributes. Trajectory prediction for dense nav areas.")
(li (strong "Eager bundles") " — extend " (code "components_for_page()") " to include linked routes' deps. Heavier initial payload, zero-latency nav.")
(li (strong "Components + data split") " — new server response mode returning data bindings only. Client renders with prefetched components. Bridges toward Phase 4.")))
(~docs/section :title "Relationship to Isomorphic Roadmap" :id "relationship"
(p "This plan sits between Phase 3 (client-side routing) and Phase 4 (client async & IO bridge) of the "
(a :href "/sx/(etc.(plan.isomorphic-architecture))" (~tw :tokens "text-violet-700 underline") "isomorphic architecture roadmap")
". It extends Phase 3 by making more navigations go client-side without needing any IO bridge — purely by ensuring component definitions are available before they're needed.")
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-3 mt-2")
(p (~tw :tokens "text-amber-800 text-sm") (strong "Depends on: ") "Phase 3 (client-side routing with deps checking). No dependency on Phase 4.")))))
;; ---------------------------------------------------------------------------
;; Plan Status Overview
;; ---------------------------------------------------------------------------

110
sx/sx/plans/reader-macros.sx
View File

@@ -1,110 +0,0 @@
;; ---------------------------------------------------------------------------
;; Reader Macros
;; ---------------------------------------------------------------------------
(defcomp ~plans/reader-macros/plan-reader-macros-content ()
(~docs/page :title "Reader Macros"
(~docs/section :title "Context" :id "context"
(p "SX has three hardcoded reader transformations: " (code "`") " → " (code "(quasiquote ...)") ", " (code ",") " → " (code "(unquote ...)") ", " (code ",@") " → " (code "(splice-unquote ...)") ". These are baked into the parser with no extensibility. The " (code "~") " prefix for components and " (code "&") " for param modifiers are just symbol characters, handled at eval time.")
(p "Reader macros add parse-time transformations triggered by a dispatch character. Motivating use case: a " (code "~md") " component that uses heredoc syntax for markdown source instead of string literals. More broadly: datum comments, raw strings, and custom literal syntax."))
(~docs/section :title "Design" :id "design"
(~docs/subsection :title "Dispatch Character: #"
(p "Lisp tradition. " (code "#") " is NOT in " (code "ident-start") " or " (code "ident-char") " — completely free. Pattern:")
(~docs/code :src (highlight "#;expr → (read and discard expr, return next)\n#|...| → raw string literal\n#'expr → (quote expr)" "lisp")))
(~docs/subsection :title "#; — Datum comment"
(p "Scheme/Racket standard. Reads and discards the next expression. Preserves balanced parens.")
(~docs/code :src (highlight "(list 1 #;(this is commented out) 2 3) → (list 1 2 3)" "lisp")))
(~docs/subsection :title "#|...| — Raw string"
(p "No escape processing. Everything between " (code "#|") " and " (code "|") " is literal. Enables inline markdown, regex patterns, code examples.")
(~docs/code :src (highlight "(~md #|## Title\n\nSome **bold** text with \"quotes\" and \\backslashes.|)" "lisp")))
(~docs/subsection :title "#' — Quote shorthand"
(p "Currently no single-char quote (" (code "`") " is quasiquote).")
(~docs/code :src (highlight "#'my-function → (quote my-function)" "lisp")))
(~docs/subsection :title "Extensible dispatch: #name"
(p "User-defined reader macros via " (code "#name expr") ". The parser reads an identifier after " (code "#") ", looks up a handler in the reader macro registry, and calls it with the next parsed expression. See the " (a :href "/sx/(etc.(plan.reader-macro-demo))" (~tw :tokens "text-violet-600 hover:underline") "#z3 demo") " for a working example that translates SX spec declarations to SMT-LIB.")))
;; -----------------------------------------------------------------------
;; Implementation
;; -----------------------------------------------------------------------
(~docs/section :title "Implementation" :id "implementation"
(~docs/subsection :title "1. Spec: parser.sx"
(p "Add " (code "#") " dispatch to " (code "read-expr") " (after the " (code ",") "/" (code ",@") " case, before number). Add " (code "read-raw-string") " helper function.")
(~docs/code :src (highlight ";; Reader macro dispatch\n(= ch \"#\")\n (do (set! pos (inc pos))\n (if (>= pos len-src)\n (error \"Unexpected end of input after #\")\n (let ((dispatch-ch (nth source pos)))\n (cond\n ;; #; — datum comment: read and discard next expr\n (= dispatch-ch \";\")\n (do (set! pos (inc pos))\n (read-expr) ;; read and discard\n (read-expr)) ;; return the NEXT expr\n\n ;; #| — raw string\n (= dispatch-ch \"|\")\n (do (set! pos (inc pos))\n (read-raw-string))\n\n ;; #' — quote shorthand\n (= dispatch-ch \"'\")\n (do (set! pos (inc pos))\n (list (make-symbol \"quote\") (read-expr)))\n\n :else\n (error (str \"Unknown reader macro: #\" dispatch-ch))))))" "lisp"))
(p "The " (code "read-raw-string") " helper:")
(~docs/code :src (highlight "(define read-raw-string\n (fn ()\n (let ((buf \"\"))\n (define raw-loop\n (fn ()\n (if (>= pos len-src)\n (error \"Unterminated raw string\")\n (let ((ch (nth source pos)))\n (if (= ch \"|\")\n (do (set! pos (inc pos)) nil) ;; done\n (do (set! buf (str buf ch))\n (set! pos (inc pos))\n (raw-loop)))))))\n (raw-loop)\n buf)))" "lisp")))
(~docs/subsection :title "2. Python: parser.py"
(p "Add " (code "#") " dispatch to " (code "_parse_expr()") " (after " (code ",") "/" (code ",@") " handling ~line 252). Add " (code "_read_raw_string()") " method to Tokenizer.")
(~docs/code :src (highlight "# In _parse_expr(), after the comma/splice-unquote block:\nif raw == \"#\":\n tok._advance(1) # consume the #\n if tok.pos >= len(tok.text):\n raise ParseError(\"Unexpected end of input after #\",\n tok.pos, tok.line, tok.col)\n dispatch = tok.text[tok.pos]\n if dispatch == \";\":\n tok._advance(1)\n _parse_expr(tok) # read and discard\n return _parse_expr(tok) # return next\n if dispatch == \"|\":\n tok._advance(1)\n return tok._read_raw_string()\n if dispatch == \"'\":\n tok._advance(1)\n return [Symbol(\"quote\"), _parse_expr(tok)]\n raise ParseError(f\"Unknown reader macro: #{dispatch}\",\n tok.pos, tok.line, tok.col)" "python"))
(p "The " (code "_read_raw_string()") " method on Tokenizer:")
(~docs/code :src (highlight "def _read_raw_string(self) -> str:\n buf = []\n while self.pos < len(self.text):\n ch = self.text[self.pos]\n if ch == \"|\":\n self._advance(1)\n return \"\".join(buf)\n buf.append(ch)\n self._advance(1)\n raise ParseError(\"Unterminated raw string\",\n self.pos, self.line, self.col)" "python")))
(~docs/subsection :title "3. JS: auto-transpiled"
(p "JS parser comes from bootstrap of parser.sx — spec change handles it automatically."))
(~docs/subsection :title "4. Rebootstrap both targets"
(p "Run " (code "bootstrap_js.py") " and " (code "bootstrap_py.py") " to regenerate " (code "sx-ref.js") " and " (code "sx_ref.py") " from the updated parser.sx spec."))
(~docs/subsection :title "5. Grammar update"
(p "Add reader macro syntax to the grammar comment at the top of parser.sx:")
(~docs/code :src (highlight ";; reader → '#;' expr (datum comment)\n;; | '#|' raw-chars '|' (raw string)\n;; | \"#'\" expr (quote shorthand)" "lisp"))))
;; -----------------------------------------------------------------------
;; Files
;; -----------------------------------------------------------------------
(~docs/section :title "Files" :id "files"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Change")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/parser.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "# dispatch in read-expr, read-raw-string helper, grammar comment"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/parser.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "# dispatch in _parse_expr(), _read_raw_string()"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/sx_ref.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Rebootstrap"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/static/scripts/sx-ref.js")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Rebootstrap"))))))
;; -----------------------------------------------------------------------
;; Verification
;; -----------------------------------------------------------------------
(~docs/section :title "Verification" :id "verification"
(~docs/subsection :title "Parse tests"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 font-mono text-sm")
(li "#;(ignored) 42 → 42")
(li "(list 1 #;2 3) → (list 1 3)")
(li "#|hello \"world\" \\n| → string: hello \"world\" \\n (literal, no escaping)")
(li "#|multi\\nline| → string with actual newline")
(li "#'foo → (quote foo)")
(li "# at EOF → error")
(li "#x unknown → error")))
(~docs/subsection :title "Regression"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "All existing parser tests pass after changes")
(li "Rebootstrapped JS and Python pass their test suites")
(li "JS parity: SX.parse('#|hello|') returns [\"hello\"]"))))))
;; ---------------------------------------------------------------------------
;; Reader Macro Demo: #z3 — SX Spec to SMT-LIB
;; ---------------------------------------------------------------------------

282
sx/sx/plans/runtime-slicing.sx
View File

@@ -1,282 +0,0 @@
;; ---------------------------------------------------------------------------
;; Runtime Slicing
;; ---------------------------------------------------------------------------
(defcomp ~plans/runtime-slicing/plan-runtime-slicing-content ()
(~docs/page :title "Runtime Slicing"
(~docs/section :title "The Problem" :id "problem"
(p "sx-browser.js is the full SX client runtime — evaluator, parser, renderer, engine, morph, signals, routing, orchestration, boot. Every page loads all of it.")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Raw")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Gzipped")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Min+Gz")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "sx-browser.js")
(td (~tw :tokens "px-3 py-2 text-stone-700") "354KB")
(td (~tw :tokens "px-3 py-2 text-stone-700") "75KB")
(td (~tw :tokens "px-3 py-2 text-stone-700") "44KB"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "sx-ref.js")
(td (~tw :tokens "px-3 py-2 text-stone-700") "244KB")
(td (~tw :tokens "px-3 py-2 text-stone-700") "49KB")
(td (~tw :tokens "px-3 py-2 text-stone-700") "29KB"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "React + ReactDOM")
(td (~tw :tokens "px-3 py-2 text-stone-700") "—")
(td (~tw :tokens "px-3 py-2 text-stone-700") "—")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~5KB + ~40KB")))))
(p "Most pages are L0 (pure hypermedia — server renders, client morphs). They don't need the parser, the evaluator, the full primitive set, signals, or client-side routing. They need morph + swap + trigger dispatch. That's a fraction of the runtime.")
(p "The runtime should be sliceable: each page declares what level it operates at, and the bootstrapper emits only the code that level requires."))
;; -----------------------------------------------------------------------
;; Tiers
;; -----------------------------------------------------------------------
(~docs/section :title "Tiers" :id "tiers"
(p "Four tiers, matching the " (a :href "/sx/(geography.(reactive.(reactive-design)))" (~tw :tokens "text-violet-700 underline") "reactive islands") " levels:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Tier")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "What")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Modules")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Target (min+gz)")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "L0 Hypermedia")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Morph, swap, trigger dispatch, history")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-600") "engine, boot (partial)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~5KB"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "L1 DOM Ops")
(td (~tw :tokens "px-3 py-2 text-stone-700") "L0 + toggle!, set-attr!, on-event, class-list ops")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-600") "+ DOM adapter (partial)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~8KB"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "L2 Islands")
(td (~tw :tokens "px-3 py-2 text-stone-700") "L1 + signals, computed, effect, defisland hydration")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-600") "+ signals, DOM adapter (full)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~15KB"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "L3 Full Eval")
(td (~tw :tokens "px-3 py-2 text-stone-700") "L2 + parser, evaluator, all primitives, client routing, component resolution")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-600") "everything")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~44KB (current)")))))
(p "90% of typical pages are L0. A blog post, a product listing, a checkout form — server renders, client morphs on navigation. The full evaluator only loads for pages that do client-side rendering or have reactive islands.")
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-4 mt-4")
(p (~tw :tokens "text-amber-900 font-medium") "Progressive loading")
(p (~tw :tokens "text-amber-800") "A user navigating an L0 page downloads ~5KB. If they navigate to an L2 page (reactive island), the delta (~10KB) loads on demand. The runtime grows with need, never speculatively.")))
;; -----------------------------------------------------------------------
;; The slicer is SX
;; -----------------------------------------------------------------------
(~docs/section :title "The Slicer is SX" :id "slicer-is-sx"
(p "Per the " (a :href "/sx/(etc.(plan.self-hosting-bootstrapper))" (~tw :tokens "text-violet-700 underline") "self-hosting principle") ", the slicer is not a build tool — it's a spec module. " (code "slice.sx") " analyzes the spec's own dependency graph and determines which defines belong to which tier.")
(p (code "js.sx") " (the self-hosting SX-to-JavaScript translator) already compiles the full spec. Slicing is a filter: " (code "js.sx") " translates only the defines that " (code "slice.sx") " selects for a given tier.")
(~docs/code :src (highlight ";; slice.sx — determine which defines each tier needs\n;;\n;; Input: the full list of defines from all spec files\n;; Output: a filtered list for the requested tier\n\n(define tier-deps\n ;; Which spec modules each tier requires\n {:L0 (list \"engine\" \"boot-partial\")\n :L1 (list \"engine\" \"boot-partial\" \"dom-partial\")\n :L2 (list \"engine\" \"boot-partial\" \"dom-partial\"\n \"signals\" \"dom-island\")\n :L3 (list \"eval\" \"render\" \"parser\"\n \"engine\" \"orchestration\" \"boot\"\n \"dom\" \"signals\" \"router\")})\n\n(define slice-defines\n (fn (tier all-defines)\n ;; 1. Get the module list for this tier\n ;; 2. Walk each define's dependency references\n ;; 3. Include a define only if ALL its deps are\n ;; satisfiable within the tier's module set\n ;; 4. Return the filtered define list\n (let ((modules (get tier-deps tier)))\n (filter\n (fn (d) (tier-satisfies? modules (define-deps d)))\n all-defines))))" "lisp"))
(p "The pipeline becomes:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-2")
(li (code "slice.sx") " analyzes the spec and produces a define list per tier")
(li (code "js.sx") " translates each define list to JavaScript (same translator, different input)")
(li "The bootstrapper wraps each tier's output with its platform interface (the hand-written JS glue)")
(li "Output: one " (code ".js") " file per tier, or a single file with tier markers for runtime splitting"))
(p "Because " (code "js.sx") " is self-hosting, slicing is just function composition: " (code "(js-translate-file (slice-defines :L0 all-defines))") ". No new translator. No new build tool. The same 1,382-line " (code "js.sx") " that produces the full runtime produces every tier."))
;; -----------------------------------------------------------------------
;; Dependency analysis
;; -----------------------------------------------------------------------
(~docs/section :title "Define-Level Dependency Analysis" :id "deps"
(p "The slicer needs to know which defines reference which other defines. This is a simpler version of what " (code "deps.sx") " does for components — but at the define level, not the component level.")
(~docs/code :src (highlight ";; Walk a define's body AST, collect all symbol references\n;; that resolve to other top-level defines.\n;;\n;; (define morph-children\n;; (fn (old-node new-node)\n;; ...uses morph-node, morph-attrs, create-element...))\n;;\n;; → deps: #{morph-node morph-attrs create-element}\n\n(define define-refs\n (fn (body all-define-names)\n ;; Collect all symbols in body that appear in all-define-names\n (let ((refs (make-set)))\n (walk-ast body\n (fn (node)\n (when (and (symbol? node)\n (set-has? all-define-names (symbol-name node)))\n (set-add! refs (symbol-name node)))))\n refs)))" "lisp"))
(p "From these per-define deps, we build the full dependency graph and compute transitive closures. A tier's define set is the transitive closure of its entry points:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Tier")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Entry points")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Pulls in")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "L0")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-600") "morph-node, process-swap, dispatch-trigger, push-url")
(td (~tw :tokens "px-3 py-2 text-stone-700") "morph-attrs, morph-children, create-element, extract-swap-config — the morph/swap subgraph"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "L1")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-600") "L0 + toggle-class!, set-attr!, add-event-listener!")
(td (~tw :tokens "px-3 py-2 text-stone-700") "DOM manipulation helpers — small subgraph"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "L2")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-600") "L1 + signal, deref, reset!, computed, effect, render-dom-island")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Signal runtime + reactive DOM adapter"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-800") "L3")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-600") "L2 + eval-expr, sx-parse, render-to-dom, resolve-component-by-cid")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Everything — full evaluator, parser, all ~80 primitives"))))))
;; -----------------------------------------------------------------------
;; Platform interface slicing
;; -----------------------------------------------------------------------
(~docs/section :title "Platform Interface Slicing" :id "platform"
(p "The bootstrapped code (from js.sx) is only half the story. Each define also depends on platform primitives — the hand-written JS glue that js.sx doesn't produce. These live in " (code "bootstrap_js.py") "'s PLATFORM_* blocks.")
(p "The slicer must track platform deps too:")
(~docs/code :src (highlight ";; Platform primitives referenced by tier\n;;\n;; L0 needs: createElement, setAttribute, morphAttrs,\n;; fetch (for sx-get/post), pushState, replaceState\n;;\n;; L1 adds: classList.toggle, addEventListener\n;;\n;; L2 adds: createComment, createTextNode, domRemove,\n;; domChildNodes — the reactive DOM primitives\n;;\n;; L3 adds: everything in PLATFORM_PARSER_JS,\n;; full DOM adapter, async IO bridge\n\n(define platform-deps\n {:L0 (list :dom-morph :fetch :history)\n :L1 (list :dom-morph :fetch :history :dom-events)\n :L2 (list :dom-morph :fetch :history :dom-events\n :dom-reactive :signal-constructors)\n :L3 (list :dom-morph :fetch :history :dom-events\n :dom-reactive :signal-constructors\n :parser :evaluator :async-io)})" "lisp"))
(p "The platform JS blocks in " (code "bootstrap_js.py") " are already organized by adapter (" (code "PLATFORM_DOM_JS") ", " (code "PLATFORM_ENGINE_PURE_JS") ", etc). Slicing further subdivides these into the minimal set each tier needs.")
(p "This subdivision also happens in SX: " (code "slice.sx") " declares which platform blocks each tier requires. " (code "js.sx") " doesn't need to change — it translates defines. The bootstrapper script reads the slice spec and assembles the platform preamble accordingly."))
;; -----------------------------------------------------------------------
;; Progressive loading
;; -----------------------------------------------------------------------
(~docs/section :title "Progressive Loading" :id "progressive"
(p "The simplest approach: one file per tier. The server knows each page's tier (from " (code "defpage") " metadata or component analysis) and serves the right script tag.")
(p "Better: a base file (L0) that all pages load, plus tier deltas loaded on demand.")
(~docs/code :src (highlight ";; Server emits the appropriate script for the page's tier\n;;\n;; L0 page (blog post, product listing):\n;; <script src=\"/static/scripts/sx-L0.js\"></script>\n;;\n;; L2 page (reactive island):\n;; <script src=\"/static/scripts/sx-L0.js\"></script>\n;; <script src=\"/static/scripts/sx-L2-delta.js\"></script>\n;;\n;; Client-side navigation from L0 → L2:\n;; 1. L0 runtime handles the swap\n;; 2. New page declares tier=L2 in response header\n;; 3. L0 runtime loads sx-L2-delta.js dynamically\n;; 4. Island hydration proceeds\n\n(define page-tier\n (fn (page)\n ;; Analyze the page's component tree\n ;; If any component is defisland → L2\n ;; If any component uses on-event/toggle! → L1\n ;; Otherwise → L0\n (cond\n ((page-has-islands? page) :L2)\n ((page-has-dom-ops? page) :L1)\n (true :L0))))" "lisp"))
(~docs/subsection :title "SX-Tier Response Header"
(p "The server includes the page's tier in the response:")
(~docs/code :src (highlight "HTTP/1.1 200 OK\nSX-Tier: L0\nSX-Components: ~card:bafy...,~plans/environment-images/nav:bafy...\n\n;; or for an island page:\nSX-Tier: L2\nSX-Components: ~counter-island:bafy..." "http"))
(p "On client-side navigation, the engine reads " (code "SX-Tier") " from the response. If the new page requires a higher tier than currently loaded, it fetches the delta script before processing the swap. The delta script registers its additional primitives and the swap proceeds."))
(~docs/subsection :title "Cache Behavior"
(p "Each tier file is content-hashed (like the current " (code "sx_js_hash") " mechanism). Cache-forever semantics. A user who visits any L0 page caches the L0 runtime permanently. If they later visit an L2 page, only the ~10KB delta downloads.")
(p "Combined with " (a :href "/sx/(etc.(plan.environment-images))" (~tw :tokens "text-violet-700 underline") "environment images") ": the image CID includes the tier. An L0 image is smaller than an L3 image — it contains fewer primitives, no parser state, no evaluator. The standalone HTML bundle for an L0 page is tiny.")))
;; -----------------------------------------------------------------------
;; Automatic tier detection
;; -----------------------------------------------------------------------
(~docs/section :title "Automatic Tier Detection" :id "auto-detect"
(p (code "deps.sx") " already classifies components as pure or IO-dependent. Extend it to classify pages by tier:")
(~docs/code :src (highlight ";; Extend deps.sx with tier analysis\n;;\n;; Walk the page's component tree:\n;; - Any defisland → L2 minimum\n;; - Any on-event, toggle!, set-attr! call → L1 minimum \n;; - Any client-eval'd component (SX wire + defcomp) → L3\n;; - Otherwise → L0\n;;\n;; The tier is the MAX of all components' requirements.\n\n(define component-tier\n (fn (comp)\n (cond\n ((island? comp) :L2)\n ((has-dom-ops? (component-body comp)) :L1)\n (true :L0))))\n\n(define page-tier\n (fn (page-def)\n (let ((comp-tiers (map component-tier\n (page-all-components page-def))))\n (max-tier comp-tiers))))" "lisp"))
(p "This runs at registration time (same phase as " (code "compute_all_deps") "). Each " (code "PageDef") " gains a " (code "tier") " field. The server uses it to select the script tag. No manual annotation needed — the tier is derived from what the page actually uses."))
;; -----------------------------------------------------------------------
;; What L0 actually needs
;; -----------------------------------------------------------------------
(~docs/section :title "What L0 Actually Needs" :id "l0-detail"
(p "L0 is the critical tier — it's what most pages load. Every byte matters. Let's be precise about what it contains:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Function")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Purpose")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Source")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "morph-node")
(td (~tw :tokens "px-3 py-2 text-stone-700") "DOM diffing — update existing DOM from new HTML")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "engine.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "morph-attrs")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Attribute diffing on a single element")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "engine.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "morph-children")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Child node reconciliation")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "engine.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "process-swap")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Apply sx-swap directive (innerHTML, outerHTML, etc)")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "engine.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "dispatch-trigger")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Process sx-trigger attributes (click, submit, load, etc)")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "engine.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "sx-fetch")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Make sx-get/sx-post requests, process response")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "orchestration.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "push-url / replace-url")
(td (~tw :tokens "px-3 py-2 text-stone-700") "History management for sx-push-url")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "engine.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "boot-triggers")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Scan DOM for sx-* attributes, wire up event listeners")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "boot.sx"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-stone-700") "resolve-suspense")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Fill in streamed suspense slots")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "boot.sx")))))
(p "That's roughly 20-30 defines from engine.sx + orchestration.sx + boot.sx, plus their transitive deps. No parser, no evaluator, no primitives beyond what those defines call internally. The platform JS is just: fetch wrapper, DOM helpers (createElement, setAttribute, morphing), history API, and event delegation.")
(p "Target: " (strong "~5KB min+gz") " — competitive with htmx (10KB) while being semantically richer (morph-based, not innerHTML-based)."))
;; -----------------------------------------------------------------------
;; Build pipeline
;; -----------------------------------------------------------------------
(~docs/section :title "Build Pipeline" :id "pipeline"
(p "The pipeline uses the same tools that already exist — " (code "js.sx") " for translation, " (code "bootstrap_js.py") " for platform assembly — but feeds them filtered define lists.")
(~docs/code :src (highlight ";; Build all tiers\n;;\n;; 1. Load all spec .sx files\n;; 2. Extract all defines (same as current bootstrap)\n;; 3. Run slice.sx to partition defines by tier\n;; 4. For each tier:\n;; a. js.sx translates the tier's define list\n;; b. Platform assembler wraps with minimal platform JS\n;; c. Output: sx-L{n}.js\n;; 5. Compute deltas: L1-delta = L1 - L0, L2-delta = L2 - L1, etc.\n\n;; The bootstrapper script orchestrates this:\n;;\n;; python bootstrap_js.py --tier L0 -o sx-L0.js\n;; python bootstrap_js.py --tier L1 --delta -o sx-L1-delta.js\n;; python bootstrap_js.py --tier L2 --delta -o sx-L2-delta.js\n;; python bootstrap_js.py -o sx-browser.js # full (L3, backward compat)" "lisp"))
(p "The " (code "--delta") " flag emits only the defines not present in the previous tier. The delta file calls " (code "Sx.extend()") " to register its additions into the already-loaded runtime.")
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mt-4")
(p (~tw :tokens "text-violet-900 font-medium") "Self-hosting all the way")
(p (~tw :tokens "text-violet-800") (code "slice.sx") " is spec. " (code "js.sx") " is spec. The bootstrapper script (" (code "bootstrap_js.py") ") is the thin host-specific glue that reads slice output, calls js.sx via the evaluator, and wraps with platform JS. The slicer could itself be bootstrapped to JavaScript and run in a browser build tool — but that's a future concern.")))
;; -----------------------------------------------------------------------
;; Spec modules
;; -----------------------------------------------------------------------
(~docs/section :title "Spec Modules" :id "spec-modules"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Module")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Functions")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Depends on")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "slice.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "define-refs") ", " (code "define-dep-graph") ", " (code "slice-defines") ", " (code "tier-entry-points") ", " (code "page-tier") ", " (code "component-tier"))
(td (~tw :tokens "px-3 py-2 text-stone-600") "deps.sx (component analysis), eval.sx (AST walking)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "js.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "js-translate-file") " — already exists, unchanged")
(td (~tw :tokens "px-3 py-2 text-stone-600") "eval.sx (runs on evaluator)")))))
(p "One new spec file (" (code "slice.sx") "), one existing translator (" (code "js.sx") "), one modified host script (" (code "bootstrap_js.py") " gains " (code "--tier") " and " (code "--delta") " flags)."))
;; -----------------------------------------------------------------------
;; Relationships
;; -----------------------------------------------------------------------
(~docs/section :title "Relationships" :id "relationships"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (a :href "/sx/(etc.(plan.environment-images))" (~tw :tokens "text-violet-700 underline") "Environment Images") " — tiered images are smaller. An L0 image omits the parser, evaluator, and most primitives.")
(li (a :href "/sx/(etc.(plan.content-addressed-components))" (~tw :tokens "text-violet-700 underline") "Content-Addressed Components") " — component CID resolution is L3-only. L0 pages don't resolve components client-side.")
(li (a :href "/sx/(geography.(reactive.(reactive-design)))" (~tw :tokens "text-violet-700 underline") "Reactive Islands") " — L2 tier is defined by island presence. The signal runtime is the L1→L2 delta.")
(li (a :href "/sx/(etc.(plan.isomorphic-architecture))" (~tw :tokens "text-violet-700 underline") "Isomorphic Architecture") " — client-side page rendering is L3. Most pages don't need it."))
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-3 mt-2")
(p (~tw :tokens "text-amber-800 text-sm") (strong "Depends on: ") (code "js.sx") " (complete), " (code "deps.sx") " (complete), " (code "bootstrap_js.py") " adapter selection (exists). " (strong "New: ") (code "slice.sx") " spec module.")))))

263
sx/sx/plans/rust-wasm-host.sx
View File

@@ -1,263 +0,0 @@
;; ---------------------------------------------------------------------------
;; Rust/WASM Host — Bootstrap the SX spec to Rust, compile to WASM
;; ---------------------------------------------------------------------------
(defcomp ~plans/rust-wasm-host/plan-rust-wasm-host-content ()
(~docs/page :title "Rust/WASM Host"
(~docs/section :title "Context" :id "context"
(p "The SX host architecture says: spec it in " (code ".sx") ", bootstrap to every target. We've now done it for Rust.")
(p "The Rust bootstrapper (" (code "bootstrap_rs.py") ") reads all 20 " (code ".sx") " spec files and emits a complete Rust crate — " (strong "9,781 lines") " of Rust that compiles with zero errors. The test suite has " (strong "92 tests passing") " across parser, evaluator, primitives, and rendering.")
(p "This is distinct from the " (a :href "/sx/(etc.(plan.wasm-bytecode-vm))" "WASM Bytecode VM") " plan. That plan designs a custom bytecode format and VM. This plan bootstraps the " (strong "same tree-walking evaluator") " directly to Rust/WASM from the same " (code ".sx") " specs — no new bytecode, no new VM, just another host.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "What exists today")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Artifact")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Lines")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Status")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "bootstrap_rs.py"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "—")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Rust bootstrapper, reads all 20 .sx spec files"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "sx_ref.rs"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "9,781")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Generated Rust — compiles clean, 0 errors"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "platform.rs"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "—")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Rust platform interface (type constructors, env ops)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "test_parser.rs"))
(td (~tw :tokens "px-3 py-2 text-stone-700") "—")
(td (~tw :tokens "px-3 py-2 text-stone-600") "92 tests passing (parser, eval, primitives, render)"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "Primitives")
(td (~tw :tokens "px-3 py-2 text-stone-700") "—")
(td (~tw :tokens "px-3 py-2 text-stone-600") "75 real implementations, 154 stubs"))))))
;; -----------------------------------------------------------------------
;; Architecture
;; -----------------------------------------------------------------------
(~docs/section :title "Architecture" :id "architecture"
(p "The key architectural insight: " (strong "factor the browser primitives into a shared platform layer") " that both the JS evaluator and the WASM module consume.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Shared platform layer")
(p (code "platform_js.py") " already contains all DOM, browser, fetch, timer, and storage implementations — bootstrapped from " (code "boundary.sx") ". These are pure JavaScript functions that call browser APIs. They don't depend on the evaluator.")
(p "Extract them into a standalone " (code "sx-platform.js") " module. Both " (code "sx-browser.js") " (the current JS evaluator) and the new " (code "sx-wasm-shim.js") " import from the same platform module:")
(~docs/code :src (highlight " ┌─────────────────┐\n │ sx-platform.js │ ← DOM, fetch, timers, storage\n └────────┬────────┘\n │\n ┌──────────────┼──────────────┐\n │ │\n ┌─────────┴─────────┐ ┌────────┴────────┐\n │ sx-browser.js │ │ sx-wasm-shim.js │\n │ (JS tree-walker) │ │ (WASM instance │\n │ │ │ + handle table) │\n └────────────────────┘ └─────────────────┘" "text"))
(p "One codebase for all browser primitives. Bug fixes apply to both targets. The evaluator is the only thing that changes — JS tree-walker vs Rust/WASM tree-walker.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Opaque handle table")
(p "Rust/WASM can't hold DOM node references directly. Instead, Rust values use " (code "Value::Handle(u32)") " — an opaque integer that indexes into a JavaScript-side handle table:")
(~docs/code :src (highlight "// JS side (in sx-wasm-shim.js)\nconst handles = []; // handle_id → DOM node\n\nfunction allocHandle(node) {\n const id = handles.length;\n handles.push(node);\n return id;\n}\n\nfunction getHandle(id) { return handles[id]; }\nfunction freeHandle(id) { handles[id] = null; }" "javascript"))
(~docs/code :src (highlight "// Rust side\n#[derive(Clone, Debug)]\nenum Value {\n Nil,\n Bool(bool),\n Number(f64),\n Str(String),\n Symbol(String),\n Keyword(String),\n List(Vec<Value>),\n Dict(Vec<(Value, Value)>),\n Lambda(Rc<Closure>),\n Handle(u32), // ← opaque DOM node reference\n}" "rust"))
(p "When Rust calls a DOM primitive (e.g. " (code "createElement") "), it gets back a " (code "Handle(id)") ". When it passes that handle to " (code "appendChild") ", the JS shim looks up the real node. Rust never sees a DOM node — only integer IDs.")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "The JS shim is thin")
(p "The WASM shim's job is minimal:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "Instantiate the WASM module")
(li "Wire up the handle table")
(li "Delegate all browser primitives to " (code "sx-platform.js"))
(li "Provide " (code "#[wasm_bindgen]") " imports for the DOM primitives that Rust calls"))
(p "Everything complex — event dispatch, morph engine, routing, SSE — lives in the shared platform layer. The shim is glue."))
;; -----------------------------------------------------------------------
;; Stubs breakdown
;; -----------------------------------------------------------------------
(~docs/section :title "Current Stub Breakdown" :id "stubs"
(p "Of the 154 stubbed primitives, most fall into a few categories that map directly to implementation phases:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Category")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Count")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Examples")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Phase")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "DOM creation & attrs")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~30")
(td (~tw :tokens "px-3 py-2 text-stone-600") "createElement, setAttribute, appendChild")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Phase 2"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Events & callbacks")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~20")
(td (~tw :tokens "px-3 py-2 text-stone-600") "addEventListener, setTimeout, requestAnimationFrame")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Phase 3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Fetch & network")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~15")
(td (~tw :tokens "px-3 py-2 text-stone-600") "fetch, XMLHttpRequest, SSE")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Phase 3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Browser APIs")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~25")
(td (~tw :tokens "px-3 py-2 text-stone-600") "history, location, localStorage, console")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Phase 5"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Morph engine")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~15")
(td (~tw :tokens "px-3 py-2 text-stone-600") "morph, sync-attrs, reconcile")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Phase 3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Signals & reactivity")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~20")
(td (~tw :tokens "px-3 py-2 text-stone-600") "signal, deref, effect, computed, batch")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Phase 6"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "Component lifecycle")
(td (~tw :tokens "px-3 py-2 text-stone-700") "~29")
(td (~tw :tokens "px-3 py-2 text-stone-600") "boot, hydrate, register-component")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Phase 4"))))))
;; -----------------------------------------------------------------------
;; Phase 1: WASM build + parse/eval
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 1: WASM Build + Parse/Eval" :id "phase-1"
(p "Get the existing Rust code compiling to WASM and running parse/eval in the browser.")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "Add " (code "wasm-bindgen") " and " (code "wasm-pack") " to the Rust crate")
(li "Export " (code "#[wasm_bindgen]") " functions: " (code "parse(src) -> JsValue") " and " (code "eval(src) -> JsValue"))
(li "All 75 real primitives work — arithmetic, string ops, list ops, dict ops, comparisons")
(li "Test page: load WASM module, parse SX source, eval expressions, display results")
(li "Benchmark: parse/eval speed vs JS evaluator on the same expressions"))
(p (strong "Milestone:") " SX expressions evaluate identically in JS and WASM. No DOM, no rendering — just computation."))
;; -----------------------------------------------------------------------
;; Phase 2: DOM rendering via handle table
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 2: DOM Rendering via Handle Table" :id "phase-2"
(p "Implement the shared platform layer and handle table. Rust can create and manipulate DOM nodes.")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "Extract " (code "sx-platform.js") " from " (code "platform_js.py") " output — all DOM primitives as standalone functions")
(li "Implement " (code "sx-wasm-shim.js") " — WASM instantiation + handle table + platform imports")
(li "Wire " (code "Value::Handle(u32)") " through the Rust evaluator for DOM node references")
(li "Implement ~30 DOM stubs: " (code "createElement") ", " (code "setAttribute") ", " (code "appendChild") ", " (code "createTextNode") ", " (code "removeChild"))
(li (code "render-to-dom") " works through WASM — evaluates component tree, produces real DOM nodes via handle table"))
(p (strong "Milestone:") " A component renders to identical DOM whether evaluated by JS or WASM."))
;; -----------------------------------------------------------------------
;; Phase 3: Events + fetch + morph
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 3: Events + Fetch + Morph" :id "phase-3"
(p "The hardest phase — callbacks cross the WASM/JS boundary. Event handlers are Rust closures that JS must be able to invoke.")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Callback table") " — mirror of the handle table but for functions. Rust registers a closure, gets back an ID. JS calls the ID when the event fires. Rust looks up and invokes the closure.")
(li (strong "Event listeners") " — " (code "addEventListener") " stores the callback ID on the element handle. JS dispatches events to the WASM callback table.")
(li (strong "Fetch") " — Rust initiates fetch via JS import, JS calls " (code "sx-platform.js") " " (code "fetch") ", returns result to Rust via callback.")
(li (strong "Morph engine") " — DOM diffing/patching runs in Rust. Morph calls produce a sequence of DOM mutations via handle operations. " (code "sync-attrs") " and " (code "morph-children") " work through the handle table."))
(p (strong "Milestone:") " Interactive pages work — click handlers, form submissions, HTMX-style requests, morph updates. This is the " (strong "MVP") " — a page could ship with the WASM evaluator and function correctly."))
;; -----------------------------------------------------------------------
;; Phase 4: Boot + hydration + components
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 4: Boot + Hydration + Components" :id "phase-4"
(p "Full page lifecycle — the WASM module replaces " (code "sx-browser.js") " for a complete page.")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Boot sequence") " — WASM module loads, reads " (code "data-components") " script tags, registers component definitions, processes " (code "data-sx-page") " content")
(li (strong "Component registration") " — " (code "defcomp") " and " (code "defisland") " evaluated in Rust, stored in the WASM-side environment")
(li (strong "Hydration") " — server-rendered HTML matched against component tree, event handlers attached, islands activated")
(li (strong "CSSX") " — style computation runs in Rust (all CSSX primitives are already in the 75 real implementations)"))
(p (strong "Milestone:") " A full SX page boots and hydrates under WASM. Component definitions, CSSX styles, page content all evaluated by Rust."))
;; -----------------------------------------------------------------------
;; Phase 5: Routing + streaming + SSE
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 5: Routing + Streaming + SSE" :id "phase-5"
(p "Client-side navigation and real-time updates.")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Client routing") " — " (code "navigate-to") ", " (code "popstate") " handling, URL matching from " (code "defpage") " routes")
(li (strong "History API") " — " (code "pushState") " / " (code "replaceState") " via JS imports")
(li (strong "SSE") " — server-sent events for live updates, morph on incoming HTML/SX")
(li (strong "Streaming responses") " — progressive rendering of large page content"))
(p (strong "Milestone:") " Full SPA navigation works under WASM. Pages load, navigate, and receive live updates."))
;; -----------------------------------------------------------------------
;; Phase 6: Signals + reactive islands
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 6: Signals + Reactive Islands" :id "phase-6"
(p "The most architecturally interesting phase — closures-as-values must work across the WASM boundary.")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Signal primitives") " — " (code "signal") ", " (code "deref") ", " (code "reset!") ", " (code "swap!") ", " (code "computed") ", " (code "effect") ", " (code "batch") " all implemented in Rust")
(li (strong "Reactive DOM updates") " — signal changes trigger DOM mutations via handle table operations. No full re-render — fine-grained updates.")
(li (strong "Island scoping") " — " (code "with-island-scope") " manages signal lifecycle. Dispose island = drop all signals, effects, and handles.")
(li (strong "Computed chains") " — dependency graph tracks which signals feed which computed values. Topological update order."))
(p "The closure challenge: " (code "effect") " and " (code "computed") " take closures that capture reactive context. In Rust, these are " (code "Rc<dyn Fn(...)>") " values that the signal graph holds. The " (code "with-island-scope") " arena pattern handles cleanup — drop the arena, drop all closures.")
(p (strong "Milestone:") " Reactive islands work under WASM. Signals, computed values, effects, fine-grained DOM updates — all in Rust."))
;; -----------------------------------------------------------------------
;; Phase 7: Full parity + gradual rollout
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 7: Full Parity + Gradual Rollout" :id "phase-7"
(p "Shadow-compare and feature-flagged rollout.")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Shadow compare") " — run both JS and WASM evaluators in parallel on every page render. Assert identical DOM output. Log divergences. Same principle as async eval convergence.")
(li (strong "Feature flag") " — server sets " (code "data-sx-runtime=\"wasm\"") " or " (code "\"js\"") " on the page. Boot script loads the corresponding evaluator. Flag can be per-page, per-user, or global.")
(li (strong "Progressive enhancement") " — try WASM first, fall back to JS if WASM instantiation fails. Ship both " (code "sx-browser.js") " and " (code "sx-wasm.wasm") ".")
(li (strong "Gradual rollout") " — start with simple pages (documentation, static content). Move to interactive pages. Finally, reactive islands. Each phase validates correctness before advancing."))
(p (strong "Milestone:") " Full parity. Any SX page renders identically under JS or WASM. The runtime is a deployment choice, not an architectural one."))
;; -----------------------------------------------------------------------
;; Shared platform layer
;; -----------------------------------------------------------------------
(~docs/section :title "Shared Platform Layer" :id "shared-platform"
(p "The architectural heart of this plan. " (code "platform_js.py") " already generates all browser primitive implementations — DOM manipulation, fetch wrappers, timer management, storage access, history API, SSE handling. Currently these are inlined into " (code "sx-browser.js") ".")
(p "The extraction:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Step 1") " — refactor " (code "platform_js.py") " to emit a standalone " (code "sx-platform.js") " module (ES module exports)")
(li (strong "Step 2") " — " (code "sx-browser.js") " imports from " (code "sx-platform.js") " instead of containing the implementations inline")
(li (strong "Step 3") " — " (code "sx-wasm-shim.js") " imports from the same " (code "sx-platform.js") " and wires the functions as WASM imports"))
(p "Result: one implementation of every browser primitive, shared by both evaluator targets. Fix a bug in " (code "sx-platform.js") " and both JS and WASM evaluators get the fix.")
(~docs/code :src (highlight "// sx-platform.js (extracted from platform_js.py output)\nexport function createElement(tag) {\n return document.createElement(tag);\n}\nexport function setAttribute(el, key, val) {\n el.setAttribute(key, val);\n}\nexport function appendChild(parent, child) {\n parent.appendChild(child);\n}\nexport function addEventListener(el, event, callback) {\n el.addEventListener(event, callback);\n}\n// ... ~150 more browser primitives\n\n// sx-browser.js\nimport * as platform from './sx-platform.js';\n// Uses platform functions directly — evaluator is JS\n\n// sx-wasm-shim.js\nimport * as platform from './sx-platform.js';\n// Wraps platform functions for WASM import — evaluator is Rust" "javascript")))
;; -----------------------------------------------------------------------
;; Interaction with other plans
;; -----------------------------------------------------------------------
(~docs/section :title "Interaction with Other Plans" :id "interactions"
(ul (~tw :tokens "list-disc list-inside space-y-2")
(li (strong "WASM Bytecode VM") " — complementary, not competing. This plan bootstraps the tree-walking evaluator to Rust/WASM. The bytecode VM plan compiles SX to a custom bytecode format and runs it in a dispatch loop. Tree-walking comes first (it's working now). Bytecode VM is a future optimisation on top of the Rust host.")
(li (strong "Runtime Slicing") " — the WASM module can be tiered. L0 pages need no WASM at all. L1 pages need a minimal WASM module (just parse + eval, no DOM). L2+ pages need the full module with DOM and signals. Tree-shake unused primitives per tier.")
(li (strong "Content-Addressed Components") " — deterministic Rust compilation means the same " (code ".sx") " source always produces the same WASM binary. CID-addressable WASM modules. Fetch the evaluator itself by content hash.")
(li (strong "Async Eval Convergence") " — must complete first. The spec must be the single source of truth before we add another compilation target. The Rust bootstrapper reads the same spec files that the Python and JS bootstrappers read.")))
;; -----------------------------------------------------------------------
;; Principles
;; -----------------------------------------------------------------------
(~docs/section :title "Principles" :id "principles"
(ul (~tw :tokens "list-disc list-inside space-y-2")
(li (strong "Same spec, another host.") " The Rust target is not special — it's the same architecture as Python and JavaScript. " (code "bootstrap_rs.py") " reads the same " (code ".sx") " files and emits Rust instead of Python or JS. The spec doesn't know which host runs it.")
(li (strong "Platform primitives stay in JavaScript.") " DOM, fetch, timers, storage — these are browser APIs. Rust doesn't reimplement them. It calls them through the shared platform layer via the handle table.")
(li (strong "Shared platform, not duplicated platform.") " The key win over a pure-WASM approach. Browser primitives exist once in " (code "sx-platform.js") ". Both evaluators use them. No divergence, no duplicate bugs.")
(li (strong "Progressive, not all-or-nothing.") " WASM is an enhancement. JS remains the fallback. Pages can opt in per-page. The server decides which runtime to ship. Rollout is gradual and reversible.")
(li (strong "Handle table is the boundary.") " Rust holds integer IDs. JavaScript holds real objects. The handle table is the only bridge. This keeps the WASM module platform-independent — swap the handle table implementation for a different host (Node, Deno, native webview) and the Rust code doesn't change.")))
;; -----------------------------------------------------------------------
;; Outcome
;; -----------------------------------------------------------------------
(~docs/section :title "Outcome" :id "outcome"
(p "After completion:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "SX bootstraps to four hosts: JavaScript, Python, Rust (native), Rust (WASM)")
(li "Browser evaluation runs at near-native speed via WASM tree-walking")
(li "All browser primitives shared between JS and WASM evaluators — zero duplication")
(li "Gradual rollout: feature flag per-page, shadow-compare for correctness, progressive enhancement for compatibility")
(li "The architecture proof is complete: one spec, every host, deployment-time target selection")
(li "Future bytecode VM plan builds on the Rust host — the platform layer and handle table are already in place")))))

449
sx/sx/plans/scoped-effects.sx
View File

@@ -1,449 +0,0 @@
;; Scoped Effects — The Deepest Primitive
;; Algebraic effects as the unified foundation for spreads, islands, lakes, and context.
(defcomp ~plans/scoped-effects/plan-scoped-effects-content ()
(~docs/page :title "Scoped Effects — The Deepest Primitive"
(p (~tw :tokens "text-stone-500 text-sm italic mb-8")
"Everything SX does — spreads, signals, islands, lakes, morph, context, collect — "
"is an instance of one pattern: a named scope with a value flowing down, "
"contributions flowing up, and a propagation mode determining when effects are realised. "
"This plan traces that pattern to its foundation in algebraic effects "
"and proposes " (code "scope") " as the single primitive underneath everything.")
;; =====================================================================
;; I. The Observation
;; =====================================================================
(~docs/section :title "The observation" :id "observation"
(p "SX has accumulated several mechanisms that all do variations of the same thing:")
(table (~tw :tokens "w-full text-sm border-collapse mb-6")
(thead
(tr (~tw :tokens "border-b border-stone-300")
(th (~tw :tokens "text-left py-2 pr-4") "Mechanism")
(th (~tw :tokens "text-left py-2 pr-4") "Direction")
(th (~tw :tokens "text-left py-2 pr-4") "When")
(th (~tw :tokens "text-left py-2") "Boundary")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "spread")
(td (~tw :tokens "py-2 pr-4") "child → parent")
(td (~tw :tokens "py-2 pr-4") "render time")
(td (~tw :tokens "py-2") "element"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "collect! / collected")
(td (~tw :tokens "py-2 pr-4") "child → ancestor")
(td (~tw :tokens "py-2 pr-4") "render time")
(td (~tw :tokens "py-2") "render tree"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "provide / context")
(td (~tw :tokens "py-2 pr-4") "ancestor → child")
(td (~tw :tokens "py-2 pr-4") "render time")
(td (~tw :tokens "py-2") "render subtree"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "signal / effect")
(td (~tw :tokens "py-2 pr-4") "value → subscribers")
(td (~tw :tokens "py-2 pr-4") "signal change")
(td (~tw :tokens "py-2") "reactive scope"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "defisland")
(td (~tw :tokens "py-2 pr-4") "server → client")
(td (~tw :tokens "py-2 pr-4") "hydration")
(td (~tw :tokens "py-2") "server/client"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "lake")
(td (~tw :tokens "py-2 pr-4") "server → client slot")
(td (~tw :tokens "py-2 pr-4") "navigation/morph")
(td (~tw :tokens "py-2") "client/server"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "reactive-spread")
(td (~tw :tokens "py-2 pr-4") "child → parent")
(td (~tw :tokens "py-2 pr-4") "signal change")
(td (~tw :tokens "py-2") "element"))
(tr
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "def-store / use-store")
(td (~tw :tokens "py-2 pr-4") "island ↔ island")
(td (~tw :tokens "py-2 pr-4") "signal change")
(td (~tw :tokens "py-2") "cross-island"))))
(p "Eight mechanisms. Each invented separately for a real use case. "
"Each with its own API surface, its own implementation path, its own section in the spec.")
(p "But look at the table. Every row is: " (em "a named region, "
"a direction of data flow, a moment when effects are realised, "
"and a boundary that determines what crosses") ". They are all the same shape."))
;; =====================================================================
;; II. Three propagation modes
;; =====================================================================
(~docs/section :title "Three propagation modes" :id "propagation"
(p "The \"when\" column has exactly three values. These are the three "
"propagation modes of the render tree:")
(~docs/subsection :title "Render propagation"
(p "Effects are realised " (strong "during the render pass") ". "
"Synchronous, one-shot, deterministic. The renderer walks the tree, "
"encounters effects, handles them immediately.")
(p "Instances: " (code "provide/context/emit!") ", " (code "spread") ", "
(code "collect!/collected") ".")
(p "Characteristic: by the time rendering finishes, all render-time effects "
"are fully resolved. The output is static HTML with everything baked in."))
(~docs/subsection :title "Reactive propagation"
(p "Effects are realised " (strong "when signals change") ". "
"Fine-grained, incremental, potentially infinite. A signal notifies "
"its subscribers, each subscriber updates its piece of the DOM.")
(p "Instances: " (code "signal/effect/computed") ", " (code "reactive-spread") ", "
(code "defisland") ", " (code "def-store/use-store") ".")
(p "Characteristic: the initial render produces a snapshot. "
"Subsequent changes propagate through the signal graph without re-rendering. "
"Each signal write touches only the DOM nodes that actually depend on it."))
(~docs/subsection :title "Morph propagation"
(p "Effects are realised " (strong "when the server sends new HTML") ". "
"Network-bound, server-initiated, structural. The morph algorithm "
"compares old and new DOM trees and surgically updates.")
(p "Instances: " (code "lake") ", full-page morph, " (code "sx-get/sx-post") " responses.")
(p "Characteristic: the server is the source of truth for content. "
"The client has reactive state that must be " (em "protected") " during morph. "
"Lakes mark the boundaries: morph enters islands, finds lake elements by ID, "
"updates their children, leaves reactive attrs untouched.")))
;; =====================================================================
;; III. The scope primitive
;; =====================================================================
(~docs/section :title "The scope primitive" :id "scope"
(p "A " (code "scope") " is a named region of the render tree with three properties:")
(ol (~tw :tokens "space-y-2 text-stone-600")
(li (strong "Value") " — data flowing " (em "downward") " to descendants (readable via " (code "context") ")")
(li (strong "Accumulator") " — data flowing " (em "upward") " from descendants (writable via " (code "emit!") ")")
(li (strong "Propagation mode") " — " (em "when") " effects on this scope are realised"))
(~docs/code :src (highlight "(scope \"name\"\n :value expr ;; downward (readable by descendants)\n :propagation :render ;; :render | :reactive | :morph\n body...)" "lisp"))
(p "Every existing mechanism is a scope with a specific configuration:")
(table (~tw :tokens "w-full text-sm border-collapse mb-6")
(thead
(tr (~tw :tokens "border-b border-stone-300")
(th (~tw :tokens "text-left py-2 pr-4") "Mechanism")
(th (~tw :tokens "text-left py-2 pr-4") "Scope equivalent")
(th (~tw :tokens "text-left py-2") "Propagation")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "provide")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "(scope name :value v body)")
(td (~tw :tokens "py-2") ":render"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "defisland")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "(scope name :propagation :reactive body)")
(td (~tw :tokens "py-2") ":reactive"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "lake")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "(scope name :propagation :morph body)")
(td (~tw :tokens "py-2") ":morph"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "spread")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "(emit! :element-attrs dict)")
(td (~tw :tokens "py-2") ":render (implicit)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "collect!")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "(emit! name value)")
(td (~tw :tokens "py-2") ":render"))
(tr
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "def-store")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "(scope name :value (signal v) :propagation :reactive)")
(td (~tw :tokens "py-2") ":reactive (cross-scope)"))))
(~docs/subsection :title "Scopes compose by nesting"
(~docs/code :src (highlight ";; An island with a theme context and a morphable lake\n(scope \"my-island\" :propagation :reactive\n (let ((colour (signal \"violet\")))\n (scope \"theme\" :value {:primary colour} :propagation :render\n (div\n (h1 :style (str \"color:\" (deref (get (context \"theme\") :primary)))\n \"Themed heading\")\n (scope \"product-details\" :propagation :morph\n ;; Server morphs this on navigation\n ;; Reactive attrs on the h1 are protected\n (~product-card :id 42))))))" "lisp"))
(p "Three scopes, three propagation modes, nested naturally. "
"The reactive scope manages signal lifecycle. "
"The render scope provides downward context. "
"The morph scope marks where server content flows in. "
"Each scope handles its own effects without knowing about the others.")))
;; =====================================================================
;; IV. The 2×3 matrix
;; =====================================================================
(~docs/section :title "The 2×3 matrix" :id "matrix"
(p "Direction (up/down) × propagation mode (render/reactive/morph) gives six cells. "
"Every mechanism in SX occupies one:")
(table (~tw :tokens "w-full text-sm border-collapse mb-6")
(thead
(tr (~tw :tokens "border-b border-stone-300")
(th (~tw :tokens "text-left py-2 pr-4") "")
(th (~tw :tokens "text-left py-2 pr-4") "Render-time")
(th (~tw :tokens "text-left py-2 pr-4") "Reactive")
(th (~tw :tokens "text-left py-2") "Morph")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-semibold") "Down ↓")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "context")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "reactive context")
(td (~tw :tokens "py-2 font-mono text-xs") "server props"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-semibold") "Up ↑")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "emit! / spread")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "reactive-spread")
(td (~tw :tokens "py-2 font-mono text-xs") "lake contributions"))
(tr
(td (~tw :tokens "py-2 pr-4 font-semibold") "Both ↕")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "provide")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "island")
(td (~tw :tokens "py-2 font-mono text-xs") "full morph"))))
(p "Some cells are already implemented. Some are new. "
"The point is that " (code "scope") " fills them all from " (strong "one primitive") ". "
"You don't need to invent a new mechanism for each cell — you configure the scope.")
(~docs/subsection :title "The new cell: reactive context"
(p "The (Down ↓, Reactive) cell — " (strong "reactive context") " — "
"is the most interesting gap. It's React's Context + signals, but without "
"the re-render avalanche that makes React Context slow.")
(~docs/code :src (highlight ";; Reactive context: value is a signal, propagation is reactive\n(scope \"theme\" :value (signal {:primary \"violet\"}) :propagation :reactive\n ;; Any descendant reads with context + deref\n ;; Only the specific DOM node that uses the value updates\n (h1 :style (str \"color:\" (get (deref (context \"theme\")) :primary))\n \"This h1 updates when the theme signal changes\")\n ;; Deep nesting doesn't matter — it's O(1) per subscriber\n (~deeply-nested-component-tree))" "lisp"))
(p "React re-renders " (em "every") " component that reads a changed context. "
"SX's reactive context updates " (em "only") " the DOM nodes that " (code "deref") " the signal. "
"Same API, fundamentally different performance characteristics.")))
;; =====================================================================
;; V. Algebraic effects
;; =====================================================================
(~docs/section :title "Algebraic effects: the deepest layer" :id "algebraic-effects"
(p "The scope primitive has a name in programming language theory: "
(strong "algebraic effects with handlers") ".")
(~docs/subsection :title "The correspondence"
(table (~tw :tokens "w-full text-sm border-collapse mb-6")
(thead
(tr (~tw :tokens "border-b border-stone-300")
(th (~tw :tokens "text-left py-2 pr-4") "Algebraic effects")
(th (~tw :tokens "text-left py-2 pr-4") "SX")
(th (~tw :tokens "text-left py-2") "What it does")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "perform effect")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "emit!")
(td (~tw :tokens "py-2") "Raise an effect upward through the tree"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "handle effect")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "scope / provide")
(td (~tw :tokens "py-2") "Catch and interpret the effect"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "ask")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "context")
(td (~tw :tokens "py-2") "Read the nearest handler's value"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "handler nesting")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "scope nesting")
(td (~tw :tokens "py-2") "Inner handlers shadow outer ones"))
(tr
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "resumption")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "reactive propagation")
(td (~tw :tokens "py-2") "Handler can re-invoke the continuation"))))
(p "The last row is the deep one. In algebraic effect theory, a handler can "
(strong "resume") " the computation that performed the effect — optionally with "
"a different value. This is exactly what reactive propagation does: when a signal "
"changes, the effect handler (the island scope) re-invokes the subscribed "
"continuations (the effect callbacks) with the new value."))
(~docs/subsection :title "Why this matters"
(p "Algebraic effects are " (strong "the most general control flow abstraction known") ". "
"Exceptions, generators, async/await, coroutines, backtracking, and "
"nondeterminism are all instances of algebraic effects with different handler strategies.")
(p "SX's three propagation modes are three handler strategies:")
(ul (~tw :tokens "list-disc pl-5 space-y-2 text-stone-600")
(li (strong "Render") " = " (em "one-shot") " handler — handle the effect immediately, "
"don't resume. Like " (code "try/catch") " for rendering.")
(li (strong "Reactive") " = " (em "multi-shot") " handler — handle the effect, "
"then re-handle it every time the value changes. Like a generator that yields "
"whenever a signal fires.")
(li (strong "Morph") " = " (em "external resumption") " handler — the server decides "
"when to resume, sending new content over the network. Like an async/await "
"where the promise is resolved by a different machine.")))
(~docs/subsection :title "The effect hierarchy"
(p "Effects form a hierarchy. Lower-level effects are handled by higher-level scopes:")
(~docs/code :src (highlight ";; Effect hierarchy (innermost handled first)\n;;\n;; spread → element scope handles (merge attrs)\n;; emit! → nearest provide handles (accumulate)\n;; signal write → reactive scope handles (notify subscribers)\n;; morph → morph scope handles (diff + patch)\n;;\n;; If no handler is found, the effect bubbles up.\n;; Unhandled effects at the root are errors (like uncaught exceptions).\n;;\n;; This is why (emit! \"cssx\" rule) without a provider\n;; should error — there's no handler for the effect." "lisp"))
(p "The current " (code "collect!") " is a global accumulator — effectively an "
"implicit top-level handler. Under the scope model, it would be an explicit "
(code "provide") " in the layout that handles " (code "\"cssx\"") " effects.")))
;; =====================================================================
;; VI. What scope enables
;; =====================================================================
(~docs/section :title "What scope enables" :id "enables"
(~docs/subsection :title "1. New propagation modes"
(p "The three modes (render, reactive, morph) are not hardcoded — they are "
"handler strategies. New strategies can be added without new primitives:")
(ul (~tw :tokens "list-disc pl-5 space-y-2 text-stone-600")
(li (strong ":animation") " — effects realised on requestAnimationFrame, "
"batched per frame, interruptible by higher-priority updates")
(li (strong ":idle") " — effects deferred to requestIdleCallback, "
"used for non-urgent background updates (the transition pattern)")
(li (strong ":stream") " — effects realised as server-sent events arrive, "
"for live data (scores, prices, notifications)")
(li (strong ":worker") " — effects computed in a Web Worker, "
"results propagated back to the main thread")))
(~docs/subsection :title "2. Effect composition"
(p "Because scopes nest, effects compose naturally:")
(~docs/code :src (highlight ";; Animation scope inside a reactive scope\n(scope \"island\" :propagation :reactive\n (let ((items (signal large-list)))\n (scope \"smooth\" :propagation :animation\n ;; Updates to items are batched per frame\n ;; The reactive scope tracks deps\n ;; The animation scope throttles DOM writes\n (for-each (fn (item)\n (div (get item \"name\"))) (deref items)))))" "lisp"))
(p "The reactive scope notifies when " (code "items") " changes. "
"The animation scope batches the resulting DOM writes to the next frame. "
"Neither scope knows about the other. They compose by nesting."))
(~docs/subsection :title "3. Server/client as effect boundary"
(p "The deepest consequence: the server/client boundary becomes " (em "just another "
"scope boundary") ". What crosses it is determined by the propagation mode:")
(~docs/code :src (highlight ";; The server renders this:\n(scope \"page\" :propagation :render\n (scope \"header\" :propagation :reactive\n ;; Client takes over — reactive scope\n (island-body...))\n (scope \"content\" :propagation :morph\n ;; Server controls — morphed on navigation\n (page-content...))\n (scope \"footer\" :propagation :render\n ;; Static — rendered once, never changes\n (footer...)))" "lisp"))
(p "The renderer walks the tree. When it hits a reactive scope, it serialises "
"state and emits a hydration marker. When it hits a morph scope, it emits "
"a lake marker. When it hits a render scope, it just renders. "
"The three scope types naturally produce the islands-and-lakes architecture — "
"not as a special case, but as the " (em "only") " case."))
(~docs/subsection :title "4. Cross-scope communication"
(p "Named stores (" (code "def-store/use-store") ") are scopes that transcend "
"the render tree. Two islands sharing a signal is two reactive scopes "
"referencing the same named scope:")
(~docs/code :src (highlight ";; Two islands, one shared scope\n(scope \"cart\" :value (signal []) :propagation :reactive :global true\n ;; Any island can read/write the cart\n ;; The scope transcends the render tree\n ;; Signal propagation handles cross-island updates)" "lisp"))
(p "The " (code ":global") " flag lifts the scope out of the tree hierarchy "
"into a named registry. " (code "def-store") " is syntax sugar for this.")))
;; =====================================================================
;; VII. Implementation path
;; =====================================================================
(~docs/section :title "Implementation path" :id "implementation"
(p "The path from current SX to the scope primitive follows the existing plan "
"and adds two phases:")
(~docs/subsection :title "Phase 1: provide/context/emit! ✓"
(p (strong "Complete. ") "Render-time dynamic scope. Four primitives: "
(code "provide") " (special form), " (code "context") ", " (code "emit!") ", "
(code "emitted") ". Platform provides " (code "scope-push!/scope-pop!") ". "
"Spreads reimplemented on provide/emit!.")
(p "See "
(a :href "/sx/(geography.(provide))" (~tw :tokens "text-violet-600 hover:underline") "provide article")
" and "
(a :href "/sx/(geography.(spreads))" (~tw :tokens "text-violet-600 hover:underline") "spreads article")
"."))
(~docs/subsection :title "Phase 2: scope as the common form ✓"
(p (strong "Complete. ") (code "scope") " is now the general form. "
(code "provide") " is sugar for " (code "(scope name :value v body...)") ". "
(code "collect!") " creates a lazy root scope with deduplication. "
"All adapters use " (code "scope-push!/scope-pop!") " directly.")
(p "The unified platform structure:")
(~docs/code :src (highlight "_scope_stacks = {} ;; {name: [{value, emitted: [], dedup: bool}]}" "python"))
(p "See "
(a :href "/sx/(geography.(scopes))" (~tw :tokens "text-violet-600 hover:underline") "scopes article")
"."))
(~docs/subsection :title "Phase 3: effect handlers (future)"
(p "Make propagation modes extensible. A " (code ":propagation") " value is a "
"handler function that determines when and how effects are realised:")
(~docs/code :src (highlight ";; Custom propagation mode\n(define :debounced\n (fn (emit-fn delay)\n ;; Returns a handler that debounces effect realisation\n (let ((timer nil))\n (fn (effect)\n (when timer (clear-timeout timer))\n (set! timer (set-timeout\n (fn () (emit-fn effect)) delay))))))\n\n;; Use it\n(scope \"search\" :propagation (:debounced 300)\n ;; Effects in this scope are debounced by 300ms\n (input :on-input (fn (e)\n (emit! \"search\" (get-value e)))))" "lisp"))
(p (strong "Delivers: ") "user-defined propagation modes (animation, debounce, throttle, "
"worker, stream), effect composition by nesting, "
"the full algebraic effects model.")))
;; =====================================================================
;; VIII. What this means for the spec
;; =====================================================================
(~docs/section :title "What this means for the spec" :id "spec"
(p "The self-hosting spec currently has separate code paths for each mechanism. "
"Under the scope model, they converge:")
(table (~tw :tokens "w-full text-sm border-collapse mb-6")
(thead
(tr (~tw :tokens "border-b border-stone-300")
(th (~tw :tokens "text-left py-2 pr-4") "Spec file")
(th (~tw :tokens "text-left py-2 pr-4") "Current")
(th (~tw :tokens "text-left py-2") "After scope")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "eval.sx")
(td (~tw :tokens "py-2 pr-4") "provide + defisland as separate special forms")
(td (~tw :tokens "py-2") "scope as one special form, sugar for provide/defisland/lake"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "adapter-html.sx")
(td (~tw :tokens "py-2 pr-4") "provide, island, lake as separate dispatch cases")
(td (~tw :tokens "py-2") "one scope dispatch, mode determines serialisation"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "adapter-dom.sx")
(td (~tw :tokens "py-2 pr-4") "render-dom-island, render-dom-lake, reactive-spread")
(td (~tw :tokens "py-2") "one render-dom-scope, mode determines lifecycle"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "engine.sx")
(td (~tw :tokens "py-2 pr-4") "morph-island-children, sync-attrs, data-sx-reactive-attrs")
(td (~tw :tokens "py-2") "morph-scope (scope boundary determines skip/update)"))
(tr
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "signals.sx")
(td (~tw :tokens "py-2 pr-4") "standalone signal runtime")
(td (~tw :tokens "py-2") "unchanged — signals are the value layer, scopes are the structure layer"))))
(p "Signals remain orthogonal. A scope's " (code ":value") " can be a signal or a plain "
"value — the scope doesn't care. The propagation mode determines whether the scope "
"re-runs effects when the value changes, not whether the value " (em "can") " change."))
;; =====================================================================
;; IX. The deepest thing
;; =====================================================================
(~docs/section :title "The deepest thing" :id "deepest"
(p "Go deep enough and there is one operation:")
(blockquote (~tw :tokens "border-l-4 border-violet-300 pl-4 my-6")
(p (~tw :tokens "text-lg font-semibold text-stone-700")
"Evaluate this expression in a scope. Handle effects that cross scope boundaries."))
(p "Rendering is evaluating expressions in a scope where " (code "emit!") " effects "
"are handled by accumulating HTML. Reactivity is re-evaluating expressions "
"when signals fire. Morphing is receiving new expressions from the server "
"and re-evaluating them in existing scopes.")
(p "Every SX mechanism — every one — is a specific answer to three questions:")
(ol (~tw :tokens "space-y-2 text-stone-600")
(li (strong "What scope") " am I in? (element / subtree / island / lake / global)")
(li (strong "What direction") " does data flow? (down via context, up via emit)")
(li (strong "When") " are effects realised? (render / signal change / network)"))
(p (code "scope") " is the primitive that makes all three questions explicit "
"and composable. It's the last primitive SX needs.")
(~docs/note
(p (strong "Status: ") "Phase 1 (" (code "provide/context/emit!") ") and "
"Phase 2 (" (code "scope") " unification) are complete. "
"574 tests pass. All four adapters use " (code "scope-push!/scope-pop!") ". "
(code "collect!") " is backed by lazy scopes with dedup. "
"Phase 3 (extensible handlers) is the research frontier — "
"it may turn out that three modes are sufficient, or it may turn out that "
"user-defined modes unlock something unexpected.")))))

236
sx/sx/plans/self-hosting-bootstrapper.sx
View File

@@ -1,236 +0,0 @@
;; ---------------------------------------------------------------------------
;; Self-Hosting Bootstrapper — py.sx (COMPLETE)
;; ---------------------------------------------------------------------------
;; @css bg-green-100 text-green-800 bg-green-50 border-green-200 text-green-700 text-green-600
(defcomp ~plans/self-hosting-bootstrapper/plan-self-hosting-bootstrapper-content ()
(~docs/page :title "Self-Hosting Bootstrapper"
;; -----------------------------------------------------------------------
;; Status banner
;; -----------------------------------------------------------------------
(div (~tw :tokens "rounded-lg bg-green-50 border border-green-200 p-4 mb-8")
(div (~tw :tokens "flex items-center gap-3")
(span (~tw :tokens "inline-flex items-center rounded-full bg-green-100 px-3 py-1 text-sm font-semibold text-green-800")
"Complete")
(p (~tw :tokens "text-green-700 text-sm")
(code "py.sx") " is implemented and verified. G0 == G1: 128/128 defines match, "
"1490 lines, 88,955 bytes — byte-for-byte identical. "
(a :href "/sx/(language.(bootstrapper.self-hosting))" (~tw :tokens "underline text-green-600 font-medium")
"See live verification."))))
;; -----------------------------------------------------------------------
;; The Idea
;; -----------------------------------------------------------------------
(~docs/section :title "The Idea" :id "idea"
(p "We have " (code "bootstrap_py.py") " — a Python program that reads "
(code ".sx") " spec files and emits " (code "sx_ref.py")
", a standalone Python evaluator. It's a compiler written in the host language.")
(p "We also have " (code "z3.sx") " — a translator written in SX itself that "
"converts SX spec declarations into SMT-LIB for Z3. It's proof that "
"SX can generate code in another language.")
(p (strong "py.sx") " — an SX-to-Python translator written in SX. "
"Feed it through the existing Python evaluator against the spec files and it produces "
(code "sx_ref.py") ". The bootstrapper bootstraps itself.")
(p (code "py.sx") " produces output identical to " (code "bootstrap_py.py")
". The Python bootstrapper is redundant. "
"The spec defines itself."))
;; -----------------------------------------------------------------------
;; Results
;; -----------------------------------------------------------------------
(~docs/section :title "Results" :id "results"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 my-4")
(table (~tw :tokens "w-full text-sm")
(thead (~tw :tokens "bg-stone-50")
(tr
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Metric")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Value")))
(tbody
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 text-stone-600") "py.sx size")
(td (~tw :tokens "px-4 py-2 font-mono") "1,182 lines"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 text-stone-600") "bootstrap_py.py size (replaced)")
(td (~tw :tokens "px-4 py-2 font-mono") "902 lines (PyEmitter class)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 text-stone-600") "Defines translated")
(td (~tw :tokens "px-4 py-2 font-mono") "128/128 exact match"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 text-stone-600") "Spec files processed")
(td (~tw :tokens "px-4 py-2 font-mono") "7 (eval, forms, render, adapter-html, adapter-sx, deps, signals)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 text-stone-600") "Output size")
(td (~tw :tokens "px-4 py-2 font-mono") "1,490 lines / 88,955 bytes"))
(tr (~tw :tokens "border-t border-stone-100 bg-green-50")
(td (~tw :tokens "px-4 py-2 font-semibold text-green-700") "G0 == G1")
(td (~tw :tokens "px-4 py-2 font-semibold text-green-700") "Identical (diff is empty)"))))))
;; -----------------------------------------------------------------------
;; Architecture
;; -----------------------------------------------------------------------
(~docs/section :title "Architecture" :id "architecture"
(p "Three bootstrapper generations:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 my-4")
(table (~tw :tokens "w-full text-sm")
(thead (~tw :tokens "bg-stone-50")
(tr
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Gen")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Source")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Runs on")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Reads")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Produces")))
(tbody
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono text-stone-600") "G0")
(td (~tw :tokens "px-4 py-2") (code "bootstrap_py.py"))
(td (~tw :tokens "px-4 py-2") "Python (manual)")
(td (~tw :tokens "px-4 py-2") (code "eval.sx") ", " (code "render.sx") ", ...")
(td (~tw :tokens "px-4 py-2") (code "sx_ref.py")))
(tr (~tw :tokens "border-t border-stone-100 bg-green-50")
(td (~tw :tokens "px-4 py-2 font-mono text-green-700") "G1")
(td (~tw :tokens "px-4 py-2") (code "py.sx"))
(td (~tw :tokens "px-4 py-2") "Python evaluator + SX")
(td (~tw :tokens "px-4 py-2") (code "eval.sx") ", " (code "render.sx") ", ...")
(td (~tw :tokens "px-4 py-2 font-semibold text-green-700") (code "sx_ref.py") " (identical)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono text-stone-600") "G2")
(td (~tw :tokens "px-4 py-2") (code "py.sx"))
(td (~tw :tokens "px-4 py-2") (code "sx_ref.py") " from G1 + SX")
(td (~tw :tokens "px-4 py-2") (code "eval.sx") ", " (code "render.sx") ", ...")
(td (~tw :tokens "px-4 py-2") (code "sx_ref.py") " (fixed-point)")))))
(p "G0 is the hand-written compiler. G1 replaces it with SX. "
"G2 proves the fixed-point: " (code "py.sx") " compiled by its own output "
"produces the same output. That's the definition of self-hosting."))
;; -----------------------------------------------------------------------
;; Translation Rules
;; -----------------------------------------------------------------------
(~docs/section :title "Translation Rules" :id "translation"
(~docs/subsection :title "Name Mangling"
(p "SX identifiers become valid Python identifiers. "
"The RENAMES dict (200+ entries) handles explicit mappings; "
"general rules handle the rest:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 my-4")
(table (~tw :tokens "w-full text-sm")
(thead (~tw :tokens "bg-stone-50")
(tr
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "SX")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Python")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Rule")))
(tbody
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "eval-expr")
(td (~tw :tokens "px-4 py-2 font-mono") "eval_expr")
(td (~tw :tokens "px-4 py-2") "kebab → snake"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "nil?")
(td (~tw :tokens "px-4 py-2 font-mono") "is_nil")
(td (~tw :tokens "px-4 py-2") "predicate rename"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "empty?")
(td (~tw :tokens "px-4 py-2 font-mono") "empty_p")
(td (~tw :tokens "px-4 py-2") "? → _p (general)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "set!")
(td (~tw :tokens "px-4 py-2 font-mono") "set_b")
(td (~tw :tokens "px-4 py-2") "! → _b"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "type")
(td (~tw :tokens "px-4 py-2 font-mono") "type_")
(td (~tw :tokens "px-4 py-2") "Python reserved word escape"))))))
(~docs/subsection :title "Special Forms"
(p "Each SX special form maps to a Python expression pattern:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 my-4")
(table (~tw :tokens "w-full text-sm")
(thead (~tw :tokens "bg-stone-50")
(tr
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "SX")
(th (~tw :tokens "px-4 py-2 text-left font-semibold text-stone-700") "Python")))
(tbody
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(if c t e)")
(td (~tw :tokens "px-4 py-2 font-mono") "(t if sx_truthy(c) else e)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(let ((a 1)) body)")
(td (~tw :tokens "px-4 py-2 font-mono") "(lambda a: body)(1)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(fn (x) body)")
(td (~tw :tokens "px-4 py-2 font-mono") "lambda x: body"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(and a b c)")
(td (~tw :tokens "px-4 py-2 font-mono") "(a if not sx_truthy(a) else ...)"))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "px-4 py-2 font-mono") "(case x \"a\" 1)")
(td (~tw :tokens "px-4 py-2 font-mono") "_sx_case(x, [(\"a\", lambda: 1)])"))))))
(~docs/subsection :title "Mutation: set! and Cell Variables"
(p "Python closures can read but not rebind outer variables. "
(code "py.sx") " detects " (code "set!") " targets that cross lambda boundaries "
"and routes them through a " (code "_cells") " dict:")
(~docs/code :src (highlight ";; SX ;; Python
(define counter def counter():
(fn () _cells = {}
(let ((n 0)) _cells['n'] = 0
(fn () return lambda: _sx_begin(
(set! n (+ n 1)) _sx_cell_set(_cells, 'n', ...),
n)))) _cells['n'])" "python"))
(p "This is the hardest translation rule. " (code "py-find-nested-set-vars")
" scans the AST for " (code "set!") " inside nested " (code "fn") " bodies. "
(code "py-emit-define-as-def") " emits a " (code "def") " with " (code "_cells")
" dict instead of a lambda.")))
;; -----------------------------------------------------------------------
;; Scope
;; -----------------------------------------------------------------------
(~docs/section :title "Scope" :id "scope"
(p (code "py.sx") " is a general-purpose SX-to-Python translator. "
"The translation rules are mechanical and apply to " (em "all") " SX, "
"not just the spec subset.")
(p "The spec files are the " (em "first test case") " — because we can verify correctness "
"by diffing G0 and G1 output. But the translator itself is target-agnostic: "
"it translates SX syntax, not SX semantics. Every SX form has a "
"straightforward Python equivalent.")
(p "The roadmap:")
(ul (~tw :tokens "list-disc pl-6 space-y-1 text-stone-700")
(li (code "py.sx") " → Python — " (strong "done"))
(li (code "js.sx") " → JavaScript (replaces " (code "bootstrap_js.py") ")")
(li (code "go.sx") " → Go (new host)")
(li (code "rs.sx") " → Rust (new host)")
(li (code "hs.sx") " → Haskell (new host)"))
(p "Each new host only needs: (1) a minimal evaluator to run "
(code "{host}.sx") " once, and (2) the platform interface in the target language."))
;; -----------------------------------------------------------------------
;; Implications
;; -----------------------------------------------------------------------
(~docs/section :title "Implications" :id "implications"
(~docs/subsection :title "Practical"
(p "One less Python file to maintain. Changes to the transpilation logic "
"are written in SX and tested with the SX test harness. The spec and its "
"compiler live in the same language."))
(~docs/subsection :title "Architectural"
(p "With " (code "z3.sx") " (SMT-LIB) and " (code "py.sx") " (Python), "
"the pattern is clear: SX translators are SX programs. "
"Adding a new target language means writing one " (code ".sx") " file, "
"not a new Python compiler."))
(~docs/subsection :title "Philosophical"
(p "A self-hosting bootstrapper is a fixed point. The spec defines behavior. "
"The translator is itself defined in terms of that behavior. Running the "
"translator on the spec produces a program that can run the translator on "
"the spec and get the same program.")
(p "Gödel showed that any sufficiently powerful formal system can encode statements "
"about itself. A self-hosting bootstrapper is the constructive version: "
"the system doesn't just " (em "talk") " about itself, it " (em "builds") " itself.")))))

62
sx/sx/plans/social-sharing.sx
View File

@@ -1,62 +0,0 @@
;; ---------------------------------------------------------------------------
;; Social Sharing
;; ---------------------------------------------------------------------------
(defcomp ~plans/social-sharing/plan-social-sharing-content ()
(~docs/page :title "Social Network Sharing"
(~docs/section :title "Context" :id "context"
(p "Rose Ash already has ActivityPub for federated social sharing. This plan adds OAuth-based sharing to mainstream social networks — Facebook, Instagram, Threads, Twitter/X, LinkedIn, and Mastodon.")
(p "All social logic lives in the " (strong "account") " microservice. Content apps get a share button that opens the account share page."))
(~docs/section :title "What remains" :id "remains"
(~docs/note "Nothing has been implemented. This is the full scope of work.")
(div (~tw :tokens "space-y-4")
(div (~tw :tokens "rounded border border-stone-200 p-4")
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "Phase 1: Data Model + Encryption")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 text-sm")
(li (code "shared/models/social_connection.py") " — SocialConnection model (user_id, platform, tokens, scopes, extra_data)")
(li (code "shared/infrastructure/social_crypto.py") " — Fernet encrypt/decrypt for tokens")
(li "Alembic migration for social_connections table")
(li "Environment variables for per-platform OAuth credentials")))
(div (~tw :tokens "rounded border border-stone-200 p-4")
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "Phase 2: Platform OAuth Clients")
(p (~tw :tokens "text-sm text-stone-600 mb-2") "All in " (code "account/services/social_platforms/") ":")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 text-sm")
(li (code "base.py") " — SocialPlatform ABC, OAuthResult, ShareResult")
(li (code "meta.py") " — Facebook + Instagram + Threads (Graph API)")
(li (code "twitter.py") " — OAuth 2.0 with PKCE")
(li (code "linkedin.py") " — LinkedIn Posts API")
(li (code "mastodon.py") " — Dynamic app registration per instance")))
(div (~tw :tokens "rounded border border-stone-200 p-4")
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "Phase 3: Account Blueprint")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 text-sm")
(li (code "account/bp/social/routes.py") " — /social/ list, /social/connect/<platform>/, /social/callback/<platform>/, /social/share/")
(li "Register before account blueprint (account has catch-all /<slug>/ route)")))
(div (~tw :tokens "rounded border border-stone-200 p-4")
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "Phase 4: Templates")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 text-sm")
(li "Social panel — platform cards, connect/disconnect")
(li "Share panel — content preview, account checkboxes, share button")
(li "Share result — per-platform success/failure with links")))
(div (~tw :tokens "rounded border border-stone-200 p-4")
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "Phase 5: Share Button in Content Apps")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 text-sm")
(li "share-button fragment from account service")
(li "Blog, events, market detail pages fetch and render the fragment")))
(div (~tw :tokens "rounded border border-stone-200 p-4")
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "Phase 6: Token Refresh + Share History")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1 text-sm")
(li "Automatic token refresh before posting")
(li "Optional social_shares table for history and duplicate prevention")))))))
;; ---------------------------------------------------------------------------
;; Isomorphic Architecture Roadmap
;; ---------------------------------------------------------------------------

213
sx/sx/plans/spec-explorer.sx
View File

@@ -1,213 +0,0 @@
;; ---------------------------------------------------------------------------
;; Spec Explorer — The Fifth Ring
;; ---------------------------------------------------------------------------
(defcomp ~plans/spec-explorer/plan-spec-explorer-content ()
(~docs/page :title "Spec Explorer"
(~docs/section :title "The Five Rings" :id "five-rings"
(p "SX has a peculiar architecture. At its centre sits a specification — a set of s-expression files that define the language. Not a description of the language. Not documentation about the language. The specification " (em "is") " the language. It is simultaneously a formal definition and executable code. You can read it as a document or run it as a program. It does not describe how to build an SX evaluator; it " (em "is") " an SX evaluator, expressed in the language it defines.")
(p "This is the nucleus. Everything else radiates outward from it.")
(div (~tw :tokens "space-y-3 my-4")
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50/50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-violet-600 text-white") "1")
(span (~tw :tokens "font-semibold text-stone-800") "The Nucleus"))
(p (~tw :tokens "text-sm text-stone-600") "The spec itself — " (code "shared/sx/ref/*.sx") ". Fourteen files, 180+ functions with type and effect annotations. Each function is simultaneously a formal definition and executable code."))
(div (~tw :tokens "rounded border border-sky-200 bg-sky-50/50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-sky-600 text-white") "2")
(span (~tw :tokens "font-semibold text-stone-800") "The Bootstrapper Ring"))
(p (~tw :tokens "text-sm text-stone-600") "Translators that read the spec and emit native implementations. " (code "bootstrap_py.py") " emits Python. " (code "js.sx") " emits JavaScript. " (code "z3.sx") " emits SMT-LIB verification conditions. The spec's knowledge is preserved in every translation. Nothing is added, nothing is lost."))
(div (~tw :tokens "rounded border border-rose-200 bg-rose-50/50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-rose-500 text-white") "3")
(span (~tw :tokens "font-semibold text-stone-800") "The Bridge Ring"))
(p (~tw :tokens "text-sm text-stone-600") "The platform boundary — " (code "boundary.sx") " is literally the membrane between the made world (the spec) and the found world (the host environment). It declares what the host must provide so the spec can function. Each spec function's dependencies trace back to either other spec functions or platform primitives."))
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50/50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-amber-500 text-white") "4")
(span (~tw :tokens "font-semibold text-stone-800") "The Runtime Ring"))
(p (~tw :tokens "text-sm text-stone-600") "Bootstrapped spec + platform bridge = working system. Tests verify behaviour. " (code "prove.sx") " verifies algebraic properties by bounded model checking. " (code "z3.sx") " translates to SMT-LIB for unbounded proofs. The spec doesn't just claim to work — it proves it."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50/50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white") "5")
(span (~tw :tokens "font-semibold text-stone-800") "The Application Ring"))
(p (~tw :tokens "text-sm text-stone-600") "This website — rendering the spec's source code using the runtime the spec produced, displayed in components written in the language the spec defines, navigated by an engine the spec specifies. The documentation is the thing documenting itself.")))
(p "The spec explorer makes all five rings visible for every function."))
;; -----------------------------------------------------------------------
;; What the explorer shows
;; -----------------------------------------------------------------------
(~docs/section :title "Per-Function Cards" :id "cards"
(p "Each function in the spec gets a card showing all five rings:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Ring")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Panel")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Content")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-violet-600 text-white") "1"))
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Nucleus")
(td (~tw :tokens "px-3 py-2 text-stone-600") "SX source with syntax highlighting. Effect badges (pure/mutation/io/render). Typed parameter list."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-sky-600 text-white") "2"))
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Translations")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Collapsible panels showing the same function in Python, JavaScript, and Z3/SMT-LIB. Each generated by the actual bootstrappers — not hand-written."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-rose-500 text-white") "3"))
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Bridge")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Cross-references: which spec functions and platform primitives this function depends on. Platform deps marked with " (code "⬡") "."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-amber-500 text-white") "4"))
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Runtime")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Tests matched from " (code "test-*.sx") " files. Proof status from " (code "prove.sx") " — sat/unknown/n/a. Algebraic properties that reference this function."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white") "5"))
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Examples")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Usage examples extracted from comments, test assertions, and curated examples. Living documentation."))))))
;; -----------------------------------------------------------------------
;; Effect system
;; -----------------------------------------------------------------------
(~docs/section :title "Effect Annotations" :id "effects"
(p "Every function in the spec now carries an " (code ":effects") " annotation declaring what kind of side effects it performs:")
(div (~tw :tokens "flex flex-wrap gap-3 my-4")
(span (~tw :tokens "inline-flex items-center gap-1 px-3 py-1 rounded bg-green-100 text-green-700 text-sm font-medium") "pure " (code ":effects []"))
(span (~tw :tokens "inline-flex items-center gap-1 px-3 py-1 rounded bg-amber-100 text-amber-700 text-sm font-medium") "mutation " (code ":effects [mutation]"))
(span (~tw :tokens "inline-flex items-center gap-1 px-3 py-1 rounded bg-orange-100 text-orange-700 text-sm font-medium") "io " (code ":effects [io]"))
(span (~tw :tokens "inline-flex items-center gap-1 px-3 py-1 rounded bg-sky-100 text-sky-700 text-sm font-medium") "render " (code ":effects [render]")))
(p "The explorer shows effect badges on each function card, and the stats bar aggregates them across the whole file. Pure functions (green) are the nucleus — no side effects, fully deterministic, safe to cache, reorder, or parallelise.")
(~docs/code :src (highlight "(define signal :effects []\n (fn ((initial-value :as any))\n (make-signal initial-value)))\n\n(define reset! :effects [mutation]\n (fn ((s :as signal) value)\n (when (signal? s)\n (let ((old (signal-value s)))\n (when (not (identical? old value))\n (signal-set-value! s value)\n (notify-subscribers s))))))" "sx")))
;; -----------------------------------------------------------------------
;; Bootstrapper translations
;; -----------------------------------------------------------------------
(~docs/section :title "Bootstrapper Translations" :id "translations"
(p "Each function is translated by the actual bootstrappers that build the production runtime. The same " (code "signal") " function shown in three target languages:")
(~docs/subsection :title "Python (via bootstrap_py.py)"
(~docs/code :src (highlight "def signal(initial_value):\n return make_signal(initial_value)" "python"))
(p (~tw :tokens "text-sm text-stone-500") (code "PyEmitter._emit_define()") " — the exact same code path that generates " (code "sx_ref.py") "."))
(~docs/subsection :title "JavaScript (via js.sx)"
(~docs/code :src (highlight "var signal = function(initial_value) {\n return make_signal(initial_value);\n};" "javascript"))
(p (~tw :tokens "text-sm text-stone-500") (code "js-emit-define") " — the self-hosting JS bootstrapper, written in SX, evaluated by the Python evaluator."))
(~docs/subsection :title "Z3 / SMT-LIB (via z3.sx)"
(~docs/code :src (highlight "; signal — Create a reactive signal container with an initial value.\n(declare-fun signal (Value) Value)" "lisp"))
(p (~tw :tokens "text-sm text-stone-500") (code "z3-translate") " — the first self-hosted bootstrapper, translating spec declarations to verification conditions for theorem provers.")))
;; -----------------------------------------------------------------------
;; Testing and proving
;; -----------------------------------------------------------------------
(~docs/section :title "Tests and Proofs" :id "runtime"
(p "Ring 4 shows that the spec does what it claims.")
(~docs/subsection :title "Tests"
(p "Test files (" (code "test-signals.sx") ", " (code "test-eval.sx") ", etc.) use the " (code "defsuite") "/" (code "deftest") " framework. The explorer matches tests to functions by suite name and shows them on the function card.")
(~docs/code :src (highlight "(defsuite \"signal basics\"\n (deftest \"creates signal with value\"\n (let ((s (signal 42)))\n (assert-equal (deref s) 42)))\n (deftest \"reset changes value\"\n (let ((s (signal 0)))\n (reset! s 99)\n (assert-equal (deref s) 99))))" "sx")))
(~docs/subsection :title "Proofs"
(p (code "prove.sx") " verifies algebraic properties of SX primitives by bounded model checking. For each " (code "define-primitive") " with a " (code ":body") ", " (code "prove-translate") " translates to SMT-LIB and verifies satisfiability by construction.")
(p "Properties from the " (code "sx-properties") " library are matched to functions and shown on their cards:")
(~docs/code :src (highlight ";; prove.sx property: +-commutative\n{:name \"+-commutative\"\n :vars (list \"a\" \"b\")\n :test (fn (a b) (= (+ a b) (+ b a)))\n :holds '(= (+ a b) (+ b a))}\n\n;; Result: verified — 1,681 ground instances tested" "sx"))))
;; -----------------------------------------------------------------------
;; Architecture
;; -----------------------------------------------------------------------
(~docs/section :title "Implementation" :id "implementation"
(p "Three layers, three increments.")
(~docs/subsection :title "Layer 1: Python helper"
(p (code "spec-explorer-data(slug)") " in " (code "helpers.py") " — parses a " (code ".sx") " spec file via " (code "parse_all()") ", extracts sections/defines/effects/params, calls each bootstrapper for per-function translations, matches tests, runs proofs.")
(p "This is the only new Python code. Everything else is SX components."))
(~docs/subsection :title "Layer 2: SX components"
(p (code "specs-explorer.sx") " — 12-15 " (code "defcomp") " components rendering the structured data:")
(div (~tw :tokens "overflow-x-auto")
(pre (~tw :tokens "text-xs bg-stone-100 rounded p-3")
(code "~specs-explorer/spec-explorer-content top-level, receives parsed data\n ~specs-explorer/spec-explorer-header filename, title, source link\n ~specs-explorer/spec-explorer-stats aggregate badges: effects, tests, proofs\n ~spec-explorer-toc section table of contents\n ~specs-explorer/spec-explorer-section one section with its defines\n ~specs-explorer/spec-explorer-define one function card (all five rings)\n ~specs-explorer/spec-effect-badge colored effect badge\n ~specs-explorer/spec-param-list typed parameter list\n ~specs-explorer/spec-ring-translations SX / Python / JS / Z3 panels\n ~specs-explorer/spec-ring-bridge cross-references + platform deps\n ~specs-explorer/spec-ring-runtime tests + proofs\n ~spec-ring-examples usage examples\n ~specs-explorer/spec-platform-interface platform primitives table"))))
(~docs/subsection :title "Layer 3: Routing"
(p "New route at " (code "/language/specs/explore/<slug>") " — parallel to existing raw source at " (code "/language/specs/<slug>") ". Each spec page gets an \"Explore\" link; the explorer gets a \"Source\" link.")))
;; -----------------------------------------------------------------------
;; Increments
;; -----------------------------------------------------------------------
(~docs/section :title "Incremental Delivery" :id "increments"
(div (~tw :tokens "space-y-4")
(div (~tw :tokens "rounded border border-stone-200 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-violet-600 text-white uppercase") "Inc 1")
(span (~tw :tokens "font-semibold text-stone-800") "Core + Translations (Ring 1-2)"))
(ul (~tw :tokens "list-disc pl-5 text-sm text-stone-600 space-y-1")
(li (code "_spec_explorer_data()") " — sections, defines, effects, params, source extraction")
(li "Python translation via " (code "PyEmitter._emit_define()"))
(li "JavaScript translation via " (code "js-emit-define") " (from " (code "run_js_sx.py") ")")
(li "Z3 translation via " (code "z3-translate") " (from " (code "z3.sx") ")")
(li "Explorer components + translation panels + routing")
(li "Test with " (code "signals.sx"))))
(div (~tw :tokens "rounded border border-stone-200 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-sky-600 text-white uppercase") "Inc 2")
(span (~tw :tokens "font-semibold text-stone-800") "Bridge + Runtime (Ring 3-4)"))
(ul (~tw :tokens "list-disc pl-5 text-sm text-stone-600 space-y-1")
(li "Cross-reference index: function→slug mapping across all spec files")
(li "Platform dependency detection (ref not in index = platform primitive)")
(li "Test file parsing: " (code "defsuite") "/" (code "deftest") " structure extraction")
(li "Test-to-function matching")
(li "Proof generation via " (code "prove-translate"))
(li "Property matching from " (code "sx-properties"))))
(div (~tw :tokens "rounded border border-stone-200 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-2")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Inc 3")
(span (~tw :tokens "font-semibold text-stone-800") "Examples + Polish (Ring 5)"))
(ul (~tw :tokens "list-disc pl-5 text-sm text-stone-600 space-y-1")
(li "Example extraction from comments + test assertions")
(li "Curated examples for key functions")
(li "Stats bar with aggregate counts")
(li "Table of contents with anchor links")
(li "Platform interface table")
(li "View switcher links between source and explorer")))))
;; -----------------------------------------------------------------------
;; The strange loop
;; -----------------------------------------------------------------------
(~docs/section :title "The Strange Loop" :id "strange-loop"
(p "When you view " (code "/language/specs/explore/eval") ", what happens is this:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-2")
(li "The SX evaluator — bootstrapped from " (code "eval.sx") " — evaluates the page definition.")
(li "The page calls " (code "spec-explorer-data(\"eval\")") ", which parses " (code "eval.sx") " using " (code "parse_all()") " — itself bootstrapped from " (code "parser.sx") ".")
(li "The parsed AST is fed to " (code "PyEmitter") " and " (code "js-emit-define") " — the same bootstrappers that produced the running evaluator.")
(li "The resulting data is rendered by SX components, written in the language " (code "eval.sx") " defines, evaluated by the runtime " (code "eval.sx") " produced.")
(li "The components use " (code "highlight") " to syntax-highlight the SX source — which is the source of the evaluator that's currently running the highlight function."))
(p "The evaluator evaluates its own definition. The parser parses its own grammar. The bootstrapper translates its own translator. The documentation documents the thing doing the documenting.")
(p "This is not a metaphor. It is the literal execution path."))))

146
sx/sx/plans/status.sx
View File

@@ -1,146 +0,0 @@
;; ---------------------------------------------------------------------------
;; Plan Status Overview
;; ---------------------------------------------------------------------------
(defcomp ~plans/status/plan-status-content ()
(~docs/page :title "Plan Status"
(p (~tw :tokens "text-lg text-stone-600 mb-6")
"Audit of all plans across the SX language and Rose Ash platform. Last updated March 2026.")
;; -----------------------------------------------------------------------
;; Completed
;; -----------------------------------------------------------------------
(~docs/section :title "Completed" :id "completed"
(div (~tw :tokens "space-y-4")
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(span (~tw :tokens "font-semibold text-stone-800") "Split Cart into Microservices"))
(p (~tw :tokens "text-sm text-stone-600") "Cart decomposed into 4 services: relations (internal, owns ContainerRelation), likes (internal, unified generic likes), orders (public, owns Order/OrderItem + SumUp checkout), and cart (thin CartItem CRUD). All three new services deployed with dedicated databases."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(span (~tw :tokens "font-semibold text-stone-800") "Ticket Purchase Through Cart"))
(p (~tw :tokens "text-sm text-stone-600") "Tickets flow through the cart like products: state=pending in cart, reserved at checkout, confirmed on payment. TicketDTO, CartSummaryDTO with ticket_count/ticket_total, CalendarService protocol methods all implemented."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(span (~tw :tokens "font-semibold text-stone-800") "Ticket UX Improvements"))
(p (~tw :tokens "text-sm text-stone-600") "+/- quantity buttons on entry pages and cart. Tickets grouped by event in cart display. Adjust quantity route, sold/basket counts, matching product card UX pattern."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(geography.(isomorphism))" (~tw :tokens "font-semibold text-green-800 underline") "Isomorphic Phase 1: Dependency Analysis"))
(p (~tw :tokens "text-sm text-stone-600") "Per-page component bundles via deps.sx. Transitive closure, scan-refs, components-needed, page-css-classes. 15 tests, bootstrapped to both hosts."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(geography.(isomorphism))" (~tw :tokens "font-semibold text-green-800 underline") "Isomorphic Phase 2: IO Detection"))
(p (~tw :tokens "text-sm text-stone-600") "Automatic IO classification. scan-io-refs, transitive-io-refs, compute-all-io-refs. Server expands IO components, serializes pure ones for client."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(geography.(isomorphism))" (~tw :tokens "font-semibold text-green-800 underline") "Isomorphic Phase 3: Client-Side Routing"))
(p (~tw :tokens "text-sm text-stone-600") "router.sx spec, page registry via <script type=\"text/sx-pages\">, client route matching, try-first/fallback to server. Pure pages render without server roundtrips."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(geography.(isomorphism))" (~tw :tokens "font-semibold text-green-800 underline") "Isomorphic Phase 4: Client Async & IO Bridge"))
(p (~tw :tokens "text-sm text-stone-600") "Server evaluates :data expressions, serializes as SX wire format. Client fetches pre-evaluated data, caches with 30s TTL, renders :content locally. 30 unit tests."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(geography.(isomorphism))" (~tw :tokens "font-semibold text-green-800 underline") "Isomorphic Phase 5: Client IO Proxy"))
(p (~tw :tokens "text-sm text-stone-600") "IO primitives (highlight, asset-url, etc.) proxied to server via registerIoDeps(). Async DOM renderer handles promises through the render tree. Components with IO deps render client-side via server round-trips — no continuations needed."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(language.(test))" (~tw :tokens "font-semibold text-green-800 underline") "Modular Test Architecture"))
(p (~tw :tokens "text-sm text-stone-600") "Per-module test specs (eval, parser, router, render) with 161 tests. Three runners: Python, Node.js, browser. 5 platform functions, everything else pure SX."))))
;; -----------------------------------------------------------------------
;; In Progress / Partial
;; -----------------------------------------------------------------------
(~docs/section :title "In Progress" :id "in-progress"
(div (~tw :tokens "space-y-4")
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-amber-600 text-white uppercase") "Partial")
(a :href "/sx/(etc.(plan.fragment-protocol))" (~tw :tokens "font-semibold text-amber-900 underline") "Fragment Protocol"))
(p (~tw :tokens "text-sm text-stone-600") "Fragment GET infrastructure works. The planned POST/sexp structured protocol for transferring component definitions between services is not yet implemented. Fragment endpoints still use legacy GET + X-Fragment-Request headers."))))
;; -----------------------------------------------------------------------
;; Not Started
;; -----------------------------------------------------------------------
(~docs/section :title "Not Started" :id "not-started"
(div (~tw :tokens "space-y-4")
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-700 text-white uppercase") "Done")
(a :href "/sx/(etc.(plan.reader-macros))" (~tw :tokens "font-semibold text-stone-800 underline") "Reader Macros"))
(p (~tw :tokens "text-sm text-stone-600") "# dispatch in parser.sx spec, Python parser.py, hand-written sx.js. Three built-ins (#;, #|...|, #') plus extensible #name dispatch. #z3 demo translates define-primitive to SMT-LIB.")
(p (~tw :tokens "text-sm text-stone-500 mt-1") "48 parser tests (SX + Python), all passing. Rebootstrapped to JS and Python."))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-stone-500 text-white uppercase") "Not Started")
(a :href "/sx/(etc.(plan.sx-activity))" (~tw :tokens "font-semibold text-stone-800 underline") "SX-Activity"))
(p (~tw :tokens "text-sm text-stone-600") "Federated SX over ActivityPub — 6 phases from SX wire format for activities to the evaluable web on IPFS. Existing AP infrastructure provides the foundation but no SX-specific federation code exists.")
(p (~tw :tokens "text-sm text-stone-500 mt-1") "Remaining: shared/sx/activity.py (SX<->JSON-LD), shared/sx/ipfs.py, shared/sx/ref/ipfs-resolve.sx, shared/sx/registry.py, shared/sx/anchor.py."))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-stone-500 text-white uppercase") "Not Started")
(a :href "/sx/(etc.(plan.glue-decoupling))" (~tw :tokens "font-semibold text-stone-800 underline") "Cross-App Decoupling via Glue"))
(p (~tw :tokens "text-sm text-stone-600") "Eliminate all cross-app model imports by routing through a glue service layer. No glue/ directory exists. Apps are currently decoupled via HTTP interfaces and DTOs instead.")
(p (~tw :tokens "text-sm text-stone-500 mt-1") "Remaining: glue/services/ for pages, page_config, calendars, marketplaces, cart_items, products, post_associations. 25+ cross-app imports to eliminate."))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-stone-500 text-white uppercase") "Not Started")
(a :href "/sx/(etc.(plan.social-sharing))" (~tw :tokens "font-semibold text-stone-800 underline") "Social Network Sharing"))
(p (~tw :tokens "text-sm text-stone-600") "OAuth-based sharing to Facebook, Instagram, Threads, Twitter/X, LinkedIn, and Mastodon via the account service. No models, blueprints, or platform clients created.")
(p (~tw :tokens "text-sm text-stone-500 mt-1") "Remaining: SocialConnection model, social_crypto.py, platform OAuth clients (6), account/bp/social/ blueprint, share button fragment."))
(div (~tw :tokens "rounded border border-green-200 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(geography.(isomorphism))" (~tw :tokens "font-semibold text-stone-800 underline") "Isomorphic Phase 6: Streaming & Suspense"))
(p (~tw :tokens "text-sm text-stone-600") "Server streams partially-evaluated SX as IO resolves. ~shared:pages/suspense component renders fallbacks, inline resolution scripts fill in content. Concurrent IO via asyncio, chunked transfer encoding.")
(p (~tw :tokens "text-sm text-stone-500 mt-1") "Demo: " (a :href "/sx/(geography.(isomorphism.streaming))" "/sx/(geography.(isomorphism.streaming))")))
(div (~tw :tokens "rounded border border-green-300 bg-green-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-green-600 text-white uppercase") "Complete")
(a :href "/sx/(geography.(isomorphism))" (~tw :tokens "font-semibold text-stone-800 underline") "Isomorphic Phase 7: Full Isomorphism"))
(p (~tw :tokens "text-sm text-stone-600") "Affinity annotations, render plans, optimistic data updates, offline mutation queue, isomorphic testing harness, universal page descriptor.")
(p (~tw :tokens "text-sm text-stone-500 mt-1") "All 6 sub-phases (7a–7f) complete."))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(div (~tw :tokens "flex items-center gap-2 mb-1")
(span (~tw :tokens "inline-block px-2 py-0.5 rounded text-xs font-bold bg-stone-500 text-white uppercase") "Not Started")
(a :href "/sx/(etc.(plan.spec-explorer))" (~tw :tokens "font-semibold text-stone-800 underline") "Spec Explorer — The Fifth Ring"))
(p (~tw :tokens "text-sm text-stone-600") "SX exploring itself. Per-function cards showing all five rings: SX source (nucleus), Python/JS/Z3 translations (bootstrapper), platform dependencies (bridge), tests and proofs (runtime), and usage examples (application). The documentation is the thing documenting itself.")
(p (~tw :tokens "text-sm text-stone-500 mt-1") "Prerequisite complete: 180+ functions annotated with :effects across all 14 spec files. Three increments: core + translations, bridge + runtime, examples + polish."))))))
;; ---------------------------------------------------------------------------
;; Fragment Protocol
;; ---------------------------------------------------------------------------

471
sx/sx/plans/sx-activity.sx
View File

@@ -1,471 +0,0 @@
;; ---------------------------------------------------------------------------
;; SX-Activity: Federated SX over ActivityPub
;; ---------------------------------------------------------------------------
(defcomp ~plans/sx-activity/plan-sx-activity-content ()
(~docs/page :title "SX-Activity"
(~docs/section :title "Context" :id "context"
(p "The web is six incompatible formats duct-taped together: HTML for structure, CSS for style, JavaScript for behavior, JSON for data, server languages for backend logic, build tools for compilation. Moving anything between layers requires serialization, template languages, API contracts, and glue code. Federation (ActivityPub) adds a seventh — JSON-LD — which is inert data that every consumer must interpret from scratch and wrap in their own UI.")
(p "SX is already one evaluable format that does all six. A component definition is simultaneously structure, style (components apply classes and respond to data), behavior (event handlers), data (the AST " (em "is") " data), server-renderable (Python evaluator), and client-renderable (JS evaluator). The pieces already exist: content-addressed DAG execution (artdag), IPFS storage with CIDs, OpenTimestamps Bitcoin anchoring, boundary-enforced sandboxing.")
(p "SX-Activity wires these together into a new web. Everything — content, UI components, markdown parsers, syntax highlighters, validation logic, media, processing pipelines — is the same executable format, stored on a content-addressed network, running within each participant's own security context. " (strong "The wire format is the programming language is the component system is the package manager.")))
(~docs/section :title "Current State" :id "current-state"
(ul (~tw :tokens "space-y-2 text-stone-700 list-disc pl-5")
(li (strong "ActivityPub: ") "Full implementation — virtual per-app actors, HTTP signatures, webfinger, inbox/outbox, followers/following, delivery with idempotent logging.")
(li (strong "Activity bus: ") "Unified event bus with NOTIFY/LISTEN wakeup, at-least-once delivery, handler registry keyed by (activity_type, object_type).")
(li (strong "Content addressing: ") "artdag nodes use SHA3-256 hashing. Cache layer tracks IPFS CIDs. IPFSPin model tracks pinned content across domains.")
(li (strong "Bitcoin anchoring: ") "APAnchor model — Merkle tree of activities, OpenTimestamps proof CID, Bitcoin txid. Infrastructure exists but isn't wired to all activity types.")
(li (strong "SX wire format: ") "Server serializes to SX source via _aser, client parses and renders. Component caching via localStorage + content hashes.")
(li (strong "Boundary enforcement: ") "SX_BOUNDARY_STRICT=1 validates all primitives at registration. Pure components can't do IO — safe to load from untrusted sources.")))
;; -----------------------------------------------------------------------
;; Phase 1: SX Wire Format for Activities
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 1: SX Wire Format for Activities" :id "phase-1"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "Activities expressed as s-expressions instead of JSON-LD. Same semantics as ActivityStreams, but compact, parseable, and directly evaluable. Dual-format support for backward compatibility with existing AP servers."))
(~docs/subsection :title "The Problem"
(p "JSON-LD activities are verbose and require context resolution:")
(~docs/code :src (highlight "{\"@context\": \"https://www.w3.org/ns/activitystreams\",\n \"type\": \"Create\",\n \"actor\": \"https://example.com/users/alice\",\n \"object\": {\n \"type\": \"Note\",\n \"content\": \"<p>Hello world</p>\",\n \"attributedTo\": \"https://example.com/users/alice\"\n }}" "json"))
(p "Every consumer parses JSON, resolves @context, extracts fields, then builds their own UI around the raw data. The content is HTML embedded in a JSON string — two formats nested, neither evaluable."))
(~docs/subsection :title "SX Activity Format"
(p "The same activity as SX:")
(~docs/code :src (highlight "(Create\n :actor \"https://example.com/users/alice\"\n :published \"2026-03-06T12:00:00Z\"\n :object (Note\n :attributed-to \"https://example.com/users/alice\"\n :content (p \"Hello world\")\n :media-type \"text/sx\"))" "lisp"))
(p "The content isn't a string containing markup — it " (em "is") " markup. The receiving server can evaluate it directly. The Note's content is a renderable SX expression."))
(~docs/subsection :title "Approach"
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. Activity vocabulary in SX")
(p "Map ActivityStreams types to SX symbols. Activities are lists with a type head and keyword properties:")
(~docs/code :src (highlight ";; Core activity types\n(Create :actor ... :object ...)\n(Update :actor ... :object ...)\n(Delete :actor ... :object ...)\n(Follow :actor ... :object ...)\n(Like :actor ... :object ...)\n(Announce :actor ... :object ...)\n\n;; Object types\n(Note :content ... :attributed-to ...)\n(Article :name ... :content ... :summary ...)\n(Image :url ... :media-type ... :cid ...)\n(Collection :total-items ... :items ...)" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. Content negotiation")
(p "Inbox accepts both formats. " (code "Accept: text/sx") " gets SX, " (code "Accept: application/activity+json") " gets JSON-LD. Outbox serves both. SX-native servers negotiate SX; legacy Mastodon/Pleroma servers get JSON-LD as today."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Bidirectional translation")
(p "Lossless mapping between JSON-LD and SX activity formats. Translate at the boundary — internal processing always uses SX. The existing " (code "APActivity") " model gains an " (code "sx_source") " column storing the canonical SX representation."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "4. HTTP Signatures over SX")
(p "Same RSA signature mechanism. Digest header computed over the SX body. Existing keypair infrastructure unchanged."))))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Round-trip: SX → JSON-LD → SX produces identical output")
(li "Legacy AP servers receive valid JSON-LD (Mastodon can display the post)")
(li "SX-native servers receive evaluable SX (client can render directly)")
(li "HTTP signatures verify over SX bodies"))))
;; -----------------------------------------------------------------------
;; Phase 2: Content-Addressed Components on IPFS
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 2: Content-Addressed Components on IPFS" :id "phase-2"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "Component definitions stored on IPFS, referenced by CID. Any server can publish components. Any browser can fetch them. No central registry — content addressing IS the registry."))
(~docs/subsection :title "The Insight"
(p "SX components are pure functions — they take data and return markup. They can't do IO (boundary enforcement guarantees this). That means they're " (strong "safe to load from any source") ". And if they're content-addressed, the CID " (em "is") " the identity — you don't need to trust the source, you just verify the hash.")
(p "Currently, component definitions travel with each page via " (code "<script type=\"text/sx\" data-components>") ". Each server bundles its own. With IPFS, components become shared infrastructure — define once, use everywhere."))
(~docs/subsection :title "Approach"
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. Component CID computation")
(p "Each " (code "defcomp") " definition gets a content address:")
(~docs/code :src (highlight ";; Component source\n(defcomp ~plans/sx-activity/card (&key title &rest children)\n (div :class \"border rounded p-4\"\n (h2 title) children))\n\n;; CID = SHA3-256 of canonical serialized form\n;; → bafy...abc123\n;; Stored: ipfs://bafy...abc123 → component source" "lisp"))
(p "Canonical form: normalize whitespace, sort keyword args alphabetically, strip comments. Same component always produces same CID regardless of formatting."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. Component references in activities")
(p "Activities declare which components they need by CID:")
(~docs/code :src (highlight "(Create\n :actor \"https://rose-ash.com/users/alice\"\n :requires (list\n \"bafy...card\" ;; ~plans/sx-activity/card component\n \"bafy...avatar\") ;; ~shared:misc/avatar component\n :object (Note\n :content (~plans/sx-activity/card :title \"Hello\"\n (~shared:misc/avatar :src \"ipfs://bafy...photo\")\n (p \"This renders with the card component.\"))))" "lisp"))
(p "The receiving browser fetches missing components from IPFS, verifies CIDs, registers them, then renders the content."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. IPFS component resolution")
(p "Client-side resolution pipeline:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Check localStorage cache (keyed by CID — cache-forever semantics)")
(li "Check local IPFS node if running (ipfs cat)")
(li "Fetch from IPFS gateway (configurable, default: dweb.link)")
(li "Verify SHA3-256 matches CID")
(li "Parse, register in component env, render")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "4. Component publication")
(p "Server-side: on component registration, compute CID and pin to IPFS. Track in " (code "IPFSPin") " model (already exists). Publish component availability via AP outbox:")
(~docs/code :src (highlight "(Create\n :actor \"https://rose-ash.com/apps/market\"\n :object (SxComponent\n :name \"~product-card\"\n :cid \"bafy...productcard\"\n :version \"1.0.0\"\n :deps (list \"bafy...card\" \"bafy...price-tag\")))" "lisp")))))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Component pinned to IPFS → fetchable via gateway → CID verifies")
(li "Browser renders federated post using IPFS-fetched components")
(li "Modified component → different CID → old content still renders with old version")
(li "Boundary enforcement: IPFS-loaded component cannot call IO primitives"))))
;; -----------------------------------------------------------------------
;; Phase 3: Federated Media & Content Store
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 3: Federated Media & Content Store" :id "phase-3"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "All media (images, video, audio, DAG outputs) stored content-addressed on IPFS. Activities reference media by CID. No hotlinking, no broken links, no dependence on the origin server staying online."))
(~docs/subsection :title "Current Mechanism"
(p "artdag already content-addresses all DAG outputs with SHA3-256 and tracks IPFS CIDs in " (code "IPFSPin") ". But media in the web platform (blog images, product photos, event banners) is stored as regular files on the origin server. Federated posts include " (code "url") " fields pointing to the origin — if the server goes down, the media is gone."))
(~docs/subsection :title "Approach"
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. Media CID pipeline")
(p "On upload: hash content → pin to IPFS → store CID in database. Activities reference media by CID alongside URL fallback:")
(~docs/code :src (highlight "(Image\n :cid \"bafy...photo123\"\n :url \"https://rose-ash.com/media/photo.jpg\" ;; fallback\n :media-type \"image/jpeg\"\n :width 1200 :height 800)" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. DAG output federation")
(p "artdag processing results (rendered video, processed images) already have CIDs. Federate them as activities:")
(~docs/code :src (highlight "(Create\n :actor \"https://rose-ash.com/users/alice\"\n :object (Artwork\n :name \"Sunset Remix\"\n :cid \"bafy...artwork\"\n :dag-cid \"bafy...dag\" ;; full DAG for reproduction\n :media-type \"video/mp4\"\n :sources (list\n (Image :cid \"bafy...src1\" :attribution \"...\")\n (Image :cid \"bafy...src2\" :attribution \"...\"))))" "lisp"))
(p "The " (code ":dag-cid") " lets anyone re-execute the processing pipeline. The artwork is both a result and a reproducible recipe."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Shared SX content store")
(p "Not just components and media — full page content can be content-addressed. An Article's body is SX, pinned to IPFS:")
(~docs/code :src (highlight "(Article\n :name \"Why S-Expressions\"\n :content-cid \"bafy...article-body\" ;; SX source on IPFS\n :requires (list \"bafy...doc-page\" \"bafy...code-block\")\n :summary \"Why SX uses s-expressions instead of HTML.\")" "lisp"))
(p "The content outlives the server. Anyone with the CID can fetch, parse, and render the article with its original components."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "4. Progressive resolution")
(p "Client resolves content progressively:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Inline content renders immediately")
(li "CID-referenced content shows placeholder → fetches from IPFS → renders")
(li "Large media uses IPFS streaming (chunked CIDs)")
(li "Integrates with Phase 6 of isomorphic plan (streaming/suspense)")))))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Origin server offline → content still resolvable via IPFS gateway")
(li "DAG CID → re-executing DAG produces identical output")
(li "Media CID verifies → tampered content rejected"))))
;; -----------------------------------------------------------------------
;; Phase 4: Component Registry & Discovery
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 4: Component Registry & Discovery" :id "phase-4"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "Federated component discovery. Servers publish component collections. Other servers follow component feeds. Like npm, but federated, content-addressed, and the packages are safe to run (pure functions, no IO)."))
(~docs/subsection :title "Approach"
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. Component collections as AP actors")
(p "Each server exposes a component registry actor:")
(~docs/code :src (highlight "(Service\n :id \"https://rose-ash.com/sx-registry\"\n :type \"SxComponentRegistry\"\n :name \"Rose Ash Components\"\n :outbox \"https://rose-ash.com/sx-registry/outbox\"\n :followers \"https://rose-ash.com/sx-registry/followers\")" "lisp"))
(p "Follow the registry to receive component updates. The outbox is a chronological feed of Create/Update/Delete activities for components."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. Component metadata")
(~docs/code :src (highlight "(SxComponent\n :name \"~data-table\"\n :cid \"bafy...datatable\"\n :version \"2.1.0\"\n :deps (list \"bafy...sortable\" \"bafy...paginator\")\n :params (list\n (dict :name \"rows\" :type \"list\" :required true)\n (dict :name \"columns\" :type \"list\" :required true)\n (dict :name \"sortable\" :type \"boolean\" :default false))\n :css-atoms (list :border :rounded :p-4 :text-sm)\n :preview-cid \"bafy...screenshot\"\n :license \"MIT\")" "lisp"))
(p "Dependencies are transitive CID references. CSS atoms declare which CSSX rules the component needs. Preview CID is a screenshot for registry browsing."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Discovery protocol")
(p "Webfinger-style lookup for components by name:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Local registry: check own component env first")
(li "Followed registries: check cached feeds from followed registries")
(li "Global search: query known registries by component name")
(li "CID resolution: if you have a CID, skip discovery — fetch directly from IPFS")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "4. Version resolution")
(p "Components are immutable (CID = identity). \"Updating\" a component publishes a new CID. Activities reference specific CIDs, so old content always renders correctly. The registry tracks version history:")
(~docs/code :src (highlight "(Update\n :actor \"https://rose-ash.com/sx-registry\"\n :object (SxComponent\n :name \"~card\"\n :cid \"bafy...card-v2\" ;; new version\n :replaces \"bafy...card-v1\" ;; previous version\n :changelog \"Added subtitle slot\"))" "lisp")))))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Follow registry → receive component Create activities → components available locally")
(li "Render post using component from foreign registry → works")
(li "Old post referencing old CID → still renders correctly with old version"))))
;; -----------------------------------------------------------------------
;; Phase 5: Bitcoin-Anchored Provenance
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 5: Bitcoin-Anchored Provenance" :id "phase-5"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What it enables")
(p (~tw :tokens "text-violet-800") "Cryptographic proof that content existed at a specific time, authored by a specific actor. Leverages the existing APAnchor/OpenTimestamps infrastructure. Unforgeable, independently verifiable, survives server shutdown."))
(~docs/subsection :title "Current Mechanism"
(p "The " (code "APAnchor") " model already batches activities into Merkle trees, stores the tree on IPFS, creates an OpenTimestamps proof, and records the Bitcoin txid. This runs but isn't surfaced to users or integrated with the full activity lifecycle."))
(~docs/subsection :title "Approach"
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "1. Automatic anchoring pipeline")
(p "Every SX activity gets queued for anchoring. Batch processor runs periodically:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Collect pending activities (content CIDs + actor signatures)")
(li "Build Merkle tree of activity hashes")
(li "Pin Merkle tree to IPFS → tree CID")
(li "Submit tree root to OpenTimestamps calendar servers")
(li "When Bitcoin confirmation arrives: store txid, update activities with anchor reference")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "2. Provenance chain in activities")
(~docs/code :src (highlight "(Create\n :actor \"https://rose-ash.com/users/alice\"\n :object (Note :content (p \"Hello\") :cid \"bafy...note\")\n :provenance (Anchor\n :tree-cid \"bafy...merkle-tree\"\n :leaf-index 42\n :ots-cid \"bafy...ots-proof\"\n :btc-txid \"abc123...def\"\n :btc-block 890123\n :anchored-at \"2026-03-06T12:00:00Z\"))" "lisp"))
(p "Any party can verify: fetch the OTS proof from IPFS, check the Merkle path from the activity's CID to the tree root, confirm the tree root is committed in the Bitcoin block."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "3. Component provenance")
(p "Components published to the registry also get anchored. This proves authorship and publication time:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "\"This component was published by rose-ash.com at block 890123\"")
(li "Prevents backdating — can't claim you published a component before you actually did")
(li "License disputes resolvable by checking anchor timestamps")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "4. Verification UI")
(p "Client-side provenance badge on federated content:")
(~docs/code :src (highlight "(defcomp ~plans/sx-activity/provenance-badge (&key anchor)\n (when anchor\n (details :class \"inline text-xs text-stone-400\"\n (summary \"✓ Anchored\")\n (dl :class \"mt-1 space-y-1\"\n (dt \"Bitcoin block\") (dd (get anchor \"btc-block\"))\n (dt \"Timestamp\") (dd (get anchor \"anchored-at\"))\n (dt \"Proof\") (dd (a :href (str \"ipfs://\" (get anchor \"ots-cid\"))\n \"OTS proof\"))))))" "lisp")))))
(~docs/subsection :title "Verification"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Activity anchored → OTS proof fetchable from IPFS → Merkle path validates → txid confirms in Bitcoin")
(li "Tampered activity → Merkle proof fails → provenance badge shows ✗")
(li "Server goes offline → provenance still verifiable (all proofs on IPFS + Bitcoin)"))))
;; -----------------------------------------------------------------------
;; Phase 6: The Evaluable Web
;; -----------------------------------------------------------------------
(~docs/section :title "Phase 6: The Evaluable Web" :id "phase-6"
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mb-4")
(p (~tw :tokens "text-violet-900 font-medium") "What this really is")
(p (~tw :tokens "text-violet-800") "Not ActivityPub-with-SX. A new web. One where everything — content, components, parsers, renderers, server logic, client logic — is the same executable format, shared on a content-addressed network, running within each participant's own security context."))
(~docs/subsection :title "The insight"
(p "The web has six layers that don't talk to each other: HTML (structure), CSS (style), JavaScript (behavior), JSON (data interchange), server frameworks (backend logic), and build tools (compilation). Each has its own syntax, its own semantics, its own ecosystem. Moving data between them requires serialization, deserialization, template languages, API contracts, type coercion, and an endless parade of glue code.")
(p "SX collapses all six into one evaluable format. A component definition is simultaneously structure, style (components apply classes and respond to data), behavior (event handlers), data (the AST is data), server-renderable (Python evaluator), and client-renderable (JS evaluator). There is no boundary between \"data\" and \"program\" — s-expressions are both.")
(p "Once that's true, " (strong "everything becomes shareable.") " Not just UI components — markdown parsers, syntax highlighters, date formatters, validation logic, layout algorithms, color systems, animation curves. Any pure function over data. All content-addressed, all on IPFS, all executable within your own security context."))
(~docs/subsection :title "What travels on the network"
(div (~tw :tokens "space-y-4")
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "Content")
(p "Blog posts, product listings, event descriptions, social media posts. Not HTML strings embedded in JSON — live SX expressions that evaluate to rendered UI. The content " (em "is") " the application."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "Components")
(p "UI building blocks: cards, tables, forms, navigation, media players. Published to IPFS, referenced by CID. A commerce site publishes " (code "~product-card") ". A blogging platform publishes " (code "~article-layout") ". A social network publishes " (code "~thread-view") ". Anyone can compose them. They're pure functions — safe to load from anywhere."))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "Parsers and transforms")
(p "A markdown parser is just an SX function: takes a string, returns an SX tree. Publish it to IPFS. Now anyone can use your markdown dialect. Same for: syntax highlighters, BBCode parsers, wiki markup, LaTeX subsets, CSV-to-table converters, JSON-to-SX adapters. " (strong "The parser ecosystem becomes shared infrastructure."))
(~docs/code :src (highlight ";; A markdown parser, published to IPFS\n;; CID: bafy...md-parser\n(define parse-markdown\n (fn (source)\n ;; tokenize → build AST → return SX tree\n ;; (parse-markdown \"# Hello\\n**bold**\")\n ;; → (h1 \"Hello\") (p (strong \"bold\"))\n ...))\n\n;; Anyone can use it in their components\n(defcomp ~plans/sx-activity/blog-post (&key markdown-source)\n (div :class \"prose\"\n (parse-markdown markdown-source)))" "lisp")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "Server-side and client-side logic")
(p "The same SX code runs on either side. A validation function published to IPFS runs server-side in Python for form processing and client-side in JavaScript for instant feedback. A price calculator runs server-side for order totals and client-side for live previews. " (em "The server/client split is a deployment decision, not a language boundary.")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700") "Media and processing pipelines")
(p "Images, video, audio — all content-addressed on IPFS. But also the " (em "processing pipelines") " that created them. artdag DAGs are SX. Publish a DAG CID alongside the output CID and anyone can verify the provenance, re-render at different resolution, or fork the pipeline for their own work."))))
(~docs/subsection :title "The security model"
(p "This only works because of boundary enforcement. Every piece of SX fetched from the network runs within the receiver's security context:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Pure functions can't do IO. ") "A component from IPFS can produce markup — it cannot read your cookies, make network requests, access localStorage, or call any IO primitive. The boundary spec (boundary.sx) is enforced at registration time. This isn't a policy — it's structural. The evaluator literally doesn't have IO primitives available when running untrusted code.")
(li (strong "IO requires explicit grant. ") "Only locally-registered IO primitives (query, frag, current-user) have access to server resources. Fetched components never see them. The host decides what capabilities to grant.")
(li (strong "Step limits cap computation. ") "Untrusted code runs with configurable eval step limits. No infinite loops, no resource exhaustion. Exceeding the limit halts evaluation and returns an error node.")
(li (strong "Content addressing prevents tampering. ") "You asked for CID X, you got CID X, the hash proves it. No MITM, no CDN poisoning, no supply chain attacks on the content itself.")
(li (strong "Provenance proves authorship. ") "Bitcoin-anchored timestamps prove who published what and when. Not \"trust me\" — independently verifiable against the Bitcoin blockchain."))
(p "This is the opposite of the npm model. npm packages run with full access to your system — a malicious package can exfiltrate secrets, install backdoors, modify the filesystem. SX components are structurally sandboxed. The worst a malicious component can do is render a " (code "(div \"haha got you\")") "."))
(~docs/subsection :title "What this replaces"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Current web")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "SX web")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Why")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "npm / package registries")
(td (~tw :tokens "px-3 py-2 text-stone-700") "IPFS + component CIDs")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Content-addressed, no central authority, structurally safe"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "CDNs for JS/CSS")
(td (~tw :tokens "px-3 py-2 text-stone-700") "IPFS gateways")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Permanent, decentralized, self-verifying"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "REST/GraphQL APIs")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SX activities over AP")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Responses are evaluable, not just data"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "HTML + CSS + JS")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SX (one format)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "No impedance mismatch, same evaluator everywhere"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Build tools (webpack, vite)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Nothing")
(td (~tw :tokens "px-3 py-2 text-stone-600") "SX evaluates directly, no compilation step"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Template languages")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Nothing")
(td (~tw :tokens "px-3 py-2 text-stone-600") "SX is the template and the language"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "JSON-LD federation")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SX federation")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Wire format is executable, content renders itself"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Trust-based package security")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Structural sandboxing")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Pure functions can't have side effects — not by policy, by construction"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Servers + hosting + DNS + TLS")
(td (~tw :tokens "px-3 py-2 text-stone-700") "IPFS CID")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Entire applications are content-addressed, no infrastructure needed"))))))
(~docs/subsection :title "Serverless applications on IPFS"
(p "The logical conclusion: " (strong "entire web applications hosted on IPFS with no server at all."))
(p "An SX application is a tree of content-addressed artifacts: a root page definition, component dependencies, media, stylesheets, parsers, transforms. Pin the root CID to IPFS and the application is live. No server, no DNS, no hosting provider, no deployment pipeline. Someone gives you a CID, you paste it into an SX-aware browser, and the application runs.")
(~docs/code :src (highlight ";; An entire blog — one CID\n;; ipfs://bafy...my-blog\n(defpage blog-home\n :path \"/\"\n :requires (list\n \"bafy...article-layout\" ;; layout component\n \"bafy...md-parser\" ;; markdown parser\n \"bafy...syntax-highlight\" ;; code highlighting\n \"bafy...nav-component\") ;; navigation\n :content\n (~article-layout\n :title \"My Blog\"\n :nav (~nav-component\n :items (list\n (dict :label \"Post 1\" :cid \"bafy...post-1\")\n (dict :label \"Post 2\" :cid \"bafy...post-2\")))\n :body (~markdown-page\n :source-cid \"bafy...homepage-md\")))" "lisp"))
(p "What this looks like in practice:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Personal sites: ") "A portfolio or blog is a handful of SX files + media. Pin to IPFS. Share the CID. No hosting costs, no domain renewal, no SSL certificates. The site is permanent.")
(li (strong "Documentation: ") "Pin your docs. They can't go offline, can't be censored, can't be altered after publication (provenance proves it). Anyone can mirror them by pinning the same CID.")
(li (strong "Collaborative applications: ") "Multiple authors contribute pages and components. Each publishes their CIDs. A root manifest composes them. Update the manifest CID to add content — old CIDs remain valid forever.")
(li (strong "Offline-first: ") "An IPFS-hosted app works the same whether you're online or have the content cached locally. The browser's SX evaluator + local IPFS node = complete offline platform.")
(li (strong "Zero-cost deployment: ") "\"Deploying\" means computing a hash. No CI/CD, no Docker, no cloud provider. Pin locally, pin to a remote IPFS node, or let others pin if they want to help host."))
(p "For applications that " (em "do") " need a server — user accounts, payments, real-time collaboration, database queries — the server provides IO primitives via the existing boundary system. The SX application fetches data from the server's IO endpoints, but the application itself (all the rendering, routing, component logic) lives on IPFS. The server is a " (em "data service") ", not an application host.")
(p "This inverts the current model. Today: server hosts the application, client is a thin renderer. SX web: IPFS hosts the application, server is an optional IO provider. " (strong "The application is the content. The content is the application. Both are just s-expressions.")))
(~docs/subsection :title "The end state"
(p "A browser with an SX evaluator and an IPFS gateway is a complete web platform. Given a CID — for a page, a post, an application — it can:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Fetch the content from IPFS")
(li "Resolve component dependencies (also from IPFS)")
(li "Resolve media (also from IPFS)")
(li "Evaluate the content (pure computation, sandboxed)")
(li "Render to DOM")
(li "Verify provenance (Bitcoin anchor)")
(li "Cache everything forever (content-addressed = immutable)"))
(p "No server needed. No DNS. No TLS certificates. No hosting provider. No build step. No framework. Just content-addressed s-expressions evaluating in a sandbox.")
(p "The server becomes optional infrastructure for " (em "IO") " — database queries, authentication, payment processing, real-time events. Everything else lives on the content-addressed network. The web stops being a collection of servers you visit and becomes a " (strong "shared evaluable space") " you participate in.")))
;; -----------------------------------------------------------------------
;; Cross-Cutting Concerns
;; -----------------------------------------------------------------------
(~docs/section :title "Cross-Cutting Concerns" :id "cross-cutting"
(~docs/subsection :title "Security"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Boundary enforcement is the foundation. ") "IPFS-fetched components are parsed and registered like any other component. SX_BOUNDARY_STRICT ensures they can't call IO primitives. A malicious component can produce ugly markup but can't exfiltrate data or make network requests.")
(li (strong "CID verification: ") "Content fetched from IPFS is hashed and compared to the expected CID before use. Tampered content is rejected.")
(li (strong "Signature chain: ") "Actor signatures (RSA/HTTP Signatures) prove authorship. Bitcoin anchors prove timing. Together they establish non-repudiable provenance.")
(li (strong "Resource limits: ") "Evaluation of untrusted components runs with step limits (max eval steps, max recursion depth). Infinite loops are caught and terminated.")))
(~docs/subsection :title "Backward Compatibility"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Content negotiation ensures legacy AP servers always receive valid JSON-LD")
(li "SX-Activity is strictly opt-in — servers that don't understand it get standard AP")
(li "Existing internal activity bus unchanged — SX format is for federation, not internal events")
(li "URL fallbacks on all media references — CID is preferred, URL is fallback")))
(~docs/subsection :title "Performance"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Component CIDs cached in localStorage forever (content-addressed = immutable)")
(li "IPFS gateway responses cached with long TTL (content can't change)")
(li "Local IPFS node (if present) eliminates gateway latency")
(li "Provenance verification is lazy — badge shows unverified until user clicks to verify")))
(~docs/subsection :title "Integration with Isomorphic Architecture"
(p "SX-Activity builds on the isomorphic architecture plan:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Phase 1 (component distribution) → IPFS replaces per-server bundles")
(li "Phase 2 (IO detection) → pure components safe for IPFS publication")
(li "Phase 3 (client routing) → client can resolve federated content without server")
(li "Phase 6 (streaming/suspense) → progressive IPFS resolution uses same infrastructure"))))
;; -----------------------------------------------------------------------
;; Critical Files
;; -----------------------------------------------------------------------
(~docs/section :title "Critical Files" :id "critical-files"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Role")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Phases")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/infrastructure/activitypub.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "AP blueprint — add SX content negotiation")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/events/bus.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Activity bus — add sx_source column, SX serialization")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/activity.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SX ↔ JSON-LD bidirectional translation (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ipfs.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Component CID computation, IPFS pinning (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2, 3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/ipfs-resolve.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client-side IPFS resolution spec (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2, 3"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/models/federation.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "IPFSPin, APAnchor models — extend for components")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2, 3, 5"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/registry.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Component registry actor, discovery protocol (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "4"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/anchor.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Anchoring pipeline — wire to activity lifecycle (new)")
(td (~tw :tokens "px-3 py-2 text-stone-600") "5"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/boot.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client boot — IPFS component loading")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2, 6"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/events/handlers/ap_delivery_handler.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Federation delivery — SX format support")
(td (~tw :tokens "px-3 py-2 text-stone-600") "1"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "artdag/core/artdag/cache.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Content addressing — shared with component CIDs")
(td (~tw :tokens "px-3 py-2 text-stone-600") "2, 3"))))))))
;; ---------------------------------------------------------------------------
;; Content-Addressed Components
;; ---------------------------------------------------------------------------

133
sx/sx/plans/sx-ci.sx
View File

@@ -1,133 +0,0 @@
;; ---------------------------------------------------------------------------
;; SX CI Pipeline
;; ---------------------------------------------------------------------------
(defcomp ~plans/sx-ci/plan-sx-ci-content ()
(~docs/page :title "SX CI Pipeline"
(p (~tw :tokens "text-stone-500 text-sm italic mb-8")
"Build, test, and deploy Rose Ash using the same language the application is written in.")
(~docs/section :title "Context" :id "context"
(p (~tw :tokens "text-stone-600")
"Rose Ash currently uses shell scripts for CI: " (code "deploy.sh") " auto-detects changed services via git diff, builds Docker images, pushes to the registry, and restarts Swarm services. " (code "dev.sh") " starts the dev environment and runs tests. These work, but they are opaque imperative scripts with no reuse, no composition, and no relationship to SX.")
(p (~tw :tokens "text-stone-600")
"The CI pipeline is the last piece of infrastructure not expressed in s-expressions. Fixing that completes the \"one representation for everything\" claim — the same language that defines the spec, the components, the pages, the essays, and the deployment config also defines the build pipeline."))
(~docs/section :title "Design" :id "design"
(p (~tw :tokens "text-stone-600")
"Pipeline definitions are " (code ".sx") " files. A minimal Python CLI runner evaluates them using " (code "sx_ref.py") ". CI-specific IO primitives (shell execution, Docker, git) are boundary-declared and only available to the pipeline runner — never to web components.")
(~docs/code :src (highlight ";; pipeline/deploy.sx\n(let ((targets (if (= (length ARGS) 0)\n (~plans/sx-ci/detect-changed :base \"HEAD~1\")\n (filter (fn (svc) (some (fn (a) (= a (get svc \"name\"))) ARGS))\n services))))\n (when (= (length targets) 0)\n (log-step \"No changes detected\")\n (exit 0))\n\n (log-step (str \"Deploying: \" (join \" \" (map (fn (s) (get s \"name\")) targets))))\n\n ;; Tests first\n (~unit-tests)\n (~sx-spec-tests)\n\n ;; Build, push, restart\n (for-each (fn (svc) (~plans/sx-ci/build-service :service svc)) targets)\n (for-each (fn (svc) (~restart-service :service svc)) targets)\n\n (log-step \"Deploy complete\"))" "lisp"))
(p (~tw :tokens "text-stone-600")
"Pipeline steps are components. " (code "~unit-tests") ", " (code "~plans/sx-ci/build-service") ", " (code "~plans/sx-ci/detect-changed") " are " (code "defcomp") " definitions that compose by nesting — the same mechanism used for page layouts, navigation, and every other piece of the system."))
(~docs/section :title "CI Primitives" :id "primitives"
(p (~tw :tokens "text-stone-600")
"New IO primitives declared in " (code "boundary.sx") ", implemented only in the CI runner context:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Primitive")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Signature")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Purpose")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shell-run")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "(command) -> dict")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Execute shell command, return " (code "{:exit N :stdout \"...\" :stderr \"...\"}") ""))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shell-run!")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "(command) -> dict")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Execute shell command, throw on non-zero exit"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "docker-build")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "(&key file tag context) -> nil")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Build Docker image"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "docker-push")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "(tag) -> nil")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Push image to registry"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "docker-restart")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "(service) -> nil")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Restart Swarm service"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "git-diff-files")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "(base head) -> list")
(td (~tw :tokens "px-3 py-2 text-stone-700") "List changed files between commits"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "git-branch")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "() -> string")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Current branch name"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "log-step")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "(message) -> nil")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Formatted pipeline output"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "fail!")
(td (~tw :tokens "px-3 py-2 font-mono text-xs text-stone-600") "(message) -> nil")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Abort pipeline with error")))))
(p (~tw :tokens "text-stone-600")
"The boundary system ensures these primitives are " (em "only") " available in the CI context. Web components cannot call " (code "shell-run!") " — the evaluator will refuse to resolve the symbol, just as it refuses to resolve any other unregistered IO primitive. The sandbox is structural, not a convention."))
(~docs/section :title "Reusable Steps" :id "steps"
(p (~tw :tokens "text-stone-600")
"Pipeline steps are components — same " (code "defcomp") " as UI components, same " (code "&key") " params, same composition by nesting:")
(~docs/code :src (highlight "(defcomp ~plans/sx-ci/detect-changed (&key base)\n (let ((files (git-diff-files (or base \"HEAD~1\") \"HEAD\")))\n (if (some (fn (f) (starts-with? f \"shared/\")) files)\n services\n (filter (fn (svc)\n (some (fn (f) (starts-with? f (str (get svc \"dir\") \"/\"))) files))\n services))))\n\n(defcomp ~plans/sx-ci/build-service (&key service)\n (let ((name (get service \"name\"))\n (tag (str registry \"/\" name \":latest\")))\n (log-step (str \"Building \" name))\n (docker-build :file (str (get service \"dir\") \"/Dockerfile\") :tag tag :context \".\")\n (docker-push tag)))\n\n(defcomp ~plans/sx-ci/bootstrap-check ()\n (log-step \"Checking bootstrapped files are up to date\")\n (shell-run! \"python shared/sx/ref/bootstrap_js.py\")\n (shell-run! \"python shared/sx/ref/bootstrap_py.py\")\n (let ((diff (shell-run \"git diff --name-only shared/static/scripts/sx-ref.js shared/sx/ref/sx_ref.py\")))\n (when (not (= (get diff \"stdout\") \"\"))\n (fail! \"Bootstrapped files are stale — rebootstrap and commit\"))))" "lisp"))
(p (~tw :tokens "text-stone-600")
"Compare this to GitHub Actions YAML, where \"reuse\" means composite actions with " (code "uses:") " references, input/output mappings, shell script blocks inside YAML strings, and a " (a :href "https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions" (~tw :tokens "text-violet-600 hover:underline") "100-page syntax reference") ". SX pipeline reuse is function composition. That is all it has ever been."))
(~docs/section :title "Pipelines" :id "pipelines"
(p (~tw :tokens "text-stone-600")
"Two primary pipelines, each a single " (code ".sx") " file:")
(div (~tw :tokens "space-y-4")
(div (~tw :tokens "rounded border border-stone-200 p-4")
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "pipeline/test.sx")
(p (~tw :tokens "text-sm text-stone-600") "Unit tests, SX spec tests (Python + Node), bootstrap staleness check, Tailwind CSS check. Run locally or in CI.")
(p (~tw :tokens "text-sm font-mono text-violet-700 mt-1") "python -m shared.sx.ci pipeline/test.sx"))
(div (~tw :tokens "rounded border border-stone-200 p-4")
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "pipeline/deploy.sx")
(p (~tw :tokens "text-sm text-stone-600") "Auto-detect changed services (or accept explicit args), run tests, build Docker images, push to registry, restart Swarm services.")
(p (~tw :tokens "text-sm font-mono text-violet-700 mt-1") "python -m shared.sx.ci pipeline/deploy.sx blog market"))))
(~docs/section :title "Why this matters" :id "why"
(p (~tw :tokens "text-stone-600")
"CI pipelines are the strongest test case for \"one representation for everything.\" GitHub Actions, GitLab CI, CircleCI — all use YAML. YAML is not a programming language. So every CI system reinvents conditionals (" (code "if:") " expressions evaluated as strings), iteration (" (code "matrix:") " strategies), composition (" (code "uses:") " references with input/output schemas), and error handling (" (code "continue-on-error:") " booleans) — all in a data format that was never designed for any of it.")
(p (~tw :tokens "text-stone-600")
"The result is a domain-specific language trapped inside YAML, with worse syntax than any language designed to be one. Every CI pipeline of sufficient complexity becomes a programming task performed in a notation that actively resists programming.")
(p (~tw :tokens "text-stone-600")
"SX pipelines use real conditionals, real functions, real composition, and real error handling — because SX is a real language. The pipeline definition and the application code are the same thing. An AI that can generate SX components can generate SX pipelines. A developer who reads SX pages can read SX deploys. The representation is universal."))
(~docs/section :title "Files" :id "files"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "File")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Purpose")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ci.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Pipeline runner CLI (~150 lines)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ci_primitives.py")
(td (~tw :tokens "px-3 py-2 text-stone-700") "CI IO primitive implementations"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "pipeline/services.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Service registry (data)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "pipeline/steps.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Reusable step components"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "pipeline/test.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Test pipeline"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "pipeline/deploy.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Deploy pipeline"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "shared/sx/ref/boundary.sx")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Add CI primitive declarations"))))))))
;; ---------------------------------------------------------------------------
;; Live Streaming — SSE & WebSocket
;; ---------------------------------------------------------------------------

157
sx/sx/plans/sx-forge.sx
View File

@@ -1,157 +0,0 @@
;; sx-forge — SX-based Git Forge
;; Plan: a Gitea/Forgejo-style git hosting platform where everything —
;; repositories, issues, pull requests, CI, permissions — is SX.
(defcomp ~plans/sx-forge/plan-sx-forge-content ()
(~docs/page :title "sx-forge: Git Forge in SX"
(~docs/section :title "Vision" :id "vision"
(p "A git forge where the entire interface, configuration, and automation layer "
"is written in SX. Repositories are browsed, issues are filed, pull requests "
"are reviewed, and CI pipelines are triggered — all through SX components "
"rendered via the same hypermedia pipeline as every other SX app.")
(p "Configuration is SX. Webhooks are SX. Access control policies are SX macros "
"that expand to permission checks. Repository templates are defcomps. "
"The forge doesn't use SX — it " (em "is") " SX."))
(~docs/section :title "Why" :id "why"
(p "Gitea/Forgejo are excellent but they're Go binaries with YAML/INI config, "
"Markdown rendering, and a template engine that's separate from the application logic. "
"Every layer speaks a different language.")
(p "sx-forge collapses these layers:")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li "Repository browsing = SX components rendering git tree objects")
(li "Issue tracking = SX forms with sx-post, stored as content-addressed SX documents")
(li "Pull requests = SX diff viewer + sx-activity for review comments")
(li "CI integration = sx-ci pipelines triggered by push hooks")
(li "Configuration = SX s-expressions, manipulated by macros")
(li "Access control = SX macros that expand to permission predicates")
(li "API = SX wire format (text/sx) alongside JSON for compatibility")))
(~docs/section :title "Architecture" :id "architecture"
(div (~tw :tokens "overflow-x-auto mt-4")
(table (~tw :tokens "w-full text-sm text-left")
(thead
(tr (~tw :tokens "border-b border-stone-200")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Layer")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Implementation")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Notes")))
(tbody (~tw :tokens "text-stone-600")
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Git backend")
(td (~tw :tokens "py-2 px-3") "libgit2 or shell-out to git")
(td (~tw :tokens "py-2 px-3") "Smart HTTP + SSH protocols. Bare repos on disk."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "UI")
(td (~tw :tokens "py-2 px-3") "SX components (defcomp)")
(td (~tw :tokens "py-2 px-3") "Tree browser, diff viewer, blame, commit log — all defcomps."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Issues / PRs")
(td (~tw :tokens "py-2 px-3") "SX documents on IPFS")
(td (~tw :tokens "py-2 px-3") "Content-addressed. Federated via sx-activity."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "CI")
(td (~tw :tokens "py-2 px-3") "sx-ci pipelines")
(td (~tw :tokens "py-2 px-3") "Push hook triggers pipeline. Results as SX components."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Auth")
(td (~tw :tokens "py-2 px-3") "OAuth2 + SX policy macros")
(td (~tw :tokens "py-2 px-3") "Permissions are macro-expanded predicates."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Config")
(td (~tw :tokens "py-2 px-3") "SX s-expressions")
(td (~tw :tokens "py-2 px-3") "forge.sx per-instance. repo.sx per-repo."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Federation")
(td (~tw :tokens "py-2 px-3") "sx-activity (ActivityPub)")
(td (~tw :tokens "py-2 px-3") "Cross-instance PRs, issues, stars, forks."))))))
(~docs/section :title "Configuration as SX" :id "config"
(p "Instance configuration is an SX file, not YAML or INI:")
(highlight "(define forge-config
{:name \"Rose Ash Forge\"
:domain \"forge.rose-ash.com\"
:ssh-port 2222
:storage {:backend :filesystem :root \"/data/repos\"}
:auth {:provider :oauth2
:issuer \"https://account.rose-ash.com\"}
:federation {:enabled true
:allowlist (list \"*.rose-ash.com\")}
:ci {:runner :sx-ci
:default-pipeline \"ci/default.sx\"}})" "lisp")
(p "Macros transform configuration:")
(highlight ";; Macro: generate mirror config from upstream
(defmacro mirror-repo (name upstream)
`(define-repo ,name
{:mirror true
:upstream ,upstream
:sync-interval 3600
:ci false}))" "lisp")
(p "Per-repository configuration lives in " (code "repo.sx") " at the repo root:")
(highlight "(define repo-config
{:default-branch \"main\"
:ci (pipeline
(stage :test (sx-ci/run \"test.sx\"))
(stage :deploy
(when-branch \"main\"
(sx-ci/deploy :target :production))))
:permissions
{:push (or (role? :maintainer) (role? :admin))
:merge-pr (and (ci-passed?) (approved-by? 1))
:admin (role? :admin)}})" "lisp"))
(~docs/section :title "SX Diff Viewer" :id "diff-viewer"
(p "Diffs rendered as SX components, not pre-formatted text:")
(highlight ";; The diff viewer is a defcomp, composable like any other
(defcomp ~plans/sx-forge/diff-view (&key (diff :as dict))
(map (fn (hunk)
(~diff-hunk
:file (get hunk \"file\")
:old-start (get hunk \"old-start\")
:new-start (get hunk \"new-start\")
:lines (get hunk \"lines\")))
(get diff \"hunks\")))" "lisp")
(p "Because diffs are SX data, macros can transform them:")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li "Syntax highlighting via the same " (code "highlight") " helper used everywhere")
(li "Inline review comments as SX forms (sx-post to comment endpoint)")
(li "Suggestion blocks — click to apply a proposed change")
(li "SX-aware diffs — show component-level changes, not just line changes")))
(~docs/section :title "Federated Forge" :id "federation"
(p "sx-activity enables cross-instance collaboration:")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li (strong "Cross-instance PRs") " — open a PR from your fork on another instance")
(li (strong "Federated issues") " — file an issue on a remote repo from your instance")
(li (strong "Stars and forks") " — ActivityPub Follow/Like activities")
(li (strong "Mirror sync") " — subscribe to upstream changes via sx-activity")
(li (strong "Review comments") " — threaded discussions federated as SX documents"))
(p "Every issue, comment, and review is a content-addressed SX document on IPFS. "
"Federation distributes references. The content is permanent and verifiable."))
(~docs/section :title "Git Operations as IO Primitives" :id "git-ops"
(p "Git operations exposed as SX IO primitives in boundary.sx:")
(highlight ";; boundary.sx additions
(io git-log (repo &key branch limit offset) list)
(io git-tree (repo ref path) list)
(io git-blob (repo ref path) string)
(io git-diff (repo base head) dict)
(io git-refs (repo) list)
(io git-commit (repo message files &key author) dict)
(io git-create-branch (repo name from) dict)
(io git-merge (repo source target &key strategy) dict)" "lisp")
(p "Pages use these directly — no controller layer, no ORM:"))
(~docs/section :title "Implementation Path" :id "implementation"
(ol (~tw :tokens "space-y-3 text-stone-600 list-decimal pl-5")
(li (strong "Phase 1: Read-only browser") " — git-tree, git-blob, git-log, git-diff as IO primitives. "
"SX components for tree view, blob view, commit log, diff view.")
(li (strong "Phase 2: Issues") " — SX forms for create/edit. Content-addressed storage. "
"Labels, milestones, assignees as SX data.")
(li (strong "Phase 3: Pull requests") " — fork model, diff + review UI, merge strategies. "
"CI status checks from sx-ci.")
(li (strong "Phase 4: CI integration") " — push hooks trigger sx-ci pipelines. "
"Results rendered as SX components on the PR page.")
(li (strong "Phase 5: Federation") " — sx-activity for cross-instance PRs, issues, stars.")
(li (strong "Phase 6: Admin") " — SX macro-based permission policies. "
"Instance and org management via SX config.")))))

288
sx/sx/plans/sx-host.sx
View File

@@ -1,288 +0,0 @@
;; sx-host: Universal Platform Primitives
;; Five irreducible IO operations beneath every host.
;; Everything above — HTTP, SQL, TLS, protocols — is SX.
(defcomp ~plans/sx-host/plan-sx-host-content ()
(~docs/page :title "sx-host: Universal Platform Primitives"
(p (~tw :tokens "text-stone-500 text-sm italic mb-8")
"Every SX host — browser, server, CLI, embedded — is the same five IO primitives. "
"Everything above that floor is SX evaluating SX. HTTP is an SX library. "
"SQL is an SX library. TLS is an SX library. The host only touches raw bytes.")
;; =====================================================================
;; The Five Primitives
;; =====================================================================
(~docs/section :title "The Five Primitives" :id "primitives"
(table (~tw :tokens "w-full mb-6 text-sm")
(thead
(tr (th (~tw :tokens "text-left p-2") "Primitive") (th (~tw :tokens "text-left p-2") "Signature") (th (~tw :tokens "text-left p-2") "What")))
(tbody
(tr (td (~tw :tokens "p-2 font-mono text-violet-700") "read") (td (~tw :tokens "p-2 font-mono text-xs") "(read channel) → bytes") (td (~tw :tokens "p-2") "Bytes in. From a socket, a file handle, stdin, a sensor."))
(tr (td (~tw :tokens "p-2 font-mono text-violet-700") "write") (td (~tw :tokens "p-2 font-mono text-xs") "(write channel bytes) → n") (td (~tw :tokens "p-2") "Bytes out. To a socket, a file handle, stdout, an actuator."))
(tr (td (~tw :tokens "p-2 font-mono text-violet-700") "store") (td (~tw :tokens "p-2 font-mono text-xs") "(store key bytes) → ok") (td (~tw :tokens "p-2") "Persist bytes by key. Filesystem, KV store, database row, IPFS block."))
(tr (td (~tw :tokens "p-2 font-mono text-violet-700") "retrieve") (td (~tw :tokens "p-2 font-mono text-xs") "(retrieve key) → bytes | nil") (td (~tw :tokens "p-2") "Get bytes by key. Same backends."))
(tr (td (~tw :tokens "p-2 font-mono text-violet-700") "hash") (td (~tw :tokens "p-2 font-mono text-xs") "(hash bytes) → digest") (td (~tw :tokens "p-2") "Content-address bytes. SHA-256. The trust anchor for everything."))))
(p "These are irreducible. You cannot implement " (code "read") " in terms of the others. "
"You cannot implement " (code "store") " without the host. You cannot implement "
(code "hash") " without CPU instructions for the algorithm. Everything else — "
"HTTP, SQL, TLS, SMTP, ActivityPub, the Postgres wire protocol — is bytes "
"interpreted by SX logic built on these five.")
(~docs/code :src (highlight
";; The platform interface — total\n(platform\n (read channel → bytes)\n (write channel bytes → n)\n (store key bytes → ok)\n (retrieve key → bytes | nil)\n (hash bytes → digest))\n\n;; Plus channel management\n(open address opts → channel) ;; obtain a channel\n(close channel → ok) ;; release a channel\n(listen address handler → server) ;; accept incoming channels"
"lisp"))
(p (code "open") ", " (code "close") ", and " (code "listen") " manage channels — "
"they're the lifecycle around " (code "read") "/" (code "write") ". "
"Whether a channel is a TCP socket, a UDP port, a serial connection, "
"or a pipe to a subprocess is determined by the address. "
"The SX layer doesn't know or care."))
;; =====================================================================
;; Everything Above Is SX
;; =====================================================================
(~docs/section :title "Everything Above Is SX" :id "layers"
(p "Each protocol is an SX library that reads and writes bytes through channels.")
(table (~tw :tokens "w-full mb-6 text-sm")
(thead
(tr (th (~tw :tokens "text-left p-2") "What") (th (~tw :tokens "text-left p-2") "Primitives Used") (th (~tw :tokens "text-left p-2") "SX Library")))
(tbody
(tr (td (~tw :tokens "p-2") "TCP connection") (td (~tw :tokens "p-2 font-mono text-xs") "open, read, write, close") (td (~tw :tokens "p-2 font-mono text-xs") "net/tcp.sx"))
(tr (td (~tw :tokens "p-2") "TLS handshake") (td (~tw :tokens "p-2 font-mono text-xs") "read, write, hash") (td (~tw :tokens "p-2 font-mono text-xs") "net/tls.sx"))
(tr (td (~tw :tokens "p-2") "HTTP server") (td (~tw :tokens "p-2 font-mono text-xs") "listen, read, write") (td (~tw :tokens "p-2 font-mono text-xs") "net/http.sx"))
(tr (td (~tw :tokens "p-2") "HTTP client") (td (~tw :tokens "p-2 font-mono text-xs") "open, read, write") (td (~tw :tokens "p-2 font-mono text-xs") "net/http-client.sx"))
(tr (td (~tw :tokens "p-2") "Postgres client") (td (~tw :tokens "p-2 font-mono text-xs") "open, read, write") (td (~tw :tokens "p-2 font-mono text-xs") "db/postgres.sx"))
(tr (td (~tw :tokens "p-2") "Redis client") (td (~tw :tokens "p-2 font-mono text-xs") "open, read, write") (td (~tw :tokens "p-2 font-mono text-xs") "db/redis.sx"))
(tr (td (~tw :tokens "p-2") "IPFS content store") (td (~tw :tokens "p-2 font-mono text-xs") "store, retrieve, hash") (td (~tw :tokens "p-2 font-mono text-xs") "net/ipfs.sx"))
(tr (td (~tw :tokens "p-2") "File system") (td (~tw :tokens "p-2 font-mono text-xs") "store, retrieve") (td (~tw :tokens "p-2 font-mono text-xs") "io/fs.sx"))
(tr (td (~tw :tokens "p-2") "DNS resolver") (td (~tw :tokens "p-2 font-mono text-xs") "open, read, write") (td (~tw :tokens "p-2 font-mono text-xs") "net/dns.sx"))
(tr (td (~tw :tokens "p-2") "WebSocket") (td (~tw :tokens "p-2 font-mono text-xs") "read, write (over HTTP upgrade)") (td (~tw :tokens "p-2 font-mono text-xs") "net/websocket.sx"))
(tr (td (~tw :tokens "p-2") "SMTP") (td (~tw :tokens "p-2 font-mono text-xs") "open, read, write") (td (~tw :tokens "p-2 font-mono text-xs") "net/smtp.sx"))
(tr (td (~tw :tokens "p-2") "ActivityPub") (td (~tw :tokens "p-2 font-mono text-xs") "HTTP + hash (signatures)") (td (~tw :tokens "p-2 font-mono text-xs") "net/activitypub.sx"))
(tr (td (~tw :tokens "p-2") "sx-sync protocol") (td (~tw :tokens "p-2 font-mono text-xs") "read, write (over WS/WebRTC)") (td (~tw :tokens "p-2 font-mono text-xs") "net/sx-sync.sx"))))
(~docs/code :src (highlight
";; HTTP server — pure SX on the five primitives\n(define http-server\n (fn (address handler)\n (listen address (fn (channel)\n (let ((request-bytes (read channel))\n (request (http-parse-request request-bytes))\n (response (handler request))\n (response-bytes (http-serialize-response response)))\n (write channel response-bytes)\n (close channel))))))\n\n;; Your app — SX all the way down\n(http-server \":8080\" (fn (req)\n (let ((page (route (get req :path))))\n (http-response 200 (render-to-html page)))))"
"lisp"))
(p "The HTTP library parses request bytes into an SX dict, "
"calls your handler (which is SX), serializes the response dict back to bytes. "
"The host never sees HTTP. It sees " (code "listen") ", " (code "read") ", "
(code "write") ", " (code "close") ". Raw bytes in, raw bytes out."))
;; =====================================================================
;; Host Implementations
;; =====================================================================
(~docs/section :title "Host Implementations" :id "hosts"
(p "Each host implements the same 5+3 primitives. The SX layer is identical across all of them.")
(table (~tw :tokens "w-full mb-6 text-sm")
(thead
(tr (th (~tw :tokens "text-left p-2") "Host") (th (~tw :tokens "text-left p-2") "read/write") (th (~tw :tokens "text-left p-2") "store/retrieve") (th (~tw :tokens "text-left p-2") "hash") (th (~tw :tokens "text-left p-2") "Target")))
(tbody
(tr (td (~tw :tokens "p-2 font-medium") "OCaml native")
(td (~tw :tokens "p-2 text-xs") "Unix.read/write")
(td (~tw :tokens "p-2 text-xs") "file IO / mmap")
(td (~tw :tokens "p-2 text-xs") "Digestif SHA256")
(td (~tw :tokens "p-2 text-xs") "Production servers"))
(tr (td (~tw :tokens "p-2 font-medium") "Rust native")
(td (~tw :tokens "p-2 text-xs") "tokio async IO")
(td (~tw :tokens "p-2 text-xs") "sled / filesystem")
(td (~tw :tokens "p-2 text-xs") "ring SHA256")
(td (~tw :tokens "p-2 text-xs") "Edge, embedded"))
(tr (td (~tw :tokens "p-2 font-medium") "Node.js")
(td (~tw :tokens "p-2 text-xs") "net.Socket")
(td (~tw :tokens "p-2 text-xs") "fs / LevelDB")
(td (~tw :tokens "p-2 text-xs") "crypto.createHash")
(td (~tw :tokens "p-2 text-xs") "Quick deploy, Lambda"))
(tr (td (~tw :tokens "p-2 font-medium") "Python/Quart")
(td (~tw :tokens "p-2 text-xs") "asyncio streams")
(td (~tw :tokens "p-2 text-xs") "aiofiles / SQLAlchemy")
(td (~tw :tokens "p-2 text-xs") "hashlib")
(td (~tw :tokens "p-2 text-xs") "Current, prototyping"))
(tr (td (~tw :tokens "p-2 font-medium") "Browser")
(td (~tw :tokens "p-2 text-xs") "WebSocket / WebRTC")
(td (~tw :tokens "p-2 text-xs") "IndexedDB / Cache API")
(td (~tw :tokens "p-2 text-xs") "SubtleCrypto")
(td (~tw :tokens "p-2 text-xs") "Client nodes"))
(tr (td (~tw :tokens "p-2 font-medium") "WASM")
(td (~tw :tokens "p-2 text-xs") "imported from host")
(td (~tw :tokens "p-2 text-xs") "imported from host")
(td (~tw :tokens "p-2 text-xs") "compiled in")
(td (~tw :tokens "p-2 text-xs") "Universal, portable"))
(tr (td (~tw :tokens "p-2 font-medium") "Embedded (Rust)")
(td (~tw :tokens "p-2 text-xs") "UART / SPI / I2C")
(td (~tw :tokens "p-2 text-xs") "flash / EEPROM")
(td (~tw :tokens "p-2 text-xs") "hardware SHA")
(td (~tw :tokens "p-2 text-xs") "IoT, sensors"))))
(p "An SX app published to sx-pub runs on any of these hosts unchanged. "
"The CIDs are the same. The evaluation is deterministic. "
"The host is an implementation detail."))
;; =====================================================================
;; The Layered Stack
;; =====================================================================
(~docs/section :title "The Layered Stack" :id "stack"
(~docs/code :src (highlight
"┌─────────────────────────────────────────────┐\n│ Your App │\n│ defpage, defhandler, defquery, defaction │\n│ defcomp, defisland, defprotocol │\n├─────────────────────────────────────────────┤\n│ Protocol Libraries (SX) │\n│ net/http.sx db/postgres.sx net/tls.sx │\n│ net/websocket.sx net/ipfs.sx io/fs.sx │\n├─────────────────────────────────────────────┤\n│ Core Libraries (SX) │\n│ signals.sx evaluator.sx render.sx │\n│ parser.sx router.sx content.sx │\n├─────────────────────────────────────────────┤\n│ Platform Primitives (5+3) │\n│ read write store retrieve hash │\n│ open close listen │\n├─────────────────────────────────────────────┤\n│ Host (native code) │\n│ OCaml / Rust / Node / Python / Browser │\n└─────────────────────────────────────────────┘"
"text"))
(p "Every layer above the platform primitives is SX, published to sx-pub, "
"content-addressed by CID. The host is the only part that varies per deployment target. "
"The host is also the only part that isn't SX — it's native code for the target platform.")
(p "This is the same architecture the browser already has. "
"The browser host provides DOM primitives (" (code "host-call, host-get, host-set!") " etc). "
"Everything above — signals, rendering, islands, routing — is SX. "
"The server host provides IO primitives (" (code "read, write, store, retrieve, hash") "). "
"Everything above — HTTP, SQL, TLS, your app — is SX. Same pattern, different primitives."))
;; =====================================================================
;; Bootstrap From sx-pub
;; =====================================================================
(~docs/section :title "Bootstrap From sx-pub" :id "bootstrap"
(p "A new server starts with the host binary and a seed peer. Everything else comes from the network.")
(~docs/code :src (highlight
";; Server bootstrap sequence\n;;\n;; 1. Host binary starts (OCaml/Rust/Node — compiled, ~5MB)\n;; Provides: read, write, store, retrieve, hash, open, close, listen\n;;\n;; 2. Fetch spec CIDs from seed peer\n;; (open \"tcp://seed.sx-web.org:4433\")\n;; (write channel (serialize (want spec-manifest-cid)))\n;; (read channel) → spec source files\n;; (store spec-cid spec-bytes) for each spec\n;;\n;; 3. Bootstrap the evaluator\n;; evaluator.sx + parser.sx + primitives.sx → working SX runtime\n;; All verified by CID before evaluation\n;;\n;; 4. Fetch protocol libraries\n;; net/http.sx, net/tls.sx, db/postgres.sx\n;; Resolved from seed peer or IPFS\n;;\n;; 5. Fetch app definition\n;; defpage, defhandler, defquery, defaction, defcomp\n;; Your app is a set of CIDs\n;;\n;; 6. Start serving\n;; (http-server \":443\" app-handler)\n;; The host calls listen → SX handles everything above"
"text"))
(p "The host binary is the only thing installed locally. "
"The evaluator, the protocol libraries, and the app all come from the network. "
"All verified by CID. If the spec at that CID evaluates correctly, the system works. "
"The bootstrap IS the proof."))
;; =====================================================================
;; Migration Path
;; =====================================================================
(~docs/section :title "Migration Path" :id "migration"
(p "The current architecture migrates incrementally. Each step is independently useful.")
(~docs/subsection :title "Step 1: SX Page Helpers"
(p "Move Python page helpers to SX. The IO bridge already supports "
(code "(service ...)") " calls from SX. Page helpers are just compositions of service calls.")
(~docs/code :src (highlight
";; Current Python:\n;; async def post_detail_data(slug):\n;; post = await fetch_data(\"blog\", \"post-by-slug\", {\"slug\": slug})\n;; related = await fetch_data(\"blog\", \"related-posts\", {\"id\": post[\"id\"]})\n;; return {\"post\": post, \"related\": related}\n\n;; SX equivalent — works today via IO bridge:\n(defhelper post-detail-data (slug)\n (let ((post (service \"blog\" \"post-by-slug\" :slug slug))\n (related (service \"blog\" \"related-posts\" :id (get post \"id\"))))\n (dict :post post :related related)))"
"lisp"))
(p "New form: " (code "defhelper") ". Registers an async data function callable from "
(code "defpage :data") ". Implementation: evaluate SX via OCaml bridge, "
"IO requests dispatched through existing bridge. ~50 LOC spec change."))
(~docs/subsection :title "Step 2: SX Config"
(p "Replace YAML config with SX. Loaded at startup, frozen after boot.")
(~docs/code :src (highlight
";; app-config.sx — replaces app-config.yaml\n(defconfig app\n {:domain \"rose-ash.com\"\n :services {:blog {:port 8001 :url \"https://blog.rose-ash.com\"}\n :cart {:port 8003 :url \"https://cart.rose-ash.com\"}}\n :features {:ipfs true :anchoring true}\n :secrets (env-get \"SECRET_KEY\")})"
"lisp"))
(p (code "env-get") " reads environment variables — secrets never stored in SX. "
"Config is SX but secrets are platform."))
(~docs/subsection :title "Step 3: Protocol Libraries"
(p "Implement HTTP, Postgres wire protocol, Redis protocol as SX libraries "
"on the five primitives. Replaces Python framework dependencies.")
(p "This is the big step. Each protocol library is substantial:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "net/http.sx") " — request/response parsing, chunked encoding, keep-alive (~800 LOC)")
(li (code "net/tls.sx") " — TLS 1.3 handshake, certificate verification (~1200 LOC)")
(li (code "db/postgres.sx") " — wire protocol, query serialization, result parsing (~600 LOC)")
(li (code "db/redis.sx") " — RESP protocol, command serialization (~200 LOC)"))
(p "Each library is independently publishable to sx-pub. "
"Any SX server can use them. They're shared infrastructure for the whole network."))
(~docs/subsection :title "Step 4: Host Binary"
(p "Minimal native binary: SX evaluator + 5+3 IO primitives. Nothing else.")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li "OCaml: CEK machine + Unix IO (~2000 LOC, single binary)")
(li "Rust: compiled evaluator + tokio IO (~3000 LOC, single binary)")
(li "Both: no Python, no Quart, no SQLAlchemy, no pip, no virtualenv"))
(p "The host binary is the only thing you install. "
"Everything above it — HTTP, TLS, SQL, your app — downloads from sx-pub on first boot."))
(~docs/subsection :title "Step 5: Self-Hosting"
(p "sx-pub itself runs on this stack. The federation infrastructure is SX. "
"The IPFS client is SX. The anchoring is SX. "
"A new node bootstraps from the network, then helps serve the network. "
"Fully self-hosting, fully federated, fully content-addressed.")))
;; =====================================================================
;; Non-Web Hosts
;; =====================================================================
(~docs/section :title "Non-Web Hosts" :id "non-web"
(p "The five primitives are not web-specific. They're IO-specific. "
"Any device that can read bytes, write bytes, and store bytes can host SX.")
(table (~tw :tokens "w-full mb-6 text-sm")
(thead
(tr (th (~tw :tokens "text-left p-2") "Host") (th (~tw :tokens "text-left p-2") "Channels") (th (~tw :tokens "text-left p-2") "Storage") (th (~tw :tokens "text-left p-2") "Use Case")))
(tbody
(tr (td (~tw :tokens "p-2 font-medium") "CLI")
(td (~tw :tokens "p-2 text-xs") "stdin/stdout/stderr, files, pipes")
(td (~tw :tokens "p-2 text-xs") "filesystem")
(td (~tw :tokens "p-2 text-xs") "Build tools, scripts, automation"))
(tr (td (~tw :tokens "p-2 font-medium") "Desktop (Tauri)")
(td (~tw :tokens "p-2 text-xs") "IPC to native windows, file dialogs")
(td (~tw :tokens "p-2 text-xs") "app data dir")
(td (~tw :tokens "p-2 text-xs") "Native apps with SX UI"))
(tr (td (~tw :tokens "p-2 font-medium") "Mobile")
(td (~tw :tokens "p-2 text-xs") "network + sensor APIs")
(td (~tw :tokens "p-2 text-xs") "SQLite / app storage")
(td (~tw :tokens "p-2 text-xs") "Apps, games, tools"))
(tr (td (~tw :tokens "p-2 font-medium") "Serverless")
(td (~tw :tokens "p-2 text-xs") "event payload in, response out")
(td (~tw :tokens "p-2 text-xs") "KV store / S3")
(td (~tw :tokens "p-2 text-xs") "Lambda, Cloudflare Workers"))
(tr (td (~tw :tokens "p-2 font-medium") "Game engine")
(td (~tw :tokens "p-2 text-xs") "GPU commands, audio buffers")
(td (~tw :tokens "p-2 text-xs") "asset cache")
(td (~tw :tokens "p-2 text-xs") "SX game logic + native rendering"))
(tr (td (~tw :tokens "p-2 font-medium") "Embedded")
(td (~tw :tokens "p-2 text-xs") "UART, SPI, I2C, GPIO")
(td (~tw :tokens "p-2 text-xs") "flash / EEPROM")
(td (~tw :tokens "p-2 text-xs") "IoT, sensors, controllers"))))
(p "Same five primitives. Same SX spec. Same content-addressed components from sx-pub. "
"A component that works in the browser works on a server works on a Raspberry Pi — "
"as long as it only uses the five primitives and the SX standard library. "
"The host provides the bytes. SX provides the meaning."))
;; =====================================================================
;; Relationship to Existing Plans
;; =====================================================================
(~docs/section :title "Relationship to Existing Plans" :id "relationships"
(table (~tw :tokens "w-full mb-6 text-sm")
(thead
(tr (th (~tw :tokens "text-left p-2") "Plan") (th (~tw :tokens "text-left p-2") "Relationship")))
(tbody
(tr (td (~tw :tokens "p-2 font-medium") "sx-web") (td (~tw :tokens "p-2") "sx-host defines what a node IS. sx-web defines how nodes CONNECT."))
(tr (td (~tw :tokens "p-2 font-medium") "Mother Language") (td (~tw :tokens "p-2") "The evaluator that runs on each host. sx-host defines the interface it evaluates against."))
(tr (td (~tw :tokens "p-2 font-medium") "Rust/WASM Host") (td (~tw :tokens "p-2") "One implementation of the five primitives. Compiles to native or WASM."))
(tr (td (~tw :tokens "p-2 font-medium") "Isolated Evaluator") (td (~tw :tokens "p-2") "Core/platform split maps directly: core = evaluator, platform = five primitives."))
(tr (td (~tw :tokens "p-2 font-medium") "sx-pub") (td (~tw :tokens "p-2") "Where the SX protocol libraries and app definitions live. Hosts fetch from sx-pub to bootstrap."))
(tr (td (~tw :tokens "p-2 font-medium") "Runtime Slicing") (td (~tw :tokens "p-2") "Slicing determines which SX libraries a host needs. Minimal host = evaluator + primitives. Full host = all protocol libraries.")))))))

197
sx/sx/plans/sx-protocol.sx
View File

@@ -1,197 +0,0 @@
;; SX Protocol — A Proposal
;; S-expressions as a universal protocol for networked hypermedia.
(defcomp ~plans/sx-protocol/plan-sx-protocol-content ()
(~docs/page :title "SX Protocol — A Proposal"
(~docs/section :title "Abstract" :id "abstract"
(p "SX is a Lisp dialect and a proposed universal protocol for networked hypermedia. "
"It replaces URLs, HTTP verbs, query strings, API query languages, and rendering layers "
"with a single unified concept: " (strong "the s-expression") ".")
(p "Everything is an expression. Everything is evaluable. Everything is composable."))
(~docs/section :title "The Problem With the Current Web" :id "problem"
(p "The modern web stack has accumulated layers of incompatible syntax to express "
"what are fundamentally the same things:")
(table (~tw :tokens "w-full text-sm border-collapse mb-4")
(thead
(tr (~tw :tokens "border-b border-stone-300")
(th (~tw :tokens "text-left py-2 pr-4") "Concern")
(th (~tw :tokens "text-left py-2 pr-4") "Current Syntax")
(th (~tw :tokens "text-left py-2") "Example")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Resource path")
(td (~tw :tokens "py-2 pr-4") "URL segments")
(td (~tw :tokens "py-2 font-mono text-xs") "/users/123/posts"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Query parameters")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "?key=value")
(td (~tw :tokens "py-2 font-mono text-xs") "?filter=published&sort=date"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "API queries")
(td (~tw :tokens "py-2 pr-4") "GraphQL / REST")
(td (~tw :tokens "py-2 font-mono text-xs") "{ posts(filter: \"published\") { title } }"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Network verb")
(td (~tw :tokens "py-2 pr-4") "HTTP method")
(td (~tw :tokens "py-2 font-mono text-xs") "GET, POST, PUT"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Real-time")
(td (~tw :tokens "py-2 pr-4") "WebSocket URL")
(td (~tw :tokens "py-2 font-mono text-xs") "wss://site.com/live"))
(tr
(td (~tw :tokens "py-2 pr-4") "Rendering")
(td (~tw :tokens "py-2 pr-4") "HTML + CSS + JS")
(td (~tw :tokens "py-2") "Three separate languages"))))
(p "Each layer invented its own syntax. None of them compose. None of them are executable. "
"None of them are data."))
(~docs/section :title "The SX Approach" :id "approach"
(h3 (~tw :tokens "text-lg font-semibold mt-6 mb-2") "URLs as S-Expressions")
(p "A conventional URL:")
(~docs/code :src (highlight "https://site.com/blog/my-post?filter=published&sort=date" "text"))
(p "As an SX expression:")
(~docs/code :src (highlight "(get.site.com.(blog.(my-post.(filter.published.sort.date))))" "lisp"))
(ul
(li "The protocol/verb is the first atom: " (code "get"))
(li "The domain follows: " (code "site.com"))
(li "Path and parameters collapse into " (strong "one unified nested structure"))
(li "No " (code "?") ", no " (code "&") ", no " (code "/") " — just lists"))
(h3 (~tw :tokens "text-lg font-semibold mt-6 mb-2") "Dots, Not Spaces")
(p "Lisp conventionally uses spaces as separators. In URLs, spaces become " (code "%20")
". SX uses dots instead, which are URL-safe and semantically meaningful — a dot between "
"two atoms is a " (strong "cons pair") ", the fundamental unit of Lisp structure.")
(~docs/code :src (highlight ";; Clean, URL-safe, valid Lisp\n(blog.(filter.published).(sort.date.desc))" "lisp"))
(h3 (~tw :tokens "text-lg font-semibold mt-6 mb-2") "Verbs Are Just Atoms")
(p "HTTP methods are not special syntax — they are simply the first element of the expression:")
(~docs/code :src (highlight "(get.site.com.(post.my-first-post)) ; read\n(post.site.com.(submit-post.(title.hello))) ; write\n(ws.site.com.(live-feed)) ; websocket / subscribe" "lisp"))
(p "No special protocol prefixes. No " (code "https://") " vs " (code "wss://")
". The verb is data, like everything else."))
(~docs/section :title "Graph-SX: Hypermedia Queries" :id "graph-sx"
(p "GraphQL was a major advance over REST, but it made two compromises:")
(ol
(li "Queries are sent as POST bodies, sacrificing cacheability and shareability")
(li "Responses are dead data — JSON that must be separately rendered"))
(p (strong "Graph-SX") " addresses both.")
(h3 (~tw :tokens "text-lg font-semibold mt-6 mb-2") "Queries Are URLs")
(p "Because SX expressions are URLs, every query is a GET request:")
(~docs/code :src (highlight ";; This is a URL and a query simultaneously\n(get.site.com.(blog.(filter.(tag.lisp)).(limit.10)))" "lisp"))
(ul
(li "Fully cacheable by CDNs")
(li "Bookmarkable and shareable")
(li "No POST body required for reads")
(li "Browser back button works correctly"))
(h3 (~tw :tokens "text-lg font-semibold mt-6 mb-2") "Responses Include Rendering")
(p "GraphQL returns data. Graph-SX returns " (strong "hypermedia")
" — data and its presentation in the same expression:")
(~docs/code :src (highlight ";; GraphQL response (dead data)\n{\"title\": \"My Post\", \"body\": \"Hello world\"}\n\n;; Graph-SX response (live hypermedia)\n(article\n (h1 \"My Post\")\n (p \"Hello world\")\n (a (href (get.site.com.(post.next-post))) \"Next\"))" "lisp"))
(p "The server returns what the resource " (strong "is") " and how to "
(strong "present") " it in one unified structure. There is no separate rendering layer.")
(h3 (~tw :tokens "text-lg font-semibold mt-6 mb-2") "Queries Are Transformations")
(p "Because SX is a full programming language, the query and the transformation "
"are the same expression:")
(~docs/code :src (highlight ";; Fetch, filter, and transform in one expression\n(map (lambda (p) (title p))\n (filter published?\n (posts (after \"2025\"))))" "lisp"))
(p "No separate processing step. No client-side data manipulation layer."))
(~docs/section :title "Components" :id "components"
(p "SX supports server-side composable components via the " (code "~") " prefix convention:")
(~docs/code :src (highlight "(~get.everything-under-the-sun)" "lisp"))
(p "A " (code "~component") " is a named server-side function that:")
(ol
(li "Receives the expression as arguments")
(li "Makes onward queries as needed")
(li "Processes and composes results")
(li "Returns hypermedia"))
(p "Components compose naturally:")
(~docs/code :src (highlight "(~page.home\n (~hero.banner)\n (~get.latest-posts.(limit.5))\n (~get.featured.(filter.pinned)))" "lisp"))
(p "This is equivalent to React Server Components — but without a framework, "
"without a build step, and without leaving Lisp."))
(~docs/section :title "Cross-Domain Composition" :id "cross-domain"
(p "Because domain and verb are just atoms, cross-domain queries are structurally "
"identical to local ones:")
(~docs/code :src (highlight ";; Local\n(post.my-first-post)\n\n;; Remote — identical structure, qualified\n(get.site.com.(post.my-first-post))\n\n;; Composed across domains\n(~render\n (get.site.com.(post.my-first-post))\n (get.cdn.com.(image.hero)))" "lisp"))
(p "Network calls are function calls. Remote resources are just namespaced expressions."))
(~docs/section :title "Self-Describing and Introspectable" :id "introspectable"
(p "Because the site is implemented in SX and served as SX, every page is introspectable:")
(~docs/code :src (highlight "(get.sx.dev.(about)) ; the about page\n(get.sx.dev.(source.(about))) ; the SX source for the about page\n(get.sx.dev.(eval.(source.about))) ; re-evaluate it live" "lisp"))
(p "The site is its own documentation. The source is always one expression away."))
(~docs/section :title "Comparison" :id "comparison"
(table (~tw :tokens "w-full text-sm border-collapse mb-4")
(thead
(tr (~tw :tokens "border-b border-stone-300")
(th (~tw :tokens "text-left py-2 pr-4") "Feature")
(th (~tw :tokens "text-left py-2 pr-4") "REST")
(th (~tw :tokens "text-left py-2 pr-4") "GraphQL")
(th (~tw :tokens "text-left py-2") "Graph-SX")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Reads are GETs")
(td (~tw :tokens "py-2 pr-4 text-green-700") "Yes")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No (POST)")
(td (~tw :tokens "py-2 text-green-700") "Yes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "CDN cacheable")
(td (~tw :tokens "py-2 pr-4 text-green-700") "Yes")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 text-green-700") "Yes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Nested queries")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 pr-4 text-green-700") "Yes")
(td (~tw :tokens "py-2 text-green-700") "Yes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Response includes rendering")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 text-green-700") "Yes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Query is a transformation")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 text-green-700") "Yes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Composable across domains")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 text-green-700") "Yes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "One syntax for everything")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 text-green-700") "Yes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Bookmarkable deep links")
(td (~tw :tokens "py-2 pr-4 text-green-700") "Yes")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 text-green-700") "Yes"))
(tr
(td (~tw :tokens "py-2 pr-4") "Self-hosting / introspectable")
(td (~tw :tokens "py-2 pr-4 text-red-700") "No")
(td (~tw :tokens "py-2 pr-4 text-stone-500") "Partial")
(td (~tw :tokens "py-2 text-green-700") "Yes")))))
(~docs/section :title "Future Direction" :id "future"
(p "The logical conclusion of SX is a " (strong "new internet protocol")
" in which the URL, the HTTP verb, the query language, the response format, "
"and the rendering layer are all unified under one evaluable expression format.")
(~docs/code :src (highlight ";; The entire network request — protocol, domain, verb, query, all one expression\n(get.sx.dev.(blog.(filter.(tag.lisp)).(limit.10)))" "lisp"))
(p "HTTP becomes one possible implementation of a more general principle:")
(blockquote (~tw :tokens "border-l-4 border-violet-300 pl-4 italic text-stone-600 my-4")
(p (strong "Evaluate this expression. Return an expression."))))
(~docs/section :title "Reference Implementation" :id "reference"
(p "SX is implemented in SX. The reference implementation is self-hosting and available at:")
(~docs/code :src (highlight "(get.sx.dev.(source.evaluator))" "lisp"))
(p (~tw :tokens "text-sm text-stone-500 mt-4 italic")
"This proposal was written in conversation with Claude (Anthropic). The ideas are the author's own."))))

186
sx/sx/plans/sx-proxy.sx
View File

@@ -1,186 +0,0 @@
;; sx-proxy — SX-based Reverse Proxy
;; Plan: a Caddy-style reverse proxy where routes, TLS, middleware,
;; and load balancing are SX s-expressions manipulated by macros.
(defcomp ~plans/sx-proxy/plan-sx-proxy-content ()
(~docs/page :title "sx-proxy: Reverse Proxy in SX"
(~docs/section :title "Vision" :id "vision"
(p "A reverse proxy where routing rules, TLS configuration, middleware chains, "
"and load balancing policies are SX. Not a config file that gets parsed — "
"actual SX that gets evaluated. Macros generate routes from service definitions. "
"The proxy config is derived from the same stack definition as the deployment.")
(p "Caddy's Caddyfile is close — it's almost a language. But it stops short. "
"No functions, no macros, no computed values, no integration with the rest of the stack. "
"sx-proxy goes all the way: the proxy configuration " (em "is") " the application."))
(~docs/section :title "Why" :id "why"
(p "Every proxy config language reinvents the same features badly:")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li "Nginx: custom DSL with if-is-evil, no real conditionals, include for composition")
(li "Caddy: Caddyfile is nice but no functions, no macros, no types")
(li "Traefik: labels on Docker containers — stringly-typed, no validation")
(li "HAProxy: line-oriented config, ACLs as string matching, no abstraction"))
(p "All of them: separate config from application. "
"SX unifies them. The proxy reads the same service definitions "
"that the orchestrator deploys."))
(~docs/section :title "Route Definitions" :id "routes"
(p "Routes as SX, with macros for common patterns:")
(highlight ";; Basic route definition
(route blog.rose-ash.com
:upstream \"http://blog:8000\"
:tls :auto)
;; Service route macro — generates from service definition
(defmacro service-route (service)
(let ((domain (service-domain service))
(port (service-port service)))
`(route ,domain
:upstream ,(str \"http://\" (get service \"name\") \":\" port)
:tls :auto
:headers {:X-Forwarded-Proto \"https\"
:X-Real-IP (client-ip)})))
;; Generate all routes from the stack definition
(define routes
(map service-route (get rose-ash-stack \"services\")))" "lisp")
(p "One macro generates all routes from the same service definitions "
"that sx-swarm uses for deployment. Change the service, the route updates."))
(~docs/section :title "Middleware as Composition" :id "middleware"
(p "Middleware chains are function composition:")
(highlight ";; Middleware are functions: request -> response -> response
(define rate-limit
(middleware :name :rate-limit
:window \"1m\" :max 100 :key (client-ip)))
(define cors
(middleware :name :cors
:origins (list \"*.rose-ash.com\")
:methods (list :GET :POST :PUT :DELETE)
:headers (list :Authorization :Content-Type)))
(define auth-required
(middleware :name :auth
:provider :oauth2
:redirect \"https://account.rose-ash.com/login\"))
;; Compose middleware chains
(define public-chain
(chain rate-limit cors))
(define private-chain
(chain rate-limit cors auth-required))
;; Apply to routes
(route account.rose-ash.com
:upstream \"http://account:8000\"
:middleware private-chain
:tls :auto)" "lisp")
(p "Middleware is data. Chains compose with " (code "chain") ". "
"Apply different chains to different routes. "
"No nginx location blocks, no Caddy handle nesting."))
(~docs/section :title "TLS Configuration" :id "tls"
(p "TLS as SX with macro-generated cert management:")
(highlight ";; Auto TLS via ACME (like Caddy)
(define tls-auto
{:provider :acme
:ca \"https://acme-v02.api.letsencrypt.org/directory\"
:email \"admin@rose-ash.com\"
:storage \"/data/certs\"})
;; Wildcard via DNS challenge
(define tls-wildcard
{:provider :acme
:ca \"https://acme-v02.api.letsencrypt.org/directory\"
:dns-challenge {:provider :cloudflare
:api-token (from-secret \"cf-token\")}
:domains (list \"*.rose-ash.com\")})
;; Per-route TLS macro
(defmacro with-tls (config &rest routes)
`(map (fn (r) (assoc r :tls ,config)) (list ,@routes)))" "lisp"))
(~docs/section :title "Load Balancing" :id "load-balancing"
(p "Load balancing policies as SX values:")
(highlight ";; Load balancing strategies
(define round-robin (lb :strategy :round-robin))
(define least-conn (lb :strategy :least-connections))
(define ip-hash (lb :strategy :ip-hash))
;; Health-checked upstream pool
(define blog-pool
(upstream-pool
:backends (list \"blog-1:8000\" \"blog-2:8000\" \"blog-3:8000\")
:strategy least-conn
:health-check {:path \"/health\"
:interval \"10s\"
:timeout \"3s\"
:unhealthy-threshold 3}))
(route blog.rose-ash.com
:upstream blog-pool
:tls :auto)" "lisp"))
(~docs/section :title "Dynamic Reconfiguration" :id "dynamic"
(p "The proxy evaluates SX — so config can be dynamic:")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li (strong "Hot reload") " — change the SX, proxy re-evaluates. No restart.")
(li (strong "API-driven") " — sx-post new routes. The proxy is an SX app with endpoints.")
(li (strong "Swarm-aware") " — subscribe to swarm-status changes, "
"auto-update upstreams when services scale or move.")
(li (strong "Feature flags") " — route traffic based on SX predicates "
"(header match, percentage split, user attribute).")
(li (strong "A/B testing") " — split traffic via SX expressions, "
"not nginx map blocks or Traefik weighted services."))
(highlight ";; Traffic splitting — 90% to v2, 10% to v1
(route blog.rose-ash.com
:upstream (traffic-split
{:weight 90 :backend \"blog-v2:8000\"}
{:weight 10 :backend \"blog-v1:8000\"})
:tls :auto)
;; Feature flag routing
(route blog.rose-ash.com
:upstream (if (header? \"X-Beta\" \"true\")
\"blog-beta:8000\"
\"blog-prod:8000\")
:tls :auto)" "lisp"))
(~docs/section :title "Integration with sx-swarm" :id "integration"
(p "sx-proxy and sx-swarm share the same service definitions:")
(highlight ";; One source of truth
(defservice blog
:image \"rose-ash/blog:latest\"
:port 8000
:domain \"blog.rose-ash.com\"
:replicas 2)
;; sx-swarm reads: image, port, replicas -> container orchestration
;; sx-proxy reads: port, domain -> route generation
;; sx-ci reads: image -> build pipeline
;; sx-forge reads: domain -> webhook URLs
;; Everything derived from the same definition.
;; Change it once, everything updates." "lisp")
(p "This is the point of using one language for everything. "
"The proxy, the orchestrator, the CI, and the forge all consume "
"the same SX service definitions. No YAML-to-Caddyfile-to-Dockerfile translation."))
(~docs/section :title "Implementation Path" :id "implementation"
(ol (~tw :tokens "space-y-3 text-stone-600 list-decimal pl-5")
(li (strong "Phase 1: Config compiler") " — SX route definitions compiled to Caddy JSON API. "
"Use Caddy as the runtime, SX as the config language.")
(li (strong "Phase 2: Middleware macros") " — defmacro for common middleware patterns. "
"Chain composition. Apply to route groups.")
(li (strong "Phase 3: Swarm integration") " — auto-generate routes from sx-swarm service defs. "
"Hot reload on swarm state changes.")
(li (strong "Phase 4: Dynamic routing") " — traffic splitting, feature flags, A/B testing. "
"SX predicates evaluated per-request.")
(li (strong "Phase 5: Native proxy") " — replace Caddy with SX-native HTTP proxy. "
"Event loop in the host language, routing logic in SX. "
"Like phase 5 of sx-swarm: the ambitious endgame."))
(p "Phase 1-3 are practical today — Caddy's JSON API is a clean compilation target. "
"The SX layer adds macros, composition, and integration that Caddyfile lacks."))))

374
sx/sx/plans/sx-pub.sx
View File

@@ -1,374 +0,0 @@
;; ---------------------------------------------------------------------------
;; sx-pub: Federated SX Publishing Protocol
;; ---------------------------------------------------------------------------
(defcomp ~sx-pub/overview-content ()
(~docs/page :title "sx-pub"
(~docs/section :title "What It Is" :id "what"
(p "A federated content publishing protocol built entirely in SX. No JSON. Content pinned to IPFS, referenced by CID. Blockchain-anchored provenance.")
(p "One actor — " (code "sx") " — publishes SX specs, platform libraries, components, and documentation as content-addressed packages. Other sx-pub servers can follow, mirror, and refer to content by CID. Human-readable paths map to immutable IPFS addresses.")
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mt-4")
(p (~tw :tokens "text-violet-900 font-medium") "The key insight")
(p (~tw :tokens "text-violet-800") "IPFS is the database. SX is the wire format. defhandlers are the endpoints. The wire format is the programming language is the component system is the package manager.")))
;; -----------------------------------------------------------------------
;; Why ActivityPub Failed — and Why LISP Fixes It
;; -----------------------------------------------------------------------
(~docs/section :title "Why ActivityPub Failed" :id "why"
(p "ActivityPub's problems aren't bugs — they're consequences of a fundamental design error: " (strong "the wire format is dead data") ". JSON-LD activities describe what happened but can't " (em "do") " anything. Every consumer must independently interpret, render, and extend. The fediverse is N incompatible applications showing the same inert data differently.")
(~docs/subsection :title "1. The Rendering Gap"
(p "An ActivityPub Note looks like this:")
(~docs/code :src "{\"type\": \"Note\",\n \"content\": \"<p>Hello <strong>world</strong></p>\",\n \"attributedTo\": \"https://example.com/users/alice\"}")
(p "The content is an HTML string embedded in a JSON string. Two formats nested, neither evaluable. Every client — Mastodon, Pleroma, Misskey, Lemmy — parses this independently and wraps it in their own UI. A poll renders differently everywhere. An event looks different everywhere. There is no shared component model because the protocol has no concept of components.")
(p "In sx-pub, the content " (em "is") " the component:")
(~docs/code :src "(Note\n :attributed-to \"https://pub.sx-web.org/pub/actor\"\n :content (p \"Hello \" (strong \"world\")))")
(p "No HTML-in-JSON. No parsing step. The receiver evaluates the s-expression and gets rendered output. The content carries its own rendering semantics."))
(~docs/subsection :title "2. Inert Data vs. Evaluable Programs"
(p "A JSON-LD activity is a description. It says \"Alice created a Note with this HTML.\" The receiving server must then decide: how do I display a Note? What if it has attachments? What about custom emoji? What about a poll? Every new feature requires every implementation to add rendering code.")
(p "An SX activity is a " (em "program") ". It evaluates to its own UI. A poll isn't a JSON blob that each client renders differently — it's a " (code "(defcomp ~poll ...)") " that evaluates identically everywhere because the CEK machine has deterministic semantics. " (strong "The wire format is the renderer.")))
(~docs/subsection :title "3. Extensions Are Impossible"
(p "ActivityPub extensions (FEPs) add new JSON fields that most implementations ignore. Want to federate a recipe, a calendar event, a code snippet with syntax highlighting? Write a spec, wait years for implementations to support it, accept that most never will.")
(p "In sx-pub, extensions are just components. Publish " (code "(defcomp ~recipe ...)") " to IPFS. Anyone who follows you gets the component. Their server can evaluate it immediately — " (strong "no implementation changes needed") ". The extension mechanism is the same as the content mechanism: pin an s-expression, reference it by CID."))
(~docs/subsection :title "4. No Content Integrity"
(p "ActivityPub has no content addressing. A server can silently edit a federated post. HTTP signatures verify the " (em "transport") " but not the " (em "content") ". There's no way to prove that what you see is what was originally published.")
(p "sx-pub content is IPFS-pinned and referenced by CID (content hash). The CID " (em "is") " the identity. Mutate a single character and the CID changes. Merkle trees of CIDs get anchored to Bitcoin via OpenTimestamps — cryptographic proof of what was published and when."))
(~docs/subsection :title "5. The N-Client Problem"
(p "Because AP content is inert, every client must build its own renderer, interaction model, and extension system. The result: Mastodon is one application, Pleroma is another, Misskey is another. They can exchange data, but the " (em "experience") " is entirely determined by the client. Rich content degrades to plain text when the client doesn't understand it.")
(p "sx-pub eliminates this entirely. Content carries its rendering logic. A server that receives " (code "(defcomp ~interactive-chart ...)") " can render it correctly without any prior knowledge of charts — because the component is self-contained, sandboxed (boundary enforcement prevents IO in pure components), and deterministic. " (strong "Every server renders the same content the same way.")))
(~docs/subsection :title "6. No Dependency System"
(p "AP has no concept of shared code. If ten ActivityPub projects all need a Markdown parser, ten teams write ten Markdown parsers. There's no mechanism to publish reusable logic that travels between servers.")
(p "sx-pub's " (code ":requires") " field on published documents declares CID dependencies. A component that uses a chart library says " (code ":requires (list \"bafy...chart-lib\")") ". Pin the dependency tree and everything works offline. " (strong "This is a package manager built into the federation protocol.") " No npm, no crates.io, no central registry — just content-addressed s-expressions on IPFS."))
(~docs/subsection :title "7. The @context Illusion"
(p "JSON-LD @context was supposed to provide semantic interoperability. In practice it's a constant source of bugs. Implementations disagree on which contexts to support, which extensions to recognise, how to resolve conflicts. The W3C ActivityStreams context document is a single point of failure that the entire fediverse depends on.")
(p "SX needs no context resolution. " (code "(Publish :actor \"...\" :object (SxDocument :cid \"...\"))") " is self-describing. The s-expression " (em "is") " the meaning. Symbols are symbols. Keywords are keywords. No indirection, no remote JSON documents, no semantic web machinery."))
(div (~tw :tokens "rounded border border-rose-200 bg-rose-50 p-4 mt-6")
(p (~tw :tokens "text-rose-900 font-medium") "The fundamental difference")
(p (~tw :tokens "text-rose-800") "ActivityPub federates " (em "descriptions") " of content. Consumers must independently figure out how to present them. sx-pub federates " (em "programs") " — self-contained, evaluable, deterministic, content-addressed. The gap between \"data\" and \"application\" disappears because in LISP, they were never separate to begin with.")))
;; -----------------------------------------------------------------------
;; Wire Format
;; -----------------------------------------------------------------------
(~docs/section :title "Wire Format" :id "wire-format"
(p "Everything is s-expressions. No JSON-LD, no @context resolution, no nested HTML-in-JSON strings.")
(~docs/subsection :title "Actor"
(~docs/code :src "(SxActor\n :id \"https://pub.sx-web.org/pub/actor\"\n :type \"SxPublisher\"\n :name \"sx\"\n :summary \"SX language — specs, platforms, components\"\n :public-key-pem \"-----BEGIN PUBLIC KEY-----\\n...\"\n :inbox \"/pub/inbox\"\n :outbox \"/pub/outbox\"\n :followers \"/pub/followers\"\n :following \"/pub/following\"\n :collections (list\n (SxCollection :name \"core-specs\" :href \"/pub/core-specs\")\n (SxCollection :name \"platforms\" :href \"/pub/platforms\")\n (SxCollection :name \"components\" :href \"/pub/components\")))"))
(~docs/subsection :title "Publish Activity"
(p "When content is published, it's pinned to IPFS and announced to followers:")
(~docs/code :src "(Publish\n :actor \"https://pub.sx-web.org/pub/actor\"\n :published \"2026-03-24T12:00:00Z\"\n :object (SxDocument\n :name \"evaluator\"\n :path \"/pub/core-specs/evaluator\"\n :cid \"bafy...eval123\"\n :content-type \"text/sx\"\n :size 48000\n :hash \"sha3-256:abc123...\"\n :requires (list \"bafy...parser\" \"bafy...primitives\")\n :summary \"CEK machine evaluator — frames, utils, step function\"))")
(p "The " (code ":requires") " field declares CID dependencies — a component that imports the parser declares that dependency. Receivers can pin the full dependency graph."))
(~docs/subsection :title "Follow — Subscribe to a Remote Server"
(~docs/code :src "(Follow\n :actor \"https://pub.sx-web.org/pub/actor\"\n :object \"https://other.example.com/pub/actor\")")
(p "When accepted, the remote server delivers " (code "(Publish ...)") " activities to our inbox. We pin their CIDs locally."))
(~docs/subsection :title "Announce — Mirror Remote Content"
(~docs/code :src "(Announce\n :actor \"https://pub.sx-web.org/pub/actor\"\n :object \"bafy...remote-cid\")")
(p "Re-publish content from a followed server to our own followers. The CID is already pinned — we're just signalling that we endorse it."))
(~docs/subsection :title "Anchor — Blockchain Provenance Record"
(~docs/code :src "(Anchor\n :tree-cid \"bafy...merkle-tree\"\n :leaf-index 42\n :ots-cid \"bafy...ots-proof\"\n :btc-txid \"abc123...def\"\n :btc-block 890123\n :anchored-at \"2026-03-24T12:00:00Z\")")
(p "Embedded in every " (code "(Publish ...)") " activity. Receivers can verify: the Merkle tree was anchored in that Bitcoin block, and the CID is a leaf in that tree. Inclusion proof is verifiable offline.")))
;; -----------------------------------------------------------------------
;; Endpoints
;; -----------------------------------------------------------------------
(~docs/section :title "Endpoints" :id "endpoints"
(p "All endpoints are " (code "defhandler") " definitions in " (code ".sx") " files. Python provides only IO primitives behind the boundary.")
(table (~tw :tokens "w-full text-sm")
(thead
(tr (~tw :tokens "border-b border-stone-200 text-left")
(th (~tw :tokens "py-2 pr-4") "Method")
(th (~tw :tokens "py-2 pr-4") "Route")
(th (~tw :tokens "py-2") "Purpose")))
(tbody (~tw :tokens "text-stone-600")
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "GET") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/actor") (td (~tw :tokens "py-2") "Actor profile"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "GET") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/outbox") (td (~tw :tokens "py-2") "Published activity feed"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "POST") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/inbox") (td (~tw :tokens "py-2") "Receive activities from remote servers"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "GET") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/⟨collection⟩") (td (~tw :tokens "py-2") "List items in a collection"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "GET") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/⟨collection⟩/⟨slug⟩") (td (~tw :tokens "py-2") "Document by path — resolves CID, fetches from IPFS"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "GET") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/cid/⟨cid⟩") (td (~tw :tokens "py-2") "Direct CID fetch (immutable, cache forever)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "POST") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/publish") (td (~tw :tokens "py-2") "Pin to IPFS + create Publish activity + announce"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "POST") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/follow") (td (~tw :tokens "py-2") "Follow a remote sx-pub server"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "GET") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/followers") (td (~tw :tokens "py-2") "Who follows us"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "GET") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/following") (td (~tw :tokens "py-2") "Who we follow"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "GET") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/.well-known/webfinger") (td (~tw :tokens "py-2") "Discovery"))
(tr
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "GET") (td (~tw :tokens "py-2 pr-4 font-mono text-xs") "/pub/anchor/⟨tree-cid⟩") (td (~tw :tokens "py-2") "Merkle tree verification")))))
;; -----------------------------------------------------------------------
;; Storage
;; -----------------------------------------------------------------------
(~docs/section :title "Storage" :id "storage"
(p "IPFS is the canonical content store. PostgreSQL is the local index.")
(table (~tw :tokens "w-full text-sm")
(thead
(tr (~tw :tokens "border-b border-stone-200 text-left")
(th (~tw :tokens "py-2 pr-4") "What")
(th (~tw :tokens "py-2") "Where")))
(tbody (~tw :tokens "text-stone-600")
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Content (SX source)") (td (~tw :tokens "py-2") "IPFS — pinned, referenced by CID"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Path → CID index") (td (~tw :tokens "py-2") "PostgreSQL"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Outbox activities") (td (~tw :tokens "py-2") "PostgreSQL"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Followers / following") (td (~tw :tokens "py-2") "PostgreSQL"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Anchor records") (td (~tw :tokens "py-2") "PostgreSQL"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Remote mirrored content") (td (~tw :tokens "py-2") "PostgreSQL + IPFS (pinned locally)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 pr-4") "Content cache") (td (~tw :tokens "py-2") "Redis"))
(tr
(td (~tw :tokens "py-2 pr-4") "Actor keypair") (td (~tw :tokens "py-2") "PostgreSQL")))))
;; -----------------------------------------------------------------------
;; SX / Python Split
;; -----------------------------------------------------------------------
(~docs/section :title "SX / Python Split" :id "split"
(p "SX handles all composition, logic, and rendering. Python provides IO primitives behind the boundary.")
(div (~tw :tokens "grid grid-cols-2 gap-6")
(div
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "Pure SX (defhandler + defcomp)")
(ul (~tw :tokens "text-sm text-stone-600 space-y-1 list-disc pl-5")
(li "All HTTP routing and request dispatch")
(li "Content negotiation (Accept header)")
(li "Response construction (SX actor docs, collections)")
(li "UI components for browsing")
(li "Activity serialization")
(li "Nav data for /pub/ sections")))
(div
(h4 (~tw :tokens "font-semibold text-stone-700 mb-2") "Python Helpers (IO boundary)")
(ul (~tw :tokens "text-sm text-stone-600 space-y-1 list-disc pl-5")
(li (code "pub-ipfs-add") " / " (code "pub-ipfs-get") " / " (code "pub-ipfs-pin") " — IPFS ops")
(li (code "pub-db-query") " / " (code "pub-db-insert") " — database")
(li (code "pub-sign") " / " (code "pub-verify-signature") " — HTTP signatures")
(li (code "pub-hash") " — SHA3-256")
(li (code "pub-anchor-batch") " — OpenTimestamps")
(li (code "pub-deliver") " — POST to follower inboxes")))))
;; -----------------------------------------------------------------------
;; Flows
;; -----------------------------------------------------------------------
(~docs/section :title "Flows" :id "flows"
(~docs/subsection :title "Publish"
(p (~tw :tokens "text-sm text-stone-600 mb-2") "Publishing is " (strong "anchor-then-announce") " — content is provenance-sealed before anyone sees it. This prevents a malicious follower from seeing content, anchoring it first, and claiming provenance.")
(ol (~tw :tokens "text-sm text-stone-600 space-y-2 list-decimal pl-5")
(li (code "POST /pub/publish") " with collection, slug, SX content")
(li "Pin to IPFS → get CID (content is now addressable but not announced)")
(li "Store path→CID in DB (status: " (code "anchoring") ")")
(li "Build Merkle tree, submit to OpenTimestamps, wait for Bitcoin confirmation")
(li "On confirmation: update status to " (code "anchored") ", record proof")
(li "Construct " (code "(Publish ...)") " with anchor proof embedded")
(li "Deliver to follower inboxes — " (strong "only now") " is it public"))
(p (~tw :tokens "text-sm text-stone-500 mt-2 italic") "This means publishing isn't instant — there's a delay while the anchor confirms. That's a feature: it forces a deliberate pace and makes provenance airtight.")
(~docs/code :src "(Publish\n :actor \"https://pub.sx-web.org/pub/actor\"\n :published \"2026-03-24T12:00:00Z\"\n :object (SxDocument\n :name \"evaluator\"\n :cid \"bafy...eval123\"\n :anchor (Anchor\n :tree-cid \"bafy...merkle\"\n :ots-cid \"bafy...proof\"\n :btc-txid \"abc123...\"\n :btc-block 890123)))")
(p (~tw :tokens "text-sm text-stone-600") "Receivers verify: the CID was anchored in that Bitcoin block " (em "before") " the Publish was sent. No way to backdate."))
(~docs/subsection :title "Follow"
(ol (~tw :tokens "text-sm text-stone-600 space-y-2 list-decimal pl-5")
(li (code "POST /pub/follow") " with remote actor URL")
(li "Fetch remote actor document (" (code "text/sx") ")")
(li "Send " (code "(Follow ...)") " to their inbox, signed with our key")
(li "They verify signature, add us to followers, send " (code "(Accept ...)"))
(li "We store them in our following list")
(li "They start delivering " (code "(Publish ...)") " activities to our inbox")))
(~docs/subsection :title "Receive (inbox)"
(ol (~tw :tokens "text-sm text-stone-600 space-y-2 list-decimal pl-5")
(li "Remote server POSTs " (code "(Publish ...)") " to our inbox")
(li "Verify HTTP signature against their actor's public key")
(li "Pin the referenced CID to our local IPFS node")
(li "Index the remote content in our DB (marked as mirrored)")
(li "Now we can resolve that CID locally — " (code "/pub/cid/bafy...") " just works")))
(~docs/subsection :title "Anchoring"
(p (~tw :tokens "text-sm text-stone-600 mb-2") "Anchoring is integrated into the publish flow — not a separate background job. Content waits for provenance before going public.")
(ol (~tw :tokens "text-sm text-stone-600 space-y-2 list-decimal pl-5")
(li "Batch pending CIDs into a Merkle tree (multiple publishes can share one tree)")
(li "Pin Merkle tree to IPFS")
(li "Submit Merkle root to OpenTimestamps calendar servers")
(li "Poll for Bitcoin confirmation (typically ~1 hour)")
(li "On confirmation: store proof, transition content to " (code "anchored") " status")
(li "Trigger delivery of Publish activities for all items in the batch"))
(p (~tw :tokens "text-sm text-stone-500 mt-2 italic") "Batching amortises the anchor cost — one Bitcoin attestation can seal hundreds of CIDs via the Merkle tree. Each CID gets an inclusion proof linking it to the anchored root.")))
;; -----------------------------------------------------------------------
;; CID ↔ Path
;; -----------------------------------------------------------------------
(~docs/section :title "CID ↔ Path Translation" :id "cid-path"
(p "Published SX refers to dependencies by CID — the immutable, universal identifier. But humans browse by path.")
(~docs/code :src ";; Canonical SX (stored in IPFS) references by CID\n(defcomp ~my-app/dashboard (&key data)\n ;; This component depends on a chart library published at another CID\n (let ((chart-lib (require \"bafy...chart-lib-cid\")))\n (div :class \"dashboard\"\n (chart-lib/bar-chart :data data))))\n\n;; But on pub.sx-web.org you browse by path:\n;; /pub/components/dashboard\n;; /pub/core-specs/evaluator\n;; /pub/platforms/javascript\n;;\n;; The DB maps: path → CID → IPFS content\n;; Following a remote server maps: their-path → CID → our local pin")
(p "When you follow another sx-pub server and they publish a component, you get the " (code "(Publish ...)") " activity with the CID. Your server pins it. Now " (code "(require \"bafy...\")") " resolves locally — no network round-trip, no central registry, no package manager."))
;; -----------------------------------------------------------------------
;; Phases
;; -----------------------------------------------------------------------
(~docs/section :title "Implementation Phases" :id "phases"
(div (~tw :tokens "space-y-4")
(div (~tw :tokens "rounded border border-emerald-200 bg-emerald-50 p-4")
(h4 (~tw :tokens "font-semibold text-emerald-800") "Phase 1: Foundation ✓")
(p (~tw :tokens "text-emerald-700 text-sm") "DB schema, async IPFS client, actor endpoint, webfinger, " (code "/pub/actor") " returns SX actor document."))
(div (~tw :tokens "rounded border border-sky-200 bg-sky-50 p-4")
(h4 (~tw :tokens "font-semibold text-sky-800") "Phase 2: Publishing ✓")
(p (~tw :tokens "text-sky-700 text-sm") "Pin to IPFS, path→CID index, collection browsing. Publish the actual SX spec files as the first content."))
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4")
(h4 (~tw :tokens "font-semibold text-violet-800") "Phase 3: Federation ✓")
(p (~tw :tokens "text-violet-700 text-sm") "Inbox/outbox, follow/accept, HTTP signature verification, activity delivery, content mirroring."))
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-4")
(h4 (~tw :tokens "font-semibold text-amber-800") "Phase 4: Anchoring ✓")
(p (~tw :tokens "text-amber-700 text-sm") "Merkle trees, OpenTimestamps, Bitcoin proof, provenance verification."))))
;; -----------------------------------------------------------------------
;; Live Dashboard
;; -----------------------------------------------------------------------
(~docs/section :title "Live Dashboard" :id "dashboard"
(p "Live data from the sx-pub API — server-rendered from the same endpoints.")
;; --- Status ---
(~docs/subsection :title "Server Status"
(let ((status (helper "pub-status-data")))
(div (~tw :tokens "grid grid-cols-2 sm:grid-cols-4 gap-3")
(div (~tw :tokens "rounded border border-stone-200 p-3 text-center")
(p (~tw :tokens "text-xs text-stone-400 uppercase") "DB")
(p (~tw :tokens "font-semibold text-sm") (get status "db")))
(div (~tw :tokens "rounded border border-stone-200 p-3 text-center")
(p (~tw :tokens "text-xs text-stone-400 uppercase") "IPFS")
(p (~tw :tokens "font-semibold text-sm") (get status "ipfs")))
(div (~tw :tokens "rounded border border-stone-200 p-3 text-center")
(p (~tw :tokens "text-xs text-stone-400 uppercase") "Actor")
(p (~tw :tokens "font-semibold text-sm") (get status "actor")))
(div (~tw :tokens "rounded border border-stone-200 p-3 text-center")
(p (~tw :tokens "text-xs text-stone-400 uppercase") "Domain")
(p (~tw :tokens "font-semibold text-sm") (or (get status "domain") "—"))))))
;; --- Actor ---
(~docs/subsection :title "Actor Identity"
(let ((actor (helper "pub-actor-data")))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4 space-y-2")
(div (~tw :tokens "flex items-center gap-3")
(div (~tw :tokens "w-10 h-10 rounded-full bg-violet-100 flex items-center justify-center text-violet-700 font-bold") "sx")
(div
(p (~tw :tokens "font-semibold") (get actor "display-name"))
(p (~tw :tokens "text-sm text-stone-500") (str "@" (get actor "preferred-username") "@" (get actor "domain")))))
(p (~tw :tokens "text-sm text-stone-600") (get actor "summary"))
(details (~tw :tokens "text-xs")
(summary (~tw :tokens "text-stone-400 cursor-pointer") "Public key")
(pre (~tw :tokens "mt-2 bg-stone-100 rounded p-2 text-xs overflow-x-auto") (get actor "public-key-pem"))))))
;; --- Collections ---
(~docs/subsection :title "Collections"
(let ((collections (helper "pub-collections-data")))
(div (~tw :tokens "grid gap-3")
(map (fn (c)
(div (~tw :tokens "rounded border border-stone-200 p-4 hover:border-violet-300 transition-colors")
(div (~tw :tokens "flex items-center justify-between")
(div
(h4 (~tw :tokens "font-semibold text-stone-800") (get c "name"))
(p (~tw :tokens "text-sm text-stone-500") (get c "description")))
(span (~tw :tokens "text-xs font-mono text-violet-600 bg-violet-50 px-2 py-1 rounded")
(str "/pub/" (get c "slug"))))))
collections))))
;; --- Published Documents ---
(~docs/subsection :title "Published Documents"
(let ((specs (helper "pub-collection-items" "core-specs")))
(when (not (get specs "error"))
(div (~tw :tokens "space-y-2")
(h4 (~tw :tokens "text-sm font-semibold text-stone-500 uppercase tracking-wide")
(get specs "name"))
(map (fn (d)
(when (!= (get d "slug") "")
(div (~tw :tokens "rounded border border-stone-200 p-3 flex items-center justify-between")
(div
(p (~tw :tokens "font-medium text-stone-800") (get d "title"))
(p (~tw :tokens "text-xs text-stone-400") (get d "summary")))
(div (~tw :tokens "text-right")
(p (~tw :tokens "text-xs font-mono text-emerald-600 truncate max-w-48") (get d "cid"))
(p (~tw :tokens "text-xs text-stone-400") (str (get d "size") " bytes"))))))
(get specs "items"))))))
;; --- Outbox ---
(~docs/subsection :title "Recent Activity"
(let ((outbox (helper "pub-outbox-data" "")))
(if (= (get outbox "total") 0)
(p (~tw :tokens "text-sm text-stone-400 italic") "No activities yet.")
(div (~tw :tokens "space-y-2")
(p (~tw :tokens "text-xs text-stone-400") (str (get outbox "total") " total activities"))
(map (fn (a)
(when (!= (get a "type") "")
(div (~tw :tokens "rounded border border-stone-200 p-3 flex items-center gap-3")
(span (~tw :tokens "text-xs font-semibold text-white bg-violet-500 px-2 py-0.5 rounded")
(get a "type"))
(span (~tw :tokens "text-xs text-stone-500") (get a "published"))
(when (!= (get a "cid") "")
(span (~tw :tokens "text-xs font-mono text-emerald-600 truncate max-w-48")
(get a "cid"))))))
(get outbox "items"))))))
;; --- Followers ---
(~docs/subsection :title "Followers"
(let ((followers (helper "pub-followers-data")))
(if (empty? followers)
(p (~tw :tokens "text-sm text-stone-400 italic") "No followers yet.")
(div (~tw :tokens "space-y-2")
(map (fn (f)
(when (!= (get f "acct") "")
(div (~tw :tokens "rounded border border-stone-200 p-3 flex items-center gap-2")
(div (~tw :tokens "w-8 h-8 rounded-full bg-sky-100 flex items-center justify-center text-sky-700 text-xs font-bold") "F")
(p (~tw :tokens "text-sm font-mono text-stone-600 truncate") (get f "acct")))))
followers))))
;; --- API Endpoints ---
(~docs/subsection :title "Try the API"
(p (~tw :tokens "text-sm text-stone-600 mb-2") "All endpoints return " (code "text/sx") ". Try them directly:")
(div (~tw :tokens "grid grid-cols-2 sm:grid-cols-3 gap-2")
(map (fn (endpoint)
(a :href (get endpoint "href")
(~tw :tokens "block rounded border border-stone-200 p-2 text-center hover:border-violet-300 hover:bg-violet-50 transition-colors text-xs font-mono text-stone-600")
(get endpoint "label")))
(list
{"label" "GET /pub/actor" "href" "/pub/actor"}
{"label" "GET /pub/status" "href" "/pub/status"}
{"label" "GET /pub/collections" "href" "/pub/collections"}
{"label" "GET /pub/outbox" "href" "/pub/outbox"}
{"label" "GET /pub/followers" "href" "/pub/followers"}
{"label" "GET /pub/following" "href" "/pub/following"}))))))))

171
sx/sx/plans/sx-swarm.sx
View File

@@ -1,171 +0,0 @@
;; sx-swarm — SX-based Container Orchestration
;; Plan: Docker Swarm management where stack definitions, service configs,
;; and deployment logic are SX s-expressions manipulated by macros.
(defcomp ~plans/sx-swarm/plan-sx-swarm-content ()
(~docs/page :title "sx-swarm: Container Orchestration in SX"
(~docs/section :title "Vision" :id "vision"
(p "Replace docker-compose.yml and Docker Swarm stack files with SX. "
"Service definitions are defcomps. Environment configs are macros that expand "
"differently per target (dev, staging, production). Deployments are SX pipelines "
"executed by sx-ci. The entire infrastructure is the same language as the application.")
(p "YAML is a data format pretending to be a configuration language. "
"It has no functions, no macros, no composition, no type checking. "
"SX has all of these. A service definition is a value. A deployment is a function. "
"An environment override is a macro."))
(~docs/section :title "Why Not YAML" :id "why-not-yaml"
(p "Docker Compose and Swarm stack files share a fundamental problem: "
"they're static data with ad-hoc templating bolted on. "
"Variable substitution (${VAR}), extension fields (x-), YAML anchors (&/*) — "
"all workarounds for not having a real language.")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li "No functions — can't abstract repeated patterns")
(li "No macros — can't generate config at definition time")
(li "No conditionals — can't branch on environment")
(li "No types — a typo in an environment variable is a runtime error")
(li "No composition — merging files is fragile and order-dependent")
(li "No verification — can't check constraints before deploy"))
(p "SX solves all of these because it's a programming language, not a data format."))
(~docs/section :title "Service Definitions" :id "services"
(p "Services defined as SX, with macros for common patterns:")
(highlight ";; Base service macro — shared defaults
(defmacro defservice (name &rest body)
`(define ,name
(merge-service
{:restart-policy {:condition :on-failure :max-attempts 3}
:logging {:driver :json-file :options {:max-size \"10m\"}}
:networks (list \"internal\")}
(dict ,@body))))
;; A concrete service
(defservice blog-service
:image \"rose-ash/blog:latest\"
:ports (list \"8001:8000\")
:environment (env-for :blog)
:volumes (list \"./blog:/app/blog:ro\")
:depends-on (list :postgres :redis)
:deploy {:replicas 2
:update-config {:parallelism 1 :delay \"10s\"}})" "lisp")
(p "The " (code "defservice") " macro injects shared defaults. "
"Every service gets restart policy, logging, and network config for free. "
"Override any field by specifying it in the body."))
(~docs/section :title "Environment Macros" :id "environments"
(p "Environment-specific config via macros, not file merging:")
(highlight ";; Environment macro — expands differently per target
(defmacro env-for (service)
(let ((base (service-env-base service))
(target (current-deploy-target)))
(case target
:dev (merge base
{:DEBUG \"true\"
:DATABASE_URL (str \"postgresql://\" service \"_dev/\" service)})
:staging (merge base
{:DATABASE_URL (from-secret (str service \"-db-staging\"))})
:production (merge base
{:DATABASE_URL (from-secret (str service \"-db-prod\"))
:SENTRY_DSN (from-secret \"sentry-dsn\")}))))
;; Volume mounts differ per environment
(defmacro dev-volumes (service)
`(list ,(str \"./\" service \":/app/\" service)
,(str \"./shared:/app/shared\")
\"./dev.sh:/app/dev.sh:ro\"))
;; Production: no bind mounts, just named volumes
(defmacro prod-volumes (service)
`(list ,(str service \"-data:/data\")))" "lisp")
(p "No more docker-compose.yml + docker-compose.dev.yml + docker-compose.prod.yml. "
"One definition, macros handle the rest."))
(~docs/section :title "Stack Composition" :id "composition"
(p "Stacks compose like functions:")
(highlight ";; Infrastructure services shared across all stacks
(define infra-services
(list
(defservice postgres
:image \"postgres:16\"
:volumes (list \"pg-data:/var/lib/postgresql/data\")
:environment {:POSTGRES_PASSWORD (from-secret \"pg-pass\")})
(defservice redis
:image \"redis:7-alpine\"
:volumes (list \"redis-data:/data\"))
(defservice pgbouncer
:image \"edoburu/pgbouncer\"
:depends-on (list :postgres))))
;; Full stack = infra + app services
(define rose-ash-stack
(stack :name \"rose-ash\"
:services (concat infra-services app-services)
:networks (list
(network :name \"internal\" :driver :overlay :internal true)
(network :name \"public\" :driver :overlay))
:volumes (list
(volume :name \"pg-data\" :driver :local)
(volume :name \"redis-data\" :driver :local))))" "lisp"))
(~docs/section :title "Deploy as SX Pipeline" :id "deploy"
(p "Deployment is an sx-ci pipeline, not a shell script:")
(highlight "(define deploy-pipeline
(pipeline :name \"deploy\"
(stage :build
(map (fn (svc)
(docker-build
:context (str \"./\" (get svc \"name\"))
:tag (str \"rose-ash/\" (get svc \"name\") \":\" (git-sha))))
(changed-services)))
(stage :push
(map (fn (svc)
(docker-push (service-image svc)))
(changed-services)))
(stage :deploy
(swarm-deploy rose-ash-stack
:target (current-deploy-target)
:rolling true
:health-check-interval \"5s\"))))" "lisp"))
(~docs/section :title "Swarm Operations as IO" :id "swarm-ops"
(p "Swarm management exposed as SX IO primitives:")
(highlight ";; boundary.sx additions
(io swarm-deploy (stack &key target rolling) dict)
(io swarm-status (&key stack service) list)
(io swarm-scale (service replicas) dict)
(io swarm-rollback (service) dict)
(io swarm-logs (service &key since tail follow) list)
(io swarm-inspect (service) dict)
(io swarm-nodes () list)
(io swarm-secrets (&key create delete) dict)
(io docker-build (&key context tag dockerfile) dict)
(io docker-push (image) dict)" "lisp")
(p "These compose with sx-ci and the forge — push to forge triggers CI, "
"CI runs tests, tests pass, deploy pipeline executes, swarm updates."))
(~docs/section :title "Health and Monitoring" :id "monitoring"
(p "Service health as live SX components:")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li (strong "Dashboard") " — defcomp rendering swarm-status, auto-refreshing via sx-get polling")
(li (strong "Log viewer") " — swarm-logs streamed via SSE, rendered as SX")
(li (strong "Alerts") " — SX predicates on service state, notifications via sx-activity")
(li (strong "Rollback") " — one-click rollback via swarm-rollback IO primitive"))
(p "The monitoring dashboard is an SX page like any other. "
"No Grafana, no separate monitoring stack. Same language, same renderer, same platform."))
(~docs/section :title "Implementation Path" :id "implementation"
(ol (~tw :tokens "space-y-3 text-stone-600 list-decimal pl-5")
(li (strong "Phase 1: Stack definition") " — SX data structures for services, networks, volumes. "
"Compiler to docker-compose.yml / docker stack deploy format.")
(li (strong "Phase 2: Environment macros") " — defmacro for env-for, dev-volumes, prod-volumes. "
"Single source file, multiple targets.")
(li (strong "Phase 3: Deploy pipeline") " — sx-ci integration. Build, push, deploy as SX stages.")
(li (strong "Phase 4: Swarm IO") " — boundary primitives wrapping Docker API. "
"Dashboard and log viewer components.")
(li (strong "Phase 5: Native orchestration") " — replace Docker Swarm entirely. "
"SX-native container scheduling, networking, service discovery. "
"The ultimate goal: no Docker, no Swarm — just SX managing containers directly."))
(p "Phase 1-3 are pragmatic — compile SX to existing Docker tooling. "
"Phase 5 is ambitious — replace Docker tooling with SX. "
"The architecture supports both because the abstraction boundary is clean."))))

302
sx/sx/plans/sx-urls.sx
View File

@@ -1,302 +0,0 @@
;; SX Expression URLs — GraphSX
;; Plan: replace path-based routing with s-expression URLs where the URL
;; IS the query, the render instruction, and the address — all at once.
(defcomp ~plans/sx-urls/plan-sx-urls-content ()
(~docs/page :title "SX Expression URLs"
(~docs/section :title "Vision" :id "vision"
(p "URLs become s-expressions. The entire routing layer collapses into eval. "
"Every page is a function, every URL is a function call, and the nav tree hierarchy "
"is encoded directly in the nesting of the expression.")
(p "Current URLs like " (code "/language/docs/introduction") " become "
(code "/sx/(language.(doc.introduction))") ". Dots replace spaces as the URL-friendly "
"separator — they are unreserved in RFC 3986, never percent-encoded, and visually clean. "
"The parser treats dot as whitespace: " (code "s/./ /") " before parsing as SX.")
(~docs/code :src (highlight
";; Current → SX URLs (dots = spaces)\n/language/specs/signals → /(language.(spec.signals))\n/language/specs/explore/signals → /(language.(spec.(explore.signals)))\n/language/docs/introduction → /(language.(doc.introduction))\n/etc/plans/spec-explorer → /(etc.(plan.spec-explorer))\n\n;; Direct component access — any defcomp is addressable\n/(~essays/sx-sucks/essay-sx-sucks)\n/(~plans/sx-urls/plan-sx-urls-content)\n/(~analyzer/bundle-analyzer-content)"
"lisp")))
(~docs/section :title "Scoping — The 30-Year Ambiguity, Fixed" :id "scoping"
(p "REST URLs have an inherent ambiguity: does a filter/parameter apply to "
"the last segment, or the whole path? Consider:")
(~docs/code :src (highlight
";; REST — ambiguous:\n/users/123/posts?filter=published\n;; Is the filter scoped to posts? Or to the user? Or the whole query?\n;; Nobody knows. Conventions vary. Documentation required.\n\n;; SX URLs — explicit scoping via nesting:\n/(hello.(sailor.(filter.hhh))) ;; filter scoped to sailor\n/(hello.sailor.(filter.hhh)) ;; filter scoped to hello\n\n;; These mean different things, both expressible.\n;; Parens make scope visible. No ambiguity. No documentation needed."
"lisp"))
(p "This is not a minor syntactic preference. REST has never been able to express "
"the difference between a parameter scoped to a sub-resource and one scoped to the "
"parent. Query strings are flat — " (code "?filter=x") " applies to... what, exactly? "
"The last path segment? The whole URL? It depends on the API.")
(p "S-expression nesting makes scope " (em "structural") ". "
"The paren boundaries are the scope boundaries. "
"What took REST 30 years of convention documents to approximate, "
"SX URLs express in the syntax itself."))
(~docs/section :title "Dots as URL-Safe Whitespace" :id "dots"
(p "Spaces in URLs are ugly — they become " (code "%20") " in copy-paste, curl, logs, and proxies. "
"Dots are unreserved in RFC 3986, never encoded, and read naturally as \"drill down.\"")
(p "The rule is simple: " (strong "dot = space, nothing more") ". "
"Parens carry all the structural meaning. Dots are syntactic sugar for URLs only:")
(~docs/code :src (highlight
";; These are identical after dot→space transform:\n/(language.(doc.introduction)) → (language (doc introduction))\n/(geography.(hypermedia.(reference.attributes)))\n → (geography (hypermedia (reference attributes)))\n\n;; Parens are still required for nesting:\n/(language.doc.introduction) → (language doc introduction)\n;; = language(\"doc\", \"introduction\") — WRONG\n\n;; Correct nesting:\n/(language.(doc.introduction)) → (language (doc introduction))\n;; = language(doc(\"introduction\")) — RIGHT"
"lisp"))
(p "The server's URL handler does one thing before parsing: "
(code "url_expr = raw_path[1:].replace('.', ' ')") ". Then standard SX parsing takes over."))
(~docs/section :title "The Lisp Tax" :id "parens"
(p "People will hate the parentheses. But consider what developers already accept:")
(~docs/code :src (highlight
";; Developers happily write this every day:\nhttps://api.site.com/v2/users/123/posts?filter=published&sort=date&order=desc&limit=10&offset=20\n\n;; And they would complain about this?\nhttps://site.com/(users.(posts.123.(filter.published.sort.date.limit.10)))\n\n;; The second is shorter, structured, unambiguous, and composable."
"lisp"))
(p "The real question: who is reading these URLs?")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li (strong "End users") " barely look at URLs anymore — they live in address bars people ignore")
(li (strong "Developers") " will love it once they get it — it is powerful and consistent")
(li (strong "Crawlers and bots") " do not care at all"))
(p "And this site is " (em "about SX, implemented in SX") ". "
"The audience is exactly the people who will love the parens. "
"Every URL on the site is a live example of SX in action. "
"Visiting a page is evaluating an expression."))
(~docs/section :title "The Site Is a REPL" :id "repl"
(p "The address bar becomes the input line of a REPL. The page is the output.")
(~docs/code :src (highlight
"/sx/(about) ;; renders the about page\n/(source.(about)) ;; returns the SX source for the about page\n/(eval.(source.(about))) ;; re-evaluates it live\n\n;; The killer demo:\n/(eval.(map.double.(list.1.2.3))) ;; actually returns (2 4 6)\n\n;; The website IS a REPL. The address bar IS the input."
"lisp"))
(p "You do not need to explain what SX is. You show someone a URL and they "
"immediately understand the entire language and philosophy. "
"The whole site becomes a self-hosting proof of concept — "
"that is not just elegant, that is the pitch."))
(~docs/section :title "Components as Query Resolvers" :id "resolvers"
(p "The " (code "~") " sigil means \"find and execute this component.\" "
"Components can make onward queries, process results, and return composed content — "
"like server-side includes but Lispy and composable.")
(~docs/code :src (highlight
";; ~get is a component that fetches, processes, and returns\n/(~get.everything-under-the-sun)\n\n;; The flow:\n;; 1. Server finds ~get component in env\n;; 2. ~get makes onward queries\n;; 3. Processes and transforms results\n;; 4. Returns composed hypermedia\n\n;; Because it's all SX, you nest and compose:\n/(~page.home\n .(~hero.banner)\n .(~get.latest-posts.(limit.5))\n .(~get.featured.(filter.pinned)))\n\n;; Each ~component is independently:\n;; - cacheable (by its expression)\n;; - reusable (same component, different args)\n;; - testable (evaluate in isolation)"
"lisp"))
(p "This is what React Server Components are trying to do — server-side data resolution "
"composed with client-side rendering. Except React needs a framework, a bundler, "
"a serialization protocol, and \"use server\" pragmas. "
"SX gets it from a sigil and an evaluator."))
(~docs/section :title "HTTP Semantics — REST Re-Aligned" :id "http"
(p "GraphQL uses POST for queries even though they are pure reads — "
"because queries can be long and the body feels more natural for structured data. "
"But this violates HTTP semantics: POST implies side effects, "
"GET requests are cacheable, bookmarkable, shareable.")
(p "SX URLs naturally align with HTTP because the query " (em "is") " the URL:")
(div (~tw :tokens "overflow-x-auto mt-4")
(table (~tw :tokens "w-full text-sm text-left")
(thead
(tr (~tw :tokens "border-b border-stone-200")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Method")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Semantics")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Example")))
(tbody (~tw :tokens "text-stone-600")
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "GET")
(td (~tw :tokens "py-2 px-3") "Pure evaluation — cacheable, bookmarkable")
(td (~tw :tokens "py-2 px-3 font-mono text-sm") "GET /(language.(doc.intro))"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "POST")
(td (~tw :tokens "py-2 px-3") "Side effects — mutations, submissions")
(td (~tw :tokens "py-2 px-3 font-mono text-sm") "POST /(submit-post)")))))
(p "CDN caching of Lisp hypermedia just works. "
"This is what REST always wanted but GraphQL abandoned. "
"SX re-aligns with HTTP while being more powerful than both."))
(~docs/section :title "GraphSX — This Is a Query Language" :id "graphsx"
(p "The SX URL scheme is not just a routing convention — it is the emergence of "
(strong "GraphSX") ": GraphQL but Lisp. The structural parallel is exact:")
(div (~tw :tokens "overflow-x-auto mt-4")
(table (~tw :tokens "w-full text-sm text-left")
(thead
(tr (~tw :tokens "border-b border-stone-200")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Concept")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "GraphQL")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "GraphSX")))
(tbody (~tw :tokens "text-stone-600")
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Single endpoint")
(td (~tw :tokens "py-2 px-3") (code "/graphql"))
(td (~tw :tokens "py-2 px-3") "Catch-all " (code "/<path:expr>")))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Query structure")
(td (~tw :tokens "py-2 px-3") "Nested fields " (code "{ language { doc { ... } } }"))
(td (~tw :tokens "py-2 px-3") "Nested s-expressions " (code "(language.(doc....))")))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Resolvers")
(td (~tw :tokens "py-2 px-3") "Per-field functions")
(td (~tw :tokens "py-2 px-3") "Page/section functions + " (code "~components")))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Fragments")
(td (~tw :tokens "py-2 px-3") "Named reusable selections")
(td (~tw :tokens "py-2 px-3") "Components (" (code "defcomp") ")"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Schema")
(td (~tw :tokens "py-2 px-3") "Type definitions")
(td (~tw :tokens "py-2 px-3") "Page function registry + component env"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Scoping")
(td (~tw :tokens "py-2 px-3") "Flat — " (code "?filter=x") " applies to... what?")
(td (~tw :tokens "py-2 px-3") "Structural — parens ARE scope boundaries"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Transport")
(td (~tw :tokens "py-2 px-3") "POST JSON (violates HTTP GET semantics)")
(td (~tw :tokens "py-2 px-3") "GET " (code "/sx/(expr)") " — cacheable, bookmarkable"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Response")
(td (~tw :tokens "py-2 px-3") "JSON data (needs separate rendering)")
(td (~tw :tokens "py-2 px-3") "Content " (em "and") " data — response is already meaningful")))))
(p "The killer difference: in GraphQL, query and rendering are separate concerns — "
"you fetch JSON, then a frontend renders it. In GraphSX, "
(strong "the query language and the rendering language are the same thing") ". "
(code "(language.(doc.introduction))") " is simultaneously:")
(ol (~tw :tokens "space-y-1 text-stone-600 list-decimal pl-5")
(li "A " (strong "query") " — resolve the docs/introduction content within the language section")
(li "A " (strong "render instruction") " — render it inside a language section wrapper with nav context")
(li "A " (strong "URL") " — addressable, bookmarkable, shareable, cacheable"))
(p "This is what Lisp does. The uniform syntax means there is no boundary between "
"routing DSL, query language, template language, and component system. "
"They are all s-expressions evaluated in the same environment. "
"GraphQL had to invent a special syntax for queries because JSON is data, not code. "
"S-expressions are both."))
(~docs/section :title "Direct Component URLs" :id "components"
(p "Any " (code "defcomp") " is directly addressable via its " (code "~plans/content-addressed-components/name") ". "
"The URL evaluator sees " (code "~essays/sx-sucks/essay-sx-sucks") ", looks it up in the component env, "
"evaluates it, wraps in " (code "~layouts/doc") ", and returns.")
(~docs/code :src (highlight
";; Page functions are convenience wrappers:\n/(etc.(essay.sx-sucks)) ;; dispatches via case statement\n\n;; But you can bypass them entirely:\n/(~essays/sx-sucks/essay-sx-sucks) ;; direct component — no routing needed\n\n;; Every defcomp is instantly URL-accessible:\n/(~plans/sx-urls/plan-sx-urls-content) ;; this very page\n/(~analyzer/bundle-analyzer-content) ;; tools\n/(~docs-content/docs-evaluator-content) ;; docs"
"lisp"))
(p "New components are instantly URL-accessible without routing wiring. "
"Debugging is trivial — render any component in isolation."))
(~docs/section :title "URL Special Forms" :id "special-forms"
(p "URL-level functions that transform how content is resolved or displayed:")
(div (~tw :tokens "overflow-x-auto mt-4")
(table (~tw :tokens "w-full text-sm text-left")
(thead
(tr (~tw :tokens "border-b border-stone-200")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Form")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Example")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Effect")))
(tbody (~tw :tokens "text-stone-600")
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-mono text-violet-700") "source")
(td (~tw :tokens "py-2 px-3 font-mono text-sm") "/sx/(source.(~essays/sx-sucks/essay-sx-sucks))")
(td (~tw :tokens "py-2 px-3") "Show defcomp source instead of rendering"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-mono text-violet-700") "inspect")
(td (~tw :tokens "py-2 px-3 font-mono text-sm") "/sx/(inspect.(language.(doc.primitives)))")
(td (~tw :tokens "py-2 px-3") "Deps, CSS classes, render plan, IO requirements"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-mono text-violet-700") "diff")
(td (~tw :tokens "py-2 px-3 font-mono text-sm") "/sx/(diff.(spec.signals).(spec.eval))")
(td (~tw :tokens "py-2 px-3") "Side-by-side two pages"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-mono text-violet-700") "search")
(td (~tw :tokens "py-2 px-3 font-mono text-sm") "/sx/(search.\"define\".:in.(spec.signals))")
(td (~tw :tokens "py-2 px-3") "Grep within a page or spec"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-mono text-violet-700") "raw")
(td (~tw :tokens "py-2 px-3 font-mono text-sm") "/sx/(raw.(~some-component))")
(td (~tw :tokens "py-2 px-3") "Skip ~layouts/doc nav wrapping"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-mono text-violet-700") "eval")
(td (~tw :tokens "py-2 px-3 font-mono text-sm") "/sx/(eval.(map.double.(list.1.2.3)))")
(td (~tw :tokens "py-2 px-3") "Arbitrary evaluation — the URL bar is a REPL"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-mono text-violet-700") "json")
(td (~tw :tokens "py-2 px-3 font-mono text-sm") "/sx/(json.(language.(doc.primitives)))")
(td (~tw :tokens "py-2 px-3") "Return data as JSON — pure query mode"))))))
(~docs/section :title "Evaluation Model" :id "eval"
(p "The URL path (after stripping " (code "/") " and replacing dots with spaces) "
"is parsed as SX and evaluated with a " (strong "soft eval") ": "
"known function names are called; unknown symbols self-evaluate to their name as a string; "
"components (" (code "~plans/content-addressed-components/name") ") are looked up in the component env.")
(~docs/code :src (highlight
"/sx/(language.(doc.introduction))\n\n;; After dot→space: (language (doc introduction))\n;; 1. Eval `introduction` → not a known function → \"introduction\"\n;; 2. Eval (doc \"introduction\") → call doc(\"introduction\") → page content\n;; 3. Eval (language content) → call language(content) → passes through\n;; 4. Router wraps result in (~layouts/doc :path \"(language (doc introduction))\" ...)\n\n/(~essays/sx-sucks/essay-sx-sucks)\n;; 1. Eval ~essays/sx-sucks/essay-sx-sucks → component lookup → evaluate → content\n;; 2. Router wraps in ~layouts/doc"
"lisp"))
(~docs/subsection :title "Section Functions"
(p "Structural functions that encode hierarchy and pass through content:")
(~docs/code :src (highlight
"(define language\n (fn (&rest args)\n (if (empty? args) (language-index-content) (first args))))\n\n(define geography\n (fn (&rest args)\n (if (empty? args) (geography-index-content) (first args))))\n\n;; Sub-sections also pass through\n(define hypermedia\n (fn (&rest args)\n (if (empty? args) (hypermedia-index-content) (first args))))"
"lisp")))
(~docs/subsection :title "Page Functions"
(p "Leaf functions that dispatch to content components. "
"Data-dependent pages call helpers directly — the async evaluator handles IO:")
(~docs/code :src (highlight
"(define doc\n (fn (&rest args)\n (let ((slug (first-or-nil args)))\n (if (nil? slug)\n (~docs-content/docs-introduction-content)\n (case slug\n \"introduction\" (~docs-content/docs-introduction-content)\n \"getting-started\" (~docs-content/docs-getting-started-content)\n ...)))))\n\n(define bootstrapper\n (fn (&rest args)\n (let ((slug (first-or-nil args))\n (data (when slug (bootstrapper-data slug))))\n (if (nil? slug)\n (~specs/bootstrappers-index-content)\n (if (get data \"bootstrapper-not-found\")\n (~specs/not-found :slug slug)\n (case slug\n \"python\" (~specs/bootstrapper-py-content ...)\n ...))))))"
"lisp"))))
(~docs/section :title "The Catch-All Route" :id "route"
(p "The entire routing layer becomes one handler:")
(~docs/code :src (highlight
"@app.get(\"/\")\nasync def sx_home():\n return await eval_sx_url(\"/\")\n\n@app.get(\"/<path:expr>\")\nasync def sx_eval_route(expr):\n return await eval_sx_url(f\"/{expr}\")"
"python"))
(p (code "eval_sx_url") " in seven steps:")
(ol (~tw :tokens "space-y-1 text-stone-600 list-decimal pl-5")
(li "URL-decode the path")
(li "Replace dots with spaces")
(li "Parse as SX expression")
(li "Auto-quote unknown symbols (slugs become strings)")
(li "Evaluate with components + helpers + page/section functions in env")
(li "Wrap result in " (code "~layouts/doc") " with the URL expression as " (code ":path"))
(li "Return HTML or SX wire format depending on HTMX request"))
(p "Defhandler API endpoints and Python demo routes are registered " (em "before") " the catch-all, "
"so they match first."))
(~docs/section :title "Composability" :id "composability"
(~docs/code :src (highlight
";; Direct component access\n/(~essays/sx-sucks/essay-sx-sucks)\n/(~specs-explorer/spec-explorer-content)\n\n;; URL special forms\n/(source.(~essays/sx-sucks/essay-sx-sucks)) ;; view defcomp source\n/(inspect.(language.(doc.primitives))) ;; deps, render plan\n/(diff.(language.(spec.signals)).(language.(spec.eval))) ;; side by side\n/(eval.(map.double.(list.1.2.3))) ;; REPL in the URL bar\n\n;; Components as query resolvers\n/(~page.home\n .(~hero.banner)\n .(~get.latest-posts.(limit.5))\n .(~get.featured.(filter.pinned)))\n\n;; Scoping is explicit\n/(users.(posts.123.(filter.published))) ;; filter scoped to posts\n/(users.posts.123.(filter.published)) ;; filter scoped to users\n\n;; Cross-service (future)\n/(market.(product.42.:fields.(name.price)))\n/(subscribe.(etc.(plan.status)))"
"lisp")))
(~docs/section :title "Implementation Phases" :id "phases"
(div (~tw :tokens "space-y-4")
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4")
(p (~tw :tokens "font-semibold text-violet-800 mb-2") "Phase 1: Page Functions + Catch-All Route")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Create " (code "page-functions.sx") " — ~20 section + ~15 page functions")
(li "Create " (code "sx_router.py") " — URL parser (dot→space), soft eval, streaming detection")
(li "Replace " (code "auto_mount_pages") " with catch-all")
(li "Direct " (code "~component") " URL support")))
(div (~tw :tokens "rounded border border-blue-200 bg-blue-50 p-4")
(p (~tw :tokens "font-semibold text-blue-800 mb-2") "Phase 2: Navigation Data")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Rewrite ~200 " (code ":href") " values to dot-separated SX URLs")
(li "Rewrite " (code "resolve-nav-path") " for tree-descent matching")))
(div (~tw :tokens "rounded border border-emerald-200 bg-emerald-50 p-4")
(p (~tw :tokens "font-semibold text-emerald-800 mb-2") "Phase 3: Backward Compatibility")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "301 redirects from all old paths to new SX URLs")
(li "Algorithmic pattern matching — ~25 regex rules")))
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-4")
(p (~tw :tokens "font-semibold text-amber-800 mb-2") "Phase 4: URL Special Forms")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "source") ", " (code "inspect") ", " (code "diff") ", " (code "raw") ", " (code "eval") ", " (code "json"))
(li "Components as query resolvers (" (code "~get") ", " (code "~page") ")")))
(div (~tw :tokens "rounded border border-rose-200 bg-rose-50 p-4")
(p (~tw :tokens "font-semibold text-rose-800 mb-2") "Phase 5: Client-Side Routing")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Update " (code "_build_pages_sx()") " for function-based page registry")
(li "Client-side dot→space→parse→eval pipeline")
(li "Rebootstrap " (code "sx-ref.js") " and " (code "sx_ref.py"))))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(p (~tw :tokens "font-semibold text-stone-800 mb-2") "Phase 6: Cleanup")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Delete " (code "docs.sx") " (all 46 defpages)")
(li "Grep content files for stale old-style hrefs")))))
(~docs/section :title "What Stays the Same" :id "unchanged"
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li (strong "Defhandler API paths") " — registered before the catch-all, match first")
(li (strong "Python demo routes") " — registered via blueprint before the catch-all")
(li (strong "All other services") " — blog, market, etc. keep defpage routing")
(li (strong "Component definitions") " — all " (code ".sx") " files unchanged")
(li (strong "Page helpers") " — called from page functions instead of defpage " (code ":data"))
(li (strong "Layout, CSS, shell, rendering pipeline") " — no changes")))))

102
sx/sx/plans/sx-web-platform.sx
View File

@@ -1,102 +0,0 @@
;; sx-web.org — Online Development Platform
;; Plan: transform sx-web.org from documentation site into a live development
;; environment where content is authored, tested, and deployed in the browser.
(defcomp ~plans/sx-web-platform/plan-sx-web-platform-content ()
(~docs/page :title "sx-web.org Development Platform"
(~docs/section :title "Vision" :id "vision"
(p "sx-web.org becomes the development environment for itself. "
"Authors write essays, examples, components, and specs directly in the browser. "
"Changes are planned, staged, tested, and deployed without leaving the site. "
"The documentation is not about the platform — it " (em "is") " the platform.")
(p "Every artifact is content-addressed on IPFS. Every change flows through sx-activity. "
"Every deployment runs through sx-ci. Claude Code is embedded as the AI pair programmer. "
"The entire development lifecycle happens over the web, using the same SX primitives "
"that the platform is built from."))
(~docs/section :title "Architecture" :id "architecture"
(p "The platform composes existing SX subsystems into a unified workflow:")
(div (~tw :tokens "overflow-x-auto mt-4")
(table (~tw :tokens "w-full text-sm text-left")
(thead
(tr (~tw :tokens "border-b border-stone-200")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Layer")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "System")
(th (~tw :tokens "py-2 px-3 font-semibold text-stone-700") "Role")))
(tbody (~tw :tokens "text-stone-600")
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Author")
(td (~tw :tokens "py-2 px-3") "Embedded editor + Claude Code")
(td (~tw :tokens "py-2 px-3") "Write SX in the browser with AI assistance"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Stage")
(td (~tw :tokens "py-2 px-3") "Content-addressed components")
(td (~tw :tokens "py-2 px-3") "CID-identified artifacts, preview before publish"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Test")
(td (~tw :tokens "py-2 px-3") "sx-ci")
(td (~tw :tokens "py-2 px-3") "Run test suites against staged changes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Publish")
(td (~tw :tokens "py-2 px-3") "sx-activity")
(td (~tw :tokens "py-2 px-3") "Federated distribution via ActivityPub"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Store")
(td (~tw :tokens "py-2 px-3") "IPFS")
(td (~tw :tokens "py-2 px-3") "Content-addressed storage, permanent availability"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "py-2 px-3 font-semibold") "Verify")
(td (~tw :tokens "py-2 px-3") "Environment images")
(td (~tw :tokens "py-2 px-3") "Spec CID → image CID → endpoint provenance"))))))
(~docs/section :title "Embedded Claude Code" :id "claude-code"
(p "Claude Code sessions run inside the browser as reactive islands. "
"The AI has access to the full SX component environment — it can read specs, "
"write components, run tests, and propose changes. All within the user's security context.")
(p "The session produces SX diffs — not text patches, but structural changes to the component tree. "
"These diffs are first-class SX values that can be inspected, composed, reverted, and replayed.")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li "Read any component definition, spec file, or plan")
(li "Write new components (essays, examples, specs)")
(li "Modify existing components with structural diffs")
(li "Run sx-ci test suites against proposed changes")
(li "Stage changes as content-addressed preview")
(li "Publish via sx-activity when approved")))
(~docs/section :title "Workflow" :id "workflow"
(p "A typical session — adding a new essay:")
(ol (~tw :tokens "space-y-3 text-stone-600 list-decimal pl-5")
(li (strong "Author: ") "Open Claude Code session on sx-web.org. "
"Describe the essay topic. Claude writes the defcomp in SX.")
(li (strong "Preview: ") "The component renders live in the browser. "
"Author reviews, requests changes. Claude iterates.")
(li (strong "Stage: ") "Component is serialized, CID computed, stored to IPFS. "
"A preview URL is generated from the CID.")
(li (strong "Test: ") "sx-ci runs: component renders without error, "
"all referenced components exist, CSS classes are valid, links resolve.")
(li (strong "Publish: ") "sx-activity broadcasts the new component. "
"Federated subscribers receive it. The nav tree updates automatically.")
(li (strong "Verify: ") "Anyone can follow the CID chain from the served page "
"back to the spec that generated the evaluator that rendered it.")))
(~docs/section :title "Content Types" :id "content-types"
(p "Anything that can be a defcomp can be authored on the platform:")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li (strong "Essays") " — opinion pieces, rationales, explorations")
(li (strong "Examples") " — interactive demos with live code")
(li (strong "Specs") " — new spec modules authored and tested in-browser")
(li (strong "Plans") " — architecture documents with embedded diagrams")
(li (strong "Components") " — reusable UI components shared via IPFS")
(li (strong "Tests") " — defsuite/deftest written and executed live")))
(~docs/section :title "Prerequisites" :id "prerequisites"
(p "Systems that must be complete before the platform can work:")
(ul (~tw :tokens "space-y-2 text-stone-600 list-disc pl-5")
(li (strong "Reactive islands (L2+)") " — for the editor and preview panes")
(li (strong "Content-addressed components") " — CID computation and IPFS storage")
(li (strong "sx-activity") " — federated publish/subscribe")
(li (strong "sx-ci") " — test pipelines as SX components")
(li (strong "Runtime slicing") " — L3 full eval in browser for live preview")
(li (strong "Environment images") " — for provenance verification")
(li (strong "Claude Code API integration") " — embedded sessions via API")))))

384
sx/sx/plans/sx-web.sx
View File

@@ -1,384 +0,0 @@
;; sx-web: Federated Component Web
;; Browser-native peer network with content-addressed SX components.
;; Builds on sx-pub (phases 1-4 complete: publishing, federation, anchoring, IPFS).
(defcomp ~plans/sx-web/plan-sx-web-content ()
(~docs/page :title "sx-web: Federated Component Web"
(p (~tw :tokens "text-stone-500 text-sm italic mb-8")
"A two-tier peer network where server nodes pin to IPFS and browser tabs "
"participate via WebTransport. Components, tests, and content are all "
"content-addressed SX expressions verified by CID regardless of source. "
"Builds on sx-pub phases 1–4 (publishing, federation, anchoring, IPFS).")
;; =====================================================================
;; Architecture
;; =====================================================================
(~docs/section :title "Architecture" :id "architecture"
(p "Two tiers, one logical network. Trust anchored on CIDs, not sources.")
(table (~tw :tokens "w-full mb-6 text-sm")
(thead
(tr (th (~tw :tokens "text-left p-2") "") (th (~tw :tokens "text-left p-2") "Server Node") (th (~tw :tokens "text-left p-2") "Browser Node")))
(tbody
(tr (td (~tw :tokens "p-2 font-medium") "Runtime") (td (~tw :tokens "p-2") "OCaml + Python") (td (~tw :tokens "p-2") "JavaScript (sx-browser.js)"))
(tr (td (~tw :tokens "p-2 font-medium") "Storage") (td (~tw :tokens "p-2") "IPFS daemon + PostgreSQL") (td (~tw :tokens "p-2") "IndexedDB + Cache API"))
(tr (td (~tw :tokens "p-2 font-medium") "IPFS") (td (~tw :tokens "p-2") "Full DHT peer, pins content") (td (~tw :tokens "p-2") "Verifies CIDs, caches locally"))
(tr (td (~tw :tokens "p-2 font-medium") "Transport") (td (~tw :tokens "p-2") "WebSocket + HTTP federation") (td (~tw :tokens "p-2") "WebSocket to home + WebRTC to peers"))
(tr (td (~tw :tokens "p-2 font-medium") "Availability") (td (~tw :tokens "p-2") "Long-lived, always on") (td (~tw :tokens "p-2") "Ephemeral, online when open"))
(tr (td (~tw :tokens "p-2 font-medium") "Trust") (td (~tw :tokens "p-2") "CID verification + blockchain anchor") (td (~tw :tokens "p-2") "CID verification (same guarantee)"))))
(p "A component fetched from IPFS, from a server, from another browser tab, "
"or from local IndexedDB cache is verified the same way: hash it, check the CID. "
"The transport is irrelevant to the trust model."))
;; =====================================================================
;; Phase 1: Browser Node Foundation
;; =====================================================================
(~docs/section :title "Phase 1: Browser Node Foundation" :id "phase-1"
(p (~tw :tokens "text-stone-600 italic mb-4")
"Make the browser a first-class participant, not just a viewer.")
(~docs/subsection :title "1a. Content Store (IndexedDB)"
(p "Browser-local content-addressed store. Same interface as server-side "
(code "content-put") "/" (code "content-get") " but backed by IndexedDB.")
(~docs/code :src (highlight
";; Browser content store — pure SX, platform provides IndexedDB ops\n(define browser-store (make-content-store\n {:backend :indexeddb\n :db-name \"sx-web\"\n :verify true})) ;; always verify CID on read\n\n;; Same API as server store\n(content-put browser-store sx-source) ;; → CID\n(content-get browser-store cid) ;; → SX source | nil\n(content-has? browser-store cid) ;; → bool\n(content-pin browser-store cid) ;; mark as persistent\n(content-unpin browser-store cid) ;; allow eviction"
"lisp"))
(p "Two IndexedDB object stores:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (strong "blobs") " — CID → SX source bytes (content-addressed, immutable)")
(li (strong "pins") " — CID → {pinned, last-accessed, size, source-node} (eviction metadata)"))
(p "Unpinned content evicted LRU when storage exceeds quota. "
"Pinned content survives until explicitly unpinned."))
(~docs/subsection :title "1b. Service Worker"
(p "Intercepts fetch requests to resolve CIDs locally before hitting the network.")
(~docs/code :src (highlight
";; Service Worker routes\n;;\n;; /sx-web/cid/<cid> → check IndexedDB first, then fetch from home node\n;; /sx-web/resolve/<path> → resolve path→CID via local index, then fetch\n;; /sx-web/eval/<cid> → fetch + evaluate, return rendered result\n;;\n;; Cache strategy:\n;; CID routes are immutable — cache forever once verified\n;; Path routes check freshness against home node"
"text"))
(p "The Service Worker makes the browser work offline for all pinned components. "
"Online, it deduplicates fetches across tabs — one IndexedDB, shared cache."))
(~docs/subsection :title "1c. CID Verification in JavaScript"
(p "SHA3-256 hashing in the browser for CID verification. "
"Uses SubtleCrypto API (hardware-accelerated on modern browsers).")
(~docs/code :src (highlight
";; New platform primitive for browser host\n(define content-hash\n (fn (source)\n (host-call (host-get (host-global \"crypto\") \"subtle\")\n \"digest\" \"SHA-256\" (encode-utf8 source))))\n\n;; Verification: hash content, compare to claimed CID\n(define verify-cid\n (fn (cid source)\n (= cid (content-hash source))))"
"lisp"))
(p "One new platform primitive: " (code "content-hash") ". "
"Everything else (verification, pinning, eviction) is pure SX."))
(~docs/subsection :title "1d. Deliverables"
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "web/lib/content-store.sx") " — browser content store (~120 LOC)")
(li (code "web/lib/sw-routes.sx") " — Service Worker CID routing (~80 LOC)")
(li "Platform primitive: " (code "content-hash") " in JS host (~15 LOC)")
(li "IndexedDB schema: blobs + pins stores"))))
;; =====================================================================
;; Phase 2: WebTransport Peer Layer
;; =====================================================================
(~docs/section :title "Phase 2: Peer Network" :id "phase-2"
(p (~tw :tokens "text-stone-600 italic mb-4")
"Three transports, one SX protocol. Browser↔server via WebSocket, "
"browser↔browser via WebRTC data channels, server↔server via federation. "
"The protocol layer doesn't know or care which pipe carries it.")
(~docs/subsection :title "2a. SX Protocol Definition"
(p "The protocol is an SX component. Nodes fetch, evaluate, and speak it.")
(~docs/code :src (highlight
"(defprotocol sx-sync \"0.1\"\n :messages {\n ;; Content exchange\n :have (fn (cids) ...) ;; I have these CIDs\n :want (fn (cids) ...) ;; I need these CIDs\n :provide (fn (cid source) ...) ;; Here's the content\n :reject (fn (cid reason) ...) ;; Can't provide\n\n ;; Discovery\n :announce (fn (cid metadata) ...) ;; New content published\n :query (fn (pattern) ...) ;; Search for components\n :results (fn (matches) ...) ;; Search results\n\n ;; Identity\n :hello (fn (node-id pubkey) ...) ;; Handshake\n :verify (fn (challenge sig) ...) ;; Prove identity\n }\n\n :flow {\n ;; Content resolution\n :want → :provide | :reject\n ;; Publishing\n :announce → :want (from interested peers)\n ;; Connection\n :hello → :verify → ready\n })"
"lisp"))
(p "The protocol definition is itself content-addressed. Nodes pin the protocol "
"version they speak. Protocol upgrades are published to sx-pub like any component."))
(~docs/subsection :title "2b. Three Transports"
(table (~tw :tokens "w-full mb-6 text-sm")
(thead
(tr (th (~tw :tokens "text-left p-2") "Link") (th (~tw :tokens "text-left p-2") "Transport") (th (~tw :tokens "text-left p-2") "Why")))
(tbody
(tr (td (~tw :tokens "p-2 font-medium") "Browser ↔ Home node") (td (~tw :tokens "p-2") "WebSocket") (td (~tw :tokens "p-2") "Reliable, always-on, works today, no infrastructure changes"))
(tr (td (~tw :tokens "p-2 font-medium") "Browser ↔ Browser") (td (~tw :tokens "p-2") "WebRTC data channel") (td (~tw :tokens "p-2") "True peer-to-peer, NAT traversal via ICE/STUN, UDP + TCP modes"))
(tr (td (~tw :tokens "p-2 font-medium") "Server ↔ Server") (td (~tw :tokens "p-2") "HTTP federation") (td (~tw :tokens "p-2") "Existing sx-pub protocol, signed activities"))))
(p "The SX protocol layer is transport-agnostic. " (code "peer-send") " and " (code "peer-listen") " "
"work identically whether the peer is connected via WebSocket, WebRTC, or HTTP. "
"Upgrade to WebTransport when HTTP/3 infrastructure matures — zero protocol changes."))
(~docs/subsection :title "2c. Browser ↔ Home Node (WebSocket)"
(p "Browser connects to home node on page load. Receives push updates, "
"sends publish requests, queries the federation.")
(~docs/code :src (highlight
";; Connect to home node — WebSocket, works today\n(define home (ws-connect (str \"wss://\" home-node \"/sx-web/ws\")))\n\n;; Same protocol regardless of transport\n(peer-send home (sx-hello (node-id) (node-pubkey)))\n(peer-listen home (fn (msg) (dispatch-protocol msg)))\n\n;; Push notifications from server:\n;; (announce \"Qm..xyz\" {:author \"...\" :type :component})\n;; Content exchange:\n;; (want (list \"Qm..abc\")) → (provide \"Qm..abc\" \"(defcomp ...)\")\n;; Peer introduction:\n;; (peers (list {:id \"...\" :offer sdp-offer}))"
"lisp")))
(~docs/subsection :title "2d. Browser ↔ Browser (WebRTC)"
(p "Direct peer-to-peer connections between browsers. Home nodes relay "
"the signaling (SDP offer/answer exchange), then browsers talk directly. "
"Essential for low-latency game state, collaborative editing, and peer content exchange.")
(~docs/code :src (highlight
";; Signaling: home nodes relay SDP offers\n;;\n;; Browser A → Home A (WebSocket): (rtc-offer peer-b-id sdp)\n;; Home A → Home B (federation): (rtc-relay peer-a-id sdp)\n;; Home B → Browser B (WebSocket): (rtc-offer peer-a-id sdp)\n;; Browser B → Home B: (rtc-answer peer-a-id sdp)\n;; ... relay back ...\n;; WebRTC connection established — browsers talk directly\n\n;; After signaling, same API as any peer\n(define peer-b (rtc-connect offer))\n(peer-send peer-b (signal-update \"player-x\" 42))\n(peer-listen peer-b (fn (msg) (dispatch-protocol msg)))"
"lisp"))
(~docs/code :src (highlight
";; WebRTC data channel modes — chosen per room\n;;\n;; Reliable ordered (TCP-like):\n;; Chat messages, component exchange, CID resolution\n;; Every message arrives, in order\n;;\n;; Unreliable unordered (UDP-like):\n;; Game state (player positions), cursor positions, audio levels\n;; Latest value matters, old values can drop\n;;\n;; A room can use both — reliable for commands, unreliable for state\n(define game-room (join-room home \"game-1\"\n {:reliable {:chat (signal (list)) :inventory (signal (dict))}\n :unreliable {:player-x (signal 0) :player-y (signal 0)\n :aim-angle (signal 0)}}))"
"lisp"))
(p "Peer-to-peer means:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li "Game state at LAN-like latency — no server round-trip")
(li "Content exchange between nearby browsers without hitting IPFS")
(li "Collaborative editing with sub-frame update propagation")
(li "Works even if both home nodes are on modest hardware")
(li "Server load scales with signaling, not with traffic")))
(~docs/subsection :title "2d. Shared Signals Over WebTransport"
(p "Signals synchronized across peers. The same reactive primitive "
"that drives local UI drives multiplayer state. A " (code "shared-signal") " "
"is a signal whose writes propagate over WebTransport to all subscribed peers.")
(~docs/code :src (highlight
";; Shared signal — local writes propagate to peers, remote writes update locally\n(define player-x (shared-signal room \"player-x\" 0))\n(define player-y (shared-signal room \"player-y\" 0))\n\n;; Local code uses normal signal ops — no awareness of networking\n(reset! player-x 42) ;; updates local + sends to peers\n(deref player-x) ;; reactive, triggers re-render\n(swap! player-y inc) ;; same — local update + peer sync\n\n;; Under the hood:\n;; (reset! shared-sig val)\n;; → (reset! local-signal val)\n;; → (peer-send room (signal-update \"player-x\" val))\n;;\n;; incoming (signal-update \"player-x\" val) from peer\n;; → (reset! local-signal val)\n;; → reactive DOM updates fire as normal"
"lisp"))
(p "A " (code "room") " is a WebTransport channel with a set of shared signals. "
"Peers join a room, receive current state, and get incremental updates. "
"The signal names are the contract — any SX component that reads "
(code "(deref player-x)") " works whether the signal is local or shared.")
(~docs/code :src (highlight
";; Room: a named set of shared signals over WebTransport\n(define room (join-room home-peer \"game-lobby\"\n {:signals {:player-x 0 :player-y 0 :chat-messages (list)}\n :on-join (fn (peer-id) (announce peer-id))\n :on-leave (fn (peer-id) (remove-player peer-id))}))\n\n;; Game loop reads shared signals — identical to local signals\n(effect (fn ()\n (draw-player (deref player-x) (deref player-y))))\n\n;; Chat is shared too — append a message, all peers see it\n(reset! chat-messages (append (deref chat-messages)\n (dict :from (node-id) :text \"hello\")))"
"lisp"))
(p "Protocol messages for shared state:")
(~docs/code :src (highlight
";; Added to sx-sync protocol\n:join-room (fn (room-id signal-names) ...) ;; subscribe to room\n:room-state (fn (room-id signals) ...) ;; full state snapshot\n:signal-update (fn (room-id name value) ...) ;; incremental update\n:leave-room (fn (room-id) ...) ;; unsubscribe\n\n;; Conflict resolution: last-writer-wins by default\n;; Opt-in CRDT merge for collaborative data (lists, sets, maps)\n:signal-merge (fn (room-id name op args) ...) ;; CRDT operation"
"text"))
(p "Games, collaborative editors, chat, multiplayer tools — all the same primitive. "
"A " (code "shared-signal") " is a signal. Existing components work unchanged. "
"The networking is invisible to the reactive layer."))
(~docs/subsection :title "2f. Deliverables"
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "web/lib/sx-sync.sx") " — protocol definition incl. shared signals (~300 LOC)")
(li (code "web/lib/peer.sx") " — transport-agnostic peer abstraction (~200 LOC)")
(li (code "web/lib/shared-signal.sx") " — shared signal primitive (~120 LOC)")
(li "Platform primitives: " (code "ws-connect, rtc-connect, peer-send, peer-listen") " (~80 LOC JS)")
(li "Server: WebSocket endpoint + signaling relay + room state (~350 LOC Python)")
(li "Conflict resolution: LWW default + CRDT opt-in (~150 LOC SX)"))))
;; =====================================================================
;; Phase 3: In-Browser Authoring
;; =====================================================================
(~docs/section :title "Phase 3: In-Browser Authoring" :id "phase-3"
(p (~tw :tokens "text-stone-600 italic mb-4")
"Edit, preview, test, and publish from a browser tab. No local dev environment.")
(~docs/subsection :title "3a. Live Editor"
(p "SX editor component with real-time preview. "
"Edit source on the left, rendered output on the right. "
"The editor is itself an SX island.")
(~docs/code :src (highlight
"(defisland ~sx-web/editor (&key initial-source)\n (let ((source (signal (or initial-source \"\")))\n (parsed (computed (fn () (try-parse (deref source)))))\n (preview (computed (fn () (try-render (deref parsed))))))\n (div :class \"flex h-full\"\n ;; Source editor (textarea or CodeMirror via foreign FFI)\n (div :class \"flex-1\"\n (textarea :bind source :class \"font-mono ...\"))\n ;; Live preview\n (div :class \"flex-1 border-l\"\n (if (get (deref preview) \"error\")\n (div :class \"text-red-600\" (get (deref preview) \"error\"))\n (div (get (deref preview) \"result\")))))))"
"lisp"))
(p "The evaluator runs client-side. No server round-trip for preview. "
"Syntax errors, type errors, and render errors show inline as you type."))
(~docs/subsection :title "3b. In-Browser Test Runner"
(p "Tests are SX expressions. The browser evaluator runs them directly.")
(~docs/code :src (highlight
";; Tests travel with components\n(deftest ~my-component/tests\n (test \"renders title\"\n (let ((html (render-to-html (~my-component :title \"Hello\"))))\n (assert-contains html \"Hello\")))\n\n (test \"handles empty props\"\n (let ((html (render-to-html (~my-component))))\n (assert-contains html \"Untitled\"))))\n\n;; Run in browser — same evaluator, same results\n(run-tests ~my-component/tests)\n;; => {:passed 2 :failed 0 :errors ()}"
"lisp"))
(p "Component + tests + docs are published as a single unit to sx-pub. "
"Any node can re-run the tests to verify the component works."))
(~docs/subsection :title "3c. Publish Flow"
(p "From browser, one action: edit → test → publish → federate.")
(~docs/code :src (highlight
";; Browser publish flow\n(define publish!\n (fn (source)\n (let ((cid (content-hash source))\n (tests (extract-tests source))\n (results (run-tests tests)))\n (when (all-passed? results)\n ;; Pin locally\n (content-pin browser-store cid)\n ;; Push to home node via WebTransport\n (peer-send home-peer\n (sx-publish cid source\n {:tests results\n :requires (extract-requires source)}))\n ;; Home node pins to IPFS + anchors + federates\n cid))))"
"lisp"))
(p "The browser handles editing, testing, hashing, and local pinning. "
"The home server handles IPFS pinning, blockchain anchoring, and federation delivery. "
"Clean split: browsers create, servers distribute."))
(~docs/subsection :title "3d. Deliverables"
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "web/lib/editor.sx") " — live editor island (~300 LOC)")
(li (code "web/lib/test-runner.sx") " — browser test runner (~150 LOC)")
(li (code "web/lib/publish.sx") " — publish flow (~100 LOC)")
(li "Test extraction from SX source (find deftest forms)"))))
;; =====================================================================
;; Phase 4: Component Discovery & Resolution
;; =====================================================================
(~docs/section :title "Phase 4: Component Discovery & Resolution" :id "phase-4"
(p (~tw :tokens "text-stone-600 italic mb-4")
"Find, resolve, and depend on components across the federation.")
(~docs/subsection :title "4a. Dependency Resolution"
(p "Components declare dependencies via " (code ":requires") ". "
"Resolution checks local cache first, then home node, then federation.")
(~docs/code :src (highlight
";; Component with declared dependencies\n(defcomp ~charts/bar-chart (&key data labels)\n :requires ((~cssx/tw \"Qm..css\") ;; pinned by CID\n (~charts/axis \"Qm..axis\") ;; specific version\n (~charts/tooltip :latest)) ;; resolve latest from publisher\n ...)\n\n;; Resolution order:\n;; 1. Local IndexedDB (by CID)\n;; 2. Service Worker cache\n;; 3. Home node (WebTransport :want)\n;; 4. Federation query (WebTransport :query)\n;; 5. IPFS gateway (HTTP fallback)"
"lisp"))
(p "CID dependencies are immutable — resolve once, cache forever. "
(code ":latest") " dependencies check the publisher's feed for the current CID."))
(~docs/subsection :title "4b. Component Browser"
(p "Browse, search, and preview components from the federated network. "
"Each component renders its own preview because it IS executable SX.")
(~docs/code :src (highlight
";; Search the federation\n(peer-send home-peer\n (sx-query {:type :component\n :tags [\"chart\" \"visualization\"]\n :has-tests true}))\n\n;; Results include CID, metadata, AND the component itself\n;; Click to preview — evaluates client-side\n;; Click to pin — stores in IndexedDB\n;; Click to fork — opens in editor with source"
"lisp"))
(p "The component browser is itself an SX component published to sx-pub."))
(~docs/subsection :title "4c. Deliverables"
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "web/lib/resolver.sx") " — dependency resolution chain (~200 LOC)")
(li (code "sx/sx/tools/component-browser.sx") " — federated browser (~400 LOC)")
(li (code "sx/sx/tools/search.sx") " — search interface (~150 LOC)")
(li "Server: search index over pinned components"))))
;; =====================================================================
;; Phase 5: Node Bootstrap
;; =====================================================================
(~docs/section :title "Phase 5: Node Bootstrap" :id "phase-5"
(p (~tw :tokens "text-stone-600 italic mb-4")
"Standing up a new node: one command, minutes to first render.")
(~docs/subsection :title "5a. Server Node Bootstrap"
(~docs/code :src (highlight
"# One command. Pulls Docker image, starts all services.\ndocker run -d --name sx-node \\\n -e SX_SEED_PEER=pub.sx-web.org \\\n -e SX_DOMAIN=my-node.example.com \\\n -p 443:443 \\\n ghcr.io/rose-ash/sx-node:latest\n\n# What happens:\n# 1. IPFS daemon starts, connects to network\n# 2. Fetches spec CIDs from seed peer\n# 3. Evaluates specs — now has a working SX evaluator\n# 4. Follows seed peer — receives component announcements\n# 5. Caddy auto-provisions TLS via Let's Encrypt\n# 6. WebTransport endpoint live\n# 7. Ready to serve, publish, and federate"
"bash"))
(p "The bootstrap IS the proof: if your node can fetch specs from the network, "
"evaluate them, and render components — the system works. "
"No manual configuration beyond domain name and seed peer."))
(~docs/subsection :title "5b. Browser Node Bootstrap"
(~docs/code :src (highlight
";; Visit any sx-web node in a browser\n;; → sx-browser.js loads (evaluator + signals + DOM adapter)\n;; → Service Worker installs (CID caching)\n;; → WebTransport connects to host node\n;; → Spec CIDs pinned to IndexedDB\n;; → You're a node.\n;;\n;; No install. No signup. No CLI.\n;; Close the tab → ephemeral node disappears\n;; Reopen → reconnects, IndexedDB still has your pins"
"text"))
(p "A browser becomes a node by visiting a URL. "
"Leaving keeps your pins. Returning restores your state. "
"The barrier to entry is typing an address."))
(~docs/subsection :title "5c. Deliverables"
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li "Docker image: " (code "sx-node") " with OCaml host, IPFS, Caddy, PostgreSQL")
(li "Bootstrap script: fetch specs, verify CIDs, initialize DB")
(li "Seed peer protocol: advertise available specs + popular components")
(li "Browser auto-bootstrap: Service Worker install + spec pinning on first visit"))))
;; =====================================================================
;; Phase 6: AI Composition Layer
;; =====================================================================
(~docs/section :title "Phase 6: AI Composition Layer" :id "phase-6"
(p (~tw :tokens "text-stone-600 italic mb-4")
"The federated graph is simultaneously training data, retrieval context, and composition target.")
(~docs/subsection :title "6a. Component Retrieval"
(p "AI resolves components by CID, not by guessing. "
"The federation is a typed, tested, searchable context window.")
(~docs/code :src (highlight
";; AI composition query\n(ai-compose\n {:intent \"dashboard with sales chart and user table\"\n :constraints {:has-tests true\n :min-pins 10 ;; popularity signal\n :verified true} ;; blockchain-anchored\n :context (deref current-page)})\n\n;; AI doesn't generate from scratch — it assembles:\n;; 1. Search federation for matching components\n;; 2. Check tests pass for each candidate\n;; 3. Resolve dependencies\n;; 4. Compose into page, respecting :requires\n;; 5. Output is auditable: list of CIDs + composition logic"
"lisp"))
(p "The output is a composition of existing, tested, content-addressed components. "
"Not generated code — assembled code. Deterministic, verifiable, traceable."))
(~docs/subsection :title "6b. Self-Improving Tooling"
(p "AI tools are SX components published to sx-pub. "
"They improve as the network grows — more components to compose from, "
"more tests to validate against, more usage patterns to learn from.")
(p "The network effect works for both humans and AI:")
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li "Every component published is more material for AI composition")
(li "Every test attached is more validation for AI output")
(li "Every fork is a preference signal for AI ranking")
(li "Every pin is a popularity signal for AI retrieval")))
(~docs/subsection :title "6c. Deliverables"
(ul (~tw :tokens "list-disc pl-6 mb-4 space-y-1")
(li (code "web/lib/ai-compose.sx") " — composition interface (~200 LOC)")
(li "Embedding index over component metadata + signatures")
(li "Retrieval API: search by intent, filter by quality signals")
(li "Composition validator: check output resolves, tests pass"))))
;; =====================================================================
;; Implementation Order
;; =====================================================================
(~docs/section :title "Implementation Order" :id "order"
(~docs/code :src (highlight
"Phase 1 Browser Node Foundation ~400 LOC SX + ~100 LOC JS\n IndexedDB store, Service Worker, CID verification\n Prerequisite: none (extends existing sx-browser.js)\n\nPhase 2 Peer Network ~770 LOC SX + ~430 LOC Python/JS\n WebSocket + WebRTC + protocol + shared signals + rooms\n Prerequisite: Phase 1 (needs content store)\n\nPhase 3 In-Browser Authoring ~550 LOC SX\n Editor, test runner, publish flow\n Prerequisite: Phase 1 + 2 (needs store + peer connection)\n\nPhase 4 Component Discovery ~750 LOC SX\n Dependency resolution, component browser, search\n Prerequisite: Phase 2 (needs federation queries)\n\nPhase 5 Node Bootstrap ~300 LOC config + scripts\n Docker image, auto-bootstrap, seed protocol\n Prerequisite: Phase 1–4 (packages everything)\n\nPhase 6 AI Composition ~400 LOC SX + embedding index\n Retrieval, composition, validation\n Prerequisite: Phase 4 (needs searchable component graph)\n\n Total new SX: ~2650 LOC\n Total new platform: ~350 LOC (JS + Python)\n Timeline: Phases are independent workstreams after Phase 1"
"text"))
(p "Phase 1 is the foundation — everything else depends on the browser being able to "
"store, verify, and cache content-addressed SX. After that, phases 2–4 can proceed "
"in parallel. Phase 5 packages it all. Phase 6 builds on the populated network."))
;; =====================================================================
;; What Already Exists
;; =====================================================================
(~docs/section :title "What Already Exists (sx-pub Phases 1–4)" :id "existing"
(table (~tw :tokens "w-full mb-6 text-sm")
(thead
(tr (th (~tw :tokens "text-left p-2") "Capability") (th (~tw :tokens "text-left p-2") "Status") (th (~tw :tokens "text-left p-2") "Location")))
(tbody
(tr (td (~tw :tokens "p-2") "IPFS async client") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "shared/utils/ipfs_client.py"))
(tr (td (~tw :tokens "p-2") "Merkle tree + proofs") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "shared/utils/anchoring.py"))
(tr (td (~tw :tokens "p-2") "OpenTimestamps anchoring") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "shared/utils/anchoring.py"))
(tr (td (~tw :tokens "p-2") "ActivityPub federation") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "shared/infrastructure/activitypub.py"))
(tr (td (~tw :tokens "p-2") "sx-pub HTTP handlers") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "sx/sx/handlers/pub-api.sx"))
(tr (td (~tw :tokens "p-2") "Publishing + CID resolution") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "sx/sxc/pages/pub_helpers.py"))
(tr (td (~tw :tokens "p-2") "SX wire format (aser)") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "web/adapter-sx.sx"))
(tr (td (~tw :tokens "p-2") "Content store (in-memory)") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "lib/content.sx"))
(tr (td (~tw :tokens "p-2") "Component localStorage cache") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "web/boot.sx"))
(tr (td (~tw :tokens "p-2") "Docker dev environment") (td (~tw :tokens "p-2 text-green-700") "Done") (td (~tw :tokens "p-2 font-mono text-xs") "docker-compose.dev-pub.yml"))))
(p "The server-side infrastructure is complete. The plan above extends it to browsers."))))

253
sx/sx/plans/theorem-prover.sx
View File

@@ -1,253 +0,0 @@
;; ---------------------------------------------------------------------------
;; Theorem Prover: prove.sx — Constraint Solving for SX Primitives
;; ---------------------------------------------------------------------------
;; Helper: render a Phase 1 result row
(defcomp ~plans/theorem-prover/prove-phase1-row (&key (name :as string) (status :as string))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "py-1.5 px-3 font-mono text-xs text-stone-700") name)
(td (~tw :tokens "py-1.5 px-3 text-xs")
(if (= status "sat")
(span (~tw :tokens "text-emerald-600 font-medium") "sat")
(span (~tw :tokens "text-red-600 font-medium") status)))))
;; Helper: render a Phase 2 result row
(defcomp ~plans/theorem-prover/prove-phase2-row (&key (name :as string) (status :as string) (tested :as number) (skipped :as number) (counterexample :as string?))
(tr (~tw :tokens "border-t border-stone-100")
(td (~tw :tokens "py-1.5 px-3 font-mono text-xs text-stone-700") name)
(td (~tw :tokens "py-1.5 px-3 text-xs")
(if (= status "verified")
(span (~tw :tokens "text-emerald-600 font-medium") "verified")
(span (~tw :tokens "text-red-600 font-medium") "falsified")))
(td (~tw :tokens "py-1.5 px-3 text-xs text-stone-500 text-right tabular-nums")
(str tested))
(td (~tw :tokens "py-1.5 px-3 text-xs text-stone-400 text-right tabular-nums")
(str skipped))
(td (~tw :tokens "py-1.5 px-3 text-xs font-mono text-red-600")
(or counterexample ""))))
(defcomp ~plans/theorem-prover/plan-theorem-prover-content ()
(~docs/page :title "Theorem Prover"
;; --- Intro ---
(~docs/section :title "SX proves itself" :id "intro"
(p (~tw :tokens "text-stone-600")
(code "prove.sx") " is a constraint solver and property prover written in SX. It takes the SX specification (" (code "primitives.sx") "), translates it to formal logic via " (code "z3.sx") ", and proves properties about the result. Every step in the pipeline is an s-expression program operating on other s-expressions.")
(p (~tw :tokens "text-stone-600")
"Two proof modes. " (strong "Phase 1") " verifies that every primitive definition is satisfiable " (em "by construction") " — the definition is the model. " (strong "Phase 2") " goes further: it states algebraic properties (commutativity, associativity, transitivity, bounds) and verifies them by " (em "bounded model checking") " — exhaustive evaluation over integer domains, searching for counterexamples.")
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mt-4")
(p (~tw :tokens "text-sm text-violet-800 font-semibold mb-2") "Everything is SX")
(p (~tw :tokens "text-sm text-violet-700")
"No external solver. No Python proof logic. " (code "prove.sx") " is 400+ lines of s-expressions that parse SMT-LIB, evaluate expressions, generate test domains, compute cartesian products, search for counterexamples, and produce verification conditions. The same code would work client-side via the bootstrapped JavaScript evaluator.")))
;; --- Phase 1 Results ---
(~docs/section :title "Phase 1: Definitional satisfiability" :id "phase1"
(p (~tw :tokens "text-stone-600")
"Every " (code "define-primitive") " with a " (code ":body") " produces a " (code "forall") " assertion in SMT-LIB. For example, " (code "(define-primitive \"inc\" :params (n) :body (+ n 1))") " becomes:")
(~docs/code :src (highlight "; inc\n(declare-fun inc (Int) Int)\n(assert (forall (((n Int)))\n (= (inc n) (+ n 1))))\n(check-sat)" "lisp"))
(p (~tw :tokens "text-stone-600")
"This is satisfiable by construction: define " (code "inc(n) = n + 1") " and the assertion holds. " (code "prove.sx") " verifies this mechanically for every primitive — it parses the SMT-LIB, extracts the definition, builds a model, and evaluates it with test values.")
(div (~tw :tokens "rounded border border-emerald-200 bg-emerald-50 p-4 my-4")
(p (~tw :tokens "text-sm font-semibold")
(if phase1-all-sat
(span (~tw :tokens "text-emerald-800") (str phase1-sat "/" phase1-total " primitives: all sat"))
(span (~tw :tokens "text-red-800") (str phase1-sat "/" phase1-total " sat (some failed)"))))
(p (~tw :tokens "text-xs text-emerald-700 mt-1")
"Computed live in " (str phase1-ms) "ms"))
(~docs/subsection :title "Results"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 max-h-64 overflow-y-auto")
(table (~tw :tokens "w-full text-left")
(thead
(tr (~tw :tokens "bg-stone-100 sticky top-0")
(th (~tw :tokens "py-2 px-3 text-xs text-stone-500 font-medium") "Primitive")
(th (~tw :tokens "py-2 px-3 text-xs text-stone-500 font-medium") "Status")))
(tbody
(map (fn (r)
(~plans/theorem-prover/prove-phase1-row
:name (get r "name")
:status (get r "status")))
phase1-results))))))
;; --- Phase 2 Results ---
(~docs/section :title "Phase 2: Algebraic properties" :id "phase2"
(p (~tw :tokens "text-stone-600")
"Phase 1 proves internal consistency. Phase 2 proves " (em "external properties") " — mathematical laws that should hold across all inputs. Each property is defined as a test function evaluated over a bounded integer domain.")
(p (~tw :tokens "text-stone-600")
"For arity 1, the domain is [-50, 50] (101 values). For arity 2, [-20, 20] (1,681 pairs). For arity 3, [-8, 8] (4,913 triples). Properties with preconditions (like " (code "clamp-in-range") " requiring " (code "(<= lo hi)") ") skip value combinations that don't satisfy the precondition.")
(div (~tw :tokens "rounded border border-emerald-200 bg-emerald-50 p-4 my-4")
(p (~tw :tokens "text-sm font-semibold")
(if phase2-all-verified
(span (~tw :tokens "text-emerald-800") (str phase2-verified "/" phase2-total " properties: all verified"))
(span (~tw :tokens "text-red-800") (str phase2-verified "/" phase2-total " verified (" phase2-falsified " falsified)"))))
(p (~tw :tokens "text-xs text-emerald-700 mt-1")
(str phase2-total-tested " constraint evaluations in " phase2-ms "ms")))
(~docs/subsection :title "Results"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 max-h-96 overflow-y-auto")
(table (~tw :tokens "w-full text-left")
(thead
(tr (~tw :tokens "bg-stone-100 sticky top-0")
(th (~tw :tokens "py-2 px-3 text-xs text-stone-500 font-medium") "Property")
(th (~tw :tokens "py-2 px-3 text-xs text-stone-500 font-medium") "Status")
(th (~tw :tokens "py-2 px-3 text-xs text-stone-500 font-medium text-right") "Tested")
(th (~tw :tokens "py-2 px-3 text-xs text-stone-500 font-medium text-right") "Skipped")
(th (~tw :tokens "py-2 px-3 text-xs text-stone-500 font-medium") "Counterexample")))
(tbody
(map (fn (r)
(~plans/theorem-prover/prove-phase2-row
:name (get r "name")
:status (get r "status")
:tested (get r "tested")
:skipped (get r "skipped")
:counterexample (get r "counterexample" "")))
phase2-results))))))
;; --- What the properties prove ---
(~docs/section :title "What the properties prove" :id "properties"
(p (~tw :tokens "text-stone-600")
"34 properties across seven categories. Each encodes a mathematical law that the SX primitives must obey.")
(div (~tw :tokens "space-y-4 mt-4")
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(p (~tw :tokens "text-sm font-semibold text-stone-800 mb-2") "Arithmetic")
(p (~tw :tokens "text-sm text-stone-600")
(code "(+ a b) = (+ b a)") " commutativity. "
(code "(+ (+ a b) c) = (+ a (+ b c))") " associativity. "
(code "(+ a 0) = a") " identity. "
(code "(* a (+ b c)) = (+ (* a b) (* a c))") " distributive law. "
"Same for " (code "*") " — commutative, associative, identity, zero annihilation. "
(code "(- a a) = 0") " subtraction inverse. Nine properties, all verified."))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(p (~tw :tokens "text-sm font-semibold text-stone-800 mb-2") "inc / dec")
(p (~tw :tokens "text-sm text-stone-600")
(code "(inc n) = (+ n 1)") " and " (code "(dec n) = (- n 1)") " — definitional relationships. "
(code "(dec (inc n)) = n") " and " (code "(inc (dec n)) = n") " — mutual inverses. "
"These confirm that " (code "inc") " and " (code "dec") " are exactly " (code "+1") " and " (code "-1") ", not off-by-one."))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(p (~tw :tokens "text-sm font-semibold text-stone-800 mb-2") "abs")
(p (~tw :tokens "text-sm text-stone-600")
(code "(abs n) >= 0") " non-negativity. "
(code "(abs (abs n)) = (abs n)") " idempotence. "
(code "(abs n) = (abs (- 0 n))") " symmetry. "
"Three properties that together characterize absolute value."))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(p (~tw :tokens "text-sm font-semibold text-stone-800 mb-2") "Predicates")
(p (~tw :tokens "text-sm text-stone-600")
(code "(odd? n) = (not (even? n))") " — " (code "odd?") " and " (code "even?") " are complementary. "
(code "(even? n) = (= (mod n 2) 0)") " — " (code "even?") " matches modular arithmetic. "
(code "(zero? n) = (= n 0)") " — " (code "zero?") " is exactly equality with zero. "
(code "(not (not p)) = p") " — double negation elimination."))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(p (~tw :tokens "text-sm font-semibold text-stone-800 mb-2") "min / max / clamp")
(p (~tw :tokens "text-sm text-stone-600")
(code "min") " and " (code "max") " are commutative and satisfy their bounds: "
(code "(min a b) <= a") " and " (code "(max a b) >= a") ". "
(code "(min a b) + (max a b) = a + b") " — the min-max sum identity. "
(code "clamp") " with precondition " (code "lo <= hi") ": result is always in " (code "[lo, hi]") ", equals " (code "x") " when already in range, and is idempotent."))
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(p (~tw :tokens "text-sm font-semibold text-stone-800 mb-2") "Comparison & inequality")
(p (~tw :tokens "text-sm text-stone-600")
(code "(< a b) = (> b a)") " — flipped arguments. "
(code "(<= a b) = (not (> a b))") " and " (code "(>= a b) = (not (< a b))") " — duality. "
"Trichotomy: exactly one of " (code "<") ", " (code "=") ", " (code ">") " holds. "
"Transitivity: " (code "(< a b)") " and " (code "(< b c)") " implies " (code "(< a c)") ". "
(code "(!= a b) = (not (= a b))") "."))))
;; --- SMT-LIB output ---
(~docs/section :title "SMT-LIB verification conditions" :id "smtlib"
(p (~tw :tokens "text-stone-600")
"Each property also generates SMT-LIB for unbounded verification by an external solver. The strategy: assert the " (em "negation") " of the universal property. If Z3 returns " (code "unsat") ", the property holds for " (em "all") " integers — not just the bounded domain.")
(p (~tw :tokens "text-stone-600")
(code "prove.sx") " reuses " (code "z3-expr") " from " (code "z3.sx") " to translate the property AST to SMT-LIB. Properties with preconditions use " (code "=>") " (implication). The same SX expression is both the bounded test and the formal verification condition.")
(~docs/code :src (highlight smtlib-sample "lisp")))
;; --- What it tells us ---
(~docs/section :title "What this tells us about SX" :id "implications"
(p (~tw :tokens "text-stone-600")
"Three things, at increasing depth.")
(~docs/subsection :title "1. The spec is internally consistent"
(p (~tw :tokens "text-stone-600")
"Phase 1 proves every " (code "define-primitive") " with a " (code ":body") " is satisfiable. The definition doesn't contradict itself. This is necessary but weak — it's true by construction. The value is mechanical verification: no typo, no copy-paste error, no accidental negation in any of the 91 definitions."))
(~docs/subsection :title "2. The primitives obey algebraic laws"
(p (~tw :tokens "text-stone-600")
"Phase 2 proves real mathematical properties hold across bounded domains. These aren't tautologies — they're constraints that " (em "could") " fail. "
(code "(+ a b) = (+ b a)") " could fail if " (code "+") " had a subtle bug. "
(code "clamp-in-range") " could fail if " (code "max") "/" (code "min") " were swapped. "
"The " (str phase2-total-tested) " evaluations found zero counterexamples.")
(p (~tw :tokens "text-stone-600")
"Bounded model checking is not a mathematical proof — it verifies over a finite domain. The SMT-LIB output bridges the gap: feed it to Z3 for a universal proof over all integers."))
(~docs/subsection :title "3. SX can reason about itself"
(p (~tw :tokens "text-stone-600")
"The deep result. The SX evaluator executes " (code "z3.sx") ", which reads SX spec files and emits formal logic. Then the SX evaluator executes " (code "prove.sx") ", which parses that logic and proves properties about it. The specification, the translator, and the prover are all written in the same language, operating on the same data structures.")
(p (~tw :tokens "text-stone-600")
"The pipeline: " (code "SX spec") " → " (code "SX translator") " → " (code "formal logic") " → " (code "SX prover") " → " (code "proof") ". At every step, SX is both the subject and the tool. The system is verifying its own foundations using its own machinery.")
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-4 mt-3")
(p (~tw :tokens "text-sm text-amber-800 font-semibold mb-2") "The strange loop")
(p (~tw :tokens "text-sm text-amber-700")
"The SX spec defines primitives. " (code "z3.sx") " (written in SX, using those primitives) translates the spec to formal logic. " (code "prove.sx") " (written in SX, using those same primitives) proves properties about the logic. The primitives being verified are the same primitives doing the verifying. This is not circular — it's a fixed point. If the primitives were wrong, the proofs would fail."))))
;; --- The pipeline ---
(~docs/section :title "The full pipeline" :id "pipeline"
(div (~tw :tokens "rounded border border-stone-200 bg-stone-50 p-4")
(table (~tw :tokens "w-full text-sm")
(thead (tr (~tw :tokens "text-left text-stone-500")
(th (~tw :tokens "pb-2 pr-4") "Step")
(th (~tw :tokens "pb-2 pr-4") "Input")
(th (~tw :tokens "pb-2 pr-4") "Tool")
(th (~tw :tokens "pb-2") "Output")))
(tbody
(tr (~tw :tokens "text-stone-700 border-t border-stone-200")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "1")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "primitives.sx")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "z3.sx")
(td (~tw :tokens "py-2") "SMT-LIB (formal logic)"))
(tr (~tw :tokens "text-stone-700 border-t border-stone-200")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "2a")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "SMT-LIB")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "prove.sx (Phase 1)")
(td (~tw :tokens "py-2") "sat (definitional consistency)"))
(tr (~tw :tokens "text-stone-700 border-t border-stone-200")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "2b")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "property defs")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "prove.sx (Phase 2)")
(td (~tw :tokens "py-2") "verified (algebraic laws)"))
(tr (~tw :tokens "text-stone-700 border-t border-stone-200")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "3")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "properties")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "prove.sx → z3-expr")
(td (~tw :tokens "py-2") "SMT-LIB for Z3 (unbounded)"))
(tr (~tw :tokens "text-stone-400 border-t border-stone-200")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "4")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs") "SMT-LIB")
(td (~tw :tokens "py-2 pr-4 font-mono text-xs italic") "Z3 (external)")
(td (~tw :tokens "py-2 italic") "unsat (universal proof)")))))
(p (~tw :tokens "text-stone-600 mt-4")
"Steps 1-3 run on this page, live, in the SX evaluator. Step 4 requires an external Z3 installation — the SMT-LIB output above is ready to feed to it."))
;; --- Source ---
(~docs/section :title "The source: prove.sx" :id "source"
(p (~tw :tokens "text-stone-600")
"The entire constraint solver is a single SX file. Key sections: "
(code "smt-eval") " evaluates SMT-LIB expressions. "
(code "prove-tuples") " generates cartesian products for bounded checking. "
(code "prove-search") " walks tuples looking for counterexamples. "
(code "sx-properties") " declares 34 algebraic laws as test functions with quoted ASTs. "
(code "prove-property-smtlib") " translates properties to SMT-LIB verification conditions via " (code "z3-expr") ".")
(~docs/code :src (highlight prove-source "lisp")))
(~docs/section :title "The translator: z3.sx" :id "z3-source"
(p (~tw :tokens "text-stone-600")
"The translator that " (code "prove.sx") " depends on. SX expressions that walk other SX expressions and emit SMT-LIB strings. Both files together: ~760 lines of s-expressions, no host language logic.")
(~docs/code :src (highlight z3-source "lisp")))))

496
sx/sx/plans/typed-sx.sx
View File

@@ -1,496 +0,0 @@
;; ---------------------------------------------------------------------------
;; Typed SX — Gradual Type System
;; ---------------------------------------------------------------------------
(defcomp ~plans/typed-sx/plan-typed-sx-content ()
(~docs/page :title "Typed SX"
(~docs/section :title "The Opportunity" :id "opportunity"
(p "SX already has types. Every primitive in " (code "primitives.sx") " declares " (code ":returns \"number\"") " or " (code ":returns \"boolean\"") ". Every IO primitive in " (code "boundary.sx") " declares " (code ":returns \"dict?\"") " or " (code ":returns \"any\"") ". Component params are named. The information exists — nobody checks it.")
(p "A gradual type system makes this information useful. Annotations are optional. Unannotated code works exactly as before. Annotated code gets checked at registration time — zero runtime cost, errors before any request is served. The checker is a spec module (" (code "types.sx") "), bootstrapped to every host.")
(p "This is not Haskell. SX doesn't need a type system to be correct — " (a :href "/sx/(etc.(plan.theorem-prover))" (~tw :tokens "text-violet-700 underline") "prove.sx") " already verifies primitive properties by exhaustive search. Types serve a different purpose: they catch " (strong "composition errors") " — wrong argument passed to a component, mismatched return type piped into another function, missing keyword arg. The kind of bug you find by reading the stack trace and slapping your forehead."))
;; -----------------------------------------------------------------------
;; What already exists
;; -----------------------------------------------------------------------
(~docs/section :title "What Already Exists" :id "existing"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Feature")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Where")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Types today")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Primitive return types")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "primitives.sx :returns")
(td (~tw :tokens "px-3 py-2 text-stone-600") "\"number\", \"string\", \"boolean\", \"list\", \"dict\", \"any\""))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "IO primitive return types")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "boundary.sx :returns")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Same + \"dict?\", \"string?\", \"element\" — nullable types already appear"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Primitive param names")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "primitives.sx :params")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Named but untyped: " (code "(a b)") ", " (code "(&rest args)")))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Component param names")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "eval.sx parse-comp-params")
(td (~tw :tokens "px-3 py-2 text-stone-600") (code "&key") " params, " (code "&rest") " children — named, untyped"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Runtime type predicates")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "primitives.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") (code "number?") ", " (code "string?") ", " (code "list?") ", " (code "dict?") ", " (code "nil?") ", " (code "symbol?") " — all exist"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Purity classification")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "deps.sx + boundary.py")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Pure vs IO — a binary type at the component level"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Property verification")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "prove.sx")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Algebraic properties (commutativity, transitivity) verified by bounded model checking")))))
(p "The foundation is solid. Primitives already have return types. Params already have names. Boundary enforcement already does structural analysis. Types extend this — they don't replace it."))
;; -----------------------------------------------------------------------
;; Type language
;; -----------------------------------------------------------------------
(~docs/section :title "Type Language" :id "type-language"
(p "Small, practical, no type theory PhD required.")
(~docs/subsection :title "Base types"
(~docs/code :src (highlight ";; Atomic types\nnumber string boolean nil symbol keyword element\n\n;; Nullable (already used in boundary.sx)\nstring? ;; (or string nil)\ndict? ;; (or dict nil)\nnumber? ;; (or number nil)\n\n;; The top type — anything goes\nany\n\n;; The bottom type — never returns (e.g. abort)\nnever" "lisp")))
(~docs/subsection :title "Compound types"
(~docs/code :src (highlight ";; Collections with element types\n(list-of number) ;; list where every element is a number\n(list-of string) ;; list of strings\n(list-of any) ;; list (same as untyped)\n(dict-of string number) ;; dict with string keys, number values\n(dict-of string any) ;; dict with string keys (typical kwargs)\n\n;; Union types\n(or string number) ;; either string or number\n(or string nil) ;; same as string?\n\n;; Function types\n(-> number number) ;; number → number\n(-> string string boolean) ;; (string, string) → boolean\n(-> (list-of any) number) ;; list → number\n(-> &rest any number) ;; variadic → number" "lisp")))
(~docs/subsection :title "Component types"
(~docs/code :src (highlight ";; Component type is its keyword signature\n;; Derived automatically from defcomp — no annotation needed\n\n(comp :title string :price number &rest element)\n;; keyword args children type\n\n;; This is NOT a new syntax for defcomp.\n;; It's the TYPE that a defcomp declaration produces.\n;; The checker infers it from parse-comp-params + annotations." "lisp")))
(p "That's the core. No higher-kinded types, no dependent types, no type classes. Just: what goes in, what comes out, can it be nil.")
(~docs/subsection :title "User-defined types"
(~docs/code :src (highlight ";; Type alias — a name for an existing type\n(deftype price number)\n(deftype html-string string)\n\n;; Union — one of several types\n(deftype renderable (union string number nil))\n(deftype key-type (union string keyword))\n\n;; Record — typed dict shape\n(deftype card-props\n {:title string\n :subtitle string?\n :price number})\n\n;; Parameterized — generic over a type variable\n(deftype (maybe a) (union nil a))\n(deftype (list-of-pairs a b) (list-of (dict-of a b)))\n(deftype (result a e) (union (ok a) (err e)))" "lisp"))
(p (code "deftype") " is a declaration form — zero runtime cost, purely for the checker. The type registry resolves user-defined type names during " (code "subtype?") " and " (code "infer-type") ". Records enable typed keyword args for components:")
(~docs/code :src (highlight ";; Define a prop shape\n(deftype card-props\n {:title string\n :subtitle string?\n :price number\n :on-click (-> any nil)?})\n\n;; Use it in a component\n(defcomp ~plans/typed-sx/product-card (&key (props :as card-props) &rest children)\n (div :class \"card\"\n (h2 (get props :title))\n (span (format-decimal (get props :price) 2))\n children))\n\n;; Checker validates dict literals against record shape:\n(~plans/typed-sx/product-card :props {:title \"Widget\" :price \"oops\"})\n;; ^^^^^^\n;; ERROR: :price expects number, got string" "lisp"))))
;; -----------------------------------------------------------------------
;; Annotation syntax
;; -----------------------------------------------------------------------
(~docs/section :title "Annotation Syntax" :id "syntax"
(p "Annotations are optional. Three places they can appear:")
(~docs/subsection :title "1. Component params"
(~docs/code :src (highlight ";; Current (unchanged, still works)\n(defcomp ~plans/typed-sx/product-card (&key title price image-url &rest children)\n (div ...))\n\n;; Annotated — colon after param name\n(defcomp ~plans/typed-sx/product-card (&key (title : string)\n (price : number)\n (image-url : string?)\n &rest children)\n (div ...))\n\n;; Parenthesized pairs: (name : type)\n;; Unannotated params default to `any`\n;; &rest children is always (list-of element)" "lisp"))
(p "The " (code "(name : type)") " syntax is unambiguous — a 3-element list where the second element is the symbol " (code ":") ". The parser already handles lists inside parameter lists. " (code "parse-comp-params") " gains a branch: if a param is a list of length 3 with " (code ":") " in the middle, extract name and type."))
(~docs/subsection :title "2. Define/lambda return types"
(~docs/code :src (highlight ";; Current (unchanged)\n(define total-price\n (fn (items)\n (reduce + 0 (map (fn (i) (get i \"price\")) items))))\n\n;; Annotated — :returns after params\n(define total-price\n (fn ((items : (list-of dict)) :returns number)\n (reduce + 0 (map (fn (i) (get i \"price\")) items))))" "lisp"))
(p (code ":returns") " is already the convention in " (code "primitives.sx") " and " (code "boundary.sx") ". Same keyword, same position (after params), same meaning."))
(~docs/subsection :title "3. Let bindings"
(~docs/code :src (highlight ";; Current (unchanged)\n(let ((x (compute-value)))\n (+ x 1))\n\n;; Annotated\n(let (((x : number) (compute-value)))\n (+ x 1))\n\n;; Usually unnecessary — the checker infers let binding\n;; types from the right-hand side. Only annotate when\n;; the inference is ambiguous (e.g. the RHS returns `any`)." "lisp"))
(p "All annotations are syntactically backward-compatible. Unannotated code parses and runs identically. The annotations are simply ignored by evaluators that don't have the type checker loaded."))
;; -----------------------------------------------------------------------
;; Type checking
;; -----------------------------------------------------------------------
(~docs/section :title "Type Checking" :id "checking"
(p "The checker runs at registration time — after " (code "compute_all_deps") ", before serving. It walks every component's body AST and verifies that call sites match declared signatures.")
(~docs/subsection :title "What it checks"
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-2")
(li (strong "Primitive calls:") " " (code "(+ \"hello\" 3)") " — " (code "+") " expects numbers, got a string. Error.")
(li (strong "Component calls:") " " (code "(~plans/typed-sx/product-card :title 42)") " — " (code ":title") " declared as " (code "string") ", got " (code "number") ". Error.")
(li (strong "Missing required params:") " " (code "(~plans/typed-sx/product-card :price 29.99)") " — " (code ":title") " not provided, no default. Error.")
(li (strong "Unknown keyword args:") " " (code "(~plans/typed-sx/product-card :title \"Hi\" :colour \"red\")") " — " (code ":colour") " not in param list. Warning.")
(li (strong "Nil safety:") " " (code "(+ 1 (get user \"age\"))") " — " (code "get") " returns " (code "any") " (might be nil). " (code "+") " expects " (code "number") ". Warning: possible nil.")
(li (strong "Thread-first type flow:") " " (code "(-> items (filter active?) (map name) (join \", \"))") " — checks each step's output matches the next step's input.")))
(~docs/subsection :title "What it does NOT check"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "Runtime values.") " " (code "(if condition 42 \"hello\")") " — the type is " (code "(or number string)") ". The checker doesn't know which branch executes.")
(li (strong "Dict key presence (yet).") " " (code "(get user \"name\")") " — the checker knows " (code "get") " returns " (code "any") " but doesn't track which keys a dict has. Phase 6 (" (code "deftype") " records) will enable this.")
(li (strong "Termination.") " That's " (a :href "/sx/(etc.(plan.theorem-prover))" (~tw :tokens "text-violet-700 underline") "prove.sx") "'s domain.")
(li (strong "Full algebraic effects.") " The effect system (Phase 7) checks static effect annotations — it does not provide algebraic effect handlers, effect polymorphism, or continuation-based effect dispatch. That door remains open for the future.")))
(~docs/subsection :title "Inference"
(p "Most types are inferred, not annotated. The checker knows:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Literal types: " (code "42") " → " (code "number") ", " (code "\"hi\"") " → " (code "string") ", " (code "true") " → " (code "boolean") ", " (code "nil") " → " (code "nil"))
(li "Primitive return types: " (code "(+ a b)") " → " (code "number") ", " (code "(str x)") " → " (code "string") ", " (code "(empty? x)") " → " (code "boolean"))
(li "Let bindings: " (code "(let ((x 42)) ...)") " → " (code "x : number"))
(li "If/cond narrowing: " (code "(if (nil? x) \"default\" (str x))") " — in the else branch, " (code "x") " is not nil")
(li "Component return: always " (code "element"))
(li "Map/filter propagation: " (code "(map name items)") " → " (code "(list-of string)") " if " (code "name") " returns " (code "string")))
(p "In practice, most component bodies need zero annotations. The checker infers types from literals and primitive return declarations. Annotations are for the edges: params coming from outside, and ambiguous flows.")))
;; -----------------------------------------------------------------------
;; Gradual semantics
;; -----------------------------------------------------------------------
(~docs/section :title "Gradual Semantics" :id "gradual"
(p "The type " (code "any") " is the escape hatch. It's compatible with everything — passes every check, accepts every value. Unannotated params are " (code "any") ". The return type of " (code "get") " is " (code "any") ". This means:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-2")
(li (strong "Fully untyped code:") " All params are " (code "any") ", all returns are " (code "any") ". The checker has nothing to verify. No errors, no warnings. Exactly the same as today.")
(li (strong "Partially typed:") " Some components annotate params, others don't. The checker verifies annotated call sites and skips untyped ones. You get value proportional to effort.")
(li (strong "Fully typed:") " Every component, every lambda, every let binding. The checker catches every composition error. Maximum value, maximum annotation cost."))
(p "The practical sweet spot: " (strong "annotate component params, nothing else.") " Components are the public API — the boundary between independent pieces of code. Their params are the contract. Internal lambdas and let bindings benefit less from annotations because the checker can infer their types from context.")
(~docs/code :src (highlight ";; Sweet spot: annotate the interface, infer the rest\n(defcomp ~plans/typed-sx/price-display (&key (price : number)\n (currency : string)\n (sale-price : number?))\n ;; Everything below is inferred:\n ;; formatted → string (str returns string)\n ;; discount → number (- returns number)\n ;; has-sale → boolean (and returns boolean)\n (let ((formatted (str currency (format-number price 2)))\n (has-sale (and sale-price (< sale-price price)))\n (discount (if has-sale\n (round (* 100 (/ (- price sale-price) price)))\n 0)))\n (div :class \"price\"\n (span :class (if has-sale \"line-through text-stone-400\" \"font-bold\")\n formatted)\n (when has-sale\n (span :class \"text-green-700 font-bold ml-2\"\n (str currency (format-number sale-price 2))\n (span :class \"text-xs ml-1\" (str \"(-\" discount \"%)\")))))))" "lisp")))
;; -----------------------------------------------------------------------
;; Error reporting
;; -----------------------------------------------------------------------
(~docs/section :title "Error Reporting" :id "errors"
(p "Type errors are reported at registration time with source location, expected type, actual type, and the full call chain.")
(~docs/code :src (highlight ";; Example error output:\n;;\n;; TYPE ERROR in ~checkout-summary (checkout.sx:34)\n;;\n;; (str \"Total: \" (compute-total items))\n;; ^^^^^^^^^^^^^^^^^\n;; Argument 2 of `str` expects: string\n;; Got: number (from compute-total :returns number)\n;;\n;; Fix: (str \"Total: \" (str (compute-total items)))\n;;\n;;\n;; TYPE ERROR in ~reactive-islands/event-bridge/product-page (products.sx:12)\n;;\n;; (~plans/typed-sx/product-card :title product-name :price \"29.99\")\n;; ^^^^^^\n;; Keyword :price of ~plans/typed-sx/product-card expects: number\n;; Got: string (literal \"29.99\")\n;;\n;; Fix: (~plans/typed-sx/product-card :title product-name :price 29.99)" "lisp"))
(p "Severity levels:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Level")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "When")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Behavior")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-red-700") "Error")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Definite type mismatch: " (code "number") " where " (code "string") " expected")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Strict mode: startup crash. Permissive: logged warning."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-amber-700") "Warning")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Possible mismatch: " (code "any") " where " (code "number") " expected, unknown kwarg")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Logged, never crashes"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-500") "Info")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Annotation suggestion: frequently-called untyped component")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Dev mode only")))))
(p (code "SX_TYPE_STRICT=1") " (env var, like " (code "SX_BOUNDARY_STRICT") ") makes type errors fatal at startup. Absent = permissive. Same pattern as boundary enforcement."))
;; -----------------------------------------------------------------------
;; Nil narrowing
;; -----------------------------------------------------------------------
(~docs/section :title "Nil Narrowing" :id "nil"
(p "The most common real-world type error in SX: passing a possibly-nil value where a non-nil is required. " (code "get") " returns " (code "any") " (might be nil). " (code "current-user") " returns " (code "dict?") " (explicitly nullable). Piping these into " (code "str") " or arithmetic without checking is the #1 source of runtime errors.")
(~docs/code :src (highlight ";; Before: runtime error if user is nil\n(defcomp ~plans/typed-sx/greeting (&key (user : dict?))\n (h1 (str \"Hello, \" (get user \"name\"))))\n ;; ^^^ TYPE WARNING: user is dict?, get needs non-nil first arg\n\n;; After: checker enforces nil handling\n(defcomp ~plans/typed-sx/greeting (&key (user : dict?))\n (if user\n (h1 (str \"Hello, \" (get user \"name\")))\n ;; In this branch, checker narrows user to `dict` (not nil)\n (h1 \"Hello, guest\")))\n ;; No warning — nil case handled" "lisp"))
(p "Narrowing rules:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "(if x then else)") " — in " (code "then") ", " (code "x") " is narrowed to exclude " (code "nil") " and " (code "false"))
(li (code "(when x body)") " — in " (code "body") ", " (code "x") " is narrowed")
(li (code "(nil? x)") " in an if test — " (code "then") " branch: " (code "x") " is " (code "nil") ", " (code "else") " branch: " (code "x") " is non-nil")
(li (code "(string? x)") " in an if test — " (code "then") " branch: " (code "x") " is " (code "string")))
(p "This is standard flow typing, nothing exotic. TypeScript does the same thing with " (code "if (x !== null)") " narrowing."))
;; -----------------------------------------------------------------------
;; Component signature verification
;; -----------------------------------------------------------------------
(~docs/section :title "Component Signature Verification" :id "signatures"
(p "The highest-value check: verifying that component call sites match declared signatures. This is where most bugs live.")
(~docs/code :src (highlight ";; Definition\n(defcomp ~plans/typed-sx/product-card (&key (title : string)\n (price : number)\n (image-url : string?)\n &rest children)\n ...)\n\n;; Call site checks:\n(~plans/typed-sx/product-card :title \"Widget\" :price 29.99) ;; OK\n(~plans/typed-sx/product-card :title \"Widget\") ;; ERROR: :price required\n(~plans/typed-sx/product-card :title 42 :price 29.99) ;; ERROR: :title expects string\n(~plans/typed-sx/product-card :title \"Widget\" :price 29.99\n (p \"Description\") (p \"Details\")) ;; OK: children\n(~plans/typed-sx/product-card :titel \"Widget\" :price 29.99) ;; WARNING: :titel unknown\n ;; (did you mean :title?)" "lisp"))
(p "The checker walks every component call in every component body. For each call:")
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-1")
(li "Look up the callee component in env")
(li "Match provided keyword args against declared params")
(li "Check each arg's inferred type against the param's declared type")
(li "Report missing required params (those without defaults)")
(li "Report unknown keyword args (with Levenshtein suggestion)"))
(p "This catches the majority of composition bugs. A renamed param, a swapped argument, a missing required field — all caught before serving."))
;; -----------------------------------------------------------------------
;; Thread-first type flow
;; -----------------------------------------------------------------------
(~docs/section :title "Thread-First Type Flow" :id "thread-first"
(p "The " (code "->") " (thread-first) form is SX's primary composition operator. Type checking it means verifying each step's output matches the next step's input:")
(~docs/code :src (highlight ";; (-> items\n;; (filter active?) ;; (list-of dict) → (list-of dict)\n;; (map name) ;; (list-of dict) → (list-of string)\n;; (join \", \")) ;; (list-of string) → string\n;;\n;; Type flow: (list-of dict) → (list-of dict) → (list-of string) → string\n;; Each step's output is the next step's first argument.\n\n;; ERROR example:\n;; (-> items\n;; (filter active?)\n;; (join \", \") ;; join expects (list-of string),\n;; (map name)) ;; got string — wrong order!\n;;\n;; TYPE ERROR: step 3 (map) expects (list-of any) as first arg\n;; got: string (from join)" "lisp"))
(p "The checker threads the inferred type through each step. If any step's input type doesn't match the previous step's output type, it reports the exact point where the pipeline breaks."))
;; -----------------------------------------------------------------------
;; Relationship to prove.sx
;; -----------------------------------------------------------------------
(~docs/section :title "Types vs Proofs" :id "types-vs-proofs"
(p (a :href "/sx/(etc.(plan.theorem-prover))" (~tw :tokens "text-violet-700 underline") "prove.sx") " and types.sx are complementary, not competing:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "types.sx")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "prove.sx")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Checks")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Composition: does A's output fit B's input?")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Properties: is + commutative? Is sort stable?"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Scope")
(td (~tw :tokens "px-3 py-2 text-stone-700") "All component bodies, every call site")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Primitives only (declared in primitives.sx)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Method")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Type inference + checking (fast, O(n) AST walk)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Bounded model checking (exhaustive, slower)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "When")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Registration time (every startup)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "CI / on-demand (not every startup)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Catches")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Wrong arg type, missing param, nil misuse")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Algebraic law violations, edge case failures")))))
(p "Types answer: " (em "\"does this code fit together?\"") " Proofs answer: " (em "\"does this code do the right thing?\"") " Both are spec modules, both bootstrapped, both run without external tools."))
;; -----------------------------------------------------------------------
;; User-defined types
;; -----------------------------------------------------------------------
(~docs/section :title "User-Defined Types" :id "deftype"
(p (code "deftype") " introduces named types — aliases, unions, records, and parameterized types. All are declaration-only, zero runtime cost, resolved at check time.")
(~docs/subsection :title "Type aliases"
(~docs/code :src (highlight ";; Simple name for an existing type\n(deftype price number)\n(deftype html-string string)\n(deftype user-id (or string number))\n\n;; Use anywhere a type is expected\n(defcomp ~plans/typed-sx/price-tag (&key (amount :as price) (label :as string))\n (span :class \"price\" (str label \": $\" (format-decimal amount 2))))" "lisp"))
(p "Aliases are transparent — " (code "price") " IS " (code "number") " for all checking purposes. They exist for documentation and domain semantics."))
(~docs/subsection :title "Union types"
(~docs/code :src (highlight ";; Named unions\n(deftype renderable (union string number nil component))\n(deftype key-type (union string keyword))\n(deftype falsy (union nil false))\n\n;; The checker narrows unions in branches:\n(define handle-input\n (fn ((val :as (union string number)))\n (if (string? val)\n (upper val) ;; narrowed to string — upper is valid\n (+ val 1)))) ;; narrowed to number — + is valid" "lisp"))
(p "Union types compose with narrowing — " (code "if (string? x)") " in the then-branch narrows " (code "(union string number)") " to " (code "string") ". Same flow typing that already works for nullable."))
(~docs/subsection :title "Record types (typed dicts)"
(~docs/code :src (highlight ";; Typed dict shape\n(deftype card-props\n {:title string\n :subtitle string?\n :price number\n :tags (list-of string)})\n\n;; Checker validates dict literals against shape:\n{:title \"Widget\" :price \"oops\"}\n;; ERROR: :price expects number, got string\n\n{:title \"Widget\" :price 29.99}\n;; WARNING: missing :tags (required field)\n\n;; Record types enable typed component props:\n(defcomp ~plans/typed-sx/product-card (&key (props :as card-props) &rest children)\n (div :class \"card\"\n (h2 (get props :title))\n children))" "lisp"))
(p "Records are the big win. Components pass dicts everywhere — config, props, context. A record type makes " (code "get") " on a known-shape dict return the field's type instead of " (code "any") ". This is where " (code "deftype") " pays for itself."))
(~docs/subsection :title "Parameterized types"
(~docs/code :src (highlight ";; Generic over type variables\n(deftype (maybe a) (union nil a))\n(deftype (result a e) (union {:ok a} {:err e}))\n(deftype (pair a b) {:fst a :snd b})\n\n;; Used in signatures:\n(define find-user : (-> number (maybe user-record))\n (fn (id) ...))\n\n;; Checker instantiates: (maybe user-record) = (union nil user-record)\n;; So the caller must handle nil." "lisp"))
(p "Parameterized types are substitution-based — " (code "(maybe string)") " expands to " (code "(union nil string)") " at check time. No inference of type parameters (that would require Hindley-Milner). You write " (code "(maybe string)") " explicitly, the checker substitutes and verifies.")))
;; -----------------------------------------------------------------------
;; Effect system
;; -----------------------------------------------------------------------
(~docs/section :title "Effect System" :id "effects"
(p "The pragmatic middle: static effect " (em "checking") " without algebraic effect " (em "handlers") ". Functions declare what side effects they use. The checker enforces that effects don't leak across boundaries. No continuations, no runtime cost.")
(~docs/subsection :title "Effect declarations"
(~docs/code :src (highlight ";; Declare named effects\n(defeffect io) ;; Database, HTTP, file system\n(defeffect dom) ;; Browser DOM manipulation\n(defeffect async) ;; Asynchronous operations\n(defeffect state) ;; Mutable state (set!, dict-set!, append!)\n\n;; Functions declare their effects in brackets\n(define fetch-user : (-> number user) [io async]\n (fn (id) (query \"SELECT * FROM users WHERE id = $1\" id)))\n\n(define toggle-class : (-> element string nil) [dom]\n (fn (el cls) (set-attr! el :class cls)))\n\n;; Pure by default — no annotation means no effects\n(define add-prices : (-> (list-of number) number)\n (fn (prices) (reduce + 0 prices)))" "lisp")))
(~docs/subsection :title "What it checks"
(ol (~tw :tokens "list-decimal pl-5 text-stone-700 space-y-2")
(li (strong "Pure functions can't call effectful ones.") " A function with no effect annotation calling " (code "fetch-user") " (which has " (code "[io async]") ") is an error. The IO leaked into pure code.")
(li (strong "Components declare their effect ceiling.") " A " (code "[pure]") " component can only call pure functions. A " (code "[io]") " component can call IO but not DOM. This is the render-mode safety guarantee.")
(li (strong "Render modes enforce effect sets.") " " (code "render-to-html") " (server) allows " (code "[io]") " but not " (code "[dom]") ". " (code "render-to-dom") " (browser) allows " (code "[dom]") " but not " (code "[io]") ". " (code "aser") " (wire format) allows " (code "[io]") " for evaluation but serializes the result.")
(li (strong "Islands are the effect boundary.") " Server effects (" (code "io") ") can't cross into client island code. Client effects (" (code "dom") ") can't leak into server rendering. Currently this is convention — effects make it a proof.")))
(~docs/subsection :title "Three effect sets match three render modes"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Render mode")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Allowed effects")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Forbidden")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "render-to-html")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "pure") ", " (code "io") ", " (code "async") ", " (code "state"))
(td (~tw :tokens "px-3 py-2 text-red-600") (code "dom")))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "render-to-dom")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "pure") ", " (code "dom") ", " (code "state"))
(td (~tw :tokens "px-3 py-2 text-red-600") (code "io") ", " (code "async")))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "aser (wire)")
(td (~tw :tokens "px-3 py-2 text-stone-700") (code "pure") ", " (code "io") ", " (code "async") ", " (code "state"))
(td (~tw :tokens "px-3 py-2 text-red-600") (code "dom"))))))
(p "This is exactly the information " (code "deps.sx") " already computes — which components have IO refs. Effects promote it from a runtime classification to a static type-level property. Pure components get an ironclad guarantee: memoize, cache, SSR anywhere, serialize for client — provably safe."))
(~docs/subsection :title "Effect propagation"
(~docs/code :src (highlight ";; Effects propagate through calls:\n(define fetch-prices : (-> (list-of number)) [io async]\n (fn () (query \"SELECT price FROM products\")))\n\n(define render-total : (-> element) [io async] ;; must declare, calls fetch-prices\n (fn ()\n (let ((prices (fetch-prices)))\n (span (str \"$\" (reduce + 0 prices))))))\n\n;; ERROR if you forget:\n(define render-total : (-> element) ;; no effects declared\n (fn ()\n (let ((prices (fetch-prices))) ;; ERROR: calls [io async] from pure context\n (span (str \"$\" (reduce + 0 prices))))))" "lisp"))
(p "The checker walks call graphs and verifies that every function's declared effects are a superset of its callees' effects. This is transitive — if A calls B calls C, and C has " (code "[io]") ", then A must also declare " (code "[io]") "."))
(~docs/subsection :title "Gradual effects"
(p "Like gradual types, effects are opt-in. Unannotated functions are assumed to have " (em "all") " effects — they can call anything, and anything can call them. This is safe (no false positives) but provides no guarantees. As you annotate more functions, the checker catches more violations.")
(p "The practical sweet spot: annotate " (code "defcomp") " declarations (they're the public API) and let the checker verify that pure components don't accidentally depend on IO. Internal helpers can stay unannotated until they matter.")
(~docs/code :src (highlight ";; Annotated component — checker enforces purity\n(defcomp ~plans/typed-sx/price-display [pure] (&key (price :as number))\n (span :class \"price\" (str \"$\" (format-decimal price 2))))\n\n;; ERROR: pure component calls IO\n(defcomp ~plans/typed-sx/price-display [pure] (&key (product-id :as number))\n (let ((product (fetch-product product-id))) ;; ERROR: [io] in [pure] context\n (span :class \"price\" (str \"$\" (get product \"price\")))))" "lisp")))
(~docs/subsection :title "Relationship to deps.sx and boundary.sx"
(p "Effects don't replace the existing systems — they formalize them:")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "boundary.sx") " declares which primitives are IO. Effects declare which " (em "functions") " use IO.")
(li (code "deps.sx") " computes transitive IO refs at registration time. Effects check them at type-check time — earlier, with better error messages.")
(li "The boundary is still the source of truth for " (em "what is IO") ". Effects are the enforcement mechanism for " (em "who can use it") "."))
(p "Long term, " (code "deps.sx") "'s IO classification can be derived from effect annotations. In the short term, both coexist — effects are checked, deps are computed, both must agree."))
(~docs/subsection :title "What it does NOT do"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (strong "No algebraic effect handlers.") " You can't intercept and resume effects. This would require delimited continuations in every bootstrapper target — massive complexity for marginal UI benefit.")
(li (strong "No effect polymorphism.") " You can't write a function generic over effects (" (code "forall e. (-> a [e] b)") "). This needs higher-kinded effect types — the same complexity as type classes.")
(li (strong "No effect inference.") " Effects are declared, not inferred. Inference would require whole-program analysis and produce confusing error messages.")
(li (strong "No runtime cost.") " Effects are erased after checking. The bootstrapped code has zero overhead. This is purely a static guarantee."))
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mt-4")
(p (~tw :tokens "text-violet-900 font-medium") "The door stays open.")
(p (~tw :tokens "text-violet-800") "Static effect checking is the pragmatic middle — 80% of the benefit of a full effect system, 20% of the complexity. If SX ever needs algebraic handlers (e.g. for suspense, or cooperative scheduling beyond what " (code "shift") "/" (code "reset") " provides), the annotation syntax is already in place. Add handlers later without changing any existing code."))))
;; -----------------------------------------------------------------------
;; Implementation
;; -----------------------------------------------------------------------
(~docs/section :title "Implementation" :id "implementation"
(~docs/subsection :title "Phase 1: Type Registry (done)"
(p "Build the type registry from existing declarations.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Parse " (code ":returns") " from " (code "primitives.sx") " and " (code "boundary.sx") " into a type map: " (code "primitive-name → return-type"))
(li "Parse " (code ":params") " declarations into param type maps (currently untyped — default to " (code "any") ")")
(li "Compute component signatures from " (code "parse-comp-params") " + any type annotations")
(li "Store in env as metadata alongside existing component/primitive objects")))
(~docs/subsection :title "Phase 2: Type Inference Engine (done)"
(p "Walk AST, infer types bottom-up.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Literals → concrete types")
(li "Primitive calls → look up return type in registry")
(li "Component calls → " (code "element"))
(li "Let bindings → RHS inferred type")
(li "If/when/cond → union of branch types, with narrowing in branches")
(li "Lambda → " (code "(-> param-types return-type)") " from body inference")
(li "Map/filter → propagate element types through the transform")))
(~docs/subsection :title "Phase 3: Type Checker (done)"
(p "Compare inferred types at call sites against declared types.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li "Subtype check: " (code "number") " <: " (code "any") ", " (code "string") " <: " (code "string?") ", " (code "nil") " <: " (code "string?"))
(li "Error on definite mismatch: " (code "number") " vs " (code "string"))
(li "Warn on possible mismatch: " (code "any") " vs " (code "number") " (might work, might not)")
(li "Component kwarg checking: required params, unknown kwargs, type mismatches")))
(~docs/subsection :title "Phase 4: Annotation Parsing (done)"
(p "Extend " (code "parse-comp-params") " and " (code "sf-defcomp") " to recognize type annotations.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "(name : type)") " in param lists → extract type, store in component metadata")
(li (code ":returns type") " in lambda/fn bodies → store as declared return type")
(li "Backward compatible: unannotated params remain " (code "any"))))
(~docs/subsection :title "Phase 5: Typed Primitives (done)"
(p "Add param types to " (code "primitives.sx") " declarations.")
(~docs/code :src (highlight ";; Current\n(define-primitive \"+\"\n :params (&rest args)\n :returns \"number\"\n :doc \"Sum all arguments.\")\n\n;; Extended\n(define-primitive \"+\"\n :params (&rest (args : number))\n :returns \"number\"\n :doc \"Sum all arguments.\")" "lisp"))
(p "This is the biggest payoff for effort: ~80 primitives gain param types, enabling the checker to catch every mistyped primitive call across the entire codebase."))
(~docs/subsection :title "Phase 6: User-Defined Types (deftype)"
(p "Extend the type system with named user-defined types.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "deftype") " special form — parsed by evaluator, stored in type registry")
(li "Type aliases: " (code "(deftype price number)") " → transparent substitution")
(li "Union types: " (code "(deftype renderable (union string number nil))") " → checked via " (code "subtype?"))
(li "Record types: " (code "(deftype card-props {:title string :price number})") " → dict shape validation")
(li "Parameterized types: " (code "(deftype (maybe a) (union nil a))") " → substitution-based instantiation")
(li "Extend " (code "infer-type") " — " (code "(get d :title)") " on a known record returns the field type instead of " (code "any"))
(li "Extend " (code "subtype?") " — resolve named types through the registry before comparing")
(li "Test: dict literal against record shape, parameterized type instantiation, field-typed " (code "get"))))
(~docs/subsection :title "Phase 7: Static Effect System"
(p "Add effect annotations and static checking. No handlers, no runtime cost.")
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (code "defeffect") " declaration form — registers named effects: " (code "io") ", " (code "dom") ", " (code "async") ", " (code "state"))
(li "Effect annotation syntax: " (code "[io async]") " after function type or on " (code "defcomp"))
(li "Extend " (code "check-body-walk") " — verify called functions' effects are subset of caller's declared effects")
(li "Transitive propagation: if A calls B calls C[io], A must declare [io]")
(li "Render mode enforcement: " (code "render-to-html") " forbids " (code "[dom]") ", " (code "render-to-dom") " forbids " (code "[io]"))
(li "Gradual: unannotated functions assumed " (code "[all]") " — safe but unchecked. Annotations opt in.")
(li "Integration with " (code "deps.sx") " — IO classification derived from effect annotations when available")
(li "Integration with " (code "boundary.sx") " — IO primitives automatically carry " (code "[io]") " effect")
(li "Test: pure calling IO = error, DOM in server render = error, island boundary enforcement"))))
;; -----------------------------------------------------------------------
;; Spec module
;; -----------------------------------------------------------------------
(~docs/section :title "Spec Module" :id "spec-module"
(p (code "types.sx") " — the type checker, written in SX, bootstrapped to every host.")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Function")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Purpose")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "infer-type")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Infer the type of an AST node in a given type environment"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "check-call")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Check a function/component call against declared signature"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "check-component")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Type-check an entire component body"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "check-all")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Check all registered components, return error/warning list"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "subtype?")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Is type A a subtype of type B?"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "narrow-type")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Narrow a type based on a predicate test (nil?, string?, etc)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "type-union")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Compute the union of two types"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "parse-type-annotation")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Parse a type expression from annotation syntax"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "build-type-registry")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Build type map from primitives.sx + boundary.sx declarations"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "check-primitive-call")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Check a primitive call against declared param types (positional + rest)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "resolve-type")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Resolve a named type through the deftype registry (Phase 6)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-mono text-sm text-violet-700") "check-effects")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Verify effect annotations: callee effects ⊆ caller effects (Phase 7)")))))
(p "The checker is a growing spec module — currently ~650 lines of SX. It's an AST walk with a type environment, structurally similar to " (code "deps.sx") " (which walks ASTs to find IO refs) and " (code "prove.sx") " (which walks ASTs to generate verification conditions). Same pattern, different question."))
;; -----------------------------------------------------------------------
;; Relationships
;; -----------------------------------------------------------------------
(~docs/section :title "Relationships" :id "relationships"
(ul (~tw :tokens "list-disc pl-5 text-stone-700 space-y-1")
(li (a :href "/sx/(etc.(plan.theorem-prover))" (~tw :tokens "text-violet-700 underline") "Theorem Prover") " — prove.sx verifies primitive properties; types.sx verifies composition. Complementary.")
(li (a :href "/sx/(etc.(plan.content-addressed-components))" (~tw :tokens "text-violet-700 underline") "Content-Addressed Components") " — component manifests gain type signatures. A consumer knows param types before fetching the source.")
(li (a :href "/sx/(etc.(plan.environment-images))" (~tw :tokens "text-violet-700 underline") "Environment Images") " — the type registry serializes into the image. Type checking happens once at image build time, not on every startup.")
(li (a :href "/sx/(etc.(plan.runtime-slicing))" (~tw :tokens "text-violet-700 underline") "Runtime Slicing") " — types.sx is a registration-time module, not a runtime module. It doesn't ship to the client. Zero impact on bundle size."))
(div (~tw :tokens "rounded border border-amber-200 bg-amber-50 p-3 mt-2")
(p (~tw :tokens "text-amber-800 text-sm") (strong "Depends on: ") "primitives.sx (return + param types), boundary.sx (IO return types + IO classification), eval.sx (defcomp/deftype/defeffect parsing), deps.sx (IO detection — effects formalize this). " (strong "New: ") (code "types.sx") " spec module, type annotations in " (code "parse-comp-params") ", " (code "deftype") " declaration form, " (code "defeffect") " declaration form."))
(div (~tw :tokens "rounded border border-violet-200 bg-violet-50 p-4 mt-4")
(p (~tw :tokens "text-violet-900 font-medium") "Why not a Haskell host?")
(p (~tw :tokens "text-violet-800") "A Haskell SX host would type-check the " (em "host") " code (the evaluator, renderer, parser). But it can't type-check " (em "SX programs") " — those are dynamically typed values passing through " (code "SxVal") ". " (code "types.sx") " checks SX programs directly, on every host. One spec, all hosts benefit. The type system lives where it matters — in the language, not in any particular implementation of it."))))))

260
sx/sx/plans/wasm-bytecode-vm.sx
View File

@@ -1,260 +0,0 @@
;; ---------------------------------------------------------------------------
;; WASM Bytecode VM — Compile SX to bytecode, run in Rust/WASM
;; ---------------------------------------------------------------------------
(defcomp ~plans/wasm-bytecode-vm/plan-wasm-bytecode-vm-content ()
(~docs/page :title "WASM Bytecode VM"
(~docs/section :title "The Idea" :id "idea"
(p "Currently the client-side SX runtime is a tree-walking interpreter bootstrapped to JavaScript. The server sends " (strong "SX source text") " — component definitions, page content — and the browser parses and evaluates it.")
(p "The alternative: compile SX to a " (strong "compact bytecode format") ", ship bytecode to the browser, and execute it in a " (strong "WebAssembly VM written in Rust") ". The VM calls out to JavaScript for DOM operations and I/O via standard WASM↔JS bindings.")
(p "This fits naturally into the SX host architecture. Rust becomes another bootstrapper target. The spec compiles to Rust the same way it compiles to Python and JavaScript. The WASM module is the client-side expression of that Rust target."))
;; -----------------------------------------------------------------------
;; Why
;; -----------------------------------------------------------------------
(~docs/section :title "Why" :id "why"
(ul (~tw :tokens "list-disc list-inside space-y-2")
(li (strong "Wire size") " — bytecode is far more compact than source text. No redundant whitespace, no comments, no repeated symbol names. A component bundle that's 40KB of SX source might be 8KB of bytecode.")
(li (strong "No parse overhead") " — the browser currently parses every SX source string (tokenize → AST → eval). Bytecode skips parsing entirely.")
(li (strong "Eval performance") " — a Rust VM with a tight dispatch loop is significantly faster than tree-walking in JavaScript. Matters for compute-heavy islands, large list rendering, complex CSSX calculations.")
(li (strong "Rust as a host target") " — an architectural goal. The spec should compile to every host. Rust/WASM proves the architecture is truly portable.")
(li (strong "Content-addressed bytecode") " — bytecode modules have deterministic content hashes (CIDs). Fits perfectly with the content-addressed components plan — fetch bytecode by CID from anywhere.")))
;; -----------------------------------------------------------------------
;; Architecture
;; -----------------------------------------------------------------------
(~docs/section :title "Architecture" :id "architecture"
(p "Three new layers, all specced in " (code ".sx") " and bootstrapped:")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "1. Bytecode format — bytecode.sx")
(p "A spec for the bytecode instruction set. Stack-based VM (simpler than register-based, natural fit for s-expressions). Instructions:")
(~docs/code :src (highlight ";; Core instructions\nPUSH_CONST idx ;; push constant from pool\nPUSH_NIL ;; push nil\nPUSH_TRUE / PUSH_FALSE\nLOOKUP idx ;; look up symbol by index\nSET idx ;; define/set symbol\nCALL n ;; call top-of-stack with n args\nTAIL_CALL n ;; tail call (TCO)\nRETURN\nJUMP offset ;; unconditional jump\nJUMP_IF_FALSE offset ;; conditional jump\nMAKE_LAMBDA idx n_params ;; create closure\nMAKE_LIST n ;; collect n stack values into list\nMAKE_DICT n ;; collect 2n stack values into dict\nPOP ;; discard top\nDUP ;; duplicate top" "lisp"))
(p "Bytecode modules contain: a " (strong "constant pool") " (strings, numbers, symbols), a " (strong "code section") " (instruction bytes), and a " (strong "metadata section") " (source maps, component/island declarations for the host to register).")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "2. Compiler — compile.sx")
(p "An SX-to-bytecode compiler, " (strong "written in SX") ". Takes parsed AST, emits bytecode modules. Handles:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Macro expansion") " — all macros expanded at compile time. The VM never sees macros.")
(li (strong "Constant folding") " — pure expressions with known values computed at compile time.")
(li (strong "Closure analysis") " — determines free variables for each lambda, emits efficient capture instructions.")
(li (strong "Tail call detection") " — emits TAIL_CALL instead of CALL + RETURN for tail positions.")
(li (strong "Component metadata") " — defcomp/defisland declarations are extracted and stored in the module metadata, so the host can register them without evaluating the body."))
(p "Bootstrapped to Python (server-side compilation) and Rust (if self-compilation is needed).")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "3. VM — bootstrap_rs.py → Rust/WASM")
(p "A Rust implementation of the SX platform interface. The bootstrapper (" (code "bootstrap_rs.py") ") translates the spec to Rust source, which compiles to both:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Native binary") " — for server-side evaluation (replaces Python evaluators entirely)")
(li (strong "WASM module") " — for browser-side evaluation (replaces sx-browser.js)")))
;; -----------------------------------------------------------------------
;; DOM interop
;; -----------------------------------------------------------------------
(~docs/section :title "DOM Interop" :id "dom-interop"
(p "The main engineering challenge. Every DOM operation crosses the WASM↔JS boundary. Two strategies:")
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Strategy A: Direct calls")
(p "Each DOM operation (" (code "createElement") ", " (code "setAttribute") ", " (code "appendChild") ") is a separate WASM→JS call. Simple, works, but ~50ns overhead per call. For a page with 1,000 DOM operations, that's ~50μs — negligible.")
(~docs/code :src (highlight "// JS side — imported by WASM\nfunction domCreateElement(tag_ptr, tag_len) {\n const tag = readString(tag_ptr, tag_len);\n return storeHandle(document.createElement(tag));\n}\n\n// Rust side\nextern \"C\" { fn dom_create_element(tag: *const u8, len: u32) -> u32; }" "javascript"))
(h4 (~tw :tokens "font-semibold mt-4 mb-2") "Strategy B: Command buffer")
(p "Batch DOM operations in WASM memory as a command buffer. Flush to JS in one call. JS walks the buffer and applies all operations. Fewer boundary crossings, but more complex.")
(~docs/code :src (highlight ";; Command buffer format (in shared WASM memory)\n;; [CREATE_ELEMENT, tag_idx, handle_out]\n;; [SET_ATTR, handle, key_idx, val_idx]\n;; [APPEND_CHILD, parent_handle, child_handle]\n;; [SET_TEXT, handle, text_idx]\n;; Then: (flush-dom-commands)" "lisp"))
(p "Strategy A is simpler and sufficient for SX workloads. Strategy B is an optimisation if profiling shows the boundary crossing matters. " (strong "Start with A, measure, switch to B only if needed.")))
;; -----------------------------------------------------------------------
;; String handling
;; -----------------------------------------------------------------------
(~docs/section :title "String Handling" :id "strings"
(p "WASM has no native string type. Strings must cross the boundary via shared " (code "ArrayBuffer") " memory. Options:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Copy on crossing") " — encode to UTF-8 in WASM linear memory, JS reads via " (code "TextDecoder") ". Simple, safe, ~1μs per string.")
(li (strong "String interning") " — the constant pool already interns all string literals. Assign each a numeric ID. DOM operations reference strings by ID. JS maintains a parallel string table. Strings never cross the boundary — only IDs do.")
(li (strong "Hybrid") " — intern constants (attribute names, tag names, class names), copy dynamic strings (computed CSS values, interpolated text)."))
(p "String interning is the right default — most DOM attribute values in SX are constant strings. Dynamic strings (like CSSX colour output) are the minority. The constant pool already has all the static strings; just share it with JS at init time."))
;; -----------------------------------------------------------------------
;; Memory management
;; -----------------------------------------------------------------------
(~docs/section :title "Memory & Closures" :id "memory"
(p "SX values that the VM must manage:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "Closures") " — lambda captures free variables. Rust: " (code "Rc<Closure>") " with captured env as " (code "Vec<Value>") ".")
(li (strong "Signals") " — reference-counted mutable cells with subscriber lists. Subscribers hold weak references to computed nodes to avoid cycles.")
(li (strong "Lists/Dicts") " — immutable by convention (SX doesn't mutate collections). Arena-allocate per evaluation, free the arena when done.")
(li (strong "DOM handles") " — opaque integers referencing JS-side DOM nodes. A handle table in JS maps handle IDs to actual DOM objects. Handles are freed when the island disposes."))
(p "The " (code "with-island-scope") " pattern already models the cleanup boundary. In Rust: each island gets its own " (code "Arena") " + signal scope. When the island is removed from the DOM, drop the arena — all closures, signals, and DOM handles for that island are freed in one shot."))
;; -----------------------------------------------------------------------
;; What gets compiled
;; -----------------------------------------------------------------------
(~docs/section :title "What Gets Compiled" :id "compilation"
(p "Not everything needs bytecode. The compilation boundary follows the existing server/client split:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Content")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Format")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Why")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Component definitions")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Bytecode")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Evaluated on every page load, benefits from fast dispatch"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Client library (@client files)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Bytecode")
(td (~tw :tokens "px-3 py-2 text-stone-600") "CSSX functions, colour computation — pure code that runs client-side"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Page content (SX wire responses)")
(td (~tw :tokens "px-3 py-2 text-stone-700") "SX source or bytecode")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Wire responses are small, parse overhead minimal. Bytecode optional."))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Macros")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Expanded at compile time")
(td (~tw :tokens "px-3 py-2 text-stone-600") "VM never sees macros — they're pure compile-time constructs"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "Server-affinity components")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Not compiled")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Expanded server-side, never sent to client"))))))
;; -----------------------------------------------------------------------
;; Bytecode vs direct WASM compilation
;; -----------------------------------------------------------------------
(~docs/section :title "Bytecode VM vs Direct WASM Compilation" :id "vm-vs-direct"
(p "Two paths to WASM. The choice matters:")
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Bytecode VM in WASM")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Compile SX → WASM directly")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Complexity")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Standard VM design — proven pattern")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Full compiler backend (SSA, register alloc, WASM codegen)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Dynamic loading")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Trivial — load bytecode module, eval")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Hard — must instantiate new WASM module per chunk"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "eval / REPL")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Works — compile + eval at runtime")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Impossible without bundling a compiler"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Performance")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Fast — WASM dispatch loop, no JS overhead")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Fastest — native WASM speed, no dispatch"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Module size")
(td (~tw :tokens "px-3 py-2 text-stone-700") "One VM module (~100KB) + bytecode per page")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Per-page WASM modules, each self-contained"))
(tr
(td (~tw :tokens "px-3 py-2 font-semibold text-stone-700") "Debugging")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Source maps over bytecode")
(td (~tw :tokens "px-3 py-2 text-stone-700") "DWARF debug info in WASM")))))
(p (strong "Bytecode VM is the right choice.") " SX needs dynamic loading (HTMX responses inject new components), runtime eval (islands, reactive updates), and incremental compilation (page-by-page). Direct WASM compilation is better for static, ahead-of-time scenarios — not for a live hypermedia system."))
;; -----------------------------------------------------------------------
;; Dual target — same spec, runtime choice
;; -----------------------------------------------------------------------
(~docs/section :title "Dual Target: JS or WASM from the Same Spec" :id "dual-target"
(p "The key insight: this is " (strong "not a replacement") " for the JS evaluator. It's " (strong "another compilation target from the same spec") ". The existing bootstrapper pipeline already proves this pattern:")
(~docs/code :src (highlight "eval.sx ──→ bootstrap_js.py ──→ sx-ref.js (browser, JS eval)\n ──→ bootstrap_py.py ──→ sx_ref.py (server, Python eval)\n ──→ bootstrap_rs.py ──→ sx-vm.wasm (browser, WASM eval) ← new" "text"))
(p "All three outputs have identical semantics because they're compiled from the same source. The choice of which to use is a " (strong "deployment decision") ", not an architectural one:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li (strong "JS-only") " — current default. Works everywhere. Zero WASM dependency. Ship sx-browser.js + SX source text.")
(li (strong "WASM-only") " — maximum performance. Ship sx-vm.wasm + bytecode. Requires WASM support (99%+ of browsers).")
(li (strong "Progressive") " — try WASM, fall back to JS. Ship both. The server sends bytecode in a " (code "<script type=\"text/sx-bytecode\">") " tag and source in " (code "<script type=\"text/sx\">") ". The boot script picks whichever runtime loaded.")
(li (strong "Per-page") " — heavy pages (data tables, complex islands) use WASM. Simple pages use JS. The server decides based on page complexity."))
(p "The server-side choice is the same pattern. The Python bootstrapped evaluator (" (code "sx_ref.py") ") and the Rust native binary are interchangeable. A deployment could use Rust for production (speed) and Python for development (debugging, hot-reload).")
(p "This is the architectural payoff of the self-hosting spec. Write the semantics once. Compile to every target. Choose at deploy time which targets to use. " (strong "The spec doesn't know or care which host runs it.")))
;; -----------------------------------------------------------------------
;; Implementation phases
;; -----------------------------------------------------------------------
(~docs/section :title "Implementation Phases" :id "phases"
(div (~tw :tokens "overflow-x-auto rounded border border-stone-200 mb-4")
(table (~tw :tokens "w-full text-left text-sm")
(thead (tr (~tw :tokens "border-b border-stone-200 bg-stone-100")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Phase")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "What")
(th (~tw :tokens "px-3 py-2 font-medium text-stone-600") "Deliverable")))
(tbody
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "1")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Spec the bytecode format in bytecode.sx. Instruction set, constant pool layout, module structure, encoding.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "bytecode.sx — format spec + serializer/deserializer"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "2")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Write the compiler in SX. AST → bytecode. Macro expansion, constant folding, tail call detection, closure analysis.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "compile.sx — bootstrapped to Python for server-side compilation"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "3")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Write bootstrap_rs.py — Rust bootstrapper. Translates spec to Rust source implementing the platform interface.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "bootstrap_rs.py + generated Rust crate"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "4")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Implement bytecode VM in Rust. Dispatch loop, value representation, closure/env model, GC strategy.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "sx-vm crate — native binary + WASM target"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "5")
(td (~tw :tokens "px-3 py-2 text-stone-700") "JS bindings for DOM. WASM imports for createElement, setAttribute, appendChild, event listeners. Handle table.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "sx-vm-dom.js — JS glue layer (~2KB)"))
(tr (~tw :tokens "border-b border-stone-100")
(td (~tw :tokens "px-3 py-2 text-stone-700") "6")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Server-side bytecode compilation pipeline. Component registration emits bytecode alongside source. Bytecode hash for caching.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Bytecode in data-components script tags, fallback to source"))
(tr
(td (~tw :tokens "px-3 py-2 text-stone-700") "7")
(td (~tw :tokens "px-3 py-2 text-stone-700") "Shadow-compare: run JS evaluator and WASM VM in parallel, assert identical DOM output on every page render.")
(td (~tw :tokens "px-3 py-2 text-stone-600") "Confidence to switch over"))))))
;; -----------------------------------------------------------------------
;; Interaction with existing plans
;; -----------------------------------------------------------------------
(~docs/section :title "Interaction with Other Plans" :id "interactions"
(ul (~tw :tokens "list-disc list-inside space-y-2")
(li (strong "Async Eval Convergence") " — must complete first. The spec must be the single evaluator before we add another target. Otherwise we'd be bootstrapping a fork.")
(li (strong "Runtime Slicing") " — the WASM module can be tiered just like the JS runtime. L0 hypermedia needs no VM at all (pure HTML). L1 DOM ops needs a minimal VM. L2 islands needs signals. The WASM module should be tree-shakeable.")
(li (strong "Content-Addressed Components") " — bytecode modules are ideal for content addressing. Deterministic compilation means the same SX source always produces the same bytecode → same CID. Fetch bytecode by CID from IPFS.")
(li (strong "Self-Hosting Bootstrappers") " — compile.sx is bootstrapped by the existing Python bootstrapper. The Rust bootstrapper translates the spec. Self-hosting chain: SX → Python → Rust → WASM.")
(li (strong "js.sx AOT compiler") " — complementary, not competing. js.sx compiles components to static JS for zero-runtime sites. The WASM VM is for dynamic sites with islands, signals, and runtime eval. Both are valid compilation targets.")))
;; -----------------------------------------------------------------------
;; Principles
;; -----------------------------------------------------------------------
(~docs/section :title "Principles" :id "principles"
(ul (~tw :tokens "list-disc list-inside space-y-2")
(li (strong "The spec remains the single source of truth.") " The bytecode format, compiler, and VM semantics are all specced in .sx. The Rust VM is just another host, like Python and JavaScript.")
(li (strong "Bytecode is an optimisation, not a requirement.") " SX source text remains a valid wire format. The system degrades gracefully — if WASM isn't available, fall back to the JS evaluator. Progressive enhancement.")
(li (strong "The VM is dumb, the compiler is smart.") " Macro expansion, constant folding, tail call detection — all done at compile time. The VM is a simple dispatch loop. This keeps the WASM module small and the compilation fast.")
(li (strong "DOM is on the other side of the wall.") " The VM never touches DOM directly. All DOM operations go through the JS binding layer. This keeps the WASM module platform-independent — the same VM could target Node.js, Deno, native desktop (via webview), or headless rendering.")
(li (strong "Shadow-compare before switching.") " Same principle as async eval convergence. Run both paths in parallel, assert identical output. No big-bang cutover.")))
;; -----------------------------------------------------------------------
;; Outcome
;; -----------------------------------------------------------------------
(~docs/section :title "Outcome" :id "outcome"
(p "After completion:")
(ul (~tw :tokens "list-disc list-inside space-y-2 mt-2")
(li "SX compiles to four targets: JavaScript, Python, Rust (native), Rust (WASM)")
(li "Client wire format is ~5x smaller (bytecode vs source text)")
(li "No parsing overhead on the client — bytecode loads directly into the VM")
(li "Reactive islands run at near-native speed inside WASM")
(li "The architecture proof is complete: one spec, every host, every target")
(li "Content-addressed bytecode modules can be fetched from any CDN or IPFS gateway")))))