giles
6f96452f70
Two bugs caused code blocks to render empty across the site: 1. ~docs/code component had parameter named `code` which collided with the HTML <code> tag name. Renamed to `src` and updated all 57 callers. Added font-mono class for explicit monospace. 2. Batched IO dispatch in ocaml_bridge.py only skipped one leading number (batch ID) but the format has two (epoch + ID): (io-request EPOCH ID "name" args...). Changed to skip all leading numbers so the string name is correctly found. This fixes highlight and other batchable helpers returning empty results. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
124 lines
11 KiB
Plaintext
124 lines
11 KiB
Plaintext
;; ---------------------------------------------------------------------------
|
|
;; Async Evaluator Convergence — Bootstrap async_eval.py from Spec
|
|
;; ---------------------------------------------------------------------------
|
|
|
|
(defcomp ~plans/async-eval-convergence/plan-async-eval-convergence-content ()
|
|
(~docs/page :title "Async Evaluator Convergence"
|
|
|
|
(~docs/section :title "The Problem" :id "problem"
|
|
(p "There are currently " (strong "three") " lambda call implementations that must be kept in sync:")
|
|
(ol :class "list-decimal list-inside space-y-2 mt-2"
|
|
(li (code "shared/sx/ref/eval.sx") " — the canonical spec, bootstrapped to " (code "sx-ref.js") " and " (code "sx_ref.py"))
|
|
(li (code "shared/sx/evaluator.py") " — hand-written synchronous Python evaluator")
|
|
(li (code "shared/sx/async_eval.py") " — hand-written asynchronous Python evaluator (the production server path)"))
|
|
(p "Every semantic change to the evaluator — lenient lambda arity, new special forms, calling convention tweaks — must be applied to all three. The spec is authoritative but " (code "async_eval.py") " is what actually serves pages. This is a maintenance hazard and a source of subtle divergence bugs.")
|
|
(p "The lenient arity change (lambda params pad missing args with nil instead of erroring) exposed this: the spec and sync evaluator were updated, but " (code "async_eval.py") " still had strict arity checking, causing production crashes."))
|
|
|
|
(~docs/section :title "Why async_eval.py Exists" :id "why"
|
|
(p "The async evaluator exists because SX page rendering needs to:")
|
|
(ul :class "list-disc list-inside space-y-2 mt-2"
|
|
(li (strong "Await IO primitives") " — page helpers like " (code "highlight") ", " (code "reference-data") ", " (code "component-source") " call Python async functions (DB queries, HTTP fetches). The spec evaluator is synchronous.")
|
|
(li (strong "Expand server-affinity components") " — " (code ":affinity :server") " components must be fully expanded server-side before serialising to SX wire format. This requires async eval of the component body.")
|
|
(li (strong "Handle the aser rendering mode") " — the " (code "_aser") " function evaluates control flow server-side but serialises HTML tags and component calls as SX source for the client. This hybrid eval/serialize mode isn't in the spec."))
|
|
(p "None of these require " (em "different") " evaluation semantics — they require the " (em "same") " semantics with async IO at the boundary."))
|
|
|
|
;; -----------------------------------------------------------------------
|
|
;; Architecture
|
|
;; -----------------------------------------------------------------------
|
|
|
|
(~docs/section :title "Target Architecture" :id "architecture"
|
|
(p "The goal is to " (strong "eliminate hand-written evaluator code entirely") ". All evaluation semantics come from the spec via bootstrapping. The host provides only:")
|
|
(ul :class "list-disc list-inside space-y-2 mt-2"
|
|
(li (strong "Platform primitives") " — type constructors, env operations, DOM/HTML primitives")
|
|
(li (strong "Async IO bridge") " — a thin wrapper that makes the bootstrapped evaluator await-compatible")
|
|
(li (strong "Rendering modes") " — aser/render-to-html dispatch, already mostly specced in " (code "render.sx")))
|
|
(p "The bootstrapped " (code "sx_ref.py") " already has correct eval semantics. The question is how to make it async-aware without forking the spec."))
|
|
|
|
;; -----------------------------------------------------------------------
|
|
;; Approach: Async Adapter Layer
|
|
;; -----------------------------------------------------------------------
|
|
|
|
(~docs/section :title "Approach: Async Adapter Layer" :id "approach"
|
|
(p "Rather than making the spec itself async (which would pollute it with Python-specific concerns), introduce a thin adapter layer between the bootstrapped evaluator and the IO boundary:")
|
|
|
|
(h4 :class "font-semibold mt-4 mb-2" "Phase 1 — Async call hook")
|
|
(p "The bootstrapped evaluator calls primitives via " (code "apply(fn, args)") ". In the Python host, " (code "apply") " is a platform primitive. Replace it with an async-aware version:")
|
|
(~docs/code :src (highlight "(define apply-fn\n (fn (f args)\n ;; Platform provides: if f returns a coroutine, await it\n (apply-maybe-async f args)))" "lisp"))
|
|
(p "The bootstrapper emits " (code "apply_maybe_async") " as a Python " (code "async def") " that checks if the result is a coroutine and awaits it if so. Pure functions return immediately. IO primitives return coroutines that get awaited. " (strong "Zero overhead for pure calls") " — just an " (code "isinstance") " check.")
|
|
|
|
(h4 :class "font-semibold mt-4 mb-2" "Phase 2 — Async trampoline")
|
|
(p "The spec's trampoline loop resolves thunks synchronously. The Python bootstrapper emits an " (code "async def trampoline") " variant that can await thunks whose bodies contain IO calls. The trampoline structure is identical — only the " (code "await") " keyword is added.")
|
|
(~docs/code :src (highlight "# Bootstrapper emits this for Python async target\nasync def trampoline(val):\n while isinstance(val, Thunk):\n val = await eval_expr(val.expr, val.env)\n return val" "python"))
|
|
|
|
(h4 :class "font-semibold mt-4 mb-2" "Phase 3 — Aser as spec module")
|
|
(p "The " (code "_aser") " rendering mode (evaluate control flow, serialize HTML/components as SX source) should be specced as a module in " (code "render.sx") " alongside " (code "render-to-html") " and " (code "render-to-dom") ". It's currently hand-written Python because it predates the spec, but its logic is pure SX: walk the AST, eval certain forms, serialize others.")
|
|
(p "Once aser is specced, the bootstrapper emits it with the same async adapter — IO calls within aser bodies get awaited transparently.")
|
|
|
|
(h4 :class "font-semibold mt-4 mb-2" "Phase 4 — Delete hand-written evaluators")
|
|
(p "With the async adapter + specced aser, " (code "evaluator.py") " and " (code "async_eval.py") " become dead code. Delete them. All evaluation flows through the bootstrapped " (code "sx_ref.py") " with async adapter."))
|
|
|
|
;; -----------------------------------------------------------------------
|
|
;; What changes in the bootstrapper
|
|
;; -----------------------------------------------------------------------
|
|
|
|
(~docs/section :title "Bootstrapper Changes" :id "bootstrapper"
|
|
(p "The Python bootstrapper (" (code "bootstrap_py.py") ") gains a new emit mode: " (code "--async") ". This emits:")
|
|
(ul :class "list-disc list-inside space-y-2 mt-2"
|
|
(li (code "async def eval_expr") " instead of " (code "def eval_expr"))
|
|
(li (code "async def trampoline") " with " (code "await") " on thunk eval")
|
|
(li (code "apply_maybe_async") " that awaits coroutine results")
|
|
(li "All higher-order forms (" (code "map") ", " (code "filter") ", " (code "reduce") ", etc.) as " (code "async def") " with " (code "await") " on callback invocations"))
|
|
(p "The JS bootstrapper is unaffected — the browser evaluator is synchronous (IO is handled by the SxEngine request pipeline, not inline eval)."))
|
|
|
|
;; -----------------------------------------------------------------------
|
|
;; Migration path
|
|
;; -----------------------------------------------------------------------
|
|
|
|
(~docs/section :title "Migration Path" :id "migration"
|
|
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
|
|
(table :class "w-full text-left text-sm"
|
|
(thead (tr :class "border-b border-stone-200 bg-stone-100"
|
|
(th :class "px-3 py-2 font-medium text-stone-600" "Step")
|
|
(th :class "px-3 py-2 font-medium text-stone-600" "What")
|
|
(th :class "px-3 py-2 font-medium text-stone-600" "Risk")))
|
|
(tbody
|
|
(tr :class "border-b border-stone-100"
|
|
(td :class "px-3 py-2 text-stone-700" "1")
|
|
(td :class "px-3 py-2 text-stone-700" "Add async emit mode to bootstrap_py.py. Generate async_sx_ref.py alongside sx_ref.py.")
|
|
(td :class "px-3 py-2 text-stone-600" "Low — new file, nothing changes yet"))
|
|
(tr :class "border-b border-stone-100"
|
|
(td :class "px-3 py-2 text-stone-700" "2")
|
|
(td :class "px-3 py-2 text-stone-700" "Run async_sx_ref.py in parallel with async_eval.py, compare outputs on every page render.")
|
|
(td :class "px-3 py-2 text-stone-600" "Low — shadow mode, no user impact"))
|
|
(tr :class "border-b border-stone-100"
|
|
(td :class "px-3 py-2 text-stone-700" "3")
|
|
(td :class "px-3 py-2 text-stone-700" "Spec aser in render.sx. Bootstrap it. Shadow-compare with hand-written _aser.")
|
|
(td :class "px-3 py-2 text-stone-600" "Medium — aser has edge cases around OOB, fragments"))
|
|
(tr :class "border-b border-stone-100"
|
|
(td :class "px-3 py-2 text-stone-700" "4")
|
|
(td :class "px-3 py-2 text-stone-700" "Switch page rendering to async_sx_ref.py. Keep async_eval.py as fallback.")
|
|
(td :class "px-3 py-2 text-stone-600" "Medium — production path changes"))
|
|
(tr
|
|
(td :class "px-3 py-2 text-stone-700" "5")
|
|
(td :class "px-3 py-2 text-stone-700" "Delete evaluator.py and async_eval.py.")
|
|
(td :class "px-3 py-2 text-stone-600" "Low — once shadow confirms parity"))))))
|
|
|
|
;; -----------------------------------------------------------------------
|
|
;; Principles
|
|
;; -----------------------------------------------------------------------
|
|
|
|
(~docs/section :title "Principles" :id "principles"
|
|
(ul :class "list-disc list-inside space-y-2"
|
|
(li (strong "The spec is the single source of truth.") " All SX evaluation semantics live in .sx files. Host code implements platform primitives, not evaluation rules.")
|
|
(li (strong "Async is a host concern, not a language concern.") " The spec is synchronous. The Python bootstrapper emits async wrappers. The JS bootstrapper emits sync code. The spec doesn't know or care.")
|
|
(li (strong "Shadow-compare before switching.") " Every migration step runs both paths in parallel and asserts identical output. No big-bang cutover.")
|
|
(li (strong "Aser is just another render mode.") " It belongs in render.sx alongside render-to-html and render-to-dom. It's not special — it's the 'evaluate some, serialize the rest' mode.")))
|
|
|
|
(~docs/section :title "Outcome" :id "outcome"
|
|
(p "After convergence:")
|
|
(ul :class "list-disc list-inside space-y-2 mt-2"
|
|
(li "One evaluator implementation (the spec), bootstrapped to every host")
|
|
(li "Semantic changes made once in .sx, automatically propagated")
|
|
(li "~2,000 lines of hand-written Python evaluator code deleted")
|
|
(li "The lenient-arity class of bug becomes impossible — it's fixed in the spec, done everywhere")))))
|