;; Foundations — The Computational Floor ;; From scoped effects to CEK: what's beneath algebraic effects and why it's the limit. (defcomp ~plans/foundations/plan-foundations-content () (~docs/page :title "Foundations \u2014 The Computational Floor" (p :class "text-stone-500 text-sm italic mb-8" "Scoped effects unify everything SX does. But they are not the bottom. " "Beneath them is a hierarchy of increasingly fundamental primitives, " "terminating at three irreducible things: an expression, an environment, and a continuation. " "This document traces that hierarchy, proves where the floor is, " "and maps the path from where SX stands to the deepest layer.") ;; ----------------------------------------------------------------------- ;; The hierarchy ;; ----------------------------------------------------------------------- (h2 :class "text-xl font-bold mt-12 mb-4" "The Hierarchy") (p "Every layer is definable in terms of the one below. " "No layer can be decomposed without the layer beneath it.") (~docs/code :code (str "Layer 0: CEK machine (expression + environment + continuation) \u2714 DONE\n" "Layer 1: Continuations (shift / reset \u2014 delimited capture) \u2714 DONE\n" "Layer 2: Algebraic effects (operations + handlers) \u2714 DONE\n" "Layer 3: Scoped effects (+ region delimitation) \u2714 DONE\n" "Layer 4: SX patterns (spread, provide, island, lake, signal) \u2714 DONE")) (p "All five layers are implemented. The entire hierarchy from patterns down to raw CEK " "is specced in " (code ".sx") " files and bootstrapped to Python and JavaScript. " "No hand-written evaluation logic remains.") ;; ----------------------------------------------------------------------- ;; What we built (status) ;; ----------------------------------------------------------------------- (h2 :class "text-xl font-bold mt-12 mb-4" "What We Built") (div :class "overflow-x-auto mb-6" (table :class "min-w-full text-sm" (thead (tr (th :class "text-left pr-4 pb-2 font-semibold" "Layer") (th :class "text-left pr-4 pb-2 font-semibold" "Spec files") (th :class "text-left pr-4 pb-2 font-semibold" "What it provides") (th :class "text-left pb-2 font-semibold" "Tests"))) (tbody (tr (td :class "pr-4 py-1" "0 \u2014 CEK") (td :class "pr-4 font-mono text-xs" "cek.sx, frames.sx") (td :class "pr-4" "Explicit step function, 20+ frame types, cek-call dispatch, CEK-native HO forms") (td "43 CEK + 26 reactive")) (tr (td :class "pr-4 py-1" "1 \u2014 Continuations") (td :class "pr-4 font-mono text-xs" "continuations.sx, callcc.sx") (td :class "pr-4" "shift/reset (delimited), call/cc (full), ReactiveResetFrame + DerefFrame") (td "Continuation tests")) (tr (td :class "pr-4 py-1" "2 \u2014 Effect signatures") (td :class "pr-4 font-mono text-xs" "boundary.sx, eval.sx") (td :class "pr-4" ":effects annotations on define, boundary enforcement at startup") (td "Boundary validation")) (tr (td :class "pr-4 py-1" "3 \u2014 Scoped effects") (td :class "pr-4 font-mono text-xs" "eval.sx, adapters") (td :class "pr-4" "scope/provide/context/emit!/emitted, scope-push!/scope-pop!") (td "Scope integration")) (tr (td :class "pr-4 py-1" "4 \u2014 Patterns") (td :class "pr-4 font-mono text-xs" "signals.sx, adapter-dom.sx, engine.sx") (td :class "pr-4" "signal/deref/computed/effect/batch, island/lake, spread/collect") (td "20 signal + 26 CEK reactive"))))) ;; ----------------------------------------------------------------------- ;; Layer 0: The CEK machine ;; ----------------------------------------------------------------------- (h2 :class "text-xl font-bold mt-12 mb-4" "Layer 0: The CEK Machine") (p "The CEK machine (Felleisen & Friedman, 1986) is a small-step evaluator with three registers:") (div :class "overflow-x-auto mb-6" (table :class "min-w-full text-sm" (thead (tr (th :class "text-left pr-4 pb-2 font-semibold" "Register") (th :class "text-left pr-4 pb-2 font-semibold" "Name") (th :class "text-left pb-2 font-semibold" "Meaning"))) (tbody (tr (td :class "pr-4 py-1 font-mono" "C") (td :class "pr-4" "Control") (td "The expression being evaluated")) (tr (td :class "pr-4 py-1 font-mono" "E") (td :class "pr-4" "Environment") (td "The bindings in scope \u2014 names to values")) (tr (td :class "pr-4 py-1 font-mono" "K") (td :class "pr-4" "Kontinuation") (td "What to do with the result"))))) (p "Every step of evaluation is a transition: take the current (C, E, K), " "produce a new (C\u2032, E\u2032, K\u2032). That's it. That's all computation is.") (p "Can you remove any register?") (ul :class "list-disc pl-6 mb-4 space-y-1" (li (strong "Remove C") " \u2014 nothing to evaluate. No computation.") (li (strong "Remove E") " \u2014 names are meaningless. Lose abstraction. " "Every value must be literal, every function must be closed. No variables, no closures, no reuse.") (li (strong "Remove K") " \u2014 no sequencing. The result of an expression goes nowhere. " "Lose composition \u2014 you can evaluate one thing but never connect it to the next.")) (p "Three things, all necessary, none decomposable further.") (h3 :class "text-lg font-semibold mt-8 mb-3" "CEK in SX") (p "The CEK machine is the default evaluator on both client (JS) and server (Python). " "Every " (code "eval-expr") " call goes through " (code "cek-run") ". " "The spec lives in two files:") (ul :class "list-disc pl-6 mb-4 space-y-1" (li (code "frames.sx") " \u2014 20+ frame types (IfFrame, ArgFrame, MapFrame, ReactiveResetFrame, ...)") (li (code "cek.sx") " \u2014 step function, run loop, special form handlers, HO form handlers, cek-call")) (p (code "cek-call") " is the universal function dispatch. " "It replaces the old " (code "invoke") " shim \u2014 SX lambdas go through " (code "cek-run") ", " "native callables through " (code "apply") ". One calling convention, bootstrapped identically to every host.") (h3 :class "text-lg font-semibold mt-8 mb-3" "CEK-native higher-order forms") (p "All higher-order forms step element-by-element through the CEK machine:") (~docs/code :code (str ";; map pushes a MapFrame, calls f on each element via continue-with-call\n" "(map (fn (x) (* 2 (deref counter))) items)\n" "\n" ";; deref inside the callback goes through CEK's DerefFrame\n" ";; \u2192 reactive-shift-deref fires if inside a reactive-reset boundary\n" ";; \u2192 the continuation from deref to the MapFrame is captured as a subscriber")) (p "This means " (code "deref") " works inside " (code "map") ", " (code "filter") ", " (code "reduce") ", " (code "for-each") ", " (code "some") ", " (code "every?") " callbacks. " "The HO forms don't escape to tree-walk \u2014 the CEK machine processes every step.") ;; ----------------------------------------------------------------------- ;; Layer 1: Delimited continuations ;; ----------------------------------------------------------------------- (h2 :class "text-xl font-bold mt-12 mb-4" "Layer 1: Delimited Continuations") (p "Delimited continuations (Felleisen 1988, Danvy & Filinski 1990) " "expose the K register as a first-class value:") (~docs/code :code (str ";; reset marks a point in the continuation\n" "(reset\n" " (+ 1 (shift k ;; k = \"the rest up to reset\"\n" " (k (k 10)))) ;; invoke it twice: 1 + (1 + 10) = 12\n" ")")) (p (code "reset") " says: \"here's a boundary.\" " (code "shift") " says: \"give me everything between here and that boundary as a callable function.\" " "The captured continuation " (em "is") " a slice of the K register.") (h3 :class "text-lg font-semibold mt-8 mb-3" "Deref as shift") (p "The reactive payoff. " (code "deref") " inside a " (code "reactive-reset") " boundary " "is shift/reset applied to signals:") (~docs/code :code (str ";; User writes:\n" "(div :class (str \"count-\" (deref counter))\n" " (str \"Value: \" (deref counter)))\n" "\n" ";; CEK sees (deref counter) \u2192 signal? \u2192 reactive-reset on stack?\n" ";; Yes: capture (str \"count-\" [HOLE]) as continuation\n" ";; Register as subscriber. Return current value.\n" ";; When counter changes: re-invoke continuation \u2192 update DOM.")) (p "No explicit " (code "effect()") " wrapping needed. " "The continuation capture IS the subscription mechanism.") (h3 :class "text-lg font-semibold mt-8 mb-3" "The Filinski embedding") (p "Filinski (1994) proved that " (code "shift/reset") " can encode " (em "any") " monadic effect. State, exceptions, nondeterminism, I/O, " "continuations themselves \u2014 all expressible as shift/reset patterns. " "This means layer 1 is already computationally complete for effects. " "Everything above is structure, not power.") ;; ----------------------------------------------------------------------- ;; The floor proof ;; ----------------------------------------------------------------------- (h2 :class "text-xl font-bold mt-12 mb-4" "Why Layer 0 Is the Floor") (p "Two independent arguments:") (h3 :class "text-lg font-semibold mt-8 mb-3" "1. Church\u2013Turing") (p "The Church\u2013Turing thesis: any effectively computable function can be computed by " "a Turing machine (equivalently, by the lambda calculus, equivalently, by the CEK machine). " "This has withstood 90 years of attempts to disprove it. " "No formal system has ever been shown to compute more.") (p "This means CEK captures " (em "all") " computation. Adding more primitives doesn't let you " "compute anything new. It only changes what's " (em "convenient") " to express.") (h3 :class "text-lg font-semibold mt-8 mb-3" "2. Irreducibility of C, E, K") (p "The three registers are independently necessary:") (ul :class "list-disc pl-6 mb-4 space-y-1" (li (strong "C without E") " = combinatory logic. Turing-complete but inhumane. " "No named bindings \u2014 everything via S, K, I combinators.") (li (strong "C without K") " = single expression evaluation. " "You can compute one thing but can't compose it with anything.") (li (strong "E without C") " = a phone book with no one to call.") (li (strong "K without C") " = a to-do list with nothing on it.")) (p "You can " (em "encode") " any register into another (CPS eliminates K, " "De Bruijn indices eliminate E), but encoding isn't elimination. " "The information is still there, just hidden in a different representation.") ;; ----------------------------------------------------------------------- ;; Digging deeper: what's next ;; ----------------------------------------------------------------------- (h2 :class "text-xl font-bold mt-12 mb-4" "Progress") (div :class "overflow-x-auto mb-6" (table :class "min-w-full text-sm" (thead (tr (th :class "text-left pr-4 pb-2 font-semibold" "Step") (th :class "text-left pr-4 pb-2 font-semibold" "What") (th :class "text-left pb-2 font-semibold" "Status"))) (tbody (tr (td :class "pr-4 py-1" "1") (td :class "pr-4" "Serializable CEK state") (td :class "text-emerald-600 font-semibold" "\u2714 Done \u2014 freeze-scope, freeze-signal, freeze-to-sx, thaw-from-sx")) (tr (td :class "pr-4 py-1" "2") (td :class "pr-4" "CEK stepping debugger") (td :class "text-emerald-600 font-semibold" "\u2714 Done \u2014 live island with reactive code view, DOM preview")) (tr (td :class "pr-4 py-1" "3") (td :class "pr-4" "Content-addressed computation") (td :class "text-emerald-600 font-semibold" "\u2714 Done \u2014 content-hash, freeze-to-cid, thaw-from-cid")) (tr (td :class "pr-4 py-1" "4") (td :class "pr-4" "Concurrent CEK") (td :class "text-amber-600 font-semibold" "Spec complete \u2014 implementation next")) (tr (td :class "pr-4 py-1" "5") (td :class "pr-4" "Linear effects") (td :class "text-stone-400" "Future"))))) ;; ----------------------------------------------------------------------- ;; Step 4: Concurrent CEK \u2014 deep spec ;; ----------------------------------------------------------------------- (h2 :class "text-xl font-bold mt-12 mb-4" "Step 4: Concurrent CEK") (p "Multiple CEK machines running concurrently. The machines are independent \u2014 " "each has its own C, E, K registers. Communication is via typed channels. " "The host provides the concurrency mechanism (Web Workers, forkIO, tokio::spawn). " "The spec defines the coordination primitives.") (h3 :class "text-lg font-semibold mt-8 mb-3" "4.1 Spawn") (p "The fundamental primitive. Create a new CEK machine from a thunk, " "run it concurrently, return a signal that resolves with the result:") (~docs/code :code (str "(let ((result (spawn (fn () (* 6 7))))) " " ;; result is a signal \u2014 nil until the computation completes " " (effect (fn () " " (when (deref result) " " (log (str \"answer: \" (deref result)))))) " " ;; => \"answer: 42\" (eventually)")) (p "Implementation: the host freezes the thunk\u2019s CEK state, ships it to a worker, " "the worker thaws and runs, posts back the result, the signal resolves. " "The reactive system propagates the change \u2014 any island deref\u2019ing the signal re-renders.") (p "The default implementation is synchronous (no actual concurrency). " "The host overrides " (code "spawn-impl") " for real concurrency:") (~docs/code :code (str ";; Host-provided spawn (Web Worker) " "(set! spawn-impl " " (fn (thunk result-sig) " " (let ((frozen (freeze-to-sx \"spawn\"))) " " (worker-post frozen) " " (worker-on-message " " (fn (val) (reset! result-sig val))))))")) (h3 :class "text-lg font-semibold mt-8 mb-3" "4.2 Channels") (p "Typed communication pipes between concurrent computations. " "Send is non-blocking. Receive suspends via " (code "shift") " until data arrives:") (~docs/code :code (str ";; Create a channel " "(make-channel \"results\") " " " ";; Producer (in spawned computation) " "(spawn (fn () " " (channel-send \"results\" (expensive-computation)))) " " " ";; Consumer (in main thread) " "(let ((val (channel-receive \"results\"))) " " ;; val is a signal if no data yet, or the value if buffered " " (when (signal? val) " " ;; Reactive: re-renders when data arrives " " (div \"Waiting...\" (deref val))))")) (p "Channels are named and registered globally. Multiple producers and consumers " "can share a channel. The buffer stores values until a consumer reads them. " "When a consumer arrives and the buffer is empty, a signal is returned " "that resolves when a producer sends.") (p "Channel semantics:") (ul :class "list-disc pl-6 mb-4 space-y-1" (li (strong "Unbuffered") " \u2014 send blocks until a receiver is ready (rendezvous)") (li (strong "Buffered") " \u2014 send succeeds immediately, values queue (current default)") (li (strong "Broadcast") " \u2014 every receiver gets every value (pub/sub)") (li (strong "Select") " \u2014 wait on multiple channels, take the first that has data")) (h3 :class "text-lg font-semibold mt-8 mb-3" "4.3 Fork / Join") (p "Spawn multiple computations, collect all results. " "The spec primitive, not a library pattern:") (~docs/code :code (str ";; Fork: run N computations concurrently " "(let ((results (fork-join (list " " (fn () (fetch-user 1)) " " (fn () (fetch-user 2)) " " (fn () (fetch-user 3)))))) " " ;; results is a list of signals " " ;; All resolve when all computations complete " " (div " " (map (fn (r) (div (deref r))) results)))")) (p "Fork/join composes with the reactive system. Each result signal drives " "a reactive DOM node. Results appear as they complete, not all-or-nothing.") (h3 :class "text-lg font-semibold mt-8 mb-3" "4.4 Scheduler") (p "The scheduler decides which CEK machine steps next. " "Different strategies for different use cases:") (ul :class "list-disc pl-6 mb-4 space-y-1" (li (strong "Round-robin") " \u2014 fair, each machine gets equal time") (li (strong "Priority") " \u2014 UI-facing computations step first") (li (strong "Work-stealing") " \u2014 idle workers steal from busy ones") (li (strong "Cooperative") " \u2014 machines yield explicitly via " (code "yield!")) (li (strong "DAG-ordered") " \u2014 step machines whose inputs are ready (Art DAG)")) (~docs/code :code (str ";; Cooperative scheduling " "(spawn (fn () " " (for-each (fn (item) " " (process item) " " (yield!) ;; let other machines run " " ) items))) " " " ";; DAG scheduling " "(dag-execute " " {:nodes {:a (fn () (load-image \"input.jpg\")) " " :b (fn (a) (resize a 512 512)) " " :c (fn (a) (blur a 3)) " " :d (fn (b c) (composite b c))} " " :edges {[:a] [:b :c] " " [:b :c] [:d]}})")) (h3 :class "text-lg font-semibold mt-8 mb-3" "4.5 Content-addressed concurrency") (p "Concurrent computation + content addressing = verifiable parallel execution:") (ul :class "list-disc pl-6 mb-4 space-y-1" (li "Each spawned computation\u2019s initial state has a CID") (li "Each result has a CID") (li "The mapping input-CID \u2192 output-CID is a proof of computation") (li "Memoize: if input-CID was computed before, return cached output-CID") (li "Distribute: send input-CID to whichever worker has the data") (li "Verify: any machine can re-run from input-CID and check output-CID matches")) (~docs/code :code (str ";; Content-addressed spawn " "(let ((input-cid (freeze-to-cid \"job\"))) " " ;; Check cache first " " (or (result-cache-get input-cid) " " ;; Not cached \u2014 compute and cache " " (let ((result (spawn (fn () (expensive-work))))) " " (effect (fn () " " (when (deref result) " " (result-cache-put input-cid " " (freeze-to-cid \"result\"))))) " " result)))")) (h3 :class "text-lg font-semibold mt-8 mb-3" "4.6 Host mapping") (p "Each host implements concurrency differently. The spec is the same:") (div :class "overflow-x-auto mb-6" (table :class "min-w-full text-sm" (thead (tr (th :class "text-left pr-4 pb-2 font-semibold" "Primitive") (th :class "text-left pr-4 pb-2 font-semibold" "JavaScript") (th :class "text-left pr-4 pb-2 font-semibold" "Python") (th :class "text-left pr-4 pb-2 font-semibold" "Haskell") (th :class "text-left pb-2 font-semibold" "Rust/WASM"))) (tbody (tr (td :class "pr-4 py-1 font-mono" "spawn") (td :class "pr-4" "Web Worker") (td :class "pr-4" "asyncio.create_task") (td :class "pr-4" "forkIO") (td "tokio::spawn")) (tr (td :class "pr-4 py-1 font-mono" "channel") (td :class "pr-4" "MessageChannel") (td :class "pr-4" "asyncio.Queue") (td :class "pr-4" "TChan (STM)") (td "mpsc::channel")) (tr (td :class "pr-4 py-1 font-mono" "yield!") (td :class "pr-4" "setTimeout(0)") (td :class "pr-4" "await asyncio.sleep(0)") (td :class "pr-4" "threadDelay 0") (td "tokio::task::yield_now")) (tr (td :class "pr-4 py-1 font-mono" "freeze/thaw") (td :class "pr-4" "postMessage + JSON") (td :class "pr-4" "pickle / SX text") (td :class "pr-4" "Serialise + SX text") (td "serde + SX text")) (tr (td :class "pr-4 py-1 font-mono" "select") (td :class "pr-4" "Promise.race") (td :class "pr-4" "asyncio.wait FIRST_COMPLETED") (td :class "pr-4" "STM orElse") (td "tokio::select!"))))) (h3 :class "text-lg font-semibold mt-8 mb-3" "4.7 Roadmap") (div :class "overflow-x-auto mb-6" (table :class "min-w-full text-sm" (thead (tr (th :class "text-left pr-4 pb-2 font-semibold" "Phase") (th :class "text-left pr-4 pb-2 font-semibold" "What") (th :class "text-left pr-4 pb-2 font-semibold" "Host") (th :class "text-left pb-2 font-semibold" "Validates"))) (tbody (tr (td :class "pr-4 py-1" "4a") (td :class "pr-4" "spawn + Web Worker") (td :class "pr-4" "JavaScript") (td "Freeze/thaw across execution contexts")) (tr (td :class "pr-4 py-1" "4b") (td :class "pr-4" "Channels (buffered)") (td :class "pr-4" "JavaScript") (td "Cross-worker communication")) (tr (td :class "pr-4 py-1" "4c") (td :class "pr-4" "Fork/join + DAG scheduler") (td :class "pr-4" "JavaScript") (td "Art DAG integration path")) (tr (td :class "pr-4 py-1" "4d") (td :class "pr-4" "Haskell bootstrapper") (td :class "pr-4" "Haskell") (td "Spec portability, native concurrency")) (tr (td :class "pr-4 py-1" "4e") (td :class "pr-4" "Rust/WASM bootstrapper") (td :class "pr-4" "Rust") (td "Browser performance, WASM target")) (tr (td :class "pr-4 py-1" "4f") (td :class "pr-4" "Content-addressed spawn") (td :class "pr-4" "Any") (td "Memoization, distribution, verification")) (tr (td :class "pr-4 py-1" "4g") (td :class "pr-4" "IPFS-backed content store") (td :class "pr-4" "Any") (td "Global content addressing, peer-to-peer")) (tr (td :class "pr-4 py-1" "4h") (td :class "pr-4" "Select + broadcast channels") (td :class "pr-4" "Any") (td "Complex coordination patterns")) (tr (td :class "pr-4 py-1" "4i") (td :class "pr-4" "Linear channels") (td :class "pr-4" "Any") (td "Resource safety, exactly-once delivery"))))) (p "Each phase is independently valuable. Phase 4a (Web Worker spawn) is the immediate next build. " "Phase 4d (Haskell) proves the spec is truly host-independent. " "Phase 4e (Rust/WASM) proves it can be fast. " "Phase 4g (IPFS) makes it distributed.") ;; ----------------------------------------------------------------------- ;; Step 5: Linear effects (future) ;; ----------------------------------------------------------------------- (h2 :class "text-xl font-bold mt-12 mb-4" "Step 5: Linear Effects") (p "Constrain the continuation: a handler " (em "must") " resume exactly once. " "No dropping (resource leak), no duplicating (nondeterminism). " "This is the linearity axis from the three-axis model.") (ul :class "list-disc pl-6 mb-4 space-y-1" (li (strong "Affine channels") " \u2014 send at most once. Prevents duplicate messages.") (li (strong "Linear scopes") " \u2014 a freeze scope that must be thawed. Prevents orphaned state.") (li (strong "Session types") " \u2014 a protocol that must complete. Prevents half-open connections.") (li (strong "Resource handles") " \u2014 a file/DB handle that must be closed. Prevents leaks.")) (p "Linear effects connect SX to Rust\u2019s ownership model. " "Affine types (use at most once) are the practical sweet spot. " "Full linearity (use exactly once) is for protocols and resources.") (p "This is the furthest horizon. The infrastructure \u2014 freeze scopes, channels, " "content addressing \u2014 must be solid first. Linearity is a discipline on top, " "not a prerequisite beneath.") (p :class "text-stone-500 text-sm italic mt-12" "The floor is thought itself. We can't go deeper, because there's no one left to dig.")))