plans: layered-stack framing + chisel sequence + loop scaffolding

Design + ops scaffolding for the next phase of work, none of it touching
substrate or guest code.

lib-guest.md: rewrites Architectural framing as a 5-layer stack
  (substrate → lib/guest → languages → shared/ → applications),
  recursive dependency-direction rule, scaled two-consumer rule. Adds
  Phase B (long-running stratification) with sub-layer matrix
  (core/typed/relational/effects/layout/lazy/oo), language profiles, and
  the long-running-discipline section. Preserves existing Phase A
  progress log and rules.

ocaml-on-sx.md: scope reduced to substrate validation + HM + reference
  oracle. Phases 1-5 + minimal stdlib slice + vendored testsuite slice.
  Dream carved out into dream-on-sx.md; Phase 8 (ReasonML) deferred.
  Records lib-guest sequencing dependency.

datalog-on-sx.md: adds Phase 4 built-in predicates + body arithmetic,
  Phase 6 magic sets, safety analysis in Phase 3, Non-goals section.

New chisel plans (forward-looking, not yet launchable):
  kernel-on-sx.md       — first-class everything, env-as-value endgame
  idris-on-sx.md        — dependent types, evidence chisel
  probabilistic-on-sx.md — weighted nondeterminism + traces
  maude-on-sx.md        — rewriting as primitive
  linear-on-sx.md       — resource model, artdag-relevant

Loop briefings (4 active, 1 cold):
  minikanren-loop.md, ocaml-loop.md, datalog-loop.md, elm-loop.md, koka-loop.md

Restore scripts mirror the loop pattern:
  restore-{minikanren,ocaml,datalog,jit-perf,lib-guest}.sh
  Each captures worktree state, plan progress, MCP health, tmux status.
  Includes the .mcp.json absolute-path patch instruction (fresh worktrees
  have no _build/, so the relative mcp_tree path fails on first launch).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-08 22:27:50 +00:00
parent e8246340fc
commit 9dd9fb9c37
19 changed files with 2283 additions and 183 deletions

View File

@@ -1,7 +1,9 @@
# lib/guest — shared toolkit for SX-hosted languages
# lib/guest — the metatheory layer for SX-hosted languages
Extract the duplicated plumbing across `lib/{haskell,common-lisp,erlang,prolog,js,lua,smalltalk,tcl,forth,ruby,apl,hyperscript}` into a small, composable kit so language N+1 costs ~200 lines instead of ~2000, without regressing any existing conformance scoreboard.
**This is a long-running, accreting plan.** Phase 03 (below) is the bootstrapping extraction — pulling the most obvious shared plumbing out of existing guests. Phase B onwards (see Stratification) is the ongoing accretion: codifying the universal patterns rose-ash's languages share, stratified by audience, refined continuously by pairs of new language consumers. The plan does not have a "done" state. The closest equivalent is "no two languages currently disagree about an abstraction in lib/guest" — and that's a moving target as new languages come online.
Branch: `architecture`. SX files via `sx-tree` MCP only. Never edit generated files.
## Thesis
@@ -10,7 +12,64 @@ The substrate (CEK, hygienic macros, records, delimited continuations, IO suspen
**Canaries:** Lua (small, conventional expression-grammar — exercises lex/Pratt/AST) and Prolog (paradigm-different — exercises pattern-match/unification). The two-canary rule prevents Lua-shaped abstractions.
**Two-language rule:** no extraction is merged until **two** guests consume it.
**Two-language rule:** no extraction is merged until **two** guests consume it. The rule scales with the universality claim — see *Stratification* for layer-appropriate pairs.
## Architectural framing — the layered stack
Rose-ash stratifies into five layers, each with a different invariant, audience, and time horizon. The same operating principles (dependency direction, two-consumer rule, layered editorial bar) work at every layer.
| Layer | rose-ash location | Time horizon | Audience |
|-------|-------------------|--------------|----------|
| **Substrate** (SX) | `spec/`, `hosts/` | years | platform maintainers |
| **lib/guest** (language metatheory) | `lib/guest/` | years, slower than substrate | guest-language authors |
| **Languages** | `lib/<lang>/` | monthsyears | application authors |
| **shared/** (application metatheory) | `shared/` | months | service authors |
| **Applications** | `blog/`, `market/`, `cart/`, `events/`, `federation/`, `account/`, `orders/`, `artdag/` | weeksmonths | village members |
What each layer *is*:
- **Substrate (SX)** — values, evaluation, continuations, effects, hygienic macros, reactivity. The *physics* of the platform. Bugs here are catastrophic for everyone.
- **lib/guest** — patterns that recur across paradigms: pattern matching, lexical primitives, precedence parsing, type inference, layout algorithms, effect handler protocols, module dispatch. *Applied PL theory.* Bugs only affect adopters; non-consumers don't care.
- **Languages** — specific syntactic and semantic commitments that *are* a particular language. The user-facing surface for code authoring.
- **shared/ (application metatheory)** — patterns that recur across domains: app factory, OAuth flow, ActivityPub, internal HMAC channel, fragments, sessions. Mature counterpart of what lib/guest is becoming, just at the application layer. The two-consumer rule there is already passed (every service is a consumer).
- **Applications** — the village system itself. Federation, blog, market, events, etc. The proof point that justifies all the layers below.
This five-layer separation is unusually clean. Most platforms collapse two adjacent layers — JVM and BEAM are pure substrate (no shared metatheory layer), Lisp images and Smalltalk environments bundle substrate + metatheory, conventional web stacks merge "shared infrastructure" with "applications." Racket's `#lang` machinery is the closest analogue at the lib/guest boundary. Treating each layer as a deliberately separate stratum is a design choice, not a code-organisation accident.
### Dependency direction (strict, at every boundary)
Higher layers may use lower; lower layers must not know higher exists.
- Applications import from `shared/`. `shared/` doesn't know which application is using it.
- `shared/` and applications import from languages (via SX modules and the host runtime). Languages don't know what application calls them.
- Languages import from lib/guest. lib/guest doesn't know which language is consuming it.
- lib/guest uses SX primitives. SX doesn't import from lib/guest.
Same invariant that makes substrate/metatheory separation work in PL theory, applied recursively up the stack. Violations show up as cyclic imports or as suspiciously language-specific code in `lib/guest/`, suspiciously domain-specific code in `shared/`, etc.
### Two-consumer rule, recursive
The pair-validation discipline applies at every layer, with audience-appropriate pairs:
- An entry in `lib/guest/core/` needs two consumers from different paradigms (e.g. lua + prolog).
- An entry in `lib/guest/typed/` needs two typed consumers.
- A pattern in `shared/` needs two services using it (largely already enforced — auth/HMAC/AP are used everywhere).
- An application's reusable abstraction promotion to `shared/` should happen only after a second domain wants the same shape.
At every layer, "shared between two consumers we happen to have" is not enough — the pair must be appropriate to the universality being claimed.
### Editorial bar
An entry belongs at layer N only if it codifies a piece of universal-or-near-universal pattern *for that layer's audience*. Same bar at every level; just the meaning of "universal" changes — universal-across-paradigms for lib/guest, universal-across-services for shared/, universal-across-domains for application metatheory.
### Leverage versus concreteness
The two directions matter at every layer.
- **Leverage compounds downward.** A substrate fix benefits every layer above. A lib/guest fix benefits every consuming language. A `shared/` fix benefits every service. So the highest-leverage work is always the layer that *enables* the most above it.
- **Concreteness flows upward.** Applications are what the village actually uses; substrate is invisible to them. Each layer is judged by its appropriate audience: substrate by correctness and speed, lib/guest by paradigm-coverage, languages by ergonomic fit, `shared/` by service reuse, applications by real users on real use cases.
The pleasant property: once you internalise the operating discipline at one layer, you know how to operate at every other. Pair-driven extraction. Two-consumer rule scaled to the layer's universality. Higher-uses-lower invariant. Codify-don't-just-deduplicate. The lib/guest plan is a working example of these principles applied at the metatheory layer; the same playbook applies all the way up.
## Current baseline
@@ -34,6 +93,10 @@ The baseline only needs to be re-snapshotted when the substrate (`spec/**`, `hos
---
## Phase A — Bootstrapping extraction (Phases 03 below)
The following four phases (0/1/2/3) are the bootstrap — pulling the most obvious shared plumbing out of existing guests. Largely shipped; partial-status entries are deferred ports waiting for their natural consumer (datalog, minikanren, ocaml, etc.) to close them. Phase B (Stratification) is the long-running successor.
## Phase 0 — Baseline snapshot (one-shot)
### Step 0: Snapshot every guest's scoreboard
@@ -147,6 +210,82 @@ Extract from `haskell/infer.sx`. Algorithm W or J, generalisation, instantiation
---
## Phase B — Stratification (long-running)
lib/guest itself decomposes by audience. Phase B accepts that and codifies the decomposition. Sub-layers emerge as paradigms reveal which abstractions are real; nothing in this section is fully fleshed out — it's the editorial direction, not a concrete queue.
### Proposed sub-layer shape
| Sub-layer | Purpose | Pair-validation requirement |
|-----------|---------|------------------------------|
| `lib/guest/core/` | True universals: lex, pratt, ast, match, prefix-rename, conformance harness | Two consumers from *different paradigms* (e.g. lua + prolog) |
| `lib/guest/typed/` | HM, generalisation, kind system, type-class-style dispatch | Two typed consumers (e.g. ocaml + haskell) |
| `lib/guest/relational/` | Unification beyond core match, occurs-check toggles, substitution composition, search strategies | Two relational consumers (e.g. minikanren + datalog) |
| `lib/guest/effects/` | Handler stacks, perform/resume protocols, dynamic-extent tracking | Two effect-typed consumers (e.g. koka + future) |
| `lib/guest/layout/` | Off-side rule, semicolon insertion, brace insertion | Two whitespace-sensitive consumers (e.g. haskell + elm or python-shape) |
| `lib/guest/lazy/` | Thunk wrapping, force/delay protocols, sharing semantics | Two lazy consumers (e.g. haskell + future lazy guest) |
| `lib/guest/oo/` | Message dispatch, method tables, super lookup | Two message-passing consumers (e.g. smalltalk + ruby) |
Future rows as paradigms emerge: constraint-domain solvers, gradual typing, capability-based effect systems, dependent types, etc. Layers should not be created speculatively — wait for two real consumers in the same paradigm before opening a sub-layer.
### Re-homing the Phase A entries
The Phase 03 entries currently at the lib/guest root are the candidates for re-homing under sub-layers as the stratification settles. Initial mapping (subject to refinement):
- `conformance.sx`, `prefix.sx` → stay at root (true infrastructure, not paradigm-specific)
- `lex.sx`, `pratt.sx`, `ast.sx`, `match.sx``core/`
- `layout.sx``layout/`
- `hm.sx``typed/` (currently the most overclaiming entry at root — has no plausible non-typed consumer)
### Two-consumer rule, scaled
The flat "two guests must consume it" rule scales with the layer's universality claim:
- A `core/` extraction must be cross-paradigm-validated (lua + prolog, not lua + tcl which are both dynamic-imperative).
- A `typed/` extraction needs two typed consumers; that's a tighter audience but still real.
- A `relational/` extraction needs two relational consumers, etc.
- Each layer's bar is *exactly* the universality it claims, no more, no less. An abstraction that claims universality (root level) but only has typed consumers belongs in `typed/`, not at root.
### Language profiles
Each language ends up consuming a *profile* of which sub-layers it uses. Profiles are aspirational until each language ports — but the matrix tells you which sub-layers to invest in based on consumer demand, and serves as a quick design document for new languages ("which existing profile does it most resemble?").
| Language | core | typed | relational | effects | layout | lazy | oo |
|----------|:----:|:-----:|:----------:|:-------:|:------:|:----:|:--:|
| ocaml | ✓ | ✓ | | | | | |
| haskell | ✓ | ✓ | | | ✓ | ✓ | |
| elm | ✓ | ✓ | | | ✓ | | |
| reasonml | ✓ | ✓ | | | | | |
| minikanren | ✓ | | ✓ | | | | |
| datalog | ✓ | | ✓ | | | | |
| prolog | ✓ | | ✓ | | | | |
| koka | ✓ | ✓ | | ✓ | | | |
| erlang | ✓ | | | (msg) | | | |
| elixir | ✓ | | | (msg) | | | |
| smalltalk | ✓ | | | | | | ✓ |
| ruby | ✓ | | | | | | ✓ |
| common-lisp | ✓ | | | | | | (CLOS) |
| lua / tcl / forth / apl | ✓ | | | | | | |
| js | ✓ | | | (async) | | | |
`(msg)`, `(async)`, `(CLOS)` denote shapes that *might* live in `effects/` or `oo/` once the paradigm gets a second consumer to validate against.
## Long-running discipline
This plan does not have a "done" state. The operating mode is *continuous pair-driven refactoring*:
- When a new guest reaches the same shape as an existing one → look for shared abstraction → consider extraction.
- When two existing consumers diverge on how they use a kit → consider a sub-layer split or a redesign.
- When a sub-layer accumulates more than ~5 entries → consider further stratification.
- When a kit has *never* been refactored after a second consumer ported → suspicious; the second port probably reshaped expectations and the kit should have flexed. Audit it.
- When a Phase A entry (currently at root) gets a second consumer in a narrower paradigm than "universal" → re-home into the appropriate sub-layer, don't wait for a third.
**Substrate work and lib/guest work feed each other.** Substrate fixes (sx-improvements queue) raise lib/guest's ceiling — every kit gets faster and more correct. lib/guest exposes substrate gaps that wouldn't show up in single-guest work — when two paradigms can't share an abstraction cleanly, the substrate may be missing a primitive. Treat lib/guest issues as substrate-investigation prompts before papering them over with kit-side workarounds.
**Extraction is not the goal — codification is.** "I refactored 800 lines of duplication into 200 lines of shared kit" is the bootstrapping mode. The long-running mode is "I codified a piece of language theory in working SX form, validated by N paradigms." The same line-count delta means very different things in those two modes. Keep the bar at codification, not just deduplication.
---
## Progress log
| Step | Status | Commit | Delta |
@@ -169,7 +308,7 @@ Extract from `haskell/infer.sx`. Algorithm W or J, generalisation, instantiation
- **Scope:** ONLY `lib/guest/**`, `lib/{lua,prolog,haskell,common-lisp,tcl}/**` (canaries + extraction targets), `plans/lib-guest.md`, `plans/agent-briefings/lib-guest-loop.md`. No `spec/`, `hosts/`, `web/`, `shared/`.
- **SX files:** `sx-tree` MCP tools only. `sx_validate` after every edit.
- **No raw dune.** Use `sx_build target="ocaml"` MCP tool.
- **Two-language rule:** never merge an extraction until two guests consume it (Step 8 excepted with explicit note).
- **Two-language rule (scaled by claim):** never merge an extraction until two guests consume it. The pair must be appropriate to the layer's universality claim — `core/` needs cross-paradigm pair, `typed/` needs two typed consumers, `relational/` needs two relational consumers, etc. (See *Phase B — Stratification* for the matrix.) Step 8 (Phase A) excepted with explicit OCaml-paired note.
- **Conformance baseline is the bar.** Any port whose scoreboard regresses by ≥1 test → revert, mark blocked, move on.
- **Substrate change → re-snapshot.** If `spec/` or `hosts/` changes underneath this loop, re-run Step 0 before continuing.
- **One step per code commit.** Plan updates as a separate commit. Short message with delta.