Tokens → list of {:head :body} / {:query} clauses. SX symbols for
constants and variables (case-distinguished). not(literal) in body
desugars to {:neg literal}. Nested compounds permitted in arg
position for arithmetic; safety analysis (Phase 3) will gate them.
Conformance harness wraps lib/guest/conformance.sh; produces
lib/datalog/scoreboard.{json,md}.
Ships the algebra for HM-style type inference, riding on
lib/guest/match.sx (terms + unify) and ast.sx (canonical AST):
• Type constructors: hm-tv, hm-arrow, hm-con, hm-int, hm-bool, hm-string
• Schemes: hm-scheme / hm-monotype + accessors
• Free type-vars: hm-ftv, hm-ftv-scheme, hm-ftv-env
• Substitution: hm-apply, hm-apply-scheme, hm-apply-env, hm-compose
• Generalize / Instantiate (with shared fresh-tv counter)
• hm-fresh-tv (counter is a (list N) the caller threads)
• hm-infer-literal (the only fully-closed inference rule)
24 self-tests in lib/guest/tests/hm.sx covering every function above.
The lambda / app / let inference rules — the substitution-threading
core of Algorithm W — intentionally live in HOST CODE rather than the
kit, because each host's AST shape and substitution-threading idiom
differ subtly enough that forcing one shared assembly here proved
brittle in practice (an earlier inline-assembled hm-infer faulted with
"Not callable: nil" only when defined in the kit, despite working when
inline-eval'd or in a separate file — a load/closure interaction not
worth chasing inside this step's budget). The host gets the algebra
plus a spec; assembly stays close to the AST it reasons over.
PARTIAL — algebra + literal rule shipped; full Algorithm W deferred
to host consumers (haskell/infer.sx, lib/ocaml/types.sx when
OCaml-on-SX Phase 5 lands per the brief's sequencing note). Haskell
infer.sx untouched; haskell scoreboard still 156/156 baseline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
apl-permutations was doing (append acc <new-perms>) which is
O(|acc|) and acc grows ~N! big — total cost O(N!²).
Swapped to (append <new-perms> acc) — append is O(|first|)
so cost is O((n+1)·N!_prev) per layer, total O(N!). q(7)
went from 32s to 12s; q(8)=92 now finishes well within the
300s timeout, so the queens(8) test is restored.
497/497. Phase 8 complete.
Configurable layout pass that inserts virtual open / close / separator
tokens based on indentation. Supports both styles the brief calls out:
• Haskell-flavour: layout opens AFTER a reserved keyword
(let/where/do/of) and resolves to the next token's column. Module
prelude wraps the whole input in an implicit block. Explicit `{`
after the keyword suppresses virtual layout.
• Python-flavour: layout opens via an :open-trailing-fn predicate
fired AFTER the trigger token (e.g. trailing `:`) — and resolves
to the column of the next token, which in real source is on a
fresh line. No module prelude.
Public entry: (layout-pass cfg tokens). Token shape: dict with at
least :type :value :line :col; everything else passes through. Newline
filler tokens are NOT used — line-break detection is via :line.
lib/guest/tests/layout.sx — 6 tests covering both flavours:
haskell-do-block / haskell-explicit-brace / haskell-do-inline /
haskell-module-prelude / python-if-block / python-nested.
Per the brief's gotcha note ("Don't ship lib/guest/layout.sx unless
the haskell scoreboard equals baseline") — haskell/layout.sx is left
UNTOUCHED. The kit isn't yet a drop-in replacement for the full
Haskell 98 algorithm (Note 5, multi-stage pre-pass, etc.) and forcing
a port would risk the 156 currently passing programs. Haskell
scoreboard remains at 156/156 baseline because no haskell file
changed. The synthetic Python-ish fixture is the second consumer per
the brief's wording.
PARTIAL — kit + synthetic fixture shipped; haskell port deferred until
the kit grows the missing Haskell-98 wrinkles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure-functional pattern-match + unification, shipped for miniKanren
(minikraken) / Datalog and any other logic-flavoured guest that wants
immutable unification without writing it from scratch.
Canonical wire format (config callbacks let other shapes plug in):
var (:var NAME)
constructor (:ctor HEAD ARGS)
literal number / string / boolean / nil
Public API:
empty-subst walk walk* extend occurs?
unify (symmetric, with occurs check)
unify-with (cfg-driven for non-canonical term shapes)
match-pat (asymmetric pattern→value, vars only in pattern)
match-pat-with (cfg-driven)
lib/guest/tests/match.sx — 25 tests covering walk chains, occurs,
unify (literal/var/ctor, head + arity mismatch, transitive vars),
match-pat. All passing.
The brief flagged this as the highest-risk step ("revert and redesign
on any regression"). The two existing engines — haskell/match.sx
(pure asymmetric, lazy, returns env-or-nil) and prolog runtime.sx
pl-unify! (mutating symmetric, trail-based, returns bool) — are
structurally divergent and forcing a shared core under either of their
contracts would risk the 746 tests they currently pass. Both are
untouched; they remain at baseline (haskell 156/156, prolog 590/590)
because none of their source files were modified.
PARTIAL — kit shipped, prolog/haskell ports deferred until a guest
chooses to migrate or until a third consumer (minikraken / datalog)
provides a less risky migration path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
programs-e2e.sx exercises the classic-algorithm shapes from
lib/apl/tests/programs/*.apl via the full pipeline (apl-run on
embedded source strings). Tests include factorial-via-∇,
triangular numbers, sum-of-squares, prime-mask building blocks
(divisor counts via outer mod), named-fn composition,
dyadic max-of-two, and a single Newton sqrt step.
The original one-liners (e.g. primes' inline ⍵←⍳⍵) need parser
features we haven't built (compress-as-fn, inline assign) — the
e2e tests use multi-statement equivalents. No file-reading
primitive in OCaml SX, so source is embedded.
Side-fix: ⌿ (first-axis reduce) and ⍀ (first-axis scan) were
silently skipped by the tokenizer — added to apl-glyph-set
and apl-parse-op-glyphs.
Defines the 10 canonical node kinds called out in the brief — literal,
var, app, lambda, let, letrec, if, match-clause, module, import — plus
predicates, ast-kind dispatch, and per-field accessors. Each node is a
tagged keyword-headed list: (:literal V), (:var N), (:app FN ARGS), …
Also lib/guest/tests/ast.sx — 33 tests exercising every constructor +
predicate + accessor, runnable via (gast-tests-run!) which returns the
{:passed :failed :total} dict the shared conformance driver expects.
PARTIAL — pending real consumers. The brief calls Step 5 "Optional —
guests may keep their own AST" and forcing lua/prolog to switch their
internal AST shape risks regressing 775 passing tests for tooling that
nothing yet calls. Both internal ASTs are untouched; lua still 185/185,
prolog still 590/590. Datalog-on-sx (in flight, see plans/datalog-on-sx.md)
will be the natural first real consumer; lua/prolog converters can land
when a cross-language tool wants them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Parser: apl-collect-fn-bindings pre-scans stmt-groups for
`name ← { ... }` patterns and populates apl-known-fn-names.
is-fn-tok? consults this list; collect-segments-loop emits
(:fn-name nm) for known names so they parse as functions.
Resolver: apl-resolve-{monadic,dyadic} handle :fn-name by
looking up env, asserting the binding is a dfn, returning
a closure that dispatches to apl-call-dfn{-m,}.
Recursion still works: `fact ← {0=⍵:1 ⋄ ⍵×∇⍵-1} ⋄ fact 5` → 120.
Three small unblockers in one iteration:
- tokenizer: read-digits! now consumes optional ".digits" suffix,
so 3.7 and ¯2.5 are single number tokens.
- tokenizer: ⎕ followed by ← emits a single :name "⎕←" token
(instead of splitting on the assign glyph). Parser registers
⎕← in apl-quad-fn-names; apl-monadic-fn maps to apl-quad-print.
- eval-ast: :str AST nodes evaluate to char arrays. Single-char
strings become rank-0 scalars; multi-char become rank-1 vectors
of single-char strings.
Extracted the data-half of Pratt-style precedence parsing: the operator
table format and lookup. The climbing loop stays per-language because
the two canaries use opposite conventions (lua: higher prec = tighter;
prolog: lower prec = tighter, with xfx/xfy/yfx assoc tags) — forcing
one shared loop adds callback indirection that obscures more than it
shares. The brief's literal ask is "Grammar is a dict, not hardcoded
cond" and that's what gets shared.
Entry shape: (NAME PREC ASSOC). Three accessors: pratt-op-name /
pratt-op-prec / pratt-op-assoc. One traversal: pratt-op-lookup.
Ported lua/parser.sx — replaced 18-clause cond and the
lua-binop-right? hardcoded `or` with a 15-entry lua-op-table, now
queried via pratt-op-lookup. Ported prolog/parser.sx — pl-op-find
(linear walk reimpl) deleted; pl-op-lookup wraps pratt-op-lookup;
pl-token-op simplified to return the entry directly.
Verification:
- lua/test.sh: 185/185 = baseline.
- prolog/conformance.sh: 590/590 = baseline (timestamp-only diff).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new SX primitives wrapping Unix socket APIs:
- socket-connect host port → "sockN" (TCP client)
- socket-server ?host? port → "sockN" listening socket (SO_REUSEADDR, backlog 8)
- socket-accept server-chan → {:channel :host :port}
Sockets reuse the channel_table from Phase 5, so existing channel-read/
write/close/select all work on them. Host arg supports localhost,
0.0.0.0, IPv4 literal, or gethostbyname lookup.
Tcl `socket` command:
- socket host port → TCP client
- socket -server cb port → listening socket; auto-registers a fileevent
on the server channel that fires `_sock-do-accept SRV CB` per readable
event. _sock-do-accept (internal) accepts the pending client and calls
the user's callback as `cb client-chan host port`.
puts channel detection now also recognizes "sockN" prefix (was only
"fileN") and dispatches to channel-write.
+4 idiom tests: socket-server-fires-callback, socket-client-server-
roundtrip, socket-server-peer-host, socket-multiple-connections.
358/358 green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
apl-throw raises a tagged ("apl-error" code msg) error.
apl-trap-matches? checks if codes list contains the error's code
(0 = catch-all, à la Dyalog).
Eval-stmt :trap clause wraps try-block with R7RS guard;
on match, runs catch-block; on mismatch, re-raises.
Bonus :throw AST node for testing.
test.sh + conformance.sh now load lib/r7rs.sx (for guard) and
include eval-ops + pipeline suites in scoreboard.
All Phase 7 unchecked items are now ticked.
Final scoreboard: 450/450 across 10 suites.
30 new source-string idioms via apl-run: triangulars, factorial,
running sum/product, parity counts, identity matrix, mult-table,
dot product, ∧.= equality, take/drop/reverse, tally, ravel,
count-of-value, etc.
Side-fix: tokenizer's apl-glyph-set was missing ≢ and ≡ — they
were silently skipped. Added them and to apl-parse-fn-glyphs.
New SX primitive io-select-channels(read-list write-list timeout-ms) wrapping
Unix.select on the registered channel table. Returns {:readable :writable}.
Tcl event loop implemented purely in Tcl (no sx_server.ml changes):
- fileevent $chan readable|writable script (or "" to unregister)
- fileevent $chan event (1 arg) returns the registered script
- after ms script — schedule one-shot timer
- after ms (no script) — sleep, driving event loop in the meantime
- vwait varname — block until var is set/changed, handlers run between polls
- update — non-blocking event drain (poll-timeout=0)
State on interp: :fileevents (list of (chan event script)) and :timers
(sorted list of (expiry-ms script)).
tcl-event-step is the inner loop: expire timers, build fd lists from
:fileevents, call io-select-channels with computed timeout, run ready
handlers. vwait polls every 1000ms or until var changes.
Scoped to script mode by design — vwait from inside a server-handled
command does not interact with sx_server's stdin scheduler.
+5 idiom tests: after-vwait-timer, after-multiple-timers-update,
fileevent-readable-fires, fileevent-query-script,
after-cancel-via-vwait-timing. 354/354 green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
apl-resolve-monadic and apl-resolve-dyadic dispatch :derived-fn,
:outer, and :derived-fn2 nodes to the matching operator helper.
:monad/:dyad in apl-eval-ast now route through these resolvers.
Removed queens(8) test (too slow under current 300s timeout).
11 new SX primitives in sx_primitives.ml wrapping Unix.openfile/read/write/
lseek/set_nonblock: channel-open/close/read/read-line/write/flush/seek/tell/
eof?/blocking?/set-blocking!.
Tcl runtime now uses real channel ops:
- open ?-mode? returns "fileN" handle (modes r/w/a/r+/w+/a+)
- close/read/gets/puts/seek/tell/eof/flush wired through
- new fconfigure command supports -blocking 0|1
- puts dispatches to channel-write when first arg starts with "file"
- gets command registration fixed (was pointing to old stub)
eof-returns-1 coro test updated to match real Tcl semantics (eof flips
only after a read hits EOF).
Test runner timeout bumped 180s→1200s (post-merge JIT is slow).
+7 idiom tests covering write+read, gets-loop, seek/tell, eof-after-read,
append mode, seek-to-end, fconfigure-blocking. 349/349 green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The bytecode compiler emitted OP_CALL_PRIM (52) for every primitive call, even
for arithmetic and comparison hot-paths. The VM had specialized opcodes
(OP_ADD, OP_SUB, OP_EQ, etc.) defined but unused.
- lib/compiler.sx (compile-call): emit specialized 1-byte opcode when the
primitive name + arity matches one of {+, -, *, /, =, <, >, cons, not, len,
first, rest}. Falls back to CALL_PRIM otherwise. fib bytecode: 50 → 38 bytes.
- hosts/ocaml/lib/sx_compiler.ml: mirror change in the auto-generated OCaml
compiler so SXBC export from mcp_tree uses the same emission.
- hosts/ocaml/lib/sx_vm.ml: extend OP_ADD/SUB/MUL/DIV to handle Integer+Integer
(not just Number+Number). Inline OP_EQ via Sx_runtime._fast_eq. Inline
OP_LT/GT mixed-numeric comparisons. Avoids Hashtbl lookup on the fallback
path for the common integer cases that dominate tight loops.
- hosts/ocaml/bin/bench_vm.ml: VM-only benchmark — loads compiler.sx via CEK,
JIT-compiles each fn, measures Sx_vm.call_closure throughput.
Median improvements (best of 3 runs of 9-min, bench_vm.exe):
fib(22) 107.87ms → 33.13ms -69%
loop(200000) 429.64ms → 161.16ms -62%
sum-to(50000) 72.85ms → 36.74ms -50%
count-lt(20000) 28.44ms → 17.58ms -38%
count-eq(20000) 37.23ms → 15.46ms -58%
Tests: 4550/4550 OCaml passing (unchanged). Zero regressions.
Last step in the sx-improvements roadmap — all 14 steps complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move `hs-prolog-hook` / `hs-set-prolog-hook!` / `prolog` out of
`lib/hyperscript/runtime.sx` into a self-contained plugin file at
`lib/hyperscript/plugins/prolog.sx`. The API surface is preserved —
`lib/prolog/hs-bridge.sx::pl-install-hs-hook!` still calls
`hs-set-prolog-hook!` exactly as before, just resolved to the plugin
file's binding rather than runtime.sx's.
Move the E39 worker stub registration out of `lib/hyperscript/parser.sx`
into `lib/hyperscript/plugins/worker.sx`. The plugin calls
`(hs-register-feature! "worker" ...)` at file load time. Behaviour is
identical — `worker MyWorker ...` raises the same helpful "plugin not
installed" error, just routed through the registry from a separate
file. The pre-existing `behavioral` test for the helpful error
("raises a helpful error when the worker plugin is not installed")
still passes via the new path.
Wire-up:
- OCaml `bin/run_tests.ml`: load `plugins/worker.sx` and
`plugins/prolog.sx` after `runtime.sx`, before `integration.sx`.
- JS `tests/hs-kernel-eval.js`: extend HS module list with
`hs-worker` / `hs-prolog`; add `HS_PLUGINS` resolver branch so the
`hs-` prefix maps to `lib/hyperscript/plugins/`.
- WASM `hosts/ocaml/browser/bundle.sh`: copy plugin files into
`dist/sx/hs-<name>.sx`.
- WASM `hosts/ocaml/browser/compile-modules.js`: add `hs-worker` /
`hs-prolog` to `FILES`, `HS_DEPS`, and `HS_LAZY` so the lazy loader
resolves them on first reference.
- Worker plugin carries a sentinel `(define hs-worker-loaded? true)`
so `extractDefines` indexes it in the module manifest (the lazy
loader skips files with no defines).
Mirrors `shared/static/wasm/sx/hs-{parser,runtime}.sx` are byte-identical
to source; new mirrors `hs-{prolog,worker}.sx` written via sx_write_file.
OCaml: 4545 passed, 1339 failed — matches baseline.
JS: 2591 passed, 2465 failed — matches baseline.
Smoke tests: `(prolog ...)` raises "prolog hook not installed" cleanly,
`(hs-set-prolog-hook! ...)` then `(prolog ...)` returns the hook result,
`(hs-compile "worker MyWorker def noop() end end")` raises the worker
stub error via the registry path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `_hs-command-registry` and `_hs-converter-registry` dicts plus
`hs-register-command!` / `hs-register-converter!` to
`lib/hyperscript/compiler.sx`. Inside `hs-to-sx`, before the existing
`cond` over head symbols, check both registries: an `as` form whose
type-name has a registered converter dispatches to that converter; any
list head whose name (`(str head)`) is in the command registry
dispatches to that compile-fn. On registry miss, the original ~180
hardcoded branches handle the form.
Each registered fn receives a ctx dict (built per call) exposing
`:hs-to-sx` for recursion plus the AST fields the dispatch needs
(`:ast :head` for commands; `:ast :value-ast :type-name` for
converters). Mirrors Step 9's parser feature registry shape.
Smoke tested: register custom command + converter, both dispatch;
built-in `(as x \"Int\")` still produces `(hs-coerce x \"Int\")`.
Mirror `shared/static/wasm/sx/hs-compiler.sx` copied byte-identical.
OCaml: 4545/1339, JS: 2591/2465 — both match baseline, zero regressions.
Second piece of plans/designs/hs-plugin-system.md (Step 11 next).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `_hs-feature-registry` dict and `hs-register-feature!` to
`lib/hyperscript/parser.sx`. Replace `parse-feat`'s hardcoded `cond`
on feature names with a registry lookup; the paren-open and
default-expression branches remain as fallthroughs.
Each parse-fn receives a `ctx` dict (built per call by `parse-feat-ctx`)
exposing parser internals (`:adv!`, `:tp-val`, `:tp-type`, `:at-end?`,
`:parse-cmd-list`, `:parse-expr`) and the per-feature handlers
(`:parse-on-feat` … `:parse-socket-feat`). All nine builtins
(`on`, `init`, `def`, `behavior`, `live`, `when`, `worker`, `bind`,
`socket`) are registered at file load time, so plugins added later via
`hs-register-feature!` persist across `hs-parse` calls.
Worker stub still raises identically. Mirror `shared/static/wasm/sx/hs-parser.sx`
copied byte-identical. OCaml: 4545/1339, JS: 2591/2465 — both match
baseline, zero regressions.
First piece of plans/designs/hs-plugin-system.md (Steps 10/11 follow).
