rose-ash/plans/designs/hs-plugin-system.md

HyperScript Plugin / Extension System

Post-Bucket-F capability work. No conformance delta on its own — the payoff is clean architecture for language embeds (Lua, Prolog, Worker runtime) and alignment with real _hyperscript's extension model.

---

1. Motivation

1a. Real _hyperscript has a plugin API

Stock _hyperscript ships a core bundle with feature stubs and a use(ext) hook that loads named extensions at runtime. The worker feature is the canonical example: the core parser has a stub that errors helpfully; loading the worker extension replaces the stub with a real implementation.

We currently have no equivalent. New grammar or compiler targets require editing parse-feat's hardcoded cond or hs-to-sx's hardcoded dispatch. This is fine for conformance work but wrong for language embeds.

1b. Ad-hoc hooks are accumulating

runtime.sx already has hs-prolog-hook / hs-set-prolog-hook! / prolog (nodes 140–142) — an informal plugin slot bolted on outside the parser and compiler. This pattern will repeat for Lua, and again for the Worker runtime. A proper registry prevents the drift.

1c. E39 worker stub is a placeholder

The stub added in E39 (parse-feat raises immediately on "worker") was explicitly designed to be replaced by a real plugin at a single site. This plan is where that replacement happens.

1d. Bucket-F Group 10 needs a converter registry

as MyType via registered converter is already in the Bucket-F plan (Group 10). A hs-register-converter! registry is the natural home for it — and the plugin system is the right time to add registries generally.

---

2. Scope

In scope:

• Parser feature registry (parse-feat dispatch)
• Compiler command registry (hs-to-sx dispatch)
• as converter registry (hs-coerce dispatch)
• Migration of E39 worker stub to use the parser registry
• Migration of hs-prolog-hook ad-hoc slot to a proper plugin
• Worker full runtime plugin (first real plugin)
• Lua embed plugin
• Prolog embed plugin

Out of scope:

• Changing the test runner or generator
• Any conformance delta (this plan doesn't target failing tests)
• Third-party plugin loading from external URLs (future)
• Hot-reload of plugins (future)

---

3. Registry design

Three registries, all SX dicts. Checked before the hardcoded cond in each dispatch. Registration functions defined alongside the registries in their respective files.

3a. Parser feature registry (lib/hyperscript/parser.sx)

(define _hs-feature-registry (dict))

(define hs-register-feature!
  (fn (keyword parse-fn)
    (set! _hs-feature-registry
      (dict-set _hs-feature-registry keyword parse-fn))))

In parse-feat, prepend a registry lookup before the existing cond:

(let ((registered (dict-get _hs-feature-registry val)))
  (if registered
    (registered)           ;; call the registered parse-fn (no args; uses closure over adv!/tp-val etc.)
    (cond                  ;; existing dispatch unchanged below
      ...)))

parse-fn is a zero-arg thunk that has access to the parser's internal state via the same closure that the existing parse-* helpers use. Since parse-feat is itself defined inside the big let in hs-parse, all the parser helpers (adv!, tp-val, tp-typ, parse-cmd-list, etc.) are in scope.

3b. Compiler command registry (lib/hyperscript/compiler.sx)

(define _hs-compiler-registry (dict))

(define hs-register-compiler!
  (fn (head compile-fn)
    (set! _hs-compiler-registry
      (dict-set _hs-compiler-registry (str head) compile-fn))))

In hs-to-sx, before the existing cond on head, check the registry:

(let ((registered (dict-get _hs-compiler-registry (str head))))
  (if registered
    (registered ast)
    (cond ...)))

compile-fn receives the full AST node and returns an SX expression.

3c. as converter registry (lib/hyperscript/runtime.sx)

(define _hs-converters (dict))

(define hs-register-converter!
  (fn (type-name converter-fn)
    (set! _hs-converters
      (dict-set _hs-converters type-name converter-fn))))

In hs-coerce, add a registry lookup as the last cond clause before the fallthrough error:

((dict-get _hs-converters type-name)
  ((dict-get _hs-converters type-name) value))

This is also the hook that Bucket-F Group 10 (can accept custom conversions) hangs on — so implementing it here kills two birds.

---

4. First-party plugins

Each plugin is a .sx file in lib/hyperscript/plugins/. Plugins call the registration functions at load time (top-level do forms). The host loads plugins explicitly after the core files.

4a. Worker plugin (lib/hyperscript/plugins/worker.sx)

Phase 1 — stub migration (immediate): Remove the inline error branch from parse-feat (the E39 stub). Replace with:

(hs-register-feature! "worker"
  (fn ()
    (error "worker plugin is not installed — see https://hyperscript.org/features/worker")))

This is identical behaviour to E39 but routed through the registry. The stub lives in the plugin file, not the core parser. No test regression.

Phase 2 — full runtime:

Parser: parse-worker-feat — consumes worker <Name> [(<url>*)] <def|js>* end, returns (worker Name urls defs) AST node.

Compiler: registered under "worker" head:

• Emits (hs-worker-define! "Name" urls defs) call.

Runtime additions in the plugin file:

• hs-worker-define! — creates a {:_hs-worker true :name N :handle H :exports (...)} record, binds it in the HS top-level env under Name.
• hs-method-call (existing) detects :_hs-worker and dispatches via postMessage.
• Worker script body compiled to a standalone SX bundle posted to a Blob URL.
• Return values are promise-wrapped; async-transparent via perform/IO suspension.

Mock env additions for the test runner: Worker constructor + synchronous message loop for the 7 sibling test.skip(...) upstream tests (the ones deferred in E39).

4b. Prolog plugin (lib/hyperscript/plugins/prolog.sx)

Replaces the ad-hoc hs-prolog-hook in runtime.sx.

Parser: Register "prolog" feature — parses prolog(<db-expr>, <goal-expr>) at feature level (alternative: keep as an expression, register a compiler extension only).

Compiler: Registered under "prolog" head — emits (prolog db goal).

Runtime: The existing prolog function in runtime.sx moves here. hs-prolog-hook and hs-set-prolog-hook! are removed from runtime.sx and the hook mechanism is replaced by the plugin loading lib/prolog/runtime.sx and wiring the solver directly.

Remove from runtime.sx nodes 140–142 once the plugin is live.

4c. Lua plugin (lib/hyperscript/plugins/lua.sx)

Parser: Register "lua" feature — parses lua ... end block, captures the body as a raw string.

Compiler: Registered under "lua" head — emits (lua-eval <body-string>).

Runtime: lua-eval calls lib/lua/runtime.sx's eval entry point, returns result as an SX value via hs-host-to-sx. Errors surface as HS catch-able exceptions.

This enables inline Lua in HyperScript:

on click
  lua
    return document.title:upper()
  end
  put it into me
end

---

5. Load order

lib/hyperscript/parser.sx       ;; defines _hs-feature-registry, hs-register-feature!
lib/hyperscript/compiler.sx     ;; defines _hs-compiler-registry, hs-register-compiler!
lib/hyperscript/runtime.sx      ;; defines _hs-converters, hs-register-converter!
lib/hyperscript/plugins/worker.sx
lib/hyperscript/plugins/prolog.sx
lib/hyperscript/plugins/lua.sx

The test runner (tests/hs-run-filtered.js) loads plugins after core. The browser WASM bundle includes all three by default (plugins are small; no reason to lazy-load them).

---

6. Migration checklist

The work below is ordered to keep main green at every commit. Each step is independently committable.

Step 1 — Registries (infrastructure, no behaviour change)

1. Add _hs-feature-registry + hs-register-feature! to parser.sx. Thread the registry check into parse-feat. No entries yet → behaviour unchanged.
2. Add _hs-compiler-registry + hs-register-compiler! to compiler.sx. Thread into hs-to-sx. No entries yet → behaviour unchanged.
3. Add _hs-converters + hs-register-converter! to runtime.sx. Thread into hs-coerce. No entries yet → behaviour unchanged.
4. sx_validate all three files. Run full HS suite — expect zero regressions.
5. Commit: HS: plugin registry infrastructure (parser + compiler + converter).

Step 2 — Worker stub migration

6. Create lib/hyperscript/plugins/worker.sx. Register the worker stub error.
7. Remove the inline ((= val "worker") ...) branch from parse-feat in parser.sx.
8. Update the test runner to load worker.sx after core.
9. Run HS_SUITE=hs-upstream-worker — expect 1/1. Run full suite — expect no regressions.
10. Commit: HS: migrate E39 worker stub to plugin registry.

Step 3 — Prolog plugin

11. Create lib/hyperscript/plugins/prolog.sx. Wire to lib/prolog/runtime.sx.
12. Remove hs-prolog-hook, hs-set-prolog-hook!, prolog from runtime.sx nodes 140–142.
13. Update test runner to load prolog.sx.
14. Validate and run full suite.
15. Commit: HS: prolog plugin replaces ad-hoc hook.

Step 4 — as converter registry (bridges Bucket-F Group 10)

16. Confirm hs-register-converter! satisfies the Group 10 test can accept custom conversions. If yes, this step may be pulled into Bucket-F Group 10 instead (no duplication — just move step 3 of §6 there).
17. Commit: HS: as-converter registry wired into hs-coerce.

Step 5 — Lua plugin

18. Create lib/hyperscript/plugins/lua.sx.
19. Add lua-eval to runtime.sx or directly in the plugin file.
20. Parser: parse-lua-feat consuming lua … end.
21. Compiler: registered "lua" head.
22. Write 3–5 tests in spec/tests/test-hyperscript-lua.sx:
    • Lua returns a string → HS uses it.
    • Lua error → HS catch.
    • Lua reads a passed argument.
23. Commit: HS: Lua plugin — inline lua...end blocks.

Step 6 — Worker full runtime plugin

24. Extend worker.sx: implement parse-worker-feat, compiler entry, hs-worker-define!, hs-method-call worker branch.
25. Extend test runner: Worker constructor + synchronous message loop.
26. Un-skip the 7 sibling worker tests from upstream.
27. Target: 7/7 worker suite.
28. Commit: HS: Worker plugin full runtime (+7 tests).

---

7. Risks

• parse-feat closure scope — hs-register-feature! stores parse-fns that need access to parser-internal helpers (adv!, tp-val, etc.). These are only in scope inside hs-parse's big let. Two options: (a) the registry stores fns that receive a parser-context dict as arg, or (b) the registry is checked inside parse-feat where helpers are in scope and fns are zero-arg closures captured at registration time. Option (b) is simpler but requires plugins to be loaded while the parser let is being evaluated — i.e., plugins must be defined inside the parser file or the context dict must be exposed. Recommended: expose a _hs-parser-ctx dict at the module level that parse-fns receive as their sole argument. This makes the API explicit and plugins independent files.

• Worker Blob URL in WASM — URL.createObjectURL is available in browsers but not in the OCaml WASM host. Worker full runtime is browser-only; flag it with a capability check and graceful fallback.

• Lua/Prolog mutual recursion — a Lua block calling back into HS calling back into Lua is theoretically possible via the IO suspension machinery. Don't try to support it initially; raise a clear error if detected.

• Plugin load-order sensitivity — hs-register-feature! must be called before any source is parsed. If a plugin is loaded lazily (future), a worker MyWorker in the page would hit the stub before the full plugin registers. Acceptable for now; document that plugins must be loaded at boot.

• runtime.sx cleanup for prolog — nodes 140–142 are referenced nowhere else in the codebase (grep confirms). Safe to delete once the plugin is live.

---

8. Non-goals

• Runtime use(ext) API (JS-style dynamic plugin install) — future.
• Plugin namespacing / versioning — future.
• Any conformance tests other than the 7 worker tests in step 6.
• Changing how the WASM bundle is built or split.