rose-ash/plans/designs/hs-plugin-system.md

# HyperScript Plugin / Extension System

Post-Bucket-F capability work. No conformance delta on its own — the payoff is
clean architecture for language embeds (Lua, Prolog, Worker runtime) and
alignment with real `_hyperscript`'s extension model.

---

## 1. Motivation

### 1a. Real `_hyperscript` has a plugin API

Stock `_hyperscript` ships a core bundle with feature stubs and a `use(ext)`
hook that loads named extensions at runtime. The worker feature is the canonical
example: the core parser has a stub that errors helpfully; loading the worker
extension replaces the stub with a real implementation.

We currently have no equivalent. New grammar or compiler targets require editing
`parse-feat`'s hardcoded `cond` or `hs-to-sx`'s hardcoded dispatch. This is
fine for conformance work but wrong for language embeds.

### 1b. Ad-hoc hooks are accumulating

`runtime.sx` already has `hs-prolog-hook` / `hs-set-prolog-hook!` / `prolog`
(nodes 140–142) — an informal plugin slot bolted on outside the parser and
compiler. This pattern will repeat for Lua, and again for the Worker runtime.
A proper registry prevents the drift.

### 1c. E39 worker stub is a placeholder

The stub added in E39 (`parse-feat` raises immediately on `"worker"`) was
explicitly designed to be replaced by a real plugin at a single site. This plan
is where that replacement happens.

### 1d. Bucket-F Group 10 needs a converter registry

`as MyType` via registered converter is already in the Bucket-F plan (Group 10).
A `hs-register-converter!` registry is the natural home for it — and the plugin
system is the right time to add registries generally.

---

## 2. Scope

**In scope:**
- Parser feature registry (`parse-feat` dispatch)
- Compiler command registry (`hs-to-sx` dispatch)
- `as` converter registry (`hs-coerce` dispatch)
- Migration of E39 worker stub to use the parser registry
- Migration of `hs-prolog-hook` ad-hoc slot to a proper plugin
- Worker full runtime plugin (first real plugin)
- Lua embed plugin
- Prolog embed plugin

**Out of scope:**
- Changing the test runner or generator
- Any conformance delta (this plan doesn't target failing tests)
- Third-party plugin loading from external URLs (future)
- Hot-reload of plugins (future)

---

## 3. Registry design

Three registries, all SX dicts. Checked before the hardcoded `cond` in each
dispatch. Registration functions defined alongside the registries in their
respective files.

### 3a. Parser feature registry (`lib/hyperscript/parser.sx`)

```lisp
(define _hs-feature-registry (dict))

(define hs-register-feature!
  (fn (keyword parse-fn)
    (set! _hs-feature-registry
      (dict-set _hs-feature-registry keyword parse-fn))))
```

In `parse-feat`, prepend a registry lookup before the existing `cond`:

```lisp
(let ((registered (dict-get _hs-feature-registry val)))
  (if registered
    (registered)           ;; call the registered parse-fn (no args; uses closure over adv!/tp-val etc.)
    (cond                  ;; existing dispatch unchanged below
      ...)))
```

`parse-fn` is a zero-arg thunk that has access to the parser's internal state
via the same closure that the existing `parse-*` helpers use. Since `parse-feat`
is itself defined inside the big `let` in `hs-parse`, all the parser helpers
(`adv!`, `tp-val`, `tp-typ`, `parse-cmd-list`, etc.) are in scope.

### 3b. Compiler command registry (`lib/hyperscript/compiler.sx`)

```lisp
(define _hs-compiler-registry (dict))

(define hs-register-compiler!
  (fn (head compile-fn)
    (set! _hs-compiler-registry
      (dict-set _hs-compiler-registry (str head) compile-fn))))
```

In `hs-to-sx`, before the existing `cond` on `head`, check the registry:

```lisp
(let ((registered (dict-get _hs-compiler-registry (str head))))
  (if registered
    (registered ast)
    (cond ...)))
```

`compile-fn` receives the full AST node and returns an SX expression.

### 3c. `as` converter registry (`lib/hyperscript/runtime.sx`)

```lisp
(define _hs-converters (dict))

(define hs-register-converter!
  (fn (type-name converter-fn)
    (set! _hs-converters
      (dict-set _hs-converters type-name converter-fn))))
```

In `hs-coerce`, add a registry lookup as the last `cond` clause before the
fallthrough error:

```lisp
((dict-get _hs-converters type-name)
  ((dict-get _hs-converters type-name) value))
```

This is also the hook that Bucket-F Group 10 (`can accept custom conversions`)
hangs on — so implementing it here kills two birds.

---

## 4. First-party plugins

Each plugin is a `.sx` file in `lib/hyperscript/plugins/`. Plugins call the
registration functions at load time (top-level `do` forms). The host loads
plugins explicitly after the core files.

### 4a. Worker plugin (`lib/hyperscript/plugins/worker.sx`)

**Phase 1 — stub migration (immediate):**
Remove the inline error branch from `parse-feat` (the E39 stub). Replace with:

```lisp
(hs-register-feature! "worker"
  (fn ()
    (error "worker plugin is not installed — see https://hyperscript.org/features/worker")))
```

This is identical behaviour to E39 but routed through the registry. The stub
lives in the plugin file, not the core parser. No test regression.

**Phase 2 — full runtime:**

Parser: `parse-worker-feat` — consumes `worker <Name> [(<url>*)] <def|js>* end`,
returns `(worker Name urls defs)` AST node.

Compiler: registered under `"worker"` head:
- Emits `(hs-worker-define! "Name" urls defs)` call.

Runtime additions in the plugin file:
- `hs-worker-define!` — creates a `{:_hs-worker true :name N :handle H :exports (...)}` record,
  binds it in the HS top-level env under `Name`.
- `hs-method-call` (existing) detects `:_hs-worker` and dispatches via `postMessage`.
- Worker script body compiled to a standalone SX bundle posted to a Blob URL.
- Return values are promise-wrapped; async-transparent via `perform`/IO suspension.

Mock env additions for the test runner: `Worker` constructor + synchronous
message loop for the 7 sibling `test.skip(...)` upstream tests (the ones
deferred in E39).

### 4b. Prolog plugin (`lib/hyperscript/plugins/prolog.sx`)

Replaces the ad-hoc `hs-prolog-hook` in `runtime.sx`.

**Parser:** Register `"prolog"` feature — parses
`prolog(<db-expr>, <goal-expr>)` at feature level (alternative: keep as an
expression, register a compiler extension only).

**Compiler:** Registered under `"prolog"` head — emits `(prolog db goal)`.

**Runtime:** The existing `prolog` function in `runtime.sx` moves here.
`hs-prolog-hook` and `hs-set-prolog-hook!` are removed from `runtime.sx` and
the hook mechanism is replaced by the plugin loading `lib/prolog/runtime.sx`
and wiring the solver directly.

Remove from `runtime.sx` nodes 140–142 once the plugin is live.

### 4c. Lua plugin (`lib/hyperscript/plugins/lua.sx`)

**Parser:** Register `"lua"` feature — parses `lua ... end` block, captures
the body as a raw string.

**Compiler:** Registered under `"lua"` head — emits `(lua-eval <body-string>)`.

**Runtime:** `lua-eval` calls `lib/lua/runtime.sx`'s eval entry point, returns
result as an SX value via `hs-host-to-sx`. Errors surface as HS `catch`-able
exceptions.

This enables inline Lua in HyperScript:

```
on click
  lua
    return document.title:upper()
  end
  put it into me
end
```

---

## 5. Load order

```
lib/hyperscript/parser.sx       ;; defines _hs-feature-registry, hs-register-feature!
lib/hyperscript/compiler.sx     ;; defines _hs-compiler-registry, hs-register-compiler!
lib/hyperscript/runtime.sx      ;; defines _hs-converters, hs-register-converter!
lib/hyperscript/plugins/worker.sx
lib/hyperscript/plugins/prolog.sx
lib/hyperscript/plugins/lua.sx
```

The test runner (`tests/hs-run-filtered.js`) loads plugins after core. The
browser WASM bundle includes all three by default (plugins are small; no
reason to lazy-load them).

---

## 6. Migration checklist

The work below is ordered to keep main green at every commit. Each step is
independently committable.

### Step 1 — Registries (infrastructure, no behaviour change)

1. Add `_hs-feature-registry` + `hs-register-feature!` to `parser.sx`.
   Thread the registry check into `parse-feat`. No entries yet → behaviour
   unchanged.
2. Add `_hs-compiler-registry` + `hs-register-compiler!` to `compiler.sx`.
   Thread into `hs-to-sx`. No entries yet → behaviour unchanged.
3. Add `_hs-converters` + `hs-register-converter!` to `runtime.sx`. Thread
   into `hs-coerce`. No entries yet → behaviour unchanged.
4. `sx_validate` all three files. Run full HS suite — expect zero regressions.
5. Commit: `HS: plugin registry infrastructure (parser + compiler + converter)`.

### Step 2 — Worker stub migration

6. Create `lib/hyperscript/plugins/worker.sx`. Register the worker stub error.
7. Remove the inline `((= val "worker") ...)` branch from `parse-feat` in
   `parser.sx`.
8. Update the test runner to load `worker.sx` after core.
9. Run `HS_SUITE=hs-upstream-worker` — expect 1/1. Run full suite — expect no
   regressions.
10. Commit: `HS: migrate E39 worker stub to plugin registry`.

### Step 3 — Prolog plugin

11. Create `lib/hyperscript/plugins/prolog.sx`. Wire to `lib/prolog/runtime.sx`.
12. Remove `hs-prolog-hook`, `hs-set-prolog-hook!`, `prolog` from `runtime.sx`
    nodes 140–142.
13. Update test runner to load `prolog.sx`.
14. Validate and run full suite.
15. Commit: `HS: prolog plugin replaces ad-hoc hook`.

### Step 4 — `as` converter registry (bridges Bucket-F Group 10)

16. Confirm `hs-register-converter!` satisfies the Group 10 test
    `can accept custom conversions`. If yes, this step may be pulled into
    Bucket-F Group 10 instead (no duplication — just move step 3 of §6 there).
17. Commit: `HS: as-converter registry wired into hs-coerce`.

### Step 5 — Lua plugin

18. Create `lib/hyperscript/plugins/lua.sx`.
19. Add `lua-eval` to `runtime.sx` or directly in the plugin file.
20. Parser: `parse-lua-feat` consuming `lua … end`.
21. Compiler: registered `"lua"` head.
22. Write 3–5 tests in `spec/tests/test-hyperscript-lua.sx`:
    - Lua returns a string → HS uses it.
    - Lua error → HS catch.
    - Lua reads a passed argument.
23. Commit: `HS: Lua plugin — inline lua...end blocks`.

### Step 6 — Worker full runtime plugin

24. Extend `worker.sx`: implement `parse-worker-feat`, compiler entry,
    `hs-worker-define!`, `hs-method-call` worker branch.
25. Extend test runner: `Worker` constructor + synchronous message loop.
26. Un-skip the 7 sibling worker tests from upstream.
27. Target: 7/7 worker suite.
28. Commit: `HS: Worker plugin full runtime (+7 tests)`.

---

## 7. Risks

- **`parse-feat` closure scope** — `hs-register-feature!` stores parse-fns
  that need access to parser-internal helpers (`adv!`, `tp-val`, etc.). These
  are only in scope inside `hs-parse`'s big `let`. Two options:
  (a) the registry stores fns that receive a parser-context dict as arg, or
  (b) the registry is checked *inside* `parse-feat` where helpers are in scope
  and fns are zero-arg closures captured at registration time.
  Option (b) is simpler but requires plugins to be loaded while the parser
  `let` is being evaluated — i.e., plugins must be defined *inside* the parser
  file or the context dict must be exposed. **Recommended:** expose a
  `_hs-parser-ctx` dict at the module level that parse-fns receive as their
  sole argument. This makes the API explicit and plugins independent files.

- **Worker Blob URL in WASM** — `URL.createObjectURL` is available in browsers
  but not in the OCaml WASM host. Worker full runtime is browser-only; flag it
  with a capability check and graceful fallback.

- **Lua/Prolog mutual recursion** — a Lua block calling back into HS calling
  back into Lua is theoretically possible via the IO suspension machinery.
  Don't try to support it initially; raise a clear error if detected.

- **Plugin load-order sensitivity** — `hs-register-feature!` must be called
  before any source is parsed. If a plugin is loaded lazily (future), a
  `worker MyWorker` in the page would hit the stub before the full plugin
  registers. Acceptable for now; document that plugins must be loaded at boot.

- **`runtime.sx` cleanup for prolog** — nodes 140–142 are referenced nowhere
  else in the codebase (grep confirms). Safe to delete once the plugin is live.

---

## 8. Non-goals

- Runtime `use(ext)` API (JS-style dynamic plugin install) — future.
- Plugin namespacing / versioning — future.
- Any conformance tests other than the 7 worker tests in step 6.
- Changing how the WASM bundle is built or split.