rose-ash/plans/tcl-sx-completion.md

# Tcl-on-SX completion plan — SX capabilities first

Tcl phases 1–6 are complete (329/329 tests). This plan covers the remaining
limitations, ordered by the SX work needed to enable them.

## Key audit findings

Several apparent gaps are already solved in SX:

- **Floats** — SX parses `3.14` natively; `(+ 1.5 2.5) → 4.0`; `str` formats
  with `%g` (compact, no trailing zeros). `floor`/`ceil`/`round`/`truncate`
  all exist in `spec/primitives.sx`.
- **Regex** — `regexp-match`, `regexp-match-all`, `regexp-replace`,
  `regexp-replace-all`, `regexp-split` are registered OCaml primitives using
  `Re.Pcre` (`hosts/ocaml/lib/sx_primitives.ml`).
- **`call/cc` multi-shot** — works. `set!` on closed-over vars works. Fibers
  are implementable as a pure SX library.
- **`perform` user-accessible** — `(perform :foo 42)` from user code suspends
  the evaluator and emits an IO request. The algebraic effects model is
  already half-built.
- **No `file-read`/`clock-seconds`** — not yet registered as OCaml primitives.
  Only string ports exist. Would need small OCaml additions.
- **No `env-as-value`** — environments are internal OCaml values, not
  inspectable from SX user code.

---

## Phase 1 — Zero-cost wins (no SX changes, only `lib/tcl/`)

Everything here is pure Tcl implementation work.

| Status | Work | Effort | Unlocks in Tcl |
|---|---|---|---|
| [x] | Float in `expr` — detect `.` in number tokens, route through float ops instead of `parse-int` | half day | `expr {3.14 * 2}`, `expr {sqrt(2.0)}`, float comparisons |
| [x] | `regexp pattern str` and `regsub pattern str repl` wrapping existing SX primitives | few hours | pattern matching, text processing |
| [x] | `apply {args body} ?arg…?` — anonymous proc call | 1 hour | higher-order functions, `lmap` idiom |
| [x] | `array get/set/names/size/exists/unset` commands | half day | array variables (tokenizer already parses `$arr(key)`) |

**Total: ~2 days. Zero SX changes.**

---

## Phase 2 — `lib/fiber.sx` (pure SX library, no OCaml)

| Status | Work |
|---|---|
| [x] | Create `lib/fiber.sx` — `make-fiber` / `fiber-resume` / `fiber-done?` |
| [x] | Rewrite `tcl-cmd-coroutine` to use `make-fiber` (true suspension) |

`call/cc` is multi-shot and `set!` on closed-over vars both work. Fibers are
implementable as a pure SX library using symmetric continuation swapping:

```scheme
; lib/fiber.sx — canonical fiber primitive for all hosted languages
(define make-fiber
  (fn (thunk)
    (define slot-k nil)
    (define slot-caller nil)
    (define slot-done false)
    (fn (resume-val)
      (call/cc (fn (caller-k)
        (set! slot-caller caller-k)
        (if (nil? slot-k)
          (begin (thunk resume-val) (set! slot-done true) (caller-k nil))
          (slot-k resume-val)))))))

(define fiber-yield
  (fn (val)
    (call/cc (fn (k)
      (set! slot-k k)
      (slot-caller val)))))
```

Each coroutine becomes a fiber. `yield` swaps to the caller; calling the
coroutine name swaps back. True suspension, not eager pre-execution.

**Broader value:** Ruby fibers, Python generators, Lua coroutines, async event
loops, cooperative schedulers all sit on top of the same library.

**Alternatively:** `perform` is user-accessible. A Tcl scheduler living outside
the SX evaluator (the OCaml host or an SX event loop) could catch
`(perform :fiber-yield val)` and dispatch it — the algebraic effects model,
already half-built.

**Total: 2–3 days. Produces `lib/fiber.sx` as a lasting SX contribution.**
Tcl coroutines then rewrite using `make-fiber` for true suspension.

---

## Phase 3 — Small OCaml additions (`sx_primitives.ml`)

Each is ~10–20 lines of OCaml. All are useful across the whole platform, not
just Tcl.

| Status | Primitive | OCaml effort | Unlocks |
|---|---|---|---|
| [x] | `(file-read path)` → string | tiny | Tcl `open`/`read`, SX scripts reading files |
| [x] | `(file-write path str)` → nil | tiny | Tcl `open`/`puts` to files |
| [x] | `(file-exists? path)` → bool | tiny | Tcl `file exists` |
| [x] | `(file-glob pattern)` → list | small | Tcl `glob` |
| [x] | `(clock-seconds)` → int | tiny | Tcl `clock seconds` |
| [x] | `(clock-format n fmt)` → string | small (wraps `strftime`) | Tcl `clock format` |

**Total: 1 day. One focused afternoon of OCaml.**

---

## Phase 4 — env-as-value (architectural) ✓



`uplevel`/`upvar` required an explicit frame stack because SX environments
aren't inspectable from user code. Adding:

```scheme
(current-env)            ; → env value
(eval-in-env env expr)   ; → result
(env-lookup env key)     ; → value or nil
(env-extend env key val) ; → new env (non-mutating)
```

...would let `uplevel N` be literally "look up env N levels up, eval in it."
The Tcl frame stack (hundreds of lines) collapses to ~10 lines.

Also benefits: metacircular evaluators, REPL tooling, live debugging (inspect
any scope), the sx_docs server's eval endpoint.

More invasive — touches `sx_types.ml` and `sx_server.ml` — but a meaningful
architectural improvement worth doing when the moment is right.

**Total: 2–3 days. High architectural value, not urgent.**

---

## Phase 5 — Channel I/O (random access + non-blocking) ✓

Real Tcl channel commands replacing the previous stubs. SX gained 11 channel
primitives in `sx_primitives.ml` (using `Unix.openfile` + `Unix.read`/`write`/
`lseek`/`set_nonblock`). Tcl `open`/`close`/`read`/`gets`/`puts`/`seek`/`tell`/
`eof`/`flush`/`fconfigure` now wrap them.

| Status | Work | Unlocks in Tcl |
|---|---|---|
| [x] | `channel-open`, `channel-close` | `open` returns "fileN", `close` actually closes |
| [x] | `channel-read`, `channel-read-line`, `channel-write` | `read`/`gets`/`puts` to/from real files |
| [x] | `channel-seek`, `channel-tell` | random access — `seek $c offset start\|current\|end`, `tell` |
| [x] | `channel-eof?`, `channel-flush` | proper EOF detection, no-op flush |
| [x] | `channel-blocking?`, `channel-set-blocking!` | `fconfigure $c -blocking 0\|1` |

Modes supported: `r`, `w`, `a`, `r+`, `w+`, `a+`. Whence: `start`, `current`, `end`.

`puts` now detects channel argument (string starting with "file") and dispatches
to `channel-write`; otherwise writes to `interp :output` as before.

**Total: ~half day. 7 new idiom tests covering write+read, gets-loop, seek/tell,
eof-after-read, append mode, seek-to-end, fconfigure-blocking.**

---

## Phase 5b — Event loop: fileevent / after / vwait / update ✓

Tcl event-driven I/O scoped to script-mode (vs. server-side commands). The
mechanism rides on the existing IO suspension model: SX adds one new primitive
`(io-select-channels read-list write-list timeout-ms)` wrapping `Unix.select`,
and the Tcl event loop is implemented in Tcl itself (no sx_server.ml changes).

| Status | Work | Unlocks in Tcl |
|---|---|---|
| [x] | `io-select-channels` SX primitive | Unix.select on registered channels |
| [x] | `fileevent $chan readable\|writable script` | event handler registration; `{}` to unregister |
| [x] | `after ms script` | one-shot timer queued in `:timers` |
| [x] | `after ms` (no script) | sleep that drives the event loop |
| [x] | `vwait varname` | block until var set/changed, runs handlers |
| [x] | `update` | non-blocking event drain (poll, fire ready handlers) |

Event loop: `tcl-event-step interp poll-timeout-ms` — fires expired timers,
calls `io-select-channels` with fd list from `:fileevents`, runs ready handlers.
`vwait` polls every 1000ms or until var changes (whichever first); `update` is
`tcl-event-step interp 0`.

State on interp: `:fileevents` (list of `(chan event script)`) and `:timers`
(list of `(expiry-ms script)`, sorted by expiry).

**Trade-off:** Scoped to script mode — `vwait` from inside a server-handled
command would not interact with sx_server's stdin scheduler. Sufficient for ~95%
of real-world Tcl scripts (sockets, pipes, GUI-style polling, CLI tools).

**Total: ~half day. 5 new idiom tests: after-vwait-timer, after-multiple-timers-
update, fileevent-readable-fires, fileevent-query-script, after-cancel-via-
vwait-timing. 354/354 green.**

---

## Phase 5c — TCP sockets (client + server) ✓

Tcl `socket` command for both connecting and listening. Reuses the channel
registry built in Phase 5 and the event loop from Phase 5b. Server channels
auto-fire user callbacks via fileevent on each accept.

| Status | Work | Unlocks in Tcl |
|---|---|---|
| [x] | `socket-connect host port` SX primitive | TCP client via `Unix.socket`+`Unix.connect` |
| [x] | `socket-server ?host? port` SX primitive | listening socket; `Unix.bind`+`Unix.listen` (backlog 8) |
| [x] | `socket-accept server-chan` SX primitive | returns `{:channel :host :port}` |
| [x] | Tcl `socket host port` | TCP client; returns "sockN" |
| [x] | Tcl `socket -server cb port` | listening socket; auto-fires `cb sock host port` per accept |
| [x] | `puts` channel detection extended | "sockN" channels also dispatch to `channel-write` |

The auto-accept mechanism is a tiny internal Tcl command `_sock-do-accept`
registered by `socket -server`. Its registered handler, fired by the event
loop, accepts the pending client, then evaluates `cb client-chan host port`.

`Unix.SO_REUSEADDR` is set on server sockets to avoid TIME_WAIT issues
during testing. Host argument supports `localhost`, `0.0.0.0`, IPv4 literal,
or DNS lookup via `Unix.gethostbyname`.

**Total: ~half day. 4 new idiom tests: socket-server-fires-callback,
socket-client-server-roundtrip, socket-server-peer-host, socket-multiple-
connections. 358/358 green.**

---

## Phase 5d — File metadata + filesystem ops ✓

Real implementations of `file isfile`/`isdir`/`readable`/`writable`/`size`/
`mtime`/`atime`/`type` (previously stubs returning `0`/`""`) and proper
`file delete`/`mkdir`/`copy`/`rename`.

| Status | Primitive | Wraps |
|---|---|---|
| [x] | `file-size`, `file-mtime`, `file-stat` | `Unix.stat` |
| [x] | `file-isfile?`, `file-isdir?` | `Unix.stat`+`st_kind` |
| [x] | `file-readable?`, `file-writable?` | `Unix.access [R_OK\|W_OK]` |
| [x] | `file-delete` | `Unix.unlink`/`rmdir` (tolerates ENOENT) |
| [x] | `file-mkdir` | recursive `Unix.mkdir 0o755` |
| [x] | `file-copy`, `file-rename` | stdlib I/O / `Sys.rename` |

`file-stat` returns a dict `{:size :mtime :atime :ctime :mode :type}` with
`:type` ∈ `file|directory|link|fifo|socket|...`. Tcl `file copy`/`rename`/
`delete` strip leading-`-` flags so `file delete -force` works.

**Total: ~half day. 10 new idiom tests covering isfile, isdir on /tmp, size,
readable, mkdir + check, copy roundtrip, rename, mtime > 0. 368/368 green.**

---

## Phase 5e — clock format options + clock scan ✓

Real `-format`, `-timezone`, and `-gmt` options on `clock format`, and a
working `clock scan` for parsing date strings back to Unix seconds.

| Status | Work |
|---|---|
| [x] | `clock-format` extended to `(t fmt tz)` with tz ∈ `utc|local` |
| [x] | More format specifiers: `%y` (2-digit year), `%I` (12h hour), `%p` (AM/PM), `%w` (weekday num), `%%` (literal) |
| [x] | `clock-scan` SX primitive: format-driven parser + manual `timegm` (OCaml stdlib lacks it) |
| [x] | Tcl `clock format $secs -format $fmt -timezone $tz -gmt 0\|1` |
| [x] | Tcl `clock scan $str -format $fmt -timezone $tz -gmt 0\|1` |

Default tz for both is UTC. Format specifiers supported by scan: `%Y %y %m
%d %e %H %I %M %S %%`. Unsupported specifiers in scan are silently skipped
(no validation).

**Total: ~half day. 5 new idiom tests: clock-format-utc, fmt-default,
scan-roundtrip, scan-returns-int, format-percent-pct. 373/373 green.**

---

## Phase 5f — `socket -async` (non-blocking connect) ✓

| Status | Work |
|---|---|
| [x] | `socket-connect-async host port` SX primitive — `Unix.set_nonblock` + `Unix.connect`, catches `EINPROGRESS` |
| [x] | `channel-async-error chan` SX primitive — `Unix.getsockopt_error` |
| [x] | Tcl `socket -async host port` — returns "sockN" immediately |
| [x] | Tcl `fconfigure $chan -error` — queries async-error |

Connection completes when the channel becomes writable; canonical pattern is
`fileevent $sock writable {handler}`. Channel buffer state is set to
`blocking=false` so subsequent reads/writes don't block.

**Total: ~few hours. 3 new idiom tests: socket-async-completes-writable,
socket-async-then-write, socket-async-no-error. 376/376 green.**

**Bug fix landed alongside:** `tcl-call-proc` was discarding `:fileevents`,
`:timers`, and `:procs` updates made inside Tcl procs (only `:commands` was
forwarded). Changed the return to forward the inner `result-interp` as the
base while restoring caller's frame/stack/result/output/code. This was
masked until socket -async made it natural to register a `fileevent` from
inside a proc body (the typical async accept pattern).

---

## Suggested order

1. **Phase 1** — immediate Tcl wins, zero risk, proves the approach
2. **Phase 2** (`lib/fiber.sx`) — the interesting SX work, benefits all hosted languages
3. **Phase 3** (OCaml primitives) — quick practical completions
4. **Phase 4** — architectural cleanup when it's worth the invasiveness

Phases 1+2+3 ≈ one focused week. Tcl is genuinely complete, and `lib/fiber.sx`
becomes a lasting SX contribution used by every future hosted language.

---

## Progress log

_Newest first._

- 2026-05-07: Phase 5f socket -async — socket-connect-async (Unix.set_nonblock+connect/EINPROGRESS) + channel-async-error (getsockopt_error); Tcl `socket -async host port` returns immediately; `fconfigure $sock -error` queries async error; +3 idiom tests; 376/376 green
- 2026-05-07: Phase 5e clock options + scan — clock-format extended with tz arg (utc/local) + more specifiers; new clock-scan primitive with manual timegm; Tcl clock format/scan support -format/-timezone/-gmt; +5 idiom tests; 373/373 green
- 2026-05-07: Phase 5d file ops — file-size/mtime/isfile?/isdir?/readable?/writable?/stat/delete/mkdir/copy/rename SX primitives; Tcl file isfile/isdir/readable/writable/size/mtime/atime/type/mkdir/copy/rename/delete now real; +10 idiom tests; 368/368 green
- 2026-05-07: Phase 5c sockets — socket-connect/socket-server/socket-accept SX primitives wrapping Unix.socket/connect/bind/listen/accept; tcl-cmd-socket dispatches client (host port) vs server (-server cb port); server auto-registers fileevent → _sock-do-accept handler that calls user callback per accept; puts now dispatches "sockN" channels to channel-write too; +4 idiom tests; 358/358 green
- 2026-05-07: Phase 5b event loop — io-select-channels SX primitive + Tcl-side fileevent/after/vwait/update; tcl-event-step drives expired timers + Unix.select on registered channels; +5 idiom tests; 354/354 green
- 2026-05-07: Phase 5 channel I/O — 11 SX primitives (channel-open/close/read/read-line/write/flush/seek/tell/eof?/blocking?/set-blocking!) wrapping Unix.openfile/read/write/lseek/set_nonblock; tcl-cmd-open/close/read/gets-chan/seek/tell/flush rewritten + new tcl-cmd-fconfigure; tcl-cmd-puts dispatches on "fileN" arg; gets registration fixed; +7 idiom tests; 349/349 green
- 2026-05-06: Phase 4 env-as-value — current-env (special form via Sx_ref.register_special_form), eval-in-env (primitive in setup_evaluator_bridge), env-lookup + env-extend (in setup_env_operations); 5 idiom tests; 342/342 green
- 2026-05-06: Phase 3 OCaml primitives — file-read/write/append/exists?/glob + clock-seconds/milliseconds/format in sx_primitives.ml + unix dep; tcl-cmd-clock/file wired up; 337/337 green
- 2026-05-06: Phase 2 coroutine rewrite — `tcl-cmd-coroutine` now creates a `make-fiber`; `tcl-cmd-yield` calls `:coro-yield-fn` (threaded through interp); true suspension; 337/337 green
- 2026-05-06: Phase 2 fiber.sx — `make-fiber`/`fiber-resume`/`fiber-done?` using call/cc + set!; bidirectional value passing; generator and echo tests pass
- 2026-05-06: Phase 1 array — `tcl-cmd-array` get/set/names/size/exists/unset; frame-local key scanning with prefix `arrname(`; 337/337 tests green
- 2026-05-06: Phase 1 apply — `tcl-cmd-apply` wraps `tcl-call-proc`, parses `{args body}` funcList, full frame isolation; 329/329 tests green
- 2026-05-06: Phase 1 regexp/regsub — `tcl-cmd-regexp`/`tcl-cmd-regsub` wrapping `make-regexp`/`regexp-match`/`regexp-match-all`/`regexp-replace`/`regexp-replace-all`; -nocase/-all/-inline/-all flags; matchVar + subgroup capture; 329/329 tests green
- 2026-05-06: Phase 1 float expr — `tcl-num-float?`, `tcl-parse-num`, float-aware `tcl-apply-binop`/`tcl-apply-func`/unary-minus/`**`; `sqrt`/`floor`/`ceil`/`round`/`sin`/`cos`/`tan`/`pow`/`exp`/`log` all float-native; 329/329 tests green

---

## What stays out of scope

- `package require` of binary loadables (would need `Dynlink` + native ABI design)
- Full `clock format` locale (translated month/day names, `LC_TIME`-aware) — Phase 5e covers `-format`/`-timezone`/`-gmt` with English names
- Tk / GUI
- Threads (mapped to coroutines only, as planned)
- Server-mode `vwait` — Phase 5b event loop is scoped to script-mode; from inside a server-handled command it can't see sx_server's stdin scheduler