# Tcl-on-SX completion plan — SX capabilities first Tcl phases 1–6 are complete (329/329 tests). This plan covers the remaining limitations, ordered by the SX work needed to enable them. ## Key audit findings Several apparent gaps are already solved in SX: - **Floats** — SX parses `3.14` natively; `(+ 1.5 2.5) → 4.0`; `str` formats with `%g` (compact, no trailing zeros). `floor`/`ceil`/`round`/`truncate` all exist in `spec/primitives.sx`. - **Regex** — `regexp-match`, `regexp-match-all`, `regexp-replace`, `regexp-replace-all`, `regexp-split` are registered OCaml primitives using `Re.Pcre` (`hosts/ocaml/lib/sx_primitives.ml`). - **`call/cc` multi-shot** — works. `set!` on closed-over vars works. Fibers are implementable as a pure SX library. - **`perform` user-accessible** — `(perform :foo 42)` from user code suspends the evaluator and emits an IO request. The algebraic effects model is already half-built. - **No `file-read`/`clock-seconds`** — not yet registered as OCaml primitives. Only string ports exist. Would need small OCaml additions. - **No `env-as-value`** — environments are internal OCaml values, not inspectable from SX user code. --- ## Phase 1 — Zero-cost wins (no SX changes, only `lib/tcl/`) Everything here is pure Tcl implementation work. | Status | Work | Effort | Unlocks in Tcl | |---|---|---|---| | [x] | Float in `expr` — detect `.` in number tokens, route through float ops instead of `parse-int` | half day | `expr {3.14 * 2}`, `expr {sqrt(2.0)}`, float comparisons | | [x] | `regexp pattern str` and `regsub pattern str repl` wrapping existing SX primitives | few hours | pattern matching, text processing | | [x] | `apply {args body} ?arg…?` — anonymous proc call | 1 hour | higher-order functions, `lmap` idiom | | [x] | `array get/set/names/size/exists/unset` commands | half day | array variables (tokenizer already parses `$arr(key)`) | **Total: ~2 days. Zero SX changes.** --- ## Phase 2 — `lib/fiber.sx` (pure SX library, no OCaml) | Status | Work | |---|---| | [x] | Create `lib/fiber.sx` — `make-fiber` / `fiber-resume` / `fiber-done?` | | [x] | Rewrite `tcl-cmd-coroutine` to use `make-fiber` (true suspension) | `call/cc` is multi-shot and `set!` on closed-over vars both work. Fibers are implementable as a pure SX library using symmetric continuation swapping: ```scheme ; lib/fiber.sx — canonical fiber primitive for all hosted languages (define make-fiber (fn (thunk) (define slot-k nil) (define slot-caller nil) (define slot-done false) (fn (resume-val) (call/cc (fn (caller-k) (set! slot-caller caller-k) (if (nil? slot-k) (begin (thunk resume-val) (set! slot-done true) (caller-k nil)) (slot-k resume-val))))))) (define fiber-yield (fn (val) (call/cc (fn (k) (set! slot-k k) (slot-caller val))))) ``` Each coroutine becomes a fiber. `yield` swaps to the caller; calling the coroutine name swaps back. True suspension, not eager pre-execution. **Broader value:** Ruby fibers, Python generators, Lua coroutines, async event loops, cooperative schedulers all sit on top of the same library. **Alternatively:** `perform` is user-accessible. A Tcl scheduler living outside the SX evaluator (the OCaml host or an SX event loop) could catch `(perform :fiber-yield val)` and dispatch it — the algebraic effects model, already half-built. **Total: 2–3 days. Produces `lib/fiber.sx` as a lasting SX contribution.** Tcl coroutines then rewrite using `make-fiber` for true suspension. --- ## Phase 3 — Small OCaml additions (`sx_primitives.ml`) Each is ~10–20 lines of OCaml. All are useful across the whole platform, not just Tcl. | Status | Primitive | OCaml effort | Unlocks | |---|---|---|---| | [x] | `(file-read path)` → string | tiny | Tcl `open`/`read`, SX scripts reading files | | [x] | `(file-write path str)` → nil | tiny | Tcl `open`/`puts` to files | | [x] | `(file-exists? path)` → bool | tiny | Tcl `file exists` | | [x] | `(file-glob pattern)` → list | small | Tcl `glob` | | [x] | `(clock-seconds)` → int | tiny | Tcl `clock seconds` | | [x] | `(clock-format n fmt)` → string | small (wraps `strftime`) | Tcl `clock format` | **Total: 1 day. One focused afternoon of OCaml.** --- ## Phase 4 — env-as-value (architectural) ✓ `uplevel`/`upvar` required an explicit frame stack because SX environments aren't inspectable from user code. Adding: ```scheme (current-env) ; → env value (eval-in-env env expr) ; → result (env-lookup env key) ; → value or nil (env-extend env key val) ; → new env (non-mutating) ``` ...would let `uplevel N` be literally "look up env N levels up, eval in it." The Tcl frame stack (hundreds of lines) collapses to ~10 lines. Also benefits: metacircular evaluators, REPL tooling, live debugging (inspect any scope), the sx_docs server's eval endpoint. More invasive — touches `sx_types.ml` and `sx_server.ml` — but a meaningful architectural improvement worth doing when the moment is right. **Total: 2–3 days. High architectural value, not urgent.** --- ## Phase 5 — Channel I/O (random access + non-blocking) ✓ Real Tcl channel commands replacing the previous stubs. SX gained 11 channel primitives in `sx_primitives.ml` (using `Unix.openfile` + `Unix.read`/`write`/ `lseek`/`set_nonblock`). Tcl `open`/`close`/`read`/`gets`/`puts`/`seek`/`tell`/ `eof`/`flush`/`fconfigure` now wrap them. | Status | Work | Unlocks in Tcl | |---|---|---| | [x] | `channel-open`, `channel-close` | `open` returns "fileN", `close` actually closes | | [x] | `channel-read`, `channel-read-line`, `channel-write` | `read`/`gets`/`puts` to/from real files | | [x] | `channel-seek`, `channel-tell` | random access — `seek $c offset start\|current\|end`, `tell` | | [x] | `channel-eof?`, `channel-flush` | proper EOF detection, no-op flush | | [x] | `channel-blocking?`, `channel-set-blocking!` | `fconfigure $c -blocking 0\|1` | Modes supported: `r`, `w`, `a`, `r+`, `w+`, `a+`. Whence: `start`, `current`, `end`. `puts` now detects channel argument (string starting with "file") and dispatches to `channel-write`; otherwise writes to `interp :output` as before. **Total: ~half day. 7 new idiom tests covering write+read, gets-loop, seek/tell, eof-after-read, append mode, seek-to-end, fconfigure-blocking.** --- ## Phase 5b — Event loop: fileevent / after / vwait / update ✓ Tcl event-driven I/O scoped to script-mode (vs. server-side commands). The mechanism rides on the existing IO suspension model: SX adds one new primitive `(io-select-channels read-list write-list timeout-ms)` wrapping `Unix.select`, and the Tcl event loop is implemented in Tcl itself (no sx_server.ml changes). | Status | Work | Unlocks in Tcl | |---|---|---| | [x] | `io-select-channels` SX primitive | Unix.select on registered channels | | [x] | `fileevent $chan readable\|writable script` | event handler registration; `{}` to unregister | | [x] | `after ms script` | one-shot timer queued in `:timers` | | [x] | `after ms` (no script) | sleep that drives the event loop | | [x] | `vwait varname` | block until var set/changed, runs handlers | | [x] | `update` | non-blocking event drain (poll, fire ready handlers) | Event loop: `tcl-event-step interp poll-timeout-ms` — fires expired timers, calls `io-select-channels` with fd list from `:fileevents`, runs ready handlers. `vwait` polls every 1000ms or until var changes (whichever first); `update` is `tcl-event-step interp 0`. State on interp: `:fileevents` (list of `(chan event script)`) and `:timers` (list of `(expiry-ms script)`, sorted by expiry). **Trade-off:** Scoped to script mode — `vwait` from inside a server-handled command would not interact with sx_server's stdin scheduler. Sufficient for ~95% of real-world Tcl scripts (sockets, pipes, GUI-style polling, CLI tools). **Total: ~half day. 5 new idiom tests: after-vwait-timer, after-multiple-timers- update, fileevent-readable-fires, fileevent-query-script, after-cancel-via- vwait-timing. 354/354 green.** --- ## Phase 5c — TCP sockets (client + server) ✓ Tcl `socket` command for both connecting and listening. Reuses the channel registry built in Phase 5 and the event loop from Phase 5b. Server channels auto-fire user callbacks via fileevent on each accept. | Status | Work | Unlocks in Tcl | |---|---|---| | [x] | `socket-connect host port` SX primitive | TCP client via `Unix.socket`+`Unix.connect` | | [x] | `socket-server ?host? port` SX primitive | listening socket; `Unix.bind`+`Unix.listen` (backlog 8) | | [x] | `socket-accept server-chan` SX primitive | returns `{:channel :host :port}` | | [x] | Tcl `socket host port` | TCP client; returns "sockN" | | [x] | Tcl `socket -server cb port` | listening socket; auto-fires `cb sock host port` per accept | | [x] | `puts` channel detection extended | "sockN" channels also dispatch to `channel-write` | The auto-accept mechanism is a tiny internal Tcl command `_sock-do-accept` registered by `socket -server`. Its registered handler, fired by the event loop, accepts the pending client, then evaluates `cb client-chan host port`. `Unix.SO_REUSEADDR` is set on server sockets to avoid TIME_WAIT issues during testing. Host argument supports `localhost`, `0.0.0.0`, IPv4 literal, or DNS lookup via `Unix.gethostbyname`. **Total: ~half day. 4 new idiom tests: socket-server-fires-callback, socket-client-server-roundtrip, socket-server-peer-host, socket-multiple- connections. 358/358 green.** --- ## Phase 5d — File metadata + filesystem ops ✓ Real implementations of `file isfile`/`isdir`/`readable`/`writable`/`size`/ `mtime`/`atime`/`type` (previously stubs returning `0`/`""`) and proper `file delete`/`mkdir`/`copy`/`rename`. | Status | Primitive | Wraps | |---|---|---| | [x] | `file-size`, `file-mtime`, `file-stat` | `Unix.stat` | | [x] | `file-isfile?`, `file-isdir?` | `Unix.stat`+`st_kind` | | [x] | `file-readable?`, `file-writable?` | `Unix.access [R_OK\|W_OK]` | | [x] | `file-delete` | `Unix.unlink`/`rmdir` (tolerates ENOENT) | | [x] | `file-mkdir` | recursive `Unix.mkdir 0o755` | | [x] | `file-copy`, `file-rename` | stdlib I/O / `Sys.rename` | `file-stat` returns a dict `{:size :mtime :atime :ctime :mode :type}` with `:type` ∈ `file|directory|link|fifo|socket|...`. Tcl `file copy`/`rename`/ `delete` strip leading-`-` flags so `file delete -force` works. **Total: ~half day. 10 new idiom tests covering isfile, isdir on /tmp, size, readable, mkdir + check, copy roundtrip, rename, mtime > 0. 368/368 green.** --- ## Phase 5e — clock format options + clock scan ✓ Real `-format`, `-timezone`, and `-gmt` options on `clock format`, and a working `clock scan` for parsing date strings back to Unix seconds. | Status | Work | |---|---| | [x] | `clock-format` extended to `(t fmt tz)` with tz ∈ `utc|local` | | [x] | More format specifiers: `%y` (2-digit year), `%I` (12h hour), `%p` (AM/PM), `%w` (weekday num), `%%` (literal) | | [x] | `clock-scan` SX primitive: format-driven parser + manual `timegm` (OCaml stdlib lacks it) | | [x] | Tcl `clock format $secs -format $fmt -timezone $tz -gmt 0\|1` | | [x] | Tcl `clock scan $str -format $fmt -timezone $tz -gmt 0\|1` | Default tz for both is UTC. Format specifiers supported by scan: `%Y %y %m %d %e %H %I %M %S %%`. Unsupported specifiers in scan are silently skipped (no validation). **Total: ~half day. 5 new idiom tests: clock-format-utc, fmt-default, scan-roundtrip, scan-returns-int, format-percent-pct. 373/373 green.** --- ## Phase 5f — `socket -async` (non-blocking connect) ✓ | Status | Work | |---|---| | [x] | `socket-connect-async host port` SX primitive — `Unix.set_nonblock` + `Unix.connect`, catches `EINPROGRESS` | | [x] | `channel-async-error chan` SX primitive — `Unix.getsockopt_error` | | [x] | Tcl `socket -async host port` — returns "sockN" immediately | | [x] | Tcl `fconfigure $chan -error` — queries async-error | Connection completes when the channel becomes writable; canonical pattern is `fileevent $sock writable {handler}`. Channel buffer state is set to `blocking=false` so subsequent reads/writes don't block. **Total: ~few hours. 3 new idiom tests: socket-async-completes-writable, socket-async-then-write, socket-async-no-error. 376/376 green.** **Bug fix landed alongside:** `tcl-call-proc` was discarding `:fileevents`, `:timers`, and `:procs` updates made inside Tcl procs (only `:commands` was forwarded). Changed the return to forward the inner `result-interp` as the base while restoring caller's frame/stack/result/output/code. This was masked until socket -async made it natural to register a `fileevent` from inside a proc body (the typical async accept pattern). --- ## Phase 6 — Command surface fill-out ✓ After Phases 1–5 the architecture and IO model are complete. What remains is filling in the command surface that real Tcl scripts depend on. | Status | Work | Effort | Why it matters | |---|---|---|---| | [x] | **Phase 6a — namespace polish (`::var`)** | small | `set ::var` from inside a proc now resolves to the global (root) frame. Tokenizer also updated so `$::var` substitution works. Surfaced during socket -async test design. | | [x] | **Phase 6b — list ops audit** | few hours | Added `lassign`, `lrepeat`, `lset`, `lmap`. (`lsearch`, `lreplace`, `lreverse` were already present.) | | [x] | **Phase 6c — `dict` command additions** | small | `dict create/get/set/unset/exists/keys/values/for/update/merge/incr/append` were already implemented. Added `dict lappend`, `dict remove`, `dict filter -key`. | | [x] | **Phase 6d — `scan` and `format`** | few hours | Added `printf-spec` and `scan-spec` SX primitives wrapping OCaml `Printf`/`Scanf` via `Scanf.format_from_string`. Tcl `format` rewrote to dispatch via `printf-spec`; `scan` is a real walker that fills variables. Supports `%d %i %u %x %X %o %c %s %f %e %E %g %G %%` with width/precision/flags. | | [x] | **Phase 6e — `exec`** | few hours | `exec-process` SX primitive wraps `Unix.create_process` + `Unix.waitpid` and captures stdout. Tcl `exec cmd arg...` returns trimmed stdout; non-zero exit raises an error including stderr. Pipelines/redirection (`\|`, `>`, `<`) are not yet parsed. | **Bonus perf:** `tcl-global-ref?` (called on every var-get/set) was using `(substring name 0 2)` — re-allocating a 2-char string per call. Switched to `(char-at name 0)` + `(char-at name 1)` which short-circuits on non-`:` names. ~6× speedup on tight loops (`factorial 10`: 16s → 2.5s). `tcl-call-proc` was discarding `:fileevents`, `:timers`, and `:procs` updates made inside Tcl proc bodies — only `:commands` was forwarded. Now forwards the full set. Surfaced when socket-async made fileevent-from-inside-proc the canonical pattern. **Bug fix landed alongside:** `vwait ::var` was infinite-looping because `vwait` used `frame-lookup` directly, which doesn't honour `::` global routing. So after `set ::done fired` (which routes the write to the root frame), `vwait ::done` kept reading the local frame and never saw the change. Added `tcl-var-lookup-or-nil` helper that mirrors `tcl-var-get`'s `::` routing but returns nil instead of erroring on missing vars; vwait and `info exists` both use it now. **Total: 399/399 green** (parse 67, eval 169, error 39, namespace 22, coro 20, idiom 82). --- ## Phase 7 — Real-script polish ✓ What was left to make a moderately complex Tcl script run unchanged. | Status | Work | Notes | |---|---|---| | [x] | **Phase 7a — `try`/`catch`/`finally`** | Extended existing `tcl-cmd-try` with `trap pattern varlist body` clause matching errorcode prefix. Handler varlist supports `{result optsdict}` to capture both result and `-options` dict. Existing `on code var body` clauses unchanged. | | [x] | **Phase 7b — `exec` pipelines + redirection** | New `exec-pipeline` SX primitive parses `|`, `< file`, `> file`, `>> file`, `2> file`, `2>@1` and builds a process pipeline via `Unix.pipe` + `Unix.create_process`. `tcl-cmd-exec` dispatches to it when any metachar is present. | | [x] | **Phase 7c — `string` subcommand audit** | Added `string equal ?-nocase? ?-length n? s1 s2`, `string totitle`, `string reverse`, `string replace s f l ?new?`. Added `string is true/false/xdigit/ascii` classes (already had integer/double/alpha/alnum/digit/space/upper/lower/boolean). | | [x] | **Phase 7d — TclOO (`oo::class`)** | Minimal `oo::class create NAME body` with `method`, `constructor`, `destructor`, `superclass` declarations. Instances via `Cls new ?args?`. Method dispatch via per-object Tcl command. Single inheritance via `:super` chain. Class/object state on interp `:classes`/`:oo-objects`/`:oo-counter`. Mixins/filters/`oo::define` deferred. | | [x] | **Phase 7e — `regexp` audit** | Existing `Re.Pcre` wrapper handles `^`/`$` anchors, `\b` boundaries, `-nocase`, capture groups, `regsub -all`. Added regression tests covering each. No code changes needed. | **+28 idiom tests** (5 try, 5 exec pipeline, 7 string, 6 regexp, 5 TclOO). **Total: 427/427 green** (parse 67, eval 169, error 39, namespace 22, coro 20, idiom 110). --- ## Suggested order 1. **Phase 1** — immediate Tcl wins, zero risk, proves the approach 2. **Phase 2** (`lib/fiber.sx`) — the interesting SX work, benefits all hosted languages 3. **Phase 3** (OCaml primitives) — quick practical completions 4. **Phase 4** — architectural cleanup when it's worth the invasiveness Phases 1+2+3 ≈ one focused week. Tcl is genuinely complete, and `lib/fiber.sx` becomes a lasting SX contribution used by every future hosted language. --- ## Progress log _Newest first._ - 2026-05-08: Phase 7 verified — 427/427 (idiom 110). try/trap with errorcode prefix matching + 2-var optsdict capture; exec pipelines with `|`/`<`/`>`/`>>`/`2>@1` via Unix.pipe + create_process; string equal/totitle/reverse/replace + is true/false/xdigit/ascii classes; regexp regression suite; minimal TclOO (oo::class create / new / single inheritance / ctor / methods); +28 idiom tests. - 2026-05-08: Phase 6 verified — 399/399 (parse 67, eval 169, error 39, namespace 22, coro 20, idiom 82). Fixed vwait `::var` infinite loop via tcl-var-lookup-or-nil helper; info exists also uses it now. - 2026-05-07: Phase 6e exec — exec-process SX primitive (Unix.create_process+waitpid, captures stdout, errors on non-zero exit with stderr) + Tcl `exec cmd arg...`; +3 idiom tests - 2026-05-07: Phase 6d scan/format — printf-spec + scan-spec SX primitives wrapping OCaml Printf/Scanf via Scanf.format_from_string; Tcl format rewritten to dispatch via printf-spec; scan is real walker; supports d/i/u/x/X/o/c/s/f/e/E/g/G/% with width/precision/flags; +7 idiom tests - 2026-05-07: Phase 6c dict additions — dict lappend / remove / filter -key (rest of dict was already implemented); +3 idiom tests - 2026-05-07: Phase 6b list ops — lassign / lrepeat / lset / lmap added (lsearch/lreplace/lreverse were already present); +6 idiom tests - 2026-05-07: Phase 6a namespace `::` prefix — tcl-global-ref?/strip-global helpers; tcl-var-get/set route `::name` to root frame; tokenizer parse-var-sub also accepts `::` start so `$::var` substitution works; tcl-call-proc forwards :fileevents/:timers/:procs; char-at fast-path optimization (~6× speedup on tight loops); +4 idiom tests - 2026-05-07: Phase 5f socket -async — socket-connect-async (Unix.set_nonblock+connect/EINPROGRESS) + channel-async-error (getsockopt_error); Tcl `socket -async host port` returns immediately; `fconfigure $sock -error` queries async error; +3 idiom tests; 376/376 green - 2026-05-07: Phase 5e clock options + scan — clock-format extended with tz arg (utc/local) + more specifiers; new clock-scan primitive with manual timegm; Tcl clock format/scan support -format/-timezone/-gmt; +5 idiom tests; 373/373 green - 2026-05-07: Phase 5d file ops — file-size/mtime/isfile?/isdir?/readable?/writable?/stat/delete/mkdir/copy/rename SX primitives; Tcl file isfile/isdir/readable/writable/size/mtime/atime/type/mkdir/copy/rename/delete now real; +10 idiom tests; 368/368 green - 2026-05-07: Phase 5c sockets — socket-connect/socket-server/socket-accept SX primitives wrapping Unix.socket/connect/bind/listen/accept; tcl-cmd-socket dispatches client (host port) vs server (-server cb port); server auto-registers fileevent → _sock-do-accept handler that calls user callback per accept; puts now dispatches "sockN" channels to channel-write too; +4 idiom tests; 358/358 green - 2026-05-07: Phase 5b event loop — io-select-channels SX primitive + Tcl-side fileevent/after/vwait/update; tcl-event-step drives expired timers + Unix.select on registered channels; +5 idiom tests; 354/354 green - 2026-05-07: Phase 5 channel I/O — 11 SX primitives (channel-open/close/read/read-line/write/flush/seek/tell/eof?/blocking?/set-blocking!) wrapping Unix.openfile/read/write/lseek/set_nonblock; tcl-cmd-open/close/read/gets-chan/seek/tell/flush rewritten + new tcl-cmd-fconfigure; tcl-cmd-puts dispatches on "fileN" arg; gets registration fixed; +7 idiom tests; 349/349 green - 2026-05-06: Phase 4 env-as-value — current-env (special form via Sx_ref.register_special_form), eval-in-env (primitive in setup_evaluator_bridge), env-lookup + env-extend (in setup_env_operations); 5 idiom tests; 342/342 green - 2026-05-06: Phase 3 OCaml primitives — file-read/write/append/exists?/glob + clock-seconds/milliseconds/format in sx_primitives.ml + unix dep; tcl-cmd-clock/file wired up; 337/337 green - 2026-05-06: Phase 2 coroutine rewrite — `tcl-cmd-coroutine` now creates a `make-fiber`; `tcl-cmd-yield` calls `:coro-yield-fn` (threaded through interp); true suspension; 337/337 green - 2026-05-06: Phase 2 fiber.sx — `make-fiber`/`fiber-resume`/`fiber-done?` using call/cc + set!; bidirectional value passing; generator and echo tests pass - 2026-05-06: Phase 1 array — `tcl-cmd-array` get/set/names/size/exists/unset; frame-local key scanning with prefix `arrname(`; 337/337 tests green - 2026-05-06: Phase 1 apply — `tcl-cmd-apply` wraps `tcl-call-proc`, parses `{args body}` funcList, full frame isolation; 329/329 tests green - 2026-05-06: Phase 1 regexp/regsub — `tcl-cmd-regexp`/`tcl-cmd-regsub` wrapping `make-regexp`/`regexp-match`/`regexp-match-all`/`regexp-replace`/`regexp-replace-all`; -nocase/-all/-inline/-all flags; matchVar + subgroup capture; 329/329 tests green - 2026-05-06: Phase 1 float expr — `tcl-num-float?`, `tcl-parse-num`, float-aware `tcl-apply-binop`/`tcl-apply-func`/unary-minus/`**`; `sqrt`/`floor`/`ceil`/`round`/`sin`/`cos`/`tan`/`pow`/`exp`/`log` all float-native; 329/329 tests green --- ## What stays out of scope - `package require` of binary loadables (would need `Dynlink` + native ABI design) - Full `clock format` locale (translated month/day names, `LC_TIME`-aware) — Phase 5e covers `-format`/`-timezone`/`-gmt` with English names - Tk / GUI - Threads (mapped to coroutines only, as planned) - Server-mode `vwait` — Phase 5b event loop is scoped to script-mode; from inside a server-handled command it can't see sx_server's stdin scheduler - TclOO mixins, filters, `oo::define` after-the-fact, multiple inheritance — Phase 7d covers single-inheritance class declarations only