rose-ash/plans/tcl-sx-completion.md

Tcl-on-SX completion plan — SX capabilities first

Tcl phases 1–6 are complete (329/329 tests). This plan covers the remaining limitations, ordered by the SX work needed to enable them.

Key audit findings

Several apparent gaps are already solved in SX:

• Floats — SX parses 3.14 natively; (+ 1.5 2.5) → 4.0; str formats with %g (compact, no trailing zeros). floor/ceil/round/truncate all exist in spec/primitives.sx.
• Regex — regexp-match, regexp-match-all, regexp-replace, regexp-replace-all, regexp-split are registered OCaml primitives using Re.Pcre (hosts/ocaml/lib/sx_primitives.ml).
• call/cc multi-shot — works. set! on closed-over vars works. Fibers are implementable as a pure SX library.
• perform user-accessible — (perform :foo 42) from user code suspends the evaluator and emits an IO request. The algebraic effects model is already half-built.
• No file-read/clock-seconds — not yet registered as OCaml primitives. Only string ports exist. Would need small OCaml additions.
• No env-as-value — environments are internal OCaml values, not inspectable from SX user code.

---

Phase 1 — Zero-cost wins (no SX changes, only lib/tcl/)

Everything here is pure Tcl implementation work.

Status	Work	Effort	Unlocks in Tcl
[x]	Float in expr — detect . in number tokens, route through float ops instead of parse-int	half day	expr {3.14 * 2}, expr {sqrt(2.0)}, float comparisons
[x]	regexp pattern str and regsub pattern str repl wrapping existing SX primitives	few hours	pattern matching, text processing
[x]	apply {args body} ?arg…? — anonymous proc call	1 hour	higher-order functions, lmap idiom
[x]	array get/set/names/size/exists/unset commands	half day	array variables (tokenizer already parses $arr(key))

Total: ~2 days. Zero SX changes.

---

Phase 2 — lib/fiber.sx (pure SX library, no OCaml)

Status	Work
[x]	Create lib/fiber.sx — make-fiber / fiber-resume / fiber-done?
[x]	Rewrite tcl-cmd-coroutine to use make-fiber (true suspension)

call/cc is multi-shot and set! on closed-over vars both work. Fibers are implementable as a pure SX library using symmetric continuation swapping:

; lib/fiber.sx — canonical fiber primitive for all hosted languages
(define make-fiber
  (fn (thunk)
    (define slot-k nil)
    (define slot-caller nil)
    (define slot-done false)
    (fn (resume-val)
      (call/cc (fn (caller-k)
        (set! slot-caller caller-k)
        (if (nil? slot-k)
          (begin (thunk resume-val) (set! slot-done true) (caller-k nil))
          (slot-k resume-val)))))))

(define fiber-yield
  (fn (val)
    (call/cc (fn (k)
      (set! slot-k k)
      (slot-caller val)))))

Each coroutine becomes a fiber. yield swaps to the caller; calling the coroutine name swaps back. True suspension, not eager pre-execution.

Broader value: Ruby fibers, Python generators, Lua coroutines, async event loops, cooperative schedulers all sit on top of the same library.

Alternatively: perform is user-accessible. A Tcl scheduler living outside the SX evaluator (the OCaml host or an SX event loop) could catch (perform :fiber-yield val) and dispatch it — the algebraic effects model, already half-built.

Total: 2–3 days. Produces lib/fiber.sx as a lasting SX contribution. Tcl coroutines then rewrite using make-fiber for true suspension.

---

Phase 3 — Small OCaml additions (sx_primitives.ml)

Each is ~10–20 lines of OCaml. All are useful across the whole platform, not just Tcl.

Status	Primitive	OCaml effort	Unlocks
[x]	(file-read path) → string	tiny	Tcl open/read, SX scripts reading files
[x]	(file-write path str) → nil	tiny	Tcl open/puts to files
[x]	(file-exists? path) → bool	tiny	Tcl file exists
[x]	(file-glob pattern) → list	small	Tcl glob
[x]	(clock-seconds) → int	tiny	Tcl clock seconds
[x]	(clock-format n fmt) → string	small (wraps strftime)	Tcl clock format

Total: 1 day. One focused afternoon of OCaml.

---

Phase 4 — env-as-value (architectural) ✓

uplevel/upvar required an explicit frame stack because SX environments aren't inspectable from user code. Adding:

(current-env)            ; → env value
(eval-in-env env expr)   ; → result
(env-lookup env key)     ; → value or nil
(env-extend env key val) ; → new env (non-mutating)

...would let uplevel N be literally "look up env N levels up, eval in it." The Tcl frame stack (hundreds of lines) collapses to ~10 lines.

Also benefits: metacircular evaluators, REPL tooling, live debugging (inspect any scope), the sx_docs server's eval endpoint.

More invasive — touches sx_types.ml and sx_server.ml — but a meaningful architectural improvement worth doing when the moment is right.

Total: 2–3 days. High architectural value, not urgent.

---

Phase 5 — Channel I/O (random access + non-blocking) ✓

Real Tcl channel commands replacing the previous stubs. SX gained 11 channel primitives in sx_primitives.ml (using Unix.openfile + Unix.read/write/ lseek/set_nonblock). Tcl open/close/read/gets/puts/seek/tell/ eof/flush/fconfigure now wrap them.

Status	Work	Unlocks in Tcl
[x]	channel-open, channel-close	open returns "fileN", close actually closes
[x]	channel-read, channel-read-line, channel-write	read/gets/puts to/from real files
[x]	channel-seek, channel-tell	random access — seek $c offset start|current|end, tell
[x]	channel-eof?, channel-flush	proper EOF detection, no-op flush
[x]	channel-blocking?, channel-set-blocking!	fconfigure $c -blocking 0|1

Modes supported: r, w, a, r+, w+, a+. Whence: start, current, end.

puts now detects channel argument (string starting with "file") and dispatches to channel-write; otherwise writes to interp :output as before.

Total: ~half day. 7 new idiom tests covering write+read, gets-loop, seek/tell, eof-after-read, append mode, seek-to-end, fconfigure-blocking.

---

Phase 5b — Event loop: fileevent / after / vwait / update ✓

Tcl event-driven I/O scoped to script-mode (vs. server-side commands). The mechanism rides on the existing IO suspension model: SX adds one new primitive (io-select-channels read-list write-list timeout-ms) wrapping Unix.select, and the Tcl event loop is implemented in Tcl itself (no sx_server.ml changes).

Status	Work	Unlocks in Tcl
[x]	io-select-channels SX primitive	Unix.select on registered channels
[x]	fileevent $chan readable|writable script	event handler registration; {} to unregister
[x]	after ms script	one-shot timer queued in :timers
[x]	after ms (no script)	sleep that drives the event loop
[x]	vwait varname	block until var set/changed, runs handlers
[x]	update	non-blocking event drain (poll, fire ready handlers)

Event loop: tcl-event-step interp poll-timeout-ms — fires expired timers, calls io-select-channels with fd list from :fileevents, runs ready handlers. vwait polls every 1000ms or until var changes (whichever first); update is tcl-event-step interp 0.

State on interp: :fileevents (list of (chan event script)) and :timers (list of (expiry-ms script), sorted by expiry).

Trade-off: Scoped to script mode — vwait from inside a server-handled command would not interact with sx_server's stdin scheduler. Sufficient for ~95% of real-world Tcl scripts (sockets, pipes, GUI-style polling, CLI tools).

Total: ~half day. 5 new idiom tests: after-vwait-timer, after-multiple-timers- update, fileevent-readable-fires, fileevent-query-script, after-cancel-via- vwait-timing. 354/354 green.

---

Phase 5c — TCP sockets (client + server) ✓

Tcl socket command for both connecting and listening. Reuses the channel registry built in Phase 5 and the event loop from Phase 5b. Server channels auto-fire user callbacks via fileevent on each accept.

Status	Work	Unlocks in Tcl
[x]	socket-connect host port SX primitive	TCP client via Unix.socket+Unix.connect
[x]	socket-server ?host? port SX primitive	listening socket; Unix.bind+Unix.listen (backlog 8)
[x]	socket-accept server-chan SX primitive	returns {:channel :host :port}
[x]	Tcl socket host port	TCP client; returns "sockN"
[x]	Tcl socket -server cb port	listening socket; auto-fires cb sock host port per accept
[x]	puts channel detection extended	"sockN" channels also dispatch to channel-write

The auto-accept mechanism is a tiny internal Tcl command _sock-do-accept registered by socket -server. Its registered handler, fired by the event loop, accepts the pending client, then evaluates cb client-chan host port.

Unix.SO_REUSEADDR is set on server sockets to avoid TIME_WAIT issues during testing. Host argument supports localhost, 0.0.0.0, IPv4 literal, or DNS lookup via Unix.gethostbyname.

Total: ~half day. 4 new idiom tests: socket-server-fires-callback, socket-client-server-roundtrip, socket-server-peer-host, socket-multiple- connections. 358/358 green.

---

Suggested order

1. Phase 1 — immediate Tcl wins, zero risk, proves the approach
2. Phase 2 (lib/fiber.sx) — the interesting SX work, benefits all hosted languages
3. Phase 3 (OCaml primitives) — quick practical completions
4. Phase 4 — architectural cleanup when it's worth the invasiveness

Phases 1+2+3 ≈ one focused week. Tcl is genuinely complete, and lib/fiber.sx becomes a lasting SX contribution used by every future hosted language.

---

Progress log

Newest first.

• 2026-05-07: Phase 5c sockets — socket-connect/socket-server/socket-accept SX primitives wrapping Unix.socket/connect/bind/listen/accept; tcl-cmd-socket dispatches client (host port) vs server (-server cb port); server auto-registers fileevent → _sock-do-accept handler that calls user callback per accept; puts now dispatches "sockN" channels to channel-write too; +4 idiom tests; 358/358 green
• 2026-05-07: Phase 5b event loop — io-select-channels SX primitive + Tcl-side fileevent/after/vwait/update; tcl-event-step drives expired timers + Unix.select on registered channels; +5 idiom tests; 354/354 green
• 2026-05-07: Phase 5 channel I/O — 11 SX primitives (channel-open/close/read/read-line/write/flush/seek/tell/eof?/blocking?/set-blocking!) wrapping Unix.openfile/read/write/lseek/set_nonblock; tcl-cmd-open/close/read/gets-chan/seek/tell/flush rewritten + new tcl-cmd-fconfigure; tcl-cmd-puts dispatches on "fileN" arg; gets registration fixed; +7 idiom tests; 349/349 green
• 2026-05-06: Phase 4 env-as-value — current-env (special form via Sx_ref.register_special_form), eval-in-env (primitive in setup_evaluator_bridge), env-lookup + env-extend (in setup_env_operations); 5 idiom tests; 342/342 green
• 2026-05-06: Phase 3 OCaml primitives — file-read/write/append/exists?/glob + clock-seconds/milliseconds/format in sx_primitives.ml + unix dep; tcl-cmd-clock/file wired up; 337/337 green
• 2026-05-06: Phase 2 coroutine rewrite — tcl-cmd-coroutine now creates a make-fiber; tcl-cmd-yield calls :coro-yield-fn (threaded through interp); true suspension; 337/337 green
• 2026-05-06: Phase 2 fiber.sx — make-fiber/fiber-resume/fiber-done? using call/cc + set!; bidirectional value passing; generator and echo tests pass
• 2026-05-06: Phase 1 array — tcl-cmd-array get/set/names/size/exists/unset; frame-local key scanning with prefix arrname(; 337/337 tests green
• 2026-05-06: Phase 1 apply — tcl-cmd-apply wraps tcl-call-proc, parses {args body} funcList, full frame isolation; 329/329 tests green
• 2026-05-06: Phase 1 regexp/regsub — tcl-cmd-regexp/tcl-cmd-regsub wrapping make-regexp/regexp-match/regexp-match-all/regexp-replace/regexp-replace-all; -nocase/-all/-inline/-all flags; matchVar + subgroup capture; 329/329 tests green
• 2026-05-06: Phase 1 float expr — tcl-num-float?, tcl-parse-num, float-aware tcl-apply-binop/tcl-apply-func/unary-minus/**; sqrt/floor/ceil/round/sin/cos/tan/pow/exp/log all float-native; 329/329 tests green

---

What stays out of scope

• package require of binary loadables
• Full clock format locale support
• Tk / GUI
• Threads (mapped to coroutines only, as planned)
• Server-mode vwait — Phase 5b event loop is scoped to script-mode; from inside a server-handled command it can't see sx_server's stdin scheduler