GUEST: step 3 — lib/guest/lex.sx character-class + token primitives

Extracted shared tokeniser primitives:
- Char-class predicates: lex-digit?, lex-hex-digit?, lex-alpha?
  (alias lex-letter?), lex-alnum?, lex-ident-start?, lex-ident-char?,
  lex-space? (no newline), lex-whitespace? (incl newline). All nil-safe.
- Token record: lex-make-token, lex-make-token-spanning, accessors.

Ported lib/lua/tokenizer.sx and lib/tcl/tokenizer.sx — 7 lua and 5 tcl
predicate definitions collapsed into prefix-rename calls that alias
lua-/tcl- names to lex- primitives. Test scripts (lua/test.sh,
tcl/test.sh, tcl/conformance.sh) load lib/guest/lex.sx and prefix.sx
before the per-language tokenizer.

Verification:
- lua/test.sh: 185/185 = baseline
- tcl/test.sh: 342/342 (parse 67 + eval 169 + error 39 + namespace 22
                       + coro 20 + idiom 25)
- tcl/conformance.sh: 3/4 = baseline (event-loop failure is pre-existing)

Two consumers verified — step complete.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-06 23:06:12 +00:00
parent ba9ab4e65a
commit 559b0df900
6 changed files with 87 additions and 42 deletions

View File

@@ -28,6 +28,8 @@ trap "rm -f $TMPFILE" EXIT
cat > "$TMPFILE" << 'EPOCHS'
(epoch 1)
(load "lib/guest/lex.sx")
(load "lib/guest/prefix.sx")
(load "lib/lua/tokenizer.sx")
(epoch 2)
(load "lib/lua/parser.sx")

View File

@@ -1,31 +1,12 @@
(define lua-make-token (fn (type value pos) {:pos pos :value value :type type}))
(prefix-rename "lua-"
'((make-token lex-make-token)
(digit? lex-digit?)
(hex-digit? lex-hex-digit?)
(letter? lex-alpha?)
(ident-start? lex-ident-start?)
(ident-char? lex-ident-char?)
(ws? lex-whitespace?)))
(define lua-digit? (fn (c) (and (not (= c nil)) (>= c "0") (<= c "9"))))
(define
lua-hex-digit?
(fn
(c)
(and
(not (= c nil))
(or
(lua-digit? c)
(and (>= c "a") (<= c "f"))
(and (>= c "A") (<= c "F"))))))
(define
lua-letter?
(fn
(c)
(and
(not (= c nil))
(or (and (>= c "a") (<= c "z")) (and (>= c "A") (<= c "Z"))))))
(define lua-ident-start? (fn (c) (or (lua-letter? c) (= c "_"))))
(define lua-ident-char? (fn (c) (or (lua-ident-start? c) (lua-digit? c))))
(define lua-ws? (fn (c) (or (= c " ") (= c "\t") (= c "\n") (= c "\r"))))
(define
lua-keywords