GUEST: step 3 — lib/guest/lex.sx character-class + token primitives
Extracted shared tokeniser primitives:
- Char-class predicates: lex-digit?, lex-hex-digit?, lex-alpha?
(alias lex-letter?), lex-alnum?, lex-ident-start?, lex-ident-char?,
lex-space? (no newline), lex-whitespace? (incl newline). All nil-safe.
- Token record: lex-make-token, lex-make-token-spanning, accessors.
Ported lib/lua/tokenizer.sx and lib/tcl/tokenizer.sx — 7 lua and 5 tcl
predicate definitions collapsed into prefix-rename calls that alias
lua-/tcl- names to lex- primitives. Test scripts (lua/test.sh,
tcl/test.sh, tcl/conformance.sh) load lib/guest/lex.sx and prefix.sx
before the per-language tokenizer.
Verification:
- lua/test.sh: 185/185 = baseline
- tcl/test.sh: 342/342 (parse 67 + eval 169 + error 39 + namespace 22
+ coro 20 + idiom 25)
- tcl/conformance.sh: 3/4 = baseline (event-loop failure is pre-existing)
Two consumers verified — step complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -63,6 +63,8 @@ for tcl_file in "${TCL_FILES[@]}"; do
|
||||
# Build epoch input using quoted heredoc for static parts; helper path via variable
|
||||
cat > "$tmpfile" << EPOCHS
|
||||
(epoch 1)
|
||||
(load "lib/guest/lex.sx")
|
||||
(load "lib/guest/prefix.sx")
|
||||
(load "lib/tcl/tokenizer.sx")
|
||||
(epoch 2)
|
||||
(load "lib/tcl/parser.sx")
|
||||
|
||||
@@ -33,6 +33,8 @@ HELPER_EOF
|
||||
|
||||
cat > "$TMPFILE" << EPOCHS
|
||||
(epoch 1)
|
||||
(load "lib/guest/lex.sx")
|
||||
(load "lib/guest/prefix.sx")
|
||||
(load "lib/tcl/tokenizer.sx")
|
||||
(epoch 2)
|
||||
(load "lib/tcl/parser.sx")
|
||||
|
||||
@@ -1,19 +1,10 @@
|
||||
(define tcl-ws? (fn (c) (or (= c " ") (= c "\t") (= c "\r"))))
|
||||
(prefix-rename "tcl-"
|
||||
'((ws? lex-space?)
|
||||
(alpha? lex-alpha?)
|
||||
(digit? lex-digit?)
|
||||
(ident-start? lex-ident-start?)
|
||||
(ident-char? lex-ident-char?)))
|
||||
|
||||
(define tcl-alpha?
|
||||
(fn (c)
|
||||
(and
|
||||
(not (= c nil))
|
||||
(or (and (>= c "a") (<= c "z")) (and (>= c "A") (<= c "Z"))))))
|
||||
|
||||
(define tcl-digit?
|
||||
(fn (c) (and (not (= c nil)) (>= c "0") (<= c "9"))))
|
||||
|
||||
(define tcl-ident-start?
|
||||
(fn (c) (or (tcl-alpha? c) (= c "_"))))
|
||||
|
||||
(define tcl-ident-char?
|
||||
(fn (c) (or (tcl-ident-start? c) (tcl-digit? c))))
|
||||
|
||||
(define tcl-tokenize
|
||||
(fn (src)
|
||||
|
||||
Reference in New Issue
Block a user