HS: E37 tokenizer API (+17 tests)
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 15s

Implements hs-tokens-of, hs-stream-token, hs-stream-consume,
hs-stream-has-more, hs-token-type, hs-token-value, hs-token-op?,
hs-raw->api-token, hs-eof-sentinel in runtime.sx.

Tokenizer emits whitespace tokens after the first content token;
stream functions skip them for look-ahead and consume. Parser
filters whitespace tokens at hs-parse entry. Dot/hash after close
brackets split into PERIOD/POUND + IDENTIFIER. Template escape \$
produces literal $.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-04-26 19:54:40 +00:00
parent 92adf9d496
commit 54b54f4e19
8 changed files with 360 additions and 16 deletions

View File

@@ -9,7 +9,9 @@
(fn
(tokens src)
(let
((p 0) (tok-len (len tokens)))
((tokens (filter (fn (t) (not (= (get t "type") "whitespace"))) tokens))
(p 0)
(tok-len (len (filter (fn (t) (not (= (get t "type") "whitespace"))) tokens))))
(define tp (fn () (if (< p tok-len) (nth tokens p) nil)))
(define
tp-type