HS: E37 tokenizer API (+17 tests)
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 15s
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 15s
Implements hs-tokens-of, hs-stream-token, hs-stream-consume, hs-stream-has-more, hs-token-type, hs-token-value, hs-token-op?, hs-raw->api-token, hs-eof-sentinel in runtime.sx. Tokenizer emits whitespace tokens after the first content token; stream functions skip them for look-ahead and consume. Parser filters whitespace tokens at hs-parse entry. Dot/hash after close brackets split into PERIOD/POUND + IDENTIFIER. Template escape \$ produces literal $. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -9,7 +9,9 @@
|
||||
(fn
|
||||
(tokens src)
|
||||
(let
|
||||
((p 0) (tok-len (len tokens)))
|
||||
((tokens (filter (fn (t) (not (= (get t "type") "whitespace"))) tokens))
|
||||
(p 0)
|
||||
(tok-len (len (filter (fn (t) (not (= (get t "type") "whitespace"))) tokens))))
|
||||
(define tp (fn () (if (< p tok-len) (nth tokens p) nil)))
|
||||
(define
|
||||
tp-type
|
||||
|
||||
Reference in New Issue
Block a user