coop/rose-ash
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HS: E37 tokenizer API (+17 tests)
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 15s
Details

Browse Source 
Implements hs-tokens-of, hs-stream-token, hs-stream-consume,
hs-stream-has-more, hs-token-type, hs-token-value, hs-token-op?,
hs-raw->api-token, hs-eof-sentinel in runtime.sx.

Tokenizer emits whitespace tokens after the first content token;
stream functions skip them for look-ahead and consume. Parser
filters whitespace tokens at hs-parse entry. Dot/hash after close
brackets split into PERIOD/POUND + IDENTIFIER. Template escape \$
produces literal $.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
giles
2026-04-26 19:54:40 +00:00
parent 92adf9d496
commit 54b54f4e19
8 changed files with 360 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
lib/hyperscript/compiler.sx
View File

@@ -895,9 +895,15 @@
(let
((ch (nth raw i)))
(if
(and (= ch "$") (< (+ i 1) n))
(and (= ch "\\") (< (+ i 1) n) (= (nth raw (+ i 1)) "$"))
(do
(set! buf (str buf "$"))
(set! i (+ i 2))
(tpl-collect))
(if
(= (nth raw (+ i 1)) "{")
(and (= ch "$") (< (+ i 1) n))
(if
(= (nth raw (+ i 1)) "{")
(let
((start (+ i 2)))
(let
@@ -931,10 +937,10 @@
(hs-to-sx (hs-compile ident)))))
(set! i end)
(tpl-collect))))))
(do
(set! buf (str buf ch))
(set! i (+ i 1))
(tpl-collect)))))))
(do
(set! buf (str buf ch))
(set! i (+ i 1))
(tpl-collect))))))))
(tpl-collect)
(tpl-flush)
(cons (quote str) parts))))