HS: tokenizer-stream API → 13 tests pass (-13 skips)
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 54s
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 54s
lib/hyperscript/tokenizer.sx — added cursor + follow-set wrapper over
the existing flat-list tokenize output:
hs-stream src → {:tokens :pos :follows :last-match :last-ws}
hs-stream-current s → next non-WS token (skips WS, captures :last-ws)
hs-stream-match s value → consume if value matches & not in follow set
hs-stream-match-type s ...types → consume if upstream type name matches
hs-stream-match-any s ...names → consume if value matches any name
hs-stream-match-any-op s ...ops → consume if op token & value matches
hs-stream-peek s value n → look n non-WS tokens ahead, no consume
hs-stream-consume-until s marker → collect tokens until marker
hs-stream-consume-until-ws s → collect until next whitespace
hs-stream-push-follow! / pop-follow!
hs-stream-push-follows! / pop-follows! n
hs-stream-clear-follows! → saved / restore-follows! saved
hs-stream-last-match / last-ws
hs-stream-type-map maps our lowercase type names to upstream's
("ident" → "IDENTIFIER", "number" → "NUMBER", etc.) so type-based
matching works against upstream test expectations.
13 tokenizer-stream tests now pass; 30/30 in hs-upstream-core/tokenizer.
Skips remaining: 5 (down from 18).
- 2 template-component scope tests
- 1 async event dispatch (until event keyword works)
- left for later: needs more architectural work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -967,25 +967,6 @@ for(let i=startTest;i<Math.min(endTest,testCount);i++){
|
||||
// 'repeat until event' loop suspends the OCaml kernel waiting for an
|
||||
// event that is never fired from outside the K.eval call chain.
|
||||
"until event keyword works",
|
||||
// === Tokenizer-stream API tests (13) — upstream exposes a streaming token
|
||||
// API on _hyperscript.internals.tokenizer (matchToken, peekToken, consumeUntil,
|
||||
// pushFollow, etc.). Our lib/hyperscript/tokenizer.sx returns a flat token list
|
||||
// and the parser keeps stream state internally as closures. Making these tests
|
||||
// pass would require exposing a token-stream wrapper as a primitive. The
|
||||
// tokenizer is correct; it just doesn't expose this API surface. ===
|
||||
"matchToken consumes and returns on match",
|
||||
"matchToken honors the follow set",
|
||||
"matchTokenType matches by type",
|
||||
"matchOpToken matches operators by value",
|
||||
"matchAnyToken and matchAnyOpToken try each option",
|
||||
"peekToken skips whitespace when looking ahead",
|
||||
"consumeUntil collects tokens up to a marker",
|
||||
"consumeUntilWhitespace stops at first whitespace",
|
||||
"pushFollow/popFollow nest follow-set boundaries",
|
||||
"pushFollows/popFollows push and pop in bulk",
|
||||
"clearFollows/restoreFollows round-trip the follow set",
|
||||
"lastMatch returns the last consumed token",
|
||||
"lastWhitespace reflects whitespace before the current token",
|
||||
// === Template-component scope tests (2) — upstream uses
|
||||
// <script type="text/hyperscript-template" component="...">
|
||||
// for HTML-template-based custom-element components. Our defcomp uses SX
|
||||
|
||||
Reference in New Issue
Block a user