Step 18 (part 1): _hyperscript tokenizer — 38 tests

lib/hyperscript/tokenizer.sx — tokenizes real _hyperscript syntax into
typed token stream. Handles:
  Keywords (on, set, add, toggle, if, then, from, etc.)
  DOM literals (.class, #id, @attr, *style, :local, <sel/>)
  Strings (single/double quoted, escapes), template literals
  Numbers (integers, decimals, time units: 100ms, 2s)
  Operators (==, !=, +, -, 's possessive)
  Punctuation (parens, brackets, braces, commas, dots)
  Line comments (// to EOL)

Parser will disambiguate .name as class vs property access from context.
Possessive 's correctly distinguished from single-quote strings.

2952/2952 tests, zero failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-05 21:49:13 +00:00
parent 9552750c4f
commit 3336c4e957
3 changed files with 831 additions and 0 deletions

View File

@@ -1308,6 +1308,8 @@ let run_spec_tests env test_files =
load_module "freeze.sx" lib_dir;
load_module "content.sx" lib_dir;
load_module "parser-combinators.sx" lib_dir;
let hs_dir = Filename.concat lib_dir "hyperscript" in
load_module "tokenizer.sx" hs_dir;
load_module "types.sx" lib_dir;
load_module "sx-swap.sx" lib_dir;
(* Shared templates: TW styling engine *)