giles 023bc2d80c sx: step 3 — add :end and :line to hs tokenizer tokens ...

Extend hs-make-token to (type value pos &rest extras) producing dicts
{:pos :end :line :value :type}. End defaults to pos+len(value); line
defaults to 1. Both tokenize loops now track current-line via newline
counting in advance!. hs-emit! and t-emit! pass the right end and
start-line to the constructor; redundant dict-set! after construction
removed.

Mirror copied to shared/static/wasm/sx/hs-tokenizer.sx (byte-identical).

Verify: (hs-make-token "NUMBER" "1" 0) returns
  {:pos 0 :end 1 :line 1 :value "1" :type "NUMBER"}.

OCaml suite: 4529 pass, 1339 pre-existing failures (baseline). All
4/4 hs-upstream-core/sourceInfo tests now pass (was 2/4 — closes E38).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-06 22:14:10 +00:00

27 KiB

Raw Blame History

View Raw