sx: step 3 — add :end and :line to hs tokenizer tokens

Extend hs-make-token to (type value pos &rest extras) producing dicts
{:pos :end :line :value :type}. End defaults to pos+len(value); line
defaults to 1. Both tokenize loops now track current-line via newline
counting in advance!. hs-emit! and t-emit! pass the right end and
start-line to the constructor; redundant dict-set! after construction
removed.

Mirror copied to shared/static/wasm/sx/hs-tokenizer.sx (byte-identical).

Verify: (hs-make-token "NUMBER" "1" 0) returns
  {:pos 0 :end 1 :line 1 :value "1" :type "NUMBER"}.

OCaml suite: 4529 pass, 1339 pre-existing failures (baseline). All
4/4 hs-upstream-core/sourceInfo tests now pass (was 2/4 — closes E38).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-06 22:14:10 +00:00
parent ccf8a0fb90
commit 023bc2d80c
3 changed files with 113 additions and 28 deletions

View File

@@ -183,7 +183,7 @@ these when operands are known numbers/lists.
|------|--------|--------|
| 1 — JIT combinator bug | [x] | 882a4b76 |
| 2 — letrec+resume | [x] | e80e655b |
| 3 — tokenizer :end/:line | [ ] | |
| 3 — tokenizer :end/:line | [x] | (pending) |
| 4 — parser spans complete | [ ] | — |
| 5 — OCaml AdtValue + define-type + match | [ ] | — |
| 6 — JS AdtValue + define-type + match | [ ] | — |