sx: step 3 — add :end and :line to hs tokenizer tokens
Extend hs-make-token to (type value pos &rest extras) producing dicts
{:pos :end :line :value :type}. End defaults to pos+len(value); line
defaults to 1. Both tokenize loops now track current-line via newline
counting in advance!. hs-emit! and t-emit! pass the right end and
start-line to the constructor; redundant dict-set! after construction
removed.
Mirror copied to shared/static/wasm/sx/hs-tokenizer.sx (byte-identical).
Verify: (hs-make-token "NUMBER" "1" 0) returns
{:pos 0 :end 1 :line 1 :value "1" :type "NUMBER"}.
OCaml suite: 4529 pass, 1339 pre-existing failures (baseline). All
4/4 hs-upstream-core/sourceInfo tests now pass (was 2/4 — closes E38).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -183,7 +183,7 @@ these when operands are known numbers/lists.
|
||||
|------|--------|--------|
|
||||
| 1 — JIT combinator bug | [x] | 882a4b76 |
|
||||
| 2 — letrec+resume | [x] | e80e655b |
|
||||
| 3 — tokenizer :end/:line | [ ] | — |
|
||||
| 3 — tokenizer :end/:line | [x] | (pending) |
|
||||
| 4 — parser spans complete | [ ] | — |
|
||||
| 5 — OCaml AdtValue + define-type + match | [ ] | — |
|
||||
| 6 — JS AdtValue + define-type + match | [ ] | — |
|
||||
|
||||
Reference in New Issue
Block a user