datalog: tokenizer raises on unexpected characters (256/256)
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 45s
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 45s
Bug: characters not recognised by any branch of `scan!` (`?`, `!`, `#`, `@`, `&`, `|`, `\\`, `^`, etc.) were silently consumed via `(else (advance! 1) (scan!))`. Programs with typos would parse to a stripped version of themselves with no warning — `?(X).` became `(X).` and produced confusing downstream errors. Fix: the else branch now raises a clear "unexpected character" error with the offending char and its position. 1 new tokenize test.
This commit is contained in:
@@ -254,7 +254,9 @@
|
||||
(dl-emit! "op" "/" start)
|
||||
(advance! 1)
|
||||
(scan!)))
|
||||
(else (do (advance! 1) (scan!)))))))))
|
||||
(else (error
|
||||
(str "Tokenizer: unexpected character '" ch
|
||||
"' at position " start)))))))))
|
||||
(scan!)
|
||||
(dl-emit! "eof" nil pos)
|
||||
tokens)))
|
||||
|
||||
Reference in New Issue
Block a user