common-lisp: Phase 1 tokenizer + 79 tests
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Has been cancelled

lib/common-lisp/reader.sx — CL tokenizer: symbols with package
qualification (pkg:sym/pkg::sym), integers, floats, ratios, hex/
binary/octal (#xFF/#b1010/#o17), strings with escapes, #\ char
literals (named + bare), reader macros (#' #( #: ,@), line and
nested block comments.

lib/common-lisp/tests/read.sx — 79 tests, all green.
lib/common-lisp/test.sh — test runner (sx_server pipe protocol).

Key SX gotcha: use str not concat for string building.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-04-25 18:06:30 +00:00
parent 3316d402fd
commit 13d0ebcce8
4 changed files with 661 additions and 2 deletions

View File

@@ -50,7 +50,7 @@ Core mapping:
## Roadmap
### Phase 1 — reader + parser
- [ ] Tokenizer: symbols (with package qualification `pkg:sym` / `pkg::sym`), numbers (int, float, ratio `1/3`, `#xFF`, `#b1010`, `#o17`), strings `"…"` with `\` escapes, characters `#\Space` `#\Newline` `#\a`, comments `;`, block comments `#| … |#`
- [x] Tokenizer: symbols (with package qualification `pkg:sym` / `pkg::sym`), numbers (int, float, ratio `1/3`, `#xFF`, `#b1010`, `#o17`), strings `"…"` with `\` escapes, characters `#\Space` `#\Newline` `#\a`, comments `;`, block comments `#| … |#`
- [ ] Reader: list, dotted pair, quote `'`, function `#'`, quasiquote `` ` ``, unquote `,`, splice `,@`, vector `#(…)`, uninterned `#:foo`, nil/t literals
- [ ] Parser: lambda lists with `&optional` `&rest` `&key` `&aux` `&allow-other-keys`, defaults, supplied-p variables
- [ ] Unit tests in `lib/common-lisp/tests/read.sx`
@@ -114,7 +114,7 @@ Core mapping:
_Newest first._
- _(none yet)_
- 2026-04-25: Phase 1 tokenizer — 79 tests green. `lib/common-lisp/reader.sx` + `tests/read.sx` + `test.sh`. Handles symbols (pkg:sym, pkg::sym), integers, floats, ratios, hex/binary/octal, strings, #\ chars, reader macros (#' #( #: ,@), line/block comments. Key gotcha: SX `str` for string concat (not `concat`), substring-based read-while.
## Blockers