giles
da8ba104a6
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 17s
Implement lib/apl/parser.sx — APL expression parser:
- Segment-based algorithm: scan L→R collecting {fn,val} segments
- build-tree constructs AST with leftmost-fn = root (right-to-left semantics)
- Handles: monadic/dyadic fns, strands (:vec), assignment (:assign)
- Operators: derived-fn (:derived-fn op fn), inner product (:derived-fn2)
- Outer product ∘.f (:outer), dfns {:dfn stmt...}, guards (:guard cond expr)
- split-statements is bracket-aware (depth tracking prevents splitting inside {})
44 new parser tests + 46 existing tokenizer = 90/90 green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
117 lines
7.5 KiB
Markdown
117 lines
7.5 KiB
Markdown
# APL-on-SX: rank-polymorphic primitives + glyph parser
|
||
|
||
The headline showcase is **rank polymorphism** — a single primitive (`+`, `⌈`, `⊂`, `⍳`) works uniformly on scalars, vectors, matrices, and higher-rank arrays. ~80 glyph primitives + 6 operators bind together with right-to-left evaluation; the entire language is a high-density combinator algebra. The JIT compiler + primitive table pay off massively here because almost every program is `array → array` pure pipelines.
|
||
|
||
End-state goal: Dyalog-flavoured APL subset, dfns + tradfns, classic programs (game-of-life, mandelbrot, prime-sieve, n-queens, conway), 100+ green tests.
|
||
|
||
## Scope decisions (defaults — override by editing before we spawn)
|
||
|
||
- **Syntax:** Dyalog APL surface, Unicode glyphs. `⎕`-quad system functions for I/O. `∇` tradfn header.
|
||
- **Conformance:** "Reads like APL, runs like APL." Not byte-compat with Dyalog; we care about right-to-left semantics and rank polymorphism.
|
||
- **Test corpus:** custom — APL idioms (Roger Hui style), classic programs, plus ~50 pattern tests for primitives.
|
||
- **Out of scope:** ⎕-namespaces beyond a handful, complex numbers, full TAO ordering, `⎕FX` runtime function definition (use static `∇` only), nested-array-of-functions higher orders, the editor.
|
||
- **Glyphs:** input via plain Unicode in `.apl` source files. Backtick-prefix shortcuts handled by the user's editor — we don't ship one.
|
||
|
||
## Ground rules
|
||
|
||
- **Scope:** only touch `lib/apl/**` and `plans/apl-on-sx.md`. Don't edit `spec/`, `hosts/`, `shared/`, or any other `lib/<lang>/**`. APL primitives go in `lib/apl/runtime.sx`.
|
||
- **SX files:** use `sx-tree` MCP tools only.
|
||
- **Commits:** one feature per commit. Keep `## Progress log` updated and tick roadmap boxes.
|
||
|
||
## Architecture sketch
|
||
|
||
```
|
||
APL source (Unicode glyphs)
|
||
│
|
||
▼
|
||
lib/apl/tokenizer.sx — glyphs, identifiers, numbers (¯ for negative), strings, strands
|
||
│
|
||
▼
|
||
lib/apl/parser.sx — right-to-left with valence resolution (mon vs dyadic by position)
|
||
│
|
||
▼
|
||
lib/apl/transpile.sx — AST → SX AST (entry: apl-eval-ast)
|
||
│
|
||
▼
|
||
lib/apl/runtime.sx — array model, ~80 primitives, 6 operators, dfns/tradfns
|
||
```
|
||
|
||
Core mapping:
|
||
- **Array** = SX dict `{:shape (d1 d2 …) :ravel #(v1 v2 …)}`. Scalar is rank-0 (empty shape), vector is rank-1, matrix rank-2, etc. Type uniformity not required (heterogeneous nested arrays via "boxed" elements `⊂x`).
|
||
- **Rank polymorphism** — every scalar primitive is broadcast: `1 2 3 + 4 5 6` ↦ `5 7 9`; `(2 3⍴⍳6) + 1` ↦ broadcast scalar to matrix.
|
||
- **Conformability** = matching shapes, or one-side scalar, or rank-1 cycling (deferred — keep strict in v1).
|
||
- **Valence** = each glyph has a monadic and a dyadic meaning; resolution is purely positional (left-arg present → dyadic).
|
||
- **Operator** = takes one or two function operands, returns a derived function (`f¨` = `each f`, `f/` = `reduce f`, `f∘g` = `compose`, `f⍨` = `commute`).
|
||
- **Tradfn** `∇R←L F R; locals` = named function with explicit header.
|
||
- **Dfn** `{⍺+⍵}` = anonymous, `⍺` = left arg, `⍵` = right arg, `∇` = recurse.
|
||
|
||
## Roadmap
|
||
|
||
### Phase 1 — tokenizer + parser
|
||
- [x] Tokenizer: Unicode glyphs (the full APL set: `+ - × ÷ * ⍟ ⌈ ⌊ | ! ? ○ ~ < ≤ = ≥ > ≠ ∊ ∧ ∨ ⍱ ⍲ , ⍪ ⍴ ⌽ ⊖ ⍉ ↑ ↓ ⊂ ⊃ ⊆ ∪ ∩ ⍳ ⍸ ⌷ ⍋ ⍒ ⊥ ⊤ ⊣ ⊢ ⍎ ⍕ ⍝`), operators (`/ \ ¨ ⍨ ∘ . ⍣ ⍤ ⍥ @`), numbers (`¯` for negative, `1E2`, `1J2` complex deferred), characters (`'a'`, `''` escape), strands (juxtaposition of literals: `1 2 3`), names, comments `⍝ …`
|
||
- [x] Parser: right-to-left; classify each token as function, operator, value, or name; resolve valence positionally; dfn `{…}` body, tradfn `∇` header, guards `:`; outer product `∘.f`, inner product `f.g`, derived fns `f/ f¨ f⍨ f⍣n`
|
||
- [x] Unit tests in `lib/apl/tests/parse.sx`
|
||
|
||
### Phase 2 — array model + scalar primitives
|
||
- [ ] Array constructor: `make-array shape ravel`, `scalar v`, `vector v…`, `enclose`/`disclose`
|
||
- [ ] Shape arithmetic: `⍴` (shape), `,` (ravel), `≢` (tally / first-axis-length), `≡` (depth)
|
||
- [ ] Scalar arithmetic primitives broadcast: `+ - × ÷ ⌈ ⌊ * ⍟ | ! ○`
|
||
- [ ] Scalar comparison primitives: `< ≤ = ≥ > ≠`
|
||
- [ ] Scalar logical: `~ ∧ ∨ ⍱ ⍲`
|
||
- [ ] Index generator: `⍳n` (vector 1..n or 0..n-1 depending on `⎕IO`)
|
||
- [ ] `⎕IO` = 1 default (Dyalog convention)
|
||
- [ ] 40+ tests in `lib/apl/tests/scalar.sx`
|
||
|
||
### Phase 3 — structural primitives + indexing
|
||
- [ ] Reshape `⍴`, ravel `,`, transpose `⍉` (full + dyadic axis spec)
|
||
- [ ] Take `↑`, drop `↓`, rotate `⌽` (last axis), `⊖` (first axis)
|
||
- [ ] Catenate `,` (last axis) and `⍪` (first axis)
|
||
- [ ] Index `⌷` (squad), bracket-indexing `A[I]` (sugar for `⌷`)
|
||
- [ ] Grade-up `⍋`, grade-down `⍒`
|
||
- [ ] Enclose `⊂`, disclose `⊃`, partition (subset deferred)
|
||
- [ ] Membership `∊`, find `⍳` (dyadic), without `~` (dyadic), unique `∪` (deferred to phase 6)
|
||
- [ ] 40+ tests in `lib/apl/tests/structural.sx`
|
||
|
||
### Phase 4 — operators (THE SHOWCASE)
|
||
- [ ] Reduce `f/` (last axis), `f⌿` (first axis) — including `∧/`, `∨/`, `+/`, `×/`, `⌈/`, `⌊/`
|
||
- [ ] Scan `f\`, `f⍀`
|
||
- [ ] Each `f¨` — applies `f` to each scalar/element
|
||
- [ ] Outer product `∘.f` — `1 2 3 ∘.× 1 2 3` ↦ multiplication table
|
||
- [ ] Inner product `f.g` — `+.×` is matrix multiply
|
||
- [ ] Commute `f⍨` — `f⍨ x` ↔ `x f x`, `x f⍨ y` ↔ `y f x`
|
||
- [ ] Compose `f∘g` — applies `g` first then `f`
|
||
- [ ] Power `f⍣n` — apply f n times; `f⍣≡` until fixed point
|
||
- [ ] Rank `f⍤k` — apply f at sub-rank k
|
||
- [ ] At `@` — selective replace
|
||
- [ ] 40+ tests in `lib/apl/tests/operators.sx`
|
||
|
||
### Phase 5 — dfns + tradfns + control flow
|
||
- [ ] Dfn `{…}` with `⍺` (left arg, may be absent → niladic/monadic), `⍵` (right arg), `∇` (recurse), guards `cond:expr`, default left arg `⍺←default`
|
||
- [ ] Local assignment via `←` (lexical inside dfn)
|
||
- [ ] Tradfn `∇` header: `R←L F R;l1;l2`, statement-by-statement, branch via `→linenum`
|
||
- [ ] Dyalog control words: `:If/:Else/:EndIf`, `:While/:EndWhile`, `:For X :In V :EndFor`, `:Select/:Case/:EndSelect`, `:Trap`/`:EndTrap`
|
||
- [ ] Niladic / monadic / dyadic dispatch (function valence at definition time)
|
||
- [ ] `lib/apl/conformance.sh` + runner, `scoreboard.json` + `scoreboard.md`
|
||
|
||
### Phase 6 — classic programs + drive corpus
|
||
- [ ] Classic programs in `lib/apl/tests/programs/`:
|
||
- [ ] `life.apl` — Conway's Game of Life as a one-liner using `⊂` `⊖` `⌽` `+/`
|
||
- [ ] `mandelbrot.apl` — complex iteration with rank-polymorphic `+ × ⌊` (or real-axis subset)
|
||
- [ ] `primes.apl` — `(2=+⌿0=A∘.|A)/A←⍳N` sieve
|
||
- [ ] `n-queens.apl` — backtracking via reduce
|
||
- [ ] `quicksort.apl` — the classic Roger Hui one-liner
|
||
- [ ] System functions: `⎕FMT`, `⎕FR` (float repr), `⎕TS` (timestamp), `⎕IO`, `⎕ML` (migration level — fixed at 1), `⎕←` (print)
|
||
- [ ] Drive corpus to 100+ green
|
||
- [ ] Idiom corpus — `lib/apl/tests/idioms.sx` covering classic Roger Hui / Phil Last idioms
|
||
|
||
## Progress log
|
||
|
||
_Newest first._
|
||
|
||
- 2026-04-26: parser (Phase 1 step 2) — 44/44 parser tests green (90/90 total); right-to-left segment algorithm; derived fns, outer/inner product, dfns with guards, strand handling; `lib/apl/parser.sx` + `lib/apl/tests/parse.sx`
|
||
- 2026-04-25: tokenizer (Phase 1 step 1) — 46/46 tests green; Unicode-aware starts-with? scanner for multi-byte APL glyphs; `lib/apl/tokenizer.sx` + `lib/apl/tests/parse.sx`
|
||
|
||
## Blockers
|
||
|
||
- _(none yet)_
|