rose-ash/plans/apl-on-sx.md

APL-on-SX: rank-polymorphic primitives + glyph parser

The headline showcase is rank polymorphism — a single primitive (+, ⌈, ⊂, ⍳) works uniformly on scalars, vectors, matrices, and higher-rank arrays. ~80 glyph primitives + 6 operators bind together with right-to-left evaluation; the entire language is a high-density combinator algebra. The JIT compiler + primitive table pay off massively here because almost every program is array → array pure pipelines.

End-state goal: Dyalog-flavoured APL subset, dfns + tradfns, classic programs (game-of-life, mandelbrot, prime-sieve, n-queens, conway), 100+ green tests.

Scope decisions (defaults — override by editing before we spawn)

• Syntax: Dyalog APL surface, Unicode glyphs. ⎕-quad system functions for I/O. ∇ tradfn header.
• Conformance: "Reads like APL, runs like APL." Not byte-compat with Dyalog; we care about right-to-left semantics and rank polymorphism.
• Test corpus: custom — APL idioms (Roger Hui style), classic programs, plus ~50 pattern tests for primitives.
• Out of scope: ⎕-namespaces beyond a handful, complex numbers, full TAO ordering, ⎕FX runtime function definition (use static ∇ only), nested-array-of-functions higher orders, the editor.
• Glyphs: input via plain Unicode in .apl source files. Backtick-prefix shortcuts handled by the user's editor — we don't ship one.

Ground rules

• Scope: only touch lib/apl/** and plans/apl-on-sx.md. Don't edit spec/, hosts/, shared/, or any other lib/<lang>/**. APL primitives go in lib/apl/runtime.sx.
• SX files: use sx-tree MCP tools only.
• Commits: one feature per commit. Keep ## Progress log updated and tick roadmap boxes.

Architecture sketch

APL source (Unicode glyphs)
    │
    ▼
lib/apl/tokenizer.sx   — glyphs, identifiers, numbers (¯ for negative), strings, strands
    │
    ▼
lib/apl/parser.sx      — right-to-left with valence resolution (mon vs dyadic by position)
    │
    ▼
lib/apl/transpile.sx   — AST → SX AST (entry: apl-eval-ast)
    │
    ▼
lib/apl/runtime.sx     — array model, ~80 primitives, 6 operators, dfns/tradfns

Core mapping:

• Array = SX dict {:shape (d1 d2 …) :ravel #(v1 v2 …)}. Scalar is rank-0 (empty shape), vector is rank-1, matrix rank-2, etc. Type uniformity not required (heterogeneous nested arrays via "boxed" elements ⊂x).
• Rank polymorphism — every scalar primitive is broadcast: 1 2 3 + 4 5 6 ↦ 5 7 9; (2 3⍴⍳6) + 1 ↦ broadcast scalar to matrix.
• Conformability = matching shapes, or one-side scalar, or rank-1 cycling (deferred — keep strict in v1).
• Valence = each glyph has a monadic and a dyadic meaning; resolution is purely positional (left-arg present → dyadic).
• Operator = takes one or two function operands, returns a derived function (f¨ = each f, f/ = reduce f, f∘g = compose, f⍨ = commute).
• Tradfn ∇R←L F R; locals = named function with explicit header.
• Dfn {⍺+⍵} = anonymous, ⍺ = left arg, ⍵ = right arg, ∇ = recurse.

Roadmap

Phase 1 — tokenizer + parser

• Tokenizer: Unicode glyphs (the full APL set: + - × ÷ * ⍟ ⌈ ⌊ | ! ? ○ ~ < ≤ = ≥ > ≠ ∊ ∧ ∨ ⍱ ⍲ , ⍪ ⍴ ⌽ ⊖ ⍉ ↑ ↓ ⊂ ⊃ ⊆ ∪ ∩ ⍳ ⍸ ⌷ ⍋ ⍒ ⊥ ⊤ ⊣ ⊢ ⍎ ⍕ ⍝), operators (/ \ ¨ ⍨ ∘ . ⍣ ⍤ ⍥ @), numbers (¯ for negative, 1E2, 1J2 complex deferred), characters ('a', '' escape), strands (juxtaposition of literals: 1 2 3), names, comments ⍝ …
• Parser: right-to-left; classify each token as function, operator, value, or name; resolve valence positionally; dfn {…} body, tradfn ∇ header, guards :; outer product ∘.f, inner product f.g, derived fns f/ f¨ f⍨ f⍣n
• Unit tests in lib/apl/tests/parse.sx

Phase 2 — array model + scalar primitives

• Array constructor: make-array shape ravel, scalar v, vector v…, enclose/disclose
• Shape arithmetic: ⍴ (shape), , (ravel), ≢ (tally / first-axis-length), ≡ (depth)
• Scalar arithmetic primitives broadcast: + - × ÷ ⌈ ⌊ * ⍟ | ! ○
• Scalar comparison primitives: < ≤ = ≥ > ≠
• Scalar logical: ~ ∧ ∨ ⍱ ⍲
• Index generator: ⍳n (vector 1..n or 0..n-1 depending on ⎕IO)
• ⎕IO = 1 default (Dyalog convention)
• 40+ tests in lib/apl/tests/scalar.sx

Phase 3 — structural primitives + indexing

• Reshape ⍴, ravel ,, transpose ⍉ (full + dyadic axis spec)
• Take ↑, drop ↓, rotate ⌽ (last axis), ⊖ (first axis)
• Catenate , (last axis) and ⍪ (first axis)
• Index ⌷ (squad), bracket-indexing A[I] (sugar for ⌷)
• Grade-up ⍋, grade-down ⍒
• Enclose ⊂, disclose ⊃, partition (subset deferred)
• Membership ∊, find ⍳ (dyadic), without ~ (dyadic), unique ∪ (deferred to phase 6)
• 40+ tests in lib/apl/tests/structural.sx

Phase 4 — operators (THE SHOWCASE)

• Reduce f/ (last axis), f⌿ (first axis) — including ∧/, ∨/, +/, ×/, ⌈/, ⌊/
• Scan f\, f⍀
• Each f¨ — applies f to each scalar/element
• Outer product ∘.f — 1 2 3 ∘.× 1 2 3 ↦ multiplication table
• Inner product f.g — +.× is matrix multiply
• Commute f⍨ — f⍨ x ↔ x f x, x f⍨ y ↔ y f x
• Compose f∘g — applies g first then f
• Power f⍣n — apply f n times; f⍣≡ until fixed point
• Rank f⍤k — apply f at sub-rank k
• At @ — selective replace
• 40+ tests in lib/apl/tests/operators.sx

Phase 5 — dfns + tradfns + control flow

• Dfn {…} with ⍺ (left arg, may be absent → niladic/monadic), ⍵ (right arg), ∇ (recurse), guards cond:expr, default left arg ⍺←default
• Local assignment via ← (lexical inside dfn)
• Tradfn ∇ header: R←L F R;l1;l2, statement-by-statement, branch via →linenum
• Dyalog control words: :If/:Else/:EndIf, :While/:EndWhile, :For X :In V :EndFor, :Select/:Case/:EndSelect, :Trap/:EndTrap
• Niladic / monadic / dyadic dispatch (function valence at definition time)
• lib/apl/conformance.sh + runner, scoreboard.json + scoreboard.md

Phase 6 — classic programs + drive corpus

• Classic programs in lib/apl/tests/programs/:
  • life.apl — Conway's Game of Life as a one-liner using ⊂ ⊖ ⌽ +/
  • mandelbrot.apl — complex iteration with rank-polymorphic + × ⌊ (or real-axis subset)
  • primes.apl — (2=+⌿0=A∘.|A)/A←⍳N sieve
  • n-queens.apl — backtracking via reduce
  • quicksort.apl — the classic Roger Hui one-liner
• System functions: ⎕FMT, ⎕FR (float repr), ⎕TS (timestamp), ⎕IO, ⎕ML (migration level — fixed at 1), ⎕← (print)
• Drive corpus to 100+ green
• Idiom corpus — lib/apl/tests/idioms.sx covering classic Roger Hui / Phil Last idioms

Progress log

Newest first.

• 2026-04-26: parser (Phase 1 step 2) — 44/44 parser tests green (90/90 total); right-to-left segment algorithm; derived fns, outer/inner product, dfns with guards, strand handling; lib/apl/parser.sx + lib/apl/tests/parse.sx
• 2026-04-25: tokenizer (Phase 1 step 1) — 46/46 tests green; Unicode-aware starts-with? scanner for multi-byte APL glyphs; lib/apl/tokenizer.sx + lib/apl/tests/parse.sx

Blockers

• (none yet)