rose-ash/plans/elixir-on-sx.md

Elixir-on-SX: Elixir on the CEK/VM

Compile Elixir source to SX AST; the existing CEK evaluator runs it. The natural companion to lib/erlang/ — Elixir compiles to the BEAM and most of its runtime semantics are Erlang's. The interesting parts are Elixir-specific: the macro system (quote/unquote), the pipe operator |>, with expressions, defmodule/def/defp, protocol dispatch, and the Stream lazy evaluation library.

End-state goal: core Elixir programs running, including modules, pattern matching, the pipe operator, macros (quote/unquote/defmacro), protocols, and actor-style processes reusing the Erlang runtime foundation.

Ground rules

• Scope: only touch lib/elixir/** and plans/elixir-on-sx.md. Do not edit spec/, hosts/, shared/, or other lib/<lang>/. Reuse lib/erlang/ runtime functions where possible — import them, don't duplicate.
• Shared-file issues go under "Blockers" below with a minimal repro; do not fix here.
• SX files: use sx-tree MCP tools only.
• Architecture: Elixir source → Elixir AST → SX AST. Reuse Erlang runtime for process/ message/pattern primitives; add Elixir-specific surface in lib/elixir/.
• Commits: one feature per commit. Keep ## Progress log updated and tick boxes.

Architecture sketch

Elixir source text
    │
    ▼
lib/elixir/tokenizer.sx   — atoms (:atom), strings (""), charlists (''), sigils (~r, ~s etc.),
    │                        operators (|>, <>, ++, :::, etc.), do/end blocks
    ▼
lib/elixir/parser.sx      — Elixir AST: defmodule, def/defp/defmacro, @attribute,
    │                        pattern matching, |> pipe, with, for comprehension, quote/unquote,
    │                        case/cond/if/unless, fn, receive, try/rescue/catch/after
    ▼
lib/elixir/transpile.sx   — Elixir AST → SX AST
    │
    ├── lib/erlang/runtime.sx    (reused: processes, message passing, pattern match)
    └── lib/elixir/runtime.sx   — Elixir-specific: Kernel, String, Enum, Stream, Map,
                                   List, Tuple, IO, protocol dispatch, macro expansion

Key semantic mappings (differences from Erlang):

• defmodule M do ... end → SX define-library + module dict {:module "M" :fns {...}}
• def f(args) do body end → named function in module dict, with pattern-match dispatch
• |> pipe → left-to-right function composition; a |> f(b) = f(a, b)
• with x <- expr, y <- expr2 do body else patterns end → chained pattern match with early exit
• for x <- list, filter, do: expr → list comprehension (SX map/filter)
• quote do expr end → returns AST as SX list (homoiconic — Elixir AST IS SX-like)
• unquote(expr) → evaluate expr and splice into surrounding quote
• defmacro → macro in module; expanded at compile time by calling the SX macro
• Protocol → dict of implementations keyed by type name; defprotocol defines interface, defimpl registers an implementation
• Stream → lazy sequences using SX promises/coroutines (Phase 9/4 of primitives)
• Agent/GenServer → SX coroutine + message queue (similar to Erlang process model)

Roadmap

Phase 1 — tokenizer + parser

• Tokenizer: atoms (:atom, :"atom with spaces"), strings (""), charlists (''), numbers (int, float, hex 0xFF, octal 0o77, binary 0b11), booleans (true/false/nil), operators (|>, <>, ++, --, :::, &&, ||, !, .., <-, =~), sigils (~r/regex/, ~s"string", ~w(word list)), do/end blocks, keywords as args f(key: val), @module_attribute
• Parser: - Module: defmodule Name do ... end → module AST with body - Functions: def f(pat) do body end, def f(pat) when guard do body end, multi-clause def f(a) do ...; def f(b) do ... → clause list - defp (private), defmacro, defmacrop - @doc, @moduledoc, @spec, @type, @behaviour module attributes - case expr do patterns end, cond do clauses end, if/unless - with x <- e, y <- e2, do: body, else: [pattern -> body] - for x <- list, filter, into: acc, do: expr comprehension - fn pat -> body end anonymous function; capture &Module.fun/arity, &(&1 + 1) - receive do patterns after timeout -> body end - try do body rescue e -> ... catch type, val -> ... after ... end - quote do ... end, unquote(expr), unquote_splicing(list) - |> pipe chain: a |> f |> g(b) → g(f(a), b)
• Tests in lib/elixir/tests/parse.sx

Phase 2 — transpile: basic Elixir (no macros, no processes)

• ex-eval-ast entry
• Arithmetic, string <>, list ++/--, comparison, boolean (and/or/not)
• Pattern matching in =, function heads, case — reuse Erlang pattern engine
• def/defp → SX define with clause dispatch (like Erlang function clauses)
• Module as a dict of named functions; ModuleName.function(args) dispatch
• |> pipe: desugar a |> f(b, c) → f(a, b, c) at transpile time
• with expression: chain of <- bindings, short-circuit on mismatch to else
• for comprehension: for x <- list, filter do body end → map/filter
• fn anonymous functions, & capture forms
• if/unless/cond/case
• String interpolation: "Hello #{name}" → string concat
• Keyword lists [key: val] → SX list of {:key val} dicts; maps %{key: val} → SX dict
• Tuples {a, b, c} → SX list (or vector); elem/2, put_elem/3
• 40+ eval tests in lib/elixir/tests/eval.sx

Phase 3 — macro system

• quote do expr end → returns Elixir AST as SX list structure (Elixir AST is 3-tuples {name, meta, args} — map to SX (list name meta args))
• unquote(expr) → evaluate and splice into surrounding quote
• unquote_splicing(list) → splice list into surrounding quote
• defmacro → define a macro in the module; macro receives AST args, returns AST
• Macro expansion: expand macros before transpiling (two-pass: collect defs, then expand)
• use Module → calls Module.__using__/1 macro, injects code into caller
• import Module → bring functions into scope without prefix
• alias Module, as: M → short name for module
• Tests: defmacro unless, defmacro my_if, use injection, __MODULE__, __DIR__

Phase 4 — protocols

• defprotocol P do @spec f(t) :: result end → defines protocol dict + dispatch fn
• defimpl P, for: Type do def f(t) do ... end end → register implementation
• Protocol dispatch: P.f(value) → look up type of value, find implementation, call it
• Built-in protocols: Enumerable, Collectable, String.Chars, Inspect
• Enumerable implementation for lists, maps, ranges — enables Enum.* on custom types
• derive — automatic protocol implementation for simple structs
• Tests: custom type implementing Enumerable, String.Chars, protocol fallback

Phase 5 — structs + behaviours

• defstruct [:field1, field2: default] → defines %ModuleName{} struct type Structs are maps with __struct__: ModuleName key + defined fields
• Struct pattern matching: %User{name: n} = user
• @behaviour Module → declares behaviour callbacks; compile-time check
• @impl true / @impl BehaviourName → marks function as behaviour implementation
• Built-in behaviours: GenServer, Supervisor, Agent, Task
• Tests: struct creation, update syntax %{struct | field: val}, behaviour callbacks

Phase 6 — processes + OTP patterns (reuses Erlang runtime)

• spawn(fn -> ... end) / spawn(M, f, args) → SX coroutine on scheduler Reuse lib/erlang/ process + message queue infrastructure
• send(pid, msg) / receive do patterns end — already in Erlang runtime
• GenServer behaviour: start_link, call, cast, handle_call, handle_cast, handle_info, init — implement as SX macros expanding to process + message loop
• Agent — simple state wrapper over GenServer; Agent.start_link, get, update
• Task — async computation; Task.async, Task.await
• Supervisor — child spec, restart strategy (one_for_one, one_for_all)
• Tests: counter GenServer, bank account Agent, parallel Task, supervised worker

Phase 7 — standard library

• Enum.* — map, filter, reduce, each, into, flat_map, zip, sort, sort_by, min_by, max_by, group_by, frequencies, count, any?, all?, find, take, drop, take_while, drop_while, chunk_every, chunk_by, flat_map_reduce, scan, uniq, uniq_by, member?, empty?, sum, product
• Stream.* — lazy versions of Enum; Stream.map, Stream.filter, Stream.take, Stream.cycle, Stream.iterate, Stream.unfold, Stream.resource Uses SX promises (Phase 9) for laziness
• String.* — length, upcase, downcase, trim, split, replace, contains?, starts_with?, ends_with?, slice, at, graphemes, codepoints, to_integer, to_float, pad_leading, pad_trailing, duplicate, match?
• Map.* — new, get, put, delete, update, merge, keys, values, to_list, from_struct, has_key?, filter, map, reject, take, drop
• List.* — first, last, flatten, zip, unzip, keystore, keyfind, wrap, duplicate, improper?, delete, insert_at, replace_at
• Tuple.* — to_list, from_list, append, insert_at, delete_at
• Integer.* / Float.* — parse, to_string, digits, pow, is_odd?, is_even?
• IO.* — puts, gets, inspect, write, read → SX IO perform
• Kernel.* — built-in functions: is_integer?, is_binary?, length, hd, tl, elem, put_elem, apply, raise, exit, inspect
• inspect/1 / IO.inspect/2 — debug printing using Inspect protocol

Phase 8 — conformance target

• Vendor or hand-build 100+ Elixir program tests in lib/elixir/tests/programs/
• Drive scoreboard

Blockers

(none yet)

Progress log

Newest first.

(awaiting phase 1)