rose-ash/plans/go-on-sx.md

Go-on-SX — Go as an SX guest language

Port Go to SX as the first static-typed, bidirectional-checked guest in the rose-ash language family. Goal isn't a production Go compiler; it's to prove the substrate from a paradigm angle the existing eleven guests don't cover, and to chisel out the lib/guest kits that statically-typed guests N+1 and N+2 will need.

Reference:

• plans/lib-guest.md — parent, chiselling discipline, two-language rule.
• plans/lib-guest-scheduler.md — sister kit; Go's scheduler pairs with Erlang's. Extraction gated on this loop reaching Phase 5.
• plans/lib-guest-static-types-bidirectional.md — sister kit; Go's checker pairs with a TBD second consumer. Extraction gated on this loop reaching Phase 3.
• plans/erlang-on-sx.md — reference implementation for paradigm-port: process model, BIF registry, hot reload, VM bytecode opcodes.

Branch: loops/go (loop-style workstream once kicked off). SX files via sx-tree MCP only.

Thesis — why Go

Eleven guests already live in lib/: apl, common-lisp, datalog, erlang, forth, haskell, hyperscript, js, kernel, lua, minikanren, ocaml, prolog, ruby, scheme, smalltalk, tcl. Every one is either dynamically typed (most) or HM-inferred (haskell, ocaml). None exercise:

1. Bidirectional static type checking — annotation-driven, locally- inferred, the dominant paradigm of modern statically-typed languages.
2. Anonymous-channel concurrency — Go's chan and select. Erlang has addressed processes + mailboxes; Go has anonymous values + structural pairing. Two different vocabularies for the same underlying scheduler machinery.
3. Structural interfaces — io.Reader is "anything with this method signature", not a declared subtype relationship. Different from Haskell typeclasses (nominal), different from Lua duck typing (no declaration).

These three together make Go an unusually high-value port for proving SX. If SX can host Go cleanly, it can host the next decade of mainstream statically-typed languages (Rust, TS, Swift, Kotlin, Scala 3, Hack) because they share these three properties.

Like Erlang-on-SX validated the actor model on the substrate, Go-on-SX validates the goroutine model + bidirectional types.

Non-goals (deliberate)

Out of scope. Reject feature requests for these without further consideration:

• unsafe package. Memory mucking. Skip entirely.
• CGo. C interop. Out of scope at every level.
• Full reflect. Provide enough for fmt.Println to render values; reject the rest.
• Build tags, modules, vendoring. Treat source as monolithic. One package per file, no real import resolution.
• Production performance. Conformance tests pass; benchmarks don't.
• Garbage collection tuning. SX's GC is what you get.
• Race detector, escape analysis, inlining. Out of scope.
• os, net/http, full stdlib. Provide a deliberately small slice (Phase 8 below).

Architecture sketch

Go source text
    │
    ▼
lib/go/lex.sx       — tokens; ASI; literals; operators
    │                 (consumes lib/guest/core/lex.sx)
    ▼
lib/go/parse.sx     — AST: package/import/var/const/type/func/struct/
    │                 interface; expressions; statements
    │                 (consumes lib/guest/core/pratt.sx + ast.sx)
    ▼
lib/go/types.sx     — bidirectional type checker. Synth + check judgments;
    │                 structural interface satisfaction; pluggable subtype
    │                 (INDEPENDENT — no lib/guest/static-types-bidirectional
    │                  yet; this loop builds the first consumer)
    ▼
lib/go/eval.sx      — tree-walk evaluator on CEK. Variables as mutable cells;
    │                 slices = (length, capacity, backing-vector); maps =
    │                 SX dict; defer stack per frame.
    ▼
lib/go/sched.sx     — goroutine scheduler + channels + select
    │                 (INDEPENDENT — no lib/guest/scheduler yet; this loop
    │                  builds the first consumer)
    ▼
lib/go/std/         — minimal stdlib slice (fmt, strings, strconv, sync,
                      time, errors)

Semantic mappings (operational):

• go fn(args) → task-spawn on the local scheduler.
• ch <- v → task-block with predicate "receiver waiting on ch".
• v := <-ch → task-block with predicate "sender waiting on ch".
• select { case ... } → task-block with predicate "any case ready".
• defer fn() → push thunk onto per-frame defer stack; runs LIFO on return or panic.
• panic(v) → raise SX exception; deferred fns run while unwinding.
• recover() → CEK exception capture inside a deferred fn.
• interface{T} → type-check matches structurally against T's method set; at runtime, the value carries its concrete-type metadata.
• struct{...} → SX dict + type tag; methods are functions in the type's method table.
• *T (pointer) → mutable cell (Common Lisp port did the same).
• []T (slice) → triple (length, capacity, backing-vector).
• map[K]V → SX dict; iteration order spec-undefined (v1 = sorted for determinism — programs that depend on indeterminism fail loudly, which is a feature not a bug).

Conformance scoreboard

Following lib/erlang/scoreboard.json precedent. Add lib/go/scoreboard.json on first iteration; populate as suites land. Suites planned:

Suite	Tests target	What it covers
lex	50+	Keywords, operators, literals, ASI
parse	80+	All statement & expression shapes
types	90+	Synth, check, interface satisfaction, generics
eval	100+	Tree-walk over typed AST
runtime	60+	Goroutines, channels, select, close
stdlib	40+	fmt, strings, strconv, sync, time, errors
e2e	10+	Complete representative programs

Phasing — one feature per commit

Loop-style. Each phase: implement → test → commit → tick [ ] → append Progress-log line → push origin/loops/go.

Phase 1 — Tokenizer (lib/go/lex.sx) ✅

• Scaffold + scoreboard + conformance runner (consumes lib/guest/lex.sx)
• Identifiers + 25 keywords
• Decimal integer literals
• Interpreted string literals "..." with \n \t \r \\ \" \' escapes
• Rune literals 'x' (single char + simple escapes)
• Line + block comments (block w/ newline triggers ASI)
• Common operator/punct set incl. := <- ++ -- == != <= >= && || ...
• Automatic semicolon insertion (Go spec § Semicolons) — newline, EOF, and block-comment-with-newline trigger ; after ident/int/string/rune/{break,continue,fallthrough,return}/{++,--,),],}}.
• Float / imaginary literals (decimal floats: 3.14 .5 1. 1e10 1.5e-3; imag: 2i 3.14i 1e2i; hex floats 0x1.fp0 deferred)
• Raw string literals `...` (multi-line, no escape processing, \r stripped per Go spec § String literals; same "string" type as interpreted strings)
• Hex/octal/binary integer literals (0x… 0o… 0b…) + underscores (legacy 0123 octal also accepted; consumes lex-hex-digit?)
• Full operator set audit (47 distinct per Go spec, plus ~ for generics type-sets). Exhaustive coverage tests in op-audit: block.
• Acceptance: lex/ suite at 50+ tests. Current: 129/129. Phase 1 done — hex floats deferred (rare). Move to Phase 2 next.

Phase 2 — Parser (lib/go/parse.sx) ✅

• Parser scaffold + Go operator-precedence table (entry shape from lib/guest/pratt.sx) + primary expressions (int/float/imag/string/ rune/ident → ast-literal / ast-var via lib/guest/ast.sx).
• Binary operators (Pratt precedence climbing using pratt-op-lookup + Go precedence table). Operator app emitted as (ast-app (ast-var OP) [LHS RHS]); left-assoc raises right-min by 1.
• Unary operators (+x, -x, !x, ^x, *p, &v, <-ch). gp-parse-unary recursive, sits between gp-parse-expr and gp-parse-primary; right-associative chains (!!x).
• Function calls f(a, b) (canonical ast-app) and member access x.field (Go-specific (list :select OBJ "field") — the AST kit doesn't ship a selector node; this is a sister-plan-static-types data point about what the canonical AST is missing).
• Index x[i] and slice x[a:b]/x[a:b:c]. Go-specific (list :index OBJ IDX) and (list :slice OBJ LOW HIGH MAX) (LOW/HIGH/MAX may be nil) — kit lacks both. Permissive parser accepts a[1::3] (strict Go rejects, but type phase can enforce).
• Type assertion v.(T). (list :assert OBJ TYPE). Includes a minimal gp-parse-type (named / qualified pkg.T / pointer *T / **T); full type grammar still pending below.
• Type expressions: slice []T, array [N]T, map map[K]V, chan chan T / chan<- T / <-chan T, pointer *T, named T, qualified pkg.T, func func(...) ..., struct struct{...} with shared-type field rows (x, y int), interface interface{...} with methods + embedded interfaces (named and qualified) all done — kit has no type primitives. Field tags, struct embeds, variadic, named func-params, Go 1.18 type sets, generics deferred.
• Composite literals: T{...}, []T{...}, [N]T{...}, map[K]V{...}, pkg.T{...}, nested. Positional and keyed (X: 1, Y: 2) elements. AST (list :composite TYPE-OR-EXPR ELEMS); elements are exprs or (list :kv KEY VALUE). Note: in statement context (e.g. if cond { ... }) my parser would WRONGLY treat the body as a composite; statement parsing will need a "no- composite-here" mode flag — to be added when statements arrive.
• Declarations: package, import, var, const, type, func (with named-greedy params + method receivers + body skipped opaquely until statement parsing arrives). All five :field consumers now exist (struct fields, var, const, func params, method receivers) — strong signal that ast-binding-group belongs in the canonical AST kit. Grouped/parenthesized decls (var (...), etc.) and variadic params deferred. Anonymous param-list disambiguation (func(int, string)) is a known parser-greedy limitation, flagged.
• Statements: return, short-decl, assign, compound assign, expr stmt, block, if/else (chained), for (4 shapes incl. range), break, continue, inc-dec, go, defer, send, switch (tagged / tagless, multi-value cases, default), select (recv-into-var / send / bare-recv / default) all done. Composite-literal { suppression active in control-flow conditions. Type-switch (switch v := x.(type)) deferred to a follow-up.
• End-to-end: hello-world, fibonacci, FizzBuzz, goroutine ping-pong, struct + method, interface, defer+select+range. go-parse extended to handle multi-form files: returns the single form for one-form input (backward compat) or (list :file FORMS) for multiple. Structural tests assert top-level decl-tag sequences via the decl-tags helper rather than full ASTs.
• Acceptance: parse/ suite at 80+ tests. Current: 176/176. Phase 2 complete. Type-switch is the one syntactic shape still deferred to a follow-up; it doesn't gate Phase 3.

Phase 3 — Bidirectional type checker, MVP (lib/go/types.sx) ⬜

• Scaffold: go-synth / go-check skeletons; context-as-value (go-ctx-empty / -extend / -lookup / -extend-field); predeclared true/false/nil; structural type equality.
• [/] Literal synth: heuristic kind detection from value strings (go-classify-literal-string) → :ty-untyped-int/-float/ -imag/-string (-rune deferred — value-shape ambiguous with single-char string). Parser-shape change to (:literal KIND VALUE) flagged as future work; the heuristic stopgap avoids breaking 66 existing parse tests.
• Binary-op synth with untyped-constant flow. Canonical pitfall handled: 42 / 7 synthesises to :ty-untyped-int, then checks successfully against float64. Untyped int + untyped float unifies to untyped float. Typed-var + untyped-int propagates the var's type. Comparison/logical ops produce bool.
• Var/const declaration checking (var x T = expr, var x = expr, var x T, const Pi = 3.14, type T int, var x, y int, plus short-decl x := 5 and a, b := 1, 2). go-check-decl returns the extended context or a :type-error. Untyped synthesized types get their default-type (untyped-int → int, untyped-float → float64, etc.) when bound in inferred-type decls.
• Function declaration: extends ctx with each :field param group, checks block body (decls thread through, returns verify against signature, assignments verify RHS assignable to LHS). The function itself is bound in the body's ctx so recursion will work once call-checking lands. Signature-only (no body) just binds.
• Call type-checking. go-synth-call: synth callee → expect :ty-func, arity-check, check each arg assignable to param, then return type by result count (0 → :ty-void, 1 → that type, N → :ty-tuple). Recursive calls now type-check because the func is bound in the body's ctx. Untyped-constant args flow through.
• Composite type element checking (slice / map / chan).
• Interface satisfaction (structural match against method sets).
• Short variable declaration := (synth RHS into LHS bindings).
• Defer: generics (Phase 7), full conversion rules, type assertions, type switches.
• Acceptance: types/ suite at 60+ tests. Current: 55/55. Chisel note shapes-static-types-bidirectional — sister-plan design diary is the cross-language record.

Phase 4 — Tree-walk evaluator (lib/go/eval.sx) ⬜

• AST-walking interpreter over CEK. Each Go statement maps to one step function (precedent: step-sf-if etc. in spec/evaluator.sx).
• Variables: mutable cells. Pointer semantics: &x returns the cell, *p dereferences.
• Slices: triple (length, capacity, backing-vector). append honours capacity-grow per spec.
• Maps: SX dict + key-type metadata.
• Structs: SX dict + type tag. Methods looked up via type's method table.
• Functions: closures over enclosing scope; multiple return values.
• Channels: stub (Phase 5 wires them).
• Tests: arithmetic, control flow, recursion, closures, slices, maps, structs, methods, pointer semantics, multiple-return.
• Acceptance: eval/ suite at 80+ tests. No concurrency yet.

Phase 5 — Goroutines + channels + select (lib/go/sched.sx) ⬜

• Independent implementation. Do NOT use lib/guest/scheduler/ — that kit doesn't exist yet and depends on this work for its design. See plans/lib-guest-scheduler.md.
• go expr — spawn a goroutine; returns nothing.
• chan T — make(chan T) creates an unbuffered channel; make(chan T,n) creates a buffered channel (Phase 5b — defer buffer to a sub-phase).
• <-ch — receive (blocks until sender ready).
• ch <- v — send (blocks until receiver ready for unbuffered, or buffer has room for buffered).
• select { case ... } — non-deterministic multiplexing; default makes it non-blocking.
• close(ch) — closes channel. Receive on closed → zero value + ok=false.
• Tests: ping-pong, fan-out/fan-in, work queue, select with default, select with timeout (via a time.After-like stub), close semantics, range over channel.
• Acceptance: runtime/ suite at 40+ tests. Chisel note shapes- scheduler — append a paragraph to the sister plan's design diary describing what task-spawn/block/wake/yield shape emerged.

Phase 5b — Buffered channels + select fairness ⬜

• Buffered: send blocks only when buffer full; recv only when empty.
• select random case ordering (spec mandates pseudo-random; v1 uses a fixed seed for determinism with a runtime-package knob to randomise).
• Tests: buffer-full blocking, buffer-empty blocking, select fairness over many iterations.
• Acceptance: runtime/ +20 tests.

Phase 6 — defer + panic/recover ⬜

• Defer stack per function frame; runs LIFO on return (normal or panic).
• panic(v) unwinds frames running deferreds; recover() inside a deferred fn captures the panic value and stops unwinding.
• Goroutine panic propagation: a panicking goroutine that doesn't recover crashes the whole program (honour Go spec, or document divergence).
• Tests: defer order (LIFO), defer + named-return mutation, panic/recover, panic across goroutines, defer in a loop (push per iter, run on fn return — common bug).
• Acceptance: eval/ +20 tests.

Phase 7 — Generics (Go 1.18+) ⬜

• Type parameters with constraints (type sets: interface{ int | float64 }, comparable, any).
• Type inference at call sites — basic; the full Go inference algorithm is notoriously complex. Implement enough for common cases; document limitations in a Blockers section below.
• Tests: generic function (func Map[T, U any](xs []T, f func(T) U) []U), generic data structure (linked list), constrained type param.
• Acceptance: types/ +30 tests.

Phase 8 — Minimal stdlib (lib/go/std/) ⬜

• Implement just what's needed for representative programs:
  • fmt — Println, Printf, Sprintf, Fprintf, Errorf, Stringer dispatch. Verbs: %d %s %v %t %f %T %+v.
  • strings — Contains, HasPrefix, HasSuffix, Split, Join, TrimSpace, ToUpper, ToLower, Replace, Index, Count, Repeat, NewReader.
  • strconv — Itoa, Atoi, FormatFloat, ParseFloat, ParseInt, FormatInt.
  • errors — New, Is, As, Unwrap.
  • sync — Mutex (cooperative — flag + waiter queue), WaitGroup, Once, RWMutex.
  • time — Now, Since, After (channel-returning timer), Sleep, Duration, Time.
  • io — Reader/Writer interfaces; ReadAll; Copy.
  • sort — Slice, Ints, Strings.
• Tests: round-trip Itoa/Atoi, fmt verb coverage, sync.WaitGroup with goroutines, time.After in a select, sort.Slice with custom less fn.
• Acceptance: stdlib/ suite at 40+ tests.

Phase 9 — End-to-end programs ⬜

• Complete programs from canonical sources (gopl.io, "concurrency patterns" talk examples) running end-to-end:
  • Concurrent prime sieve
  • HTTP-ish ping-pong over stubbed transport
  • Word frequency counter
  • Pipeline (channel chain)
  • Producer/consumer with sync.WaitGroup
  • "Bounded parallelism" pattern (worker pool over a job channel)
• Acceptance: e2e/ suite at 10+ tests, all passing.

Phase 10 — lib/guest extraction enabler ⬜

• Now that Go has lex+parse+types+eval+sched, sister plans are unblocked on the Go side. This phase is doc-only in loops/go:
  • Cross-reference plans/lib-guest-scheduler.md — mark its Phase 1 (Go scheduler independent) as complete from Go's side.
  • Cross-reference plans/lib-guest-static-types-bidirectional.md — mark its Phase 1 as complete from Go's side.
  • Update the chiselling diary in each sister plan with the actual Go-side surface that emerged.
• Acceptance: sister plans cross-referenced + diaries updated. No new Go code.

Phase 11 — VM bytecode opcodes (deferred, optional) ⬜

• Following Erlang-on-SX Phase 10 precedent: identify hot paths in the tree-walk evaluator, define Go-specific bytecode opcodes, compile hot fns through them. Substantial work; only justified if Go programs exercise enough volume that performance starts mattering.
• Acceptance: TBD on demand.

Ground rules (loop-style)

• Scope: only lib/go/** and this plan. Do not touch spec/, hosts/, shared/, lib/guest/** (read-only consumer at this phase), or other lib/<lang>/.
• Consume lib/guest/core/ for lex/parse/ast/match/layout. Hand- rolling defeats the chiselling goal.
• Do NOT extract into lib/guest/scheduler/ or lib/guest/static- types-bidirectional/ from this loop. Those extractions are gated on two consumers AND the discipline of writing each consumer independently. Extraction is its own workstream after Go and the second consumer both exist.
• Substrate gaps → Blockers entry with minimal repro. Don't fix the substrate from this loop. Belongs to sx-improvements.md.
• NEVER call sx_build without timeout awareness — 600s watchdog.
• SX files: sx-tree MCP tools ONLY. sx_validate after every edit.
• Worktree: branch loops/go, push origin/loops/go. Never main, never architecture.
• Commit granularity: one feature per commit. Short factual messages: go: parse short-decl + 6 tests [consumes-pratt]. Chisel note at end in brackets.
• Plan file: update Progress log + tick boxes every commit.
• If blocked for two iterations on the same issue, add to Blockers and move on. Phases 1-4 are sequential; Phases 5-8 are largely independent once 4 lands.

Chisel discipline (per parent lib-guest plan)

Every commit ends its message with a chisel note in brackets:

• [consumes-X] — used lib/guest/X kit.
• [shapes-scheduler] / [shapes-static-types-bidirectional] — revealed something about what the sister lib-guest kits should look like. Add a paragraph to the relevant sister plan's design diary.
• [proposes-Y] — revealed a gap in another existing kit. Blockers entry in the kit's plan.
• [nothing] — pure Go work that didn't touch substrate or lib/guest story. Acceptable; if it shows up twice in a row, stop and reflect.

Go-specific gotchas

• ASI (automatic semicolon insertion). Newline becomes ; after identifier/literal/)/]/}. Build into the tokenizer; the Go spec's "Semicolons" section is unusually precise — follow it literally.
• Untyped constants. 42 has type untyped int until used in a context that forces a type. The canonical example: var x float64 = 42 / 7 — must compute as untyped int / untyped int = 6 then convert to float64 = 6.0. Wrong: float-coercing eagerly gives 6.0 prematurely. Wrong: integer-truncating after coercion gives 5.something. Test it.
• Methods vs functions. func (r Receiver) Method() is a method bound to a type; func Function(r Receiver) is just a function. Methods on pointer-receivers vs value-receivers have asymmetric satisfaction in interfaces — pointer-receiver methods are NOT in the value's method set for interface satisfaction.
• Interface satisfaction is structural and silent. Type satisfies an interface if its method set contains all the interface's methods. Lazy check: at every point a value flows into an interface-typed slot.
• Channels are first-class values. Pass them, store them, send them through other channels. Each channel has identity.
• select with default = non-blocking. Without default, blocks until a case is ready.
• nil is typed. var x *int makes x a (*int)(nil). Comparison x == nil works on typed nil; but var i interface{} = (*int)(nil); i == nil is false — i holds a typed-nil-of-type-*int, not untyped nil. The classic Go footgun. Test it.
• Goroutine panic propagation. A panicking goroutine that doesn't recover crashes the whole program. Implement faithfully or document divergence.
• defer in a loop. Each iteration pushes; they all run on function return. Common bug; tests should cover.
• Iteration order of maps. Spec: unspecified. v1 = sorted by SX- canonical key order for determinism; document that programs depending on iteration order are not Go-conformant. Add a runtime-package knob to enable randomisation later.

Style

• No comments in .sx unless non-obvious. Cite Go spec sections inline for non-obvious decisions (Go's spec is rigorous; citations work).
• No new planning docs — update this plan inline.
• One feature per iteration. Commit. Log. Push. Next.

Open questions

1. Module/import model. Go has packages and import paths. Probably model "package" as one or more .sx files in a directory, no real import resolution against a remote module graph. Decide in Phase 2.
2. Goroutine identity. Spec says goroutines have no identity; the scheduler does internally. Expose to user code? No (not Go). Expose for debugging? Yes via a runtime-package stub.
3. Error handling: panic-as-exception vs explicit error returns. Go strongly prefers explicit errors. Stdlib stubs follow that: strconv. Atoi("x") returns (0, err), not panic.
4. Memory model. Go has a happens-before model for atomics + channel ops. SX runtime is single-threaded under the scheduler — every channel op is a synchronization point automatically. Don't model relaxed memory; document the simplification.
5. Iteration order of maps. Already addressed in Gotchas; flagged here as a known divergence from spec.

Blockers

Kit-gap proposals against lib/guest/ast.sx

Observed from building the Go parser:

1. No selector / field-access node. obj.field is a universal shape across nominally-typed languages — Go, Rust, Swift, TS, JS, Python, Ruby, Java, C#. The kit ships ast-app (function application) but not ast-select. We rolled (list :select OBJ "field") locally as a Go-specific tag. Worth promoting once a second consumer hits the same need (likely immediately — almost every guest needs it).

2. No index / subscript node. x[i] is universal across nearly every guest with arrays/maps. Rolled (list :index OBJ IDX) locally.

3. No slice node. Go's two- and three-index slice expressions are distinctive but the basic two-index x[a:b] shape covers Python, Rust, Swift, JS, Ruby slicing too. Rolled (list :slice OBJ LOW HIGH MAX) (LOW/HIGH/MAX may be nil for omitted indices). MAX-as-fourth-field is Go-specific; the canonical kit shape could ship as (list :slice OBJ LOW HIGH) for the common case and a separate :slice3 or :full-slice for the Go variant.

Minimal repro: see lib/go/parse.sx#gp-parse-postfix + gp-parse-bracket.

4. No "named binding(s) of a type" node. Building struct types surfaced a shape that recurs everywhere:

   (list :field NAMES TYPE)
   

   Same shape appears in: struct fields (x, y int), func parameters (func(a, b int, c string)), method receivers (m(a, b int)), variable declarations (var x, y int). Three Phase-2 sub-deliverables (struct fields, func decls, var decls) all want this shape. Promoting it once means Rust struct fields, Swift parameters, TS class fields, Java method signatures all get a free home. Candidate canonical name: ast-binding-group or ast-named-of-type.

5. No type-expression primitives. Every statically-typed guest needs to express types in source. Proposed canonical shapes:

   (list :ty-name "T")           — named type
   (list :ty-sel  "pkg" "T")     — qualified type
   (list :ty-ptr  T)             — pointer to T
   (list :ty-slice T)            — slice / dynamic array of T
   (list :ty-array N T)          — fixed array, N is an expr
   (list :ty-map  K V)           — map type (also Python dict, Rust HashMap)
   

   The first six are universal: Rust, Swift, TS, Kotlin, Scala, Hack all need them. Go-specific extensions like :ty-chan (channel with direction) and :ty-func (parameter+return) should stay guest-specific until a second consumer wants them.

Minimal repro: see lib/go/parse.sx#gp-parse-type.

Kit-gap proposals against lib/guest/lex.sx

Observed from building the Go tokenizer. Not blocking Phase 2; surfaced here for the substrate-maintainer / next statically-typed-guest loop:

1. No lex-oct-digit? / lex-bin-digit?. Go's prefixed integer forms 0o17 and 0b1010 need digit-class predicates that the kit doesn't provide. We rolled local gl-oct-digit? and gl-bin-digit?. Rust and Swift's lexers will need the same. Cheap to promote.

2. No table-driven longest-prefix matcher. Go has 47+ operator sequences with longest-match semantics. Our gl-match-op is a 25-clause cond ladder; Rust/Swift/TS will each need ~50+. A kit helper like (lex-match-longest TABLE SOURCE POS) that takes a sorted prefix table would collapse this. Worth proposing once a second statically-typed guest hits the same pattern.

Minimal repro: see lib/go/lex.sx#gl-oct-digit? and #gl-match-op.

Progress log

Newest first. Append one dated entry per commit.

• 2026-05-27 — Phase 3 cont.: call type-checking. go-synth-call synthesises the callee's type, asserts it's a :ty-func, arity- checks args, then go-check-args-against runs each arg through go-check against the corresponding param type (untyped-constant flow works). Result: :ty-void for 0-result funcs, the result type for 1-result, (list :ty-tuple TYPES) for multi-return. The :app dispatch in go-synth now routes via go-is-binop-call? (operator name + 2 args + op in the binop tables) — binops short- circuit; everything else goes through the call path. Recursive functions now type-check because the func is bound in its own body's ctx by go-check-func-decl. +8 tests, types 55/55, total 360/360. [nothing] — Go-side composition on top of established primitives; no new kit-relevant shapes (call semantics are uniform across statically-typed guests).

• 2026-05-27 — Phase 3 cont.: function-declaration checking + statement-level dispatch. go-check-func-decl binds the function in the outer ctx (so the body can see itself), extends the body's ctx with each :field param group via go-ctx-extend-field (the binding-group shape's third consumer in the type checker — now five total across parser+typer combined), then runs go-check-block through every statement. go-check-stmt dispatches on :return, :assign, :var-decl/:const-decl/:short-decl/:type-decl, :block, falling back to go-synth for expression statements. Return-list and assign-pair count mismatches are typed errors. +7 tests, types 47/47, total 352/352. [nothing] — pure Go-side composition; the kit-relevant insights are already in the sister- plan diary.

• 2026-05-27 — Phase 3 cont.: declaration checking — var/const/type

  • short-decl :=. go-check-decl returns the extended context (or a :type-error). New helpers: go-default-type (untyped-int → int, untyped-float → float64, etc.), go-check-exprs-against, go-bind-names-to-synth. Annotated decls check each init expression is assignable to the declared type; inferred decls bind names to the default-typed synthesis of the init. var x float64 = 42 / 7 and const C int = 42 both bind x to float64 / C to int correctly via the assignability relation from the previous commit. +12 tests, types 40/40, total 345/345. [nothing] — the kit-relevant insights (synth/check + assignable predicate) already in the diary; this is pure Go-side composition on top.

• 2026-05-27 — Phase 3 cont.: literal synth + binop synth + assignability. Heuristic go-classify-literal-string decodes the parser's untagged literal values back into :int/:float/:imag/:string kinds (rune defers); these become :ty-untyped-* types. go-synth-binop handles arithmetic / bitwise / comparison / logical operators with untyped-constant unification: untyped int + untyped float → untyped float; untyped + typed → typed. Canonical Go pitfall now handled: 42 / 7 synthesises to :ty-untyped-int, then go-check against float64 returns :ok via go-type-assignable?. +16 tests, types 28/28, total 333/333. [shapes-static-types-bidirectional] — sister plan diary updated with the assignable-relation insight (kit's check should accept a subtype?/assignable? predicate parameter).

• 2026-05-27 — Phase 3 scaffold. First lib/go/types.sx cut: context as an association list (go-ctx-empty + -extend + -lookup), a load-bearing go-ctx-extend-field that consumes the :field binding- group shape (validating the Phase 2 cross-deliverable observation), predeclared true/false/nil, go-synth for identifier lookup, go-check deferring to synth + structural equality. types suite 12/12, total 317/317. Literal kinds (untyped int/float/string/rune)

  • binop synth + var-decl checking next. [shapes-static-types- bidirectional] — sister-plan diary updated with the synth/check Go-side surface as it emerges.

• 2026-05-27 — Phase 2 complete. End-to-end multi-form file parsing. go-parse now returns single forms for backward compat (~169 tests unchanged) or (list :file FORMS) for multi-form input. Tests cover hello-world, fibonacci, FizzBuzz, goroutine ping-pong, struct+method, interface+method, and defer+select+range — each asserted via top- level decl-tags. Type-switch is the one syntactic shape still deferred. +7 tests, parse 176/176, total 305/305. Next: Phase 3 (bidirectional type checker). [nothing] — pure Go parser composition; the cross-language insights are already in the sister- plan diaries from earlier Phase 2 commits.

• 2026-05-27 — Phase 2 cont.: switch and select statements. Tagged + tagless switch, multi-value cases, default, and select with recv-into-var / send / bare-recv / default cases. New gp-parse-case-body reads stmts until the next case/default/} without consuming the terminator. AST shapes: (list :switch TAG CASES), (list :case VALUES BODY), (list :select CASES), (list :select-case COMM-STMT BODY), (list :default BODY). With this, Phase 2 statement coverage is complete — type-switch is the one remaining shape (deferred). +8 tests, parse 169/169, total 298/298. [shapes-scheduler] — sister-plan diary updated with the :select-case uniform shape insight (single kit primitive covers all four Go case kinds; default vs no-default determines blocking semantics; cross-references to Erlang's receive ... after).

  Sister-plan diary update follows.

• 2026-05-27 — Phase 2 cont.: concurrency + iteration statements. go EXPR, defer EXPR, channel send ch <- v, and the four for ... range shapes (no-kv / k-only / k,v / assign-form). New gp-for-find-range pre-scans the for-header to dispatch between range and C-style/while forms cleanly. Send-stmt detection added to the LHS-list branch (after lhs, <- → send). +9 tests, parse 161/161, total 290/290. [shapes-scheduler] — Go's concurrency- primitive AST shapes (:go, :defer, :send, :range-for) all landed; sister-plan diary updated with the corresponding kit-API insights (uniform spawn-thunk shape, channel-recv ⇄ iteration polymorphism at the range-coll dispatch).

  Sister-plan diary update follows.

• 2026-05-27 — Phase 2 cont.: control-flow statements. if cond { } [else { }] with chained else-if, for { } (infinite), for cond { } (while-like), for init; cond; post { } (C-style), break, continue, plus x++ / x-- inc-dec statements. Closed the parser-mode caveat flagged when composite literals landed: gp-no-comp-lit is a re-entrant counter that suppresses the postfix {...} → composite-lit interpretation inside control-flow condition positions, matching Go spec § Composite literals. gp-parse-control- cond wraps the increment/decrement so callers can't forget. +11 tests, parse 152/152, total 281/281. [nothing] — pure Go parser shape work; the bidirectional-checker-relevant shapes (cond/body) are already covered by the earlier :field insight.

• 2026-05-27 — Phase 2 cont.: statements. First slice covers return [exprs], short-decl lhs := exprs, assignment lhs = exprs, compound assignment (+= -= *= /= %= &= |= ^= <<= >>= &^=), bare expression statements, and nested blocks { ... }. New gp-parse-stmt dispatches on the leading token; gp-parse-block-body replaces the func-decl :body sentinel with real (:block STMTS). Existing func/method tests updated to the new body shape. Progress guards added to gp-block-body-loop and gp-parse-composite-elems — unsupported syntax (if, for, etc.) now advances one token instead of spinning. gp-skip-block! left as dead code; will be deleted once control-flow stmts land. +9 tests, parse 141/141, total 270/270. [nothing] — pure Go parser work; the cross-language statement shapes will become a chiselling target once a second statically-typed guest hits them.

• 2026-05-27 — Phase 2 cont.: func declarations. func f() {}, func add(x, y int) int { ... }, multi-group params, multi-return, signature-only (no body), pointer-receiver and value-receiver methods, nested-brace body. New gp-parse-decl-param-group uses a named-greedy algorithm: collects consecutive ident [, ident]* then parses a type. gp-skip-block! brace-balances over the body opaquely; the AST stores :body as a sentinel pending statement parsing. With this, all five :field binding-group consumers now exist (struct fields, var, const, func params, method receivers) — strong cross- deliverable validation of the ast-binding-group kit proposal. Anonymous-param-list disambiguation (func(int, string)) is a known greedy-parser limitation flagged in plan. +8 tests, parse 132/132, total 261/261. [shapes-static-types-bidirectional] — the consistent use of :field across decls is what the sister kit's bidirectional checker will use to propagate types from declarations to bindings.

  Sister-plan diary update follows.

• 2026-05-27 — Phase 2 cont.: declarations — package N, import "p", var name [TYPE] [= EXPRS], const name [TYPE] [= EXPRS], type NAME TYPE. New gp-parse-top dispatcher routes the five decl keywords to gp-parse-decl while preserving expression parsing for everything else. var and const reuse the :field binding- group shape from Blockers — first cross-deliverable use of the proposed kit shape: struct fields, func params, and now var/const decls all share the same (list :field NAMES TYPE) envelope. import uses canonical ast-import directly. Grouped forms (var (...)) and func decls deferred. +10 tests, parse 124/124, total 253/253. [consumes-ast] — first concrete use of ast-import from the kit; also validates the :field shape across three contexts.

• 2026-05-27 — Phase 2 cont.: composite literals. T{}, T{1, 2}, T{X: 1, Y: 2}, []T{...}, [N]T{...}, map[K]V{...}, pkg.T{...}, nested composites. AST shape (list :composite TYPE-OR-EXPR ELEMS); each element is an expression or (list :kv KEY VALUE). Two parser entry points: type-prefixed (gp-parse-primary adds [/map/struct branches) and ident-prefixed (postfix loop adds { branch). Known limitation flagged in plan: when statement parsing arrives, the postfix { branch will misread if cond { ... } as composite literal — needs a "no-composite-here" parser-mode flag. +8 tests, parse 114/114, total 243/243. [nothing] — pure Go parser shape work.

• 2026-05-27 — Phase 2 cont.: interface-type expressions. interface {}, interface { Close() }, interface { String() string }, interface { Read([]byte) (int, error) }, plus embedded interfaces (Stringer, io.Reader). AST shape: (list :ty-interface ELEMS) where each element is either (list :method NAME PARAMS RESULTS) or (list :embed TYPE). Method params reuse gp-parse-func-type-params — same anonymous-only shape as func types. Go 1.18+ type sets (~int | ~float64) deferred to generics work. With this, all Phase-2 type expressions are complete. +8 tests, parse 106/106, total 235/235. [nothing] — pure Go parser; the field-binding-group kit-gap proposal from the previous commit covers the cross-language angle.

• 2026-05-27 — Phase 2 cont.: struct-type expressions. struct {}, struct { x int }, struct { x int; y string }, struct { x, y int } (shared type), nested struct fields. gp-parse-struct-fields walks field rows tolerating ASI semis; each row is a name list + type. AST: (list :ty-struct FIELDS) with each field (list :field NAMES TYPE). Embedded fields, tags, and methods deferred. +8 tests, parse 98/98, total 227/227. [proposes-ast] — the :field shape (NAMES + TYPE) recurs in func params, method receivers, var decls; flagged in Blockers as a unified ast-binding-group candidate for the kit.

• 2026-05-27 — Phase 2 cont.: func-type expressions. func(), func() int, func(int, string), func(int) string, func() (int, error). AST shape (list :ty-func PARAMS RESULTS) where both are lists of type nodes. Results parsing reuses param parser for the multi-return (T, T, ...) case. Anonymous-only params for now — named params (func(a int, b string)) need a different shape and are required mainly for func DECLARATIONS not pure func-type expressions. Variadic deferred. Covers nested func-as-return and chan-of-func. +9 tests, parse 90/90, total 219/219. [nothing] — pure Go parser; type AST proposals already in Blockers.

• 2026-05-27 — Phase 2 cont.: type expressions — slice []T, array [N]T, map map[K]V, chan in all three directions (chan T, chan<- T, <-chan T). gp-parse-type now dispatches on */[/map/chan/<-/ident; each branch recurses for nested types. Channel direction is :both/:send/:recv. AST stays Go-specific tagged lists — kit has no type primitives at all. Covers nested types end-to-end (slice-of-pointer, slice-of-slice, map-with-slice-value, chan-of-map, pointer-to-slice). Parse acceptance bar (80+) crossed: +11 tests, parse 81/81, total 210/210. Func / struct / interface types and generics still pending in Phase 2. [proposes-ast] — surfaces concrete type-node proposals (slice / array / map are universal across statically-typed guests; channel direction is Go-specific). Logged in Blockers.

• 2026-05-27 — Phase 2 cont.: type assertion v.(T) postfix form. Postfix . branch now disambiguates between .field (selector) and .(...) (type assertion) by peeking at the next token. New gp-parse-type handles the minimum needed: named (int, MyType), qualified (pkg.T), pointer (*T, **T). AST shapes are Go-specific tagged lists — kit has no notion of types at all yet (this is a meta-gap: full bidirectional types arrive in Phase 3, but even the parser needs a type substrate). Covers chained, call-result, after-selector, and binary-precedence interactions. +9 tests, parse 70/70, total 199/199. [nothing].

• 2026-05-27 — Phase 2 cont.: index x[i] and slice x[a:b] / x[a:b:c] postfix forms. New gp-parse-bracket + gp-parse-bracket-expr branch off the same postfix loop as calls/selectors. AST: Go-specific (list :index OBJ IDX) and (list :slice OBJ LOW HIGH MAX) — LOW/HIGH/MAX may be nil for omitted indices. Two more kit gaps logged (no :index, no :slice in canonical AST). Permissive on a[1::3]. Covers: literal idx, var idx, expr idx, string idx, chained a[0][1], mixed a[0].field, full slice with three indices. +12 tests, parse 61/61, total 190/190. [proposes-ast].

• 2026-05-27 — Phase 2 cont.: postfix forms — function calls f(a, b) via canonical ast-app, and member access x.field via Go-specific (list :select OBJ "field"). The AST kit has no selector node; logged in Blockers as [proposes-ast] — every nominally-typed guest will hit the same gap, worth promoting on the next consumer. Postfix loop sits between unary and primary so calls bind tighter than unary (-f(x) = -(f(x))). Covers nested calls, chained selectors, methods obj.m(x), mixed precedence. +12 tests, parse 49/49, total 178/178. [consumes-ast proposes-ast].

• 2026-05-27 — Phase 2 cont.: unary prefix operators (+, -, !, ^, *, &, <-). gp-parse-unary is recursive (!!x) and sits between gp-parse-expr and gp-parse-primary so unary always binds tighter than any binary. Symbols + - * & ^ are shared with binary; the positional split (expression-start vs mid-expression) disambiguates cleanly without lookback. Unary nodes are single-arg ast-app. +11 tests, parse 37/37, total 166/166. [nothing] — pure Go parser work.

• 2026-05-27 — Phase 2 cont.: binary operators via Pratt precedence climbing. gp-pratt-loop consumes pratt-op-lookup against go-precedence-table; left-assoc bumps right-min by 1, right-assoc keeps prec. Binary op nodes are (ast-app (ast-var OP) [LHS RHS]) — uses the canonical ast-app shape rather than inventing a Go-specific binary node. Covers: equal-prec left-to-right, * tighter than +, && tighter than ||, comparison tighter than &&, long chains. +9 tests, parse 26/26, total 155/155. [consumes-pratt].

• 2026-05-27 — Phase 2 first slice: lib/go/parse.sx parser scaffold. Defines go-precedence-table using lib/guest/pratt.sx entry shape (NAME PREC ASSOC) — five Go precedence levels, all left-associative per Go spec § Operator precedence. go-parse tokenises via go-tokenize, then gp-parse-primary reads one literal / identifier and emits a canonical AST node via lib/guest/ast.sx's ast-literal / ast-var. parse 17/17, lex still 129/129, total 146/146. [consumes-pratt consumes-ast].

• 2026-05-27 — Phase 1 complete. Operator-set audit: added missing ~ (Go 1.18+ generics type-set), exhaustive op coverage tests grouped by category. Two kit gaps observed and logged in Blockers: lex-oct-digit?/lex-bin-digit? predicates + lex-match-longest table-driven prefix matcher — both useful for future statically-typed guests. +6 tests, lex 129/129. [proposes-lex]. Phase 2 (parser) next.

• 2026-05-27 — Phase 1 cont.: raw string literals (backtick-delimited). Multi-line, no escape processing, \r stripped per Go spec § String literals. Same "string" token type as interpreted strings — parsers / type checkers don't need to distinguish. +9 tests, lex 123/123. [nothing] — pure Go work; raw strings don't touch the substrate or lib/guest story.

• 2026-05-27 — Phase 1 cont.: decimal float + imaginary literals. 3.14, .5, 1., 1e10, 1.5e-3, 2i, 3.14i. gl-finish-number! handles exponent + i suffix; gl-read-number! returns the type string (int/float/imag). ASI trigger list extended to float/imag. Greedy-grammar pin: 1.method lexes as float ident. Hex floats (0x1.fp0) deferred. +22 tests, lex 114/114. [consumes-lex].

• 2026-05-27 — Phase 1 cont.: prefixed integer literals (0x.., 0X.., 0b.., 0B.., 0o.., 0O.., legacy 0123) + underscore separators in any digit run. Dispatch in gl-read-number!; consumes lex-hex-digit? from the kit. +14 tests, lex 92/92. [consumes-lex].

• 2026-05-26 — Phase 1 first slice: lib/go/lex.sx tokenizer consuming lib/guest/lex.sx predicates. 25 keywords, ident/int/string/rune lits, line+block comments, common operators, automatic semicolon insertion per Go spec § Semicolons (newline / EOF / block-comment-with-newline triggers). Scoreboard + conformance.sh wired. 78/78 tests. [consumes-lex].

• 2026-05-26 — Plan rewritten to integrate the lib/guest framework (chiselling discipline, sister plans for scheduler + bidirectional types, type-checker phase added, conformance scoreboard model adopted). Original 2026-04-26 draft preserved in git history. Loop not yet kicked off; Phase 1 (tokenizer) is the first iteration when this loop spins up.