nearDocs k t1 t2 returns docs where both terms occur within k positions
(unordered); candidates from the posting intersection, filtered on positional
postings. 205/205.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deterministic English suffix stripping (stem), stemText/stemTokens, indexStemmed.
Worked around two haskell-on-sx string gotchas: take/drop over a String yield
char codes (rebuild via joinChars . map chr), and isSuffixOf's reverse trips ++
(manual suffix compare). 196/196.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
highlight marks query-matching (normalized) tokens with [..]; snippet extracts a
context window around the first match. 178/178.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
editDist as an O(m*n) row-based Levenshtein DP (naive recursion is exponential
and times out under load); fuzzyTerms/fuzzyDocs/fuzzyRankTfIdf expand a term to
indexed terms within a max edit distance. 166/166.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
paginate windows a ranked list (take lim . drop off); pageTfIdf/pageBm25 and
resultCount. 148/148.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
prefixTerms matches indexed terms by prefix (allTerms + isPrefixOf); prefixDocs
unions their docs; prefixRankTfIdf ranks via the matched terms. 136/136.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fedIndex merges per-peer inverted indices (union posting lists per term) after
relabelling local DocIds to global gid = peer*1000 + local — dedupe by
(peer,doc-id) is automatic and positions survive, so ranking runs once over the
merge and interleaves peers by score. ACL is a post-rank filter over an injected
permit predicate (searchTfIdfAcl/topNTfIdfAcl/searchBm25Acl). Roadmap complete,
122/122.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rankTfIdf and rankBm25 (configurable k1/b) over the candidate set, float scores
with deterministic DocId tiebreak; topNTfIdf/topNBm25. df/idf derived from
posting-list length. Tests cover tf/idf behavior, a BM25-vs-TF-IDF flip from
length-norm + tf-saturation, the b-parameter effect, tiebreak stability. 101/101.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Query ADT (Term|And|Or|Not|Phrase) and evalQuery over docid-sorted posting
lists: boolean ops as linear merges, Not over the allDocs universe, Phrase via
positional adjacency. Batched both test suites into one program eval each
(search-batch) so they finish under heavy CPU load. 46/46.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tokenizer (lowercase, strip punctuation, positions) and a sorted assoc-list
inverted index [(Term,[(DocId,[Pos])])] with indexDoc/deleteDoc/lookupTerm/
docFreq/allTerms. Search lib is haskell-on-sx source assembled into search/src;
tests reuse hk-test counters via a search-eval helper. conformance.sh models
lib/haskell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plans for acl-on-sx (Datalog), flow-on-sx (Scheme), feed-on-sx (APL),
mod-on-sx (Prolog), search-on-sx (Haskell). Each is a 4-phase queue
sitting on its respective guest language, targeting rose-ash needs:
access control, durable workflows, activity feeds, moderation, search.
Federation extension in Phase 4 of each (plugs into fed-sx).
Briefings for the three loops we're kicking off now: acl-loop,
flow-loop, feed-loop. mod-sx and search-sx briefings will follow
once the first three have surfaced any shared infrastructure
worth extracting to lib/guest/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four small, contained substrate fixes that came out of the fed-sx-m1 milestone work — all scoped to
lib/erlang/, no other-language regressions:
c6f397c3 register binary_to_list/1 + list_to_binary/1 BIFs (+9 ffi tests, 738/738)
9fe5c904 $X char literals decode to char code in tokenizer (+12 eval tests, 750/750)
5098a8f0 atom_to_list/integer_to_list return Erlang charlists; list_to_* accept both (+9 eval, 759/759)
bcabed6b integer literals truncate to strict int (was float; broke integer->char)
Together these complete the byte-level term-codec primitive set:
binary_to_list / list_to_binary (iolist-aware; round-trips for free)
$X char literals decoding to int char codes
atom_to_list / integer_to_list returning standard Erlang charlists
integer literals coercing to strict int (not float)
Any Erlang-on-SX consumer that needs to construct/deconstruct byte sequences or work with charlists now
does so with standard Erlang semantics. Scoreboard: 759/759 (full Erlang suite).
Loop branch loops/erlang stays alive for future Erlang substrate work; this just lands the closed deliverables.
plans/lib-guest-scheduler.md and plans/lib-guest-static-types-
bidirectional.md both have Phase 1 ticked complete from Go's side
with status blocks enumerating what landed.
Each sister diary received a consolidated chisel-summary entry:
the kit primitives the Go consumer chiselled out, the three
pluggable predicates / orthogonal first-class-tag axes, and the
v0 limitations the eventual kit must lift.
No new Go code — Phase 10 is doc-only per plan. Go-on-SX loop
fully landed: 11 phases, 7 test suites, 609/609 passing.
Two-consumer rule per sister plan now waits on TypeScript (Phase 2
of the bidirectional sister plan, owned outside this loop).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12 canonical Go programs running through the full pipeline (lex +
parse + types + eval + sched + stdlib): sieve-of-Eratosthenes via
boolean slice (modulo-free), linear search, slice reverse, fib(10),
sum-of-squares via generic Map+Reduce, word-freq counter, channel
pipeline (gen→sq→sum), worker pool, bubble sort, sentence-reverse,
Filter+len, Ackermann, defer+recover on div-by-zero.
Each test threads ONE self-contained Go program through go-eval-
program. The v0 limitations chiselled in earlier phases (float
division, sync spawn, type erasure, nil-as-unbound) are now
durable as commit-trail artifacts; e2e variants written to avoid
them where possible. HTTP-ish ping-pong + WaitGroup deferred
(real preemption + sync package needed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New :go-package NAME ENTRIES value type with field lookup via
extended go-eval-select. New :go-builtin-fn callable for closure-
based stdlib functions. lib/go/std/strings.sx ships 12 functions
(Contains, HasPrefix, HasSuffix, Index, Count, Repeat, Join,
ToUpper, ToLower, TrimSpace, Split, Replace) + lib/go/std/strconv.sx
ships Itoa/Atoi.
Pre-existing bug fixed: parser was emitting (:literal V) for both
`42` and `"42"`, relying on first-char heuristic in eval/types.
Now emits :literal-string for string/rune literals so Atoi("42")
correctly receives the string. 3 parse tests + 2 in-composite-key
tests updated to new shape.
Total 597/597. Stdlib 41/41 — +40 acceptance bar cleared. Sister
diary documents the 11 value-type kinds (struct/slice/map/chan/
fn/method/builtin/builtin-fn/package/panic/defer) all sharing the
"(:KIND PAYLOAD...)" shape, alongside AST nodes and sentinel signals
as the kit's three orthogonal first-class-tag axes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Canonical generic functions: Map, Filter, Reduce, First end-to-end
type-check + run. Plus 20+ typer-only shape tests covering Apply,
Compose, ToMap, Swap, Box, Triple, ToSlice, Take, Send, Fill, Eq,
Values, Pair, Inspect, etc. Index synth (slice/array/map →
element type) added to typer.
v0 limitations stamped in tests: SX `/` is float (no int mod
emulation), `var r []T` indistinguishable from unbound, single-name
constraints opaque (no type-set arithmetic).
Shape locked in: "the parser recognizes shapes, the validator
recognizes roles." Same AST + different role-validators = different
guest semantics. Diary documents this as the lemma the kit should
extract — three deliverables (binding-groups, control-flow sentinels,
index synthesis) now all instantiate it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gp-parse-type-params consumes the optional [NAMES CONSTRAINT, ...]
clause after a func name. AST stays backward-compatible: 5-slot
func-decl when no [...] is present, 6-slot when it is.
Typer binds each type-param name as (:ty-param NAME CONSTRAINT) so
body's (:ty-name "T") references resolve. Eval is type-erasing —
ignores type info, dispatches by name + arity.
10 new tests: parse (3), types (5), eval (2). Total 527/527.
Shape: the field binding-group from the canonical kit now feeds
6 consumers (struct fields, var-decls, const-decls, params,
receivers, type-params). Confirms it as a TRUE cross-deliverable
shape — sister-plan diary documents the 5 roles binding-groups
take and why the kit should expose ONE parser + pluggable validators.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wired panic through :go stmt (v0 sync surfaces back to spawner —
matches real Go's "crash whole program" end-effect) and through
go-eval-for (was swallowing panic at the loop boundary).
8 tests added: goroutine-panic-surfaces, goroutine-recover-via-
spawner-defer, multi-defer-LIFO-with-recover, defer-fires-on-panic-
path, panic(nil), panic-in-loop, defer-still-runs-in-panicking-fn,
args-eager-on-panic-path. 20 Phase-6 tests total; +20 acceptance
bar cleared (eval/ 80 → 100).
Shape: 4 control-flow sites now repeat the same sentinel dispatch
arm (return-value, break, continue, eval-error, go-panic). The
scheduler kit should bake in a single propagates? helper rather
than have each guest evaluator list every sentinel inline — diary
documents the cross-cutting abstraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Panic/recover builtins + per-frame __go-panic-cell of shape
(STATE V). Body panic flips cell :none→:raised BEFORE defers drain
so recover() can find it. recover() walks env chain past shadowing
cells to the outermost :raised one — flips it :recovered, returns V.
Frame exit checks cell: :recovered → return clean; :raised →
propagate (:go-panic V).
6 tests: uncaught-from-program, panic-from-fn, defer-recover-swallow,
recover-captures-via-channel, propagation-through-no-defer-chain,
middle-frame-catches-deeper-panic.
Shape: panic cell is a frame-attached out-of-band channel that
survives function boundaries via env-chain walk. Same primitive
slots into the scheduler kit's termination-record + cleanup-with-
error-context hook. Maps cleanly to Erlang try/catch/after.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 6 first slice. New :defer stmt dispatch, go-eval-defer-stmt
captures (callee, eagerly-evaluated args) onto a frame-local
__go-defer-stack mutable list. go-eval-call installs the stack and
drains LIFO before returning; go-eval-program does the same for
the implicit main frame. New :quoted-value AST node lets defer
re-invoke calls with the frozen arg values.
6 eval tests: single defer, multi-LIFO, args-eager-at-defer-time,
fires-on-early-return, frame-local (no bleed to outer), defer-in-loop.
Shape: defer is a per-frame cleanup queue (LIFO on frame exit) that
the scheduler kit will reuse for panic-unwind + clean-exit + select-
case-rollback paths. Distinct from the scheduler's ready-queue —
diary updated to keep that distinction explicit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Acceptance bar hit (40 runtime, 497 total). Tests: timer ready,
select-with-timeout, fan-in (3 producers), worker queue, pipeline,
fan-out-then-fan-in, select source-order, fallback case, default,
producer-consumer, two-stage pipeline, channel-counter, after+default,
tick-collector.
Shape chiselled: timer collapses "after duration" into
"channel ready immediately" — select needs only ready? from each
case. Real time is when the flip happens, not what the protocol is.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 cont. New go-eval-range-for handles the parser's :range-for
AST shape. Dispatches on the collection's runtime type:
:go-slice → bind index + element, iterate by position
:go-map → bind key + value, walk entries assoc list
:go-chan → bind value, drain until buffer empty (v0 limitation)
Each loop carries:
- go-range-extend: handles 0/1/2-name binding patterns uniformly
- go-range-body: evaluates body whether it's a :block or other shape
- per-collection loop helper: threads env, catches :break/:continue/
:return-value/:eval-error sentinels
**Subtle break fix:** loops were previously returning the *pre-loop*
env when break fired, clobbering all assignments made in prior
iterations. Now returns the current iteration's input env (which
carries forward successful iterations' state). Patched for the three
range variants and for the regular for-loop where the same pattern
applied. The shape:
(= r :break) env ;; was: (= r :break) original-env
Tests:
range: slice — sum of 1..5 = 15
range: slice — key only (index)
range: map — sum values
range: channel — collect all buffered
range: slice with break exits early
range: slice with continue skips an element
range: empty slice — body never runs
range: chan + goroutine producer
runtime 26/26, total 483/483.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 cont. Adds `select` statement evaluation:
go-select-try-case env COMM →
:not-ready / extended-env / :eval-error
go-select-pick env CASES DEFAULT-OR-NIL →
body-result / blocked-error
go-eval-select-stmt env STMT — public entry
Walks cases in declared order:
* :send case — always ready in v0 (unbounded buffer). Sends value
via go-chan-send! and returns env unchanged.
* :short-decl / :assign case — RHS expected to be unary <- on a
channel. Ready iff go-chan-len > 0; on success, recv-into-var
binds the new value in env.
* Bare recv (:app (:var "<-") [CHAN]) — ready iff len > 0; consumes
the value (discarded).
* :default — deferred until end of walk. Runs if no other case
ready. Absence + no ready case → (:eval-error :select-blocked-
no-default).
New `go-chan-len` accessor on the channel closure-bundle so the
select can peek without consuming.
Subtle bug fix: the :select stmt branch in go-eval-stmt was returning
the old env instead of the env returned by the case body. Assignments
inside select cases (`select { case <-ch: x = 1 ; default: x = 99 }`)
now stick.
Tests (6):
default fires when no case ready
recv case fires when ready
recv-into-var binds the value
send case always ready
picks first ready case (deterministic order in v0)
no default + nothing ready → blocked error
combined with goroutine fan-in
runtime 18/18, total 475/475.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 (goroutines + channels) opens.
lib/go/sched.sx is the **independent implementation** referenced by
plans/lib-guest-scheduler.md — the first-consumer cut whose realised
shape will inform the eventual sister kit.
Channel representation:
(list :go-chan SEND-FN RECV-FN CLOSED?-FN CLOSE!-FN)
Each closure shares a mutable `buf` (a list mutated via append! and
set!) and a `closed` flag. Channel identity is closure-instance —
two `make()` calls produce distinct values per Go spec § Channel types.
Primitive API in sched.sx:
go-make-chan / go-chan? / go-chan-send! / go-chan-recv! /
go-chan-closed? / go-chan-close!
Eval integration in eval.sx:
* `make` and `close` added as builtins. v0 `make()` takes no args
and returns an unbounded-buffer channel.
* `:send` stmt → go-chan-send! on the channel.
* Unary `<-` recv on channel values → go-chan-recv!. `:empty`
sentinel converted to nil (stand-in for blocking semantics).
* `:go expr` → synchronous eval (v0 limitation, see sched.sx
header).
**v0 concurrency model — synchronous goroutines.** SX doesn't expose
first-class continuations to guest code, so v0 runs `go f()`
immediately and depends on the spawned goroutine running to
completion before the main goroutine receives. This is the right
semantics for the simple producer/consumer patterns covered here.
True preemption with blocking send/recv is Phase 5b — requires either
a CEK-style trampolining eval rewrite or kit-level continuation
support. Logged in sched.sx header and in the sister-plan diary.
Runtime suite (12 tests):
* 6 direct API tests: identity, FIFO order, closed-flag
* 6 source-level: make + send + recv, go ping-pong, close,
multi-goroutine fan-in, worker-with-result
Sister-plan scheduler diary updated with the channel-as-closure-
bundle insight and the v0 synchronous-spawn caveat.
runtime 12/12, total 469/469.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 4 cont. The crossings:
* Method dispatch — Methods record under #method/TYPE/NAME (same
mangled-key scheme the type checker uses, intentionally so eval
and type checker can converge on a shared method-table protocol
later). go-eval-method-call: lookup the receiver type's method,
bind receiver param to the struct value, evaluate body. Value and
pointer receivers treated the same in v0 (pointer semantics not
modelled yet).
* Method-call dispatch — In go-eval's :app branch, head=:select
routes to go-eval-method-call. If the receiver is not a struct,
falls back to the field-as-callable path.
* Unary prefix ops — go-eval's :app branch checks for 1-arg :var
head with op name "-" / "+" / "!". (Other unary ops like
*p / &v / <-ch / ^x deferred until pointer / channel / bitwise
semantics arrive.)
End-to-end programs verified:
* recursive fib(10) = 55
* struct + method + iterative loop (counter bump 7 times)
* linear search (returns index or -1)
* factorial via method on Counter (= 120)
* count odd numbers in 1..10 = 5
**Phase 4 acceptance bar (80+) crossed: eval 80/80, total 457/457.**
Remaining Phase 4 work (closures, multi-return, full slice triple,
pointer semantics) refines but doesn't gate Phase 5 (goroutines).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 4 cont. Adds runtime support for Go's struct type.
Struct representation: (list :go-struct TYPE-NAME FIELDS) where
FIELDS is an association list of (field-name value) pairs.
`type T struct { ... }` is now significant at eval-time. The new
go-eval-type-decl registers field-name lists in env under
(:go-struct-type FIELD-NAMES) so positional composite literals can
map argument positions to field names. Non-struct type aliases are
silent no-ops in v0.
go-eval-composite extended:
* If type is (:var TYPE-NAME), look up in env. Must be a
:go-struct-type entry — error otherwise.
* go-eval-struct-lit branches on whether the first elem is :kv
(keyed) or not (positional). Keyed mode reads key-name from each
:kv's key (which is a :var node). Positional mode arity-checks
against the field-names list and zips positionally.
go-eval-select handles (:select OBJ FIELD-NAME) — field lookup with
go-map-get on the FIELDS assoc list.
go-eval-assign-pairs gets a new (:select OBJ FIELD) LHS branch:
- var-rooted only for v0
- rebuilds the struct via go-map-set, rebinds the var
**Functions taking and returning structs round-trip end-to-end:**
type Point struct { x, y int }
func add(a, b Point) Point { return Point{a.x + b.x, a.y + b.y} }
add(Point{1, 2}, Point{3, 4}) // Point{4, 6}
Method-dispatch (calling p.M() where M is a method on Point's type)
is the next step; needs threading the type checker's #method/T/N
scheme into eval-time so functions can be looked up by receiver type.
eval 66/66, total 443/443.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 4 cont. Adds map values and index-assignment for both
slices and maps.
Map representation: (list :go-map ENTRIES) where ENTRIES is an
association list of (key value) pairs.
go-map-get / go-map-set — primitive lookup + functional-update.
go-slice-set — same idea for slices.
go-extract-map-entries reads each :kv element in a composite literal,
evaluating key and value. go-eval-composite dispatches on :ty-map to
build the :go-map value.
go-eval-index extended: when OBJ is a :go-map, look up the key via
go-map-get. Missing keys return nil in v0 (Go's real semantics is
the zero value of the value type — needs runtime type info that this
slice doesn't yet thread through).
go-eval-builtin's len handles :go-map alongside :go-slice and strings.
go-eval-assign-pairs gets a new branch for (:index OBJ IDX) LHS:
- var-rooted indexing only (a[i] = v / m["k"] = v)
- slice → go-slice-set then rebind the var
- map → go-map-set then rebind the var
**Word-counter via map[string]int works end-to-end:**
words := []string{"a", "b", "a", "c", "a"}
counts := map[string]int{}
for i := 0; i < len(words); i++ {
counts[words[i]] = counts[words[i]] + 1
}
// counts["a"] == 3
Builds on:
- map composite literal eval
- map index lookup
- map index-assign
- slice indexing
- len() builtin
- nil + 1 = 1 (numeric-coercion of missing-key default)
eval 58/58, total 435/435.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 4 cont. Adds runtime support for Go's slice type.
Slice representation: (list :go-slice ELEMS) — a simple wrapper around
a list of element values. v0 deferring the full
(length, capacity, backing-vector) triple from the Go spec until
programs need it.
go-eval-composite → for (:composite TYPE-OR-EXPR ELEMS) where
TYPE is :ty-slice / :ty-array, eval each
element (handling :kv index-keyed
shorthand by taking only the value) and
wrap in :go-slice.
go-eval-index → (:index OBJ IDX). Bounds-checked; out-of-
range returns (:eval-error :index-out-of-range).
go-eval-slice → (:slice OBJ LOW HIGH MAX). Two-index slice
with omitted low → 0, omitted high → len.
Returns a new :go-slice.
go-list-slice → primitive list-slicing helper.
Builtins live in a new starter env go-env-builtins:
len(slice|string) → count
append(slice, ...x) → new slice with x appended
print(...) → no-op in v0
Builtins are bound as (:go-builtin NAME); go-eval-call recognises the
shape and routes to go-eval-builtin instead of go-eval-fn.
**Summing a slice via the canonical Go for-loop works end-to-end:**
a := []int{1, 2, 3, 4, 5}
sum := 0
for i := 0; i < len(a); i++ {
sum = sum + a[i]
}
// sum == 15
eval 50/50, total 427/427.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
