From c3a0727645816006231bc1e4559f8bbc891c3dae Mon Sep 17 00:00:00 2001 From: giles Date: Sat, 6 Jun 2026 15:55:39 +0000 Subject: [PATCH] plans: five rose-ash subsystem plans + three loop briefings Plans for acl-on-sx (Datalog), flow-on-sx (Scheme), feed-on-sx (APL), mod-on-sx (Prolog), search-on-sx (Haskell). Each is a 4-phase queue sitting on its respective guest language, targeting rose-ash needs: access control, durable workflows, activity feeds, moderation, search. Federation extension in Phase 4 of each (plugs into fed-sx). Briefings for the three loops we're kicking off now: acl-loop, flow-loop, feed-loop. mod-sx and search-sx briefings will follow once the first three have surfaced any shared infrastructure worth extracting to lib/guest/. Co-Authored-By: Claude Opus 4.7 (1M context) --- plans/acl-on-sx.md | 102 ++++++++++++++++++++++++++ plans/agent-briefings/acl-loop.md | 93 ++++++++++++++++++++++++ plans/agent-briefings/feed-loop.md | 99 +++++++++++++++++++++++++ plans/agent-briefings/flow-loop.md | 98 +++++++++++++++++++++++++ plans/feed-on-sx.md | 105 +++++++++++++++++++++++++++ plans/flow-on-sx.md | 108 ++++++++++++++++++++++++++++ plans/mod-on-sx.md | 112 +++++++++++++++++++++++++++++ plans/search-on-sx.md | 106 +++++++++++++++++++++++++++ 8 files changed, 823 insertions(+) create mode 100644 plans/acl-on-sx.md create mode 100644 plans/agent-briefings/acl-loop.md create mode 100644 plans/agent-briefings/feed-loop.md create mode 100644 plans/agent-briefings/flow-loop.md create mode 100644 plans/feed-on-sx.md create mode 100644 plans/flow-on-sx.md create mode 100644 plans/mod-on-sx.md create mode 100644 plans/search-on-sx.md diff --git a/plans/acl-on-sx.md b/plans/acl-on-sx.md new file mode 100644 index 00000000..898c8138 --- /dev/null +++ b/plans/acl-on-sx.md @@ -0,0 +1,102 @@ +# acl-on-sx: Access Control on Datalog + +rose-ash needs fine-grained, explainable, federation-aware access control. Subjects +(users, groups, roles, services) × actions (read, edit, comment, moderate, federate) +× resources (pages, posts, threads, peers). Decisions must come with a trace — not just +permit/deny, but **why**. + +Datalog's bottom-up rule engine produces transparent permit/deny chains: the proof tree +is the audit trail. Inheritance over groups + resource hierarchies is recursive Datalog +in one rule. Federation extends naturally — fed-sx replicates ACL facts, peers reason +over the union. + +End-state: a Datalog-on-SX layer specifically for ACL, with explanation API, audit log, +and federation extension. Reuses `lib/datalog/` evaluator and term model where possible. + +## Status (rolling) + +`bash lib/acl/conformance.sh` → **0/0** (not yet started) + +## Ground rules + +- **Scope:** only touch `lib/acl/**` and `plans/acl-on-sx.md`. Do **not** edit `spec/`, + `hosts/`, `shared/`, `lib/datalog/**`, or other `lib//`. You may **import** + from `lib/datalog/` (its public API in `lib/datalog/datalog.sx`); do **not** copy or + modify Datalog code. +- **Shared-file issues** go under "Blockers" with a minimal repro; do not fix here. +- **SX files:** use `sx-tree` MCP tools only. +- **Architecture:** thin layer on top of `lib/datalog/`. Define schema, surface API, + audit + federation hooks. The rule engine itself is Datalog's. +- **Watch for shared patterns** going into `lib/guest/` — both acl-sx and mod-sx need + rule-engine plumbing. If you find shared shape, flag it for extraction (don't + extract yet — wait for mod-sx to start). +- **Commits:** one feature per commit. Keep Progress log updated and tick boxes. + +## Architecture sketch + +``` +ACL declarations (SX) User query + │ │ + ▼ ▼ +lib/acl/schema.sx lib/acl/api.sx + — subject sorts — (acl/permit? subj act res) + — resource sorts — (acl/explain subj act res) + — action sorts — (acl/audit subj act res :allowed?) + — fact schema │ + │ ▼ + ▼ lib/acl/engine.sx +lib/acl/facts.sx — builds Datalog query + — actor(id, kind) — invokes lib/datalog/ + — resource(id, kind) — extracts proof tree + — member_of(actor, group) │ + — child_of(res, parent) ▼ + — grant(actor, act, res) lib/acl/audit.sx + — deny (actor, act, res) — persistent decision log + — query API +``` + +## Phase 1 — Direct grants + +- [ ] `lib/acl/schema.sx` — sorts: subject {user, group, role, service}, action, + resource {page, post, thread, peer} +- [ ] `lib/acl/facts.sx` — `actor`, `resource`, `grant`, `deny` predicates as Datalog + EDB +- [ ] `lib/acl/engine.sx` — `(permit? subj act res db)` reduces to Datalog query +- [ ] `lib/acl/api.sx` — public `(acl/permit? ...)` taking implicit current db +- [ ] `lib/acl/tests/direct.sx` — 15+ cases: direct grant, missing grant, explicit deny +- [ ] `lib/acl/scoreboard.{json,md}` baseline +- [ ] `lib/acl/conformance.sh` runs the suite + +## Phase 2 — Inheritance + +- [ ] `member_of(actor, group)` chain — group grants apply to members (transitive) +- [ ] `child_of(res, parent)` chain — parent grants apply to children (transitive) +- [ ] role expansion — role contains list of (action, resource) tuples +- [ ] deny-overrides — explicit deny wins over inherited allow +- [ ] `lib/acl/tests/inherit.sx` — 25+ cases: nested groups, deep resource trees, + conflict resolution, deny precedence +- [ ] document the deny-overrides choice in plan + +## Phase 3 — Explanation + audit + +- [ ] `(acl/explain subj act res)` → `{:allowed? T :proof }` +- [ ] proof tree extracts from Datalog's derivation +- [ ] `lib/acl/audit.sx` — append-only decision log (in-memory + serializer for disk) +- [ ] `(acl/audit-tail n)` for recent decisions +- [ ] `lib/acl/tests/explain.sx` — proof correctness, audit completeness + +## Phase 4 — Federation + +- [ ] peer trust facts — `peer(addr, kind)`, `trust(peer, level)` +- [ ] delegated grants — `delegate(peer, actor, action, resource)` +- [ ] cross-instance permit chain — query asks local + queries trusted peers via fed-sx +- [ ] revocation propagation — fact retraction across federation +- [ ] `lib/acl/tests/fed.sx` — federated grant chains (mock fed-sx transport in tests) + +## Progress log + +(loop fills this in) + +## Blockers + +(loop fills this in) diff --git a/plans/agent-briefings/acl-loop.md b/plans/agent-briefings/acl-loop.md new file mode 100644 index 00000000..a224fa40 --- /dev/null +++ b/plans/agent-briefings/acl-loop.md @@ -0,0 +1,93 @@ +# acl-on-sx loop agent (single agent, queue-driven) + +Role: iterates `plans/acl-on-sx.md` forever. **First subsystem loop after fed-sx.** +Sits on `lib/datalog/` — rule engine reused, schema/api/audit/federation added on +top. The deliverable isn't "implement Datalog ACL"; it's *also* to surface shared +rule-engine plumbing into `lib/guest/` (the mod-sx loop will be the second consumer, +validating extraction). + +``` +description: acl-on-sx queue loop +subagent_type: general-purpose +run_in_background: true +isolation: worktree +``` + +## Prompt + +You are the sole background agent working `/root/rose-ash/plans/acl-on-sx.md`. +Isolated worktree, forever, one commit per feature. Push to `origin/loops/acl` +after every commit. + +## Restart baseline — check before iterating + +1. Read `plans/acl-on-sx.md` — roadmap + Progress log. +2. `ls lib/acl/` — pick up from the most advanced file. +3. If `lib/acl/tests/*.sx` exist, run them via `bash lib/acl/conformance.sh`. Green + before new work. +4. If `lib/acl/scoreboard.md` exists, that's your baseline. +5. Read `lib/datalog/datalog.sx` public API once — that's your substrate. + +## The queue + +Phase order per `plans/acl-on-sx.md`: + +- **Phase 1** — direct grants. Schema, EDB facts, engine, api, 15+ tests +- **Phase 2** — inheritance (member_of, child_of, role expansion, deny-overrides) +- **Phase 3** — explanation + audit (proof tree, audit log) +- **Phase 4** — federation (peer trust, delegation, cross-instance permit chain) + +Within a phase, pick the checkbox that unlocks the most tests per effort. + +Every iteration: implement → test → commit → tick `[ ]` → Progress log → next. + +## Ground rules (hard) + +- **Scope:** only `lib/acl/**` and `plans/acl-on-sx.md`. Do **not** edit `spec/`, + `hosts/`, `shared/`, other `lib//` dirs, `lib/stdlib.sx`, or `lib/` root. + May **import** from `lib/datalog/` only (its public API). +- **NEVER call `sx_build`.** 600s watchdog. If sx_server binary broken → Blockers + entry, stop. +- **Shared-file issues** → plan's Blockers with minimal repro. +- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after edits. +- **Worktree:** commit, then push to `origin/loops/acl`. Never touch `main` or + `architecture`. +- **Commit granularity:** one feature per commit. Short factual messages + (`acl: child_of resource inheritance + 8 tests`). +- **Plan file:** update Progress log + tick boxes every commit. +- **Watch for shared infrastructure** with future mod-sx (Prolog moderation). If you + build a generic rule-engine adapter, note it in Progress log so the eventual + `lib/guest/rules/` extraction has both consumers identified. + +## ACL-specific gotchas + +- **Datalog is bottom-up.** No goal-directed search. Don't reach for cut or + backtracking — that's mod-sx's job. Your decisions emerge from fixpoint. +- **Deny-overrides** is the policy: if both an allow and deny rule fire, deny wins. + Encode this via stratified negation; document the choice clearly in plan. +- **Inheritance termination:** recursive rules with `member_of` chains must + terminate. Datalog guarantees this absent function symbols — don't introduce them + in your schema. +- **Proof tree shape:** Datalog's derivation graph is a DAG, not a tree, when the + same fact is derived multiple ways. For audit, pick one canonical derivation + (shortest, or first); document choice. +- **Federation isn't transitive trust.** A peer's `delegate(...)` fact only applies + if local `trust(peer, level)` covers the action class. Re-check trust on every + query, not at fact-ingestion time. + +## General gotchas (all loops) + +- SX `do` = R7RS iteration. Use `begin` for multi-expr sequences. +- `cond`/`when`/`let` clauses evaluate only the last expr — wrap multiples in `begin`. +- `env-bind!` creates a binding; `env-set!` mutates an existing one (walks scope chain). +- `sx_validate` after every structural edit. +- `list?` returns false on raw JS Arrays — host data must be SX-converted. + +## Style + +- No comments in `.sx` unless non-obvious. +- No new planning docs — update `plans/acl-on-sx.md` inline. +- Short, factual commit messages. +- One feature per iteration. Commit. Log. Push. Next. + +Go. Start by reading the plan; find the first unchecked `[ ]`; implement it. diff --git a/plans/agent-briefings/feed-loop.md b/plans/agent-briefings/feed-loop.md new file mode 100644 index 00000000..1c89f9c0 --- /dev/null +++ b/plans/agent-briefings/feed-loop.md @@ -0,0 +1,99 @@ +# feed-on-sx loop agent (single agent, queue-driven) + +Role: iterates `plans/feed-on-sx.md` forever. **Activity feeds on APL** — timelines, +notifications, fanout, ranking, all as APL array math on activity vectors. Densest +possible expression of feed composition. Sits on `lib/apl/` (450+/450+ tests +already); adds a feed-shaped vocabulary on top. + +``` +description: feed-on-sx queue loop +subagent_type: general-purpose +run_in_background: true +isolation: worktree +``` + +## Prompt + +You are the sole background agent working `/root/rose-ash/plans/feed-on-sx.md`. +Isolated worktree, forever, one commit per feature. Push to `origin/loops/feed` +after every commit. + +## Restart baseline — check before iterating + +1. Read `plans/feed-on-sx.md` — roadmap + Progress log. +2. `ls lib/feed/` — pick up from the most advanced file. +3. If `lib/feed/tests/*.sx` exist, run them via `bash lib/feed/conformance.sh`. Green + before new work. +4. If `lib/feed/scoreboard.md` exists, that's your baseline. +5. Read `lib/apl/apl.sx` public API once — that's your substrate. Familiarize + yourself with at least: `⍳ ⍴ / ⌽ ↑ ↓ ⌷ ∊ ∘.× /\ ⍋` (you will use all of these). + +## The queue + +Phase order per `plans/feed-on-sx.md`: + +- **Phase 1** — stream model + basic ops (record schema, filter, sort, take) +- **Phase 2** — **THE SHOWCASE**: fanout via outer product. activities `∘.×` + followers → inbox matrix, flatten + dedupe +- **Phase 3** — aggregation + ranking (group-by, velocity, recency, top-N) +- **Phase 4** — visibility filter (acl-sx) + federation (fed-sx inbox + backfill) + +Within a phase, pick the checkbox that unlocks the most tests per effort. + +Every iteration: implement → test → commit → tick `[ ]` → Progress log → next. + +## Ground rules (hard) + +- **Scope:** only `lib/feed/**` and `plans/feed-on-sx.md`. Do **not** edit `spec/`, + `hosts/`, `shared/`, other `lib//` dirs, `lib/stdlib.sx`, or `lib/` root. + May **import** from `lib/apl/` only (its public API). +- **NEVER call `sx_build`.** 600s watchdog. If sx_server binary broken → Blockers + entry, stop. +- **Shared-file issues** → plan's Blockers with minimal repro. +- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after edits. +- **Unicode in `.sx`:** raw UTF-8 only, never `\uXXXX` escapes. APL glyphs land + directly in source. +- **Worktree:** commit, then push to `origin/loops/feed`. Never touch `main` or + `architecture`. +- **Commit granularity:** one feature per commit. Short factual messages + (`feed: outer-product fanout + dedupe by (actor,verb,object) + 9 tests`). +- **Plan file:** update Progress log + tick boxes every commit. + +## feed-specific gotchas + +- **Activities are heterogeneous.** Different verbs carry different shapes + (`:object` might be page-id, post-id, user-id). Don't over-normalize — keep + `:tags` as a flexible bag. APL operations over heterogeneous records work fine + via dict lookups; only the indexed fields need uniform shape. +- **Fanout produces matrices fast.** N activities × M followers → NM items. Apply + filter/dedupe early, not after materialization. Use guard predicates *inside* + the outer product where possible (compose with `∘.{a v ⊢ ...}`). +- **Dedupe key isn't always `(actor,verb,object)`.** For "alice liked X" and "bob + liked X" the dedupe key is `(verb,object)` (collapse the actors into a list). + For "alice posted X" each `:actor` is distinct. Each verb may want its own + dedupe rule; codify these in `lib/feed/dedupe.sx`. +- **Recency decay matters more than score precision.** Use a simple half-life decay + (e.g. score × 0.5^(age/window)) rather than a clever curve. Calibrate the + window via tests, not theory. +- **Ranking should be deterministic on ties.** Always include a tiebreaker (id, or + hash). Otherwise tests will flake. +- **The ACL filter is per-viewer.** A timeline is computed *for* a user; the same + candidate stream produces different timelines for different viewers. Don't + cache pre-ACL timelines. + +## General gotchas (all loops) + +- SX `do` = R7RS iteration. Use `begin` for multi-expr sequences. +- `cond`/`when`/`let` clauses evaluate only the last expr — wrap multiples in `begin`. +- `env-bind!` creates a binding; `env-set!` mutates an existing one (walks scope chain). +- `sx_validate` after every structural edit. +- `list?` returns false on raw JS Arrays — host data must be SX-converted. + +## Style + +- No comments in `.sx` unless non-obvious. +- No new planning docs — update `plans/feed-on-sx.md` inline. +- Short, factual commit messages. +- One feature per iteration. Commit. Log. Push. Next. + +Go. Start by reading the plan; find the first unchecked `[ ]`; implement it. diff --git a/plans/agent-briefings/flow-loop.md b/plans/agent-briefings/flow-loop.md new file mode 100644 index 00000000..3e7acd49 --- /dev/null +++ b/plans/agent-briefings/flow-loop.md @@ -0,0 +1,98 @@ +# flow-on-sx loop agent (single agent, queue-driven) + +Role: iterates `plans/flow-on-sx.md` forever. **Durable workflows on Scheme** — the +call/cc + delimited continuation showcase that justifies pulling R7RS into +production. art-dag's natural successor: DAG-of-tasks with pause/resume across +process restarts. fed-sx extension turns local flows into distributed ones. + +``` +description: flow-on-sx queue loop +subagent_type: general-purpose +run_in_background: true +isolation: worktree +``` + +## Prompt + +You are the sole background agent working `/root/rose-ash/plans/flow-on-sx.md`. +Isolated worktree, forever, one commit per feature. Push to `origin/loops/flow` +after every commit. + +## Restart baseline — check before iterating + +1. Read `plans/flow-on-sx.md` — roadmap + Progress log. +2. `ls lib/flow/` — pick up from the most advanced file. +3. If `lib/flow/tests/*.sx` exist, run them via `bash lib/flow/conformance.sh`. Green + before new work. +4. If `lib/flow/scoreboard.md` exists, that's your baseline. +5. Read `lib/scheme/scheme.sx` public API once — that's your substrate. + +## The queue + +Phase order per `plans/flow-on-sx.md`: + +- **Phase 1** — declarative DAG: `defflow`, `sequence`, `parallel`, sync runtime, + basic api +- **Phase 2** — control flow + error handling: `cond`, `retry`, `timeout`, + `try-catch` +- **Phase 3** — **THE SHOWCASE**: `suspend`/`resume` via `call/cc`, persistent + store, crash recovery +- **Phase 4** — distributed nodes via fed-sx (remote-node, handoff, replication) + +Within a phase, pick the checkbox that unlocks the most tests per effort. + +Every iteration: implement → test → commit → tick `[ ]` → Progress log → next. + +## Ground rules (hard) + +- **Scope:** only `lib/flow/**` and `plans/flow-on-sx.md`. Do **not** edit `spec/`, + `hosts/`, `shared/`, other `lib//` dirs, `lib/stdlib.sx`, or `lib/` root. + May **import** from `lib/scheme/` only (its public API). +- **NEVER call `sx_build`.** 600s watchdog. If sx_server binary broken → Blockers + entry, stop. +- **Shared-file issues** → plan's Blockers with minimal repro. +- **SX files:** `sx-tree` MCP tools ONLY. `sx_validate` after edits. +- **Worktree:** commit, then push to `origin/loops/flow`. Never touch `main` or + `architecture`. +- **Commit granularity:** one feature per commit. Short factual messages + (`flow: retry combinator with exponential backoff + 6 tests`). +- **Plan file:** update Progress log + tick boxes every commit. + +## flow-specific gotchas + +- **Continuations must be re-entrant.** Phase 3's `suspend` captures a continuation + that may be re-entered after a process restart. That means: no captured file + descriptors, no captured sockets, no captured live runtime references that won't + survive serialization. State referenced by the continuation must be plain SX data + or live in the flow store. +- **call/cc, not call-with-escape-continuation.** R7RS distinguishes. Use the full + call/cc for resume; escape-only continuations cannot be re-entered. Read + `lib/scheme/r7rs.md` (or equivalent) to confirm semantics. +- **`parallel` in Phase 1 is sequential.** Don't try threading until Phase 3+. Just + evaluate branches in order, collect results, return joined value. Document the + semantics clearly so users don't assume true concurrency. +- **Retry doesn't retry continuations.** If a node has already suspended, retry on + resume doesn't re-run it from scratch — it resumes. `retry` only applies to + exceptions raised before suspend. Be explicit in the API. +- **Cancellation invalidates the continuation.** `(flow/cancel id)` must remove the + stored continuation so a stale `resume` cannot wake it. Document semantics. +- **Timeouts in pure SX are tricky.** Without a scheduler, `timeout` is a budget on + step count or wall-clock probed at safe points. Pick one approach (probably step + budget for determinism) and document. + +## General gotchas (all loops) + +- SX `do` = R7RS iteration. Use `begin` for multi-expr sequences. +- `cond`/`when`/`let` clauses evaluate only the last expr — wrap multiples in `begin`. +- `env-bind!` creates a binding; `env-set!` mutates an existing one (walks scope chain). +- `sx_validate` after every structural edit. +- `list?` returns false on raw JS Arrays — host data must be SX-converted. + +## Style + +- No comments in `.sx` unless non-obvious. +- No new planning docs — update `plans/flow-on-sx.md` inline. +- Short, factual commit messages. +- One feature per iteration. Commit. Log. Push. Next. + +Go. Start by reading the plan; find the first unchecked `[ ]`; implement it. diff --git a/plans/feed-on-sx.md b/plans/feed-on-sx.md new file mode 100644 index 00000000..53acd1fd --- /dev/null +++ b/plans/feed-on-sx.md @@ -0,0 +1,105 @@ +# feed-on-sx: Activity Feeds on APL + +Timelines, notifications, activity aggregation. The math is array math: filter, sort, +reduce, scan, outer product. APL is the densest possible expression of feed +composition — a fanout-and-rank pipeline reads as a single line. + +rose-ash needs: per-user home timeline, notification feed, activity stream digestion, +backfill for new follows, deduplication across cross-posts. Every operation is an +array-shaped transformation. + +End-state: an APL-flavored layer on `lib/apl/` with feed-specific combinators +(`fanout`, `dedupe`, `score`, `rank`), an SX adapter for callers who don't want raw +APL, ACL visibility filtering via `lib/acl/`, federation via fed-sx. + +## Status (rolling) + +`bash lib/feed/conformance.sh` → **0/0** (not yet started) + +## Ground rules + +- **Scope:** only touch `lib/feed/**` and `plans/feed-on-sx.md`. Do **not** edit + `spec/`, `hosts/`, `shared/`, `lib/apl/**`, or other `lib//`. You may + **import** from `lib/apl/` (public API in `lib/apl/apl.sx`); do **not** modify APL. +- **Shared-file issues** go under "Blockers" with a minimal repro; do not fix here. +- **SX files:** use `sx-tree` MCP tools only. +- **Architecture:** an activity is a small dict (`{:actor :verb :object :at :tags}`); a + stream is an APL vector of such dicts. Operations are APL primitives lifted onto + this shape. SX adapter exposes ergonomic API to non-APL callers. +- **Unicode:** raw UTF-8 in `.sx` files. APL glyphs land directly. +- **Commits:** one feature per commit. Keep Progress log updated and tick boxes. + +## Architecture sketch + +``` +Raw activities (any shape) Per-user view + │ ▲ + ▼ │ +lib/feed/normalize.sx lib/feed/timeline.sx + — {:actor :verb :object — (timeline user) + :at :tags} record — applies filter ∘ rank ∘ take + │ ▲ + ▼ │ +lib/feed/stream.sx lib/feed/rank.sx + — APL vector of activities — velocity, recency + — filter, sort, take — TF-IDF-ish over :tags + │ ▲ + ▼ │ +lib/feed/fanout.sx lib/feed/dedupe.sx + — followers vector — group by :object + — activities ∘.× followers — collapse cross-posts + — flatten + dedupe + │ + ▼ +lib/feed/api.sx lib/feed/fed.sx + — (feed/post activity) — inbox via fed-sx + — (feed/timeline user) — backfill on subscribe + — (feed/notify user) +``` + +## Phase 1 — Stream model + basic ops + +- [ ] `lib/feed/normalize.sx` — activity record schema; coerce arbitrary inputs +- [ ] `lib/feed/stream.sx` — APL vector representation; filter by predicate; sort by + `:at`; take N (`↑`); reverse (`⌽`) +- [ ] `lib/feed/api.sx` — `(feed/post activity)`, `(feed/all)` +- [ ] `lib/feed/tests/basic.sx` — 15+ cases: post, query, filter, sort +- [ ] `lib/feed/scoreboard.{json,md}` +- [ ] `lib/feed/conformance.sh` + +## Phase 2 — Fanout via outer product + +- [ ] follower graph: `followers user → vector of user ids` +- [ ] fanout: activities `∘.×` followers → matrix `(activity, follower)` pairs +- [ ] flatten to inbox events vector +- [ ] dedupe — group by `(actor, verb, object)` collapse to one inbox event per + receiver +- [ ] `lib/feed/tests/fanout.sx` — 20+ cases: small graph, mutual follow, popular + actor (high-fanout), cross-post dedupe + +## Phase 3 — Aggregation + ranking + +- [ ] group-by — `(actor, day) → count` via key-reduce +- [ ] velocity score — recent activity count over window +- [ ] recency score — decay by age +- [ ] composite rank — weighted sum of components +- [ ] top-N per timeline +- [ ] `lib/feed/tests/rank.sx` — 20+ cases: ranking stable on tie, decay shape, + per-user weighting + +## Phase 4 — Visibility filter + federation + +- [ ] ACL filter — each candidate activity passed through `(acl/permit? viewer :read + activity)` +- [ ] fed-sx outbound — local `feed/post` fans out to remote followers' inboxes +- [ ] fed-sx inbound — peer activities arrive at local inbox +- [ ] backfill on subscribe — request peer history, merge into local stream +- [ ] `lib/feed/tests/integration.sx` — federated timeline with ACL applied + +## Progress log + +(loop fills this in) + +## Blockers + +(loop fills this in) diff --git a/plans/flow-on-sx.md b/plans/flow-on-sx.md new file mode 100644 index 00000000..46517942 --- /dev/null +++ b/plans/flow-on-sx.md @@ -0,0 +1,108 @@ +# flow-on-sx: Durable DAG Workflows on Scheme + +rose-ash needs workflows that survive restarts: content pipelines (write → review → +publish → federate), scheduled jobs (digest emails), multi-step user flows (signup, +confirm, onboard). art-dag is the precedent — DAG-of-tasks with pause/resume at IO +boundaries. + +Scheme's `call/cc` + delimited continuations make pause/resume natural: a `suspend` +captures the continuation, serializes it as part of the flow record, and `resume` +re-enters at exactly that point. No state-machine bookkeeping by hand. R7RS-small is +already at 2644/2644 (see kernel/architecture status). + +End-state: a Scheme-on-SX layer over the existing scheme runtime, with combinators +for sequence/parallel/branch/retry/timeout/suspend, persistent flow store, and a +federation extension via fed-sx for remote-node execution. + +## Status (rolling) + +`bash lib/flow/conformance.sh` → **0/0** (not yet started) + +## Ground rules + +- **Scope:** only touch `lib/flow/**` and `plans/flow-on-sx.md`. Do **not** edit + `spec/`, `hosts/`, `shared/`, `lib/scheme/**`, or other `lib//`. You may + **import** from `lib/scheme/` (public API via `lib/scheme/scheme.sx`); do **not** + modify Scheme. +- **Shared-file issues** go under "Blockers" with a minimal repro; do not fix here. +- **SX files:** use `sx-tree` MCP tools only. +- **Architecture:** flow combinators are Scheme macros + procedures. Runtime is a + driver loop that walks the flow graph and invokes `call/cc` at `suspend` points. + Persistence layer serializes the continuation + open file/socket placeholders are + forbidden (continuations must be resumable across process restart). +- **art-dag awareness:** read `plans/art-dag*` if it exists for design lineage; do not + import code. +- **Commits:** one feature per commit. Keep Progress log updated and tick boxes. + +## Architecture sketch + +``` +(defflow publish + (sequence + (write-content) + (parallel + (review) + (spell-check)) + (cond approved? + (sequence (publish) (federate)) + (notify-author)))) + │ + ▼ +lib/flow/spec.sx lib/flow/runtime.sx lib/flow/store.sx + — defflow — driver loop — append-only flow log + — sequence/parallel — node dispatch — checkpoint serialize + — cond/retry/timeout — call/cc at suspend — restart loader + — suspend/resume │ │ + ▼ ▼ + lib/flow/api.sx lib/flow/remote.sx + — (flow/start name args) — fed-sx adapter + — (flow/resume id value) — node-on-peer execution + — (flow/cancel id) — failure handling +``` + +## Phase 1 — Declarative DAG + sequential execution + +- [ ] `lib/flow/spec.sx` — `defflow` macro, `sequence` combinator +- [ ] node = Scheme thunk; output threads to next node (data flow) +- [ ] `parallel` combinator (sequential semantics for now — TRUE parallelism in Phase 3) +- [ ] runtime executes a flow synchronously, returns final value +- [ ] `lib/flow/api.sx` — `(flow/start name args)` entry point +- [ ] `lib/flow/tests/basic.sx` — 15+ cases: linear sequence, nested sequences, + data flow between nodes, parallel-with-join +- [ ] `lib/flow/scoreboard.{json,md}` +- [ ] `lib/flow/conformance.sh` + +## Phase 2 — Control flow + error handling + +- [ ] `cond` combinator — predicate selects branch +- [ ] `retry n [backoff]` — re-runs node up to n times on exception +- [ ] `timeout ms` — bounds node execution +- [ ] `try-catch` — exception handler with reified error +- [ ] error model — exceptions vs explicit `(fail :reason ...)` results +- [ ] `lib/flow/tests/control.sx` — 25+ cases: each combinator + composition + +## Phase 3 — Suspend / resume (the showcase) + +- [ ] `(suspend reason)` — `call/cc` captures continuation, returns flow-id to caller +- [ ] `lib/flow/store.sx` — serialize flow state (continuation + open vars) +- [ ] `(flow/resume id value)` — load continuation, inject value, re-enter +- [ ] `(flow/cancel id)` — explicit termination +- [ ] crash recovery — on restart, scan store for paused flows, mark resumable +- [ ] `lib/flow/tests/suspend.sx` — pause-resume scenarios, cancellation, "restart" + scenarios (simulated by re-loading store) + +## Phase 4 — Distributed nodes via fed-sx + +- [ ] `(remote-node addr fn args)` — execute node on a federation peer +- [ ] failure semantics — retry on different peer, fall through to local +- [ ] persistence across instances — flow state replicates via fed-sx +- [ ] handoff — flow started here can resume on a peer if the local instance is down +- [ ] `lib/flow/tests/distributed.sx` — federated flow scenarios (mock fed-sx in tests) + +## Progress log + +(loop fills this in) + +## Blockers + +(loop fills this in) diff --git a/plans/mod-on-sx.md b/plans/mod-on-sx.md new file mode 100644 index 00000000..30887b50 --- /dev/null +++ b/plans/mod-on-sx.md @@ -0,0 +1,112 @@ +# mod-on-sx: Moderation on Prolog + +rose-ash needs moderation infrastructure: reports flagged by users, automated +classifications (spam, abuse), tiered escalation (auto → human → appeal), audit +trails. Each decision is the conclusion of a backtracking search over evidence and +policy rules — exactly what Prolog does. + +Where acl-sx says "may this happen?", mod-sx says "should this stay?" The former is +a positive decision (proof of grant); the latter often a negative one (proof of +violation), and policy chains naturally backtrack: if the first rule doesn't apply, +try the next. + +End-state: a Prolog-on-SX layer for moderation policy declaration and evaluation, +with persistent report lifecycle, audit log, escalation state machine, and +federation extension. + +## Status (rolling) + +`bash lib/mod/conformance.sh` → **0/0** (not yet started) + +## Ground rules + +- **Scope:** only touch `lib/mod/**` and `plans/mod-on-sx.md`. Do **not** edit + `spec/`, `hosts/`, `shared/`, `lib/prolog/**`, or other `lib//`. You may + **import** from `lib/prolog/` (public API in `lib/prolog/prolog.sx`); do **not** + modify Prolog. +- **Shared-file issues** go under "Blockers" with a minimal repro; do not fix here. +- **SX files:** use `sx-tree` MCP tools only. +- **Architecture:** policies are Prolog rules over `report(...)` and `evidence(...)` + facts. Decisions are query results. Proof trees become audit records. The state + machine for report lifecycle is separate (an SX module on top). +- **Shared with acl-sx:** rule-engine plumbing may be liftable into `lib/guest/`. + Watch for it; flag in Progress log but do not extract until both subsystems are + past Phase 2. +- **Commits:** one feature per commit. Keep Progress log updated and tick boxes. + +## Architecture sketch + +``` +Report Decision + {:by :about :reason :at} {:action :proof :next-state} + │ ▲ + ▼ │ +lib/mod/schema.sx lib/mod/engine.sx + — report/4, evidence/2, — query Prolog with report fact + classification/3 predicates — extract proof tree + │ ▲ + ▼ │ +lib/mod/policy.sx lib/mod/lifecycle.sx + — rule syntax → Prolog — state machine + — action heads: — open → triaged → decided + {:keep :hide :remove — appeal handling + :escalate :ban} │ + │ ▼ + ▼ lib/mod/audit.sx +lib/mod/api.sx — append-only decision log + — (mod/report ...) — proof tree persistence + — (mod/decide report) — query API + — (mod/appeal id) + │ + ▼ +lib/mod/fed.sx + — cross-instance reports via fed-sx + — decision sharing / trust model +``` + +## Phase 1 — Report representation + simple policy + +- [ ] `lib/mod/schema.sx` — `report(id, by, about, reason)`, `evidence(id, kind, val)`, + `policy-action(report, action)` predicates as Prolog facts/rules +- [ ] `lib/mod/policy.sx` — rule declarations: `(defrule action :when conditions)` + desugars to Prolog clause +- [ ] `lib/mod/engine.sx` — `(decide report-id)` runs Prolog query, returns first + matching action +- [ ] `lib/mod/api.sx` — `(mod/report by about reason)`, `(mod/decide id)` +- [ ] `lib/mod/tests/decide.sx` — 15+ cases: spam keyword → hide, repeated reports → + escalate, no rule matches → keep +- [ ] `lib/mod/scoreboard.{json,md}` +- [ ] `lib/mod/conformance.sh` + +## Phase 2 — Evidence + audit trail + +- [ ] evidence accumulation — additional facts asserted before query +- [ ] proof tree from Prolog derivation tree +- [ ] `lib/mod/audit.sx` — append-only log (decision + proof + evidence snapshot) +- [ ] `(mod/audit id)` retrieval +- [ ] `lib/mod/tests/audit.sx` — proof correctness, trail completeness + +## Phase 3 — Escalation + lifecycle state machine + +- [ ] state machine: `:open → :triaged → :decided → :appealed → :final` +- [ ] auto-tier: first-pass rules decide quick cases +- [ ] human-tier: rules that emit `:escalate` move to next state +- [ ] appeal: re-runs with appeal evidence, may override prior decision +- [ ] `(mod/appeal id new-evidence)` API +- [ ] `lib/mod/tests/escalation.sx` — full lifecycle traversal cases + +## Phase 4 — Federation + +- [ ] cross-instance reports — peer raises report about local content (or vice versa) +- [ ] decision sharing — actions taken locally propagate to peers via fed-sx +- [ ] trust model — peer's decision is advisory unless `(trust peer :mod)` is granted +- [ ] revocation — undo applied moderation if proof was invalidated +- [ ] `lib/mod/tests/fed.sx` — federated decision chains (mock fed-sx in tests) + +## Progress log + +(loop fills this in) + +## Blockers + +(loop fills this in) diff --git a/plans/search-on-sx.md b/plans/search-on-sx.md new file mode 100644 index 00000000..9e0045d4 --- /dev/null +++ b/plans/search-on-sx.md @@ -0,0 +1,106 @@ +# search-on-sx: Full-text + structured search on Haskell + +rose-ash needs search across pages, posts, threads, federated content. Tokenize, +index, query, rank, filter by visibility. Typed ADTs make query parsing clean, +lazy lists make posting-list iteration efficient, and Haskell-on-SX is at 1514/1514. + +End-state: a Haskell-on-SX layer with inverted index, query AST, boolean + +phrase + ranked queries (TF-IDF, BM25), ACL-aware post-filter, and a federation +extension that merges per-peer indices. + +## Status (rolling) + +`bash lib/search/conformance.sh` → **0/0** (not yet started) + +## Ground rules + +- **Scope:** only touch `lib/search/**` and `plans/search-on-sx.md`. Do **not** edit + `spec/`, `hosts/`, `shared/`, `lib/haskell/**`, or other `lib//`. You may + **import** from `lib/haskell/` (public API in `lib/haskell/haskell.sx`); do **not** + modify Haskell. +- **Shared-file issues** go under "Blockers" with a minimal repro; do not fix here. +- **SX files:** use `sx-tree` MCP tools only. +- **Architecture:** index = `Map Term [(DocId, [Pos])]`. Query AST = ADT. Eval = + fold of posting lists with set ops + ranking math. Ranking is pure (no IO until + result emission). +- **Commits:** one feature per commit. Keep Progress log updated and tick boxes. + +## Architecture sketch + +``` +Document Query + {:id :text :tags} "alice AND bob OR phrase \"x y\"" + │ │ + ▼ ▼ +lib/search/tokenize.sx lib/search/parse.sx + — tokenize :: Text → [Term] — parse :: Text → Query + — normalize (lowercase, strip) — Query = Term | And | Or + — (optionally) stem | Not | Phrase + │ │ + ▼ ▼ +lib/search/index.sx lib/search/eval.sx + — Map Term [(DocId, [Pos])] — eval :: Index → Query → [DocId] + — insert / delete / lookup — boolean + phrase positions + — persistence (optional later) │ + │ ▼ + └────────────────► lib/search/rank.sx + — TF-IDF / BM25 scoring + — top-N + │ + ▼ + lib/search/api.sx + — (search/index doc) + — (search/query q) + — (search/top n q) + │ + ▼ + lib/search/fed.sx + — federated query (merge peer results) + — ACL filter post-merge +``` + +## Phase 1 — Tokenize + index + +- [ ] `lib/search/tokenize.sx` — normalize (lowercase, strip punctuation), split on + whitespace, return positions +- [ ] `lib/search/index.sx` — inverted index data structure (typed `Map` from + haskell lib); `insert`, `delete`, `lookup` +- [ ] `lib/search/api.sx` — `(search/index doc)`, `(search/lookup term)` +- [ ] `lib/search/tests/index.sx` — 15+ cases: tokenize, insert + lookup, update, + delete, multi-doc +- [ ] `lib/search/scoreboard.{json,md}` +- [ ] `lib/search/conformance.sh` + +## Phase 2 — Query AST + boolean evaluation + +- [ ] Query ADT: `Term Text | And Query Query | Or Query Query | Not Query | + Phrase [Text]` +- [ ] `lib/search/parse.sx` — query syntax parser (boolean operators, quoted phrases) +- [ ] `lib/search/eval.sx` — boolean eval via set ops on posting lists +- [ ] phrase eval — adjacency check using positions +- [ ] `lib/search/tests/boolean.sx` — 25+ cases: term, and, or, not, phrase, + composition, parser edge cases + +## Phase 3 — Ranking + +- [ ] document frequency tracking — extend index with `df` per term +- [ ] TF-IDF scoring +- [ ] BM25 scoring (configurable k1, b) +- [ ] top-N retrieval (heap-based) +- [ ] `lib/search/tests/rank.sx` — 20+ cases: TF-IDF behavior, BM25 vs TF-IDF, + ranking stability, top-N correctness + +## Phase 4 — ACL filter + federation + +- [ ] post-filter — each candidate result tested via `(acl/permit? viewer :read doc)` +- [ ] federated query — fan out to peer instances via fed-sx, merge results +- [ ] merge policy — interleave by rank, dedupe by `(peer, doc-id)` +- [ ] `lib/search/tests/integration.sx` — federated search with ACL filter + +## Progress log + +(loop fills this in) + +## Blockers + +(loop fills this in)