rose-ash/plans/feed-on-sx.md

feed-on-sx: Activity Feeds on APL

Timelines, notifications, activity aggregation. The math is array math: filter, sort, reduce, scan, outer product. APL is the densest possible expression of feed composition — a fanout-and-rank pipeline reads as a single line.

rose-ash needs: per-user home timeline, notification feed, activity stream digestion, backfill for new follows, deduplication across cross-posts. Every operation is an array-shaped transformation.

End-state: an APL-flavored layer on lib/apl/ with feed-specific combinators (fanout, dedupe, score, rank), an SX adapter for callers who don't want raw APL, ACL visibility filtering via lib/acl/, federation via fed-sx.

Status (rolling)

bash lib/feed/conformance.sh → 59/59 (Phases 1–2 complete)

Ground rules

• Scope: only touch lib/feed/** and plans/feed-on-sx.md. Do not edit spec/, hosts/, shared/, lib/apl/**, or other lib/<lang>/. You may import from lib/apl/ (public API in lib/apl/apl.sx); do not modify APL.
• Shared-file issues go under "Blockers" with a minimal repro; do not fix here.
• SX files: use sx-tree MCP tools only.
• Architecture: an activity is a small dict ({:actor :verb :object :at :tags}); a stream is an APL vector of such dicts. Operations are APL primitives lifted onto this shape. SX adapter exposes ergonomic API to non-APL callers.
• Unicode: raw UTF-8 in .sx files. APL glyphs land directly.
• Commits: one feature per commit. Keep Progress log updated and tick boxes.

Architecture sketch

Raw activities (any shape)              Per-user view
        │                                    ▲
        ▼                                    │
lib/feed/normalize.sx              lib/feed/timeline.sx
  — {:actor :verb :object             — (timeline user)
     :at :tags} record                — applies filter ∘ rank ∘ take
        │                                    ▲
        ▼                                    │
lib/feed/stream.sx                  lib/feed/rank.sx
  — APL vector of activities          — velocity, recency
  — filter, sort, take                — TF-IDF-ish over :tags
        │                                    ▲
        ▼                                    │
lib/feed/fanout.sx                  lib/feed/dedupe.sx
  — followers vector                  — group by :object
  — activities ∘.× followers          — collapse cross-posts
  — flatten + dedupe
        │
        ▼
lib/feed/api.sx           lib/feed/fed.sx
  — (feed/post activity)    — inbox via fed-sx
  — (feed/timeline user)    — backfill on subscribe
  — (feed/notify user)

Phase 1 — Stream model + basic ops

• lib/feed/normalize.sx — activity record schema; coerce arbitrary inputs
• lib/feed/stream.sx — APL vector representation; filter by predicate; sort by :at; take N (↑); reverse (⌽)
• lib/feed/api.sx — (feed/post activity), (feed/all)
• lib/feed/tests/basic.sx — 30 cases: normalize defaults, filter, sort, take, api
• lib/feed/scoreboard.{json,md}
• lib/feed/conformance.sh

Phase 2 — Fanout via outer product

• follower graph: followers user → vector of user ids (feed/follow-graph, feed/followers; graph = {followee -> (followers)} dict)
• fanout: activities ∘.× audience → matrix via apl-outer feed/-mk-event
• flatten to inbox events vector (feed/-flatten rank-2 → rank-1)
• dedupe — feed/dedupe-inbox by (to, actor, verb, object); also feed/dedupe-activities (actor verb object) and feed/dedupe-collapse (verb object) for cross-actor likes
• lib/feed/tests/fanout.sx — 29 cases: small graph, mutual follow, star (high-fanout), empty graph, unfollowed actor, cross-post dedupe

Phase 3 — Aggregation + ranking

• group-by — (actor, day) → count via key-reduce
• velocity score — recent activity count over window
• recency score — decay by age
• composite rank — weighted sum of components
• top-N per timeline
• lib/feed/tests/rank.sx — 20+ cases: ranking stable on tie, decay shape, per-user weighting

Phase 4 — Visibility filter + federation

• ACL filter — each candidate activity passed through (acl/permit? viewer :read activity)
• fed-sx outbound — local feed/post fans out to remote followers' inboxes
• fed-sx inbound — peer activities arrive at local inbox
• backfill on subscribe — request peer history, merge into local stream
• lib/feed/tests/integration.sx — federated timeline with ACL applied

Progress log

• Phase 1 done (30/30). Stream = APL rank-1 array whose ravel holds activity dicts. normalize.sx (record schema + accessors), stream.sx (filter via / compress, sort via ⍋ grade-up [stable], take via ↑, reverse via ⌽, by-actor/verb/object/since predicates), api.sx (mutable log: post/all/reset!/size). Substrate: apl-compress, apl-grade-up, apl-take, apl-reverse, make-array. Grade-up returns 1-based indices (⎕IO=1), is stable on ties → deterministic sort.
• Phase 2 done (59/59 total). fanout.sx (graph + apl-outer showcase), dedupe.sx (per-key dedupe, first-wins stable). Key APL gotcha: scalar? is true for ANY dict and disclose nils a non-array dict, so an apl-outer combiner MUST enclose its event dict — apl-outer discloses it back intact. apl-unique preserves first-occurrence order; dict keys order is NOT stable, so feed/audience sorts (else recipient ordering flakes). apl-compress needs a rank-1 array, so the (activity×follower) matrix is flattened to its ravel before the edge-guard filter.

(none)

Notes for next iteration

• sx-tree MCP tools take file: NOT path: (CLAUDE.md is stale). Wrong key → Yojson Type_error("Expected string, got null"). Looks like a broken binary, isn't.
• sx_server binary lives in main repo: /root/rose-ash/hosts/ocaml/_build/default/bin/sx_server.exe (worktree has no _build). conformance.sh already points there with relative fallback.
• Phase 2 substrate verified available: apl-outer (∘.×), apl-member (∊), apl-unique, apl-iota (1-based).