rose-ash/plans/feed-on-sx.md

feed-on-sx: Activity Feeds on APL

Timelines, notifications, activity aggregation. The math is array math: filter, sort, reduce, scan, outer product. APL is the densest possible expression of feed composition — a fanout-and-rank pipeline reads as a single line.

rose-ash needs: per-user home timeline, notification feed, activity stream digestion, backfill for new follows, deduplication across cross-posts. Every operation is an array-shaped transformation.

End-state: an APL-flavored layer on lib/apl/ with feed-specific combinators (fanout, dedupe, score, rank), an SX adapter for callers who don't want raw APL, ACL visibility filtering via lib/acl/, federation via fed-sx.

Status (rolling)

bash lib/feed/conformance.sh → 189/189 (Phases 1–4 + TF-IDF, notifications, home, smart-dedupe, trending, mute, pagination, threading)

Ground rules

• Scope: only touch lib/feed/** and plans/feed-on-sx.md. Do not edit spec/, hosts/, shared/, lib/apl/**, or other lib/<lang>/. You may import from lib/apl/ (public API in lib/apl/apl.sx); do not modify APL.
• Shared-file issues go under "Blockers" with a minimal repro; do not fix here.
• SX files: use sx-tree MCP tools only.
• Architecture: an activity is a small dict ({:actor :verb :object :at :tags}); a stream is an APL vector of such dicts. Operations are APL primitives lifted onto this shape. SX adapter exposes ergonomic API to non-APL callers.
• Unicode: raw UTF-8 in .sx files. APL glyphs land directly.
• Commits: one feature per commit. Keep Progress log updated and tick boxes.

Architecture sketch

Raw activities (any shape)              Per-user view
        │                                    ▲
        ▼                                    │
lib/feed/normalize.sx              lib/feed/timeline.sx
  — {:actor :verb :object             — (timeline user)
     :at :tags} record                — applies filter ∘ rank ∘ take
        │                                    ▲
        ▼                                    │
lib/feed/stream.sx                  lib/feed/rank.sx
  — APL vector of activities          — velocity, recency
  — filter, sort, take                — TF-IDF-ish over :tags
        │                                    ▲
        ▼                                    │
lib/feed/fanout.sx                  lib/feed/dedupe.sx
  — followers vector                  — group by :object
  — activities ∘.× followers          — collapse cross-posts
  — flatten + dedupe
        │
        ▼
lib/feed/api.sx           lib/feed/fed.sx
  — (feed/post activity)    — inbox via fed-sx
  — (feed/timeline user)    — backfill on subscribe
  — (feed/notify user)

Phase 1 — Stream model + basic ops

• lib/feed/normalize.sx — activity record schema; coerce arbitrary inputs
• lib/feed/stream.sx — APL vector representation; filter by predicate; sort by :at; take N (↑); reverse (⌽)
• lib/feed/api.sx — (feed/post activity), (feed/all)
• lib/feed/tests/basic.sx — 30 cases: normalize defaults, filter, sort, take, api
• lib/feed/scoreboard.{json,md}
• lib/feed/conformance.sh

Phase 2 — Fanout via outer product

• follower graph: followers user → vector of user ids (feed/follow-graph, feed/followers; graph = {followee -> (followers)} dict)
• fanout: activities ∘.× audience → matrix via apl-outer feed/-mk-event
• flatten to inbox events vector (feed/-flatten rank-2 → rank-1)
• dedupe — feed/dedupe-inbox by (to, actor, verb, object); also feed/dedupe-activities (actor verb object) and feed/dedupe-collapse (verb object) for cross-actor likes
• lib/feed/tests/fanout.sx — 29 cases: small graph, mutual follow, star (high-fanout), empty graph, unfollowed actor, cross-post dedupe

Phase 3 — Aggregation + ranking

• group-by — feed/group-by/feed/group-count key-reduce; feed/by-actor-day buckets (actor, day) via feed/day (string-joined keys)
• velocity score — feed/velocity counts actor's activities in (at-window, at]
• recency score — feed/recency half-life decay 0.5^(age/hl)
• composite rank — feed/composite weighted sum of (weight scorer) parts
• top-N per timeline — feed/top = rank then take
• lib/feed/tests/rank.sx — 24 cases: decay shape, velocity burst, stable tie-break, top-N, composite

Phase 4 — Visibility filter + federation

lib/acl/ and fed-sx don't exist yet and are out of scope (import lib/apl/ only), so ACL/transport are injected: permit?, remote?, send-fn, fetch-fn are function parameters. Real acl-sx / fed-sx wire in at the call site unchanged.

• ACL filter — feed/visible stream viewer permit?; default feed/permit-acl? reads :visible-to allowlist (+ author-sees-own); per-viewer, never cached
• fed-sx outbound — feed/federate/feed/deliver fan out then partition local vs remote inboxes; remote events handed to injected send-fn
• fed-sx inbound — feed/inbound normalizes + feed/ingest dedupes peer activities into the local stream
• backfill on subscribe — feed/backfill local fetch-fn peer-id
• lib/feed/tests/integration.sx — 22 cases incl. end-to-end feed/timeline (federated → ACL for viewer → recency rank → top-N)

Progress log

• Phase 1 done (30/30). Stream = APL rank-1 array whose ravel holds activity dicts. normalize.sx (record schema + accessors), stream.sx (filter via / compress, sort via ⍋ grade-up [stable], take via ↑, reverse via ⌽, by-actor/verb/object/since predicates), api.sx (mutable log: post/all/reset!/size). Substrate: apl-compress, apl-grade-up, apl-take, apl-reverse, make-array. Grade-up returns 1-based indices (⎕IO=1), is stable on ties → deterministic sort.
• Phase 2 done (59/59 total). fanout.sx (graph + apl-outer showcase), dedupe.sx (per-key dedupe, first-wins stable). Key APL gotcha: scalar? is true for ANY dict and disclose nils a non-array dict, so an apl-outer combiner MUST enclose its event dict — apl-outer discloses it back intact. apl-unique preserves first-occurrence order; dict keys order is NOT stable, so feed/audience sorts (else recipient ordering flakes). apl-compress needs a rank-1 array, so the (activity×follower) matrix is flattened to its ravel before the edge-guard filter.
• Phase 3 done (83/83 total). aggregate.sx (group-by/count, day buckets) + rank.sx (recency/velocity/engagement scorers, composite, top-N). sort is single-arg ascending only — no comparator — so ranking uses a stable two-pass apl-grade-down (by :at desc, then by score desc) for deterministic tie-breaks. Dict keys must be strings, so composite group keys are string-joined ("actor#day").
• Phase 4 done (105/105 total). acl.sx (per-viewer feed/visible, feed/timeline capstone) + fed.sx (merge/ingest/inbound/backfill/federate/ deliver). ACL/transport are dependency-injected (permit?/remote?/send-fn/fetch-fn) since lib/acl + fed-sx don't exist. feed/normalize now MERGEs defaults over the raw dict (was projecting to 5 keys) so extra metadata (:visible-to, peer fields) survives — matches the "flexible bag" principle.

Roadmap is complete (all 4 phases). Possible follow-ups:

• Wire real acl-sx once lib/acl/ exists (swap injected permit?).
• Wire real fed-sx transport (swap send-fn/fetch-fn).
• TF-IDF over :tags for content ranking — content.sx: feed/tag-df, feed/tag-idf (log N/df), feed/tfidf-score, feed/by-relevance; 15 tests. Composes as a scorer with rank.sx. (120/120 total.)
• Notification feed (verb-filtered, per-recipient) — notify.sx: feed/notifications, feed/notify-verbs, feed/notify-digest (collapses "X, Y liked Z" by (verb,object), sorted-deterministic); 8 tests. (128/128 total.)
• Capstone feed/home — the whole pipeline as one line: fanout ∘ inbox ∘ dedupe ∘ ACL ∘ rank ∘ take (home.sx); 6 tests incl. per-viewer ACL + cross-post dedupe. (134/134 total.)
• Per-verb dedupe rules (briefing gotcha #3) — feed/dedupe-smart / feed/smart-key: reactions (like/follow/boost/...) collapse cross-actor on (verb,object); posts stay distinct per actor. feed/collapse-verbs is rebindable policy; 9 tests. (143/143 total.)
• Trending — feed/trending / feed/trending-actors: objects/actors ranked by activity count in a recency window, count-desc with key-asc tiebreak (trending.sx); 11 tests. (154/154 total.)
• Mute/block — feed/mute-actors / feed/mute-tags / feed/mute-objects / feed/apply-prefs: viewer-controlled per-request filtering (complements ACL's author-controlled visibility) (mute.sx); 9 tests. (163/163 total.)
• Pagination — feed/page/feed/page-count (offset) + feed/before/ feed/after/feed/page-before/feed/next-cursor (cursor by :at, stable under inserts) (page.sx); 14 tests. (177/177 total.)
• Threading — feed/replies/feed/reply-count/feed/thread/ feed/thread-objects/feed/thread-size: conversation closure over :reply-to (transitive fixpoint), chronological (thread.sx); 12 tests. (189/189 total.)

(none)

Notes for next iteration

• sx-tree MCP tools take file: NOT path: (CLAUDE.md is stale). Wrong key → Yojson Type_error("Expected string, got null"). Looks like a broken binary, isn't.
• sx_server binary lives in main repo: /root/rose-ash/hosts/ocaml/_build/default/bin/sx_server.exe (worktree has no _build). conformance.sh already points there with relative fallback.
• Phase 2 substrate verified available: apl-outer (∘.×), apl-member (∊), apl-unique, apl-iota (1-based).