rose-ash/plans/feed-on-sx.md

# feed-on-sx: Activity Feeds on APL

Timelines, notifications, activity aggregation. The math is array math: filter, sort,
reduce, scan, outer product. APL is the densest possible expression of feed
composition — a fanout-and-rank pipeline reads as a single line.

rose-ash needs: per-user home timeline, notification feed, activity stream digestion,
backfill for new follows, deduplication across cross-posts. Every operation is an
array-shaped transformation.

End-state: an APL-flavored layer on `lib/apl/` with feed-specific combinators
(`fanout`, `dedupe`, `score`, `rank`), an SX adapter for callers who don't want raw
APL, ACL visibility filtering via `lib/acl/`, federation via fed-sx.

## Status (rolling)

`bash lib/feed/conformance.sh` → **189/189** (Phases 1–4 + TF-IDF, notifications, home, smart-dedupe, trending, mute, pagination, threading)

## Ground rules

- **Scope:** only touch `lib/feed/**` and `plans/feed-on-sx.md`. Do **not** edit
  `spec/`, `hosts/`, `shared/`, `lib/apl/**`, or other `lib/<lang>/`. You may
  **import** from `lib/apl/` (public API in `lib/apl/apl.sx`); do **not** modify APL.
- **Shared-file issues** go under "Blockers" with a minimal repro; do not fix here.
- **SX files:** use `sx-tree` MCP tools only.
- **Architecture:** an activity is a small dict (`{:actor :verb :object :at :tags}`); a
  stream is an APL vector of such dicts. Operations are APL primitives lifted onto
  this shape. SX adapter exposes ergonomic API to non-APL callers.
- **Unicode:** raw UTF-8 in `.sx` files. APL glyphs land directly.
- **Commits:** one feature per commit. Keep Progress log updated and tick boxes.

## Architecture sketch

```
Raw activities (any shape)              Per-user view
        │                                    ▲
        ▼                                    │
lib/feed/normalize.sx              lib/feed/timeline.sx
  — {:actor :verb :object             — (timeline user)
     :at :tags} record                — applies filter ∘ rank ∘ take
        │                                    ▲
        ▼                                    │
lib/feed/stream.sx                  lib/feed/rank.sx
  — APL vector of activities          — velocity, recency
  — filter, sort, take                — TF-IDF-ish over :tags
        │                                    ▲
        ▼                                    │
lib/feed/fanout.sx                  lib/feed/dedupe.sx
  — followers vector                  — group by :object
  — activities ∘.× followers          — collapse cross-posts
  — flatten + dedupe
        │
        ▼
lib/feed/api.sx           lib/feed/fed.sx
  — (feed/post activity)    — inbox via fed-sx
  — (feed/timeline user)    — backfill on subscribe
  — (feed/notify user)
```

## Phase 1 — Stream model + basic ops

- [x] `lib/feed/normalize.sx` — activity record schema; coerce arbitrary inputs
- [x] `lib/feed/stream.sx` — APL vector representation; filter by predicate; sort by
  `:at`; take N (`↑`); reverse (`⌽`)
- [x] `lib/feed/api.sx` — `(feed/post activity)`, `(feed/all)`
- [x] `lib/feed/tests/basic.sx` — 30 cases: normalize defaults, filter, sort, take, api
- [x] `lib/feed/scoreboard.{json,md}`
- [x] `lib/feed/conformance.sh`

## Phase 2 — Fanout via outer product

- [x] follower graph: `followers user → vector of user ids` (`feed/follow-graph`,
  `feed/followers`; graph = `{followee -> (followers)}` dict)
- [x] fanout: activities `∘.×` audience → matrix via `apl-outer feed/-mk-event`
- [x] flatten to inbox events vector (`feed/-flatten` rank-2 → rank-1)
- [x] dedupe — `feed/dedupe-inbox` by `(to, actor, verb, object)`; also
  `feed/dedupe-activities` `(actor verb object)` and `feed/dedupe-collapse`
  `(verb object)` for cross-actor likes
- [x] `lib/feed/tests/fanout.sx` — 29 cases: small graph, mutual follow, star
  (high-fanout), empty graph, unfollowed actor, cross-post dedupe

## Phase 3 — Aggregation + ranking

- [x] group-by — `feed/group-by`/`feed/group-count` key-reduce; `feed/by-actor-day`
  buckets `(actor, day)` via `feed/day` (string-joined keys)
- [x] velocity score — `feed/velocity` counts actor's activities in `(at-window, at]`
- [x] recency score — `feed/recency` half-life decay `0.5^(age/hl)`
- [x] composite rank — `feed/composite` weighted sum of `(weight scorer)` parts
- [x] top-N per timeline — `feed/top` = rank then take
- [x] `lib/feed/tests/rank.sx` — 24 cases: decay shape, velocity burst, stable
  tie-break, top-N, composite

## Phase 4 — Visibility filter + federation

`lib/acl/` and fed-sx don't exist yet and are out of scope (import `lib/apl/`
only), so ACL/transport are injected: `permit?`, `remote?`, `send-fn`, `fetch-fn`
are function parameters. Real acl-sx / fed-sx wire in at the call site unchanged.

- [x] ACL filter — `feed/visible stream viewer permit?`; default `feed/permit-acl?`
  reads `:visible-to` allowlist (+ author-sees-own); per-viewer, never cached
- [x] fed-sx outbound — `feed/federate`/`feed/deliver` fan out then partition
  local vs remote inboxes; remote events handed to injected `send-fn`
- [x] fed-sx inbound — `feed/inbound` normalizes + `feed/ingest` dedupes peer
  activities into the local stream
- [x] backfill on subscribe — `feed/backfill local fetch-fn peer-id`
- [x] `lib/feed/tests/integration.sx` — 22 cases incl. end-to-end
  `feed/timeline` (federated → ACL for viewer → recency rank → top-N)

## Progress log

- **Phase 1 done (30/30).** Stream = APL rank-1 array whose ravel holds activity
  dicts. `normalize.sx` (record schema + accessors), `stream.sx` (filter via `/`
  compress, sort via `⍋` grade-up [stable], take via `↑`, reverse via `⌽`,
  by-actor/verb/object/since predicates), `api.sx` (mutable log: post/all/reset!/size).
  Substrate: `apl-compress`, `apl-grade-up`, `apl-take`, `apl-reverse`, `make-array`.
  Grade-up returns 1-based indices (⎕IO=1), is stable on ties → deterministic sort.
- **Phase 2 done (59/59 total).** `fanout.sx` (graph + `apl-outer` showcase),
  `dedupe.sx` (per-key dedupe, first-wins stable). Key APL gotcha: `scalar?` is
  true for ANY dict and `disclose` nils a non-array dict, so an apl-outer combiner
  MUST `enclose` its event dict — apl-outer discloses it back intact. `apl-unique`
  preserves first-occurrence order; dict `keys` order is NOT stable, so
  `feed/audience` sorts (else recipient ordering flakes). `apl-compress` needs a
  rank-1 array, so the (activity×follower) matrix is flattened to its ravel before
  the edge-guard filter.
- **Phase 3 done (83/83 total).** `aggregate.sx` (group-by/count, day buckets) +
  `rank.sx` (recency/velocity/engagement scorers, composite, top-N). `sort` is
  single-arg ascending only — no comparator — so ranking uses a stable two-pass
  `apl-grade-down` (by :at desc, then by score desc) for deterministic tie-breaks.
  Dict keys must be strings, so composite group keys are string-joined ("actor#day").
- **Phase 4 done (105/105 total).** `acl.sx` (per-viewer `feed/visible`,
  `feed/timeline` capstone) + `fed.sx` (merge/ingest/inbound/backfill/federate/
  deliver). ACL/transport are dependency-injected (permit?/remote?/send-fn/fetch-fn)
  since lib/acl + fed-sx don't exist. `feed/normalize` now MERGEs defaults over the
  raw dict (was projecting to 5 keys) so extra metadata (:visible-to, peer fields)
  survives — matches the "flexible bag" principle.

## Roadmap is complete (all 4 phases). Possible follow-ups:

- Wire real acl-sx once `lib/acl/` exists (swap injected `permit?`).
- Wire real fed-sx transport (swap `send-fn`/`fetch-fn`).
- [x] TF-IDF over `:tags` for content ranking — `content.sx`: `feed/tag-df`,
  `feed/tag-idf` (log N/df), `feed/tfidf-score`, `feed/by-relevance`; 15 tests.
  Composes as a scorer with rank.sx. (120/120 total.)
- [x] Notification feed (verb-filtered, per-recipient) — `notify.sx`:
  `feed/notifications`, `feed/notify-verbs`, `feed/notify-digest` (collapses
  "X, Y liked Z" by (verb,object), sorted-deterministic); 8 tests. (128/128 total.)
- [x] **Capstone** `feed/home` — the whole pipeline as one line: fanout ∘ inbox ∘
  dedupe ∘ ACL ∘ rank ∘ take (`home.sx`); 6 tests incl. per-viewer ACL + cross-post
  dedupe. (134/134 total.)
- [x] Per-verb dedupe rules (briefing gotcha #3) — `feed/dedupe-smart` /
  `feed/smart-key`: reactions (like/follow/boost/...) collapse cross-actor on
  (verb,object); posts stay distinct per actor. `feed/collapse-verbs` is
  rebindable policy; 9 tests. (143/143 total.)
- [x] Trending — `feed/trending` / `feed/trending-actors`: objects/actors ranked
  by activity count in a recency window, count-desc with key-asc tiebreak
  (`trending.sx`); 11 tests. (154/154 total.)
- [x] Mute/block — `feed/mute-actors` / `feed/mute-tags` / `feed/mute-objects` /
  `feed/apply-prefs`: viewer-controlled per-request filtering (complements ACL's
  author-controlled visibility) (`mute.sx`); 9 tests. (163/163 total.)
- [x] Pagination — `feed/page`/`feed/page-count` (offset) + `feed/before`/
  `feed/after`/`feed/page-before`/`feed/next-cursor` (cursor by :at, stable under
  inserts) (`page.sx`); 14 tests. (177/177 total.)
- [x] Threading — `feed/replies`/`feed/reply-count`/`feed/thread`/
  `feed/thread-objects`/`feed/thread-size`: conversation closure over `:reply-to`
  (transitive fixpoint), chronological (`thread.sx`); 12 tests. (189/189 total.)

(none)

## Notes for next iteration

- sx-tree MCP tools take `file:` NOT `path:` (CLAUDE.md is stale). Wrong key →
  `Yojson Type_error("Expected string, got null")`. Looks like a broken binary, isn't.
- sx_server binary lives in main repo: `/root/rose-ash/hosts/ocaml/_build/default/bin/sx_server.exe`
  (worktree has no `_build`). conformance.sh already points there with relative fallback.
- Phase 2 substrate verified available: `apl-outer` (∘.×), `apl-member` (∊),
  `apl-unique`, `apl-iota` (1-based).