diff --git a/docs/relation-system-plan.md b/docs/relation-system-plan.md new file mode 100644 index 0000000..d7ae9cc --- /dev/null +++ b/docs/relation-system-plan.md @@ -0,0 +1,473 @@ +# Flexible Entity Relationship System + +## Context + +The rose-ash platform has 9 microservices with decoupled entities (posts, pages, markets, calendars, entries, products, etc.) connected via a generic `ContainerRelation` table. The current system is limited: + +- **Parent-child only** — no cardinality, no relation types, no many-to-many +- **Feature flags gate everything** — `PageConfig.features["market"]=true` must be set before creating a market on a page +- **Navigation is disconnected** — `rebuild_navigation()` is a no-op; MenuNode is manually managed +- **Per-domain boilerplate** — each service has its own container-nav fragment handler, its own create-and-attach flow +- **No relation semantics** — you can't distinguish *how* two entities are connected, only *that* they are + +The s-expression engine (phases 1-7 complete) gives us a declarative language to define relations and auto-generate UI. Building this now means the remaining sexp work gets designed around the relation model. + +**Goal**: A registry-driven system where relation types are declared in s-expressions, cardinality is enforced, navigation is auto-generated, and attach/detach is generic. + +--- + +## Key Decisions + +1. **Relations is a graph store, not an orchestrator.** Domain services create their own entities, then call `relate` on the relations service. Relations validates cardinality and stores the link. It never calls out to domain services to create entities. + +2. **All entity connections live in ContainerRelation.** Including many-to-many (e.g. post↔calendar_entry). Single source of truth. Domain-specific junction tables (like `calendar_entry_posts`) migrate here. + +## Architecture: Three Layers + +``` +┌─────────────────────────────────────────────────────────┐ +│ RELATION REGISTRY (shared/sexp/relations.py) │ +│ defrelation declarations — loaded at startup by all │ +│ services from shared code. Pure data, no DB. │ +├─────────────────────────────────────────────────────────┤ +│ RELATION STORAGE (relations service, db_relations) │ +│ container_relations table + relation_type column. │ +│ Graph store. Validates against registry. │ +│ Domain services create entities, then call relate. │ +├─────────────────────────────────────────────────────────┤ +│ RELATION CONSUMERS (all services) │ +│ Nav auto-generation, UI components, fragment rendering │ +│ Read registry to know how to display/manage relations │ +└─────────────────────────────────────────────────────────┘ +``` + +--- + +## Phase A: Relation Registry + +### A.1 — `defrelation` s-expression form + +New special form for the evaluator, alongside `defcomp`: + +```scheme +(defrelation :page->market + :from "page" + :to "market" + :cardinality :one-to-many + :inverse :market->page + :nav :submenu + :nav-icon "fa fa-shopping-bag" + :nav-label "markets") + +(defrelation :page->calendar + :from "page" + :to "calendar" + :cardinality :one-to-many + :inverse :calendar->page + :nav :submenu + :nav-icon "fa fa-calendar" + :nav-label "calendars") + +(defrelation :post->calendar_entry + :from "post" + :to "calendar_entry" + :cardinality :many-to-many + :inverse :calendar_entry->post + :nav :inline + :nav-icon "fa fa-file-alt" + :nav-label "events") + +(defrelation :page->menu_node + :from "page" + :to "menu_node" + :cardinality :one-to-one + :nav :hidden) +``` + +### A.2 — RelationDef type + +**File: `shared/sexp/types.py`** — add alongside `Component`, `Lambda`: + +```python +@dataclass(frozen=True) +class RelationDef: + name: str # "page->market" + from_type: str # "page" + to_type: str # "market" + cardinality: str # "one-to-one" | "one-to-many" | "many-to-many" + inverse: str | None # "market->page" + nav: str # "submenu" | "tab" | "badge" | "inline" | "hidden" + nav_icon: str | None # "fa fa-shopping-bag" + nav_label: str | None # "markets" — display label for nav sections +``` + +### A.3 — Registry module + +**New file: `shared/sexp/relations.py`** + +```python +_RELATION_REGISTRY: dict[str, RelationDef] = {} + +def load_relation_registry() -> None: + """Parse defrelation s-expressions, populate registry.""" + for source in [_PAGE_MARKET, _PAGE_CALENDAR, _POST_ENTRY, _PAGE_MENU_NODE]: + register_relations(source) + +def get_relation(name: str) -> RelationDef | None: ... +def relations_from(entity_type: str) -> list[RelationDef]: ... +def relations_to(entity_type: str) -> list[RelationDef]: ... +``` + +Called from `create_base_app()` at startup alongside `load_shared_components()`. + +### A.4 — `defrelation` in the evaluator + +**File: `shared/sexp/evaluator.py`** — add to `_SPECIAL_FORMS`: + +Parses keyword args from the s-expression, creates a `RelationDef`, stores it in `_RELATION_REGISTRY` and the current env. + +### A.5 — Replaces PageConfig feature flags + +- The existence of `defrelation :page->market` means pages *can* have markets. No feature flag needed. +- Admin UI queries `relations_from("page")` to show create buttons. +- `PageConfig` survives only for SumUp payment credentials. The `features` JSON column is deprecated. + +### A.6 — Tests + +**New file: `shared/sexp/tests/test_relations.py`** +- Parse `defrelation`, verify `RelationDef` fields +- Registry lookup: `get_relation()`, `relations_from()`, `relations_to()` +- Cardinality values validated +- Inverse relation lookup + +### Files touched +- `shared/sexp/types.py` — add `RelationDef` +- `shared/sexp/evaluator.py` — add `defrelation` special form +- `shared/sexp/relations.py` — NEW: registry + definitions +- `shared/sexp/tests/test_relations.py` — NEW: registry tests +- `shared/infrastructure/factory.py` — call `load_relation_registry()` at startup + +--- + +## Phase B: Schema Evolution + +### B.1 — Add `relation_type` column + +**File: `shared/models/container_relation.py`** + +```python +relation_type: Mapped[Optional[str]] = mapped_column(String(64), nullable=True, index=True) +``` + +New composite index: `(relation_type, parent_type, parent_id)` for filtered child queries. + +### B.2 — Alembic migration + +**File: `relations/alembic/versions/xxx_add_relation_type.py`** + +1. Add nullable `relation_type` column +2. Add indexes +3. Backfill existing rows: + - `(page, *, market, *)` → `relation_type = "page->market"` + - `(page, *, calendar, *)` → `relation_type = "page->calendar"` + - `(page, *, menu_node, *)` → `relation_type = "page->menu_node"` + +### B.3 — Add `get_parents()` query + +**File: `shared/services/relationships.py`** + +Inverse of `get_children()` — query by `(child_type, child_id)` with optional `relation_type` filter. Uses existing `ix_container_relations_child` index. + +### B.4 — Update `attach_child()` / `detach_child()` + +Add optional `relation_type` parameter. When provided: +- Stored in the relation row +- Included in emitted activity `object_data` +- Used for cardinality enforcement (Phase C) + +Existing callers without `relation_type` continue to work (backward compatible). + +### Files touched +- `shared/models/container_relation.py` — add column + index +- `shared/services/relationships.py` — add `relation_type` param, add `get_parents()` +- `relations/alembic/versions/` — NEW migration + +--- + +## Phase C: Generic Relate/Unrelate API + +### C.1 — New action endpoints on relations service + +**File: `relations/bp/actions/routes.py`** + +**`relate`** — validates against registry, enforces cardinality, delegates to `attach_child()`: +```json +POST /internal/actions/relate +{ + "relation_type": "page->market", + "from_id": 42, + "to_id": 7, + "label": "Farm Shop", + "metadata": {"slug": "farm-shop"} +} +``` + +**`unrelate`** — validates, delegates to `detach_child()`: +```json +POST /internal/actions/unrelate +{ + "relation_type": "page->market", + "from_id": 42, + "to_id": 7 +} +``` + +**`can-relate`** — pre-flight check (cardinality, registry validation) without creating anything: +```json +POST /internal/actions/can-relate +{ + "relation_type": "page->market", + "from_id": 42 +} +→ {"allowed": true} or {"allowed": false, "reason": "one-to-one already exists"} +``` + +Domain services call `can-relate` *before* creating the entity to avoid orphans on cardinality failure. + +### C.2 — Typical domain flow + +``` +1. Domain service receives "create market on page 42" request +2. call_action("relations", "can-relate", {relation_type: "page->market", from_id: 42}) +3. If not allowed → return error to user (no entity created) +4. Create the entity locally (market service creates MarketPlace row) +5. call_action("relations", "relate", {relation_type: "page->market", from_id: 42, to_id: 7, ...}) +``` + +### C.3 — Cardinality enforcement + +In the `relate` and `can-relate` handlers: + +- **one-to-one**: Check no active relation of this type exists for `from_id`. Reject if found. +- **one-to-many**: No limit (current behavior). +- **many-to-many**: No limit in either direction. + +### C.4 — Enhanced activity emission + +Activities now include `relation_type` in `object_data`: +```python +object_data={ + "relation_type": "page->market", + "parent_type": "page", + "parent_id": 42, + "child_type": "market", + "child_id": 7, +} +``` + +### C.5 — Existing endpoints stay as aliases + +`attach-child` and `detach-child` remain during transition. They infer `relation_type` from `(parent_type, child_type)` when not provided. + +### Files touched +- `relations/bp/actions/routes.py` — add `relate`, `unrelate`, `can-relate` +- `shared/services/relationships.py` — cardinality check in `attach_child()` + +--- + +## Phase D: Navigation Auto-Generation + +### D.1 — Generic container-nav fragment + +**New handler: `relations/bp/fragments/routes.py`** + +The relations service becomes a fragment provider. A single generic handler replaces per-service container-nav handlers in market and events: + +```python +async def _container_nav_handler(): + container_type = request.args["container_type"] + container_id = int(request.args["container_id"]) + post_slug = request.args.get("post_slug", "") + + nav_defs = [d for d in relations_from(container_type) if d.nav != "hidden"] + parts = [] + + for defn in nav_defs: + children = await get_children( + g.s, container_type, container_id, + child_type=defn.to_type, + relation_type=defn.name, + ) + for child in children: + parts.append(render_sexp( + '(~relation-nav :href href :name name :icon icon :nav-class nc)', + href=_build_href(defn, child, post_slug, ctx), + name=child.label or "", + icon=defn.nav_icon or "", + nc=nav_class, + )) + + return "\n".join(parts) +``` + +### D.2 — Nav hints + +| `nav` value | Behavior | +|-------------|----------| +| `submenu` | Clickable item in container-nav header/sidebar | +| `tab` | Tab in a tabbed interface | +| `badge` | Count badge on parent | +| `inline` | Inline within parent content | +| `hidden` | Not shown in navigation | + +### D.3 — `rebuild_navigation()` becomes real + +**File: `shared/services/navigation.py`** + +For now: invalidate nav caches when relations change. The top-level MenuNode entries (depth=0, main nav bar) remain manually managed. Child navigation (markets, calendars under a page) is generated dynamically from the relation registry via the generic container-nav fragment. + +Future: `rebuild_navigation()` could materialize the full nav tree from the relation graph for performance. + +### D.4 — Href computation + +The `ContainerRelation.label` column stores the display name. Add a `metadata` JSON column for additional denormalized data (slug, icon) needed to compute hrefs without fetching from the owner service: + +```python +metadata: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True) +# e.g., {"slug": "farm-shop", "icon": "..."} +``` + +The `relate` action populates this from the caller's payload. + +### Files touched +- `relations/bp/fragments/routes.py` — NEW: generic container-nav +- `shared/services/navigation.py` — cache invalidation +- `shared/models/container_relation.py` — add `metadata` JSON column +- `relations/alembic/versions/` — migration for metadata column +- Blog, market, events fragment routes — deprecate per-service container-nav + +--- + +## Phase E: S-Expression UI Components + +### E.1 — `~relation-nav` component + +**File: `shared/sexp/components.py`** — replaces `~market-link-nav`, `~calendar-link-nav`: + +```scheme +(defcomp ~relation-nav (&key href name icon nav-class relation-type) + (a :href href :class (or nav-class "...") + (when icon (i :class icon :aria-hidden "true")) + (div name))) +``` + +### E.2 — `~relation-attach` component + +Generic "add child" button driven by registry: + +```scheme +(defcomp ~relation-attach (&key create-url label icon) + (a :href create-url + :hx-get create-url + :hx-target "#main-panel" + :hx-swap "outerHTML" + :hx-push-url "true" + :class "flex items-center gap-2 text-sm p-2 rounded hover:bg-stone-100" + (when icon (i :class icon)) + (span (or label "Add")))) +``` + +### E.3 — `~relation-detach` component + +Confirm-and-remove button: + +```scheme +(defcomp ~relation-detach (&key detach-url name) + (button :hx-delete detach-url + :hx-confirm (str "Remove " name "?") + :class "text-red-500 hover:text-red-700 text-sm" + (i :class "fa fa-times"))) +``` + +### E.4 — Keep specializations where needed + +`~calendar-entry-nav` has date display logic — keep it as a specialization. The generic handler can delegate to specific components based on `to_type` when richer rendering is needed. + +### Files touched +- `shared/sexp/components.py` — add `~relation-nav`, `~relation-attach`, `~relation-detach` + +--- + +## Phase F: Migration of Existing Relations + +### F.1 — Migration order + +1. Add `relation_type` column + backfill (Phase B) +2. Deploy registry + new endpoints (Phases A, C) +3. Switch callers one-by-one to use `can-relate` + `relate`: + - `shared/services/market_impl.py` — market creation: check can-relate, create, relate + - `events/bp/calendars/services/calendars.py` — calendar creation + - `blog/bp/menu_items/services/menu_items.py` — menu node creation +4. Deploy generic container-nav fragment (Phase D) +5. Switch fragment consumers from per-service to generic +6. Deprecate per-service container-nav handlers +7. Deprecate `PageConfig.features` column + +### F.2 — CalendarEntryPost migration (many-to-many) + +The `calendar_entry_posts` junction table in db_events is a domain-specific many-to-many. Strategy: + +- Add `defrelation :post->calendar_entry` with `:cardinality :many-to-many` +- Migrate existing rows to `ContainerRelation` with `relation_type="post->calendar_entry"` +- Update `shared/services/entry_associations.py` to use `relate`/`unrelate` +- Eventually drop `calendar_entry_posts` table + +### F.3 — PageConfig simplification + +- Remove `features` JSON column (after all feature-flag checks are replaced by registry lookups) +- Keep `PageConfig` for SumUp credentials only +- Consider renaming to `PaymentConfig` + +### Files touched (incremental) +- `shared/services/market_impl.py` — use `relate` +- `events/bp/calendars/services/calendars.py` — use `relate` +- `blog/bp/menu_items/services/menu_items.py` — use `relate` +- `shared/services/entry_associations.py` — use `relate`/`unrelate` +- `blog/bp/post/services/markets.py` — simplify (no feature flag check) +- `market/bp/fragments/routes.py` — deprecate container-nav +- `events/bp/fragments/routes.py` — deprecate container-nav + +--- + +## Verification + +### Unit tests +- `shared/sexp/tests/test_relations.py` — registry parsing, lookup, cardinality rules +- `shared/sexp/tests/test_evaluator.py` — `defrelation` form evaluation +- `shared/sexp/tests/test_components.py` — `~relation-nav`, `~relation-attach` rendering + +### Integration tests +- Relations service: `relate` with valid/invalid types, cardinality enforcement +- `can-relate` pre-flight: returns allowed/denied correctly +- `unrelate`: soft-deletes, emits Remove activity +- Generic container-nav fragment: returns correct HTML for various relation types + +### Manual testing +- Browse site with Playwright after each phase +- Verify nav renders correctly +- Test create/delete flows for markets and calendars +- Check activity bus still fires on relation changes + +--- + +## Phase Order & Dependencies + +``` +Phase A (Registry) ──────┐ + ├──▶ Phase C (API) ──▶ Phase D (Nav) ──▶ Phase F (Migration) +Phase B (Schema) ───────┘ ▲ + │ + Phase E (UI Components) +``` + +A and B can run in parallel. C needs both. D and E need C. F is incremental, runs alongside D and E.