relation plan

docs/relation-system-plan.md

@@ -0,0 +1,473 @@
+# Flexible Entity Relationship System
+
+## Context
+
+The rose-ash platform has 9 microservices with decoupled entities (posts, pages, markets, calendars, entries, products, etc.) connected via a generic `ContainerRelation` table. The current system is limited:
+
+- **Parent-child only** — no cardinality, no relation types, no many-to-many
+- **Feature flags gate everything** — `PageConfig.features["market"]=true` must be set before creating a market on a page
+- **Navigation is disconnected** — `rebuild_navigation()` is a no-op; MenuNode is manually managed
+- **Per-domain boilerplate** — each service has its own container-nav fragment handler, its own create-and-attach flow
+- **No relation semantics** — you can't distinguish *how* two entities are connected, only *that* they are
+
+The s-expression engine (phases 1-7 complete) gives us a declarative language to define relations and auto-generate UI. Building this now means the remaining sexp work gets designed around the relation model.
+
+**Goal**: A registry-driven system where relation types are declared in s-expressions, cardinality is enforced, navigation is auto-generated, and attach/detach is generic.
+
+---
+
+## Key Decisions
+
+1. **Relations is a graph store, not an orchestrator.** Domain services create their own entities, then call `relate` on the relations service. Relations validates cardinality and stores the link. It never calls out to domain services to create entities.
+
+2. **All entity connections live in ContainerRelation.** Including many-to-many (e.g. post↔calendar_entry). Single source of truth. Domain-specific junction tables (like `calendar_entry_posts`) migrate here.
+
+## Architecture: Three Layers
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  RELATION REGISTRY  (shared/sexp/relations.py)          │
+│  defrelation declarations — loaded at startup by all    │
+│  services from shared code. Pure data, no DB.           │
+├─────────────────────────────────────────────────────────┤
+│  RELATION STORAGE  (relations service, db_relations)    │
+│  container_relations table + relation_type column.      │
+│  Graph store. Validates against registry.               │
+│  Domain services create entities, then call relate.     │
+├─────────────────────────────────────────────────────────┤
+│  RELATION CONSUMERS  (all services)                     │
+│  Nav auto-generation, UI components, fragment rendering │
+│  Read registry to know how to display/manage relations  │
+└─────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Phase A: Relation Registry
+
+### A.1 — `defrelation` s-expression form
+
+New special form for the evaluator, alongside `defcomp`:
+
+```scheme
+(defrelation :page->market
+  :from "page"
+  :to "market"
+  :cardinality :one-to-many
+  :inverse :market->page
+  :nav :submenu
+  :nav-icon "fa fa-shopping-bag"
+  :nav-label "markets")
+
+(defrelation :page->calendar
+  :from "page"
+  :to "calendar"
+  :cardinality :one-to-many
+  :inverse :calendar->page
+  :nav :submenu
+  :nav-icon "fa fa-calendar"
+  :nav-label "calendars")
+
+(defrelation :post->calendar_entry
+  :from "post"
+  :to "calendar_entry"
+  :cardinality :many-to-many
+  :inverse :calendar_entry->post
+  :nav :inline
+  :nav-icon "fa fa-file-alt"
+  :nav-label "events")
+
+(defrelation :page->menu_node
+  :from "page"
+  :to "menu_node"
+  :cardinality :one-to-one
+  :nav :hidden)
+```
+
+### A.2 — RelationDef type
+
+**File: `shared/sexp/types.py`** — add alongside `Component`, `Lambda`:
+
+```python
+@dataclass(frozen=True)
+class RelationDef:
+    name: str               # "page->market"
+    from_type: str          # "page"
+    to_type: str            # "market"
+    cardinality: str        # "one-to-one" | "one-to-many" | "many-to-many"
+    inverse: str | None     # "market->page"
+    nav: str                # "submenu" | "tab" | "badge" | "inline" | "hidden"
+    nav_icon: str | None    # "fa fa-shopping-bag"
+    nav_label: str | None   # "markets" — display label for nav sections
+```
+
+### A.3 — Registry module
+
+**New file: `shared/sexp/relations.py`**
+
+```python
+_RELATION_REGISTRY: dict[str, RelationDef] = {}
+
+def load_relation_registry() -> None:
+    """Parse defrelation s-expressions, populate registry."""
+    for source in [_PAGE_MARKET, _PAGE_CALENDAR, _POST_ENTRY, _PAGE_MENU_NODE]:
+        register_relations(source)
+
+def get_relation(name: str) -> RelationDef | None: ...
+def relations_from(entity_type: str) -> list[RelationDef]: ...
+def relations_to(entity_type: str) -> list[RelationDef]: ...
+```
+
+Called from `create_base_app()` at startup alongside `load_shared_components()`.
+
+### A.4 — `defrelation` in the evaluator
+
+**File: `shared/sexp/evaluator.py`** — add to `_SPECIAL_FORMS`:
+
+Parses keyword args from the s-expression, creates a `RelationDef`, stores it in `_RELATION_REGISTRY` and the current env.
+
+### A.5 — Replaces PageConfig feature flags
+
+- The existence of `defrelation :page->market` means pages *can* have markets. No feature flag needed.
+- Admin UI queries `relations_from("page")` to show create buttons.
+- `PageConfig` survives only for SumUp payment credentials. The `features` JSON column is deprecated.
+
+### A.6 — Tests
+
+**New file: `shared/sexp/tests/test_relations.py`**
+- Parse `defrelation`, verify `RelationDef` fields
+- Registry lookup: `get_relation()`, `relations_from()`, `relations_to()`
+- Cardinality values validated
+- Inverse relation lookup
+
+### Files touched
+- `shared/sexp/types.py` — add `RelationDef`
+- `shared/sexp/evaluator.py` — add `defrelation` special form
+- `shared/sexp/relations.py` — NEW: registry + definitions
+- `shared/sexp/tests/test_relations.py` — NEW: registry tests
+- `shared/infrastructure/factory.py` — call `load_relation_registry()` at startup
+
+---
+
+## Phase B: Schema Evolution
+
+### B.1 — Add `relation_type` column
+
+**File: `shared/models/container_relation.py`**
+
+```python
+relation_type: Mapped[Optional[str]] = mapped_column(String(64), nullable=True, index=True)
+```
+
+New composite index: `(relation_type, parent_type, parent_id)` for filtered child queries.
+
+### B.2 — Alembic migration
+
+**File: `relations/alembic/versions/xxx_add_relation_type.py`**
+
+1. Add nullable `relation_type` column
+2. Add indexes
+3. Backfill existing rows:
+   - `(page, *, market, *)` → `relation_type = "page->market"`
+   - `(page, *, calendar, *)` → `relation_type = "page->calendar"`
+   - `(page, *, menu_node, *)` → `relation_type = "page->menu_node"`
+
+### B.3 — Add `get_parents()` query
+
+**File: `shared/services/relationships.py`**
+
+Inverse of `get_children()` — query by `(child_type, child_id)` with optional `relation_type` filter. Uses existing `ix_container_relations_child` index.
+
+### B.4 — Update `attach_child()` / `detach_child()`
+
+Add optional `relation_type` parameter. When provided:
+- Stored in the relation row
+- Included in emitted activity `object_data`
+- Used for cardinality enforcement (Phase C)
+
+Existing callers without `relation_type` continue to work (backward compatible).
+
+### Files touched
+- `shared/models/container_relation.py` — add column + index
+- `shared/services/relationships.py` — add `relation_type` param, add `get_parents()`
+- `relations/alembic/versions/` — NEW migration
+
+---
+
+## Phase C: Generic Relate/Unrelate API
+
+### C.1 — New action endpoints on relations service
+
+**File: `relations/bp/actions/routes.py`**
+
+**`relate`** — validates against registry, enforces cardinality, delegates to `attach_child()`:
+```json
+POST /internal/actions/relate
+{
+  "relation_type": "page->market",
+  "from_id": 42,
+  "to_id": 7,
+  "label": "Farm Shop",
+  "metadata": {"slug": "farm-shop"}
+}
+```
+
+**`unrelate`** — validates, delegates to `detach_child()`:
+```json
+POST /internal/actions/unrelate
+{
+  "relation_type": "page->market",
+  "from_id": 42,
+  "to_id": 7
+}
+```
+
+**`can-relate`** — pre-flight check (cardinality, registry validation) without creating anything:
+```json
+POST /internal/actions/can-relate
+{
+  "relation_type": "page->market",
+  "from_id": 42
+}
+→ {"allowed": true} or {"allowed": false, "reason": "one-to-one already exists"}
+```
+
+Domain services call `can-relate` *before* creating the entity to avoid orphans on cardinality failure.
+
+### C.2 — Typical domain flow
+
+```
+1. Domain service receives "create market on page 42" request
+2. call_action("relations", "can-relate", {relation_type: "page->market", from_id: 42})
+3. If not allowed → return error to user (no entity created)
+4. Create the entity locally (market service creates MarketPlace row)
+5. call_action("relations", "relate", {relation_type: "page->market", from_id: 42, to_id: 7, ...})
+```
+
+### C.3 — Cardinality enforcement
+
+In the `relate` and `can-relate` handlers:
+
+- **one-to-one**: Check no active relation of this type exists for `from_id`. Reject if found.
+- **one-to-many**: No limit (current behavior).
+- **many-to-many**: No limit in either direction.
+
+### C.4 — Enhanced activity emission
+
+Activities now include `relation_type` in `object_data`:
+```python
+object_data={
+    "relation_type": "page->market",
+    "parent_type": "page",
+    "parent_id": 42,
+    "child_type": "market",
+    "child_id": 7,
+}
+```
+
+### C.5 — Existing endpoints stay as aliases
+
+`attach-child` and `detach-child` remain during transition. They infer `relation_type` from `(parent_type, child_type)` when not provided.
+
+### Files touched
+- `relations/bp/actions/routes.py` — add `relate`, `unrelate`, `can-relate`
+- `shared/services/relationships.py` — cardinality check in `attach_child()`
+
+---
+
+## Phase D: Navigation Auto-Generation
+
+### D.1 — Generic container-nav fragment
+
+**New handler: `relations/bp/fragments/routes.py`**
+
+The relations service becomes a fragment provider. A single generic handler replaces per-service container-nav handlers in market and events:
+
+```python
+async def _container_nav_handler():
+    container_type = request.args["container_type"]
+    container_id = int(request.args["container_id"])
+    post_slug = request.args.get("post_slug", "")
+
+    nav_defs = [d for d in relations_from(container_type) if d.nav != "hidden"]
+    parts = []
+
+    for defn in nav_defs:
+        children = await get_children(
+            g.s, container_type, container_id,
+            child_type=defn.to_type,
+            relation_type=defn.name,
+        )
+        for child in children:
+            parts.append(render_sexp(
+                '(~relation-nav :href href :name name :icon icon :nav-class nc)',
+                href=_build_href(defn, child, post_slug, ctx),
+                name=child.label or "",
+                icon=defn.nav_icon or "",
+                nc=nav_class,
+            ))
+
+    return "\n".join(parts)
+```
+
+### D.2 — Nav hints
+
+| `nav` value | Behavior |
+|-------------|----------|
+| `submenu` | Clickable item in container-nav header/sidebar |
+| `tab` | Tab in a tabbed interface |
+| `badge` | Count badge on parent |
+| `inline` | Inline within parent content |
+| `hidden` | Not shown in navigation |
+
+### D.3 — `rebuild_navigation()` becomes real
+
+**File: `shared/services/navigation.py`**
+
+For now: invalidate nav caches when relations change. The top-level MenuNode entries (depth=0, main nav bar) remain manually managed. Child navigation (markets, calendars under a page) is generated dynamically from the relation registry via the generic container-nav fragment.
+
+Future: `rebuild_navigation()` could materialize the full nav tree from the relation graph for performance.
+
+### D.4 — Href computation
+
+The `ContainerRelation.label` column stores the display name. Add a `metadata` JSON column for additional denormalized data (slug, icon) needed to compute hrefs without fetching from the owner service:
+
+```python
+metadata: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
+# e.g., {"slug": "farm-shop", "icon": "..."}
+```
+
+The `relate` action populates this from the caller's payload.
+
+### Files touched
+- `relations/bp/fragments/routes.py` — NEW: generic container-nav
+- `shared/services/navigation.py` — cache invalidation
+- `shared/models/container_relation.py` — add `metadata` JSON column
+- `relations/alembic/versions/` — migration for metadata column
+- Blog, market, events fragment routes — deprecate per-service container-nav
+
+---
+
+## Phase E: S-Expression UI Components
+
+### E.1 — `~relation-nav` component
+
+**File: `shared/sexp/components.py`** — replaces `~market-link-nav`, `~calendar-link-nav`:
+
+```scheme
+(defcomp ~relation-nav (&key href name icon nav-class relation-type)
+  (a :href href :class (or nav-class "...")
+    (when icon (i :class icon :aria-hidden "true"))
+    (div name)))
+```
+
+### E.2 — `~relation-attach` component
+
+Generic "add child" button driven by registry:
+
+```scheme
+(defcomp ~relation-attach (&key create-url label icon)
+  (a :href create-url
+     :hx-get create-url
+     :hx-target "#main-panel"
+     :hx-swap "outerHTML"
+     :hx-push-url "true"
+     :class "flex items-center gap-2 text-sm p-2 rounded hover:bg-stone-100"
+    (when icon (i :class icon))
+    (span (or label "Add"))))
+```
+
+### E.3 — `~relation-detach` component
+
+Confirm-and-remove button:
+
+```scheme
+(defcomp ~relation-detach (&key detach-url name)
+  (button :hx-delete detach-url
+          :hx-confirm (str "Remove " name "?")
+          :class "text-red-500 hover:text-red-700 text-sm"
+    (i :class "fa fa-times")))
+```
+
+### E.4 — Keep specializations where needed
+
+`~calendar-entry-nav` has date display logic — keep it as a specialization. The generic handler can delegate to specific components based on `to_type` when richer rendering is needed.
+
+### Files touched
+- `shared/sexp/components.py` — add `~relation-nav`, `~relation-attach`, `~relation-detach`
+
+---
+
+## Phase F: Migration of Existing Relations
+
+### F.1 — Migration order
+
+1. Add `relation_type` column + backfill (Phase B)
+2. Deploy registry + new endpoints (Phases A, C)
+3. Switch callers one-by-one to use `can-relate` + `relate`:
+   - `shared/services/market_impl.py` — market creation: check can-relate, create, relate
+   - `events/bp/calendars/services/calendars.py` — calendar creation
+   - `blog/bp/menu_items/services/menu_items.py` — menu node creation
+4. Deploy generic container-nav fragment (Phase D)
+5. Switch fragment consumers from per-service to generic
+6. Deprecate per-service container-nav handlers
+7. Deprecate `PageConfig.features` column
+
+### F.2 — CalendarEntryPost migration (many-to-many)
+
+The `calendar_entry_posts` junction table in db_events is a domain-specific many-to-many. Strategy:
+
+- Add `defrelation :post->calendar_entry` with `:cardinality :many-to-many`
+- Migrate existing rows to `ContainerRelation` with `relation_type="post->calendar_entry"`
+- Update `shared/services/entry_associations.py` to use `relate`/`unrelate`
+- Eventually drop `calendar_entry_posts` table
+
+### F.3 — PageConfig simplification
+
+- Remove `features` JSON column (after all feature-flag checks are replaced by registry lookups)
+- Keep `PageConfig` for SumUp credentials only
+- Consider renaming to `PaymentConfig`
+
+### Files touched (incremental)
+- `shared/services/market_impl.py` — use `relate`
+- `events/bp/calendars/services/calendars.py` — use `relate`
+- `blog/bp/menu_items/services/menu_items.py` — use `relate`
+- `shared/services/entry_associations.py` — use `relate`/`unrelate`
+- `blog/bp/post/services/markets.py` — simplify (no feature flag check)
+- `market/bp/fragments/routes.py` — deprecate container-nav
+- `events/bp/fragments/routes.py` — deprecate container-nav
+
+---
+
+## Verification
+
+### Unit tests
+- `shared/sexp/tests/test_relations.py` — registry parsing, lookup, cardinality rules
+- `shared/sexp/tests/test_evaluator.py` — `defrelation` form evaluation
+- `shared/sexp/tests/test_components.py` — `~relation-nav`, `~relation-attach` rendering
+
+### Integration tests
+- Relations service: `relate` with valid/invalid types, cardinality enforcement
+- `can-relate` pre-flight: returns allowed/denied correctly
+- `unrelate`: soft-deletes, emits Remove activity
+- Generic container-nav fragment: returns correct HTML for various relation types
+
+### Manual testing
+- Browse site with Playwright after each phase
+- Verify nav renders correctly
+- Test create/delete flows for markets and calendars
+- Check activity bus still fires on relation changes
+
+---
+
+## Phase Order & Dependencies
+
+```
+Phase A (Registry)  ──────┐
+                          ├──▶ Phase C (API)  ──▶ Phase D (Nav)  ──▶ Phase F (Migration)
+Phase B (Schema)   ───────┘                        ▲
+                                                   │
+                                    Phase E (UI Components)
+```
+
+A and B can run in parallel. C needs both. D and E need C. F is incremental, runs alongside D and E.