# Flexible Entity Relationship System

## Context

The rose-ash platform has 9 microservices with decoupled entities (posts, pages, markets, calendars, entries, products, etc.) connected via a generic `ContainerRelation` table. The current system is limited:

- **Parent-child only** — no cardinality, no relation types, no many-to-many
- **Feature flags gate everything** — `PageConfig.features["market"]=true` must be set before creating a market on a page
- **Navigation is disconnected** — `rebuild_navigation()` is a no-op; MenuNode is manually managed
- **Per-domain boilerplate** — each service has its own container-nav fragment handler, its own create-and-attach flow
- **No relation semantics** — you can't distinguish *how* two entities are connected, only *that* they are

The s-expression engine (phases 1-7 complete) gives us a declarative language to define relations and auto-generate UI. Building this now means the remaining sexp work gets designed around the relation model.

**Goal**: A registry-driven system where relation types are declared in s-expressions, cardinality is enforced, navigation is auto-generated, and attach/detach is generic.

---

## Key Decisions

1. **Relations is a graph store, not an orchestrator.** Domain services create their own entities, then call `relate` on the relations service. Relations validates cardinality and stores the link. It never calls out to domain services to create entities.

2. **All entity connections live in ContainerRelation.** Including many-to-many (e.g. post↔calendar_entry). Single source of truth. Domain-specific junction tables (like `calendar_entry_posts`) migrate here.

## Architecture: Three Layers

```
┌─────────────────────────────────────────────────────────┐
│  RELATION REGISTRY  (shared/sexp/relations.py)          │
│  defrelation declarations — loaded at startup by all    │
│  services from shared code. Pure data, no DB.           │
├─────────────────────────────────────────────────────────┤
│  RELATION STORAGE  (relations service, db_relations)    │
│  container_relations table + relation_type column.      │
│  Graph store. Validates against registry.               │
│  Domain services create entities, then call relate.     │
├─────────────────────────────────────────────────────────┤
│  RELATION CONSUMERS  (all services)                     │
│  Nav auto-generation, UI components, fragment rendering │
│  Read registry to know how to display/manage relations  │
└─────────────────────────────────────────────────────────┘
```

---

## Phase A: Relation Registry

### A.1 — `defrelation` s-expression form

New special form for the evaluator, alongside `defcomp`:

```scheme
(defrelation :page->market
  :from "page"
  :to "market"
  :cardinality :one-to-many
  :inverse :market->page
  :nav :submenu
  :nav-icon "fa fa-shopping-bag"
  :nav-label "markets")

(defrelation :page->calendar
  :from "page"
  :to "calendar"
  :cardinality :one-to-many
  :inverse :calendar->page
  :nav :submenu
  :nav-icon "fa fa-calendar"
  :nav-label "calendars")

(defrelation :post->calendar_entry
  :from "post"
  :to "calendar_entry"
  :cardinality :many-to-many
  :inverse :calendar_entry->post
  :nav :inline
  :nav-icon "fa fa-file-alt"
  :nav-label "events")

(defrelation :page->menu_node
  :from "page"
  :to "menu_node"
  :cardinality :one-to-one
  :nav :hidden)
```

### A.2 — RelationDef type

**File: `shared/sexp/types.py`** — add alongside `Component`, `Lambda`:

```python
@dataclass(frozen=True)
class RelationDef:
    name: str               # "page->market"
    from_type: str          # "page"
    to_type: str            # "market"
    cardinality: str        # "one-to-one" | "one-to-many" | "many-to-many"
    inverse: str | None     # "market->page"
    nav: str                # "submenu" | "tab" | "badge" | "inline" | "hidden"
    nav_icon: str | None    # "fa fa-shopping-bag"
    nav_label: str | None   # "markets" — display label for nav sections
```

### A.3 — Registry module

**New file: `shared/sexp/relations.py`**

```python
_RELATION_REGISTRY: dict[str, RelationDef] = {}

def load_relation_registry() -> None:
    """Parse defrelation s-expressions, populate registry."""
    for source in [_PAGE_MARKET, _PAGE_CALENDAR, _POST_ENTRY, _PAGE_MENU_NODE]:
        register_relations(source)

def get_relation(name: str) -> RelationDef | None: ...
def relations_from(entity_type: str) -> list[RelationDef]: ...
def relations_to(entity_type: str) -> list[RelationDef]: ...
```

Called from `create_base_app()` at startup alongside `load_shared_components()`.

### A.4 — `defrelation` in the evaluator

**File: `shared/sexp/evaluator.py`** — add to `_SPECIAL_FORMS`:

Parses keyword args from the s-expression, creates a `RelationDef`, stores it in `_RELATION_REGISTRY` and the current env.

### A.5 — Replaces PageConfig feature flags

- The existence of `defrelation :page->market` means pages *can* have markets. No feature flag needed.
- Admin UI queries `relations_from("page")` to show create buttons.
- `PageConfig` survives only for SumUp payment credentials. The `features` JSON column is deprecated.

### A.6 — Tests

**New file: `shared/sexp/tests/test_relations.py`**
- Parse `defrelation`, verify `RelationDef` fields
- Registry lookup: `get_relation()`, `relations_from()`, `relations_to()`
- Cardinality values validated
- Inverse relation lookup

### Files touched
- `shared/sexp/types.py` — add `RelationDef`
- `shared/sexp/evaluator.py` — add `defrelation` special form
- `shared/sexp/relations.py` — NEW: registry + definitions
- `shared/sexp/tests/test_relations.py` — NEW: registry tests
- `shared/infrastructure/factory.py` — call `load_relation_registry()` at startup

---

## Phase B: Schema Evolution

### B.1 — Add `relation_type` column

**File: `shared/models/container_relation.py`**

```python
relation_type: Mapped[Optional[str]] = mapped_column(String(64), nullable=True, index=True)
```

New composite index: `(relation_type, parent_type, parent_id)` for filtered child queries.

### B.2 — Alembic migration

**File: `relations/alembic/versions/xxx_add_relation_type.py`**

1. Add nullable `relation_type` column
2. Add indexes
3. Backfill existing rows:
   - `(page, *, market, *)` → `relation_type = "page->market"`
   - `(page, *, calendar, *)` → `relation_type = "page->calendar"`
   - `(page, *, menu_node, *)` → `relation_type = "page->menu_node"`

### B.3 — Add `get_parents()` query

**File: `shared/services/relationships.py`**

Inverse of `get_children()` — query by `(child_type, child_id)` with optional `relation_type` filter. Uses existing `ix_container_relations_child` index.

### B.4 — Update `attach_child()` / `detach_child()`

Add optional `relation_type` parameter. When provided:
- Stored in the relation row
- Included in emitted activity `object_data`
- Used for cardinality enforcement (Phase C)

Existing callers without `relation_type` continue to work (backward compatible).

### Files touched
- `shared/models/container_relation.py` — add column + index
- `shared/services/relationships.py` — add `relation_type` param, add `get_parents()`
- `relations/alembic/versions/` — NEW migration

---

## Phase C: Generic Relate/Unrelate API

### C.1 — New action endpoints on relations service

**File: `relations/bp/actions/routes.py`**

**`relate`** — validates against registry, enforces cardinality, delegates to `attach_child()`:
```json
POST /internal/actions/relate
{
  "relation_type": "page->market",
  "from_id": 42,
  "to_id": 7,
  "label": "Farm Shop",
  "metadata": {"slug": "farm-shop"}
}
```

**`unrelate`** — validates, delegates to `detach_child()`:
```json
POST /internal/actions/unrelate
{
  "relation_type": "page->market",
  "from_id": 42,
  "to_id": 7
}
```

**`can-relate`** — pre-flight check (cardinality, registry validation) without creating anything:
```json
POST /internal/actions/can-relate
{
  "relation_type": "page->market",
  "from_id": 42
}
→ {"allowed": true} or {"allowed": false, "reason": "one-to-one already exists"}
```

Domain services call `can-relate` *before* creating the entity to avoid orphans on cardinality failure.

### C.2 — Typical domain flow

```
1. Domain service receives "create market on page 42" request
2. call_action("relations", "can-relate", {relation_type: "page->market", from_id: 42})
3. If not allowed → return error to user (no entity created)
4. Create the entity locally (market service creates MarketPlace row)
5. call_action("relations", "relate", {relation_type: "page->market", from_id: 42, to_id: 7, ...})
```

### C.3 — Cardinality enforcement

In the `relate` and `can-relate` handlers:

- **one-to-one**: Check no active relation of this type exists for `from_id`. Reject if found.
- **one-to-many**: No limit (current behavior).
- **many-to-many**: No limit in either direction.

### C.4 — Enhanced activity emission

Activities now include `relation_type` in `object_data`:
```python
object_data={
    "relation_type": "page->market",
    "parent_type": "page",
    "parent_id": 42,
    "child_type": "market",
    "child_id": 7,
}
```

### C.5 — Existing endpoints stay as aliases

`attach-child` and `detach-child` remain during transition. They infer `relation_type` from `(parent_type, child_type)` when not provided.

### Files touched
- `relations/bp/actions/routes.py` — add `relate`, `unrelate`, `can-relate`
- `shared/services/relationships.py` — cardinality check in `attach_child()`

---

## Phase D: Navigation Auto-Generation

### D.1 — Generic container-nav fragment

**New handler: `relations/bp/fragments/routes.py`**

The relations service becomes a fragment provider. A single generic handler replaces per-service container-nav handlers in market and events:

```python
async def _container_nav_handler():
    container_type = request.args["container_type"]
    container_id = int(request.args["container_id"])
    post_slug = request.args.get("post_slug", "")

    nav_defs = [d for d in relations_from(container_type) if d.nav != "hidden"]
    parts = []

    for defn in nav_defs:
        children = await get_children(
            g.s, container_type, container_id,
            child_type=defn.to_type,
            relation_type=defn.name,
        )
        for child in children:
            parts.append(render_sexp(
                '(~relation-nav :href href :name name :icon icon :nav-class nc)',
                href=_build_href(defn, child, post_slug, ctx),
                name=child.label or "",
                icon=defn.nav_icon or "",
                nc=nav_class,
            ))

    return "\n".join(parts)
```

### D.2 — Nav hints

| `nav` value | Behavior |
|-------------|----------|
| `submenu` | Clickable item in container-nav header/sidebar |
| `tab` | Tab in a tabbed interface |
| `badge` | Count badge on parent |
| `inline` | Inline within parent content |
| `hidden` | Not shown in navigation |

### D.3 — `rebuild_navigation()` becomes real

**File: `shared/services/navigation.py`**

For now: invalidate nav caches when relations change. The top-level MenuNode entries (depth=0, main nav bar) remain manually managed. Child navigation (markets, calendars under a page) is generated dynamically from the relation registry via the generic container-nav fragment.

Future: `rebuild_navigation()` could materialize the full nav tree from the relation graph for performance.

### D.4 — Href computation

The `ContainerRelation.label` column stores the display name. Add a `metadata` JSON column for additional denormalized data (slug, icon) needed to compute hrefs without fetching from the owner service:

```python
metadata: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
# e.g., {"slug": "farm-shop", "icon": "..."}
```

The `relate` action populates this from the caller's payload.

### Files touched
- `relations/bp/fragments/routes.py` — NEW: generic container-nav
- `shared/services/navigation.py` — cache invalidation
- `shared/models/container_relation.py` — add `metadata` JSON column
- `relations/alembic/versions/` — migration for metadata column
- Blog, market, events fragment routes — deprecate per-service container-nav

---

## Phase E: S-Expression UI Components

### E.1 — `~relation-nav` component

**File: `shared/sexp/components.py`** — replaces `~market-link-nav`, `~calendar-link-nav`:

```scheme
(defcomp ~relation-nav (&key href name icon nav-class relation-type)
  (a :href href :class (or nav-class "...")
    (when icon (i :class icon :aria-hidden "true"))
    (div name)))
```

### E.2 — `~relation-attach` component

Generic "add child" button driven by registry:

```scheme
(defcomp ~relation-attach (&key create-url label icon)
  (a :href create-url
     :hx-get create-url
     :hx-target "#main-panel"
     :hx-swap "outerHTML"
     :hx-push-url "true"
     :class "flex items-center gap-2 text-sm p-2 rounded hover:bg-stone-100"
    (when icon (i :class icon))
    (span (or label "Add"))))
```

### E.3 — `~relation-detach` component

Confirm-and-remove button:

```scheme
(defcomp ~relation-detach (&key detach-url name)
  (button :hx-delete detach-url
          :hx-confirm (str "Remove " name "?")
          :class "text-red-500 hover:text-red-700 text-sm"
    (i :class "fa fa-times")))
```

### E.4 — Keep specializations where needed

`~calendar-entry-nav` has date display logic — keep it as a specialization. The generic handler can delegate to specific components based on `to_type` when richer rendering is needed.

### Files touched
- `shared/sexp/components.py` — add `~relation-nav`, `~relation-attach`, `~relation-detach`

---

## Phase F: Migration of Existing Relations

### F.1 — Migration order

1. Add `relation_type` column + backfill (Phase B)
2. Deploy registry + new endpoints (Phases A, C)
3. Switch callers one-by-one to use `can-relate` + `relate`:
   - `shared/services/market_impl.py` — market creation: check can-relate, create, relate
   - `events/bp/calendars/services/calendars.py` — calendar creation
   - `blog/bp/menu_items/services/menu_items.py` — menu node creation
4. Deploy generic container-nav fragment (Phase D)
5. Switch fragment consumers from per-service to generic
6. Deprecate per-service container-nav handlers
7. Deprecate `PageConfig.features` column

### F.2 — CalendarEntryPost migration (many-to-many)

The `calendar_entry_posts` junction table in db_events is a domain-specific many-to-many. Strategy:

- Add `defrelation :post->calendar_entry` with `:cardinality :many-to-many`
- Migrate existing rows to `ContainerRelation` with `relation_type="post->calendar_entry"`
- Update `shared/services/entry_associations.py` to use `relate`/`unrelate`
- Eventually drop `calendar_entry_posts` table

### F.3 — PageConfig simplification

- Remove `features` JSON column (after all feature-flag checks are replaced by registry lookups)
- Keep `PageConfig` for SumUp credentials only
- Consider renaming to `PaymentConfig`

### Files touched (incremental)
- `shared/services/market_impl.py` — use `relate`
- `events/bp/calendars/services/calendars.py` — use `relate`
- `blog/bp/menu_items/services/menu_items.py` — use `relate`
- `shared/services/entry_associations.py` — use `relate`/`unrelate`
- `blog/bp/post/services/markets.py` — simplify (no feature flag check)
- `market/bp/fragments/routes.py` — deprecate container-nav
- `events/bp/fragments/routes.py` — deprecate container-nav

---

## Verification

### Unit tests
- `shared/sexp/tests/test_relations.py` — registry parsing, lookup, cardinality rules
- `shared/sexp/tests/test_evaluator.py` — `defrelation` form evaluation
- `shared/sexp/tests/test_components.py` — `~relation-nav`, `~relation-attach` rendering

### Integration tests
- Relations service: `relate` with valid/invalid types, cardinality enforcement
- `can-relate` pre-flight: returns allowed/denied correctly
- `unrelate`: soft-deletes, emits Remove activity
- Generic container-nav fragment: returns correct HTML for various relation types

### Manual testing
- Browse site with Playwright after each phase
- Verify nav renders correctly
- Test create/delete flows for markets and calendars
- Check activity bus still fires on relation changes

---

## Phase Order & Dependencies

```
Phase A (Registry)  ──────┐
                          ├──▶ Phase C (API)  ──▶ Phase D (Nav)  ──▶ Phase F (Migration)
Phase B (Schema)   ───────┘                        ▲
                                                   │
                                    Phase E (UI Components)
```

A and B can run in parallel. C needs both. D and E need C. F is incremental, runs alongside D and E.