mono/.claude/plans/glittery-discovering-kahn.md

# Sexp Fragment Protocol: Component Defs Between Services

## Context

Fragment endpoints return raw sexp source (e.g., `(~blog-nav-wrapper :items ...)`). The consuming service embeds this in its page sexp, which the client evaluates. But blog-specific components like `~blog-nav-wrapper` are only in blog's `_COMPONENT_ENV` — not in market's. So market's `client_components_tag()` never sends them to the client, causing "Unknown component" errors.

The fix: transfer component definitions alongside fragments. Services tell the provider what they already have; the provider sends only what's missing. The consuming service registers received defs into its `_COMPONENT_ENV` so they're included in `client_components_tag()` output for the client.

## Approach: Structured Sexp Request/Response

Replace the current GET + `X-Fragment-Request` header protocol with POST + sexp body. This aligns with the vision in `docs/sexpr-internal-protocol-first.md`.

### Request format (POST body)
```scheme
(fragment-request
  :type "nav-tree"
  :params (:app-name "market" :path "/")
  :components (~blog-nav-wrapper ~blog-nav-item-link ~header-row-sx ...))
```

`:components` lists component names already in the consumer's `_COMPONENT_ENV`. Provider skips these.

### Response format
```scheme
(fragment-response
  :defs ((defcomp ~blog-nav-wrapper (&key ...) ...) (defcomp ~blog-nav-item-link ...))
  :content (~blog-nav-wrapper :items ...))
```

`:defs` contains only components the consumer doesn't have. `:content` is the fragment sexp (same as current response body).

## Changes

### 1. `shared/infrastructure/fragments.py` — Client side

**`fetch_fragment()`**: Switch from GET to POST with sexp body.

- Build request body using `sexp_call`:
  ```python
  from shared.sexp.helpers import sexp_call, SexpExpr
  from shared.sexp.jinja_bridge import _COMPONENT_ENV

  comp_names = [k for k in _COMPONENT_ENV if k.startswith("~")]
  body = sexp_call("fragment-request",
      type=fragment_type,
      params=params or {},
      components=SexpExpr("(" + " ".join(comp_names) + ")"))
  ```
- POST to same URL, body as `text/sexp`, keep `X-Fragment-Request` header for backward compat
- Parse response: extract `:defs` and `:content` from the sexp response
- Register defs into `_COMPONENT_ENV` via `register_components()`
- Return `:content` wrapped as `SexpExpr`

**New helper `_parse_fragment_response(text)`**:
- `parse()` the response sexp
- Extract keyword args (reuse the keyword-extraction pattern from `evaluator.py`)
- Return `(defs_source, content_source)` tuple

### 2. `shared/sexp/helpers.py` — Response builder

**New `fragment_response(content, request_text)`**:

```python
def fragment_response(content: str, request_text: str) -> str:
    """Build a structured fragment response with missing component defs."""
    from .parser import parse, serialize
    from .types import Keyword, Component
    from .jinja_bridge import _COMPONENT_ENV

    # Parse request to get :components list
    req = parse(request_text)
    loaded = set()
    # extract :components keyword value
    ...

    # Diff against _COMPONENT_ENV, serialize missing defs
    defs_parts = []
    for key, val in _COMPONENT_ENV.items():
        if not isinstance(val, Component):
            continue
        if key in loaded or f"~{val.name}" in loaded:
            continue
        defs_parts.append(_serialize_defcomp(val))

    defs_sexp = "(" + " ".join(defs_parts) + ")" if defs_parts else "nil"
    return sexp_call("fragment-response",
        defs=SexpExpr(defs_sexp),
        content=SexpExpr(content))
```

### 3. Fragment endpoints — All services

**Generic change in each `bp/fragments/routes.py`**: Update the route handler to accept POST, read sexp body, use `fragment_response()` for the response.

The `get_fragment` handler becomes:
```python
@bp.route("/<fragment_type>", methods=["GET", "POST"])
async def get_fragment(fragment_type: str):
    handler = _handlers.get(fragment_type)
    if handler is None:
        return Response("", status=200, content_type="text/sexp")
    content = await handler()

    # Structured sexp protocol (POST with sexp body)
    request_body = await request.get_data(as_text=True)
    if request_body and request.content_type == "text/sexp":
        from shared.sexp.helpers import fragment_response
        body = fragment_response(content, request_body)
        return Response(body, status=200, content_type="text/sexp")

    # Legacy GET fallback
    return Response(content, status=200, content_type="text/sexp")
```

Since all fragment endpoints follow the identical `_handlers` + `get_fragment` pattern, we can extract this into a shared helper in `fragments.py` or a new `shared/infrastructure/fragment_endpoint.py`.

### 4. Extract shared fragment endpoint helper

To avoid touching every service's fragment routes, create a shared blueprint factory:

**`shared/infrastructure/fragment_endpoint.py`**:
```python
def create_fragment_blueprint(handlers: dict) -> Blueprint:
    """Create a fragment endpoint blueprint with sexp protocol support."""
    bp = Blueprint("fragments", __name__, url_prefix="/internal/fragments")

    @bp.before_request
    async def _require_fragment_header():
        if not request.headers.get(FRAGMENT_HEADER):
            return Response("", status=403)

    @bp.route("/<fragment_type>", methods=["GET", "POST"])
    async def get_fragment(fragment_type: str):
        handler = handlers.get(fragment_type)
        if handler is None:
            return Response("", status=200, content_type="text/sexp")
        content = await handler()

        # Sexp protocol: POST with structured request/response
        if request.method == "POST" and request.content_type == "text/sexp":
            request_body = await request.get_data(as_text=True)
            from shared.sexp.helpers import fragment_response
            body = fragment_response(content, request_body)
            return Response(body, status=200, content_type="text/sexp")

        return Response(content, status=200, content_type="text/sexp")

    return bp
```

Then each service's `register()` just returns `create_fragment_blueprint(_handlers)`. This is a small refactor since they all duplicate the same boilerplate today.

## Files to modify

| File | Change |
|------|--------|
| `shared/infrastructure/fragments.py` | POST sexp body, parse response, register defs |
| `shared/sexp/helpers.py` | `fragment_response()` builder, `_serialize_defcomp()` |
| `shared/infrastructure/fragment_endpoint.py` | **New** — shared blueprint factory |
| `blog/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
| `market/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
| `events/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
| `cart/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
| `account/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
| `orders/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
| `federation/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
| `relations/bp/fragments/routes.py` | Use `create_fragment_blueprint` |

## Verification

1. Start blog + market services: `./dev.sh blog market`
2. Load market page — should fetch nav-tree from blog with sexp protocol
3. Check market logs: no "Unknown component" errors
4. Inspect page source: `client_components_tag()` output includes `~blog-nav-wrapper` etc.
5. Cross-domain sx-get navigation (blog → market) works without reload
6. Run sexp tests: `python3 -m pytest shared/sexp/tests/ -x -q`
7. Second page load: `:components` list in request includes blog nav components, response `:defs` is empty