rose-ash/.claude/plans/glittery-discovering-kahn.md

Sexp Fragment Protocol: Component Defs Between Services

Context

Fragment endpoints return raw sexp source (e.g., (~blog-nav-wrapper :items ...)). The consuming service embeds this in its page sexp, which the client evaluates. But blog-specific components like ~blog-nav-wrapper are only in blog's _COMPONENT_ENV — not in market's. So market's client_components_tag() never sends them to the client, causing "Unknown component" errors.

The fix: transfer component definitions alongside fragments. Services tell the provider what they already have; the provider sends only what's missing. The consuming service registers received defs into its _COMPONENT_ENV so they're included in client_components_tag() output for the client.

Approach: Structured Sexp Request/Response

Replace the current GET + X-Fragment-Request header protocol with POST + sexp body. This aligns with the vision in docs/sexpr-internal-protocol-first.md.

Request format (POST body)

(fragment-request
  :type "nav-tree"
  :params (:app-name "market" :path "/")
  :components (~blog-nav-wrapper ~blog-nav-item-link ~header-row-sx ...))

:components lists component names already in the consumer's _COMPONENT_ENV. Provider skips these.

Response format

(fragment-response
  :defs ((defcomp ~blog-nav-wrapper (&key ...) ...) (defcomp ~blog-nav-item-link ...))
  :content (~blog-nav-wrapper :items ...))

:defs contains only components the consumer doesn't have. :content is the fragment sexp (same as current response body).

Changes

1. shared/infrastructure/fragments.py — Client side

fetch_fragment(): Switch from GET to POST with sexp body.

• Build request body using sexp_call:

  from shared.sexp.helpers import sexp_call, SexpExpr
  from shared.sexp.jinja_bridge import _COMPONENT_ENV
  
  comp_names = [k for k in _COMPONENT_ENV if k.startswith("~")]
  body = sexp_call("fragment-request",
      type=fragment_type,
      params=params or {},
      components=SexpExpr("(" + " ".join(comp_names) + ")"))
  

• POST to same URL, body as text/sexp, keep X-Fragment-Request header for backward compat
• Parse response: extract :defs and :content from the sexp response
• Register defs into _COMPONENT_ENV via register_components()
• Return :content wrapped as SexpExpr

New helper _parse_fragment_response(text):

• parse() the response sexp
• Extract keyword args (reuse the keyword-extraction pattern from evaluator.py)
• Return (defs_source, content_source) tuple

2. shared/sexp/helpers.py — Response builder

New fragment_response(content, request_text):

def fragment_response(content: str, request_text: str) -> str:
    """Build a structured fragment response with missing component defs."""
    from .parser import parse, serialize
    from .types import Keyword, Component
    from .jinja_bridge import _COMPONENT_ENV

    # Parse request to get :components list
    req = parse(request_text)
    loaded = set()
    # extract :components keyword value
    ...

    # Diff against _COMPONENT_ENV, serialize missing defs
    defs_parts = []
    for key, val in _COMPONENT_ENV.items():
        if not isinstance(val, Component):
            continue
        if key in loaded or f"~{val.name}" in loaded:
            continue
        defs_parts.append(_serialize_defcomp(val))

    defs_sexp = "(" + " ".join(defs_parts) + ")" if defs_parts else "nil"
    return sexp_call("fragment-response",
        defs=SexpExpr(defs_sexp),
        content=SexpExpr(content))

3. Fragment endpoints — All services

Generic change in each bp/fragments/routes.py: Update the route handler to accept POST, read sexp body, use fragment_response() for the response.

The get_fragment handler becomes:

@bp.route("/<fragment_type>", methods=["GET", "POST"])
async def get_fragment(fragment_type: str):
    handler = _handlers.get(fragment_type)
    if handler is None:
        return Response("", status=200, content_type="text/sexp")
    content = await handler()

    # Structured sexp protocol (POST with sexp body)
    request_body = await request.get_data(as_text=True)
    if request_body and request.content_type == "text/sexp":
        from shared.sexp.helpers import fragment_response
        body = fragment_response(content, request_body)
        return Response(body, status=200, content_type="text/sexp")

    # Legacy GET fallback
    return Response(content, status=200, content_type="text/sexp")

Since all fragment endpoints follow the identical _handlers + get_fragment pattern, we can extract this into a shared helper in fragments.py or a new shared/infrastructure/fragment_endpoint.py.

4. Extract shared fragment endpoint helper

To avoid touching every service's fragment routes, create a shared blueprint factory:

shared/infrastructure/fragment_endpoint.py:

def create_fragment_blueprint(handlers: dict) -> Blueprint:
    """Create a fragment endpoint blueprint with sexp protocol support."""
    bp = Blueprint("fragments", __name__, url_prefix="/internal/fragments")

    @bp.before_request
    async def _require_fragment_header():
        if not request.headers.get(FRAGMENT_HEADER):
            return Response("", status=403)

    @bp.route("/<fragment_type>", methods=["GET", "POST"])
    async def get_fragment(fragment_type: str):
        handler = handlers.get(fragment_type)
        if handler is None:
            return Response("", status=200, content_type="text/sexp")
        content = await handler()

        # Sexp protocol: POST with structured request/response
        if request.method == "POST" and request.content_type == "text/sexp":
            request_body = await request.get_data(as_text=True)
            from shared.sexp.helpers import fragment_response
            body = fragment_response(content, request_body)
            return Response(body, status=200, content_type="text/sexp")

        return Response(content, status=200, content_type="text/sexp")

    return bp

Then each service's register() just returns create_fragment_blueprint(_handlers). This is a small refactor since they all duplicate the same boilerplate today.

Files to modify

File	Change
shared/infrastructure/fragments.py	POST sexp body, parse response, register defs
shared/sexp/helpers.py	fragment_response() builder, _serialize_defcomp()
shared/infrastructure/fragment_endpoint.py	New — shared blueprint factory
blog/bp/fragments/routes.py	Use create_fragment_blueprint
market/bp/fragments/routes.py	Use create_fragment_blueprint
events/bp/fragments/routes.py	Use create_fragment_blueprint
cart/bp/fragments/routes.py	Use create_fragment_blueprint
account/bp/fragments/routes.py	Use create_fragment_blueprint
orders/bp/fragments/routes.py	Use create_fragment_blueprint
federation/bp/fragments/routes.py	Use create_fragment_blueprint
relations/bp/fragments/routes.py	Use create_fragment_blueprint

Verification

1. Start blog + market services: ./dev.sh blog market
2. Load market page — should fetch nav-tree from blog with sexp protocol
3. Check market logs: no "Unknown component" errors
4. Inspect page source: client_components_tag() output includes ~blog-nav-wrapper etc.
5. Cross-domain sx-get navigation (blog → market) works without reload
6. Run sexp tests: python3 -m pytest shared/sexp/tests/ -x -q
7. Second page load: :components list in request includes blog nav components, response :defs is empty