# Rose-Ash S-Expression Architecture Transformation

## Context

Rose-ash is a federated cooperative platform built as a Quart microservice monorepo (blog, market, cart, events, federation, account, likes, relations). It currently uses Jinja2 templates for rendering, ad-hoc Python code for fragment composition, and HTTP-based inter-service communication (data, actions, fragments, inbox).

Separately, the art-dag and art-dag-mono repos contain a DAG execution engine, s-expression parser/evaluator, Celery rendering pipeline, and 104k lines of media processing primitives — but are not integrated with rose-ash.

**The transformation**: Unify these into a single architecture where s-expressions are the application language. Pages and media renders become the same computation — DAG execution over content-addressed nodes. Python drops to the role of runtime primitives (DB, HTTP, IPFS, GPU). The app logic, layouts, components, routes, and data bindings are all expressed in s-expressions.

**Key insight 1**: Rendering a page and rendering a video are the same function — walk a DAG of content-addressed nodes, check cache, compute what's missing, assemble the result. The only difference is the leaf executor (Jinja/HTML vs JAX/pixels).

**Key insight 2**: S-expressions are the natural output language for an LLM. An LLM trained on the primitive vocabulary generates s-expressions in response to natural language prompts and HTTP requests. The primitives are the guardrails — the LLM can compose freely but can only invoke what's registered. The s-expressions are then resolved and rendered as HTML pages, media assets, or any other output format. The app becomes a conversation: user speaks natural language → LLM speaks s-expressions → resolver renders results. The LLM learns continually from the data flowing through the system — the s-expressions it generates, the content they resolve to, the user interactions they produce.

---

## Architecture Overview

```
┌──────────────────────────────────────────────┐
│  Natural language (the interface)             │
│  user prompts, HTTP requests, AP activities   │
├──────────────────────────────────────────────┤
│  LLM (the compiler)                          │
│  natural language → s-expressions             │
│  trained on primitives, learns from data      │
├──────────────────────────────────────────────┤
│  S-expressions (the application)             │
│  components, pages, routes, data bindings,   │
│  media effects, composition, federation      │
├──────────────────────────────────────────────┤
│  Resolver (the engine)                       │
│  parse → analyze → plan → execute → cache    │
│  content-addressed at every node             │
├──────────────────────────────────────────────┤
│  Python primitives (the runtime)             │
│  HTTP/DB/Redis/IPFS/Celery/JAX/AP/OAuth      │
└──────────────────────────────────────────────┘

Storage tiers:
  Hot:  Redis (seconds-minutes, user-specific, ephemeral)
  Warm: IPFS  (immutable, content-addressed, shared)
  Pointers: IPNS (mutable "current version" references)

Feedback loop:
  LLM generates s-expression → resolver executes → result cached
  → user interacts → interaction data feeds back to LLM training
  → LLM improves its s-expression generation
```

---

## Phase 1: S-Expression Core Library

**Goal**: A standalone s-expression parser, evaluator, and primitive registry that every service can import. Pure data manipulation — no I/O, no rendering, no HTTP.

**Source material**: `~/art-dag-mono/core/artdag/sexp/` (parser.py, evaluator.py, compiler.py, primitives.py)

**Deliverables**:
```
shared/sexp/
  __init__.py
  parser.py         # tokenize + parse s-expressions to AST
  types.py          # SExp, Symbol, Keyword, Atom types
  evaluator.py      # evaluate with environments, closures, let-bindings
  primitives.py     # register_primitive decorator + base primitives
  env.py            # environment/scope management
```

**Base primitives** (no I/O, pure transforms):
- `seq`, `list`, `map`, `filter`, `reduce`
- `let`, `lambda`, `defcomp`, `if`/`when`/`cond`
- `str`, `concat`, `format`
- `slot` (access keyword fields from data)
- Arithmetic, comparison, logic

**Tasks**:
1. Port parser from art-dag-mono, adapt to rose-ash conventions
2. Port evaluator, add `defcomp` (component definition) and `defroute` forms
3. Define type system (SExp, Symbol, Keyword, String, Number, List, Nil)
4. Implement environment/scope chain
5. Write primitive registry with `@register_primitive` decorator
6. Unit tests

**Verification**: Pure unit tests — parse → evaluate → assert result.

---

## Phase 2: HTML Renderer

**Goal**: An HSX-style renderer that walks an s-expression tree and emits HTML strings. Handles elements, attributes, components, fragments, raw HTML, escaping.

**Reference**: HSX (Common Lisp) — s-expressions map directly to HTML elements.

**Deliverables**:
```
shared/sexp/
  html.py           # s-expression → HTML string renderer
  escape.py         # attribute and text escaping (XSS prevention)
  components.py     # defcomp registry, component resolution
```

**Syntax conventions**:
```scheme
(div :class "foo" :id "bar"       ;; HTML element with attributes
  (h1 "Title")                     ;; text children
  (p "Paragraph"))

(defcomp ~card (&key title children)  ;; component (~ prefix)
  (div :class "card"
    (h2 title)
    children))

(~card :title "Hello"             ;; component invocation
  (p "Body"))

(<> (li "One") (li "Two"))        ;; fragment (no wrapper element)

(raw! "<b>trusted</b>")           ;; unescaped HTML

(when condition (p "shown"))      ;; conditional rendering
(map fn items)                    ;; list rendering
```

**Tasks**:
1. Implement HTML element rendering (tag, attributes, children)
2. Implement text escaping (prevent XSS — escape &, <, >, ", ')
3. Implement `raw!` for trusted HTML (existing Jinja `| safe` equivalent)
4. Implement fragment (`<>`) rendering
5. Implement `defcomp` / component registry and invocation
6. Implement void elements (img, br, input, meta, link)
7. Boolean attributes (disabled, checked, required)
8. Unit tests — render s-expression, assert HTML output

**Verification**: Render existing fragment templates as s-expressions, diff against current Jinja output.

---

## Phase 3: Async Resolver

**Goal**: Walk an s-expression tree, identify nodes that need I/O (service fragments, data queries), fetch them in parallel, substitute results. This is the DAG execution engine applied to page rendering.

**Source material**:
- `~/art-dag-mono/core/artdag/engine.py` (analyze → plan → execute)
- `~/rose-ash/shared/infrastructure/fragments.py` (fetch_fragment, fetch_fragments, fetch_fragment_batch)

**Deliverables**:
```
shared/sexp/
  resolver.py       # async tree walker — identify, fetch, substitute
  cache.py          # content-addressed caching (SHA3-256 → Redis/IPFS)
  primitives_io.py  # I/O primitives (frag, query, action)
```

**I/O primitives** (async, registered separately from pure primitives):
- `(frag service type :key val ...)` → fetch_fragment
- `(query service query-name :key val ...)` → fetch_data
- `(action service action-name :key val ...)` → call_action
- `(current-user)` → load user from request context
- `(htmx-request?)` → check HX-Request header

**Resolution strategy**:
1. Parse the s-expression tree
2. Walk the tree, identify all `frag`/`query` nodes
3. Group independent fetches, dispatch via `asyncio.gather()`
4. Substitute results into the tree
5. Render resolved tree to HTML

**Content addressing**:
- Hash each subtree (SHA3-256 of the s-expression text)
- Check Redis (hot cache) → check IPFS (warm cache) → compute
- Cache rendered subtrees at configurable granularity

**Tasks**:
1. Implement async tree walker with parallel fetch grouping
2. Port content-addressed caching from art-dag-mono/core/artdag/cache.py
3. Implement `frag` primitive (wraps existing fetch_fragment)
4. Implement `query` primitive (wraps existing fetch_data)
5. Implement `action` primitive (wraps existing call_action)
6. Implement request-context primitives (current-user, htmx-request?)
7. Integration tests against running services

**Verification**: Render a blog post page via resolver, compare output to current Jinja render.

---

## Phase 4: Bridge — Coexistence with Jinja

**Goal**: Allow s-expression components and Jinja templates to coexist. Migrate incrementally — one component at a time, one page at a time.

**Deliverables**:
```
shared/sexp/
  jinja_bridge.py   # Jinja filter/global to render s-expressions in templates
                    # + helper to embed Jinja output in s-expressions via raw!
```

**Bridge patterns**:

```python
# In Jinja: render an s-expression component
{{ sexp('(~link-card :slug "apple" :title "Apple")') | safe }}

# In s-expression: embed existing Jinja template output
(raw! (jinja "fragments/nav_tree.html" :items nav-items))
```

**Migration order for fragments** (leaf nodes first):
1. `link-card` (blog, market, events, federation) — simplest, self-contained
2. `cart-mini` — small, user-specific
3. `auth-menu` — small, user-specific
4. `nav-tree` — recursive structure, good test of composition
5. `container-nav`, `container-cards` — cross-service composites

**Tasks**:
1. Implement `sexp()` Jinja global function
2. Implement `jinja` s-expression primitive
3. Rewrite `link-card` fragment as s-expression component (all services)
4. Rewrite `cart-mini` and `auth-menu` fragments
5. Rewrite `nav-tree` fragment
6. Rewrite `container-nav` and `container-cards` fragments
7. Verify each rewritten fragment produces identical HTML

**Verification**: A/B test — render via Jinja, render via s-expression, diff output.

---

## Phase 5: Page Layouts as S-Expressions

**Goal**: Replace Jinja template inheritance (`{% extends %}`, `{% block %}`) with s-expression component composition. Layouts become components with slots.

**Current template hierarchy**:
```
_types/root/index.html          → base HTML shell
  _types/root/_index.html       → layout with aside, filter, content slots
    _types/blog/index.html      → blog-specific layout
      _types/post/index.html    → post page
```

**Becomes**:
```scheme
(defcomp ~base-layout (&key title user cart &rest content)
  (html :lang "en"
    (head (title title) ...)
    (body :class "min-h-screen"
      (~header :user user :cart cart)
      (main :class "flex" content))))

(defcomp ~app-layout (&key title user cart aside filter content)
  (~base-layout :title title :user user :cart cart
    (when filter (div :id "filter" filter))
    (aside :id "aside" aside)
    (section :id "main-panel" content)))

(defcomp ~post-page (&key post nav-items user cart)
  (~app-layout
    :title (:slot post :title)
    :user user :cart cart
    :aside (~nav-tree :items nav-items)
    :content
      (article
        (h1 (:slot post :title))
        (div :class "prose" (raw! (:slot post :body))))))
```

**OOB updates** (HTMX partial renders):
```scheme
(defcomp ~post-oob (&key post nav-items)
  (<>
    (div :id "filter" :hx-swap-oob "outerHTML"
      (~post-filter :post post))
    (aside :id "aside" :hx-swap-oob "outerHTML"
      (~nav-tree :items nav-items))
    (section :id "main-panel"
      (article ...))))
```

**Tasks**:
1. Define `~base-layout` component (replaces `_types/root/index.html`)
2. Define `~app-layout` component (replaces `_types/root/_index.html`)
3. Define `~header` component (replaces header block)
4. Define OOB rendering pattern (replaces `oob_elements.html`)
5. Rewrite blog post page as s-expression
6. Rewrite market product page
7. Rewrite cart page
8. Rewrite events calendar page
9. Update route handlers to use resolver instead of render_template
10. Remove migrated Jinja templates

**Verification**: Visual comparison — deploy both paths, screenshot diff.

---

## Phase 6: Routes as S-Expressions

**Goal**: Route definitions move from Python decorators + handler functions to s-expression declarations. Python route handlers become thin dispatchers.

**Current**:
```python
@post_bp.get("/<slug>/")
async def post_view(slug):
    post = await services.blog.get_post_by_slug(g.s, slug)
    ctx = await post_data(slug, g.s)
    if is_htmx_request():
        return render_template("_types/post/_oob_elements.html", **ctx)
    return render_template("_types/post/index.html", **ctx)
```

**Becomes**:
```scheme
(defroute "/blog/:slug/"
  (let ((post (query blog post-by-slug :slug slug))
        (nav (query blog nav-tree))
        (user (current-user))
        (cart (when user (query cart cart-summary :user_id (:slot user :id)))))
    (if (htmx-request?)
      (render (~post-oob :post post :nav-items nav))
      (render (~post-page :post post :nav-items nav :user user :cart cart)))))
```

**Python dispatcher**:
```python
# One generic route handler per service
@bp.route("/<path:path>")
async def dispatch(path):
    route_expr = match_route(path)  # find matching defroute
    return await resolve_and_render(route_expr, request)
```

**Tasks**:
1. Implement `defroute` form with path pattern matching
2. Implement route registry (load s-expression route files at startup)
3. Implement request context binding (path params, query params, headers)
4. Write generic Quart dispatcher
5. Migrate blog routes
6. Migrate market routes
7. Migrate cart routes
8. Migrate events routes
9. Migrate account/federation routes

**Verification**: Full integration test — HTTP requests produce correct responses.

---

## Phase 7: Content Addressing + IPFS + IPNS

**Goal**: Resolved fragment trees are content-addressed and cached on IPFS. IPNS provides mutable pointers to current versions. Cache invalidation becomes IPNS pointer updates.

**Source material**:
- `~/rose-ash/shared/utils/ipfs_client.py` (already in rose-ash)
- `~/rose-ash/shared/utils/anchoring.py` (merkle trees, OTS)
- `~/art-dag-mono/core/artdag/cache.py` (content-addressed caching)

**Two-tier caching**:
```
Hot tier (Redis):   user-specific, short TTL, ephemeral
  key: sha3(s-expression) → rendered HTML
  examples: cart-mini, auth-menu

Warm tier (IPFS):   deterministic, immutable, shared
  CID: sha3(s-expression) → rendered HTML on IPFS
  IPNS name: stable identity → current CID
  examples: post-body, nav-tree, link-card, full pages
```

**Invalidation**:
- Content changes → new s-expression → new hash → new CID
- Service publishes new CID to IPNS name
- ActivityPub `Update` activity propagates to federated instances
- No TTL-based expiry for warm tier — immutable content, versioned pointers

**Tasks**:
1. Implement SHA3-256 hashing of s-expression subtrees
2. Implement two-tier cache lookup (Redis → IPFS → compute)
3. Implement IPNS name management per fragment type
4. Implement cache warming (pre-render and pin stable content)
5. Wire invalidation into event bus (content change → IPNS update)
6. Wire IPNS updates into AP federation (Update activities)

**Verification**: Cache hit rates, IPFS pin counts, IPNS resolution latency.

---

## Phase 8: Media Pipeline Integration

**Goal**: Bring art-dag's Celery rendering pipeline into rose-ash as the `render` service. Same s-expression language, same resolver, different leaf executors (JAX/FFmpeg instead of HTML).

**Source material**:
- `~/art-dag-mono/l1/` (Celery app, tasks, sexp_effects, streaming)
- `~/art-dag-mono/core/artdag/` (engine, analysis, planning, nodes, effects)

**Deliverables**:
```
rose-ash/
  render/                    # new service
    celery_app.py            # from art-dag-mono/l1/celery_app.py
    tasks/                   # from art-dag-mono/l1/tasks/
    sexp_effects/            # from art-dag-mono/l1/sexp_effects/
      primitives.py          # 104k lines of media primitives
      interpreter.py
      wgsl_compiler.py       # GPU shaders
    effects/                 # effect plugins
    streaming/               # video streaming output
    Dockerfile
    Dockerfile.gpu
    docker-compose.yml
```

**Integration points**:
- `(render-media expr)` primitive dispatches to Celery task
- Results stored on IPFS, CID returned
- Event bus activity emitted on completion
- Same content-addressing — same s-expression → same output CID

**Tasks**:
1. Move L1 codebase into `rose-ash/render/`
2. Adapt imports to use `shared/sexp/` parser (replace local copy)
3. Register media primitives alongside web primitives
4. Implement `render-media` primitive (dispatch to Celery)
5. Wire Celery task completion into event bus
6. Integration tests — submit recipe, verify output on IPFS

**Verification**: Submit an s-expression media recipe via the resolver, get back an IPFS CID with the rendered output.

---

## Phase 9: Unified DAG Executor

**Goal**: One execution engine that handles both page renders and media renders. The executor dispatches to different primitive sets based on node type, but the resolution, caching, and content-addressing logic is shared.

**Deliverables**:
```
shared/sexp/
  executor.py        # unified DAG executor
  registry.py        # executor registry (HTML executors, media executors)
```

**Unified flow**:
```
input s-expression
  → parse (shared/sexp/parser.py)
  → analyze (identify node types, owners, dependencies)
  → plan (check cache tiers, determine fetch/compute order)
  → execute (dispatch to registered executors in parallel)
  → cache (store results at appropriate tier)
  → return (HTML string, media CID, or composed result)
```

**Tasks**:
1. Abstract the resolver into a generic DAG executor
2. Implement executor registry (register by node type/prefix)
3. Register HTML executors (frag, query, render, defcomp)
4. Register media executors (transcode, filter, compose, source)
5. Implement mixed-mode execution (page with embedded media)
6. Provenance tracking (link executor output to AP activities)

**Verification**: A single `resolve()` call handles a page that contains both HTML components and embedded media references.

---

## Phase 10: Federation of S-Expressions

**Goal**: S-expression components and pages are first-class ActivityPub objects. Remote instances can fetch, render, cache, and re-style federated content expressed as s-expressions.

**Integration with existing AP infrastructure**:
- `shared/infrastructure/activitypub.py` — actor endpoints
- `shared/events/bus.py` — activity emission
- `shared/utils/ipfs_client.py` — content storage
- `shared/utils/anchoring.py` — provenance

**Tasks**:
1. Define AP object type for s-expression content (`rose:SExpression`)
2. Publish component definitions as AP Create activities
3. Federate page updates as AP Update activities with IPNS pointers
4. Implement remote component resolution (fetch s-expr from remote instance)
5. Implement content verification (signature on s-expression CID)
6. Implement re-styling (apply local theme to remote s-expression)

**Verification**: Instance A publishes a post, Instance B resolves and renders it from the federated s-expression.

---

## Phase 11: LLM as S-Expression Compiler

**Goal**: An LLM trained on the primitive vocabulary generates s-expressions from natural language. Users describe what they want in plain English. The LLM outputs valid s-expressions. The resolver renders them. Pages are generated on the fly from conversation.

**The LLM speaks s-expressions because**:
- The primitive vocabulary is small and well-defined (unlike HTML/CSS/JS)
- S-expressions are structurally simple — easy for an LLM to generate correctly
- The resolver validates and sandboxes — the LLM can't produce unsafe output
- Every generated s-expression is content-addressed — same prompt → cacheable result
- The primitives are the training data — the LLM learns what `~product-card`, `~nav-tree`, `(query market ...)` do from examples

**How it works**:

```
User: "show me a page of seasonal vegetables under £5"

LLM generates:
(~app-layout :title "Seasonal Vegetables Under £5"
  :content
    (let ((products (query market products-search
                     :category "vegetables"
                     :seasonal true
                     :max_price 5.00
                     :sort "price-asc")))
      (div :class "grid grid-cols-3 gap-4"
        (if (empty? products)
          (p :class "text-gray-500" "No vegetables found matching your criteria.")
          (map (lambda (p) (~product-card :slug (:slot p :slug)))
               products)))))

Resolver: parse → fetch products → render cards → HTML page
```

**Three modes of LLM integration**:

1. **Generative pages**: User prompt → LLM → s-expression → rendered page
   - Conversational UI: user refines via follow-up prompts
   - Each generated page is a CID on IPFS — shareable, cacheable

2. **Adaptive layouts**: LLM observes user behavior → generates personalized component arrangements
   - Home page adapts: frequent buyer sees cart-heavy layout
   - Event organizer sees calendar-first layout
   - Same primitives, different composition

3. **Content authoring**: LLM assists in creating blog posts, product descriptions, event listings
   - Author describes intent → LLM generates structured s-expression content
   - Content is data (s-expression), not just text — queryable, composable, versionable

**Training the LLM on primitives**:
- Primitive catalog: every registered primitive with its signature, description, examples
- Component library: every `defcomp` with usage examples
- Query catalog: every `(query service name)` with parameter schemas and return types
- Interaction logs: successful s-expressions that produced good user outcomes
- Continuous learning: new primitives/components automatically extend the vocabulary

**Safety model**:
- S-expressions can only invoke registered primitives — no arbitrary code execution
- The resolver validates the tree before execution
- I/O primitives respect existing auth (HMAC, OAuth, user context)
- Rate limiting on LLM generation endpoint
- Content-addressed caching prevents regeneration of identical requests
- Generated s-expressions are logged as AP activities (provenance tracking)

**Deliverables**:
```
shared/sexp/
  llm.py              # LLM integration — prompt → s-expression generation
  catalog.py          # primitive/component catalog for LLM context
  validation.py       # validate generated s-expressions before execution

rose-ash/
  llm/                # LLM service (or integration with external LLM API)
    routes.py         # conversational endpoint
    training.py       # continuous learning from interaction data
    prompts/          # system prompts with primitive catalog
```

**Tasks**:
1. Build primitive catalog generator (introspect registry → structured docs)
2. Build component catalog generator (introspect defcomp registry → examples)
3. Build query catalog generator (introspect data endpoints → schemas)
4. Design system prompt that teaches LLM the s-expression grammar + primitives
5. Implement generation endpoint (natural language → s-expression)
6. Implement validation layer (parse + type-check generated expressions)
7. Implement conversational refinement (user feedback → modified s-expression)
8. Implement caching of generated s-expressions (prompt hash → CID)
9. Wire into AP for provenance (LLM-generated content attributed to LLM actor)
10. Implement feedback loop (interaction data → training signal)

**Verification**: User prompt → generated page → visual inspection + primitive coverage audit.

---

## Summary of Phases

| Phase | What | Depends On | Scope |
|-------|------|-----------|-------|
| 1 | S-expression core library | — | `shared/sexp/` |
| 2 | HTML renderer (HSX-style) | 1 | `shared/sexp/html.py` |
| 3 | Async resolver | 1, 2 | `shared/sexp/resolver.py` |
| 4 | Jinja bridge + fragment migration | 2, 3 | All services' fragment templates |
| 5 | Page layouts as s-expressions | 4 | All services' page templates |
| 6 | Routes as s-expressions | 5 | All services' route handlers |
| 7 | Content addressing + IPFS/IPNS | 3 | `shared/sexp/cache.py` |
| 8 | Media pipeline integration | 1, 7 | `render/` service |
| 9 | Unified DAG executor | 3, 8 | `shared/sexp/executor.py` |
| 10 | Federation of s-expressions | 7, 9 | AP + IPFS integration |
| 11 | LLM as s-expression compiler | 1-6, 9 | `shared/sexp/llm.py`, `llm/` service |

**Foundation** (Phases 1-3): The s-expression language, HTML rendering, and async resolver. Everything else builds on this.

**Migration** (Phases 4-6): Incremental replacement of Jinja templates and Python route handlers with s-expressions. The bridge ensures coexistence — the app never breaks during migration.

**Infrastructure** (Phases 7-9): Content addressing, IPFS/IPNS caching, media pipeline, and unified DAG execution. Pages and video renders become the same computation.

**Intelligence** (Phases 10-11): Federation makes s-expressions portable across instances. The LLM makes s-expressions accessible to non-programmers — natural language in, rendered pages out. The system learns from its own data, continuously improving the quality of generated s-expressions.

Each phase is independently deployable. The end state: a platform where the application logic is expressed in a small, composable, content-addressed language that humans author, LLMs generate, resolvers execute, IPFS stores, and ActivityPub federates.

---

## Progress Log

### Phase 1: S-Expression Core Library — COMPLETE

**Branch:** `sexpression`

**Delivered** (`shared/sexp/`):
- `types.py` — Symbol, Keyword, Lambda (callable closure), Component (defcomp), NIL singleton
- `parser.py` — Tokenizer + parse/parse_all/serialize. Supports lists, vectors, maps, symbols (~component, <>fragment), keywords, strings, numbers, comments, &key/&rest
- `env.py` — Lexical environment with parent-chain scoping
- `evaluator.py` — Full evaluator with special forms (if, when, cond, case, and, or, let/let*, lambda/fn, define, defcomp, begin/do, quote, ->, set!) and higher-order forms (map, map-indexed, filter, reduce, some, every?, for-each)
- `primitives.py` — 60+ pure builtins: arithmetic, comparison, predicates, strings (str, concat, upper, lower, join, split, starts-with?, ends-with?), collections (list, dict, get, first, last, rest, nth, cons, append, keys, vals, merge, assoc, dissoc, into, range, chunk-every, zip-pairs)
- `__init__.py` — Public API

**Tests** (`shared/sexp/tests/`):
- `test_parser.py` — 28 tests (atoms, lists, maps, vectors, comments, errors, serialization, roundtrip)
- `test_evaluator.py` — 81 tests (literals, arithmetic, comparison, predicates, special forms, lambda/closures, collections, higher-order, strings, defcomp, dict literals, set!)
- **109 tests, all passing**

**Source material ported from:** `artdag/core/artdag/sexp/parser.py` and `evaluator.py`. Stripped DAG-specific types (Binding), replaced Lambda dataclass with callable closure, added defcomp/Component, added web-oriented string primitives, added &key/&rest support in parser.

### Phase 2: HTML Renderer — COMPLETE

**Branch:** `sexpression`

**Delivered** (`shared/sexp/html.py`):
- HSX-style renderer: s-expression AST → HTML string
- ~100 HTML tags recognised (sections, headings, grouping, text, embedded, table, forms, interactive, template)
- 14 void elements (br, img, input, meta, link, etc.) — no closing tag
- 23 boolean attributes (disabled, checked, required, hidden, etc.)
- Text and attribute escaping (XSS prevention: &, <, >, ")
- `raw!` for trusted unescaped HTML
- `<>` fragment rendering (no wrapper element)
- Render-aware special forms: `if`, `when`, `cond`, `let`/`let*`, `begin`/`do`, `map`, `map-indexed`, `filter`, `for-each`, `define`, `defcomp` — these call `_render` on result branches so HTML tags inside control flow work correctly
- `_render_component()` — render-aware component calling (vs evaluator's `_call_component` which only evaluates)
- `_render_lambda_call()` — lambda bodies containing HTML tags are rendered directly
- `_RawHTML` marker type — pre-rendered children pass through without double-escaping
- Component children rendered to HTML string and wrapped as `_RawHTML` for safe embedding

**Key architectural decision:** The renderer maintains a parallel set of special form handlers (`_RENDER_FORMS`) that mirror the evaluator's special forms but call `_render` on results instead of `_eval`. This is necessary because the evaluator doesn't know about HTML tags — `_eval((p "Hello"))` fails with "Undefined symbol: p". The renderer intercepts these forms before they reach the evaluator.

**Dispatch order in `_render_list`:**
1. `raw!` → unescaped HTML
2. `<>` → fragment
3. `_RENDER_FORMS` (checked before HTML_TAGS because `map` is both a render form and an HTML tag)
4. `HTML_TAGS` → element rendering
5. `~prefix` → component rendering
6. Fallthrough → `_eval` then `_render`

**Tests** (`shared/sexp/tests/test_html.py`):
- 63 tests: escaping (4), atoms (8), elements (6), attributes (8), boolean attrs (4), void elements (7), fragments (3), raw! (3), components (4), expressions with control flow (8), full pages (3), edge cases (5)
- **172 total tests across all 3 files, all passing**

### Phase 3: Async Resolver — COMPLETE

**Branch:** `sexpression`

**Delivered** (`shared/sexp/`):
- `resolver.py` — Async tree walker: collects I/O nodes from parsed tree, executes them in parallel via `asyncio.gather()`, substitutes results back, renders to HTML. Multi-pass resolution (up to 5 depth) for cases where resolved values contain further I/O. Graceful degradation: failed I/O nodes substitute empty string instead of crashing.
- `primitives_io.py` — I/O primitive registry and handlers:
  - `(frag "service" "type" :key val ...)` → wraps `fetch_fragment`
  - `(query "service" "query-name" :key val ...)` → wraps `fetch_data`
  - `(action "service" "action-name" :key val ...)` → wraps `call_action`
  - `(current-user)` → user dict from `RequestContext`
  - `(htmx-request?)` → boolean from `RequestContext`
- `RequestContext` — per-request state (user, is_htmx, extras) passed to I/O handlers

**Resolution strategy:**
1. Parse s-expression tree
2. Walk tree, collect all I/O nodes (frag, query, action, current-user, htmx-request?)
3. Parse each node's positional args + keyword kwargs, evaluating expressions
4. Dispatch all I/O in parallel via `asyncio.gather(return_exceptions=True)`
5. Substitute results back into tree (fragments wrapped as `_RawHTML` to prevent escaping)
6. Repeat up to 5 passes if resolved values introduce new I/O nodes
7. Render fully-resolved tree to HTML via Phase 2 renderer

**Design decisions:**
- I/O handlers use deferred imports (inside functions) so `shared.sexp` doesn't depend on infrastructure at import time — only when actually executing I/O
- Tests mock at the `execute_io` boundary (patching `shared.sexp.resolver.execute_io`) rather than patching infrastructure imports, keeping tests self-contained with no external dependencies
- Fragment results wrapped as `_RawHTML` since they're already-rendered HTML
- Identity-based substitution (`id(expr)`) maps I/O nodes back to their tree position

**Tests** (`shared/sexp/tests/test_resolver.py`):
- 27 tests: passthrough rendering (4), I/O collection (8), fragment resolution (3), query resolution (2), parallel I/O (1), request context (4), error handling (2), mixed content (3)
- **199 total tests across all 4 files, all passing**

### Phase 4: Jinja Bridge — COMPLETE

**Branch:** `sexpression`

**Delivered** (`shared/sexp/`):
- `jinja_bridge.py` — Two-way bridge between Jinja and s-expressions:
  - `sexp(source, **kwargs)` — sync render for Jinja templates: `{{ sexp('(~card :title "Hi")') | safe }}`
  - `sexp_async(source, **kwargs)` — async render with I/O resolution
  - `register_components(sexp_source)` — load component definitions at startup
  - `get_component_env()` — access the shared component registry
  - `setup_sexp_bridge(app)` — register `sexp` and `sexp_async` as Jinja globals
  - `_get_request_context()` — auto-builds RequestContext from Quart request

**Integration point**: Call `setup_sexp_bridge(app)` after `setup_jinja(app)` in app factories. Components registered via `register_components()` are available globally across all templates.

**First migration target: `~link-card`** — unified component replacing 5 separate Jinja templates (`blog/fragments/link_card.html`, `market/fragments/link_card.html`, `events/fragments/link_card.html`, `federation/fragments/link_card.html`, `artdag/l1/app/templates/fragments/link_card.html`). The s-expression component handles image/no-image, brand, and is usable from both Jinja templates and s-expression trees.

**Tests** (`shared/sexp/tests/test_jinja_bridge.py`):
- 19 tests: sexp() rendering (5), component registration (6), link-card migration (5), sexp_async (3)
- **218 total tests across all 5 files, all passing**

### Test Infrastructure — COMPLETE

**Delivered:**
- `test/Dockerfile.unit` — Tier 1: all unit tests (shared + artdag core + L1), pure Python, fast
- `test/Dockerfile.integration` — Tier 2: integration tests needing ffmpeg/media pipeline
- `docker-compose.dev.yml` — `test-unit` (watch mode) and `test-integration` services, `profiles: [test]`
- `dev.sh` — `./dev.sh watch` (auto-rerun on save), `./dev.sh test` (one-shot), `./dev.sh test-integration`
- `deploy.sh` — Unit test gate: tests must pass before any images are pushed