Rebrand sexp → sx across web platform (173 files)

Rename all sexp directories, files, identifiers, and references to sx.
artdag/ excluded (separate media processing DSL).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.claude/plans/glittery-discovering-kahn.md

@@ -0,0 +1,177 @@
+# Sexp Fragment Protocol: Component Defs Between Services
+
+## Context
+
+Fragment endpoints return raw sexp source (e.g., `(~blog-nav-wrapper :items ...)`). The consuming service embeds this in its page sexp, which the client evaluates. But blog-specific components like `~blog-nav-wrapper` are only in blog's `_COMPONENT_ENV` — not in market's. So market's `client_components_tag()` never sends them to the client, causing "Unknown component" errors.
+
+The fix: transfer component definitions alongside fragments. Services tell the provider what they already have; the provider sends only what's missing. The consuming service registers received defs into its `_COMPONENT_ENV` so they're included in `client_components_tag()` output for the client.
+
+## Approach: Structured Sexp Request/Response
+
+Replace the current GET + `X-Fragment-Request` header protocol with POST + sexp body. This aligns with the vision in `docs/sexpr-internal-protocol-first.md`.
+
+### Request format (POST body)
+```scheme
+(fragment-request
+  :type "nav-tree"
+  :params (:app-name "market" :path "/")
+  :components (~blog-nav-wrapper ~blog-nav-item-link ~header-row-sx ...))
+```
+
+`:components` lists component names already in the consumer's `_COMPONENT_ENV`. Provider skips these.
+
+### Response format
+```scheme
+(fragment-response
+  :defs ((defcomp ~blog-nav-wrapper (&key ...) ...) (defcomp ~blog-nav-item-link ...))
+  :content (~blog-nav-wrapper :items ...))
+```
+
+`:defs` contains only components the consumer doesn't have. `:content` is the fragment sexp (same as current response body).
+
+## Changes
+
+### 1. `shared/infrastructure/fragments.py` — Client side
+
+**`fetch_fragment()`**: Switch from GET to POST with sexp body.
+
+- Build request body using `sexp_call`:
+  ```python
+  from shared.sexp.helpers import sexp_call, SexpExpr
+  from shared.sexp.jinja_bridge import _COMPONENT_ENV
+
+  comp_names = [k for k in _COMPONENT_ENV if k.startswith("~")]
+  body = sexp_call("fragment-request",
+      type=fragment_type,
+      params=params or {},
+      components=SexpExpr("(" + " ".join(comp_names) + ")"))
+  ```
+- POST to same URL, body as `text/sexp`, keep `X-Fragment-Request` header for backward compat
+- Parse response: extract `:defs` and `:content` from the sexp response
+- Register defs into `_COMPONENT_ENV` via `register_components()`
+- Return `:content` wrapped as `SexpExpr`
+
+**New helper `_parse_fragment_response(text)`**:
+- `parse()` the response sexp
+- Extract keyword args (reuse the keyword-extraction pattern from `evaluator.py`)
+- Return `(defs_source, content_source)` tuple
+
+### 2. `shared/sexp/helpers.py` — Response builder
+
+**New `fragment_response(content, request_text)`**:
+
+```python
+def fragment_response(content: str, request_text: str) -> str:
+    """Build a structured fragment response with missing component defs."""
+    from .parser import parse, serialize
+    from .types import Keyword, Component
+    from .jinja_bridge import _COMPONENT_ENV
+
+    # Parse request to get :components list
+    req = parse(request_text)
+    loaded = set()
+    # extract :components keyword value
+    ...
+
+    # Diff against _COMPONENT_ENV, serialize missing defs
+    defs_parts = []
+    for key, val in _COMPONENT_ENV.items():
+        if not isinstance(val, Component):
+            continue
+        if key in loaded or f"~{val.name}" in loaded:
+            continue
+        defs_parts.append(_serialize_defcomp(val))
+
+    defs_sexp = "(" + " ".join(defs_parts) + ")" if defs_parts else "nil"
+    return sexp_call("fragment-response",
+        defs=SexpExpr(defs_sexp),
+        content=SexpExpr(content))
+```
+
+### 3. Fragment endpoints — All services
+
+**Generic change in each `bp/fragments/routes.py`**: Update the route handler to accept POST, read sexp body, use `fragment_response()` for the response.
+
+The `get_fragment` handler becomes:
+```python
+@bp.route("/<fragment_type>", methods=["GET", "POST"])
+async def get_fragment(fragment_type: str):
+    handler = _handlers.get(fragment_type)
+    if handler is None:
+        return Response("", status=200, content_type="text/sexp")
+    content = await handler()
+
+    # Structured sexp protocol (POST with sexp body)
+    request_body = await request.get_data(as_text=True)
+    if request_body and request.content_type == "text/sexp":
+        from shared.sexp.helpers import fragment_response
+        body = fragment_response(content, request_body)
+        return Response(body, status=200, content_type="text/sexp")
+
+    # Legacy GET fallback
+    return Response(content, status=200, content_type="text/sexp")
+```
+
+Since all fragment endpoints follow the identical `_handlers` + `get_fragment` pattern, we can extract this into a shared helper in `fragments.py` or a new `shared/infrastructure/fragment_endpoint.py`.
+
+### 4. Extract shared fragment endpoint helper
+
+To avoid touching every service's fragment routes, create a shared blueprint factory:
+
+**`shared/infrastructure/fragment_endpoint.py`**:
+```python
+def create_fragment_blueprint(handlers: dict) -> Blueprint:
+    """Create a fragment endpoint blueprint with sexp protocol support."""
+    bp = Blueprint("fragments", __name__, url_prefix="/internal/fragments")
+
+    @bp.before_request
+    async def _require_fragment_header():
+        if not request.headers.get(FRAGMENT_HEADER):
+            return Response("", status=403)
+
+    @bp.route("/<fragment_type>", methods=["GET", "POST"])
+    async def get_fragment(fragment_type: str):
+        handler = handlers.get(fragment_type)
+        if handler is None:
+            return Response("", status=200, content_type="text/sexp")
+        content = await handler()
+
+        # Sexp protocol: POST with structured request/response
+        if request.method == "POST" and request.content_type == "text/sexp":
+            request_body = await request.get_data(as_text=True)
+            from shared.sexp.helpers import fragment_response
+            body = fragment_response(content, request_body)
+            return Response(body, status=200, content_type="text/sexp")
+
+        return Response(content, status=200, content_type="text/sexp")
+
+    return bp
+```
+
+Then each service's `register()` just returns `create_fragment_blueprint(_handlers)`. This is a small refactor since they all duplicate the same boilerplate today.
+
+## Files to modify
+
+| File | Change |
+|------|--------|
+| `shared/infrastructure/fragments.py` | POST sexp body, parse response, register defs |
+| `shared/sexp/helpers.py` | `fragment_response()` builder, `_serialize_defcomp()` |
+| `shared/infrastructure/fragment_endpoint.py` | **New** — shared blueprint factory |
+| `blog/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
+| `market/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
+| `events/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
+| `cart/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
+| `account/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
+| `orders/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
+| `federation/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
+| `relations/bp/fragments/routes.py` | Use `create_fragment_blueprint` |
+
+## Verification
+
+1. Start blog + market services: `./dev.sh blog market`
+2. Load market page — should fetch nav-tree from blog with sexp protocol
+3. Check market logs: no "Unknown component" errors
+4. Inspect page source: `client_components_tag()` output includes `~blog-nav-wrapper` etc.
+5. Cross-domain sx-get navigation (blog → market) works without reload
+6. Run sexp tests: `python3 -m pytest shared/sexp/tests/ -x -q`
+7. Second page load: `:components` list in request includes blog nav components, response `:defs` is empty

.claude/plans/hazy-sniffing-sphinx.md

@@ -0,0 +1,325 @@
+# Split Cart into Microservices
+
+## Context
+The cart app currently owns too much: CartItem, Order/OrderItem, PageConfig, ContainerRelation, plus all checkout/payment logic. We're splitting it into 4 pieces:
+
+1. **Relations service** — internal only, owns ContainerRelation
+2. **Likes service** — internal only, unified generic likes replacing ProductLike + PostLike
+3. **PageConfig → blog** — move to blog (which already owns pages)
+4. **Orders service** — public (orders.rose-ash.com), owns Order/OrderItem + SumUp checkout
+
+After the split, cart becomes a thin CartItem CRUD + inbox service.
+
+---
+
+## Phase 1: Relations Service (internal only)
+
+### 1.1 Scaffold `relations/`
+Create minimal internal-only app (no templates, no context_fn):
+
+| File | Notes |
+|------|-------|
+| `relations/__init__.py` | Empty |
+| `relations/path_setup.py` | Copy from cart |
+| `relations/app.py` | `create_base_app("relations")`, register data + actions BPs only |
+| `relations/services/__init__.py` | Empty `register_domain_services()` |
+| `relations/models/__init__.py` | `from shared.models.container_relation import ContainerRelation` |
+| `relations/bp/__init__.py` | Export `register_data`, `register_actions` |
+| `relations/bp/data/routes.py` | Move `get-children` handler from `cart/bp/data/routes.py:175-198` |
+| `relations/bp/actions/routes.py` | Move `attach-child` + `detach-child` from `cart/bp/actions/routes.py:112-153` |
+| `relations/alembic.ini` | Copy from cart, adjust path |
+| `relations/alembic/env.py` | MODELS=`["shared.models.container_relation"]`, TABLES=`{"container_relations"}` |
+| `relations/alembic/versions/0001_initial.py` | Create `container_relations` table |
+| `relations/Dockerfile` | Follow cart pattern, `COPY relations/ ./` |
+| `relations/entrypoint.sh` | Standard pattern, db=`db_relations` |
+
+### 1.2 Retarget callers (`"cart"` → `"relations"`)
+
+| File | Lines | Change |
+|------|-------|--------|
+| `events/bp/calendars/services/calendars.py` | 74, 111, 121 | `call_action("cart", ...)` → `call_action("relations", ...)` |
+| `blog/bp/menu_items/services/menu_items.py` | 83, 137, 141, 157 | Same |
+| `shared/services/market_impl.py` | 96, 109, 133 | Same |
+
+### 1.3 Clean up cart
+- Remove `get-children` from `cart/bp/data/routes.py:175-198`
+- Remove `attach-child`, `detach-child` from `cart/bp/actions/routes.py:112-153`
+- Remove `"shared.models.container_relation"` and `"container_relations"` from `cart/alembic/env.py`
+
+---
+
+## Phase 2: Likes Service (internal only)
+
+### 2.1 New unified model
+Single `likes` table in `db_likes`:
+
+```python
+class Like(Base):
+    __tablename__ = "likes"
+    id: Mapped[int] (pk)
+    user_id: Mapped[int] (not null, indexed)
+    target_type: Mapped[str] (String 32, not null)  # "product" or "post"
+    target_slug: Mapped[str | None] (String 255)     # for products
+    target_id: Mapped[int | None] (Integer)           # for posts
+    created_at, updated_at, deleted_at
+
+    UniqueConstraint("user_id", "target_type", "target_slug")
+    UniqueConstraint("user_id", "target_type", "target_id")
+    Index("ix_likes_target", "target_type", "target_slug")
+```
+
+Products use `target_type="product"`, `target_slug=slug`. Posts use `target_type="post"`, `target_id=post.id`.
+
+### 2.2 Scaffold `likes/`
+
+| File | Notes |
+|------|-------|
+| `likes/__init__.py` | Empty |
+| `likes/path_setup.py` | Standard |
+| `likes/app.py` | Internal-only, `create_base_app("likes")`, data + actions BPs |
+| `likes/services/__init__.py` | Empty `register_domain_services()` |
+| `likes/models/__init__.py` | Import Like |
+| `likes/models/like.py` | Generic Like model (above) |
+| `likes/bp/__init__.py` | Export register functions |
+| `likes/bp/data/routes.py` | `is-liked`, `liked-slugs`, `liked-ids` |
+| `likes/bp/actions/routes.py` | `toggle` action |
+| `likes/alembic.ini` | Standard |
+| `likes/alembic/env.py` | MODELS=`["likes.models.like"]`, TABLES=`{"likes"}` |
+| `likes/alembic/versions/0001_initial.py` | Create `likes` table |
+| `likes/Dockerfile` | Standard pattern |
+| `likes/entrypoint.sh` | Standard, db=`db_likes` |
+
+### 2.3 Data endpoints (`likes/bp/data/routes.py`)
+- `is-liked`: params `user_id, target_type, target_slug/target_id` → `{"liked": bool}`
+- `liked-slugs`: params `user_id, target_type` → `["slug1", "slug2"]`
+- `liked-ids`: params `user_id, target_type` → `[1, 2, 3]`
+
+### 2.4 Action endpoints (`likes/bp/actions/routes.py`)
+- `toggle`: payload `{user_id, target_type, target_slug?, target_id?}` → `{"liked": bool}`
+
+### 2.5 Retarget market app
+
+**`market/bp/product/routes.py`** (like_toggle, ~line 119):
+Replace `toggle_product_like(g.s, user_id, product_slug)` with:
+```python
+result = await call_action("likes", "toggle", payload={
+    "user_id": user_id, "target_type": "product", "target_slug": product_slug
+})
+liked = result["liked"]
+```
+
+**`market/bp/browse/services/db_backend.py`** (most complex):
+- `db_product_full` / `db_product_full_id`: Replace `ProductLike` subquery with `fetch_data("likes", "is-liked", ...)`. Annotate `is_liked` after query.
+- `db_products_nocounts` / `db_products_counts`: Fetch `liked_slugs` once via `fetch_data("likes", "liked-slugs", ...)`, filter `Product.slug.in_(liked_slugs)` for `?liked=true`, annotate `is_liked` post-query.
+
+**Delete**: `toggle_product_like` from `market/bp/product/services/product_operations.py`
+
+### 2.6 Retarget blog app
+
+**`blog/bp/post/routes.py`** (like_toggle):
+Replace `toggle_post_like(g.s, user_id, post_id)` with `call_action("likes", "toggle", payload={...})`.
+
+**Delete**: `toggle_post_like` from `blog/bp/post/services/post_operations.py`
+
+### 2.7 Remove old like models
+- Remove `ProductLike` from `shared/models/market.py` (lines 118-131) + `Product.likes` relationship (lines 110-114)
+- Remove `PostLike` from `shared/models/ghost_content.py` + `Post.likes` relationship
+- Remove `product_likes` from market alembic TABLES
+- Remove `post_likes` from blog alembic TABLES
+
+---
+
+## Phase 3: PageConfig → Blog
+
+### 3.1 Replace blog proxy endpoints with direct DB queries
+
+**`blog/bp/data/routes.py`** (lines 77-102): Replace the 3 proxy handlers that currently call `fetch_data("cart", ...)` with direct DB queries. Copy logic from `cart/bp/data/routes.py`:
+- `page-config` (cart lines 114-134)
+- `page-config-by-id` (cart lines 136-149)
+- `page-configs-batch` (cart lines 151-172)
+- `page-config-ensure` (cart lines 49-81) — add new
+
+Also add the `_page_config_dict` helper (cart lines 203-213).
+
+### 3.2 Move action to blog
+
+**`blog/bp/actions/routes.py`** (~line 40): Replace `call_action("cart", "update-page-config", ...)` proxy with direct handler. Copy logic from `cart/bp/actions/routes.py:51-110`.
+
+### 3.3 Blog callers become local
+
+| File | Current | After |
+|------|---------|-------|
+| `blog/bp/post/admin/routes.py:34` | `fetch_data("cart", "page-config", ...)` | Direct DB query (blog now owns table) |
+| `blog/bp/post/admin/routes.py:87,132` | `call_action("cart", "update-page-config", ...)` | Direct call to local handler |
+| `blog/bp/post/services/markets.py:44` | `fetch_data("cart", "page-config", ...)` | Direct DB query |
+| `blog/bp/blog/ghost_db.py:295` | `fetch_data("cart", "page-configs-batch", ...)` | Direct DB query |
+
+### 3.4 Retarget cross-service callers (`"cart"` → `"blog"`)
+
+| File | Change |
+|------|--------|
+| `cart/bp/cart/services/page_cart.py:181` | `fetch_data("cart", "page-configs-batch", ...)` → `fetch_data("blog", "page-configs-batch", ...)` |
+| `cart/bp/cart/global_routes.py:274` | `fetch_data("cart", "page-config-by-id", ...)` → `fetch_data("blog", "page-config-by-id", ...)` |
+
+(Note: `checkout.py:117` and `cart/app.py:177` already target `"blog"`)
+
+### 3.5 Update blog alembic
+**`blog/alembic/env.py`**: Add `"shared.models.page_config"` to MODELS and `"page_configs"` to TABLES.
+
+### 3.6 Clean up cart
+- Remove all `page-config*` handlers from `cart/bp/data/routes.py` (lines 49-172)
+- Remove `update-page-config` from `cart/bp/actions/routes.py` (lines 50-110)
+- Remove `"shared.models.page_config"` and `"page_configs"` from `cart/alembic/env.py`
+
+---
+
+## Phase 4: Orders Service (public, orders.rose-ash.com)
+
+### 4.1 Scaffold `orders/`
+
+| File | Notes |
+|------|-------|
+| `orders/__init__.py` | Empty |
+| `orders/path_setup.py` | Standard |
+| `orders/app.py` | Public app with `context_fn`, templates, fragments, page slug hydration |
+| `orders/services/__init__.py` | `register_domain_services()` |
+| `orders/models/__init__.py` | `from shared.models.order import Order, OrderItem` |
+| `orders/bp/__init__.py` | Export all BPs |
+| `orders/bp/order/` | Move from `cart/bp/order/` (single order: detail, pay, recheck) |
+| `orders/bp/orders/` | Move from `cart/bp/orders/` (order list + pagination) |
+| `orders/bp/checkout/routes.py` | Webhook + return routes from `cart/bp/cart/global_routes.py` |
+| `orders/bp/data/routes.py` | Minimal |
+| `orders/bp/actions/routes.py` | `create-order` action (called by cart during checkout) |
+| `orders/bp/fragments/routes.py` | `account-nav-item` fragment (orders link) |
+| `orders/templates/` | Move `_types/order/`, `_types/orders/`, checkout templates from cart |
+| `orders/alembic.ini` | Standard |
+| `orders/alembic/env.py` | MODELS=`["shared.models.order"]`, TABLES=`{"orders", "order_items"}` |
+| `orders/alembic/versions/0001_initial.py` | Create `orders` + `order_items` tables |
+| `orders/Dockerfile` | Standard, public-facing |
+| `orders/entrypoint.sh` | Standard, db=`db_orders` |
+
+### 4.2 Move checkout services to orders
+
+**Move to `orders/services/`:**
+- `checkout.py` — from `cart/bp/cart/services/checkout.py` (move: `create_order_from_cart`, `resolve_page_config`, `build_sumup_*`, `get_order_with_details`. Keep `find_or_create_cart_item` in cart.)
+- `check_sumup_status.py` — from `cart/bp/cart/services/check_sumup_status.py`
+
+**`clear_cart_for_order`** stays in cart as new action:
+- Add `clear-cart-for-order` to `cart/bp/actions/routes.py`
+- Orders calls `call_action("cart", "clear-cart-for-order", payload={user_id, session_id, page_post_id})`
+
+### 4.3 `create-order` action endpoint (`orders/bp/actions/routes.py`)
+Cart's `POST /checkout/` calls this:
+```
+Payload: {cart_items: [{product_id, product_title, product_slug, product_image,
+           product_special_price, product_regular_price, product_price_currency,
+           quantity, market_place_container_id}],
+          calendar_entries, tickets, user_id, session_id,
+          product_total, calendar_total, ticket_total,
+          page_post_id, redirect_url, webhook_base_url}
+Returns: {order_id, sumup_hosted_url, page_config_id, sumup_reference, description}
+```
+
+### 4.4 Refactor cart's checkout route
+`cart/bp/cart/global_routes.py` `POST /checkout/`:
+1. Load local cart data (get_cart, calendar entries, tickets, totals)
+2. Serialize cart items to dicts
+3. `result = await call_action("orders", "create-order", payload={...})`
+4. Redirect to `result["sumup_hosted_url"]`
+
+Same for page-scoped checkout in `cart/bp/cart/page_routes.py`.
+
+### 4.5 Move webhook + return routes to orders
+- `POST /checkout/webhook/<order_id>/` → `orders/bp/checkout/routes.py`
+- `GET /checkout/return/<order_id>/` → `orders/bp/checkout/routes.py`
+- SumUp redirect/webhook URLs must now point to orders.rose-ash.com
+
+### 4.6 Move order list/detail routes
+- `cart/bp/order/` → `orders/bp/order/`
+- `cart/bp/orders/` → `orders/bp/orders/`
+
+### 4.7 Move startup reconciliation
+`_reconcile_pending_orders` from `cart/app.py:209-265` → `orders/app.py`
+
+### 4.8 Clean up cart
+- Remove `cart/bp/order/`, `cart/bp/orders/`
+- Remove checkout webhook/return from `cart/bp/cart/global_routes.py`
+- Remove `_reconcile_pending_orders` from `cart/app.py`
+- Remove order templates from `cart/templates/`
+- Remove `"shared.models.order"` and `"orders", "order_items"` from `cart/alembic/env.py`
+
+---
+
+## Phase 5: Infrastructure (applies to all new services)
+
+### 5.1 docker-compose.yml
+Add 3 new services (relations, likes, orders) with own DATABASE_URL (db_relations, db_likes, db_orders), own REDIS_URL (Redis DB 7, 8, 9).
+
+Add to `x-app-env`:
+```yaml
+INTERNAL_URL_RELATIONS: http://relations:8000
+INTERNAL_URL_LIKES: http://likes:8000
+INTERNAL_URL_ORDERS: http://orders:8000
+APP_URL_ORDERS: https://orders.rose-ash.com
+```
+
+### 5.2 docker-compose.dev.yml
+Add all 3 services with dev volumes (ports 8008, 8009, 8010).
+Add to `x-sibling-models` for all 3 new services.
+
+### 5.3 deploy.sh
+Add `relations likes orders` to APPS list.
+
+### 5.4 Caddyfile (`/root/caddy/Caddyfile`)
+Add only orders (public):
+```
+orders.rose-ash.com { reverse_proxy rose-ash-dev-orders-1:8000 }
+```
+
+### 5.5 shared/infrastructure/factory.py
+Add to model import loop: `"relations.models", "likes.models", "orders.models"`
+
+### 5.6 shared/infrastructure/urls.py
+Add `orders_url(path)` helper.
+
+### 5.7 All existing Dockerfiles
+Add sibling model COPY lines for the 3 new services to every existing Dockerfile (blog, market, cart, events, federation, account).
+
+### 5.8 CLAUDE.md
+Update project structure and add notes about the new services.
+
+---
+
+## Data Migration (one-time, run before code switch)
+
+1. `container_relations` from `db_cart` → `db_relations`
+2. `product_likes` from `db_market` + `post_likes` from `db_blog` → `db_likes.likes`
+3. `page_configs` from `db_cart` → `db_blog`
+4. `orders` + `order_items` from `db_cart` → `db_orders`
+
+Use `pg_dump`/`pg_restore` or direct SQL for migration.
+
+---
+
+## Post-Split Cart State
+
+After all 4 phases, cart owns only:
+- **Model**: CartItem (table in db_cart)
+- **Alembic**: `cart_items` only
+- **Data endpoints**: `cart-summary`, `cart-items`
+- **Action endpoints**: `adopt-cart-for-user`, `clear-cart-for-order` (new)
+- **Inbox handlers**: Add/Remove/Update `rose:CartItem`
+- **Public routes**: cart overview, page cart, add-to-cart, quantity, delete
+- **Fragments**: `cart-mini`
+- **Checkout**: POST /checkout/ (creates order via `call_action("orders", "create-order")`, redirects to SumUp)
+
+---
+
+## Verification
+1. **Relations**: Blog attach/detach marketplace to page; events attach/detach calendar
+2. **Likes**: Toggle product like on market page; toggle post like on blog; `?liked=true` filter
+3. **PageConfig**: Blog admin page config update; cart checkout resolves page config from blog
+4. **Orders**: Add to cart → checkout → SumUp redirect → webhook → order paid; order list/detail on orders.rose-ash.com
+5. No remaining `call_action("cart", "attach-child|detach-child|update-page-config")`
+6. No remaining `fetch_data("cart", "page-config*|get-children")`
+7. Cart alembic only manages `cart_items` table

.gitea/workflows/ci.yml

@@ -2,7 +2,7 @@ name: Build and Deploy
 
 on:
   push:
-    branches: [main, decoupling]
+    branches: ['**']
 
 env:
   REGISTRY: registry.rose-ash.com:5000
@@ -58,7 +58,7 @@ jobs:
               fi
             fi
 
-            for app in blog market cart events federation account; do
+            for app in blog market cart events federation account relations likes orders test; do
               IMAGE_EXISTS=\$(docker image ls -q ${{ env.REGISTRY }}/\$app:latest 2>/dev/null)
               if [ \"\$REBUILD_ALL\" = true ] || echo \"\$CHANGED\" | grep -q \"^\$app/\" || [ -z \"\$IMAGE_EXISTS\" ]; then
                 echo \"Building \$app...\"
@@ -75,13 +75,18 @@ jobs:
               fi
             done
 
-            source .env
-            docker stack deploy -c docker-compose.yml rose-ash
-            echo 'Waiting for swarm services to update...'
-            sleep 10
-            docker stack services rose-ash
+            # Deploy swarm stack only on main branch
+            if [ '${{ github.ref_name }}' = 'main' ]; then
+              source .env
+              docker stack deploy -c docker-compose.yml rose-ash
+              echo 'Waiting for swarm services to update...'
+              sleep 10
+              docker stack services rose-ash
+            else
+              echo 'Skipping swarm deploy (branch: ${{ github.ref_name }})'
+            fi
 
-            # Deploy dev stack (bind-mounted source + auto-reload)
+            # Dev stack always deployed (bind-mounted source + auto-reload)
             echo 'Deploying dev stack...'
             docker compose -p rose-ash-dev -f docker-compose.yml -f docker-compose.dev.yml up -d
             echo 'Dev stack deployed'

CLAUDE.md

@@ -1,6 +1,6 @@
-# Art DAG Monorepo
+# Rose Ash Monorepo
 
-Federated content-addressed DAG execution engine for distributed media processing with ActivityPub ownership and provenance tracking.
+Cooperative web platform: federated content, commerce, events, and media processing. Each domain runs as an independent Quart microservice with its own database, communicating via HMAC-signed internal HTTP and ActivityPub events.
 
 ## Deployment
 
@@ -9,71 +9,134 @@ Federated content-addressed DAG execution engine for distributed media processin
 ## Project Structure
 
 ```
-core/       # DAG engine (artdag package) - nodes, effects, analysis, planning
-l1/         # L1 Celery rendering server (FastAPI + Celery + Redis + PostgreSQL)
-l2/         # L2 ActivityPub registry (FastAPI + PostgreSQL)
-common/     # Shared templates, middleware, models (artdag_common package)
-client/     # CLI client
-test/       # Integration & e2e tests
+blog/           # Content management, Ghost CMS sync, navigation, WYSIWYG editor
+market/         # Product catalog, marketplace pages, web scraping
+cart/           # Shopping cart CRUD, checkout (delegates order creation to orders)
+events/         # Calendar & event management, ticketing
+federation/     # ActivityPub social hub, user profiles
+account/        # OAuth2 authorization server, user dashboard, membership
+orders/         # Order history, SumUp payment/webhook handling, reconciliation
+relations/      # (internal) Cross-domain parent/child relationship tracking
+likes/          # (internal) Unified like/favourite tracking across domains
+shared/         # Shared library: models, infrastructure, templates, static assets
+artdag/         # Art DAG — media processing engine (separate codebase, see below)
+```
+
+### Shared Library (`shared/`)
+
+```
+shared/
+  models/          # Canonical SQLAlchemy ORM models for all domains
+  db/              # Async session management, per-domain DB support, alembic helpers
+  infrastructure/  # App factory, OAuth, ActivityPub, fragments, internal auth, Jinja
+  services/        # Domain service implementations + DI registry
+  contracts/       # DTOs and service protocols
+  browser/         # Middleware, Redis caching, CSRF, error handlers
+  events/          # Activity bus + background processor (AP-shaped events)
+  config/          # YAML config loading (frozen/readonly)
+  static/          # Shared CSS, JS, images
+  templates/       # Base HTML layouts, partials (inherited by all apps)
+```
+
+### Art DAG (`artdag/`)
+
+Federated content-addressed DAG execution engine for distributed media processing.
+
+```
+artdag/
+  core/      # DAG engine (artdag package) — nodes, effects, analysis, planning
+  l1/        # L1 Celery rendering server (FastAPI + Celery + Redis + PostgreSQL)
+  l2/        # L2 ActivityPub registry (FastAPI + PostgreSQL)
+  common/    # Shared templates, middleware, models (artdag_common package)
+  client/    # CLI client
+  test/      # Integration & e2e tests
 ```
 
 ## Tech Stack
 
-Python 3.11+, FastAPI, Celery, Redis, PostgreSQL (asyncpg for L1), SQLAlchemy, Pydantic, JAX (CPU/GPU), IPFS/Kubo, Docker Swarm, HTMX + Jinja2 for web UI.
+**Web platform:** Python 3.11+, Quart (async Flask), SQLAlchemy (asyncpg), Jinja2, HTMX, PostgreSQL, Redis, Docker Swarm, Hypercorn.
+
+**Art DAG:** FastAPI, Celery, JAX (CPU/GPU), IPFS/Kubo, Pydantic.
 
 ## Key Commands
 
-### Testing
+### Development
 ```bash
-cd l1 && pytest tests/                    # L1 unit tests
-cd core && pytest tests/                  # Core unit tests
-cd test && python run.py                  # Full integration pipeline
+./dev.sh                        # Start all services + infra (db, redis, pgbouncer)
+./dev.sh blog market            # Start specific services + infra
+./dev.sh --build blog           # Rebuild image then start
+./dev.sh down                   # Stop everything
+./dev.sh logs blog              # Tail service logs
 ```
-- pytest uses `asyncio_mode = "auto"` for async tests
-- Test files: `test_*.py`, fixtures in `conftest.py`
 
-### Linting & Type Checking (L1)
+### Deployment
 ```bash
-cd l1 && ruff check .                     # Lint (E, F, I, UP rules)
-cd l1 && mypy app/types.py app/routers/recipes.py tests/
+./deploy.sh                     # Auto-detect changed apps, build + push + restart
+./deploy.sh blog market         # Deploy specific apps
+./deploy.sh --all               # Deploy everything
 ```
-- Line length: 100 chars (E501 ignored)
-- Mypy: strict on `app/types.py`, `app/routers/recipes.py`, `tests/`; gradual elsewhere
-- Mypy ignores imports for: celery, redis, artdag, artdag_common, ipfs_client
 
-### Docker
+### Art DAG
 ```bash
-docker build -f l1/Dockerfile -t celery-l1-server:latest .
-docker build -f l1/Dockerfile.gpu -t celery-l1-gpu:latest .
-docker build -f l2/Dockerfile -t l2-server:latest .
-./deploy.sh                               # Build, push, deploy stacks
+cd artdag/l1 && pytest tests/              # L1 unit tests
+cd artdag/core && pytest tests/            # Core unit tests
+cd artdag/test && python run.py            # Full integration pipeline
+cd artdag/l1 && ruff check .               # Lint
+cd artdag/l1 && mypy app/types.py app/routers/recipes.py tests/
 ```
 
 ## Architecture Patterns
 
-- **3-Phase Execution**: Analyze -> Plan -> Execute (tasks in `l1/tasks/`)
-- **Content-Addressed**: All data identified by SHA3-256 hashes or IPFS CIDs
-- **Services Pattern**: Business logic in `app/services/`, API endpoints in `app/routers/`
-- **Types Module**: Pydantic models and TypedDicts in `app/types.py`
-- **Celery Tasks**: In `l1/tasks/`, decorated with `@app.task`
-- **S-Expression Effects**: Composable effect language in `l1/sexp_effects/`
-- **Storage**: Local filesystem, S3, or IPFS backends (`storage_providers.py`)
-- **Inter-Service Reads**: `fetch_data()` → GET `/internal/data/{query}` (HMAC-signed)
-- **Inter-Service Actions**: `call_action()` → POST `/internal/actions/{name}` (HMAC-signed)
-- **Inter-Service AP Inbox**: `send_internal_activity()` → POST `/internal/inbox` (HMAC-signed, AP-shaped activities for cross-service writes with denormalized data)
+### Web Platform
 
-## Auth
+- **App factory:** `create_base_app(name, context_fn, before_request_fns, domain_services_fn)` in `shared/infrastructure/factory.py` — creates Quart app with DB, Redis, CSRF, OAuth, AP, session management
+- **Blueprint pattern:** Each blueprint exposes `register() -> Blueprint`, handlers stored in `_handlers` dict
+- **Per-service database:** Each service has own PostgreSQL DB via PgBouncer; cross-domain data fetched via HTTP
+- **Alembic per-service:** Each service declares `MODELS` and `TABLES` in `alembic/env.py`, delegates to `shared.db.alembic_env.run_alembic()`
+- **Inter-service reads:** `fetch_data(service, query, params)` → GET `/internal/data/{query}` (HMAC-signed, 3s timeout)
+- **Inter-service writes:** `call_action(service, action, payload)` → POST `/internal/actions/{action}` (HMAC-signed, 5s timeout)
+- **Inter-service AP inbox:** `send_internal_activity()` → POST `/internal/inbox` (HMAC-signed, AP-shaped activities for cross-service writes)
+- **Fragments:** HTML fragments fetched cross-service via `fetch_fragments()` for composing shared UI (nav, cart mini, auth menu)
+- **Soft deletes:** Models use `deleted_at` column pattern
+- **Context processors:** Each app provides its own `context_fn` that assembles template context from local DB + cross-service fragments
 
-- L1 <-> L2: scoped JWT tokens (no shared secrets)
-- L2: password + OAuth SSO, token revocation in Redis (30-day expiry)
-- Federation: ActivityPub RSA signatures (`core/artdag/activitypub/`)
+### Auth
+
+- **Account** is the OAuth2 authorization server; all other apps are OAuth clients
+- Per-app first-party session cookies (Safari ITP compatible), synchronized via device ID
+- Grant verification: apps check grant validity against account DB (cached in Redis)
+- Silent SSO: `prompt=none` OAuth flow for automatic cross-app login
+- ActivityPub: RSA signatures, per-app virtual actor projections sharing same keypair
+
+### Art DAG
+
+- **3-Phase Execution:** Analyze → Plan → Execute (tasks in `artdag/l1/tasks/`)
+- **Content-Addressed:** All data identified by SHA3-256 hashes or IPFS CIDs
+- **S-Expression Effects:** Composable effect language in `artdag/l1/sexp_effects/`
+- **Storage:** Local filesystem, S3, or IPFS backends
+- L1 ↔ L2: scoped JWT tokens; L2: password + OAuth SSO
+
+## Domains
+
+| Service | Public URL | Dev Port |
+|---------|-----------|----------|
+| blog | blog.rose-ash.com | 8001 |
+| market | market.rose-ash.com | 8002 |
+| cart | cart.rose-ash.com | 8003 |
+| events | events.rose-ash.com | 8004 |
+| federation | federation.rose-ash.com | 8005 |
+| account | account.rose-ash.com | 8006 |
+| relations | (internal only) | 8008 |
+| likes | (internal only) | 8009 |
+| orders | orders.rose-ash.com | 8010 |
 
 ## Key Config Files
 
-- `l1/pyproject.toml` - mypy, pytest, ruff config for L1
-- `l1/celery_app.py` - Celery initialization
-- `l1/database.py` / `l2/db.py` - SQLAlchemy models
-- `l1/docker-compose.yml` / `l2/docker-compose.yml` - Swarm stacks
+- `docker-compose.yml` / `docker-compose.dev.yml` — service definitions, env vars, volumes
+- `deploy.sh` / `dev.sh` — deployment and development scripts
+- `shared/infrastructure/factory.py` — app factory (all services use this)
+- `{service}/alembic/env.py` — per-service migration config
+- `_config/app-config.yaml` — runtime YAML config (mounted into containers)
 
 ## Tools
 

account/Dockerfile

@@ -38,6 +38,12 @@ COPY events/__init__.py ./events/__init__.py
 COPY events/models/ ./events/models/
 COPY federation/__init__.py ./federation/__init__.py
 COPY federation/models/ ./federation/models/
+COPY relations/__init__.py ./relations/__init__.py
+COPY relations/models/ ./relations/models/
+COPY likes/__init__.py ./likes/__init__.py
+COPY likes/models/ ./likes/models/
+COPY orders/__init__.py ./orders/__init__.py
+COPY orders/models/ ./orders/models/
 
 # ---------- Runtime setup ----------
 COPY account/entrypoint.sh /usr/local/bin/entrypoint.sh

account/app.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 import path_setup  # noqa: F401  # adds shared/ to sys.path
+import sx.sx_components as sx_components  # noqa: F401  # ensure Hypercorn --reload watches this file
 from pathlib import Path
 
 from quart import g, request
@@ -43,14 +44,14 @@ async def account_context() -> dict:
     if ident["session_id"] is not None:
         cart_params["session_id"] = ident["session_id"]
 
-    cart_mini_html, auth_menu_html, nav_tree_html = await fetch_fragments([
+    cart_mini, auth_menu, nav_tree = await fetch_fragments([
         ("cart", "cart-mini", cart_params or None),
         ("account", "auth-menu", {"email": user.email} if user else None),
         ("blog", "nav-tree", {"app_name": "account", "path": request.path}),
     ])
-    ctx["cart_mini_html"] = cart_mini_html
-    ctx["auth_menu_html"] = auth_menu_html
-    ctx["nav_tree_html"] = nav_tree_html
+    ctx["cart_mini"] = cart_mini
+    ctx["auth_menu"] = auth_menu
+    ctx["nav_tree"] = nav_tree
 
     return ctx
 

account/bp/account/routes.py

@@ -8,7 +8,6 @@ from __future__ import annotations
 from quart import (
     Blueprint,
     request,
-    render_template,
     make_response,
     redirect,
     g,
@@ -19,6 +18,7 @@ from shared.models import UserNewsletter
 from shared.models.ghost_membership_entities import GhostNewsletter
 from shared.infrastructure.urls import login_url
 from shared.infrastructure.fragments import fetch_fragment, fetch_fragments
+from shared.sx.helpers import sx_response
 
 oob = {
     "oob_extends": "oob_elements.html",
@@ -42,21 +42,24 @@ def register(url_prefix="/"):
             ("cart", "account-nav-item", {}),
             ("artdag", "nav-item", {}),
         ], required=False)
-        return {"oob": oob, "account_nav_html": events_nav + cart_nav + artdag_nav}
+        return {"oob": oob, "account_nav": events_nav + cart_nav + artdag_nav}
 
     @account_bp.get("/")
     async def account():
         from shared.browser.app.utils.htmx import is_htmx_request
+        from shared.sx.page import get_template_context
+        from sx.sx_components import render_account_page, render_account_oob
 
         if not g.get("user"):
             return redirect(login_url("/"))
 
+        ctx = await get_template_context()
         if not is_htmx_request():
-            html = await render_template("_types/auth/index.html")
+            html = await render_account_page(ctx)
+            return await make_response(html)
         else:
-            html = await render_template("_types/auth/_oob_elements.html")
-
-        return await make_response(html)
+            sx_src = await render_account_oob(ctx)
+            return sx_response(sx_src)
 
     @account_bp.get("/newsletters/")
     async def newsletters():
@@ -86,22 +89,16 @@ def register(url_prefix="/"):
                 "subscribed": un.subscribed if un else False,
             })
 
-        nl_oob = {**oob, "main": "_types/auth/_newsletters_panel.html"}
+        from shared.sx.page import get_template_context
+        from sx.sx_components import render_newsletters_page, render_newsletters_oob
 
+        ctx = await get_template_context()
         if not is_htmx_request():
-            html = await render_template(
-                "_types/auth/index.html",
-                oob=nl_oob,
-                newsletter_list=newsletter_list,
-            )
+            html = await render_newsletters_page(ctx, newsletter_list)
+            return await make_response(html)
         else:
-            html = await render_template(
-                "_types/auth/_oob_elements.html",
-                oob=nl_oob,
-                newsletter_list=newsletter_list,
-            )
-
-        return await make_response(html)
+            sx_src = await render_newsletters_oob(ctx, newsletter_list)
+            return sx_response(sx_src)
 
     @account_bp.post("/newsletter/<int:newsletter_id>/toggle/")
     async def toggle_newsletter(newsletter_id: int):
@@ -128,10 +125,8 @@ def register(url_prefix="/"):
 
         await g.s.flush()
 
-        return await render_template(
-            "_types/auth/_newsletter_toggle.html",
-            un=un,
-        )
+        from sx.sx_components import render_newsletter_toggle
+        return sx_response(render_newsletter_toggle(un))
 
     # Catch-all for fragment-provided pages — must be last
     @account_bp.get("/<slug>/")
@@ -149,21 +144,15 @@ def register(url_prefix="/"):
         if not fragment_html:
             abort(404)
 
-        w_oob = {**oob, "main": "_types/auth/_fragment_panel.html"}
+        from shared.sx.page import get_template_context
+        from sx.sx_components import render_fragment_page, render_fragment_oob
 
+        ctx = await get_template_context()
         if not is_htmx_request():
-            html = await render_template(
-                "_types/auth/index.html",
-                oob=w_oob,
-                page_fragment_html=fragment_html,
-            )
+            html = await render_fragment_page(ctx, fragment_html)
+            return await make_response(html)
         else:
-            html = await render_template(
-                "_types/auth/_oob_elements.html",
-                oob=w_oob,
-                page_fragment_html=fragment_html,
-            )
-
-        return await make_response(html)
+            sx_src = await render_fragment_oob(ctx, fragment_html)
+            return sx_response(sx_src)
 
     return account_bp

account/bp/auth/routes.py

@@ -12,7 +12,6 @@ from datetime import datetime, timezone, timedelta
 from quart import (
     Blueprint,
     request,
-    render_template,
     redirect,
     url_for,
     session as qsession,
@@ -45,7 +44,7 @@ from .services import (
 SESSION_USER_KEY = "uid"
 ACCOUNT_SESSION_KEY = "account_sid"
 
-ALLOWED_CLIENTS = {"blog", "market", "cart", "events", "federation", "artdag", "artdag_l2"}
+ALLOWED_CLIENTS = {"blog", "market", "cart", "events", "federation", "orders", "test", "artdag", "artdag_l2"}
 
 
 def register(url_prefix="/auth"):
@@ -123,7 +122,7 @@ def register(url_prefix="/auth"):
         sep = "&" if "?" in redirect_uri else "?"
         return redirect(
             f"{redirect_uri}{sep}code={code}&state={state}"
-            f"&account_did={account_did}"
+            f"&account_did={account_did}&grant_token={grant_token}"
         )
 
     # --- OAuth2 token exchange (for external clients like artdag) -------------
@@ -275,7 +274,11 @@ def register(url_prefix="/auth"):
         if g.get("user"):
             redirect_url = pop_login_redirect_target()
             return redirect(redirect_url)
-        return await render_template("auth/login.html")
+
+        from shared.sx.page import get_template_context
+        from sx.sx_components import render_login_page
+        ctx = await get_template_context()
+        return await render_login_page(ctx)
 
     @rate_limit(
         key_func=lambda: request.headers.get("X-Forwarded-For", request.remote_addr),
@@ -288,28 +291,20 @@ def register(url_prefix="/auth"):
 
         is_valid, email = validate_email(email_input)
         if not is_valid:
-            return (
-                await render_template(
-                    "auth/login.html",
-                    error="Please enter a valid email address.",
-                    email=email_input,
-                ),
-                400,
-            )
+            from shared.sx.page import get_template_context
+            from sx.sx_components import render_login_page
+            ctx = await get_template_context(error="Please enter a valid email address.", email=email_input)
+            return await render_login_page(ctx), 400
 
         # Per-email rate limit: 5 magic links per 15 minutes
         from shared.infrastructure.rate_limit import _check_rate_limit
         try:
             allowed, _ = await _check_rate_limit(f"magic_email:{email}", 5, 900)
             if not allowed:
-                return (
-                    await render_template(
-                        "auth/check_email.html",
-                        email=email,
-                        email_error=None,
-                    ),
-                    200,
-                )
+                from shared.sx.page import get_template_context
+                from sx.sx_components import render_check_email_page
+                ctx = await get_template_context(email=email, email_error=None)
+                return await render_check_email_page(ctx), 200
         except Exception:
             pass  # Redis down — allow the request
 
@@ -329,11 +324,10 @@ def register(url_prefix="/auth"):
                 "Please try again in a moment."
             )
 
-        return await render_template(
-            "auth/check_email.html",
-            email=email,
-            email_error=email_error,
-        )
+        from shared.sx.page import get_template_context
+        from sx.sx_components import render_check_email_page
+        ctx = await get_template_context(email=email, email_error=email_error)
+        return await render_check_email_page(ctx)
 
     @auth_bp.get("/magic/<token>/")
     async def magic(token: str):
@@ -346,20 +340,17 @@ def register(url_prefix="/auth"):
                     user, error = await validate_magic_link(s, token)
 
                     if error:
-                        return (
-                            await render_template("auth/login.html", error=error),
-                            400,
-                        )
+                        from shared.sx.page import get_template_context
+                        from sx.sx_components import render_login_page
+                        ctx = await get_template_context(error=error)
+                        return await render_login_page(ctx), 400
                     user_id = user.id
 
         except Exception:
-            return (
-                await render_template(
-                    "auth/login.html",
-                    error="Could not sign you in right now. Please try again.",
-                ),
-                502,
-            )
+            from shared.sx.page import get_template_context
+            from sx.sx_components import render_login_page
+            ctx = await get_template_context(error="Could not sign you in right now. Please try again.")
+            return await render_login_page(ctx), 502
 
         assert user_id is not None
 
@@ -688,8 +679,11 @@ def register(url_prefix="/auth"):
     @auth_bp.get("/device/")
     async def device_form():
         """Browser form where user enters the code displayed in terminal."""
+        from shared.sx.page import get_template_context
+        from sx.sx_components import render_device_page
         code = request.args.get("code", "")
-        return await render_template("auth/device.html", code=code)
+        ctx = await get_template_context(code=code)
+        return await render_device_page(ctx)
 
     @auth_bp.post("/device")
     @auth_bp.post("/device/")
@@ -699,22 +693,20 @@ def register(url_prefix="/auth"):
         user_code = (form.get("code") or "").strip().replace("-", "").upper()
 
         if not user_code or len(user_code) != 8:
-            return await render_template(
-                "auth/device.html",
-                error="Please enter a valid 8-character code.",
-                code=form.get("code", ""),
-            ), 400
+            from shared.sx.page import get_template_context
+            from sx.sx_components import render_device_page
+            ctx = await get_template_context(error="Please enter a valid 8-character code.", code=form.get("code", ""))
+            return await render_device_page(ctx), 400
 
         from shared.infrastructure.auth_redis import get_auth_redis
 
         r = await get_auth_redis()
         device_code = await r.get(f"devflow_uc:{user_code}")
         if not device_code:
-            return await render_template(
-                "auth/device.html",
-                error="Code not found or expired. Please try again.",
-                code=form.get("code", ""),
-            ), 400
+            from shared.sx.page import get_template_context
+            from sx.sx_components import render_device_page
+            ctx = await get_template_context(error="Code not found or expired. Please try again.", code=form.get("code", ""))
+            return await render_device_page(ctx), 400
 
         if isinstance(device_code, bytes):
             device_code = device_code.decode()
@@ -728,17 +720,23 @@ def register(url_prefix="/auth"):
         # Logged in — approve immediately
         ok = await _approve_device(device_code, g.user)
         if not ok:
-            return await render_template(
-                "auth/device.html",
-                error="Code expired or already used.",
-            ), 400
+            from shared.sx.page import get_template_context
+            from sx.sx_components import render_device_page
+            ctx = await get_template_context(error="Code expired or already used.")
+            return await render_device_page(ctx), 400
 
-        return await render_template("auth/device_approved.html")
+        from shared.sx.page import get_template_context
+        from sx.sx_components import render_device_approved_page
+        ctx = await get_template_context()
+        return await render_device_approved_page(ctx)
 
     @auth_bp.get("/device/complete")
     @auth_bp.get("/device/complete/")
     async def device_complete():
         """Post-login redirect — completes approval after magic link auth."""
+        from shared.sx.page import get_template_context
+        from sx.sx_components import render_device_page, render_device_approved_page
+
         device_code = request.args.get("code", "")
 
         if not device_code:
@@ -750,11 +748,12 @@ def register(url_prefix="/auth"):
 
         ok = await _approve_device(device_code, g.user)
         if not ok:
-            return await render_template(
-                "auth/device.html",
+            ctx = await get_template_context(
                 error="Code expired or already used. Please start the login process again in your terminal.",
-            ), 400
+            )
+            return await render_device_page(ctx), 400
 
-        return await render_template("auth/device_approved.html")
+        ctx = await get_template_context()
+        return await render_device_approved_page(ctx)
 
     return auth_bp

account/bp/fragments/routes.py

@@ -1,6 +1,6 @@
 """Account app fragment endpoints.
 
-Exposes HTML fragments at ``/internal/fragments/<type>`` for consumption
+Exposes sx fragments at ``/internal/fragments/<type>`` for consumption
 by other coop apps via the fragment client.
 
 Fragments:
@@ -9,7 +9,7 @@ Fragments:
 
 from __future__ import annotations
 
-from quart import Blueprint, Response, request, render_template
+from quart import Blueprint, Response, request
 
 from shared.infrastructure.fragments import FRAGMENT_HEADER
 
@@ -18,15 +18,17 @@ def register():
     bp = Blueprint("fragments", __name__, url_prefix="/internal/fragments")
 
     # ---------------------------------------------------------------
-    # Fragment handlers
+    # Fragment handlers — return sx source text
     # ---------------------------------------------------------------
 
     async def _auth_menu():
+        from shared.infrastructure.urls import account_url
+        from shared.sx.helpers import sx_call
+
         user_email = request.args.get("email", "")
-        return await render_template(
-            "fragments/auth_menu.html",
-            user_email=user_email,
-        )
+        return sx_call("auth-menu",
+                         user_email=user_email or None,
+                         account_url=account_url(""))
 
     _handlers = {
         "auth-menu": _auth_menu,
@@ -45,8 +47,8 @@ def register():
     async def get_fragment(fragment_type: str):
         handler = _handlers.get(fragment_type)
         if handler is None:
-            return Response("", status=200, content_type="text/html")
-        html = await handler()
-        return Response(html, status=200, content_type="text/html")
+            return Response("", status=200, content_type="text/sx")
+        src = await handler()
+        return Response(src, status=200, content_type="text/sx")
 
     return bp

account/entrypoint.sh

@@ -54,6 +54,7 @@ fi
 RELOAD_FLAG=""
 if [[ "${RELOAD:-}" == "true" ]]; then
   RELOAD_FLAG="--reload"
+  python3 -m shared.dev_watcher &
   echo "Starting Hypercorn (${APP_MODULE:-app:app}) with auto-reload..."
 else
   echo "Starting Hypercorn (${APP_MODULE:-app:app})..."

account/sx/auth.sx

@@ -0,0 +1,58 @@
+;; Auth page components (login, device, check email)
+
+(defcomp ~account-login-error (&key error)
+  (when error
+    (div :class "bg-red-50 border border-red-200 text-red-700 p-3 rounded mb-4"
+      error)))
+
+(defcomp ~account-login-form (&key error action csrf-token email)
+  (div :class "py-8 max-w-md mx-auto"
+    (h1 :class "text-2xl font-bold mb-6" "Sign in")
+    error
+    (form :method "post" :action action :class "space-y-4"
+      (input :type "hidden" :name "csrf_token" :value csrf-token)
+      (div
+        (label :for "email" :class "block text-sm font-medium mb-1" "Email address")
+        (input :type "email" :name "email" :id "email" :value email :required true :autofocus true
+          :class "w-full border border-stone-300 rounded px-3 py-2 focus:outline-none focus:ring-2 focus:ring-stone-500"))
+      (button :type "submit"
+        :class "w-full bg-stone-800 text-white py-2 px-4 rounded hover:bg-stone-700 transition"
+        "Send magic link"))))
+
+(defcomp ~account-device-error (&key error)
+  (when error
+    (div :class "bg-red-50 border border-red-200 text-red-700 p-3 rounded mb-4"
+      error)))
+
+(defcomp ~account-device-form (&key error action csrf-token code)
+  (div :class "py-8 max-w-md mx-auto"
+    (h1 :class "text-2xl font-bold mb-6" "Authorize device")
+    (p :class "text-stone-600 mb-4" "Enter the code shown in your terminal to sign in.")
+    error
+    (form :method "post" :action action :class "space-y-4"
+      (input :type "hidden" :name "csrf_token" :value csrf-token)
+      (div
+        (label :for "code" :class "block text-sm font-medium mb-1" "Device code")
+        (input :type "text" :name "code" :id "code" :value code :placeholder "XXXX-XXXX"
+          :required true :autofocus true :maxlength "9" :autocomplete "off" :spellcheck "false"
+          :class "w-full border border-stone-300 rounded px-3 py-3 text-center text-2xl tracking-widest font-mono uppercase focus:outline-none focus:ring-2 focus:ring-stone-500"))
+      (button :type "submit"
+        :class "w-full bg-stone-800 text-white py-2 px-4 rounded hover:bg-stone-700 transition"
+        "Authorize"))))
+
+(defcomp ~account-device-approved ()
+  (div :class "py-8 max-w-md mx-auto text-center"
+    (h1 :class "text-2xl font-bold mb-4" "Device authorized")
+    (p :class "text-stone-600" "You can close this window and return to your terminal.")))
+
+(defcomp ~account-check-email-error (&key error)
+  (when error
+    (div :class "bg-yellow-50 border border-yellow-200 text-yellow-700 p-3 rounded mt-4"
+      error)))
+
+(defcomp ~account-check-email (&key email error)
+  (div :class "py-8 max-w-md mx-auto text-center"
+    (h1 :class "text-2xl font-bold mb-4" "Check your email")
+    (p :class "text-stone-600 mb-2" "We sent a sign-in link to " (strong email) ".")
+    (p :class "text-stone-500 text-sm" "Click the link in the email to sign in. The link expires in 15 minutes.")
+    error))

account/sx/dashboard.sx

@@ -0,0 +1,48 @@
+;; Account dashboard components
+
+(defcomp ~account-error-banner (&key error)
+  (when error
+    (div :class "rounded-lg border border-red-200 bg-red-50 text-red-800 px-4 py-3 text-sm"
+      error)))
+
+(defcomp ~account-user-email (&key email)
+  (when email
+    (p :class "text-sm text-stone-500 mt-1" email)))
+
+(defcomp ~account-user-name (&key name)
+  (when name
+    (p :class "text-sm text-stone-600" name)))
+
+(defcomp ~account-logout-form (&key csrf-token)
+  (form :action "/auth/logout/" :method "post"
+    (input :type "hidden" :name "csrf_token" :value csrf-token)
+    (button :type "submit"
+      :class "inline-flex items-center gap-2 rounded-full border border-stone-300 px-4 py-2 text-sm font-medium text-stone-700 hover:bg-stone-50 transition"
+      (i :class "fa-solid fa-right-from-bracket text-xs") " Sign out")))
+
+(defcomp ~account-label-item (&key name)
+  (span :class "inline-flex items-center rounded-full border border-stone-200 px-3 py-1 text-xs font-medium bg-white/60"
+    name))
+
+(defcomp ~account-labels-section (&key items)
+  (when items
+    (div
+      (h2 :class "text-base font-semibold tracking-tight mb-3" "Labels")
+      (div :class "flex flex-wrap gap-2" items))))
+
+(defcomp ~account-main-panel (&key error email name logout labels)
+  (div :class "w-full max-w-3xl mx-auto px-4 py-6"
+    (div :class "bg-white/70 backdrop-blur rounded-2xl shadow border border-stone-200 p-6 sm:p-8 space-y-8"
+      error
+      (div :class "flex items-center justify-between"
+        (div
+          (h1 :class "text-xl font-semibold tracking-tight" "Account")
+          email
+          name)
+        logout)
+      labels)))
+
+;; Header child wrapper
+(defcomp ~account-header-child (&key inner)
+  (div :id "root-header-child" :class "flex flex-col w-full items-center"
+    inner))

account/sx/newsletters.sx

@@ -0,0 +1,37 @@
+;; Newsletter management components
+
+(defcomp ~account-newsletter-desc (&key description)
+  (when description
+    (p :class "text-xs text-stone-500 mt-0.5 truncate" description)))
+
+(defcomp ~account-newsletter-toggle (&key id url hdrs target cls checked knob-cls)
+  (div :id id :class "flex items-center"
+    (button :sx-post url :sx-headers hdrs :sx-target target :sx-swap "outerHTML"
+      :class cls :role "switch" :aria-checked checked
+      (span :class knob-cls))))
+
+(defcomp ~account-newsletter-toggle-off (&key id url hdrs target)
+  (div :id id :class "flex items-center"
+    (button :sx-post url :sx-headers hdrs :sx-target target :sx-swap "outerHTML"
+      :class "relative inline-flex h-6 w-11 items-center rounded-full transition-colors focus:outline-none focus:ring-2 focus:ring-emerald-500 focus:ring-offset-2 bg-stone-300"
+      :role "switch" :aria-checked "false"
+      (span :class "inline-block h-4 w-4 rounded-full bg-white shadow transform transition-transform translate-x-1"))))
+
+(defcomp ~account-newsletter-item (&key name desc toggle)
+  (div :class "flex items-center justify-between py-4 first:pt-0 last:pb-0"
+    (div :class "min-w-0 flex-1"
+      (p :class "text-sm font-medium text-stone-800" name)
+      desc)
+    (div :class "ml-4 flex-shrink-0" toggle)))
+
+(defcomp ~account-newsletter-list (&key items)
+  (div :class "divide-y divide-stone-100" items))
+
+(defcomp ~account-newsletter-empty ()
+  (p :class "text-sm text-stone-500" "No newsletters available."))
+
+(defcomp ~account-newsletters-panel (&key list)
+  (div :class "w-full max-w-3xl mx-auto px-4 py-6"
+    (div :class "bg-white/70 backdrop-blur rounded-2xl shadow border border-stone-200 p-6 sm:p-8 space-y-6"
+      (h1 :class "text-xl font-semibold tracking-tight" "Newsletters")
+      list)))

account/sx/sx_components.py

@@ -0,0 +1,392 @@
+"""
+Account service s-expression page components.
+
+Renders account dashboard, newsletters, fragment pages, login, and device
+auth pages.  Called from route handlers in place of ``render_template()``.
+"""
+from __future__ import annotations
+
+import os
+from typing import Any
+
+from shared.sx.jinja_bridge import load_service_components
+from shared.sx.helpers import (
+    call_url, sx_call, SxExpr,
+    root_header_sx, full_page_sx, header_child_sx, oob_page_sx,
+)
+
+# Load account-specific .sx components at import time
+load_service_components(os.path.dirname(os.path.dirname(__file__)))
+
+
+# ---------------------------------------------------------------------------
+# Header helpers
+# ---------------------------------------------------------------------------
+
+def _auth_nav_sx(ctx: dict) -> str:
+    """Auth section desktop nav items."""
+    parts = [
+        sx_call("nav-link",
+            href=call_url(ctx, "account_url", "/newsletters/"),
+            label="newsletters",
+            select_colours=ctx.get("select_colours", ""),
+        )
+    ]
+    account_nav = ctx.get("account_nav")
+    if account_nav:
+        parts.append(account_nav)
+    return "(<> " + " ".join(parts) + ")"
+
+
+def _auth_header_sx(ctx: dict, *, oob: bool = False) -> str:
+    """Build the account section header row."""
+    return sx_call(
+        "menu-row-sx",
+        id="auth-row", level=1, colour="sky",
+        link_href=call_url(ctx, "account_url", "/"),
+        link_label="account", icon="fa-solid fa-user",
+        nav=SxExpr(_auth_nav_sx(ctx)),
+        child_id="auth-header-child", oob=oob,
+    )
+
+
+def _auth_nav_mobile_sx(ctx: dict) -> str:
+    """Mobile nav menu for auth section."""
+    parts = [
+        sx_call("nav-link",
+            href=call_url(ctx, "account_url", "/newsletters/"),
+            label="newsletters",
+            select_colours=ctx.get("select_colours", ""),
+        )
+    ]
+    account_nav = ctx.get("account_nav")
+    if account_nav:
+        parts.append(account_nav)
+    return "(<> " + " ".join(parts) + ")"
+
+
+# ---------------------------------------------------------------------------
+# Account dashboard (GET /)
+# ---------------------------------------------------------------------------
+
+def _account_main_panel_sx(ctx: dict) -> str:
+    """Account info panel with user details and logout."""
+    from quart import g
+    from shared.browser.app.csrf import generate_csrf_token
+
+    user = getattr(g, "user", None)
+    error = ctx.get("error", "")
+
+    error_sx = sx_call("account-error-banner", error=error) if error else ""
+
+    user_email_sx = ""
+    user_name_sx = ""
+    if user:
+        user_email_sx = sx_call("account-user-email", email=user.email)
+        if user.name:
+            user_name_sx = sx_call("account-user-name", name=user.name)
+
+    logout_sx = sx_call("account-logout-form", csrf_token=generate_csrf_token())
+
+    labels_sx = ""
+    if user and hasattr(user, "labels") and user.labels:
+        label_items = " ".join(
+            sx_call("account-label-item", name=label.name)
+            for label in user.labels
+        )
+        labels_sx = sx_call("account-labels-section",
+                                items=SxExpr("(<> " + label_items + ")"))
+
+    return sx_call(
+        "account-main-panel",
+        error=SxExpr(error_sx) if error_sx else None,
+        email=SxExpr(user_email_sx) if user_email_sx else None,
+        name=SxExpr(user_name_sx) if user_name_sx else None,
+        logout=SxExpr(logout_sx),
+        labels=SxExpr(labels_sx) if labels_sx else None,
+    )
+
+
+# ---------------------------------------------------------------------------
+# Newsletters (GET /newsletters/)
+# ---------------------------------------------------------------------------
+
+def _newsletter_toggle_sx(un: Any, account_url_fn: Any, csrf_token: str) -> str:
+    """Render a single newsletter toggle switch."""
+    nid = un.newsletter_id
+    toggle_url = account_url_fn(f"/newsletter/{nid}/toggle/")
+    if un.subscribed:
+        bg = "bg-emerald-500"
+        translate = "translate-x-6"
+        checked = "true"
+    else:
+        bg = "bg-stone-300"
+        translate = "translate-x-1"
+        checked = "false"
+    return sx_call(
+        "account-newsletter-toggle",
+        id=f"nl-{nid}", url=toggle_url,
+        hdrs=f'{{"X-CSRFToken": "{csrf_token}"}}',
+        target=f"#nl-{nid}",
+        cls=f"relative inline-flex h-6 w-11 items-center rounded-full transition-colors focus:outline-none focus:ring-2 focus:ring-emerald-500 focus:ring-offset-2 {bg}",
+        checked=checked,
+        knob_cls=f"inline-block h-4 w-4 rounded-full bg-white shadow transform transition-transform {translate}",
+    )
+
+
+def _newsletter_toggle_off_sx(nid: int, toggle_url: str, csrf_token: str) -> str:
+    """Render an unsubscribed newsletter toggle (no subscription record yet)."""
+    return sx_call(
+        "account-newsletter-toggle-off",
+        id=f"nl-{nid}", url=toggle_url,
+        hdrs=f'{{"X-CSRFToken": "{csrf_token}"}}',
+        target=f"#nl-{nid}",
+    )
+
+
+def _newsletters_panel_sx(ctx: dict, newsletter_list: list) -> str:
+    """Newsletters management panel."""
+    from shared.browser.app.csrf import generate_csrf_token
+
+    account_url_fn = ctx.get("account_url") or (lambda p: p)
+    csrf = generate_csrf_token()
+
+    if newsletter_list:
+        items = []
+        for item in newsletter_list:
+            nl = item["newsletter"]
+            un = item.get("un")
+
+            desc_sx = sx_call(
+                "account-newsletter-desc", description=nl.description
+            ) if nl.description else ""
+
+            if un:
+                toggle = _newsletter_toggle_sx(un, account_url_fn, csrf)
+            else:
+                toggle_url = account_url_fn(f"/newsletter/{nl.id}/toggle/")
+                toggle = _newsletter_toggle_off_sx(nl.id, toggle_url, csrf)
+
+            items.append(sx_call(
+                "account-newsletter-item",
+                name=nl.name,
+                desc=SxExpr(desc_sx) if desc_sx else None,
+                toggle=SxExpr(toggle),
+            ))
+        list_sx = sx_call(
+            "account-newsletter-list",
+            items=SxExpr("(<> " + " ".join(items) + ")"),
+        )
+    else:
+        list_sx = sx_call("account-newsletter-empty")
+
+    return sx_call("account-newsletters-panel", list=SxExpr(list_sx))
+
+
+# ---------------------------------------------------------------------------
+# Auth pages (login, device, check_email)
+# ---------------------------------------------------------------------------
+
+def _login_page_content(ctx: dict) -> str:
+    """Login form content."""
+    from shared.browser.app.csrf import generate_csrf_token
+    from quart import url_for
+
+    error = ctx.get("error", "")
+    email = ctx.get("email", "")
+    action = url_for("auth.start_login")
+
+    error_sx = sx_call("account-login-error", error=error) if error else ""
+
+    return sx_call(
+        "account-login-form",
+        error=SxExpr(error_sx) if error_sx else None,
+        action=action,
+        csrf_token=generate_csrf_token(), email=email,
+    )
+
+
+def _device_page_content(ctx: dict) -> str:
+    """Device authorization form content."""
+    from shared.browser.app.csrf import generate_csrf_token
+    from quart import url_for
+
+    error = ctx.get("error", "")
+    code = ctx.get("code", "")
+    action = url_for("auth.device_submit")
+
+    error_sx = sx_call("account-device-error", error=error) if error else ""
+
+    return sx_call(
+        "account-device-form",
+        error=SxExpr(error_sx) if error_sx else None,
+        action=action,
+        csrf_token=generate_csrf_token(), code=code,
+    )
+
+
+def _device_approved_content() -> str:
+    """Device approved success content."""
+    return sx_call("account-device-approved")
+
+
+# ---------------------------------------------------------------------------
+# Public API: Account dashboard
+# ---------------------------------------------------------------------------
+
+async def render_account_page(ctx: dict) -> str:
+    """Full page: account dashboard."""
+    main = _account_main_panel_sx(ctx)
+
+    hdr = root_header_sx(ctx)
+    hdr_child = header_child_sx(_auth_header_sx(ctx))
+    header_rows = "(<> " + hdr + " " + hdr_child + ")"
+
+    return full_page_sx(ctx, header_rows=header_rows,
+                          content=main,
+                          menu=_auth_nav_mobile_sx(ctx))
+
+
+async def render_account_oob(ctx: dict) -> str:
+    """OOB response for account dashboard."""
+    main = _account_main_panel_sx(ctx)
+
+    oobs = "(<> " + _auth_header_sx(ctx, oob=True) + " " + root_header_sx(ctx, oob=True) + ")"
+
+    return oob_page_sx(oobs=oobs,
+                         content=main,
+                         menu=_auth_nav_mobile_sx(ctx))
+
+
+# ---------------------------------------------------------------------------
+# Public API: Newsletters
+# ---------------------------------------------------------------------------
+
+async def render_newsletters_page(ctx: dict, newsletter_list: list) -> str:
+    """Full page: newsletters."""
+    main = _newsletters_panel_sx(ctx, newsletter_list)
+
+    hdr = root_header_sx(ctx)
+    hdr_child = header_child_sx(_auth_header_sx(ctx))
+    header_rows = "(<> " + hdr + " " + hdr_child + ")"
+
+    return full_page_sx(ctx, header_rows=header_rows,
+                          content=main,
+                          menu=_auth_nav_mobile_sx(ctx))
+
+
+async def render_newsletters_oob(ctx: dict, newsletter_list: list) -> str:
+    """OOB response for newsletters."""
+    main = _newsletters_panel_sx(ctx, newsletter_list)
+
+    oobs = "(<> " + _auth_header_sx(ctx, oob=True) + " " + root_header_sx(ctx, oob=True) + ")"
+
+    return oob_page_sx(oobs=oobs,
+                         content=main,
+                         menu=_auth_nav_mobile_sx(ctx))
+
+
+# ---------------------------------------------------------------------------
+# Public API: Fragment pages
+# ---------------------------------------------------------------------------
+
+async def render_fragment_page(ctx: dict, page_fragment_html: str) -> str:
+    """Full page: fragment-provided content."""
+    hdr = root_header_sx(ctx)
+    hdr_child = header_child_sx(_auth_header_sx(ctx))
+    header_rows = "(<> " + hdr + " " + hdr_child + ")"
+
+    return full_page_sx(ctx, header_rows=header_rows,
+                          content=f'(~rich-text :html "{_sx_escape(page_fragment_html)}")',
+                          menu=_auth_nav_mobile_sx(ctx))
+
+
+async def render_fragment_oob(ctx: dict, page_fragment_html: str) -> str:
+    """OOB response for fragment pages."""
+    oobs = "(<> " + _auth_header_sx(ctx, oob=True) + " " + root_header_sx(ctx, oob=True) + ")"
+
+    return oob_page_sx(oobs=oobs,
+                         content=f'(~rich-text :html "{_sx_escape(page_fragment_html)}")',
+                         menu=_auth_nav_mobile_sx(ctx))
+
+
+# ---------------------------------------------------------------------------
+# Public API: Auth pages (login, device)
+# ---------------------------------------------------------------------------
+
+async def render_login_page(ctx: dict) -> str:
+    """Full page: login form."""
+    hdr = root_header_sx(ctx)
+    return full_page_sx(ctx, header_rows=hdr,
+                          content=_login_page_content(ctx),
+                          meta_html='<title>Login \u2014 Rose Ash</title>')
+
+
+async def render_device_page(ctx: dict) -> str:
+    """Full page: device authorization form."""
+    hdr = root_header_sx(ctx)
+    return full_page_sx(ctx, header_rows=hdr,
+                          content=_device_page_content(ctx),
+                          meta_html='<title>Authorize Device \u2014 Rose Ash</title>')
+
+
+async def render_device_approved_page(ctx: dict) -> str:
+    """Full page: device approved."""
+    hdr = root_header_sx(ctx)
+    return full_page_sx(ctx, header_rows=hdr,
+                          content=_device_approved_content(),
+                          meta_html='<title>Device Authorized \u2014 Rose Ash</title>')
+
+
+# ---------------------------------------------------------------------------
+# Public API: Check email page (POST /start/ success)
+# ---------------------------------------------------------------------------
+
+def _check_email_content(email: str, email_error: str | None = None) -> str:
+    """Check email confirmation content."""
+    from markupsafe import escape
+
+    error_sx = sx_call(
+        "account-check-email-error", error=str(escape(email_error))
+    ) if email_error else ""
+
+    return sx_call(
+        "account-check-email",
+        email=str(escape(email)),
+        error=SxExpr(error_sx) if error_sx else None,
+    )
+
+
+async def render_check_email_page(ctx: dict) -> str:
+    """Full page: check email after magic link sent."""
+    email = ctx.get("email", "")
+    email_error = ctx.get("email_error")
+    hdr = root_header_sx(ctx)
+    return full_page_sx(ctx, header_rows=hdr,
+                          content=_check_email_content(email, email_error),
+                          meta_html='<title>Check your email \u2014 Rose Ash</title>')
+
+
+# ---------------------------------------------------------------------------
+# Public API: Fragment renderers for POST handlers
+# ---------------------------------------------------------------------------
+
+
+def render_newsletter_toggle(un) -> str:
+    """Render a newsletter toggle switch for POST response (uses account_url)."""
+    from shared.browser.app.csrf import generate_csrf_token
+    from quart import g
+    account_url_fn = getattr(g, "_account_url", None)
+    if account_url_fn is None:
+        from shared.infrastructure.urls import account_url
+        account_url_fn = account_url
+    return _newsletter_toggle_sx(un, account_url_fn, generate_csrf_token())
+
+
+# ---------------------------------------------------------------------------
+# Internal helpers
+# ---------------------------------------------------------------------------
+
+def _sx_escape(s: str) -> str:
+    """Escape a string for embedding in sx string literals."""
+    return s.replace("\\", "\\\\").replace('"', '\\"')

account/templates/_types/auth/_newsletter_toggle.html

@@ -1,9 +1,9 @@
 <div id="nl-{{ un.newsletter_id }}" class="flex items-center">
   <button
-    hx-post="{{ account_url('/newsletter/' ~ un.newsletter_id ~ '/toggle/') }}"
-    hx-headers='{"X-CSRFToken": "{{ csrf_token() }}"}'
-    hx-target="#nl-{{ un.newsletter_id }}"
-    hx-swap="outerHTML"
+    sx-post="{{ account_url('/newsletter/' ~ un.newsletter_id ~ '/toggle/') }}"
+    sx-headers='{"X-CSRFToken": "{{ csrf_token() }}"}'
+    sx-target="#nl-{{ un.newsletter_id }}"
+    sx-swap="outerHTML"
     class="relative inline-flex h-6 w-11 items-center rounded-full transition-colors focus:outline-none focus:ring-2 focus:ring-emerald-500 focus:ring-offset-2
       {% if un.subscribed %}bg-emerald-500{% else %}bg-stone-300{% endif %}"
     role="switch"

account/templates/_types/auth/_newsletters_panel.html

@@ -22,10 +22,10 @@
                 {# No subscription row yet — show an off toggle that will create one #}
                 <div id="nl-{{ item.newsletter.id }}" class="flex items-center">
                   <button
-                    hx-post="{{ account_url('/newsletter/' ~ item.newsletter.id ~ '/toggle/') }}"
-                    hx-headers='{"X-CSRFToken": "{{ csrf_token() }}"}'
-                    hx-target="#nl-{{ item.newsletter.id }}"
-                    hx-swap="outerHTML"
+                    sx-post="{{ account_url('/newsletter/' ~ item.newsletter.id ~ '/toggle/') }}"
+                    sx-headers='{"X-CSRFToken": "{{ csrf_token() }}"}'
+                    sx-target="#nl-{{ item.newsletter.id }}"
+                    sx-swap="outerHTML"
                     class="relative inline-flex h-6 w-11 items-center rounded-full transition-colors focus:outline-none focus:ring-2 focus:ring-emerald-500 focus:ring-offset-2 bg-stone-300"
                     role="switch"
                     aria-checked="false"

account/templates/fragments/auth_menu.html

@@ -1,36 +0,0 @@
-{# Desktop auth menu #}
-<span id="auth-menu-desktop" class="hidden md:inline-flex">
-{% if user_email %}
-  <a
-    href="{{ account_url('/') }}"
-    class="justify-center cursor-pointer flex flex-row items-center p-3 gap-2 rounded bg-stone-200 text-black"
-    data-close-details
-  >
-    <i class="fa-solid fa-user"></i>
-    <span>{{ user_email }}</span>
-  </a>
-{% else %}
-  <a
-    href="{{ account_url('/') }}"
-    class="justify-center cursor-pointer flex flex-row items-center p-3 gap-2 rounded bg-stone-200 text-black"
-    data-close-details
-  >
-    <i class="fa-solid fa-key"></i>
-    <span>sign in or register</span>
-  </a>
-{% endif %}
-</span>
-{# Mobile auth menu #}
-<span id="auth-menu-mobile" class="block md:hidden text-md font-bold">
-{% if user_email %}
-  <a href="{{ account_url('/') }}" data-close-details>
-    <i class="fa-solid fa-user"></i>
-    <span>{{ user_email }}</span>
-  </a>
-{% else %}
-  <a href="{{ account_url('/') }}">
-    <i class="fa-solid fa-key"></i>
-    <span>sign in or register</span>
-  </a>
-{% endif %}
-</span>

account/tests/test_auth_operations.py

@@ -0,0 +1,39 @@
+"""Unit tests for account auth operations."""
+from __future__ import annotations
+
+import pytest
+
+from account.bp.auth.services.auth_operations import validate_email
+
+
+class TestValidateEmail:
+    def test_valid_email(self):
+        ok, email = validate_email("user@example.com")
+        assert ok is True
+        assert email == "user@example.com"
+
+    def test_uppercase_lowered(self):
+        ok, email = validate_email("USER@EXAMPLE.COM")
+        assert ok is True
+        assert email == "user@example.com"
+
+    def test_whitespace_stripped(self):
+        ok, email = validate_email("  user@example.com  ")
+        assert ok is True
+        assert email == "user@example.com"
+
+    def test_empty_string(self):
+        ok, email = validate_email("")
+        assert ok is False
+
+    def test_no_at_sign(self):
+        ok, email = validate_email("notanemail")
+        assert ok is False
+
+    def test_just_at(self):
+        ok, email = validate_email("@")
+        assert ok is True  # has "@", passes the basic check
+
+    def test_spaces_only(self):
+        ok, email = validate_email("   ")
+        assert ok is False

account/tests/test_ghost_membership.py

@@ -0,0 +1,164 @@
+"""Unit tests for Ghost membership helpers."""
+from __future__ import annotations
+
+from datetime import datetime
+
+import pytest
+
+from account.services.ghost_membership import (
+    _iso, _to_str_or_none, _member_email,
+    _price_cents, _sanitize_member_payload,
+)
+
+
+class TestIso:
+    def test_none(self):
+        assert _iso(None) is None
+
+    def test_empty(self):
+        assert _iso("") is None
+
+    def test_z_suffix(self):
+        result = _iso("2024-06-15T12:00:00Z")
+        assert isinstance(result, datetime)
+        assert result.year == 2024
+
+    def test_offset(self):
+        result = _iso("2024-06-15T12:00:00+00:00")
+        assert isinstance(result, datetime)
+
+
+class TestToStrOrNone:
+    def test_none(self):
+        assert _to_str_or_none(None) is None
+
+    def test_dict(self):
+        assert _to_str_or_none({"a": 1}) is None
+
+    def test_list(self):
+        assert _to_str_or_none([1, 2]) is None
+
+    def test_bytes(self):
+        assert _to_str_or_none(b"hello") is None
+
+    def test_empty_string(self):
+        assert _to_str_or_none("") is None
+
+    def test_whitespace_only(self):
+        assert _to_str_or_none("   ") is None
+
+    def test_valid_string(self):
+        assert _to_str_or_none("hello") == "hello"
+
+    def test_int(self):
+        assert _to_str_or_none(42) == "42"
+
+    def test_strips_whitespace(self):
+        assert _to_str_or_none("  hi  ") == "hi"
+
+    def test_set(self):
+        assert _to_str_or_none({1, 2}) is None
+
+    def test_tuple(self):
+        assert _to_str_or_none((1,)) is None
+
+    def test_bytearray(self):
+        assert _to_str_or_none(bytearray(b"x")) is None
+
+
+class TestMemberEmail:
+    def test_normal(self):
+        assert _member_email({"email": "USER@EXAMPLE.COM"}) == "user@example.com"
+
+    def test_none(self):
+        assert _member_email({"email": None}) is None
+
+    def test_empty(self):
+        assert _member_email({"email": ""}) is None
+
+    def test_whitespace(self):
+        assert _member_email({"email": "  "}) is None
+
+    def test_missing_key(self):
+        assert _member_email({}) is None
+
+    def test_strips(self):
+        assert _member_email({"email": "  a@b.com  "}) == "a@b.com"
+
+
+class TestPriceCents:
+    def test_valid(self):
+        assert _price_cents({"price": {"amount": 1500}}) == 1500
+
+    def test_string_amount(self):
+        assert _price_cents({"price": {"amount": "2000"}}) == 2000
+
+    def test_missing_price(self):
+        assert _price_cents({}) is None
+
+    def test_missing_amount(self):
+        assert _price_cents({"price": {}}) is None
+
+    def test_none_amount(self):
+        assert _price_cents({"price": {"amount": None}}) is None
+
+    def test_nested_none(self):
+        assert _price_cents({"price": None}) is None
+
+
+class TestSanitizeMemberPayload:
+    def test_email_lowercased(self):
+        result = _sanitize_member_payload({"email": "USER@EXAMPLE.COM"})
+        assert result["email"] == "user@example.com"
+
+    def test_empty_email_excluded(self):
+        result = _sanitize_member_payload({"email": ""})
+        assert "email" not in result
+
+    def test_name_included(self):
+        result = _sanitize_member_payload({"name": "Alice"})
+        assert result["name"] == "Alice"
+
+    def test_note_included(self):
+        result = _sanitize_member_payload({"note": "VIP"})
+        assert result["note"] == "VIP"
+
+    def test_subscribed_bool(self):
+        result = _sanitize_member_payload({"subscribed": 1})
+        assert result["subscribed"] is True
+
+    def test_labels_with_id(self):
+        result = _sanitize_member_payload({
+            "labels": [{"id": "abc"}, {"name": "VIP"}]
+        })
+        assert result["labels"] == [{"id": "abc"}, {"name": "VIP"}]
+
+    def test_labels_empty_items_excluded(self):
+        result = _sanitize_member_payload({
+            "labels": [{"id": None, "name": None}]
+        })
+        assert "labels" not in result
+
+    def test_newsletters_with_id(self):
+        result = _sanitize_member_payload({
+            "newsletters": [{"id": "n1", "subscribed": True}]
+        })
+        assert result["newsletters"] == [{"subscribed": True, "id": "n1"}]
+
+    def test_newsletters_default_subscribed(self):
+        result = _sanitize_member_payload({
+            "newsletters": [{"name": "Weekly"}]
+        })
+        assert result["newsletters"][0]["subscribed"] is True
+
+    def test_dict_email_excluded(self):
+        result = _sanitize_member_payload({"email": {"bad": "input"}})
+        assert "email" not in result
+
+    def test_id_passthrough(self):
+        result = _sanitize_member_payload({"id": "ghost-member-123"})
+        assert result["id"] == "ghost-member-123"
+
+    def test_empty_payload(self):
+        result = _sanitize_member_payload({})
+        assert result == {}

artdag/.dockerignore

@@ -0,0 +1,8 @@
+.git
+.gitea
+**/.env
+**/.env.gpu
+**/__pycache__
+**/.pytest_cache
+**/*.pyc
+test/

artdag/.gitea/workflows/ci.yml

@@ -0,0 +1,114 @@
+name: Build and Deploy
+
+on:
+  push:
+    branches: [main]
+
+env:
+  REGISTRY: registry.rose-ash.com:5000
+  ARTDAG_DIR: /root/art-dag-mono
+
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install tools
+        run: |
+          apt-get update && apt-get install -y --no-install-recommends openssh-client
+
+      - name: Set up SSH
+        env:
+          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
+          DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
+        run: |
+          mkdir -p ~/.ssh
+          echo "$SSH_KEY" > ~/.ssh/id_rsa
+          chmod 600 ~/.ssh/id_rsa
+          ssh-keyscan -H "$DEPLOY_HOST" >> ~/.ssh/known_hosts 2>/dev/null || true
+
+      - name: Build and deploy
+        env:
+          DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
+        run: |
+          ssh "root@$DEPLOY_HOST" "
+            cd ${{ env.ARTDAG_DIR }}
+
+            OLD_HEAD=\$(git rev-parse HEAD 2>/dev/null || echo none)
+
+            git fetch origin main
+            git reset --hard origin/main
+
+            NEW_HEAD=\$(git rev-parse HEAD)
+
+            # Change detection
+            BUILD_L1=false
+            BUILD_L2=false
+            if [ \"\$OLD_HEAD\" = \"none\" ] || [ \"\$OLD_HEAD\" = \"\$NEW_HEAD\" ]; then
+              BUILD_L1=true
+              BUILD_L2=true
+            else
+              CHANGED=\$(git diff --name-only \$OLD_HEAD \$NEW_HEAD)
+              # common/ or core/ change -> rebuild both
+              if echo \"\$CHANGED\" | grep -qE '^(common|core)/'; then
+                BUILD_L1=true
+                BUILD_L2=true
+              fi
+              if echo \"\$CHANGED\" | grep -q '^l1/'; then
+                BUILD_L1=true
+              fi
+              if echo \"\$CHANGED\" | grep -q '^l2/'; then
+                BUILD_L2=true
+              fi
+              if echo \"\$CHANGED\" | grep -q '^client/'; then
+                BUILD_L1=true
+              fi
+            fi
+
+            # Build L1
+            if [ \"\$BUILD_L1\" = true ]; then
+              echo 'Building L1...'
+              docker build \
+                --build-arg CACHEBUST=\$(date +%s) \
+                -f l1/Dockerfile \
+                -t ${{ env.REGISTRY }}/celery-l1-server:latest \
+                -t ${{ env.REGISTRY }}/celery-l1-server:${{ github.sha }} \
+                .
+              docker push ${{ env.REGISTRY }}/celery-l1-server:latest
+              docker push ${{ env.REGISTRY }}/celery-l1-server:${{ github.sha }}
+            else
+              echo 'Skipping L1 (no changes)'
+            fi
+
+            # Build L2
+            if [ \"\$BUILD_L2\" = true ]; then
+              echo 'Building L2...'
+              docker build \
+                --build-arg CACHEBUST=\$(date +%s) \
+                -f l2/Dockerfile \
+                -t ${{ env.REGISTRY }}/l2-server:latest \
+                -t ${{ env.REGISTRY }}/l2-server:${{ github.sha }} \
+                .
+              docker push ${{ env.REGISTRY }}/l2-server:latest
+              docker push ${{ env.REGISTRY }}/l2-server:${{ github.sha }}
+            else
+              echo 'Skipping L2 (no changes)'
+            fi
+
+            # Deploy stacks (--resolve-image always forces re-pull of :latest)
+            if [ \"\$BUILD_L1\" = true ]; then
+              cd l1 && source .env && docker stack deploy --resolve-image always -c docker-compose.yml celery && cd ..
+              echo 'L1 stack deployed'
+            fi
+            if [ \"\$BUILD_L2\" = true ]; then
+              cd l2 && source .env && docker stack deploy --resolve-image always -c docker-compose.yml activitypub && cd ..
+              echo 'L2 stack deployed'
+            fi
+
+            sleep 10
+            echo '=== L1 Services ==='
+            docker stack services celery
+            echo '=== L2 Services ==='
+            docker stack services activitypub
+          "

artdag/CLAUDE.md

@@ -0,0 +1,74 @@
+# Art DAG Monorepo
+
+Federated content-addressed DAG execution engine for distributed media processing with ActivityPub ownership and provenance tracking.
+
+## Project Structure
+
+```
+core/       # DAG engine (artdag package) - nodes, effects, analysis, planning
+l1/         # L1 Celery rendering server (FastAPI + Celery + Redis + PostgreSQL)
+l2/         # L2 ActivityPub registry (FastAPI + PostgreSQL)
+common/     # Shared templates, middleware, models (artdag_common package)
+client/     # CLI client
+test/       # Integration & e2e tests
+```
+
+## Tech Stack
+
+Python 3.11+, FastAPI, Celery, Redis, PostgreSQL (asyncpg for L1), SQLAlchemy, Pydantic, JAX (CPU/GPU), IPFS/Kubo, Docker Swarm, HTMX + Jinja2 for web UI.
+
+## Key Commands
+
+### Testing
+```bash
+cd l1 && pytest tests/                    # L1 unit tests
+cd core && pytest tests/                  # Core unit tests
+cd test && python run.py                  # Full integration pipeline
+```
+- pytest uses `asyncio_mode = "auto"` for async tests
+- Test files: `test_*.py`, fixtures in `conftest.py`
+
+### Linting & Type Checking (L1)
+```bash
+cd l1 && ruff check .                     # Lint (E, F, I, UP rules)
+cd l1 && mypy app/types.py app/routers/recipes.py tests/
+```
+- Line length: 100 chars (E501 ignored)
+- Mypy: strict on `app/types.py`, `app/routers/recipes.py`, `tests/`; gradual elsewhere
+- Mypy ignores imports for: celery, redis, artdag, artdag_common, ipfs_client
+
+### Docker
+```bash
+docker build -f l1/Dockerfile -t celery-l1-server:latest .
+docker build -f l1/Dockerfile.gpu -t celery-l1-gpu:latest .
+docker build -f l2/Dockerfile -t l2-server:latest .
+./deploy.sh                               # Build, push, deploy stacks
+```
+
+## Architecture Patterns
+
+- **3-Phase Execution**: Analyze -> Plan -> Execute (tasks in `l1/tasks/`)
+- **Content-Addressed**: All data identified by SHA3-256 hashes or IPFS CIDs
+- **Services Pattern**: Business logic in `app/services/`, API endpoints in `app/routers/`
+- **Types Module**: Pydantic models and TypedDicts in `app/types.py`
+- **Celery Tasks**: In `l1/tasks/`, decorated with `@app.task`
+- **S-Expression Effects**: Composable effect language in `l1/sexp_effects/`
+- **Storage**: Local filesystem, S3, or IPFS backends (`storage_providers.py`)
+
+## Auth
+
+- L1 <-> L2: scoped JWT tokens (no shared secrets)
+- L2: password + OAuth SSO, token revocation in Redis (30-day expiry)
+- Federation: ActivityPub RSA signatures (`core/artdag/activitypub/`)
+
+## Key Config Files
+
+- `l1/pyproject.toml` - mypy, pytest, ruff config for L1
+- `l1/celery_app.py` - Celery initialization
+- `l1/database.py` / `l2/db.py` - SQLAlchemy models
+- `l1/docker-compose.yml` / `l2/docker-compose.yml` - Swarm stacks
+
+## Tools
+
+- Use Context7 MCP for up-to-date library documentation
+- Playwright MCP is available for browser automation/testing

artdag/client/.gitignore

@@ -0,0 +1,5 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+.scripts

artdag/client/README.md

@@ -0,0 +1,263 @@
+# Art DAG Client
+
+CLI for interacting with the Art DAG L1 rendering server.
+
+## Setup
+
+```bash
+pip install -r requirements.txt
+```
+
+## Configuration
+
+```bash
+# Set L1 server URL (default: http://localhost:8100)
+export ARTDAG_SERVER=http://localhost:8100
+
+# Set L2 server URL for auth (default: http://localhost:8200)
+export ARTDAG_L2=https://artdag.rose-ash.com
+
+# Or pass with commands
+./artdag.py --server http://localhost:8100 --l2 https://artdag.rose-ash.com <command>
+```
+
+## Authentication
+
+Most commands require authentication. Login credentials are stored locally in `~/.artdag/token.json`.
+
+```bash
+# Register a new account
+artdag register <username> [--email user@example.com]
+
+# Login
+artdag login <username>
+
+# Check current user
+artdag whoami
+
+# Logout
+artdag logout
+```
+
+## Commands Reference
+
+### Server & Stats
+
+```bash
+# Show server info
+artdag info
+
+# Show user stats (counts of runs, recipes, effects, media, storage)
+artdag stats
+
+# List known named assets
+artdag assets
+```
+
+### Runs
+
+```bash
+# List runs (with pagination)
+artdag runs [--limit N] [--offset N]
+
+# Start a run
+artdag run <recipe> <input_cid> [--name output_name] [--wait]
+
+# Get run status
+artdag status <run_id>
+
+# Get detailed run info
+artdag status <run_id> --plan       # Show execution plan with steps
+artdag status <run_id> --artifacts  # Show output artifacts
+artdag status <run_id> --analysis   # Show audio analysis data
+
+# Delete a run
+artdag delete-run <run_id> [--force]
+```
+
+### Recipes
+
+```bash
+# List recipes (with pagination)
+artdag recipes [--limit N] [--offset N]
+
+# Show recipe details
+artdag recipe <recipe_id>
+
+# Upload a recipe (YAML or S-expression)
+artdag upload-recipe <filepath>
+
+# Run a recipe with inputs
+artdag run-recipe <recipe_id> -i node_id:cid [--wait]
+
+# Delete a recipe
+artdag delete-recipe <recipe_id> [--force]
+```
+
+### Effects
+
+```bash
+# List effects (with pagination)
+artdag effects [--limit N] [--offset N]
+
+# Show effect details
+artdag effect <cid>
+
+# Show effect with source code
+artdag effect <cid> --source
+
+# Upload an effect (.py file)
+artdag upload-effect <filepath>
+```
+
+### Media / Cache
+
+```bash
+# List cached content (with pagination and type filter)
+artdag cache [--limit N] [--offset N] [--type all|image|video|audio]
+
+# View/download cached content
+artdag view <cid>                    # Show metadata (size, type, friendly name)
+artdag view <cid> --raw              # Get raw content info
+artdag view <cid> -o output.mp4     # Download raw file
+artdag view <cid> -o - | mpv -      # Pipe raw content to player
+
+# Upload file to cache and IPFS
+artdag upload <filepath>
+
+# Import local file to cache (local server only)
+artdag import <filepath>
+
+# View/update metadata
+artdag meta <cid>                              # View metadata
+artdag meta <cid> -d "Description"             # Set description
+artdag meta <cid> -t "tag1,tag2"               # Set tags
+artdag meta <cid> --publish "my-video"         # Publish to L2
+
+# Delete cached content
+artdag delete-cache <cid> [--force]
+```
+
+### Storage Providers
+
+```bash
+# List storage providers
+artdag storage list
+
+# Add a provider (interactive)
+artdag storage add <type> [--name friendly_name] [--capacity GB]
+# Types: pinata, web3storage, nftstorage, infura, filebase, storj, local
+
+# Test provider connectivity
+artdag storage test <id>
+
+# Delete a provider
+artdag storage delete <id> [--force]
+```
+
+### Folders & Collections
+
+```bash
+# Folders
+artdag folder list
+artdag folder create <path>
+artdag folder delete <path>
+
+# Collections
+artdag collection list
+artdag collection create <name>
+artdag collection delete <name>
+```
+
+### v2 API (3-Phase Execution)
+
+```bash
+# Generate execution plan
+artdag plan <recipe_file> -i name:cid [--features beats,energy] [--output plan.json]
+
+# Execute a plan
+artdag execute-plan <plan_file> [--wait]
+
+# Run recipe (plan + execute in one step)
+artdag run-v2 <recipe_file> -i name:cid [--wait]
+
+# Check v2 run status
+artdag run-status <run_id>
+```
+
+### Publishing to L2
+
+```bash
+# Publish a run output to L2
+artdag publish <run_id> <output_name>
+```
+
+### Data Management
+
+```bash
+# Clear all user data (preserves storage configs)
+artdag clear-data [--force]
+```
+
+## Example Workflows
+
+### Basic Rendering
+
+```bash
+# Login
+artdag login myuser
+
+# Check available assets
+artdag assets
+
+# Run an effect on an input
+artdag run dog cat --wait
+
+# View runs
+artdag runs
+
+# Download result
+artdag view <output_cid> -o result.mp4
+```
+
+### Recipe-Based Processing
+
+```bash
+# Upload a recipe
+artdag upload-recipe my-recipe.yaml
+
+# View recipes
+artdag recipes
+
+# Run with inputs
+artdag run-recipe <recipe_id> -i video:bafkrei... --wait
+
+# View run plan
+artdag status <run_id> --plan
+```
+
+### Managing Storage
+
+```bash
+# Add Pinata storage
+artdag storage add pinata --name "My Pinata"
+
+# Test connection
+artdag storage test 1
+
+# View all providers
+artdag storage list
+```
+
+### Browsing Media
+
+```bash
+# List all media
+artdag cache
+
+# Filter by type
+artdag cache --type video --limit 20
+
+# View with pagination
+artdag cache --offset 20 --limit 20
+```

artdag/client/requirements.txt

@@ -0,0 +1,3 @@
+click>=8.0.0
+requests>=2.31.0
+PyYAML>=6.0

artdag/client/test_gpu_effects.sexp

@@ -0,0 +1,38 @@
+;; GPU Effects Performance Test
+;; Tests rotation, zoom, hue-shift, ripple
+
+(stream "gpu_effects_test"
+  :fps 30
+  :width 1920
+  :height 1080
+  :seed 42
+
+  ;; Load primitives
+  (require-primitives "geometry")
+  (require-primitives "core")
+  (require-primitives "math")
+  (require-primitives "image")
+  (require-primitives "color_ops")
+
+  ;; Frame pipeline - test GPU effects
+  (frame
+    (let [;; Create a base gradient image
+          r (+ 0.5 (* 0.5 (math:sin (* t 1))))
+          g (+ 0.5 (* 0.5 (math:sin (* t 1.3))))
+          b (+ 0.5 (* 0.5 (math:sin (* t 1.7))))
+          color [(* r 255) (* g 255) (* b 255)]
+          base (image:make-image 1920 1080 color)
+
+          ;; Apply rotation (this is the main GPU bottleneck we optimized)
+          angle (* t 30)
+          rotated (geometry:rotate base angle)
+
+          ;; Apply hue shift
+          hue-shift (* 180 (math:sin (* t 0.5)))
+          hued (color_ops:hue-shift rotated hue-shift)
+
+          ;; Apply brightness based on time
+          brightness (+ 0.8 (* 0.4 (math:sin (* t 2))))
+          bright (color_ops:brightness hued brightness)]
+
+      bright)))

artdag/client/test_simple.sexp

@@ -0,0 +1,26 @@
+;; Simple Test - No external assets required
+;; Just generates a color gradient that changes over time
+
+(stream "simple_test"
+  :fps 30
+  :width 720
+  :height 720
+  :seed 42
+
+  ;; Load standard primitives
+  (require-primitives "geometry")
+  (require-primitives "core")
+  (require-primitives "math")
+  (require-primitives "image")
+  (require-primitives "color_ops")
+
+  ;; Frame pipeline - animated gradient
+  (frame
+    (let [;; Time-based color cycling (0-1 range)
+          r (+ 0.5 (* 0.5 (math:sin (* t 1))))
+          g (+ 0.5 (* 0.5 (math:sin (* t 1.3))))
+          b (+ 0.5 (* 0.5 (math:sin (* t 1.7))))
+          ;; Convert to 0-255 range and create solid color frame
+          color [(* r 255) (* g 255) (* b 255)]
+          frame (image:make-image 720 720 color)]
+      frame)))

artdag/common/README.md

@@ -0,0 +1,293 @@
+# artdag-common
+
+Shared components for Art-DAG L1 (celery) and L2 (activity-pub) servers.
+
+## Features
+
+- **Jinja2 Templating**: Unified template environment with shared base templates
+- **Reusable Components**: Cards, tables, pagination, DAG visualization, media preview
+- **Authentication Middleware**: Cookie and JWT token parsing
+- **Content Negotiation**: HTML/JSON/ActivityPub format detection
+- **Utility Functions**: Hash truncation, file size formatting, status colors
+
+## Installation
+
+```bash
+pip install -e /path/to/artdag-common
+
+# Or add to requirements.txt
+-e file:../common
+```
+
+## Quick Start
+
+```python
+from fastapi import FastAPI, Request
+from artdag_common import create_jinja_env, render
+
+app = FastAPI()
+
+# Initialize templates with app-specific directory
+templates = create_jinja_env("app/templates")
+
+@app.get("/")
+async def home(request: Request):
+    return render(templates, "home.html", request, title="Home")
+```
+
+## Package Structure
+
+```
+artdag_common/
+├── __init__.py              # Package exports
+├── constants.py             # CDN URLs, colors, configs
+├── rendering.py             # Jinja2 environment and helpers
+├── middleware/
+│   ├── auth.py              # Authentication utilities
+│   └── content_negotiation.py  # Accept header parsing
+├── models/
+│   ├── requests.py          # Shared request models
+│   └── responses.py         # Shared response models
+├── utils/
+│   ├── formatting.py        # Text/date formatting
+│   ├── media.py             # Media type detection
+│   └── pagination.py        # Pagination helpers
+└── templates/
+    ├── base.html            # Base layout template
+    └── components/
+        ├── badge.html       # Status/type badges
+        ├── card.html        # Info cards
+        ├── dag.html         # Cytoscape DAG visualization
+        ├── media_preview.html  # Video/image/audio preview
+        ├── pagination.html  # HTMX pagination
+        └── table.html       # Styled tables
+```
+
+## Jinja2 Templates
+
+### Base Template
+
+The `base.html` template provides:
+- Dark theme with Tailwind CSS
+- HTMX integration
+- Navigation slot
+- Content block
+- Optional Cytoscape.js block
+
+```html
+{% extends "base.html" %}
+
+{% block title %}My Page{% endblock %}
+
+{% block content %}
+<h1>Hello World</h1>
+{% endblock %}
+```
+
+### Reusable Components
+
+#### Card
+
+```html
+{% include "components/card.html" %}
+```
+
+```html
+<!-- Usage in your template -->
+<div class="...card styles...">
+    {% block card_title %}Title{% endblock %}
+    {% block card_content %}Content{% endblock %}
+</div>
+```
+
+#### Badge
+
+Status and type badges with appropriate colors:
+
+```html
+{% from "components/badge.html" import status_badge, type_badge %}
+
+{{ status_badge("completed") }}  <!-- Green -->
+{{ status_badge("failed") }}     <!-- Red -->
+{{ type_badge("video") }}
+```
+
+#### DAG Visualization
+
+Interactive Cytoscape.js graph:
+
+```html
+{% include "components/dag.html" %}
+```
+
+Requires passing `nodes` and `edges` data to template context.
+
+#### Media Preview
+
+Responsive media preview with format detection:
+
+```html
+{% include "components/media_preview.html" %}
+```
+
+Supports video, audio, and image formats.
+
+#### Pagination
+
+HTMX-powered infinite scroll pagination:
+
+```html
+{% include "components/pagination.html" %}
+```
+
+## Template Rendering
+
+### Full Page Render
+
+```python
+from artdag_common import render
+
+@app.get("/runs/{run_id}")
+async def run_detail(run_id: str, request: Request):
+    run = get_run(run_id)
+    return render(templates, "runs/detail.html", request, run=run)
+```
+
+### Fragment Render (HTMX)
+
+```python
+from artdag_common import render_fragment
+
+@app.get("/runs/{run_id}/status")
+async def run_status_fragment(run_id: str):
+    run = get_run(run_id)
+    html = render_fragment(templates, "components/status.html", status=run.status)
+    return HTMLResponse(html)
+```
+
+## Authentication Middleware
+
+### UserContext
+
+```python
+from artdag_common.middleware.auth import UserContext, get_user_from_cookie
+
+@app.get("/profile")
+async def profile(request: Request):
+    user = get_user_from_cookie(request)
+    if not user:
+        return RedirectResponse("/login")
+    return {"username": user.username, "actor_id": user.actor_id}
+```
+
+### Token Parsing
+
+```python
+from artdag_common.middleware.auth import get_user_from_header, decode_jwt_claims
+
+@app.get("/api/me")
+async def api_me(request: Request):
+    user = get_user_from_header(request)
+    if not user:
+        raise HTTPException(401, "Not authenticated")
+    return {"user": user.username}
+```
+
+## Content Negotiation
+
+Detect what response format the client wants:
+
+```python
+from artdag_common.middleware.content_negotiation import wants_html, wants_json, wants_activity_json
+
+@app.get("/users/{username}")
+async def user_profile(username: str, request: Request):
+    user = get_user(username)
+
+    if wants_activity_json(request):
+        return ActivityPubActor(user)
+    elif wants_json(request):
+        return user.dict()
+    else:
+        return render(templates, "users/profile.html", request, user=user)
+```
+
+## Constants
+
+### CDN URLs
+
+```python
+from artdag_common import TAILWIND_CDN, HTMX_CDN, CYTOSCAPE_CDN
+
+# Available in templates as globals:
+# {{ TAILWIND_CDN }}
+# {{ HTMX_CDN }}
+# {{ CYTOSCAPE_CDN }}
+```
+
+### Node Colors
+
+```python
+from artdag_common import NODE_COLORS
+
+# {
+#     "SOURCE": "#3b82f6",    # Blue
+#     "EFFECT": "#22c55e",    # Green
+#     "OUTPUT": "#a855f7",    # Purple
+#     "ANALYSIS": "#f59e0b",  # Amber
+#     "_LIST": "#6366f1",     # Indigo
+#     "default": "#6b7280",   # Gray
+# }
+```
+
+### Status Colors
+
+```python
+STATUS_COLORS = {
+    "completed": "bg-green-600",
+    "cached": "bg-blue-600",
+    "running": "bg-yellow-600",
+    "pending": "bg-gray-600",
+    "failed": "bg-red-600",
+}
+```
+
+## Custom Jinja2 Filters
+
+The following filters are available in all templates:
+
+| Filter | Usage | Description |
+|--------|-------|-------------|
+| `truncate_hash` | `{{ hash\|truncate_hash }}` | Shorten hash to 16 chars with ellipsis |
+| `format_size` | `{{ bytes\|format_size }}` | Format bytes as KB/MB/GB |
+| `status_color` | `{{ status\|status_color }}` | Get Tailwind class for status |
+
+Example:
+
+```html
+<span class="{{ run.status|status_color }}">
+    {{ run.status }}
+</span>
+
+<code>{{ content_hash|truncate_hash }}</code>
+
+<span>{{ file_size|format_size }}</span>
+```
+
+## Development
+
+```bash
+cd /root/art-dag/common
+
+# Install in development mode
+pip install -e .
+
+# Run tests
+pytest
+```
+
+## Dependencies
+
+- `fastapi>=0.100.0` - Web framework
+- `jinja2>=3.1.0` - Templating engine
+- `pydantic>=2.0.0` - Data validation

artdag/common/artdag_common/__init__.py

@@ -0,0 +1,18 @@
+"""
+Art-DAG Common Library
+
+Shared components for L1 (celery) and L2 (activity-pub) servers.
+"""
+
+from .constants import NODE_COLORS, TAILWIND_CDN, HTMX_CDN, CYTOSCAPE_CDN
+from .rendering import create_jinja_env, render, render_fragment
+
+__all__ = [
+    "NODE_COLORS",
+    "TAILWIND_CDN",
+    "HTMX_CDN",
+    "CYTOSCAPE_CDN",
+    "create_jinja_env",
+    "render",
+    "render_fragment",
+]

artdag/common/artdag_common/constants.py

@@ -0,0 +1,76 @@
+"""
+Shared constants for Art-DAG servers.
+"""
+
+# CDN URLs
+TAILWIND_CDN = "https://cdn.tailwindcss.com?plugins=typography"
+HTMX_CDN = "https://unpkg.com/htmx.org@1.9.10"
+CYTOSCAPE_CDN = "https://cdnjs.cloudflare.com/ajax/libs/cytoscape/3.28.1/cytoscape.min.js"
+DAGRE_CDN = "https://cdnjs.cloudflare.com/ajax/libs/dagre/0.8.5/dagre.min.js"
+CYTOSCAPE_DAGRE_CDN = "https://cdn.jsdelivr.net/npm/cytoscape-dagre@2.5.0/cytoscape-dagre.min.js"
+
+# Node colors for DAG visualization
+NODE_COLORS = {
+    "SOURCE": "#3b82f6",    # Blue - input sources
+    "EFFECT": "#22c55e",    # Green - processing effects
+    "OUTPUT": "#a855f7",    # Purple - final outputs
+    "ANALYSIS": "#f59e0b",  # Amber - analysis nodes
+    "_LIST": "#6366f1",     # Indigo - list aggregation
+    "default": "#6b7280",   # Gray - unknown types
+}
+
+# Status colors
+STATUS_COLORS = {
+    "completed": "bg-green-600",
+    "cached": "bg-blue-600",
+    "running": "bg-yellow-600",
+    "pending": "bg-gray-600",
+    "failed": "bg-red-600",
+}
+
+# Tailwind dark theme configuration
+TAILWIND_CONFIG = """
+<script>
+tailwind.config = {
+    darkMode: 'class',
+    theme: {
+        extend: {
+            colors: {
+                dark: {
+                    600: '#374151',
+                    700: '#1f2937',
+                    800: '#111827',
+                    900: '#030712',
+                }
+            }
+        }
+    }
+}
+</script>
+<style type="text/tailwindcss">
+@layer utilities {
+    .prose-invert {
+        --tw-prose-body: #d1d5db;
+        --tw-prose-headings: #f9fafb;
+        --tw-prose-lead: #9ca3af;
+        --tw-prose-links: #60a5fa;
+        --tw-prose-bold: #f9fafb;
+        --tw-prose-counters: #9ca3af;
+        --tw-prose-bullets: #6b7280;
+        --tw-prose-hr: #374151;
+        --tw-prose-quotes: #f3f4f6;
+        --tw-prose-quote-borders: #374151;
+        --tw-prose-captions: #9ca3af;
+        --tw-prose-code: #f9fafb;
+        --tw-prose-pre-code: #e5e7eb;
+        --tw-prose-pre-bg: #1f2937;
+        --tw-prose-th-borders: #4b5563;
+        --tw-prose-td-borders: #374151;
+    }
+}
+</style>
+"""
+
+# Default pagination settings
+DEFAULT_PAGE_SIZE = 20
+MAX_PAGE_SIZE = 100

artdag/common/artdag_common/fragments.py

@@ -0,0 +1,91 @@
+"""Fragment client for fetching HTML fragments from coop apps.
+
+Lightweight httpx-based client (no Quart dependency) for Art-DAG to consume
+coop app fragments like nav-tree, auth-menu, and cart-mini.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import os
+from typing import Sequence
+
+import httpx
+
+log = logging.getLogger(__name__)
+
+FRAGMENT_HEADER = "X-Fragment-Request"
+
+_client: httpx.AsyncClient | None = None
+_DEFAULT_TIMEOUT = 2.0
+
+
+def _get_client() -> httpx.AsyncClient:
+    global _client
+    if _client is None or _client.is_closed:
+        _client = httpx.AsyncClient(
+            timeout=httpx.Timeout(_DEFAULT_TIMEOUT),
+            follow_redirects=False,
+        )
+    return _client
+
+
+def _internal_url(app_name: str) -> str:
+    """Resolve internal base URL for a coop app.
+
+    Looks up ``INTERNAL_URL_{APP}`` first, falls back to ``http://{app}:8000``.
+    """
+    env_key = f"INTERNAL_URL_{app_name.upper()}"
+    return os.getenv(env_key, f"http://{app_name}:8000").rstrip("/")
+
+
+async def fetch_fragment(
+    app_name: str,
+    fragment_type: str,
+    *,
+    params: dict | None = None,
+    timeout: float = _DEFAULT_TIMEOUT,
+    required: bool = False,
+) -> str:
+    """Fetch an HTML fragment from a coop app.
+
+    Returns empty string on failure by default (required=False).
+    """
+    base = _internal_url(app_name)
+    url = f"{base}/internal/fragments/{fragment_type}"
+    try:
+        resp = await _get_client().get(
+            url,
+            params=params,
+            headers={FRAGMENT_HEADER: "1"},
+            timeout=timeout,
+        )
+        if resp.status_code == 200:
+            return resp.text
+        msg = f"Fragment {app_name}/{fragment_type} returned {resp.status_code}"
+        log.warning(msg)
+        if required:
+            raise RuntimeError(msg)
+        return ""
+    except RuntimeError:
+        raise
+    except Exception as exc:
+        msg = f"Fragment {app_name}/{fragment_type} failed: {exc}"
+        log.warning(msg)
+        if required:
+            raise RuntimeError(msg) from exc
+        return ""
+
+
+async def fetch_fragments(
+    requests: Sequence[tuple[str, str, dict | None]],
+    *,
+    timeout: float = _DEFAULT_TIMEOUT,
+    required: bool = False,
+) -> list[str]:
+    """Fetch multiple fragments concurrently."""
+    return list(await asyncio.gather(*(
+        fetch_fragment(app, ftype, params=params, timeout=timeout, required=required)
+        for app, ftype, params in requests
+    )))

artdag/common/artdag_common/middleware/__init__.py

@@ -0,0 +1,16 @@
+"""
+Middleware and FastAPI dependencies for Art-DAG servers.
+"""
+
+from .auth import UserContext, get_user_from_cookie, get_user_from_header, require_auth
+from .content_negotiation import wants_html, wants_json, ContentType
+
+__all__ = [
+    "UserContext",
+    "get_user_from_cookie",
+    "get_user_from_header",
+    "require_auth",
+    "wants_html",
+    "wants_json",
+    "ContentType",
+]

artdag/common/artdag_common/middleware/auth.py

@@ -0,0 +1,276 @@
+"""
+Authentication middleware and dependencies.
+
+Provides common authentication patterns for L1 and L2 servers.
+Each server can extend or customize these as needed.
+"""
+
+from dataclasses import dataclass
+from typing import Callable, Optional, Awaitable, Any
+import base64
+import json
+
+from fastapi import Request, HTTPException, Depends
+from fastapi.responses import RedirectResponse
+
+
+@dataclass
+class UserContext:
+    """User context extracted from authentication."""
+    username: str
+    actor_id: str  # Full actor ID like "@user@server.com"
+    token: Optional[str] = None
+    l2_server: Optional[str] = None  # L2 server URL for this user
+    email: Optional[str] = None  # User's email address
+
+    @property
+    def display_name(self) -> str:
+        """Get display name (username without @ prefix)."""
+        return self.username.lstrip("@")
+
+
+def get_user_from_cookie(request: Request) -> Optional[UserContext]:
+    """
+    Extract user context from session cookie.
+
+    Supports two cookie formats:
+    1. artdag_session: base64-encoded JSON {"username": "user", "actor_id": "@user@server.com"}
+    2. auth_token: raw JWT token (used by L1 servers)
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        UserContext if valid cookie found, None otherwise
+    """
+    # Try artdag_session cookie first (base64-encoded JSON)
+    cookie = request.cookies.get("artdag_session")
+    if cookie:
+        try:
+            data = json.loads(base64.b64decode(cookie))
+            username = data.get("username", "")
+            actor_id = data.get("actor_id", "")
+            if not actor_id and username:
+                actor_id = f"@{username}"
+            return UserContext(
+                username=username,
+                actor_id=actor_id,
+                email=data.get("email", ""),
+            )
+        except (json.JSONDecodeError, ValueError, KeyError):
+            pass
+
+    # Try auth_token cookie (raw JWT, used by L1)
+    token = request.cookies.get("auth_token")
+    if token:
+        claims = decode_jwt_claims(token)
+        if claims:
+            username = claims.get("username") or claims.get("sub", "")
+            actor_id = claims.get("actor_id") or claims.get("actor")
+            if not actor_id and username:
+                actor_id = f"@{username}"
+            return UserContext(
+                username=username,
+                actor_id=actor_id or "",
+                token=token,
+                email=claims.get("email", ""),
+            )
+
+    return None
+
+
+def get_user_from_header(request: Request) -> Optional[UserContext]:
+    """
+    Extract user context from Authorization header.
+
+    Supports:
+    - Bearer <token> format (JWT or opaque token)
+    - Basic <base64(user:pass)> format
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        UserContext if valid header found, None otherwise
+    """
+    auth_header = request.headers.get("Authorization", "")
+
+    if auth_header.startswith("Bearer "):
+        token = auth_header[7:]
+        # Attempt to decode JWT claims
+        claims = decode_jwt_claims(token)
+        if claims:
+            username = claims.get("username") or claims.get("sub", "")
+            actor_id = claims.get("actor_id") or claims.get("actor")
+            # Default actor_id to @username if not provided
+            if not actor_id and username:
+                actor_id = f"@{username}"
+            return UserContext(
+                username=username,
+                actor_id=actor_id or "",
+                token=token,
+            )
+
+    return None
+
+
+def decode_jwt_claims(token: str) -> Optional[dict]:
+    """
+    Decode JWT claims without verification.
+
+    This is useful for extracting user info from a token
+    when full verification is handled elsewhere.
+
+    Args:
+        token: JWT token string
+
+    Returns:
+        Claims dict if valid JWT format, None otherwise
+    """
+    try:
+        parts = token.split(".")
+        if len(parts) != 3:
+            return None
+
+        # Decode payload (second part)
+        payload = parts[1]
+        # Add padding if needed
+        padding = 4 - len(payload) % 4
+        if padding != 4:
+            payload += "=" * padding
+
+        return json.loads(base64.urlsafe_b64decode(payload))
+    except (json.JSONDecodeError, ValueError):
+        return None
+
+
+def create_auth_dependency(
+    token_validator: Optional[Callable[[str], Awaitable[Optional[dict]]]] = None,
+    allow_cookie: bool = True,
+    allow_header: bool = True,
+):
+    """
+    Create a customized auth dependency for a specific server.
+
+    Args:
+        token_validator: Optional async function to validate tokens with backend
+        allow_cookie: Whether to check cookies for auth
+        allow_header: Whether to check Authorization header
+
+    Returns:
+        FastAPI dependency function
+    """
+    async def get_current_user(request: Request) -> Optional[UserContext]:
+        ctx = None
+
+        # Try header first (API clients)
+        if allow_header:
+            ctx = get_user_from_header(request)
+            if ctx and token_validator:
+                # Validate token with backend
+                validated = await token_validator(ctx.token)
+                if not validated:
+                    ctx = None
+
+        # Fall back to cookie (browser)
+        if ctx is None and allow_cookie:
+            ctx = get_user_from_cookie(request)
+
+        return ctx
+
+    return get_current_user
+
+
+async def require_auth(request: Request) -> UserContext:
+    """
+    Dependency that requires authentication.
+
+    Raises HTTPException 401 if not authenticated.
+    Use with Depends() in route handlers.
+
+    Example:
+        @app.get("/protected")
+        async def protected_route(user: UserContext = Depends(require_auth)):
+            return {"user": user.username}
+    """
+    # Try header first
+    ctx = get_user_from_header(request)
+    if ctx is None:
+        ctx = get_user_from_cookie(request)
+
+    if ctx is None:
+        # Check Accept header to determine response type
+        accept = request.headers.get("accept", "")
+        if "text/html" in accept:
+            raise HTTPException(
+                status_code=302,
+                headers={"Location": "/login"}
+            )
+        raise HTTPException(
+            status_code=401,
+            detail="Authentication required"
+        )
+
+    return ctx
+
+
+def require_owner(resource_owner_field: str = "username"):
+    """
+    Dependency factory that requires the user to own the resource.
+
+    Args:
+        resource_owner_field: Field name on the resource that contains owner username
+
+    Returns:
+        Dependency function
+
+    Example:
+        @app.delete("/items/{item_id}")
+        async def delete_item(
+            item: Item = Depends(get_item),
+            user: UserContext = Depends(require_owner("created_by"))
+        ):
+            ...
+    """
+    async def check_ownership(
+        request: Request,
+        user: UserContext = Depends(require_auth),
+    ) -> UserContext:
+        # The actual ownership check must be done in the route
+        # after fetching the resource
+        return user
+
+    return check_ownership
+
+
+def set_auth_cookie(response: Any, user: UserContext, max_age: int = 86400 * 30) -> None:
+    """
+    Set authentication cookie on response.
+
+    Args:
+        response: FastAPI response object
+        user: User context to store
+        max_age: Cookie max age in seconds (default 30 days)
+    """
+    cookie_data = {
+        "username": user.username,
+        "actor_id": user.actor_id,
+    }
+    if user.email:
+        cookie_data["email"] = user.email
+    data = json.dumps(cookie_data)
+    cookie_value = base64.b64encode(data.encode()).decode()
+
+    response.set_cookie(
+        key="artdag_session",
+        value=cookie_value,
+        max_age=max_age,
+        httponly=True,
+        samesite="lax",
+        secure=True,  # Require HTTPS in production
+    )
+
+
+def clear_auth_cookie(response: Any) -> None:
+    """Clear authentication cookie."""
+    response.delete_cookie(key="artdag_session")

artdag/common/artdag_common/middleware/content_negotiation.py

@@ -0,0 +1,174 @@
+"""
+Content negotiation utilities.
+
+Helps determine what response format the client wants.
+"""
+
+from enum import Enum
+from typing import Optional
+
+from fastapi import Request
+
+
+class ContentType(Enum):
+    """Response content types."""
+    HTML = "text/html"
+    JSON = "application/json"
+    ACTIVITY_JSON = "application/activity+json"
+    XML = "application/xml"
+
+
+def wants_html(request: Request) -> bool:
+    """
+    Check if the client wants HTML response.
+
+    Returns True if:
+    - Accept header contains text/html
+    - Accept header contains application/xhtml+xml
+    - No Accept header (browser default)
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        True if HTML is preferred
+    """
+    accept = request.headers.get("accept", "")
+
+    # No accept header usually means browser
+    if not accept:
+        return True
+
+    # Check for HTML preferences
+    if "text/html" in accept:
+        return True
+    if "application/xhtml" in accept:
+        return True
+
+    return False
+
+
+def wants_json(request: Request) -> bool:
+    """
+    Check if the client wants JSON response.
+
+    Returns True if:
+    - Accept header contains application/json
+    - Accept header does NOT contain text/html
+    - Request has .json suffix (convention)
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        True if JSON is preferred
+    """
+    accept = request.headers.get("accept", "")
+
+    # Explicit JSON preference
+    if "application/json" in accept:
+        # But not if HTML is also requested (browsers often send both)
+        if "text/html" not in accept:
+            return True
+
+    # Check URL suffix convention
+    if request.url.path.endswith(".json"):
+        return True
+
+    return False
+
+
+def wants_activity_json(request: Request) -> bool:
+    """
+    Check if the client wants ActivityPub JSON-LD response.
+
+    Used for federation with other ActivityPub servers.
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        True if ActivityPub format is preferred
+    """
+    accept = request.headers.get("accept", "")
+
+    if "application/activity+json" in accept:
+        return True
+    if "application/ld+json" in accept:
+        return True
+
+    return False
+
+
+def get_preferred_type(request: Request) -> ContentType:
+    """
+    Determine the preferred content type from Accept header.
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        ContentType enum value
+    """
+    if wants_activity_json(request):
+        return ContentType.ACTIVITY_JSON
+    if wants_json(request):
+        return ContentType.JSON
+    return ContentType.HTML
+
+
+def is_htmx_request(request: Request) -> bool:
+    """
+    Check if this is an HTMX request (partial page update).
+
+    HTMX requests set the HX-Request header.
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        True if this is an HTMX request
+    """
+    return request.headers.get("HX-Request") == "true"
+
+
+def get_htmx_target(request: Request) -> Optional[str]:
+    """
+    Get the HTMX target element ID.
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        Target element ID or None
+    """
+    return request.headers.get("HX-Target")
+
+
+def get_htmx_trigger(request: Request) -> Optional[str]:
+    """
+    Get the HTMX trigger element ID.
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        Trigger element ID or None
+    """
+    return request.headers.get("HX-Trigger")
+
+
+def is_ios_request(request: Request) -> bool:
+    """
+    Check if request is from iOS device.
+
+    Useful for video format selection (iOS prefers MP4).
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        True if iOS user agent detected
+    """
+    user_agent = request.headers.get("user-agent", "").lower()
+    return "iphone" in user_agent or "ipad" in user_agent

artdag/common/artdag_common/models/__init__.py

@@ -0,0 +1,25 @@
+"""
+Shared Pydantic models for Art-DAG servers.
+"""
+
+from .requests import (
+    PaginationParams,
+    PublishRequest,
+    StorageConfigRequest,
+    MetadataUpdateRequest,
+)
+from .responses import (
+    PaginatedResponse,
+    ErrorResponse,
+    SuccessResponse,
+)
+
+__all__ = [
+    "PaginationParams",
+    "PublishRequest",
+    "StorageConfigRequest",
+    "MetadataUpdateRequest",
+    "PaginatedResponse",
+    "ErrorResponse",
+    "SuccessResponse",
+]

artdag/common/artdag_common/models/requests.py

@@ -0,0 +1,74 @@
+"""
+Request models shared across L1 and L2 servers.
+"""
+
+from typing import Optional, List, Dict, Any
+from pydantic import BaseModel, Field
+
+from ..constants import DEFAULT_PAGE_SIZE, MAX_PAGE_SIZE
+
+
+class PaginationParams(BaseModel):
+    """Common pagination parameters."""
+    page: int = Field(default=1, ge=1, description="Page number (1-indexed)")
+    limit: int = Field(
+        default=DEFAULT_PAGE_SIZE,
+        ge=1,
+        le=MAX_PAGE_SIZE,
+        description="Items per page"
+    )
+
+    @property
+    def offset(self) -> int:
+        """Calculate offset for database queries."""
+        return (self.page - 1) * self.limit
+
+
+class PublishRequest(BaseModel):
+    """Request to publish content to L2/storage."""
+    name: str = Field(..., min_length=1, max_length=255)
+    description: Optional[str] = Field(default=None, max_length=2000)
+    tags: List[str] = Field(default_factory=list)
+    storage_id: Optional[str] = Field(default=None, description="Target storage provider")
+
+
+class MetadataUpdateRequest(BaseModel):
+    """Request to update content metadata."""
+    name: Optional[str] = Field(default=None, max_length=255)
+    description: Optional[str] = Field(default=None, max_length=2000)
+    tags: Optional[List[str]] = Field(default=None)
+    metadata: Optional[Dict[str, Any]] = Field(default=None)
+
+
+class StorageConfigRequest(BaseModel):
+    """Request to configure a storage provider."""
+    provider_type: str = Field(..., description="Provider type (pinata, web3storage, local, etc.)")
+    name: str = Field(..., min_length=1, max_length=100)
+    api_key: Optional[str] = Field(default=None)
+    api_secret: Optional[str] = Field(default=None)
+    endpoint: Optional[str] = Field(default=None)
+    config: Optional[Dict[str, Any]] = Field(default_factory=dict)
+    is_default: bool = Field(default=False)
+
+
+class RecipeRunRequest(BaseModel):
+    """Request to run a recipe."""
+    recipe_id: str = Field(..., description="Recipe content hash or ID")
+    inputs: Dict[str, str] = Field(..., description="Map of input name to content hash")
+    features: List[str] = Field(
+        default=["beats", "energy"],
+        description="Analysis features to extract"
+    )
+
+
+class PlanRequest(BaseModel):
+    """Request to generate an execution plan."""
+    recipe_yaml: str = Field(..., description="Recipe YAML content")
+    input_hashes: Dict[str, str] = Field(..., description="Map of input name to content hash")
+    features: List[str] = Field(default=["beats", "energy"])
+
+
+class ExecutePlanRequest(BaseModel):
+    """Request to execute a generated plan."""
+    plan_json: str = Field(..., description="JSON-serialized execution plan")
+    run_id: Optional[str] = Field(default=None, description="Optional run ID for tracking")

artdag/common/artdag_common/models/responses.py

@@ -0,0 +1,96 @@
+"""
+Response models shared across L1 and L2 servers.
+"""
+
+from typing import Optional, List, Dict, Any, Generic, TypeVar
+from pydantic import BaseModel, Field
+
+T = TypeVar("T")
+
+
+class PaginatedResponse(BaseModel, Generic[T]):
+    """Generic paginated response."""
+    data: List[Any] = Field(default_factory=list)
+    pagination: Dict[str, Any] = Field(default_factory=dict)
+
+    @classmethod
+    def create(
+        cls,
+        items: List[Any],
+        page: int,
+        limit: int,
+        total: int,
+    ) -> "PaginatedResponse":
+        """Create a paginated response."""
+        return cls(
+            data=items,
+            pagination={
+                "page": page,
+                "limit": limit,
+                "total": total,
+                "has_more": page * limit < total,
+                "total_pages": (total + limit - 1) // limit,
+            }
+        )
+
+
+class ErrorResponse(BaseModel):
+    """Standard error response."""
+    error: str = Field(..., description="Error message")
+    detail: Optional[str] = Field(default=None, description="Detailed error info")
+    code: Optional[str] = Field(default=None, description="Error code")
+
+
+class SuccessResponse(BaseModel):
+    """Standard success response."""
+    success: bool = Field(default=True)
+    message: Optional[str] = Field(default=None)
+    data: Optional[Dict[str, Any]] = Field(default=None)
+
+
+class RunStatus(BaseModel):
+    """Run execution status."""
+    run_id: str
+    status: str = Field(..., description="pending, running, completed, failed")
+    recipe: Optional[str] = None
+    plan_id: Optional[str] = None
+    output_hash: Optional[str] = None
+    output_ipfs_cid: Optional[str] = None
+    total_steps: int = 0
+    cached_steps: int = 0
+    completed_steps: int = 0
+    error: Optional[str] = None
+
+
+class CacheItemResponse(BaseModel):
+    """Cached content item response."""
+    content_hash: str
+    media_type: Optional[str] = None
+    size: Optional[int] = None
+    name: Optional[str] = None
+    description: Optional[str] = None
+    tags: List[str] = Field(default_factory=list)
+    ipfs_cid: Optional[str] = None
+    created_at: Optional[str] = None
+
+
+class RecipeResponse(BaseModel):
+    """Recipe response."""
+    recipe_id: str
+    name: str
+    description: Optional[str] = None
+    inputs: List[Dict[str, Any]] = Field(default_factory=list)
+    outputs: List[str] = Field(default_factory=list)
+    node_count: int = 0
+    created_at: Optional[str] = None
+
+
+class StorageProviderResponse(BaseModel):
+    """Storage provider configuration response."""
+    storage_id: str
+    provider_type: str
+    name: str
+    is_default: bool = False
+    is_connected: bool = False
+    usage_bytes: Optional[int] = None
+    pin_count: int = 0

artdag/common/artdag_common/rendering.py

@@ -0,0 +1,160 @@
+"""
+Jinja2 template rendering system for Art-DAG servers.
+
+Provides a unified template environment that can load from:
+1. The shared artdag_common/templates directory
+2. App-specific template directories
+
+Usage:
+    from artdag_common import create_jinja_env, render
+
+    # In app initialization
+    templates = create_jinja_env("app/templates")
+
+    # In route handler
+    return render(templates, "runs/detail.html", request, run=run, user=user)
+"""
+
+from pathlib import Path
+from typing import Any, Optional, Union
+
+from fastapi import Request
+from fastapi.responses import HTMLResponse
+from jinja2 import Environment, ChoiceLoader, FileSystemLoader, PackageLoader, select_autoescape
+
+from .constants import (
+    TAILWIND_CDN,
+    HTMX_CDN,
+    CYTOSCAPE_CDN,
+    DAGRE_CDN,
+    CYTOSCAPE_DAGRE_CDN,
+    TAILWIND_CONFIG,
+    NODE_COLORS,
+    STATUS_COLORS,
+)
+
+
+def create_jinja_env(*template_dirs: Union[str, Path]) -> Environment:
+    """
+    Create a Jinja2 environment with the shared templates and optional app-specific dirs.
+
+    Args:
+        *template_dirs: Additional template directories to search (app-specific)
+
+    Returns:
+        Configured Jinja2 Environment
+
+    Example:
+        env = create_jinja_env("/app/templates", "/app/custom")
+    """
+    loaders = []
+
+    # Add app-specific directories first (higher priority)
+    for template_dir in template_dirs:
+        path = Path(template_dir)
+        if path.exists():
+            loaders.append(FileSystemLoader(str(path)))
+
+    # Add shared templates from this package (lower priority, fallback)
+    loaders.append(PackageLoader("artdag_common", "templates"))
+
+    env = Environment(
+        loader=ChoiceLoader(loaders),
+        autoescape=select_autoescape(["html", "xml"]),
+        trim_blocks=True,
+        lstrip_blocks=True,
+    )
+
+    # Add global context available to all templates
+    env.globals.update({
+        "TAILWIND_CDN": TAILWIND_CDN,
+        "HTMX_CDN": HTMX_CDN,
+        "CYTOSCAPE_CDN": CYTOSCAPE_CDN,
+        "DAGRE_CDN": DAGRE_CDN,
+        "CYTOSCAPE_DAGRE_CDN": CYTOSCAPE_DAGRE_CDN,
+        "TAILWIND_CONFIG": TAILWIND_CONFIG,
+        "NODE_COLORS": NODE_COLORS,
+        "STATUS_COLORS": STATUS_COLORS,
+    })
+
+    # Add custom filters
+    env.filters["truncate_hash"] = truncate_hash
+    env.filters["format_size"] = format_size
+    env.filters["status_color"] = status_color
+
+    return env
+
+
+def render(
+    env: Environment,
+    template_name: str,
+    request: Request,
+    status_code: int = 200,
+    **context: Any,
+) -> HTMLResponse:
+    """
+    Render a template to an HTMLResponse.
+
+    Args:
+        env: Jinja2 environment
+        template_name: Template file path (e.g., "runs/detail.html")
+        request: FastAPI request object
+        status_code: HTTP status code (default 200)
+        **context: Template context variables
+
+    Returns:
+        HTMLResponse with rendered content
+    """
+    template = env.get_template(template_name)
+    html = template.render(request=request, **context)
+    return HTMLResponse(html, status_code=status_code)
+
+
+def render_fragment(
+    env: Environment,
+    template_name: str,
+    **context: Any,
+) -> str:
+    """
+    Render a template fragment to a string (for HTMX partial updates).
+
+    Args:
+        env: Jinja2 environment
+        template_name: Template file path
+        **context: Template context variables
+
+    Returns:
+        Rendered HTML string
+    """
+    template = env.get_template(template_name)
+    return template.render(**context)
+
+
+# Custom Jinja2 filters
+
+def truncate_hash(value: str, length: int = 16) -> str:
+    """Truncate a hash to specified length with ellipsis."""
+    if not value:
+        return ""
+    if len(value) <= length:
+        return value
+    return f"{value[:length]}..."
+
+
+def format_size(size_bytes: Optional[int]) -> str:
+    """Format file size in human-readable form."""
+    if size_bytes is None:
+        return "Unknown"
+    if size_bytes < 1024:
+        return f"{size_bytes} B"
+    elif size_bytes < 1024 * 1024:
+        return f"{size_bytes / 1024:.1f} KB"
+    elif size_bytes < 1024 * 1024 * 1024:
+        return f"{size_bytes / (1024 * 1024):.1f} MB"
+    else:
+        return f"{size_bytes / (1024 * 1024 * 1024):.1f} GB"
+
+
+def status_color(status: str) -> str:
+    """Get Tailwind CSS class for a status."""
+    return STATUS_COLORS.get(status, STATUS_COLORS["pending"])

artdag/common/artdag_common/templates/_base.html

@@ -0,0 +1,96 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>{% block title %}Art-DAG{% endblock %}</title>
+
+    <!-- Tailwind CSS (same CDN as coop) -->
+    <script src="https://cdn.tailwindcss.com?plugins=typography"></script>
+    <script>
+    tailwind.config = {
+        theme: {
+            extend: {
+                colors: {
+                    dark: {
+                        600: '#374151',
+                        700: '#1f2937',
+                        800: '#111827',
+                        900: '#030712',
+                    }
+                }
+            }
+        }
+    }
+    </script>
+    <!-- HTMX -->
+    <script src="https://unpkg.com/htmx.org@2.0.8"></script>
+    <!-- Hyperscript (for nav-tree scrolling arrows) -->
+    <script src="https://unpkg.com/hyperscript.org@0.9.12"></script>
+    <!-- Font Awesome (for auth-menu + nav icons) -->
+    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css">
+
+    {% block head %}{% endblock %}
+
+    <style>
+        /* HTMX loading indicator */
+        .htmx-indicator { display: none; }
+        .htmx-request .htmx-indicator { display: inline-flex; }
+        img { max-width: 100%; height: auto; }
+        .no-scrollbar::-webkit-scrollbar { display: none; }
+        .no-scrollbar { -ms-overflow-style: none; scrollbar-width: none; }
+        .scrollbar-hide::-webkit-scrollbar { display: none; }
+        .scrollbar-hide { -ms-overflow-style: none; scrollbar-width: none; }
+    </style>
+    <script>
+        if (matchMedia('(hover: hover) and (pointer: fine)').matches) {
+            document.documentElement.classList.add('hover-capable');
+        }
+    </script>
+</head>
+<body class="bg-stone-50 text-stone-900 min-h-screen">
+    <div class="max-w-screen-2xl mx-auto py-1 px-1">
+    {% block header %}
+    {# Coop-style header: sky banner with title, nav-tree, auth-menu, cart-mini #}
+    <div class="w-full">
+      <div class="flex flex-col items-center md:flex-row justify-center md:justify-between w-full p-1 bg-sky-500">
+        <div class="w-full flex flex-row items-top">
+          {# Cart mini #}
+          {% block cart_mini %}{% endblock %}
+
+          {# Site title #}
+          <div class="font-bold text-5xl flex-1">
+            <a href="/" class="flex justify-center md:justify-start">
+              <h1>{% block brand %}Art-DAG{% endblock %}</h1>
+            </a>
+          </div>
+
+          {# Desktop nav: nav-tree + auth-menu #}
+          <nav class="hidden md:flex gap-4 text-sm ml-2 justify-end items-center flex-0">
+            {% block nav_tree %}{% endblock %}
+            {% block auth_menu %}{% endblock %}
+          </nav>
+        </div>
+      </div>
+      {# Mobile auth #}
+      <div class="block md:hidden text-md font-bold">
+        {% block auth_menu_mobile %}{% endblock %}
+      </div>
+    </div>
+    {% endblock %}
+
+    {# App-specific sub-nav (Runs, Recipes, Effects, etc.) #}
+    {% block sub_nav %}{% endblock %}
+
+    </div>{# close max-w-screen-2xl wrapper #}
+
+    <main class="bg-dark-800 text-gray-100 min-h-screen">
+      <div class="max-w-screen-2xl mx-auto px-4 py-4">
+        {% block content %}{% endblock %}
+      </div>
+    </main>
+
+    {% block footer %}{% endblock %}
+    {% block scripts %}{% endblock %}
+</body>
+</html>

artdag/common/artdag_common/templates/components/badge.html

@@ -0,0 +1,64 @@
+{#
+Badge component for status and type indicators.
+
+Usage:
+    {% from "components/badge.html" import badge, status_badge, type_badge %}
+
+    {{ badge("Active", "green") }}
+    {{ status_badge("completed") }}
+    {{ type_badge("EFFECT") }}
+#}
+
+{% macro badge(text, color="gray", class="") %}
+<span class="inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium bg-{{ color }}-600/20 text-{{ color }}-400 {{ class }}">
+    {{ text }}
+</span>
+{% endmacro %}
+
+{% macro status_badge(status, class="") %}
+{% set colors = {
+    "completed": "green",
+    "cached": "blue",
+    "running": "yellow",
+    "pending": "gray",
+    "failed": "red",
+    "active": "green",
+    "inactive": "gray",
+} %}
+{% set color = colors.get(status, "gray") %}
+<span class="inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium bg-{{ color }}-600/20 text-{{ color }}-400 {{ class }}">
+    {% if status == "running" %}
+    <svg class="animate-spin -ml-0.5 mr-1.5 h-3 w-3" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24">
+        <circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4"></circle>
+        <path class="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"></path>
+    </svg>
+    {% endif %}
+    {{ status | capitalize }}
+</span>
+{% endmacro %}
+
+{% macro type_badge(node_type, class="") %}
+{% set colors = {
+    "SOURCE": "blue",
+    "EFFECT": "green",
+    "OUTPUT": "purple",
+    "ANALYSIS": "amber",
+    "_LIST": "indigo",
+} %}
+{% set color = colors.get(node_type, "gray") %}
+<span class="inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium bg-{{ color }}-600/20 text-{{ color }}-400 {{ class }}">
+    {{ node_type }}
+</span>
+{% endmacro %}
+
+{% macro role_badge(role, class="") %}
+{% set colors = {
+    "input": "blue",
+    "output": "purple",
+    "intermediate": "gray",
+} %}
+{% set color = colors.get(role, "gray") %}
+<span class="inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium bg-{{ color }}-600/20 text-{{ color }}-400 {{ class }}">
+    {{ role | capitalize }}
+</span>
+{% endmacro %}

artdag/common/artdag_common/templates/components/card.html

@@ -0,0 +1,45 @@
+{#
+Card component for displaying information.
+
+Usage:
+    {% include "components/card.html" with title="Status", content="Active", class="col-span-2" %}
+
+Or as a block:
+    {% call card(title="Details") %}
+        <p>Card content here</p>
+    {% endcall %}
+#}
+
+{% macro card(title=None, class="") %}
+<div class="bg-dark-600 rounded-lg p-4 {{ class }}">
+    {% if title %}
+    <h3 class="text-sm font-medium text-gray-400 mb-2">{{ title }}</h3>
+    {% endif %}
+    <div class="text-white">
+        {{ caller() if caller else "" }}
+    </div>
+</div>
+{% endmacro %}
+
+{% macro stat_card(title, value, color="white", class="") %}
+<div class="bg-dark-600 rounded-lg p-4 text-center {{ class }}">
+    <div class="text-2xl font-bold text-{{ color }}-400">{{ value }}</div>
+    <div class="text-sm text-gray-400">{{ title }}</div>
+</div>
+{% endmacro %}
+
+{% macro info_card(title, items, class="") %}
+<div class="bg-dark-600 rounded-lg p-4 {{ class }}">
+    {% if title %}
+    <h3 class="text-sm font-medium text-gray-400 mb-3">{{ title }}</h3>
+    {% endif %}
+    <dl class="space-y-2">
+        {% for label, value in items %}
+        <div class="flex justify-between">
+            <dt class="text-gray-400">{{ label }}</dt>
+            <dd class="text-white font-mono text-sm">{{ value }}</dd>
+        </div>
+        {% endfor %}
+    </dl>
+</div>
+{% endmacro %}

artdag/common/artdag_common/templates/components/dag.html

@@ -0,0 +1,176 @@
+{#
+Cytoscape.js DAG visualization component.
+
+Usage:
+    {% from "components/dag.html" import dag_container, dag_scripts, dag_legend %}
+
+    {# In head block #}
+    {{ dag_scripts() }}
+
+    {# In content #}
+    {{ dag_container(id="plan-dag", height="400px") }}
+    {{ dag_legend() }}
+
+    {# In scripts block #}
+    <script>
+        initDag('plan-dag', {{ nodes | tojson }}, {{ edges | tojson }});
+    </script>
+#}
+
+{% macro dag_scripts() %}
+<script src="{{ CYTOSCAPE_CDN }}"></script>
+<script src="{{ DAGRE_CDN }}"></script>
+<script src="{{ CYTOSCAPE_DAGRE_CDN }}"></script>
+<script>
+// Global Cytoscape instance for WebSocket updates
+window.artdagCy = null;
+
+function initDag(containerId, nodes, edges) {
+    const nodeColors = {{ NODE_COLORS | tojson }};
+
+    window.artdagCy = cytoscape({
+        container: document.getElementById(containerId),
+        elements: {
+            nodes: nodes,
+            edges: edges
+        },
+        style: [
+            {
+                selector: 'node',
+                style: {
+                    'label': 'data(label)',
+                    'text-valign': 'center',
+                    'text-halign': 'center',
+                    'background-color': function(ele) {
+                        return nodeColors[ele.data('nodeType')] || nodeColors['default'];
+                    },
+                    'color': '#fff',
+                    'font-size': '10px',
+                    'width': 80,
+                    'height': 40,
+                    'shape': 'round-rectangle',
+                    'text-wrap': 'wrap',
+                    'text-max-width': '70px',
+                }
+            },
+            {
+                selector: 'node[status="cached"], node[status="completed"]',
+                style: {
+                    'border-width': 3,
+                    'border-color': '#22c55e'
+                }
+            },
+            {
+                selector: 'node[status="running"]',
+                style: {
+                    'border-width': 3,
+                    'border-color': '#eab308',
+                    'border-style': 'dashed'
+                }
+            },
+            {
+                selector: 'node:selected',
+                style: {
+                    'border-width': 3,
+                    'border-color': '#3b82f6'
+                }
+            },
+            {
+                selector: 'edge',
+                style: {
+                    'width': 2,
+                    'line-color': '#6b7280',
+                    'target-arrow-color': '#6b7280',
+                    'target-arrow-shape': 'triangle',
+                    'curve-style': 'bezier'
+                }
+            }
+        ],
+        layout: {
+            name: 'dagre',
+            rankDir: 'TB',
+            nodeSep: 50,
+            rankSep: 80,
+            padding: 20
+        },
+        userZoomingEnabled: true,
+        userPanningEnabled: true,
+        boxSelectionEnabled: false
+    });
+
+    // Click handler for node details
+    window.artdagCy.on('tap', 'node', function(evt) {
+        const node = evt.target;
+        const data = node.data();
+        showNodeDetails(data);
+    });
+
+    return window.artdagCy;
+}
+
+function showNodeDetails(data) {
+    const panel = document.getElementById('node-details');
+    if (!panel) return;
+
+    panel.innerHTML = `
+        <h4 class="font-medium text-white mb-2">${data.label || data.id}</h4>
+        <dl class="space-y-1 text-sm">
+            <div class="flex justify-between">
+                <dt class="text-gray-400">Type</dt>
+                <dd class="text-white">${data.nodeType || 'Unknown'}</dd>
+            </div>
+            <div class="flex justify-between">
+                <dt class="text-gray-400">Status</dt>
+                <dd class="text-white">${data.status || 'pending'}</dd>
+            </div>
+            ${data.cacheId ? `
+            <div class="flex justify-between">
+                <dt class="text-gray-400">Cache ID</dt>
+                <dd class="text-white font-mono text-xs">${data.cacheId.substring(0, 16)}...</dd>
+            </div>
+            ` : ''}
+            ${data.level !== undefined ? `
+            <div class="flex justify-between">
+                <dt class="text-gray-400">Level</dt>
+                <dd class="text-white">${data.level}</dd>
+            </div>
+            ` : ''}
+        </dl>
+    `;
+    panel.classList.remove('hidden');
+}
+
+// Future WebSocket support: update node status in real-time
+function updateNodeStatus(stepId, status, cacheId) {
+    if (!window.artdagCy) return;
+    const node = window.artdagCy.getElementById(stepId);
+    if (node && node.length > 0) {
+        node.data('status', status);
+        if (cacheId) {
+            node.data('cacheId', cacheId);
+        }
+    }
+}
+</script>
+{% endmacro %}
+
+{% macro dag_container(id="dag-container", height="400px", class="") %}
+<div id="{{ id }}" class="w-full bg-dark-700 rounded-lg {{ class }}" style="height: {{ height }};"></div>
+<div id="node-details" class="hidden mt-4 p-4 bg-dark-600 rounded-lg"></div>
+{% endmacro %}
+
+{% macro dag_legend(node_types=None) %}
+{% set types = node_types or ["SOURCE", "EFFECT", "_LIST"] %}
+<div class="flex gap-4 text-sm flex-wrap mt-4">
+    {% for type in types %}
+    <span class="flex items-center gap-2">
+        <span class="w-4 h-4 rounded" style="background-color: {{ NODE_COLORS.get(type, NODE_COLORS.default) }}"></span>
+        {{ type }}
+    </span>
+    {% endfor %}
+    <span class="flex items-center gap-2">
+        <span class="w-4 h-4 rounded border-2 border-green-500 bg-dark-600"></span>
+        Cached
+    </span>
+</div>
+{% endmacro %}

artdag/common/artdag_common/templates/components/media_preview.html

@@ -0,0 +1,98 @@
+{#
+Media preview component for videos, images, and audio.
+
+Usage:
+    {% from "components/media_preview.html" import media_preview, video_player, image_preview, audio_player %}
+
+    {{ media_preview(content_hash, media_type, title="Preview") }}
+    {{ video_player(src="/cache/abc123/mp4", poster="/cache/abc123/thumb") }}
+#}
+
+{% macro media_preview(content_hash, media_type, title=None, class="", show_download=True) %}
+<div class="bg-dark-600 rounded-lg overflow-hidden {{ class }}">
+    {% if title %}
+    <div class="px-4 py-2 border-b border-dark-500">
+        <h3 class="text-sm font-medium text-gray-400">{{ title }}</h3>
+    </div>
+    {% endif %}
+
+    <div class="aspect-video bg-dark-700 flex items-center justify-center">
+        {% if media_type == "video" %}
+        {{ video_player("/cache/" + content_hash + "/mp4") }}
+        {% elif media_type == "image" %}
+        {{ image_preview("/cache/" + content_hash + "/raw") }}
+        {% elif media_type == "audio" %}
+        {{ audio_player("/cache/" + content_hash + "/raw") }}
+        {% else %}
+        <div class="text-gray-400 text-center p-4">
+            <svg class="w-12 h-12 mx-auto mb-2" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+                <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 12h6m-6 4h6m2 5H7a2 2 0 01-2-2V5a2 2 0 012-2h5.586a1 1 0 01.707.293l5.414 5.414a1 1 0 01.293.707V19a2 2 0 01-2 2z"/>
+            </svg>
+            <p>Preview not available</p>
+        </div>
+        {% endif %}
+    </div>
+
+    {% if show_download %}
+    <div class="px-4 py-2 border-t border-dark-500">
+        <a href="/cache/{{ content_hash }}/raw" download
+           class="text-blue-400 hover:text-blue-300 text-sm">
+            Download original
+        </a>
+    </div>
+    {% endif %}
+</div>
+{% endmacro %}
+
+{% macro video_player(src, poster=None, autoplay=False, muted=True, loop=False, class="") %}
+<video
+    class="w-full h-full object-contain {{ class }}"
+    controls
+    playsinline
+    {% if poster %}poster="{{ poster }}"{% endif %}
+    {% if autoplay %}autoplay{% endif %}
+    {% if muted %}muted{% endif %}
+    {% if loop %}loop{% endif %}
+>
+    <source src="{{ src }}" type="video/mp4">
+    Your browser does not support the video tag.
+</video>
+{% endmacro %}
+
+{% macro image_preview(src, alt="", class="") %}
+<img
+    src="{{ src }}"
+    alt="{{ alt }}"
+    class="w-full h-full object-contain {{ class }}"
+    loading="lazy"
+>
+{% endmacro %}
+
+{% macro audio_player(src, class="") %}
+<div class="w-full px-4 {{ class }}">
+    <audio controls class="w-full">
+        <source src="{{ src }}">
+        Your browser does not support the audio element.
+    </audio>
+</div>
+{% endmacro %}
+
+{% macro thumbnail(content_hash, media_type, size="w-24 h-24", class="") %}
+<div class="bg-dark-700 rounded {{ size }} flex items-center justify-center overflow-hidden {{ class }}">
+    {% if media_type == "image" %}
+    <img src="/cache/{{ content_hash }}/raw" alt="" class="w-full h-full object-cover" loading="lazy">
+    {% elif media_type == "video" %}
+    <svg class="w-8 h-8 text-gray-400" fill="currentColor" viewBox="0 0 24 24">
+        <path d="M8 5v14l11-7z"/>
+    </svg>
+    {% elif media_type == "audio" %}
+    <svg class="w-8 h-8 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+        <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 19V6l12-3v13M9 19c0 1.105-1.343 2-3 2s-3-.895-3-2 1.343-2 3-2 3 .895 3 2zm12-3c0 1.105-1.343 2-3 2s-3-.895-3-2 1.343-2 3-2 3 .895 3 2zM9 10l12-3"/>
+    </svg>
+    {% else %}
+    <svg class="w-8 h-8 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+        <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 12h6m-6 4h6m2 5H7a2 2 0 01-2-2V5a2 2 0 012-2h5.586a1 1 0 01.707.293l5.414 5.414a1 1 0 01.293.707V19a2 2 0 01-2 2z"/>
+    </svg>
+    {% endif %}
+</div>
+{% endmacro %}

artdag/common/artdag_common/templates/components/pagination.html

@@ -0,0 +1,82 @@
+{#
+Pagination component with HTMX infinite scroll support.
+
+Usage:
+    {% from "components/pagination.html" import infinite_scroll_trigger, page_links %}
+
+    {# Infinite scroll (HTMX) #}
+    {{ infinite_scroll_trigger(url="/items?page=2", colspan=3, has_more=True) }}
+
+    {# Traditional pagination #}
+    {{ page_links(current_page=1, total_pages=5, base_url="/items") }}
+#}
+
+{% macro infinite_scroll_trigger(url, colspan=1, has_more=True, target=None) %}
+{% if has_more %}
+<tr hx-get="{{ url }}"
+    hx-trigger="revealed"
+    hx-swap="afterend"
+    {% if target %}hx-target="{{ target }}"{% endif %}
+    class="htmx-indicator-row">
+    <td colspan="{{ colspan }}" class="text-center py-4">
+        <span class="text-gray-400 htmx-indicator">
+            <svg class="animate-spin h-5 w-5 inline mr-2" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24">
+                <circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4"></circle>
+                <path class="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"></path>
+            </svg>
+            Loading more...
+        </span>
+    </td>
+</tr>
+{% endif %}
+{% endmacro %}
+
+{% macro page_links(current_page, total_pages, base_url, class="") %}
+<nav class="flex items-center justify-center space-x-2 {{ class }}">
+    {# Previous button #}
+    {% if current_page > 1 %}
+    <a href="{{ base_url }}?page={{ current_page - 1 }}"
+       class="px-3 py-2 rounded-lg bg-dark-600 text-gray-300 hover:bg-dark-500 transition-colors">
+        &larr; Previous
+    </a>
+    {% else %}
+    <span class="px-3 py-2 rounded-lg bg-dark-700 text-gray-500 cursor-not-allowed">
+        &larr; Previous
+    </span>
+    {% endif %}
+
+    {# Page numbers #}
+    <div class="flex items-center space-x-1">
+        {% for page in range(1, total_pages + 1) %}
+            {% if page == current_page %}
+            <span class="px-3 py-2 rounded-lg bg-blue-600 text-white">{{ page }}</span>
+            {% elif page == 1 or page == total_pages or (page >= current_page - 2 and page <= current_page + 2) %}
+            <a href="{{ base_url }}?page={{ page }}"
+               class="px-3 py-2 rounded-lg bg-dark-600 text-gray-300 hover:bg-dark-500 transition-colors">
+                {{ page }}
+            </a>
+            {% elif page == current_page - 3 or page == current_page + 3 %}
+            <span class="px-2 text-gray-500">...</span>
+            {% endif %}
+        {% endfor %}
+    </div>
+
+    {# Next button #}
+    {% if current_page < total_pages %}
+    <a href="{{ base_url }}?page={{ current_page + 1 }}"
+       class="px-3 py-2 rounded-lg bg-dark-600 text-gray-300 hover:bg-dark-500 transition-colors">
+        Next &rarr;
+    </a>
+    {% else %}
+    <span class="px-3 py-2 rounded-lg bg-dark-700 text-gray-500 cursor-not-allowed">
+        Next &rarr;
+    </span>
+    {% endif %}
+</nav>
+{% endmacro %}
+
+{% macro page_info(page, limit, total) %}
+<div class="text-sm text-gray-400">
+    Showing {{ (page - 1) * limit + 1 }}-{{ [page * limit, total] | min }} of {{ total }}
+</div>
+{% endmacro %}

artdag/common/artdag_common/templates/components/table.html

@@ -0,0 +1,51 @@
+{#
+Table component with dark theme styling.
+
+Usage:
+    {% from "components/table.html" import table, table_row %}
+
+    {% call table(columns=["Name", "Status", "Actions"]) %}
+        {% for item in items %}
+        {{ table_row([item.name, item.status, actions_html]) }}
+        {% endfor %}
+    {% endcall %}
+#}
+
+{% macro table(columns, class="", id="") %}
+<div class="overflow-x-auto {{ class }}" {% if id %}id="{{ id }}"{% endif %}>
+    <table class="w-full text-sm">
+        <thead class="text-gray-400 border-b border-dark-600">
+            <tr>
+                {% for col in columns %}
+                <th class="text-left py-3 px-4 font-medium">{{ col }}</th>
+                {% endfor %}
+            </tr>
+        </thead>
+        <tbody class="divide-y divide-dark-600">
+            {{ caller() }}
+        </tbody>
+    </table>
+</div>
+{% endmacro %}
+
+{% macro table_row(cells, class="", href=None) %}
+<tr class="hover:bg-dark-600/50 transition-colors {{ class }}">
+    {% for cell in cells %}
+    <td class="py-3 px-4">
+        {% if href and loop.first %}
+        <a href="{{ href }}" class="text-blue-400 hover:text-blue-300">{{ cell }}</a>
+        {% else %}
+        {{ cell | safe }}
+        {% endif %}
+    </td>
+    {% endfor %}
+</tr>
+{% endmacro %}
+
+{% macro empty_row(colspan, message="No items found") %}
+<tr>
+    <td colspan="{{ colspan }}" class="py-8 text-center text-gray-400">
+        {{ message }}
+    </td>
+</tr>
+{% endmacro %}

artdag/common/artdag_common/utils/__init__.py

@@ -0,0 +1,19 @@
+"""
+Utility functions shared across Art-DAG servers.
+"""
+
+from .pagination import paginate, get_pagination_params
+from .media import detect_media_type, get_media_extension, is_streamable
+from .formatting import format_date, format_size, truncate_hash, format_duration
+
+__all__ = [
+    "paginate",
+    "get_pagination_params",
+    "detect_media_type",
+    "get_media_extension",
+    "is_streamable",
+    "format_date",
+    "format_size",
+    "truncate_hash",
+    "format_duration",
+]

artdag/common/artdag_common/utils/formatting.py

@@ -0,0 +1,165 @@
+"""
+Formatting utilities for display.
+"""
+
+from datetime import datetime
+from typing import Optional, Union
+
+
+def format_date(
+    value: Optional[Union[str, datetime]],
+    length: int = 10,
+    include_time: bool = False,
+) -> str:
+    """
+    Format a date/datetime for display.
+
+    Args:
+        value: Date string or datetime object
+        length: Length to truncate to (default 10 for YYYY-MM-DD)
+        include_time: Whether to include time portion
+
+    Returns:
+        Formatted date string
+    """
+    if value is None:
+        return ""
+
+    if isinstance(value, str):
+        # Parse ISO format string
+        try:
+            if "T" in value:
+                dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
+            else:
+                return value[:length]
+        except ValueError:
+            return value[:length]
+    else:
+        dt = value
+
+    if include_time:
+        return dt.strftime("%Y-%m-%d %H:%M")
+    return dt.strftime("%Y-%m-%d")
+
+
+def format_size(size_bytes: Optional[int]) -> str:
+    """
+    Format file size in human-readable form.
+
+    Args:
+        size_bytes: Size in bytes
+
+    Returns:
+        Human-readable size string (e.g., "1.5 MB")
+    """
+    if size_bytes is None:
+        return "Unknown"
+    if size_bytes < 0:
+        return "Unknown"
+    if size_bytes == 0:
+        return "0 B"
+
+    units = ["B", "KB", "MB", "GB", "TB"]
+    unit_index = 0
+    size = float(size_bytes)
+
+    while size >= 1024 and unit_index < len(units) - 1:
+        size /= 1024
+        unit_index += 1
+
+    if unit_index == 0:
+        return f"{int(size)} {units[unit_index]}"
+    return f"{size:.1f} {units[unit_index]}"
+
+
+def truncate_hash(value: str, length: int = 16, suffix: str = "...") -> str:
+    """
+    Truncate a hash or long string with ellipsis.
+
+    Args:
+        value: String to truncate
+        length: Maximum length before truncation
+        suffix: Suffix to add when truncated
+
+    Returns:
+        Truncated string
+    """
+    if not value:
+        return ""
+    if len(value) <= length:
+        return value
+    return f"{value[:length]}{suffix}"
+
+
+def format_duration(seconds: Optional[float]) -> str:
+    """
+    Format duration in human-readable form.
+
+    Args:
+        seconds: Duration in seconds
+
+    Returns:
+        Human-readable duration string (e.g., "2m 30s")
+    """
+    if seconds is None or seconds < 0:
+        return "Unknown"
+
+    if seconds < 1:
+        return f"{int(seconds * 1000)}ms"
+
+    if seconds < 60:
+        return f"{seconds:.1f}s"
+
+    minutes = int(seconds // 60)
+    remaining_seconds = int(seconds % 60)
+
+    if minutes < 60:
+        if remaining_seconds:
+            return f"{minutes}m {remaining_seconds}s"
+        return f"{minutes}m"
+
+    hours = minutes // 60
+    remaining_minutes = minutes % 60
+
+    if remaining_minutes:
+        return f"{hours}h {remaining_minutes}m"
+    return f"{hours}h"
+
+
+def format_count(count: int) -> str:
+    """
+    Format a count with abbreviation for large numbers.
+
+    Args:
+        count: Number to format
+
+    Returns:
+        Formatted string (e.g., "1.2K", "3.5M")
+    """
+    if count < 1000:
+        return str(count)
+    if count < 1000000:
+        return f"{count / 1000:.1f}K"
+    if count < 1000000000:
+        return f"{count / 1000000:.1f}M"
+    return f"{count / 1000000000:.1f}B"
+
+
+def format_percentage(value: float, decimals: int = 1) -> str:
+    """
+    Format a percentage value.
+
+    Args:
+        value: Percentage value (0-100 or 0-1)
+        decimals: Number of decimal places
+
+    Returns:
+        Formatted percentage string
+    """
+    # Assume 0-1 if less than 1
+    if value <= 1:
+        value *= 100
+
+    if decimals == 0:
+        return f"{int(value)}%"
+    return f"{value:.{decimals}f}%"

artdag/common/artdag_common/utils/media.py

@@ -0,0 +1,166 @@
+"""
+Media type detection and handling utilities.
+"""
+
+from pathlib import Path
+from typing import Optional
+import mimetypes
+
+# Initialize mimetypes database
+mimetypes.init()
+
+# Media type categories
+VIDEO_TYPES = {"video/mp4", "video/webm", "video/quicktime", "video/x-msvideo", "video/avi"}
+IMAGE_TYPES = {"image/jpeg", "image/png", "image/gif", "image/webp", "image/svg+xml"}
+AUDIO_TYPES = {"audio/mpeg", "audio/wav", "audio/ogg", "audio/flac", "audio/aac", "audio/mp3"}
+
+# File extension mappings
+EXTENSION_TO_CATEGORY = {
+    # Video
+    ".mp4": "video",
+    ".webm": "video",
+    ".mov": "video",
+    ".avi": "video",
+    ".mkv": "video",
+    # Image
+    ".jpg": "image",
+    ".jpeg": "image",
+    ".png": "image",
+    ".gif": "image",
+    ".webp": "image",
+    ".svg": "image",
+    # Audio
+    ".mp3": "audio",
+    ".wav": "audio",
+    ".ogg": "audio",
+    ".flac": "audio",
+    ".aac": "audio",
+    ".m4a": "audio",
+}
+
+
+def detect_media_type(path: Path) -> str:
+    """
+    Detect the media category for a file.
+
+    Args:
+        path: Path to the file
+
+    Returns:
+        Category string: "video", "image", "audio", or "unknown"
+    """
+    if not path:
+        return "unknown"
+
+    # Try extension first
+    ext = path.suffix.lower()
+    if ext in EXTENSION_TO_CATEGORY:
+        return EXTENSION_TO_CATEGORY[ext]
+
+    # Try mimetypes
+    mime_type, _ = mimetypes.guess_type(str(path))
+    if mime_type:
+        if mime_type in VIDEO_TYPES or mime_type.startswith("video/"):
+            return "video"
+        if mime_type in IMAGE_TYPES or mime_type.startswith("image/"):
+            return "image"
+        if mime_type in AUDIO_TYPES or mime_type.startswith("audio/"):
+            return "audio"
+
+    return "unknown"
+
+
+def get_mime_type(path: Path) -> str:
+    """
+    Get the MIME type for a file.
+
+    Args:
+        path: Path to the file
+
+    Returns:
+        MIME type string or "application/octet-stream"
+    """
+    mime_type, _ = mimetypes.guess_type(str(path))
+    return mime_type or "application/octet-stream"
+
+
+def get_media_extension(media_type: str) -> str:
+    """
+    Get the typical file extension for a media type.
+
+    Args:
+        media_type: Media category or MIME type
+
+    Returns:
+        File extension with dot (e.g., ".mp4")
+    """
+    if media_type == "video":
+        return ".mp4"
+    if media_type == "image":
+        return ".png"
+    if media_type == "audio":
+        return ".mp3"
+
+    # Try as MIME type
+    ext = mimetypes.guess_extension(media_type)
+    return ext or ""
+
+
+def is_streamable(path: Path) -> bool:
+    """
+    Check if a file type is streamable (video/audio).
+
+    Args:
+        path: Path to the file
+
+    Returns:
+        True if the file can be streamed
+    """
+    media_type = detect_media_type(path)
+    return media_type in ("video", "audio")
+
+
+def needs_conversion(path: Path, target_format: str = "mp4") -> bool:
+    """
+    Check if a video file needs format conversion.
+
+    Args:
+        path: Path to the file
+        target_format: Target format (default mp4)
+
+    Returns:
+        True if conversion is needed
+    """
+    media_type = detect_media_type(path)
+    if media_type != "video":
+        return False
+
+    ext = path.suffix.lower().lstrip(".")
+    return ext != target_format
+
+
+def get_video_src(
+    content_hash: str,
+    original_path: Optional[Path] = None,
+    is_ios: bool = False,
+) -> str:
+    """
+    Get the appropriate video source URL.
+
+    For iOS devices, prefer MP4 format.
+
+    Args:
+        content_hash: Content hash for the video
+        original_path: Optional original file path
+        is_ios: Whether the client is iOS
+
+    Returns:
+        URL path for the video source
+    """
+    if is_ios:
+        return f"/cache/{content_hash}/mp4"
+
+    if original_path and original_path.suffix.lower() in (".mp4", ".webm"):
+        return f"/cache/{content_hash}/raw"
+
+    return f"/cache/{content_hash}/mp4"

artdag/common/artdag_common/utils/pagination.py

@@ -0,0 +1,85 @@
+"""
+Pagination utilities.
+"""
+
+from typing import List, Any, Tuple, Optional
+
+from fastapi import Request
+
+from ..constants import DEFAULT_PAGE_SIZE, MAX_PAGE_SIZE
+
+
+def get_pagination_params(request: Request) -> Tuple[int, int]:
+    """
+    Extract pagination parameters from request query string.
+
+    Args:
+        request: FastAPI request
+
+    Returns:
+        Tuple of (page, limit)
+    """
+    try:
+        page = int(request.query_params.get("page", 1))
+        page = max(1, page)
+    except ValueError:
+        page = 1
+
+    try:
+        limit = int(request.query_params.get("limit", DEFAULT_PAGE_SIZE))
+        limit = max(1, min(limit, MAX_PAGE_SIZE))
+    except ValueError:
+        limit = DEFAULT_PAGE_SIZE
+
+    return page, limit
+
+
+def paginate(
+    items: List[Any],
+    page: int = 1,
+    limit: int = DEFAULT_PAGE_SIZE,
+) -> Tuple[List[Any], dict]:
+    """
+    Paginate a list of items.
+
+    Args:
+        items: Full list of items
+        page: Page number (1-indexed)
+        limit: Items per page
+
+    Returns:
+        Tuple of (paginated items, pagination info dict)
+    """
+    total = len(items)
+    start = (page - 1) * limit
+    end = start + limit
+
+    paginated = items[start:end]
+
+    return paginated, {
+        "page": page,
+        "limit": limit,
+        "total": total,
+        "has_more": end < total,
+        "total_pages": (total + limit - 1) // limit if total > 0 else 1,
+    }
+
+
+def calculate_offset(page: int, limit: int) -> int:
+    """Calculate database offset from page and limit."""
+    return (page - 1) * limit
+
+
+def build_pagination_info(
+    page: int,
+    limit: int,
+    total: int,
+) -> dict:
+    """Build pagination info dictionary."""
+    return {
+        "page": page,
+        "limit": limit,
+        "total": total,
+        "has_more": page * limit < total,
+        "total_pages": (total + limit - 1) // limit if total > 0 else 1,
+    }

artdag/common/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "artdag-common"
+version = "0.1.3"
+description = "Shared components for Art-DAG L1 and L2 servers"
+requires-python = ">=3.10"
+dependencies = [
+    "fastapi>=0.100.0",
+    "jinja2>=3.1.0",
+    "pydantic>=2.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0.0",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["artdag_common"]

artdag/core/.gitignore

@@ -0,0 +1,47 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# Environment
+.env
+.venv
+env/
+venv/
+
+# Private keys (ActivityPub secrets)
+.cache/
+
+# Test outputs
+test_cache/
+test_plan_output.json
+analysis.json
+plan.json
+plan_with_analysis.json

artdag/core/README.md

@@ -0,0 +1,110 @@
+# artdag
+
+Content-addressed DAG execution engine with ActivityPub ownership.
+
+## Features
+
+- **Content-addressed nodes**: `node_id = SHA3-256(type + config + inputs)` for automatic deduplication
+- **Quantum-resistant hashing**: SHA-3 throughout for future-proof integrity
+- **ActivityPub ownership**: Cryptographically signed ownership claims
+- **Federated identity**: `@user@artdag.rose-ash.com` style identities
+- **Pluggable executors**: Register custom node types
+- **Built-in video primitives**: SOURCE, SEGMENT, RESIZE, TRANSFORM, SEQUENCE, MUX, BLEND
+
+## Installation
+
+```bash
+pip install -e .
+```
+
+### Optional: External Effect Tools
+
+Some effects can use external tools for better performance:
+
+**Pixelsort** (glitch art pixel sorting):
+```bash
+# Rust CLI (recommended - fast)
+cargo install --git https://github.com/Void-ux/pixelsort.git pixelsort
+
+# Or Python CLI
+pip install git+https://github.com/Blotz/pixelsort-cli
+```
+
+**Datamosh** (video glitch/corruption):
+```bash
+# FFglitch (recommended)
+./scripts/install-ffglitch.sh
+
+# Or Python CLI
+pip install git+https://github.com/tiberiuiancu/datamoshing
+```
+
+Check available tools:
+```bash
+python -m artdag.sexp.external_tools
+```
+
+## Quick Start
+
+```python
+from artdag import Engine, DAGBuilder, Registry
+from artdag.activitypub import OwnershipManager
+
+# Create ownership manager
+manager = OwnershipManager("./my_registry")
+
+# Create your identity
+actor = manager.create_actor("alice", "Alice")
+print(f"Created: {actor.handle}")  # @alice@artdag.rose-ash.com
+
+# Register an asset with ownership
+asset, activity = manager.register_asset(
+    actor=actor,
+    name="my_image",
+    path="/path/to/image.jpg",
+    tags=["photo", "art"],
+)
+print(f"Owned: {asset.name} (hash: {asset.content_hash})")
+
+# Build and execute a DAG
+engine = Engine("./cache")
+builder = DAGBuilder()
+
+source = builder.source(str(asset.path))
+resized = builder.resize(source, width=1920, height=1080)
+builder.set_output(resized)
+
+result = engine.execute(builder.build())
+print(f"Output: {result.output_path}")
+```
+
+## Architecture
+
+```
+artdag/
+├── dag.py           # Node, DAG, DAGBuilder
+├── cache.py         # Content-addressed file cache
+├── executor.py      # Base executor + registry
+├── engine.py        # DAG execution engine
+├── activitypub/     # Identity + ownership
+│   ├── actor.py     # Actor identity with RSA keys
+│   ├── activity.py  # Create, Announce activities
+│   ├── signatures.py # RSA signing/verification
+│   └── ownership.py # Links actors to assets
+├── nodes/           # Built-in executors
+│   ├── source.py    # SOURCE
+│   ├── transform.py # SEGMENT, RESIZE, TRANSFORM
+│   ├── compose.py   # SEQUENCE, LAYER, MUX, BLEND
+│   └── effect.py    # EFFECT (identity, etc.)
+└── effects/         # Effect implementations
+    └── identity/    # The foundational identity effect
+```
+
+## Related Repos
+
+- **Registry**: https://git.rose-ash.com/art-dag/registry - Asset registry with ownership proofs
+- **Recipes**: https://git.rose-ash.com/art-dag/recipes - DAG recipes using effects
+
+## License
+
+MIT

artdag/core/artdag/__init__.py

@@ -0,0 +1,61 @@
+# artdag - Content-addressed DAG execution engine with ActivityPub ownership
+#
+# A standalone execution engine that processes directed acyclic graphs (DAGs)
+# where each node represents an operation. Nodes are content-addressed for
+# automatic caching and deduplication.
+#
+# Core concepts:
+# - Node: An operation with type, config, and inputs
+# - DAG: A graph of nodes with a designated output node
+# - Executor: Implements the actual operation for a node type
+# - Engine: Executes DAGs by resolving dependencies and running executors
+
+from .dag import Node, DAG, DAGBuilder, NodeType
+from .cache import Cache, CacheEntry
+from .executor import Executor, register_executor, get_executor
+from .engine import Engine
+from .registry import Registry, Asset
+from .activities import Activity, ActivityStore, ActivityManager, make_is_shared_fn
+
+# Analysis and planning modules (optional, require extra dependencies)
+try:
+    from .analysis import Analyzer, AnalysisResult
+except ImportError:
+    Analyzer = None
+    AnalysisResult = None
+
+try:
+    from .planning import RecipePlanner, ExecutionPlan, ExecutionStep
+except ImportError:
+    RecipePlanner = None
+    ExecutionPlan = None
+    ExecutionStep = None
+
+__all__ = [
+    # Core
+    "Node",
+    "DAG",
+    "DAGBuilder",
+    "NodeType",
+    "Cache",
+    "CacheEntry",
+    "Executor",
+    "register_executor",
+    "get_executor",
+    "Engine",
+    "Registry",
+    "Asset",
+    "Activity",
+    "ActivityStore",
+    "ActivityManager",
+    "make_is_shared_fn",
+    # Analysis (optional)
+    "Analyzer",
+    "AnalysisResult",
+    # Planning (optional)
+    "RecipePlanner",
+    "ExecutionPlan",
+    "ExecutionStep",
+]
+
+__version__ = "0.1.0"

artdag/core/artdag/activities.py

@@ -0,0 +1,371 @@
+# artdag/activities.py
+"""
+Persistent activity (job) tracking for cache management.
+
+Activities represent executions of DAGs. They track:
+- Input node IDs (sources)
+- Output node ID (terminal node)
+- Intermediate node IDs (everything in between)
+
+This enables deletion rules:
+- Shared items (ActivityPub published) cannot be deleted
+- Inputs/outputs of activities cannot be deleted
+- Intermediates can be deleted (reconstructible)
+- Activities can only be discarded if no items are shared
+"""
+
+import json
+import logging
+import time
+import uuid
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Callable, Dict, List, Optional, Set
+
+from .cache import Cache, CacheEntry
+from .dag import DAG
+
+logger = logging.getLogger(__name__)
+
+
+def make_is_shared_fn(activitypub_store: "ActivityStore") -> Callable[[str], bool]:
+    """
+    Create an is_shared function from an ActivityPub ActivityStore.
+
+    Args:
+        activitypub_store: The ActivityPub activity store
+            (from artdag.activitypub.activity)
+
+    Returns:
+        Function that checks if a cid has been published
+    """
+    def is_shared(cid: str) -> bool:
+        activities = activitypub_store.find_by_object_hash(cid)
+        return any(a.activity_type == "Create" for a in activities)
+    return is_shared
+
+
+@dataclass
+class Activity:
+    """
+    A recorded execution of a DAG.
+
+    Tracks which cache entries are inputs, outputs, and intermediates
+    to enforce deletion rules.
+    """
+    activity_id: str
+    input_ids: List[str]        # Source node cache IDs
+    output_id: str              # Terminal node cache ID
+    intermediate_ids: List[str] # Everything in between
+    created_at: float
+    status: str = "completed"   # pending|running|completed|failed
+    dag_snapshot: Optional[Dict[str, Any]] = None  # Serialized DAG for reconstruction
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "activity_id": self.activity_id,
+            "input_ids": self.input_ids,
+            "output_id": self.output_id,
+            "intermediate_ids": self.intermediate_ids,
+            "created_at": self.created_at,
+            "status": self.status,
+            "dag_snapshot": self.dag_snapshot,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "Activity":
+        return cls(
+            activity_id=data["activity_id"],
+            input_ids=data["input_ids"],
+            output_id=data["output_id"],
+            intermediate_ids=data["intermediate_ids"],
+            created_at=data["created_at"],
+            status=data.get("status", "completed"),
+            dag_snapshot=data.get("dag_snapshot"),
+        )
+
+    @classmethod
+    def from_dag(cls, dag: DAG, activity_id: str = None) -> "Activity":
+        """
+        Create an Activity from a DAG.
+
+        Classifies nodes as inputs, output, or intermediates.
+        """
+        if activity_id is None:
+            activity_id = str(uuid.uuid4())
+
+        # Find input nodes (nodes with no inputs - sources)
+        input_ids = []
+        for node_id, node in dag.nodes.items():
+            if not node.inputs:
+                input_ids.append(node_id)
+
+        # Output is the terminal node
+        output_id = dag.output_id
+
+        # Intermediates are everything else
+        intermediate_ids = []
+        for node_id in dag.nodes:
+            if node_id not in input_ids and node_id != output_id:
+                intermediate_ids.append(node_id)
+
+        return cls(
+            activity_id=activity_id,
+            input_ids=sorted(input_ids),
+            output_id=output_id,
+            intermediate_ids=sorted(intermediate_ids),
+            created_at=time.time(),
+            status="completed",
+            dag_snapshot=dag.to_dict(),
+        )
+
+    @property
+    def all_node_ids(self) -> List[str]:
+        """All node IDs involved in this activity."""
+        return self.input_ids + [self.output_id] + self.intermediate_ids
+
+
+class ActivityStore:
+    """
+    Persistent storage for activities.
+
+    Provides methods to check deletion eligibility and perform deletions.
+    """
+
+    def __init__(self, store_dir: Path | str):
+        self.store_dir = Path(store_dir)
+        self.store_dir.mkdir(parents=True, exist_ok=True)
+        self._activities: Dict[str, Activity] = {}
+        self._load()
+
+    def _index_path(self) -> Path:
+        return self.store_dir / "activities.json"
+
+    def _load(self):
+        """Load activities from disk."""
+        index_path = self._index_path()
+        if index_path.exists():
+            try:
+                with open(index_path) as f:
+                    data = json.load(f)
+                self._activities = {
+                    a["activity_id"]: Activity.from_dict(a)
+                    for a in data.get("activities", [])
+                }
+            except (json.JSONDecodeError, KeyError) as e:
+                logger.warning(f"Failed to load activities: {e}")
+                self._activities = {}
+
+    def _save(self):
+        """Save activities to disk."""
+        data = {
+            "version": "1.0",
+            "activities": [a.to_dict() for a in self._activities.values()],
+        }
+        with open(self._index_path(), "w") as f:
+            json.dump(data, f, indent=2)
+
+    def add(self, activity: Activity) -> None:
+        """Add an activity."""
+        self._activities[activity.activity_id] = activity
+        self._save()
+
+    def get(self, activity_id: str) -> Optional[Activity]:
+        """Get an activity by ID."""
+        return self._activities.get(activity_id)
+
+    def remove(self, activity_id: str) -> bool:
+        """Remove an activity record (does not delete cache entries)."""
+        if activity_id not in self._activities:
+            return False
+        del self._activities[activity_id]
+        self._save()
+        return True
+
+    def list(self) -> List[Activity]:
+        """List all activities."""
+        return list(self._activities.values())
+
+    def find_by_input_ids(self, input_ids: List[str]) -> List[Activity]:
+        """Find activities with the same inputs (for UI grouping)."""
+        sorted_inputs = sorted(input_ids)
+        return [
+            a for a in self._activities.values()
+            if sorted(a.input_ids) == sorted_inputs
+        ]
+
+    def find_using_node(self, node_id: str) -> List[Activity]:
+        """Find all activities that reference a node ID."""
+        return [
+            a for a in self._activities.values()
+            if node_id in a.all_node_ids
+        ]
+
+    def __len__(self) -> int:
+        return len(self._activities)
+
+
+class ActivityManager:
+    """
+    Manages activities and cache deletion with sharing rules.
+
+    Deletion rules:
+    1. Shared items (ActivityPub published) cannot be deleted
+    2. Inputs/outputs of activities cannot be deleted
+    3. Intermediates can be deleted (reconstructible)
+    4. Activities can only be discarded if no items are shared
+    """
+
+    def __init__(
+        self,
+        cache: Cache,
+        activity_store: ActivityStore,
+        is_shared_fn: Callable[[str], bool],
+    ):
+        """
+        Args:
+            cache: The L1 cache
+            activity_store: Activity persistence
+            is_shared_fn: Function that checks if a cid is shared
+                          (published via ActivityPub)
+        """
+        self.cache = cache
+        self.activities = activity_store
+        self._is_shared = is_shared_fn
+
+    def record_activity(self, dag: DAG) -> Activity:
+        """Record a completed DAG execution as an activity."""
+        activity = Activity.from_dag(dag)
+        self.activities.add(activity)
+        return activity
+
+    def is_shared(self, node_id: str) -> bool:
+        """Check if a cache entry is shared (published via ActivityPub)."""
+        entry = self.cache.get_entry(node_id)
+        if not entry or not entry.cid:
+            return False
+        return self._is_shared(entry.cid)
+
+    def can_delete_cache_entry(self, node_id: str) -> bool:
+        """
+        Check if a cache entry can be deleted.
+
+        Returns False if:
+        - Entry is shared (ActivityPub published)
+        - Entry is an input or output of any activity
+        """
+        # Check if shared
+        if self.is_shared(node_id):
+            return False
+
+        # Check if it's an input or output of any activity
+        for activity in self.activities.list():
+            if node_id in activity.input_ids:
+                return False
+            if node_id == activity.output_id:
+                return False
+
+        # It's either an intermediate or orphaned - can delete
+        return True
+
+    def can_discard_activity(self, activity_id: str) -> bool:
+        """
+        Check if an activity can be discarded.
+
+        Returns False if any cache entry (input, output, or intermediate)
+        is shared via ActivityPub.
+        """
+        activity = self.activities.get(activity_id)
+        if not activity:
+            return False
+
+        # Check if any item is shared
+        for node_id in activity.all_node_ids:
+            if self.is_shared(node_id):
+                return False
+
+        return True
+
+    def discard_activity(self, activity_id: str) -> bool:
+        """
+        Discard an activity and delete its intermediate cache entries.
+
+        Returns False if the activity cannot be discarded (has shared items).
+
+        When discarded:
+        - Intermediate cache entries are deleted
+        - The activity record is removed
+        - Inputs remain (may be used by other activities)
+        - Output is deleted if orphaned (not shared, not used elsewhere)
+        """
+        if not self.can_discard_activity(activity_id):
+            return False
+
+        activity = self.activities.get(activity_id)
+        if not activity:
+            return False
+
+        output_id = activity.output_id
+        intermediate_ids = list(activity.intermediate_ids)
+
+        # Remove the activity record first
+        self.activities.remove(activity_id)
+
+        # Delete intermediates
+        for node_id in intermediate_ids:
+            self.cache.remove(node_id)
+            logger.debug(f"Deleted intermediate: {node_id}")
+
+        # Check if output is now orphaned
+        if self._is_orphaned(output_id) and not self.is_shared(output_id):
+            self.cache.remove(output_id)
+            logger.debug(f"Deleted orphaned output: {output_id}")
+
+        # Inputs remain - they may be used by other activities
+        # But check if any are orphaned now
+        for input_id in activity.input_ids:
+            if self._is_orphaned(input_id) and not self.is_shared(input_id):
+                self.cache.remove(input_id)
+                logger.debug(f"Deleted orphaned input: {input_id}")
+
+        return True
+
+    def _is_orphaned(self, node_id: str) -> bool:
+        """Check if a node is not referenced by any activity."""
+        for activity in self.activities.list():
+            if node_id in activity.all_node_ids:
+                return False
+        return True
+
+    def get_deletable_entries(self) -> List[CacheEntry]:
+        """Get all cache entries that can be deleted."""
+        deletable = []
+        for entry in self.cache.list_entries():
+            if self.can_delete_cache_entry(entry.node_id):
+                deletable.append(entry)
+        return deletable
+
+    def get_discardable_activities(self) -> List[Activity]:
+        """Get all activities that can be discarded."""
+        return [
+            a for a in self.activities.list()
+            if self.can_discard_activity(a.activity_id)
+        ]
+
+    def cleanup_intermediates(self) -> int:
+        """
+        Delete all intermediate cache entries.
+
+        Intermediates are safe to delete as they can be reconstructed
+        from inputs using the DAG.
+
+        Returns:
+            Number of entries deleted
+        """
+        deleted = 0
+        for activity in self.activities.list():
+            for node_id in activity.intermediate_ids:
+                if self.cache.has(node_id):
+                    self.cache.remove(node_id)
+                    deleted += 1
+        return deleted

artdag/core/artdag/activitypub/__init__.py

@@ -0,0 +1,33 @@
+# primitive/activitypub/__init__.py
+"""
+ActivityPub implementation for Art DAG.
+
+Provides decentralized identity and ownership for assets.
+Domain: artdag.rose-ash.com
+
+Core concepts:
+- Actor: A user identity with cryptographic keys
+- Object: An asset (image, video, etc.)
+- Activity: An action (Create, Announce, Like, etc.)
+- Signature: Cryptographic proof of authorship
+"""
+
+from .actor import Actor, ActorStore
+from .activity import Activity, CreateActivity, ActivityStore
+from .signatures import sign_activity, verify_signature, verify_activity_ownership
+from .ownership import OwnershipManager, OwnershipRecord
+
+__all__ = [
+    "Actor",
+    "ActorStore",
+    "Activity",
+    "CreateActivity",
+    "ActivityStore",
+    "sign_activity",
+    "verify_signature",
+    "verify_activity_ownership",
+    "OwnershipManager",
+    "OwnershipRecord",
+]
+
+DOMAIN = "artdag.rose-ash.com"

artdag/core/artdag/activitypub/activity.py

@@ -0,0 +1,203 @@
+# primitive/activitypub/activity.py
+"""
+ActivityPub Activity types.
+
+Activities represent actions taken by actors on objects.
+Key activity types for Art DAG:
+- Create: Actor creates/claims ownership of an object
+- Announce: Actor shares/boosts an object
+- Like: Actor endorses an object
+"""
+
+import json
+import time
+import uuid
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from .actor import Actor, DOMAIN
+
+
+def _generate_id() -> str:
+    """Generate unique activity ID."""
+    return str(uuid.uuid4())
+
+
+@dataclass
+class Activity:
+    """
+    Base ActivityPub Activity.
+
+    Attributes:
+        activity_id: Unique identifier
+        activity_type: Type (Create, Announce, Like, etc.)
+        actor_id: ID of the actor performing the activity
+        object_data: The object of the activity
+        published: ISO timestamp
+        signature: Cryptographic signature (added after signing)
+    """
+    activity_id: str
+    activity_type: str
+    actor_id: str
+    object_data: Dict[str, Any]
+    published: str = field(default_factory=lambda: time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()))
+    signature: Optional[Dict[str, Any]] = None
+
+    def to_activitypub(self) -> Dict[str, Any]:
+        """Return ActivityPub JSON-LD representation."""
+        activity = {
+            "@context": "https://www.w3.org/ns/activitystreams",
+            "type": self.activity_type,
+            "id": f"https://{DOMAIN}/activities/{self.activity_id}",
+            "actor": self.actor_id,
+            "object": self.object_data,
+            "published": self.published,
+        }
+        if self.signature:
+            activity["signature"] = self.signature
+        return activity
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Serialize for storage."""
+        return {
+            "activity_id": self.activity_id,
+            "activity_type": self.activity_type,
+            "actor_id": self.actor_id,
+            "object_data": self.object_data,
+            "published": self.published,
+            "signature": self.signature,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "Activity":
+        """Deserialize from storage."""
+        return cls(
+            activity_id=data["activity_id"],
+            activity_type=data["activity_type"],
+            actor_id=data["actor_id"],
+            object_data=data["object_data"],
+            published=data.get("published", ""),
+            signature=data.get("signature"),
+        )
+
+
+@dataclass
+class CreateActivity(Activity):
+    """
+    Create activity - establishes ownership of an object.
+
+    Used when an actor creates or claims an asset.
+    """
+    activity_type: str = field(default="Create", init=False)
+
+    @classmethod
+    def for_asset(
+        cls,
+        actor: Actor,
+        asset_name: str,
+        cid: str,
+        asset_type: str = "Image",
+        metadata: Dict[str, Any] = None,
+    ) -> "CreateActivity":
+        """
+        Create a Create activity for an asset.
+
+        Args:
+            actor: The actor claiming ownership
+            asset_name: Name of the asset
+            cid: SHA-3 hash of the asset content
+            asset_type: ActivityPub object type (Image, Video, Audio, etc.)
+            metadata: Additional metadata
+
+        Returns:
+            CreateActivity establishing ownership
+        """
+        object_data = {
+            "type": asset_type,
+            "name": asset_name,
+            "id": f"https://{DOMAIN}/objects/{cid}",
+            "contentHash": {
+                "algorithm": "sha3-256",
+                "value": cid,
+            },
+            "attributedTo": actor.id,
+        }
+        if metadata:
+            object_data["metadata"] = metadata
+
+        return cls(
+            activity_id=_generate_id(),
+            actor_id=actor.id,
+            object_data=object_data,
+        )
+
+
+class ActivityStore:
+    """
+    Persistent storage for activities.
+
+    Activities are stored as an append-only log for auditability.
+    """
+
+    def __init__(self, store_dir: Path | str):
+        self.store_dir = Path(store_dir)
+        self.store_dir.mkdir(parents=True, exist_ok=True)
+        self._activities: List[Activity] = []
+        self._load()
+
+    def _log_path(self) -> Path:
+        return self.store_dir / "activities.json"
+
+    def _load(self):
+        """Load activities from disk."""
+        log_path = self._log_path()
+        if log_path.exists():
+            with open(log_path) as f:
+                data = json.load(f)
+            self._activities = [
+                Activity.from_dict(a) for a in data.get("activities", [])
+            ]
+
+    def _save(self):
+        """Save activities to disk."""
+        data = {
+            "version": "1.0",
+            "activities": [a.to_dict() for a in self._activities],
+        }
+        with open(self._log_path(), "w") as f:
+            json.dump(data, f, indent=2)
+
+    def add(self, activity: Activity) -> None:
+        """Add an activity to the log."""
+        self._activities.append(activity)
+        self._save()
+
+    def get(self, activity_id: str) -> Optional[Activity]:
+        """Get an activity by ID."""
+        for a in self._activities:
+            if a.activity_id == activity_id:
+                return a
+        return None
+
+    def list(self) -> List[Activity]:
+        """List all activities."""
+        return list(self._activities)
+
+    def find_by_actor(self, actor_id: str) -> List[Activity]:
+        """Find activities by actor."""
+        return [a for a in self._activities if a.actor_id == actor_id]
+
+    def find_by_object_hash(self, cid: str) -> List[Activity]:
+        """Find activities referencing an object by hash."""
+        results = []
+        for a in self._activities:
+            obj_hash = a.object_data.get("contentHash", {})
+            if isinstance(obj_hash, dict) and obj_hash.get("value") == cid:
+                results.append(a)
+            elif a.object_data.get("contentHash") == cid:
+                results.append(a)
+        return results
+
+    def __len__(self) -> int:
+        return len(self._activities)

artdag/core/artdag/activitypub/actor.py

@@ -0,0 +1,206 @@
+# primitive/activitypub/actor.py
+"""
+ActivityPub Actor management.
+
+An Actor is an identity with:
+- Username and display name
+- RSA key pair for signing
+- ActivityPub-compliant JSON-LD representation
+"""
+
+import json
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Dict, Optional
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import rsa, padding
+
+DOMAIN = "artdag.rose-ash.com"
+
+
+def _generate_keypair() -> tuple[bytes, bytes]:
+    """Generate RSA key pair for signing."""
+    private_key = rsa.generate_private_key(
+        public_exponent=65537,
+        key_size=2048,
+    )
+    private_pem = private_key.private_bytes(
+        encoding=serialization.Encoding.PEM,
+        format=serialization.PrivateFormat.PKCS8,
+        encryption_algorithm=serialization.NoEncryption(),
+    )
+    public_pem = private_key.public_key().public_bytes(
+        encoding=serialization.Encoding.PEM,
+        format=serialization.PublicFormat.SubjectPublicKeyInfo,
+    )
+    return private_pem, public_pem
+
+
+@dataclass
+class Actor:
+    """
+    An ActivityPub Actor (identity).
+
+    Attributes:
+        username: Unique username (e.g., "giles")
+        display_name: Human-readable name
+        public_key: PEM-encoded public key
+        private_key: PEM-encoded private key (kept secret)
+        created_at: Timestamp of creation
+    """
+    username: str
+    display_name: str
+    public_key: bytes
+    private_key: bytes
+    created_at: float = field(default_factory=time.time)
+    domain: str = DOMAIN
+
+    @property
+    def id(self) -> str:
+        """ActivityPub actor ID (URL)."""
+        return f"https://{self.domain}/users/{self.username}"
+
+    @property
+    def handle(self) -> str:
+        """Fediverse handle."""
+        return f"@{self.username}@{self.domain}"
+
+    @property
+    def inbox(self) -> str:
+        """ActivityPub inbox URL."""
+        return f"{self.id}/inbox"
+
+    @property
+    def outbox(self) -> str:
+        """ActivityPub outbox URL."""
+        return f"{self.id}/outbox"
+
+    @property
+    def key_id(self) -> str:
+        """Key ID for HTTP Signatures."""
+        return f"{self.id}#main-key"
+
+    def to_activitypub(self) -> Dict[str, Any]:
+        """Return ActivityPub JSON-LD representation."""
+        return {
+            "@context": [
+                "https://www.w3.org/ns/activitystreams",
+                "https://w3id.org/security/v1",
+            ],
+            "type": "Person",
+            "id": self.id,
+            "preferredUsername": self.username,
+            "name": self.display_name,
+            "inbox": self.inbox,
+            "outbox": self.outbox,
+            "publicKey": {
+                "id": self.key_id,
+                "owner": self.id,
+                "publicKeyPem": self.public_key.decode("utf-8"),
+            },
+        }
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Serialize for storage."""
+        return {
+            "username": self.username,
+            "display_name": self.display_name,
+            "public_key": self.public_key.decode("utf-8"),
+            "private_key": self.private_key.decode("utf-8"),
+            "created_at": self.created_at,
+            "domain": self.domain,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "Actor":
+        """Deserialize from storage."""
+        return cls(
+            username=data["username"],
+            display_name=data["display_name"],
+            public_key=data["public_key"].encode("utf-8"),
+            private_key=data["private_key"].encode("utf-8"),
+            created_at=data.get("created_at", time.time()),
+            domain=data.get("domain", DOMAIN),
+        )
+
+    @classmethod
+    def create(cls, username: str, display_name: str = None) -> "Actor":
+        """Create a new actor with generated keys."""
+        private_pem, public_pem = _generate_keypair()
+        return cls(
+            username=username,
+            display_name=display_name or username,
+            public_key=public_pem,
+            private_key=private_pem,
+        )
+
+
+class ActorStore:
+    """
+    Persistent storage for actors.
+
+    Structure:
+        store_dir/
+            actors.json       # Index of all actors
+            keys/
+                <username>.private.pem
+                <username>.public.pem
+    """
+
+    def __init__(self, store_dir: Path | str):
+        self.store_dir = Path(store_dir)
+        self.store_dir.mkdir(parents=True, exist_ok=True)
+        self._actors: Dict[str, Actor] = {}
+        self._load()
+
+    def _index_path(self) -> Path:
+        return self.store_dir / "actors.json"
+
+    def _load(self):
+        """Load actors from disk."""
+        index_path = self._index_path()
+        if index_path.exists():
+            with open(index_path) as f:
+                data = json.load(f)
+            self._actors = {
+                username: Actor.from_dict(actor_data)
+                for username, actor_data in data.get("actors", {}).items()
+            }
+
+    def _save(self):
+        """Save actors to disk."""
+        data = {
+            "version": "1.0",
+            "domain": DOMAIN,
+            "actors": {
+                username: actor.to_dict()
+                for username, actor in self._actors.items()
+            },
+        }
+        with open(self._index_path(), "w") as f:
+            json.dump(data, f, indent=2)
+
+    def create(self, username: str, display_name: str = None) -> Actor:
+        """Create and store a new actor."""
+        if username in self._actors:
+            raise ValueError(f"Actor {username} already exists")
+
+        actor = Actor.create(username, display_name)
+        self._actors[username] = actor
+        self._save()
+        return actor
+
+    def get(self, username: str) -> Optional[Actor]:
+        """Get an actor by username."""
+        return self._actors.get(username)
+
+    def list(self) -> list[Actor]:
+        """List all actors."""
+        return list(self._actors.values())
+
+    def __contains__(self, username: str) -> bool:
+        return username in self._actors
+
+    def __len__(self) -> int:
+        return len(self._actors)

artdag/core/artdag/activitypub/ownership.py

@@ -0,0 +1,226 @@
+# primitive/activitypub/ownership.py
+"""
+Ownership integration between ActivityPub and Registry.
+
+Connects actors, activities, and assets to establish provable ownership.
+"""
+
+import json
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from .actor import Actor, ActorStore
+from .activity import Activity, CreateActivity, ActivityStore
+from .signatures import sign_activity, verify_activity_ownership
+from ..registry import Registry, Asset
+
+
+@dataclass
+class OwnershipRecord:
+    """
+    A verified ownership record linking actor to asset.
+
+    Attributes:
+        actor_handle: The actor's fediverse handle
+        asset_name: Name of the owned asset
+        cid: SHA-3 hash of the asset
+        activity_id: ID of the Create activity establishing ownership
+        verified: Whether the signature has been verified
+    """
+    actor_handle: str
+    asset_name: str
+    cid: str
+    activity_id: str
+    verified: bool = False
+
+
+class OwnershipManager:
+    """
+    Manages ownership relationships between actors and assets.
+
+    Integrates:
+    - ActorStore: Identity management
+    - Registry: Asset storage
+    - ActivityStore: Ownership activities
+    """
+
+    def __init__(self, base_dir: Path | str):
+        self.base_dir = Path(base_dir)
+        self.base_dir.mkdir(parents=True, exist_ok=True)
+
+        # Initialize stores
+        self.actors = ActorStore(self.base_dir / "actors")
+        self.activities = ActivityStore(self.base_dir / "activities")
+        self.registry = Registry(self.base_dir / "registry")
+
+    def create_actor(self, username: str, display_name: str = None) -> Actor:
+        """Create a new actor identity."""
+        return self.actors.create(username, display_name)
+
+    def get_actor(self, username: str) -> Optional[Actor]:
+        """Get an actor by username."""
+        return self.actors.get(username)
+
+    def register_asset(
+        self,
+        actor: Actor,
+        name: str,
+        cid: str,
+        url: str = None,
+        local_path: Path | str = None,
+        tags: List[str] = None,
+        metadata: Dict[str, Any] = None,
+    ) -> tuple[Asset, Activity]:
+        """
+        Register an asset and establish ownership.
+
+        Creates the asset in the registry and a signed Create activity
+        proving the actor's ownership.
+
+        Args:
+            actor: The actor claiming ownership
+            name: Name for the asset
+            cid: SHA-3-256 hash of the content
+            url: Public URL (canonical location)
+            local_path: Optional local path
+            tags: Optional tags
+            metadata: Optional metadata
+
+        Returns:
+            Tuple of (Asset, signed CreateActivity)
+        """
+        # Add to registry
+        asset = self.registry.add(
+            name=name,
+            cid=cid,
+            url=url,
+            local_path=local_path,
+            tags=tags,
+            metadata=metadata,
+        )
+
+        # Create ownership activity
+        activity = CreateActivity.for_asset(
+            actor=actor,
+            asset_name=name,
+            cid=asset.cid,
+            asset_type=self._asset_type_to_ap(asset.asset_type),
+            metadata=metadata,
+        )
+
+        # Sign the activity
+        signed_activity = sign_activity(activity, actor)
+
+        # Store the activity
+        self.activities.add(signed_activity)
+
+        return asset, signed_activity
+
+    def _asset_type_to_ap(self, asset_type: str) -> str:
+        """Convert registry asset type to ActivityPub type."""
+        type_map = {
+            "image": "Image",
+            "video": "Video",
+            "audio": "Audio",
+            "unknown": "Document",
+        }
+        return type_map.get(asset_type, "Document")
+
+    def get_owner(self, asset_name: str) -> Optional[Actor]:
+        """
+        Get the owner of an asset.
+
+        Finds the earliest Create activity for the asset and returns
+        the actor if the signature is valid.
+        """
+        asset = self.registry.get(asset_name)
+        if not asset:
+            return None
+
+        # Find Create activities for this asset
+        activities = self.activities.find_by_object_hash(asset.cid)
+        create_activities = [a for a in activities if a.activity_type == "Create"]
+
+        if not create_activities:
+            return None
+
+        # Get the earliest (first owner)
+        earliest = min(create_activities, key=lambda a: a.published)
+
+        # Extract username from actor_id
+        # Format: https://artdag.rose-ash.com/users/{username}
+        actor_id = earliest.actor_id
+        if "/users/" in actor_id:
+            username = actor_id.split("/users/")[-1]
+            actor = self.actors.get(username)
+            if actor and verify_activity_ownership(earliest, actor):
+                return actor
+
+        return None
+
+    def verify_ownership(self, asset_name: str, actor: Actor) -> bool:
+        """
+        Verify that an actor owns an asset.
+
+        Checks for a valid signed Create activity linking the actor
+        to the asset.
+        """
+        asset = self.registry.get(asset_name)
+        if not asset:
+            return False
+
+        activities = self.activities.find_by_object_hash(asset.cid)
+        for activity in activities:
+            if activity.activity_type == "Create" and activity.actor_id == actor.id:
+                if verify_activity_ownership(activity, actor):
+                    return True
+
+        return False
+
+    def list_owned_assets(self, actor: Actor) -> List[Asset]:
+        """List all assets owned by an actor."""
+        activities = self.activities.find_by_actor(actor.id)
+        owned = []
+
+        for activity in activities:
+            if activity.activity_type == "Create":
+                # Find asset by hash
+                obj_hash = activity.object_data.get("contentHash", {})
+                if isinstance(obj_hash, dict):
+                    hash_value = obj_hash.get("value")
+                else:
+                    hash_value = obj_hash
+
+                if hash_value:
+                    asset = self.registry.find_by_hash(hash_value)
+                    if asset:
+                        owned.append(asset)
+
+        return owned
+
+    def get_ownership_records(self) -> List[OwnershipRecord]:
+        """Get all ownership records."""
+        records = []
+
+        for activity in self.activities.list():
+            if activity.activity_type != "Create":
+                continue
+
+            # Extract info
+            actor_id = activity.actor_id
+            username = actor_id.split("/users/")[-1] if "/users/" in actor_id else "unknown"
+            actor = self.actors.get(username)
+
+            obj_hash = activity.object_data.get("contentHash", {})
+            hash_value = obj_hash.get("value") if isinstance(obj_hash, dict) else obj_hash
+
+            records.append(OwnershipRecord(
+                actor_handle=actor.handle if actor else f"@{username}@unknown",
+                asset_name=activity.object_data.get("name", "unknown"),
+                cid=hash_value or "unknown",
+                activity_id=activity.activity_id,
+                verified=verify_activity_ownership(activity, actor) if actor else False,
+            ))
+
+        return records

artdag/core/artdag/activitypub/signatures.py

@@ -0,0 +1,163 @@
+# primitive/activitypub/signatures.py
+"""
+Cryptographic signatures for ActivityPub.
+
+Uses RSA-SHA256 signatures compatible with HTTP Signatures spec
+and Linked Data Signatures for ActivityPub.
+"""
+
+import base64
+import hashlib
+import json
+import time
+from typing import Any, Dict
+
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import padding, rsa
+from cryptography.exceptions import InvalidSignature
+
+from .actor import Actor
+from .activity import Activity
+
+
+def _canonicalize(data: Dict[str, Any]) -> str:
+    """
+    Canonicalize JSON for signing.
+
+    Uses JCS (JSON Canonicalization Scheme) - sorted keys, no whitespace.
+    """
+    return json.dumps(data, sort_keys=True, separators=(",", ":"))
+
+
+def _hash_sha256(data: str) -> bytes:
+    """Hash string with SHA-256."""
+    return hashlib.sha256(data.encode()).digest()
+
+
+def sign_activity(activity: Activity, actor: Actor) -> Activity:
+    """
+    Sign an activity with the actor's private key.
+
+    Uses Linked Data Signatures with RsaSignature2017.
+
+    Args:
+        activity: The activity to sign
+        actor: The actor whose key signs the activity
+
+    Returns:
+        Activity with signature attached
+    """
+    # Load private key
+    private_key = serialization.load_pem_private_key(
+        actor.private_key,
+        password=None,
+    )
+
+    # Create signature options
+    created = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
+
+    # Canonicalize the activity (without signature)
+    activity_data = activity.to_activitypub()
+    activity_data.pop("signature", None)
+    canonical = _canonicalize(activity_data)
+
+    # Create the data to sign: hash of options + hash of document
+    options = {
+        "@context": "https://w3id.org/security/v1",
+        "type": "RsaSignature2017",
+        "creator": actor.key_id,
+        "created": created,
+    }
+    options_hash = _hash_sha256(_canonicalize(options))
+    document_hash = _hash_sha256(canonical)
+    to_sign = options_hash + document_hash
+
+    # Sign with RSA-SHA256
+    signature_bytes = private_key.sign(
+        to_sign,
+        padding.PKCS1v15(),
+        hashes.SHA256(),
+    )
+    signature_value = base64.b64encode(signature_bytes).decode("utf-8")
+
+    # Attach signature to activity
+    activity.signature = {
+        "type": "RsaSignature2017",
+        "creator": actor.key_id,
+        "created": created,
+        "signatureValue": signature_value,
+    }
+
+    return activity
+
+
+def verify_signature(activity: Activity, public_key_pem: bytes) -> bool:
+    """
+    Verify an activity's signature.
+
+    Args:
+        activity: The activity with signature
+        public_key_pem: PEM-encoded public key
+
+    Returns:
+        True if signature is valid
+    """
+    if not activity.signature:
+        return False
+
+    try:
+        # Load public key
+        public_key = serialization.load_pem_public_key(public_key_pem)
+
+        # Reconstruct signature options
+        options = {
+            "@context": "https://w3id.org/security/v1",
+            "type": activity.signature["type"],
+            "creator": activity.signature["creator"],
+            "created": activity.signature["created"],
+        }
+
+        # Canonicalize activity without signature
+        activity_data = activity.to_activitypub()
+        activity_data.pop("signature", None)
+        canonical = _canonicalize(activity_data)
+
+        # Recreate signed data
+        options_hash = _hash_sha256(_canonicalize(options))
+        document_hash = _hash_sha256(canonical)
+        signed_data = options_hash + document_hash
+
+        # Decode and verify signature
+        signature_bytes = base64.b64decode(activity.signature["signatureValue"])
+        public_key.verify(
+            signature_bytes,
+            signed_data,
+            padding.PKCS1v15(),
+            hashes.SHA256(),
+        )
+        return True
+
+    except (InvalidSignature, KeyError, ValueError):
+        return False
+
+
+def verify_activity_ownership(activity: Activity, actor: Actor) -> bool:
+    """
+    Verify that an activity was signed by the claimed actor.
+
+    Args:
+        activity: The activity to verify
+        actor: The claimed actor
+
+    Returns:
+        True if the activity was signed by this actor
+    """
+    if not activity.signature:
+        return False
+
+    # Check creator matches actor
+    if activity.signature.get("creator") != actor.key_id:
+        return False
+
+    # Verify signature
+    return verify_signature(activity, actor.public_key)

artdag/core/artdag/analysis/__init__.py

@@ -0,0 +1,26 @@
+# artdag/analysis - Audio and video feature extraction
+#
+# Provides the Analysis phase of the 3-phase execution model:
+# 1. ANALYZE - Extract features from inputs
+# 2. PLAN - Generate execution plan with cache IDs
+# 3. EXECUTE - Run steps with caching
+
+from .schema import (
+    AnalysisResult,
+    AudioFeatures,
+    VideoFeatures,
+    BeatInfo,
+    EnergyEnvelope,
+    SpectrumBands,
+)
+from .analyzer import Analyzer
+
+__all__ = [
+    "Analyzer",
+    "AnalysisResult",
+    "AudioFeatures",
+    "VideoFeatures",
+    "BeatInfo",
+    "EnergyEnvelope",
+    "SpectrumBands",
+]

artdag/core/artdag/analysis/analyzer.py

@@ -0,0 +1,282 @@
+# artdag/analysis/analyzer.py
+"""
+Main Analyzer class for the Analysis phase.
+
+Coordinates audio and video feature extraction with caching.
+"""
+
+import json
+import logging
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Dict, List, Optional
+
+from .schema import AnalysisResult, AudioFeatures, VideoFeatures
+from .audio import analyze_audio, FEATURE_ALL as AUDIO_ALL
+from .video import analyze_video, FEATURE_ALL as VIDEO_ALL
+
+logger = logging.getLogger(__name__)
+
+
+class AnalysisCache:
+    """
+    Simple file-based cache for analysis results.
+
+    Stores results as JSON files keyed by analysis cache_id.
+    """
+
+    def __init__(self, cache_dir: Path):
+        self.cache_dir = Path(cache_dir)
+        self.cache_dir.mkdir(parents=True, exist_ok=True)
+
+    def _path_for(self, cache_id: str) -> Path:
+        """Get cache file path for a cache_id."""
+        return self.cache_dir / f"{cache_id}.json"
+
+    def get(self, cache_id: str) -> Optional[AnalysisResult]:
+        """Retrieve cached analysis result."""
+        path = self._path_for(cache_id)
+        if not path.exists():
+            return None
+
+        try:
+            with open(path, "r") as f:
+                data = json.load(f)
+            return AnalysisResult.from_dict(data)
+        except (json.JSONDecodeError, KeyError) as e:
+            logger.warning(f"Failed to load analysis cache {cache_id}: {e}")
+            return None
+
+    def put(self, result: AnalysisResult) -> None:
+        """Store analysis result in cache."""
+        path = self._path_for(result.cache_id)
+        with open(path, "w") as f:
+            json.dump(result.to_dict(), f, indent=2)
+
+    def has(self, cache_id: str) -> bool:
+        """Check if analysis result is cached."""
+        return self._path_for(cache_id).exists()
+
+    def remove(self, cache_id: str) -> bool:
+        """Remove cached analysis result."""
+        path = self._path_for(cache_id)
+        if path.exists():
+            path.unlink()
+            return True
+        return False
+
+
+class Analyzer:
+    """
+    Analyzes media inputs to extract features.
+
+    The Analyzer is the first phase of the 3-phase execution model.
+    It extracts features from inputs that inform downstream processing.
+
+    Example:
+        analyzer = Analyzer(cache_dir=Path("./analysis_cache"))
+
+        # Analyze a music file for beats
+        result = analyzer.analyze(
+            input_path=Path("/path/to/music.mp3"),
+            input_hash="abc123...",
+            features=["beats", "energy"]
+        )
+
+        print(f"Tempo: {result.tempo} BPM")
+        print(f"Beats: {result.beat_times}")
+    """
+
+    def __init__(
+        self,
+        cache_dir: Optional[Path] = None,
+        content_cache: Optional["Cache"] = None,  # artdag.Cache for input lookup
+    ):
+        """
+        Initialize the Analyzer.
+
+        Args:
+            cache_dir: Directory for analysis cache. If None, no caching.
+            content_cache: artdag Cache for looking up inputs by hash
+        """
+        self.cache = AnalysisCache(cache_dir) if cache_dir else None
+        self.content_cache = content_cache
+
+    def get_input_path(self, input_hash: str, input_path: Optional[Path] = None) -> Path:
+        """
+        Resolve input to a file path.
+
+        Args:
+            input_hash: Content hash of the input
+            input_path: Optional direct path to file
+
+        Returns:
+            Path to the input file
+
+        Raises:
+            ValueError: If input cannot be resolved
+        """
+        if input_path and input_path.exists():
+            return input_path
+
+        if self.content_cache:
+            entry = self.content_cache.get(input_hash)
+            if entry:
+                return Path(entry.output_path)
+
+        raise ValueError(f"Cannot resolve input {input_hash}: no path provided and not in cache")
+
+    def analyze(
+        self,
+        input_hash: str,
+        features: List[str],
+        input_path: Optional[Path] = None,
+        media_type: Optional[str] = None,
+    ) -> AnalysisResult:
+        """
+        Analyze an input file and extract features.
+
+        Args:
+            input_hash: Content hash of the input (for cache key)
+            features: List of features to extract:
+                Audio: "beats", "tempo", "energy", "spectrum", "onsets"
+                Video: "metadata", "motion_tempo", "scene_changes"
+                Meta: "all" (extracts all relevant features)
+            input_path: Optional direct path to file
+            media_type: Optional hint ("audio", "video", or None for auto-detect)
+
+        Returns:
+            AnalysisResult with extracted features
+        """
+        # Compute cache ID
+        temp_result = AnalysisResult(
+            input_hash=input_hash,
+            features_requested=sorted(features),
+        )
+        cache_id = temp_result.cache_id
+
+        # Check cache
+        if self.cache and self.cache.has(cache_id):
+            cached = self.cache.get(cache_id)
+            if cached:
+                logger.info(f"Analysis cache hit: {cache_id[:16]}...")
+                return cached
+
+        # Resolve input path
+        path = self.get_input_path(input_hash, input_path)
+        logger.info(f"Analyzing {path} for features: {features}")
+
+        # Detect media type if not specified
+        if media_type is None:
+            media_type = self._detect_media_type(path)
+
+        # Extract features
+        audio_features = None
+        video_features = None
+
+        # Normalize features
+        if "all" in features:
+            audio_features_list = [AUDIO_ALL]
+            video_features_list = [VIDEO_ALL]
+        else:
+            audio_features_list = [f for f in features if f in ("beats", "tempo", "energy", "spectrum", "onsets")]
+            video_features_list = [f for f in features if f in ("metadata", "motion_tempo", "scene_changes")]
+
+        if media_type in ("audio", "video") and audio_features_list:
+            try:
+                audio_features = analyze_audio(path, features=audio_features_list)
+            except Exception as e:
+                logger.warning(f"Audio analysis failed: {e}")
+
+        if media_type == "video" and video_features_list:
+            try:
+                video_features = analyze_video(path, features=video_features_list)
+            except Exception as e:
+                logger.warning(f"Video analysis failed: {e}")
+
+        result = AnalysisResult(
+            input_hash=input_hash,
+            features_requested=sorted(features),
+            audio=audio_features,
+            video=video_features,
+            analyzed_at=datetime.now(timezone.utc).isoformat(),
+        )
+
+        # Cache result
+        if self.cache:
+            self.cache.put(result)
+
+        return result
+
+    def analyze_multiple(
+        self,
+        inputs: Dict[str, Path],
+        features: List[str],
+    ) -> Dict[str, AnalysisResult]:
+        """
+        Analyze multiple inputs.
+
+        Args:
+            inputs: Dict mapping input_hash to file path
+            features: Features to extract from all inputs
+
+        Returns:
+            Dict mapping input_hash to AnalysisResult
+        """
+        results = {}
+        for input_hash, input_path in inputs.items():
+            try:
+                results[input_hash] = self.analyze(
+                    input_hash=input_hash,
+                    features=features,
+                    input_path=input_path,
+                )
+            except Exception as e:
+                logger.error(f"Analysis failed for {input_hash}: {e}")
+                raise
+
+        return results
+
+    def _detect_media_type(self, path: Path) -> str:
+        """
+        Detect if file is audio or video.
+
+        Args:
+            path: Path to media file
+
+        Returns:
+            "audio" or "video"
+        """
+        import subprocess
+        import json
+
+        cmd = [
+            "ffprobe", "-v", "quiet",
+            "-print_format", "json",
+            "-show_streams",
+            str(path)
+        ]
+
+        try:
+            result = subprocess.run(cmd, capture_output=True, text=True, check=True)
+            data = json.loads(result.stdout)
+            streams = data.get("streams", [])
+
+            has_video = any(s.get("codec_type") == "video" for s in streams)
+            has_audio = any(s.get("codec_type") == "audio" for s in streams)
+
+            if has_video:
+                return "video"
+            elif has_audio:
+                return "audio"
+            else:
+                return "unknown"
+
+        except (subprocess.CalledProcessError, json.JSONDecodeError):
+            # Fall back to extension-based detection
+            ext = path.suffix.lower()
+            if ext in (".mp4", ".mov", ".avi", ".mkv", ".webm"):
+                return "video"
+            elif ext in (".mp3", ".wav", ".flac", ".ogg", ".m4a", ".aac"):
+                return "audio"
+            return "unknown"

artdag/core/artdag/analysis/audio.py

@@ -0,0 +1,336 @@
+# artdag/analysis/audio.py
+"""
+Audio feature extraction.
+
+Uses librosa for beat detection, energy analysis, and spectral features.
+Falls back to basic ffprobe if librosa is not available.
+"""
+
+import json
+import logging
+import subprocess
+from pathlib import Path
+from typing import List, Optional, Tuple
+
+from .schema import AudioFeatures, BeatInfo, EnergyEnvelope, SpectrumBands
+
+logger = logging.getLogger(__name__)
+
+# Feature names for requesting specific analysis
+FEATURE_BEATS = "beats"
+FEATURE_TEMPO = "tempo"
+FEATURE_ENERGY = "energy"
+FEATURE_SPECTRUM = "spectrum"
+FEATURE_ONSETS = "onsets"
+FEATURE_ALL = "all"
+
+
+def _get_audio_info_ffprobe(path: Path) -> Tuple[float, int, int]:
+    """Get basic audio info using ffprobe."""
+    cmd = [
+        "ffprobe", "-v", "quiet",
+        "-print_format", "json",
+        "-show_streams",
+        "-select_streams", "a:0",
+        str(path)
+    ]
+    try:
+        result = subprocess.run(cmd, capture_output=True, text=True, check=True)
+        data = json.loads(result.stdout)
+        if not data.get("streams"):
+            raise ValueError("No audio stream found")
+
+        stream = data["streams"][0]
+        duration = float(stream.get("duration", 0))
+        sample_rate = int(stream.get("sample_rate", 44100))
+        channels = int(stream.get("channels", 2))
+        return duration, sample_rate, channels
+    except (subprocess.CalledProcessError, json.JSONDecodeError, KeyError) as e:
+        logger.warning(f"ffprobe failed: {e}")
+        raise ValueError(f"Could not read audio info: {e}")
+
+
+def _extract_audio_to_wav(path: Path, duration: Optional[float] = None) -> Path:
+    """Extract audio to temporary WAV file for librosa processing."""
+    import tempfile
+    wav_path = Path(tempfile.mktemp(suffix=".wav"))
+
+    cmd = ["ffmpeg", "-y", "-i", str(path)]
+    if duration:
+        cmd.extend(["-t", str(duration)])
+    cmd.extend([
+        "-vn",  # No video
+        "-acodec", "pcm_s16le",
+        "-ar", "22050",  # Resample to 22050 Hz for librosa
+        "-ac", "1",  # Mono
+        str(wav_path)
+    ])
+
+    try:
+        subprocess.run(cmd, capture_output=True, check=True)
+        return wav_path
+    except subprocess.CalledProcessError as e:
+        logger.error(f"Audio extraction failed: {e.stderr}")
+        raise ValueError(f"Could not extract audio: {e}")
+
+
+def analyze_beats(path: Path, sample_rate: int = 22050) -> BeatInfo:
+    """
+    Detect beats and tempo using librosa.
+
+    Args:
+        path: Path to audio file (or pre-extracted WAV)
+        sample_rate: Sample rate for analysis
+
+    Returns:
+        BeatInfo with beat times, tempo, and confidence
+    """
+    try:
+        import librosa
+    except ImportError:
+        raise ImportError("librosa required for beat detection. Install with: pip install librosa")
+
+    # Load audio
+    y, sr = librosa.load(str(path), sr=sample_rate, mono=True)
+
+    # Detect tempo and beats
+    tempo, beat_frames = librosa.beat.beat_track(y=y, sr=sr)
+
+    # Convert frames to times
+    beat_times = librosa.frames_to_time(beat_frames, sr=sr).tolist()
+
+    # Estimate confidence from onset strength consistency
+    onset_env = librosa.onset.onset_strength(y=y, sr=sr)
+    beat_strength = onset_env[beat_frames] if len(beat_frames) > 0 else []
+    confidence = float(beat_strength.mean() / onset_env.max()) if len(beat_strength) > 0 and onset_env.max() > 0 else 0.5
+
+    # Detect downbeats (first beat of each bar)
+    # Use beat phase to estimate bar positions
+    downbeat_times = None
+    if len(beat_times) >= 4:
+        # Assume 4/4 time signature, downbeats every 4 beats
+        downbeat_times = [beat_times[i] for i in range(0, len(beat_times), 4)]
+
+    return BeatInfo(
+        beat_times=beat_times,
+        tempo=float(tempo) if hasattr(tempo, '__float__') else float(tempo[0]) if len(tempo) > 0 else 120.0,
+        confidence=min(1.0, max(0.0, confidence)),
+        downbeat_times=downbeat_times,
+        time_signature=4,
+    )
+
+
+def analyze_energy(path: Path, window_ms: float = 50.0, sample_rate: int = 22050) -> EnergyEnvelope:
+    """
+    Extract energy (loudness) envelope.
+
+    Args:
+        path: Path to audio file
+        window_ms: Analysis window size in milliseconds
+        sample_rate: Sample rate for analysis
+
+    Returns:
+        EnergyEnvelope with times and normalized values
+    """
+    try:
+        import librosa
+        import numpy as np
+    except ImportError:
+        raise ImportError("librosa and numpy required. Install with: pip install librosa numpy")
+
+    y, sr = librosa.load(str(path), sr=sample_rate, mono=True)
+
+    # Calculate frame size from window_ms
+    hop_length = int(sr * window_ms / 1000)
+
+    # RMS energy
+    rms = librosa.feature.rms(y=y, hop_length=hop_length)[0]
+
+    # Normalize to 0-1
+    rms_max = rms.max()
+    if rms_max > 0:
+        rms_normalized = rms / rms_max
+    else:
+        rms_normalized = rms
+
+    # Generate time points
+    times = librosa.frames_to_time(np.arange(len(rms)), sr=sr, hop_length=hop_length)
+
+    return EnergyEnvelope(
+        times=times.tolist(),
+        values=rms_normalized.tolist(),
+        window_ms=window_ms,
+    )
+
+
+def analyze_spectrum(
+    path: Path,
+    band_ranges: Optional[dict] = None,
+    window_ms: float = 50.0,
+    sample_rate: int = 22050
+) -> SpectrumBands:
+    """
+    Extract frequency band envelopes.
+
+    Args:
+        path: Path to audio file
+        band_ranges: Dict mapping band name to (low_hz, high_hz)
+        window_ms: Analysis window size
+        sample_rate: Sample rate
+
+    Returns:
+        SpectrumBands with bass, mid, high envelopes
+    """
+    try:
+        import librosa
+        import numpy as np
+    except ImportError:
+        raise ImportError("librosa and numpy required")
+
+    if band_ranges is None:
+        band_ranges = {
+            "bass": (20, 200),
+            "mid": (200, 2000),
+            "high": (2000, 20000),
+        }
+
+    y, sr = librosa.load(str(path), sr=sample_rate, mono=True)
+    hop_length = int(sr * window_ms / 1000)
+
+    # Compute STFT
+    n_fft = 2048
+    stft = np.abs(librosa.stft(y, n_fft=n_fft, hop_length=hop_length))
+
+    # Frequency bins
+    freqs = librosa.fft_frequencies(sr=sr, n_fft=n_fft)
+
+    def band_energy(low_hz: float, high_hz: float) -> List[float]:
+        """Sum energy in frequency band."""
+        mask = (freqs >= low_hz) & (freqs <= high_hz)
+        if not mask.any():
+            return [0.0] * stft.shape[1]
+        band = stft[mask, :].sum(axis=0)
+        # Normalize
+        band_max = band.max()
+        if band_max > 0:
+            band = band / band_max
+        return band.tolist()
+
+    times = librosa.frames_to_time(np.arange(stft.shape[1]), sr=sr, hop_length=hop_length)
+
+    return SpectrumBands(
+        bass=band_energy(*band_ranges["bass"]),
+        mid=band_energy(*band_ranges["mid"]),
+        high=band_energy(*band_ranges["high"]),
+        times=times.tolist(),
+        band_ranges=band_ranges,
+    )
+
+
+def analyze_onsets(path: Path, sample_rate: int = 22050) -> List[float]:
+    """
+    Detect onset times (note/sound starts).
+
+    Args:
+        path: Path to audio file
+        sample_rate: Sample rate
+
+    Returns:
+        List of onset times in seconds
+    """
+    try:
+        import librosa
+    except ImportError:
+        raise ImportError("librosa required")
+
+    y, sr = librosa.load(str(path), sr=sample_rate, mono=True)
+
+    # Detect onsets
+    onset_frames = librosa.onset.onset_detect(y=y, sr=sr)
+    onset_times = librosa.frames_to_time(onset_frames, sr=sr)
+
+    return onset_times.tolist()
+
+
+def analyze_audio(
+    path: Path,
+    features: Optional[List[str]] = None,
+) -> AudioFeatures:
+    """
+    Extract audio features from file.
+
+    Args:
+        path: Path to audio/video file
+        features: List of features to extract. Options:
+            - "beats": Beat detection (tempo, beat times)
+            - "energy": Loudness envelope
+            - "spectrum": Frequency band envelopes
+            - "onsets": Note onset times
+            - "all": All features
+
+    Returns:
+        AudioFeatures with requested analysis
+    """
+    if features is None:
+        features = [FEATURE_ALL]
+
+    # Normalize features
+    if FEATURE_ALL in features:
+        features = [FEATURE_BEATS, FEATURE_ENERGY, FEATURE_SPECTRUM, FEATURE_ONSETS]
+
+    # Get basic info via ffprobe
+    duration, sample_rate, channels = _get_audio_info_ffprobe(path)
+
+    result = AudioFeatures(
+        duration=duration,
+        sample_rate=sample_rate,
+        channels=channels,
+    )
+
+    # Check if librosa is available for advanced features
+    try:
+        import librosa  # noqa: F401
+        has_librosa = True
+    except ImportError:
+        has_librosa = False
+        if any(f in features for f in [FEATURE_BEATS, FEATURE_ENERGY, FEATURE_SPECTRUM, FEATURE_ONSETS]):
+            logger.warning("librosa not available, skipping advanced audio features")
+
+    if not has_librosa:
+        return result
+
+    # Extract audio to WAV for librosa
+    wav_path = None
+    try:
+        wav_path = _extract_audio_to_wav(path)
+
+        if FEATURE_BEATS in features or FEATURE_TEMPO in features:
+            try:
+                result.beats = analyze_beats(wav_path)
+            except Exception as e:
+                logger.warning(f"Beat detection failed: {e}")
+
+        if FEATURE_ENERGY in features:
+            try:
+                result.energy = analyze_energy(wav_path)
+            except Exception as e:
+                logger.warning(f"Energy analysis failed: {e}")
+
+        if FEATURE_SPECTRUM in features:
+            try:
+                result.spectrum = analyze_spectrum(wav_path)
+            except Exception as e:
+                logger.warning(f"Spectrum analysis failed: {e}")
+
+        if FEATURE_ONSETS in features:
+            try:
+                result.onsets = analyze_onsets(wav_path)
+            except Exception as e:
+                logger.warning(f"Onset detection failed: {e}")
+
+    finally:
+        # Clean up temporary WAV file
+        if wav_path and wav_path.exists():
+            wav_path.unlink()
+
+    return result

artdag/core/artdag/analysis/schema.py

@@ -0,0 +1,352 @@
+# artdag/analysis/schema.py
+"""
+Data structures for analysis results.
+
+Analysis extracts features from input media that inform downstream processing.
+Results are cached by: analysis_cache_id = SHA3-256(input_hash + sorted(features))
+"""
+
+import hashlib
+import json
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional, Tuple
+
+
+def _stable_hash(data: Any, algorithm: str = "sha3_256") -> str:
+    """Create stable hash from arbitrary data."""
+    json_str = json.dumps(data, sort_keys=True, separators=(",", ":"))
+    hasher = hashlib.new(algorithm)
+    hasher.update(json_str.encode())
+    return hasher.hexdigest()
+
+
+@dataclass
+class BeatInfo:
+    """
+    Beat detection results.
+
+    Attributes:
+        beat_times: List of beat positions in seconds
+        tempo: Estimated tempo in BPM
+        confidence: Tempo detection confidence (0-1)
+        downbeat_times: First beat of each bar (if detected)
+        time_signature: Detected or assumed time signature (e.g., 4)
+    """
+    beat_times: List[float]
+    tempo: float
+    confidence: float = 1.0
+    downbeat_times: Optional[List[float]] = None
+    time_signature: int = 4
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "beat_times": self.beat_times,
+            "tempo": self.tempo,
+            "confidence": self.confidence,
+            "downbeat_times": self.downbeat_times,
+            "time_signature": self.time_signature,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "BeatInfo":
+        return cls(
+            beat_times=data["beat_times"],
+            tempo=data["tempo"],
+            confidence=data.get("confidence", 1.0),
+            downbeat_times=data.get("downbeat_times"),
+            time_signature=data.get("time_signature", 4),
+        )
+
+
+@dataclass
+class EnergyEnvelope:
+    """
+    Energy (loudness) over time.
+
+    Attributes:
+        times: Time points in seconds
+        values: Energy values (0-1, normalized)
+        window_ms: Analysis window size in milliseconds
+    """
+    times: List[float]
+    values: List[float]
+    window_ms: float = 50.0
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "times": self.times,
+            "values": self.values,
+            "window_ms": self.window_ms,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "EnergyEnvelope":
+        return cls(
+            times=data["times"],
+            values=data["values"],
+            window_ms=data.get("window_ms", 50.0),
+        )
+
+    def at_time(self, t: float) -> float:
+        """Interpolate energy value at given time."""
+        if not self.times:
+            return 0.0
+        if t <= self.times[0]:
+            return self.values[0]
+        if t >= self.times[-1]:
+            return self.values[-1]
+
+        # Binary search for bracketing indices
+        lo, hi = 0, len(self.times) - 1
+        while hi - lo > 1:
+            mid = (lo + hi) // 2
+            if self.times[mid] <= t:
+                lo = mid
+            else:
+                hi = mid
+
+        # Linear interpolation
+        t0, t1 = self.times[lo], self.times[hi]
+        v0, v1 = self.values[lo], self.values[hi]
+        alpha = (t - t0) / (t1 - t0) if t1 != t0 else 0
+        return v0 + alpha * (v1 - v0)
+
+
+@dataclass
+class SpectrumBands:
+    """
+    Frequency band envelopes over time.
+
+    Attributes:
+        bass: Low frequency envelope (20-200 Hz typical)
+        mid: Mid frequency envelope (200-2000 Hz typical)
+        high: High frequency envelope (2000-20000 Hz typical)
+        times: Time points in seconds
+        band_ranges: Frequency ranges for each band in Hz
+    """
+    bass: List[float]
+    mid: List[float]
+    high: List[float]
+    times: List[float]
+    band_ranges: Dict[str, Tuple[float, float]] = field(default_factory=lambda: {
+        "bass": (20, 200),
+        "mid": (200, 2000),
+        "high": (2000, 20000),
+    })
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "bass": self.bass,
+            "mid": self.mid,
+            "high": self.high,
+            "times": self.times,
+            "band_ranges": self.band_ranges,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "SpectrumBands":
+        return cls(
+            bass=data["bass"],
+            mid=data["mid"],
+            high=data["high"],
+            times=data["times"],
+            band_ranges=data.get("band_ranges", {
+                "bass": (20, 200),
+                "mid": (200, 2000),
+                "high": (2000, 20000),
+            }),
+        )
+
+
+@dataclass
+class AudioFeatures:
+    """
+    All extracted audio features.
+
+    Attributes:
+        duration: Audio duration in seconds
+        sample_rate: Sample rate in Hz
+        channels: Number of audio channels
+        beats: Beat detection results
+        energy: Energy envelope
+        spectrum: Frequency band envelopes
+        onsets: Note/sound onset times
+    """
+    duration: float
+    sample_rate: int
+    channels: int
+    beats: Optional[BeatInfo] = None
+    energy: Optional[EnergyEnvelope] = None
+    spectrum: Optional[SpectrumBands] = None
+    onsets: Optional[List[float]] = None
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "duration": self.duration,
+            "sample_rate": self.sample_rate,
+            "channels": self.channels,
+            "beats": self.beats.to_dict() if self.beats else None,
+            "energy": self.energy.to_dict() if self.energy else None,
+            "spectrum": self.spectrum.to_dict() if self.spectrum else None,
+            "onsets": self.onsets,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "AudioFeatures":
+        return cls(
+            duration=data["duration"],
+            sample_rate=data["sample_rate"],
+            channels=data["channels"],
+            beats=BeatInfo.from_dict(data["beats"]) if data.get("beats") else None,
+            energy=EnergyEnvelope.from_dict(data["energy"]) if data.get("energy") else None,
+            spectrum=SpectrumBands.from_dict(data["spectrum"]) if data.get("spectrum") else None,
+            onsets=data.get("onsets"),
+        )
+
+
+@dataclass
+class VideoFeatures:
+    """
+    Extracted video features.
+
+    Attributes:
+        duration: Video duration in seconds
+        frame_rate: Frames per second
+        width: Frame width in pixels
+        height: Frame height in pixels
+        codec: Video codec name
+        motion_tempo: Estimated tempo from motion analysis (optional)
+        scene_changes: Times of detected scene changes
+    """
+    duration: float
+    frame_rate: float
+    width: int
+    height: int
+    codec: str = ""
+    motion_tempo: Optional[float] = None
+    scene_changes: Optional[List[float]] = None
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "duration": self.duration,
+            "frame_rate": self.frame_rate,
+            "width": self.width,
+            "height": self.height,
+            "codec": self.codec,
+            "motion_tempo": self.motion_tempo,
+            "scene_changes": self.scene_changes,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "VideoFeatures":
+        return cls(
+            duration=data["duration"],
+            frame_rate=data["frame_rate"],
+            width=data["width"],
+            height=data["height"],
+            codec=data.get("codec", ""),
+            motion_tempo=data.get("motion_tempo"),
+            scene_changes=data.get("scene_changes"),
+        )
+
+
+@dataclass
+class AnalysisResult:
+    """
+    Complete analysis result for an input.
+
+    Combines audio and video features with metadata for caching.
+
+    Attributes:
+        input_hash: Content hash of the analyzed input
+        features_requested: List of features that were requested
+        audio: Audio features (if input has audio)
+        video: Video features (if input has video)
+        cache_id: Computed cache ID for this analysis
+        analyzed_at: Timestamp of analysis
+    """
+    input_hash: str
+    features_requested: List[str]
+    audio: Optional[AudioFeatures] = None
+    video: Optional[VideoFeatures] = None
+    cache_id: Optional[str] = None
+    analyzed_at: Optional[str] = None
+
+    def __post_init__(self):
+        """Compute cache_id if not provided."""
+        if self.cache_id is None:
+            self.cache_id = self._compute_cache_id()
+
+    def _compute_cache_id(self) -> str:
+        """
+        Compute cache ID from input hash and requested features.
+
+        cache_id = SHA3-256(input_hash + sorted(features_requested))
+        """
+        content = {
+            "input_hash": self.input_hash,
+            "features": sorted(self.features_requested),
+        }
+        return _stable_hash(content)
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "input_hash": self.input_hash,
+            "features_requested": self.features_requested,
+            "audio": self.audio.to_dict() if self.audio else None,
+            "video": self.video.to_dict() if self.video else None,
+            "cache_id": self.cache_id,
+            "analyzed_at": self.analyzed_at,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "AnalysisResult":
+        return cls(
+            input_hash=data["input_hash"],
+            features_requested=data["features_requested"],
+            audio=AudioFeatures.from_dict(data["audio"]) if data.get("audio") else None,
+            video=VideoFeatures.from_dict(data["video"]) if data.get("video") else None,
+            cache_id=data.get("cache_id"),
+            analyzed_at=data.get("analyzed_at"),
+        )
+
+    def to_json(self) -> str:
+        """Serialize to JSON string."""
+        return json.dumps(self.to_dict(), indent=2)
+
+    @classmethod
+    def from_json(cls, json_str: str) -> "AnalysisResult":
+        """Deserialize from JSON string."""
+        return cls.from_dict(json.loads(json_str))
+
+    # Convenience accessors
+    @property
+    def tempo(self) -> Optional[float]:
+        """Get tempo if beats were analyzed."""
+        return self.audio.beats.tempo if self.audio and self.audio.beats else None
+
+    @property
+    def beat_times(self) -> Optional[List[float]]:
+        """Get beat times if beats were analyzed."""
+        return self.audio.beats.beat_times if self.audio and self.audio.beats else None
+
+    @property
+    def downbeat_times(self) -> Optional[List[float]]:
+        """Get downbeat times if analyzed."""
+        return self.audio.beats.downbeat_times if self.audio and self.audio.beats else None
+
+    @property
+    def duration(self) -> float:
+        """Get duration from video or audio."""
+        if self.video:
+            return self.video.duration
+        if self.audio:
+            return self.audio.duration
+        return 0.0
+
+    @property
+    def dimensions(self) -> Optional[Tuple[int, int]]:
+        """Get video dimensions if available."""
+        if self.video:
+            return (self.video.width, self.video.height)
+        return None

artdag/core/artdag/analysis/video.py

@@ -0,0 +1,266 @@
+# artdag/analysis/video.py
+"""
+Video feature extraction.
+
+Uses ffprobe for basic metadata and optional OpenCV for motion analysis.
+"""
+
+import json
+import logging
+import subprocess
+from fractions import Fraction
+from pathlib import Path
+from typing import List, Optional
+
+from .schema import VideoFeatures
+
+logger = logging.getLogger(__name__)
+
+# Feature names
+FEATURE_METADATA = "metadata"
+FEATURE_MOTION_TEMPO = "motion_tempo"
+FEATURE_SCENE_CHANGES = "scene_changes"
+FEATURE_ALL = "all"
+
+
+def _parse_frame_rate(rate_str: str) -> float:
+    """Parse frame rate string like '30000/1001' or '30'."""
+    try:
+        if "/" in rate_str:
+            frac = Fraction(rate_str)
+            return float(frac)
+        return float(rate_str)
+    except (ValueError, ZeroDivisionError):
+        return 30.0  # Default
+
+
+def analyze_metadata(path: Path) -> VideoFeatures:
+    """
+    Extract video metadata using ffprobe.
+
+    Args:
+        path: Path to video file
+
+    Returns:
+        VideoFeatures with basic metadata
+    """
+    cmd = [
+        "ffprobe", "-v", "quiet",
+        "-print_format", "json",
+        "-show_streams",
+        "-show_format",
+        "-select_streams", "v:0",
+        str(path)
+    ]
+
+    try:
+        result = subprocess.run(cmd, capture_output=True, text=True, check=True)
+        data = json.loads(result.stdout)
+    except (subprocess.CalledProcessError, json.JSONDecodeError) as e:
+        raise ValueError(f"Could not read video info: {e}")
+
+    if not data.get("streams"):
+        raise ValueError("No video stream found")
+
+    stream = data["streams"][0]
+    fmt = data.get("format", {})
+
+    # Get duration from format or stream
+    duration = float(fmt.get("duration", stream.get("duration", 0)))
+
+    # Parse frame rate
+    frame_rate = _parse_frame_rate(stream.get("avg_frame_rate", "30"))
+
+    return VideoFeatures(
+        duration=duration,
+        frame_rate=frame_rate,
+        width=int(stream.get("width", 0)),
+        height=int(stream.get("height", 0)),
+        codec=stream.get("codec_name", ""),
+    )
+
+
+def analyze_scene_changes(path: Path, threshold: float = 0.3) -> List[float]:
+    """
+    Detect scene changes using ffmpeg scene detection.
+
+    Args:
+        path: Path to video file
+        threshold: Scene change threshold (0-1, lower = more sensitive)
+
+    Returns:
+        List of scene change times in seconds
+    """
+    cmd = [
+        "ffmpeg", "-i", str(path),
+        "-vf", f"select='gt(scene,{threshold})',showinfo",
+        "-f", "null", "-"
+    ]
+
+    try:
+        result = subprocess.run(cmd, capture_output=True, text=True)
+        stderr = result.stderr
+    except subprocess.CalledProcessError as e:
+        logger.warning(f"Scene detection failed: {e}")
+        return []
+
+    # Parse scene change times from ffmpeg output
+    scene_times = []
+    for line in stderr.split("\n"):
+        if "pts_time:" in line:
+            try:
+                # Extract pts_time value
+                for part in line.split():
+                    if part.startswith("pts_time:"):
+                        time_str = part.split(":")[1]
+                        scene_times.append(float(time_str))
+                        break
+            except (ValueError, IndexError):
+                continue
+
+    return scene_times
+
+
+def analyze_motion_tempo(path: Path, sample_duration: float = 30.0) -> Optional[float]:
+    """
+    Estimate tempo from video motion periodicity.
+
+    Analyzes optical flow or frame differences to detect rhythmic motion.
+    This is useful for matching video speed to audio tempo.
+
+    Args:
+        path: Path to video file
+        sample_duration: Duration to analyze (seconds)
+
+    Returns:
+        Estimated motion tempo in BPM, or None if not detectable
+    """
+    try:
+        import cv2
+        import numpy as np
+    except ImportError:
+        logger.warning("OpenCV not available, skipping motion tempo analysis")
+        return None
+
+    cap = cv2.VideoCapture(str(path))
+    if not cap.isOpened():
+        logger.warning(f"Could not open video: {path}")
+        return None
+
+    try:
+        fps = cap.get(cv2.CAP_PROP_FPS)
+        if fps <= 0:
+            fps = 30.0
+
+        max_frames = int(sample_duration * fps)
+        frame_diffs = []
+        prev_gray = None
+
+        frame_count = 0
+        while frame_count < max_frames:
+            ret, frame = cap.read()
+            if not ret:
+                break
+
+            # Convert to grayscale and resize for speed
+            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+            gray = cv2.resize(gray, (160, 90))
+
+            if prev_gray is not None:
+                # Calculate frame difference
+                diff = cv2.absdiff(gray, prev_gray)
+                frame_diffs.append(np.mean(diff))
+
+            prev_gray = gray
+            frame_count += 1
+
+        if len(frame_diffs) < 60:  # Need at least 2 seconds at 30fps
+            return None
+
+        # Convert to numpy array
+        motion = np.array(frame_diffs)
+
+        # Normalize
+        motion = motion - motion.mean()
+        if motion.std() > 0:
+            motion = motion / motion.std()
+
+        # Autocorrelation to find periodicity
+        n = len(motion)
+        acf = np.correlate(motion, motion, mode="full")[n-1:]
+        acf = acf / acf[0]  # Normalize
+
+        # Find peaks in autocorrelation (potential beat periods)
+        # Look for periods between 0.3s (200 BPM) and 2s (30 BPM)
+        min_lag = int(0.3 * fps)
+        max_lag = min(int(2.0 * fps), len(acf) - 1)
+
+        if max_lag <= min_lag:
+            return None
+
+        # Find the highest peak in the valid range
+        search_range = acf[min_lag:max_lag]
+        if len(search_range) == 0:
+            return None
+
+        peak_idx = np.argmax(search_range) + min_lag
+        peak_value = acf[peak_idx]
+
+        # Only report if peak is significant
+        if peak_value < 0.1:
+            return None
+
+        # Convert lag to BPM
+        period_seconds = peak_idx / fps
+        bpm = 60.0 / period_seconds
+
+        # Sanity check
+        if 30 <= bpm <= 200:
+            return round(bpm, 1)
+
+        return None
+
+    finally:
+        cap.release()
+
+
+def analyze_video(
+    path: Path,
+    features: Optional[List[str]] = None,
+) -> VideoFeatures:
+    """
+    Extract video features from file.
+
+    Args:
+        path: Path to video file
+        features: List of features to extract. Options:
+            - "metadata": Basic video info (always included)
+            - "motion_tempo": Estimated tempo from motion
+            - "scene_changes": Scene change detection
+            - "all": All features
+
+    Returns:
+        VideoFeatures with requested analysis
+    """
+    if features is None:
+        features = [FEATURE_METADATA]
+
+    if FEATURE_ALL in features:
+        features = [FEATURE_METADATA, FEATURE_MOTION_TEMPO, FEATURE_SCENE_CHANGES]
+
+    # Basic metadata is always extracted
+    result = analyze_metadata(path)
+
+    if FEATURE_MOTION_TEMPO in features:
+        try:
+            result.motion_tempo = analyze_motion_tempo(path)
+        except Exception as e:
+            logger.warning(f"Motion tempo analysis failed: {e}")
+
+    if FEATURE_SCENE_CHANGES in features:
+        try:
+            result.scene_changes = analyze_scene_changes(path)
+        except Exception as e:
+            logger.warning(f"Scene change detection failed: {e}")
+
+    return result

artdag/core/artdag/cache.py

@@ -0,0 +1,464 @@
+# primitive/cache.py
+"""
+Content-addressed file cache for node outputs.
+
+Each node's output is stored at: cache_dir / node_id / output_file
+This enables automatic reuse when the same operation is requested.
+"""
+
+import json
+import logging
+import shutil
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Dict, List, Optional
+
+logger = logging.getLogger(__name__)
+
+
+def _file_hash(path: Path, algorithm: str = "sha3_256") -> str:
+    """
+    Compute content hash of a file.
+
+    Uses SHA-3 (Keccak) by default for quantum resistance.
+    """
+    import hashlib
+    hasher = hashlib.new(algorithm)
+    with open(path, "rb") as f:
+        for chunk in iter(lambda: f.read(65536), b""):
+            hasher.update(chunk)
+    return hasher.hexdigest()
+
+
+@dataclass
+class CacheEntry:
+    """Metadata about a cached output."""
+    node_id: str
+    output_path: Path
+    created_at: float
+    size_bytes: int
+    node_type: str
+    cid: str = ""  # Content identifier (IPFS CID or local hash)
+    execution_time: float = 0.0
+
+    def to_dict(self) -> Dict:
+        return {
+            "node_id": self.node_id,
+            "output_path": str(self.output_path),
+            "created_at": self.created_at,
+            "size_bytes": self.size_bytes,
+            "node_type": self.node_type,
+            "cid": self.cid,
+            "execution_time": self.execution_time,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict) -> "CacheEntry":
+        # Support both "cid" and legacy "content_hash"
+        cid = data.get("cid") or data.get("content_hash", "")
+        return cls(
+            node_id=data["node_id"],
+            output_path=Path(data["output_path"]),
+            created_at=data["created_at"],
+            size_bytes=data["size_bytes"],
+            node_type=data["node_type"],
+            cid=cid,
+            execution_time=data.get("execution_time", 0.0),
+        )
+
+
+@dataclass
+class CacheStats:
+    """Statistics about cache usage."""
+    total_entries: int = 0
+    total_size_bytes: int = 0
+    hits: int = 0
+    misses: int = 0
+    hit_rate: float = 0.0
+
+    def record_hit(self):
+        self.hits += 1
+        self._update_rate()
+
+    def record_miss(self):
+        self.misses += 1
+        self._update_rate()
+
+    def _update_rate(self):
+        total = self.hits + self.misses
+        self.hit_rate = self.hits / total if total > 0 else 0.0
+
+
+class Cache:
+    """
+    Code-addressed file cache.
+
+    The filesystem IS the index - no JSON index files needed.
+    Each node's hash is its directory name.
+
+    Structure:
+        cache_dir/
+            <hash>/
+                output.ext       # Actual output file
+                metadata.json    # Per-node metadata (optional)
+    """
+
+    def __init__(self, cache_dir: Path | str):
+        self.cache_dir = Path(cache_dir)
+        self.cache_dir.mkdir(parents=True, exist_ok=True)
+        self.stats = CacheStats()
+
+    def _node_dir(self, node_id: str) -> Path:
+        """Get the cache directory for a node."""
+        return self.cache_dir / node_id
+
+    def _find_output_file(self, node_dir: Path) -> Optional[Path]:
+        """Find the output file in a node directory."""
+        if not node_dir.exists() or not node_dir.is_dir():
+            return None
+        for f in node_dir.iterdir():
+            if f.is_file() and f.name.startswith("output."):
+                return f
+        return None
+
+    def get(self, node_id: str) -> Optional[Path]:
+        """
+        Get cached output path for a node.
+
+        Checks filesystem directly - no in-memory index.
+        Returns the output path if cached, None otherwise.
+        """
+        node_dir = self._node_dir(node_id)
+        output_file = self._find_output_file(node_dir)
+
+        if output_file:
+            self.stats.record_hit()
+            logger.debug(f"Cache hit: {node_id[:16]}...")
+            return output_file
+
+        self.stats.record_miss()
+        return None
+
+    def put(self, node_id: str, source_path: Path, node_type: str,
+            execution_time: float = 0.0, move: bool = False) -> Path:
+        """
+        Store a file in the cache.
+
+        Args:
+            node_id: The code-addressed node ID (hash)
+            source_path: Path to the file to cache
+            node_type: Type of the node (for metadata)
+            execution_time: How long the node took to execute
+            move: If True, move the file instead of copying
+
+        Returns:
+            Path to the cached file
+        """
+        node_dir = self._node_dir(node_id)
+        node_dir.mkdir(parents=True, exist_ok=True)
+
+        # Preserve extension
+        ext = source_path.suffix or ".out"
+        output_path = node_dir / f"output{ext}"
+
+        # Copy or move file (skip if already in place)
+        source_resolved = Path(source_path).resolve()
+        output_resolved = output_path.resolve()
+        if source_resolved != output_resolved:
+            if move:
+                shutil.move(source_path, output_path)
+            else:
+                shutil.copy2(source_path, output_path)
+
+        # Compute content hash (IPFS CID of the result)
+        cid = _file_hash(output_path)
+
+        # Store per-node metadata (optional, for stats/debugging)
+        metadata = {
+            "node_id": node_id,
+            "output_path": str(output_path),
+            "created_at": time.time(),
+            "size_bytes": output_path.stat().st_size,
+            "node_type": node_type,
+            "cid": cid,
+            "execution_time": execution_time,
+        }
+        metadata_path = node_dir / "metadata.json"
+        with open(metadata_path, "w") as f:
+            json.dump(metadata, f, indent=2)
+
+        logger.debug(f"Cached: {node_id[:16]}... ({metadata['size_bytes']} bytes)")
+        return output_path
+
+    def has(self, node_id: str) -> bool:
+        """Check if a node is cached (without affecting stats)."""
+        return self._find_output_file(self._node_dir(node_id)) is not None
+
+    def remove(self, node_id: str) -> bool:
+        """Remove a node from the cache."""
+        node_dir = self._node_dir(node_id)
+        if node_dir.exists():
+            shutil.rmtree(node_dir)
+            return True
+        return False
+
+    def clear(self):
+        """Clear all cached entries."""
+        for node_dir in self.cache_dir.iterdir():
+            if node_dir.is_dir() and not node_dir.name.startswith("_"):
+                shutil.rmtree(node_dir)
+        self.stats = CacheStats()
+
+    def get_stats(self) -> CacheStats:
+        """Get cache statistics (scans filesystem)."""
+        stats = CacheStats()
+        for node_dir in self.cache_dir.iterdir():
+            if node_dir.is_dir() and not node_dir.name.startswith("_"):
+                output_file = self._find_output_file(node_dir)
+                if output_file:
+                    stats.total_entries += 1
+                    stats.total_size_bytes += output_file.stat().st_size
+        stats.hits = self.stats.hits
+        stats.misses = self.stats.misses
+        stats.hit_rate = self.stats.hit_rate
+        return stats
+
+    def list_entries(self) -> List[CacheEntry]:
+        """List all cache entries (scans filesystem)."""
+        entries = []
+        for node_dir in self.cache_dir.iterdir():
+            if node_dir.is_dir() and not node_dir.name.startswith("_"):
+                entry = self._load_entry_from_disk(node_dir.name)
+                if entry:
+                    entries.append(entry)
+        return entries
+
+    def _load_entry_from_disk(self, node_id: str) -> Optional[CacheEntry]:
+        """Load entry metadata from disk."""
+        node_dir = self._node_dir(node_id)
+        metadata_path = node_dir / "metadata.json"
+        output_file = self._find_output_file(node_dir)
+
+        if not output_file:
+            return None
+
+        if metadata_path.exists():
+            try:
+                with open(metadata_path) as f:
+                    data = json.load(f)
+                return CacheEntry.from_dict(data)
+            except (json.JSONDecodeError, KeyError):
+                pass
+
+        # Fallback: create entry from filesystem
+        return CacheEntry(
+            node_id=node_id,
+            output_path=output_file,
+            created_at=output_file.stat().st_mtime,
+            size_bytes=output_file.stat().st_size,
+            node_type="unknown",
+            cid=_file_hash(output_file),
+        )
+
+    def get_entry(self, node_id: str) -> Optional[CacheEntry]:
+        """Get cache entry metadata (without affecting stats)."""
+        return self._load_entry_from_disk(node_id)
+
+    def find_by_cid(self, cid: str) -> Optional[CacheEntry]:
+        """Find a cache entry by its content hash (scans filesystem)."""
+        for entry in self.list_entries():
+            if entry.cid == cid:
+                return entry
+        return None
+
+    def prune(self, max_size_bytes: int = None, max_age_seconds: float = None) -> int:
+        """
+        Prune cache based on size or age.
+
+        Args:
+            max_size_bytes: Remove oldest entries until under this size
+            max_age_seconds: Remove entries older than this
+
+        Returns:
+            Number of entries removed
+        """
+        removed = 0
+        now = time.time()
+        entries = self.list_entries()
+
+        # Remove by age first
+        if max_age_seconds is not None:
+            for entry in entries:
+                if now - entry.created_at > max_age_seconds:
+                    self.remove(entry.node_id)
+                    removed += 1
+
+        # Then by size (remove oldest first)
+        if max_size_bytes is not None:
+            stats = self.get_stats()
+            if stats.total_size_bytes > max_size_bytes:
+                sorted_entries = sorted(entries, key=lambda e: e.created_at)
+                total_size = stats.total_size_bytes
+                for entry in sorted_entries:
+                    if total_size <= max_size_bytes:
+                        break
+                    self.remove(entry.node_id)
+                    total_size -= entry.size_bytes
+                    removed += 1
+
+        return removed
+
+    def get_output_path(self, node_id: str, extension: str = ".mkv") -> Path:
+        """Get the output path for a node (creates directory if needed)."""
+        node_dir = self._node_dir(node_id)
+        node_dir.mkdir(parents=True, exist_ok=True)
+        return node_dir / f"output{extension}"
+
+    # Effect storage methods
+
+    def _effects_dir(self) -> Path:
+        """Get the effects subdirectory."""
+        effects_dir = self.cache_dir / "_effects"
+        effects_dir.mkdir(parents=True, exist_ok=True)
+        return effects_dir
+
+    def store_effect(self, source: str) -> str:
+        """
+        Store an effect in the cache.
+
+        Args:
+            source: Effect source code
+
+        Returns:
+            Content hash (cache ID) of the effect
+        """
+        import hashlib as _hashlib
+
+        # Compute content hash
+        cid = _hashlib.sha3_256(source.encode("utf-8")).hexdigest()
+
+        # Try to load full metadata if effects module available
+        try:
+            from .effects.loader import load_effect
+            loaded = load_effect(source)
+            meta_dict = loaded.meta.to_dict()
+            dependencies = loaded.dependencies
+            requires_python = loaded.requires_python
+        except ImportError:
+            # Fallback: store without parsed metadata
+            meta_dict = {}
+            dependencies = []
+            requires_python = ">=3.10"
+
+        effect_dir = self._effects_dir() / cid
+        effect_dir.mkdir(parents=True, exist_ok=True)
+
+        # Store source
+        source_path = effect_dir / "effect.py"
+        source_path.write_text(source, encoding="utf-8")
+
+        # Store metadata
+        metadata = {
+            "cid": cid,
+            "meta": meta_dict,
+            "dependencies": dependencies,
+            "requires_python": requires_python,
+            "stored_at": time.time(),
+        }
+        metadata_path = effect_dir / "metadata.json"
+        with open(metadata_path, "w") as f:
+            json.dump(metadata, f, indent=2)
+
+        logger.info(f"Stored effect '{loaded.meta.name}' with hash {cid[:16]}...")
+        return cid
+
+    def get_effect(self, cid: str) -> Optional[str]:
+        """
+        Get effect source by content hash.
+
+        Args:
+            cid: SHA3-256 hash of effect source
+
+        Returns:
+            Effect source code if found, None otherwise
+        """
+        effect_dir = self._effects_dir() / cid
+        source_path = effect_dir / "effect.py"
+
+        if not source_path.exists():
+            return None
+
+        return source_path.read_text(encoding="utf-8")
+
+    def get_effect_path(self, cid: str) -> Optional[Path]:
+        """
+        Get path to effect source file.
+
+        Args:
+            cid: SHA3-256 hash of effect source
+
+        Returns:
+            Path to effect.py if found, None otherwise
+        """
+        effect_dir = self._effects_dir() / cid
+        source_path = effect_dir / "effect.py"
+
+        if not source_path.exists():
+            return None
+
+        return source_path
+
+    def get_effect_metadata(self, cid: str) -> Optional[dict]:
+        """
+        Get effect metadata by content hash.
+
+        Args:
+            cid: SHA3-256 hash of effect source
+
+        Returns:
+            Metadata dict if found, None otherwise
+        """
+        effect_dir = self._effects_dir() / cid
+        metadata_path = effect_dir / "metadata.json"
+
+        if not metadata_path.exists():
+            return None
+
+        try:
+            with open(metadata_path) as f:
+                return json.load(f)
+        except (json.JSONDecodeError, KeyError):
+            return None
+
+    def has_effect(self, cid: str) -> bool:
+        """Check if an effect is cached."""
+        effect_dir = self._effects_dir() / cid
+        return (effect_dir / "effect.py").exists()
+
+    def list_effects(self) -> List[dict]:
+        """List all cached effects with their metadata."""
+        effects = []
+        effects_dir = self._effects_dir()
+
+        if not effects_dir.exists():
+            return effects
+
+        for effect_dir in effects_dir.iterdir():
+            if effect_dir.is_dir():
+                metadata = self.get_effect_metadata(effect_dir.name)
+                if metadata:
+                    effects.append(metadata)
+
+        return effects
+
+    def remove_effect(self, cid: str) -> bool:
+        """Remove an effect from the cache."""
+        effect_dir = self._effects_dir() / cid
+
+        if not effect_dir.exists():
+            return False
+
+        shutil.rmtree(effect_dir)
+        logger.info(f"Removed effect {cid[:16]}...")
+        return True

artdag/core/artdag/cli.py

@@ -0,0 +1,724 @@
+#!/usr/bin/env python3
+"""
+Art DAG CLI
+
+Command-line interface for the 3-phase execution model:
+  artdag analyze - Extract features from inputs
+  artdag plan - Generate execution plan
+  artdag execute - Run the plan
+  artdag run-recipe - Full pipeline
+
+Usage:
+  artdag analyze <recipe> -i <name>:<hash>[@<path>] [--features <list>]
+  artdag plan <recipe> -i <name>:<hash> [--analysis <file>]
+  artdag execute <plan.json> [--dry-run]
+  artdag run-recipe <recipe> -i <name>:<hash>[@<path>]
+"""
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+
+
+def parse_input(input_str: str) -> Tuple[str, str, Optional[str]]:
+    """
+    Parse input specification: name:hash[@path]
+
+    Returns (name, hash, path or None)
+    """
+    if "@" in input_str:
+        name_hash, path = input_str.rsplit("@", 1)
+    else:
+        name_hash = input_str
+        path = None
+
+    if ":" not in name_hash:
+        raise ValueError(f"Invalid input format: {input_str}. Expected name:hash[@path]")
+
+    name, hash_value = name_hash.split(":", 1)
+    return name, hash_value, path
+
+
+def parse_inputs(input_list: List[str]) -> Tuple[Dict[str, str], Dict[str, str]]:
+    """
+    Parse list of input specifications.
+
+    Returns (input_hashes, input_paths)
+    """
+    input_hashes = {}
+    input_paths = {}
+
+    for input_str in input_list:
+        name, hash_value, path = parse_input(input_str)
+        input_hashes[name] = hash_value
+        if path:
+            input_paths[name] = path
+
+    return input_hashes, input_paths
+
+
+def cmd_analyze(args):
+    """Run analysis phase."""
+    from .analysis import Analyzer
+
+    # Parse inputs
+    input_hashes, input_paths = parse_inputs(args.input)
+
+    # Parse features
+    features = args.features.split(",") if args.features else ["all"]
+
+    # Create analyzer
+    cache_dir = Path(args.cache_dir) if args.cache_dir else Path("./analysis_cache")
+    analyzer = Analyzer(cache_dir=cache_dir)
+
+    # Analyze each input
+    results = {}
+    for name, hash_value in input_hashes.items():
+        path = input_paths.get(name)
+        if path:
+            path = Path(path)
+
+        print(f"Analyzing {name} ({hash_value[:16]}...)...")
+
+        result = analyzer.analyze(
+            input_hash=hash_value,
+            features=features,
+            input_path=path,
+        )
+
+        results[hash_value] = result.to_dict()
+
+        # Print summary
+        if result.audio and result.audio.beats:
+            print(f"  Tempo: {result.audio.beats.tempo:.1f} BPM")
+            print(f"  Beats: {len(result.audio.beats.beat_times)}")
+        if result.video:
+            print(f"  Duration: {result.video.duration:.1f}s")
+            print(f"  Dimensions: {result.video.width}x{result.video.height}")
+
+    # Write output
+    output_path = Path(args.output) if args.output else Path("analysis.json")
+    with open(output_path, "w") as f:
+        json.dump(results, f, indent=2)
+
+    print(f"\nAnalysis saved to: {output_path}")
+
+
+def cmd_plan(args):
+    """Run planning phase."""
+    from .analysis import AnalysisResult
+    from .planning import RecipePlanner, Recipe
+
+    # Load recipe
+    recipe = Recipe.from_file(Path(args.recipe))
+    print(f"Recipe: {recipe.name} v{recipe.version}")
+
+    # Parse inputs
+    input_hashes, _ = parse_inputs(args.input)
+
+    # Load analysis if provided
+    analysis = {}
+    if args.analysis:
+        with open(args.analysis, "r") as f:
+            analysis_data = json.load(f)
+        for hash_value, data in analysis_data.items():
+            analysis[hash_value] = AnalysisResult.from_dict(data)
+
+    # Create planner
+    planner = RecipePlanner(use_tree_reduction=not args.no_tree_reduction)
+
+    # Generate plan
+    print("Generating execution plan...")
+    plan = planner.plan(
+        recipe=recipe,
+        input_hashes=input_hashes,
+        analysis=analysis,
+    )
+
+    # Print summary
+    print(f"\nPlan ID: {plan.plan_id[:16]}...")
+    print(f"Steps: {len(plan.steps)}")
+
+    steps_by_level = plan.get_steps_by_level()
+    max_level = max(steps_by_level.keys()) if steps_by_level else 0
+    print(f"Levels: {max_level + 1}")
+
+    for level in sorted(steps_by_level.keys()):
+        steps = steps_by_level[level]
+        print(f"  Level {level}: {len(steps)} steps (parallel)")
+
+    # Write output
+    output_path = Path(args.output) if args.output else Path("plan.json")
+    with open(output_path, "w") as f:
+        f.write(plan.to_json())
+
+    print(f"\nPlan saved to: {output_path}")
+
+
+def cmd_execute(args):
+    """Run execution phase."""
+    from .planning import ExecutionPlan
+    from .cache import Cache
+    from .executor import get_executor
+    from .dag import NodeType
+    from . import nodes  # Register built-in executors
+
+    # Load plan
+    with open(args.plan, "r") as f:
+        plan = ExecutionPlan.from_json(f.read())
+
+    print(f"Executing plan: {plan.plan_id[:16]}...")
+    print(f"Steps: {len(plan.steps)}")
+
+    if args.dry_run:
+        print("\n=== DRY RUN ===")
+
+        # Check cache status
+        cache = Cache(Path(args.cache_dir) if args.cache_dir else Path("./cache"))
+        steps_by_level = plan.get_steps_by_level()
+
+        cached_count = 0
+        pending_count = 0
+
+        for level in sorted(steps_by_level.keys()):
+            steps = steps_by_level[level]
+            print(f"\nLevel {level}:")
+            for step in steps:
+                if cache.has(step.cache_id):
+                    print(f"  [CACHED] {step.step_id}: {step.node_type}")
+                    cached_count += 1
+                else:
+                    print(f"  [PENDING] {step.step_id}: {step.node_type}")
+                    pending_count += 1
+
+        print(f"\nSummary: {cached_count} cached, {pending_count} pending")
+        return
+
+    # Execute locally (for testing - production uses Celery)
+    cache = Cache(Path(args.cache_dir) if args.cache_dir else Path("./cache"))
+
+    cache_paths = {}
+    for name, hash_value in plan.input_hashes.items():
+        if cache.has(hash_value):
+            entry = cache.get(hash_value)
+            cache_paths[hash_value] = str(entry.output_path)
+
+    steps_by_level = plan.get_steps_by_level()
+    executed = 0
+    cached = 0
+
+    for level in sorted(steps_by_level.keys()):
+        steps = steps_by_level[level]
+        print(f"\nLevel {level}: {len(steps)} steps")
+
+        for step in steps:
+            if cache.has(step.cache_id):
+                cached_path = cache.get(step.cache_id)
+                cache_paths[step.cache_id] = str(cached_path)
+                cache_paths[step.step_id] = str(cached_path)
+                print(f"  [CACHED] {step.step_id}")
+                cached += 1
+                continue
+
+            print(f"  [RUNNING] {step.step_id}: {step.node_type}...")
+
+            # Get executor
+            try:
+                node_type = NodeType[step.node_type]
+            except KeyError:
+                node_type = step.node_type
+
+            executor = get_executor(node_type)
+            if executor is None:
+                print(f"    ERROR: No executor for {step.node_type}")
+                continue
+
+            # Resolve inputs
+            input_paths = []
+            for input_id in step.input_steps:
+                if input_id in cache_paths:
+                    input_paths.append(Path(cache_paths[input_id]))
+                else:
+                    input_step = plan.get_step(input_id)
+                    if input_step and input_step.cache_id in cache_paths:
+                        input_paths.append(Path(cache_paths[input_step.cache_id]))
+
+            if len(input_paths) != len(step.input_steps):
+                print(f"    ERROR: Missing inputs")
+                continue
+
+            # Execute
+            output_path = cache.get_output_path(step.cache_id)
+            try:
+                result_path = executor.execute(step.config, input_paths, output_path)
+                cache.put(step.cache_id, result_path, node_type=step.node_type)
+                cache_paths[step.cache_id] = str(result_path)
+                cache_paths[step.step_id] = str(result_path)
+                print(f"    [DONE] -> {result_path}")
+                executed += 1
+            except Exception as e:
+                print(f"    [FAILED] {e}")
+
+    # Final output
+    output_step = plan.get_step(plan.output_step)
+    output_path = cache_paths.get(output_step.cache_id) if output_step else None
+
+    print(f"\n=== Complete ===")
+    print(f"Cached: {cached}")
+    print(f"Executed: {executed}")
+    if output_path:
+        print(f"Output: {output_path}")
+
+
+def cmd_run_recipe(args):
+    """Run complete pipeline: analyze → plan → execute."""
+    from .analysis import Analyzer, AnalysisResult
+    from .planning import RecipePlanner, Recipe
+    from .cache import Cache
+    from .executor import get_executor
+    from .dag import NodeType
+    from . import nodes  # Register built-in executors
+
+    # Load recipe
+    recipe = Recipe.from_file(Path(args.recipe))
+    print(f"Recipe: {recipe.name} v{recipe.version}")
+
+    # Parse inputs
+    input_hashes, input_paths = parse_inputs(args.input)
+
+    # Parse features
+    features = args.features.split(",") if args.features else ["beats", "energy"]
+
+    cache_dir = Path(args.cache_dir) if args.cache_dir else Path("./cache")
+
+    # Phase 1: Analyze
+    print("\n=== Phase 1: Analysis ===")
+    analyzer = Analyzer(cache_dir=cache_dir / "analysis")
+
+    analysis = {}
+    for name, hash_value in input_hashes.items():
+        path = input_paths.get(name)
+        if path:
+            path = Path(path)
+            print(f"Analyzing {name}...")
+
+            result = analyzer.analyze(
+                input_hash=hash_value,
+                features=features,
+                input_path=path,
+            )
+            analysis[hash_value] = result
+
+            if result.audio and result.audio.beats:
+                print(f"  Tempo: {result.audio.beats.tempo:.1f} BPM, {len(result.audio.beats.beat_times)} beats")
+
+    # Phase 2: Plan
+    print("\n=== Phase 2: Planning ===")
+
+    # Check for cached plan
+    plans_dir = cache_dir / "plans"
+    plans_dir.mkdir(parents=True, exist_ok=True)
+
+    # Generate plan to get plan_id (deterministic hash)
+    planner = RecipePlanner(use_tree_reduction=True)
+    plan = planner.plan(
+        recipe=recipe,
+        input_hashes=input_hashes,
+        analysis=analysis,
+    )
+
+    plan_cache_path = plans_dir / f"{plan.plan_id}.json"
+
+    if plan_cache_path.exists():
+        print(f"Plan cached: {plan.plan_id[:16]}...")
+        from .planning import ExecutionPlan
+        with open(plan_cache_path, "r") as f:
+            plan = ExecutionPlan.from_json(f.read())
+    else:
+        # Save plan to cache
+        with open(plan_cache_path, "w") as f:
+            f.write(plan.to_json())
+        print(f"Plan saved: {plan.plan_id[:16]}...")
+
+    print(f"Plan: {len(plan.steps)} steps")
+    steps_by_level = plan.get_steps_by_level()
+    print(f"Levels: {len(steps_by_level)}")
+
+    # Phase 3: Execute
+    print("\n=== Phase 3: Execution ===")
+
+    cache = Cache(cache_dir)
+
+    # Build initial cache paths
+    cache_paths = {}
+    for name, hash_value in input_hashes.items():
+        path = input_paths.get(name)
+        if path:
+            cache_paths[hash_value] = path
+            cache_paths[name] = path
+
+    executed = 0
+    cached = 0
+
+    for level in sorted(steps_by_level.keys()):
+        steps = steps_by_level[level]
+        print(f"\nLevel {level}: {len(steps)} steps")
+
+        for step in steps:
+            if cache.has(step.cache_id):
+                cached_path = cache.get(step.cache_id)
+                cache_paths[step.cache_id] = str(cached_path)
+                cache_paths[step.step_id] = str(cached_path)
+                print(f"  [CACHED] {step.step_id}")
+                cached += 1
+                continue
+
+            # Handle SOURCE specially
+            if step.node_type == "SOURCE":
+                cid = step.config.get("cid")
+                if cid in cache_paths:
+                    cache_paths[step.cache_id] = cache_paths[cid]
+                    cache_paths[step.step_id] = cache_paths[cid]
+                    print(f"  [SOURCE] {step.step_id}")
+                    continue
+
+            print(f"  [RUNNING] {step.step_id}: {step.node_type}...")
+
+            try:
+                node_type = NodeType[step.node_type]
+            except KeyError:
+                node_type = step.node_type
+
+            executor = get_executor(node_type)
+            if executor is None:
+                print(f"    SKIP: No executor for {step.node_type}")
+                continue
+
+            # Resolve inputs
+            input_paths_list = []
+            for input_id in step.input_steps:
+                if input_id in cache_paths:
+                    input_paths_list.append(Path(cache_paths[input_id]))
+                else:
+                    input_step = plan.get_step(input_id)
+                    if input_step and input_step.cache_id in cache_paths:
+                        input_paths_list.append(Path(cache_paths[input_step.cache_id]))
+
+            if len(input_paths_list) != len(step.input_steps):
+                print(f"    ERROR: Missing inputs for {step.step_id}")
+                continue
+
+            output_path = cache.get_output_path(step.cache_id)
+            try:
+                result_path = executor.execute(step.config, input_paths_list, output_path)
+                cache.put(step.cache_id, result_path, node_type=step.node_type)
+                cache_paths[step.cache_id] = str(result_path)
+                cache_paths[step.step_id] = str(result_path)
+                print(f"    [DONE]")
+                executed += 1
+            except Exception as e:
+                print(f"    [FAILED] {e}")
+
+    # Final output
+    output_step = plan.get_step(plan.output_step)
+    output_path = cache_paths.get(output_step.cache_id) if output_step else None
+
+    print(f"\n=== Complete ===")
+    print(f"Cached: {cached}")
+    print(f"Executed: {executed}")
+    if output_path:
+        print(f"Output: {output_path}")
+
+
+def cmd_run_recipe_ipfs(args):
+    """Run complete pipeline with IPFS-primary mode.
+
+    Everything stored on IPFS:
+    - Inputs (media files)
+    - Analysis results (JSON)
+    - Execution plans (JSON)
+    - Step outputs (media files)
+    """
+    import hashlib
+    import shutil
+    import tempfile
+
+    from .analysis import Analyzer, AnalysisResult
+    from .planning import RecipePlanner, Recipe, ExecutionPlan
+    from .executor import get_executor
+    from .dag import NodeType
+    from . import nodes  # Register built-in executors
+
+    # Check for ipfs_client
+    try:
+        from art_celery import ipfs_client
+    except ImportError:
+        # Try relative import for when running from art-celery
+        try:
+            import ipfs_client
+        except ImportError:
+            print("Error: ipfs_client not available. Install art-celery or run from art-celery directory.")
+            sys.exit(1)
+
+    # Check IPFS availability
+    if not ipfs_client.is_available():
+        print("Error: IPFS daemon not available. Start IPFS with 'ipfs daemon'")
+        sys.exit(1)
+
+    print("=== IPFS-Primary Mode ===")
+    print(f"IPFS Node: {ipfs_client.get_node_id()[:16]}...")
+
+    # Load recipe
+    recipe_path = Path(args.recipe)
+    recipe = Recipe.from_file(recipe_path)
+    print(f"\nRecipe: {recipe.name} v{recipe.version}")
+
+    # Parse inputs
+    input_hashes, input_paths = parse_inputs(args.input)
+
+    # Parse features
+    features = args.features.split(",") if args.features else ["beats", "energy"]
+
+    # Phase 0: Register on IPFS
+    print("\n=== Phase 0: Register on IPFS ===")
+
+    # Register recipe
+    recipe_bytes = recipe_path.read_bytes()
+    recipe_cid = ipfs_client.add_bytes(recipe_bytes)
+    print(f"Recipe CID: {recipe_cid}")
+
+    # Register inputs
+    input_cids = {}
+    for name, hash_value in input_hashes.items():
+        path = input_paths.get(name)
+        if path:
+            cid = ipfs_client.add_file(Path(path))
+            if cid:
+                input_cids[name] = cid
+                print(f"Input '{name}': {cid}")
+            else:
+                print(f"Error: Failed to add input '{name}' to IPFS")
+                sys.exit(1)
+
+    # Phase 1: Analyze
+    print("\n=== Phase 1: Analysis ===")
+
+    # Create temp dir for analysis
+    work_dir = Path(tempfile.mkdtemp(prefix="artdag_ipfs_"))
+    analysis_cids = {}
+    analysis = {}
+
+    try:
+        for name, hash_value in input_hashes.items():
+            input_cid = input_cids.get(name)
+            if not input_cid:
+                continue
+
+            print(f"Analyzing {name}...")
+
+            # Fetch from IPFS to temp
+            temp_input = work_dir / f"input_{name}.mkv"
+            if not ipfs_client.get_file(input_cid, temp_input):
+                print(f"  Error: Failed to fetch from IPFS")
+                continue
+
+            # Run analysis
+            analyzer = Analyzer(cache_dir=None)
+            result = analyzer.analyze(
+                input_hash=hash_value,
+                features=features,
+                input_path=temp_input,
+            )
+
+            if result.audio and result.audio.beats:
+                print(f"  Tempo: {result.audio.beats.tempo:.1f} BPM, {len(result.audio.beats.beat_times)} beats")
+
+            # Store analysis on IPFS
+            analysis_cid = ipfs_client.add_json(result.to_dict())
+            if analysis_cid:
+                analysis_cids[hash_value] = analysis_cid
+                analysis[hash_value] = result
+                print(f"  Analysis CID: {analysis_cid}")
+
+        # Phase 2: Plan
+        print("\n=== Phase 2: Planning ===")
+
+        planner = RecipePlanner(use_tree_reduction=True)
+        plan = planner.plan(
+            recipe=recipe,
+            input_hashes=input_hashes,
+            analysis=analysis if analysis else None,
+        )
+
+        # Store plan on IPFS
+        import json
+        plan_dict = json.loads(plan.to_json())
+        plan_cid = ipfs_client.add_json(plan_dict)
+        print(f"Plan ID: {plan.plan_id[:16]}...")
+        print(f"Plan CID: {plan_cid}")
+        print(f"Steps: {len(plan.steps)}")
+
+        steps_by_level = plan.get_steps_by_level()
+        print(f"Levels: {len(steps_by_level)}")
+
+        # Phase 3: Execute
+        print("\n=== Phase 3: Execution ===")
+
+        # CID results
+        cid_results = dict(input_cids)
+        step_cids = {}
+
+        executed = 0
+        cached = 0
+
+        for level in sorted(steps_by_level.keys()):
+            steps = steps_by_level[level]
+            print(f"\nLevel {level}: {len(steps)} steps")
+
+            for step in steps:
+                # Handle SOURCE
+                if step.node_type == "SOURCE":
+                    source_name = step.config.get("name") or step.step_id
+                    cid = cid_results.get(source_name)
+                    if cid:
+                        step_cids[step.step_id] = cid
+                        print(f"  [SOURCE] {step.step_id}")
+                    continue
+
+                print(f"  [RUNNING] {step.step_id}: {step.node_type}...")
+
+                try:
+                    node_type = NodeType[step.node_type]
+                except KeyError:
+                    node_type = step.node_type
+
+                executor = get_executor(node_type)
+                if executor is None:
+                    print(f"    SKIP: No executor for {step.node_type}")
+                    continue
+
+                # Fetch inputs from IPFS
+                input_paths_list = []
+                for i, input_step_id in enumerate(step.input_steps):
+                    input_cid = step_cids.get(input_step_id) or cid_results.get(input_step_id)
+                    if not input_cid:
+                        print(f"    ERROR: Missing input CID for {input_step_id}")
+                        continue
+
+                    temp_path = work_dir / f"step_{step.step_id}_input_{i}.mkv"
+                    if not ipfs_client.get_file(input_cid, temp_path):
+                        print(f"    ERROR: Failed to fetch {input_cid}")
+                        continue
+                    input_paths_list.append(temp_path)
+
+                if len(input_paths_list) != len(step.input_steps):
+                    print(f"    ERROR: Missing inputs")
+                    continue
+
+                # Execute
+                output_path = work_dir / f"step_{step.step_id}_output.mkv"
+                try:
+                    result_path = executor.execute(step.config, input_paths_list, output_path)
+
+                    # Add to IPFS
+                    output_cid = ipfs_client.add_file(result_path)
+                    if output_cid:
+                        step_cids[step.step_id] = output_cid
+                        print(f"    [DONE] CID: {output_cid}")
+                        executed += 1
+                    else:
+                        print(f"    [FAILED] Could not add to IPFS")
+                except Exception as e:
+                    print(f"    [FAILED] {e}")
+
+        # Final output
+        output_step = plan.get_step(plan.output_step)
+        output_cid = step_cids.get(output_step.step_id) if output_step else None
+
+        print(f"\n=== Complete ===")
+        print(f"Executed: {executed}")
+        if output_cid:
+            print(f"Output CID: {output_cid}")
+            print(f"Fetch with: ipfs get {output_cid}")
+
+        # Summary of all CIDs
+        print(f"\n=== All CIDs ===")
+        print(f"Recipe: {recipe_cid}")
+        print(f"Plan: {plan_cid}")
+        for name, cid in input_cids.items():
+            print(f"Input '{name}': {cid}")
+        for hash_val, cid in analysis_cids.items():
+            print(f"Analysis '{hash_val[:16]}...': {cid}")
+        if output_cid:
+            print(f"Output: {output_cid}")
+
+    finally:
+        # Cleanup temp
+        shutil.rmtree(work_dir, ignore_errors=True)
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog="artdag",
+        description="Art DAG - Declarative media composition",
+    )
+    subparsers = parser.add_subparsers(dest="command", help="Commands")
+
+    # analyze command
+    analyze_parser = subparsers.add_parser("analyze", help="Extract features from inputs")
+    analyze_parser.add_argument("recipe", help="Recipe YAML file")
+    analyze_parser.add_argument("-i", "--input", action="append", required=True,
+                                help="Input: name:hash[@path]")
+    analyze_parser.add_argument("--features", help="Features to extract (comma-separated)")
+    analyze_parser.add_argument("-o", "--output", help="Output file (default: analysis.json)")
+    analyze_parser.add_argument("--cache-dir", help="Analysis cache directory")
+
+    # plan command
+    plan_parser = subparsers.add_parser("plan", help="Generate execution plan")
+    plan_parser.add_argument("recipe", help="Recipe YAML file")
+    plan_parser.add_argument("-i", "--input", action="append", required=True,
+                             help="Input: name:hash")
+    plan_parser.add_argument("--analysis", help="Analysis JSON file")
+    plan_parser.add_argument("-o", "--output", help="Output file (default: plan.json)")
+    plan_parser.add_argument("--no-tree-reduction", action="store_true",
+                             help="Disable tree reduction optimization")
+
+    # execute command
+    execute_parser = subparsers.add_parser("execute", help="Execute a plan")
+    execute_parser.add_argument("plan", help="Plan JSON file")
+    execute_parser.add_argument("--dry-run", action="store_true",
+                                help="Show what would execute")
+    execute_parser.add_argument("--cache-dir", help="Cache directory")
+
+    # run-recipe command
+    run_parser = subparsers.add_parser("run-recipe", help="Full pipeline: analyze → plan → execute")
+    run_parser.add_argument("recipe", help="Recipe YAML file")
+    run_parser.add_argument("-i", "--input", action="append", required=True,
+                            help="Input: name:hash[@path]")
+    run_parser.add_argument("--features", help="Features to extract (comma-separated)")
+    run_parser.add_argument("--cache-dir", help="Cache directory")
+    run_parser.add_argument("--ipfs-primary", action="store_true",
+                            help="Use IPFS-primary mode (everything on IPFS, no local cache)")
+
+    args = parser.parse_args()
+
+    if args.command == "analyze":
+        cmd_analyze(args)
+    elif args.command == "plan":
+        cmd_plan(args)
+    elif args.command == "execute":
+        cmd_execute(args)
+    elif args.command == "run-recipe":
+        if getattr(args, 'ipfs_primary', False):
+            cmd_run_recipe_ipfs(args)
+        else:
+            cmd_run_recipe(args)
+    else:
+        parser.print_help()
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

artdag/core/artdag/client.py

@@ -0,0 +1,201 @@
+# primitive/client.py
+"""
+Client SDK for the primitive execution server.
+
+Provides a simple API for submitting DAGs and retrieving results.
+
+Usage:
+    client = PrimitiveClient("http://localhost:8080")
+
+    # Build a DAG
+    builder = DAGBuilder()
+    source = builder.source("/path/to/video.mp4")
+    segment = builder.segment(source, duration=5.0)
+    builder.set_output(segment)
+    dag = builder.build()
+
+    # Execute and wait for result
+    result = client.execute(dag)
+    print(f"Output: {result.output_path}")
+"""
+
+import json
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Optional
+from urllib.request import urlopen, Request
+from urllib.error import HTTPError, URLError
+
+from .dag import DAG, DAGBuilder
+
+
+@dataclass
+class ExecutionResult:
+    """Result from server execution."""
+    success: bool
+    output_path: Optional[Path] = None
+    error: Optional[str] = None
+    execution_time: float = 0.0
+    nodes_executed: int = 0
+    nodes_cached: int = 0
+
+
+@dataclass
+class CacheStats:
+    """Cache statistics from server."""
+    total_entries: int = 0
+    total_size_bytes: int = 0
+    hits: int = 0
+    misses: int = 0
+    hit_rate: float = 0.0
+
+
+class PrimitiveClient:
+    """
+    Client for the primitive execution server.
+
+    Args:
+        base_url: Server URL (e.g., "http://localhost:8080")
+        timeout: Request timeout in seconds
+    """
+
+    def __init__(self, base_url: str = "http://localhost:8080", timeout: float = 300):
+        self.base_url = base_url.rstrip("/")
+        self.timeout = timeout
+
+    def _request(self, method: str, path: str, data: dict = None) -> dict:
+        """Make HTTP request to server."""
+        url = f"{self.base_url}{path}"
+
+        if data is not None:
+            body = json.dumps(data).encode()
+            headers = {"Content-Type": "application/json"}
+        else:
+            body = None
+            headers = {}
+
+        req = Request(url, data=body, headers=headers, method=method)
+
+        try:
+            with urlopen(req, timeout=self.timeout) as response:
+                return json.loads(response.read().decode())
+        except HTTPError as e:
+            error_body = e.read().decode()
+            try:
+                error_data = json.loads(error_body)
+                raise RuntimeError(error_data.get("error", str(e)))
+            except json.JSONDecodeError:
+                raise RuntimeError(f"HTTP {e.code}: {error_body}")
+        except URLError as e:
+            raise ConnectionError(f"Failed to connect to server: {e}")
+
+    def health(self) -> bool:
+        """Check if server is healthy."""
+        try:
+            result = self._request("GET", "/health")
+            return result.get("status") == "ok"
+        except Exception:
+            return False
+
+    def submit(self, dag: DAG) -> str:
+        """
+        Submit a DAG for execution.
+
+        Args:
+            dag: The DAG to execute
+
+        Returns:
+            Job ID for tracking
+        """
+        result = self._request("POST", "/execute", dag.to_dict())
+        return result["job_id"]
+
+    def status(self, job_id: str) -> str:
+        """
+        Get job status.
+
+        Args:
+            job_id: Job ID from submit()
+
+        Returns:
+            Status: "pending", "running", "completed", or "failed"
+        """
+        result = self._request("GET", f"/status/{job_id}")
+        return result["status"]
+
+    def result(self, job_id: str) -> Optional[ExecutionResult]:
+        """
+        Get job result (non-blocking).
+
+        Args:
+            job_id: Job ID from submit()
+
+        Returns:
+            ExecutionResult if complete, None if still running
+        """
+        data = self._request("GET", f"/result/{job_id}")
+
+        if not data.get("ready", False):
+            return None
+
+        return ExecutionResult(
+            success=data.get("success", False),
+            output_path=Path(data["output_path"]) if data.get("output_path") else None,
+            error=data.get("error"),
+            execution_time=data.get("execution_time", 0),
+            nodes_executed=data.get("nodes_executed", 0),
+            nodes_cached=data.get("nodes_cached", 0),
+        )
+
+    def wait(self, job_id: str, poll_interval: float = 0.5) -> ExecutionResult:
+        """
+        Wait for job completion and return result.
+
+        Args:
+            job_id: Job ID from submit()
+            poll_interval: Seconds between status checks
+
+        Returns:
+            ExecutionResult
+        """
+        while True:
+            result = self.result(job_id)
+            if result is not None:
+                return result
+            time.sleep(poll_interval)
+
+    def execute(self, dag: DAG, poll_interval: float = 0.5) -> ExecutionResult:
+        """
+        Submit DAG and wait for result.
+
+        Convenience method combining submit() and wait().
+
+        Args:
+            dag: The DAG to execute
+            poll_interval: Seconds between status checks
+
+        Returns:
+            ExecutionResult
+        """
+        job_id = self.submit(dag)
+        return self.wait(job_id, poll_interval)
+
+    def cache_stats(self) -> CacheStats:
+        """Get cache statistics."""
+        data = self._request("GET", "/cache/stats")
+        return CacheStats(
+            total_entries=data.get("total_entries", 0),
+            total_size_bytes=data.get("total_size_bytes", 0),
+            hits=data.get("hits", 0),
+            misses=data.get("misses", 0),
+            hit_rate=data.get("hit_rate", 0.0),
+        )
+
+    def clear_cache(self) -> None:
+        """Clear the server cache."""
+        self._request("DELETE", "/cache")
+
+
+# Re-export DAGBuilder for convenience
+__all__ = ["PrimitiveClient", "ExecutionResult", "CacheStats", "DAGBuilder"]

artdag/core/artdag/dag.py

@@ -0,0 +1,344 @@
+# primitive/dag.py
+"""
+Core DAG data structures.
+
+Nodes are content-addressed: node_id = hash(type + config + input_ids)
+This enables automatic caching and deduplication.
+"""
+
+import hashlib
+import json
+from dataclasses import dataclass, field
+from enum import Enum, auto
+from typing import Any, Dict, List, Optional
+
+
+class NodeType(Enum):
+    """Built-in node types."""
+    # Source operations
+    SOURCE = auto()      # Load file from path
+
+    # Transform operations
+    SEGMENT = auto()     # Extract time range
+    RESIZE = auto()      # Scale/crop/pad
+    TRANSFORM = auto()   # Visual effects (color, blur, etc.)
+
+    # Compose operations
+    SEQUENCE = auto()    # Concatenate in time
+    LAYER = auto()       # Stack spatially (overlay)
+    MUX = auto()         # Combine video + audio streams
+    BLEND = auto()       # Blend two inputs
+    AUDIO_MIX = auto()   # Mix multiple audio streams
+    SWITCH = auto()      # Time-based input switching
+
+    # Analysis operations
+    ANALYZE = auto()     # Extract features (audio, motion, etc.)
+
+    # Generation operations
+    GENERATE = auto()    # Create content (text, graphics, etc.)
+
+
+def _stable_hash(data: Any, algorithm: str = "sha3_256") -> str:
+    """
+    Create stable hash from arbitrary data.
+
+    Uses SHA-3 (Keccak) for quantum resistance.
+    Returns full hash - no truncation.
+
+    Args:
+        data: Data to hash (will be JSON serialized)
+        algorithm: Hash algorithm (default: sha3_256)
+
+    Returns:
+        Full hex digest
+    """
+    # Convert to JSON with sorted keys for stability
+    json_str = json.dumps(data, sort_keys=True, separators=(",", ":"))
+    hasher = hashlib.new(algorithm)
+    hasher.update(json_str.encode())
+    return hasher.hexdigest()
+
+
+@dataclass
+class Node:
+    """
+    A node in the execution DAG.
+
+    Attributes:
+        node_type: The operation type (NodeType enum or string for custom types)
+        config: Operation-specific configuration
+        inputs: List of input node IDs (resolved during execution)
+        node_id: Content-addressed ID (computed from type + config + inputs)
+        name: Optional human-readable name for debugging
+    """
+    node_type: NodeType | str
+    config: Dict[str, Any] = field(default_factory=dict)
+    inputs: List[str] = field(default_factory=list)
+    node_id: Optional[str] = None
+    name: Optional[str] = None
+
+    def __post_init__(self):
+        """Compute node_id if not provided."""
+        if self.node_id is None:
+            self.node_id = self._compute_id()
+
+    def _compute_id(self) -> str:
+        """Compute content-addressed ID from node contents."""
+        type_str = self.node_type.name if isinstance(self.node_type, NodeType) else str(self.node_type)
+        content = {
+            "type": type_str,
+            "config": self.config,
+            "inputs": sorted(self.inputs),  # Sort for stability
+        }
+        return _stable_hash(content)
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Serialize node to dictionary."""
+        type_str = self.node_type.name if isinstance(self.node_type, NodeType) else str(self.node_type)
+        return {
+            "node_id": self.node_id,
+            "node_type": type_str,
+            "config": self.config,
+            "inputs": self.inputs,
+            "name": self.name,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "Node":
+        """Deserialize node from dictionary."""
+        type_str = data["node_type"]
+        try:
+            node_type = NodeType[type_str]
+        except KeyError:
+            node_type = type_str  # Custom type as string
+
+        return cls(
+            node_type=node_type,
+            config=data.get("config", {}),
+            inputs=data.get("inputs", []),
+            node_id=data.get("node_id"),
+            name=data.get("name"),
+        )
+
+
+@dataclass
+class DAG:
+    """
+    A directed acyclic graph of nodes.
+
+    Attributes:
+        nodes: Dictionary mapping node_id -> Node
+        output_id: The ID of the final output node
+        metadata: Optional metadata about the DAG (source, version, etc.)
+    """
+    nodes: Dict[str, Node] = field(default_factory=dict)
+    output_id: Optional[str] = None
+    metadata: Dict[str, Any] = field(default_factory=dict)
+
+    def add_node(self, node: Node) -> str:
+        """Add a node to the DAG, returning its ID."""
+        if node.node_id in self.nodes:
+            # Node already exists (deduplication via content addressing)
+            return node.node_id
+        self.nodes[node.node_id] = node
+        return node.node_id
+
+    def set_output(self, node_id: str) -> None:
+        """Set the output node."""
+        if node_id not in self.nodes:
+            raise ValueError(f"Node {node_id} not in DAG")
+        self.output_id = node_id
+
+    def get_node(self, node_id: str) -> Node:
+        """Get a node by ID."""
+        if node_id not in self.nodes:
+            raise KeyError(f"Node {node_id} not found")
+        return self.nodes[node_id]
+
+    def topological_order(self) -> List[str]:
+        """Return nodes in topological order (dependencies first)."""
+        visited = set()
+        order = []
+
+        def visit(node_id: str):
+            if node_id in visited:
+                return
+            visited.add(node_id)
+            node = self.nodes[node_id]
+            for input_id in node.inputs:
+                visit(input_id)
+            order.append(node_id)
+
+        # Visit all nodes (not just output, in case of disconnected components)
+        for node_id in self.nodes:
+            visit(node_id)
+
+        return order
+
+    def validate(self) -> List[str]:
+        """Validate DAG structure. Returns list of errors (empty if valid)."""
+        errors = []
+
+        if self.output_id is None:
+            errors.append("No output node set")
+        elif self.output_id not in self.nodes:
+            errors.append(f"Output node {self.output_id} not in DAG")
+
+        # Check all input references are valid
+        for node_id, node in self.nodes.items():
+            for input_id in node.inputs:
+                if input_id not in self.nodes:
+                    errors.append(f"Node {node_id} references missing input {input_id}")
+
+        # Check for cycles (skip if we already found missing inputs)
+        if not any("missing" in e for e in errors):
+            try:
+                self.topological_order()
+            except (RecursionError, KeyError):
+                errors.append("DAG contains cycles or invalid references")
+
+        return errors
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Serialize DAG to dictionary."""
+        return {
+            "nodes": {nid: node.to_dict() for nid, node in self.nodes.items()},
+            "output_id": self.output_id,
+            "metadata": self.metadata,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "DAG":
+        """Deserialize DAG from dictionary."""
+        dag = cls(metadata=data.get("metadata", {}))
+        for node_data in data.get("nodes", {}).values():
+            dag.add_node(Node.from_dict(node_data))
+        dag.output_id = data.get("output_id")
+        return dag
+
+    def to_json(self) -> str:
+        """Serialize DAG to JSON string."""
+        return json.dumps(self.to_dict(), indent=2)
+
+    @classmethod
+    def from_json(cls, json_str: str) -> "DAG":
+        """Deserialize DAG from JSON string."""
+        return cls.from_dict(json.loads(json_str))
+
+
+class DAGBuilder:
+    """
+    Fluent builder for constructing DAGs.
+
+    Example:
+        builder = DAGBuilder()
+        source = builder.source("/path/to/video.mp4")
+        segment = builder.segment(source, duration=5.0)
+        builder.set_output(segment)
+        dag = builder.build()
+    """
+
+    def __init__(self):
+        self.dag = DAG()
+
+    def _add(self, node_type: NodeType | str, config: Dict[str, Any],
+             inputs: List[str] = None, name: str = None) -> str:
+        """Add a node and return its ID."""
+        node = Node(
+            node_type=node_type,
+            config=config,
+            inputs=inputs or [],
+            name=name,
+        )
+        return self.dag.add_node(node)
+
+    # Source operations
+
+    def source(self, path: str, name: str = None) -> str:
+        """Add a SOURCE node."""
+        return self._add(NodeType.SOURCE, {"path": path}, name=name)
+
+    # Transform operations
+
+    def segment(self, input_id: str, duration: float = None,
+                offset: float = 0, precise: bool = True, name: str = None) -> str:
+        """Add a SEGMENT node."""
+        config = {"offset": offset, "precise": precise}
+        if duration is not None:
+            config["duration"] = duration
+        return self._add(NodeType.SEGMENT, config, [input_id], name=name)
+
+    def resize(self, input_id: str, width: int, height: int,
+               mode: str = "fit", name: str = None) -> str:
+        """Add a RESIZE node."""
+        return self._add(
+            NodeType.RESIZE,
+            {"width": width, "height": height, "mode": mode},
+            [input_id],
+            name=name
+        )
+
+    def transform(self, input_id: str, effects: Dict[str, Any],
+                  name: str = None) -> str:
+        """Add a TRANSFORM node."""
+        return self._add(NodeType.TRANSFORM, {"effects": effects}, [input_id], name=name)
+
+    # Compose operations
+
+    def sequence(self, input_ids: List[str], transition: Dict[str, Any] = None,
+                 name: str = None) -> str:
+        """Add a SEQUENCE node."""
+        config = {"transition": transition or {"type": "cut"}}
+        return self._add(NodeType.SEQUENCE, config, input_ids, name=name)
+
+    def layer(self, input_ids: List[str], configs: List[Dict] = None,
+              name: str = None) -> str:
+        """Add a LAYER node."""
+        return self._add(
+            NodeType.LAYER,
+            {"inputs": configs or [{}] * len(input_ids)},
+            input_ids,
+            name=name
+        )
+
+    def mux(self, video_id: str, audio_id: str, shortest: bool = True,
+            name: str = None) -> str:
+        """Add a MUX node."""
+        return self._add(
+            NodeType.MUX,
+            {"video_stream": 0, "audio_stream": 1, "shortest": shortest},
+            [video_id, audio_id],
+            name=name
+        )
+
+    def blend(self, input1_id: str, input2_id: str, mode: str = "overlay",
+              opacity: float = 0.5, name: str = None) -> str:
+        """Add a BLEND node."""
+        return self._add(
+            NodeType.BLEND,
+            {"mode": mode, "opacity": opacity},
+            [input1_id, input2_id],
+            name=name
+        )
+
+    def audio_mix(self, input_ids: List[str], gains: List[float] = None,
+                  normalize: bool = True, name: str = None) -> str:
+        """Add an AUDIO_MIX node to mix multiple audio streams."""
+        config = {"normalize": normalize}
+        if gains is not None:
+            config["gains"] = gains
+        return self._add(NodeType.AUDIO_MIX, config, input_ids, name=name)
+
+    # Output
+
+    def set_output(self, node_id: str) -> "DAGBuilder":
+        """Set the output node."""
+        self.dag.set_output(node_id)
+        return self
+
+    def build(self) -> DAG:
+        """Build and validate the DAG."""
+        errors = self.dag.validate()
+        if errors:
+            raise ValueError(f"Invalid DAG: {errors}")
+        return self.dag

artdag/core/artdag/effects/__init__.py

@@ -0,0 +1,55 @@
+"""
+Cacheable effect system.
+
+Effects are single Python files with:
+- PEP 723 embedded dependencies
+- @-tag metadata in docstrings
+- Frame-by-frame or whole-video API
+
+Effects are cached by content hash (SHA3-256) and executed in
+sandboxed environments for determinism.
+"""
+
+from .meta import EffectMeta, ParamSpec, ExecutionContext
+from .loader import load_effect, load_effect_file, LoadedEffect, compute_cid
+from .binding import (
+    AnalysisData,
+    ResolvedBinding,
+    resolve_binding,
+    resolve_all_bindings,
+    bindings_to_lookup_table,
+    has_bindings,
+    extract_binding_sources,
+)
+from .sandbox import Sandbox, SandboxConfig, SandboxResult, is_bwrap_available, get_venv_path
+from .runner import run_effect, run_effect_from_cache, EffectExecutor
+
+__all__ = [
+    # Meta types
+    "EffectMeta",
+    "ParamSpec",
+    "ExecutionContext",
+    # Loader
+    "load_effect",
+    "load_effect_file",
+    "LoadedEffect",
+    "compute_cid",
+    # Binding
+    "AnalysisData",
+    "ResolvedBinding",
+    "resolve_binding",
+    "resolve_all_bindings",
+    "bindings_to_lookup_table",
+    "has_bindings",
+    "extract_binding_sources",
+    # Sandbox
+    "Sandbox",
+    "SandboxConfig",
+    "SandboxResult",
+    "is_bwrap_available",
+    "get_venv_path",
+    # Runner
+    "run_effect",
+    "run_effect_from_cache",
+    "EffectExecutor",
+]

artdag/core/artdag/effects/binding.py

@@ -0,0 +1,311 @@
+"""
+Parameter binding resolution.
+
+Resolves bind expressions to per-frame lookup tables at plan time.
+Binding options:
+  - :range [lo hi] - map 0-1 to output range
+  - :smooth N - smoothing window in seconds
+  - :offset N - time offset in seconds
+  - :on-event V - value on discrete events
+  - :decay N - exponential decay after event
+  - :noise N - add deterministic noise (seeded)
+  - :seed N - explicit RNG seed
+"""
+
+import hashlib
+import math
+import random
+from dataclasses import dataclass
+from typing import Any, Dict, List, Optional, Tuple
+
+
+@dataclass
+class AnalysisData:
+    """
+    Analysis data for binding resolution.
+
+    Attributes:
+        frame_rate: Video frame rate
+        total_frames: Total number of frames
+        features: Dict mapping feature name to per-frame values
+        events: Dict mapping event name to list of frame indices
+    """
+
+    frame_rate: float
+    total_frames: int
+    features: Dict[str, List[float]]  # feature -> [value_per_frame]
+    events: Dict[str, List[int]]  # event -> [frame_indices]
+
+    def get_feature(self, name: str, frame: int) -> float:
+        """Get feature value at frame, interpolating if needed."""
+        if name not in self.features:
+            return 0.0
+        values = self.features[name]
+        if not values:
+            return 0.0
+        if frame >= len(values):
+            return values[-1]
+        return values[frame]
+
+    def get_events_in_range(
+        self, name: str, start_frame: int, end_frame: int
+    ) -> List[int]:
+        """Get event frames in range."""
+        if name not in self.events:
+            return []
+        return [f for f in self.events[name] if start_frame <= f < end_frame]
+
+
+@dataclass
+class ResolvedBinding:
+    """
+    Resolved binding with per-frame values.
+
+    Attributes:
+        param_name: Parameter this binding applies to
+        values: List of values, one per frame
+    """
+
+    param_name: str
+    values: List[float]
+
+    def get(self, frame: int) -> float:
+        """Get value at frame."""
+        if frame >= len(self.values):
+            return self.values[-1] if self.values else 0.0
+        return self.values[frame]
+
+
+def resolve_binding(
+    binding: Dict[str, Any],
+    analysis: AnalysisData,
+    param_name: str,
+    cache_id: str = None,
+) -> ResolvedBinding:
+    """
+    Resolve a binding specification to per-frame values.
+
+    Args:
+        binding: Binding spec with source, feature, and options
+        analysis: Analysis data with features and events
+        param_name: Name of the parameter being bound
+        cache_id: Cache ID for deterministic seeding
+
+    Returns:
+        ResolvedBinding with values for each frame
+    """
+    feature = binding.get("feature")
+    if not feature:
+        raise ValueError(f"Binding for {param_name} missing feature")
+
+    # Get base values
+    values = []
+    is_event = feature in analysis.events
+
+    if is_event:
+        # Event-based binding
+        on_event = binding.get("on_event", 1.0)
+        decay = binding.get("decay", 0.0)
+        values = _resolve_event_binding(
+            analysis.events.get(feature, []),
+            analysis.total_frames,
+            analysis.frame_rate,
+            on_event,
+            decay,
+        )
+    else:
+        # Continuous feature binding
+        feature_values = analysis.features.get(feature, [])
+        if not feature_values:
+            # No data, use zeros
+            values = [0.0] * analysis.total_frames
+        else:
+            # Extend to total frames if needed
+            values = list(feature_values)
+            while len(values) < analysis.total_frames:
+                values.append(values[-1] if values else 0.0)
+
+    # Apply offset
+    offset = binding.get("offset")
+    if offset:
+        offset_frames = int(offset * analysis.frame_rate)
+        values = _apply_offset(values, offset_frames)
+
+    # Apply smoothing
+    smooth = binding.get("smooth")
+    if smooth:
+        window_frames = int(smooth * analysis.frame_rate)
+        values = _apply_smoothing(values, window_frames)
+
+    # Apply range mapping
+    range_spec = binding.get("range")
+    if range_spec:
+        lo, hi = range_spec
+        values = _apply_range(values, lo, hi)
+
+    # Apply noise
+    noise = binding.get("noise")
+    if noise:
+        seed = binding.get("seed")
+        if seed is None and cache_id:
+            # Derive seed from cache_id for determinism
+            seed = int(hashlib.sha256(cache_id.encode()).hexdigest()[:8], 16)
+        values = _apply_noise(values, noise, seed or 0)
+
+    return ResolvedBinding(param_name=param_name, values=values)
+
+
+def _resolve_event_binding(
+    event_frames: List[int],
+    total_frames: int,
+    frame_rate: float,
+    on_event: float,
+    decay: float,
+) -> List[float]:
+    """
+    Resolve event-based binding with optional decay.
+
+    Args:
+        event_frames: List of frame indices where events occur
+        total_frames: Total number of frames
+        frame_rate: Video frame rate
+        on_event: Value at event
+        decay: Decay time constant in seconds (0 = instant)
+
+    Returns:
+        List of values per frame
+    """
+    values = [0.0] * total_frames
+
+    if not event_frames:
+        return values
+
+    event_set = set(event_frames)
+
+    if decay <= 0:
+        # No decay - just mark event frames
+        for f in event_frames:
+            if 0 <= f < total_frames:
+                values[f] = on_event
+    else:
+        # Apply exponential decay
+        decay_frames = decay * frame_rate
+        for f in event_frames:
+            if f < 0 or f >= total_frames:
+                continue
+            # Apply decay from this event forward
+            for i in range(f, total_frames):
+                elapsed = i - f
+                decayed = on_event * math.exp(-elapsed / decay_frames)
+                if decayed < 0.001:
+                    break
+                values[i] = max(values[i], decayed)
+
+    return values
+
+
+def _apply_offset(values: List[float], offset_frames: int) -> List[float]:
+    """Shift values by offset frames (positive = delay)."""
+    if offset_frames == 0:
+        return values
+
+    n = len(values)
+    result = [0.0] * n
+
+    for i in range(n):
+        src = i - offset_frames
+        if 0 <= src < n:
+            result[i] = values[src]
+
+    return result
+
+
+def _apply_smoothing(values: List[float], window_frames: int) -> List[float]:
+    """Apply moving average smoothing."""
+    if window_frames <= 1:
+        return values
+
+    n = len(values)
+    result = []
+    half = window_frames // 2
+
+    for i in range(n):
+        start = max(0, i - half)
+        end = min(n, i + half + 1)
+        avg = sum(values[start:end]) / (end - start)
+        result.append(avg)
+
+    return result
+
+
+def _apply_range(values: List[float], lo: float, hi: float) -> List[float]:
+    """Map values from 0-1 to lo-hi range."""
+    return [lo + v * (hi - lo) for v in values]
+
+
+def _apply_noise(values: List[float], amount: float, seed: int) -> List[float]:
+    """Add deterministic noise to values."""
+    rng = random.Random(seed)
+    return [v + rng.uniform(-amount, amount) for v in values]
+
+
+def resolve_all_bindings(
+    config: Dict[str, Any],
+    analysis: AnalysisData,
+    cache_id: str = None,
+) -> Dict[str, ResolvedBinding]:
+    """
+    Resolve all bindings in a config dict.
+
+    Looks for values with _binding: True marker.
+
+    Args:
+        config: Node config with potential bindings
+        analysis: Analysis data
+        cache_id: Cache ID for seeding
+
+    Returns:
+        Dict mapping param name to resolved binding
+    """
+    resolved = {}
+
+    for key, value in config.items():
+        if isinstance(value, dict) and value.get("_binding"):
+            resolved[key] = resolve_binding(value, analysis, key, cache_id)
+
+    return resolved
+
+
+def bindings_to_lookup_table(
+    bindings: Dict[str, ResolvedBinding],
+) -> Dict[str, List[float]]:
+    """
+    Convert resolved bindings to simple lookup tables.
+
+    Returns dict mapping param name to list of per-frame values.
+    This format is JSON-serializable for inclusion in execution plans.
+    """
+    return {name: binding.values for name, binding in bindings.items()}
+
+
+def has_bindings(config: Dict[str, Any]) -> bool:
+    """Check if config contains any bindings."""
+    for value in config.values():
+        if isinstance(value, dict) and value.get("_binding"):
+            return True
+    return False
+
+
+def extract_binding_sources(config: Dict[str, Any]) -> List[str]:
+    """
+    Extract all analysis source references from bindings.
+
+    Returns list of node IDs that provide analysis data.
+    """
+    sources = []
+    for value in config.values():
+        if isinstance(value, dict) and value.get("_binding"):
+            source = value.get("source")
+            if source and source not in sources:
+                sources.append(source)
+    return sources

artdag/core/artdag/effects/frame_processor.py

@@ -0,0 +1,347 @@
+"""
+FFmpeg pipe-based frame processing.
+
+Processes video through Python frame-by-frame effects using FFmpeg pipes:
+  FFmpeg decode -> Python process_frame -> FFmpeg encode
+
+This avoids writing intermediate frames to disk.
+"""
+
+import logging
+import subprocess
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Callable, Dict, List, Optional, Tuple
+
+import numpy as np
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class VideoInfo:
+    """Video metadata."""
+
+    width: int
+    height: int
+    frame_rate: float
+    total_frames: int
+    duration: float
+    pixel_format: str = "rgb24"
+
+
+def probe_video(path: Path) -> VideoInfo:
+    """
+    Get video information using ffprobe.
+
+    Args:
+        path: Path to video file
+
+    Returns:
+        VideoInfo with dimensions, frame rate, etc.
+    """
+    cmd = [
+        "ffprobe",
+        "-v", "error",
+        "-select_streams", "v:0",
+        "-show_entries", "stream=width,height,r_frame_rate,nb_frames,duration",
+        "-of", "csv=p=0",
+        str(path),
+    ]
+
+    result = subprocess.run(cmd, capture_output=True, text=True)
+    if result.returncode != 0:
+        raise RuntimeError(f"ffprobe failed: {result.stderr}")
+
+    parts = result.stdout.strip().split(",")
+    if len(parts) < 4:
+        raise RuntimeError(f"Unexpected ffprobe output: {result.stdout}")
+
+    width = int(parts[0])
+    height = int(parts[1])
+
+    # Parse frame rate (could be "30/1" or "30")
+    fr_parts = parts[2].split("/")
+    if len(fr_parts) == 2:
+        frame_rate = float(fr_parts[0]) / float(fr_parts[1])
+    else:
+        frame_rate = float(fr_parts[0])
+
+    # nb_frames might be N/A
+    total_frames = 0
+    duration = 0.0
+    try:
+        total_frames = int(parts[3])
+    except (ValueError, IndexError):
+        pass
+
+    try:
+        duration = float(parts[4]) if len(parts) > 4 else 0.0
+    except (ValueError, IndexError):
+        pass
+
+    if total_frames == 0 and duration > 0:
+        total_frames = int(duration * frame_rate)
+
+    return VideoInfo(
+        width=width,
+        height=height,
+        frame_rate=frame_rate,
+        total_frames=total_frames,
+        duration=duration,
+    )
+
+
+FrameProcessor = Callable[[np.ndarray, Dict[str, Any], Any], Tuple[np.ndarray, Any]]
+
+
+def process_video(
+    input_path: Path,
+    output_path: Path,
+    process_frame: FrameProcessor,
+    params: Dict[str, Any],
+    bindings: Dict[str, List[float]] = None,
+    initial_state: Any = None,
+    pixel_format: str = "rgb24",
+    output_codec: str = "libx264",
+    output_options: List[str] = None,
+) -> Tuple[Path, Any]:
+    """
+    Process video through frame-by-frame effect.
+
+    Args:
+        input_path: Input video path
+        output_path: Output video path
+        process_frame: Function (frame, params, state) -> (frame, state)
+        params: Static parameter dict
+        bindings: Per-frame parameter lookup tables
+        initial_state: Initial state for process_frame
+        pixel_format: Pixel format for frame data
+        output_codec: Video codec for output
+        output_options: Additional ffmpeg output options
+
+    Returns:
+        Tuple of (output_path, final_state)
+    """
+    bindings = bindings or {}
+    output_options = output_options or []
+
+    # Probe input
+    info = probe_video(input_path)
+    logger.info(f"Processing {info.width}x{info.height} @ {info.frame_rate}fps")
+
+    # Calculate bytes per frame
+    if pixel_format == "rgb24":
+        bytes_per_pixel = 3
+    elif pixel_format == "rgba":
+        bytes_per_pixel = 4
+    else:
+        bytes_per_pixel = 3  # Default to RGB
+
+    frame_size = info.width * info.height * bytes_per_pixel
+
+    # Start decoder process
+    decode_cmd = [
+        "ffmpeg",
+        "-i", str(input_path),
+        "-f", "rawvideo",
+        "-pix_fmt", pixel_format,
+        "-",
+    ]
+
+    # Start encoder process
+    encode_cmd = [
+        "ffmpeg",
+        "-y",
+        "-f", "rawvideo",
+        "-pix_fmt", pixel_format,
+        "-s", f"{info.width}x{info.height}",
+        "-r", str(info.frame_rate),
+        "-i", "-",
+        "-i", str(input_path),  # For audio
+        "-map", "0:v",
+        "-map", "1:a?",
+        "-c:v", output_codec,
+        "-c:a", "aac",
+        *output_options,
+        str(output_path),
+    ]
+
+    logger.debug(f"Decoder: {' '.join(decode_cmd)}")
+    logger.debug(f"Encoder: {' '.join(encode_cmd)}")
+
+    decoder = subprocess.Popen(
+        decode_cmd,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.DEVNULL,
+    )
+
+    encoder = subprocess.Popen(
+        encode_cmd,
+        stdin=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    )
+
+    state = initial_state
+    frame_idx = 0
+
+    try:
+        while True:
+            # Read frame from decoder
+            raw_frame = decoder.stdout.read(frame_size)
+            if len(raw_frame) < frame_size:
+                break
+
+            # Convert to numpy
+            frame = np.frombuffer(raw_frame, dtype=np.uint8)
+            frame = frame.reshape((info.height, info.width, bytes_per_pixel))
+
+            # Build per-frame params
+            frame_params = dict(params)
+            for param_name, values in bindings.items():
+                if frame_idx < len(values):
+                    frame_params[param_name] = values[frame_idx]
+
+            # Process frame
+            processed, state = process_frame(frame, frame_params, state)
+
+            # Ensure correct shape and dtype
+            if processed.shape != frame.shape:
+                raise ValueError(
+                    f"Frame shape mismatch: {processed.shape} vs {frame.shape}"
+                )
+            processed = processed.astype(np.uint8)
+
+            # Write to encoder
+            encoder.stdin.write(processed.tobytes())
+            frame_idx += 1
+
+            if frame_idx % 100 == 0:
+                logger.debug(f"Processed frame {frame_idx}")
+
+    except Exception as e:
+        logger.error(f"Frame processing failed at frame {frame_idx}: {e}")
+        raise
+    finally:
+        decoder.stdout.close()
+        decoder.wait()
+        encoder.stdin.close()
+        encoder.wait()
+
+    if encoder.returncode != 0:
+        stderr = encoder.stderr.read().decode() if encoder.stderr else ""
+        raise RuntimeError(f"Encoder failed: {stderr}")
+
+    logger.info(f"Processed {frame_idx} frames")
+    return output_path, state
+
+
+def process_video_batch(
+    input_path: Path,
+    output_path: Path,
+    process_frames: Callable[[List[np.ndarray], Dict[str, Any]], List[np.ndarray]],
+    params: Dict[str, Any],
+    batch_size: int = 30,
+    pixel_format: str = "rgb24",
+    output_codec: str = "libx264",
+) -> Path:
+    """
+    Process video in batches for effects that need temporal context.
+
+    Args:
+        input_path: Input video path
+        output_path: Output video path
+        process_frames: Function (frames_batch, params) -> processed_batch
+        params: Parameter dict
+        batch_size: Number of frames per batch
+        pixel_format: Pixel format
+        output_codec: Output codec
+
+    Returns:
+        Output path
+    """
+    info = probe_video(input_path)
+
+    if pixel_format == "rgb24":
+        bytes_per_pixel = 3
+    elif pixel_format == "rgba":
+        bytes_per_pixel = 4
+    else:
+        bytes_per_pixel = 3
+
+    frame_size = info.width * info.height * bytes_per_pixel
+
+    decode_cmd = [
+        "ffmpeg",
+        "-i", str(input_path),
+        "-f", "rawvideo",
+        "-pix_fmt", pixel_format,
+        "-",
+    ]
+
+    encode_cmd = [
+        "ffmpeg",
+        "-y",
+        "-f", "rawvideo",
+        "-pix_fmt", pixel_format,
+        "-s", f"{info.width}x{info.height}",
+        "-r", str(info.frame_rate),
+        "-i", "-",
+        "-i", str(input_path),
+        "-map", "0:v",
+        "-map", "1:a?",
+        "-c:v", output_codec,
+        "-c:a", "aac",
+        str(output_path),
+    ]
+
+    decoder = subprocess.Popen(
+        decode_cmd,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.DEVNULL,
+    )
+
+    encoder = subprocess.Popen(
+        encode_cmd,
+        stdin=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    )
+
+    batch = []
+    total_processed = 0
+
+    try:
+        while True:
+            raw_frame = decoder.stdout.read(frame_size)
+            if len(raw_frame) < frame_size:
+                # Process remaining batch
+                if batch:
+                    processed = process_frames(batch, params)
+                    for frame in processed:
+                        encoder.stdin.write(frame.astype(np.uint8).tobytes())
+                        total_processed += 1
+                break
+
+            frame = np.frombuffer(raw_frame, dtype=np.uint8)
+            frame = frame.reshape((info.height, info.width, bytes_per_pixel))
+            batch.append(frame)
+
+            if len(batch) >= batch_size:
+                processed = process_frames(batch, params)
+                for frame in processed:
+                    encoder.stdin.write(frame.astype(np.uint8).tobytes())
+                    total_processed += 1
+                batch = []
+
+    finally:
+        decoder.stdout.close()
+        decoder.wait()
+        encoder.stdin.close()
+        encoder.wait()
+
+    if encoder.returncode != 0:
+        stderr = encoder.stderr.read().decode() if encoder.stderr else ""
+        raise RuntimeError(f"Encoder failed: {stderr}")
+
+    logger.info(f"Processed {total_processed} frames in batches of {batch_size}")
+    return output_path

artdag/core/artdag/effects/loader.py

@@ -0,0 +1,455 @@
+"""
+Effect file loader.
+
+Parses effect files with:
+- PEP 723 inline script metadata for dependencies
+- @-tag docstrings for effect metadata
+- META object for programmatic access
+"""
+
+import ast
+import hashlib
+import re
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple
+
+from .meta import EffectMeta, ParamSpec
+
+
+@dataclass
+class LoadedEffect:
+    """
+    A loaded effect with all metadata.
+
+    Attributes:
+        source: Original source code
+        cid: SHA3-256 hash of source
+        meta: Extracted EffectMeta
+        dependencies: List of pip dependencies
+        requires_python: Python version requirement
+        module: Compiled module (if loaded)
+    """
+
+    source: str
+    cid: str
+    meta: EffectMeta
+    dependencies: List[str] = field(default_factory=list)
+    requires_python: str = ">=3.10"
+    module: Any = None
+
+    def has_frame_api(self) -> bool:
+        """Check if effect has frame-by-frame API."""
+        return self.meta.api_type == "frame"
+
+    def has_video_api(self) -> bool:
+        """Check if effect has whole-video API."""
+        return self.meta.api_type == "video"
+
+
+def compute_cid(source: str) -> str:
+    """Compute SHA3-256 hash of effect source."""
+    return hashlib.sha3_256(source.encode("utf-8")).hexdigest()
+
+
+def parse_pep723_metadata(source: str) -> Tuple[List[str], str]:
+    """
+    Parse PEP 723 inline script metadata.
+
+    Looks for:
+        # /// script
+        # requires-python = ">=3.10"
+        # dependencies = ["numpy", "opencv-python"]
+        # ///
+
+    Returns:
+        Tuple of (dependencies list, requires_python string)
+    """
+    dependencies = []
+    requires_python = ">=3.10"
+
+    # Match the script block
+    pattern = r"# /// script\n(.*?)# ///"
+    match = re.search(pattern, source, re.DOTALL)
+
+    if not match:
+        return dependencies, requires_python
+
+    block = match.group(1)
+
+    # Parse dependencies
+    deps_match = re.search(r'# dependencies = \[(.*?)\]', block, re.DOTALL)
+    if deps_match:
+        deps_str = deps_match.group(1)
+        # Extract quoted strings
+        dependencies = re.findall(r'"([^"]+)"', deps_str)
+
+    # Parse requires-python
+    python_match = re.search(r'# requires-python = "([^"]+)"', block)
+    if python_match:
+        requires_python = python_match.group(1)
+
+    return dependencies, requires_python
+
+
+def parse_docstring_metadata(docstring: str) -> Dict[str, Any]:
+    """
+    Parse @-tag metadata from docstring.
+
+    Supports:
+        @effect name
+        @version 1.0.0
+        @author @user@domain
+        @temporal false
+        @description
+        Multi-line description text.
+
+        @param name type
+          @range lo hi
+          @default value
+          Description text.
+
+        @example
+          (fx effect :param value)
+
+    Returns:
+        Dictionary with extracted metadata
+    """
+    if not docstring:
+        return {}
+
+    result = {
+        "name": "",
+        "version": "1.0.0",
+        "author": "",
+        "temporal": False,
+        "description": "",
+        "params": [],
+        "examples": [],
+    }
+
+    lines = docstring.strip().split("\n")
+    i = 0
+    current_param = None
+
+    while i < len(lines):
+        line = lines[i].strip()
+
+        if line.startswith("@effect "):
+            result["name"] = line[8:].strip()
+
+        elif line.startswith("@version "):
+            result["version"] = line[9:].strip()
+
+        elif line.startswith("@author "):
+            result["author"] = line[8:].strip()
+
+        elif line.startswith("@temporal "):
+            val = line[10:].strip().lower()
+            result["temporal"] = val in ("true", "yes", "1")
+
+        elif line.startswith("@description"):
+            # Collect multi-line description
+            desc_lines = []
+            i += 1
+            while i < len(lines):
+                next_line = lines[i]
+                if next_line.strip().startswith("@"):
+                    i -= 1  # Back up to process this tag
+                    break
+                desc_lines.append(next_line)
+                i += 1
+            result["description"] = "\n".join(desc_lines).strip()
+
+        elif line.startswith("@param "):
+            # Parse parameter: @param name type
+            parts = line[7:].split()
+            if len(parts) >= 2:
+                current_param = {
+                    "name": parts[0],
+                    "type": parts[1],
+                    "range": None,
+                    "default": None,
+                    "description": "",
+                }
+                # Collect param details
+                desc_lines = []
+                i += 1
+                while i < len(lines):
+                    next_line = lines[i]
+                    stripped = next_line.strip()
+
+                    if stripped.startswith("@range "):
+                        range_parts = stripped[7:].split()
+                        if len(range_parts) >= 2:
+                            try:
+                                current_param["range"] = (
+                                    float(range_parts[0]),
+                                    float(range_parts[1]),
+                                )
+                            except ValueError:
+                                pass
+
+                    elif stripped.startswith("@default "):
+                        current_param["default"] = stripped[9:].strip()
+
+                    elif stripped.startswith("@param ") or stripped.startswith("@example"):
+                        i -= 1  # Back up
+                        break
+
+                    elif stripped.startswith("@"):
+                        i -= 1
+                        break
+
+                    elif stripped:
+                        desc_lines.append(stripped)
+
+                    i += 1
+
+                current_param["description"] = " ".join(desc_lines)
+                result["params"].append(current_param)
+                current_param = None
+
+        elif line.startswith("@example"):
+            # Collect example
+            example_lines = []
+            i += 1
+            while i < len(lines):
+                next_line = lines[i]
+                if next_line.strip().startswith("@") and not next_line.strip().startswith("@example"):
+                    if next_line.strip().startswith("@example"):
+                        i -= 1
+                    break
+                if next_line.strip().startswith("@example"):
+                    i -= 1
+                    break
+                example_lines.append(next_line)
+                i += 1
+            example = "\n".join(example_lines).strip()
+            if example:
+                result["examples"].append(example)
+
+        i += 1
+
+    return result
+
+
+def extract_meta_from_ast(source: str) -> Optional[Dict[str, Any]]:
+    """
+    Extract META object from source AST.
+
+    Looks for:
+        META = EffectMeta(...)
+
+    Returns the keyword arguments if found.
+    """
+    try:
+        tree = ast.parse(source)
+    except SyntaxError:
+        return None
+
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Assign):
+            for target in node.targets:
+                if isinstance(target, ast.Name) and target.id == "META":
+                    if isinstance(node.value, ast.Call):
+                        return _extract_call_kwargs(node.value)
+    return None
+
+
+def _extract_call_kwargs(call: ast.Call) -> Dict[str, Any]:
+    """Extract keyword arguments from an AST Call node."""
+    result = {}
+
+    for keyword in call.keywords:
+        if keyword.arg is None:
+            continue
+        value = _ast_to_value(keyword.value)
+        if value is not None:
+            result[keyword.arg] = value
+
+    return result
+
+
+def _ast_to_value(node: ast.expr) -> Any:
+    """Convert AST node to Python value."""
+    if isinstance(node, ast.Constant):
+        return node.value
+    elif isinstance(node, ast.Str):  # Python 3.7 compat
+        return node.s
+    elif isinstance(node, ast.Num):  # Python 3.7 compat
+        return node.n
+    elif isinstance(node, ast.NameConstant):  # Python 3.7 compat
+        return node.value
+    elif isinstance(node, ast.List):
+        return [_ast_to_value(elt) for elt in node.elts]
+    elif isinstance(node, ast.Tuple):
+        return tuple(_ast_to_value(elt) for elt in node.elts)
+    elif isinstance(node, ast.Dict):
+        return {
+            _ast_to_value(k): _ast_to_value(v)
+            for k, v in zip(node.keys, node.values)
+            if k is not None
+        }
+    elif isinstance(node, ast.Call):
+        # Handle ParamSpec(...) calls
+        if isinstance(node.func, ast.Name) and node.func.id == "ParamSpec":
+            return _extract_call_kwargs(node)
+    return None
+
+
+def get_module_docstring(source: str) -> str:
+    """Extract the module-level docstring from source."""
+    try:
+        tree = ast.parse(source)
+    except SyntaxError:
+        return ""
+
+    if tree.body and isinstance(tree.body[0], ast.Expr):
+        if isinstance(tree.body[0].value, ast.Constant):
+            return tree.body[0].value.value
+        elif isinstance(tree.body[0].value, ast.Str):  # Python 3.7 compat
+            return tree.body[0].value.s
+    return ""
+
+
+def load_effect(source: str) -> LoadedEffect:
+    """
+    Load an effect from source code.
+
+    Parses:
+    1. PEP 723 metadata for dependencies
+    2. Module docstring for @-tag metadata
+    3. META object for programmatic metadata
+
+    Priority: META object > docstring > defaults
+
+    Args:
+        source: Effect source code
+
+    Returns:
+        LoadedEffect with all metadata
+
+    Raises:
+        ValueError: If effect is invalid
+    """
+    cid = compute_cid(source)
+
+    # Parse PEP 723 metadata
+    dependencies, requires_python = parse_pep723_metadata(source)
+
+    # Parse docstring metadata
+    docstring = get_module_docstring(source)
+    doc_meta = parse_docstring_metadata(docstring)
+
+    # Try to extract META from AST
+    ast_meta = extract_meta_from_ast(source)
+
+    # Build EffectMeta, preferring META object over docstring
+    name = ""
+    if ast_meta and "name" in ast_meta:
+        name = ast_meta["name"]
+    elif doc_meta.get("name"):
+        name = doc_meta["name"]
+
+    if not name:
+        raise ValueError("Effect must have a name (@effect or META.name)")
+
+    version = ast_meta.get("version") if ast_meta else doc_meta.get("version", "1.0.0")
+    temporal = ast_meta.get("temporal") if ast_meta else doc_meta.get("temporal", False)
+    author = ast_meta.get("author") if ast_meta else doc_meta.get("author", "")
+    description = ast_meta.get("description") if ast_meta else doc_meta.get("description", "")
+    examples = ast_meta.get("examples") if ast_meta else doc_meta.get("examples", [])
+
+    # Build params
+    params = []
+    if ast_meta and "params" in ast_meta:
+        for p in ast_meta["params"]:
+            if isinstance(p, dict):
+                type_map = {"float": float, "int": int, "bool": bool, "str": str}
+                param_type = type_map.get(p.get("param_type", "float"), float)
+                if isinstance(p.get("param_type"), type):
+                    param_type = p["param_type"]
+                params.append(
+                    ParamSpec(
+                        name=p.get("name", ""),
+                        param_type=param_type,
+                        default=p.get("default"),
+                        range=p.get("range"),
+                        description=p.get("description", ""),
+                    )
+                )
+    elif doc_meta.get("params"):
+        for p in doc_meta["params"]:
+            type_map = {"float": float, "int": int, "bool": bool, "str": str}
+            param_type = type_map.get(p.get("type", "float"), float)
+
+            default = p.get("default")
+            if default is not None:
+                try:
+                    default = param_type(default)
+                except (ValueError, TypeError):
+                    pass
+
+            params.append(
+                ParamSpec(
+                    name=p["name"],
+                    param_type=param_type,
+                    default=default,
+                    range=p.get("range"),
+                    description=p.get("description", ""),
+                )
+            )
+
+    # Determine API type by checking for function definitions
+    api_type = "frame"  # default
+    try:
+        tree = ast.parse(source)
+        for node in ast.walk(tree):
+            if isinstance(node, ast.FunctionDef):
+                if node.name == "process":
+                    api_type = "video"
+                    break
+                elif node.name == "process_frame":
+                    api_type = "frame"
+                    break
+    except SyntaxError:
+        pass
+
+    meta = EffectMeta(
+        name=name,
+        version=version if isinstance(version, str) else "1.0.0",
+        temporal=bool(temporal),
+        params=params,
+        author=author if isinstance(author, str) else "",
+        description=description if isinstance(description, str) else "",
+        examples=examples if isinstance(examples, list) else [],
+        dependencies=dependencies,
+        requires_python=requires_python,
+        api_type=api_type,
+    )
+
+    return LoadedEffect(
+        source=source,
+        cid=cid,
+        meta=meta,
+        dependencies=dependencies,
+        requires_python=requires_python,
+    )
+
+
+def load_effect_file(path: Path) -> LoadedEffect:
+    """Load an effect from a file path."""
+    source = path.read_text(encoding="utf-8")
+    return load_effect(source)
+
+
+def compute_deps_hash(dependencies: List[str]) -> str:
+    """
+    Compute hash of sorted dependencies.
+
+    Used for venv caching - same deps = same hash = reuse venv.
+    """
+    sorted_deps = sorted(dep.lower().strip() for dep in dependencies)
+    deps_str = "\n".join(sorted_deps)
+    return hashlib.sha3_256(deps_str.encode("utf-8")).hexdigest()

artdag/core/artdag/effects/meta.py

@@ -0,0 +1,247 @@
+"""
+Effect metadata types.
+
+Defines the core dataclasses for effect metadata:
+- ParamSpec: Parameter specification with type, range, and default
+- EffectMeta: Complete effect metadata including params and flags
+"""
+
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional, Tuple, Type, Union
+
+
+@dataclass
+class ParamSpec:
+    """
+    Specification for an effect parameter.
+
+    Attributes:
+        name: Parameter name (used in recipes as :name)
+        param_type: Python type (float, int, bool, str)
+        default: Default value if not specified
+        range: Optional (min, max) tuple for numeric types
+        description: Human-readable description
+        choices: Optional list of allowed values (for enums)
+    """
+
+    name: str
+    param_type: Type
+    default: Any = None
+    range: Optional[Tuple[float, float]] = None
+    description: str = ""
+    choices: Optional[List[Any]] = None
+
+    def validate(self, value: Any) -> Any:
+        """
+        Validate and coerce a parameter value.
+
+        Args:
+            value: Input value to validate
+
+        Returns:
+            Validated and coerced value
+
+        Raises:
+            ValueError: If value is invalid
+        """
+        if value is None:
+            if self.default is not None:
+                return self.default
+            raise ValueError(f"Parameter '{self.name}' requires a value")
+
+        # Type coercion
+        try:
+            if self.param_type == bool:
+                if isinstance(value, str):
+                    value = value.lower() in ("true", "1", "yes")
+                else:
+                    value = bool(value)
+            elif self.param_type == int:
+                value = int(value)
+            elif self.param_type == float:
+                value = float(value)
+            elif self.param_type == str:
+                value = str(value)
+            else:
+                value = self.param_type(value)
+        except (ValueError, TypeError) as e:
+            raise ValueError(
+                f"Parameter '{self.name}' expects {self.param_type.__name__}, "
+                f"got {type(value).__name__}: {e}"
+            )
+
+        # Range check for numeric types
+        if self.range is not None and self.param_type in (int, float):
+            min_val, max_val = self.range
+            if value < min_val or value > max_val:
+                raise ValueError(
+                    f"Parameter '{self.name}' must be in range "
+                    f"[{min_val}, {max_val}], got {value}"
+                )
+
+        # Choices check
+        if self.choices is not None and value not in self.choices:
+            raise ValueError(
+                f"Parameter '{self.name}' must be one of {self.choices}, got {value}"
+            )
+
+        return value
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for serialization."""
+        d = {
+            "name": self.name,
+            "type": self.param_type.__name__,
+            "description": self.description,
+        }
+        if self.default is not None:
+            d["default"] = self.default
+        if self.range is not None:
+            d["range"] = list(self.range)
+        if self.choices is not None:
+            d["choices"] = self.choices
+        return d
+
+
+@dataclass
+class EffectMeta:
+    """
+    Complete metadata for an effect.
+
+    Attributes:
+        name: Effect name (used in recipes)
+        version: Semantic version string
+        temporal: If True, effect needs complete input (can't be collapsed)
+        params: List of parameter specifications
+        author: Optional author identifier
+        description: Human-readable description
+        examples: List of example S-expression usages
+        dependencies: List of Python package dependencies
+        requires_python: Minimum Python version
+        api_type: "frame" for frame-by-frame, "video" for whole-video
+    """
+
+    name: str
+    version: str = "1.0.0"
+    temporal: bool = False
+    params: List[ParamSpec] = field(default_factory=list)
+    author: str = ""
+    description: str = ""
+    examples: List[str] = field(default_factory=list)
+    dependencies: List[str] = field(default_factory=list)
+    requires_python: str = ">=3.10"
+    api_type: str = "frame"  # "frame" or "video"
+
+    def get_param(self, name: str) -> Optional[ParamSpec]:
+        """Get a parameter spec by name."""
+        for param in self.params:
+            if param.name == name:
+                return param
+        return None
+
+    def validate_params(self, params: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Validate all parameters.
+
+        Args:
+            params: Dictionary of parameter values
+
+        Returns:
+            Dictionary with validated/coerced values including defaults
+
+        Raises:
+            ValueError: If any parameter is invalid
+        """
+        result = {}
+        for spec in self.params:
+            value = params.get(spec.name)
+            result[spec.name] = spec.validate(value)
+        return result
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dictionary for serialization."""
+        return {
+            "name": self.name,
+            "version": self.version,
+            "temporal": self.temporal,
+            "params": [p.to_dict() for p in self.params],
+            "author": self.author,
+            "description": self.description,
+            "examples": self.examples,
+            "dependencies": self.dependencies,
+            "requires_python": self.requires_python,
+            "api_type": self.api_type,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "EffectMeta":
+        """Create from dictionary."""
+        params = []
+        for p in data.get("params", []):
+            # Map type name back to Python type
+            type_map = {"float": float, "int": int, "bool": bool, "str": str}
+            param_type = type_map.get(p.get("type", "float"), float)
+            params.append(
+                ParamSpec(
+                    name=p["name"],
+                    param_type=param_type,
+                    default=p.get("default"),
+                    range=tuple(p["range"]) if p.get("range") else None,
+                    description=p.get("description", ""),
+                    choices=p.get("choices"),
+                )
+            )
+
+        return cls(
+            name=data["name"],
+            version=data.get("version", "1.0.0"),
+            temporal=data.get("temporal", False),
+            params=params,
+            author=data.get("author", ""),
+            description=data.get("description", ""),
+            examples=data.get("examples", []),
+            dependencies=data.get("dependencies", []),
+            requires_python=data.get("requires_python", ">=3.10"),
+            api_type=data.get("api_type", "frame"),
+        )
+
+
+@dataclass
+class ExecutionContext:
+    """
+    Context passed to effect execution.
+
+    Provides controlled access to resources within sandbox.
+    """
+
+    input_paths: List[str]
+    output_path: str
+    params: Dict[str, Any]
+    seed: int  # Deterministic seed for RNG
+    frame_rate: float = 30.0
+    width: int = 1920
+    height: int = 1080
+
+    # Resolved bindings (frame -> param value lookup)
+    bindings: Dict[str, List[float]] = field(default_factory=dict)
+
+    def get_param_at_frame(self, param_name: str, frame: int) -> Any:
+        """
+        Get parameter value at a specific frame.
+
+        If parameter has a binding, looks up the bound value.
+        Otherwise returns the static parameter value.
+        """
+        if param_name in self.bindings:
+            binding_values = self.bindings[param_name]
+            if frame < len(binding_values):
+                return binding_values[frame]
+            # Past end of binding data, use last value
+            return binding_values[-1] if binding_values else self.params.get(param_name)
+        return self.params.get(param_name)
+
+    def get_rng(self) -> "random.Random":
+        """Get a seeded random number generator."""
+        import random
+
+        return random.Random(self.seed)

artdag/core/artdag/effects/runner.py

@@ -0,0 +1,259 @@
+"""
+Effect runner.
+
+Main entry point for executing cached effects with sandboxing.
+"""
+
+import logging
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from .binding import AnalysisData, bindings_to_lookup_table, resolve_all_bindings
+from .loader import load_effect, LoadedEffect
+from .meta import ExecutionContext
+from .sandbox import Sandbox, SandboxConfig, SandboxResult, get_venv_path
+
+logger = logging.getLogger(__name__)
+
+
+def run_effect(
+    effect_source: str,
+    input_paths: List[Path],
+    output_path: Path,
+    params: Dict[str, Any],
+    analysis: Optional[AnalysisData] = None,
+    cache_id: str = None,
+    seed: int = 0,
+    trust_level: str = "untrusted",
+    timeout: int = 3600,
+) -> SandboxResult:
+    """
+    Run an effect with full sandboxing.
+
+    This is the main entry point for effect execution.
+
+    Args:
+        effect_source: Effect source code
+        input_paths: List of input file paths
+        output_path: Output file path
+        params: Effect parameters (may contain bindings)
+        analysis: Optional analysis data for binding resolution
+        cache_id: Cache ID for deterministic seeding
+        seed: RNG seed (overrides cache_id-based seed)
+        trust_level: "untrusted" or "trusted"
+        timeout: Maximum execution time in seconds
+
+    Returns:
+        SandboxResult with success status and output
+    """
+    # Load and validate effect
+    loaded = load_effect(effect_source)
+    logger.info(f"Running effect '{loaded.meta.name}' v{loaded.meta.version}")
+
+    # Resolve bindings if analysis data available
+    bindings = {}
+    if analysis:
+        resolved = resolve_all_bindings(params, analysis, cache_id)
+        bindings = bindings_to_lookup_table(resolved)
+        # Remove binding dicts from params, keeping only resolved values
+        params = {
+            k: v for k, v in params.items()
+            if not (isinstance(v, dict) and v.get("_binding"))
+        }
+
+    # Validate parameters
+    validated_params = loaded.meta.validate_params(params)
+
+    # Get or create venv for dependencies
+    venv_path = None
+    if loaded.dependencies:
+        venv_path = get_venv_path(loaded.dependencies)
+
+    # Configure sandbox
+    config = SandboxConfig(
+        trust_level=trust_level,
+        venv_path=venv_path,
+        timeout=timeout,
+    )
+
+    # Write effect to temp file
+    import tempfile
+    with tempfile.NamedTemporaryFile(
+        mode="w",
+        suffix=".py",
+        delete=False,
+    ) as f:
+        f.write(effect_source)
+        effect_path = Path(f.name)
+
+    try:
+        with Sandbox(config) as sandbox:
+            result = sandbox.run_effect(
+                effect_path=effect_path,
+                input_paths=input_paths,
+                output_path=output_path,
+                params=validated_params,
+                bindings=bindings,
+                seed=seed,
+            )
+    finally:
+        effect_path.unlink(missing_ok=True)
+
+    return result
+
+
+def run_effect_from_cache(
+    cache,
+    effect_hash: str,
+    input_paths: List[Path],
+    output_path: Path,
+    params: Dict[str, Any],
+    analysis: Optional[AnalysisData] = None,
+    cache_id: str = None,
+    seed: int = 0,
+    trust_level: str = "untrusted",
+    timeout: int = 3600,
+) -> SandboxResult:
+    """
+    Run an effect from cache by content hash.
+
+    Args:
+        cache: Cache instance
+        effect_hash: Content hash of effect
+        input_paths: Input file paths
+        output_path: Output file path
+        params: Effect parameters
+        analysis: Optional analysis data
+        cache_id: Cache ID for seeding
+        seed: RNG seed
+        trust_level: "untrusted" or "trusted"
+        timeout: Max execution time
+
+    Returns:
+        SandboxResult
+    """
+    effect_source = cache.get_effect(effect_hash)
+    if not effect_source:
+        return SandboxResult(
+            success=False,
+            error=f"Effect not found in cache: {effect_hash[:16]}...",
+        )
+
+    return run_effect(
+        effect_source=effect_source,
+        input_paths=input_paths,
+        output_path=output_path,
+        params=params,
+        analysis=analysis,
+        cache_id=cache_id,
+        seed=seed,
+        trust_level=trust_level,
+        timeout=timeout,
+    )
+
+
+def check_effect_temporal(cache, effect_hash: str) -> bool:
+    """
+    Check if an effect is temporal (can't be collapsed).
+
+    Args:
+        cache: Cache instance
+        effect_hash: Content hash of effect
+
+    Returns:
+        True if effect is temporal
+    """
+    metadata = cache.get_effect_metadata(effect_hash)
+    if not metadata:
+        return False
+
+    meta = metadata.get("meta", {})
+    return meta.get("temporal", False)
+
+
+def get_effect_api_type(cache, effect_hash: str) -> str:
+    """
+    Get the API type of an effect.
+
+    Args:
+        cache: Cache instance
+        effect_hash: Content hash of effect
+
+    Returns:
+        "frame" or "video"
+    """
+    metadata = cache.get_effect_metadata(effect_hash)
+    if not metadata:
+        return "frame"
+
+    meta = metadata.get("meta", {})
+    return meta.get("api_type", "frame")
+
+
+class EffectExecutor:
+    """
+    Executor for cached effects.
+
+    Provides a higher-level interface for effect execution.
+    """
+
+    def __init__(self, cache, trust_level: str = "untrusted"):
+        """
+        Initialize executor.
+
+        Args:
+            cache: Cache instance
+            trust_level: Default trust level
+        """
+        self.cache = cache
+        self.trust_level = trust_level
+
+    def execute(
+        self,
+        effect_hash: str,
+        input_paths: List[Path],
+        output_path: Path,
+        params: Dict[str, Any],
+        analysis: Optional[AnalysisData] = None,
+        step_cache_id: str = None,
+    ) -> SandboxResult:
+        """
+        Execute an effect.
+
+        Args:
+            effect_hash: Content hash of effect
+            input_paths: Input file paths
+            output_path: Output path
+            params: Effect parameters
+            analysis: Analysis data for bindings
+            step_cache_id: Step cache ID for seeding
+
+        Returns:
+            SandboxResult
+        """
+        # Check effect metadata for trust level override
+        metadata = self.cache.get_effect_metadata(effect_hash)
+        trust_level = self.trust_level
+        if metadata:
+            # L1 owner can mark effect as trusted
+            if metadata.get("trust_level") == "trusted":
+                trust_level = "trusted"
+
+        return run_effect_from_cache(
+            cache=self.cache,
+            effect_hash=effect_hash,
+            input_paths=input_paths,
+            output_path=output_path,
+            params=params,
+            analysis=analysis,
+            cache_id=step_cache_id,
+            trust_level=trust_level,
+        )
+
+    def is_temporal(self, effect_hash: str) -> bool:
+        """Check if effect is temporal."""
+        return check_effect_temporal(self.cache, effect_hash)
+
+    def get_api_type(self, effect_hash: str) -> str:
+        """Get effect API type."""
+        return get_effect_api_type(self.cache, effect_hash)

artdag/core/artdag/effects/sandbox.py

@@ -0,0 +1,431 @@
+"""
+Sandbox for effect execution.
+
+Uses bubblewrap (bwrap) for Linux namespace isolation.
+Provides controlled access to:
+  - Input files (read-only)
+  - Output file (write)
+  - stderr (logging)
+  - Seeded RNG
+"""
+
+import hashlib
+import json
+import logging
+import os
+import shutil
+import subprocess
+import tempfile
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class SandboxConfig:
+    """
+    Sandbox configuration.
+
+    Attributes:
+        trust_level: "untrusted" (full isolation) or "trusted" (allows subprocess)
+        venv_path: Path to effect's virtual environment
+        wheel_cache: Shared wheel cache directory
+        timeout: Maximum execution time in seconds
+        memory_limit: Memory limit in bytes (0 = unlimited)
+        allow_network: Whether to allow network access
+    """
+
+    trust_level: str = "untrusted"
+    venv_path: Optional[Path] = None
+    wheel_cache: Path = field(default_factory=lambda: Path("/var/cache/artdag/wheels"))
+    timeout: int = 3600  # 1 hour default
+    memory_limit: int = 0
+    allow_network: bool = False
+
+
+def is_bwrap_available() -> bool:
+    """Check if bubblewrap is available."""
+    try:
+        result = subprocess.run(
+            ["bwrap", "--version"],
+            capture_output=True,
+            text=True,
+        )
+        return result.returncode == 0
+    except FileNotFoundError:
+        return False
+
+
+def get_venv_path(dependencies: List[str], cache_dir: Path = None) -> Path:
+    """
+    Get or create venv for given dependencies.
+
+    Uses hash of sorted dependencies for cache key.
+
+    Args:
+        dependencies: List of pip package specifiers
+        cache_dir: Base directory for venv cache
+
+    Returns:
+        Path to venv directory
+    """
+    cache_dir = cache_dir or Path("/var/cache/artdag/venvs")
+    cache_dir.mkdir(parents=True, exist_ok=True)
+
+    # Compute deps hash
+    sorted_deps = sorted(dep.lower().strip() for dep in dependencies)
+    deps_str = "\n".join(sorted_deps)
+    deps_hash = hashlib.sha3_256(deps_str.encode()).hexdigest()[:16]
+
+    venv_path = cache_dir / deps_hash
+
+    if venv_path.exists():
+        logger.debug(f"Reusing venv at {venv_path}")
+        return venv_path
+
+    # Create new venv
+    logger.info(f"Creating venv for {len(dependencies)} deps at {venv_path}")
+
+    subprocess.run(
+        ["python", "-m", "venv", str(venv_path)],
+        check=True,
+    )
+
+    # Install dependencies
+    pip_path = venv_path / "bin" / "pip"
+    wheel_cache = Path("/var/cache/artdag/wheels")
+
+    if dependencies:
+        cmd = [
+            str(pip_path),
+            "install",
+            "--cache-dir", str(wheel_cache),
+            *dependencies,
+        ]
+        subprocess.run(cmd, check=True)
+
+    return venv_path
+
+
+@dataclass
+class SandboxResult:
+    """Result of sandboxed execution."""
+
+    success: bool
+    output_path: Optional[Path] = None
+    stderr: str = ""
+    exit_code: int = 0
+    error: Optional[str] = None
+
+
+class Sandbox:
+    """
+    Sandboxed effect execution environment.
+
+    Uses bubblewrap for namespace isolation when available,
+    falls back to subprocess with restricted permissions.
+    """
+
+    def __init__(self, config: SandboxConfig = None):
+        self.config = config or SandboxConfig()
+        self._temp_dirs: List[Path] = []
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.cleanup()
+
+    def cleanup(self):
+        """Clean up temporary directories."""
+        for temp_dir in self._temp_dirs:
+            if temp_dir.exists():
+                shutil.rmtree(temp_dir, ignore_errors=True)
+        self._temp_dirs = []
+
+    def _create_temp_dir(self) -> Path:
+        """Create a temporary directory for sandbox use."""
+        temp_dir = Path(tempfile.mkdtemp(prefix="artdag_sandbox_"))
+        self._temp_dirs.append(temp_dir)
+        return temp_dir
+
+    def run_effect(
+        self,
+        effect_path: Path,
+        input_paths: List[Path],
+        output_path: Path,
+        params: Dict[str, Any],
+        bindings: Dict[str, List[float]] = None,
+        seed: int = 0,
+    ) -> SandboxResult:
+        """
+        Run an effect in the sandbox.
+
+        Args:
+            effect_path: Path to effect.py
+            input_paths: List of input file paths
+            output_path: Output file path
+            params: Effect parameters
+            bindings: Per-frame parameter bindings
+            seed: RNG seed for determinism
+
+        Returns:
+            SandboxResult with success status and output
+        """
+        bindings = bindings or {}
+
+        # Create work directory
+        work_dir = self._create_temp_dir()
+        config_path = work_dir / "config.json"
+        effect_copy = work_dir / "effect.py"
+
+        # Copy effect to work dir
+        shutil.copy(effect_path, effect_copy)
+
+        # Write config file
+        config_data = {
+            "input_paths": [str(p) for p in input_paths],
+            "output_path": str(output_path),
+            "params": params,
+            "bindings": bindings,
+            "seed": seed,
+        }
+        config_path.write_text(json.dumps(config_data))
+
+        if is_bwrap_available() and self.config.trust_level == "untrusted":
+            return self._run_with_bwrap(
+                effect_copy, config_path, input_paths, output_path, work_dir
+            )
+        else:
+            return self._run_subprocess(
+                effect_copy, config_path, input_paths, output_path, work_dir
+            )
+
+    def _run_with_bwrap(
+        self,
+        effect_path: Path,
+        config_path: Path,
+        input_paths: List[Path],
+        output_path: Path,
+        work_dir: Path,
+    ) -> SandboxResult:
+        """Run effect with bubblewrap isolation."""
+        logger.info("Running effect in bwrap sandbox")
+
+        # Build bwrap command
+        cmd = [
+            "bwrap",
+            # New PID namespace
+            "--unshare-pid",
+            # No network
+            "--unshare-net",
+            # Read-only root filesystem
+            "--ro-bind", "/", "/",
+            # Read-write work directory
+            "--bind", str(work_dir), str(work_dir),
+            # Read-only input files
+        ]
+
+        for input_path in input_paths:
+            cmd.extend(["--ro-bind", str(input_path), str(input_path)])
+
+        # Bind output directory as writable
+        output_dir = output_path.parent
+        output_dir.mkdir(parents=True, exist_ok=True)
+        cmd.extend(["--bind", str(output_dir), str(output_dir)])
+
+        # Bind venv if available
+        if self.config.venv_path and self.config.venv_path.exists():
+            cmd.extend(["--ro-bind", str(self.config.venv_path), str(self.config.venv_path)])
+            python_path = self.config.venv_path / "bin" / "python"
+        else:
+            python_path = Path("/usr/bin/python3")
+
+        # Add runner script
+        runner_script = self._get_runner_script()
+        runner_path = work_dir / "runner.py"
+        runner_path.write_text(runner_script)
+
+        # Run the effect
+        cmd.extend([
+            str(python_path),
+            str(runner_path),
+            str(effect_path),
+            str(config_path),
+        ])
+
+        try:
+            result = subprocess.run(
+                cmd,
+                capture_output=True,
+                text=True,
+                timeout=self.config.timeout,
+            )
+
+            if result.returncode == 0 and output_path.exists():
+                return SandboxResult(
+                    success=True,
+                    output_path=output_path,
+                    stderr=result.stderr,
+                    exit_code=0,
+                )
+            else:
+                return SandboxResult(
+                    success=False,
+                    stderr=result.stderr,
+                    exit_code=result.returncode,
+                    error=result.stderr or "Effect execution failed",
+                )
+
+        except subprocess.TimeoutExpired:
+            return SandboxResult(
+                success=False,
+                error=f"Effect timed out after {self.config.timeout}s",
+                exit_code=-1,
+            )
+        except Exception as e:
+            return SandboxResult(
+                success=False,
+                error=str(e),
+                exit_code=-1,
+            )
+
+    def _run_subprocess(
+        self,
+        effect_path: Path,
+        config_path: Path,
+        input_paths: List[Path],
+        output_path: Path,
+        work_dir: Path,
+    ) -> SandboxResult:
+        """Run effect in subprocess (fallback without bwrap)."""
+        logger.warning("Running effect without sandbox isolation")
+
+        # Create runner script
+        runner_script = self._get_runner_script()
+        runner_path = work_dir / "runner.py"
+        runner_path.write_text(runner_script)
+
+        # Determine Python path
+        if self.config.venv_path and self.config.venv_path.exists():
+            python_path = self.config.venv_path / "bin" / "python"
+        else:
+            python_path = "python3"
+
+        cmd = [
+            str(python_path),
+            str(runner_path),
+            str(effect_path),
+            str(config_path),
+        ]
+
+        try:
+            result = subprocess.run(
+                cmd,
+                capture_output=True,
+                text=True,
+                timeout=self.config.timeout,
+                cwd=str(work_dir),
+            )
+
+            if result.returncode == 0 and output_path.exists():
+                return SandboxResult(
+                    success=True,
+                    output_path=output_path,
+                    stderr=result.stderr,
+                    exit_code=0,
+                )
+            else:
+                return SandboxResult(
+                    success=False,
+                    stderr=result.stderr,
+                    exit_code=result.returncode,
+                    error=result.stderr or "Effect execution failed",
+                )
+
+        except subprocess.TimeoutExpired:
+            return SandboxResult(
+                success=False,
+                error=f"Effect timed out after {self.config.timeout}s",
+                exit_code=-1,
+            )
+        except Exception as e:
+            return SandboxResult(
+                success=False,
+                error=str(e),
+                exit_code=-1,
+            )
+
+    def _get_runner_script(self) -> str:
+        """Get the runner script that executes effects."""
+        return '''#!/usr/bin/env python3
+"""Effect runner script - executed in sandbox."""
+
+import importlib.util
+import json
+import sys
+from pathlib import Path
+
+def load_effect(effect_path):
+    """Load effect module from path."""
+    spec = importlib.util.spec_from_file_location("effect", effect_path)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+def main():
+    if len(sys.argv) < 3:
+        print("Usage: runner.py <effect_path> <config_path>", file=sys.stderr)
+        sys.exit(1)
+
+    effect_path = Path(sys.argv[1])
+    config_path = Path(sys.argv[2])
+
+    # Load config
+    config = json.loads(config_path.read_text())
+
+    input_paths = [Path(p) for p in config["input_paths"]]
+    output_path = Path(config["output_path"])
+    params = config["params"]
+    bindings = config.get("bindings", {})
+    seed = config.get("seed", 0)
+
+    # Load effect
+    effect = load_effect(effect_path)
+
+    # Check API type
+    if hasattr(effect, "process"):
+        # Whole-video API
+        from artdag.effects.meta import ExecutionContext
+        ctx = ExecutionContext(
+            input_paths=[str(p) for p in input_paths],
+            output_path=str(output_path),
+            params=params,
+            seed=seed,
+            bindings=bindings,
+        )
+        effect.process(input_paths, output_path, params, ctx)
+
+    elif hasattr(effect, "process_frame"):
+        # Frame-by-frame API
+        from artdag.effects.frame_processor import process_video
+
+        result_path, _ = process_video(
+            input_path=input_paths[0],
+            output_path=output_path,
+            process_frame=effect.process_frame,
+            params=params,
+            bindings=bindings,
+        )
+
+    else:
+        print("Effect must have process() or process_frame()", file=sys.stderr)
+        sys.exit(1)
+
+    print(f"Success: {output_path}", file=sys.stderr)
+
+if __name__ == "__main__":
+    main()
+'''

artdag/core/artdag/engine.py

@@ -0,0 +1,246 @@
+# primitive/engine.py
+"""
+DAG execution engine.
+
+Executes DAGs by:
+1. Resolving nodes in topological order
+2. Checking cache for each node
+3. Running executors for cache misses
+4. Storing results in cache
+"""
+
+import logging
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Callable, Dict, List, Optional
+
+from .dag import DAG, Node, NodeType
+from .cache import Cache
+from .executor import Executor, get_executor
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class ExecutionResult:
+    """Result of executing a DAG."""
+    success: bool
+    output_path: Optional[Path] = None
+    error: Optional[str] = None
+    execution_time: float = 0.0
+    nodes_executed: int = 0
+    nodes_cached: int = 0
+    node_results: Dict[str, Path] = field(default_factory=dict)
+
+
+@dataclass
+class NodeProgress:
+    """Progress update for a node."""
+    node_id: str
+    node_type: str
+    status: str  # "pending", "running", "cached", "completed", "failed"
+    progress: float = 0.0  # 0.0 to 1.0
+    message: str = ""
+
+
+# Progress callback type
+ProgressCallback = Callable[[NodeProgress], None]
+
+
+class Engine:
+    """
+    DAG execution engine.
+
+    Manages cache, resolves dependencies, and runs executors.
+    """
+
+    def __init__(self, cache_dir: Path | str):
+        self.cache = Cache(cache_dir)
+        self._progress_callback: Optional[ProgressCallback] = None
+
+    def set_progress_callback(self, callback: ProgressCallback):
+        """Set callback for progress updates."""
+        self._progress_callback = callback
+
+    def _report_progress(self, progress: NodeProgress):
+        """Report progress to callback if set."""
+        if self._progress_callback:
+            try:
+                self._progress_callback(progress)
+            except Exception as e:
+                logger.warning(f"Progress callback error: {e}")
+
+    def execute(self, dag: DAG) -> ExecutionResult:
+        """
+        Execute a DAG and return the result.
+
+        Args:
+            dag: The DAG to execute
+
+        Returns:
+            ExecutionResult with output path or error
+        """
+        start_time = time.time()
+        node_results: Dict[str, Path] = {}
+        nodes_executed = 0
+        nodes_cached = 0
+
+        # Validate DAG
+        errors = dag.validate()
+        if errors:
+            return ExecutionResult(
+                success=False,
+                error=f"Invalid DAG: {errors}",
+                execution_time=time.time() - start_time,
+            )
+
+        # Get topological order
+        try:
+            order = dag.topological_order()
+        except Exception as e:
+            return ExecutionResult(
+                success=False,
+                error=f"Failed to order DAG: {e}",
+                execution_time=time.time() - start_time,
+            )
+
+        # Execute each node
+        for node_id in order:
+            node = dag.get_node(node_id)
+            type_str = node.node_type.name if isinstance(node.node_type, NodeType) else str(node.node_type)
+
+            # Report starting
+            self._report_progress(NodeProgress(
+                node_id=node_id,
+                node_type=type_str,
+                status="pending",
+                message=f"Processing {type_str}",
+            ))
+
+            # Check cache first
+            cached_path = self.cache.get(node_id)
+            if cached_path is not None:
+                node_results[node_id] = cached_path
+                nodes_cached += 1
+                self._report_progress(NodeProgress(
+                    node_id=node_id,
+                    node_type=type_str,
+                    status="cached",
+                    progress=1.0,
+                    message="Using cached result",
+                ))
+                continue
+
+            # Get executor
+            executor = get_executor(node.node_type)
+            if executor is None:
+                return ExecutionResult(
+                    success=False,
+                    error=f"No executor for node type: {node.node_type}",
+                    execution_time=time.time() - start_time,
+                    node_results=node_results,
+                )
+
+            # Resolve input paths
+            input_paths = []
+            for input_id in node.inputs:
+                if input_id not in node_results:
+                    return ExecutionResult(
+                        success=False,
+                        error=f"Missing input {input_id} for node {node_id}",
+                        execution_time=time.time() - start_time,
+                        node_results=node_results,
+                    )
+                input_paths.append(node_results[input_id])
+
+            # Determine output path
+            output_path = self.cache.get_output_path(node_id, ".mkv")
+
+            # Execute
+            self._report_progress(NodeProgress(
+                node_id=node_id,
+                node_type=type_str,
+                status="running",
+                progress=0.5,
+                message=f"Executing {type_str}",
+            ))
+
+            node_start = time.time()
+            try:
+                result_path = executor.execute(
+                    config=node.config,
+                    inputs=input_paths,
+                    output_path=output_path,
+                )
+                node_time = time.time() - node_start
+
+                # Store in cache (file is already at output_path)
+                self.cache.put(
+                    node_id=node_id,
+                    source_path=result_path,
+                    node_type=type_str,
+                    execution_time=node_time,
+                    move=False,  # Already in place
+                )
+
+                node_results[node_id] = result_path
+                nodes_executed += 1
+
+                self._report_progress(NodeProgress(
+                    node_id=node_id,
+                    node_type=type_str,
+                    status="completed",
+                    progress=1.0,
+                    message=f"Completed in {node_time:.2f}s",
+                ))
+
+            except Exception as e:
+                logger.error(f"Node {node_id} failed: {e}")
+                self._report_progress(NodeProgress(
+                    node_id=node_id,
+                    node_type=type_str,
+                    status="failed",
+                    message=str(e),
+                ))
+                return ExecutionResult(
+                    success=False,
+                    error=f"Node {node_id} ({type_str}) failed: {e}",
+                    execution_time=time.time() - start_time,
+                    node_results=node_results,
+                    nodes_executed=nodes_executed,
+                    nodes_cached=nodes_cached,
+                )
+
+        # Get final output
+        output_path = node_results.get(dag.output_id)
+
+        return ExecutionResult(
+            success=True,
+            output_path=output_path,
+            execution_time=time.time() - start_time,
+            nodes_executed=nodes_executed,
+            nodes_cached=nodes_cached,
+            node_results=node_results,
+        )
+
+    def execute_node(self, node: Node, inputs: List[Path]) -> Path:
+        """
+        Execute a single node (bypassing DAG structure).
+
+        Useful for testing individual executors.
+        """
+        executor = get_executor(node.node_type)
+        if executor is None:
+            raise ValueError(f"No executor for node type: {node.node_type}")
+
+        output_path = self.cache.get_output_path(node.node_id, ".mkv")
+        return executor.execute(node.config, inputs, output_path)
+
+    def get_cache_stats(self):
+        """Get cache statistics."""
+        return self.cache.get_stats()
+
+    def clear_cache(self):
+        """Clear the cache."""
+        self.cache.clear()

artdag/core/artdag/executor.py

@@ -0,0 +1,106 @@
+# primitive/executor.py
+"""
+Executor base class and registry.
+
+Executors implement the actual operations for each node type.
+They are registered by node type and looked up during execution.
+"""
+
+import logging
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import Any, Callable, Dict, List, Optional, Type
+
+from .dag import NodeType
+
+logger = logging.getLogger(__name__)
+
+# Global executor registry
+_EXECUTORS: Dict[NodeType | str, Type["Executor"]] = {}
+
+
+class Executor(ABC):
+    """
+    Base class for node executors.
+
+    Subclasses implement execute() to perform the actual operation.
+    """
+
+    @abstractmethod
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        """
+        Execute the node operation.
+
+        Args:
+            config: Node configuration
+            inputs: Paths to input files (from resolved input nodes)
+            output_path: Where to write the output
+
+        Returns:
+            Path to the output file
+        """
+        pass
+
+    def validate_config(self, config: Dict[str, Any]) -> List[str]:
+        """
+        Validate node configuration.
+
+        Returns list of error messages (empty if valid).
+        Override in subclasses for specific validation.
+        """
+        return []
+
+    def estimate_output_size(self, config: Dict[str, Any], input_sizes: List[int]) -> int:
+        """
+        Estimate output size in bytes.
+
+        Override for better estimates. Default returns sum of inputs.
+        """
+        return sum(input_sizes) if input_sizes else 0
+
+
+def register_executor(node_type: NodeType | str) -> Callable:
+    """
+    Decorator to register an executor for a node type.
+
+    Usage:
+        @register_executor(NodeType.SOURCE)
+        class SourceExecutor(Executor):
+            ...
+    """
+    def decorator(cls: Type[Executor]) -> Type[Executor]:
+        if node_type in _EXECUTORS:
+            logger.warning(f"Overwriting executor for {node_type}")
+        _EXECUTORS[node_type] = cls
+        return cls
+    return decorator
+
+
+def get_executor(node_type: NodeType | str) -> Optional[Executor]:
+    """
+    Get an executor instance for a node type.
+
+    Returns None if no executor is registered.
+    """
+    executor_cls = _EXECUTORS.get(node_type)
+    if executor_cls is None:
+        return None
+    return executor_cls()
+
+
+def list_executors() -> Dict[str, Type[Executor]]:
+    """List all registered executors."""
+    return {
+        (k.name if isinstance(k, NodeType) else k): v
+        for k, v in _EXECUTORS.items()
+    }
+
+
+def clear_executors():
+    """Clear all registered executors (for testing)."""
+    _EXECUTORS.clear()

artdag/core/artdag/nodes/__init__.py

@@ -0,0 +1,11 @@
+# primitive/nodes/__init__.py
+"""
+Built-in node executors.
+
+Import this module to register all built-in executors.
+"""
+
+from . import source
+from . import transform
+from . import compose
+from . import effect

artdag/core/artdag/nodes/compose.py

@@ -0,0 +1,548 @@
+# primitive/nodes/compose.py
+"""
+Compose executors: Combine multiple media inputs.
+
+Primitives: SEQUENCE, LAYER, MUX, BLEND
+"""
+
+import logging
+import shutil
+import subprocess
+from pathlib import Path
+from typing import Any, Dict, List
+
+from ..dag import NodeType
+from ..executor import Executor, register_executor
+from .encoding import WEB_ENCODING_ARGS_STR, get_web_encoding_args
+
+logger = logging.getLogger(__name__)
+
+
+def _get_duration(path: Path) -> float:
+    """Get media duration in seconds."""
+    cmd = [
+        "ffprobe", "-v", "error",
+        "-show_entries", "format=duration",
+        "-of", "csv=p=0",
+        str(path)
+    ]
+    result = subprocess.run(cmd, capture_output=True, text=True)
+    return float(result.stdout.strip())
+
+
+def _get_video_info(path: Path) -> dict:
+    """Get video width, height, frame rate, and sample rate."""
+    cmd = [
+        "ffprobe", "-v", "error",
+        "-select_streams", "v:0",
+        "-show_entries", "stream=width,height,r_frame_rate",
+        "-of", "csv=p=0",
+        str(path)
+    ]
+    result = subprocess.run(cmd, capture_output=True, text=True)
+    parts = result.stdout.strip().split(",")
+    width = int(parts[0]) if len(parts) > 0 and parts[0] else 1920
+    height = int(parts[1]) if len(parts) > 1 and parts[1] else 1080
+    fps_str = parts[2] if len(parts) > 2 else "30/1"
+    # Parse frame rate (e.g., "30/1" or "30000/1001")
+    if "/" in fps_str:
+        num, den = fps_str.split("/")
+        fps = float(num) / float(den) if float(den) != 0 else 30
+    else:
+        fps = float(fps_str) if fps_str else 30
+
+    # Get audio sample rate
+    cmd_audio = [
+        "ffprobe", "-v", "error",
+        "-select_streams", "a:0",
+        "-show_entries", "stream=sample_rate",
+        "-of", "csv=p=0",
+        str(path)
+    ]
+    result_audio = subprocess.run(cmd_audio, capture_output=True, text=True)
+    sample_rate = int(result_audio.stdout.strip()) if result_audio.stdout.strip() else 44100
+
+    return {"width": width, "height": height, "fps": fps, "sample_rate": sample_rate}
+
+
+@register_executor(NodeType.SEQUENCE)
+class SequenceExecutor(Executor):
+    """
+    Concatenate inputs in time order.
+
+    Config:
+        transition: Transition config
+            type: "cut" | "crossfade" | "fade"
+            duration: Transition duration in seconds
+        target_size: How to determine output dimensions when inputs differ
+            "first": Use first input's dimensions (default)
+            "last": Use last input's dimensions
+            "largest": Use largest width and height from all inputs
+            "explicit": Use width/height config values
+        width: Target width (when target_size="explicit")
+        height: Target height (when target_size="explicit")
+        background: Padding color for letterbox/pillarbox (default: "black")
+    """
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        if len(inputs) < 1:
+            raise ValueError("SEQUENCE requires at least one input")
+
+        if len(inputs) == 1:
+            output_path.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy2(inputs[0], output_path)
+            return output_path
+
+        transition = config.get("transition", {"type": "cut"})
+        transition_type = transition.get("type", "cut")
+        transition_duration = transition.get("duration", 0.5)
+
+        # Size handling config
+        target_size = config.get("target_size", "first")
+        width = config.get("width")
+        height = config.get("height")
+        background = config.get("background", "black")
+
+        if transition_type == "cut":
+            return self._concat_cut(inputs, output_path, target_size, width, height, background)
+        elif transition_type == "crossfade":
+            return self._concat_crossfade(inputs, output_path, transition_duration)
+        elif transition_type == "fade":
+            return self._concat_fade(inputs, output_path, transition_duration)
+        else:
+            raise ValueError(f"Unknown transition type: {transition_type}")
+
+    def _concat_cut(
+        self,
+        inputs: List[Path],
+        output_path: Path,
+        target_size: str = "first",
+        width: int = None,
+        height: int = None,
+        background: str = "black",
+    ) -> Path:
+        """
+        Concatenate with scaling/padding to handle different resolutions.
+
+        Args:
+            inputs: Input video paths
+            output_path: Output path
+            target_size: How to determine output size:
+                - "first": Use first input's dimensions (default)
+                - "last": Use last input's dimensions
+                - "largest": Use largest dimensions from all inputs
+                - "explicit": Use width/height params
+            width: Explicit width (when target_size="explicit")
+            height: Explicit height (when target_size="explicit")
+            background: Padding color (default: black)
+        """
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        n = len(inputs)
+        input_args = []
+        for p in inputs:
+            input_args.extend(["-i", str(p)])
+
+        # Get video info for all inputs
+        infos = [_get_video_info(p) for p in inputs]
+
+        # Determine target dimensions
+        if target_size == "explicit" and width and height:
+            target_w, target_h = width, height
+        elif target_size == "last":
+            target_w, target_h = infos[-1]["width"], infos[-1]["height"]
+        elif target_size == "largest":
+            target_w = max(i["width"] for i in infos)
+            target_h = max(i["height"] for i in infos)
+        else:  # "first" or default
+            target_w, target_h = infos[0]["width"], infos[0]["height"]
+
+        # Use common frame rate (from first input) and sample rate
+        target_fps = infos[0]["fps"]
+        target_sr = max(i["sample_rate"] for i in infos)
+
+        # Build filter for each input: scale to fit + pad to target size
+        filter_parts = []
+        for i in range(n):
+            # Scale to fit within target, maintaining aspect ratio, then pad
+            vf = (
+                f"[{i}:v]scale={target_w}:{target_h}:force_original_aspect_ratio=decrease,"
+                f"pad={target_w}:{target_h}:(ow-iw)/2:(oh-ih)/2:color={background},"
+                f"setsar=1,fps={target_fps:.6f}[v{i}]"
+            )
+            # Resample audio to common rate
+            af = f"[{i}:a]aresample={target_sr}[a{i}]"
+            filter_parts.append(vf)
+            filter_parts.append(af)
+
+        # Build concat filter
+        stream_labels = "".join(f"[v{i}][a{i}]" for i in range(n))
+        filter_parts.append(f"{stream_labels}concat=n={n}:v=1:a=1[outv][outa]")
+
+        filter_complex = ";".join(filter_parts)
+
+        cmd = [
+            "ffmpeg", "-y",
+            *input_args,
+            "-filter_complex", filter_complex,
+            "-map", "[outv]",
+            "-map", "[outa]",
+            *get_web_encoding_args(),
+            str(output_path)
+        ]
+
+        logger.debug(f"SEQUENCE cut: {n} clips -> {target_w}x{target_h} (web-optimized)")
+        result = subprocess.run(cmd, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            raise RuntimeError(f"Concat failed: {result.stderr}")
+
+        return output_path
+
+    def _concat_crossfade(
+        self,
+        inputs: List[Path],
+        output_path: Path,
+        duration: float,
+    ) -> Path:
+        """Concatenate with crossfade transitions."""
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        durations = [_get_duration(p) for p in inputs]
+        n = len(inputs)
+        input_args = " ".join(f"-i {p}" for p in inputs)
+
+        # Build xfade filter chain
+        filter_parts = []
+        current = "[0:v]"
+
+        for i in range(1, n):
+            offset = sum(durations[:i]) - duration * i
+            next_input = f"[{i}:v]"
+            output_label = f"[v{i}]" if i < n - 1 else "[outv]"
+            filter_parts.append(
+                f"{current}{next_input}xfade=transition=fade:duration={duration}:offset={offset}{output_label}"
+            )
+            current = output_label
+
+        # Audio crossfade chain
+        audio_current = "[0:a]"
+        for i in range(1, n):
+            next_input = f"[{i}:a]"
+            output_label = f"[a{i}]" if i < n - 1 else "[outa]"
+            filter_parts.append(
+                f"{audio_current}{next_input}acrossfade=d={duration}{output_label}"
+            )
+            audio_current = output_label
+
+        filter_complex = ";".join(filter_parts)
+
+        cmd = f'ffmpeg -y {input_args} -filter_complex "{filter_complex}" -map [outv] -map [outa] {WEB_ENCODING_ARGS_STR} {output_path}'
+
+        logger.debug(f"SEQUENCE crossfade: {n} clips (web-optimized)")
+        result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            logger.warning(f"Crossfade failed, falling back to cut: {result.stderr[:200]}")
+            return self._concat_cut(inputs, output_path)
+
+        return output_path
+
+    def _concat_fade(
+        self,
+        inputs: List[Path],
+        output_path: Path,
+        duration: float,
+    ) -> Path:
+        """Concatenate with fade out/in transitions."""
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        faded_paths = []
+        for i, path in enumerate(inputs):
+            clip_dur = _get_duration(path)
+            faded_path = output_path.parent / f"_faded_{i}.mkv"
+
+            cmd = [
+                "ffmpeg", "-y",
+                "-i", str(path),
+                "-vf", f"fade=in:st=0:d={duration},fade=out:st={clip_dur - duration}:d={duration}",
+                "-af", f"afade=in:st=0:d={duration},afade=out:st={clip_dur - duration}:d={duration}",
+                "-c:v", "libx264", "-preset", "ultrafast", "-crf", "18",
+                "-c:a", "aac",
+                str(faded_path)
+            ]
+            subprocess.run(cmd, capture_output=True, check=True)
+            faded_paths.append(faded_path)
+
+        result = self._concat_cut(faded_paths, output_path)
+
+        for p in faded_paths:
+            p.unlink()
+
+        return result
+
+
+@register_executor(NodeType.LAYER)
+class LayerExecutor(Executor):
+    """
+    Layer inputs spatially (overlay/composite).
+
+    Config:
+        inputs: List of per-input configs
+            position: [x, y] offset
+            opacity: 0.0-1.0
+            scale: Scale factor
+    """
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        if len(inputs) < 1:
+            raise ValueError("LAYER requires at least one input")
+
+        if len(inputs) == 1:
+            output_path.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy2(inputs[0], output_path)
+            return output_path
+
+        input_configs = config.get("inputs", [{}] * len(inputs))
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        input_args = " ".join(f"-i {p}" for p in inputs)
+        n = len(inputs)
+        filter_parts = []
+        current = "[0:v]"
+
+        for i in range(1, n):
+            cfg = input_configs[i] if i < len(input_configs) else {}
+            x, y = cfg.get("position", [0, 0])
+            opacity = cfg.get("opacity", 1.0)
+            scale = cfg.get("scale", 1.0)
+
+            scale_label = f"[s{i}]"
+            if scale != 1.0:
+                filter_parts.append(f"[{i}:v]scale=iw*{scale}:ih*{scale}{scale_label}")
+                overlay_input = scale_label
+            else:
+                overlay_input = f"[{i}:v]"
+
+            output_label = f"[v{i}]" if i < n - 1 else "[outv]"
+
+            if opacity < 1.0:
+                filter_parts.append(
+                    f"{overlay_input}format=rgba,colorchannelmixer=aa={opacity}[a{i}]"
+                )
+                overlay_input = f"[a{i}]"
+
+            filter_parts.append(
+                f"{current}{overlay_input}overlay=x={x}:y={y}:format=auto{output_label}"
+            )
+            current = output_label
+
+        filter_complex = ";".join(filter_parts)
+
+        cmd = f'ffmpeg -y {input_args} -filter_complex "{filter_complex}" -map [outv] -map 0:a? {WEB_ENCODING_ARGS_STR} {output_path}'
+
+        logger.debug(f"LAYER: {n} inputs (web-optimized)")
+        result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            raise RuntimeError(f"Layer failed: {result.stderr}")
+
+        return output_path
+
+
+@register_executor(NodeType.MUX)
+class MuxExecutor(Executor):
+    """
+    Combine video and audio streams.
+
+    Config:
+        video_stream: Index of video input (default: 0)
+        audio_stream: Index of audio input (default: 1)
+        shortest: End when shortest stream ends (default: True)
+    """
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        if len(inputs) < 2:
+            raise ValueError("MUX requires at least 2 inputs (video + audio)")
+
+        video_idx = config.get("video_stream", 0)
+        audio_idx = config.get("audio_stream", 1)
+        shortest = config.get("shortest", True)
+
+        video_path = inputs[video_idx]
+        audio_path = inputs[audio_idx]
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        cmd = [
+            "ffmpeg", "-y",
+            "-i", str(video_path),
+            "-i", str(audio_path),
+            "-c:v", "copy",
+            "-c:a", "aac",
+            "-map", "0:v:0",
+            "-map", "1:a:0",
+        ]
+
+        if shortest:
+            cmd.append("-shortest")
+
+        cmd.append(str(output_path))
+
+        logger.debug(f"MUX: video={video_path.name} + audio={audio_path.name}")
+        result = subprocess.run(cmd, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            raise RuntimeError(f"Mux failed: {result.stderr}")
+
+        return output_path
+
+
+@register_executor(NodeType.BLEND)
+class BlendExecutor(Executor):
+    """
+    Blend two inputs using a blend mode.
+
+    Config:
+        mode: Blend mode (multiply, screen, overlay, add, etc.)
+        opacity: 0.0-1.0 for second input
+    """
+
+    BLEND_MODES = {
+        "multiply": "multiply",
+        "screen": "screen",
+        "overlay": "overlay",
+        "add": "addition",
+        "subtract": "subtract",
+        "average": "average",
+        "difference": "difference",
+        "lighten": "lighten",
+        "darken": "darken",
+    }
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        if len(inputs) != 2:
+            raise ValueError("BLEND requires exactly 2 inputs")
+
+        mode = config.get("mode", "overlay")
+        opacity = config.get("opacity", 0.5)
+
+        if mode not in self.BLEND_MODES:
+            raise ValueError(f"Unknown blend mode: {mode}")
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        blend_mode = self.BLEND_MODES[mode]
+
+        if opacity < 1.0:
+            filter_complex = (
+                f"[1:v]format=rgba,colorchannelmixer=aa={opacity}[b];"
+                f"[0:v][b]blend=all_mode={blend_mode}"
+            )
+        else:
+            filter_complex = f"[0:v][1:v]blend=all_mode={blend_mode}"
+
+        cmd = [
+            "ffmpeg", "-y",
+            "-i", str(inputs[0]),
+            "-i", str(inputs[1]),
+            "-filter_complex", filter_complex,
+            "-map", "0:a?",
+            *get_web_encoding_args(),
+            str(output_path)
+        ]
+
+        logger.debug(f"BLEND: {mode} (opacity={opacity}) (web-optimized)")
+        result = subprocess.run(cmd, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            raise RuntimeError(f"Blend failed: {result.stderr}")
+
+        return output_path
+
+
+@register_executor(NodeType.AUDIO_MIX)
+class AudioMixExecutor(Executor):
+    """
+    Mix multiple audio streams.
+
+    Config:
+        gains: List of gain values per input (0.0-2.0, default 1.0)
+        normalize: Normalize output to prevent clipping (default True)
+    """
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        if len(inputs) < 2:
+            raise ValueError("AUDIO_MIX requires at least 2 inputs")
+
+        gains = config.get("gains", [1.0] * len(inputs))
+        normalize = config.get("normalize", True)
+
+        # Pad gains list if too short
+        while len(gains) < len(inputs):
+            gains.append(1.0)
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        # Build filter: apply volume to each input, then mix
+        filter_parts = []
+        mix_inputs = []
+
+        for i, gain in enumerate(gains[:len(inputs)]):
+            if gain != 1.0:
+                filter_parts.append(f"[{i}:a]volume={gain}[a{i}]")
+                mix_inputs.append(f"[a{i}]")
+            else:
+                mix_inputs.append(f"[{i}:a]")
+
+        # amix filter
+        normalize_flag = 1 if normalize else 0
+        mix_filter = f"{''.join(mix_inputs)}amix=inputs={len(inputs)}:normalize={normalize_flag}[aout]"
+        filter_parts.append(mix_filter)
+
+        filter_complex = ";".join(filter_parts)
+
+        cmd = [
+            "ffmpeg", "-y",
+        ]
+        for p in inputs:
+            cmd.extend(["-i", str(p)])
+
+        cmd.extend([
+            "-filter_complex", filter_complex,
+            "-map", "[aout]",
+            "-c:a", "aac",
+            str(output_path)
+        ])
+
+        logger.debug(f"AUDIO_MIX: {len(inputs)} inputs, gains={gains[:len(inputs)]}")
+        result = subprocess.run(cmd, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            raise RuntimeError(f"Audio mix failed: {result.stderr}")
+
+        return output_path

artdag/core/artdag/nodes/effect.py

@@ -0,0 +1,520 @@
+# artdag/nodes/effect.py
+"""
+Effect executor: Apply effects from the registry or IPFS.
+
+Primitives: EFFECT
+
+Effects can be:
+1. Built-in (registered with @register_effect)
+2. Stored in IPFS (referenced by CID)
+"""
+
+import importlib.util
+import logging
+import os
+import re
+import shutil
+import tempfile
+from pathlib import Path
+from types import ModuleType
+from typing import Any, Callable, Dict, List, Optional, TypeVar
+
+import requests
+
+from ..executor import Executor, register_executor
+
+logger = logging.getLogger(__name__)
+
+# Type alias for effect functions: (input_path, output_path, config) -> output_path
+EffectFn = Callable[[Path, Path, Dict[str, Any]], Path]
+
+# Type variable for decorator
+F = TypeVar("F", bound=Callable[..., Any])
+
+# IPFS API multiaddr - same as ipfs_client.py for consistency
+# Docker uses /dns/ipfs/tcp/5001, local dev uses /ip4/127.0.0.1/tcp/5001
+IPFS_API = os.environ.get("IPFS_API", "/ip4/127.0.0.1/tcp/5001")
+
+# Connection timeout in seconds
+IPFS_TIMEOUT = int(os.environ.get("IPFS_TIMEOUT", "30"))
+
+
+def _get_ipfs_base_url() -> str:
+    """
+    Convert IPFS multiaddr to HTTP URL.
+
+    Matches the conversion logic in ipfs_client.py for consistency.
+    """
+    multiaddr = IPFS_API
+
+    # Handle /dns/hostname/tcp/port format (Docker)
+    dns_match = re.match(r"/dns[46]?/([^/]+)/tcp/(\d+)", multiaddr)
+    if dns_match:
+        return f"http://{dns_match.group(1)}:{dns_match.group(2)}"
+
+    # Handle /ip4/address/tcp/port format (local)
+    ip4_match = re.match(r"/ip4/([^/]+)/tcp/(\d+)", multiaddr)
+    if ip4_match:
+        return f"http://{ip4_match.group(1)}:{ip4_match.group(2)}"
+
+    # Fallback: assume it's already a URL or use default
+    if multiaddr.startswith("http"):
+        return multiaddr
+    return "http://127.0.0.1:5001"
+
+
+def _get_effects_cache_dir() -> Optional[Path]:
+    """Get the effects cache directory from environment or default."""
+    # Check both env var names (CACHE_DIR used by art-celery, ARTDAG_CACHE_DIR for standalone)
+    for env_var in ["CACHE_DIR", "ARTDAG_CACHE_DIR"]:
+        cache_dir = os.environ.get(env_var)
+        if cache_dir:
+            effects_dir = Path(cache_dir) / "_effects"
+            if effects_dir.exists():
+                return effects_dir
+
+    # Try default locations
+    for base in [Path.home() / ".artdag" / "cache", Path("/var/cache/artdag")]:
+        effects_dir = base / "_effects"
+        if effects_dir.exists():
+            return effects_dir
+
+    return None
+
+
+def _fetch_effect_from_ipfs(cid: str, effect_path: Path) -> bool:
+    """
+    Fetch an effect from IPFS and cache locally.
+
+    Uses the IPFS API endpoint (/api/v0/cat) for consistency with ipfs_client.py.
+    This works reliably in Docker where IPFS_API=/dns/ipfs/tcp/5001.
+
+    Returns True on success, False on failure.
+    """
+    try:
+        # Use IPFS API (same as ipfs_client.py)
+        base_url = _get_ipfs_base_url()
+        url = f"{base_url}/api/v0/cat"
+        params = {"arg": cid}
+
+        response = requests.post(url, params=params, timeout=IPFS_TIMEOUT)
+        response.raise_for_status()
+
+        # Cache locally
+        effect_path.parent.mkdir(parents=True, exist_ok=True)
+        effect_path.write_bytes(response.content)
+        logger.info(f"Fetched effect from IPFS: {cid[:16]}...")
+        return True
+
+    except Exception as e:
+        logger.error(f"Failed to fetch effect from IPFS {cid[:16]}...: {e}")
+        return False
+
+
+def _parse_pep723_dependencies(source: str) -> List[str]:
+    """
+    Parse PEP 723 dependencies from effect source code.
+
+    Returns list of package names (e.g., ["numpy", "opencv-python"]).
+    """
+    match = re.search(r"# /// script\n(.*?)# ///", source, re.DOTALL)
+    if not match:
+        return []
+
+    block = match.group(1)
+    deps_match = re.search(r'# dependencies = \[(.*?)\]', block, re.DOTALL)
+    if not deps_match:
+        return []
+
+    return re.findall(r'"([^"]+)"', deps_match.group(1))
+
+
+def _ensure_dependencies(dependencies: List[str], effect_cid: str) -> bool:
+    """
+    Ensure effect dependencies are installed.
+
+    Installs missing packages using pip. Returns True on success.
+    """
+    if not dependencies:
+        return True
+
+    missing = []
+    for dep in dependencies:
+        # Extract package name (strip version specifiers)
+        pkg_name = re.split(r'[<>=!]', dep)[0].strip()
+        # Normalize name (pip uses underscores, imports use underscores or hyphens)
+        pkg_name_normalized = pkg_name.replace('-', '_').lower()
+
+        try:
+            __import__(pkg_name_normalized)
+        except ImportError:
+            # Some packages have different import names
+            try:
+                # Try original name with hyphens replaced
+                __import__(pkg_name.replace('-', '_'))
+            except ImportError:
+                missing.append(dep)
+
+    if not missing:
+        return True
+
+    logger.info(f"Installing effect dependencies for {effect_cid[:16]}...: {missing}")
+
+    try:
+        import subprocess
+        import sys
+
+        result = subprocess.run(
+            [sys.executable, "-m", "pip", "install", "--quiet"] + missing,
+            capture_output=True,
+            text=True,
+            timeout=120,
+        )
+
+        if result.returncode != 0:
+            logger.error(f"Failed to install dependencies: {result.stderr}")
+            return False
+
+        logger.info(f"Installed dependencies: {missing}")
+        return True
+
+    except Exception as e:
+        logger.error(f"Error installing dependencies: {e}")
+        return False
+
+
+def _load_cached_effect(effect_cid: str) -> Optional[EffectFn]:
+    """
+    Load an effect by CID, fetching from IPFS if not cached locally.
+
+    Returns the effect function or None if not found.
+    """
+    effects_dir = _get_effects_cache_dir()
+
+    # Create cache dir if needed
+    if not effects_dir:
+        # Try to create default cache dir
+        for env_var in ["CACHE_DIR", "ARTDAG_CACHE_DIR"]:
+            cache_dir = os.environ.get(env_var)
+            if cache_dir:
+                effects_dir = Path(cache_dir) / "_effects"
+                effects_dir.mkdir(parents=True, exist_ok=True)
+                break
+
+        if not effects_dir:
+            effects_dir = Path.home() / ".artdag" / "cache" / "_effects"
+            effects_dir.mkdir(parents=True, exist_ok=True)
+
+    effect_path = effects_dir / effect_cid / "effect.py"
+
+    # If not cached locally, fetch from IPFS
+    if not effect_path.exists():
+        if not _fetch_effect_from_ipfs(effect_cid, effect_path):
+            logger.warning(f"Effect not found: {effect_cid[:16]}...")
+            return None
+
+    # Parse and install dependencies before loading
+    try:
+        source = effect_path.read_text()
+        dependencies = _parse_pep723_dependencies(source)
+        if dependencies:
+            logger.info(f"Effect {effect_cid[:16]}... requires: {dependencies}")
+            if not _ensure_dependencies(dependencies, effect_cid):
+                logger.error(f"Failed to install dependencies for effect {effect_cid[:16]}...")
+                return None
+    except Exception as e:
+        logger.error(f"Error parsing effect dependencies: {e}")
+        # Continue anyway - the effect might work without the deps check
+
+    # Load the effect module
+    try:
+        spec = importlib.util.spec_from_file_location("cached_effect", effect_path)
+        module = importlib.util.module_from_spec(spec)
+        spec.loader.exec_module(module)
+
+        # Check for frame-by-frame API
+        if hasattr(module, "process_frame"):
+            return _wrap_frame_effect(module, effect_path)
+
+        # Check for whole-video API
+        if hasattr(module, "process"):
+            return _wrap_video_effect(module)
+
+        # Check for old-style effect function
+        if hasattr(module, "effect"):
+            return module.effect
+
+        logger.warning(f"Effect has no recognized API: {effect_cid[:16]}...")
+        return None
+
+    except Exception as e:
+        logger.error(f"Failed to load effect {effect_cid[:16]}...: {e}")
+        return None
+
+
+def _wrap_frame_effect(module: ModuleType, effect_path: Path) -> EffectFn:
+    """Wrap a frame-by-frame effect to work with the executor API."""
+
+    def wrapped_effect(input_path: Path, output_path: Path, config: Dict[str, Any]) -> Path:
+        """Run frame-by-frame effect through FFmpeg pipes."""
+        try:
+            from ..effects.frame_processor import process_video
+        except ImportError:
+            logger.error("Frame processor not available - falling back to copy")
+            shutil.copy2(input_path, output_path)
+            return output_path
+
+        # Extract params from config (excluding internal keys)
+        params = {k: v for k, v in config.items()
+                  if k not in ("effect", "hash", "_binding")}
+
+        # Get bindings if present
+        bindings = {}
+        for key, value in config.items():
+            if isinstance(value, dict) and value.get("_resolved_values"):
+                bindings[key] = value["_resolved_values"]
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        actual_output = output_path.with_suffix(".mp4")
+
+        process_video(
+            input_path=input_path,
+            output_path=actual_output,
+            process_frame=module.process_frame,
+            params=params,
+            bindings=bindings,
+        )
+
+        return actual_output
+
+    return wrapped_effect
+
+
+def _wrap_video_effect(module: ModuleType) -> EffectFn:
+    """Wrap a whole-video effect to work with the executor API."""
+
+    def wrapped_effect(input_path: Path, output_path: Path, config: Dict[str, Any]) -> Path:
+        """Run whole-video effect."""
+        from ..effects.meta import ExecutionContext
+
+        params = {k: v for k, v in config.items()
+                  if k not in ("effect", "hash", "_binding")}
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        ctx = ExecutionContext(
+            input_paths=[str(input_path)],
+            output_path=str(output_path),
+            params=params,
+            seed=hash(str(input_path)) & 0xFFFFFFFF,
+        )
+
+        module.process([input_path], output_path, params, ctx)
+        return output_path
+
+    return wrapped_effect
+
+
+# Effect registry - maps effect names to implementations
+_EFFECTS: Dict[str, EffectFn] = {}
+
+
+def register_effect(name: str) -> Callable[[F], F]:
+    """Decorator to register an effect implementation."""
+    def decorator(func: F) -> F:
+        _EFFECTS[name] = func  # type: ignore[assignment]
+        return func
+    return decorator
+
+
+def get_effect(name: str) -> Optional[EffectFn]:
+    """Get an effect implementation by name."""
+    return _EFFECTS.get(name)
+
+
+# Built-in effects
+
+@register_effect("identity")
+def effect_identity(input_path: Path, output_path: Path, config: Dict[str, Any]) -> Path:
+    """
+    Identity effect - returns input unchanged.
+
+    This is the foundational effect: identity(x) = x
+    """
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+
+    # Remove existing output if any
+    if output_path.exists() or output_path.is_symlink():
+        output_path.unlink()
+
+    # Preserve extension from input
+    actual_output = output_path.with_suffix(input_path.suffix)
+    if actual_output.exists() or actual_output.is_symlink():
+        actual_output.unlink()
+
+    # Symlink to input (zero-copy identity)
+    os.symlink(input_path.resolve(), actual_output)
+    logger.debug(f"EFFECT identity: {input_path.name} -> {actual_output}")
+
+    return actual_output
+
+
+def _get_sexp_effect(effect_path: str, recipe_dir: Path = None) -> Optional[EffectFn]:
+    """
+    Load a sexp effect from a .sexp file.
+
+    Args:
+        effect_path: Relative path to the .sexp effect file
+        recipe_dir: Base directory for resolving paths
+
+    Returns:
+        Effect function or None if not a sexp effect
+    """
+    if not effect_path or not effect_path.endswith(".sexp"):
+        return None
+
+    try:
+        from ..sexp.effect_loader import SexpEffectLoader
+    except ImportError:
+        logger.warning("Sexp effect loader not available")
+        return None
+
+    try:
+        loader = SexpEffectLoader(recipe_dir or Path.cwd())
+        return loader.load_effect_path(effect_path)
+    except Exception as e:
+        logger.error(f"Failed to load sexp effect from {effect_path}: {e}")
+        return None
+
+
+def _get_python_primitive_effect(effect_name: str) -> Optional[EffectFn]:
+    """
+    Get a Python primitive frame processor effect.
+
+    Checks if the effect has a python_primitive in FFmpegCompiler.EFFECT_MAPPINGS
+    and wraps it for the executor API.
+    """
+    try:
+        from ..sexp.ffmpeg_compiler import FFmpegCompiler
+        from ..sexp.primitives import get_primitive
+        from ..effects.frame_processor import process_video
+    except ImportError:
+        return None
+
+    compiler = FFmpegCompiler()
+    primitive_name = compiler.has_python_primitive(effect_name)
+    if not primitive_name:
+        return None
+
+    primitive_fn = get_primitive(primitive_name)
+    if not primitive_fn:
+        logger.warning(f"Python primitive '{primitive_name}' not found for effect '{effect_name}'")
+        return None
+
+    def wrapped_effect(input_path: Path, output_path: Path, config: Dict[str, Any]) -> Path:
+        """Run Python primitive effect via frame processor."""
+        # Extract params (excluding internal keys)
+        params = {k: v for k, v in config.items()
+                  if k not in ("effect", "cid", "hash", "effect_path", "_binding")}
+
+        # Get bindings if present
+        bindings = {}
+        for key, value in config.items():
+            if isinstance(value, dict) and value.get("_resolved_values"):
+                bindings[key] = value["_resolved_values"]
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        actual_output = output_path.with_suffix(".mp4")
+
+        # Wrap primitive to match frame processor signature
+        def process_frame(frame, frame_params, state):
+            # Call primitive with frame and params
+            result = primitive_fn(frame, **frame_params)
+            return result, state
+
+        process_video(
+            input_path=input_path,
+            output_path=actual_output,
+            process_frame=process_frame,
+            params=params,
+            bindings=bindings,
+        )
+
+        logger.info(f"Processed effect '{effect_name}' via Python primitive '{primitive_name}'")
+        return actual_output
+
+    return wrapped_effect
+
+
+@register_executor("EFFECT")
+class EffectExecutor(Executor):
+    """
+    Apply an effect from the registry or IPFS.
+
+    Config:
+        effect: Name of the effect to apply
+        cid: IPFS CID for the effect (fetched from IPFS if not cached)
+        hash: Legacy alias for cid (backwards compatibility)
+        params: Optional parameters for the effect
+
+    Inputs:
+        Single input file to transform
+    """
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        effect_name = config.get("effect")
+        # Support both "cid" (new) and "hash" (legacy)
+        effect_cid = config.get("cid") or config.get("hash")
+
+        if not effect_name:
+            raise ValueError("EFFECT requires 'effect' config")
+
+        if len(inputs) != 1:
+            raise ValueError(f"EFFECT expects 1 input, got {len(inputs)}")
+
+        # Try IPFS effect first if CID provided
+        effect_fn: Optional[EffectFn] = None
+        if effect_cid:
+            effect_fn = _load_cached_effect(effect_cid)
+            if effect_fn:
+                logger.info(f"Running effect '{effect_name}' (cid={effect_cid[:16]}...)")
+
+        # Try sexp effect from effect_path (.sexp file)
+        if effect_fn is None:
+            effect_path = config.get("effect_path")
+            if effect_path and effect_path.endswith(".sexp"):
+                effect_fn = _get_sexp_effect(effect_path)
+                if effect_fn:
+                    logger.info(f"Running effect '{effect_name}' via sexp definition")
+
+        # Try Python primitive (from FFmpegCompiler.EFFECT_MAPPINGS)
+        if effect_fn is None:
+            effect_fn = _get_python_primitive_effect(effect_name)
+            if effect_fn:
+                logger.info(f"Running effect '{effect_name}' via Python primitive")
+
+        # Fall back to built-in effect
+        if effect_fn is None:
+            effect_fn = get_effect(effect_name)
+
+        if effect_fn is None:
+            raise ValueError(f"Unknown effect: {effect_name}")
+
+        # Pass full config (effect can extract what it needs)
+        return effect_fn(inputs[0], output_path, config)
+
+    def validate_config(self, config: Dict[str, Any]) -> List[str]:
+        errors = []
+        if "effect" not in config:
+            errors.append("EFFECT requires 'effect' config")
+        else:
+            # If CID provided, we'll load from IPFS - skip built-in check
+            has_cid = config.get("cid") or config.get("hash")
+            if not has_cid and get_effect(config["effect"]) is None:
+                errors.append(f"Unknown effect: {config['effect']}")
+        return errors

artdag/core/artdag/nodes/encoding.py

@@ -0,0 +1,50 @@
+# artdag/nodes/encoding.py
+"""
+Web-optimized video encoding settings.
+
+Provides common FFmpeg arguments for producing videos that:
+- Stream efficiently (faststart)
+- Play on all browsers (H.264 High profile)
+- Support seeking (regular keyframes)
+"""
+
+from typing import List
+
+# Standard web-optimized video encoding arguments
+WEB_VIDEO_ARGS: List[str] = [
+    "-c:v", "libx264",
+    "-preset", "fast",
+    "-crf", "18",
+    "-profile:v", "high",
+    "-level", "4.1",
+    "-pix_fmt", "yuv420p",  # Ensure broad compatibility
+    "-movflags", "+faststart",  # Enable streaming before full download
+    "-g", "48",  # Keyframe every ~2 seconds at 24fps (for seeking)
+]
+
+# Standard audio encoding arguments
+WEB_AUDIO_ARGS: List[str] = [
+    "-c:a", "aac",
+    "-b:a", "192k",
+]
+
+
+def get_web_encoding_args() -> List[str]:
+    """Get FFmpeg args for web-optimized video+audio encoding."""
+    return WEB_VIDEO_ARGS + WEB_AUDIO_ARGS
+
+
+def get_web_video_args() -> List[str]:
+    """Get FFmpeg args for web-optimized video encoding only."""
+    return WEB_VIDEO_ARGS.copy()
+
+
+def get_web_audio_args() -> List[str]:
+    """Get FFmpeg args for web-optimized audio encoding only."""
+    return WEB_AUDIO_ARGS.copy()
+
+
+# For shell commands (string format)
+WEB_VIDEO_ARGS_STR = " ".join(WEB_VIDEO_ARGS)
+WEB_AUDIO_ARGS_STR = " ".join(WEB_AUDIO_ARGS)
+WEB_ENCODING_ARGS_STR = f"{WEB_VIDEO_ARGS_STR} {WEB_AUDIO_ARGS_STR}"

artdag/core/artdag/nodes/source.py

@@ -0,0 +1,62 @@
+# primitive/nodes/source.py
+"""
+Source executors: Load media from paths.
+
+Primitives: SOURCE
+"""
+
+import logging
+import os
+import shutil
+from pathlib import Path
+from typing import Any, Dict, List
+
+from ..dag import NodeType
+from ..executor import Executor, register_executor
+
+logger = logging.getLogger(__name__)
+
+
+@register_executor(NodeType.SOURCE)
+class SourceExecutor(Executor):
+    """
+    Load source media from a path.
+
+    Config:
+        path: Path to source file
+
+    Creates a symlink to the source file for zero-copy loading.
+    """
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        source_path = Path(config["path"])
+
+        if not source_path.exists():
+            raise FileNotFoundError(f"Source file not found: {source_path}")
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        # Use symlink for zero-copy
+        if output_path.exists() or output_path.is_symlink():
+            output_path.unlink()
+
+        # Preserve extension from source
+        actual_output = output_path.with_suffix(source_path.suffix)
+        if actual_output.exists() or actual_output.is_symlink():
+            actual_output.unlink()
+
+        os.symlink(source_path.resolve(), actual_output)
+        logger.debug(f"SOURCE: {source_path.name} -> {actual_output}")
+
+        return actual_output
+
+    def validate_config(self, config: Dict[str, Any]) -> List[str]:
+        errors = []
+        if "path" not in config:
+            errors.append("SOURCE requires 'path' config")
+        return errors

artdag/core/artdag/nodes/transform.py

@@ -0,0 +1,224 @@
+# primitive/nodes/transform.py
+"""
+Transform executors: Modify single media inputs.
+
+Primitives: SEGMENT, RESIZE, TRANSFORM
+"""
+
+import logging
+import subprocess
+from pathlib import Path
+from typing import Any, Dict, List
+
+from ..dag import NodeType
+from ..executor import Executor, register_executor
+from .encoding import get_web_encoding_args, get_web_video_args
+
+logger = logging.getLogger(__name__)
+
+
+@register_executor(NodeType.SEGMENT)
+class SegmentExecutor(Executor):
+    """
+    Extract a time segment from media.
+
+    Config:
+        offset: Start time in seconds (default: 0)
+        duration: Duration in seconds (optional, default: to end)
+        precise: Use frame-accurate seeking (default: True)
+    """
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        if len(inputs) != 1:
+            raise ValueError("SEGMENT requires exactly one input")
+
+        input_path = inputs[0]
+        offset = config.get("offset", 0)
+        duration = config.get("duration")
+        precise = config.get("precise", True)
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        if precise:
+            # Frame-accurate: decode-seek (slower but precise)
+            cmd = ["ffmpeg", "-y", "-i", str(input_path)]
+            if offset > 0:
+                cmd.extend(["-ss", str(offset)])
+            if duration:
+                cmd.extend(["-t", str(duration)])
+            cmd.extend([*get_web_encoding_args(), str(output_path)])
+        else:
+            # Fast: input-seek at keyframes (may be slightly off)
+            cmd = ["ffmpeg", "-y"]
+            if offset > 0:
+                cmd.extend(["-ss", str(offset)])
+            cmd.extend(["-i", str(input_path)])
+            if duration:
+                cmd.extend(["-t", str(duration)])
+            cmd.extend(["-c", "copy", str(output_path)])
+
+        logger.debug(f"SEGMENT: offset={offset}, duration={duration}, precise={precise}")
+        result = subprocess.run(cmd, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            raise RuntimeError(f"Segment failed: {result.stderr}")
+
+        return output_path
+
+
+@register_executor(NodeType.RESIZE)
+class ResizeExecutor(Executor):
+    """
+    Resize media to target dimensions.
+
+    Config:
+        width: Target width
+        height: Target height
+        mode: "fit" (letterbox), "fill" (crop), "stretch", "pad"
+        background: Background color for pad mode (default: black)
+    """
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        if len(inputs) != 1:
+            raise ValueError("RESIZE requires exactly one input")
+
+        input_path = inputs[0]
+        width = config["width"]
+        height = config["height"]
+        mode = config.get("mode", "fit")
+        background = config.get("background", "black")
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        if mode == "fit":
+            # Scale to fit, add letterboxing
+            vf = f"scale={width}:{height}:force_original_aspect_ratio=decrease,pad={width}:{height}:(ow-iw)/2:(oh-ih)/2:color={background}"
+        elif mode == "fill":
+            # Scale to fill, crop excess
+            vf = f"scale={width}:{height}:force_original_aspect_ratio=increase,crop={width}:{height}"
+        elif mode == "stretch":
+            # Stretch to exact size
+            vf = f"scale={width}:{height}"
+        elif mode == "pad":
+            # Scale down only if larger, then pad
+            vf = f"scale='min({width},iw)':'min({height},ih)':force_original_aspect_ratio=decrease,pad={width}:{height}:(ow-iw)/2:(oh-ih)/2:color={background}"
+        else:
+            raise ValueError(f"Unknown resize mode: {mode}")
+
+        cmd = [
+            "ffmpeg", "-y",
+            "-i", str(input_path),
+            "-vf", vf,
+            *get_web_video_args(),
+            "-c:a", "copy",
+            str(output_path)
+        ]
+
+        logger.debug(f"RESIZE: {width}x{height} ({mode}) (web-optimized)")
+        result = subprocess.run(cmd, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            raise RuntimeError(f"Resize failed: {result.stderr}")
+
+        return output_path
+
+
+@register_executor(NodeType.TRANSFORM)
+class TransformExecutor(Executor):
+    """
+    Apply visual effects to media.
+
+    Config:
+        effects: Dict of effect -> value
+            saturation: 0.0-2.0 (1.0 = normal)
+            contrast: 0.0-2.0 (1.0 = normal)
+            brightness: -1.0 to 1.0 (0.0 = normal)
+            gamma: 0.1-10.0 (1.0 = normal)
+            hue: degrees shift
+            blur: blur radius
+            sharpen: sharpen amount
+            speed: playback speed multiplier
+    """
+
+    def execute(
+        self,
+        config: Dict[str, Any],
+        inputs: List[Path],
+        output_path: Path,
+    ) -> Path:
+        if len(inputs) != 1:
+            raise ValueError("TRANSFORM requires exactly one input")
+
+        input_path = inputs[0]
+        effects = config.get("effects", {})
+
+        if not effects:
+            # No effects - just copy
+            import shutil
+            output_path.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy2(input_path, output_path)
+            return output_path
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+
+        # Build filter chain
+        vf_parts = []
+        af_parts = []
+
+        # Color adjustments via eq filter
+        eq_parts = []
+        if "saturation" in effects:
+            eq_parts.append(f"saturation={effects['saturation']}")
+        if "contrast" in effects:
+            eq_parts.append(f"contrast={effects['contrast']}")
+        if "brightness" in effects:
+            eq_parts.append(f"brightness={effects['brightness']}")
+        if "gamma" in effects:
+            eq_parts.append(f"gamma={effects['gamma']}")
+        if eq_parts:
+            vf_parts.append(f"eq={':'.join(eq_parts)}")
+
+        # Hue adjustment
+        if "hue" in effects:
+            vf_parts.append(f"hue=h={effects['hue']}")
+
+        # Blur
+        if "blur" in effects:
+            vf_parts.append(f"boxblur={effects['blur']}")
+
+        # Sharpen
+        if "sharpen" in effects:
+            vf_parts.append(f"unsharp=5:5:{effects['sharpen']}:5:5:0")
+
+        # Speed change
+        if "speed" in effects:
+            speed = effects["speed"]
+            vf_parts.append(f"setpts={1/speed}*PTS")
+            af_parts.append(f"atempo={speed}")
+
+        cmd = ["ffmpeg", "-y", "-i", str(input_path)]
+
+        if vf_parts:
+            cmd.extend(["-vf", ",".join(vf_parts)])
+        if af_parts:
+            cmd.extend(["-af", ",".join(af_parts)])
+
+        cmd.extend([*get_web_encoding_args(), str(output_path)])
+
+        logger.debug(f"TRANSFORM: {list(effects.keys())} (web-optimized)")
+        result = subprocess.run(cmd, capture_output=True, text=True)
+
+        if result.returncode != 0:
+            raise RuntimeError(f"Transform failed: {result.stderr}")
+
+        return output_path

artdag/core/artdag/planning/__init__.py

@@ -0,0 +1,29 @@
+# artdag/planning - Execution plan generation
+#
+# Provides the Planning phase of the 3-phase execution model:
+# 1. ANALYZE - Extract features from inputs
+# 2. PLAN - Generate execution plan with cache IDs
+# 3. EXECUTE - Run steps with caching
+
+from .schema import (
+    ExecutionStep,
+    ExecutionPlan,
+    StepStatus,
+    StepOutput,
+    StepInput,
+    PlanInput,
+)
+from .planner import RecipePlanner, Recipe
+from .tree_reduction import TreeReducer
+
+__all__ = [
+    "ExecutionStep",
+    "ExecutionPlan",
+    "StepStatus",
+    "StepOutput",
+    "StepInput",
+    "PlanInput",
+    "RecipePlanner",
+    "Recipe",
+    "TreeReducer",
+]

artdag/core/artdag/planning/planner.py

@@ -0,0 +1,756 @@
+# artdag/planning/planner.py
+"""
+Recipe planner - converts recipes into execution plans.
+
+The planner is the second phase of the 3-phase execution model.
+It takes a recipe and analysis results and generates a complete
+execution plan with pre-computed cache IDs.
+"""
+
+import hashlib
+import json
+import logging
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple
+
+import yaml
+
+from .schema import ExecutionPlan, ExecutionStep, StepOutput, StepInput, PlanInput
+from .tree_reduction import TreeReducer, reduce_sequence
+from ..analysis import AnalysisResult
+
+
+def _infer_media_type(node_type: str, config: Dict[str, Any] = None) -> str:
+    """Infer media type from node type and config."""
+    config = config or {}
+
+    # Audio operations
+    if node_type in ("AUDIO", "MIX_AUDIO", "EXTRACT_AUDIO"):
+        return "audio/wav"
+    if "audio" in node_type.lower():
+        return "audio/wav"
+
+    # Image operations
+    if node_type in ("FRAME", "THUMBNAIL", "IMAGE"):
+        return "image/png"
+
+    # Default to video
+    return "video/mp4"
+
+logger = logging.getLogger(__name__)
+
+
+def _stable_hash(data: Any, algorithm: str = "sha3_256") -> str:
+    """Create stable hash from arbitrary data."""
+    json_str = json.dumps(data, sort_keys=True, separators=(",", ":"))
+    hasher = hashlib.new(algorithm)
+    hasher.update(json_str.encode())
+    return hasher.hexdigest()
+
+
+@dataclass
+class RecipeNode:
+    """A node in the recipe DAG."""
+    id: str
+    type: str
+    config: Dict[str, Any]
+    inputs: List[str]
+
+
+@dataclass
+class Recipe:
+    """Parsed recipe structure."""
+    name: str
+    version: str
+    description: str
+    nodes: List[RecipeNode]
+    output: str
+    registry: Dict[str, Any]
+    owner: str
+    raw_yaml: str
+
+    @property
+    def recipe_hash(self) -> str:
+        """Compute hash of recipe content."""
+        return _stable_hash({"yaml": self.raw_yaml})
+
+    @classmethod
+    def from_yaml(cls, yaml_content: str) -> "Recipe":
+        """Parse recipe from YAML string."""
+        data = yaml.safe_load(yaml_content)
+
+        nodes = []
+        for node_data in data.get("dag", {}).get("nodes", []):
+            # Handle both 'inputs' as list and 'inputs' as dict
+            inputs = node_data.get("inputs", [])
+            if isinstance(inputs, dict):
+                # Extract input references from dict structure
+                input_list = []
+                for key, value in inputs.items():
+                    if isinstance(value, str):
+                        input_list.append(value)
+                    elif isinstance(value, list):
+                        input_list.extend(value)
+                inputs = input_list
+            elif isinstance(inputs, str):
+                inputs = [inputs]
+
+            nodes.append(RecipeNode(
+                id=node_data["id"],
+                type=node_data["type"],
+                config=node_data.get("config", {}),
+                inputs=inputs,
+            ))
+
+        return cls(
+            name=data.get("name", "unnamed"),
+            version=data.get("version", "1.0"),
+            description=data.get("description", ""),
+            nodes=nodes,
+            output=data.get("dag", {}).get("output", ""),
+            registry=data.get("registry", {}),
+            owner=data.get("owner", ""),
+            raw_yaml=yaml_content,
+        )
+
+    @classmethod
+    def from_file(cls, path: Path) -> "Recipe":
+        """Load recipe from YAML file."""
+        with open(path, "r") as f:
+            return cls.from_yaml(f.read())
+
+
+class RecipePlanner:
+    """
+    Generates execution plans from recipes.
+
+    The planner:
+    1. Parses the recipe
+    2. Resolves fixed inputs from registry
+    3. Maps variable inputs to provided hashes
+    4. Expands MAP/iteration nodes
+    5. Applies tree reduction for SEQUENCE nodes
+    6. Computes cache IDs for all steps
+    """
+
+    def __init__(self, use_tree_reduction: bool = True):
+        """
+        Initialize the planner.
+
+        Args:
+            use_tree_reduction: Whether to use tree reduction for SEQUENCE
+        """
+        self.use_tree_reduction = use_tree_reduction
+
+    def plan(
+        self,
+        recipe: Recipe,
+        input_hashes: Dict[str, str],
+        analysis: Optional[Dict[str, AnalysisResult]] = None,
+        seed: Optional[int] = None,
+    ) -> ExecutionPlan:
+        """
+        Generate an execution plan from a recipe.
+
+        Args:
+            recipe: The parsed recipe
+            input_hashes: Mapping from input name to content hash
+            analysis: Analysis results for inputs (keyed by hash)
+            seed: Random seed for deterministic planning
+
+        Returns:
+            ExecutionPlan with pre-computed cache IDs
+        """
+        logger.info(f"Planning recipe: {recipe.name}")
+
+        # Build node lookup
+        nodes_by_id = {n.id: n for n in recipe.nodes}
+
+        # Topologically sort nodes
+        sorted_ids = self._topological_sort(recipe.nodes)
+
+        # Resolve registry references
+        registry_hashes = self._resolve_registry(recipe.registry)
+
+        # Build PlanInput objects from input_hashes
+        plan_inputs = []
+        for name, cid in input_hashes.items():
+            # Try to find matching SOURCE node for media type
+            media_type = "application/octet-stream"
+            for node in recipe.nodes:
+                if node.id == name and node.type == "SOURCE":
+                    media_type = _infer_media_type("SOURCE", node.config)
+                    break
+
+            plan_inputs.append(PlanInput(
+                name=name,
+                cache_id=cid,
+                cid=cid,
+                media_type=media_type,
+            ))
+
+        # Generate steps
+        steps = []
+        step_id_map = {}  # Maps recipe node ID to step ID(s)
+        step_name_map = {}  # Maps recipe node ID to human-readable name
+        analysis_cache_ids = {}
+
+        for node_id in sorted_ids:
+            node = nodes_by_id[node_id]
+            logger.debug(f"Processing node: {node.id} ({node.type})")
+
+            new_steps, output_step_id = self._process_node(
+                node=node,
+                step_id_map=step_id_map,
+                step_name_map=step_name_map,
+                input_hashes=input_hashes,
+                registry_hashes=registry_hashes,
+                analysis=analysis or {},
+                recipe_name=recipe.name,
+            )
+
+            steps.extend(new_steps)
+            step_id_map[node_id] = output_step_id
+            # Track human-readable name for this node
+            if new_steps:
+                step_name_map[node_id] = new_steps[-1].name
+
+        # Find output step
+        output_step = step_id_map.get(recipe.output)
+        if not output_step:
+            raise ValueError(f"Output node '{recipe.output}' not found")
+
+        # Determine output name
+        output_name = f"{recipe.name}.output"
+        output_step_obj = next((s for s in steps if s.step_id == output_step), None)
+        if output_step_obj and output_step_obj.outputs:
+            output_name = output_step_obj.outputs[0].name
+
+        # Build analysis cache IDs
+        if analysis:
+            analysis_cache_ids = {
+                h: a.cache_id for h, a in analysis.items()
+                if a.cache_id
+            }
+
+        # Create plan
+        plan = ExecutionPlan(
+            plan_id=None,  # Computed in __post_init__
+            name=f"{recipe.name}_plan",
+            recipe_id=recipe.name,
+            recipe_name=recipe.name,
+            recipe_hash=recipe.recipe_hash,
+            seed=seed,
+            inputs=plan_inputs,
+            steps=steps,
+            output_step=output_step,
+            output_name=output_name,
+            analysis_cache_ids=analysis_cache_ids,
+            input_hashes=input_hashes,
+            metadata={
+                "recipe_version": recipe.version,
+                "recipe_description": recipe.description,
+                "owner": recipe.owner,
+            },
+        )
+
+        # Compute all cache IDs and then generate outputs
+        plan.compute_all_cache_ids()
+        plan.compute_levels()
+
+        # Now add outputs to each step (needs cache_id to be computed first)
+        self._add_step_outputs(plan, recipe.name)
+
+        # Recompute plan_id after outputs are added
+        plan.plan_id = plan._compute_plan_id()
+
+        logger.info(f"Generated plan with {len(steps)} steps")
+        return plan
+
+    def _add_step_outputs(self, plan: ExecutionPlan, recipe_name: str) -> None:
+        """Add output definitions to each step after cache_ids are computed."""
+        for step in plan.steps:
+            if step.outputs:
+                continue  # Already has outputs
+
+            # Generate output name from step name
+            base_name = step.name or step.step_id
+            output_name = f"{recipe_name}.{base_name}.out"
+
+            media_type = _infer_media_type(step.node_type, step.config)
+
+            step.add_output(
+                name=output_name,
+                media_type=media_type,
+                index=0,
+                metadata={},
+            )
+
+    def plan_from_yaml(
+        self,
+        yaml_content: str,
+        input_hashes: Dict[str, str],
+        analysis: Optional[Dict[str, AnalysisResult]] = None,
+    ) -> ExecutionPlan:
+        """
+        Generate plan from YAML string.
+
+        Args:
+            yaml_content: Recipe YAML content
+            input_hashes: Mapping from input name to content hash
+            analysis: Analysis results
+
+        Returns:
+            ExecutionPlan
+        """
+        recipe = Recipe.from_yaml(yaml_content)
+        return self.plan(recipe, input_hashes, analysis)
+
+    def plan_from_file(
+        self,
+        recipe_path: Path,
+        input_hashes: Dict[str, str],
+        analysis: Optional[Dict[str, AnalysisResult]] = None,
+    ) -> ExecutionPlan:
+        """
+        Generate plan from recipe file.
+
+        Args:
+            recipe_path: Path to recipe YAML file
+            input_hashes: Mapping from input name to content hash
+            analysis: Analysis results
+
+        Returns:
+            ExecutionPlan
+        """
+        recipe = Recipe.from_file(recipe_path)
+        return self.plan(recipe, input_hashes, analysis)
+
+    def _topological_sort(self, nodes: List[RecipeNode]) -> List[str]:
+        """Topologically sort recipe nodes."""
+        nodes_by_id = {n.id: n for n in nodes}
+        visited = set()
+        order = []
+
+        def visit(node_id: str):
+            if node_id in visited:
+                return
+            if node_id not in nodes_by_id:
+                return  # External input
+            visited.add(node_id)
+            node = nodes_by_id[node_id]
+            for input_id in node.inputs:
+                visit(input_id)
+            order.append(node_id)
+
+        for node in nodes:
+            visit(node.id)
+
+        return order
+
+    def _resolve_registry(self, registry: Dict[str, Any]) -> Dict[str, str]:
+        """
+        Resolve registry references to content hashes.
+
+        Args:
+            registry: Registry section from recipe
+
+        Returns:
+            Mapping from name to content hash
+        """
+        hashes = {}
+
+        # Assets
+        for name, asset_data in registry.get("assets", {}).items():
+            if isinstance(asset_data, dict) and "hash" in asset_data:
+                hashes[name] = asset_data["hash"]
+            elif isinstance(asset_data, str):
+                hashes[name] = asset_data
+
+        # Effects
+        for name, effect_data in registry.get("effects", {}).items():
+            if isinstance(effect_data, dict) and "hash" in effect_data:
+                hashes[f"effect:{name}"] = effect_data["hash"]
+            elif isinstance(effect_data, str):
+                hashes[f"effect:{name}"] = effect_data
+
+        return hashes
+
+    def _process_node(
+        self,
+        node: RecipeNode,
+        step_id_map: Dict[str, str],
+        step_name_map: Dict[str, str],
+        input_hashes: Dict[str, str],
+        registry_hashes: Dict[str, str],
+        analysis: Dict[str, AnalysisResult],
+        recipe_name: str = "",
+    ) -> Tuple[List[ExecutionStep], str]:
+        """
+        Process a recipe node into execution steps.
+
+        Args:
+            node: Recipe node to process
+            step_id_map: Mapping from processed node IDs to step IDs
+            step_name_map: Mapping from node IDs to human-readable names
+            input_hashes: User-provided input hashes
+            registry_hashes: Registry-resolved hashes
+            analysis: Analysis results
+            recipe_name: Name of the recipe (for generating readable names)
+
+        Returns:
+            Tuple of (new steps, output step ID)
+        """
+        # SOURCE nodes
+        if node.type == "SOURCE":
+            return self._process_source(node, input_hashes, registry_hashes, recipe_name)
+
+        # SOURCE_LIST nodes
+        if node.type == "SOURCE_LIST":
+            return self._process_source_list(node, input_hashes, recipe_name)
+
+        # ANALYZE nodes
+        if node.type == "ANALYZE":
+            return self._process_analyze(node, step_id_map, analysis, recipe_name)
+
+        # MAP nodes
+        if node.type == "MAP":
+            return self._process_map(node, step_id_map, input_hashes, analysis, recipe_name)
+
+        # SEQUENCE nodes (may use tree reduction)
+        if node.type == "SEQUENCE":
+            return self._process_sequence(node, step_id_map, recipe_name)
+
+        # SEGMENT_AT nodes
+        if node.type == "SEGMENT_AT":
+            return self._process_segment_at(node, step_id_map, analysis, recipe_name)
+
+        # Standard nodes (SEGMENT, RESIZE, TRANSFORM, LAYER, MUX, BLEND, etc.)
+        return self._process_standard(node, step_id_map, recipe_name)
+
+    def _process_source(
+        self,
+        node: RecipeNode,
+        input_hashes: Dict[str, str],
+        registry_hashes: Dict[str, str],
+        recipe_name: str = "",
+    ) -> Tuple[List[ExecutionStep], str]:
+        """Process SOURCE node."""
+        config = dict(node.config)
+
+        # Variable input?
+        if config.get("input"):
+            # Look up in user-provided inputs
+            if node.id not in input_hashes:
+                raise ValueError(f"Missing input for SOURCE node '{node.id}'")
+            cid = input_hashes[node.id]
+        # Fixed asset from registry?
+        elif config.get("asset"):
+            asset_name = config["asset"]
+            if asset_name not in registry_hashes:
+                raise ValueError(f"Asset '{asset_name}' not found in registry")
+            cid = registry_hashes[asset_name]
+        else:
+            raise ValueError(f"SOURCE node '{node.id}' has no input or asset")
+
+        # Human-readable name
+        display_name = config.get("name", node.id)
+        step_name = f"{recipe_name}.inputs.{display_name}" if recipe_name else display_name
+
+        step = ExecutionStep(
+            step_id=node.id,
+            node_type="SOURCE",
+            config={"input_ref": node.id, "cid": cid},
+            input_steps=[],
+            cache_id=cid,  # SOURCE cache_id is just the content hash
+            name=step_name,
+        )
+
+        return [step], step.step_id
+
+    def _process_source_list(
+        self,
+        node: RecipeNode,
+        input_hashes: Dict[str, str],
+        recipe_name: str = "",
+    ) -> Tuple[List[ExecutionStep], str]:
+        """
+        Process SOURCE_LIST node.
+
+        Creates individual SOURCE steps for each item in the list.
+        """
+        # Look for list input
+        if node.id not in input_hashes:
+            raise ValueError(f"Missing input for SOURCE_LIST node '{node.id}'")
+
+        input_value = input_hashes[node.id]
+
+        # Parse as comma-separated list if string
+        if isinstance(input_value, str):
+            items = [h.strip() for h in input_value.split(",")]
+        else:
+            items = list(input_value)
+
+        display_name = node.config.get("name", node.id)
+        base_name = f"{recipe_name}.{display_name}" if recipe_name else display_name
+
+        steps = []
+        for i, cid in enumerate(items):
+            step = ExecutionStep(
+                step_id=f"{node.id}_{i}",
+                node_type="SOURCE",
+                config={"input_ref": f"{node.id}[{i}]", "cid": cid},
+                input_steps=[],
+                cache_id=cid,
+                name=f"{base_name}[{i}]",
+            )
+            steps.append(step)
+
+        # Return list marker as output
+        list_step = ExecutionStep(
+            step_id=node.id,
+            node_type="_LIST",
+            config={"items": [s.step_id for s in steps]},
+            input_steps=[s.step_id for s in steps],
+            name=f"{base_name}.list",
+        )
+        steps.append(list_step)
+
+        return steps, list_step.step_id
+
+    def _process_analyze(
+        self,
+        node: RecipeNode,
+        step_id_map: Dict[str, str],
+        analysis: Dict[str, AnalysisResult],
+        recipe_name: str = "",
+    ) -> Tuple[List[ExecutionStep], str]:
+        """
+        Process ANALYZE node.
+
+        ANALYZE nodes reference pre-computed analysis results.
+        """
+        input_step = step_id_map.get(node.inputs[0]) if node.inputs else None
+        if not input_step:
+            raise ValueError(f"ANALYZE node '{node.id}' has no input")
+
+        feature = node.config.get("feature", "all")
+        step_name = f"{recipe_name}.analysis.{feature}" if recipe_name else f"analysis.{feature}"
+
+        step = ExecutionStep(
+            step_id=node.id,
+            node_type="ANALYZE",
+            config={
+                "feature": feature,
+                **node.config,
+            },
+            input_steps=[input_step],
+            name=step_name,
+        )
+
+        return [step], step.step_id
+
+    def _process_map(
+        self,
+        node: RecipeNode,
+        step_id_map: Dict[str, str],
+        input_hashes: Dict[str, str],
+        analysis: Dict[str, AnalysisResult],
+        recipe_name: str = "",
+    ) -> Tuple[List[ExecutionStep], str]:
+        """
+        Process MAP node - expand iteration over list.
+
+        MAP applies an operation to each item in a list.
+        """
+        operation = node.config.get("operation", "TRANSFORM")
+        base_name = f"{recipe_name}.{node.id}" if recipe_name else node.id
+
+        # Get items input
+        items_ref = node.config.get("items") or (
+            node.inputs[0] if isinstance(node.inputs, list) else
+            node.inputs.get("items") if isinstance(node.inputs, dict) else None
+        )
+
+        if not items_ref:
+            raise ValueError(f"MAP node '{node.id}' has no items input")
+
+        # Resolve items to list of step IDs
+        if items_ref in step_id_map:
+            # Reference to SOURCE_LIST output
+            items_step = step_id_map[items_ref]
+            # TODO: expand list items
+            logger.warning(f"MAP node '{node.id}' references list step, expansion TBD")
+            item_steps = [items_step]
+        else:
+            item_steps = [items_ref]
+
+        # Generate step for each item
+        steps = []
+        output_steps = []
+
+        for i, item_step in enumerate(item_steps):
+            step_id = f"{node.id}_{i}"
+
+            if operation == "RANDOM_SLICE":
+                step = ExecutionStep(
+                    step_id=step_id,
+                    node_type="SEGMENT",
+                    config={
+                        "random": True,
+                        "seed_from": node.config.get("seed_from"),
+                        "index": i,
+                    },
+                    input_steps=[item_step],
+                    name=f"{base_name}.slice[{i}]",
+                )
+            elif operation == "TRANSFORM":
+                step = ExecutionStep(
+                    step_id=step_id,
+                    node_type="TRANSFORM",
+                    config=node.config.get("effects", {}),
+                    input_steps=[item_step],
+                    name=f"{base_name}.transform[{i}]",
+                )
+            elif operation == "ANALYZE":
+                step = ExecutionStep(
+                    step_id=step_id,
+                    node_type="ANALYZE",
+                    config={"feature": node.config.get("feature", "all")},
+                    input_steps=[item_step],
+                    name=f"{base_name}.analyze[{i}]",
+                )
+            else:
+                step = ExecutionStep(
+                    step_id=step_id,
+                    node_type=operation,
+                    config=node.config,
+                    input_steps=[item_step],
+                    name=f"{base_name}.{operation.lower()}[{i}]",
+                )
+
+            steps.append(step)
+            output_steps.append(step_id)
+
+        # Create list output
+        list_step = ExecutionStep(
+            step_id=node.id,
+            node_type="_LIST",
+            config={"items": output_steps},
+            input_steps=output_steps,
+            name=f"{base_name}.results",
+        )
+        steps.append(list_step)
+
+        return steps, list_step.step_id
+
+    def _process_sequence(
+        self,
+        node: RecipeNode,
+        step_id_map: Dict[str, str],
+        recipe_name: str = "",
+    ) -> Tuple[List[ExecutionStep], str]:
+        """
+        Process SEQUENCE node.
+
+        Uses tree reduction for parallel composition if enabled.
+        """
+        base_name = f"{recipe_name}.{node.id}" if recipe_name else node.id
+
+        # Resolve input steps
+        input_steps = []
+        for input_id in node.inputs:
+            if input_id in step_id_map:
+                input_steps.append(step_id_map[input_id])
+            else:
+                input_steps.append(input_id)
+
+        if len(input_steps) == 0:
+            raise ValueError(f"SEQUENCE node '{node.id}' has no inputs")
+
+        if len(input_steps) == 1:
+            # Single input, no sequence needed
+            return [], input_steps[0]
+
+        transition_config = node.config.get("transition", {"type": "cut"})
+        config = {"transition": transition_config}
+
+        if self.use_tree_reduction and len(input_steps) > 2:
+            # Use tree reduction
+            reduction_steps, output_id = reduce_sequence(
+                input_steps,
+                transition_config=config,
+                id_prefix=node.id,
+            )
+
+            steps = []
+            for i, (step_id, inputs, step_config) in enumerate(reduction_steps):
+                step = ExecutionStep(
+                    step_id=step_id,
+                    node_type="SEQUENCE",
+                    config=step_config,
+                    input_steps=inputs,
+                    name=f"{base_name}.reduce[{i}]",
+                )
+                steps.append(step)
+
+            return steps, output_id
+        else:
+            # Direct sequence
+            step = ExecutionStep(
+                step_id=node.id,
+                node_type="SEQUENCE",
+                config=config,
+                input_steps=input_steps,
+                name=f"{base_name}.concat",
+            )
+            return [step], step.step_id
+
+    def _process_segment_at(
+        self,
+        node: RecipeNode,
+        step_id_map: Dict[str, str],
+        analysis: Dict[str, AnalysisResult],
+        recipe_name: str = "",
+    ) -> Tuple[List[ExecutionStep], str]:
+        """
+        Process SEGMENT_AT node - cut at specific times.
+
+        Creates SEGMENT steps for each time range.
+        """
+        base_name = f"{recipe_name}.{node.id}" if recipe_name else node.id
+        times_from = node.config.get("times_from")
+        distribute = node.config.get("distribute", "round_robin")
+
+        # TODO: Resolve times from analysis
+        # For now, create a placeholder
+        step = ExecutionStep(
+            step_id=node.id,
+            node_type="SEGMENT_AT",
+            config=node.config,
+            input_steps=[step_id_map.get(i, i) for i in node.inputs],
+            name=f"{base_name}.segment",
+        )
+
+        return [step], step.step_id
+
+    def _process_standard(
+        self,
+        node: RecipeNode,
+        step_id_map: Dict[str, str],
+        recipe_name: str = "",
+    ) -> Tuple[List[ExecutionStep], str]:
+        """Process standard transformation/composition node."""
+        base_name = f"{recipe_name}.{node.id}" if recipe_name else node.id
+        input_steps = [step_id_map.get(i, i) for i in node.inputs]
+
+        step = ExecutionStep(
+            step_id=node.id,
+            node_type=node.type,
+            config=node.config,
+            input_steps=input_steps,
+            name=f"{base_name}.{node.type.lower()}",
+        )
+
+        return [step], step.step_id

artdag/core/artdag/planning/schema.py

@@ -0,0 +1,594 @@
+# artdag/planning/schema.py
+"""
+Data structures for execution plans.
+
+An ExecutionPlan contains all steps needed to execute a recipe,
+with pre-computed cache IDs for each step.
+"""
+
+import hashlib
+import json
+import os
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from enum import Enum
+from typing import Any, Dict, List, Optional
+
+
+# Cluster key for trust domains
+# Systems with the same key produce the same cache_ids and can share work
+# Systems with different keys have isolated cache namespaces
+CLUSTER_KEY: Optional[str] = os.environ.get("ARTDAG_CLUSTER_KEY")
+
+
+def set_cluster_key(key: Optional[str]) -> None:
+    """Set the cluster key programmatically."""
+    global CLUSTER_KEY
+    CLUSTER_KEY = key
+
+
+def get_cluster_key() -> Optional[str]:
+    """Get the current cluster key."""
+    return CLUSTER_KEY
+
+
+def _stable_hash(data: Any, algorithm: str = "sha3_256") -> str:
+    """
+    Create stable hash from arbitrary data.
+
+    If ARTDAG_CLUSTER_KEY is set, it's mixed into the hash to create
+    isolated trust domains. Systems with the same key can share work;
+    systems with different keys have separate cache namespaces.
+    """
+    # Mix in cluster key if set
+    if CLUSTER_KEY:
+        data = {"_cluster_key": CLUSTER_KEY, "_data": data}
+
+    json_str = json.dumps(data, sort_keys=True, separators=(",", ":"))
+    hasher = hashlib.new(algorithm)
+    hasher.update(json_str.encode())
+    return hasher.hexdigest()
+
+
+class StepStatus(Enum):
+    """Status of an execution step."""
+    PENDING = "pending"
+    CLAIMED = "claimed"
+    RUNNING = "running"
+    COMPLETED = "completed"
+    CACHED = "cached"
+    FAILED = "failed"
+    SKIPPED = "skipped"
+
+
+@dataclass
+class StepOutput:
+    """
+    A single output from an execution step.
+
+    Nodes may produce multiple outputs (e.g., split_on_beats produces N segments).
+    Each output has a human-readable name and a cache_id for storage.
+
+    Attributes:
+        name: Human-readable name (e.g., "beats.split.segment[0]")
+        cache_id: Content-addressed hash for caching
+        media_type: MIME type of the output (e.g., "video/mp4", "audio/wav")
+        index: Output index for multi-output nodes
+        metadata: Optional additional metadata (time_range, etc.)
+    """
+    name: str
+    cache_id: str
+    media_type: str = "application/octet-stream"
+    index: int = 0
+    metadata: Dict[str, Any] = field(default_factory=dict)
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "name": self.name,
+            "cache_id": self.cache_id,
+            "media_type": self.media_type,
+            "index": self.index,
+            "metadata": self.metadata,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "StepOutput":
+        return cls(
+            name=data["name"],
+            cache_id=data["cache_id"],
+            media_type=data.get("media_type", "application/octet-stream"),
+            index=data.get("index", 0),
+            metadata=data.get("metadata", {}),
+        )
+
+
+@dataclass
+class StepInput:
+    """
+    Reference to an input for a step.
+
+    Inputs can reference outputs from other steps by name.
+
+    Attributes:
+        name: Input slot name (e.g., "video", "audio", "segments")
+        source: Source output name (e.g., "beats.split.segment[0]")
+        cache_id: Resolved cache_id of the source (populated during planning)
+    """
+    name: str
+    source: str
+    cache_id: Optional[str] = None
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "name": self.name,
+            "source": self.source,
+            "cache_id": self.cache_id,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "StepInput":
+        return cls(
+            name=data["name"],
+            source=data["source"],
+            cache_id=data.get("cache_id"),
+        )
+
+
+@dataclass
+class ExecutionStep:
+    """
+    A single step in the execution plan.
+
+    Each step has a pre-computed cache_id that uniquely identifies
+    its output based on its configuration and input cache_ids.
+
+    Steps can produce multiple outputs (e.g., split_on_beats produces N segments).
+    Each output has its own cache_id derived from the step's cache_id + index.
+
+    Attributes:
+        name: Human-readable name relating to recipe (e.g., "beats.split")
+        step_id: Unique identifier (hash) for this step
+        node_type: The primitive type (SOURCE, SEQUENCE, TRANSFORM, etc.)
+        config: Configuration for the primitive
+        input_steps: IDs of steps this depends on (legacy, use inputs for new code)
+        inputs: Structured input references with names and sources
+        cache_id: Pre-computed cache ID (hash of config + input cache_ids)
+        outputs: List of outputs this step produces
+        estimated_duration: Optional estimated execution time
+        level: Dependency level (0 = no dependencies, higher = more deps)
+    """
+    step_id: str
+    node_type: str
+    config: Dict[str, Any]
+    input_steps: List[str] = field(default_factory=list)
+    inputs: List[StepInput] = field(default_factory=list)
+    cache_id: Optional[str] = None
+    outputs: List[StepOutput] = field(default_factory=list)
+    name: Optional[str] = None
+    estimated_duration: Optional[float] = None
+    level: int = 0
+
+    def compute_cache_id(self, input_cache_ids: Dict[str, str]) -> str:
+        """
+        Compute cache ID from configuration and input cache IDs.
+
+        cache_id = SHA3-256(node_type + config + sorted(input_cache_ids))
+
+        Args:
+            input_cache_ids: Mapping from input step_id/name to their cache_id
+
+        Returns:
+            The computed cache_id
+        """
+        # Use structured inputs if available, otherwise fall back to input_steps
+        if self.inputs:
+            resolved_inputs = [
+                inp.cache_id or input_cache_ids.get(inp.source, inp.source)
+                for inp in sorted(self.inputs, key=lambda x: x.name)
+            ]
+        else:
+            resolved_inputs = [input_cache_ids.get(s, s) for s in sorted(self.input_steps)]
+
+        content = {
+            "node_type": self.node_type,
+            "config": self.config,
+            "inputs": resolved_inputs,
+        }
+        self.cache_id = _stable_hash(content)
+        return self.cache_id
+
+    def compute_output_cache_id(self, index: int) -> str:
+        """
+        Compute cache ID for a specific output index.
+
+        output_cache_id = SHA3-256(step_cache_id + index)
+
+        Args:
+            index: The output index
+
+        Returns:
+            Cache ID for that output
+        """
+        if not self.cache_id:
+            raise ValueError("Step cache_id must be computed first")
+        content = {"step_cache_id": self.cache_id, "output_index": index}
+        return _stable_hash(content)
+
+    def add_output(
+        self,
+        name: str,
+        media_type: str = "application/octet-stream",
+        index: Optional[int] = None,
+        metadata: Optional[Dict[str, Any]] = None,
+    ) -> StepOutput:
+        """
+        Add an output to this step.
+
+        Args:
+            name: Human-readable output name
+            media_type: MIME type of the output
+            index: Output index (defaults to next available)
+            metadata: Optional metadata
+
+        Returns:
+            The created StepOutput
+        """
+        if index is None:
+            index = len(self.outputs)
+
+        cache_id = self.compute_output_cache_id(index)
+        output = StepOutput(
+            name=name,
+            cache_id=cache_id,
+            media_type=media_type,
+            index=index,
+            metadata=metadata or {},
+        )
+        self.outputs.append(output)
+        return output
+
+    def get_output(self, index: int = 0) -> Optional[StepOutput]:
+        """Get output by index."""
+        if index < len(self.outputs):
+            return self.outputs[index]
+        return None
+
+    def get_output_by_name(self, name: str) -> Optional[StepOutput]:
+        """Get output by name."""
+        for output in self.outputs:
+            if output.name == name:
+                return output
+        return None
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "step_id": self.step_id,
+            "name": self.name,
+            "node_type": self.node_type,
+            "config": self.config,
+            "input_steps": self.input_steps,
+            "inputs": [inp.to_dict() for inp in self.inputs],
+            "cache_id": self.cache_id,
+            "outputs": [out.to_dict() for out in self.outputs],
+            "estimated_duration": self.estimated_duration,
+            "level": self.level,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "ExecutionStep":
+        inputs = [StepInput.from_dict(i) for i in data.get("inputs", [])]
+        outputs = [StepOutput.from_dict(o) for o in data.get("outputs", [])]
+        return cls(
+            step_id=data["step_id"],
+            node_type=data["node_type"],
+            config=data.get("config", {}),
+            input_steps=data.get("input_steps", []),
+            inputs=inputs,
+            cache_id=data.get("cache_id"),
+            outputs=outputs,
+            name=data.get("name"),
+            estimated_duration=data.get("estimated_duration"),
+            level=data.get("level", 0),
+        )
+
+    def to_json(self) -> str:
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> "ExecutionStep":
+        return cls.from_dict(json.loads(json_str))
+
+
+@dataclass
+class PlanInput:
+    """
+    An input to the execution plan.
+
+    Attributes:
+        name: Human-readable name from recipe (e.g., "source_video")
+        cache_id: Content hash of the input file
+        cid: Same as cache_id (for clarity)
+        media_type: MIME type of the input
+    """
+    name: str
+    cache_id: str
+    cid: str
+    media_type: str = "application/octet-stream"
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "name": self.name,
+            "cache_id": self.cache_id,
+            "cid": self.cid,
+            "media_type": self.media_type,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "PlanInput":
+        return cls(
+            name=data["name"],
+            cache_id=data["cache_id"],
+            cid=data.get("cid", data["cache_id"]),
+            media_type=data.get("media_type", "application/octet-stream"),
+        )
+
+
+@dataclass
+class ExecutionPlan:
+    """
+    Complete execution plan for a recipe.
+
+    Contains all steps in topological order with pre-computed cache IDs.
+    The plan is deterministic: same recipe + same inputs = same plan.
+
+    Attributes:
+        name: Human-readable plan name from recipe
+        plan_id: Hash of the entire plan (for deduplication)
+        recipe_id: Source recipe identifier
+        recipe_name: Human-readable recipe name
+        recipe_hash: Hash of the recipe content
+        seed: Random seed used for planning
+        steps: List of steps in execution order
+        output_step: ID of the final output step
+        output_name: Human-readable name of the final output
+        inputs: Structured input definitions
+        analysis_cache_ids: Cache IDs of analysis results used
+        input_hashes: Content hashes of input files (legacy, use inputs)
+        created_at: When the plan was generated
+        metadata: Optional additional metadata
+    """
+    plan_id: Optional[str]
+    recipe_id: str
+    recipe_hash: str
+    steps: List[ExecutionStep]
+    output_step: str
+    name: Optional[str] = None
+    recipe_name: Optional[str] = None
+    seed: Optional[int] = None
+    output_name: Optional[str] = None
+    inputs: List[PlanInput] = field(default_factory=list)
+    analysis_cache_ids: Dict[str, str] = field(default_factory=dict)
+    input_hashes: Dict[str, str] = field(default_factory=dict)
+    created_at: Optional[str] = None
+    metadata: Dict[str, Any] = field(default_factory=dict)
+
+    def __post_init__(self):
+        if self.created_at is None:
+            self.created_at = datetime.now(timezone.utc).isoformat()
+        if self.plan_id is None:
+            self.plan_id = self._compute_plan_id()
+
+    def _compute_plan_id(self) -> str:
+        """Compute plan ID from contents."""
+        content = {
+            "recipe_hash": self.recipe_hash,
+            "steps": [s.to_dict() for s in self.steps],
+            "input_hashes": self.input_hashes,
+            "analysis_cache_ids": self.analysis_cache_ids,
+        }
+        return _stable_hash(content)
+
+    def compute_all_cache_ids(self) -> None:
+        """
+        Compute cache IDs for all steps in dependency order.
+
+        Must be called after all steps are added to ensure
+        cache IDs propagate correctly through dependencies.
+        """
+        # Build step lookup
+        step_by_id = {s.step_id: s for s in self.steps}
+
+        # Cache IDs start with input hashes
+        cache_ids = dict(self.input_hashes)
+
+        # Process in order (assumes topological order)
+        for step in self.steps:
+            # For SOURCE steps referencing inputs, use input hash
+            if step.node_type == "SOURCE" and step.config.get("input_ref"):
+                ref = step.config["input_ref"]
+                if ref in self.input_hashes:
+                    step.cache_id = self.input_hashes[ref]
+                    cache_ids[step.step_id] = step.cache_id
+                    continue
+
+            # For other steps, compute from inputs
+            input_cache_ids = {}
+            for input_step_id in step.input_steps:
+                if input_step_id in cache_ids:
+                    input_cache_ids[input_step_id] = cache_ids[input_step_id]
+                elif input_step_id in step_by_id:
+                    # Step should have been processed already
+                    input_cache_ids[input_step_id] = step_by_id[input_step_id].cache_id
+                else:
+                    raise ValueError(f"Input step {input_step_id} not found for {step.step_id}")
+
+            step.compute_cache_id(input_cache_ids)
+            cache_ids[step.step_id] = step.cache_id
+
+        # Recompute plan_id with final cache IDs
+        self.plan_id = self._compute_plan_id()
+
+    def compute_levels(self) -> int:
+        """
+        Compute dependency levels for all steps.
+
+        Level 0 = no dependencies (can start immediately)
+        Level N = depends on steps at level N-1
+
+        Returns:
+            Maximum level (number of sequential dependency levels)
+        """
+        step_by_id = {s.step_id: s for s in self.steps}
+        levels = {}
+
+        def compute_level(step_id: str) -> int:
+            if step_id in levels:
+                return levels[step_id]
+
+            step = step_by_id.get(step_id)
+            if step is None:
+                return 0  # Input from outside the plan
+
+            if not step.input_steps:
+                levels[step_id] = 0
+                step.level = 0
+                return 0
+
+            max_input_level = max(compute_level(s) for s in step.input_steps)
+            level = max_input_level + 1
+            levels[step_id] = level
+            step.level = level
+            return level
+
+        for step in self.steps:
+            compute_level(step.step_id)
+
+        return max(levels.values()) if levels else 0
+
+    def get_steps_by_level(self) -> Dict[int, List[ExecutionStep]]:
+        """
+        Group steps by dependency level.
+
+        Steps at the same level can execute in parallel.
+
+        Returns:
+            Dict mapping level -> list of steps at that level
+        """
+        by_level: Dict[int, List[ExecutionStep]] = {}
+        for step in self.steps:
+            by_level.setdefault(step.level, []).append(step)
+        return by_level
+
+    def get_step(self, step_id: str) -> Optional[ExecutionStep]:
+        """Get step by ID."""
+        for step in self.steps:
+            if step.step_id == step_id:
+                return step
+        return None
+
+    def get_step_by_cache_id(self, cache_id: str) -> Optional[ExecutionStep]:
+        """Get step by cache ID."""
+        for step in self.steps:
+            if step.cache_id == cache_id:
+                return step
+        return None
+
+    def get_step_by_name(self, name: str) -> Optional[ExecutionStep]:
+        """Get step by human-readable name."""
+        for step in self.steps:
+            if step.name == name:
+                return step
+        return None
+
+    def get_all_outputs(self) -> Dict[str, StepOutput]:
+        """
+        Get all outputs from all steps, keyed by output name.
+
+        Returns:
+            Dict mapping output name -> StepOutput
+        """
+        outputs = {}
+        for step in self.steps:
+            for output in step.outputs:
+                outputs[output.name] = output
+        return outputs
+
+    def get_output_cache_ids(self) -> Dict[str, str]:
+        """
+        Get mapping of output names to cache IDs.
+
+        Returns:
+            Dict mapping output name -> cache_id
+        """
+        return {
+            output.name: output.cache_id
+            for step in self.steps
+            for output in step.outputs
+        }
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {
+            "plan_id": self.plan_id,
+            "name": self.name,
+            "recipe_id": self.recipe_id,
+            "recipe_name": self.recipe_name,
+            "recipe_hash": self.recipe_hash,
+            "seed": self.seed,
+            "inputs": [i.to_dict() for i in self.inputs],
+            "steps": [s.to_dict() for s in self.steps],
+            "output_step": self.output_step,
+            "output_name": self.output_name,
+            "analysis_cache_ids": self.analysis_cache_ids,
+            "input_hashes": self.input_hashes,
+            "created_at": self.created_at,
+            "metadata": self.metadata,
+        }
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "ExecutionPlan":
+        inputs = [PlanInput.from_dict(i) for i in data.get("inputs", [])]
+        return cls(
+            plan_id=data.get("plan_id"),
+            name=data.get("name"),
+            recipe_id=data["recipe_id"],
+            recipe_name=data.get("recipe_name"),
+            recipe_hash=data["recipe_hash"],
+            seed=data.get("seed"),
+            inputs=inputs,
+            steps=[ExecutionStep.from_dict(s) for s in data.get("steps", [])],
+            output_step=data["output_step"],
+            output_name=data.get("output_name"),
+            analysis_cache_ids=data.get("analysis_cache_ids", {}),
+            input_hashes=data.get("input_hashes", {}),
+            created_at=data.get("created_at"),
+            metadata=data.get("metadata", {}),
+        )
+
+    def to_json(self, indent: int = 2) -> str:
+        return json.dumps(self.to_dict(), indent=indent)
+
+    @classmethod
+    def from_json(cls, json_str: str) -> "ExecutionPlan":
+        return cls.from_dict(json.loads(json_str))
+
+    def summary(self) -> str:
+        """Get a human-readable summary of the plan."""
+        by_level = self.get_steps_by_level()
+        max_level = max(by_level.keys()) if by_level else 0
+
+        lines = [
+            f"Execution Plan: {self.plan_id[:16]}...",
+            f"Recipe: {self.recipe_id}",
+            f"Steps: {len(self.steps)}",
+            f"Levels: {max_level + 1}",
+            "",
+        ]
+
+        for level in sorted(by_level.keys()):
+            steps = by_level[level]
+            lines.append(f"Level {level}: ({len(steps)} steps, can run in parallel)")
+            for step in steps:
+                cache_status = f"[{step.cache_id[:8]}...]" if step.cache_id else "[no cache_id]"
+                lines.append(f"  - {step.step_id}: {step.node_type} {cache_status}")
+
+        return "\n".join(lines)

artdag/core/artdag/planning/tree_reduction.py

@@ -0,0 +1,231 @@
+# artdag/planning/tree_reduction.py
+"""
+Tree reduction for parallel composition.
+
+Instead of sequential pairwise composition:
+    A → AB → ABC → ABCD  (3 sequential steps)
+
+Use parallel tree reduction:
+    A ─┬─ AB ─┬─ ABCD
+    B ─┘      │
+    C ─┬─ CD ─┘
+    D ─┘
+
+This reduces O(N) to O(log N) levels of sequential dependency.
+"""
+
+import math
+from dataclasses import dataclass
+from typing import List, Tuple, Any, Dict
+
+
+@dataclass
+class ReductionNode:
+    """A node in the reduction tree."""
+    node_id: str
+    input_ids: List[str]
+    level: int
+    position: int  # Position within level
+
+
+class TreeReducer:
+    """
+    Generates tree reduction plans for parallel composition.
+
+    Used to convert N inputs into optimal parallel SEQUENCE operations.
+    """
+
+    def __init__(self, node_type: str = "SEQUENCE"):
+        """
+        Initialize the reducer.
+
+        Args:
+            node_type: The composition node type (SEQUENCE, AUDIO_MIX, etc.)
+        """
+        self.node_type = node_type
+
+    def reduce(
+        self,
+        input_ids: List[str],
+        id_prefix: str = "reduce",
+    ) -> Tuple[List[ReductionNode], str]:
+        """
+        Generate a tree reduction plan for the given inputs.
+
+        Args:
+            input_ids: List of input step IDs to reduce
+            id_prefix: Prefix for generated node IDs
+
+        Returns:
+            Tuple of (list of reduction nodes, final output node ID)
+        """
+        if len(input_ids) == 0:
+            raise ValueError("Cannot reduce empty input list")
+
+        if len(input_ids) == 1:
+            # Single input, no reduction needed
+            return [], input_ids[0]
+
+        if len(input_ids) == 2:
+            # Two inputs, single reduction
+            node_id = f"{id_prefix}_final"
+            node = ReductionNode(
+                node_id=node_id,
+                input_ids=input_ids,
+                level=0,
+                position=0,
+            )
+            return [node], node_id
+
+        # Build tree levels
+        nodes = []
+        current_level = list(input_ids)
+        level_num = 0
+
+        while len(current_level) > 1:
+            next_level = []
+            position = 0
+
+            # Pair up nodes at current level
+            i = 0
+            while i < len(current_level):
+                if i + 1 < len(current_level):
+                    # Pair available
+                    left = current_level[i]
+                    right = current_level[i + 1]
+                    node_id = f"{id_prefix}_L{level_num}_P{position}"
+                    node = ReductionNode(
+                        node_id=node_id,
+                        input_ids=[left, right],
+                        level=level_num,
+                        position=position,
+                    )
+                    nodes.append(node)
+                    next_level.append(node_id)
+                    i += 2
+                else:
+                    # Odd one out, promote to next level
+                    next_level.append(current_level[i])
+                    i += 1
+
+                position += 1
+
+            current_level = next_level
+            level_num += 1
+
+        # The last remaining node is the output
+        output_id = current_level[0]
+
+        # Rename final node for clarity
+        if nodes and nodes[-1].node_id == output_id:
+            nodes[-1].node_id = f"{id_prefix}_final"
+            output_id = f"{id_prefix}_final"
+
+        return nodes, output_id
+
+    def get_reduction_depth(self, n: int) -> int:
+        """
+        Calculate the number of reduction levels needed.
+
+        Args:
+            n: Number of inputs
+
+        Returns:
+            Number of sequential reduction levels (log2(n) ceiling)
+        """
+        if n <= 1:
+            return 0
+        return math.ceil(math.log2(n))
+
+    def get_total_operations(self, n: int) -> int:
+        """
+        Calculate total number of reduction operations.
+
+        Args:
+            n: Number of inputs
+
+        Returns:
+            Total composition operations (always n-1)
+        """
+        return max(0, n - 1)
+
+    def reduce_with_config(
+        self,
+        input_ids: List[str],
+        base_config: Dict[str, Any],
+        id_prefix: str = "reduce",
+    ) -> Tuple[List[Tuple[ReductionNode, Dict[str, Any]]], str]:
+        """
+        Generate reduction plan with configuration for each node.
+
+        Args:
+            input_ids: List of input step IDs
+            base_config: Base configuration to use for each reduction
+            id_prefix: Prefix for generated node IDs
+
+        Returns:
+            Tuple of (list of (node, config) pairs, final output ID)
+        """
+        nodes, output_id = self.reduce(input_ids, id_prefix)
+        result = [(node, dict(base_config)) for node in nodes]
+        return result, output_id
+
+
+def reduce_sequence(
+    input_ids: List[str],
+    transition_config: Dict[str, Any] = None,
+    id_prefix: str = "seq",
+) -> Tuple[List[Tuple[str, List[str], Dict[str, Any]]], str]:
+    """
+    Convenience function for SEQUENCE reduction.
+
+    Args:
+        input_ids: Input step IDs to sequence
+        transition_config: Transition configuration (default: cut)
+        id_prefix: Prefix for generated step IDs
+
+    Returns:
+        Tuple of (list of (step_id, inputs, config), final step ID)
+    """
+    if transition_config is None:
+        transition_config = {"transition": {"type": "cut"}}
+
+    reducer = TreeReducer("SEQUENCE")
+    nodes, output_id = reducer.reduce(input_ids, id_prefix)
+
+    result = [
+        (node.node_id, node.input_ids, dict(transition_config))
+        for node in nodes
+    ]
+
+    return result, output_id
+
+
+def reduce_audio_mix(
+    input_ids: List[str],
+    mix_config: Dict[str, Any] = None,
+    id_prefix: str = "mix",
+) -> Tuple[List[Tuple[str, List[str], Dict[str, Any]]], str]:
+    """
+    Convenience function for AUDIO_MIX reduction.
+
+    Args:
+        input_ids: Input step IDs to mix
+        mix_config: Mix configuration
+        id_prefix: Prefix for generated step IDs
+
+    Returns:
+        Tuple of (list of (step_id, inputs, config), final step ID)
+    """
+    if mix_config is None:
+        mix_config = {"normalize": True}
+
+    reducer = TreeReducer("AUDIO_MIX")
+    nodes, output_id = reducer.reduce(input_ids, id_prefix)
+
+    result = [
+        (node.node_id, node.input_ids, dict(mix_config))
+        for node in nodes
+    ]
+
+    return result, output_id

artdag/core/artdag/registry/__init__.py

@@ -0,0 +1,20 @@
+# primitive/registry/__init__.py
+"""
+Art DAG Registry.
+
+The registry is the foundational data structure that maps named assets
+to their source paths or content-addressed IDs. Assets in the registry
+can be referenced by DAGs.
+
+Example:
+    registry = Registry("/path/to/registry")
+    registry.add("cat", "/path/to/cat.jpg", tags=["animal", "photo"])
+
+    # Later, in a DAG:
+    builder = DAGBuilder()
+    cat = builder.source(registry.get("cat").path)
+"""
+
+from .registry import Registry, Asset
+
+__all__ = ["Registry", "Asset"]

artdag/core/artdag/registry/registry.py

@@ -0,0 +1,294 @@
+# primitive/registry/registry.py
+"""
+Asset registry for the Art DAG.
+
+The registry stores named assets with metadata, enabling:
+- Named references to source files
+- Tagging and categorization
+- Content-addressed deduplication
+- Asset discovery and search
+"""
+
+import hashlib
+import json
+import shutil
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+
+def _file_hash(path: Path, algorithm: str = "sha3_256") -> str:
+    """
+    Compute content hash of a file.
+
+    Uses SHA-3 (Keccak) by default for quantum resistance.
+    SHA-3-256 provides 128-bit security against quantum attacks (Grover's algorithm).
+
+    Args:
+        path: File to hash
+        algorithm: Hash algorithm (sha3_256, sha3_512, sha256, blake2b)
+
+    Returns:
+        Full hex digest (no truncation)
+    """
+    hasher = hashlib.new(algorithm)
+    with open(path, "rb") as f:
+        for chunk in iter(lambda: f.read(65536), b""):
+            hasher.update(chunk)
+    return hasher.hexdigest()
+
+
+@dataclass
+class Asset:
+    """
+    A registered asset in the Art DAG.
+
+    The cid is the true identifier. URL and local_path are
+    locations where the content can be fetched.
+
+    Attributes:
+        name: Unique name for the asset
+        cid: SHA-3-256 hash - the canonical identifier
+        url: Public URL (canonical location)
+        local_path: Optional local path (for local execution)
+        asset_type: Type of asset (image, video, audio, etc.)
+        tags: List of tags for categorization
+        metadata: Additional metadata (dimensions, duration, etc.)
+        created_at: Timestamp when added to registry
+    """
+    name: str
+    cid: str
+    url: Optional[str] = None
+    local_path: Optional[Path] = None
+    asset_type: str = "unknown"
+    tags: List[str] = field(default_factory=list)
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    created_at: float = field(default_factory=time.time)
+
+    @property
+    def path(self) -> Optional[Path]:
+        """Backwards compatible path property."""
+        return self.local_path
+
+    def to_dict(self) -> Dict[str, Any]:
+        data = {
+            "name": self.name,
+            "cid": self.cid,
+            "asset_type": self.asset_type,
+            "tags": self.tags,
+            "metadata": self.metadata,
+            "created_at": self.created_at,
+        }
+        if self.url:
+            data["url"] = self.url
+        if self.local_path:
+            data["local_path"] = str(self.local_path)
+        return data
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "Asset":
+        local_path = data.get("local_path") or data.get("path")  # backwards compat
+        return cls(
+            name=data["name"],
+            cid=data["cid"],
+            url=data.get("url"),
+            local_path=Path(local_path) if local_path else None,
+            asset_type=data.get("asset_type", "unknown"),
+            tags=data.get("tags", []),
+            metadata=data.get("metadata", {}),
+            created_at=data.get("created_at", time.time()),
+        )
+
+
+class Registry:
+    """
+    The Art DAG registry.
+
+    Stores named assets that can be referenced by DAGs.
+
+    Structure:
+        registry_dir/
+            registry.json     # Index of all assets
+            assets/           # Optional: copied asset files
+                <hash>/
+                    <filename>
+    """
+
+    def __init__(self, registry_dir: Path | str, copy_assets: bool = False):
+        """
+        Initialize the registry.
+
+        Args:
+            registry_dir: Directory to store registry data
+            copy_assets: If True, copy assets into registry (content-addressed)
+        """
+        self.registry_dir = Path(registry_dir)
+        self.registry_dir.mkdir(parents=True, exist_ok=True)
+        self.copy_assets = copy_assets
+        self._assets: Dict[str, Asset] = {}
+        self._load()
+
+    def _index_path(self) -> Path:
+        return self.registry_dir / "registry.json"
+
+    def _assets_dir(self) -> Path:
+        return self.registry_dir / "assets"
+
+    def _load(self):
+        """Load registry from disk."""
+        index_path = self._index_path()
+        if index_path.exists():
+            with open(index_path) as f:
+                data = json.load(f)
+            self._assets = {
+                name: Asset.from_dict(asset_data)
+                for name, asset_data in data.get("assets", {}).items()
+            }
+
+    def _save(self):
+        """Save registry to disk."""
+        data = {
+            "version": "1.0",
+            "assets": {name: asset.to_dict() for name, asset in self._assets.items()},
+        }
+        with open(self._index_path(), "w") as f:
+            json.dump(data, f, indent=2)
+
+    def add(
+        self,
+        name: str,
+        cid: str,
+        url: str = None,
+        local_path: Path | str = None,
+        asset_type: str = None,
+        tags: List[str] = None,
+        metadata: Dict[str, Any] = None,
+    ) -> Asset:
+        """
+        Add an asset to the registry.
+
+        Args:
+            name: Unique name for the asset
+            cid: SHA-3-256 hash of the content (the canonical identifier)
+            url: Public URL where the asset can be fetched
+            local_path: Optional local path (for local execution)
+            asset_type: Type of asset (image, video, audio, etc.)
+            tags: List of tags for categorization
+            metadata: Additional metadata
+
+        Returns:
+            The created Asset
+        """
+        # Auto-detect asset type from URL or path extension
+        if asset_type is None:
+            ext = None
+            if url:
+                ext = Path(url.split("?")[0]).suffix.lower()
+            elif local_path:
+                ext = Path(local_path).suffix.lower()
+            if ext:
+                type_map = {
+                    ".jpg": "image", ".jpeg": "image", ".png": "image",
+                    ".gif": "image", ".webp": "image", ".bmp": "image",
+                    ".mp4": "video", ".mkv": "video", ".avi": "video",
+                    ".mov": "video", ".webm": "video",
+                    ".mp3": "audio", ".wav": "audio", ".flac": "audio",
+                    ".ogg": "audio", ".aac": "audio",
+                }
+                asset_type = type_map.get(ext, "unknown")
+            else:
+                asset_type = "unknown"
+
+        asset = Asset(
+            name=name,
+            cid=cid,
+            url=url,
+            local_path=Path(local_path).resolve() if local_path else None,
+            asset_type=asset_type,
+            tags=tags or [],
+            metadata=metadata or {},
+        )
+
+        self._assets[name] = asset
+        self._save()
+        return asset
+
+    def add_from_file(
+        self,
+        name: str,
+        path: Path | str,
+        url: str = None,
+        asset_type: str = None,
+        tags: List[str] = None,
+        metadata: Dict[str, Any] = None,
+    ) -> Asset:
+        """
+        Add an asset from a local file (computes hash automatically).
+
+        Args:
+            name: Unique name for the asset
+            path: Path to the source file
+            url: Optional public URL
+            asset_type: Type of asset (auto-detected if not provided)
+            tags: List of tags for categorization
+            metadata: Additional metadata
+
+        Returns:
+            The created Asset
+        """
+        path = Path(path)
+        if not path.exists():
+            raise FileNotFoundError(f"Asset file not found: {path}")
+
+        cid = _file_hash(path)
+
+        return self.add(
+            name=name,
+            cid=cid,
+            url=url,
+            local_path=path,
+            asset_type=asset_type,
+            tags=tags,
+            metadata=metadata,
+        )
+
+    def get(self, name: str) -> Optional[Asset]:
+        """Get an asset by name."""
+        return self._assets.get(name)
+
+    def remove(self, name: str) -> bool:
+        """Remove an asset from the registry."""
+        if name not in self._assets:
+            return False
+        del self._assets[name]
+        self._save()
+        return True
+
+    def list(self) -> List[Asset]:
+        """List all assets."""
+        return list(self._assets.values())
+
+    def find_by_tag(self, tag: str) -> List[Asset]:
+        """Find assets with a specific tag."""
+        return [a for a in self._assets.values() if tag in a.tags]
+
+    def find_by_type(self, asset_type: str) -> List[Asset]:
+        """Find assets of a specific type."""
+        return [a for a in self._assets.values() if a.asset_type == asset_type]
+
+    def find_by_hash(self, cid: str) -> Optional[Asset]:
+        """Find an asset by content hash."""
+        for asset in self._assets.values():
+            if asset.cid == cid:
+                return asset
+        return None
+
+    def __contains__(self, name: str) -> bool:
+        return name in self._assets
+
+    def __len__(self) -> int:
+        return len(self._assets)
+
+    def __iter__(self):
+        return iter(self._assets.values())

artdag/core/artdag/server.py

@@ -0,0 +1,253 @@
+# primitive/server.py
+"""
+HTTP server for primitive execution engine.
+
+Provides a REST API for submitting DAGs and retrieving results.
+
+Endpoints:
+    POST /execute     - Submit DAG for execution
+    GET  /status/:id  - Get execution status
+    GET  /result/:id  - Get execution result
+    GET  /cache/stats - Get cache statistics
+    DELETE /cache     - Clear cache
+"""
+
+import json
+import logging
+import threading
+import uuid
+from dataclasses import dataclass, field
+from http.server import HTTPServer, BaseHTTPRequestHandler
+from pathlib import Path
+from typing import Any, Dict, Optional
+from urllib.parse import urlparse
+
+from .dag import DAG
+from .engine import Engine, ExecutionResult
+from . import nodes  # Register built-in executors
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class Job:
+    """A pending or completed execution job."""
+    job_id: str
+    dag: DAG
+    status: str = "pending"  # pending, running, completed, failed
+    result: Optional[ExecutionResult] = None
+    error: Optional[str] = None
+
+
+class PrimitiveServer:
+    """
+    HTTP server for the primitive engine.
+
+    Usage:
+        server = PrimitiveServer(cache_dir="/tmp/primitive_cache", port=8080)
+        server.start()  # Blocking
+    """
+
+    def __init__(self, cache_dir: Path | str, host: str = "127.0.0.1", port: int = 8080):
+        self.cache_dir = Path(cache_dir)
+        self.host = host
+        self.port = port
+        self.engine = Engine(self.cache_dir)
+        self.jobs: Dict[str, Job] = {}
+        self._lock = threading.Lock()
+
+    def submit_job(self, dag: DAG) -> str:
+        """Submit a DAG for execution, return job ID."""
+        job_id = str(uuid.uuid4())[:8]
+        job = Job(job_id=job_id, dag=dag)
+
+        with self._lock:
+            self.jobs[job_id] = job
+
+        # Execute in background thread
+        thread = threading.Thread(target=self._execute_job, args=(job_id,))
+        thread.daemon = True
+        thread.start()
+
+        return job_id
+
+    def _execute_job(self, job_id: str):
+        """Execute a job in background."""
+        with self._lock:
+            job = self.jobs.get(job_id)
+            if not job:
+                return
+            job.status = "running"
+
+        try:
+            result = self.engine.execute(job.dag)
+            with self._lock:
+                job.result = result
+                job.status = "completed" if result.success else "failed"
+                if not result.success:
+                    job.error = result.error
+        except Exception as e:
+            logger.exception(f"Job {job_id} failed")
+            with self._lock:
+                job.status = "failed"
+                job.error = str(e)
+
+    def get_job(self, job_id: str) -> Optional[Job]:
+        """Get job by ID."""
+        with self._lock:
+            return self.jobs.get(job_id)
+
+    def _create_handler(server_instance):
+        """Create request handler with access to server instance."""
+
+        class RequestHandler(BaseHTTPRequestHandler):
+            server_ref = server_instance
+
+            def log_message(self, format, *args):
+                logger.debug(format % args)
+
+            def _send_json(self, data: Any, status: int = 200):
+                self.send_response(status)
+                self.send_header("Content-Type", "application/json")
+                self.end_headers()
+                self.wfile.write(json.dumps(data).encode())
+
+            def _send_error(self, message: str, status: int = 400):
+                self._send_json({"error": message}, status)
+
+            def do_GET(self):
+                parsed = urlparse(self.path)
+                path = parsed.path
+
+                if path.startswith("/status/"):
+                    job_id = path[8:]
+                    job = self.server_ref.get_job(job_id)
+                    if not job:
+                        self._send_error("Job not found", 404)
+                        return
+                    self._send_json({
+                        "job_id": job.job_id,
+                        "status": job.status,
+                        "error": job.error,
+                    })
+
+                elif path.startswith("/result/"):
+                    job_id = path[8:]
+                    job = self.server_ref.get_job(job_id)
+                    if not job:
+                        self._send_error("Job not found", 404)
+                        return
+                    if job.status == "pending" or job.status == "running":
+                        self._send_json({
+                            "job_id": job.job_id,
+                            "status": job.status,
+                            "ready": False,
+                        })
+                        return
+
+                    result = job.result
+                    self._send_json({
+                        "job_id": job.job_id,
+                        "status": job.status,
+                        "ready": True,
+                        "success": result.success if result else False,
+                        "output_path": str(result.output_path) if result and result.output_path else None,
+                        "error": job.error,
+                        "execution_time": result.execution_time if result else 0,
+                        "nodes_executed": result.nodes_executed if result else 0,
+                        "nodes_cached": result.nodes_cached if result else 0,
+                    })
+
+                elif path == "/cache/stats":
+                    stats = self.server_ref.engine.get_cache_stats()
+                    self._send_json({
+                        "total_entries": stats.total_entries,
+                        "total_size_bytes": stats.total_size_bytes,
+                        "hits": stats.hits,
+                        "misses": stats.misses,
+                        "hit_rate": stats.hit_rate,
+                    })
+
+                elif path == "/health":
+                    self._send_json({"status": "ok"})
+
+                else:
+                    self._send_error("Not found", 404)
+
+            def do_POST(self):
+                if self.path == "/execute":
+                    try:
+                        content_length = int(self.headers.get("Content-Length", 0))
+                        body = self.rfile.read(content_length).decode()
+                        data = json.loads(body)
+
+                        dag = DAG.from_dict(data)
+                        job_id = self.server_ref.submit_job(dag)
+
+                        self._send_json({
+                            "job_id": job_id,
+                            "status": "pending",
+                        })
+                    except json.JSONDecodeError as e:
+                        self._send_error(f"Invalid JSON: {e}")
+                    except Exception as e:
+                        self._send_error(str(e), 500)
+                else:
+                    self._send_error("Not found", 404)
+
+            def do_DELETE(self):
+                if self.path == "/cache":
+                    self.server_ref.engine.clear_cache()
+                    self._send_json({"status": "cleared"})
+                else:
+                    self._send_error("Not found", 404)
+
+        return RequestHandler
+
+    def start(self):
+        """Start the HTTP server (blocking)."""
+        handler = self._create_handler()
+        server = HTTPServer((self.host, self.port), handler)
+        logger.info(f"Primitive server starting on {self.host}:{self.port}")
+        print(f"Primitive server running on http://{self.host}:{self.port}")
+        try:
+            server.serve_forever()
+        except KeyboardInterrupt:
+            print("\nShutting down...")
+            server.shutdown()
+
+    def start_background(self) -> threading.Thread:
+        """Start the server in a background thread."""
+        thread = threading.Thread(target=self.start)
+        thread.daemon = True
+        thread.start()
+        return thread
+
+
+def main():
+    """CLI entry point."""
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Primitive execution server")
+    parser.add_argument("--host", default="127.0.0.1", help="Host to bind to")
+    parser.add_argument("--port", type=int, default=8080, help="Port to bind to")
+    parser.add_argument("--cache-dir", default="/tmp/primitive_cache", help="Cache directory")
+    parser.add_argument("--verbose", "-v", action="store_true", help="Verbose logging")
+
+    args = parser.parse_args()
+
+    logging.basicConfig(
+        level=logging.DEBUG if args.verbose else logging.INFO,
+        format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
+    )
+
+    server = PrimitiveServer(
+        cache_dir=args.cache_dir,
+        host=args.host,
+        port=args.port,
+    )
+    server.start()
+
+
+if __name__ == "__main__":
+    main()

artdag/core/artdag/sexp/__init__.py

@@ -0,0 +1,75 @@
+"""
+S-expression parsing, compilation, and planning for ArtDAG.
+
+This module provides:
+- parser: Parse S-expression text into Python data structures
+- compiler: Compile recipe S-expressions into DAG format
+- planner: Generate execution plans from recipes
+"""
+
+from .parser import (
+    parse,
+    parse_all,
+    serialize,
+    Symbol,
+    Keyword,
+    ParseError,
+)
+
+from .compiler import (
+    compile_recipe,
+    compile_string,
+    CompiledRecipe,
+    CompileError,
+    ParamDef,
+    _parse_params,
+)
+
+from .planner import (
+    create_plan,
+    ExecutionPlanSexp,
+    PlanStep,
+    step_to_task_sexp,
+    task_cache_id,
+)
+
+from .scheduler import (
+    PlanScheduler,
+    PlanResult,
+    StepResult,
+    schedule_plan,
+    step_to_sexp,
+    step_sexp_to_string,
+    verify_step_cache_id,
+)
+
+__all__ = [
+    # Parser
+    'parse',
+    'parse_all',
+    'serialize',
+    'Symbol',
+    'Keyword',
+    'ParseError',
+    # Compiler
+    'compile_recipe',
+    'compile_string',
+    'CompiledRecipe',
+    'CompileError',
+    'ParamDef',
+    '_parse_params',
+    # Planner
+    'create_plan',
+    'ExecutionPlanSexp',
+    'PlanStep',
+    'step_to_task_sexp',
+    'task_cache_id',
+    # Scheduler
+    'PlanScheduler',
+    'PlanResult',
+    'StepResult',
+    'schedule_plan',
+    'step_to_sexp',
+    'step_sexp_to_string',
+    'verify_step_cache_id',
+]

artdag/core/artdag/sexp/effect_loader.py

@@ -0,0 +1,337 @@
+"""
+Sexp effect loader.
+
+Loads sexp effect definitions (define-effect forms) and creates
+frame processors that evaluate the sexp body with primitives.
+
+Effects must use :params syntax:
+
+    (define-effect name
+      :params (
+        (param1 :type int :default 8 :range [4 32] :desc "description")
+        (param2 :type string :default "value" :desc "description")
+      )
+      body)
+
+For effects with no parameters, use empty :params ():
+
+    (define-effect name
+      :params ()
+      body)
+
+Unknown parameters passed to effects will raise an error.
+
+Usage:
+    loader = SexpEffectLoader()
+    effect_fn = loader.load_effect_file(Path("effects/ascii_art.sexp"))
+    output = effect_fn(input_path, output_path, config)
+"""
+
+import logging
+from pathlib import Path
+from typing import Any, Callable, Dict, List, Optional
+
+import numpy as np
+
+from .parser import parse_all, Symbol, Keyword
+from .evaluator import evaluate
+from .primitives import PRIMITIVES
+from .compiler import ParamDef, _parse_params, CompileError
+
+logger = logging.getLogger(__name__)
+
+
+def _parse_define_effect(sexp) -> tuple:
+    """
+    Parse a define-effect form.
+
+    Required syntax:
+        (define-effect name
+          :params (
+            (param1 :type int :default 8 :range [4 32] :desc "description")
+          )
+          body)
+
+    Effects MUST use :params syntax. Legacy ((param default) ...) syntax is not supported.
+
+    Returns (name, params_with_defaults, param_defs, body)
+    where param_defs is a list of ParamDef objects
+    """
+    if not isinstance(sexp, list) or len(sexp) < 3:
+        raise ValueError(f"Invalid define-effect form: {sexp}")
+
+    head = sexp[0]
+    if not (isinstance(head, Symbol) and head.name == "define-effect"):
+        raise ValueError(f"Expected define-effect, got {head}")
+
+    name = sexp[1]
+    if isinstance(name, Symbol):
+        name = name.name
+
+    params_with_defaults = {}
+    param_defs: List[ParamDef] = []
+    body = None
+    found_params = False
+
+    # Parse :params and body
+    i = 2
+    while i < len(sexp):
+        item = sexp[i]
+        if isinstance(item, Keyword) and item.name == "params":
+            # :params syntax
+            if i + 1 >= len(sexp):
+                raise ValueError(f"Effect '{name}': Missing params list after :params keyword")
+            try:
+                param_defs = _parse_params(sexp[i + 1])
+                # Build params_with_defaults from ParamDef objects
+                for pd in param_defs:
+                    params_with_defaults[pd.name] = pd.default
+            except CompileError as e:
+                raise ValueError(f"Effect '{name}': Error parsing :params: {e}")
+            found_params = True
+            i += 2
+        elif isinstance(item, Keyword):
+            # Skip other keywords we don't recognize
+            i += 2
+        elif body is None:
+            # First non-keyword item is the body
+            if isinstance(item, list) and item:
+                first_elem = item[0]
+                # Check for legacy syntax and reject it
+                if isinstance(first_elem, list) and len(first_elem) >= 2:
+                    raise ValueError(
+                        f"Effect '{name}': Legacy parameter syntax ((name default) ...) is not supported. "
+                        f"Use :params block instead:\n"
+                        f"  :params (\n"
+                        f"    (param_name :type int :default 0 :desc \"description\")\n"
+                        f"  )"
+                    )
+            body = item
+            i += 1
+        else:
+            i += 1
+
+    if body is None:
+        raise ValueError(f"Effect '{name}': No body found")
+
+    if not found_params:
+        raise ValueError(
+            f"Effect '{name}': Missing :params block. Effects must declare parameters.\n"
+            f"For effects with no parameters, use empty :params ():\n"
+            f"  (define-effect {name}\n"
+            f"    :params ()\n"
+            f"    body)"
+        )
+
+    return name, params_with_defaults, param_defs, body
+
+
+def _create_process_frame(
+    effect_name: str,
+    params_with_defaults: Dict[str, Any],
+    param_defs: List[ParamDef],
+    body: Any,
+) -> Callable:
+    """
+    Create a process_frame function that evaluates the sexp body.
+
+    The function signature is: (frame, params, state) -> (frame, state)
+    """
+    import math
+
+    def process_frame(frame: np.ndarray, params: Dict[str, Any], state: Any):
+        """Evaluate sexp effect body on a frame."""
+        # Build environment with primitives
+        env = dict(PRIMITIVES)
+
+        # Add math functions
+        env["floor"] = lambda x: int(math.floor(x))
+        env["ceil"] = lambda x: int(math.ceil(x))
+        env["round"] = lambda x: int(round(x))
+        env["abs"] = abs
+        env["min"] = min
+        env["max"] = max
+        env["sqrt"] = math.sqrt
+        env["sin"] = math.sin
+        env["cos"] = math.cos
+
+        # Add list operations
+        env["list"] = lambda *args: tuple(args)
+        env["nth"] = lambda coll, i: coll[int(i)] if coll else None
+
+        # Bind frame
+        env["frame"] = frame
+
+        # Validate that all provided params are known
+        known_params = set(params_with_defaults.keys())
+        for k in params.keys():
+            if k not in known_params:
+                raise ValueError(
+                    f"Effect '{effect_name}': Unknown parameter '{k}'. "
+                    f"Valid parameters are: {', '.join(sorted(known_params)) if known_params else '(none)'}"
+                )
+
+        # Bind parameters (defaults + overrides from config)
+        for param_name, default in params_with_defaults.items():
+            # Use config value if provided, otherwise default
+            if param_name in params:
+                env[param_name] = params[param_name]
+            elif default is not None:
+                env[param_name] = default
+
+        # Evaluate the body
+        try:
+            result = evaluate(body, env)
+            if isinstance(result, np.ndarray):
+                return result, state
+            else:
+                logger.warning(f"Effect {effect_name} returned {type(result)}, expected ndarray")
+                return frame, state
+        except Exception as e:
+            logger.error(f"Error evaluating effect {effect_name}: {e}")
+            raise
+
+    return process_frame
+
+
+def load_sexp_effect(source: str, base_path: Optional[Path] = None) -> tuple:
+    """
+    Load a sexp effect from source code.
+
+    Args:
+        source: Sexp source code
+        base_path: Base path for resolving relative imports
+
+    Returns:
+        (effect_name, process_frame_fn, params_with_defaults, param_defs)
+        where param_defs is a list of ParamDef objects for introspection
+    """
+    exprs = parse_all(source)
+
+    # Find define-effect form
+    define_effect = None
+    if isinstance(exprs, list):
+        for expr in exprs:
+            if isinstance(expr, list) and expr and isinstance(expr[0], Symbol):
+                if expr[0].name == "define-effect":
+                    define_effect = expr
+                    break
+    elif isinstance(exprs, list) and exprs and isinstance(exprs[0], Symbol):
+        if exprs[0].name == "define-effect":
+            define_effect = exprs
+
+    if not define_effect:
+        raise ValueError("No define-effect form found in sexp effect")
+
+    name, params_with_defaults, param_defs, body = _parse_define_effect(define_effect)
+    process_frame = _create_process_frame(name, params_with_defaults, param_defs, body)
+
+    return name, process_frame, params_with_defaults, param_defs
+
+
+def load_sexp_effect_file(path: Path) -> tuple:
+    """
+    Load a sexp effect from file.
+
+    Returns:
+        (effect_name, process_frame_fn, params_with_defaults, param_defs)
+        where param_defs is a list of ParamDef objects for introspection
+    """
+    source = path.read_text()
+    return load_sexp_effect(source, base_path=path.parent)
+
+
+class SexpEffectLoader:
+    """
+    Loader for sexp effect definitions.
+
+    Creates effect functions compatible with the EffectExecutor.
+    """
+
+    def __init__(self, recipe_dir: Optional[Path] = None):
+        """
+        Initialize loader.
+
+        Args:
+            recipe_dir: Base directory for resolving relative effect paths
+        """
+        self.recipe_dir = recipe_dir or Path.cwd()
+        # Cache loaded effects with their param_defs for introspection
+        self._loaded_effects: Dict[str, tuple] = {}
+
+    def load_effect_path(self, effect_path: str) -> Callable:
+        """
+        Load a sexp effect from a relative path.
+
+        Args:
+            effect_path: Relative path to effect .sexp file
+
+        Returns:
+            Effect function (input_path, output_path, config) -> output_path
+        """
+        from ..effects.frame_processor import process_video
+
+        full_path = self.recipe_dir / effect_path
+        if not full_path.exists():
+            raise FileNotFoundError(f"Sexp effect not found: {full_path}")
+
+        name, process_frame_fn, params_defaults, param_defs = load_sexp_effect_file(full_path)
+        logger.info(f"Loaded sexp effect: {name} from {effect_path}")
+
+        # Cache for introspection
+        self._loaded_effects[effect_path] = (name, params_defaults, param_defs)
+
+        def effect_fn(input_path: Path, output_path: Path, config: Dict[str, Any]) -> Path:
+            """Run sexp effect via frame processor."""
+            # Extract params (excluding internal keys)
+            params = dict(params_defaults)  # Start with defaults
+            for k, v in config.items():
+                if k not in ("effect", "cid", "hash", "effect_path", "_binding"):
+                    params[k] = v
+
+            # Get bindings if present
+            bindings = {}
+            for key, value in config.items():
+                if isinstance(value, dict) and value.get("_resolved_values"):
+                    bindings[key] = value["_resolved_values"]
+
+            output_path.parent.mkdir(parents=True, exist_ok=True)
+            actual_output = output_path.with_suffix(".mp4")
+
+            process_video(
+                input_path=input_path,
+                output_path=actual_output,
+                process_frame=process_frame_fn,
+                params=params,
+                bindings=bindings,
+            )
+
+            logger.info(f"Processed sexp effect '{name}' from {effect_path}")
+            return actual_output
+
+        return effect_fn
+
+    def get_effect_params(self, effect_path: str) -> List[ParamDef]:
+        """
+        Get parameter definitions for an effect.
+
+        Args:
+            effect_path: Relative path to effect .sexp file
+
+        Returns:
+            List of ParamDef objects describing the effect's parameters
+        """
+        if effect_path not in self._loaded_effects:
+            # Load the effect to get its params
+            full_path = self.recipe_dir / effect_path
+            if not full_path.exists():
+                raise FileNotFoundError(f"Sexp effect not found: {full_path}")
+            name, _, params_defaults, param_defs = load_sexp_effect_file(full_path)
+            self._loaded_effects[effect_path] = (name, params_defaults, param_defs)
+
+        return self._loaded_effects[effect_path][2]
+
+
+def get_sexp_effect_loader(recipe_dir: Optional[Path] = None) -> SexpEffectLoader:
+    """Get a sexp effect loader instance."""
+    return SexpEffectLoader(recipe_dir)

artdag/core/artdag/sexp/evaluator.py

@@ -0,0 +1,869 @@
+"""
+Expression evaluator for S-expression DSL.
+
+Supports:
+- Arithmetic: +, -, *, /, mod, sqrt, pow, abs, floor, ceil, round, min, max, clamp
+- Comparison: =, <, >, <=, >=
+- Logic: and, or, not
+- Predicates: odd?, even?, zero?, nil?
+- Conditionals: if, cond, case
+- Data: list, dict/map construction, get
+- Lambda calls
+"""
+
+from typing import Any, Dict, List, Callable
+from .parser import Symbol, Keyword, Lambda, Binding
+
+
+class EvalError(Exception):
+    """Error during expression evaluation."""
+    pass
+
+
+# Built-in functions
+BUILTINS: Dict[str, Callable] = {}
+
+
+def builtin(name: str):
+    """Decorator to register a builtin function."""
+    def decorator(fn):
+        BUILTINS[name] = fn
+        return fn
+    return decorator
+
+
+@builtin("+")
+def add(*args):
+    return sum(args)
+
+
+@builtin("-")
+def sub(a, b=None):
+    if b is None:
+        return -a
+    return a - b
+
+
+@builtin("*")
+def mul(*args):
+    result = 1
+    for a in args:
+        result *= a
+    return result
+
+
+@builtin("/")
+def div(a, b):
+    return a / b
+
+
+@builtin("mod")
+def mod(a, b):
+    return a % b
+
+
+@builtin("sqrt")
+def sqrt(x):
+    return x ** 0.5
+
+
+@builtin("pow")
+def power(x, n):
+    return x ** n
+
+
+@builtin("abs")
+def absolute(x):
+    return abs(x)
+
+
+@builtin("floor")
+def floor_fn(x):
+    import math
+    return math.floor(x)
+
+
+@builtin("ceil")
+def ceil_fn(x):
+    import math
+    return math.ceil(x)
+
+
+@builtin("round")
+def round_fn(x, ndigits=0):
+    return round(x, int(ndigits))
+
+
+@builtin("min")
+def min_fn(*args):
+    if len(args) == 1 and isinstance(args[0], (list, tuple)):
+        return min(args[0])
+    return min(args)
+
+
+@builtin("max")
+def max_fn(*args):
+    if len(args) == 1 and isinstance(args[0], (list, tuple)):
+        return max(args[0])
+    return max(args)
+
+
+@builtin("clamp")
+def clamp(x, lo, hi):
+    return max(lo, min(hi, x))
+
+
+@builtin("=")
+def eq(a, b):
+    return a == b
+
+
+@builtin("<")
+def lt(a, b):
+    return a < b
+
+
+@builtin(">")
+def gt(a, b):
+    return a > b
+
+
+@builtin("<=")
+def lte(a, b):
+    return a <= b
+
+
+@builtin(">=")
+def gte(a, b):
+    return a >= b
+
+
+@builtin("odd?")
+def is_odd(n):
+    return n % 2 == 1
+
+
+@builtin("even?")
+def is_even(n):
+    return n % 2 == 0
+
+
+@builtin("zero?")
+def is_zero(n):
+    return n == 0
+
+
+@builtin("nil?")
+def is_nil(x):
+    return x is None
+
+
+@builtin("not")
+def not_fn(x):
+    return not x
+
+
+@builtin("inc")
+def inc(n):
+    return n + 1
+
+
+@builtin("dec")
+def dec(n):
+    return n - 1
+
+
+@builtin("list")
+def make_list(*args):
+    return list(args)
+
+
+@builtin("assert")
+def assert_true(condition, message="Assertion failed"):
+    if not condition:
+        raise RuntimeError(f"Assertion error: {message}")
+    return True
+
+
+@builtin("get")
+def get(coll, key, default=None):
+    if isinstance(coll, dict):
+        # Try the key directly first
+        result = coll.get(key, None)
+        if result is not None:
+            return result
+        # If key is a Keyword, also try its string name (for Python dicts with string keys)
+        if isinstance(key, Keyword):
+            result = coll.get(key.name, None)
+            if result is not None:
+                return result
+        # Return the default
+        return default
+    elif isinstance(coll, list):
+        return coll[key] if 0 <= key < len(coll) else default
+    else:
+        raise EvalError(f"get: expected dict or list, got {type(coll).__name__}: {str(coll)[:100]}")
+
+
+@builtin("dict?")
+def is_dict(x):
+    return isinstance(x, dict)
+
+
+@builtin("list?")
+def is_list(x):
+    return isinstance(x, list)
+
+
+@builtin("nil?")
+def is_nil(x):
+    return x is None
+
+
+@builtin("number?")
+def is_number(x):
+    return isinstance(x, (int, float))
+
+
+@builtin("string?")
+def is_string(x):
+    return isinstance(x, str)
+
+
+@builtin("len")
+def length(coll):
+    return len(coll)
+
+
+@builtin("first")
+def first(coll):
+    return coll[0] if coll else None
+
+
+@builtin("last")
+def last(coll):
+    return coll[-1] if coll else None
+
+
+@builtin("chunk-every")
+def chunk_every(coll, n):
+    """Split collection into chunks of n elements."""
+    n = int(n)
+    return [coll[i:i+n] for i in range(0, len(coll), n)]
+
+
+@builtin("rest")
+def rest(coll):
+    return coll[1:] if coll else []
+
+
+@builtin("nth")
+def nth(coll, n):
+    return coll[n] if 0 <= n < len(coll) else None
+
+
+@builtin("concat")
+def concat(*colls):
+    """Concatenate multiple lists/sequences."""
+    result = []
+    for c in colls:
+        if c is not None:
+            result.extend(c)
+    return result
+
+
+@builtin("cons")
+def cons(x, coll):
+    """Prepend x to collection."""
+    return [x] + list(coll) if coll else [x]
+
+
+@builtin("append")
+def append(coll, x):
+    """Append x to collection."""
+    return list(coll) + [x] if coll else [x]
+
+
+@builtin("range")
+def make_range(start, end, step=1):
+    """Create a range of numbers."""
+    return list(range(int(start), int(end), int(step)))
+
+
+@builtin("zip-pairs")
+def zip_pairs(coll):
+    """Zip consecutive pairs: [a,b,c,d] -> [[a,b],[b,c],[c,d]]."""
+    if not coll or len(coll) < 2:
+        return []
+    return [[coll[i], coll[i+1]] for i in range(len(coll)-1)]
+
+
+@builtin("dict")
+def make_dict(*pairs):
+    """Create dict from key-value pairs: (dict :a 1 :b 2)."""
+    result = {}
+    i = 0
+    while i < len(pairs) - 1:
+        key = pairs[i]
+        if isinstance(key, Keyword):
+            key = key.name
+        result[key] = pairs[i + 1]
+        i += 2
+    return result
+
+
+@builtin("keys")
+def keys(d):
+    """Get the keys of a dict as a list."""
+    if not isinstance(d, dict):
+        raise EvalError(f"keys: expected dict, got {type(d).__name__}")
+    return list(d.keys())
+
+
+@builtin("vals")
+def vals(d):
+    """Get the values of a dict as a list."""
+    if not isinstance(d, dict):
+        raise EvalError(f"vals: expected dict, got {type(d).__name__}")
+    return list(d.values())
+
+
+@builtin("merge")
+def merge(*dicts):
+    """Merge multiple dicts, later dicts override earlier."""
+    result = {}
+    for d in dicts:
+        if d is not None:
+            if not isinstance(d, dict):
+                raise EvalError(f"merge: expected dict, got {type(d).__name__}")
+            result.update(d)
+    return result
+
+
+@builtin("assoc")
+def assoc(d, *pairs):
+    """Associate keys with values in a dict: (assoc d :a 1 :b 2)."""
+    if d is None:
+        result = {}
+    elif isinstance(d, dict):
+        result = dict(d)
+    else:
+        raise EvalError(f"assoc: expected dict or nil, got {type(d).__name__}")
+
+    i = 0
+    while i < len(pairs) - 1:
+        key = pairs[i]
+        if isinstance(key, Keyword):
+            key = key.name
+        result[key] = pairs[i + 1]
+        i += 2
+    return result
+
+
+@builtin("dissoc")
+def dissoc(d, *keys_to_remove):
+    """Remove keys from a dict: (dissoc d :a :b)."""
+    if d is None:
+        return {}
+    if not isinstance(d, dict):
+        raise EvalError(f"dissoc: expected dict or nil, got {type(d).__name__}")
+
+    result = dict(d)
+    for key in keys_to_remove:
+        if isinstance(key, Keyword):
+            key = key.name
+        result.pop(key, None)
+    return result
+
+
+@builtin("into")
+def into(target, coll):
+    """Convert a collection into another collection type.
+
+    (into [] {:a 1 :b 2}) -> [["a" 1] ["b" 2]]
+    (into {} [[:a 1] [:b 2]]) -> {"a": 1, "b": 2}
+    (into [] [1 2 3]) -> [1 2 3]
+    """
+    if isinstance(target, list):
+        if isinstance(coll, dict):
+            return [[k, v] for k, v in coll.items()]
+        elif isinstance(coll, (list, tuple)):
+            return list(coll)
+        else:
+            raise EvalError(f"into: cannot convert {type(coll).__name__} into list")
+    elif isinstance(target, dict):
+        if isinstance(coll, dict):
+            return dict(coll)
+        elif isinstance(coll, (list, tuple)):
+            result = {}
+            for item in coll:
+                if isinstance(item, (list, tuple)) and len(item) >= 2:
+                    key = item[0]
+                    if isinstance(key, Keyword):
+                        key = key.name
+                    result[key] = item[1]
+                else:
+                    raise EvalError(f"into: expected [key value] pairs, got {item}")
+            return result
+        else:
+            raise EvalError(f"into: cannot convert {type(coll).__name__} into dict")
+    else:
+        raise EvalError(f"into: unsupported target type {type(target).__name__}")
+
+
+@builtin("filter")
+def filter_fn(pred, coll):
+    """Filter collection by predicate. Pred must be a lambda."""
+    if not isinstance(pred, Lambda):
+        raise EvalError(f"filter: expected lambda as predicate, got {type(pred).__name__}")
+
+    result = []
+    for item in coll:
+        # Evaluate predicate with item
+        local_env = {}
+        if pred.closure:
+            local_env.update(pred.closure)
+        local_env[pred.params[0]] = item
+
+        # Inline evaluation of pred.body
+        from . import evaluator
+        if evaluator.evaluate(pred.body, local_env):
+            result.append(item)
+    return result
+
+
+@builtin("some")
+def some(pred, coll):
+    """Return first truthy value of (pred item) for items in coll, or nil."""
+    if not isinstance(pred, Lambda):
+        raise EvalError(f"some: expected lambda as predicate, got {type(pred).__name__}")
+
+    for item in coll:
+        local_env = {}
+        if pred.closure:
+            local_env.update(pred.closure)
+        local_env[pred.params[0]] = item
+
+        from . import evaluator
+        result = evaluator.evaluate(pred.body, local_env)
+        if result:
+            return result
+    return None
+
+
+@builtin("every?")
+def every(pred, coll):
+    """Return true if (pred item) is truthy for all items in coll."""
+    if not isinstance(pred, Lambda):
+        raise EvalError(f"every?: expected lambda as predicate, got {type(pred).__name__}")
+
+    for item in coll:
+        local_env = {}
+        if pred.closure:
+            local_env.update(pred.closure)
+        local_env[pred.params[0]] = item
+
+        from . import evaluator
+        if not evaluator.evaluate(pred.body, local_env):
+            return False
+    return True
+
+
+@builtin("empty?")
+def is_empty(coll):
+    """Return true if collection is empty."""
+    if coll is None:
+        return True
+    return len(coll) == 0
+
+
+@builtin("contains?")
+def contains(coll, key):
+    """Check if collection contains key (for dicts) or element (for lists)."""
+    if isinstance(coll, dict):
+        if isinstance(key, Keyword):
+            key = key.name
+        return key in coll
+    elif isinstance(coll, (list, tuple)):
+        return key in coll
+    return False
+
+
+def evaluate(expr: Any, env: Dict[str, Any] = None) -> Any:
+    """
+    Evaluate an S-expression in the given environment.
+
+    Args:
+        expr: The expression to evaluate
+        env: Variable bindings (name -> value)
+
+    Returns:
+        The result of evaluation
+    """
+    if env is None:
+        env = {}
+
+    # Literals
+    if isinstance(expr, (int, float, str, bool)) or expr is None:
+        return expr
+
+    # Symbol - variable lookup
+    if isinstance(expr, Symbol):
+        name = expr.name
+        if name in env:
+            return env[name]
+        if name in BUILTINS:
+            return BUILTINS[name]
+        if name == "true":
+            return True
+        if name == "false":
+            return False
+        if name == "nil":
+            return None
+        raise EvalError(f"Undefined symbol: {name}")
+
+    # Keyword - return as-is (used as map keys)
+    if isinstance(expr, Keyword):
+        return expr.name
+
+    # Lambda - return as-is (it's a value)
+    if isinstance(expr, Lambda):
+        return expr
+
+    # Binding - return as-is (resolved at execution time)
+    if isinstance(expr, Binding):
+        return expr
+
+    # Dict literal
+    if isinstance(expr, dict):
+        return {k: evaluate(v, env) for k, v in expr.items()}
+
+    # List - function call or special form
+    if isinstance(expr, list):
+        if not expr:
+            return []
+
+        head = expr[0]
+
+        # If head is a string/number/etc (not Symbol), treat as data list
+        if not isinstance(head, (Symbol, Lambda, list)):
+            return [evaluate(x, env) for x in expr]
+
+        # Special forms
+        if isinstance(head, Symbol):
+            name = head.name
+
+            # if - conditional
+            if name == "if":
+                if len(expr) < 3:
+                    raise EvalError("if requires condition and then-branch")
+                cond_result = evaluate(expr[1], env)
+                if cond_result:
+                    return evaluate(expr[2], env)
+                elif len(expr) > 3:
+                    return evaluate(expr[3], env)
+                return None
+
+            # cond - multi-way conditional
+            # Supports both Clojure style: (cond test1 result1 test2 result2 :else default)
+            # and Scheme style: (cond (test1 result1) (test2 result2) (else default))
+            if name == "cond":
+                clauses = expr[1:]
+                # Check if Clojure style (flat list) or Scheme style (nested pairs)
+                # Scheme style: first clause is (test result) - exactly 2 elements
+                # Clojure style: first clause is a test expression like (= x 0) - 3+ elements
+                first_is_scheme_clause = (
+                    clauses and
+                    isinstance(clauses[0], list) and
+                    len(clauses[0]) == 2 and
+                    not (isinstance(clauses[0][0], Symbol) and clauses[0][0].name in ('=', '<', '>', '<=', '>=', '!=', 'not=', 'and', 'or'))
+                )
+                if first_is_scheme_clause:
+                    # Scheme style: ((test result) ...)
+                    for clause in clauses:
+                        if not isinstance(clause, list) or len(clause) < 2:
+                            raise EvalError("cond clause must be (test result)")
+                        test = clause[0]
+                        # Check for else/default
+                        if isinstance(test, Symbol) and test.name in ("else", ":else"):
+                            return evaluate(clause[1], env)
+                        if isinstance(test, Keyword) and test.name == "else":
+                            return evaluate(clause[1], env)
+                        if evaluate(test, env):
+                            return evaluate(clause[1], env)
+                else:
+                    # Clojure style: test1 result1 test2 result2 ...
+                    i = 0
+                    while i < len(clauses) - 1:
+                        test = clauses[i]
+                        result = clauses[i + 1]
+                        # Check for :else
+                        if isinstance(test, Keyword) and test.name == "else":
+                            return evaluate(result, env)
+                        if isinstance(test, Symbol) and test.name == ":else":
+                            return evaluate(result, env)
+                        if evaluate(test, env):
+                            return evaluate(result, env)
+                        i += 2
+                return None
+
+            # case - switch on value
+            # (case expr val1 result1 val2 result2 :else default)
+            if name == "case":
+                if len(expr) < 2:
+                    raise EvalError("case requires expression to match")
+                match_val = evaluate(expr[1], env)
+                clauses = expr[2:]
+                i = 0
+                while i < len(clauses) - 1:
+                    test = clauses[i]
+                    result = clauses[i + 1]
+                    # Check for :else / else
+                    if isinstance(test, Keyword) and test.name == "else":
+                        return evaluate(result, env)
+                    if isinstance(test, Symbol) and test.name in (":else", "else"):
+                        return evaluate(result, env)
+                    # Evaluate test value and compare
+                    test_val = evaluate(test, env)
+                    if match_val == test_val:
+                        return evaluate(result, env)
+                    i += 2
+                return None
+
+            # and - short-circuit
+            if name == "and":
+                result = True
+                for arg in expr[1:]:
+                    result = evaluate(arg, env)
+                    if not result:
+                        return result
+                return result
+
+            # or - short-circuit
+            if name == "or":
+                result = False
+                for arg in expr[1:]:
+                    result = evaluate(arg, env)
+                    if result:
+                        return result
+                return result
+
+            # let and let* - local bindings (both bind sequentially in this impl)
+            if name in ("let", "let*"):
+                if len(expr) < 3:
+                    raise EvalError(f"{name} requires bindings and body")
+                bindings = expr[1]
+
+                local_env = dict(env)
+
+                if isinstance(bindings, list):
+                    # Check if it's ((name value) ...) style (Lisp let* style)
+                    if bindings and isinstance(bindings[0], list):
+                        for binding in bindings:
+                            if len(binding) != 2:
+                                raise EvalError(f"{name} binding must be (name value)")
+                            var_name = binding[0]
+                            if isinstance(var_name, Symbol):
+                                var_name = var_name.name
+                            value = evaluate(binding[1], local_env)
+                            local_env[var_name] = value
+                    # Vector-style [name value ...]
+                    elif len(bindings) % 2 == 0:
+                        for i in range(0, len(bindings), 2):
+                            var_name = bindings[i]
+                            if isinstance(var_name, Symbol):
+                                var_name = var_name.name
+                            value = evaluate(bindings[i + 1], local_env)
+                            local_env[var_name] = value
+                    else:
+                        raise EvalError(f"{name} bindings must be [name value ...] or ((name value) ...)")
+                else:
+                    raise EvalError(f"{name} bindings must be a list")
+
+                return evaluate(expr[2], local_env)
+
+            # lambda / fn - create function with closure
+            if name in ("lambda", "fn"):
+                if len(expr) < 3:
+                    raise EvalError("lambda requires params and body")
+                params = expr[1]
+                if not isinstance(params, list):
+                    raise EvalError("lambda params must be a list")
+                param_names = []
+                for p in params:
+                    if isinstance(p, Symbol):
+                        param_names.append(p.name)
+                    elif isinstance(p, str):
+                        param_names.append(p)
+                    else:
+                        raise EvalError(f"Invalid param: {p}")
+                # Capture current environment as closure
+                return Lambda(param_names, expr[2], dict(env))
+
+            # quote - return unevaluated
+            if name == "quote":
+                return expr[1] if len(expr) > 1 else None
+
+            # bind - create binding to analysis data
+            # (bind analysis-var)
+            # (bind analysis-var :range [0.3 1.0])
+            # (bind analysis-var :range [0 100] :transform sqrt)
+            if name == "bind":
+                if len(expr) < 2:
+                    raise EvalError("bind requires analysis reference")
+                analysis_ref = expr[1]
+                if isinstance(analysis_ref, Symbol):
+                    symbol_name = analysis_ref.name
+                    # Look up the symbol in environment
+                    if symbol_name in env:
+                        resolved = env[symbol_name]
+                        # If resolved is actual analysis data (dict with times/values or
+                        # S-expression list with Keywords), keep the symbol name as reference
+                        # for later lookup at execution time
+                        if isinstance(resolved, dict) and ("times" in resolved or "values" in resolved):
+                            analysis_ref = symbol_name  # Use name as reference, not the data
+                        elif isinstance(resolved, list) and any(isinstance(x, Keyword) for x in resolved):
+                            # Parsed S-expression analysis data ([:times [...] :duration ...])
+                            analysis_ref = symbol_name
+                        else:
+                            analysis_ref = resolved
+                    else:
+                        raise EvalError(f"bind: undefined symbol '{symbol_name}' - must reference analysis data")
+
+                # Parse optional :range [min max] and :transform
+                range_min, range_max = 0.0, 1.0
+                transform = None
+                i = 2
+                while i < len(expr):
+                    if isinstance(expr[i], Keyword):
+                        kw = expr[i].name
+                        if kw == "range" and i + 1 < len(expr):
+                            range_val = evaluate(expr[i + 1], env)  # Evaluate to get actual value
+                            if isinstance(range_val, list) and len(range_val) >= 2:
+                                range_min = float(range_val[0])
+                                range_max = float(range_val[1])
+                            i += 2
+                        elif kw == "transform" and i + 1 < len(expr):
+                            t = expr[i + 1]
+                            if isinstance(t, Symbol):
+                                transform = t.name
+                            elif isinstance(t, str):
+                                transform = t
+                            i += 2
+                        else:
+                            i += 1
+                    else:
+                        i += 1
+
+                return Binding(analysis_ref, range_min=range_min, range_max=range_max, transform=transform)
+
+            # Vector literal [a b c]
+            if name == "vec" or name == "vector":
+                return [evaluate(e, env) for e in expr[1:]]
+
+            # map - (map fn coll)
+            if name == "map":
+                if len(expr) != 3:
+                    raise EvalError("map requires fn and collection")
+                fn = evaluate(expr[1], env)
+                coll = evaluate(expr[2], env)
+                if not isinstance(fn, Lambda):
+                    raise EvalError(f"map requires lambda, got {type(fn)}")
+                result = []
+                for item in coll:
+                    local_env = {}
+                    if fn.closure:
+                        local_env.update(fn.closure)
+                    local_env.update(env)
+                    local_env[fn.params[0]] = item
+                    result.append(evaluate(fn.body, local_env))
+                return result
+
+            # map-indexed - (map-indexed fn coll)
+            if name == "map-indexed":
+                if len(expr) != 3:
+                    raise EvalError("map-indexed requires fn and collection")
+                fn = evaluate(expr[1], env)
+                coll = evaluate(expr[2], env)
+                if not isinstance(fn, Lambda):
+                    raise EvalError(f"map-indexed requires lambda, got {type(fn)}")
+                if len(fn.params) < 2:
+                    raise EvalError("map-indexed lambda needs (i item) params")
+                result = []
+                for i, item in enumerate(coll):
+                    local_env = {}
+                    if fn.closure:
+                        local_env.update(fn.closure)
+                    local_env.update(env)
+                    local_env[fn.params[0]] = i
+                    local_env[fn.params[1]] = item
+                    result.append(evaluate(fn.body, local_env))
+                return result
+
+            # reduce - (reduce fn init coll)
+            if name == "reduce":
+                if len(expr) != 4:
+                    raise EvalError("reduce requires fn, init, and collection")
+                fn = evaluate(expr[1], env)
+                acc = evaluate(expr[2], env)
+                coll = evaluate(expr[3], env)
+                if not isinstance(fn, Lambda):
+                    raise EvalError(f"reduce requires lambda, got {type(fn)}")
+                if len(fn.params) < 2:
+                    raise EvalError("reduce lambda needs (acc item) params")
+                for item in coll:
+                    local_env = {}
+                    if fn.closure:
+                        local_env.update(fn.closure)
+                    local_env.update(env)
+                    local_env[fn.params[0]] = acc
+                    local_env[fn.params[1]] = item
+                    acc = evaluate(fn.body, local_env)
+                return acc
+
+            # for-each - (for-each fn coll) - iterate with side effects
+            if name == "for-each":
+                if len(expr) != 3:
+                    raise EvalError("for-each requires fn and collection")
+                fn = evaluate(expr[1], env)
+                coll = evaluate(expr[2], env)
+                if not isinstance(fn, Lambda):
+                    raise EvalError(f"for-each requires lambda, got {type(fn)}")
+                for item in coll:
+                    local_env = {}
+                    if fn.closure:
+                        local_env.update(fn.closure)
+                    local_env.update(env)
+                    local_env[fn.params[0]] = item
+                    evaluate(fn.body, local_env)
+                return None
+
+        # Function call
+        fn = evaluate(head, env)
+        args = [evaluate(arg, env) for arg in expr[1:]]
+
+        # Call builtin
+        if callable(fn):
+            return fn(*args)
+
+        # Call lambda
+        if isinstance(fn, Lambda):
+            if len(args) != len(fn.params):
+                raise EvalError(f"Lambda expects {len(fn.params)} args, got {len(args)}")
+            # Start with closure (captured env), then overlay calling env, then params
+            local_env = {}
+            if fn.closure:
+                local_env.update(fn.closure)
+            local_env.update(env)
+            for name, value in zip(fn.params, args):
+                local_env[name] = value
+            return evaluate(fn.body, local_env)
+
+        raise EvalError(f"Not callable: {fn}")
+
+    raise EvalError(f"Cannot evaluate: {expr!r}")
+
+
+def make_env(**kwargs) -> Dict[str, Any]:
+    """Create an environment with initial bindings."""
+    return dict(kwargs)

artdag/core/artdag/sexp/external_tools.py

@@ -0,0 +1,292 @@
+"""
+External tool runners for effects that can't be done in FFmpeg.
+
+Supports:
+- datamosh: via ffglitch or datamoshing Python CLI
+- pixelsort: via Rust pixelsort or Python pixelsort-cli
+"""
+
+import shutil
+import subprocess
+import tempfile
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple
+
+
+def find_tool(tool_names: List[str]) -> Optional[str]:
+    """Find first available tool from a list of candidates."""
+    for name in tool_names:
+        path = shutil.which(name)
+        if path:
+            return path
+    return None
+
+
+def check_python_package(package: str) -> bool:
+    """Check if a Python package is installed."""
+    try:
+        result = subprocess.run(
+            ["python3", "-c", f"import {package}"],
+            capture_output=True,
+            timeout=5,
+        )
+        return result.returncode == 0
+    except Exception:
+        return False
+
+
+# Tool detection
+DATAMOSH_TOOLS = ["ffgac", "ffedit"]  # ffglitch tools
+PIXELSORT_TOOLS = ["pixelsort"]  # Rust CLI
+
+
+def get_available_tools() -> Dict[str, Optional[str]]:
+    """Get dictionary of available external tools."""
+    return {
+        "datamosh": find_tool(DATAMOSH_TOOLS),
+        "pixelsort": find_tool(PIXELSORT_TOOLS),
+        "datamosh_python": "datamoshing" if check_python_package("datamoshing") else None,
+        "pixelsort_python": "pixelsort" if check_python_package("pixelsort") else None,
+    }
+
+
+def run_datamosh(
+    input_path: Path,
+    output_path: Path,
+    params: Dict[str, Any],
+) -> Tuple[bool, str]:
+    """
+    Run datamosh effect using available tool.
+
+    Args:
+        input_path: Input video file
+        output_path: Output video file
+        params: Effect parameters (corruption, block_size, etc.)
+
+    Returns:
+        (success, error_message)
+    """
+    tools = get_available_tools()
+
+    corruption = params.get("corruption", 0.3)
+
+    # Try ffglitch first
+    if tools.get("datamosh"):
+        ffgac = tools["datamosh"]
+        try:
+            # ffglitch approach: remove I-frames to create datamosh effect
+            # This is a simplified version - full datamosh needs custom scripts
+            with tempfile.NamedTemporaryFile(mode='w', suffix='.js', delete=False) as f:
+                # Write a simple ffglitch script that corrupts motion vectors
+                f.write(f"""
+// Datamosh script - corrupt motion vectors
+let corruption = {corruption};
+
+export function glitch_frame(frame, stream) {{
+    if (frame.pict_type === 'P' && Math.random() < corruption) {{
+        // Corrupt motion vectors
+        let dominated = frame.mv?.forward?.overflow;
+        if (dominated) {{
+            for (let i = 0; i < dominated.length; i++) {{
+                if (Math.random() < corruption) {{
+                    dominated[i] = [
+                        Math.floor(Math.random() * 64 - 32),
+                        Math.floor(Math.random() * 64 - 32)
+                    ];
+                }}
+            }}
+        }}
+    }}
+    return frame;
+}}
+""")
+                script_path = f.name
+
+            cmd = [
+                ffgac,
+                "-i", str(input_path),
+                "-s", script_path,
+                "-o", str(output_path),
+            ]
+            result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
+            Path(script_path).unlink(missing_ok=True)
+
+            if result.returncode == 0:
+                return True, ""
+            return False, result.stderr[:500]
+
+        except subprocess.TimeoutExpired:
+            return False, "Datamosh timeout"
+        except Exception as e:
+            return False, str(e)
+
+    # Fall back to Python datamoshing package
+    if tools.get("datamosh_python"):
+        try:
+            cmd = [
+                "python3", "-m", "datamoshing",
+                str(input_path),
+                str(output_path),
+                "--mode", "iframe_removal",
+            ]
+            result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
+            if result.returncode == 0:
+                return True, ""
+            return False, result.stderr[:500]
+        except Exception as e:
+            return False, str(e)
+
+    return False, "No datamosh tool available. Install ffglitch or: pip install datamoshing"
+
+
+def run_pixelsort(
+    input_path: Path,
+    output_path: Path,
+    params: Dict[str, Any],
+) -> Tuple[bool, str]:
+    """
+    Run pixelsort effect using available tool.
+
+    Args:
+        input_path: Input image/frame file
+        output_path: Output image file
+        params: Effect parameters (sort_by, threshold_low, threshold_high, angle)
+
+    Returns:
+        (success, error_message)
+    """
+    tools = get_available_tools()
+
+    sort_by = params.get("sort_by", "lightness")
+    threshold_low = params.get("threshold_low", 50)
+    threshold_high = params.get("threshold_high", 200)
+    angle = params.get("angle", 0)
+
+    # Try Rust pixelsort first (faster)
+    if tools.get("pixelsort"):
+        try:
+            cmd = [
+                tools["pixelsort"],
+                str(input_path),
+                "-o", str(output_path),
+                "--sort", sort_by,
+                "-r", str(angle),
+            ]
+            result = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
+            if result.returncode == 0:
+                return True, ""
+            return False, result.stderr[:500]
+        except Exception as e:
+            return False, str(e)
+
+    # Fall back to Python pixelsort-cli
+    if tools.get("pixelsort_python"):
+        try:
+            cmd = [
+                "python3", "-m", "pixelsort",
+                "--image_path", str(input_path),
+                "--output", str(output_path),
+                "--angle", str(angle),
+                "--threshold", str(threshold_low / 255),  # Normalize to 0-1
+            ]
+            result = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
+            if result.returncode == 0:
+                return True, ""
+            return False, result.stderr[:500]
+        except Exception as e:
+            return False, str(e)
+
+    return False, "No pixelsort tool available. Install: cargo install pixelsort or pip install pixelsort-cli"
+
+
+def run_pixelsort_video(
+    input_path: Path,
+    output_path: Path,
+    params: Dict[str, Any],
+    fps: float = 30,
+) -> Tuple[bool, str]:
+    """
+    Run pixelsort on a video by processing each frame.
+
+    This extracts frames, processes them, then reassembles.
+    """
+    with tempfile.TemporaryDirectory() as tmpdir:
+        tmpdir = Path(tmpdir)
+        frames_in = tmpdir / "frame_%04d.png"
+        frames_out = tmpdir / "sorted_%04d.png"
+
+        # Extract frames
+        extract_cmd = [
+            "ffmpeg", "-y",
+            "-i", str(input_path),
+            "-vf", f"fps={fps}",
+            str(frames_in),
+        ]
+        result = subprocess.run(extract_cmd, capture_output=True, timeout=300)
+        if result.returncode != 0:
+            return False, "Failed to extract frames"
+
+        # Process each frame
+        frame_files = sorted(tmpdir.glob("frame_*.png"))
+        for i, frame in enumerate(frame_files):
+            out_frame = tmpdir / f"sorted_{i:04d}.png"
+            success, error = run_pixelsort(frame, out_frame, params)
+            if not success:
+                return False, f"Frame {i}: {error}"
+
+        # Reassemble
+        # Get audio from original
+        reassemble_cmd = [
+            "ffmpeg", "-y",
+            "-framerate", str(fps),
+            "-i", str(tmpdir / "sorted_%04d.png"),
+            "-i", str(input_path),
+            "-map", "0:v", "-map", "1:a?",
+            "-c:v", "libx264", "-preset", "fast",
+            "-c:a", "copy",
+            str(output_path),
+        ]
+        result = subprocess.run(reassemble_cmd, capture_output=True, timeout=300)
+        if result.returncode != 0:
+            return False, "Failed to reassemble video"
+
+        return True, ""
+
+
+def run_external_effect(
+    effect_name: str,
+    input_path: Path,
+    output_path: Path,
+    params: Dict[str, Any],
+    is_video: bool = True,
+) -> Tuple[bool, str]:
+    """
+    Run an external effect tool.
+
+    Args:
+        effect_name: Name of effect (datamosh, pixelsort)
+        input_path: Input file
+        output_path: Output file
+        params: Effect parameters
+        is_video: Whether input is video (vs single image)
+
+    Returns:
+        (success, error_message)
+    """
+    if effect_name == "datamosh":
+        return run_datamosh(input_path, output_path, params)
+    elif effect_name == "pixelsort":
+        if is_video:
+            return run_pixelsort_video(input_path, output_path, params)
+        else:
+            return run_pixelsort(input_path, output_path, params)
+    else:
+        return False, f"Unknown external effect: {effect_name}"
+
+
+if __name__ == "__main__":
+    # Print available tools
+    print("Available external tools:")
+    for name, path in get_available_tools().items():
+        status = path if path else "NOT INSTALLED"
+        print(f"  {name}: {status}")

artdag/core/artdag/sexp/ffmpeg_compiler.py

@@ -0,0 +1,616 @@
+"""
+FFmpeg filter compiler for sexp effects.
+
+Compiles sexp effect definitions to FFmpeg filter expressions,
+with support for dynamic parameters via sendcmd scripts.
+
+Usage:
+    compiler = FFmpegCompiler()
+
+    # Compile an effect with static params
+    filter_str = compiler.compile_effect("brightness", {"amount": 50})
+    # -> "eq=brightness=0.196"
+
+    # Compile with dynamic binding to analysis data
+    filter_str, sendcmd = compiler.compile_effect_with_binding(
+        "brightness",
+        {"amount": {"_bind": "bass-data", "range_min": 0, "range_max": 100}},
+        analysis_data={"bass-data": {"times": [...], "values": [...]}},
+        segment_start=0.0,
+        segment_duration=5.0,
+    )
+    # -> ("eq=brightness=0.5", "0.0 [eq] brightness 0.5;\n0.05 [eq] brightness 0.6;...")
+"""
+
+import math
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple
+
+
+# FFmpeg filter mappings for common effects
+# Maps effect name -> {filter: str, params: {param_name: {ffmpeg_param, scale, offset}}}
+EFFECT_MAPPINGS = {
+    "invert": {
+        "filter": "negate",
+        "params": {},
+    },
+    "grayscale": {
+        "filter": "colorchannelmixer",
+        "static": "0.3:0.4:0.3:0:0.3:0.4:0.3:0:0.3:0.4:0.3",
+        "params": {},
+    },
+    "sepia": {
+        "filter": "colorchannelmixer",
+        "static": "0.393:0.769:0.189:0:0.349:0.686:0.168:0:0.272:0.534:0.131",
+        "params": {},
+    },
+    "brightness": {
+        "filter": "eq",
+        "params": {
+            "amount": {"ffmpeg_param": "brightness", "scale": 1/255, "offset": 0},
+        },
+    },
+    "contrast": {
+        "filter": "eq",
+        "params": {
+            "amount": {"ffmpeg_param": "contrast", "scale": 1.0, "offset": 0},
+        },
+    },
+    "saturation": {
+        "filter": "eq",
+        "params": {
+            "amount": {"ffmpeg_param": "saturation", "scale": 1.0, "offset": 0},
+        },
+    },
+    "hue_shift": {
+        "filter": "hue",
+        "params": {
+            "degrees": {"ffmpeg_param": "h", "scale": 1.0, "offset": 0},
+        },
+    },
+    "blur": {
+        "filter": "gblur",
+        "params": {
+            "radius": {"ffmpeg_param": "sigma", "scale": 1.0, "offset": 0},
+        },
+    },
+    "sharpen": {
+        "filter": "unsharp",
+        "params": {
+            "amount": {"ffmpeg_param": "la", "scale": 1.0, "offset": 0},
+        },
+    },
+    "pixelate": {
+        # Scale down then up to create pixelation effect
+        "filter": "scale",
+        "static": "iw/8:ih/8:flags=neighbor,scale=iw*8:ih*8:flags=neighbor",
+        "params": {},
+    },
+    "vignette": {
+        "filter": "vignette",
+        "params": {
+            "strength": {"ffmpeg_param": "a", "scale": 1.0, "offset": 0},
+        },
+    },
+    "noise": {
+        "filter": "noise",
+        "params": {
+            "amount": {"ffmpeg_param": "alls", "scale": 1.0, "offset": 0},
+        },
+    },
+    "flip": {
+        "filter": "hflip",  # Default horizontal
+        "params": {},
+    },
+    "mirror": {
+        "filter": "hflip",
+        "params": {},
+    },
+    "rotate": {
+        "filter": "rotate",
+        "params": {
+            "angle": {"ffmpeg_param": "a", "scale": math.pi/180, "offset": 0},  # degrees to radians
+        },
+    },
+    "zoom": {
+        "filter": "zoompan",
+        "params": {
+            "factor": {"ffmpeg_param": "z", "scale": 1.0, "offset": 0},
+        },
+    },
+    "posterize": {
+        # Use lutyuv to quantize levels (approximate posterization)
+        "filter": "lutyuv",
+        "static": "y=floor(val/32)*32:u=floor(val/32)*32:v=floor(val/32)*32",
+        "params": {},
+    },
+    "threshold": {
+        # Use geq for thresholding
+        "filter": "geq",
+        "static": "lum='if(gt(lum(X,Y),128),255,0)':cb=128:cr=128",
+        "params": {},
+    },
+    "edge_detect": {
+        "filter": "edgedetect",
+        "params": {
+            "low": {"ffmpeg_param": "low", "scale": 1/255, "offset": 0},
+            "high": {"ffmpeg_param": "high", "scale": 1/255, "offset": 0},
+        },
+    },
+    "swirl": {
+        "filter": "lenscorrection",  # Approximate with lens distortion
+        "params": {
+            "strength": {"ffmpeg_param": "k1", "scale": 0.1, "offset": 0},
+        },
+    },
+    "fisheye": {
+        "filter": "lenscorrection",
+        "params": {
+            "strength": {"ffmpeg_param": "k1", "scale": 1.0, "offset": 0},
+        },
+    },
+    "wave": {
+        # Wave displacement using geq - need r/g/b for RGB mode
+        "filter": "geq",
+        "static": "r='r(X+10*sin(Y/20),Y)':g='g(X+10*sin(Y/20),Y)':b='b(X+10*sin(Y/20),Y)'",
+        "params": {},
+    },
+    "rgb_split": {
+        # Chromatic aberration using geq
+        "filter": "geq",
+        "static": "r='p(X+5,Y)':g='p(X,Y)':b='p(X-5,Y)'",
+        "params": {},
+    },
+    "scanlines": {
+        "filter": "drawgrid",
+        "params": {
+            "spacing": {"ffmpeg_param": "h", "scale": 1.0, "offset": 0},
+        },
+    },
+    "film_grain": {
+        "filter": "noise",
+        "params": {
+            "intensity": {"ffmpeg_param": "alls", "scale": 100, "offset": 0},
+        },
+    },
+    "crt": {
+        "filter": "vignette",  # Simplified - just vignette for CRT look
+        "params": {},
+    },
+    "bloom": {
+        "filter": "gblur",  # Simplified bloom = blur overlay
+        "params": {
+            "radius": {"ffmpeg_param": "sigma", "scale": 1.0, "offset": 0},
+        },
+    },
+    "color_cycle": {
+        "filter": "hue",
+        "params": {
+            "speed": {"ffmpeg_param": "h", "scale": 360.0, "offset": 0, "time_expr": True},
+        },
+        "time_based": True,  # Uses time expression
+    },
+    "strobe": {
+        # Strobe using select to drop frames
+        "filter": "select",
+        "static": "'mod(n,4)'",
+        "params": {},
+    },
+    "echo": {
+        # Echo using tmix
+        "filter": "tmix",
+        "static": "frames=4:weights='1 0.5 0.25 0.125'",
+        "params": {},
+    },
+    "trails": {
+        # Trails using tblend
+        "filter": "tblend",
+        "static": "all_mode=average",
+        "params": {},
+    },
+    "kaleidoscope": {
+        # 4-way mirror kaleidoscope using FFmpeg filter chain
+        # Crops top-left quadrant, mirrors horizontally, then vertically
+        "filter": "crop",
+        "complex": True,
+        "static": "iw/2:ih/2:0:0[q];[q]split[q1][q2];[q1]hflip[qr];[q2][qr]hstack[top];[top]split[t1][t2];[t2]vflip[bot];[t1][bot]vstack",
+        "params": {},
+    },
+    "emboss": {
+        "filter": "convolution",
+        "static": "-2 -1 0 -1 1 1 0 1 2:-2 -1 0 -1 1 1 0 1 2:-2 -1 0 -1 1 1 0 1 2:-2 -1 0 -1 1 1 0 1 2",
+        "params": {},
+    },
+    "neon_glow": {
+        # Edge detect + negate for neon-like effect
+        "filter": "edgedetect",
+        "static": "mode=colormix:high=0.1",
+        "params": {},
+    },
+    "ascii_art": {
+        # Requires Python frame processing - no FFmpeg equivalent
+        "filter": None,
+        "python_primitive": "ascii_art_frame",
+        "params": {
+            "char_size": {"default": 8},
+            "alphabet": {"default": "standard"},
+            "color_mode": {"default": "color"},
+        },
+    },
+    "ascii_zones": {
+        # Requires Python frame processing - zone-based ASCII
+        "filter": None,
+        "python_primitive": "ascii_zones_frame",
+        "params": {
+            "char_size": {"default": 8},
+            "zone_threshold": {"default": 128},
+        },
+    },
+    "datamosh": {
+        # External tool: ffglitch or datamoshing CLI, falls back to Python
+        "filter": None,
+        "external_tool": "datamosh",
+        "python_primitive": "datamosh_frame",
+        "params": {
+            "block_size": {"default": 32},
+            "corruption": {"default": 0.3},
+        },
+    },
+    "pixelsort": {
+        # External tool: pixelsort CLI (Rust or Python), falls back to Python
+        "filter": None,
+        "external_tool": "pixelsort",
+        "python_primitive": "pixelsort_frame",
+        "params": {
+            "sort_by": {"default": "lightness"},
+            "threshold_low": {"default": 50},
+            "threshold_high": {"default": 200},
+            "angle": {"default": 0},
+        },
+    },
+    "ripple": {
+        # Use geq for ripple displacement
+        "filter": "geq",
+        "static": "lum='lum(X+5*sin(hypot(X-W/2,Y-H/2)/10),Y+5*cos(hypot(X-W/2,Y-H/2)/10))'",
+        "params": {},
+    },
+    "tile_grid": {
+        # Use tile filter for grid
+        "filter": "tile",
+        "static": "2x2",
+        "params": {},
+    },
+    "outline": {
+        "filter": "edgedetect",
+        "params": {},
+    },
+    "color-adjust": {
+        "filter": "eq",
+        "params": {
+            "brightness": {"ffmpeg_param": "brightness", "scale": 1/255, "offset": 0},
+            "contrast": {"ffmpeg_param": "contrast", "scale": 1.0, "offset": 0},
+        },
+    },
+}
+
+
+class FFmpegCompiler:
+    """Compiles sexp effects to FFmpeg filters with sendcmd support."""
+
+    def __init__(self, effect_mappings: Dict = None):
+        self.mappings = effect_mappings or EFFECT_MAPPINGS
+
+    def get_full_mapping(self, effect_name: str) -> Optional[Dict]:
+        """Get full mapping for an effect (including external tools and python primitives)."""
+        mapping = self.mappings.get(effect_name)
+        if not mapping:
+            # Try with underscores/hyphens converted
+            normalized = effect_name.replace("-", "_").replace(" ", "_").lower()
+            mapping = self.mappings.get(normalized)
+        return mapping
+
+    def get_mapping(self, effect_name: str) -> Optional[Dict]:
+        """Get FFmpeg filter mapping for an effect (returns None for non-FFmpeg effects)."""
+        mapping = self.get_full_mapping(effect_name)
+        # Return None if no mapping or no FFmpeg filter
+        if mapping and mapping.get("filter") is None:
+            return None
+        return mapping
+
+    def has_external_tool(self, effect_name: str) -> Optional[str]:
+        """Check if effect uses an external tool. Returns tool name or None."""
+        mapping = self.get_full_mapping(effect_name)
+        if mapping:
+            return mapping.get("external_tool")
+        return None
+
+    def has_python_primitive(self, effect_name: str) -> Optional[str]:
+        """Check if effect uses a Python primitive. Returns primitive name or None."""
+        mapping = self.get_full_mapping(effect_name)
+        if mapping:
+            return mapping.get("python_primitive")
+        return None
+
+    def is_complex_filter(self, effect_name: str) -> bool:
+        """Check if effect uses a complex filter chain."""
+        mapping = self.get_full_mapping(effect_name)
+        return bool(mapping and mapping.get("complex"))
+
+    def compile_effect(
+        self,
+        effect_name: str,
+        params: Dict[str, Any],
+    ) -> Optional[str]:
+        """
+        Compile an effect to an FFmpeg filter string with static params.
+
+        Returns None if effect has no FFmpeg mapping.
+        """
+        mapping = self.get_mapping(effect_name)
+        if not mapping:
+            return None
+
+        filter_name = mapping["filter"]
+
+        # Handle static filters (no params)
+        if "static" in mapping:
+            return f"{filter_name}={mapping['static']}"
+
+        if not mapping.get("params"):
+            return filter_name
+
+        # Build param string
+        filter_params = []
+        for param_name, param_config in mapping["params"].items():
+            if param_name in params:
+                value = params[param_name]
+                # Skip if it's a binding (handled separately)
+                if isinstance(value, dict) and ("_bind" in value or "_binding" in value):
+                    continue
+                ffmpeg_param = param_config["ffmpeg_param"]
+                scale = param_config.get("scale", 1.0)
+                offset = param_config.get("offset", 0)
+                # Handle various value types
+                if isinstance(value, (int, float)):
+                    ffmpeg_value = value * scale + offset
+                    filter_params.append(f"{ffmpeg_param}={ffmpeg_value:.4f}")
+                elif isinstance(value, str):
+                    filter_params.append(f"{ffmpeg_param}={value}")
+                elif isinstance(value, list) and value and isinstance(value[0], (int, float)):
+                    ffmpeg_value = value[0] * scale + offset
+                    filter_params.append(f"{ffmpeg_param}={ffmpeg_value:.4f}")
+
+        if filter_params:
+            return f"{filter_name}={':'.join(filter_params)}"
+        return filter_name
+
+    def compile_effect_with_bindings(
+        self,
+        effect_name: str,
+        params: Dict[str, Any],
+        analysis_data: Dict[str, Dict],
+        segment_start: float,
+        segment_duration: float,
+        sample_interval: float = 0.04,  # ~25 fps
+    ) -> Tuple[Optional[str], Optional[str], List[str]]:
+        """
+        Compile an effect with dynamic bindings to a filter + sendcmd script.
+
+        Returns:
+            (filter_string, sendcmd_script, bound_param_names)
+            - filter_string: Initial FFmpeg filter (may have placeholder values)
+            - sendcmd_script: Script content for sendcmd filter
+            - bound_param_names: List of params that have bindings
+        """
+        mapping = self.get_mapping(effect_name)
+        if not mapping:
+            return None, None, []
+
+        filter_name = mapping["filter"]
+        static_params = []
+        bound_params = []
+        sendcmd_lines = []
+
+        # Handle time-based effects (use FFmpeg expressions with 't')
+        if mapping.get("time_based"):
+            for param_name, param_config in mapping.get("params", {}).items():
+                if param_name in params:
+                    value = params[param_name]
+                    ffmpeg_param = param_config["ffmpeg_param"]
+                    scale = param_config.get("scale", 1.0)
+                    if isinstance(value, (int, float)):
+                        # Create time expression: h='t*speed*scale'
+                        static_params.append(f"{ffmpeg_param}='t*{value}*{scale}'")
+                    else:
+                        static_params.append(f"{ffmpeg_param}='t*{scale}'")
+            if static_params:
+                filter_str = f"{filter_name}={':'.join(static_params)}"
+            else:
+                filter_str = f"{filter_name}=h='t*360'"  # Default rotation
+            return filter_str, None, []
+
+        # Process each param
+        for param_name, param_config in mapping.get("params", {}).items():
+            if param_name not in params:
+                continue
+
+            value = params[param_name]
+            ffmpeg_param = param_config["ffmpeg_param"]
+            scale = param_config.get("scale", 1.0)
+            offset = param_config.get("offset", 0)
+
+            # Check if it's a binding
+            if isinstance(value, dict) and ("_bind" in value or "_binding" in value):
+                bind_ref = value.get("_bind") or value.get("_binding")
+                range_min = value.get("range_min", 0.0)
+                range_max = value.get("range_max", 1.0)
+                transform = value.get("transform")
+
+                # Get analysis data
+                analysis = analysis_data.get(bind_ref)
+                if not analysis:
+                    # Try without -data suffix
+                    analysis = analysis_data.get(bind_ref.replace("-data", ""))
+
+                if analysis and "times" in analysis and "values" in analysis:
+                    times = analysis["times"]
+                    values = analysis["values"]
+
+                    # Generate sendcmd entries for this segment
+                    segment_end = segment_start + segment_duration
+                    t = 0.0  # Time relative to segment start
+
+                    while t < segment_duration:
+                        abs_time = segment_start + t
+
+                        # Find analysis value at this time
+                        raw_value = self._interpolate_value(times, values, abs_time)
+
+                        # Apply transform
+                        if transform == "sqrt":
+                            raw_value = math.sqrt(max(0, raw_value))
+                        elif transform == "pow2":
+                            raw_value = raw_value ** 2
+                        elif transform == "log":
+                            raw_value = math.log(max(0.001, raw_value))
+
+                        # Map to range
+                        mapped_value = range_min + raw_value * (range_max - range_min)
+
+                        # Apply FFmpeg scaling
+                        ffmpeg_value = mapped_value * scale + offset
+
+                        # Add sendcmd line (time relative to segment)
+                        sendcmd_lines.append(f"{t:.3f} [{filter_name}] {ffmpeg_param} {ffmpeg_value:.4f};")
+
+                        t += sample_interval
+
+                    bound_params.append(param_name)
+                    # Use initial value for the filter string
+                    initial_value = self._interpolate_value(times, values, segment_start)
+                    initial_mapped = range_min + initial_value * (range_max - range_min)
+                    initial_ffmpeg = initial_mapped * scale + offset
+                    static_params.append(f"{ffmpeg_param}={initial_ffmpeg:.4f}")
+                else:
+                    # No analysis data, use range midpoint
+                    mid_value = (range_min + range_max) / 2
+                    ffmpeg_value = mid_value * scale + offset
+                    static_params.append(f"{ffmpeg_param}={ffmpeg_value:.4f}")
+            else:
+                # Static value - handle various types
+                if isinstance(value, (int, float)):
+                    ffmpeg_value = value * scale + offset
+                    static_params.append(f"{ffmpeg_param}={ffmpeg_value:.4f}")
+                elif isinstance(value, str):
+                    # String value - use as-is (e.g., for direction parameters)
+                    static_params.append(f"{ffmpeg_param}={value}")
+                elif isinstance(value, list) and value:
+                    # List - try to use first numeric element
+                    first = value[0]
+                    if isinstance(first, (int, float)):
+                        ffmpeg_value = first * scale + offset
+                        static_params.append(f"{ffmpeg_param}={ffmpeg_value:.4f}")
+                # Skip other types
+
+        # Handle static filters
+        if "static" in mapping:
+            filter_str = f"{filter_name}={mapping['static']}"
+        elif static_params:
+            filter_str = f"{filter_name}={':'.join(static_params)}"
+        else:
+            filter_str = filter_name
+
+        # Combine sendcmd lines
+        sendcmd_script = "\n".join(sendcmd_lines) if sendcmd_lines else None
+
+        return filter_str, sendcmd_script, bound_params
+
+    def _interpolate_value(
+        self,
+        times: List[float],
+        values: List[float],
+        target_time: float,
+    ) -> float:
+        """Interpolate a value from analysis data at a given time."""
+        if not times or not values:
+            return 0.5
+
+        # Find surrounding indices
+        if target_time <= times[0]:
+            return values[0]
+        if target_time >= times[-1]:
+            return values[-1]
+
+        # Binary search for efficiency
+        lo, hi = 0, len(times) - 1
+        while lo < hi - 1:
+            mid = (lo + hi) // 2
+            if times[mid] <= target_time:
+                lo = mid
+            else:
+                hi = mid
+
+        # Linear interpolation
+        t0, t1 = times[lo], times[hi]
+        v0, v1 = values[lo], values[hi]
+
+        if t1 == t0:
+            return v0
+
+        alpha = (target_time - t0) / (t1 - t0)
+        return v0 + alpha * (v1 - v0)
+
+
+def generate_sendcmd_filter(
+    effects: List[Dict],
+    analysis_data: Dict[str, Dict],
+    segment_start: float,
+    segment_duration: float,
+) -> Tuple[str, Optional[Path]]:
+    """
+    Generate FFmpeg filter chain with sendcmd for dynamic effects.
+
+    Args:
+        effects: List of effect configs with name and params
+        analysis_data: Analysis data keyed by name
+        segment_start: Segment start time in source
+        segment_duration: Segment duration
+
+    Returns:
+        (filter_chain_string, sendcmd_file_path or None)
+    """
+    import tempfile
+
+    compiler = FFmpegCompiler()
+    filters = []
+    all_sendcmd_lines = []
+
+    for effect in effects:
+        effect_name = effect.get("effect")
+        params = {k: v for k, v in effect.items() if k != "effect"}
+
+        filter_str, sendcmd, _ = compiler.compile_effect_with_bindings(
+            effect_name,
+            params,
+            analysis_data,
+            segment_start,
+            segment_duration,
+        )
+
+        if filter_str:
+            filters.append(filter_str)
+            if sendcmd:
+                all_sendcmd_lines.append(sendcmd)
+
+    if not filters:
+        return "", None
+
+    filter_chain = ",".join(filters)
+
+    # NOTE: sendcmd disabled - FFmpeg's sendcmd filter has compatibility issues.
+    # Bindings use their initial value (sampled at segment start time).
+    # This is acceptable since each segment is only ~8 seconds.
+    # The binding value is still music-reactive (varies per segment).
+    sendcmd_path = None
+
+    return filter_chain, sendcmd_path