celery/README.md

# Art Celery

L1 rendering server for the Art DAG system. Manages distributed rendering jobs via Celery workers.

## Dependencies

- **artdag** (GitHub): Core DAG execution engine
- **artdag-effects** (rose-ash): Effect implementations
- **Redis**: Message broker, result backend, and run persistence

## Setup

```bash
# Install Redis
sudo apt install redis-server

# Install Python dependencies
pip install -r requirements.txt

# Start a worker
celery -A celery_app worker --loglevel=info

# Start the L1 server
python server.py
```

## Web UI

The server provides a web interface at the root URL:

| Path | Description |
|------|-------------|
| `/` | Home page with server info |
| `/runs` | View and manage rendering runs |
| `/run/{id}` | Run detail page |
| `/recipes` | Browse and run available recipes |
| `/recipe/{id}` | Recipe detail page |
| `/media` | Browse cached media files |
| `/auth` | Receive auth token from L2 |
| `/logout` | Log out |
| `/download/client` | Download CLI client |

## API

Interactive docs: http://localhost:8100/docs

### Endpoints

| Method | Path | Description |
|--------|------|-------------|
| GET | `/` | Server info |
| POST | `/runs` | Start a rendering run |
| GET | `/runs` | List all runs |
| GET | `/runs/{run_id}` | Get run status |
| DELETE | `/runs/{run_id}` | Delete a run |
| GET | `/cache` | List cached content hashes |
| GET | `/cache/{hash}` | Download cached content |
| DELETE | `/cache/{hash}` | Delete cached content |
| POST | `/cache/import?path=` | Import local file to cache |
| POST | `/cache/upload` | Upload file to cache |
| GET | `/assets` | List known assets |
| POST | `/configs/upload` | Upload a config YAML |
| GET | `/configs` | List configs |
| GET | `/configs/{id}` | Get config details |
| DELETE | `/configs/{id}` | Delete a config |
| POST | `/configs/{id}/run` | Run a config |

### Configs

Configs are YAML files that define reusable DAG pipelines. They can have:
- **Fixed inputs**: Assets with pre-defined content hashes
- **Variable inputs**: Placeholders filled at run time

Example config:
```yaml
name: my-effect
version: "1.0"
description: "Apply effect to user image"

registry:
  effects:
    dog:
      hash: "abc123..."

dag:
  nodes:
    - id: user_image
      type: SOURCE
      config:
        input: true        # Variable input
        name: "input_image"

    - id: apply_dog
      type: EFFECT
      config:
        effect: dog
      inputs:
        - user_image

  output: apply_dog
```

### Start a run

```bash
curl -X POST http://localhost:8100/runs \
  -H "Content-Type: application/json" \
  -d '{"recipe": "dog", "inputs": ["33268b6e..."], "output_name": "my-output"}'
```

### Check run status

```bash
curl http://localhost:8100/runs/{run_id}
```

### Delete a run

```bash
curl -X DELETE http://localhost:8100/runs/{run_id} \
  -H "Authorization: Bearer <token>"
```

Note: Failed runs can always be deleted. Completed runs can only be deleted if their outputs haven't been published to L2.

### Delete cached content

```bash
curl -X DELETE http://localhost:8100/cache/{hash} \
  -H "Authorization: Bearer <token>"
```

Note: Items that are inputs/outputs of runs, or published to L2, cannot be deleted.

## Storage

- **Cache**: `~/.artdag/cache/` (content-addressed files)
- **Runs**: Redis db 5, keys `artdag:run:*` (persists across restarts)

## CLI Usage

```bash
# Render cat through dog effect
python render.py dog cat --sync

# Render cat through identity effect
python render.py identity cat --sync

# Submit async (don't wait)
python render.py dog cat
```

## Architecture

```
server.py (L1 Server - FastAPI)
    │
    ├── POST /runs → Submit job
    │       │
    │       ▼
    │   celery_app.py (Celery broker)
    │       │
    │       ▼
    │   tasks.py (render_effect task)
    │       │
    │       ├── artdag (GitHub) - DAG execution
    │       └── artdag-effects (rose-ash) - Effects
    │               │
    │               ▼
    │           Output + Provenance
    │
    └── GET /cache/{hash} → Retrieve output
```

## Provenance

Every render produces a provenance record:

```json
{
  "task_id": "celery-task-uuid",
  "rendered_at": "2026-01-07T...",
  "rendered_by": "@giles@artdag.rose-ash.com",
  "output": {"name": "...", "content_hash": "..."},
  "inputs": [...],
  "effects": [...],
  "infrastructure": {
    "software": {"name": "infra:artdag", "content_hash": "..."},
    "hardware": {"name": "infra:giles-hp", "content_hash": "..."}
  }
}
```