diff --git a/CLAUDE.md b/CLAUDE.md index a9588b1..438bd41 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,6 +1,6 @@ -# Art DAG Monorepo +# Rose Ash Monorepo -Federated content-addressed DAG execution engine for distributed media processing with ActivityPub ownership and provenance tracking. +Cooperative web platform: federated content, commerce, events, and media processing. Each domain runs as an independent Quart microservice with its own database, communicating via HMAC-signed internal HTTP and ActivityPub events. ## Deployment @@ -9,71 +9,134 @@ Federated content-addressed DAG execution engine for distributed media processin ## Project Structure ``` -core/ # DAG engine (artdag package) - nodes, effects, analysis, planning -l1/ # L1 Celery rendering server (FastAPI + Celery + Redis + PostgreSQL) -l2/ # L2 ActivityPub registry (FastAPI + PostgreSQL) -common/ # Shared templates, middleware, models (artdag_common package) -client/ # CLI client -test/ # Integration & e2e tests +blog/ # Content management, Ghost CMS sync, navigation, WYSIWYG editor +market/ # Product catalog, marketplace pages, web scraping +cart/ # Shopping cart CRUD, checkout (delegates order creation to orders) +events/ # Calendar & event management, ticketing +federation/ # ActivityPub social hub, user profiles +account/ # OAuth2 authorization server, user dashboard, membership +orders/ # Order history, SumUp payment/webhook handling, reconciliation +relations/ # (internal) Cross-domain parent/child relationship tracking +likes/ # (internal) Unified like/favourite tracking across domains +shared/ # Shared library: models, infrastructure, templates, static assets +artdag/ # Art DAG — media processing engine (separate codebase, see below) +``` + +### Shared Library (`shared/`) + +``` +shared/ + models/ # Canonical SQLAlchemy ORM models for all domains + db/ # Async session management, per-domain DB support, alembic helpers + infrastructure/ # App factory, OAuth, ActivityPub, fragments, internal auth, Jinja + services/ # Domain service implementations + DI registry + contracts/ # DTOs and service protocols + browser/ # Middleware, Redis caching, CSRF, error handlers + events/ # Activity bus + background processor (AP-shaped events) + config/ # YAML config loading (frozen/readonly) + static/ # Shared CSS, JS, images + templates/ # Base HTML layouts, partials (inherited by all apps) +``` + +### Art DAG (`artdag/`) + +Federated content-addressed DAG execution engine for distributed media processing. + +``` +artdag/ + core/ # DAG engine (artdag package) — nodes, effects, analysis, planning + l1/ # L1 Celery rendering server (FastAPI + Celery + Redis + PostgreSQL) + l2/ # L2 ActivityPub registry (FastAPI + PostgreSQL) + common/ # Shared templates, middleware, models (artdag_common package) + client/ # CLI client + test/ # Integration & e2e tests ``` ## Tech Stack -Python 3.11+, FastAPI, Celery, Redis, PostgreSQL (asyncpg for L1), SQLAlchemy, Pydantic, JAX (CPU/GPU), IPFS/Kubo, Docker Swarm, HTMX + Jinja2 for web UI. +**Web platform:** Python 3.11+, Quart (async Flask), SQLAlchemy (asyncpg), Jinja2, HTMX, PostgreSQL, Redis, Docker Swarm, Hypercorn. + +**Art DAG:** FastAPI, Celery, JAX (CPU/GPU), IPFS/Kubo, Pydantic. ## Key Commands -### Testing +### Development ```bash -cd l1 && pytest tests/ # L1 unit tests -cd core && pytest tests/ # Core unit tests -cd test && python run.py # Full integration pipeline +./dev.sh # Start all services + infra (db, redis, pgbouncer) +./dev.sh blog market # Start specific services + infra +./dev.sh --build blog # Rebuild image then start +./dev.sh down # Stop everything +./dev.sh logs blog # Tail service logs ``` -- pytest uses `asyncio_mode = "auto"` for async tests -- Test files: `test_*.py`, fixtures in `conftest.py` -### Linting & Type Checking (L1) +### Deployment ```bash -cd l1 && ruff check . # Lint (E, F, I, UP rules) -cd l1 && mypy app/types.py app/routers/recipes.py tests/ +./deploy.sh # Auto-detect changed apps, build + push + restart +./deploy.sh blog market # Deploy specific apps +./deploy.sh --all # Deploy everything ``` -- Line length: 100 chars (E501 ignored) -- Mypy: strict on `app/types.py`, `app/routers/recipes.py`, `tests/`; gradual elsewhere -- Mypy ignores imports for: celery, redis, artdag, artdag_common, ipfs_client -### Docker +### Art DAG ```bash -docker build -f l1/Dockerfile -t celery-l1-server:latest . -docker build -f l1/Dockerfile.gpu -t celery-l1-gpu:latest . -docker build -f l2/Dockerfile -t l2-server:latest . -./deploy.sh # Build, push, deploy stacks +cd artdag/l1 && pytest tests/ # L1 unit tests +cd artdag/core && pytest tests/ # Core unit tests +cd artdag/test && python run.py # Full integration pipeline +cd artdag/l1 && ruff check . # Lint +cd artdag/l1 && mypy app/types.py app/routers/recipes.py tests/ ``` ## Architecture Patterns -- **3-Phase Execution**: Analyze -> Plan -> Execute (tasks in `l1/tasks/`) -- **Content-Addressed**: All data identified by SHA3-256 hashes or IPFS CIDs -- **Services Pattern**: Business logic in `app/services/`, API endpoints in `app/routers/` -- **Types Module**: Pydantic models and TypedDicts in `app/types.py` -- **Celery Tasks**: In `l1/tasks/`, decorated with `@app.task` -- **S-Expression Effects**: Composable effect language in `l1/sexp_effects/` -- **Storage**: Local filesystem, S3, or IPFS backends (`storage_providers.py`) -- **Inter-Service Reads**: `fetch_data()` → GET `/internal/data/{query}` (HMAC-signed) -- **Inter-Service Actions**: `call_action()` → POST `/internal/actions/{name}` (HMAC-signed) -- **Inter-Service AP Inbox**: `send_internal_activity()` → POST `/internal/inbox` (HMAC-signed, AP-shaped activities for cross-service writes with denormalized data) +### Web Platform -## Auth +- **App factory:** `create_base_app(name, context_fn, before_request_fns, domain_services_fn)` in `shared/infrastructure/factory.py` — creates Quart app with DB, Redis, CSRF, OAuth, AP, session management +- **Blueprint pattern:** Each blueprint exposes `register() -> Blueprint`, handlers stored in `_handlers` dict +- **Per-service database:** Each service has own PostgreSQL DB via PgBouncer; cross-domain data fetched via HTTP +- **Alembic per-service:** Each service declares `MODELS` and `TABLES` in `alembic/env.py`, delegates to `shared.db.alembic_env.run_alembic()` +- **Inter-service reads:** `fetch_data(service, query, params)` → GET `/internal/data/{query}` (HMAC-signed, 3s timeout) +- **Inter-service writes:** `call_action(service, action, payload)` → POST `/internal/actions/{action}` (HMAC-signed, 5s timeout) +- **Inter-service AP inbox:** `send_internal_activity()` → POST `/internal/inbox` (HMAC-signed, AP-shaped activities for cross-service writes) +- **Fragments:** HTML fragments fetched cross-service via `fetch_fragments()` for composing shared UI (nav, cart mini, auth menu) +- **Soft deletes:** Models use `deleted_at` column pattern +- **Context processors:** Each app provides its own `context_fn` that assembles template context from local DB + cross-service fragments -- L1 <-> L2: scoped JWT tokens (no shared secrets) -- L2: password + OAuth SSO, token revocation in Redis (30-day expiry) -- Federation: ActivityPub RSA signatures (`core/artdag/activitypub/`) +### Auth + +- **Account** is the OAuth2 authorization server; all other apps are OAuth clients +- Per-app first-party session cookies (Safari ITP compatible), synchronized via device ID +- Grant verification: apps check grant validity against account DB (cached in Redis) +- Silent SSO: `prompt=none` OAuth flow for automatic cross-app login +- ActivityPub: RSA signatures, per-app virtual actor projections sharing same keypair + +### Art DAG + +- **3-Phase Execution:** Analyze → Plan → Execute (tasks in `artdag/l1/tasks/`) +- **Content-Addressed:** All data identified by SHA3-256 hashes or IPFS CIDs +- **S-Expression Effects:** Composable effect language in `artdag/l1/sexp_effects/` +- **Storage:** Local filesystem, S3, or IPFS backends +- L1 ↔ L2: scoped JWT tokens; L2: password + OAuth SSO + +## Domains + +| Service | Public URL | Dev Port | +|---------|-----------|----------| +| blog | blog.rose-ash.com | 8001 | +| market | market.rose-ash.com | 8002 | +| cart | cart.rose-ash.com | 8003 | +| events | events.rose-ash.com | 8004 | +| federation | federation.rose-ash.com | 8005 | +| account | account.rose-ash.com | 8006 | +| relations | (internal only) | 8008 | +| likes | (internal only) | 8009 | +| orders | orders.rose-ash.com | 8010 | ## Key Config Files -- `l1/pyproject.toml` - mypy, pytest, ruff config for L1 -- `l1/celery_app.py` - Celery initialization -- `l1/database.py` / `l2/db.py` - SQLAlchemy models -- `l1/docker-compose.yml` / `l2/docker-compose.yml` - Swarm stacks +- `docker-compose.yml` / `docker-compose.dev.yml` — service definitions, env vars, volumes +- `deploy.sh` / `dev.sh` — deployment and development scripts +- `shared/infrastructure/factory.py` — app factory (all services use this) +- `{service}/alembic/env.py` — per-service migration config +- `_config/app-config.yaml` — runtime YAML config (mounted into containers) ## Tools