giles
81112c716b
Remove cross-DB relationships (CartItem.product, CartItem.market_place, OrderItem.product) that break with per-service databases. Denormalize product and marketplace fields onto cart_items/order_items at write time. - Add AP internal inbox infrastructure (shared/infrastructure/internal_inbox*) for synchronous inter-service writes via HMAC-authenticated POST - Cart inbox blueprint handles Add/Remove/Update rose:CartItem activities - Market app sends AP activities to cart inbox instead of writing CartItem directly - Cart services use denormalized columns instead of cross-DB hydration/joins - Add marketplaces-by-ids data endpoint to market service - Alembic migration adds denormalized columns to cart_items and order_items - Add OAuth device flow auth to market scraper persist_api (artdag client pattern) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3.2 KiB
3.2 KiB
Art DAG Monorepo
Federated content-addressed DAG execution engine for distributed media processing with ActivityPub ownership and provenance tracking.
Deployment
- Do NOT push until explicitly told to. Pushes reload code to dev automatically.
Project Structure
core/ # DAG engine (artdag package) - nodes, effects, analysis, planning
l1/ # L1 Celery rendering server (FastAPI + Celery + Redis + PostgreSQL)
l2/ # L2 ActivityPub registry (FastAPI + PostgreSQL)
common/ # Shared templates, middleware, models (artdag_common package)
client/ # CLI client
test/ # Integration & e2e tests
Tech Stack
Python 3.11+, FastAPI, Celery, Redis, PostgreSQL (asyncpg for L1), SQLAlchemy, Pydantic, JAX (CPU/GPU), IPFS/Kubo, Docker Swarm, HTMX + Jinja2 for web UI.
Key Commands
Testing
cd l1 && pytest tests/ # L1 unit tests
cd core && pytest tests/ # Core unit tests
cd test && python run.py # Full integration pipeline
- pytest uses
asyncio_mode = "auto"for async tests - Test files:
test_*.py, fixtures inconftest.py
Linting & Type Checking (L1)
cd l1 && ruff check . # Lint (E, F, I, UP rules)
cd l1 && mypy app/types.py app/routers/recipes.py tests/
- Line length: 100 chars (E501 ignored)
- Mypy: strict on
app/types.py,app/routers/recipes.py,tests/; gradual elsewhere - Mypy ignores imports for: celery, redis, artdag, artdag_common, ipfs_client
Docker
docker build -f l1/Dockerfile -t celery-l1-server:latest .
docker build -f l1/Dockerfile.gpu -t celery-l1-gpu:latest .
docker build -f l2/Dockerfile -t l2-server:latest .
./deploy.sh # Build, push, deploy stacks
Architecture Patterns
- 3-Phase Execution: Analyze -> Plan -> Execute (tasks in
l1/tasks/) - Content-Addressed: All data identified by SHA3-256 hashes or IPFS CIDs
- Services Pattern: Business logic in
app/services/, API endpoints inapp/routers/ - Types Module: Pydantic models and TypedDicts in
app/types.py - Celery Tasks: In
l1/tasks/, decorated with@app.task - S-Expression Effects: Composable effect language in
l1/sexp_effects/ - Storage: Local filesystem, S3, or IPFS backends (
storage_providers.py) - Inter-Service Reads:
fetch_data()→ GET/internal/data/{query}(HMAC-signed) - Inter-Service Actions:
call_action()→ POST/internal/actions/{name}(HMAC-signed) - Inter-Service AP Inbox:
send_internal_activity()→ POST/internal/inbox(HMAC-signed, AP-shaped activities for cross-service writes with denormalized data)
Auth
- L1 <-> L2: scoped JWT tokens (no shared secrets)
- L2: password + OAuth SSO, token revocation in Redis (30-day expiry)
- Federation: ActivityPub RSA signatures (
core/artdag/activitypub/)
Key Config Files
l1/pyproject.toml- mypy, pytest, ruff config for L1l1/celery_app.py- Celery initializationl1/database.py/l2/db.py- SQLAlchemy modelsl1/docker-compose.yml/l2/docker-compose.yml- Swarm stacks
Tools
- Use Context7 MCP for up-to-date library documentation
- Playwright MCP is available for browser automation/testing