Commit Graph

129 Commits

Author SHA1 Message Date
gilesb
3db606bf15 Plan-based caching with artifact playback in UI
RunStatus now stores:
- plan_id, plan_name for linking to execution plan
- step_results for per-step execution status
- all_outputs for all artifacts from all steps

Plan visualization:
- Shows human-readable step names from recipe structure
- Video/audio artifact preview on node click
- Outputs list with links to cached artifacts
- Stats reflect actual execution status (completed/cached/pending)

Execution:
- Step results include outputs list with cache_ids
- run_plan returns all outputs from all steps
- Support for completed_by_other status

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 00:20:19 +00:00
giles
36bf0cd3f7 Generate execution plans on-the-fly for plan visualization
Instead of only showing cached plans, now attempts to generate
a plan from the recipe if no cached plan is found. This works
for recipe-based runs even if they were started via the legacy
execute_dag path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 23:33:38 +00:00
giles
b498d6fe4b Enable Celery events for Flower monitoring
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 23:27:36 +00:00
giles
e7d3a5ed6c Add DAG visualization for runs and recipes
- Add Cytoscape.js infrastructure for interactive DAG visualization
- New run sub-pages: /run/{id}/plan, /run/{id}/analysis, /run/{id}/artifacts
- Plan page shows execution DAG with cached/pending status
- Analysis page displays tempo, beats, energy per input
- Artifacts page lists all cached items with thumbnails
- New /recipe/{id}/dag page for recipe structure visualization
- Add sub-tabs navigation to run detail page
- Add JSON API endpoints for future WebSocket support
- Architecture designed for real-time updates (global Cytoscape instance)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 23:27:28 +00:00
gilesb
1feecc8c05 Add Flower for Celery monitoring
Flower provides a web UI to monitor:
- Active/completed/failed tasks
- Worker status and stats
- Task history and details

Accessible on port 5555.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 22:06:17 +00:00
gilesb
e56b4792c7 Show all inputs with names on run detail page
Previously only displayed the first input. Now shows all inputs with
their names from the recipe (variable_inputs and fixed_inputs).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 19:48:07 +00:00
gilesb
9a3d6949fc Store and use activity_id for L2 links
- Add activity_id column to l2_shares table
- Store activity_id when publishing to L2
- Link to /activities/{activity_id} instead of /assets/{name}
- Falls back to asset link if no activity_id

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 18:56:23 +00:00
gilesb
3d45efa90b Link to activity instead of asset in "View on L2"
After publishing, link to /activities/{activity_id} instead of
/assets/{asset_name} so user sees the published run.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 18:38:07 +00:00
gilesb
837f9a6442 Add defensive check for null asset in publish
Prevents NoneType subscript error if L2 returns null asset.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 15:08:03 +00:00
gilesb
eaff758395 Rename tasks.py to legacy_tasks.py to avoid conflict with tasks/ package
The tasks/ directory for 3-phase execution was shadowing the old tasks.py.
Renamed to legacy_tasks.py and updated all imports.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 12:05:27 +00:00
gilesb
f7890dd1ad Add 3-phase execution with IPFS cache and hash-based task claiming
New files:
- claiming.py - Redis Lua scripts for atomic task claiming
- tasks/analyze.py - Analysis Celery task
- tasks/execute.py - Step execution with IPFS-backed cache
- tasks/orchestrate.py - Plan orchestration (run_plan, run_recipe)

New API endpoints (/api/v2/):
- POST /api/v2/plan - Generate execution plan
- POST /api/v2/execute - Execute a plan
- POST /api/v2/run-recipe - Full 3-phase pipeline
- GET /api/v2/run/{run_id} - Get run status

Features:
- Hash-based task claiming prevents duplicate work
- Parallel execution within dependency levels
- IPFS-backed cache for durability
- Integration with artdag planning module

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 11:44:00 +00:00
gilesb
7d05011daa Add Storage link to L1 navigation bars
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 02:44:52 +00:00
gilesb
2a4f24b7ee Add storage engine configuration to L1, matching L2 implementation
- Copy storage_providers.py from L2 (Pinata, web3.storage, NFT.Storage,
  Infura, Filebase, Storj, local storage providers)
- Add storage management endpoints: GET/POST/PATCH/DELETE /storage
- Add provider-specific pages at /storage/type/{provider_type}
- Include connection testing via POST /storage/{id}/test
- Add HTML UI pages with dark theme matching L2

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 02:41:50 +00:00
gilesb
5ce28abe52 Phase 2: Multiple storage configs per type with new UI structure
- Database: Add description field, remove unique constraint to allow
  multiple configs of same provider type
- UI: Main page shows provider types as cards with counts
- UI: Per-type page (/storage/type/{type}) for managing configs
- API: Add get_user_storage_by_type() for filtered queries
- Form: Add description field for distinguishing configs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 00:52:28 +00:00
gilesb
58b3240430 Store IPFS CID in database for DAG outputs
Properly unpack cache_manager.put() tuple to get IPFS CID and store
it in PostgreSQL via database.create_cache_item(). This fixes the
"Output has no IPFS CID - cannot publish" error when publishing
from L1 to L2.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 23:35:50 +00:00
gilesb
5cb8d4ec17 Add storage_backends table for user storage config
Prepares L1 for distributed storage integration with user-provided
storage backends (Pinata, web3.storage, local). The storage config
is synced from L2 or can be configured locally per actor.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 23:22:28 +00:00
gilesb
2cfe0f2bd2 Update README with /auth/revoke-user documentation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 22:23:05 +00:00
gilesb
e485384d2f Add /auth/revoke-user to revoke all tokens for a user
- Track user tokens in Redis set (artdag:user_tokens:{username})
- Register token when user authenticates via /auth
- Add /auth/revoke-user endpoint to revoke all user tokens
- L2 calls this on logout to revoke scoped tokens

Fixes logout not working across L1/L2 (scoped tokens differ from L2 token).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 22:22:08 +00:00
gilesb
9a83722273 Document authentication and token revocation in README
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 18:21:36 +00:00
gilesb
cd60016558 Add token revocation for federated logout
- Add revoke_token() and is_token_revoked() functions using Redis
- Check revocation in get_verified_user_context()
- Add /auth/revoke endpoint for L2 to call on logout
- Revoked tokens stored with 30-day expiry (matching token lifetime)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 18:07:50 +00:00
gilesb
ce4d38b943 Update Web UI routes in README
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 17:56:02 +00:00
gilesb
38e988586f Pass L1_PUBLIC_URL when verifying tokens with L2
L2 now requires L1 servers to identify themselves for authorization.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 17:43:19 +00:00
gilesb
9a51d66aec Add /auth endpoint for iOS Safari cross-subdomain auth
iOS Safari blocks shared cookies. Now L2 can redirect to
L1/auth?auth_token=xxx after login, and L1 sets its own
first-party cookie.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 17:06:10 +00:00
gilesb
304f3ad56f Remove login links from L1 - L1 doesn't know where L2 is
L1 can't redirect users to login because it doesn't know which L2
server they use. Users must log in directly at their L2 server,
and the shared cookie will authenticate them on L1.

- Replace "Log in" links with "Not logged in" text
- Remove /login and /register routes
- Keep /logout to just clear cookie and redirect home
- Remove unused DEFAULT_L2_SERVER config

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 16:52:32 +00:00
gilesb
780c223714 Fix video display and download with proper media types
- video_src_for_request now returns /raw endpoint for video source
- /raw endpoint sets proper Content-Type based on file format
- /raw endpoint sets filename with extension for downloads

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 12:46:01 +00:00
gilesb
f67b33119f Register dog effect for DAG EFFECT nodes
The EFFECT node executor uses register_effect() to look up effects.
Added dog effect registration so DAG recipes can use effect: dog.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 12:30:44 +00:00
gilesb
e53e5a5ee6 Fix embedded media to use /raw endpoint
After content negotiation fix, /cache/{hash} returns HTML for
browsers. Embedded <img> tags need /raw to get actual image data.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 12:24:04 +00:00
gilesb
55878d46ac Use content_hash as asset names when publishing runs
- Remove output_name from publish form and endpoint
- Assets on L2 are now named by their content_hash
- All inputs, recipe, and output referenced by content_hash
- Simplified publish flow - no user input needed

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 12:06:32 +00:00
gilesb
b225151d99 Fix content negotiation - default to HTML, not raw data
The /cache/{hash} endpoint now defaults to HTML for browsers. Raw
data was being returned for requests without explicit text/html in
Accept header (e.g., link clicks). JSON is only returned when
explicitly requested. Raw data is served only from /cache/{hash}/raw.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 11:59:26 +00:00
gilesb
501626fa1d Return recipe media_type from cache endpoint
When cache metadata has type "recipe", return that instead of
auto-detecting (which returns "unknown" for YAML files). This
ensures L2 can properly register recipes as inputs.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 11:50:59 +00:00
gilesb
179e7e0fcb Improve recipe display - show YAML source and L2 link
- Recipe detail page now shows the full YAML source
- Shows "View on L2" link if recipe is shared to L2

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 11:33:42 +00:00
gilesb
43788108c0 Fix Celery workers to use Redis for shared cache index
The get_cache_manager() singleton wasn't initializing with Redis,
so workers couldn't see files uploaded via the API server.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 11:25:38 +00:00
gilesb
dd3b0e1c73 Add CLI client download
- Include artdag-client.tar.gz package
- Add /download/client route to serve the package
- Add "Download Client" link in nav bar

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 11:16:49 +00:00
gilesb
ebb692ee94 Fix run_recipe endpoint to use content-addressable run_id
The run_recipe endpoint was still using uuid.uuid4() instead of
compute_run_id(). Now it:
- Computes deterministic run_id from inputs + recipe
- Checks L1 cache before running
- Checks L2 and pulls from IPFS if needed
- Only runs Celery if output not found

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 11:12:52 +00:00
gilesb
e4b9657d1b Add content-addressable runs - runs identified by hash of inputs + recipe
- Add run_cache table for fast run_id -> output lookup
- compute_run_id() computes deterministic run_id from inputs + recipe
- create_run checks L1 cache then L2 before running Celery
- If output exists on L2 but not L1, pulls from IPFS
- Saves run results to cache on completion

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 11:05:18 +00:00
gilesb
5344fe263f Remove /ui redirect routes, add 404 page, update README
- Remove obsolete /ui, /ui/login, /ui/register, /ui/logout redirects
- Fix /ui/login links to use /login directly
- Add styled 404 page for HTML requests
- Add Web UI section to README documenting routes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 10:42:14 +00:00
gilesb
1194251788 Fix broken View on L2 links - remove incorrect /ui prefix
The L2 server route is /asset/{name}, not /ui/asset/{name}.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 10:25:17 +00:00
gilesb
9c158ff884 Fix media list duplicates and cache browse link
- Database: Use DISTINCT ON to deduplicate items by content_hash
- Database: Count unique content_hashes in count_user_items
- Server: Fix media card link from /ui/cache to /cache
- Server: Use /raw endpoint for image thumbnails
- Server: Add seen_hashes dedup in media list iteration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 10:17:35 +00:00
gilesb
ad63774acd Add L2 links to run detail page and publish success message
- Run detail page now shows "Published to L2" with link when already published
- Publish success message now includes "View on L2" link

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 09:48:25 +00:00
gilesb
a0a4c08b9a Use Redis for cache indexes - enables multi-worker scaling
The cache_manager now uses Redis hashes for the content_index and
ipfs_cids mappings. This allows multiple uvicorn workers to share
state, so files added by one worker are immediately visible to all
others.

- Added redis_client parameter to L1CacheManager
- Index lookups check Redis first, then fall back to in-memory
- Index updates go to both Redis and JSON file (backup)
- Migrates existing JSON indexes to Redis on first load
- Re-enabled workers=4 in uvicorn

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 04:19:00 +00:00
gilesb
6fc3562d22 Remove workers from L1 - causes cache inconsistency
With multiple workers, each process has its own in-memory cache index.
Files added by one worker aren't visible to others, causing intermittent
404 errors.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 04:02:51 +00:00
gilesb
83244aaa05 Filter out recipes from media list
Recipes have their own section under /recipes, so exclude them
from the media list by checking node_type == "recipe".

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 03:59:03 +00:00
gilesb
5ba66ed11e Record L2 share when publishing runs from UI
When ui_publish_run publishes a run to L2, now properly records the
share in the local database so the UI can display the L2 badge and
published status.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 03:51:36 +00:00
gilesb
e88f9812a6 Fix content negotiation for browser navigation
Check Sec-Fetch-Mode: navigate header for direct browser access.
This ensures /cache/{hash} shows HTML detail page when navigating
directly, but still serves raw files for embedded <img>/<video> tags.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 03:48:35 +00:00
gilesb
92c64f1037 Fix uvicorn workers - use import string
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 03:41:30 +00:00
gilesb
c40c681d2e Add uvicorn workers and Redis timeouts
- Run uvicorn with 4 workers to handle concurrent requests
- Add socket_timeout and socket_connect_timeout to Redis client (5s)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 03:34:13 +00:00
gilesb
121496eb53 Fix get_user_context_from_cookie to be async
Make get_user_context_from_cookie and get_user_from_cookie async
since they call async get_verified_user_context. Update all callers.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 03:21:47 +00:00
gilesb
d3468cd47c Add L2 sync button and improve publish status display
- Add "Sync with L2" button on media page to fetch user's outbox
- Link asset names to L2 asset pages in publish status
- Add green "L2" badge to media list for published items
- Create /user/sync-l2 and /ui/sync-l2 endpoints

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 03:18:48 +00:00
gilesb
91f23123cd Remove .env from git tracking
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 03:03:13 +00:00
gilesb
3077cd9185 Add .env to gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 02:58:56 +00:00