Commit Graph

163 Commits

Author SHA1 Message Date
giles
b372d02df2 Fix list_recipes to use offset parameter
Match the router's expected signature and return type.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 12:43:54 +00:00
giles
d19d6d6e42 Fix RecipeService initialization with cache manager
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 12:34:39 +00:00
giles
eed4596af8 Update base.html to extend _base.html
Matches renamed template in artdag-common package.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 12:28:26 +00:00
giles
83cce09b1a Fix list_runs to use offset parameter
Match the router's expected signature and return type.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 12:22:59 +00:00
giles
0ec1de3cb7 Add httpx dependency
Required by auth_service module.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 12:18:11 +00:00
giles
cc6be54a92 Add artdag-common dependency 2026-01-11 11:55:40 +00:00
giles
29da91e01a Refactor to modular app factory architecture
- Replace 6847-line monolithic server.py with 26-line entry point
- All routes now in app/routers/ using Jinja2 templates
- Add plan_node.html template for step details
- Add plan node route to runs router with cache_id lookup
- Backup old server as server_legacy.py

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 11:48:24 +00:00
giles
73b7f173c5 Use cache_id for plan node details and show input/output media
- Changed node click to use cache_id in URL instead of step_id
- Updated route to lookup step by cache_id
- Added input media previews showing thumbnails of each input step
- Enhanced output preview with video/image/audio support
- Added parameters section showing step config
- Updated JavaScript to pass cacheId when clicking nodes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 11:44:21 +00:00
giles
17b92c77ef Fix async function call for load_plan_for_run_with_fallback
Use await directly instead of asyncio.to_thread() since the
function is already async.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 11:31:27 +00:00
giles
64ef9396d6 Add /help routes to display README documentation
Provides /help index and /help/{doc_name} routes to view
L1 server and Common library READMEs in the web UI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 11:30:36 +00:00
giles
255d44fbf6 Remove redundant documentation UI routes
/docs now correctly points to FastAPI's Swagger API docs.
README files can be viewed directly in the git repository.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 11:21:16 +00:00
giles
de9fbaed4a Add debug logging for plan loading 2026-01-11 10:24:58 +00:00
giles
c68c0cedba Fix plan node detail to generate plan from recipe if not cached
- Add load_plan_for_run_with_fallback() that generates plan from recipe
  when not found in cache
- Share this helper between plan page and node detail endpoint
- Removes code duplication

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 10:23:03 +00:00
giles
45eb9c0b1c Fix plan loading and add inline DAG view for recipes
- Add load_plan_for_run() helper that tries plan_id first, then matches by inputs
- Fix "plan not found" error when clicking plan nodes
- Add inline DAG visualization to recipe detail page with tabs (DAG View / YAML Source)
- Recipe page now uses render_page_with_cytoscape for proper DAG rendering

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 10:09:07 +00:00
giles
28349ad944 Add documentation routes and update README
- Update README with comprehensive documentation covering IPFS-primary mode,
  3-phase execution, storage providers, and all API endpoints
- Add /docs routes to serve markdown documentation as styled HTML
- Include common library documentation in web interface

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 09:58:31 +00:00
gilesb
ca8bfd8705 Add hybrid state manager for distributed L1 coordination
Implements HybridStateManager providing fast local Redis operations
with background IPNS sync for eventual consistency across L1 nodes.

- hybrid_state.py: Centralized state management (cache, claims, analysis, plans, runs)
- Updated execute_cid.py, analyze_cid.py, orchestrate_cid.py to use state manager
- Background IPNS sync (configurable interval, disabled by default)
- Atomic claiming with Redis SETNX for preventing duplicate work

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 09:36:14 +00:00
giles
f11cec9d48 Add SPA-style navigation for plan nodes
- Add /run/{run_id}/plan/node/{step_id} endpoint for node details
- Node click updates URL without full page reload (pushState)
- Browser back/forward works correctly
- Refreshing page preserves selected node via ?node= parameter
- Node details loaded via fetch with partial HTML response

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 09:30:27 +00:00
gilesb
0634142576 Add ARTDAG_CLUSTER_KEY to docker-compose
Reads from environment or .env file.
Must be same on server and workers for consistent cache_ids.

Generate with: openssl rand -hex 32

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 09:17:41 +00:00
giles
c145d4a427 Add IPFS_PRIMARY mode UI support
- Add output_ipfs_cid field to RunStatus model
- Handle output_cid from IPFS-primary task results
- Add /ipfs/{cid} redirect route to IPFS gateway
- Add /ipfs/{cid}/raw to fetch and serve IPFS content
- Show IPFS output in run detail when output_hash unavailable
- Display step CIDs on plan page for IPFS_PRIMARY runs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 08:32:27 +00:00
gilesb
68c2e45541 Add IPFS_PRIMARY environment variable for server
When IPFS_PRIMARY=true:
- /api/run-recipe uses run_recipe_cid task
- Recipe registered on IPFS before execution
- Input CIDs fetched from cache manager
- Everything flows through IPFS, no local cache

Usage in docker-compose:
  environment:
    - IPFS_PRIMARY=true

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 08:27:24 +00:00
giles
791fe31483 Fix double-encoded JSON in Plan JSON display
Parse nested plan_json field before displaying to avoid escaped newlines.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 08:19:44 +00:00
gilesb
25f7213741 Add IPFS-primary orchestration
Complete pipeline with everything on IPFS:
- register_input_cid / register_recipe_cid
- generate_plan_cid (stores plan on IPFS)
- execute_plan_from_cid (fetches plan from IPFS)
- run_recipe_cid (full pipeline, returns output CID)
- run_from_local (convenience: local files → IPFS → run)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 08:18:34 +00:00
gilesb
92d154f524 Add IPFS-primary analysis task
- Fetches input from IPFS by CID
- Stores analysis JSON on IPFS
- Returns analysis_cid
- Redis cache: input_hash:features → analysis CID

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 08:15:55 +00:00
giles
1869a76157 Fix CACHE_DIR default path mismatch across files
Changed default from /data/cache to ~/.artdag/cache for local runs.
Docker sets CACHE_DIR=/data/cache via environment variable.

Files updated:
- tasks/analyze.py
- tasks/orchestrate.py
- app/config.py

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 08:14:39 +00:00
giles
8fd9acce28 Add step list with cache links and Plan JSON to plan page
- Show numbered step list with cache_id links for each completed step
- Add collapsible "Show Plan JSON" section with full plan data
- Steps show type, name, status badge, and clickable output link

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 08:12:39 +00:00
giles
945fb3b413 Fix analysis display for recipe-based runs
Add get_run_analysis() to RunService to load per-input analysis from
CACHE_DIR/analysis/{hash}.json files. Update runs router and template
to display tempo, beats, energy, and beat timeline visualization.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 08:05:40 +00:00
giles
29d8d06d76 Fix plan artifacts display for completed runs
- Add get_run_plan() and get_run_artifacts() methods to RunService
- Merge step_results into plan steps to show cache_id per step
- Display output hash links under each completed step
- Use cache manager for artifact path lookups
- Fix RunService constructor to accept database param

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 08:01:12 +00:00
giles
ae68eeef52 Add Plan JSON view to run detail page
Shows collapsible JSON of execution plan below steps list.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 07:57:56 +00:00
giles
022f88bf0c Complete L1 router and template migration
- Full implementation of runs, recipes, cache routers with templates
- Auth and storage routers fully migrated
- Jinja2 templates for all L1 pages
- Service layer for auth and storage

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 07:49:37 +00:00
gilesb
383dbf6e03 Add IPFS-primary execute_step_cid implementation
Simplified step execution where:
- Steps receive CIDs, produce CIDs
- No local cache management (IPFS handles it)
- Minimal Redis: just claims + cache_id→CID mapping
- Temp workspace for execution, cleaned up after

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 07:42:34 +00:00
giles
adc876dbd6 Add modular app structure for L1 server refactoring
Phase 2 of the full modernization:
- App factory pattern with create_app()
- Settings via dataclass with env vars
- Dependency injection container
- Router stubs for auth, storage, api, recipes, cache, runs
- Service layer stubs for run, recipe, cache
- Repository layer placeholder

Routes are stubs that import from legacy server.py during migration.
Next: Migrate each router fully with templates.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 07:08:08 +00:00
giles
3e24156c99 Fix undefined completed_count variable in plan visualization
Added completed_count stat for steps executed during run (vs cached_count for
steps already in cache). Also fixed pending_count calculation to account for
completed steps.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 01:24:22 +00:00
giles
b9f8b5538b Use orchestrated run_recipe for all recipe runs, remove v2 prefix
- UI recipe run now uses tasks.orchestrate.run_recipe (3-phase)
- Deterministic run_id via compute_run_id for cache deduplication
- Check for already-completed runs before starting
- Rename /api/v2/* endpoints to /api/* (plan, execute, run-recipe, run)
- All recipe runs now go through: analyze → plan → execute

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 01:07:14 +00:00
giles
41ceae1e6c Fix plan visualization to show completed status and artifact links
- Check cache_manager.has_content(cache_id) to determine if step is cached
- Show green border for cached nodes in completed runs
- Display artifact preview (video) when clicking on cached nodes
- Add "View" button to access cached artifacts directly
- Simplify node data structure (hasCached instead of outputs array)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 01:03:11 +00:00
gilesb
3db606bf15 Plan-based caching with artifact playback in UI
RunStatus now stores:
- plan_id, plan_name for linking to execution plan
- step_results for per-step execution status
- all_outputs for all artifacts from all steps

Plan visualization:
- Shows human-readable step names from recipe structure
- Video/audio artifact preview on node click
- Outputs list with links to cached artifacts
- Stats reflect actual execution status (completed/cached/pending)

Execution:
- Step results include outputs list with cache_ids
- run_plan returns all outputs from all steps
- Support for completed_by_other status

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 00:20:19 +00:00
giles
36bf0cd3f7 Generate execution plans on-the-fly for plan visualization
Instead of only showing cached plans, now attempts to generate
a plan from the recipe if no cached plan is found. This works
for recipe-based runs even if they were started via the legacy
execute_dag path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 23:33:38 +00:00
giles
b498d6fe4b Enable Celery events for Flower monitoring
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 23:27:36 +00:00
giles
e7d3a5ed6c Add DAG visualization for runs and recipes
- Add Cytoscape.js infrastructure for interactive DAG visualization
- New run sub-pages: /run/{id}/plan, /run/{id}/analysis, /run/{id}/artifacts
- Plan page shows execution DAG with cached/pending status
- Analysis page displays tempo, beats, energy per input
- Artifacts page lists all cached items with thumbnails
- New /recipe/{id}/dag page for recipe structure visualization
- Add sub-tabs navigation to run detail page
- Add JSON API endpoints for future WebSocket support
- Architecture designed for real-time updates (global Cytoscape instance)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 23:27:28 +00:00
gilesb
1feecc8c05 Add Flower for Celery monitoring
Flower provides a web UI to monitor:
- Active/completed/failed tasks
- Worker status and stats
- Task history and details

Accessible on port 5555.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 22:06:17 +00:00
gilesb
e56b4792c7 Show all inputs with names on run detail page
Previously only displayed the first input. Now shows all inputs with
their names from the recipe (variable_inputs and fixed_inputs).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 19:48:07 +00:00
gilesb
9a3d6949fc Store and use activity_id for L2 links
- Add activity_id column to l2_shares table
- Store activity_id when publishing to L2
- Link to /activities/{activity_id} instead of /assets/{name}
- Falls back to asset link if no activity_id

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 18:56:23 +00:00
gilesb
3d45efa90b Link to activity instead of asset in "View on L2"
After publishing, link to /activities/{activity_id} instead of
/assets/{asset_name} so user sees the published run.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 18:38:07 +00:00
gilesb
837f9a6442 Add defensive check for null asset in publish
Prevents NoneType subscript error if L2 returns null asset.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 15:08:03 +00:00
gilesb
eaff758395 Rename tasks.py to legacy_tasks.py to avoid conflict with tasks/ package
The tasks/ directory for 3-phase execution was shadowing the old tasks.py.
Renamed to legacy_tasks.py and updated all imports.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 12:05:27 +00:00
gilesb
f7890dd1ad Add 3-phase execution with IPFS cache and hash-based task claiming
New files:
- claiming.py - Redis Lua scripts for atomic task claiming
- tasks/analyze.py - Analysis Celery task
- tasks/execute.py - Step execution with IPFS-backed cache
- tasks/orchestrate.py - Plan orchestration (run_plan, run_recipe)

New API endpoints (/api/v2/):
- POST /api/v2/plan - Generate execution plan
- POST /api/v2/execute - Execute a plan
- POST /api/v2/run-recipe - Full 3-phase pipeline
- GET /api/v2/run/{run_id} - Get run status

Features:
- Hash-based task claiming prevents duplicate work
- Parallel execution within dependency levels
- IPFS-backed cache for durability
- Integration with artdag planning module

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 11:44:00 +00:00
gilesb
7d05011daa Add Storage link to L1 navigation bars
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 02:44:52 +00:00
gilesb
2a4f24b7ee Add storage engine configuration to L1, matching L2 implementation
- Copy storage_providers.py from L2 (Pinata, web3.storage, NFT.Storage,
  Infura, Filebase, Storj, local storage providers)
- Add storage management endpoints: GET/POST/PATCH/DELETE /storage
- Add provider-specific pages at /storage/type/{provider_type}
- Include connection testing via POST /storage/{id}/test
- Add HTML UI pages with dark theme matching L2

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 02:41:50 +00:00
gilesb
5ce28abe52 Phase 2: Multiple storage configs per type with new UI structure
- Database: Add description field, remove unique constraint to allow
  multiple configs of same provider type
- UI: Main page shows provider types as cards with counts
- UI: Per-type page (/storage/type/{type}) for managing configs
- API: Add get_user_storage_by_type() for filtered queries
- Form: Add description field for distinguishing configs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 00:52:28 +00:00
gilesb
58b3240430 Store IPFS CID in database for DAG outputs
Properly unpack cache_manager.put() tuple to get IPFS CID and store
it in PostgreSQL via database.create_cache_item(). This fixes the
"Output has no IPFS CID - cannot publish" error when publishing
from L1 to L2.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 23:35:50 +00:00
gilesb
5cb8d4ec17 Add storage_backends table for user storage config
Prepares L1 for distributed storage integration with user-provided
storage backends (Pinata, web3.storage, local). The storage config
is synced from L2 or can be configured locally per actor.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 23:22:28 +00:00