Commit Graph

10 Commits

Author SHA1 Message Date
gilesb
8ab0f05a7d Add durable pending runs and recipe list debugging
- Store pending runs in PostgreSQL for durability across restarts
- Add recovery method for orphaned runs
- Increase Celery result_expires to 7 days
- Add task_reject_on_worker_lost for automatic re-queuing
- Add logging to recipe list to debug filter issues

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 20:35:00 +00:00
giles
854396680f Refactor storage: remove Redis duplication, use proper data tiers
- Recipes: Now content-addressed only (cache + IPFS), removed Redis storage
- Runs: Completed runs stored in PostgreSQL, Redis only for task_id mapping
- Add list_runs_by_actor() to database.py for paginated run queries
- Add list_by_type() to cache_manager for filtering by node_type
- Fix upload endpoint to return size and filename fields
- Fix recipe run endpoint with proper DAG input binding
- Fix get_run_service() dependency to pass database module

Storage architecture:
- Redis: Ephemeral only (sessions, task mappings with TTL)
- PostgreSQL: Permanent records (completed runs, metadata)
- Cache: Content-addressed files (recipes, media, outputs)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 14:05:31 +00:00
gilesb
9a3d6949fc Store and use activity_id for L2 links
- Add activity_id column to l2_shares table
- Store activity_id when publishing to L2
- Link to /activities/{activity_id} instead of /assets/{name}
- Falls back to asset link if no activity_id

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 18:56:23 +00:00
gilesb
5ce28abe52 Phase 2: Multiple storage configs per type with new UI structure
- Database: Add description field, remove unique constraint to allow
  multiple configs of same provider type
- UI: Main page shows provider types as cards with counts
- UI: Per-type page (/storage/type/{type}) for managing configs
- API: Add get_user_storage_by_type() for filtered queries
- Form: Add description field for distinguishing configs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 00:52:28 +00:00
gilesb
5cb8d4ec17 Add storage_backends table for user storage config
Prepares L1 for distributed storage integration with user-provided
storage backends (Pinata, web3.storage, local). The storage config
is synced from L2 or can be configured locally per actor.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 23:22:28 +00:00
gilesb
e4b9657d1b Add content-addressable runs - runs identified by hash of inputs + recipe
- Add run_cache table for fast run_id -> output lookup
- compute_run_id() computes deterministic run_id from inputs + recipe
- create_run checks L1 cache then L2 before running Celery
- If output exists on L2 but not L1, pulls from IPFS
- Saves run results to cache on completion

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 11:05:18 +00:00
gilesb
9c158ff884 Fix media list duplicates and cache browse link
- Database: Use DISTINCT ON to deduplicate items by content_hash
- Database: Count unique content_hashes in count_user_items
- Server: Fix media card link from /ui/cache to /cache
- Server: Use /raw endpoint for image thumbnails
- Server: Add seen_hashes dedup in media list iteration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 10:17:35 +00:00
gilesb
c917a0539d Migrate server.py from JSON metadata to PostgreSQL database
- Replace all load_cache_meta/save_cache_meta calls with database functions
- Update get_user_cache_hashes to async, query both DB and legacy JSON
- Replace get_user_from_cookie with get_user_context_from_cookie throughout
- Update all endpoints to use UserContext (ctx) instead of plain username
- Update render_page calls to use ctx.actor_id
- Add high-level database helper functions:
  - save_item_metadata, load_item_metadata, update_item_metadata
  - save_l2_share, get_user_items, count_user_items
- Keep legacy JSON functions for backwards compatibility during migration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 16:45:22 +00:00
gilesb
bc07a101d1 Add actor_id (user) to item_types and l2_shares tables
Multi-user support for cache items:
- item_types now includes actor_id (@user@server format)
- l2_shares now includes actor_id
- Same cache item can be owned by multiple users with different types
- Deleting severs user's connection, not the actual file
- Cache files only removed when no users reference them
- Added has_remaining_references() and cleanup_orphaned_cache_item()
- Updated all CRUD functions to include actor_id parameter

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 15:26:09 +00:00
gilesb
ba244b9ebc Add PostgreSQL + IPFS backend, rename configs to recipes
- Add PostgreSQL database for cache metadata storage with schema for
  cache_items, item_types, pin_reasons, and l2_shares tables
- Add IPFS integration as durable backing store (local cache as hot storage)
- Add postgres and ipfs services to docker-compose.yml
- Update cache_manager to upload to IPFS and track CIDs
- Rename all config references to recipe throughout server.py
- Update API endpoints: /configs/* -> /recipes/*
- Update models: ConfigStatus -> RecipeStatus, ConfigRunRequest -> RecipeRunRequest
- Update UI tabs and pages to show Recipes instead of Configs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 14:58:29 +00:00