Commit Graph

18 Commits

Author SHA1 Message Date
gilesb
5b05dbd31e Fix clear-data to actually delete run_cache entries
- discard_run now deletes from run_cache and pending_runs tables
- Add delete_run_cache() and delete_pending_run() database functions
- Previously clear-data only cleared Redis, leaving DB cache intact

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 17:42:12 +00:00
gilesb
19634a4ac5 Add friendly names system for recipes, effects, and media
- Add friendly_names table with unique constraints per actor
- Create NamingService with HMAC-signed timestamp version IDs
- Version IDs use base32-crockford encoding, always increase alphabetically
- Name normalization: spaces/underscores to dashes, lowercase, strip special chars
- Format: "my-effect 01hw3x9k" (space separator ensures uniqueness)
- Integrate naming into recipe, effect, and media uploads
- Resolve friendly names to CIDs during DAG execution
- Update effects UI to display friendly names
- Add 30 tests covering normalization, parsing, and service structure

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 14:02:17 +00:00
gilesb
00b4934bf6 Remove database migration blocks
Clean schema without migration code for fresh deployments.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 08:19:40 +00:00
gilesb
faa54b2e85 Add database migration for content_hash -> cid rename
- Add DO block to migrate existing columns:
  - cache_items.content_hash -> cid
  - item_types.content_hash -> cid
  - l2_shares.content_hash -> cid
  - storage_pins.content_hash -> cid
  - run_cache.output_hash -> output_cid
- Fix duplicate key bug in upload response

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 08:18:23 +00:00
gilesb
92d26b2b72 Rename content_hash/output_hash to cid throughout
Refactor to use IPFS CID as the primary content identifier:
- Update database schema: content_hash -> cid, output_hash -> output_cid
- Update all services, routers, and tasks to use cid terminology
- Update HTML templates to display CID instead of hash
- Update cache_manager parameter names
- Update README documentation

This completes the transition to CID-only content addressing.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 08:02:44 +00:00
gilesb
5c3558e1ba Fix DAG visualization and step link handling
- Handle dict inputs ({"node": "id"}) when building DAG edges
- Add normalize_inputs() to convert dict inputs to node IDs for steps
- Fix _parse_inputs to use _json.loads (correct import alias)
- Add SOURCE/EFFECT/SEQUENCE colors to node color maps

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 00:05:50 +00:00
gilesb
7a4cd3d413 Fix inputs parsing in runs list
- Parse inputs JSON string to list in list_runs_by_actor
- Add status field to completed runs
- Prevents "136 files" display when inputs is a string

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 23:57:49 +00:00
gilesb
d92e493689 Fix inputs JSON parsing in pending_runs database functions
JSONB columns may return strings in some cases - explicitly parse
inputs field to ensure it's always a list.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 20:38:19 +00:00
gilesb
8ab0f05a7d Add durable pending runs and recipe list debugging
- Store pending runs in PostgreSQL for durability across restarts
- Add recovery method for orphaned runs
- Increase Celery result_expires to 7 days
- Add task_reject_on_worker_lost for automatic re-queuing
- Add logging to recipe list to debug filter issues

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 20:35:00 +00:00
giles
854396680f Refactor storage: remove Redis duplication, use proper data tiers
- Recipes: Now content-addressed only (cache + IPFS), removed Redis storage
- Runs: Completed runs stored in PostgreSQL, Redis only for task_id mapping
- Add list_runs_by_actor() to database.py for paginated run queries
- Add list_by_type() to cache_manager for filtering by node_type
- Fix upload endpoint to return size and filename fields
- Fix recipe run endpoint with proper DAG input binding
- Fix get_run_service() dependency to pass database module

Storage architecture:
- Redis: Ephemeral only (sessions, task mappings with TTL)
- PostgreSQL: Permanent records (completed runs, metadata)
- Cache: Content-addressed files (recipes, media, outputs)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 14:05:31 +00:00
gilesb
9a3d6949fc Store and use activity_id for L2 links
- Add activity_id column to l2_shares table
- Store activity_id when publishing to L2
- Link to /activities/{activity_id} instead of /assets/{name}
- Falls back to asset link if no activity_id

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 18:56:23 +00:00
gilesb
5ce28abe52 Phase 2: Multiple storage configs per type with new UI structure
- Database: Add description field, remove unique constraint to allow
  multiple configs of same provider type
- UI: Main page shows provider types as cards with counts
- UI: Per-type page (/storage/type/{type}) for managing configs
- API: Add get_user_storage_by_type() for filtered queries
- Form: Add description field for distinguishing configs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-10 00:52:28 +00:00
gilesb
5cb8d4ec17 Add storage_backends table for user storage config
Prepares L1 for distributed storage integration with user-provided
storage backends (Pinata, web3.storage, local). The storage config
is synced from L2 or can be configured locally per actor.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 23:22:28 +00:00
gilesb
e4b9657d1b Add content-addressable runs - runs identified by hash of inputs + recipe
- Add run_cache table for fast run_id -> output lookup
- compute_run_id() computes deterministic run_id from inputs + recipe
- create_run checks L1 cache then L2 before running Celery
- If output exists on L2 but not L1, pulls from IPFS
- Saves run results to cache on completion

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 11:05:18 +00:00
gilesb
9c158ff884 Fix media list duplicates and cache browse link
- Database: Use DISTINCT ON to deduplicate items by content_hash
- Database: Count unique content_hashes in count_user_items
- Server: Fix media card link from /ui/cache to /cache
- Server: Use /raw endpoint for image thumbnails
- Server: Add seen_hashes dedup in media list iteration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-09 10:17:35 +00:00
gilesb
c917a0539d Migrate server.py from JSON metadata to PostgreSQL database
- Replace all load_cache_meta/save_cache_meta calls with database functions
- Update get_user_cache_hashes to async, query both DB and legacy JSON
- Replace get_user_from_cookie with get_user_context_from_cookie throughout
- Update all endpoints to use UserContext (ctx) instead of plain username
- Update render_page calls to use ctx.actor_id
- Add high-level database helper functions:
  - save_item_metadata, load_item_metadata, update_item_metadata
  - save_l2_share, get_user_items, count_user_items
- Keep legacy JSON functions for backwards compatibility during migration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 16:45:22 +00:00
gilesb
bc07a101d1 Add actor_id (user) to item_types and l2_shares tables
Multi-user support for cache items:
- item_types now includes actor_id (@user@server format)
- l2_shares now includes actor_id
- Same cache item can be owned by multiple users with different types
- Deleting severs user's connection, not the actual file
- Cache files only removed when no users reference them
- Added has_remaining_references() and cleanup_orphaned_cache_item()
- Updated all CRUD functions to include actor_id parameter

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 15:26:09 +00:00
gilesb
ba244b9ebc Add PostgreSQL + IPFS backend, rename configs to recipes
- Add PostgreSQL database for cache metadata storage with schema for
  cache_items, item_types, pin_reasons, and l2_shares tables
- Add IPFS integration as durable backing store (local cache as hot storage)
- Add postgres and ipfs services to docker-compose.yml
- Update cache_manager to upload to IPFS and track CIDs
- Rename all config references to recipe throughout server.py
- Update API endpoints: /configs/* -> /recipes/*
- Update models: ConfigStatus -> RecipeStatus, ConfigRunRequest -> RecipeRunRequest
- Update UI tabs and pages to show Recipes instead of Configs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 14:58:29 +00:00