Commit Graph

72 Commits

Author SHA1 Message Date
gilesb
f52ec79860 Use direct HTTP API for IPFS instead of ipfshttpclient
Replace ipfshttpclient library with direct HTTP requests to IPFS API.
This fixes compatibility with newer Kubo versions (0.39.0+) which are
not supported by the outdated ipfshttpclient library.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 19:39:49 +00:00
gilesb
c73e79fe28 Save IPFS CID to database when caching files
- Make cache_file() async and save ipfs_cid to cache_items table
- Update all call sites to use await
- Add create_cache_item call in upload endpoint

Fixes IPFS info not showing for uploaded files.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 19:02:52 +00:00
gilesb
552c4590c2 Expose IPFS swarm port 4001 for P2P connectivity
Enables full IPFS network participation - other nodes can
discover and fetch content from this node.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 18:51:25 +00:00
gilesb
66be6b3c9a Fix f-string escaping in home page template
Escape {run_id} and {hash} as {{run_id}} and {{hash}} to
prevent NameError in f-string interpolation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 18:32:00 +00:00
gilesb
9877e61b5d Send ipfs_cid when publishing cache to L2
L1 now includes IPFS CID in publish requests so L2 can
pin content on its own IPFS node for federation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 18:20:01 +00:00
gilesb
c0c3096e48 Add configurable IPFS gateway URL with local gateway as primary
- IPFS_GATEWAY_URL env var for local gateway (e.g., https://ipfs.celery-artdag.rose-ash.com)
- Local gateway shown first with green button when configured
- Removed Pinata from public gateways list

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 18:09:51 +00:00
gilesb
e25b61e6ab Add IPFS to external network for gateway access
Allows nginx to proxy to IPFS gateway at ipfs.celery-artdag.rose-ash.com

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 18:07:51 +00:00
gilesb
808e754e7f Add IPFS CID display with public gateway links
- Cache detail page shows IPFS section with CID and links to
  ipfs.io, dweb.link, Cloudflare, and Pinata gateways
- Media list cards show purple "IPFS" badge for items with CID
- JSON API response includes ipfs_cid field

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 17:56:29 +00:00
gilesb
591c0aad4c Show logged-in user on home page instead of login button
Convert static HOME_HTML to render_home_html() function that
dynamically shows user info with link to L2 profile when
authenticated, or login button when not logged in.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 17:51:09 +00:00
gilesb
c917a0539d Migrate server.py from JSON metadata to PostgreSQL database
- Replace all load_cache_meta/save_cache_meta calls with database functions
- Update get_user_cache_hashes to async, query both DB and legacy JSON
- Replace get_user_from_cookie with get_user_context_from_cookie throughout
- Update all endpoints to use UserContext (ctx) instead of plain username
- Update render_page calls to use ctx.actor_id
- Add high-level database helper functions:
  - save_item_metadata, load_item_metadata, update_item_metadata
  - save_l2_share, get_user_items, count_user_items
- Keep legacy JSON functions for backwards compatibility during migration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 16:45:22 +00:00
gilesb
66beda70c4 Move auth to L2 only, display @user@server format
- L1 never handles credentials - redirects to L2 for login/register
- L2 sets shared cookie (domain=.rose-ash.com) for cross-subdomain auth
- Display logged-in user as @user@server (ActivityPub format)
- Remove login/register form handling from L1
- Add L1_PUBLIC_URL env var for redirect callbacks
- Rename /ui/cache-list to /ui/media-list
- Update nav links to use clean URLs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 15:32:55 +00:00
gilesb
bc07a101d1 Add actor_id (user) to item_types and l2_shares tables
Multi-user support for cache items:
- item_types now includes actor_id (@user@server format)
- l2_shares now includes actor_id
- Same cache item can be owned by multiple users with different types
- Deleting severs user's connection, not the actual file
- Cache files only removed when no users reference them
- Added has_remaining_references() and cleanup_orphaned_cache_item()
- Updated all CRUD functions to include actor_id parameter

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 15:26:09 +00:00
gilesb
c0a98dd0ff Rename Cache to Media in navigation, add login to home page
- Change navigation tab from "Cache" to "Media"
- Change /cache list route to /media
- Add Recipes link to home page navigation
- Add Login button to home page
- Update all active_tab="cache" to active_tab="media"
- Update "Back to cache" links to "Back to media"

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 15:18:40 +00:00
gilesb
5e6353d150 Fix /recipes redirect loop by merging duplicate routes
Removed duplicate /recipes route that was causing infinite redirect.
Now single route handles both HTML and JSON responses directly.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 15:10:45 +00:00
gilesb
0c69c9c136 Fix ipfshttpclient version (0.7.0 is latest)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 15:04:13 +00:00
gilesb
ba244b9ebc Add PostgreSQL + IPFS backend, rename configs to recipes
- Add PostgreSQL database for cache metadata storage with schema for
  cache_items, item_types, pin_reasons, and l2_shares tables
- Add IPFS integration as durable backing store (local cache as hot storage)
- Add postgres and ipfs services to docker-compose.yml
- Update cache_manager to upload to IPFS and track CIDs
- Rename all config references to recipe throughout server.py
- Update API endpoints: /configs/* -> /recipes/*
- Update models: ConfigStatus -> RecipeStatus, ConfigRunRequest -> RecipeRunRequest
- Update UI tabs and pages to show Recipes instead of Configs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 14:58:29 +00:00
gilesb
4639a98231 lists of shares. job deletion only deltes outputs 2026-01-08 03:38:14 +00:00
gilesb
63ec895514 Update README with config endpoints
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 03:20:04 +00:00
gilesb
4a99866602 Add config/recipe support for DAG-based jobs
- Add PyYAML dependency for parsing config files
- Add Pydantic models: VariableInput, FixedInput, ConfigStatus, ConfigRunRequest
- Add Redis storage functions for configs
- Add config YAML parsing with variable and fixed input detection
- Add config API endpoints: upload, list, get, delete, run
- Add config UI: Configs tab, list page, detail page with run form
- Add HTMX endpoints for config operations
- Add pinning on publish: configs and their fixed inputs are pinned
  when runs from configs are published to L2
- Clean up debug logging in cache_manager

Config YAML format supports:
- Fixed inputs: resolve asset hashes from registry
- Variable inputs: marked with `input: true`, filled at run time
- DAG definition with nodes and edges
- Registry of assets and effects

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 03:17:50 +00:00
gilesb
f23a721816 Use local pinned metadata for deletion checks instead of L2 API
- Add is_pinned(), pin(), _load_meta(), _save_meta() to L1CacheManager
- Update can_delete() and can_discard_activity() to check local pinned status
- Update run deletion endpoints (API and UI) to check pinned metadata
- Remove L2 shared check fallback from run deletion
- Fix L2SharedChecker to return True on error (safer - prevents accidental deletion)
- Update tests for new pinned behavior

When items are published to L2, the publish flow marks them as pinned
locally. This ensures items remain non-deletable even if L2 is unreachable,
and both outputs AND inputs of published runs are protected.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 02:44:18 +00:00
gilesb
f61c82dd51 Add PYTHONPATH to Dockerfile for worker imports 2026-01-08 02:21:05 +00:00
gilesb
01f6db0621 Add debug logging to cache lookup 2026-01-08 02:11:37 +00:00
gilesb
3e12596dbf Fix cache lookup to work across processes
cache_manager.py:
- get_by_content_hash() now tries direct cache.get(content_hash)
  since uploads use content_hash as node_id
- This works even if cache index hasn't been reloaded

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 02:02:47 +00:00
gilesb
13b93ab17c Add delete button for runs in UI
- Add delete_html section to run detail page with delete button
- Add /ui/runs/{run_id}/discard HTMX endpoint
- Failed runs can always be deleted without restrictions

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 01:53:26 +00:00
gilesb
6142c8bbab Allow failed runs to always be deleted
Failed runs have no output to protect, so they can be deleted
without checking L2 shared status or activity records.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 01:48:23 +00:00
gilesb
0e4f596401 Fix render_effect to find uploads in new cache location
The legacy render_effect task was looking for inputs only at
CACHE_DIR/{hash}, but uploads now go through cache_manager which
stores them in CACHE_DIR/nodes/{node_id}/. Now uses
cache_manager.get_by_content_hash() which checks both locations.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 01:47:23 +00:00
gilesb
ff195a7ce5 Add multi-step DAG execution support
tasks.py:
- Import artdag DAG, Node, Engine, Executor
- Register executors for effect:dog, effect:identity, SOURCE
- Add execute_dag task for running full DAG workflows
- Add build_effect_dag helper for simple effect-to-DAG conversion

server.py:
- Add use_dag and dag_json fields to RunRequest
- Update create_run to support DAG mode
- Handle both legacy render_effect and new execute_dag result formats
- Import new tasks (execute_dag, build_effect_dag)

The DAG engine executes nodes in topological order with automatic
caching. This enables multi-step pipelines like: source -> effect1 ->
effect2 -> output.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 01:45:29 +00:00
gilesb
034c7542c4 Fix cache listing to include files from new structure
- Update list_all() to scan cache_dir for legacy files directly
  (old files stored as CACHE_DIR/{hash}, not CACHE_DIR/legacy/)
- Update cache listing endpoints to use cache_manager.list_all()
  instead of iterating CACHE_DIR.iterdir() directly
- This ensures uploaded files appear in the cache UI regardless
  of whether they're in the old or new cache structure

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 01:37:17 +00:00
gilesb
a97c6309d5 Fix deletion rules: runs deletable, cache items protected
- Run deletion: Handle legacy runs without activity records by
  checking L2 shared status directly (instead of failing)
- Cache deletion: Check Redis runs in addition to activity store
  to prevent deleting inputs/outputs that belong to runs
- Add find_runs_using_content() helper to check if content_hash
  is used as input or output of any run

This fixes the inverted deletion logic where runs couldn't be
deleted but their cache items could.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 01:29:03 +00:00
gilesb
f8ec42b445 Refactor cache access to use cache_manager consistently
- Remove symlink hack from cache_file() - no longer needed
- Add get_cache_path() helper for content_hash lookups
- Update all CACHE_DIR / content_hash patterns to use cache_manager
- Fix cache_manager.get_by_content_hash() to check path.exists()
- Fix legacy path lookup (cache_dir not legacy_dir)
- Update upload endpoint to use cache_manager.put()

This ensures cache lookups work correctly for both legacy files
(stored directly in CACHE_DIR) and new files (stored in nodes/).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 01:21:11 +00:00
gilesb
26768b5942 Fix cache file serving after cache_manager integration
- Update get_cached endpoint to check cache_manager first, then legacy
- Update get_cached_mp4 endpoint similarly
- Make cache_file() create symlink in legacy location for backward compat

Fixes issue where newly cached outputs weren't being served because
they were stored in new location but served from legacy location.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 01:08:40 +00:00
gilesb
ea60afb624 Force pip cache bust on deploy
- Add CACHEBUST build arg to Dockerfile
- Pass timestamp to --build-arg in deploy.sh
- Ensures artdag is re-fetched from GitHub on each deploy

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 00:59:20 +00:00
gilesb
e4fd5eb010 Integrate artdag cache with deletion rules
- Add cache_manager.py with L1CacheManager wrapping artdag Cache
- Add L2SharedChecker for checking published status via L2 API
- Update server.py to use cache_manager for storage
- Update DELETE /cache/{content_hash} to enforce deletion rules
- Add DELETE /runs/{run_id} endpoint for discarding runs
- Record activities when runs complete for deletion tracking
- Add comprehensive tests for cache manager

Deletion rules enforced:
- Cannot delete items published to L2
- Cannot delete inputs/outputs of runs
- Can delete orphaned items
- Runs can only be discarded if no items are shared

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-08 00:51:18 +00:00
gilesb
9a399cfcee Add clean URLs with content negotiation and pagination for L1
- Add content negotiation (HTML for browsers, JSON for APIs)
- Clean URLs: /runs, /cache, /run/{id}, /cache/{hash}/detail
- Auth routes: /login, /logout, /register with clean URLs
- Add render_page() helper for consistent page layout
- Add infinite scroll pagination for HTML views
- Add API pagination with ?page=1&limit=20
- Redirect old /ui/* routes to clean URLs
- Update nav links to use clean URLs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 22:55:44 +00:00
gilesb
b2e4f81edf Add cache pinning and discard functionality
- Pin items when published (published, input_to_published)
- Discard endpoint (DELETE /cache/{hash}) refuses pinned items
- UI shows pin status and discard button on cache detail
- Run publish also pins output and all inputs for provenance

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 21:36:52 +00:00
gilesb
a3a37c568a Add metadata editing UI to cache detail page
- Added /ui/cache/{hash}/meta-form endpoint for HTMX form
- Origin selector (self vs external URL)
- Description and tags fields
- Publish to L2 with asset name
- Update on L2 for already-published items

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 21:22:48 +00:00
gilesb
5f58bf117e Increase L2 publish timeout to 30 seconds
L2 needs to call back to L1 to fetch run details, which can take
longer than 10 seconds especially with network latency.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 21:13:44 +00:00
gilesb
3436a22a94 Fix iOS video playback in list views
Use video_src_for_request() for iOS MP4 transcoding in:
- ui_runs partial
- ui_cache_list partial
- ui_run_partial (also added request parameter)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 20:58:40 +00:00
gilesb
1e95badddb Update UI to use Tailwind CSS with dark theme
- Replace custom CSS with Tailwind CSS via CDN
- Consistent dark theme across all pages
- Responsive layouts with mobile-friendly breakpoints
- Updated: home, login/register, runs list, cache list, run detail, cache detail
- Modern card-based UI with hover states and status badges

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 20:44:11 +00:00
gilesb
a0a719cda5 fix: add ffmpeg to Docker image for video transcoding
Required for iOS MKV→MP4 on-the-fly transcoding.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 20:06:44 +00:00
gilesb
a5187c0f45 feat: iOS video support with on-the-fly MKV→MP4 transcoding
- Add /cache/{hash}/mp4 endpoint that transcodes MKV to MP4
- First request transcodes (using ffmpeg), result cached as {hash}.mp4
- Subsequent requests serve cached MP4 directly
- Detect iOS devices and use MP4 endpoint for video src
- Works with Cloudflare caching for efficiency

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 20:04:30 +00:00
gilesb
809d8e452b fix: add playsinline for iOS video support
iOS Safari requires playsinline attribute for inline video playback.

Note: MKV videos still won't play on iOS - only MP4/H.264 is supported.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 19:57:27 +00:00
gilesb
f9e22171de feat: add L1_PUBLIC_URL config for many-to-many L1/L2 support
- Add L1_PUBLIC_URL environment variable
- UI publish sends L1 URL to L2 so it knows where to fetch run data
- Update .env.example with new config

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 19:40:59 +00:00
gilesb
223568df3f fix: use env_file for L2 server config
- Add env_file support for L2_SERVER and L2_DOMAIN
- Fix typo in L2_SERVER URL (ros-ash -> rose-ash)
- Add .env.example with required variables

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 19:29:40 +00:00
gilesb
917146a7f3 feat: add publish to L2 button in run detail web UI
- Add POST /ui/publish-run/{run_id} endpoint for UI publishing
- Add publish form with HTMX to run detail page
- Shows publish button for completed runs with outputs
- Uses existing L2 /registry/record-run endpoint

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 19:12:46 +00:00
gilesb
85f349a7ab feat: add L1-L2 publish integration endpoints
- Add POST /cache/{hash}/publish to publish cache items to L2
- Add PATCH /cache/{hash}/republish to sync metadata updates
- Validates origin is set before publishing
- Tracks publish status in cache metadata (to_l2, asset_name, timestamps)
- Forwards auth token to L2 for authentication

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 19:08:03 +00:00
gilesb
bfa7568db4 feat: add cache metadata, folders, and collections API
- Update save_cache_meta() to support updates (not just create)
- Add GET/PATCH /cache/{hash}/meta endpoints for metadata management
- Add user data storage for folders/collections (per-user .json files)
- Add folder CRUD endpoints (/user/folders)
- Add collection CRUD endpoints (/user/collections)
- Add cache list filtering by folder, collection, and tags
- Support origin tracking (self vs external URL)
- Support tags, description, and organization metadata

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 18:57:38 +00:00
gilesb
0cbfd87711 feat: per-user cache visibility and login protection
- Run detail page requires login and ownership check
- Cache uploads require login and track uploader metadata
- Cache list filtered to show only user's own files (uploaded or from runs)
- Cache detail view requires login and ownership check
- Add helper functions for cache metadata and user hash lookup

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 18:18:16 +00:00
gilesb
e01db2ed26 fix: require login to view runs and cache in UI
- Runs list now requires login, shows only user's own runs
- Cache list now requires login to view

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 18:02:34 +00:00
gilesb
fb354d8afe fix: match both username formats in runs filter
Filter now matches both plain username and ActivityPub format
(@user@domain) to support runs created before and after the
actor ID change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-07 17:56:54 +00:00