Implement atomic publishing with IPFS and DB transactions

All publishing operations now use three-phase atomic approach:
1. Phase 1: Preparation - validate inputs, gather IPFS CIDs
2. Phase 2: IPFS operations - pin all content before any DB changes
3. Phase 3: DB transaction - all-or-nothing database commits

Changes:

ipfs_client.py:
- Add IPFSError exception class
- Add add_bytes() to store content on IPFS
- Add add_json() to store JSON documents on IPFS
- Add pin_or_raise() for synchronous pinning with error handling

db.py:
- Add transaction() context manager for atomic DB operations
- Add create_asset_tx() for transactional asset creation
- Add create_activity_tx() for transactional activity creation
- Add get_asset_by_hash_tx() for lookup within transactions
- Add asset_exists_by_name_tx() for existence check within transactions

server.py:
- Rewrite record_run:
  - Check L2 first for inputs, fall back to L1
  - Store recipe JSON on IPFS with CID in provenance
  - Auto-register input assets if not already on L2
  - All operations atomic
- Rewrite publish_cache:
  - IPFS CID now required
  - Synchronous pinning before DB commit
  - Transaction for asset + activity
- Rewrite _register_asset_impl:
  - IPFS CID now required
  - Synchronous pinning before DB commit
  - Transaction for asset + activity

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
gilesb
2026-01-09 00:59:12 +00:00
parent a0ed1ae5ae
commit 647c564c47
3 changed files with 488 additions and 137 deletions

View File

@@ -2,10 +2,11 @@
"""
IPFS client for Art DAG L2 server.
Provides functions to fetch and pin content from IPFS.
Provides functions to fetch, pin, and add content to IPFS.
Uses direct HTTP API calls for compatibility with all Kubo versions.
"""
import json
import logging
import os
import re
@@ -13,6 +14,11 @@ from typing import Optional
import requests
class IPFSError(Exception):
"""Raised when an IPFS operation fails."""
pass
logger = logging.getLogger(__name__)
# IPFS API multiaddr - default to local, docker uses /dns/ipfs/tcp/5001
@@ -147,3 +153,74 @@ def get_node_id() -> Optional[str]:
except Exception as e:
logger.error(f"Failed to get node ID: {e}")
return None
def add_bytes(data: bytes, pin: bool = True) -> str:
"""
Add bytes data to IPFS and optionally pin it.
Args:
data: Bytes to add
pin: Whether to pin the data (default: True)
Returns:
IPFS CID
Raises:
IPFSError: If adding fails
"""
try:
url = f"{IPFS_BASE_URL}/api/v0/add"
params = {"pin": str(pin).lower()}
files = {"file": ("data", data)}
response = requests.post(url, params=params, files=files, timeout=IPFS_TIMEOUT)
response.raise_for_status()
result = response.json()
cid = result["Hash"]
logger.info(f"Added to IPFS: {len(data)} bytes -> {cid}")
return cid
except Exception as e:
logger.error(f"Failed to add bytes to IPFS: {e}")
raise IPFSError(f"Failed to add bytes to IPFS: {e}") from e
def add_json(data: dict) -> str:
"""
Serialize dict to JSON and add to IPFS.
Args:
data: Dictionary to serialize and store
Returns:
IPFS CID
Raises:
IPFSError: If adding fails
"""
json_bytes = json.dumps(data, indent=2, sort_keys=True).encode('utf-8')
return add_bytes(json_bytes, pin=True)
def pin_or_raise(cid: str) -> None:
"""
Pin a CID on IPFS. Raises exception on failure.
Args:
cid: IPFS CID to pin
Raises:
IPFSError: If pinning fails
"""
try:
url = f"{IPFS_BASE_URL}/api/v0/pin/add"
params = {"arg": cid}
response = requests.post(url, params=params, timeout=IPFS_TIMEOUT)
response.raise_for_status()
logger.info(f"Pinned on IPFS: {cid}")
except Exception as e:
logger.error(f"Failed to pin on IPFS: {e}")
raise IPFSError(f"Failed to pin {cid}: {e}") from e