Commit Graph

53 Commits

Author SHA1 Message Date
98aee1f656 Add scrape_to_snapshot.py entry point with OAuth device flow login
Restores the missing entry point script for API-mode scraping.
Calls get_access_token() before starting to trigger device flow
login if no token is saved.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 16:06:28 +00:00
81112c716b Decouple cart/market DBs: denormalize product data, AP internal inbox, OAuth scraper auth
Remove cross-DB relationships (CartItem.product, CartItem.market_place,
OrderItem.product) that break with per-service databases. Denormalize
product and marketplace fields onto cart_items/order_items at write time.

- Add AP internal inbox infrastructure (shared/infrastructure/internal_inbox*)
  for synchronous inter-service writes via HMAC-authenticated POST
- Cart inbox blueprint handles Add/Remove/Update rose:CartItem activities
- Market app sends AP activities to cart inbox instead of writing CartItem directly
- Cart services use denormalized columns instead of cross-DB hydration/joins
- Add marketplaces-by-ids data endpoint to market service
- Alembic migration adds denormalized columns to cart_items and order_items
- Add OAuth device flow auth to market scraper persist_api (artdag client pattern)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 14:49:04 +00:00
cf7fbd8e9b Fix cross-DB query: market app cart_items via data endpoint
market_context() was querying CartItem directly via g.s (db_market),
but cart_items lives in db_cart. Replace with fetch_data("cart",
"cart-items") and add the corresponding data endpoint.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 13:52:09 +00:00
00249dd2a9 Fix nh3 panic: use link_rel param instead of rel in attributes
nh3 manages the rel attribute internally — setting it in
tag_attributes triggers an assertion. Use link_rel parameter instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 13:35:06 +00:00
c015f3f02f Security audit: fix IDOR, add rate limiting, HMAC auth, token hashing, XSS sanitization
Critical: Add ownership checks to all order routes (IDOR fix).
High: Redis rate limiting on auth endpoints, HMAC-signed internal
service calls replacing header-presence-only checks, nh3 HTML
sanitization on ghost_sync and product import, internal auth on
market API endpoints, SHA-256 hashed OAuth grant/code tokens.
Medium: SECRET_KEY production guard, AP signature enforcement,
is_admin param removal, cart_sid validation, SSRF protection on
remote actor fetch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 13:30:27 +00:00
404449fcab Fix auth ordering: validate grant before loading user
_load_user ran before _check_auth_state, so g.user was set to the wrong
user before the grant check could clear the stale session. Now grant
verification runs first, ensuring stale sessions are cleared before
the user is loaded.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 12:22:35 +00:00
984ef9c65e Fix session security: clear stale sessions after DB rebuild
Two issues fixed:
- Sessions with uid but no grant_token (legacy or corrupt) were not
  validated at all, allowing a user to be logged in as whoever got
  their old numeric user ID after a DB rebuild
- DB errors during grant verification silently kept stale sessions
  alive; now treated as invalid to fail-safe

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 12:17:53 +00:00
6f0965aa9c Fix alembic path in entrypoints for both Docker and dev
In Docker images, alembic.ini is at /app/alembic.ini (no service subdir).
In dev, it's at /app/{service}/alembic.ini (bind-mounted). Now tries the
service dir first, falls back to current dir.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 12:13:03 +00:00
e65bd41ebe Decouple per-service Alembic migrations and fix cross-DB queries
Each service (blog, market, cart, events, federation, account) now owns
its own database schema with independent Alembic migrations. Removes the
monolithic shared/alembic/ that ran all migrations against a single DB.

- Add per-service alembic.ini, env.py, and 0001_initial.py migrations
- Add shared/db/alembic_env.py helper with table-name filtering
- Fix cross-DB FK in blog/models/snippet.py (users lives in db_account)
- Fix cart_impl.py cross-DB queries: fetch products and market_places
  via internal data endpoints instead of direct SQL joins
- Fix blog ghost_sync to fetch page_configs from cart via data endpoint
- Add products-by-ids and page-config-ensure data endpoints
- Update all entrypoint.sh to create own DB and run own migrations
- Cart now uses db_cart instead of db_market
- Add docker-compose.dev.yml, dev.sh for local development
- CI deploys both rose-ash swarm stack and rose-ash-dev compose stack
- Fix Quart namespace package crash (root_path in factory.py)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 12:07:24 +00:00
bde2fd73b8 Rename stack from coop to rose-ash in CI deployment
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 09:40:22 +00:00
5cca66574e Trigger deployment
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 01:02:51 +00:00
giles
e9848653d7 Reduce Hypercorn workers to 1 per app to save ~600MB RAM
Each app was running 2 workers (~100MB each). On a 3.8GB system with
6 apps, the 12 workers consumed ~1.2GB and caused swap pressure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 23:09:14 +00:00
giles
c3ba28ea03 Add device authorization flow (RFC 8628) for CLI login
Implements the device code grant flow so artdag CLI can authenticate
via browser approval. Includes device/authorize, device/token endpoints,
user code verification page, and approval confirmation template.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 19:41:09 +00:00
giles
b9fe884ab9 Fix order.page_config → page_config_id in checkout return
The page_config relationship was removed during DB split; use the
page_config_id column and fetch page config via HTTP instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 15:00:04 +00:00
giles
3797a0c7c9 Fix error page loop + account startup timeout
- Error handlers for FragmentError and generic Exception now return
  self-contained HTML (no render_template) to avoid the infinite loop
  where context processor → fetch_fragments → error → render_template
  → context processor → fetch_fragments → error ...
- Account Ghost membership sync moved to background task so it doesn't
  block Hypercorn's startup timeout (was causing crash-loop).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 14:45:29 +00:00
giles
1ea9ae4050 Show friendly error page when a service is unavailable
FragmentError now renders a 503 page naming which service is down
instead of a generic 500 error. Helps debug during deploys.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 14:23:47 +00:00
giles
507a5a66ff Remove cross-DB menu_nodes query from non-blog apps
menu_nodes table lives in db_blog. Non-blog apps (cart, market, events,
account, federation) were querying it on their own DB session, causing
UndefinedTableError. The nav-tree fragment from blog provides the real
navigation; menu_items is now an empty fallback list.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 14:18:02 +00:00
giles
094b6c55cd Fix AP blueprint cross-DB queries + harden Ghost sync init
AP blueprints (activitypub.py, ap_social.py) were querying federation
tables (ap_actor_profiles etc.) on g.s which points to the app's own DB
after the per-app split. Now uses g._ap_s backed by get_federation_session()
for non-federation apps.

Also hardens Ghost sync before_app_serving to catch/rollback on failure
instead of crashing the Hypercorn worker.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 14:06:42 +00:00
giles
97d2021a00 Rollback session when advisory lock not acquired
Prevents PgBouncer connection pool from inheriting dirty
transaction state when the non-syncing worker returns.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 13:40:05 +00:00
giles
9f29073cda Fix Ghost sync race: advisory lock for multi-worker startup
Two Hypercorn workers both run sync_all_content_from_ghost on startup,
racing on PostAuthor/PostTag rows. Use pg_try_advisory_lock so only
one worker runs the sync.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 13:32:28 +00:00
giles
c53f3025d9 Fix no_autoflush: use manual toggle for async session
AsyncSession.no_autoflush is a sync context manager, can't use
with 'async with'. Toggle autoflush manually instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 12:06:54 +00:00
giles
3053cb321d Decouple PageConfig cross-domain queries + merge cart into db_market
PageConfig (db_blog) decoupling:
- Blog: add page-config, page-config-by-id, page-configs-batch data endpoints
- Blog: add update-page-config action endpoint for events payment admin
- Cart: hydrate_page, resolve_page_config, get_cart_grouped_by_page all
  fetch PageConfig from blog via HTTP instead of direct DB query
- Cart: check_sumup_status auto-fetches page_config from blog when needed
- Events: payment routes read/write PageConfig via blog HTTP endpoints
- Order model: remove cross-domain page_config ORM relationship (keep column)

Cart + Market DB merge:
- Cart tables (cart_items, orders, order_items) moved into db_market
- Cart app DATABASE_URL now points to db_market (same bounded context)
- CartItem.product / CartItem.market_place relationships work again
  (same database, no cross-domain join issues)
- Updated split-databases.sh, init-databases.sql, docker-compose.yml

Ghost sync fix:
- Wrap PostAuthor/PostTag delete+re-add in no_autoflush block
- Use synchronize_session="fetch" to keep identity map consistent
- Prevents query-invoked autoflush IntegrityError on composite PK

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 11:59:35 +00:00
giles
3be287532d Fix post_authors duplicate key during Ghost sync
Add explicit flush after DELETE and dedup authors/tags to prevent
autoflush-triggered IntegrityError on composite PK.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 11:47:50 +00:00
giles
95bd32bd71 Decouple cross-domain DB queries for per-app database split
Move Ghost membership sync from blog to account service so blog no
longer queries account tables (users, ghost_labels, etc.). Account
runs membership sync at startup and exposes HTTP action/data endpoints
for webhook-triggered syncs and user lookups.

Key changes:
- account/services/ghost_membership.py: all membership sync functions
- account/bp/actions + data: ghost-sync-member, user-by-email, newsletters
- blog ghost_sync.py: stripped to content-only (posts, authors, tags)
- blog webhook member: delegates to account via call_action()
- try_publish: opens federation session when DBs differ
- oauth.py callback: uses get_account_session() for OAuthCode
- page_configs moved from db_events to db_blog in split script

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 11:32:14 +00:00
giles
50a9e5d952 Fix pgbouncer port and activate DB split
edoburu/pgbouncer listens on 5432, not 6432. Update all DATABASE_URL
entries to use pgbouncer:5432. Activate per-app DATABASE_URL split
(no longer commented out).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 10:59:51 +00:00
giles
961067841e Tier 0 scalability: PgBouncer, Redis split, DB split, workers
T0.1: Separate redis-auth service (64mb, noeviction) for auth state
T0.2: Bump data Redis from 256mb to 1gb
T0.3: Per-app DATABASE_URL via PgBouncer to per-domain databases
T0.4: PgBouncer service (transaction mode, pool=20, max_conn=300);
      session.py pools reduced to 3+5 with timeout and recycle
T0.5: Hypercorn --workers 2 --keep-alive 75 on all 6 entrypoints

Deploy requires running split-databases.sh first to create per-domain
databases from the existing appdb.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 10:10:18 +00:00
giles
0ccf897f74 Route outbound Follow through EventProcessor for retry
send_follow now emits a Follow activity via emit_activity() instead of
inline HTTP POST. New ap_follow_handler delivers to the remote inbox;
EventProcessor retries on failure. Wildcard delivery handler skips
Follow type to avoid duplicate broadcast.

Also add /social/ index page to per-app social blueprint.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 09:25:08 +00:00
giles
c6271931a6 Show per-app actor in follow notifications on Hub
Add app_domain to APNotification model and NotificationDTO so follow
notifications display "followed you on blog" instead of just "followed
you" when the follow targets a per-app actor.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 09:17:34 +00:00
giles
99ab363cfd Add per-app AP social UI for blog, market, and events
Lightweight social pages (search, follow/unfollow, followers, following,
actor timeline) auto-registered for AP-enabled apps via shared blueprint.
Federation keeps the full social hub. Followers scoped per app_domain;
post cards show "View on Hub" link instead of interaction buttons.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 08:45:59 +00:00
giles
8680ec37d6 Fix container nav template to render fragment HTML
Templates still used the old container_nav_widgets variable from the
widget system. Updated to render container_nav_html (fragment output)
which the context processor already provides.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:28:36 +00:00
giles
b91a58f30a Exempt internal action/data requests from CSRF protection
Internal service-to-service POSTs (call_action) were blocked by CSRF
middleware since they have no session cookie. These requests are already
gated by X-Internal-Action/X-Internal-Data headers and only reachable
on the Docker overlay network.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:19:10 +00:00
giles
8f4104a4bf Add error handling to action endpoint dispatchers
Unhandled exceptions in action handlers were returned as opaque
400/500 by Quart's default error handler. Now we catch, log the
full traceback, and return a JSON error body with 500 status so
the caller gets useful diagnostics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:18:06 +00:00
giles
e454187035 Fix dict attribute access in market hydrate_market
post is a raw dict from fetch_data, not a DTO — use post["id"]
instead of post.id.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:11:29 +00:00
giles
f6cdf126e4 Fix blog home route — replace services.cart with fetch_data
Missed cross-app call in blog/bp/blog/routes.py:127 caused
CartService not registered error on blog.rose-ash.com homepage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:08:24 +00:00
giles
3b707ec8a0 Decouple all cross-app service calls to HTTP endpoints
Replace every direct cross-app services.* call with HTTP-based
communication: call_action() for writes, fetch_data() for reads.
Each app now registers only its own domain service.

Infrastructure:
- shared/infrastructure/actions.py — POST client for /internal/actions/
- shared/infrastructure/data_client.py — GET client for /internal/data/
- shared/contracts/dtos.py — dto_to_dict/dto_from_dict serialization

Action endpoints (writes):
- events: 8 handlers (ticket adjust, claim/confirm, toggle, adopt)
- market: 2 handlers (create/soft-delete marketplace)
- cart: 1 handler (adopt cart for user)

Data endpoints (reads):
- blog: 4 (post-by-slug/id, posts-by-ids, search-posts)
- events: 10 (pending entries/tickets, entries/tickets for page/order,
  entry-ids, associated-entries, calendars, visible-entries-for-period)
- market: 1 (marketplaces-for-container)
- cart: 1 (cart-summary)

Service registration cleanup:
- blog→blog+federation, events→calendar+federation,
  market→market+federation, cart→cart only,
  federation→federation only, account→nothing
- Stubs reduced to minimal StubFederationService

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 03:01:38 +00:00
giles
5dafbdbda9 Fix FK constraint names in migration to match actual database
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 02:27:09 +00:00
giles
580f551700 Split databases and Redis — prepare infrastructure for per-domain isolation
Redis: per-app DB index (0-5) with shared auth DB 15 for SSO keys;
flushdb replaces flushall so deploys don't wipe cross-app auth state.

Postgres: drop 13 cross-domain FK constraints (migration v2t0p8q9r0),
remove dead ORM relationships, add explicit joins for 4 live ones.
Multi-engine sessions (account + federation) ready for per-domain DBs
via DATABASE_URL_ACCOUNT / DATABASE_URL_FEDERATION env vars.

All URLs initially point to the same appdb — zero behaviour change
until split-databases.sh is run to migrate data to per-domain DBs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 02:20:34 +00:00
giles
57d2a6a6e3 Fix stale g.user after grant revocation on logout
_load_user runs before _check_auth_state, so g.user was already set
when the grant was found revoked. The session was cleared but g.user
stayed populated, causing the template to render the signed-in UI
for one request after logout. Now sets g.user = None alongside the
session clear.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 01:42:02 +00:00
giles
e4c7432303 Remove widget system — fully replaced by fragment composition
Delete shared/contracts/widgets.py, shared/services/widget_registry.py,
and shared/services/widgets/ (empty stubs). Remove register_all_widgets()
from factory and widgets Jinja global from jinja_setup. Zero consumers
remain — all cross-app UI composition now uses the fragment API.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 01:20:58 +00:00
giles
1dc87d0f64 Add OAuth SSO, device ID, and silent auth to L2
- Replace L2's username/password auth with OAuth SSO via account.rose-ash.com
- Add device_id middleware (artdag_did cookie)
- Add silent auth check (prompt=none with 5-min cooldown)
- Add OAuth config settings and itsdangerous dependency
- Register artdag_l2 client with account service

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 23:50:24 +00:00
giles
66c0c23de9 Add Art-DAG to coop nav-tree fragment
Append synthetic artdag nav entry to blog's nav-tree handler so
Art-DAG appears in the shared navigation across all 6 coop apps.
Register artdag_url as Jinja global.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:28:21 +00:00
giles
660a6db6fb Fix INTERNAL_URL_ARTDAG: use Docker internal URL not external
Art-DAG l1-server runs on port 8100 and shares the externalnet overlay
network with coop apps. Using the internal URL avoids the reverse proxy
and the silent auth middleware that was 302-redirecting fragment requests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:14:12 +00:00
giles
2fe27fb34a Wire Art-DAG nav-item fragment into account dashboard nav
Fetches artdag nav-item alongside events and cart account-nav-items,
so Art-DAG link appears in the account dashboard sidebar.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:07:52 +00:00
giles
b3d853ad35 Add Phase 5: link-card fragments, oEmbed endpoints, OG meta
- fetch_fragment_batch() for N+1 avoidance with per-key Redis cache
- link-card fragment handlers in blog, market, events, federation (single + batch mode)
- link_card.html templates per app with content-specific previews
- shared/infrastructure/oembed.py: build_oembed_response, build_og_meta, build_oembed_link_tag
- GET /oembed routes on blog, market, events
- og_meta + oembed_link rendering in base template <head>
- INTERNAL_URL_ARTDAG in docker-compose.yml for cross-stack fragment fetches

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 21:44:11 +00:00
giles
4d7f8cfea2 Add fragment composition plan to docs
Includes batch fragments, consumer-side styling, viral video
distribution (AP attachments + oEmbed + Open Graph), and link-card
fragment design.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:57:51 +00:00
giles
567888c9e0 Fix cart template cross-app url_for crash and favicon 404
- Cart _cart.html: replace url_for('market.browse.product...') with
  market_product_url() for links and cart_global.update_quantity for
  quantity forms (market endpoints don't exist in cart app)
- Factory favicon route: use STATIC_DIR instead of relative "static"
  (resolves to shared/static/ where favicon.ico actually lives)
- Cart context processor: fetch all 3 fragments (cart-mini, auth-menu,
  nav-tree) concurrently, matching pattern in all other apps

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:48:23 +00:00
giles
8b52a11b67 Add cart-mini + auth-menu fragments to all apps, fix like button
- Market, events, federation, account now fetch cart-mini, auth-menu,
  and nav-tree fragments concurrently (matching blog's pattern)
- Move like button template to shared/browser/templates/ so blog can
  find it without needing market's templates in its container

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:26:20 +00:00
giles
e1f9c964f5 Update app READMEs for monorepo
Remove stale submodule references, port numbers, and Running sections.
Add fragment composition details. Create READMEs for federation and account.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:13:00 +00:00
giles
796443c06d Add monorepo README
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:10:23 +00:00
giles
a8d1c7a130 Add local deploy script for quick testing
Builds and restarts individual services without going through CI.
Supports explicit app names, --all, or auto-detection from git diff.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:09:18 +00:00