Fix AP blueprint cross-DB queries + harden Ghost sync init
All checks were successful
Build and Deploy / build-and-deploy (push) Successful in 1m10s
All checks were successful
Build and Deploy / build-and-deploy (push) Successful in 1m10s
AP blueprints (activitypub.py, ap_social.py) were querying federation tables (ap_actor_profiles etc.) on g.s which points to the app's own DB after the per-app split. Now uses g._ap_s backed by get_federation_session() for non-federation apps. Also hardens Ghost sync before_app_serving to catch/rollback on failure instead of crashing the Hypercorn worker. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
giles
236
.claude/plans/flickering-gathering-wilkes.md
Normal file
236
.claude/plans/flickering-gathering-wilkes.md
Normal file
@@ -0,0 +1,236 @@
|
||||
# Ticket Purchase Through Cart
|
||||
|
||||
## Context
|
||||
|
||||
Tickets (Ticket model) are currently created with state="reserved" immediately when a user clicks "Buy" (`POST /tickets/buy/`). They bypass the cart and checkout entirely — no cart display, no SumUp payment, no order linkage. The user wants tickets to flow through the cart exactly like products and calendar bookings: appear in the cart, go through checkout, get confirmed on payment. Login required. No reservation — if the event sells out before payment completes, the user gets refunded (admin handles refund; we show a notice).
|
||||
|
||||
## Current Flow vs Desired Flow
|
||||
|
||||
**Now:** Click Buy → Ticket created (state="reserved") → done (no cart, no payment)
|
||||
|
||||
**Desired:** Click Buy → Ticket created (state="pending", in cart) → Checkout → SumUp payment → Ticket confirmed
|
||||
|
||||
## Approach
|
||||
|
||||
Mirror the CalendarEntry pattern: CalendarEntry uses state="pending" to mean "in cart". We add state="pending" for Ticket. Pending tickets don't count toward availability (not allocated). At checkout, pending→reserved + linked to order. On payment, reserved→confirmed.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Update TicketDTO
|
||||
|
||||
**File:** `shared/contracts/dtos.py`
|
||||
|
||||
Add fields needed for cart display and page-grouping:
|
||||
- `entry_id: int` (for linking back)
|
||||
- `cost: Decimal` (ticket price — from ticket_type.cost or entry.ticket_price)
|
||||
- `calendar_container_id: int | None` (for page-grouping in cart)
|
||||
- `calendar_container_type: str | None`
|
||||
|
||||
Also add `ticket_count` and `ticket_total` to `CartSummaryDTO`.
|
||||
|
||||
## Step 2: Add ticket methods to CalendarService protocol
|
||||
|
||||
**File:** `shared/contracts/protocols.py`
|
||||
|
||||
```python
|
||||
async def pending_tickets(
|
||||
self, session: AsyncSession, *, user_id: int,
|
||||
) -> list[TicketDTO]: ...
|
||||
|
||||
async def claim_tickets_for_order(
|
||||
self, session: AsyncSession, order_id: int, user_id: int,
|
||||
page_post_id: int | None = None,
|
||||
) -> None: ...
|
||||
|
||||
async def confirm_tickets_for_order(
|
||||
self, session: AsyncSession, order_id: int,
|
||||
) -> None: ...
|
||||
```
|
||||
|
||||
## Step 3: Implement in SqlCalendarService
|
||||
|
||||
**File:** `shared/services/calendar_impl.py`
|
||||
|
||||
- **`pending_tickets`**: Query `Ticket` where `user_id` matches, `state="pending"`, eager-load entry→calendar + ticket_type. Map to TicketDTO with cost from `ticket_type.cost` or `entry.ticket_price`.
|
||||
- **`claim_tickets_for_order`**: UPDATE Ticket SET state="reserved", order_id=? WHERE user_id=? AND state="pending". If `page_post_id`, filter via entry→calendar→container.
|
||||
- **`confirm_tickets_for_order`**: UPDATE Ticket SET state="confirmed" WHERE order_id=? AND state="reserved".
|
||||
|
||||
Update `_ticket_to_dto` to populate the new fields (entry_id, cost, calendar_container_id/type).
|
||||
|
||||
## Step 4: Add stubs
|
||||
|
||||
**File:** `shared/services/stubs.py`
|
||||
|
||||
Add no-op stubs returning `[]`/`None` for the 3 new methods.
|
||||
|
||||
## Step 5: Update SqlCartService
|
||||
|
||||
**File:** `shared/services/cart_impl.py`
|
||||
|
||||
In `cart_summary()`, also query pending tickets via `services.calendar.pending_tickets()` and include `ticket_count` + `ticket_total` in the returned `CartSummaryDTO`.
|
||||
|
||||
## Step 6: Update cart internal API
|
||||
|
||||
**File:** `cart/bp/cart/api.py`
|
||||
|
||||
Add `ticket_count` and `ticket_total` to the JSON summary response. Query via `services.calendar.pending_tickets()`.
|
||||
|
||||
## Step 7: Add ticket cart service functions
|
||||
|
||||
**File:** `cart/bp/cart/services/calendar_cart.py`
|
||||
|
||||
Add:
|
||||
```python
|
||||
async def get_ticket_cart_entries(session):
|
||||
ident = current_cart_identity()
|
||||
if ident["user_id"] is None:
|
||||
return []
|
||||
return await services.calendar.pending_tickets(session, user_id=ident["user_id"])
|
||||
|
||||
def ticket_total(tickets) -> float:
|
||||
return sum((t.cost or 0) for t in tickets if t.cost is not None)
|
||||
```
|
||||
|
||||
**File:** `cart/bp/cart/services/__init__.py` — export the new functions.
|
||||
|
||||
## Step 8: Update cart page grouping
|
||||
|
||||
**File:** `cart/bp/cart/services/page_cart.py`
|
||||
|
||||
In `get_cart_grouped_by_page()`:
|
||||
- Fetch ticket cart entries via `get_ticket_cart_entries()`
|
||||
- Attach tickets to page groups by `calendar_container_id` (same pattern as calendar entries)
|
||||
- Add `ticket_count` and `ticket_total` to each group dict
|
||||
|
||||
## Step 9: Modify ticket buy route
|
||||
|
||||
**File:** `events/bp/tickets/routes.py` — `buy_tickets()`
|
||||
|
||||
- **Require login**: If `ident["user_id"]` is None, return error prompting sign-in
|
||||
- **Create with state="pending"** instead of "reserved"
|
||||
- **Remove availability check** at buy time (pending tickets not allocated)
|
||||
- Update response template to say "added to cart" instead of "reserved"
|
||||
|
||||
## Step 10: Update availability count
|
||||
|
||||
**File:** `events/bp/tickets/services/tickets.py` — `get_available_ticket_count()`
|
||||
|
||||
Change from counting `state != "cancelled"` to counting `state.in_(("reserved", "confirmed", "checked_in"))`. This excludes "pending" (in-cart) tickets from sold count.
|
||||
|
||||
## Step 11: Update buy form template
|
||||
|
||||
**File:** `events/templates/_types/tickets/_buy_form.html`
|
||||
|
||||
- If user not logged in, show "Sign in to buy tickets" link instead of buy form
|
||||
- Keep existing form for logged-in users
|
||||
|
||||
**File:** `events/templates/_types/tickets/_buy_result.html`
|
||||
|
||||
- Change "reserved" messaging to "added to cart"
|
||||
- Add link to cart app
|
||||
- Add sold-out refund notice: "If the event sells out before payment, you will be refunded."
|
||||
|
||||
## Step 12: Update cart display templates
|
||||
|
||||
**File:** `shared/browser/templates/_types/cart/_cart.html`
|
||||
|
||||
In `show_cart()` macro:
|
||||
- Add empty check: `{% if not cart and not calendar_cart_entries and not ticket_cart_entries %}`
|
||||
- Add tickets section after calendar bookings (same style)
|
||||
- Add sold-out notice under tickets section
|
||||
|
||||
In `summary()` and `cart_grand_total()` macros:
|
||||
- Include ticket_total in the grand total calculation
|
||||
|
||||
**File:** `shared/browser/templates/_types/cart/_mini.html`
|
||||
|
||||
- Add ticket count to the badge total
|
||||
|
||||
## Step 13: Update cart overview template
|
||||
|
||||
**File:** `cart/templates/_types/cart/overview/_main_panel.html`
|
||||
|
||||
- Add ticket count badge alongside product and calendar count badges
|
||||
|
||||
## Step 14: Update checkout flow
|
||||
|
||||
**File:** `cart/bp/cart/global_routes.py` — `checkout()`
|
||||
|
||||
- Fetch pending tickets: `get_ticket_cart_entries(g.s)`
|
||||
- Include ticket total in cart_total calculation
|
||||
- Include `not ticket_entries` in empty check
|
||||
- Pass tickets to `create_order_from_cart()` (or claim separately after)
|
||||
|
||||
**File:** `cart/bp/cart/page_routes.py` — `page_checkout()`
|
||||
|
||||
Same changes, scoped to page.
|
||||
|
||||
**File:** `cart/bp/cart/services/checkout.py` — `create_order_from_cart()`
|
||||
|
||||
- Accept new param `ticket_total: float` (add to order total)
|
||||
- After claiming calendar entries, also claim tickets: `services.calendar.claim_tickets_for_order()`
|
||||
- Include tickets in `resolve_page_config` page detection
|
||||
|
||||
## Step 15: Update payment confirmation
|
||||
|
||||
**File:** `cart/bp/cart/services/check_sumup_status.py`
|
||||
|
||||
When status == "PAID", also call `services.calendar.confirm_tickets_for_order(session, order.id)` alongside `confirm_entries_for_order`.
|
||||
|
||||
## Step 16: Update checkout return page
|
||||
|
||||
**File:** `cart/bp/cart/global_routes.py` — `checkout_return()`
|
||||
|
||||
- Also fetch tickets for order: `services.calendar.user_tickets()` filtered by order_id (or add a `get_tickets_for_order` method)
|
||||
|
||||
**File:** `shared/browser/templates/_types/order/_calendar_items.html`
|
||||
|
||||
- Add a tickets section showing ordered/confirmed tickets.
|
||||
|
||||
## Step 17: Sync shared files
|
||||
|
||||
Copy all changed shared files to blog/, cart/, events/, market/ submodules.
|
||||
|
||||
---
|
||||
|
||||
## Files Modified (Summary)
|
||||
|
||||
### Shared contracts/services:
|
||||
- `shared/contracts/dtos.py` — update TicketDTO, CartSummaryDTO
|
||||
- `shared/contracts/protocols.py` — add 3 methods to CalendarService
|
||||
- `shared/services/calendar_impl.py` — implement 3 new methods, update _ticket_to_dto
|
||||
- `shared/services/stubs.py` — add stubs
|
||||
- `shared/services/cart_impl.py` — include tickets in cart_summary
|
||||
|
||||
### Cart app:
|
||||
- `cart/bp/cart/api.py` — add ticket info to summary API
|
||||
- `cart/bp/cart/services/calendar_cart.py` — add ticket functions
|
||||
- `cart/bp/cart/services/__init__.py` — export new functions
|
||||
- `cart/bp/cart/services/page_cart.py` — include tickets in grouped view
|
||||
- `cart/bp/cart/global_routes.py` — include tickets in checkout + return
|
||||
- `cart/bp/cart/page_routes.py` — include tickets in page checkout
|
||||
- `cart/bp/cart/services/checkout.py` — include ticket total in order
|
||||
- `cart/bp/cart/services/check_sumup_status.py` — confirm tickets on payment
|
||||
|
||||
### Events app:
|
||||
- `events/bp/tickets/routes.py` — require login, state="pending"
|
||||
- `events/bp/tickets/services/tickets.py` — update availability count
|
||||
- `events/templates/_types/tickets/_buy_form.html` — login gate
|
||||
- `events/templates/_types/tickets/_buy_result.html` — "added to cart" messaging
|
||||
|
||||
### Templates (shared):
|
||||
- `shared/browser/templates/_types/cart/_cart.html` — ticket section + totals
|
||||
- `shared/browser/templates/_types/cart/_mini.html` — ticket count in badge
|
||||
- `cart/templates/_types/cart/overview/_main_panel.html` — ticket badge
|
||||
- `shared/browser/templates/_types/order/_calendar_items.html` — ticket section
|
||||
|
||||
## Verification
|
||||
|
||||
1. Go to an event entry with tickets configured (state="confirmed", ticket_price set)
|
||||
2. Click "Buy Tickets" while not logged in → should see "sign in" prompt
|
||||
3. Log in, click "Buy Tickets" → ticket created with state="pending"
|
||||
4. Navigate to cart → ticket appears alongside any products/bookings
|
||||
5. Proceed to checkout → SumUp payment page
|
||||
6. Complete payment → ticket state becomes "confirmed"
|
||||
7. Check cart mini badge shows ticket count
|
||||
8. Verify availability count doesn't include pending tickets
|
||||
425
.claude/plans/glittery-zooming-hummingbird.md
Normal file
425
.claude/plans/glittery-zooming-hummingbird.md
Normal file
@@ -0,0 +1,425 @@
|
||||
# Phase 6: Full Cross-App Decoupling via Glue Services
|
||||
|
||||
## Context
|
||||
|
||||
Phases 1-5 are complete. All cross-domain FK constraints have been dropped (except `OrderItem.product_id` and `CartItem.product_id`/`market_place_id`/`user_id`, kept as pragmatic exceptions). Cross-domain **writes** go through glue services.
|
||||
|
||||
However, **25+ cross-app model imports** remain — apps still `from blog.models.ghost_content import Post`, `from market.models.market import CartItem`, etc. This means every app needs every other app's code on disk to start, making separate databases or independent deployment impossible.
|
||||
|
||||
**Goal:** Eliminate all cross-app model imports. Every app only imports from its own `models/`, from `shared/`, and from `glue/`. Cross-domain access goes through glue services. After this phase, each app could theoretically run against its own database.
|
||||
|
||||
---
|
||||
|
||||
## Inventory of Cross-App Imports to Eliminate
|
||||
|
||||
### Cart app imports (9 files, 4 foreign models):
|
||||
| File | Import | Usage |
|
||||
|------|--------|-------|
|
||||
| `cart/bp/cart/api.py` | `market.models.market.CartItem` | Query cart items |
|
||||
| `cart/bp/cart/api.py` | `market.models.market_place.MarketPlace` | Filter by container |
|
||||
| `cart/bp/cart/api.py` | `events.models.calendars.CalendarEntry, Calendar` | Query pending entries |
|
||||
| `cart/bp/cart/api.py` | `blog.models.ghost_content.Post` | Resolve page slug |
|
||||
| `cart/bp/cart/services/checkout.py` | `market.models.market.Product, CartItem` | Find cart items, validate products |
|
||||
| `cart/bp/cart/services/checkout.py` | `events.models.calendars.CalendarEntry, Calendar` | Resolve page containers |
|
||||
| `cart/bp/cart/services/checkout.py` | `market.models.market_place.MarketPlace` | Get container_id |
|
||||
| `cart/bp/cart/services/page_cart.py` | `market.models.market.CartItem` | Query page cart |
|
||||
| `cart/bp/cart/services/page_cart.py` | `market.models.market_place.MarketPlace` | Join for container |
|
||||
| `cart/bp/cart/services/page_cart.py` | `events.models.calendars.CalendarEntry, Calendar` | Query page entries |
|
||||
| `cart/bp/cart/services/page_cart.py` | `blog.models.ghost_content.Post` | Batch-load posts |
|
||||
| `cart/bp/cart/services/get_cart.py` | `market.models.market.CartItem` | Query cart items |
|
||||
| `cart/bp/cart/services/calendar_cart.py` | `events.models.calendars.CalendarEntry` | Query pending entries |
|
||||
| `cart/bp/cart/services/clear_cart_for_order.py` | `market.models.market.CartItem` | Soft-delete items |
|
||||
| `cart/bp/cart/services/clear_cart_for_order.py` | `market.models.market_place.MarketPlace` | Filter by page |
|
||||
| `cart/bp/orders/routes.py` | `market.models.market.Product` | Join for search |
|
||||
| `cart/bp/order/routes.py` | `market.models.market.Product` | Load product details |
|
||||
| `cart/app.py` | `blog.models.ghost_content.Post` | Page slug hydration |
|
||||
|
||||
### Blog app imports (8 files, 3 foreign models):
|
||||
| File | Import | Usage |
|
||||
|------|--------|-------|
|
||||
| `blog/bp/post/admin/routes.py` | `cart.models.page_config.PageConfig` (3 places) | Load/update page config |
|
||||
| `blog/bp/post/admin/routes.py` | `events.models.calendars.Calendar` (3 places) | Query calendars |
|
||||
| `blog/bp/post/admin/routes.py` | `market.models.market_place.MarketPlace` (3 places) | Query/create/delete markets |
|
||||
| `blog/bp/post/services/markets.py` | `market.models.market_place.MarketPlace` | Create/delete markets |
|
||||
| `blog/bp/post/services/markets.py` | `cart.models.page_config.PageConfig` | Check feature flag |
|
||||
| `blog/bp/post/services/entry_associations.py` | `events.models.calendars.CalendarEntry, CalendarEntryPost, Calendar` | Post-entry associations |
|
||||
| `blog/bp/post/routes.py` | `events.models.calendars.Calendar` | Page context |
|
||||
| `blog/bp/post/routes.py` | `market.models.market_place.MarketPlace` | Page context |
|
||||
| `blog/bp/blog/ghost_db.py` | `cart.models.page_config.PageConfig` | Query page configs |
|
||||
| `blog/bp/blog/ghost/ghost_sync.py` | `cart.models.page_config.PageConfig` | Sync page config |
|
||||
| `blog/bp/blog/services/posts_data.py` | `events.models.calendars.CalendarEntry, CalendarEntryPost` | Fetch associated entries |
|
||||
|
||||
### Events app imports (5 files, 3 foreign models):
|
||||
| File | Import | Usage |
|
||||
|------|--------|-------|
|
||||
| `events/app.py` | `blog.models.ghost_content.Post` | Page slug hydration |
|
||||
| `events/app.py` | `market.models.market_place.MarketPlace` | Context processor |
|
||||
| `events/bp/markets/services/markets.py` | `market.models.market_place.MarketPlace` | Create/delete markets |
|
||||
| `events/bp/markets/services/markets.py` | `blog.models.ghost_content.Post` | Validate post exists |
|
||||
| `events/bp/markets/routes.py` | `market.models.market_place.MarketPlace` | Query/delete markets |
|
||||
| `events/bp/calendars/services/calendars.py` | `blog.models.ghost_content.Post` | Validate post exists |
|
||||
| `events/bp/calendar_entry/services/post_associations.py` | `blog.models.ghost_content.Post` | Manage post-entry assocs |
|
||||
| `events/bp/payments/routes.py` | `cart.models.page_config.PageConfig` | Load/update SumUp config |
|
||||
|
||||
### Market app imports (1 file):
|
||||
| File | Import | Usage |
|
||||
|------|--------|-------|
|
||||
| `market/app.py` | `blog.models.ghost_content.Post` | Page slug hydration |
|
||||
|
||||
### Glue layer imports (2 files):
|
||||
| File | Import | Usage |
|
||||
|------|--------|-------|
|
||||
| `glue/services/cart_adoption.py` | `market.models.market.CartItem` | Adopt cart items |
|
||||
| `glue/services/cart_adoption.py` | `events.models.calendars.CalendarEntry` | Adopt entries |
|
||||
| `glue/services/order_lifecycle.py` | `events.models.calendars.CalendarEntry, Calendar` | Claim/confirm entries |
|
||||
|
||||
---
|
||||
|
||||
## Design Decisions
|
||||
|
||||
1. **Glue services return ORM objects** (not dicts) when the model is standalone — PageConfig, MarketPlace, Calendar, CalendarEntry. This avoids template changes and keeps SQLAlchemy lazy-load working.
|
||||
|
||||
2. **Glue services for Post return dicts** — other apps only need `{id, slug, title, is_page, feature_image}`. Returning the full ORM object would couple them to the blog schema.
|
||||
|
||||
3. **CartItem stays in `market/models/market.py`** — it has FKs to `products.id`, `market_places.id`, and `users.id`, plus relationships to `Product`, `MarketPlace`, and `User`. Moving it to cart/ would just reverse the cross-app import direction. Instead, cart reads CartItem through glue.
|
||||
|
||||
4. **OrderItem.product relationship uses string forward-ref** — already works via SQLAlchemy string resolution as long as Product is registered in the mapper. Glue setup handles this.
|
||||
|
||||
5. **Glue services are allowed to import from any app's models** — that's the glue layer's job. Apps call glue; glue touches models.
|
||||
|
||||
6. **blog/bp/post/services/markets.py and entry_associations.py move to glue** — these are pure cross-domain CRUD (blog writes to MarketPlace, blog reads CalendarEntry). They belong in glue.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Glue service for pages (Post access)
|
||||
|
||||
New file: `glue/services/pages.py`
|
||||
|
||||
Provides dict-based Post access for non-blog apps:
|
||||
|
||||
```python
|
||||
async def get_page_by_slug(session, slug) -> dict | None:
|
||||
"""Return {id, slug, title, is_page, feature_image, ...} or None."""
|
||||
|
||||
async def get_page_by_id(session, post_id) -> dict | None:
|
||||
"""Return page dict by id."""
|
||||
|
||||
async def get_pages_by_ids(session, post_ids) -> dict[int, dict]:
|
||||
"""Batch-load pages. Returns {id: page_dict}."""
|
||||
|
||||
async def page_exists(session, post_id) -> bool:
|
||||
"""Check if post exists (for validation before creating calendars/markets)."""
|
||||
|
||||
async def is_page(session, post_id) -> bool:
|
||||
"""Check if post exists and is_page=True."""
|
||||
|
||||
async def search_posts(session, query, page=1, per_page=10) -> tuple[list[dict], int]:
|
||||
"""Search posts by title (for events post_associations)."""
|
||||
```
|
||||
|
||||
All functions import `from blog.models.ghost_content import Post` internally.
|
||||
|
||||
**Files changed:**
|
||||
- `market/app.py` — replace `from blog.models.ghost_content import Post` with `from glue.services.pages import get_page_by_slug`
|
||||
- `events/app.py` — same
|
||||
- `cart/app.py` — same
|
||||
- `cart/bp/cart/api.py` — replace Post import with `from glue.services.pages import get_page_by_slug`
|
||||
- `cart/bp/cart/services/page_cart.py` — replace Post import with `from glue.services.pages import get_pages_by_ids`
|
||||
- `events/bp/calendars/services/calendars.py` — replace `from blog.models.ghost_content import Post` with `from glue.services.pages import page_exists, is_page`
|
||||
- `events/bp/markets/services/markets.py` — replace `from blog.models.ghost_content import Post` with `from glue.services.pages import page_exists, is_page`
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Glue service for page config
|
||||
|
||||
New file: `glue/services/page_config.py`
|
||||
|
||||
```python
|
||||
async def get_page_config(session, post_id) -> PageConfig | None:
|
||||
"""Load PageConfig for a page."""
|
||||
|
||||
async def get_or_create_page_config(session, post_id) -> PageConfig:
|
||||
"""Load or create PageConfig. Emits container.child_attached if created."""
|
||||
|
||||
async def get_page_configs_by_ids(session, post_ids) -> dict[int, PageConfig]:
|
||||
"""Batch-load PageConfigs by container_id."""
|
||||
```
|
||||
|
||||
Imports `from cart.models.page_config import PageConfig` internally.
|
||||
|
||||
**Files changed:**
|
||||
- `blog/bp/post/admin/routes.py` — replace `from cart.models.page_config import PageConfig` with glue service calls
|
||||
- `blog/bp/post/services/markets.py` — replace PageConfig import
|
||||
- `blog/bp/blog/ghost_db.py` — replace PageConfig import
|
||||
- `blog/bp/blog/ghost/ghost_sync.py` — replace PageConfig import
|
||||
- `events/bp/payments/routes.py` — replace PageConfig import
|
||||
- `cart/bp/cart/services/checkout.py` — replace `from models.page_config import PageConfig` stays (same app)
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Glue service for calendars (events access from blog)
|
||||
|
||||
New file: `glue/services/calendars.py`
|
||||
|
||||
```python
|
||||
async def get_calendars_for_page(session, post_id) -> list[Calendar]:
|
||||
"""Return active calendars for a page."""
|
||||
|
||||
async def get_calendar_entries_for_posts(session, post_ids) -> dict[int, list]:
|
||||
"""Fetch confirmed CalendarEntries associated with posts (via CalendarEntryPost).
|
||||
Returns {post_id: [entry, ...]}."""
|
||||
```
|
||||
|
||||
Move and adapt from `blog/bp/post/services/entry_associations.py`:
|
||||
|
||||
```python
|
||||
async def toggle_entry_association(session, post_id, entry_id) -> tuple[bool, str | None]:
|
||||
async def get_post_entry_ids(session, post_id) -> set[int]:
|
||||
async def get_associated_entries(session, post_id, page=1, per_page=10) -> dict:
|
||||
```
|
||||
|
||||
These functions import from `events.models.calendars` internally.
|
||||
|
||||
**Files changed:**
|
||||
- `blog/bp/post/routes.py` — replace `from events.models.calendars import Calendar` + `from market.models.market_place import MarketPlace` with glue service calls
|
||||
- `blog/bp/post/admin/routes.py` — replace Calendar imports with glue service calls
|
||||
- `blog/bp/post/services/entry_associations.py` — **delete file**, moved to glue
|
||||
- `blog/bp/blog/services/posts_data.py` — replace `from events.models.calendars import CalendarEntry, CalendarEntryPost` with glue service call
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Glue service for marketplaces
|
||||
|
||||
New file: `glue/services/marketplaces.py`
|
||||
|
||||
```python
|
||||
async def get_marketplaces_for_page(session, post_id) -> list[MarketPlace]:
|
||||
"""Return active marketplaces for a page."""
|
||||
|
||||
async def create_marketplace(session, post_id, name) -> MarketPlace:
|
||||
"""Create marketplace (validates page exists via pages service)."""
|
||||
|
||||
async def soft_delete_marketplace(session, post_slug, market_slug) -> bool:
|
||||
"""Soft-delete a marketplace."""
|
||||
```
|
||||
|
||||
Move the logic from `blog/bp/post/services/markets.py` and `events/bp/markets/services/markets.py` (they're nearly identical).
|
||||
|
||||
**Files changed:**
|
||||
- `blog/bp/post/services/markets.py` — **delete file**, moved to glue
|
||||
- `blog/bp/post/admin/routes.py` — replace MarketPlace imports + service calls with glue
|
||||
- `blog/bp/post/routes.py` — replace MarketPlace import with glue service
|
||||
- `events/bp/markets/services/markets.py` — **delete file**, moved to glue
|
||||
- `events/bp/markets/routes.py` — replace MarketPlace import, use glue
|
||||
- `events/app.py` — replace MarketPlace import with glue service
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Glue service for cart items (market model access from cart)
|
||||
|
||||
New file: `glue/services/cart_items.py`
|
||||
|
||||
```python
|
||||
async def get_cart_items(session, user_id=None, session_id=None, *, page_post_id=None) -> list[CartItem]:
|
||||
"""Get cart items for identity, optionally scoped to page."""
|
||||
|
||||
async def find_or_create_cart_item(session, product_id, user_id, session_id) -> CartItem | None:
|
||||
"""Find existing or create new cart item. Returns None if product missing."""
|
||||
|
||||
async def clear_cart_for_order(session, order, *, page_post_id=None) -> None:
|
||||
"""Soft-delete cart items for order identity."""
|
||||
|
||||
async def get_calendar_cart_entries(session, user_id=None, session_id=None, *, page_post_id=None) -> list[CalendarEntry]:
|
||||
"""Get pending calendar entries for identity, optionally scoped to page."""
|
||||
```
|
||||
|
||||
Imports `CartItem`, `Product`, `MarketPlace` from market, `CalendarEntry`, `Calendar` from events internally.
|
||||
|
||||
**Files changed:**
|
||||
- `cart/bp/cart/services/get_cart.py` — replace CartItem import with glue call
|
||||
- `cart/bp/cart/services/calendar_cart.py` — replace CalendarEntry import with glue call
|
||||
- `cart/bp/cart/services/clear_cart_for_order.py` — replace CartItem/MarketPlace imports with glue call
|
||||
- `cart/bp/cart/services/checkout.py` — replace CartItem/Product/MarketPlace/CalendarEntry/Calendar imports with glue calls
|
||||
- `cart/bp/cart/api.py` — replace CartItem/MarketPlace/CalendarEntry/Calendar imports with glue calls
|
||||
- `cart/bp/cart/services/page_cart.py` — replace CartItem/MarketPlace/CalendarEntry/Calendar imports with glue calls
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Glue service for products (market access from cart orders)
|
||||
|
||||
New file: `glue/services/products.py`
|
||||
|
||||
```python
|
||||
async def get_product(session, product_id) -> Product | None:
|
||||
"""Get product by ID."""
|
||||
```
|
||||
|
||||
This is minimal — only needed by `cart/bp/order/routes.py` and `cart/bp/orders/routes.py` for search/display. However, `OrderItem.product` relationship already resolves via string forward-ref. We only need Product for the join-based search in orders listing.
|
||||
|
||||
**Files changed:**
|
||||
- `cart/bp/orders/routes.py` — replace `from market.models.market import Product` with glue import or use `OrderItem.product` relationship
|
||||
- `cart/bp/order/routes.py` — replace `from market.models.market import Product` (already uses OrderItem.product relationship for display)
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Glue service for post associations (events-side)
|
||||
|
||||
Move `events/bp/calendar_entry/services/post_associations.py` into glue:
|
||||
|
||||
New additions to `glue/services/pages.py` (or separate file `glue/services/post_associations.py`):
|
||||
|
||||
```python
|
||||
async def add_post_to_entry(session, entry_id, post_id) -> tuple[bool, str | None]:
|
||||
async def remove_post_from_entry(session, entry_id, post_id) -> tuple[bool, str | None]:
|
||||
async def get_entry_posts(session, entry_id) -> list[dict]:
|
||||
async def search_posts_for_entry(session, query, page=1, per_page=10) -> tuple[list[dict], int]:
|
||||
```
|
||||
|
||||
**Files changed:**
|
||||
- `events/bp/calendar_entry/services/post_associations.py` — **delete file**, moved to glue
|
||||
- Update any routes in events that call this service to use glue instead
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Update glue model registration
|
||||
|
||||
`glue/setup.py` needs to ensure all models from all apps are registered in SQLAlchemy's mapper when starting any app. This is because string-based relationship references (like `OrderItem.product → "Product"`) need the target model class registered.
|
||||
|
||||
```python
|
||||
def register_models():
|
||||
"""Import all model modules to register them with SQLAlchemy mapper."""
|
||||
# These are already imported by each app, but ensure completeness:
|
||||
try:
|
||||
import blog.models.ghost_content # noqa
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
import market.models.market # noqa
|
||||
import market.models.market_place # noqa
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
import cart.models.order # noqa
|
||||
import cart.models.page_config # noqa
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
import events.models.calendars # noqa
|
||||
except ImportError:
|
||||
pass
|
||||
```
|
||||
|
||||
Each app's `app.py` calls `register_models()` at startup. The try/except guards handle Docker where only one app's code is present — but since all apps share `glue/` and the DB, all model files need to be importable.
|
||||
|
||||
**Note:** In Docker, each container only has its own app + shared + glue. For glue services that import from other apps' models, those models must be available. This means either:
|
||||
- (a) Include all model files in each container (symlinks or copies), or
|
||||
- (b) Have glue services that import other apps' models use try/except at import time
|
||||
|
||||
Since all apps already share one DB and all model files are available in development, option (a) is cleaner for production. Alternatively, the current Docker setup could be extended to include cross-app model files in each image.
|
||||
|
||||
---
|
||||
|
||||
## Step 9: Update existing glue services
|
||||
|
||||
**`glue/services/cart_adoption.py`** — already imports from market and events (correct — this is glue's job). No change needed.
|
||||
|
||||
**`glue/services/order_lifecycle.py`** — already imports from events. No change needed.
|
||||
|
||||
---
|
||||
|
||||
## Step 10: Clean up dead imports and update app.py files
|
||||
|
||||
After all glue services are wired:
|
||||
|
||||
- `cart/app.py` — remove `from blog.models.ghost_content import Post`, use `from glue.services.pages import get_page_by_slug`
|
||||
- `market/app.py` — remove `from blog.models.ghost_content import Post`, use `from glue.services.pages import get_page_by_slug`
|
||||
- `events/app.py` — remove `from blog.models.ghost_content import Post` and `from market.models.market_place import MarketPlace`
|
||||
- Remove any now-empty cross-app model directories if they exist
|
||||
|
||||
---
|
||||
|
||||
## Files Summary
|
||||
|
||||
| Repo | File | Change |
|
||||
|------|------|--------|
|
||||
| **glue** | `services/pages.py` | **NEW** — Post access (slug, id, exists, search) |
|
||||
| **glue** | `services/page_config.py` | **NEW** — PageConfig CRUD |
|
||||
| **glue** | `services/calendars.py` | **NEW** — Calendar queries + entry associations (from blog) |
|
||||
| **glue** | `services/marketplaces.py` | **NEW** — MarketPlace CRUD (from blog+events) |
|
||||
| **glue** | `services/cart_items.py` | **NEW** — CartItem/CalendarEntry queries for cart |
|
||||
| **glue** | `services/products.py` | **NEW** — Product access for cart orders |
|
||||
| **glue** | `services/post_associations.py` | **NEW** — Post-CalendarEntry associations (from events) |
|
||||
| **glue** | `setup.py` | Add `register_models()` |
|
||||
| **cart** | `app.py` | Replace Post import with glue |
|
||||
| **cart** | `bp/cart/api.py` | Replace all 4 cross-app imports with glue |
|
||||
| **cart** | `bp/cart/services/checkout.py` | Replace cross-app imports with glue |
|
||||
| **cart** | `bp/cart/services/page_cart.py` | Replace all cross-app imports with glue |
|
||||
| **cart** | `bp/cart/services/get_cart.py` | Replace CartItem import with glue |
|
||||
| **cart** | `bp/cart/services/calendar_cart.py` | Replace CalendarEntry import with glue |
|
||||
| **cart** | `bp/cart/services/clear_cart_for_order.py` | Replace CartItem/MarketPlace with glue |
|
||||
| **cart** | `bp/orders/routes.py` | Replace Product import with glue |
|
||||
| **cart** | `bp/order/routes.py` | Replace Product import with glue |
|
||||
| **blog** | `bp/post/admin/routes.py` | Replace PageConfig/Calendar/MarketPlace with glue |
|
||||
| **blog** | `bp/post/routes.py` | Replace Calendar/MarketPlace with glue |
|
||||
| **blog** | `bp/post/services/entry_associations.py` | **DELETE** — moved to `glue/services/calendars.py` |
|
||||
| **blog** | `bp/post/services/markets.py` | **DELETE** — moved to `glue/services/marketplaces.py` |
|
||||
| **blog** | `bp/blog/ghost_db.py` | Replace PageConfig import with glue |
|
||||
| **blog** | `bp/blog/ghost/ghost_sync.py` | Replace PageConfig import with glue |
|
||||
| **blog** | `bp/blog/services/posts_data.py` | Replace CalendarEntry/CalendarEntryPost with glue |
|
||||
| **events** | `app.py` | Replace Post + MarketPlace imports with glue |
|
||||
| **events** | `bp/markets/services/markets.py` | **DELETE** — moved to `glue/services/marketplaces.py` |
|
||||
| **events** | `bp/markets/routes.py` | Replace MarketPlace import, use glue |
|
||||
| **events** | `bp/calendars/services/calendars.py` | Replace Post import with glue |
|
||||
| **events** | `bp/calendar_entry/services/post_associations.py` | **DELETE** — moved to `glue/services/post_associations.py` |
|
||||
| **events** | `bp/payments/routes.py` | Replace PageConfig import with glue |
|
||||
| **market** | `app.py` | Replace Post import with glue |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Step 1** (pages.py) — unlocks Steps 2-4 which depend on page validation
|
||||
2. **Step 2** (page_config.py) — independent after Step 1
|
||||
3. **Steps 3-4** (calendars.py, marketplaces.py) — can be done in parallel, both use pages.py
|
||||
4. **Step 5** (cart_items.py) — depends on steps 1, 3 for calendar queries
|
||||
5. **Step 6** (products.py) — independent
|
||||
6. **Step 7** (post_associations.py) — independent, uses pages.py
|
||||
7. **Steps 8-10** (registration, cleanup) — after all services exist
|
||||
|
||||
---
|
||||
|
||||
## What's NOT changing
|
||||
|
||||
- **CartItem stays in `market/models/market.py`** — moving it creates equal or worse coupling
|
||||
- **OrderItem stays in `cart/models/order.py`** with `product_id` FK — pragmatic exception
|
||||
- **OrderItem.product_id FK** — kept, denormalized `product_title` makes it non-critical
|
||||
- **CartItem.product_id FK** — kept, same DB
|
||||
- **CartItem.market_place_id FK** — kept, same DB
|
||||
- **CartItem.user_id FK** — kept, shared model
|
||||
- **Internal HTTP APIs** (cart/summary, coop/*, events/*) — not changing
|
||||
- **`shared/` models** (User, MagicLink, etc.) — shared across all apps by design
|
||||
|
||||
---
|
||||
|
||||
## Docker Consideration
|
||||
|
||||
For glue services to work in Docker (single app per container), model files from other apps must be importable. Options:
|
||||
1. **Copy model files** into each Docker image during build (just the `models/` dirs)
|
||||
2. **Use try/except** in glue services at import time (degrade gracefully)
|
||||
3. **Mount shared volume** with all model files
|
||||
|
||||
Recommend option 2 for now — glue services that can't import a model simply raise ImportError at call time, which only happens if the service is called from the wrong app (shouldn't happen in practice).
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
1. `grep -r "from blog\.models" cart/ market/ events/ glue/` — should return zero results (only in blog/ itself)
|
||||
2. `grep -r "from market\.models" blog/ cart/ events/` — should return zero results (only in market/ and glue/)
|
||||
3. `grep -r "from cart\.models" blog/ market/ events/` — should return zero results (only in cart/ and glue/)
|
||||
4. `grep -r "from events\.models" blog/ cart/ market/` — should return zero results (only in events/ and glue/)
|
||||
5. All 4 apps start without import errors
|
||||
6. Checkout flow works end-to-end
|
||||
7. Blog admin: can toggle features, create/delete markets, manage calendar entries
|
||||
8. Events admin: can create calendars, manage markets, configure payments
|
||||
9. Market app: markets listing page loads correctly
|
||||
149
.claude/plans/rippling-tumbling-cocke.md
Normal file
149
.claude/plans/rippling-tumbling-cocke.md
Normal file
@@ -0,0 +1,149 @@
|
||||
# Ticket UX Improvements: +/- Buttons, Sold Count, Cart Grouping
|
||||
|
||||
## Context
|
||||
The entry page currently uses a numeric input + "Buy Tickets" button, which replaces itself with a confirmation after purchase. The cart lists each ticket individually. The user wants the ticket UX to match the product pattern: +/- buttons, "in basket" count, tickets grouped by event on cart.
|
||||
|
||||
## Requirements
|
||||
1. **Entry page**: Show tickets sold count + current user's "in basket" count
|
||||
2. **Entry page**: Replace qty input with "Add to basket" / +/- buttons (product pattern)
|
||||
3. **Entry page**: Keep form active after adding (don't replace with confirmation)
|
||||
4. **Cart page**: Group tickets by event (entry_id + ticket_type), show quantity with +/- buttons
|
||||
|
||||
---
|
||||
|
||||
## 1. Add `ticket_type_id` to TicketDTO
|
||||
|
||||
**File**: `shared/contracts/dtos.py`
|
||||
- Add `ticket_type_id: int | None = None` field to `TicketDTO`
|
||||
|
||||
**File**: `shared/services/calendar_impl.py`
|
||||
- In `_ticket_to_dto()`, populate `ticket_type_id=ticket.ticket_type_id`
|
||||
|
||||
**Sync**: Copy to all 4 app submodule copies.
|
||||
|
||||
## 2. New ticket service functions
|
||||
|
||||
**File**: `events/bp/tickets/services/tickets.py`
|
||||
- Add `get_user_reserved_count(session, entry_id, user_id, session_id, ticket_type_id=None) -> int`
|
||||
- Counts reserved tickets for this user+entry+type
|
||||
- Add `get_sold_ticket_count(session, entry_id) -> int`
|
||||
- Counts all non-cancelled tickets for this entry
|
||||
- Add `cancel_latest_reserved_ticket(session, entry_id, user_id, session_id, ticket_type_id=None) -> bool`
|
||||
- Finds the most recently created reserved ticket for this user+entry+type, sets state='cancelled'. Returns True if one was cancelled.
|
||||
|
||||
## 3. Add `adjust_quantity` route to events tickets blueprint
|
||||
|
||||
**File**: `events/bp/tickets/routes.py`
|
||||
- New route: `POST /tickets/adjust/`
|
||||
- Form fields: `entry_id`, `ticket_type_id` (optional), `count` (target quantity)
|
||||
- Logic:
|
||||
- Get current user reserved count for this entry/type
|
||||
- If count > current: create `(count - current)` tickets via `create_ticket()`
|
||||
- If count < current: cancel `(current - count)` tickets via `cancel_latest_reserved_ticket()` in a loop
|
||||
- If count == 0: cancel all
|
||||
- Check availability before adding (like existing `buy_tickets`)
|
||||
- Response: re-render `_buy_form.html` (HTMX swap replaces form, keeps it active)
|
||||
- Include OOB cart-mini update: `{{ mini(oob='true') }}`
|
||||
|
||||
## 4. Inject ticket counts into entry page context
|
||||
|
||||
**File**: `events/bp/calendar_entry/routes.py` — `inject_root` context processor
|
||||
- Add `ticket_sold_count`: total non-cancelled tickets for entry (via `get_sold_ticket_count`)
|
||||
- Add `user_ticket_count`: current user's reserved count (via `get_user_reserved_count`)
|
||||
- For multi-type entries, add `user_ticket_counts_by_type`: dict mapping ticket_type_id → count
|
||||
|
||||
## 5. Rewrite entry page buy form
|
||||
|
||||
**File**: `events/templates/_types/tickets/_buy_form.html`
|
||||
- Show "X sold" (from `ticket_sold_count`) alongside "X remaining"
|
||||
- Show "X in basket" for current user
|
||||
|
||||
**For single-price entries (no ticket types)**:
|
||||
- If `user_ticket_count == 0`: show "Add to basket" button (posts to `/tickets/adjust/` with count=1)
|
||||
- If `user_ticket_count > 0`: show `[-]` [count badge] `[+]` buttons
|
||||
- Minus: posts count=user_ticket_count-1
|
||||
- Plus: posts count=user_ticket_count+1
|
||||
- All forms: `hx-post`, `hx-target="#ticket-buy-{{ entry.id }}"`, `hx-swap="outerHTML"`
|
||||
|
||||
**For multi-type entries**:
|
||||
- Same pattern per ticket type row, using `user_ticket_counts_by_type[tt.id]`
|
||||
|
||||
Style: match product pattern exactly — emerald circular buttons, w-8 h-8, cart icon with badge.
|
||||
|
||||
## 6. Add ticket quantity route to cart app
|
||||
|
||||
**File**: `cart/bp/cart/global_routes.py`
|
||||
- New route: `POST /cart/ticket-quantity/`
|
||||
- Form fields: `entry_id`, `ticket_type_id` (optional), `count` (target quantity)
|
||||
- Logic: call into CalendarService or directly use ticket functions
|
||||
- Since cart app uses service contracts, add `adjust_ticket_quantity` to CalendarService protocol
|
||||
|
||||
**File**: `shared/contracts/protocols.py` — CalendarService
|
||||
- Add: `adjust_ticket_quantity(session, entry_id, count, *, user_id, session_id, ticket_type_id=None) -> int`
|
||||
|
||||
**File**: `shared/services/calendar_impl.py`
|
||||
- Implement `adjust_ticket_quantity`:
|
||||
- Same logic as events adjust route (create/cancel to match target count)
|
||||
- Return new count
|
||||
|
||||
**File**: `shared/services/stubs.py`
|
||||
- Add stub: returns 0
|
||||
|
||||
Response: `HX-Refresh: true` (same as product quantity route).
|
||||
|
||||
## 7. Cart page: group tickets by event with +/- buttons
|
||||
|
||||
**File**: `cart/templates/_types/cart/_cart.html` — ticket section (lines 63-95)
|
||||
- Replace individual ticket list with grouped display
|
||||
- Group `ticket_cart_entries` by `(entry_id, ticket_type_id)`:
|
||||
- Use Jinja `groupby` on `entry_id` first, then sub-group by `ticket_type_name`
|
||||
- Or pre-group in the route handler and pass as a dict
|
||||
|
||||
**Approach**: Pre-group in the route handler for cleaner templates.
|
||||
|
||||
**File**: `cart/bp/cart/page_routes.py` — `page_view`
|
||||
- After getting `page_tickets`, group them into a list of dicts:
|
||||
```
|
||||
[{"entry_name": ..., "entry_id": ..., "ticket_type_name": ..., "ticket_type_id": ...,
|
||||
"entry_start_at": ..., "entry_end_at": ..., "price": ..., "quantity": N}]
|
||||
```
|
||||
- Pass as `ticket_groups` to template
|
||||
|
||||
**File**: `cart/bp/cart/global_routes.py` — overview/checkout routes
|
||||
- Same grouping for global cart view if tickets appear there
|
||||
|
||||
**Cart ticket group template**: Each group shows:
|
||||
- Event name + ticket type (if any)
|
||||
- Date/time
|
||||
- Price per ticket
|
||||
- `-` [qty] `+` buttons (posting to `/cart/ticket-quantity/`)
|
||||
- Line total (price × qty)
|
||||
|
||||
Match product `cart_item` macro style (article card with quantity controls).
|
||||
|
||||
## 8. Cart summary update
|
||||
|
||||
**File**: `cart/templates/_types/cart/_cart.html` — `summary` macro
|
||||
- Update Items count: include ticket quantities in total (currently just product quantities)
|
||||
|
||||
## Files to modify (summary)
|
||||
- `shared/contracts/dtos.py` — add ticket_type_id to TicketDTO
|
||||
- `shared/contracts/protocols.py` — add adjust_ticket_quantity to CalendarService
|
||||
- `shared/services/calendar_impl.py` — implement adjust_ticket_quantity, update _ticket_to_dto
|
||||
- `shared/services/stubs.py` — add stub
|
||||
- `events/bp/tickets/services/tickets.py` — add count/cancel functions
|
||||
- `events/bp/tickets/routes.py` — add adjust route
|
||||
- `events/bp/calendar_entry/routes.py` — inject sold/user counts
|
||||
- `events/templates/_types/tickets/_buy_form.html` — rewrite with +/- pattern
|
||||
- `cart/bp/cart/global_routes.py` — add ticket-quantity route
|
||||
- `cart/bp/cart/page_routes.py` — group tickets
|
||||
- `cart/templates/_types/cart/_cart.html` — grouped ticket display with +/-
|
||||
- All 4 app `shared/` submodule copies synced
|
||||
|
||||
## Verification
|
||||
1. Visit entry page → see "X sold", "X in basket", "Add to basket" button
|
||||
2. Click "Add to basket" → form stays, shows `-` [1] `+`, basket count shows "1 in basket"
|
||||
3. Click `+` → count increases, sold count increases
|
||||
4. Click `-` → count decreases, ticket cancelled
|
||||
5. Visit cart page → tickets grouped by event, +/- buttons work
|
||||
6. Checkout flow still works (existing tests)
|
||||
171
.claude/plans/unified-inventing-kay.md
Normal file
171
.claude/plans/unified-inventing-kay.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# Social Network Sharing Integration
|
||||
|
||||
## Context
|
||||
|
||||
Rose Ash already has ActivityPub for federated social sharing. This plan adds OAuth-based sharing to mainstream social networks — Facebook, Instagram, Threads, Twitter/X, LinkedIn, and Mastodon. Users connect their social accounts via the account dashboard, then manually share content (blog posts, events, products) via a share button on content pages.
|
||||
|
||||
All social logic lives in the **account** microservice. Content apps get a share button that opens the account share page.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Data Model + Encryption
|
||||
|
||||
### 1a. `shared/models/social_connection.py` (NEW)
|
||||
- SQLAlchemy 2.0 model following `oauth_grant.py` pattern
|
||||
- Table `social_connections` in db_account
|
||||
- Columns: `id`, `user_id` (FK to users.id with CASCADE), `platform` (facebook/instagram/threads/twitter/linkedin/mastodon), `platform_user_id`, `platform_username`, `display_name`, `access_token_enc`, `refresh_token_enc`, `token_expires_at`, `scopes`, `extra_data` (JSONB — mastodon instance URL, facebook page ID, etc.), `created_at`, `updated_at`, `revoked_at`
|
||||
- Indexes: `(user_id, platform)`, unique `(platform, platform_user_id)`
|
||||
|
||||
### 1b. `shared/models/__init__.py` (MODIFY)
|
||||
- Add `from .social_connection import SocialConnection`
|
||||
|
||||
### 1c. `shared/infrastructure/social_crypto.py` (NEW)
|
||||
- Fernet encrypt/decrypt using `SOCIAL_ENCRYPTION_KEY` env var
|
||||
- `encrypt_token(plaintext) -> str`, `decrypt_token(ciphertext) -> str`
|
||||
|
||||
### 1d. Alembic migration (NEW)
|
||||
- Creates `social_connections` table
|
||||
|
||||
### 1e. `docker-compose.yml` (MODIFY)
|
||||
- Add to `x-app-env`: `SOCIAL_ENCRYPTION_KEY`, plus per-platform credentials (`SOCIAL_FACEBOOK_APP_ID`, `SOCIAL_FACEBOOK_APP_SECRET`, `SOCIAL_TWITTER_CLIENT_ID`, `SOCIAL_TWITTER_CLIENT_SECRET`, `SOCIAL_LINKEDIN_CLIENT_ID`, `SOCIAL_LINKEDIN_CLIENT_SECRET`)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Platform OAuth Clients
|
||||
|
||||
All in `account/services/social_platforms/`:
|
||||
|
||||
### 2a. `base.py` (NEW)
|
||||
- `OAuthResult` dataclass (platform_user_id, tokens, expiry, extra_data)
|
||||
- `ShareResult` dataclass (success, platform_post_id, platform_post_url, error)
|
||||
- `SocialPlatform` abstract base class: `get_authorize_url()`, `exchange_code()`, `refresh_access_token()`, `share_link()`, `verify_token()`
|
||||
|
||||
### 2b. `meta.py` (NEW) — Facebook + Instagram + Threads
|
||||
- **Facebook**: OAuth2 via Graph API, `pages_manage_posts` scope, exchange user token → long-lived → page token, post via `/{page_id}/feed`
|
||||
- **Instagram**: Same Meta OAuth, `instagram_basic` + `instagram_content_publish` scopes, business/creator accounts only, container → publish workflow
|
||||
- **Threads**: Separate OAuth at threads.net, `threads_basic` + `threads_content_publish` scopes, container → publish
|
||||
|
||||
### 2c. `twitter.py` (NEW) — Twitter/X
|
||||
- OAuth 2.0 with PKCE, `tweet.write` + `offline.access` scopes
|
||||
- Post via `POST https://api.twitter.com/2/tweets`
|
||||
|
||||
### 2d. `linkedin.py` (NEW) — LinkedIn
|
||||
- OAuth 2.0, `w_member_social` + `openid` scopes
|
||||
- Post via LinkedIn Posts API
|
||||
|
||||
### 2e. `mastodon.py` (NEW) — Mastodon
|
||||
- Dynamic app registration per instance (`POST /api/v1/apps`)
|
||||
- OAuth 2.0, `write:statuses` scope
|
||||
- Post via `POST /api/v1/statuses`
|
||||
- Instance URL stored in `extra_data["instance_url"]`
|
||||
|
||||
### 2f. `__init__.py` (NEW) — Platform registry
|
||||
- `PLATFORMS` dict, lazy-initialized from env vars
|
||||
- Mastodon always available (no pre-configured credentials)
|
||||
- `get_platform(name)`, `available_platforms()`
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Account Social Blueprint
|
||||
|
||||
### 3a. `account/bp/social/__init__.py` (NEW)
|
||||
### 3b. `account/bp/social/routes.py` (NEW)
|
||||
|
||||
Routes (all require login):
|
||||
- `GET /social/` — list connected accounts + available platforms
|
||||
- `GET /social/connect/<platform>/` — start OAuth redirect (Mastodon: accept instance URL param)
|
||||
- `GET /social/callback/<platform>/` — OAuth callback, exchange code, encrypt & store tokens
|
||||
- `POST /social/disconnect/<int:id>/` — soft-delete (set revoked_at)
|
||||
- `GET /social/share/` — share page (params: url, title, description, image)
|
||||
- `POST /social/share/` — execute share to selected accounts, return results
|
||||
|
||||
OAuth state stored in session (nonce + platform + redirect params).
|
||||
|
||||
### 3c. `account/bp/__init__.py` (MODIFY)
|
||||
- Add `from .social.routes import register as register_social_bp`
|
||||
|
||||
### 3d. `account/app.py` (MODIFY)
|
||||
- Register social blueprint **before** account blueprint (account has catch-all `/<slug>/`)
|
||||
```python
|
||||
app.register_blueprint(register_auth_bp())
|
||||
app.register_blueprint(register_social_bp()) # <-- NEW, before account
|
||||
app.register_blueprint(register_account_bp())
|
||||
app.register_blueprint(register_fragments())
|
||||
```
|
||||
|
||||
### 3e. `account/templates/_types/auth/_nav.html` (MODIFY)
|
||||
- Add "social" link between newsletters and `account_nav_html`
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Templates
|
||||
|
||||
### 4a. `account/templates/_types/auth/_social_panel.html` (NEW)
|
||||
- Platform cards with icons (Font Awesome: `fa-facebook`, `fa-instagram`, `fa-threads`, `fa-x-twitter`, `fa-linkedin`, `fa-mastodon`)
|
||||
- Connected accounts per platform: display name, username, disconnect button
|
||||
- "Connect" button per platform
|
||||
- Mastodon: instance URL input before connecting
|
||||
|
||||
### 4b. `account/templates/_types/auth/_share_panel.html` (NEW)
|
||||
- Content preview card (title, image, URL)
|
||||
- Connected accounts as checkboxes grouped by platform
|
||||
- Optional message textarea
|
||||
- Share button → HTMX POST to `/social/share/`
|
||||
|
||||
### 4c. `account/templates/_types/auth/_share_result.html` (NEW)
|
||||
- Per-platform success/failure with links to created posts
|
||||
|
||||
### 4d. `account/templates/_types/auth/_mastodon_connect.html` (NEW)
|
||||
- Instance URL input form
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Share Button in Content Apps
|
||||
|
||||
### 5a. `account/bp/fragments/routes.py` (MODIFY)
|
||||
- Add `share-button` handler: accepts url, title, description, image params
|
||||
- Returns a share icon/link pointing to `account.rose-ash.com/social/share/?url=...&title=...`
|
||||
|
||||
### 5b. `account/templates/fragments/share_button.html` (NEW)
|
||||
- Small button: `<a href="..." target="_blank"><i class="fa-solid fa-share-nodes"></i> Share</a>`
|
||||
|
||||
### 5c. Content app integration
|
||||
- Blog post detail: fetch `share-button` fragment from account, render in post template
|
||||
- Events detail: same pattern
|
||||
- Market product detail: same pattern
|
||||
- Each passes its own public URL, title, description, image to the fragment
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Token Refresh + Share History
|
||||
|
||||
### 6a. Token refresh in share flow
|
||||
- Before posting, check `token_expires_at`; if expired, call `refresh_access_token()`
|
||||
- Update encrypted tokens in DB
|
||||
- If refresh fails, mark connection with error and prompt reconnect
|
||||
|
||||
### 6b. `shared/models/social_share.py` (NEW, optional)
|
||||
- Table `social_shares`: connection_id, shared_url, shared_title, platform_post_id, platform_post_url, status, error_message, created_at
|
||||
- Prevents duplicate shares, enables "shared" indicator on content pages
|
||||
|
||||
---
|
||||
|
||||
## Key Patterns to Follow
|
||||
|
||||
| Pattern | Reference File |
|
||||
|---------|---------------|
|
||||
| ORM model (mapped_column, FK, indexes) | `shared/models/oauth_grant.py` |
|
||||
| Blueprint registration + OOB template | `account/bp/account/routes.py` |
|
||||
| Fragment handler dict | `account/bp/fragments/routes.py` |
|
||||
| Account nav link | `account/templates/_types/auth/_nav.html` |
|
||||
| httpx async client | `shared/infrastructure/actions.py` |
|
||||
|
||||
## Verification
|
||||
|
||||
1. Generate `SOCIAL_ENCRYPTION_KEY`, add to `.env`
|
||||
2. Run Alembic migration
|
||||
3. Start account app, navigate to `/social/`
|
||||
4. Connect a test Mastodon account (easiest — no app review needed)
|
||||
5. Navigate to a blog post, click Share, select Mastodon account, verify post appears
|
||||
6. Disconnect account, verify soft-delete
|
||||
7. Test token refresh by connecting Facebook with short-lived token
|
||||
@@ -49,6 +49,8 @@ def register(url_prefix, title):
|
||||
async def init():
|
||||
from .ghost.ghost_sync import sync_all_content_from_ghost
|
||||
from sqlalchemy import text
|
||||
import logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Advisory lock prevents multiple Hypercorn workers from
|
||||
# running the sync concurrently (which causes PK conflicts).
|
||||
@@ -60,9 +62,18 @@ def register(url_prefix, title):
|
||||
try:
|
||||
await sync_all_content_from_ghost(s)
|
||||
await s.commit()
|
||||
except Exception:
|
||||
logger.exception("Ghost sync failed — will retry on next deploy")
|
||||
try:
|
||||
await s.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
try:
|
||||
await s.execute(text("SELECT pg_advisory_unlock(900001)"))
|
||||
await s.commit()
|
||||
except Exception:
|
||||
pass # lock auto-releases when session closes
|
||||
|
||||
@blogs_bp.before_request
|
||||
def route():
|
||||
|
||||
418
docs/decoupling-plan.md
Normal file
418
docs/decoupling-plan.md
Normal file
@@ -0,0 +1,418 @@
|
||||
# Rose Ash Decoupling Plan
|
||||
|
||||
## Context
|
||||
|
||||
The four Rose Ash apps (blog, market, cart, events) are tightly coupled through:
|
||||
- A shared model layer (`blog/shared_lib/models/`) containing ALL models for ALL apps
|
||||
- Cross-app foreign keys (calendars→posts, cart_items→market_places, calendar_entries→orders, etc.)
|
||||
- `Post` as the universal parent — calendars, markets, page_configs all hang off `post_id`
|
||||
- Internal HTTP calls for menu items, cart summaries, and login adoption
|
||||
|
||||
This makes it impossible to attach services to anything other than a Post, and means apps can't have independent databases. The goal is to decouple so apps are independently deployable, new services can be added easily, and the composition of "what's attached to what" is defined in a separate glue layer.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Extract `shared_lib` out of `blog/`
|
||||
|
||||
**What:** Move shared infrastructure into a top-level `shared/` package. Split models by ownership.
|
||||
|
||||
### New structure
|
||||
|
||||
```
|
||||
/root/rose-ash/
|
||||
shared/ # Extracted from blog/shared_lib/
|
||||
db/base.py, session.py # Unchanged
|
||||
models/ # ONLY shared models:
|
||||
user.py # User (used by all apps)
|
||||
kv.py # KV (settings)
|
||||
magic_link.py # MagicLink (auth)
|
||||
ghost_membership_entities.py # Ghost labels/newsletters/tiers/subscriptions
|
||||
menu_item.py # MenuItem (temporary, moves to glue in Phase 4)
|
||||
infrastructure/ # Renamed from shared/
|
||||
factory.py # create_base_app()
|
||||
internal_api.py # HTTP client for inter-app calls
|
||||
context.py # base_context()
|
||||
user_loader.py, jinja_setup.py, cart_identity.py, cart_loader.py, urls.py, http_utils.py
|
||||
browser/ # Renamed from suma_browser/
|
||||
(middleware, templates, csrf, errors, filters, redis, payments, authz)
|
||||
config.py, config/
|
||||
alembic/, static/, editor/
|
||||
|
||||
blog/models/ # Blog-owned models
|
||||
ghost_content.py # Post, Author, Tag, PostAuthor, PostTag, PostLike
|
||||
snippet.py # Snippet
|
||||
tag_group.py # TagGroup, TagGroupTag
|
||||
|
||||
market/models/ # Market-owned models
|
||||
market.py # Product, CartItem, NavTop, NavSub, Listing, etc.
|
||||
market_place.py # MarketPlace
|
||||
|
||||
cart/models/ # Cart-owned models
|
||||
order.py # Order, OrderItem
|
||||
page_config.py # PageConfig
|
||||
|
||||
events/models/ # Events-owned models
|
||||
calendars.py # Calendar, CalendarEntry, CalendarSlot, Ticket, TicketType
|
||||
```
|
||||
|
||||
### Key changes
|
||||
- Update `path_setup.py` in each app to add project root to `sys.path`
|
||||
- Update all `from models import X` → `from blog.models import X` / `from shared.models import X` etc.
|
||||
- Update `from db.base import Base` → `from shared.db.base import Base` in every model file
|
||||
- Update `from shared.factory import` → `from shared.infrastructure.factory import` in each `app.py`
|
||||
- Alembic `env.py` imports from all locations so `Base.metadata` sees every table
|
||||
- Add a transitional compat layer in old location that re-exports everything (remove later)
|
||||
|
||||
### Critical files to modify
|
||||
- `blog/app.py` (line 9: `from shared.factory`), `market/app.py`, `cart/app.py`, `events/app.py`
|
||||
- `blog/shared_lib/shared/factory.py` → `shared/infrastructure/factory.py`
|
||||
- Every model file (Base import)
|
||||
- `blog/shared_lib/alembic/env.py` → `shared/alembic/env.py`
|
||||
- Each app's `path_setup.py`
|
||||
|
||||
### Verify
|
||||
- All four apps start without import errors
|
||||
- `alembic check` produces no diff (schema unchanged)
|
||||
- All routes return correct responses
|
||||
- Internal API calls between apps still work
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Event Infrastructure + Logging
|
||||
|
||||
**What:** Add the durable event system (transactional outbox) and shared structured logging.
|
||||
|
||||
### 2a. DomainEvent model (the outbox)
|
||||
|
||||
New file: `shared/models/domain_event.py`
|
||||
|
||||
```python
|
||||
class DomainEvent(Base):
|
||||
__tablename__ = "domain_events"
|
||||
|
||||
id = Integer, primary_key
|
||||
event_type = String(128), indexed # "calendar.created", "order.completed"
|
||||
aggregate_type = String(64) # "calendar", "order"
|
||||
aggregate_id = Integer # ID of the thing that changed
|
||||
payload = JSONB # Event-specific data
|
||||
state = String(20), default "pending" # pending → processing → completed | failed
|
||||
attempts = Integer, default 0
|
||||
max_attempts = Integer, default 5
|
||||
last_error = Text, nullable
|
||||
created_at = DateTime, server_default now()
|
||||
processed_at = DateTime, nullable
|
||||
```
|
||||
|
||||
The critical property: `emit_event()` writes to this table **in the same DB transaction** as the domain change. If the app crashes after commit, the event is already persisted. If it crashes before commit, neither the domain change nor the event exists. This is atomic.
|
||||
|
||||
### 2b. Event bus
|
||||
|
||||
New directory: `shared/events/`
|
||||
|
||||
```
|
||||
shared/events/
|
||||
__init__.py # exports emit_event, register_handler, EventProcessor
|
||||
bus.py # emit_event(session, event_type, aggregate_type, aggregate_id, payload)
|
||||
# register_handler(event_type, async_handler_fn)
|
||||
processor.py # EventProcessor: polls domain_events table, dispatches to handlers
|
||||
```
|
||||
|
||||
**`emit_event(session, ...)`** — called within service functions, writes to outbox in current transaction
|
||||
**`register_handler(event_type, fn)`** — called at app startup (by glue layer) to register handlers
|
||||
**`EventProcessor`** — background polling loop:
|
||||
1. `SELECT ... FROM domain_events WHERE state='pending' FOR UPDATE SKIP LOCKED`
|
||||
2. Run all registered handlers for that event_type
|
||||
3. Mark completed or retry on failure
|
||||
4. Runs as an `asyncio.create_task` within each app process (started in `factory.py`)
|
||||
|
||||
### 2c. Structured logging
|
||||
|
||||
New directory: `shared/logging/`
|
||||
|
||||
```
|
||||
shared/logging/
|
||||
__init__.py
|
||||
setup.py # configure_logging(app_name), get_logger(name)
|
||||
```
|
||||
|
||||
- JSON-structured output to stdout (timestamp, level, app, message, plus optional fields: event_type, user_id, request_id, duration_ms)
|
||||
- `configure_logging(app_name)` called in `create_base_app()`
|
||||
- All apps get consistent log format; in production these go to a log aggregator
|
||||
|
||||
### 2d. Integration
|
||||
|
||||
Update `shared/infrastructure/factory.py`:
|
||||
- Call `configure_logging(name)` at app creation
|
||||
- Start `EventProcessor` as background task in `@app.before_serving`
|
||||
- Stop it in `@app.after_serving`
|
||||
|
||||
### Verify
|
||||
- `domain_events` table exists after migration
|
||||
- Call `emit_event()` in a test, verify row appears in table
|
||||
- `EventProcessor` picks up pending events and marks them completed
|
||||
- JSON logs appear on stdout with correct structure
|
||||
- No behavioral changes — this is purely additive infrastructure
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Generic Container Concept
|
||||
|
||||
**What:** Replace cross-app `post_id` FKs with `container_type` + `container_id` soft references.
|
||||
|
||||
### Models to change
|
||||
|
||||
**Calendar** (`events/models/calendars.py`):
|
||||
```python
|
||||
# REMOVE: post_id = Column(Integer, ForeignKey("posts.id"), ...)
|
||||
# REMOVE: post = relationship("Post", ...)
|
||||
# ADD:
|
||||
container_type = Column(String(32), nullable=False) # "page", "market", etc.
|
||||
container_id = Column(Integer, nullable=False)
|
||||
```
|
||||
|
||||
**MarketPlace** (`market/models/market_place.py`):
|
||||
```python
|
||||
# Same pattern: remove post_id FK, add container_type + container_id
|
||||
```
|
||||
|
||||
**PageConfig** (`cart/models/page_config.py`):
|
||||
```python
|
||||
# Same pattern
|
||||
```
|
||||
|
||||
**CalendarEntryPost** → rename to **CalendarEntryContent**:
|
||||
```python
|
||||
# REMOVE: post_id FK
|
||||
# ADD: content_type + content_id (generic reference)
|
||||
```
|
||||
|
||||
### From Post model (`blog/models/ghost_content.py`), remove:
|
||||
- `calendars` relationship
|
||||
- `markets` relationship
|
||||
- `page_config` relationship
|
||||
- `calendar_entries` relationship (via CalendarEntryPost)
|
||||
- `menu_items` relationship (moves to glue in Phase 4)
|
||||
|
||||
### Helper in `shared/containers.py`:
|
||||
```python
|
||||
class ContainerType:
|
||||
PAGE = "page"
|
||||
# Future: MARKET = "market", GROUP = "group", etc.
|
||||
|
||||
def container_filter(model, container_type, container_id):
|
||||
"""Return SQLAlchemy filter clauses."""
|
||||
return [model.container_type == container_type, model.container_id == container_id]
|
||||
```
|
||||
|
||||
### Three-step migration (non-breaking)
|
||||
1. **Add columns** (nullable) — keeps old post_id FK intact
|
||||
2. **Backfill** — `UPDATE calendars SET container_type='page', container_id=post_id`; make NOT NULL
|
||||
3. **Drop old FK** — remove post_id column and FK constraint
|
||||
|
||||
### Update all queries
|
||||
Key files that reference `Calendar.post_id`, `MarketPlace.post_id`, `PageConfig.post_id`:
|
||||
- `events/app.py` (~line 108)
|
||||
- `market/app.py` (~line 119)
|
||||
- `cart/app.py` (~line 131)
|
||||
- `cart/bp/cart/services/checkout.py` (lines 77-85, 160-163) — `resolve_page_config()` and `create_order_from_cart()`
|
||||
- `cart/bp/cart/services/page_cart.py`
|
||||
- `cart/bp/cart/api.py`
|
||||
|
||||
All change from `X.post_id == post.id` to `X.container_type == "page", X.container_id == post.id`.
|
||||
|
||||
### Verify
|
||||
- Creating a calendar/market/page_config uses container_type + container_id
|
||||
- Cart checkout still resolves correct page config via container references
|
||||
- No cross-app FKs remain for these three models
|
||||
- Alembic migration is clean
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Glue Layer
|
||||
|
||||
**What:** New top-level `glue/` package that owns container relationships, navigation, and event handlers.
|
||||
|
||||
### Structure
|
||||
|
||||
```
|
||||
/root/rose-ash/glue/
|
||||
__init__.py
|
||||
models/
|
||||
container_relation.py # Parent-child container relationships
|
||||
menu_node.py # Navigation tree (replaces MenuItem)
|
||||
services/
|
||||
navigation.py # Build menu from relationship tree
|
||||
relationships.py # attach_child(), get_children(), detach_child()
|
||||
handlers/
|
||||
calendar_handlers.py # on calendar attached → rebuild nav
|
||||
market_handlers.py # on market attached → rebuild nav
|
||||
order_handlers.py # on order completed → confirm calendar entries
|
||||
login_handlers.py # on login → adopt anonymous cart/calendar items
|
||||
setup.py # Registers all handlers at app startup
|
||||
```
|
||||
|
||||
### ContainerRelation model
|
||||
```python
|
||||
class ContainerRelation(Base):
|
||||
__tablename__ = "container_relations"
|
||||
id, parent_type, parent_id, child_type, child_id, sort_order, label, created_at, deleted_at
|
||||
# Unique constraint: (parent_type, parent_id, child_type, child_id)
|
||||
```
|
||||
|
||||
This is the central truth about "what's attached to what." A page has calendars and markets attached to it — defined here, not by FKs on the calendar/market tables.
|
||||
|
||||
### MenuNode model (replaces MenuItem)
|
||||
```python
|
||||
class MenuNode(Base):
|
||||
__tablename__ = "menu_nodes"
|
||||
id, container_type, container_id,
|
||||
parent_id (self-referential tree), sort_order, depth,
|
||||
label, slug, href, icon, feature_image,
|
||||
created_at, updated_at, deleted_at
|
||||
```
|
||||
|
||||
This is a **cached navigation tree** built FROM ContainerRelations. A page doesn't know it has markets — but its MenuNode has child MenuNodes for the market because the glue layer put them there.
|
||||
|
||||
### Navigation service (`glue/services/navigation.py`)
|
||||
- `get_navigation_tree(session)` → nested dict for templates (replaces `/internal/menu-items` API)
|
||||
- `rebuild_navigation(session)` → reads ContainerRelations, creates/updates MenuNodes
|
||||
- Called by event handlers when relationships change
|
||||
|
||||
### Relationship service (`glue/services/relationships.py`)
|
||||
- `attach_child(session, parent_type, parent_id, child_type, child_id)` → creates ContainerRelation + emits `container.child_attached` event
|
||||
- `get_children(session, parent_type, parent_id, child_type=None)` → query children
|
||||
- `detach_child(...)` → soft delete + emit `container.child_detached` event
|
||||
|
||||
### Event handlers (the "real code" in the glue layer)
|
||||
```python
|
||||
# glue/handlers/calendar_handlers.py
|
||||
@handler("container.child_attached")
|
||||
async def on_child_attached(payload, session):
|
||||
if payload["child_type"] in ("calendar", "market"):
|
||||
await rebuild_navigation(session)
|
||||
|
||||
# glue/handlers/order_handlers.py (Phase 5 but registered here)
|
||||
@handler("order.created")
|
||||
async def on_order_created(payload, session):
|
||||
# Confirm calendar entries for this order
|
||||
...
|
||||
|
||||
# glue/handlers/login_handlers.py (Phase 5 but registered here)
|
||||
@handler("user.logged_in")
|
||||
async def on_user_logged_in(payload, session):
|
||||
# Adopt anonymous cart items and calendar entries
|
||||
...
|
||||
```
|
||||
|
||||
### Replace menu_items flow
|
||||
**Old:** Each app calls `GET /internal/menu-items` → coop queries MenuItem → returns JSON
|
||||
**New:** Each app calls `glue.services.navigation.get_navigation_tree(g.s)` → direct DB query of MenuNode
|
||||
|
||||
Update context functions in all four `app.py` files:
|
||||
```python
|
||||
# REMOVE: menu_data = await api_get("coop", "/internal/menu-items")
|
||||
# ADD: from glue.services.navigation import get_navigation_tree
|
||||
# ctx["menu_items"] = await get_navigation_tree(g.s)
|
||||
```
|
||||
|
||||
### Data migration
|
||||
- Backfill `menu_nodes` from existing `menu_items` + posts
|
||||
- Backfill `container_relations` from existing calendar/market/page_config container references
|
||||
- Deprecate (then remove) old MenuItem model and `/internal/menu-items` endpoint
|
||||
- Update menu admin UI (`blog/bp/menu_items/`) to manage ContainerRelations + MenuNodes
|
||||
|
||||
### Verify
|
||||
- Navigation renders correctly in all four apps without HTTP calls
|
||||
- Adding a market to a page (via ContainerRelation) triggers nav rebuild and market appears in menu
|
||||
- Adding a calendar to a page does the same
|
||||
- Menu admin UI works with new models
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Event-Driven Cross-App Workflows
|
||||
|
||||
**What:** Replace remaining cross-app FKs and HTTP calls with event-driven flows.
|
||||
|
||||
### 5a. Replace `CalendarEntry.order_id` FK with soft reference
|
||||
|
||||
```python
|
||||
# REMOVE: order_id = Column(Integer, ForeignKey("orders.id"), ...)
|
||||
# ADD: order_ref_id = Column(Integer, nullable=True, index=True)
|
||||
# (No FK constraint — just stores the order ID as an integer)
|
||||
```
|
||||
|
||||
Same for `Ticket.order_id`. Three-step migration (add, backfill, drop FK).
|
||||
|
||||
### 5b. Replace `CartItem.market_place_id` FK with soft reference
|
||||
```python
|
||||
# REMOVE: market_place_id = ForeignKey("market_places.id")
|
||||
# ADD: market_ref_id = Column(Integer, nullable=True, index=True)
|
||||
```
|
||||
|
||||
### 5c. Event-driven order completion
|
||||
Currently `cart/bp/cart/services/checkout.py` line 166 directly writes `CalendarEntry` rows (cross-domain). Replace:
|
||||
|
||||
```python
|
||||
# In create_order_from_cart(), instead of direct UPDATE on CalendarEntry:
|
||||
await emit_event(session, "order.created", "order", order.id, {
|
||||
"order_id": order.id,
|
||||
"user_id": user_id,
|
||||
"calendar_entry_ids": [...],
|
||||
})
|
||||
```
|
||||
|
||||
Glue handler picks it up and updates calendar entries via events-domain code.
|
||||
|
||||
### 5d. Event-driven login adoption
|
||||
Currently `blog/bp/auth/routes.py` line 265 calls `POST /internal/cart/adopt`. Replace:
|
||||
|
||||
```python
|
||||
# In magic() route, instead of api_post("cart", "/internal/cart/adopt"):
|
||||
await emit_event(session, "user.logged_in", "user", user_id, {
|
||||
"user_id": user_id,
|
||||
"anonymous_session_id": anon_session_id,
|
||||
})
|
||||
```
|
||||
|
||||
Glue handler adopts cart items and calendar entries.
|
||||
|
||||
### 5e. Remove cross-domain ORM relationships
|
||||
From models, remove:
|
||||
- `Order.calendar_entries` (relationship to CalendarEntry)
|
||||
- `CalendarEntry.order` (relationship to Order)
|
||||
- `Ticket.order` (relationship to Order)
|
||||
- `CartItem.market_place` (relationship to MarketPlace)
|
||||
|
||||
### 5f. Move cross-domain queries to glue services
|
||||
`cart/bp/cart/services/checkout.py` currently imports `CalendarEntry`, `Calendar`, `MarketPlace` directly. Move these queries to glue service functions that bridge the domains:
|
||||
- `glue/services/cart_calendar.py` — query calendar entries for a cart identity
|
||||
- `glue/services/page_resolution.py` — determine which page/container a cart belongs to using ContainerRelation
|
||||
|
||||
### Final FK audit after Phase 5
|
||||
All remaining FKs are either:
|
||||
- **Within the same app domain** (Order→OrderItem, Calendar→CalendarSlot, etc.)
|
||||
- **To shared models** (anything→User)
|
||||
- **One pragmatic exception:** `OrderItem.product_id → products.id` (cross cart→market, but OrderItem already snapshots title/price, so this FK is just for reporting)
|
||||
|
||||
### Verify
|
||||
- Login triggers `user.logged_in` event → cart/calendar adoption happens via glue handler
|
||||
- Order creation triggers `order.created` event → calendar entries confirmed via glue handler
|
||||
- No cross-app FKs remain (except the pragmatic OrderItem→Product)
|
||||
- All apps could theoretically point at separate databases
|
||||
- Event processor reliably processes and retries all events
|
||||
|
||||
---
|
||||
|
||||
## Execution Order
|
||||
|
||||
Each phase leaves the system fully working. No big-bang migration.
|
||||
|
||||
| Phase | Risk | Size | Depends On |
|
||||
|-------|------|------|------------|
|
||||
| 1. Extract shared_lib | Low (mechanical refactor) | Medium | Nothing |
|
||||
| 2. Event infra + logging | Low (purely additive) | Small | Phase 1 |
|
||||
| 3. Generic containers | Medium (schema + query changes) | Medium | Phase 1 |
|
||||
| 4. Glue layer | Medium (new subsystem, menu migration) | Large | Phases 2 + 3 |
|
||||
| 5. Event-driven workflows | Medium (behavioral change in checkout/login) | Medium | Phase 4 |
|
||||
|
||||
Phases 2 and 3 can run in parallel after Phase 1. Phase 4 needs both. Phase 5 needs Phase 4.
|
||||
302
docs/scalability-plan.md
Normal file
302
docs/scalability-plan.md
Normal file
@@ -0,0 +1,302 @@
|
||||
# Rose Ash Scalability Plan
|
||||
|
||||
## Context
|
||||
|
||||
The coop runs 6 Quart microservices (blog, market, cart, events, federation, account) + Art-DAG (L1/L2 FastAPI) on a single-node Docker Swarm. Current architecture handles light traffic but has concrete bottlenecks: single Postgres, single Redis (256MB), no replicas, single Hypercorn worker per container, sequential AP delivery, no circuit breakers, and 6 competing EventProcessors. The decoupling work (contracts, HTTP data/actions, fragment composition) is complete — the code is structurally ready to scale, but the deployment and runtime aren't.
|
||||
|
||||
This plan covers everything from quick config wins to massive federation scale, organized in 4 tiers. Each tier unlocks roughly an order of magnitude.
|
||||
|
||||
---
|
||||
|
||||
## TIER 0 — Deploy Existing Code + Quick Config
|
||||
|
||||
**Target: low thousands concurrent. Effort: hours.**
|
||||
|
||||
### T0.1: Separate Auth Redis
|
||||
|
||||
**Why:** Auth keys (`grant:*`, `did_auth:*`, `prompt:*`) on DB 15 share a single Redis instance (256MB, `allkeys-lru`). Cache pressure from fragment/page caching can silently evict auth state, causing spurious logouts.
|
||||
|
||||
**Files:**
|
||||
- `docker-compose.yml` — add `redis-auth` service: `redis:7-alpine`, `--maxmemory 64mb --maxmemory-policy noeviction`
|
||||
- `docker-compose.yml` — update `REDIS_AUTH_URL: redis://redis-auth:6379/0` in `x-app-env`
|
||||
|
||||
**No code changes** — `shared/infrastructure/auth_redis.py` already reads `REDIS_AUTH_URL` from env.
|
||||
|
||||
### T0.2: Bump Redis Memory
|
||||
|
||||
**File:** `docker-compose.yml` (line 168)
|
||||
- Change `--maxmemory 256mb` to `--maxmemory 1gb` (or 512mb minimum)
|
||||
- Keep `allkeys-lru` for the data Redis (fragments + page cache)
|
||||
|
||||
### T0.3: Deploy the Database Split
|
||||
|
||||
**What exists:** `_config/init-databases.sql` (creates 6 DBs) and `_config/split-databases.sh` (migrates table groups). Code in `session.py` lines 46-101 already creates separate engines when URLs differ. `bus.py`, `user_loader.py`, and `factory.py` all have conditional cross-DB paths.
|
||||
|
||||
**Files:**
|
||||
- `docker-compose.yml` — per-service `DATABASE_URL` overrides (blog→`db_blog`, market→`db_market`, etc.) plus `DATABASE_URL_ACCOUNT`→`db_account`, `DATABASE_URL_FEDERATION`→`db_federation`
|
||||
- `_config/split-databases.sh` — add `menu_nodes` + `container_relations` to ALL target DBs (small read-only tables needed by `get_navigation_tree()` in non-blog apps until T1.7 replaces this)
|
||||
|
||||
**Deployment:** run `init-databases.sql`, stop services, run `split-databases.sh`, update compose env, redeploy.
|
||||
|
||||
### T0.4: Add PgBouncer
|
||||
|
||||
**Why:** 6 apps x pool_size 5 + overflow 10 = 90 connections to one Postgres. After DB split + workers, this multiplies. Postgres default `max_connections=100` will be hit.
|
||||
|
||||
**Files:**
|
||||
- `docker-compose.yml` — add `pgbouncer` service (transaction-mode pooling, `default_pool_size=20`, `max_client_conn=300`)
|
||||
- `docker-compose.yml` — change all `DATABASE_URL` values from `@db:5432` to `@pgbouncer:6432`
|
||||
- `shared/db/session.py` (lines 13-20) — add `pool_timeout=10`, `pool_recycle=1800`. Consider reducing `pool_size` to 3 since PgBouncer handles pooling.
|
||||
|
||||
### T0.5: Hypercorn Workers
|
||||
|
||||
**Why:** Single async event loop per container. CPU-bound work (Jinja2 rendering, RSA signing) blocks everything.
|
||||
|
||||
**Files:**
|
||||
- All 6 `{app}/entrypoint.sh` — change Hypercorn command to:
|
||||
```
|
||||
exec hypercorn "${APP_MODULE:-app:app}" --bind 0.0.0.0:${PORT:-8000} --workers ${WORKERS:-2} --keep-alive 75
|
||||
```
|
||||
|
||||
**No code changes needed** — `EventProcessor` uses `SKIP LOCKED` so multiple workers competing is safe. Each worker gets its own httpx client singleton and SQLAlchemy pool (correct fork behavior).
|
||||
|
||||
**Depends on:** T0.4 (PgBouncer) — doubling workers doubles connection count.
|
||||
|
||||
---
|
||||
|
||||
## TIER 1 — Fix Hot Paths
|
||||
|
||||
**Target: tens of thousands concurrent. Effort: days.**
|
||||
|
||||
### T1.1: Concurrent AP Delivery
|
||||
|
||||
**The single biggest bottleneck.** `ap_delivery_handler.py` delivers to inboxes sequentially (lines 230-246). 100 followers = up to 100 x 15s = 25 minutes.
|
||||
|
||||
**File:** `shared/events/handlers/ap_delivery_handler.py`
|
||||
- Add `DELIVERY_CONCURRENCY = 10` semaphore
|
||||
- Replace sequential inbox loop with `asyncio.gather()` bounded by semaphore
|
||||
- Reuse a single `httpx.AsyncClient` per `on_any_activity` call with explicit pool limits: `httpx.Limits(max_connections=20, max_keepalive_connections=10)`
|
||||
- Batch `APDeliveryLog` inserts (one flush per activity, not per inbox)
|
||||
|
||||
**Result:** 100 followers drops from 25 min → ~2.5 min (10 concurrent) or less.
|
||||
|
||||
### T1.2: Fragment Circuit Breaker + Stale-While-Revalidate
|
||||
|
||||
**Why:** Every page render makes 3-4 internal HTTP calls (`fetch_fragments()`). A slow/down service blocks all page renders for up to 2s per fragment. No graceful degradation.
|
||||
|
||||
**File:** `shared/infrastructure/fragments.py`
|
||||
|
||||
**Circuit breaker** (add near line 27):
|
||||
- Per-app `_CircuitState` tracking consecutive failures
|
||||
- Threshold: 3 failures → circuit opens for 30s
|
||||
- When open: skip HTTP, fall through to stale cache
|
||||
|
||||
**Stale-while-revalidate** (modify `fetch_fragment_cached()`, lines 246-288):
|
||||
- Store fragments in Redis as `{"html": "...", "ts": 1234567890.0}` instead of plain string
|
||||
- Soft TTL = normal TTL (30s). Hard TTL = soft + 300s (5 min stale window)
|
||||
- Within soft TTL: return cached. Between soft and hard: return cached + background revalidate. Past hard: block on fetch. On fetch failure: return stale if available, empty string if not.
|
||||
|
||||
### T1.3: Partition Event Processors
|
||||
|
||||
**Why:** All 6 apps register the wildcard AP delivery handler via `register_shared_handlers()`. All 6 `EventProcessor` instances compete for every activity with `SKIP LOCKED`. 5 of 6 do wasted work on every public activity.
|
||||
|
||||
**Files:**
|
||||
- `shared/events/handlers/__init__.py` — add `app_name` param to `register_shared_handlers()`. Only import `ap_delivery_handler` and `external_delivery_handler` when `app_name == "federation"`.
|
||||
- `shared/infrastructure/factory.py` (line 277) — pass `name` to `register_shared_handlers(name)`
|
||||
- `shared/infrastructure/factory.py` (line 41) — add `event_processor_all_origins: bool = False` param to `create_base_app()`
|
||||
- `shared/infrastructure/factory.py` (line 271) — `EventProcessor(app_name=None if event_processor_all_origins else name)`
|
||||
- `federation/app.py` — pass `event_processor_all_origins=True`
|
||||
|
||||
**Result:** Federation processes ALL activities (no origin filter) for delivery. Other apps process only their own origin activities for domain-specific handlers. No wasted lock contention.
|
||||
|
||||
### T1.4: httpx Client Pool Limits
|
||||
|
||||
**Why:** All three HTTP clients (`fragments.py`, `data_client.py`, `actions.py`) have no `limits` parameter. Default is 100 connections per client. With workers + replicas this fans out unbounded.
|
||||
|
||||
**Files:**
|
||||
- `shared/infrastructure/fragments.py` (lines 38-45) — add `limits=httpx.Limits(max_connections=20, max_keepalive_connections=10)`
|
||||
- `shared/infrastructure/data_client.py` (lines 36-43) — same
|
||||
- `shared/infrastructure/actions.py` (lines 37-44) — `max_connections=10, max_keepalive_connections=5`
|
||||
|
||||
### T1.5: Data Client Caching
|
||||
|
||||
**Why:** `fetch_data()` has no caching. `cart-summary` is called on every page load across 4 apps. Blog post lookups by slug happen on every market/events page.
|
||||
|
||||
**File:** `shared/infrastructure/data_client.py`
|
||||
- Add `fetch_data_cached()` following the `fetch_fragment_cached()` pattern
|
||||
- Redis key: `data:{app}:{query}:{sorted_params}`, default TTL=10s
|
||||
- Same circuit breaker + SWR as T1.2
|
||||
|
||||
**Callers updated:** all `app.py` context functions that call `fetch_data()` for repeated reads.
|
||||
|
||||
### T1.6: Navigation via HTTP Data Endpoint
|
||||
|
||||
**Why:** After DB split, `get_navigation_tree()` queries `menu_nodes` via `g.s`. But `menu_nodes` lives in `db_blog`. The T0.3 workaround (replicate table) works short-term; this is the proper fix.
|
||||
|
||||
**Files:**
|
||||
- `shared/contracts/dtos.py` — add `MenuNodeDTO`
|
||||
- `blog/bp/data/routes.py` — add `nav-tree` handler returning `[dto_to_dict(node) for node in nodes]`
|
||||
- All non-blog `app.py` context functions — replace `get_navigation_tree(g.s)` with `fetch_data_cached("blog", "nav-tree", ttl=60)`
|
||||
- Remove `menu_nodes` from non-blog DBs in `split-databases.sh` (no longer needed)
|
||||
|
||||
**Depends on:** T0.3 (DB split), T1.5 (data caching).
|
||||
|
||||
### T1.7: Fix Fragment Batch Parser O(n^2)
|
||||
|
||||
**File:** `shared/infrastructure/fragments.py` `_parse_fragment_markers()` (lines 217-243)
|
||||
- Replace nested loop with single-pass: find all `<!-- fragment:{key} -->` markers in one scan, extract content between consecutive markers
|
||||
- O(n) instead of O(n^2)
|
||||
|
||||
### T1.8: Read Replicas
|
||||
|
||||
**Why:** After DB split, each domain DB has one writer. Read-heavy pages (listings, calendars, product pages) can saturate it.
|
||||
|
||||
**Files:**
|
||||
- `docker-compose.yml` — add read replicas for high-traffic domains (blog, events)
|
||||
- `shared/db/session.py` — add `DATABASE_URL_RO` env var, create read-only engine, add `get_read_session()` context manager
|
||||
- All `bp/data/routes.py` and `bp/fragments/routes.py` — use read session (these endpoints are inherently read-only)
|
||||
|
||||
**Depends on:** T0.3 (DB split), T0.4 (PgBouncer).
|
||||
|
||||
---
|
||||
|
||||
## TIER 2 — Decouple the Runtime
|
||||
|
||||
**Target: hundreds of thousands concurrent. Effort: ~1 week.**
|
||||
|
||||
### T2.1: Edge-Side Fragment Composition (Nginx SSI)
|
||||
|
||||
**Why:** Currently every Quart app fetches 3-4 fragments per request via HTTP (`fetch_fragments()` in context processors). This adds latency and creates liveness coupling. SSI moves fragment assembly to Nginx, which caches each fragment independently.
|
||||
|
||||
**Changes:**
|
||||
- `shared/infrastructure/jinja_setup.py` — change `_fragment()` Jinja global to emit SSI directives: `<!--#include virtual="/_ssi/{app}/{type}?{params}" -->`
|
||||
- New Nginx config — map `/_ssi/{app}/{path}` to `http://{app}:8000/internal/fragments/{path}`, enable `ssi on`, proxy_cache with short TTL
|
||||
- All fragment blueprint routes — add `Cache-Control: public, max-age={ttl}, stale-while-revalidate=300` headers
|
||||
- Remove `fetch_fragments()` calls from all `app.py` context processors (templates emit SSI directly)
|
||||
|
||||
### T2.2: Replace Outbox Polling with Redis Streams
|
||||
|
||||
**Why:** `EventProcessor` polls `ap_activities` every 2s with `SELECT FOR UPDATE SKIP LOCKED`. This creates persistent DB load even when idle. LISTEN/NOTIFY helps but is fragile.
|
||||
|
||||
**Changes:**
|
||||
- `shared/events/bus.py` `emit_activity()` — after writing to DB, also `XADD coop:activities:pending` with `{activity_id, origin_app, type}`
|
||||
- `shared/events/processor.py` — replace `_poll_loop` + `_listen_for_notify` with `XREADGROUP` (blocking read, no polling). Consumer groups handle partitioning. `XPENDING` + `XCLAIM` replace the stuck activity reaper.
|
||||
- Redis Stream config: `MAXLEN ~10000` to cap memory
|
||||
|
||||
### T2.3: CDN for Static Assets
|
||||
|
||||
- Route `*.rose-ash.com/static/*` through CDN (Cloudflare, BunnyCDN)
|
||||
- `_asset_url()` already adds `?v={hash}` fingerprint — CDN can cache with `max-age=31536000, immutable`
|
||||
- No code changes, just DNS + CDN config
|
||||
|
||||
### T2.4: Horizontal Scaling (Docker Swarm Replicas)
|
||||
|
||||
**Changes:**
|
||||
- `docker-compose.yml` — add `replicas: 2` (or 3) to blog, market, events, cart. Keep federation at 1 (handles all AP delivery).
|
||||
- `blog/entrypoint.sh` — wrap Alembic in PostgreSQL advisory lock (`SELECT pg_advisory_lock(42)`) so only one replica runs migrations
|
||||
- `docker-compose.yml` — add health checks per service
|
||||
- `shared/db/session.py` — make pool sizes configurable via env vars (`DB_POOL_SIZE`, `DB_MAX_OVERFLOW`) so replicas can use smaller pools
|
||||
|
||||
**Depends on:** T0.4 (PgBouncer), T0.5 (workers).
|
||||
|
||||
---
|
||||
|
||||
## TIER 3 — Federation Scale
|
||||
|
||||
**Target: millions (federated network effects). Effort: weeks.**
|
||||
|
||||
### T3.1: Dedicated AP Delivery Service
|
||||
|
||||
**Why:** AP delivery is CPU-intensive (RSA signing) and I/O-intensive (HTTP to remote servers). Running inside federation's web worker blocks request processing.
|
||||
|
||||
**Changes:**
|
||||
- New `delivery/` service — standalone asyncio app (no Quart/web server)
|
||||
- Reads from Redis Stream `coop:delivery:pending` (from T2.2)
|
||||
- Loads activity from federation DB, loads followers, delivers with semaphore (from T1.1)
|
||||
- `ap_delivery_handler.py` `on_any_activity` → enqueues to stream instead of delivering inline
|
||||
- `docker-compose.yml` — add `delivery-worker` service with `replicas: 2`, no port binding
|
||||
|
||||
**Depends on:** T2.2 (Redis Streams), T1.1 (concurrent delivery).
|
||||
|
||||
### T3.2: Per-Domain Health Tracking + Backoff
|
||||
|
||||
**Why:** Dead remote servers waste delivery slots. Current code retries 5 times with no backoff.
|
||||
|
||||
**Changes:**
|
||||
- New `shared/events/domain_health.py` — Redis hash `domain:health:{domain}` tracking consecutive failures, exponential backoff schedule (30s → 1min → 5min → 15min → 1hr → 6hr → 24hr)
|
||||
- Delivery worker checks domain health before attempting delivery; skips domains in backoff
|
||||
- On success: reset. On failure: increment + extend backoff.
|
||||
|
||||
### T3.3: Shared Inbox Optimization
|
||||
|
||||
**Why:** 100 Mastodon followers from one instance = 100 POSTs to the same server. Mastodon supports `sharedInbox` — one POST covers all followers on that instance.
|
||||
|
||||
**Changes:**
|
||||
- `shared/models/federation.py` `APFollower` — add `shared_inbox_url` column
|
||||
- Migration to backfill from `ap_remote_actors.shared_inbox_url`
|
||||
- `ap_delivery_handler.py` — group followers by domain, prefer shared inbox when available
|
||||
- **Impact:** 100 followers on one instance → 1 HTTP POST instead of 100
|
||||
|
||||
### T3.4: Table Partitioning for `ap_activities`
|
||||
|
||||
**Why:** At millions of activities, the `ap_activities` table becomes the query bottleneck. The `EventProcessor` query orders by `created_at` — native range partitioning fits perfectly.
|
||||
|
||||
**Changes:**
|
||||
- Alembic migration — convert `ap_activities` to `PARTITION BY RANGE (created_at)` with monthly partitions
|
||||
- Add a cron job or startup hook to create future partitions
|
||||
- No application code changes needed (transparent to SQLAlchemy)
|
||||
|
||||
### T3.5: Read-Through DTO Cache
|
||||
|
||||
**Why:** Hot cross-app reads (`cart-summary`, `post-by-slug`) go through HTTP even with T1.5 caching. A Redis-backed DTO cache with event-driven invalidation eliminates HTTP for repeated reads entirely.
|
||||
|
||||
**Changes:**
|
||||
- New `shared/infrastructure/dto_cache.py` — `get(app, query, params)` / `set(...)` backed by Redis
|
||||
- Integrate into `fetch_data_cached()` as L1 cache (check before HTTP)
|
||||
- Action endpoints invalidate relevant cache keys after successful writes
|
||||
|
||||
**Depends on:** T1.5 (data caching), T2.2 (event-driven invalidation via streams).
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
```
|
||||
TIER 0 (hours):
|
||||
T0.1 Auth Redis ──┐
|
||||
T0.2 Redis memory ├── all independent, do in parallel
|
||||
T0.3 DB split ────┘
|
||||
T0.4 PgBouncer ────── after T0.3
|
||||
T0.5 Workers ──────── after T0.4
|
||||
|
||||
TIER 1 (days):
|
||||
T1.1 Concurrent AP ──┐
|
||||
T1.3 Partition procs ─├── independent, do in parallel
|
||||
T1.4 Pool limits ─────┤
|
||||
T1.7 Parse fix ───────┘
|
||||
T1.2 Circuit breaker ── after T1.4
|
||||
T1.5 Data caching ───── after T1.2 (same pattern)
|
||||
T1.6 Nav endpoint ───── after T1.5
|
||||
T1.8 Read replicas ──── after T0.3 + T0.4
|
||||
|
||||
TIER 2 (~1 week):
|
||||
T2.3 CDN ──────────── independent, anytime
|
||||
T2.2 Redis Streams ─── foundation for T3.1
|
||||
T2.1 Nginx SSI ────── after T1.2 (fragments stable)
|
||||
T2.4 Replicas ──────── after T0.4 + T0.5
|
||||
|
||||
TIER 3 (weeks):
|
||||
T3.1 Delivery service ── after T2.2 + T1.1
|
||||
T3.2 Domain health ───── after T3.1
|
||||
T3.3 Shared inbox ────── after T3.1
|
||||
T3.4 Table partitioning ─ independent
|
||||
T3.5 DTO cache ────────── after T1.5 + T2.2
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
Each tier has a clear "it works" test:
|
||||
|
||||
- **Tier 0:** All 6 apps respond. Login works across apps. `pg_stat_activity` shows connections through PgBouncer. `redis-cli -p 6380 INFO memory` shows auth Redis separate.
|
||||
- **Tier 1:** Emit a public AP activity with 50+ followers — delivery completes in seconds not minutes. Stop account service — blog pages still render with stale auth-menu. Only federation's processor handles delivery.
|
||||
- **Tier 2:** Nginx access log shows SSI fragment cache HITs. `XINFO GROUPS coop:activities:pending` shows active consumer groups. CDN cache-status headers show HITs on static assets. Multiple replicas serve traffic.
|
||||
- **Tier 3:** Delivery worker scales independently. Domain health Redis hash shows backoff state for unreachable servers. `EXPLAIN` on `ap_activities` shows partition pruning. Shared inbox delivery logs show 1 POST per domain.
|
||||
@@ -105,6 +105,16 @@ async def get_federation_session():
|
||||
await sess.close()
|
||||
|
||||
|
||||
def needs_federation_session() -> bool:
|
||||
"""True when the federation DB is separate from the app's main DB."""
|
||||
return DATABASE_URL_FEDERATION != DATABASE_URL
|
||||
|
||||
|
||||
def create_federation_session() -> AsyncSession:
|
||||
"""Create an unmanaged session targeting the federation database."""
|
||||
return _FederationSession()
|
||||
|
||||
|
||||
def register_db(app: Quart):
|
||||
|
||||
@app.before_request
|
||||
|
||||
@@ -69,6 +69,56 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
# For per-app outboxes, filter by origin_app; for federation, show all
|
||||
outbox_origin_app: str | None = None if aggregate else app_name
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Federation session management — AP tables live in db_federation,
|
||||
# which is separate from each app's own DB after the per-app split.
|
||||
# g._ap_s points to the federation DB; on the federation app itself
|
||||
# it's just an alias for g.s.
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
from shared.db.session import needs_federation_session, create_federation_session
|
||||
|
||||
@bp.before_request
|
||||
async def _open_ap_session():
|
||||
if needs_federation_session():
|
||||
sess = create_federation_session()
|
||||
g._ap_s = sess
|
||||
g._ap_tx = await sess.begin()
|
||||
g._ap_own = True
|
||||
else:
|
||||
g._ap_s = g.s
|
||||
g._ap_own = False
|
||||
|
||||
@bp.after_request
|
||||
async def _commit_ap_session(response):
|
||||
if getattr(g, "_ap_own", False):
|
||||
if 200 <= response.status_code < 400:
|
||||
try:
|
||||
await g._ap_tx.commit()
|
||||
except Exception:
|
||||
try:
|
||||
await g._ap_tx.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
return response
|
||||
|
||||
@bp.teardown_request
|
||||
async def _close_ap_session(exc):
|
||||
if getattr(g, "_ap_own", False):
|
||||
s = getattr(g, "_ap_s", None)
|
||||
if s:
|
||||
if exc is not None or s.in_transaction():
|
||||
tx = getattr(g, "_ap_tx", None)
|
||||
if tx and tx.is_active:
|
||||
try:
|
||||
await tx.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
await s.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Well-known endpoints
|
||||
# ------------------------------------------------------------------
|
||||
@@ -87,7 +137,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
if res_domain != domain:
|
||||
abort(404, "User not on this server")
|
||||
|
||||
actor = await services.federation.get_actor_by_username(g.s, username)
|
||||
actor = await services.federation.get_actor_by_username(g._ap_s, username)
|
||||
if not actor:
|
||||
abort(404, "User not found")
|
||||
|
||||
@@ -128,7 +178,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
|
||||
@bp.get("/nodeinfo/2.0")
|
||||
async def nodeinfo():
|
||||
stats = await services.federation.get_stats(g.s)
|
||||
stats = await services.federation.get_stats(g._ap_s)
|
||||
return Response(
|
||||
response=json.dumps({
|
||||
"version": "2.0",
|
||||
@@ -170,7 +220,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
|
||||
@bp.get("/users/<username>")
|
||||
async def actor_profile(username: str):
|
||||
actor = await services.federation.get_actor_by_username(g.s, username)
|
||||
actor = await services.federation.get_actor_by_username(g._ap_s, username)
|
||||
if not actor:
|
||||
abort(404)
|
||||
|
||||
@@ -224,7 +274,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
if aggregate:
|
||||
from quart import render_template
|
||||
activities, total = await services.federation.get_outbox(
|
||||
g.s, username, page=1, per_page=20,
|
||||
g._ap_s, username, page=1, per_page=20,
|
||||
)
|
||||
return await render_template(
|
||||
"federation/profile.html",
|
||||
@@ -242,7 +292,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
@csrf_exempt
|
||||
@bp.post("/users/<username>/inbox")
|
||||
async def inbox(username: str):
|
||||
actor = await services.federation.get_actor_by_username(g.s, username)
|
||||
actor = await services.federation.get_actor_by_username(g._ap_s, username)
|
||||
if not actor:
|
||||
abort(404)
|
||||
|
||||
@@ -284,7 +334,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
|
||||
# Load actor row for DB operations
|
||||
actor_row = (
|
||||
await g.s.execute(
|
||||
await g._ap_s.execute(
|
||||
select(ActorProfile).where(
|
||||
ActorProfile.preferred_username == username
|
||||
)
|
||||
@@ -298,13 +348,13 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
activity_type=activity_type,
|
||||
from_actor=from_actor_url,
|
||||
)
|
||||
g.s.add(item)
|
||||
await g.s.flush()
|
||||
g._ap_s.add(item)
|
||||
await g._ap_s.flush()
|
||||
|
||||
# Dispatch to shared handlers
|
||||
from shared.infrastructure.ap_inbox_handlers import dispatch_inbox_activity
|
||||
await dispatch_inbox_activity(
|
||||
g.s, actor_row, body, from_actor_url,
|
||||
g._ap_s, actor_row, body, from_actor_url,
|
||||
domain=domain,
|
||||
app_domain=follower_app_domain,
|
||||
)
|
||||
@@ -312,7 +362,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
# Mark as processed
|
||||
item.state = "processed"
|
||||
item.processed_at = datetime.now(timezone.utc)
|
||||
await g.s.flush()
|
||||
await g._ap_s.flush()
|
||||
|
||||
return Response(status=202)
|
||||
|
||||
@@ -322,7 +372,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
|
||||
@bp.get("/users/<username>/outbox")
|
||||
async def outbox(username: str):
|
||||
actor = await services.federation.get_actor_by_username(g.s, username)
|
||||
actor = await services.federation.get_actor_by_username(g._ap_s, username)
|
||||
if not actor:
|
||||
abort(404)
|
||||
|
||||
@@ -331,7 +381,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
|
||||
if not page_param:
|
||||
_, total = await services.federation.get_outbox(
|
||||
g.s, username, page=1, per_page=1,
|
||||
g._ap_s, username, page=1, per_page=1,
|
||||
origin_app=outbox_origin_app,
|
||||
)
|
||||
return Response(
|
||||
@@ -347,7 +397,7 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
|
||||
page_num = int(page_param)
|
||||
activities, total = await services.federation.get_outbox(
|
||||
g.s, username, page=page_num, per_page=20,
|
||||
g._ap_s, username, page=page_num, per_page=20,
|
||||
origin_app=outbox_origin_app,
|
||||
)
|
||||
|
||||
@@ -383,13 +433,13 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
|
||||
@bp.get("/users/<username>/followers")
|
||||
async def followers(username: str):
|
||||
actor = await services.federation.get_actor_by_username(g.s, username)
|
||||
actor = await services.federation.get_actor_by_username(g._ap_s, username)
|
||||
if not actor:
|
||||
abort(404)
|
||||
|
||||
collection_id = f"https://{domain}/users/{username}/followers"
|
||||
follower_list = await services.federation.get_followers(
|
||||
g.s, username, app_domain=follower_app_domain,
|
||||
g._ap_s, username, app_domain=follower_app_domain,
|
||||
)
|
||||
page_param = request.args.get("page")
|
||||
|
||||
@@ -419,12 +469,12 @@ def create_activitypub_blueprint(app_name: str) -> Blueprint:
|
||||
|
||||
@bp.get("/users/<username>/following")
|
||||
async def following(username: str):
|
||||
actor = await services.federation.get_actor_by_username(g.s, username)
|
||||
actor = await services.federation.get_actor_by_username(g._ap_s, username)
|
||||
if not actor:
|
||||
abort(404)
|
||||
|
||||
collection_id = f"https://{domain}/users/{username}/following"
|
||||
following_list, total = await services.federation.get_following(g.s, username)
|
||||
following_list, total = await services.federation.get_following(g._ap_s, username)
|
||||
page_param = request.args.get("page")
|
||||
|
||||
if not page_param:
|
||||
|
||||
@@ -19,10 +19,56 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
"""Create a per-app social blueprint scoped to *app_name*."""
|
||||
bp = Blueprint("ap_social", __name__, url_prefix="/social")
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Federation session — AP tables live in db_federation.
|
||||
# ------------------------------------------------------------------
|
||||
from shared.db.session import needs_federation_session, create_federation_session
|
||||
|
||||
@bp.before_request
|
||||
async def _open_ap_session():
|
||||
if needs_federation_session():
|
||||
sess = create_federation_session()
|
||||
g._ap_s = sess
|
||||
g._ap_tx = await sess.begin()
|
||||
g._ap_own = True
|
||||
else:
|
||||
g._ap_s = g.s
|
||||
g._ap_own = False
|
||||
|
||||
@bp.after_request
|
||||
async def _commit_ap_session(response):
|
||||
if getattr(g, "_ap_own", False):
|
||||
if 200 <= response.status_code < 400:
|
||||
try:
|
||||
await g._ap_tx.commit()
|
||||
except Exception:
|
||||
try:
|
||||
await g._ap_tx.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
return response
|
||||
|
||||
@bp.teardown_request
|
||||
async def _close_ap_session(exc):
|
||||
if getattr(g, "_ap_own", False):
|
||||
s = getattr(g, "_ap_s", None)
|
||||
if s:
|
||||
if exc is not None or s.in_transaction():
|
||||
tx = getattr(g, "_ap_tx", None)
|
||||
if tx and tx.is_active:
|
||||
try:
|
||||
await tx.rollback()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
await s.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@bp.before_request
|
||||
async def load_actor():
|
||||
if g.get("user"):
|
||||
actor = await services.federation.get_actor_by_user_id(g.s, g.user.id)
|
||||
actor = await services.federation.get_actor_by_user_id(g._ap_s, g.user.id)
|
||||
g._social_actor = actor
|
||||
|
||||
def _require_actor():
|
||||
@@ -51,10 +97,10 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
total = 0
|
||||
followed_urls: set[str] = set()
|
||||
if query:
|
||||
actors, total = await services.federation.search_actors(g.s, query)
|
||||
actors, total = await services.federation.search_actors(g._ap_s, query)
|
||||
if actor:
|
||||
following, _ = await services.federation.get_following(
|
||||
g.s, actor.preferred_username, page=1, per_page=1000,
|
||||
g._ap_s, actor.preferred_username, page=1, per_page=1000,
|
||||
)
|
||||
followed_urls = {a.actor_url for a in following}
|
||||
return await render_template(
|
||||
@@ -77,11 +123,11 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
followed_urls: set[str] = set()
|
||||
if query:
|
||||
actors, total = await services.federation.search_actors(
|
||||
g.s, query, page=page,
|
||||
g._ap_s, query, page=page,
|
||||
)
|
||||
if actor:
|
||||
following, _ = await services.federation.get_following(
|
||||
g.s, actor.preferred_username, page=1, per_page=1000,
|
||||
g._ap_s, actor.preferred_username, page=1, per_page=1000,
|
||||
)
|
||||
followed_urls = {a.actor_url for a in following}
|
||||
return await render_template(
|
||||
@@ -103,7 +149,7 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
remote_actor_url = form.get("actor_url", "")
|
||||
if remote_actor_url:
|
||||
await services.federation.send_follow(
|
||||
g.s, actor.preferred_username, remote_actor_url,
|
||||
g._ap_s, actor.preferred_username, remote_actor_url,
|
||||
)
|
||||
if request.headers.get("HX-Request"):
|
||||
return await _actor_card_response(actor, remote_actor_url, is_followed=True)
|
||||
@@ -116,7 +162,7 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
remote_actor_url = form.get("actor_url", "")
|
||||
if remote_actor_url:
|
||||
await services.federation.unfollow(
|
||||
g.s, actor.preferred_username, remote_actor_url,
|
||||
g._ap_s, actor.preferred_username, remote_actor_url,
|
||||
)
|
||||
if request.headers.get("HX-Request"):
|
||||
return await _actor_card_response(actor, remote_actor_url, is_followed=False)
|
||||
@@ -125,7 +171,7 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
async def _actor_card_response(actor, remote_actor_url, is_followed):
|
||||
"""Re-render a single actor card after follow/unfollow via HTMX."""
|
||||
remote_dto = await services.federation.get_or_fetch_remote_actor(
|
||||
g.s, remote_actor_url,
|
||||
g._ap_s, remote_actor_url,
|
||||
)
|
||||
if not remote_dto:
|
||||
return Response("", status=200)
|
||||
@@ -151,10 +197,10 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
async def followers_list():
|
||||
actor = _require_actor()
|
||||
actors, total = await services.federation.get_followers_paginated(
|
||||
g.s, actor.preferred_username, app_domain=app_name,
|
||||
g._ap_s, actor.preferred_username, app_domain=app_name,
|
||||
)
|
||||
following, _ = await services.federation.get_following(
|
||||
g.s, actor.preferred_username, page=1, per_page=1000,
|
||||
g._ap_s, actor.preferred_username, page=1, per_page=1000,
|
||||
)
|
||||
followed_urls = {a.actor_url for a in following}
|
||||
return await render_template(
|
||||
@@ -171,10 +217,10 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
actor = _require_actor()
|
||||
page = request.args.get("page", 1, type=int)
|
||||
actors, total = await services.federation.get_followers_paginated(
|
||||
g.s, actor.preferred_username, page=page, app_domain=app_name,
|
||||
g._ap_s, actor.preferred_username, page=page, app_domain=app_name,
|
||||
)
|
||||
following, _ = await services.federation.get_following(
|
||||
g.s, actor.preferred_username, page=1, per_page=1000,
|
||||
g._ap_s, actor.preferred_username, page=1, per_page=1000,
|
||||
)
|
||||
followed_urls = {a.actor_url for a in following}
|
||||
return await render_template(
|
||||
@@ -193,7 +239,7 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
async def following_list():
|
||||
actor = _require_actor()
|
||||
actors, total = await services.federation.get_following(
|
||||
g.s, actor.preferred_username,
|
||||
g._ap_s, actor.preferred_username,
|
||||
)
|
||||
return await render_template(
|
||||
"social/following.html",
|
||||
@@ -208,7 +254,7 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
actor = _require_actor()
|
||||
page = request.args.get("page", 1, type=int)
|
||||
actors, total = await services.federation.get_following(
|
||||
g.s, actor.preferred_username, page=page,
|
||||
g._ap_s, actor.preferred_username, page=page,
|
||||
)
|
||||
return await render_template(
|
||||
"social/_actor_list_items.html",
|
||||
@@ -228,7 +274,7 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
from shared.models.federation import RemoteActor
|
||||
from sqlalchemy import select as sa_select
|
||||
remote = (
|
||||
await g.s.execute(
|
||||
await g._ap_s.execute(
|
||||
sa_select(RemoteActor).where(RemoteActor.id == id)
|
||||
)
|
||||
).scalar_one_or_none()
|
||||
@@ -236,12 +282,12 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
abort(404)
|
||||
from shared.services.federation_impl import _remote_actor_to_dto
|
||||
remote_dto = _remote_actor_to_dto(remote)
|
||||
items = await services.federation.get_actor_timeline(g.s, id)
|
||||
items = await services.federation.get_actor_timeline(g._ap_s, id)
|
||||
is_following = False
|
||||
if actor:
|
||||
from shared.models.federation import APFollowing
|
||||
existing = (
|
||||
await g.s.execute(
|
||||
await g._ap_s.execute(
|
||||
sa_select(APFollowing).where(
|
||||
APFollowing.actor_profile_id == actor.id,
|
||||
APFollowing.remote_actor_id == id,
|
||||
@@ -268,7 +314,7 @@ def create_ap_social_blueprint(app_name: str) -> Blueprint:
|
||||
except ValueError:
|
||||
pass
|
||||
items = await services.federation.get_actor_timeline(
|
||||
g.s, id, before=before,
|
||||
g._ap_s, id, before=before,
|
||||
)
|
||||
return await render_template(
|
||||
"social/_timeline_items.html",
|
||||
|
||||
Reference in New Issue
Block a user