Refactor SX templates: shared components, Python migration, cleanup

- Extract shared components (empty-state, delete-btn, sentinel, crud-*,
  view-toggle, img-or-placeholder, avatar, sumup-settings-form, auth
  forms, order tables/detail/checkout)
- Migrate all Python sx_call() callers to use shared components directly
- Remove 55+ thin wrapper defcomps from domain .sx files
- Remove trivial passthrough wrappers (blog-header-label, market-card-text, etc)
- Unify duplicate auth flows (account + federation) into shared/sx/templates/auth.sx
- Unify duplicate order views (cart + orders) into shared/sx/templates/orders.sx
- Disable static file caching in dev (SEND_FILE_MAX_AGE_DEFAULT=0)
- Add SX response validation and debug headers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-01 20:34:34 +00:00
parent 755313bd29
commit c0d369eb8e
58 changed files with 3473 additions and 1210 deletions

113
docs/cssx.md Normal file
View File

@@ -0,0 +1,113 @@
# On-Demand CSS System — Replace Tailwind with Server-Driven Style Delivery
## Context
The app recently moved from Tailwind CDN to a pre-built tw.css (92KB, Tailwind v4), but v4 broke styles since the classes target v3. Currently reverted to CDN for dev. Rather than fixing the v3/v4 mismatch, we replace Tailwind entirely with an on-demand CSS system: the server knows exactly which classes each response uses (because it renders sx→HTML), so it sends only the CSS rules needed — zero unused CSS, zero build step, zero Tailwind dependency.
This mirrors the existing `SX-Components` dedup protocol: client tells server what it has, server sends only what's new.
## Phase 1: Minimal Viable System
### Step 1: CSS Registry — `shared/sx/css_registry.py` (new file)
Parse `tw.css` at startup into a dict mapping HTML class names → CSS rule text.
- **`load_css_registry(path)`** — parse tw.css once at startup, populate module-level `_REGISTRY: dict[str, str]` and `_PREAMBLE: str` (the `@property` / `@layer` declarations that define `--tw-*` vars)
- **`_css_selector_to_class(selector)`** — unescape CSS selectors (`.sm\:hidden``sm:hidden`, `.h-\[60vh\]``h-[60vh]`)
- **`lookup_rules(classes: set[str]) -> str`** — return concatenated CSS for a set of class names, preserving source order from tw.css
- **`get_preamble() -> str`** — return the preamble (sent once per page load)
The parser uses brace-depth tracking to split minified CSS into individual rules, extracts selectors, unescapes to get HTML class names. Rules inside `@media` blocks are stored with their wrapping `@media`.
### Step 2: Class Collection During Render — `shared/sx/html.py`
Add a `contextvars.ContextVar[set[str] | None]` for collecting classes. In `_render_element()` (line ~460-469), when processing a `class` attribute, split the value and add to the collector if active.
```python
# ~3 lines added in _render_element after the attr loop
if attr_name == "class" and attr_val:
collector = _css_class_collector.get(None)
if collector is not None:
collector.update(str(attr_val).split())
```
### Step 3: Sx Source Scanner — `shared/sx/css_registry.py`
For sx pages where the body is rendered *client-side* (sx source sent as text, not pre-rendered HTML), we can't use the render-time collector. Instead, scan sx source text for `:class "..."` patterns:
```python
def scan_classes_from_sx(source: str) -> set[str]:
"""Extract class names from :class "..." in sx source text."""
```
This runs on both component definitions and page sx source at response time.
### Step 4: Wire Into Responses — `shared/sx/helpers.py`
**`sx_page()` (full page loads, line 416):**
1. Scan `component_defs` + `page_sx` for classes via `scan_classes_from_sx()`
2. Look up rules + preamble from registry
3. Replace `<script src="https://cdn.tailwindcss.com...">` in `_SX_PAGE_TEMPLATE` with `<style id="sx-css">{preamble}\n{rules}</style>`
4. Add `<meta name="sx-css-classes" content="{comma-separated classes}">` so client knows what it has
**`sx_response()` (fragment swaps, line 325):**
1. Read `SX-Css` header from request (client's known classes)
2. Scan the sx source for classes
3. Compute new classes = found - known
4. If new rules exist, prepend `<style data-sx-css>{rules}</style>` to response body
5. Set `SX-Css-Add` response header with the new class names (so client can track without parsing CSS)
### Step 5: Client-Side Tracking — `shared/static/scripts/sx.js`
**Boot (page load):**
- Read `<meta name="sx-css-classes">` → populate `_sxCssKnown` dict
**Request header (in `_doFetch`, line ~1516):**
- After sending `SX-Components`, also send `SX-Css: flex,p-2,sm:hidden,...`
**Response handling (line ~1641 for text/sx, ~1698 for HTML):**
- Strip `<style data-sx-css>` blocks from response, inject into `<style id="sx-css">` in `<head>`
- Read `SX-Css-Add` response header, merge class names into `_sxCssKnown`
### Step 6: Jinja Path Fallback — `shared/browser/templates/_types/root/_head.html`
For non-sx pages still using Jinja templates, serve all CSS rules (full registry dump) in `<style id="sx-css">`. Add a Jinja global `sx_css_all()` in `shared/sx/jinja_bridge.py` that returns preamble + all rules. This is the same 92KB but self-hosted with no Tailwind dependency. These pages can be optimized later.
### Step 7: Startup — `shared/sx/jinja_bridge.py`
Call `load_css_registry()` in `setup_sx_bridge()` after loading components.
## Files to Modify
| File | Change |
|------|--------|
| `shared/sx/css_registry.py` | **NEW** — registry, parser, scanner, lookup |
| `shared/sx/html.py` | Add contextvar + 3 lines in `_render_element` |
| `shared/sx/helpers.py` | Modify `_SX_PAGE_TEMPLATE`, `sx_page()`, `sx_response()` |
| `shared/sx/jinja_bridge.py` | Call `load_css_registry()` at startup, add `sx_css_all()` Jinja global |
| `shared/static/scripts/sx.js` | `_sxCssKnown` tracking, `SX-Css` header, response CSS injection |
| `shared/browser/templates/_types/root/_head.html` | Replace Tailwind CDN with `{{ sx_css_all() }}` |
## Key Design Decisions
- **Source of truth:** tw.css (already compiled, known to be correct for current classes)
- **Dedup model:** Mirrors `SX-Components` — client declares what it has, server sends the diff
- **Header size:** ~599 class names × ~10 chars ≈ 6KB header — within limits, can optimize later with hashing
- **CSS ordering:** Registry preserves tw.css source order (later rules win for equal specificity)
- **Preamble:** `--tw-*` custom property defaults (~2KB) always included on first page load
## Verification
1. Start blog service: `./dev.sh blog`
2. Load `https://blog.rose-ash.com/index/` — verify styles match current CDN appearance
3. View page source — should have `<style id="sx-css">` instead of Tailwind CDN script
4. Navigate via sx swap (click a post) — check DevTools Network tab for `SX-Css` request header and `SX-Css-Add` response header
5. Inspect `<style id="sx-css">` — should grow as new pages introduce new classes
6. Check non-sx pages still render correctly (full CSS dump fallback)
## Phase 2 (Future)
- **Component-level pre-computation:** Pre-scan classes per component at registration time
- **Own rule generator:** Replace tw.css parsing with a Python rule engine (no Tailwind dependency at all)
- **Header compression:** Use bitfield or hash instead of full class list
- **Critical CSS:** Only inline above-fold CSS, lazy-load rest

View File

@@ -0,0 +1,366 @@
# Remove Ghost CMS — Native Content Pipeline
## Context
Ghost CMS is the write-primary for all blog content. The blog service mirrors everything to its own DB, proxies editor uploads through Ghost, and relies on Ghost for newsletter dispatch. This creates a hard dependency on an external process that duplicates what we already have locally.
We execute phases in order: implement → commit → push → test in dev → next phase. Ghost can be shut down after Phase 4.
---
## Phase 0 — Initial Sync & Cutover Preparation
One-time migration script to ensure `db_blog` is a complete, verified copy of production Ghost before we cut writes over.
### 0.1 Final sync script — `blog/scripts/final_ghost_sync.py`
Standalone CLI script (not a route) that:
1. Calls Ghost Admin API for ALL posts, pages, authors, tags with `?limit=all&formats=html,plaintext,mobiledoc,lexical&include=authors,tags`
2. Upserts everything into `db_blog` (reuses existing `ghost_sync.py` logic)
3. **Re-renders HTML** from `lexical` via `lexical_renderer.py` for every post — ensures our renderer matches Ghost's output
4. Compares our rendered HTML vs Ghost's HTML, logs diffs (catch rendering gaps before cutover)
5. Prints summary: X posts, Y pages, Z authors, W tags synced
### 0.2 Verification queries
After running the sync script:
- `SELECT count(*) FROM posts WHERE deleted_at IS NULL` matches Ghost post count
- `SELECT count(*) FROM posts WHERE html IS NULL AND status='published'` = 0
- `SELECT count(*) FROM posts WHERE lexical IS NULL AND status='published'` = 0
- Spot-check 5 posts: compare rendered HTML vs Ghost HTML
### 0.3 Cutover sequence (deploy day)
1. Run `final_ghost_sync.py` against production
2. Deploy Phase 1 code
3. Verify post create/edit works without Ghost
4. Disable Ghost webhooks (Phase 1.5 neuters them in code)
5. Ghost continues running for image serving only
### Commit: `Prepare Ghost cutover: final sync script with HTML verification`
---
## Phase 1 — Native Post Writes
Blog service creates/updates posts directly in `db_blog`. Ghost is no longer called for any content write.
### 1.1 DB migration (`blog/alembic/`)
- Make `ghost_id` nullable on `posts`, `authors`, `tags` in `shared/models/ghost_content.py`
- Add `server_default=gen_random_uuid()` on `Post.uuid`
- Add `server_default=func.now()` on `Post.updated_at`
### 1.2 New `PostWriter` — `blog/services/post_writer.py`
Replaces `ghost_posts.py`. Direct DB writes:
```
create_post(sess, title, lexical_json, status, feature_image, ..., user_id) -> Post
create_page(sess, title, lexical_json, ..., user_id) -> Post
update_post(sess, post_id, lexical_json, title, expected_updated_at, ...) -> Post
update_post_settings(sess, post_id, expected_updated_at, **kwargs) -> Post
delete_post(sess, post_id) -> None (soft delete via deleted_at)
```
Key logic:
- Render `html` + `plaintext` from `lexical_json` using existing `lexical_renderer.py`
- Calculate `reading_time` (word count in plaintext / 265)
- Generate slug from title (reuse existing slugify pattern in admin routes)
- Upsert tags by name with generated slugs (`ghost_id=None` for new tags)
- Optimistic lock: compare submitted `updated_at` vs DB value, 409 on mismatch
- Fire AP federation publish on status transitions (extract from `ghost_sync.py:_build_ap_post_data`)
- Invalidate Redis cache after writes
### 1.3 Rewrite post save routes
**`blog/bp/blog/admin/routes.py`** — `new_post_save`, `new_page_save`:
- Replace `ghost_posts.create_post()` + `sync_single_post()``PostWriter.create_post()`
**`blog/bp/post/admin/routes.py`** — `edit_save`, `settings_save`:
- Replace `ghost_posts.update_post()` + `sync_single_post()``PostWriter.update_post()`
- Replace `ghost_posts.update_post_settings()` + `sync_single_post()``PostWriter.update_post_settings()`
### 1.4 Rewrite post edit GET routes
**`blog/bp/post/admin/routes.py`** — `edit`, `settings`:
- Currently call `get_post_for_edit(ghost_id)` which fetches from Ghost Admin API
- Change to read `Post` row from local DB via `DBClient`
- Build a `post_data` dict from the ORM object matching the shape the templates expect
### 1.5 Neuter Ghost webhooks
**`blog/bp/blog/web_hooks/routes.py`**:
- Post/page/author/tag handlers → return 204 (no-op)
- Keep member webhook active (membership sync handled in Phase 3)
### 1.6 Newsletter sending — temporary disable
The editor's "publish + email" flow currently triggers Ghost's email dispatch. For Phase 1, **disable email sending in the UI** (hide publish-mode dropdown). Phase 3 restores it natively.
### Critical files
| File | Action |
|------|--------|
| `shared/models/ghost_content.py` | Make `ghost_id` nullable, add defaults |
| `blog/services/post_writer.py` | **New** — replaces `ghost_posts.py` |
| `blog/bp/blog/ghost/ghost_sync.py` | Extract AP publish logic, rest unused |
| `blog/bp/blog/ghost/ghost_db.py` | Keep — local DB reads, no Ghost coupling |
| `blog/bp/blog/ghost/lexical_renderer.py` | Keep — renders HTML from Lexical JSON |
| `blog/bp/blog/ghost/lexical_validator.py` | Keep — validates Lexical docs |
| `blog/bp/blog/admin/routes.py` | Rewrite create handlers |
| `blog/bp/post/admin/routes.py` | Rewrite edit/settings handlers |
| `blog/bp/blog/web_hooks/routes.py` | Neuter post/page/author/tag handlers |
### Verification
- Create a new post → check it lands in `db_blog` with rendered HTML
- Edit a post → verify `updated_at` optimistic lock works (edit in two tabs, second save gets 409)
- Edit post settings (slug, tags, visibility) → check DB updates
- Publish a post → verify AP federation fires
- Verify Ghost is never called (grep active code paths for `GHOST_ADMIN_API_URL`)
### Commit: `Phase 1: native post writes — blog service owns CRUD, Ghost no longer write-primary`
---
## Phase 2 — Local Image Storage
Replace Ghost upload proxy with local filesystem. Existing Ghost-hosted image URLs continue working.
### 2.1 Docker volume for media
Add `blog_media` volume in `docker-compose.yml` / `docker-compose.dev.yml`, mounted at `/app/media`.
### 2.2 Rewrite upload handlers — `blog/bp/blog/ghost/editor_api.py`
Replace Ghost-proxying handlers with local filesystem writes:
- Validate type + size (existing logic stays)
- Save to `MEDIA_ROOT/YYYY/MM/filename.ext` (content-hash prefix to avoid collisions)
- Return same JSON shape: `{"images": [{"url": "/media/YYYY/MM/filename.ext"}]}`
- `useFileUpload.js` needs zero changes (reads `data[responseKey][0].url`)
### 2.3 Static file route
Add `GET /media/<path:filename>``send_from_directory(MEDIA_ROOT, filename)`
### 2.4 OEmbed replacement
Replace Ghost oembed proxy with direct httpx call to `https://oembed.com/providers.json` endpoint registry + provider-specific URLs.
### 2.5 Legacy Ghost image URLs
Add catch-all `GET /content/images/<path:rest>` that reverse-proxies to Ghost. Keeps all existing post images working. Removed when Phase 2b migration runs.
### 2.6 (later) Bulk image migration script — `blog/scripts/migrate_ghost_images.py`
- Scan all posts for `/content/images/` URLs in `html`, `feature_image`, `lexical`, `og_image`, `twitter_image`
- Download each from Ghost
- Save to `MEDIA_ROOT` with same path structure
- Rewrite URLs in DB rows
- After running: remove the `/content/images/` proxy route
### Verification
- Upload an image in the editor → appears at `/media/...` URL
- Upload media and file → same
- OEmbed: paste a YouTube link in editor → embed renders
- Existing posts with Ghost image URLs still display correctly
- New posts only reference `/media/...` URLs
### Commit: `Phase 2: local image storage — uploads bypass Ghost, legacy URLs proxied`
---
## Phase 3 — Native Newsletter Sending
Account service sends newsletter emails directly via SMTP. Re-enable the publish+email UI.
### 3.1 New model — `NewsletterSend`
In `shared/models/` (lives in `db_account`):
```python
class NewsletterSend(Base):
__tablename__ = "newsletter_sends"
id, post_id (int), newsletter_id (FK), status (str),
recipient_count (int), sent_at (datetime), error_message (text nullable)
```
### 3.2 Newsletter email template
Create `account/templates/_email/newsletter_post.html` and `.txt`. Nick structure from Ghost's email templates (MIT-licensed). Template vars:
- `post_title`, `post_html`, `post_url`, `post_excerpt`, `feature_image`
- `newsletter_name`, `site_name`, `unsubscribe_url`
### 3.3 Send function — `account/services/newsletter_sender.py`
```python
async def send_newsletter_for_post(sess, post_id, newsletter_id, post_data) -> int:
```
- Query `UserNewsletter` where `subscribed=True` for this newsletter, join `User` for emails
- Render template per recipient (unique unsubscribe URL each)
- Send via existing `aiosmtplib` SMTP infrastructure (same config as magic links)
- Insert `NewsletterSend` record
- Return recipient count
### 3.4 Internal action — `account/bp/actions/routes.py`
Add `send-newsletter` action handler. Blog calls `call_action("account", "send-newsletter", payload={...})` after publish.
### 3.5 Unsubscribe route — `account/bp/auth/routes.py`
`GET /unsubscribe/<token>/` — HMAC-signed token of `(user_id, newsletter_id)`. Sets `UserNewsletter.subscribed = False`. Shows confirmation page.
### 3.6 Re-enable publish+email UI in blog editor
Restore the publish-mode dropdown in `blog/bp/post/admin/routes.py` `edit_save`:
- If `publish_mode in ("email", "both")` and `newsletter_id` set:
- Render post HTML from lexical
- `call_action("account", "send-newsletter", {...})`
- Check `NewsletterSend` record to show "already emailed" badge (replaces Ghost's `post.email.status`)
### Verification
- Publish a post with "Web + Email" → newsletter lands in test inbox
- Check `newsletter_sends` table has correct record
- Click unsubscribe link → subscription toggled off
- Re-publishing same post → "already emailed" badge shown, email controls disabled
- Toggle newsletter subscription in account dashboard → still works
### Commit: `Phase 3: native newsletter sending — SMTP via account service, Ghost email fully replaced`
---
## Phase 4 — Decouple Blog Models from Shared & Rename
Blog models currently live in `shared/models/ghost_content.py` — this couples every service to blog's schema. The `SqlBlogService` in `shared/services/blog_impl.py` gives shared code direct DB access to blog tables, bypassing the HTTP boundary that all other cross-domain reads use.
### 4.1 Move blog models out of shared
- **Move** `shared/models/ghost_content.py``blog/models/content.py`
- Models: `Post`, `Author`, `Tag`, `PostTag`, `PostAuthor`, `PostUser`
- **Delete** `blog/models/ghost_content.py` (the re-export shim) — imports now go to `blog/models/content.py`
- **Remove** `ghost_content` line from `shared/models/__init__.py`
- Update all blog-internal imports
### 4.2 Remove `SqlBlogService` from shared
- **Delete** `shared/services/blog_impl.py`
- **Remove** `BlogService` protocol from `shared/contracts/protocols.py`
- **Remove** `services.blog` slot from `shared/services/registry.py`
- Blog's own `blog/services/__init__.py` keeps its local service — not shared
### 4.3 Fix `entry_associations.py` to use HTTP
`shared/services/entry_associations.py` calls `services.blog.get_post_by_id(session)` — direct DB access despite its docstring claiming otherwise. Replace with:
```python
post = await fetch_data("blog", "post-by-id", params={"id": post_id})
```
This aligns with the architecture: cross-domain reads go via HTTP.
### 4.4 Rename ghost directories and files
- `blog/bp/blog/ghost/``blog/bp/blog/legacy/` or inline into `blog/bp/blog/`
- `ghost_sync.py``sync.py` (if still needed)
- `ghost_db.py``queries.py` or merge into services
- `editor_api.py` — stays, not Ghost-specific
- `lexical_renderer.py`, `lexical_validator.py` — stay
### 4.5 Delete Ghost integration code
- `blog/bp/blog/ghost/ghost_sync.py` — delete (AP logic already extracted in Phase 1)
- `blog/bp/blog/ghost/ghost_posts.py` — delete (replaced by PostWriter)
- `blog/bp/blog/ghost/ghost_admin_token.py` — delete
- `shared/infrastructure/ghost_admin_token.py` — delete
- `blog/bp/blog/web_hooks/routes.py` — delete entire blueprint (or gut to empty)
- `account/services/ghost_membership.py` — delete (sync functions, not needed)
### 4.6 Rename membership models
- `shared/models/ghost_membership_entities.py``shared/models/membership.py`
### 4.7 Rename models (with DB migrations)
| Old | New | Table rename |
|-----|-----|-------------|
| `GhostLabel` | `Label` | `ghost_labels``labels` |
| `GhostNewsletter` | `Newsletter` | `ghost_newsletters``newsletters` |
Drop entirely: `GhostTier` (`ghost_tiers`), `GhostSubscription` (`ghost_subscriptions`)
### 4.4 Rename User columns (migration)
- `ghost_status``member_status`
- `ghost_subscribed``email_subscribed`
- `ghost_note``member_note`
- Drop: `ghost_raw` (JSONB blob of Ghost member data)
- Keep `ghost_id` for now (dropped in Phase 5 with all other `ghost_id` columns)
### 4.5 Remove Ghost env vars
From `docker-compose.yml`, `docker-compose.dev.yml`, `.env`:
- `GHOST_API_URL`, `GHOST_ADMIN_API_URL`, `GHOST_PUBLIC_URL`
- `GHOST_CONTENT_API_KEY`, `GHOST_WEBHOOK_SECRET`, `GHOST_ADMIN_API_KEY`
From `shared/infrastructure/factory.py`: remove Ghost config lines (~97-99)
### Verification
- `grep -r "ghost" --include="*.py" . | grep -v alembic | grep -v __pycache__ | grep -v artdag` — should be minimal (only migration files, model `ghost_id` column refs)
- All services start cleanly without Ghost env vars
- Blog CRUD still works end-to-end
- Newsletter sending still works
### Commit: `Phase 4: remove Ghost code — rename models, drop Ghost env vars, clean codebase`
---
## Phase 5 — Author/Tag/Label/Newsletter Admin
Native admin UI for entities that were previously managed in Ghost Admin.
### 5.1 Author admin in blog
- Routes at `/settings/authors/` (follows existing `/settings/tag-groups/` pattern)
- List, create, edit, soft-delete
- Link author to User via email match
### 5.2 Tag admin enhancement
- Add create/edit/delete to existing tag admin
- Tags fully managed in `db_blog`
### 5.3 Newsletter admin in account
- Routes at `/settings/newsletters/`
- List, create, edit, soft-delete newsletters
### 5.4 Label admin in account
- Routes at `/settings/labels/`
- CRUD + assign/unassign labels to users
### 5.5 Drop all `ghost_id` columns
Final migration: drop `ghost_id` from `posts`, `authors`, `tags`, `users`, `labels`, `newsletters`.
Drop `Post.mobiledoc` (Ghost legacy format, always null for new posts).
### Verification
- Create/edit/delete authors, tags, newsletters, labels through admin UI
- Assigned labels show on user's account dashboard
- All CRUD operations persist correctly
### Commit: `Phase 5: native admin for authors, tags, labels, newsletters — drop ghost_id columns`
---
## Phase 6 (future) — SX-Native Posts
- New `body_sx` column on Post
- Block-based SX editor (Notion-style: sequence of typed blocks, not WYSIWYG)
- Each block = one SX form (paragraph, heading, image, quote, code)
- Posts render through SX pipeline instead of Lexical renderer
- Lexical → SX migration script for existing content