mono/docs/ghost-removal-execution-plan.md

# Remove Ghost CMS — Native Content Pipeline

## Context

Ghost CMS is the write-primary for all blog content. The blog service mirrors everything to its own DB, proxies editor uploads through Ghost, and relies on Ghost for newsletter dispatch. This creates a hard dependency on an external process that duplicates what we already have locally.

We execute phases in order: implement → commit → push → test in dev → next phase. Ghost can be shut down after Phase 4.

---

## Phase 0 — Initial Sync & Cutover Preparation

One-time migration script to ensure `db_blog` is a complete, verified copy of production Ghost before we cut writes over.

### 0.1 Final sync script — `blog/scripts/final_ghost_sync.py`

Standalone CLI script (not a route) that:
1. Calls Ghost Admin API for ALL posts, pages, authors, tags with `?limit=all&formats=html,plaintext,mobiledoc,lexical&include=authors,tags`
2. Upserts everything into `db_blog` (reuses existing `ghost_sync.py` logic)
3. **Re-renders HTML** from `lexical` via `lexical_renderer.py` for every post — ensures our renderer matches Ghost's output
4. Compares our rendered HTML vs Ghost's HTML, logs diffs (catch rendering gaps before cutover)
5. Prints summary: X posts, Y pages, Z authors, W tags synced

### 0.2 Verification queries

After running the sync script:
- `SELECT count(*) FROM posts WHERE deleted_at IS NULL` matches Ghost post count
- `SELECT count(*) FROM posts WHERE html IS NULL AND status='published'` = 0
- `SELECT count(*) FROM posts WHERE lexical IS NULL AND status='published'` = 0
- Spot-check 5 posts: compare rendered HTML vs Ghost HTML

### 0.3 Cutover sequence (deploy day)

1. Run `final_ghost_sync.py` against production
2. Deploy Phase 1 code
3. Verify post create/edit works without Ghost
4. Disable Ghost webhooks (Phase 1.5 neuters them in code)
5. Ghost continues running for image serving only

### Commit: `Prepare Ghost cutover: final sync script with HTML verification`

---

## Phase 1 — Native Post Writes

Blog service creates/updates posts directly in `db_blog`. Ghost is no longer called for any content write.

### 1.1 DB migration (`blog/alembic/`)

- Make `ghost_id` nullable on `posts`, `authors`, `tags` in `shared/models/ghost_content.py`
- Add `server_default=gen_random_uuid()` on `Post.uuid`
- Add `server_default=func.now()` on `Post.updated_at`

### 1.2 New `PostWriter` — `blog/services/post_writer.py`

Replaces `ghost_posts.py`. Direct DB writes:

```
create_post(sess, title, lexical_json, status, feature_image, ..., user_id) -> Post
create_page(sess, title, lexical_json, ..., user_id) -> Post
update_post(sess, post_id, lexical_json, title, expected_updated_at, ...) -> Post
update_post_settings(sess, post_id, expected_updated_at, **kwargs) -> Post
delete_post(sess, post_id) -> None  (soft delete via deleted_at)
```

Key logic:
- Render `html` + `plaintext` from `lexical_json` using existing `lexical_renderer.py`
- Calculate `reading_time` (word count in plaintext / 265)
- Generate slug from title (reuse existing slugify pattern in admin routes)
- Upsert tags by name with generated slugs (`ghost_id=None` for new tags)
- Optimistic lock: compare submitted `updated_at` vs DB value, 409 on mismatch
- Fire AP federation publish on status transitions (extract from `ghost_sync.py:_build_ap_post_data`)
- Invalidate Redis cache after writes

### 1.3 Rewrite post save routes

**`blog/bp/blog/admin/routes.py`** — `new_post_save`, `new_page_save`:
- Replace `ghost_posts.create_post()` + `sync_single_post()` → `PostWriter.create_post()`

**`blog/bp/post/admin/routes.py`** — `edit_save`, `settings_save`:
- Replace `ghost_posts.update_post()` + `sync_single_post()` → `PostWriter.update_post()`
- Replace `ghost_posts.update_post_settings()` + `sync_single_post()` → `PostWriter.update_post_settings()`

### 1.4 Rewrite post edit GET routes

**`blog/bp/post/admin/routes.py`** — `edit`, `settings`:
- Currently call `get_post_for_edit(ghost_id)` which fetches from Ghost Admin API
- Change to read `Post` row from local DB via `DBClient`
- Build a `post_data` dict from the ORM object matching the shape the templates expect

### 1.5 Neuter Ghost webhooks

**`blog/bp/blog/web_hooks/routes.py`**:
- Post/page/author/tag handlers → return 204 (no-op)
- Keep member webhook active (membership sync handled in Phase 3)

### 1.6 Newsletter sending — temporary disable

The editor's "publish + email" flow currently triggers Ghost's email dispatch. For Phase 1, **disable email sending in the UI** (hide publish-mode dropdown). Phase 3 restores it natively.

### Critical files

| File | Action |
|------|--------|
| `shared/models/ghost_content.py` | Make `ghost_id` nullable, add defaults |
| `blog/services/post_writer.py` | **New** — replaces `ghost_posts.py` |
| `blog/bp/blog/ghost/ghost_sync.py` | Extract AP publish logic, rest unused |
| `blog/bp/blog/ghost/ghost_db.py` | Keep — local DB reads, no Ghost coupling |
| `blog/bp/blog/ghost/lexical_renderer.py` | Keep — renders HTML from Lexical JSON |
| `blog/bp/blog/ghost/lexical_validator.py` | Keep — validates Lexical docs |
| `blog/bp/blog/admin/routes.py` | Rewrite create handlers |
| `blog/bp/post/admin/routes.py` | Rewrite edit/settings handlers |
| `blog/bp/blog/web_hooks/routes.py` | Neuter post/page/author/tag handlers |

### Verification

- Create a new post → check it lands in `db_blog` with rendered HTML
- Edit a post → verify `updated_at` optimistic lock works (edit in two tabs, second save gets 409)
- Edit post settings (slug, tags, visibility) → check DB updates
- Publish a post → verify AP federation fires
- Verify Ghost is never called (grep active code paths for `GHOST_ADMIN_API_URL`)

### Commit: `Phase 1: native post writes — blog service owns CRUD, Ghost no longer write-primary`

---

## Phase 2 — Local Image Storage

Replace Ghost upload proxy with local filesystem. Existing Ghost-hosted image URLs continue working.

### 2.1 Docker volume for media

Add `blog_media` volume in `docker-compose.yml` / `docker-compose.dev.yml`, mounted at `/app/media`.

### 2.2 Rewrite upload handlers — `blog/bp/blog/ghost/editor_api.py`

Replace Ghost-proxying handlers with local filesystem writes:
- Validate type + size (existing logic stays)
- Save to `MEDIA_ROOT/YYYY/MM/filename.ext` (content-hash prefix to avoid collisions)
- Return same JSON shape: `{"images": [{"url": "/media/YYYY/MM/filename.ext"}]}`
- `useFileUpload.js` needs zero changes (reads `data[responseKey][0].url`)

### 2.3 Static file route

Add `GET /media/<path:filename>` → `send_from_directory(MEDIA_ROOT, filename)`

### 2.4 OEmbed replacement

Replace Ghost oembed proxy with direct httpx call to `https://oembed.com/providers.json` endpoint registry + provider-specific URLs.

### 2.5 Legacy Ghost image URLs

Add catch-all `GET /content/images/<path:rest>` that reverse-proxies to Ghost. Keeps all existing post images working. Removed when Phase 2b migration runs.

### 2.6 (later) Bulk image migration script — `blog/scripts/migrate_ghost_images.py`

- Scan all posts for `/content/images/` URLs in `html`, `feature_image`, `lexical`, `og_image`, `twitter_image`
- Download each from Ghost
- Save to `MEDIA_ROOT` with same path structure
- Rewrite URLs in DB rows
- After running: remove the `/content/images/` proxy route

### Verification

- Upload an image in the editor → appears at `/media/...` URL
- Upload media and file → same
- OEmbed: paste a YouTube link in editor → embed renders
- Existing posts with Ghost image URLs still display correctly
- New posts only reference `/media/...` URLs

### Commit: `Phase 2: local image storage — uploads bypass Ghost, legacy URLs proxied`

---

## Phase 3 — Native Newsletter Sending

Account service sends newsletter emails directly via SMTP. Re-enable the publish+email UI.

### 3.1 New model — `NewsletterSend`

In `shared/models/` (lives in `db_account`):
```python
class NewsletterSend(Base):
    __tablename__ = "newsletter_sends"
    id, post_id (int), newsletter_id (FK), status (str),
    recipient_count (int), sent_at (datetime), error_message (text nullable)
```

### 3.2 Newsletter email template

Create `account/templates/_email/newsletter_post.html` and `.txt`. Nick structure from Ghost's email templates (MIT-licensed). Template vars:
- `post_title`, `post_html`, `post_url`, `post_excerpt`, `feature_image`
- `newsletter_name`, `site_name`, `unsubscribe_url`

### 3.3 Send function — `account/services/newsletter_sender.py`

```python
async def send_newsletter_for_post(sess, post_id, newsletter_id, post_data) -> int:
```
- Query `UserNewsletter` where `subscribed=True` for this newsletter, join `User` for emails
- Render template per recipient (unique unsubscribe URL each)
- Send via existing `aiosmtplib` SMTP infrastructure (same config as magic links)
- Insert `NewsletterSend` record
- Return recipient count

### 3.4 Internal action — `account/bp/actions/routes.py`

Add `send-newsletter` action handler. Blog calls `call_action("account", "send-newsletter", payload={...})` after publish.

### 3.5 Unsubscribe route — `account/bp/auth/routes.py`

`GET /unsubscribe/<token>/` — HMAC-signed token of `(user_id, newsletter_id)`. Sets `UserNewsletter.subscribed = False`. Shows confirmation page.

### 3.6 Re-enable publish+email UI in blog editor

Restore the publish-mode dropdown in `blog/bp/post/admin/routes.py` `edit_save`:
- If `publish_mode in ("email", "both")` and `newsletter_id` set:
  - Render post HTML from lexical
  - `call_action("account", "send-newsletter", {...})`
- Check `NewsletterSend` record to show "already emailed" badge (replaces Ghost's `post.email.status`)

### Verification

- Publish a post with "Web + Email" → newsletter lands in test inbox
- Check `newsletter_sends` table has correct record
- Click unsubscribe link → subscription toggled off
- Re-publishing same post → "already emailed" badge shown, email controls disabled
- Toggle newsletter subscription in account dashboard → still works

### Commit: `Phase 3: native newsletter sending — SMTP via account service, Ghost email fully replaced`

---

## Phase 4 — Decouple Blog Models from Shared & Rename

Blog models currently live in `shared/models/ghost_content.py` — this couples every service to blog's schema. The `SqlBlogService` in `shared/services/blog_impl.py` gives shared code direct DB access to blog tables, bypassing the HTTP boundary that all other cross-domain reads use.

### 4.1 Move blog models out of shared

- **Move** `shared/models/ghost_content.py` → `blog/models/content.py`
  - Models: `Post`, `Author`, `Tag`, `PostTag`, `PostAuthor`, `PostUser`
- **Delete** `blog/models/ghost_content.py` (the re-export shim) — imports now go to `blog/models/content.py`
- **Remove** `ghost_content` line from `shared/models/__init__.py`
- Update all blog-internal imports

### 4.2 Remove `SqlBlogService` from shared

- **Delete** `shared/services/blog_impl.py`
- **Remove** `BlogService` protocol from `shared/contracts/protocols.py`
- **Remove** `services.blog` slot from `shared/services/registry.py`
- Blog's own `blog/services/__init__.py` keeps its local service — not shared

### 4.3 Fix `entry_associations.py` to use HTTP

`shared/services/entry_associations.py` calls `services.blog.get_post_by_id(session)` — direct DB access despite its docstring claiming otherwise. Replace with:

```python
post = await fetch_data("blog", "post-by-id", params={"id": post_id})
```

This aligns with the architecture: cross-domain reads go via HTTP.

### 4.4 Rename ghost directories and files

- `blog/bp/blog/ghost/` → `blog/bp/blog/legacy/` or inline into `blog/bp/blog/`
  - `ghost_sync.py` → `sync.py` (if still needed)
  - `ghost_db.py` → `queries.py` or merge into services
  - `editor_api.py` — stays, not Ghost-specific
  - `lexical_renderer.py`, `lexical_validator.py` — stay

### 4.5 Delete Ghost integration code

- `blog/bp/blog/ghost/ghost_sync.py` — delete (AP logic already extracted in Phase 1)
- `blog/bp/blog/ghost/ghost_posts.py` — delete (replaced by PostWriter)
- `blog/bp/blog/ghost/ghost_admin_token.py` — delete
- `shared/infrastructure/ghost_admin_token.py` — delete
- `blog/bp/blog/web_hooks/routes.py` — delete entire blueprint (or gut to empty)
- `account/services/ghost_membership.py` — delete (sync functions, not needed)

### 4.6 Rename membership models

- `shared/models/ghost_membership_entities.py` → `shared/models/membership.py`

### 4.7 Rename models (with DB migrations)

| Old | New | Table rename |
|-----|-----|-------------|
| `GhostLabel` | `Label` | `ghost_labels` → `labels` |
| `GhostNewsletter` | `Newsletter` | `ghost_newsletters` → `newsletters` |

Drop entirely: `GhostTier` (`ghost_tiers`), `GhostSubscription` (`ghost_subscriptions`)

### 4.4 Rename User columns (migration)

- `ghost_status` → `member_status`
- `ghost_subscribed` → `email_subscribed`
- `ghost_note` → `member_note`
- Drop: `ghost_raw` (JSONB blob of Ghost member data)
- Keep `ghost_id` for now (dropped in Phase 5 with all other `ghost_id` columns)

### 4.5 Remove Ghost env vars

From `docker-compose.yml`, `docker-compose.dev.yml`, `.env`:
- `GHOST_API_URL`, `GHOST_ADMIN_API_URL`, `GHOST_PUBLIC_URL`
- `GHOST_CONTENT_API_KEY`, `GHOST_WEBHOOK_SECRET`, `GHOST_ADMIN_API_KEY`

From `shared/infrastructure/factory.py`: remove Ghost config lines (~97-99)

### Verification

- `grep -r "ghost" --include="*.py" . | grep -v alembic | grep -v __pycache__ | grep -v artdag` — should be minimal (only migration files, model `ghost_id` column refs)
- All services start cleanly without Ghost env vars
- Blog CRUD still works end-to-end
- Newsletter sending still works

### Commit: `Phase 4: remove Ghost code — rename models, drop Ghost env vars, clean codebase`

---

## Phase 5 — Author/Tag/Label/Newsletter Admin

Native admin UI for entities that were previously managed in Ghost Admin.

### 5.1 Author admin in blog

- Routes at `/settings/authors/` (follows existing `/settings/tag-groups/` pattern)
- List, create, edit, soft-delete
- Link author to User via email match

### 5.2 Tag admin enhancement

- Add create/edit/delete to existing tag admin
- Tags fully managed in `db_blog`

### 5.3 Newsletter admin in account

- Routes at `/settings/newsletters/`
- List, create, edit, soft-delete newsletters

### 5.4 Label admin in account

- Routes at `/settings/labels/`
- CRUD + assign/unassign labels to users

### 5.5 Drop all `ghost_id` columns

Final migration: drop `ghost_id` from `posts`, `authors`, `tags`, `users`, `labels`, `newsletters`.
Drop `Post.mobiledoc` (Ghost legacy format, always null for new posts).

### Verification

- Create/edit/delete authors, tags, newsletters, labels through admin UI
- Assigned labels show on user's account dashboard
- All CRUD operations persist correctly

### Commit: `Phase 5: native admin for authors, tags, labels, newsletters — drop ghost_id columns`

---

## Phase 6 (future) — SX-Native Posts

- New `body_sx` column on Post
- Block-based SX editor (Notion-style: sequence of typed blocks, not WYSIWYG)
- Each block = one SX form (paragraph, heading, image, quote, code)
- Posts render through SX pipeline instead of Lexical renderer
- Lexical → SX migration script for existing content