mono/docs/ghost-removal-plan.md

# Ghost Removal Plan

**Replace Ghost CMS entirely with native infrastructure.**

---

## What Ghost Currently Provides

Ghost is deeply integrated across three major areas:

### 1. Content Management (blog service)
- Post/page storage in Lexical JSON format
- Author and tag entities with many-to-many relationships
- WYSIWYG editing via Ghost Admin API
- Media uploads (images, audio/video, files) via Ghost's upload endpoints
- OEmbed lookups for embedded media
- Content sync: Ghost → local DB via Content API + webhooks
- ActivityPub publishing triggered after post sync

### 2. Membership & Subscriptions (account service)
- **Members**: Ghost is the member store — users have `ghost_id`, synced bidirectionally
- **Labels**: tagging/segmentation of members (M2M via `GhostLabel` / `UserLabel`)
- **Newsletters**: newsletter entities with per-user subscription tracking (`GhostNewsletter` / `UserNewsletter` with `subscribed` flag)
- **Tiers**: membership levels (free/paid) stored in `GhostTier`
- **Subscriptions**: paid plans with Stripe integration — cadence, price, Stripe customer/subscription IDs stored in `GhostSubscription`
- **Bidirectional sync**: Ghost → DB (`sync_all_membership_from_ghost`, `sync_single_member`) and DB → Ghost (`sync_member_to_ghost`)

### 3. Infrastructure
- JWT token generation for Admin API (`ghost_admin_token.py`)
- Webhook handlers for real-time sync (member, post, page, author, tag events)
- Email campaign sending (newsletter selection on publish, `email_segment` parameter)
- Stripe payment processing for paid subscriptions (handled entirely by Ghost)
- Ghost Docker container (Node.js app alongside our Python stack)

### Environment Variables
```
GHOST_API_URL, GHOST_ADMIN_API_URL, GHOST_PUBLIC_URL
GHOST_CONTENT_API_KEY, GHOST_ADMIN_API_KEY
GHOST_WEBHOOK_SECRET
```

### Ghost-Related Files
```
blog/bp/blog/ghost/ghost_sync.py          # Content fetch & sync
blog/bp/blog/ghost/ghost_posts.py         # Post CRUD via Admin API
blog/bp/blog/ghost/ghost_admin_token.py   # JWT generation
blog/bp/blog/ghost/lexical_validator.py   # Lexical JSON validation
blog/bp/blog/ghost/editor_api.py          # Media upload proxy
blog/bp/blog/ghost_db.py                  # Ghost DB client
blog/bp/blog/web_hooks/routes.py          # Webhook handlers
shared/infrastructure/ghost_admin_token.py # JWT generation (shared copy)
shared/models/ghost_content.py            # Post, Author, Tag, junction tables
shared/models/ghost_membership_entities.py # Label, Newsletter, Tier, Subscription
account/services/ghost_membership.py       # Membership sync service
```

### Ghost-Related Database Tables
```
# Content
posts           (ghost_id, uuid, slug, title, status, lexical, html, ...)
authors         (ghost_id, slug, name, email, ...)
tags            (ghost_id, slug, name, ...)
post_authors    (post_id, author_id, sort_order)
post_tags       (post_id, tag_id, sort_order)

# Membership
ghost_labels        (ghost_id, name, slug)
user_labels         (user_id, label_id)
ghost_newsletters   (ghost_id, name, slug, description)
user_newsletters    (user_id, newsletter_id, subscribed)
ghost_tiers         (ghost_id, name, slug, type, visibility)
ghost_subscriptions (ghost_id, user_id, status, cadence, price_amount,
                     price_currency, stripe_customer_id, stripe_subscription_id,
                     tier_id, raw)

# User model fields
users.ghost_id, users.ghost_status, users.ghost_subscribed,
users.ghost_note, users.ghost_raw, users.stripe_customer_id
```

---

## Problems

- **Two sources of truth** for content AND membership — constant sync overhead
- Every edit round-trips through Ghost's API — we don't own the write path
- Ghost sync is fragile (advisory locks, error recovery, partial sync states)
- Lexical JSON is opaque — we validate but never truly control the format
- Ghost is an entire Node.js application running alongside our Python stack
- Stripe integration is locked inside Ghost — we can't customize payment flows
- Newsletter/email is Ghost-native — no control over templates, scheduling, deliverability
- Membership tiers are Ghost concepts that don't map cleanly to our cooperative model

---

## Target State

Everything Ghost does is handled natively by our services:

| Ghost Feature | Replacement |
|---|---|
| Post/page content | Sexp in `posts.body_sexp` column |
| Lexical editor | WYSIWYG editor saving sexp directly to DB |
| Media uploads | Direct upload to our storage (S3/local) — blog service endpoint |
| Authors | Already in our DB — just drop `ghost_id` column |
| Tags | Already in our DB — just drop `ghost_id` column |
| Members | Already our `users` table — drop Ghost sync, Ghost fields |
| Labels | Rename `ghost_labels` → `labels`, drop `ghost_id` |
| Newsletters | Native newsletter service (see Phase 7 below) |
| Tiers | Native membership tiers on `account` service |
| Subscriptions | Direct Stripe integration on `orders` service (already has SumUp) |
| Email sending | Transactional email service (Postmark/SES/SMTP) |
| Webhooks | Not needed — we own the write path |
| Ghost Docker container | Removed entirely |

---

## Migration Phases

### Phase 1: Lexical → Sexp Converter

Write a one-time conversion script that transforms Lexical JSON into equivalent sexp.

| Lexical Node | S-expression |
|---|---|
| `paragraph` | `(p ...)` |
| `heading` (level 1-6) | `(h1 ...)` ... `(h6 ...)` |
| `text` (plain) | `"string"` |
| `text` (bold) | `(strong "string")` |
| `text` (italic) | `(em "string")` |
| `text` (bold+italic) | `(strong (em "string"))` |
| `text` (code) | `(code "string")` |
| `link` | `(a :href "url" "text")` |
| `list` (bullet) | `(ul (li ...) ...)` |
| `list` (number) | `(ol (li ...) ...)` |
| `quote` | `(blockquote ...)` |
| `image` | `(use "image" :src "url" :alt "text" :caption "text")` |
| `code-block` | `(pre (code :class "language-x" "..."))` |
| `divider` | `(hr)` |
| `embed` | `(use "embed" :url "..." :type "...")` |

Run against all existing posts, verify round-trip fidelity by rendering both versions and comparing HTML output.

### Phase 2: Schema Changes — Content

- Add `body_sexp` text column to `Post` model (or repurpose `lexical` column)
- Keep all existing metadata columns (title, slug, status, published_at, feature_image, etc.)
- Drop `ghost_id` from `Post`, `Author`, `Tag` tables (after full migration)
- Drop `mobiledoc` column (legacy Ghost format, unused)

### Phase 3: Editor Integration

Update the WYSIWYG editor to save sexp instead of Lexical JSON:

- Editor toolbar actions produce sexp nodes
- Save endpoint writes directly to our DB (no Ghost Admin API call)
- Preview renders via the same sexp pipeline used for the public view
- Draft/publish workflow stays the same — just a `status` column update

### Phase 4: Media Uploads

Replace Ghost's upload proxy with native endpoints on the blog service:

- `POST /admin/upload/image/` — accept image, store to S3/local, return URL
- `POST /admin/upload/media/` — audio/video
- `POST /admin/upload/file/` — generic files
- `GET /admin/oembed/?url=...` — OEmbed lookup (call providers directly)

The editor already posts to proxy endpoints in `editor_api.py` — just retarget them to store directly rather than forwarding to Ghost.

### Phase 5: Rendering Pipeline

Update `post_data()` and related functions:

- Parse `body_sexp` through the sexp evaluator
- Render to HTML via the existing `shared/sexp/html.py` pipeline
- Components referenced in post content (`use "image-gallery"`, etc.) resolve from the component registry
- Context variables (author data, related posts, etc.) passed as environment bindings

### Phase 6: Membership Decoupling

Migrate membership from Ghost to native account service:

**Labels → native labels:**
- Rename `ghost_labels` → `labels`, drop `ghost_id` column
- `user_labels` stays as-is
- Admin UI manages labels directly (no Ghost sync)

**Tiers → native membership tiers:**
- Rename `ghost_tiers` → `membership_tiers`, drop `ghost_id`
- Add tier management to account admin UI
- Tier assignment logic moves from Ghost webhook handler to account service

**User model cleanup:**
- Drop: `ghost_id`, `ghost_status`, `ghost_subscribed`, `ghost_note`, `ghost_raw`
- Keep: `stripe_customer_id` (needed for direct Stripe integration)
- Add: `membership_tier_id` FK, `membership_status` enum (free/active/cancelled)

### Phase 7: Newsletter System

Replace Ghost's newsletter infrastructure with a native implementation:

**Newsletter model (replaces `ghost_newsletters`):**
```
newsletters (id, name, slug, description, from_email, reply_to, template_sexp, created_at)
user_newsletters (user_id, newsletter_id, subscribed, subscribed_at, unsubscribed_at)
```

**Email sending:**
- Integrate a transactional email provider (Postmark, AWS SES, or direct SMTP)
- Newsletter templates as sexp — rendered to HTML email via the same pipeline
- Send endpoint on account or blog service: select newsletter, select segment (by label/tier), queue sends
- Unsubscribe handling: tokenized unsubscribe links, one-click List-Unsubscribe header

**Post → email campaign:**
- On publish, optionally select newsletter + segment (replaces Ghost's `?newsletter=slug&email_segment=...`)
- Render post body sexp to email-safe HTML (inline styles, table layout for email clients)
- Queue via background task (Celery or async worker)

**What we gain over Ghost:**
- Email templates are sexp — same format as everything else
- Full control over deliverability (SPF/DKIM/DMARC on our domain)
- Segment by any user attribute, not just Ghost's limited filter syntax
- Send analytics stored in our DB

### Phase 8: Subscription & Payment

Replace Ghost's Stripe integration with direct Stripe on the orders service:

**Current state:** Orders service already handles SumUp payments for marketplace/events. Adding Stripe for recurring subscriptions follows the same pattern.

**Implementation:**
- Stripe Checkout for subscription creation (redirect flow, PCI compliant)
- Stripe Webhooks for subscription lifecycle (created, updated, cancelled, payment_failed)
- `subscriptions` table (replaces `ghost_subscriptions`):
  ```
  subscriptions (id, user_id, tier_id, stripe_subscription_id, stripe_customer_id,
                 status, cadence, price_amount, price_currency,
                 current_period_start, current_period_end, cancelled_at)
  ```
- Customer portal: link to Stripe's hosted portal for card updates/cancellation
- Webhook handler on orders service (same pattern as SumUp webhooks)

**What we gain:**
- Unified payment handling (SumUp for one-off, Stripe for recurring)
- Custom subscription logic (cooperative membership models, sliding scale, etc.)
- Direct access to Stripe customer data without Ghost intermediary

### Phase 9: Remove Ghost

Delete all Ghost integration code:

| File/Directory | Action |
|---|---|
| `blog/bp/blog/ghost/` | Delete entire directory |
| `blog/bp/blog/ghost_db.py` | Delete |
| `blog/bp/blog/web_hooks/` | Delete |
| `shared/infrastructure/ghost_admin_token.py` | Delete |
| `account/services/ghost_membership.py` | Delete |
| Ghost Docker service | Remove from docker-compose |
| Ghost env vars | Remove all `GHOST_*` variables |
| Ghost webhook blueprint registration | Remove from blog routes |
| Startup sync (`sync_all_content_from_ghost`) | Remove from blog init |
| Startup sync (`sync_all_membership_from_ghost`) | Remove from account init |
| Advisory lock `900001` | Remove from blog init |

Rename models:
- `ghost_content.py` → `content.py`
- `ghost_membership_entities.py` → `membership.py`
- Drop all `ghost_id` columns via Alembic migration

### Phase 10: Content-Addressable Caching (ties into sexpr.js)

Once posts are sexp and the JS client runtime exists:

- Hash post body → content address
- Client caches post bodies in localStorage keyed by hash
- Server sends manifest of slug → hash mappings
- Unchanged posts served entirely from client cache
- Only the data envelope (metadata, component params) travels on repeat visits

---

## What Stays the Same

- `Post` model and all its metadata fields (minus ghost-specific ones)
- URL structure (`/slug/`)
- Tag, author, and tag group systems
- Draft/publish workflow
- Admin edit UI (updated to save sexp instead of Lexical)
- RSS feeds (rendered from sexp → HTML)
- Search indexing (extract text content from sexp)
- ActivityPub federation (triggered on publish, same as now)
- Alembic migrations (add/modify/drop columns)
- OAuth2 auth system (already independent of Ghost)

---

## Ordering & Dependencies

```
Phase 1-2 (Content schema)     ──→ Phase 3 (Editor)    ──→ Phase 5 (Rendering)
                                   Phase 4 (Uploads)    ──┘
Phase 6 (Membership)            ──→ Phase 8 (Payments)
Phase 7 (Newsletters)           ── independent, needs email provider choice
Phase 9 (Remove Ghost)          ── after all above complete
Phase 10 (Content-addressed)    ── after sexpr.js runtime exists
```

Phases 1-5 (content) and Phases 6-8 (membership/payments) can proceed in parallel — they touch different services.

---

## Risk Mitigation

- **Data safety**: Run Lexical → sexp converter in dry-run mode first, diff HTML output for every post
- **Rollback**: Keep `lexical` column and Ghost running during transition, feature flag to switch renderers
- **Editor UX**: Editor remains WYSIWYG — authors never see sexp syntax
- **SEO continuity**: URLs don't change, HTML output structurally identical
- **Email deliverability**: Set up SPF/DKIM/DMARC before sending first newsletter from our domain
- **Payment migration**: Run Ghost Stripe and direct Stripe in parallel during transition, migrate active subscriptions via Stripe API (change the subscription's application)
- **Membership data**: One-time migration script to clean User model fields, verified against Ghost export

---

## Dependencies

- Stable sexp parser + evaluator (already built: `shared/sexp/`)
- Component registry with post-relevant components: image, embed, gallery, code-block
- Editor sexp serialization (new work)
- Email provider account (Postmark/SES/SMTP)
- Stripe account with recurring billing enabled (may already exist via Ghost)
- Optional: sexpr.js client runtime for content-addressable caching (see `sexpr-js-runtime-plan.md`)