rose-ash/docs/ghost-removal-execution-plan.md

Remove Ghost CMS — Native Content Pipeline

Context

Ghost CMS is the write-primary for all blog content. The blog service mirrors everything to its own DB, proxies editor uploads through Ghost, and relies on Ghost for newsletter dispatch. This creates a hard dependency on an external process that duplicates what we already have locally.

We execute phases in order: implement → commit → push → test in dev → next phase. Ghost can be shut down after Phase 4.

---

Phase 0 — Initial Sync & Cutover Preparation

One-time migration script to ensure db_blog is a complete, verified copy of production Ghost before we cut writes over.

0.1 Final sync script — blog/scripts/final_ghost_sync.py

Standalone CLI script (not a route) that:

1. Calls Ghost Admin API for ALL posts, pages, authors, tags with ?limit=all&formats=html,plaintext,mobiledoc,lexical&include=authors,tags
2. Upserts everything into db_blog (reuses existing ghost_sync.py logic)
3. Re-renders HTML from lexical via lexical_renderer.py for every post — ensures our renderer matches Ghost's output
4. Compares our rendered HTML vs Ghost's HTML, logs diffs (catch rendering gaps before cutover)
5. Prints summary: X posts, Y pages, Z authors, W tags synced

0.2 Verification queries

After running the sync script:

• SELECT count(*) FROM posts WHERE deleted_at IS NULL matches Ghost post count
• SELECT count(*) FROM posts WHERE html IS NULL AND status='published' = 0
• SELECT count(*) FROM posts WHERE lexical IS NULL AND status='published' = 0
• Spot-check 5 posts: compare rendered HTML vs Ghost HTML

0.3 Cutover sequence (deploy day)

1. Run final_ghost_sync.py against production
2. Deploy Phase 1 code
3. Verify post create/edit works without Ghost
4. Disable Ghost webhooks (Phase 1.5 neuters them in code)
5. Ghost continues running for image serving only

Commit: Prepare Ghost cutover: final sync script with HTML verification

---

Phase 1 — Native Post Writes

Blog service creates/updates posts directly in db_blog. Ghost is no longer called for any content write.

1.1 DB migration (blog/alembic/)

• Make ghost_id nullable on posts, authors, tags in shared/models/ghost_content.py
• Add server_default=gen_random_uuid() on Post.uuid
• Add server_default=func.now() on Post.updated_at

1.2 New PostWriter — blog/services/post_writer.py

Replaces ghost_posts.py. Direct DB writes:

create_post(sess, title, lexical_json, status, feature_image, ..., user_id) -> Post
create_page(sess, title, lexical_json, ..., user_id) -> Post
update_post(sess, post_id, lexical_json, title, expected_updated_at, ...) -> Post
update_post_settings(sess, post_id, expected_updated_at, **kwargs) -> Post
delete_post(sess, post_id) -> None  (soft delete via deleted_at)

Key logic:

• Render html + plaintext from lexical_json using existing lexical_renderer.py
• Calculate reading_time (word count in plaintext / 265)
• Generate slug from title (reuse existing slugify pattern in admin routes)
• Upsert tags by name with generated slugs (ghost_id=None for new tags)
• Optimistic lock: compare submitted updated_at vs DB value, 409 on mismatch
• Fire AP federation publish on status transitions (extract from ghost_sync.py:_build_ap_post_data)
• Invalidate Redis cache after writes

1.3 Rewrite post save routes

blog/bp/blog/admin/routes.py — new_post_save, new_page_save:

• Replace ghost_posts.create_post() + sync_single_post() → PostWriter.create_post()

blog/bp/post/admin/routes.py — edit_save, settings_save:

• Replace ghost_posts.update_post() + sync_single_post() → PostWriter.update_post()
• Replace ghost_posts.update_post_settings() + sync_single_post() → PostWriter.update_post_settings()

1.4 Rewrite post edit GET routes

blog/bp/post/admin/routes.py — edit, settings:

• Currently call get_post_for_edit(ghost_id) which fetches from Ghost Admin API
• Change to read Post row from local DB via DBClient
• Build a post_data dict from the ORM object matching the shape the templates expect

1.5 Neuter Ghost webhooks

blog/bp/blog/web_hooks/routes.py:

• Post/page/author/tag handlers → return 204 (no-op)
• Keep member webhook active (membership sync handled in Phase 3)

1.6 Newsletter sending — temporary disable

The editor's "publish + email" flow currently triggers Ghost's email dispatch. For Phase 1, disable email sending in the UI (hide publish-mode dropdown). Phase 3 restores it natively.

Critical files

File	Action
shared/models/ghost_content.py	Make ghost_id nullable, add defaults
blog/services/post_writer.py	New — replaces ghost_posts.py
blog/bp/blog/ghost/ghost_sync.py	Extract AP publish logic, rest unused
blog/bp/blog/ghost/ghost_db.py	Keep — local DB reads, no Ghost coupling
blog/bp/blog/ghost/lexical_renderer.py	Keep — renders HTML from Lexical JSON
blog/bp/blog/ghost/lexical_validator.py	Keep — validates Lexical docs
blog/bp/blog/admin/routes.py	Rewrite create handlers
blog/bp/post/admin/routes.py	Rewrite edit/settings handlers
blog/bp/blog/web_hooks/routes.py	Neuter post/page/author/tag handlers

Verification

• Create a new post → check it lands in db_blog with rendered HTML
• Edit a post → verify updated_at optimistic lock works (edit in two tabs, second save gets 409)
• Edit post settings (slug, tags, visibility) → check DB updates
• Publish a post → verify AP federation fires
• Verify Ghost is never called (grep active code paths for GHOST_ADMIN_API_URL)

Commit: Phase 1: native post writes — blog service owns CRUD, Ghost no longer write-primary

---

Phase 2 — Local Image Storage

Replace Ghost upload proxy with local filesystem. Existing Ghost-hosted image URLs continue working.

2.1 Docker volume for media

Add blog_media volume in docker-compose.yml / docker-compose.dev.yml, mounted at /app/media.

2.2 Rewrite upload handlers — blog/bp/blog/ghost/editor_api.py

Replace Ghost-proxying handlers with local filesystem writes:

• Validate type + size (existing logic stays)
• Save to MEDIA_ROOT/YYYY/MM/filename.ext (content-hash prefix to avoid collisions)
• Return same JSON shape: {"images": [{"url": "/media/YYYY/MM/filename.ext"}]}
• useFileUpload.js needs zero changes (reads data[responseKey][0].url)

2.3 Static file route

Add GET /media/<path:filename> → send_from_directory(MEDIA_ROOT, filename)

2.4 OEmbed replacement

Replace Ghost oembed proxy with direct httpx call to https://oembed.com/providers.json endpoint registry + provider-specific URLs.

2.5 Legacy Ghost image URLs

Add catch-all GET /content/images/<path:rest> that reverse-proxies to Ghost. Keeps all existing post images working. Removed when Phase 2b migration runs.

2.6 (later) Bulk image migration script — blog/scripts/migrate_ghost_images.py

• Scan all posts for /content/images/ URLs in html, feature_image, lexical, og_image, twitter_image
• Download each from Ghost
• Save to MEDIA_ROOT with same path structure
• Rewrite URLs in DB rows
• After running: remove the /content/images/ proxy route

Verification

• Upload an image in the editor → appears at /media/... URL
• Upload media and file → same
• OEmbed: paste a YouTube link in editor → embed renders
• Existing posts with Ghost image URLs still display correctly
• New posts only reference /media/... URLs

Commit: Phase 2: local image storage — uploads bypass Ghost, legacy URLs proxied

---

Phase 3 — Native Newsletter Sending

Account service sends newsletter emails directly via SMTP. Re-enable the publish+email UI.

3.1 New model — NewsletterSend

In shared/models/ (lives in db_account):

class NewsletterSend(Base):
    __tablename__ = "newsletter_sends"
    id, post_id (int), newsletter_id (FK), status (str),
    recipient_count (int), sent_at (datetime), error_message (text nullable)

3.2 Newsletter email template

Create account/templates/_email/newsletter_post.html and .txt. Nick structure from Ghost's email templates (MIT-licensed). Template vars:

• post_title, post_html, post_url, post_excerpt, feature_image
• newsletter_name, site_name, unsubscribe_url

3.3 Send function — account/services/newsletter_sender.py

async def send_newsletter_for_post(sess, post_id, newsletter_id, post_data) -> int:

• Query UserNewsletter where subscribed=True for this newsletter, join User for emails
• Render template per recipient (unique unsubscribe URL each)
• Send via existing aiosmtplib SMTP infrastructure (same config as magic links)
• Insert NewsletterSend record
• Return recipient count

3.4 Internal action — account/bp/actions/routes.py

Add send-newsletter action handler. Blog calls call_action("account", "send-newsletter", payload={...}) after publish.

3.5 Unsubscribe route — account/bp/auth/routes.py

GET /unsubscribe/<token>/ — HMAC-signed token of (user_id, newsletter_id). Sets UserNewsletter.subscribed = False. Shows confirmation page.

3.6 Re-enable publish+email UI in blog editor

Restore the publish-mode dropdown in blog/bp/post/admin/routes.py edit_save:

• If publish_mode in ("email", "both") and newsletter_id set:
  • Render post HTML from lexical
  • call_action("account", "send-newsletter", {...})
• Check NewsletterSend record to show "already emailed" badge (replaces Ghost's post.email.status)

Verification

• Publish a post with "Web + Email" → newsletter lands in test inbox
• Check newsletter_sends table has correct record
• Click unsubscribe link → subscription toggled off
• Re-publishing same post → "already emailed" badge shown, email controls disabled
• Toggle newsletter subscription in account dashboard → still works

Commit: Phase 3: native newsletter sending — SMTP via account service, Ghost email fully replaced

---

Phase 4 — Decouple Blog Models from Shared & Rename

Blog models currently live in shared/models/ghost_content.py — this couples every service to blog's schema. The SqlBlogService in shared/services/blog_impl.py gives shared code direct DB access to blog tables, bypassing the HTTP boundary that all other cross-domain reads use.

4.1 Move blog models out of shared

• Move shared/models/ghost_content.py → blog/models/content.py
  • Models: Post, Author, Tag, PostTag, PostAuthor, PostUser
• Delete blog/models/ghost_content.py (the re-export shim) — imports now go to blog/models/content.py
• Remove ghost_content line from shared/models/__init__.py
• Update all blog-internal imports

4.2 Remove SqlBlogService from shared

• Delete shared/services/blog_impl.py
• Remove BlogService protocol from shared/contracts/protocols.py
• Remove services.blog slot from shared/services/registry.py
• Blog's own blog/services/__init__.py keeps its local service — not shared

4.3 Fix entry_associations.py to use HTTP

shared/services/entry_associations.py calls services.blog.get_post_by_id(session) — direct DB access despite its docstring claiming otherwise. Replace with:

post = await fetch_data("blog", "post-by-id", params={"id": post_id})

This aligns with the architecture: cross-domain reads go via HTTP.

4.4 Rename ghost directories and files

• blog/bp/blog/ghost/ → blog/bp/blog/legacy/ or inline into blog/bp/blog/
  • ghost_sync.py → sync.py (if still needed)
  • ghost_db.py → queries.py or merge into services
  • editor_api.py — stays, not Ghost-specific
  • lexical_renderer.py, lexical_validator.py — stay

4.5 Delete Ghost integration code

• blog/bp/blog/ghost/ghost_sync.py — delete (AP logic already extracted in Phase 1)
• blog/bp/blog/ghost/ghost_posts.py — delete (replaced by PostWriter)
• blog/bp/blog/ghost/ghost_admin_token.py — delete
• shared/infrastructure/ghost_admin_token.py — delete
• blog/bp/blog/web_hooks/routes.py — delete entire blueprint (or gut to empty)
• account/services/ghost_membership.py — delete (sync functions, not needed)

4.6 Rename membership models

• shared/models/ghost_membership_entities.py → shared/models/membership.py

4.7 Rename models (with DB migrations)

Old	New	Table rename
GhostLabel	Label	ghost_labels → labels
GhostNewsletter	Newsletter	ghost_newsletters → newsletters

Drop entirely: GhostTier (ghost_tiers), GhostSubscription (ghost_subscriptions)

4.4 Rename User columns (migration)

• ghost_status → member_status
• ghost_subscribed → email_subscribed
• ghost_note → member_note
• Drop: ghost_raw (JSONB blob of Ghost member data)
• Keep ghost_id for now (dropped in Phase 5 with all other ghost_id columns)

4.5 Remove Ghost env vars

From docker-compose.yml, docker-compose.dev.yml, .env:

• GHOST_API_URL, GHOST_ADMIN_API_URL, GHOST_PUBLIC_URL
• GHOST_CONTENT_API_KEY, GHOST_WEBHOOK_SECRET, GHOST_ADMIN_API_KEY

From shared/infrastructure/factory.py: remove Ghost config lines (~97-99)

Verification

• grep -r "ghost" --include="*.py" . | grep -v alembic | grep -v __pycache__ | grep -v artdag — should be minimal (only migration files, model ghost_id column refs)
• All services start cleanly without Ghost env vars
• Blog CRUD still works end-to-end
• Newsletter sending still works

Commit: Phase 4: remove Ghost code — rename models, drop Ghost env vars, clean codebase

---

Phase 5 — Author/Tag/Label/Newsletter Admin

Native admin UI for entities that were previously managed in Ghost Admin.

5.1 Author admin in blog

• Routes at /settings/authors/ (follows existing /settings/tag-groups/ pattern)
• List, create, edit, soft-delete
• Link author to User via email match

5.2 Tag admin enhancement

• Add create/edit/delete to existing tag admin
• Tags fully managed in db_blog

5.3 Newsletter admin in account

• Routes at /settings/newsletters/
• List, create, edit, soft-delete newsletters

5.4 Label admin in account

• Routes at /settings/labels/
• CRUD + assign/unassign labels to users

5.5 Drop all ghost_id columns

Final migration: drop ghost_id from posts, authors, tags, users, labels, newsletters. Drop Post.mobiledoc (Ghost legacy format, always null for new posts).

Verification

• Create/edit/delete authors, tags, newsletters, labels through admin UI
• Assigned labels show on user's account dashboard
• All CRUD operations persist correctly

Commit: Phase 5: native admin for authors, tags, labels, newsletters — drop ghost_id columns

---

Phase 6 (future) — SX-Native Posts

• New body_sx column on Post
• Block-based SX editor (Notion-style: sequence of typed blocks, not WYSIWYG)
• Each block = one SX form (paragraph, heading, image, quote, code)
• Posts render through SX pipeline instead of Lexical renderer
• Lexical → SX migration script for existing content