diff --git a/docs/ghost-removal-plan.md b/docs/ghost-removal-plan.md new file mode 100644 index 0000000..015c752 --- /dev/null +++ b/docs/ghost-removal-plan.md @@ -0,0 +1,333 @@ +# Ghost Removal Plan + +**Replace Ghost CMS entirely with native infrastructure.** + +--- + +## What Ghost Currently Provides + +Ghost is deeply integrated across three major areas: + +### 1. Content Management (blog service) +- Post/page storage in Lexical JSON format +- Author and tag entities with many-to-many relationships +- WYSIWYG editing via Ghost Admin API +- Media uploads (images, audio/video, files) via Ghost's upload endpoints +- OEmbed lookups for embedded media +- Content sync: Ghost → local DB via Content API + webhooks +- ActivityPub publishing triggered after post sync + +### 2. Membership & Subscriptions (account service) +- **Members**: Ghost is the member store — users have `ghost_id`, synced bidirectionally +- **Labels**: tagging/segmentation of members (M2M via `GhostLabel` / `UserLabel`) +- **Newsletters**: newsletter entities with per-user subscription tracking (`GhostNewsletter` / `UserNewsletter` with `subscribed` flag) +- **Tiers**: membership levels (free/paid) stored in `GhostTier` +- **Subscriptions**: paid plans with Stripe integration — cadence, price, Stripe customer/subscription IDs stored in `GhostSubscription` +- **Bidirectional sync**: Ghost → DB (`sync_all_membership_from_ghost`, `sync_single_member`) and DB → Ghost (`sync_member_to_ghost`) + +### 3. Infrastructure +- JWT token generation for Admin API (`ghost_admin_token.py`) +- Webhook handlers for real-time sync (member, post, page, author, tag events) +- Email campaign sending (newsletter selection on publish, `email_segment` parameter) +- Stripe payment processing for paid subscriptions (handled entirely by Ghost) +- Ghost Docker container (Node.js app alongside our Python stack) + +### Environment Variables +``` +GHOST_API_URL, GHOST_ADMIN_API_URL, GHOST_PUBLIC_URL +GHOST_CONTENT_API_KEY, GHOST_ADMIN_API_KEY +GHOST_WEBHOOK_SECRET +``` + +### Ghost-Related Files +``` +blog/bp/blog/ghost/ghost_sync.py # Content fetch & sync +blog/bp/blog/ghost/ghost_posts.py # Post CRUD via Admin API +blog/bp/blog/ghost/ghost_admin_token.py # JWT generation +blog/bp/blog/ghost/lexical_validator.py # Lexical JSON validation +blog/bp/blog/ghost/editor_api.py # Media upload proxy +blog/bp/blog/ghost_db.py # Ghost DB client +blog/bp/blog/web_hooks/routes.py # Webhook handlers +shared/infrastructure/ghost_admin_token.py # JWT generation (shared copy) +shared/models/ghost_content.py # Post, Author, Tag, junction tables +shared/models/ghost_membership_entities.py # Label, Newsletter, Tier, Subscription +account/services/ghost_membership.py # Membership sync service +``` + +### Ghost-Related Database Tables +``` +# Content +posts (ghost_id, uuid, slug, title, status, lexical, html, ...) +authors (ghost_id, slug, name, email, ...) +tags (ghost_id, slug, name, ...) +post_authors (post_id, author_id, sort_order) +post_tags (post_id, tag_id, sort_order) + +# Membership +ghost_labels (ghost_id, name, slug) +user_labels (user_id, label_id) +ghost_newsletters (ghost_id, name, slug, description) +user_newsletters (user_id, newsletter_id, subscribed) +ghost_tiers (ghost_id, name, slug, type, visibility) +ghost_subscriptions (ghost_id, user_id, status, cadence, price_amount, + price_currency, stripe_customer_id, stripe_subscription_id, + tier_id, raw) + +# User model fields +users.ghost_id, users.ghost_status, users.ghost_subscribed, +users.ghost_note, users.ghost_raw, users.stripe_customer_id +``` + +--- + +## Problems + +- **Two sources of truth** for content AND membership — constant sync overhead +- Every edit round-trips through Ghost's API — we don't own the write path +- Ghost sync is fragile (advisory locks, error recovery, partial sync states) +- Lexical JSON is opaque — we validate but never truly control the format +- Ghost is an entire Node.js application running alongside our Python stack +- Stripe integration is locked inside Ghost — we can't customize payment flows +- Newsletter/email is Ghost-native — no control over templates, scheduling, deliverability +- Membership tiers are Ghost concepts that don't map cleanly to our cooperative model + +--- + +## Target State + +Everything Ghost does is handled natively by our services: + +| Ghost Feature | Replacement | +|---|---| +| Post/page content | Sexp in `posts.body_sexp` column | +| Lexical editor | WYSIWYG editor saving sexp directly to DB | +| Media uploads | Direct upload to our storage (S3/local) — blog service endpoint | +| Authors | Already in our DB — just drop `ghost_id` column | +| Tags | Already in our DB — just drop `ghost_id` column | +| Members | Already our `users` table — drop Ghost sync, Ghost fields | +| Labels | Rename `ghost_labels` → `labels`, drop `ghost_id` | +| Newsletters | Native newsletter service (see Phase 7 below) | +| Tiers | Native membership tiers on `account` service | +| Subscriptions | Direct Stripe integration on `orders` service (already has SumUp) | +| Email sending | Transactional email service (Postmark/SES/SMTP) | +| Webhooks | Not needed — we own the write path | +| Ghost Docker container | Removed entirely | + +--- + +## Migration Phases + +### Phase 1: Lexical → Sexp Converter + +Write a one-time conversion script that transforms Lexical JSON into equivalent sexp. + +| Lexical Node | S-expression | +|---|---| +| `paragraph` | `(p ...)` | +| `heading` (level 1-6) | `(h1 ...)` ... `(h6 ...)` | +| `text` (plain) | `"string"` | +| `text` (bold) | `(strong "string")` | +| `text` (italic) | `(em "string")` | +| `text` (bold+italic) | `(strong (em "string"))` | +| `text` (code) | `(code "string")` | +| `link` | `(a :href "url" "text")` | +| `list` (bullet) | `(ul (li ...) ...)` | +| `list` (number) | `(ol (li ...) ...)` | +| `quote` | `(blockquote ...)` | +| `image` | `(use "image" :src "url" :alt "text" :caption "text")` | +| `code-block` | `(pre (code :class "language-x" "..."))` | +| `divider` | `(hr)` | +| `embed` | `(use "embed" :url "..." :type "...")` | + +Run against all existing posts, verify round-trip fidelity by rendering both versions and comparing HTML output. + +### Phase 2: Schema Changes — Content + +- Add `body_sexp` text column to `Post` model (or repurpose `lexical` column) +- Keep all existing metadata columns (title, slug, status, published_at, feature_image, etc.) +- Drop `ghost_id` from `Post`, `Author`, `Tag` tables (after full migration) +- Drop `mobiledoc` column (legacy Ghost format, unused) + +### Phase 3: Editor Integration + +Update the WYSIWYG editor to save sexp instead of Lexical JSON: + +- Editor toolbar actions produce sexp nodes +- Save endpoint writes directly to our DB (no Ghost Admin API call) +- Preview renders via the same sexp pipeline used for the public view +- Draft/publish workflow stays the same — just a `status` column update + +### Phase 4: Media Uploads + +Replace Ghost's upload proxy with native endpoints on the blog service: + +- `POST /admin/upload/image/` — accept image, store to S3/local, return URL +- `POST /admin/upload/media/` — audio/video +- `POST /admin/upload/file/` — generic files +- `GET /admin/oembed/?url=...` — OEmbed lookup (call providers directly) + +The editor already posts to proxy endpoints in `editor_api.py` — just retarget them to store directly rather than forwarding to Ghost. + +### Phase 5: Rendering Pipeline + +Update `post_data()` and related functions: + +- Parse `body_sexp` through the sexp evaluator +- Render to HTML via the existing `shared/sexp/html.py` pipeline +- Components referenced in post content (`use "image-gallery"`, etc.) resolve from the component registry +- Context variables (author data, related posts, etc.) passed as environment bindings + +### Phase 6: Membership Decoupling + +Migrate membership from Ghost to native account service: + +**Labels → native labels:** +- Rename `ghost_labels` → `labels`, drop `ghost_id` column +- `user_labels` stays as-is +- Admin UI manages labels directly (no Ghost sync) + +**Tiers → native membership tiers:** +- Rename `ghost_tiers` → `membership_tiers`, drop `ghost_id` +- Add tier management to account admin UI +- Tier assignment logic moves from Ghost webhook handler to account service + +**User model cleanup:** +- Drop: `ghost_id`, `ghost_status`, `ghost_subscribed`, `ghost_note`, `ghost_raw` +- Keep: `stripe_customer_id` (needed for direct Stripe integration) +- Add: `membership_tier_id` FK, `membership_status` enum (free/active/cancelled) + +### Phase 7: Newsletter System + +Replace Ghost's newsletter infrastructure with a native implementation: + +**Newsletter model (replaces `ghost_newsletters`):** +``` +newsletters (id, name, slug, description, from_email, reply_to, template_sexp, created_at) +user_newsletters (user_id, newsletter_id, subscribed, subscribed_at, unsubscribed_at) +``` + +**Email sending:** +- Integrate a transactional email provider (Postmark, AWS SES, or direct SMTP) +- Newsletter templates as sexp — rendered to HTML email via the same pipeline +- Send endpoint on account or blog service: select newsletter, select segment (by label/tier), queue sends +- Unsubscribe handling: tokenized unsubscribe links, one-click List-Unsubscribe header + +**Post → email campaign:** +- On publish, optionally select newsletter + segment (replaces Ghost's `?newsletter=slug&email_segment=...`) +- Render post body sexp to email-safe HTML (inline styles, table layout for email clients) +- Queue via background task (Celery or async worker) + +**What we gain over Ghost:** +- Email templates are sexp — same format as everything else +- Full control over deliverability (SPF/DKIM/DMARC on our domain) +- Segment by any user attribute, not just Ghost's limited filter syntax +- Send analytics stored in our DB + +### Phase 8: Subscription & Payment + +Replace Ghost's Stripe integration with direct Stripe on the orders service: + +**Current state:** Orders service already handles SumUp payments for marketplace/events. Adding Stripe for recurring subscriptions follows the same pattern. + +**Implementation:** +- Stripe Checkout for subscription creation (redirect flow, PCI compliant) +- Stripe Webhooks for subscription lifecycle (created, updated, cancelled, payment_failed) +- `subscriptions` table (replaces `ghost_subscriptions`): + ``` + subscriptions (id, user_id, tier_id, stripe_subscription_id, stripe_customer_id, + status, cadence, price_amount, price_currency, + current_period_start, current_period_end, cancelled_at) + ``` +- Customer portal: link to Stripe's hosted portal for card updates/cancellation +- Webhook handler on orders service (same pattern as SumUp webhooks) + +**What we gain:** +- Unified payment handling (SumUp for one-off, Stripe for recurring) +- Custom subscription logic (cooperative membership models, sliding scale, etc.) +- Direct access to Stripe customer data without Ghost intermediary + +### Phase 9: Remove Ghost + +Delete all Ghost integration code: + +| File/Directory | Action | +|---|---| +| `blog/bp/blog/ghost/` | Delete entire directory | +| `blog/bp/blog/ghost_db.py` | Delete | +| `blog/bp/blog/web_hooks/` | Delete | +| `shared/infrastructure/ghost_admin_token.py` | Delete | +| `account/services/ghost_membership.py` | Delete | +| Ghost Docker service | Remove from docker-compose | +| Ghost env vars | Remove all `GHOST_*` variables | +| Ghost webhook blueprint registration | Remove from blog routes | +| Startup sync (`sync_all_content_from_ghost`) | Remove from blog init | +| Startup sync (`sync_all_membership_from_ghost`) | Remove from account init | +| Advisory lock `900001` | Remove from blog init | + +Rename models: +- `ghost_content.py` → `content.py` +- `ghost_membership_entities.py` → `membership.py` +- Drop all `ghost_id` columns via Alembic migration + +### Phase 10: Content-Addressable Caching (ties into sexpr.js) + +Once posts are sexp and the JS client runtime exists: + +- Hash post body → content address +- Client caches post bodies in localStorage keyed by hash +- Server sends manifest of slug → hash mappings +- Unchanged posts served entirely from client cache +- Only the data envelope (metadata, component params) travels on repeat visits + +--- + +## What Stays the Same + +- `Post` model and all its metadata fields (minus ghost-specific ones) +- URL structure (`/slug/`) +- Tag, author, and tag group systems +- Draft/publish workflow +- Admin edit UI (updated to save sexp instead of Lexical) +- RSS feeds (rendered from sexp → HTML) +- Search indexing (extract text content from sexp) +- ActivityPub federation (triggered on publish, same as now) +- Alembic migrations (add/modify/drop columns) +- OAuth2 auth system (already independent of Ghost) + +--- + +## Ordering & Dependencies + +``` +Phase 1-2 (Content schema) ──→ Phase 3 (Editor) ──→ Phase 5 (Rendering) + Phase 4 (Uploads) ──┘ +Phase 6 (Membership) ──→ Phase 8 (Payments) +Phase 7 (Newsletters) ── independent, needs email provider choice +Phase 9 (Remove Ghost) ── after all above complete +Phase 10 (Content-addressed) ── after sexpr.js runtime exists +``` + +Phases 1-5 (content) and Phases 6-8 (membership/payments) can proceed in parallel — they touch different services. + +--- + +## Risk Mitigation + +- **Data safety**: Run Lexical → sexp converter in dry-run mode first, diff HTML output for every post +- **Rollback**: Keep `lexical` column and Ghost running during transition, feature flag to switch renderers +- **Editor UX**: Editor remains WYSIWYG — authors never see sexp syntax +- **SEO continuity**: URLs don't change, HTML output structurally identical +- **Email deliverability**: Set up SPF/DKIM/DMARC before sending first newsletter from our domain +- **Payment migration**: Run Ghost Stripe and direct Stripe in parallel during transition, migrate active subscriptions via Stripe API (change the subscription's application) +- **Membership data**: One-time migration script to clean User model fields, verified against Ghost export + +--- + +## Dependencies + +- Stable sexp parser + evaluator (already built: `shared/sexp/`) +- Component registry with post-relevant components: image, embed, gallery, code-block +- Editor sexp serialization (new work) +- Email provider account (Postmark/SES/SMTP) +- Stripe account with recurring billing enabled (may already exist via Ghost) +- Optional: sexpr.js client runtime for content-addressable caching (see `sexpr-js-runtime-plan.md`) diff --git a/docs/sexpr-js-runtime-plan.md b/docs/sexpr-js-runtime-plan.md new file mode 100644 index 0000000..23f1d08 --- /dev/null +++ b/docs/sexpr-js-runtime-plan.md @@ -0,0 +1,757 @@ +# sexpr.js — Development Plan + +**An isomorphic S-expression runtime for the web.** +**Code is data is DOM.** + +--- + +## Vision + +Replace HTML as the document/wire format with S-expressions. A single JavaScript library runs on both server (Node/Deno/Bun) and client (browser). The server composes and sends S-expressions over HTTP or WebSocket. The client parses them and renders/mutates the DOM. Because S-expressions are homoiconic, hypermedia controls (fetching, swapping, transitions) are native to the format — not bolted on as special attributes. + +The framework is not a Lisp. It is a document runtime that happens to use S-expression syntax because that syntax makes documents and commands interchangeable. + +--- + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────┐ +│ sexpr.js (shared) │ +│ │ +│ ┌───────────┐ ┌───────────┐ ┌────────────────┐ │ +│ │ Parser / │ │ Component │ │ Mutation │ │ +│ │ Serializer │ │ Registry │ │ Engine │ │ +│ └───────────┘ └───────────┘ └────────────────┘ │ +│ ┌───────────┐ ┌───────────┐ ┌────────────────┐ │ +│ │ Style │ │ Event │ │ VTree Diff │ │ +│ │ Compiler │ │ System │ │ & Patch │ │ +│ └───────────┘ └───────────┘ └────────────────┘ │ +└─────────────┬───────────────────────────┬───────────┘ + │ │ + ┌────────▼────────┐ ┌─────────▼─────────┐ + │ Server Adapter │ │ Client Adapter │ + │ │ │ │ + │ • renderToStr() │ │ • renderToDOM() │ + │ • diff on AST │ │ • mount/hydrate │ + │ • HTTP handler │ │ • WebSocket recv │ + │ • WS push │ │ • event dispatch │ + │ • SSR bootstrap │ │ • service worker │ + └──────────────────┘ └────────────────────┘ +``` + +The core is environment-agnostic. Thin adapters provide DOM APIs (client) or string serialization (server). Both sides share the parser, component system, style compiler, and mutation engine. + +--- + +## Phase 1: Core Runtime (Weeks 1–4) + +The foundation. A single ES module that works in any JS environment. + +### 1.1 Parser & Serializer + +**Parser** — tokenizer + recursive descent, producing an AST of plain JS objects. + +- Atoms: strings (`"hello"`), numbers (`42`, `3.14`), booleans (`#t`, `#f`), symbols (`div`, `my-component`), keywords (`:class`, `:on-click`) +- Lists: `(tag :attr "val" children...)` +- Comments: `; line comment` and `#| block comment |#` +- Quasiquote / unquote: `` ` `` and `,` for template interpolation on the server +- Streaming parser variant for large documents (SAX-style) + +**Serializer** — AST back to S-expression string. Round-trip fidelity. Pretty-printer with configurable indentation. + +**Deliverables:** +- `parse(string) → AST` +- `serialize(AST) → string` +- `prettyPrint(AST, opts) → string` +- Streaming: `createParser()` returning a push-based parser +- Comprehensive test suite (edge cases: nested strings, escapes, unicode, deeply nested structures) +- Benchmark: parse speed vs JSON.parse for equivalent data + +### 1.2 AST Representation + +The AST should be cheap to construct, diff, and serialize. Plain objects, not classes: + +```javascript +// Atoms +{ type: 'symbol', value: 'div' } +{ type: 'keyword', value: 'class' } +{ type: 'string', value: 'hello' } +{ type: 'number', value: 42 } +{ type: 'boolean', value: true } + +// List (the fundamental structure) +[head, ...rest] // plain arrays — cheap, diffable, JSON-compatible + +// Element sugar (derived during render, not stored) +// (div :class "box" (p "hi")) → +// [sym('div'), kw('class'), str('box'), [sym('p'), str('hi')]] +``` + +**Design decision:** ASTs are plain arrays and objects. No custom classes. This means they serialize to JSON trivially — enabling WebSocket transmission, IndexedDB caching, and worker postMessage without structured clone overhead. + +### 1.3 Element Rendering + +The core render function: AST → target output. + +**Shared logic** (environment-agnostic): +- Parse keyword attributes from element expressions +- Resolve component references +- Evaluate special forms (`if`, `each`, `list`, `let`, `slot`) +- Compile inline styles + +**Client adapter** — `renderToDOM(ast, env) → Node`: +- Creates real DOM nodes via `document.createElement` +- Handles SVG namespace detection +- Registers event handlers +- Returns live DOM node + +**Server adapter** — `renderToString(ast, env) → string`: +- Produces HTML string for initial page load (SEO, fast first paint) +- Inserts hydration markers so the client can attach without full re-render +- Escapes text content for safety + +### 1.4 Style System + +Styles as S-expressions, compiled to CSS strings. Isomorphic: the same style expressions produce CSS on server (injected into `