art-dag/mono
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add master plan: 9 tracks from stability to federated protocol

Browse Source 
Schedules all existing plans into coherent 20-week roadmap with parallel
tracks: platform stability, decoupling, entities/relations, Ghost removal,
sexp pages, internal protocol, client-side runtime, native client, and
scalability. Critical path identified through Ghost removal as linchpin.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
giles
2026-02-28 16:19:39 +00:00
parent f9d9697c67
commit c220fe21d6
1 changed files with 466 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

466
docs/masterplan.md Normal file
View File

@@ -0,0 +1,466 @@
# Rose Ash Master Plan
**From cooperative web platform to federated sexp protocol — scheduled, ordered, and realistic.**
---
## What's Done
The foundation is solid. These are complete and in production:
| Area | Status | Evidence |
|---|---|---|
| Sexp core (parser, evaluator, primitives) | Complete | 199 tests passing |
| Sexp HTML renderer | Complete | HSX-style, escaping, components |
| Sexp async resolver | Complete | Tree walker, parallel I/O |
| Sexp Jinja bridge | Complete | `sexp()` global in all apps |
| Sexp component templates | Complete | `.sexpr` files in all 8 services |
| Relation registry | Complete | `defrelation` form, 4 relations defined |
| Relation API | Complete | `relate`, `unrelate`, `can-relate` actions |
| Relation container-nav | Complete | Generic fragment, sexp-rendered |
| Relation caller migration (Phase F) | Complete | All callers on `relate`/`unrelate` |
| Decoupling Phases 1-3 | Complete | Models extracted, generic containers |
| Fragment infrastructure | Complete | Client, Redis cache, all apps serving |
| Link-card unification | Complete | Sexp component replaces 5 Jinja templates |
---
## What's Planned
Everything below is pending, grouped into tracks that can run in parallel where noted.
---
## Track 1: Platform Stability (Immediate)
*Bug fixes, test coverage, and operational reliability. Do this first — everything else builds on a stable base.*
### 1.1 — Unit Test Coverage
**When:** Now (ongoing, alongside all other work)
Expand from current 199 sexp tests to cover pure-logic modules:
- [ ] DTOs and contracts (`shared/contracts/`)
- [ ] URL utilities, HTTP signatures, HMAC auth
- [ ] Jinja filters and template helpers
- [ ] Calendar date helpers
- [ ] Config freeze/readonly enforcement
- [ ] Activity bus serialisation
- [ ] Sexp relation registry edge cases
Wire into `test-unit` Docker container. Run on every push.
### 1.2 — Scalability Tier 0
**When:** First available window (hours of work, 10x capacity)
- [ ] Deploy per-service database split
- [ ] Add PgBouncer connection pooling
- [ ] Separate auth Redis from cache Redis
- [ ] Tune Hypercorn worker count per service
- [ ] Add health check endpoints
### 1.3 — Bug Sweep
**When:** Before each major feature phase
- [ ] Audit remaining `attach-child`/`detach-child` calls (should be zero)
- [ ] Fix any broken fragment rendering from relations migration
- [ ] Verify all container-nav renders correctly across blog, events, market
- [ ] Test cart checkout flow end-to-end
- [ ] Validate ActivityPub federation still works (follow, like, boost, create)
---
## Track 2: Complete Decoupling (Weeks 1-3)
*Finish what's started. Clean separation enables everything that follows.*
### 2.1 — Cart Microservices Split
**When:** Week 1-2
The cart service currently owns too much. Split into focused services:
- [ ] Scaffold likes service (internal, port 8009) — unified like/favourite tracking
- [ ] Migrate `PageConfig` to blog service (page owner)
- [ ] Extract order history and checkout to orders service
- [ ] Cart becomes thin CartItem CRUD + checkout delegation
### 2.2 — Fragment Composition Phases 6-8
**When:** Week 2-3 (parallel with cart split)
- [ ] Account widget fragments (login state, profile mini)
- [ ] Template migration (move all domain templates out of `shared/templates/`)
- [ ] Delete shared template inheritance (apps own their layouts fully)
- [ ] Remove old widget system
### 2.3 — Decoupling Phase 5: Event-Driven Workflows
**When:** Week 3
- [ ] Order creation triggers domain events (not direct service calls)
- [ ] Login/signup events propagate via activity bus
- [ ] Replace remaining cross-service direct DB access with events
---
## Track 3: New Entities & Relations (Weeks 2-5)
*Extend the relation system with new entity types. Best done after decoupling is clean, before Ghost removal changes the content model.*
### 3.1 — Define New Relations
Add to the relation registry:
```scheme
;; Content relations
(defrelation :post->post
:from "post" :to "post" :cardinality :many-to-many
:nav :inline :nav-icon "fa fa-link" :nav-label "related")
(defrelation :post->tag
:from "post" :to "tag" :cardinality :many-to-many
:nav :hidden)
(defrelation :post->author
:from "post" :to "author" :cardinality :many-to-many
:nav :hidden)
;; Commerce relations
(defrelation :market->product
:from "market" :to "product" :cardinality :one-to-many
:inverse :product->market
:nav :submenu :nav-icon "fa fa-box" :nav-label "products")
(defrelation :page->marketplace
:from "page" :to "marketplace" :cardinality :one-to-one
:inverse :marketplace->page
:nav :submenu :nav-icon "fa fa-store" :nav-label "marketplace")
;; Event relations
(defrelation :calendar->calendar_entry
:from "calendar" :to "calendar_entry" :cardinality :one-to-many
:inverse :calendar_entry->calendar
:nav :submenu :nav-icon "fa fa-calendar-day" :nav-label "entries")
(defrelation :calendar_entry->ticket_type
:from "calendar_entry" :to "ticket_type" :cardinality :one-to-many
:nav :hidden)
;; Federation relations
(defrelation :user->ap_follower
:from "user" :to "ap_follower" :cardinality :one-to-many
:nav :hidden)
;; Cooperative governance (future)
(defrelation :page->proposal
:from "page" :to "proposal" :cardinality :one-to-many
:nav :submenu :nav-icon "fa fa-vote-yea" :nav-label "proposals")
```
### 3.2 — CalendarEntryPost Junction Migration
**When:** Week 3
The deferred work from Phase F — migrate the `CalendarEntryPost` junction table to use the relation system:
- [ ] Migrate existing junction rows to `ContainerRelation` with `relation_type = "post->calendar_entry"`
- [ ] Update events service toggle/query endpoints
- [ ] Remove old junction table
### 3.3 — Post-Tag and Post-Author via Relations
**When:** Week 4 (after Ghost removal Phase 2 changes the schema)
Currently `post_tags` and `post_authors` are Ghost-synced junction tables. Once Ghost is removed:
- [ ] Migrate to `ContainerRelation` or keep as dedicated junction tables (simpler for many-to-many with sort_order)
- [ ] Decision: relations for discovery/navigation, dedicated tables for core content model
### 3.4 — Product-Market Relations
**When:** Week 4-5
- [ ] Products currently linked to markets via `market_id` FK
- [ ] Add relation for cross-market product listing (a product can appear in multiple markets)
- [ ] Marketplace pages as relation containers
---
## Track 4: Ghost Removal (Weeks 3-8)
*The big migration. Depends on stable sexp core (done) and clean decoupling (Track 2). Content phases and membership phases can run in parallel.*
### 4.1 — Content Migration (Weeks 3-5)
**Phase 1: Lexical → Sexp Converter**
- [ ] Write one-time conversion script (Lexical JSON → sexp)
- [ ] Dry-run against all existing posts
- [ ] Diff HTML output (Lexical render vs sexp render) for every post
- [ ] Fix any conversion gaps (embeds, cards, special blocks)
**Phase 2: Schema Changes**
- [ ] Add `body_sexp` text column to Post model
- [ ] Run converter, populate `body_sexp` for all posts
- [ ] Keep `lexical` column during transition (rollback safety)
**Phase 3: Editor Integration**
- [ ] Update WYSIWYG editor to save sexp directly to DB
- [ ] Save endpoint writes to our DB (no Ghost Admin API)
- [ ] Preview renders via sexp pipeline
**Phase 4: Media Uploads**
- [ ] Native upload endpoints on blog service (image, audio/video, files)
- [ ] OEmbed lookup endpoint (call providers directly)
- [ ] Retarget editor upload calls to native endpoints
**Phase 5: Rendering Pipeline**
- [ ] `post_data()` reads `body_sexp` instead of `html`
- [ ] Render through sexp evaluator + HTML renderer
- [ ] Components in post content resolve from registry
### 4.2 — Membership Migration (Weeks 5-7)
**Phase 6: Membership Decoupling**
- [ ] Rename `ghost_labels` → `labels`, drop `ghost_id`
- [ ] Rename `ghost_tiers` → `membership_tiers`, drop `ghost_id`
- [ ] Clean User model: drop `ghost_id`, `ghost_status`, `ghost_subscribed`, `ghost_note`, `ghost_raw`
- [ ] Add `membership_tier_id` FK, `membership_status` enum
**Phase 7: Newsletter System**
- [ ] Native newsletter model (replaces `ghost_newsletters`)
- [ ] Integrate transactional email provider (Postmark/SES/SMTP)
- [ ] Newsletter templates as sexp (rendered to email-safe HTML)
- [ ] Unsubscribe handling (tokenised links, List-Unsubscribe header)
- [ ] Post → email campaign workflow
**Phase 8: Subscription & Payment**
- [ ] Stripe Checkout for subscription creation
- [ ] Stripe Webhooks for subscription lifecycle
- [ ] Native `subscriptions` table (replaces `ghost_subscriptions`)
- [ ] Customer portal via Stripe hosted portal
- [ ] Unified payment handling (SumUp one-off + Stripe recurring)
### 4.3 — Ghost Deletion (Week 8)
**Phase 9: Remove Ghost**
- [ ] Delete `blog/bp/blog/ghost/` directory
- [ ] Delete `blog/bp/blog/ghost_db.py`
- [ ] Delete `blog/bp/blog/web_hooks/`
- [ ] Delete `shared/infrastructure/ghost_admin_token.py`
- [ ] Delete `account/services/ghost_membership.py`
- [ ] Remove Ghost Docker service
- [ ] Remove all `GHOST_*` env vars
- [ ] Rename `ghost_content.py` → `content.py`
- [ ] Rename `ghost_membership_entities.py` → `membership.py`
- [ ] Alembic migration to drop all `ghost_id` columns
---
## Track 5: Sexp Page Architecture (Weeks 4-7)
*Convert remaining Jinja templates to sexp. Can run in parallel with Ghost removal membership phases.*
### 5.1 — Page Layouts as Sexp (Phase 5 of sexp-architecture)
**When:** Week 4-5
- [ ] Convert base page layout to sexp (head, nav, main, footer)
- [ ] Convert blog post page layout
- [ ] Convert market product page layout
- [ ] Convert event calendar page layout
- [ ] Component composition replaces template inheritance
### 5.2 — Routes as Sexp (Phase 6 of sexp-architecture)
**When:** Week 6-7
- [ ] `defroute` form for declaring routes as sexp
- [ ] Route dispatch from sexp expressions
- [ ] Middleware chain as sexp pipeline
- [ ] Content negotiation: same route serves HTML or sexp based on `Accept` header
---
## Track 6: Sexp Internal Protocol (Weeks 6-10)
*Replace HTTP+HMAC between services with native sexp protocol. This is the protocol playground.*
### 6.1 — Wire Format (Week 6)
- [ ] Define framing: length-prefixed sexp over Unix sockets (internal) and TCP/TLS (external)
- [ ] Lock the spec: `:key value` attrs, `#t/#f` booleans, `()` empty list — no alternatives
- [ ] Error model: `(err :code 404 :message "not found")`
### 6.2 — Python Client + Server Library (Weeks 7-8)
- [ ] `SexpConnection` class: connect, send, receive, stream
- [ ] Quart integration: handler that accepts sexp requests, returns sexp responses
- [ ] Drop-in replacements: `fetch_sexp()` replacing `fetch_data()` + `fetch_fragment()`
- [ ] HMAC auth over sexp (same signing, different envelope)
### 6.3 — Internal Mesh Migration (Weeks 9-10)
- [ ] Blog ↔ relations speaking sexp natively
- [ ] Events ↔ relations speaking sexp
- [ ] Market ↔ relations speaking sexp
- [ ] Measure: latency, throughput, error rates vs HTTP+JSON
- [ ] All services on sexp internally, HTTP externally
### 6.4 — ActivityPub over Sexp (Week 10)
- [ ] Federation between two rose-ash instances using sexp protocol
- [ ] Activities as sexp (Create, Follow, Like, Accept)
- [ ] Identity verification over sexp (RSA signatures in sexp envelope)
- [ ] Stress test: latency, reconnection, error recovery over real network
---
## Track 7: Client-Side Sexp (Weeks 8-14)
*Depends on stable internal protocol and sexp page architecture.*
### 7.1 — sexpr.js Core Runtime (Weeks 8-10)
- [ ] Parser + serialiser (<5KB gzipped)
- [ ] DOM renderer (sexp tree → DOM nodes)
- [ ] Mutation engine: `swap!`, `batch!`, `class!`, `request!`
- [ ] Component system: `defcomp`, slots, content-addressed caching
### 7.2 — sexpr.js Hypermedia (Weeks 10-12)
- [ ] Form handling (sexp forms → HTTP POST or sexp verb)
- [ ] Navigation (client-side routing, partial page updates)
- [ ] Streaming (bidirectional sexp over WebSocket, then QUIC)
- [ ] DevTools: sexp inspector, component browser, REPL
### 7.3 — Content-Addressed Caching (Week 12)
**Ghost Removal Phase 10**
- [ ] Hash post body → content address
- [ ] Client caches in localStorage keyed by SHA3 hash
- [ ] Server sends manifest (slug → hash)
- [ ] Unchanged content served entirely from client cache
### 7.4 — Browser Extension (Weeks 12-14)
- [ ] Chrome/Firefox extension intercepts `Accept` header, requests sexp
- [ ] Extension runs sexpr.js, renders directly to DOM
- [ ] Bypasses HTML serialisation entirely (Tier 1 client)
- [ ] Performance target: ~80ms page load
---
## Track 8: Native Client & Federation (Weeks 14-20)
*The long-term vision. Depends on everything above being stable.*
### 8.1 — Rust Client Library (Weeks 14-16)
- [ ] Sexp parser + serialiser in Rust
- [ ] QUIC transport (quinn library)
- [ ] Connection management, multiplexing, backpressure
- [ ] Auth: OAuth bearer tokens over sexp
### 8.2 — Rust Native Client (Weeks 16-18)
- [ ] TUI or GUI rendering of sexp pages
- [ ] Full Tier 2 client: ~20ms page load
- [ ] Platform accessibility APIs (AccessKit)
- [ ] Content-addressed local cache (SQLite or filesystem)
### 8.3 — Client-as-Node (Weeks 18-20)
- [ ] Rust client is an AP instance with inbox/outbox
- [ ] IPFS node for content persistence when offline
- [ ] Posts queue when recipient is offline, deliver on reconnect
- [ ] GPU sharing: artdag workers on member desktops
- [ ] Cooperative compute mesh: relay server as lightweight matchmaker
### 8.4 — Multi-Instance Federation (Week 20)
- [ ] Second rose-ash instance for another cooperative
- [ ] Full federation over sexp protocol
- [ ] Cross-instance commerce (browse, cart, checkout with federated identity)
- [ ] Cross-instance governance (propose, vote, ratify across co-ops)
---
## Track 9: Scalability (As Needed)
*Apply each tier when traffic demands it, not before.*
### Tier 1 (when hitting Tier 0 limits)
- [ ] Concurrent AP delivery
- [ ] Fragment circuit breaker
- [ ] Read replicas
- [ ] Data caching layer
### Tier 2 (when hitting Tier 1 limits)
- [ ] Edge-side fragment composition (Nginx SSI)
- [ ] Redis Streams event delivery
- [ ] CDN for static + cached content
- [ ] Horizontal scaling with replicas
### Tier 3 (when hitting Tier 2 limits)
- [ ] Dedicated AP delivery service
- [ ] Domain health tracking
- [ ] Table partitioning
- [ ] DTO caching layer
---
## Schedule Overview
```
Week 1-2: Track 1 (stability) + Track 2.1 (cart split)
Week 2-3: Track 2.2-2.3 (fragments, events) + Track 3.1 (new relations)
Week 3-5: Track 4.1 (Ghost content migration) + Track 3.2-3.3 (junction migrations)
Week 4-5: Track 5.1 (sexp page layouts)
Week 5-7: Track 4.2 (Ghost membership) + Track 5.2 (sexp routes)
Week 6-8: Track 6.1-6.2 (sexp protocol wire format + library)
Week 8: Track 4.3 (delete Ghost)
Week 8-10: Track 6.3-6.4 (internal mesh + AP federation) + Track 7.1 (sexpr.js core)
Week 10-12: Track 7.2-7.3 (sexpr.js hypermedia + caching)
Week 12-14: Track 7.4 (browser extension)
Week 14-18: Track 8.1-8.2 (Rust client)
Week 18-20: Track 8.3-8.4 (client-as-node + multi-instance federation)
```
**Tracks 1 (stability) and 9 (scalability) run continuously as needed.**
---
## Critical Path
The longest dependency chain:
```
Decoupling (done) → Cart split (W1-2) → Ghost content (W3-5) → Sexp pages (W4-5)
→ Sexp routes (W6-7) → Internal protocol (W6-10) → sexpr.js (W8-14)
→ Rust client (W14-18) → Client-as-node (W18-20)
```
**Ghost removal is the linchpin.** Everything before it enables it. Everything after it depends on it. Content must be sexp before the protocol can serve sexp to clients.
---
## When to Add New Features vs Fix Bugs
**Rule: stabilise before extending.**
- **Before each Track**: run the bug sweep (Track 1.3). Fix what's broken.
- **New entities and relations** (Track 3): best done in Weeks 2-5, after decoupling is clean but before Ghost removal changes the content model. The relation registry makes adding new entity types trivial — just add a `defrelation` and the nav/UI auto-generates.
- **New features on existing entities**: during or after the Track that touches them. Don't add features to Ghost-backed content — wait for sexp content.
- **Performance work**: only when measured. Don't optimise before Track 1.2 (Tier 0 scalability).
---
## One Developer Advantage
This plan looks like a year of committee work. For a single developer with AI tools, it's ~20 weeks of focused building. No coordination overhead, no design reviews, no waiting for other teams. Each track produces running code, not specifications. The protocol grows from working infrastructure, not from RFCs.
The Tier 0 strategy means rose-ash works at every stage. No big bang migration. No "it'll work when everything's done." Every week ships something that makes the platform better, and the protocol emerges from the practice.