mono/docs/sexp-architecture-plan.md

Rose-Ash S-Expression Architecture Transformation

Context

Rose-ash is a federated cooperative platform built as a Quart microservice monorepo (blog, market, cart, events, federation, account, likes, relations). It currently uses Jinja2 templates for rendering, ad-hoc Python code for fragment composition, and HTTP-based inter-service communication (data, actions, fragments, inbox).

Separately, the art-dag and art-dag-mono repos contain a DAG execution engine, s-expression parser/evaluator, Celery rendering pipeline, and 104k lines of media processing primitives — but are not integrated with rose-ash.

The transformation: Unify these into a single architecture where s-expressions are the application language. Pages and media renders become the same computation — DAG execution over content-addressed nodes. Python drops to the role of runtime primitives (DB, HTTP, IPFS, GPU). The app logic, layouts, components, routes, and data bindings are all expressed in s-expressions.

Key insight 1: Rendering a page and rendering a video are the same function — walk a DAG of content-addressed nodes, check cache, compute what's missing, assemble the result. The only difference is the leaf executor (Jinja/HTML vs JAX/pixels).

Key insight 2: S-expressions are the natural output language for an LLM. An LLM trained on the primitive vocabulary generates s-expressions in response to natural language prompts and HTTP requests. The primitives are the guardrails — the LLM can compose freely but can only invoke what's registered. The s-expressions are then resolved and rendered as HTML pages, media assets, or any other output format. The app becomes a conversation: user speaks natural language → LLM speaks s-expressions → resolver renders results. The LLM learns continually from the data flowing through the system — the s-expressions it generates, the content they resolve to, the user interactions they produce.

---

Architecture Overview

┌──────────────────────────────────────────────┐
│  Natural language (the interface)             │
│  user prompts, HTTP requests, AP activities   │
├──────────────────────────────────────────────┤
│  LLM (the compiler)                          │
│  natural language → s-expressions             │
│  trained on primitives, learns from data      │
├──────────────────────────────────────────────┤
│  S-expressions (the application)             │
│  components, pages, routes, data bindings,   │
│  media effects, composition, federation      │
├──────────────────────────────────────────────┤
│  Resolver (the engine)                       │
│  parse → analyze → plan → execute → cache    │
│  content-addressed at every node             │
├──────────────────────────────────────────────┤
│  Python primitives (the runtime)             │
│  HTTP/DB/Redis/IPFS/Celery/JAX/AP/OAuth      │
└──────────────────────────────────────────────┘

Storage tiers:
  Hot:  Redis (seconds-minutes, user-specific, ephemeral)
  Warm: IPFS  (immutable, content-addressed, shared)
  Pointers: IPNS (mutable "current version" references)

Feedback loop:
  LLM generates s-expression → resolver executes → result cached
  → user interacts → interaction data feeds back to LLM training
  → LLM improves its s-expression generation

---

Phase 1: S-Expression Core Library

Goal: A standalone s-expression parser, evaluator, and primitive registry that every service can import. Pure data manipulation — no I/O, no rendering, no HTTP.

Source material: ~/art-dag-mono/core/artdag/sexp/ (parser.py, evaluator.py, compiler.py, primitives.py)

Deliverables:

shared/sexp/
  __init__.py
  parser.py         # tokenize + parse s-expressions to AST
  types.py          # SExp, Symbol, Keyword, Atom types
  evaluator.py      # evaluate with environments, closures, let-bindings
  primitives.py     # register_primitive decorator + base primitives
  env.py            # environment/scope management

Base primitives (no I/O, pure transforms):

• seq, list, map, filter, reduce
• let, lambda, defcomp, if/when/cond
• str, concat, format
• slot (access keyword fields from data)
• Arithmetic, comparison, logic

Tasks:

1. Port parser from art-dag-mono, adapt to rose-ash conventions
2. Port evaluator, add defcomp (component definition) and defroute forms
3. Define type system (SExp, Symbol, Keyword, String, Number, List, Nil)
4. Implement environment/scope chain
5. Write primitive registry with @register_primitive decorator
6. Unit tests

Verification: Pure unit tests — parse → evaluate → assert result.

---

Phase 2: HTML Renderer

Goal: An HSX-style renderer that walks an s-expression tree and emits HTML strings. Handles elements, attributes, components, fragments, raw HTML, escaping.

Reference: HSX (Common Lisp) — s-expressions map directly to HTML elements.

Deliverables:

shared/sexp/
  html.py           # s-expression → HTML string renderer
  escape.py         # attribute and text escaping (XSS prevention)
  components.py     # defcomp registry, component resolution

Syntax conventions:

(div :class "foo" :id "bar"       ;; HTML element with attributes
  (h1 "Title")                     ;; text children
  (p "Paragraph"))

(defcomp ~card (&key title children)  ;; component (~ prefix)
  (div :class "card"
    (h2 title)
    children))

(~card :title "Hello"             ;; component invocation
  (p "Body"))

(<> (li "One") (li "Two"))        ;; fragment (no wrapper element)

(raw! "<b>trusted</b>")           ;; unescaped HTML

(when condition (p "shown"))      ;; conditional rendering
(map fn items)                    ;; list rendering

Tasks:

1. Implement HTML element rendering (tag, attributes, children)
2. Implement text escaping (prevent XSS — escape &, <, >, ", ')
3. Implement raw! for trusted HTML (existing Jinja | safe equivalent)
4. Implement fragment (<>) rendering
5. Implement defcomp / component registry and invocation
6. Implement void elements (img, br, input, meta, link)
7. Boolean attributes (disabled, checked, required)
8. Unit tests — render s-expression, assert HTML output

Verification: Render existing fragment templates as s-expressions, diff against current Jinja output.

---

Phase 3: Async Resolver

Goal: Walk an s-expression tree, identify nodes that need I/O (service fragments, data queries), fetch them in parallel, substitute results. This is the DAG execution engine applied to page rendering.

Source material:

• ~/art-dag-mono/core/artdag/engine.py (analyze → plan → execute)
• ~/rose-ash/shared/infrastructure/fragments.py (fetch_fragment, fetch_fragments, fetch_fragment_batch)

Deliverables:

shared/sexp/
  resolver.py       # async tree walker — identify, fetch, substitute
  cache.py          # content-addressed caching (SHA3-256 → Redis/IPFS)
  primitives_io.py  # I/O primitives (frag, query, action)

I/O primitives (async, registered separately from pure primitives):

• (frag service type :key val ...) → fetch_fragment
• (query service query-name :key val ...) → fetch_data
• (action service action-name :key val ...) → call_action
• (current-user) → load user from request context
• (htmx-request?) → check HX-Request header

Resolution strategy:

1. Parse the s-expression tree
2. Walk the tree, identify all frag/query nodes
3. Group independent fetches, dispatch via asyncio.gather()
4. Substitute results into the tree
5. Render resolved tree to HTML

Content addressing:

• Hash each subtree (SHA3-256 of the s-expression text)
• Check Redis (hot cache) → check IPFS (warm cache) → compute
• Cache rendered subtrees at configurable granularity

Tasks:

1. Implement async tree walker with parallel fetch grouping
2. Port content-addressed caching from art-dag-mono/core/artdag/cache.py
3. Implement frag primitive (wraps existing fetch_fragment)
4. Implement query primitive (wraps existing fetch_data)
5. Implement action primitive (wraps existing call_action)
6. Implement request-context primitives (current-user, htmx-request?)
7. Integration tests against running services

Verification: Render a blog post page via resolver, compare output to current Jinja render.

---

Phase 4: Bridge — Coexistence with Jinja

Goal: Allow s-expression components and Jinja templates to coexist. Migrate incrementally — one component at a time, one page at a time.

Deliverables:

shared/sexp/
  jinja_bridge.py   # Jinja filter/global to render s-expressions in templates
                    # + helper to embed Jinja output in s-expressions via raw!

Bridge patterns:

# In Jinja: render an s-expression component
{{ sexp('(~link-card :slug "apple" :title "Apple")') | safe }}

# In s-expression: embed existing Jinja template output
(raw! (jinja "fragments/nav_tree.html" :items nav-items))

Migration order for fragments (leaf nodes first):

1. link-card (blog, market, events, federation) — simplest, self-contained
2. cart-mini — small, user-specific
3. auth-menu — small, user-specific
4. nav-tree — recursive structure, good test of composition
5. container-nav, container-cards — cross-service composites

Tasks:

1. Implement sexp() Jinja global function
2. Implement jinja s-expression primitive
3. Rewrite link-card fragment as s-expression component (all services)
4. Rewrite cart-mini and auth-menu fragments
5. Rewrite nav-tree fragment
6. Rewrite container-nav and container-cards fragments
7. Verify each rewritten fragment produces identical HTML

Verification: A/B test — render via Jinja, render via s-expression, diff output.

---

Phase 5: Page Layouts as S-Expressions

Goal: Replace Jinja template inheritance ({% extends %}, {% block %}) with s-expression component composition. Layouts become components with slots.

Current template hierarchy:

_types/root/index.html          → base HTML shell
  _types/root/_index.html       → layout with aside, filter, content slots
    _types/blog/index.html      → blog-specific layout
      _types/post/index.html    → post page

Becomes:

(defcomp ~base-layout (&key title user cart &rest content)
  (html :lang "en"
    (head (title title) ...)
    (body :class "min-h-screen"
      (~header :user user :cart cart)
      (main :class "flex" content))))

(defcomp ~app-layout (&key title user cart aside filter content)
  (~base-layout :title title :user user :cart cart
    (when filter (div :id "filter" filter))
    (aside :id "aside" aside)
    (section :id "main-panel" content)))

(defcomp ~post-page (&key post nav-items user cart)
  (~app-layout
    :title (:slot post :title)
    :user user :cart cart
    :aside (~nav-tree :items nav-items)
    :content
      (article
        (h1 (:slot post :title))
        (div :class "prose" (raw! (:slot post :body))))))

OOB updates (HTMX partial renders):

(defcomp ~post-oob (&key post nav-items)
  (<>
    (div :id "filter" :hx-swap-oob "outerHTML"
      (~post-filter :post post))
    (aside :id "aside" :hx-swap-oob "outerHTML"
      (~nav-tree :items nav-items))
    (section :id "main-panel"
      (article ...))))

Tasks:

1. Define ~base-layout component (replaces _types/root/index.html)
2. Define ~app-layout component (replaces _types/root/_index.html)
3. Define ~header component (replaces header block)
4. Define OOB rendering pattern (replaces oob_elements.html)
5. Rewrite blog post page as s-expression
6. Rewrite market product page
7. Rewrite cart page
8. Rewrite events calendar page
9. Update route handlers to use resolver instead of render_template
10. Remove migrated Jinja templates

Verification: Visual comparison — deploy both paths, screenshot diff.

---

Phase 6: Routes as S-Expressions

Goal: Route definitions move from Python decorators + handler functions to s-expression declarations. Python route handlers become thin dispatchers.

Current:

@post_bp.get("/<slug>/")
async def post_view(slug):
    post = await services.blog.get_post_by_slug(g.s, slug)
    ctx = await post_data(slug, g.s)
    if is_htmx_request():
        return render_template("_types/post/_oob_elements.html", **ctx)
    return render_template("_types/post/index.html", **ctx)

Becomes:

(defroute "/blog/:slug/"
  (let ((post (query blog post-by-slug :slug slug))
        (nav (query blog nav-tree))
        (user (current-user))
        (cart (when user (query cart cart-summary :user_id (:slot user :id)))))
    (if (htmx-request?)
      (render (~post-oob :post post :nav-items nav))
      (render (~post-page :post post :nav-items nav :user user :cart cart)))))

Python dispatcher:

# One generic route handler per service
@bp.route("/<path:path>")
async def dispatch(path):
    route_expr = match_route(path)  # find matching defroute
    return await resolve_and_render(route_expr, request)

Tasks:

1. Implement defroute form with path pattern matching
2. Implement route registry (load s-expression route files at startup)
3. Implement request context binding (path params, query params, headers)
4. Write generic Quart dispatcher
5. Migrate blog routes
6. Migrate market routes
7. Migrate cart routes
8. Migrate events routes
9. Migrate account/federation routes

Verification: Full integration test — HTTP requests produce correct responses.

---

Phase 7: Content Addressing + IPFS + IPNS

Goal: Resolved fragment trees are content-addressed and cached on IPFS. IPNS provides mutable pointers to current versions. Cache invalidation becomes IPNS pointer updates.

Source material:

• ~/rose-ash/shared/utils/ipfs_client.py (already in rose-ash)
• ~/rose-ash/shared/utils/anchoring.py (merkle trees, OTS)
• ~/art-dag-mono/core/artdag/cache.py (content-addressed caching)

Two-tier caching:

Hot tier (Redis):   user-specific, short TTL, ephemeral
  key: sha3(s-expression) → rendered HTML
  examples: cart-mini, auth-menu

Warm tier (IPFS):   deterministic, immutable, shared
  CID: sha3(s-expression) → rendered HTML on IPFS
  IPNS name: stable identity → current CID
  examples: post-body, nav-tree, link-card, full pages

Invalidation:

• Content changes → new s-expression → new hash → new CID
• Service publishes new CID to IPNS name
• ActivityPub Update activity propagates to federated instances
• No TTL-based expiry for warm tier — immutable content, versioned pointers

Tasks:

1. Implement SHA3-256 hashing of s-expression subtrees
2. Implement two-tier cache lookup (Redis → IPFS → compute)
3. Implement IPNS name management per fragment type
4. Implement cache warming (pre-render and pin stable content)
5. Wire invalidation into event bus (content change → IPNS update)
6. Wire IPNS updates into AP federation (Update activities)

Verification: Cache hit rates, IPFS pin counts, IPNS resolution latency.

---

Phase 8: Media Pipeline Integration

Goal: Bring art-dag's Celery rendering pipeline into rose-ash as the render service. Same s-expression language, same resolver, different leaf executors (JAX/FFmpeg instead of HTML).

Source material:

• ~/art-dag-mono/l1/ (Celery app, tasks, sexp_effects, streaming)
• ~/art-dag-mono/core/artdag/ (engine, analysis, planning, nodes, effects)

Deliverables:

rose-ash/
  render/                    # new service
    celery_app.py            # from art-dag-mono/l1/celery_app.py
    tasks/                   # from art-dag-mono/l1/tasks/
    sexp_effects/            # from art-dag-mono/l1/sexp_effects/
      primitives.py          # 104k lines of media primitives
      interpreter.py
      wgsl_compiler.py       # GPU shaders
    effects/                 # effect plugins
    streaming/               # video streaming output
    Dockerfile
    Dockerfile.gpu
    docker-compose.yml

Integration points:

• (render-media expr) primitive dispatches to Celery task
• Results stored on IPFS, CID returned
• Event bus activity emitted on completion
• Same content-addressing — same s-expression → same output CID

Tasks:

1. Move L1 codebase into rose-ash/render/
2. Adapt imports to use shared/sexp/ parser (replace local copy)
3. Register media primitives alongside web primitives
4. Implement render-media primitive (dispatch to Celery)
5. Wire Celery task completion into event bus
6. Integration tests — submit recipe, verify output on IPFS

Verification: Submit an s-expression media recipe via the resolver, get back an IPFS CID with the rendered output.

---

Phase 9: Unified DAG Executor

Goal: One execution engine that handles both page renders and media renders. The executor dispatches to different primitive sets based on node type, but the resolution, caching, and content-addressing logic is shared.

Deliverables:

shared/sexp/
  executor.py        # unified DAG executor
  registry.py        # executor registry (HTML executors, media executors)

Unified flow:

input s-expression
  → parse (shared/sexp/parser.py)
  → analyze (identify node types, owners, dependencies)
  → plan (check cache tiers, determine fetch/compute order)
  → execute (dispatch to registered executors in parallel)
  → cache (store results at appropriate tier)
  → return (HTML string, media CID, or composed result)

Tasks:

1. Abstract the resolver into a generic DAG executor
2. Implement executor registry (register by node type/prefix)
3. Register HTML executors (frag, query, render, defcomp)
4. Register media executors (transcode, filter, compose, source)
5. Implement mixed-mode execution (page with embedded media)
6. Provenance tracking (link executor output to AP activities)

Verification: A single resolve() call handles a page that contains both HTML components and embedded media references.

---

Phase 10: Federation of S-Expressions

Goal: S-expression components and pages are first-class ActivityPub objects. Remote instances can fetch, render, cache, and re-style federated content expressed as s-expressions.

Integration with existing AP infrastructure:

• shared/infrastructure/activitypub.py — actor endpoints
• shared/events/bus.py — activity emission
• shared/utils/ipfs_client.py — content storage
• shared/utils/anchoring.py — provenance

Tasks:

1. Define AP object type for s-expression content (rose:SExpression)
2. Publish component definitions as AP Create activities
3. Federate page updates as AP Update activities with IPNS pointers
4. Implement remote component resolution (fetch s-expr from remote instance)
5. Implement content verification (signature on s-expression CID)
6. Implement re-styling (apply local theme to remote s-expression)

Verification: Instance A publishes a post, Instance B resolves and renders it from the federated s-expression.

---

Phase 11: LLM as S-Expression Compiler

Goal: An LLM trained on the primitive vocabulary generates s-expressions from natural language. Users describe what they want in plain English. The LLM outputs valid s-expressions. The resolver renders them. Pages are generated on the fly from conversation.

The LLM speaks s-expressions because:

• The primitive vocabulary is small and well-defined (unlike HTML/CSS/JS)
• S-expressions are structurally simple — easy for an LLM to generate correctly
• The resolver validates and sandboxes — the LLM can't produce unsafe output
• Every generated s-expression is content-addressed — same prompt → cacheable result
• The primitives are the training data — the LLM learns what ~product-card, ~nav-tree, (query market ...) do from examples

How it works:

User: "show me a page of seasonal vegetables under £5"

LLM generates:
(~app-layout :title "Seasonal Vegetables Under £5"
  :content
    (let ((products (query market products-search
                     :category "vegetables"
                     :seasonal true
                     :max_price 5.00
                     :sort "price-asc")))
      (div :class "grid grid-cols-3 gap-4"
        (if (empty? products)
          (p :class "text-gray-500" "No vegetables found matching your criteria.")
          (map (lambda (p) (~product-card :slug (:slot p :slug)))
               products)))))

Resolver: parse → fetch products → render cards → HTML page

Three modes of LLM integration:

1. Generative pages: User prompt → LLM → s-expression → rendered page

   • Conversational UI: user refines via follow-up prompts
   • Each generated page is a CID on IPFS — shareable, cacheable

2. Adaptive layouts: LLM observes user behavior → generates personalized component arrangements

   • Home page adapts: frequent buyer sees cart-heavy layout
   • Event organizer sees calendar-first layout
   • Same primitives, different composition

3. Content authoring: LLM assists in creating blog posts, product descriptions, event listings

   • Author describes intent → LLM generates structured s-expression content
   • Content is data (s-expression), not just text — queryable, composable, versionable

Training the LLM on primitives:

• Primitive catalog: every registered primitive with its signature, description, examples
• Component library: every defcomp with usage examples
• Query catalog: every (query service name) with parameter schemas and return types
• Interaction logs: successful s-expressions that produced good user outcomes
• Continuous learning: new primitives/components automatically extend the vocabulary

Safety model:

• S-expressions can only invoke registered primitives — no arbitrary code execution
• The resolver validates the tree before execution
• I/O primitives respect existing auth (HMAC, OAuth, user context)
• Rate limiting on LLM generation endpoint
• Content-addressed caching prevents regeneration of identical requests
• Generated s-expressions are logged as AP activities (provenance tracking)

Deliverables:

shared/sexp/
  llm.py              # LLM integration — prompt → s-expression generation
  catalog.py          # primitive/component catalog for LLM context
  validation.py       # validate generated s-expressions before execution

rose-ash/
  llm/                # LLM service (or integration with external LLM API)
    routes.py         # conversational endpoint
    training.py       # continuous learning from interaction data
    prompts/          # system prompts with primitive catalog

Tasks:

1. Build primitive catalog generator (introspect registry → structured docs)
2. Build component catalog generator (introspect defcomp registry → examples)
3. Build query catalog generator (introspect data endpoints → schemas)
4. Design system prompt that teaches LLM the s-expression grammar + primitives
5. Implement generation endpoint (natural language → s-expression)
6. Implement validation layer (parse + type-check generated expressions)
7. Implement conversational refinement (user feedback → modified s-expression)
8. Implement caching of generated s-expressions (prompt hash → CID)
9. Wire into AP for provenance (LLM-generated content attributed to LLM actor)
10. Implement feedback loop (interaction data → training signal)

Verification: User prompt → generated page → visual inspection + primitive coverage audit.

---

Summary of Phases

Phase	What	Depends On	Scope
1	S-expression core library	—	shared/sexp/
2	HTML renderer (HSX-style)	1	shared/sexp/html.py
3	Async resolver	1, 2	shared/sexp/resolver.py
4	Jinja bridge + fragment migration	2, 3	All services' fragment templates
5	Page layouts as s-expressions	4	All services' page templates
6	Routes as s-expressions	5	All services' route handlers
7	Content addressing + IPFS/IPNS	3	shared/sexp/cache.py
8	Media pipeline integration	1, 7	render/ service
9	Unified DAG executor	3, 8	shared/sexp/executor.py
10	Federation of s-expressions	7, 9	AP + IPFS integration
11	LLM as s-expression compiler	1-6, 9	shared/sexp/llm.py, llm/ service

Foundation (Phases 1-3): The s-expression language, HTML rendering, and async resolver. Everything else builds on this.

Migration (Phases 4-6): Incremental replacement of Jinja templates and Python route handlers with s-expressions. The bridge ensures coexistence — the app never breaks during migration.

Infrastructure (Phases 7-9): Content addressing, IPFS/IPNS caching, media pipeline, and unified DAG execution. Pages and video renders become the same computation.

Intelligence (Phases 10-11): Federation makes s-expressions portable across instances. The LLM makes s-expressions accessible to non-programmers — natural language in, rendered pages out. The system learns from its own data, continuously improving the quality of generated s-expressions.

Each phase is independently deployable. The end state: a platform where the application logic is expressed in a small, composable, content-addressed language that humans author, LLMs generate, resolvers execute, IPFS stores, and ActivityPub federates.