rose-ash/docs/sexpr-js-runtime-plan.md

sexpr.js — Development Plan

An isomorphic S-expression runtime for the web. Code is data is DOM.

---

Vision

Replace HTML as the document/wire format with S-expressions. A single JavaScript library runs on both server (Node/Deno/Bun) and client (browser). The server composes and sends S-expressions over HTTP or WebSocket. The client parses them and renders/mutates the DOM. Because S-expressions are homoiconic, hypermedia controls (fetching, swapping, transitions) are native to the format — not bolted on as special attributes.

The framework is not a Lisp. It is a document runtime that happens to use S-expression syntax because that syntax makes documents and commands interchangeable.

---

Architecture Overview

┌─────────────────────────────────────────────────────┐
│                    sexpr.js (shared)                 │
│                                                     │
│  ┌───────────┐  ┌───────────┐  ┌────────────────┐  │
│  │  Parser /  │  │ Component │  │   Mutation      │  │
│  │ Serializer │  │  Registry │  │   Engine        │  │
│  └───────────┘  └───────────┘  └────────────────┘  │
│  ┌───────────┐  ┌───────────┐  ┌────────────────┐  │
│  │   Style   │  │   Event   │  │   VTree Diff    │  │
│  │  Compiler │  │  System   │  │   & Patch       │  │
│  └───────────┘  └───────────┘  └────────────────┘  │
└─────────────┬───────────────────────────┬───────────┘
              │                           │
     ┌────────▼────────┐        ┌─────────▼─────────┐
     │  Server Adapter  │        │  Client Adapter    │
     │                  │        │                    │
     │  • renderToStr() │        │  • renderToDOM()   │
     │  • diff on AST   │        │  • mount/hydrate   │
     │  • HTTP handler  │        │  • WebSocket recv  │
     │  • WS push       │        │  • event dispatch  │
     │  • SSR bootstrap │        │  • service worker  │
     └──────────────────┘        └────────────────────┘

The core is environment-agnostic. Thin adapters provide DOM APIs (client) or string serialization (server). Both sides share the parser, component system, style compiler, and mutation engine.

---

Phase 1: Core Runtime (Weeks 1–4)

The foundation. A single ES module that works in any JS environment.

1.1 Parser & Serializer

Parser — tokenizer + recursive descent, producing an AST of plain JS objects.

• Atoms: strings ("hello"), numbers (42, 3.14), booleans (#t, #f), symbols (div, my-component), keywords (:class, :on-click)
• Lists: (tag :attr "val" children...)
• Comments: ; line comment and #| block comment |#
• Quasiquote / unquote: ` and , for template interpolation on the server
• Streaming parser variant for large documents (SAX-style)

Serializer — AST back to S-expression string. Round-trip fidelity. Pretty-printer with configurable indentation.

Deliverables:

• parse(string) → AST
• serialize(AST) → string
• prettyPrint(AST, opts) → string
• Streaming: createParser() returning a push-based parser
• Comprehensive test suite (edge cases: nested strings, escapes, unicode, deeply nested structures)
• Benchmark: parse speed vs JSON.parse for equivalent data

1.2 AST Representation

The AST should be cheap to construct, diff, and serialize. Plain objects, not classes:

// Atoms
{ type: 'symbol', value: 'div' }
{ type: 'keyword', value: 'class' }
{ type: 'string', value: 'hello' }
{ type: 'number', value: 42 }
{ type: 'boolean', value: true }

// List (the fundamental structure)
[head, ...rest]   // plain arrays — cheap, diffable, JSON-compatible

// Element sugar (derived during render, not stored)
// (div :class "box" (p "hi")) →
// [sym('div'), kw('class'), str('box'), [sym('p'), str('hi')]]

Design decision: ASTs are plain arrays and objects. No custom classes. This means they serialize to JSON trivially — enabling WebSocket transmission, IndexedDB caching, and worker postMessage without structured clone overhead.

1.3 Element Rendering

The core render function: AST → target output.

Shared logic (environment-agnostic):

• Parse keyword attributes from element expressions
• Resolve component references
• Evaluate special forms (if, each, list, let, slot)
• Compile inline styles

Client adapter — renderToDOM(ast, env) → Node:

• Creates real DOM nodes via document.createElement
• Handles SVG namespace detection
• Registers event handlers
• Returns live DOM node

Server adapter — renderToString(ast, env) → string:

• Produces HTML string for initial page load (SEO, fast first paint)
• Inserts hydration markers so the client can attach without full re-render
• Escapes text content for safety

1.4 Style System

Styles as S-expressions, compiled to CSS strings. Isomorphic: the same style expressions produce CSS on server (injected into <style> tags in HTML) and client (injected into document).

(style ".card"
  :background "#1a1a2e"
  :border-radius "8px"
  :padding "1.5rem"
  :hover (:background "#2a2a3e")        ; nested pseudo-classes
  :media "(max-width: 600px)"
    (:padding "1rem"))

(style "@keyframes fade-in"
  (from :opacity "0")
  (to :opacity "1"))

Features:

• Nested selectors (like Sass)
• @media, @keyframes, @container as nested S-expressions
• CSS variables as regular properties
• Optional: scoped styles per component (auto-prefix class names)
• Output: raw CSS string or <style> DOM node

---

Phase 2: Hypermedia Engine (Weeks 5–8)

The mutation layer. This is where homoiconicity pays off.

2.1 Mutation Primitives

Since commands and content share the same syntax, the server can send either — or both — in a single response:

;; Content (renders to DOM)
(div :class "card" (p "Hello"))

;; Command (mutates existing DOM)
(swap! "#card-1" :inner (p "Updated"))

;; Compound (multiple mutations atomically)
(batch!
  (swap! "#notifications" :append
    (div :class "toast" "Saved!"))
  (class! "#save-btn" :remove "loading")
  (transition! "#toast" :type "slide-in"))

Full primitive set:

Primitive	Purpose
swap!	Replace/insert content (:inner, :outer, :before, :after, :prepend, :append, :delete, :morph)
batch!	Execute multiple mutations atomically
class!	Add/remove/toggle CSS classes
attr!	Set/remove attributes
style!	Inline style manipulation
transition!	CSS transitions and animations
wait!	Delay between batched mutations
dispatch!	Fire custom events

2.2 Request/Response Cycle

The equivalent of HTMX's hx-get, hx-post, etc. — but as native S-expressions:

(request!
  :method "POST"
  :url "/api/todos"
  :target "#todo-list"
  :swap inner
  :include "#todo-form"         ; serialize form data
  :indicator "#spinner"         ; show during request
  :confirm "Are you sure?"      ; browser confirm dialog
  :timeout 5000
  :retry 3
  :on-error (swap! "#errors" :inner
    (p :class "error" "Request failed")))

Client-side implementation:

1. Serialize form data (if :include specified)
2. Show indicator
3. fetch() with Accept: text/x-sexpr header
4. Parse response as S-expression
5. If response is a mutation command → execute it
6. If response is content → wrap in swap! using :target and :swap
7. Hide indicator
8. Handle errors

Server-side helpers:

// Server constructs response using the same library
const { s, sym, kw, str } = require('sexpr');

app.post('/api/todos', (req, res) => {
  const todo = createTodo(req.body);
  res.type('text/x-sexpr').send(
    s.serialize(
      s.batch(
        s.swap('#todo-list', 'append', todoComponent(todo)),
        s.swap('#todo-count', 'inner', s.text(`${count} remaining`)),
        s.swap('#todo-input', 'attr', { value: '' })
      )
    )
  );
});

2.3 WebSocket Channel

For real-time updates. The server pushes S-expression mutations over WebSocket:

;; Server → Client (push)
(batch!
  (swap! "#user-42-status" :inner
    (span :class "online" "Online"))
  (swap! "#chat-messages" :append
    (div :class "message"
      (strong "Alice") " just joined")))

Protocol:

• Content-type negotiation: text/x-sexpr over HTTP, raw S-expr strings over WS
• Client reconnects automatically with exponential backoff
• Server can send any mutation at any time — the client just evaluates it
• Optional: message IDs for acknowledgment, ordering guarantees

2.4 Event Binding

Declarative event binding that works both inline and as post-render setup:

;; Inline (in element definition)
(button :on-click (request! :method "POST" :url "/api/like"
                    :target "#like-count" :swap inner)
  "Like")

;; Declarative (standalone, for progressive enhancement)
(on-event! "#search-input" "input"
  :debounce 300
  (request! :method "GET"
    :url (concat "/api/search?q=" (value event.target))
    :target "#results" :swap inner))

;; Keyboard shortcuts
(on-event! "body" "keydown"
  :filter (= event.key "Escape")
  (class! "#modal" :remove "open"))

Event modifiers (inspired by Vue/Svelte):

• :debounce 300 — debounce in ms
• :throttle 500 — throttle in ms
• :once — fire once then unbind
• :prevent — preventDefault
• :stop — stopPropagation
• :filter (expr) — conditional guard

---

Phase 3: Component System (Weeks 9–12)

3.1 Component Definition & Instantiation

Components are parameterized S-expression templates. Not classes. Not functions. Data.

(component "todo-item" (id text done)
  (style ".todo-item" :display "flex" :align-items "center" :gap "0.75rem")
  (style ".todo-item.done .text" :text-decoration "line-through" :opacity "0.5")

  (li :class (if done "todo-item done" "todo-item") :id (concat "todo-" id)
    (span :class "check"
      :on-click (request! :method "POST"
                  :url (concat "/api/todos/" id "/toggle")
                  :target (concat "#todo-" id) :swap outer)
      (if done "◉" "○"))
    (span :class "text" text)
    (button :class "delete"
      :on-click (request! :method "DELETE"
                  :url (concat "/api/todos/" id)
                  :target (concat "#todo-" id) :swap delete)
      "×")))

;; Usage
(use "todo-item" :id "1" :text "Buy milk" :done #f)

Features:

• Lexically scoped parameters
• Styles co-located with component (auto-scoped or global, configurable)
• Slots for content projection: (slot) for default, (slot "header") for named
• Isomorphic: same component definition renders on server (to string) or client (to DOM)

3.2 Slots & Composition

(component "card" (title)
  (div :class "card"
    (div :class "card-header" (h3 title))
    (div :class "card-body" (slot))           ; default slot
    (div :class "card-footer" (slot "footer"))))  ; named slot

(use "card" :title "My Card"
  (p "This goes in the default slot")
  (template :slot "footer"
    (button "OK") (button "Cancel")))

3.3 Layouts & Pages

Higher-level composition for full pages:

(layout "main" ()
  (style "body" :font-family "var(--font)" :margin "0")
  (style ".layout" :display "grid" :grid-template-rows "auto 1fr auto" :min-height "100vh")

  (div :class "layout"
    (header (use "nav-bar"))
    (main (slot))
    (footer (use "site-footer"))))

;; A page uses a layout
(page "/about"
  :layout "main"
  :title "About Us"
  (section
    (h1 "About")
    (p "We replaced HTML with S-expressions.")))

3.4 Server-Side Component Registry

On the server, components are registered globally and can be shared between routes:

const { registry, component, page, serve } = require('sexpr/server');

// Register components (can also load from .sexpr files)
registry.loadDir('./components');

// Or inline
registry.define('greeting', ['name'],
  s`(div :class "greeting" (h1 "Hello, " name))`
);

// Route handler returns S-expressions
app.get('/about', (req, res) => {
  res.sexpr(
    page('/about', { layout: 'main', title: 'About' },
      s`(section (h1 "About") (p "Hello world"))`)
  );
});

---

Phase 4: Virtual Tree & Diffing (Weeks 13–16)

4.1 VTree Representation

The parsed AST already is the virtual tree — no separate representation needed. This is a direct benefit of homoiconicity. While React needs createElement() to build a virtual DOM from JSX, our AST is the VDOM.

S-expression string  →  parse()  →  AST  ≡  VTree
                                       │
                          renderToDOM() │ diff()
                                       ▼
                                    DOM / Patches

4.2 Diff Algorithm

O(n) same-level comparison, similar to React's reconciliation but operating on S-expression ASTs:

// Both sides can call this
const patches = diff(oldTree, newTree);

// Client applies to DOM
applyPatches(rootNode, patches);

// Server serializes as mutation commands
const mutations = patchesToSexpr(patches);
// → (batch! (swap! "#el-3" :inner (p "new")) (attr! "#el-7" :set :class "active"))

Patch types:

• REPLACE — replace entire node
• PROPS — update attributes
• TEXT — update text content
• INSERT — insert child at index
• REMOVE — remove child at index
• REORDER — reorder children (using :key hints)

Key insight: Because patches are also S-expressions, the server can compute a diff and send it as a batch! of mutations. The client doesn't need to diff at all — it just executes the mutations. This means the server does the expensive work and the client stays thin.

4.3 Keyed Reconciliation

For efficient list updates:

(each todos (lambda (t)
  (use "todo-item" :key (get t "id") :text (get t "text") :done (get t "done"))))

The :key attribute enables the diff algorithm to match nodes across re-renders, minimizing DOM operations for list insertions, deletions, and reorderings.

4.4 Hydration

Server sends pre-rendered HTML (for SEO and fast first paint). Client attaches to existing DOM without re-rendering:

1. Server renders S-expression → HTML string with hydration markers
2. Browser displays HTML immediately (fast first contentful paint)
3. Client JS loads, parses the original S-expression source (embedded in a <script type="text/x-sexpr"> tag)
4. Client walks the existing DOM and attaches event handlers without rebuilding it
5. Subsequent updates go through the normal S-expression channel

---

Phase 5: Developer Experience (Weeks 17–20)

5.1 CLI Tool

npx sexpr init my-app          # scaffold project
npx sexpr dev                   # dev server with hot reload
npx sexpr build                 # production build
npx sexpr serve                 # production server

Project structure:

my-app/
  components/
    nav-bar.sexpr
    todo-item.sexpr
    card.sexpr
  pages/
    index.sexpr
    about.sexpr
  layouts/
    main.sexpr
  styles/
    theme.sexpr
  server.js
  sexpr.config.js

5.2 .sexpr File Format

Single-file components with co-located styles, markup, and metadata:

; components/todo-item.sexpr

(meta
  :name "todo-item"
  :params (id text done)
  :description "A single todo list item with toggle and delete")

(style ".todo-item"
  :display "flex"
  :align-items "center"
  :gap "0.75rem")

(li :class (if done "todo-item done" "todo-item")
  :id (concat "todo-" id)
  :key id
  (span :class "check"
    :on-click (request! :method "POST"
                :url (concat "/api/todos/" id "/toggle")
                :target (concat "#todo-" id) :swap outer)
    (if done "◉" "○"))
  (span :class "text" text)
  (button :class "delete"
    :on-click (request! :method "DELETE"
                :url (concat "/api/todos/" id)
                :target (concat "#todo-" id) :swap delete)
    "×"))

5.3 DevTools

Browser extension (or embedded panel):

• AST inspector: visualize the S-expression tree alongside the DOM
• Mutation log: every swap!, class!, batch! logged with timestamp
• Network tab: S-expression request/response viewer (not raw text)
• Component tree: hierarchical view of instantiated components
• Time-travel: replay mutations forward/backward

Server-side:

• Request logger showing S-expression responses
• Component dependency graph
• Hot reload: file watcher on .sexpr files, push updates via WebSocket

5.4 Editor Support

• VS Code extension: syntax highlighting for .sexpr files, bracket matching (parentheses), auto-indentation, component name completion, attribute completion for HTML tags
• Tree-sitter grammar: for Neovim, Helix, Zed, etc.
• Prettier plugin: auto-format .sexpr files
• LSP server: go-to-definition for components, find-references, rename symbol

5.5 Error Handling

• Parse errors: line/column reporting with context (show the offending line)
• Render errors: error boundaries like React — a component crash renders a fallback, not a blank page
• Network errors: :on-error handler in request!, plus global error handler
• Dev mode: verbose errors with suggestions; production mode: compact

---

Phase 6: Ecosystem & Production (Weeks 21–28)

6.1 Middleware & Plugins

Server middleware (Express/Koa/Hono compatible):

const { sexprMiddleware } = require('sexpr/server');

app.use(sexprMiddleware({
  componentsDir: './components',
  layoutsDir: './layouts',
  hotReload: process.env.NODE_ENV !== 'production'
}));

// Route handlers return S-expressions directly
app.get('/', (req, res) => {
  res.sexpr(homePage(req.user));
});

Plugin system for extending the runtime:

// A plugin that adds a (markdown "...") special form
sexpr.plugin('markdown', {
  transform(ast) { /* convert markdown to sexpr AST */ },
  serverRender(ast) { /* render to HTML string */ },
  clientRender(ast) { /* render to DOM */ }
});

6.2 Router

Client-side navigation without full page reloads:

(router
  (route "/" (use "home-page"))
  (route "/about" (use "about-page"))
  (route "/todos/:id" (use "todo-detail" :id params.id))
  (route "*" (use "not-found")))

• Intercepts <a> clicks for internal links
• pushState / popState navigation
• Server-side: same route definitions, used for SSR
• Prefetch: (link :href "/about" :prefetch #t "About")

6.3 Forms

Declarative form handling:

(form :action "/api/users" :method "POST"
  :target "#result" :swap inner
  :validate #t                    ; client-side validation
  :reset-on-success #t

  (input :type "text" :name "username"
    :required #t
    :minlength 3
    :pattern "[a-zA-Z0-9]+"
    :error-message "Alphanumeric, 3+ chars")

  (input :type "email" :name "email" :required #t)

  (button :type "submit" "Create User"))

The server validates identically — same validation rules expressed as S-expressions run on both sides.

6.4 Content-Type & MIME

Register text/x-sexpr as a proper MIME type:

• HTTP responses: Content-Type: text/x-sexpr; charset=utf-8
• Accept header negotiation: client sends Accept: text/x-sexpr, text/html;q=0.9
• Fallback: if client doesn't accept text/x-sexpr, server renders to HTML (graceful degradation)
• File extension: .sexpr

6.5 Caching & Service Worker

• Service worker: caches S-expression responses, serves offline
• Incremental cache: cache individual components, not whole pages
• ETag/304: standard HTTP caching works because responses are text
• Compression: S-expressions compress well with gzip/brotli (repetitive keywords)

6.6 Security

• No eval(): S-expressions are parsed, not evaluated as code. The runtime only understands the defined special forms.
• XSS prevention: text content is always escaped when rendered to DOM (via textContent, not innerHTML). The raw-html escape hatch requires explicit opt-in.
• CSP compatible: no inline scripts generated. Event handlers are registered via JS, not onclick attributes (move away from the v1 prototype approach).
• CSRF: standard token-based approach, with (meta :csrf-token "...") in the page head.

6.7 Accessibility

• S-expressions map 1:1 to semantic HTML elements — (nav ...), (main ...), (article ...), (aside ...) all render to their HTML equivalents
• ARIA attributes work naturally: :aria-label "Close" :aria-expanded #f :role "dialog"
• Focus management primitives: (focus! "#element"), (trap-focus! "#modal")

---

Phase 7: Testing & Documentation (Weeks 25–28, overlapping)

7.1 Testing Utilities

const { render, fireEvent, waitForMutation } = require('sexpr/test');

test('todo toggle', async () => {
  const tree = render('(use "todo-item" :id "1" :text "Buy milk" :done #f)');

  expect(tree.query('.todo-item')).not.toHaveClass('done');

  fireEvent.click(tree.query('.check'));
  await waitForMutation('#todo-1');

  expect(tree.query('.todo-item')).toHaveClass('done');
});

• render() works in Node (using JSDOM) or browser
• Snapshot testing: compare AST snapshots, not HTML string snapshots
• Mutation assertions: expectMutation(swap!(...)) to test server responses

7.2 Documentation Site

Built with sexpr.js itself (dogfooding). Includes:

• Tutorial: build a todo app from scratch
• API reference: every special form, mutation primitive, configuration option
• Cookbook: common patterns (modals, infinite scroll, real-time chat, auth flows)
• Interactive playground: edit S-expressions, see live DOM output
• Migration guide: "coming from HTMX" and "coming from React"

---

Technical Decisions

Why Not JSON?

JSON could represent the same tree structure. But:

• S-expressions are more compact for deeply nested structures (no commas, colons, or quotes on keys)
• The syntax is its own DSL — :keywords, symbols, and lists feel natural for document description
• Comments are supported (JSON has none)
• Human-writeable: developers will author .sexpr files directly
• Cultural signal: this is a Lisp-inspired project, and the syntax communicates the homoiconicity thesis immediately

Why Not Compile to WebAssembly?

Tempting for parser performance, but:

• JS engines already optimize parsing hot paths well
• WASM has overhead for DOM interaction (must cross the JS boundary anyway)
• Staying in pure JS means the library works everywhere JS does with zero build step
• Future option: WASM parser module for very large documents

Module Format

• ES modules (.mjs) as primary
• CommonJS (.cjs) build for older Node.js
• UMD build for <script> tag usage (the "drop it in and it works" story)
• TypeScript type definitions (.d.ts) shipped alongside

Bundle Size Target

• Core parser + renderer: < 5KB gzipped
• With mutation engine: < 8KB gzipped
• Full framework (router, forms, devtools hook): < 15KB gzipped

For comparison: HTMX is ~14KB, Alpine.js is ~15KB, Preact is ~3KB.

---

Milestones

Milestone	Target	Deliverable
M1: Parser	Week 2	parse(), serialize(), full test suite, benchmarks
M2: Client renderer	Week 4	renderToDOM(), styles, events — the v1 prototype, cleaned up
M3: Server renderer	Week 6	renderToString(), Express middleware, SSR bootstrap
M4: Mutations	Week 8	swap!, batch!, request!, class! — full hypermedia engine
M5: WebSocket	Week 10	Real-time server push, reconnection, protocol spec
M6: Components	Week 12	Component system, slots, .sexpr file format, registry
M7: Diffing	Week 16	VTree diff, keyed reconciliation, hydration
M8: CLI & DX	Week 20	sexpr init/dev/build, hot reload, VS Code extension
M9: Ecosystem	Week 24	Router, forms, plugins, service worker caching
M10: Launch	Week 28	Docs site, npm publish, example apps, announcement

---

Example Apps (for launch)

1. Todo MVC — the classic benchmark, fully server-driven
2. Real-time chat — WebSocket mutations, presence indicators
3. Dashboard — data tables, charts, polling, search
4. Blog — SSR, routing, SEO, markdown integration
5. E-commerce product page — forms, validation, cart mutations

---

Open Questions

1. Macro system? Should the server support defmacro-style macros for transforming S-expressions before rendering? This adds power but also complexity and potential security concerns.

2. TypeScript integration? Should component params be typed? Could generate TS interfaces from (meta :params ...) declarations.

3. Compilation? An optional ahead-of-time compiler could pre-parse .sexpr files into JS AST constructors, eliminating parse time at runtime. Worth the complexity?

4. CSS-in-sexpr or external stylesheets? The current approach co-locates styles. Should there also be a way to import .css files directly, or should all styling go through the S-expression syntax?

5. Interop with existing HTML? Can you embed an S-expression island inside an existing HTML page (like Astro islands)? Useful for incremental adoption.

6. Binary wire format? A compact binary encoding of the AST (like MessagePack for S-expressions) could reduce bandwidth for large pages. Worth the complexity vs. gzip?

---

Name Candidates

• sexpr.js — direct, memorable, says what it is
• sdom — S-expression DOM
• paren — the defining character
• lispr — Lisp + render
• homoDOM — homoiconic DOM

---

The document is the program. The program is the document.