Plan: SX CI Pipeline — build/test/deploy in s-expressions

Pipeline definitions as .sx files evaluated by a minimal Python runner.
CI primitives (shell-run, docker-build, git-diff-files) are boundary-declared
IO, only available to the runner. Steps are defcomp components composable
by nesting. Fixes pre-existing unclosed parens in isomorphic roadmap section.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-07 13:36:50 +00:00
parent d618530f29
commit fab9bffc49
4 changed files with 413 additions and 2 deletions

View File

@@ -0,0 +1,278 @@
# SX CI Pipeline — Build/Test/Deploy in S-Expressions
## Context
Rose Ash currently uses shell scripts for CI:
- `deploy.sh` — auto-detect changed apps via git diff, build Docker images, push to registry, restart services
- `dev.sh` — start/stop dev environment, run tests
- `test/Dockerfile.unit` — headless test runner in Docker
- Tailwind CSS build via CLI command with v3 config
- SX bootstrapping via `python bootstrap_js.py`, `python bootstrap_py.py`, `python bootstrap_test.py`
These work, but they are opaque shell scripts with imperative logic, no reuse, and no relationship to the language the application is written in. The CI pipeline is the one remaining piece of Rose Ash infrastructure that is not expressed in s-expressions.
## Goal
Replace the shell-based CI with pipeline definitions written in SX. The pipeline runner is a minimal Python CLI that evaluates `.sx` files using the SX spec. Pipeline steps are s-expressions. Conditionals, composition, and reuse are the same language constructs used everywhere else in the codebase.
This is not a generic CI framework — it is a project-specific pipeline that happens to be written in SX, proving the "one representation for everything" claim from the essays.
## Design Principles
1. **Same language** — Pipeline definitions use the same SX syntax, parser, and evaluator as the application
2. **Boundary-enforced** — CI primitives (shell, docker, git) are IO primitives declared in boundary.sx, sandboxed like everything else
3. **Composable** — Pipeline steps are components; complex pipelines compose them by nesting
4. **Self-testing** — The pipeline can run the SX spec tests as a pipeline step, using the same spec that defines the pipeline language
5. **Incremental** — Each phase is independently useful; shell scripts remain as fallback
## Implementation
### Phase 1: CI Spec + Runner
#### `shared/sx/ref/ci.sx` — CI primitives spec
Declare CI-specific IO primitives in the boundary:
```lisp
;; Shell execution
(define shell-run ...) ;; (shell-run "pytest shared/ -v") → {:exit 0 :stdout "..." :stderr "..."}
(define shell-run! ...) ;; Like shell-run but throws on non-zero exit
;; Docker
(define docker-build ...) ;; (docker-build :file "sx/Dockerfile" :tag "registry/sx:latest" :context ".")
(define docker-push ...) ;; (docker-push "registry/sx:latest")
(define docker-restart ...) ;; (docker-restart "coop_sx_docs")
;; Git
(define git-diff-files ...) ;; (git-diff-files "HEAD~1" "HEAD") → ("shared/sx/parser.py" "sx/sx/essays.sx")
(define git-branch ...) ;; (git-branch) → "macros"
;; Filesystem
(define file-exists? ...) ;; (file-exists? "sx/Dockerfile") → true
(define read-file ...) ;; (read-file "version.txt") → "1.2.3"
;; Pipeline control
(define log-step ...) ;; (log-step "Building sx_docs") — formatted output
(define fail! ...) ;; (fail! "Unit tests failed") — abort pipeline
```
#### `sx-ci` — CLI runner
Minimal Python script (~100 lines):
1. Loads SX evaluator (sx_ref.py)
2. Registers CI IO primitives (subprocess, docker SDK, git)
3. Evaluates the pipeline `.sx` file
4. Exit code = pipeline result
```bash
python -m shared.sx.ci pipeline/deploy.sx
python -m shared.sx.ci pipeline/test.sx
```
### Phase 2: Pipeline Definitions
#### `pipeline/services.sx` — Service registry (data)
```lisp
(define services
(list
{:name "blog" :dir "blog" :compose "blog" :port 8001}
{:name "market" :dir "market" :compose "market" :port 8002}
{:name "cart" :dir "cart" :compose "cart" :port 8003}
{:name "events" :dir "events" :compose "events" :port 8004}
{:name "federation" :dir "federation" :compose "federation" :port 8005}
{:name "account" :dir "account" :compose "account" :port 8006}
{:name "relations" :dir "relations" :compose "relations" :port 8008}
{:name "likes" :dir "likes" :compose "likes" :port 8009}
{:name "orders" :dir "orders" :compose "orders" :port 8010}
{:name "sx_docs" :dir "sx" :compose "sx_docs" :port 8011}))
(define registry "registry.rose-ash.com:5000")
```
#### `pipeline/steps.sx` — Reusable step components
```lisp
(defcomp ~detect-changed (&key base)
;; Returns list of services whose source dirs have changes
(let ((files (git-diff-files (or base "HEAD~1") "HEAD")))
(if (some (fn (f) (starts-with? f "shared/")) files)
services ;; shared changed → rebuild all
(filter (fn (svc)
(some (fn (f) (starts-with? f (str (get svc "dir") "/"))) files))
services))))
(defcomp ~unit-tests ()
(log-step "Running unit tests")
(shell-run! "docker build -f test/Dockerfile.unit -t rose-ash-test-unit:latest . -q")
(shell-run! "docker run --rm rose-ash-test-unit:latest"))
(defcomp ~sx-spec-tests ()
(log-step "Running SX spec tests")
(shell-run! "cd shared/sx/ref && python bootstrap_test.py")
(shell-run! "node shared/sx/ref/test_sx_ref.js"))
(defcomp ~bootstrap-check ()
(log-step "Checking bootstrapped files are up to date")
;; Rebootstrap and check for diff
(shell-run! "python shared/sx/ref/bootstrap_js.py")
(shell-run! "python shared/sx/ref/bootstrap_py.py")
(let ((diff (shell-run "git diff --name-only shared/static/scripts/sx-ref.js shared/sx/ref/sx_ref.py")))
(when (not (= (get diff "stdout") ""))
(fail! "Bootstrapped files are stale — rebootstrap and commit"))))
(defcomp ~tailwind-check ()
(log-step "Checking tw.css is up to date")
(shell-run! "cat <<'CSS' | npx tailwindcss -i /dev/stdin -o /tmp/tw-check.css --minify -c shared/static/styles/tailwind.config.js\n@tailwind base;\n@tailwind components;\n@tailwind utilities;\nCSS")
(let ((diff (shell-run "diff shared/static/styles/tw.css /tmp/tw-check.css")))
(when (not (= (get diff "exit") 0))
(log-step "WARNING: tw.css may be stale"))))
(defcomp ~build-service (&key service)
(let ((name (get service "name"))
(dir (get service "dir"))
(tag (str registry "/" name ":latest")))
(log-step (str "Building " name))
(docker-build :file (str dir "/Dockerfile") :tag tag :context ".")
(docker-push tag)))
(defcomp ~restart-service (&key service)
(let ((name (get service "compose")))
(log-step (str "Restarting coop_" name))
(docker-restart (str "coop_" name))))
```
#### `pipeline/test.sx` — Test pipeline
```lisp
(load "pipeline/steps.sx")
(do
(~unit-tests)
(~sx-spec-tests)
(~bootstrap-check)
(~tailwind-check)
(log-step "All checks passed"))
```
#### `pipeline/deploy.sx` — Deploy pipeline
```lisp
(load "pipeline/services.sx")
(load "pipeline/steps.sx")
(let ((targets (if (= (length ARGS) 0)
(~detect-changed :base "HEAD~1")
(filter (fn (svc) (some (fn (a) (= a (get svc "name"))) ARGS)) services))))
(when (= (length targets) 0)
(log-step "No changes detected")
(exit 0))
(log-step (str "Deploying: " (join " " (map (fn (s) (get s "name")) targets))))
;; Tests first
(~unit-tests)
(~sx-spec-tests)
;; Build and push
(for-each (fn (svc) (~build-service :service svc)) targets)
;; Restart
(for-each (fn (svc) (~restart-service :service svc)) targets)
(log-step "Deploy complete"))
```
### Phase 3: Boundary Integration
Add CI primitives to `boundary.sx`:
```lisp
;; CI primitives (pipeline runner only — not available in web context)
(io-primitive shell-run (command) -> dict)
(io-primitive shell-run! (command) -> dict)
(io-primitive docker-build (&key file tag context) -> nil)
(io-primitive docker-push (tag) -> nil)
(io-primitive docker-restart (service) -> nil)
(io-primitive git-diff-files (base head) -> list)
(io-primitive git-branch () -> string)
(io-primitive file-exists? (path) -> boolean)
(io-primitive read-file (path) -> string)
(io-primitive log-step (message) -> nil)
(io-primitive fail! (message) -> nil)
```
These are only registered by the CI runner, never by the web app. The boundary enforces that web components cannot call `shell-run!`.
### Phase 4: Bootstrap + Runner Implementation
#### `shared/sx/ci.py` — Runner module
```python
"""SX CI pipeline runner.
Usage: python -m shared.sx.ci pipeline/deploy.sx [args...]
"""
import sys, subprocess, os
from .ref.sx_ref import evaluate, parse, create_env
from .ref.boundary_parser import parse_boundary
def register_ci_primitives(env):
"""Register CI IO primitives into the evaluation environment."""
# shell-run, docker-build, git-diff-files, etc.
# Each is a thin Python wrapper around subprocess/docker SDK
...
def main():
pipeline_file = sys.argv[1]
args = sys.argv[2:]
env = create_env()
register_ci_primitives(env)
env_set(env, "ARGS", args)
with open(pipeline_file) as f:
source = f.read()
result = evaluate(parse(source), env)
sys.exit(0 if result else 1)
```
### Phase 5: Documentation + Essay Section
- Add a section to the "No Alternative" essay about CI pipelines as proof of universality
- Add a plan page at `/plans/sx-ci` documenting the pipeline architecture
- The pipeline definitions themselves serve as examples of SX beyond web rendering
## Files
| File | Change |
|------|--------|
| `shared/sx/ref/ci.sx` | **NEW** — CI primitive declarations |
| `shared/sx/ci.py` | **NEW** — Pipeline runner (~150 lines) |
| `shared/sx/ci_primitives.py` | **NEW** — CI IO primitive implementations |
| `pipeline/services.sx` | **NEW** — Service registry data |
| `pipeline/steps.sx` | **NEW** — Reusable pipeline step components |
| `pipeline/test.sx` | **NEW** — Test pipeline |
| `pipeline/deploy.sx` | **NEW** — Deploy pipeline |
| `shared/sx/ref/boundary.sx` | Add CI primitive declarations |
| `sx/sx/plans.sx` | Add plan page |
| `sx/sx/essays.sx` | Add CI section to "No Alternative" |
## Verification
1. `python -m shared.sx.ci pipeline/test.sx` — runs all checks, same results as manual
2. `python -m shared.sx.ci pipeline/deploy.sx blog` — builds and deploys blog only
3. `python -m shared.sx.ci pipeline/deploy.sx` — auto-detects changes, deploys affected services
4. Pipeline output is readable: step names, pass/fail, timing
5. Shell scripts remain as fallback — nothing is deleted
## Order of Implementation
1. Phase 1 first — get the runner evaluating simple pipeline files
2. Phase 2 — define the actual pipeline steps
3. Phase 3 — formal boundary declarations
4. Phase 4 — full runner with all CI primitives
5. Phase 5 — documentation and essay content
Each phase is independently committable and testable.

View File

@@ -140,7 +140,9 @@
(dict :label "Glue Decoupling" :href "/plans/glue-decoupling"
:summary "Eliminate all cross-app model imports via glue service layer.")
(dict :label "Social Sharing" :href "/plans/social-sharing"
:summary "OAuth-based sharing to Facebook, Instagram, Threads, Twitter/X, LinkedIn, and Mastodon.")))
:summary "OAuth-based sharing to Facebook, Instagram, Threads, Twitter/X, LinkedIn, and Mastodon.")
(dict :label "SX CI Pipeline" :href "/plans/sx-ci"
:summary "Build, test, and deploy in s-expressions — CI pipelines as SX components.")))
(define bootstrappers-nav-items (list
(dict :label "Overview" :href "/bootstrappers/")

View File

@@ -2014,4 +2014,134 @@
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ref/suspense.sx")
(td :class "px-3 py-2 text-stone-700" "Streaming/suspension (new)")
(td :class "px-3 py-2 text-stone-600" "5"))))))))
(td :class "px-3 py-2 text-stone-600" "5")))))))))))))
;; ---------------------------------------------------------------------------
;; SX CI Pipeline
;; ---------------------------------------------------------------------------
(defcomp ~plan-sx-ci-content ()
(~doc-page :title "SX CI Pipeline"
(p :class "text-stone-500 text-sm italic mb-8"
"Build, test, and deploy Rose Ash using the same language the application is written in.")
(~doc-section :title "Context" :id "context"
(p :class "text-stone-600"
"Rose Ash currently uses shell scripts for CI: " (code "deploy.sh") " auto-detects changed services via git diff, builds Docker images, pushes to the registry, and restarts Swarm services. " (code "dev.sh") " starts the dev environment and runs tests. These work, but they are opaque imperative scripts with no reuse, no composition, and no relationship to SX.")
(p :class "text-stone-600"
"The CI pipeline is the last piece of infrastructure not expressed in s-expressions. Fixing that completes the \"one representation for everything\" claim — the same language that defines the spec, the components, the pages, the essays, and the deployment config also defines the build pipeline."))
(~doc-section :title "Design" :id "design"
(p :class "text-stone-600"
"Pipeline definitions are " (code ".sx") " files. A minimal Python CLI runner evaluates them using " (code "sx_ref.py") ". CI-specific IO primitives (shell execution, Docker, git) are boundary-declared and only available to the pipeline runner — never to web components.")
(~doc-code :code (highlight ";; pipeline/deploy.sx\n(let ((targets (if (= (length ARGS) 0)\n (~detect-changed :base \"HEAD~1\")\n (filter (fn (svc) (some (fn (a) (= a (get svc \"name\"))) ARGS))\n services))))\n (when (= (length targets) 0)\n (log-step \"No changes detected\")\n (exit 0))\n\n (log-step (str \"Deploying: \" (join \" \" (map (fn (s) (get s \"name\")) targets))))\n\n ;; Tests first\n (~unit-tests)\n (~sx-spec-tests)\n\n ;; Build, push, restart\n (for-each (fn (svc) (~build-service :service svc)) targets)\n (for-each (fn (svc) (~restart-service :service svc)) targets)\n\n (log-step \"Deploy complete\"))" "lisp"))
(p :class "text-stone-600"
"Pipeline steps are components. " (code "~unit-tests") ", " (code "~build-service") ", " (code "~detect-changed") " are " (code "defcomp") " definitions that compose by nesting — the same mechanism used for page layouts, navigation, and every other piece of the system."))
(~doc-section :title "CI Primitives" :id "primitives"
(p :class "text-stone-600"
"New IO primitives declared in " (code "boundary.sx") ", implemented only in the CI runner context:")
(div :class "overflow-x-auto rounded border border-stone-200 mb-4"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "Primitive")
(th :class "px-3 py-2 font-medium text-stone-600" "Signature")
(th :class "px-3 py-2 font-medium text-stone-600" "Purpose")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shell-run")
(td :class "px-3 py-2 font-mono text-xs text-stone-600" "(command) -> dict")
(td :class "px-3 py-2 text-stone-700" "Execute shell command, return " (code "{:exit N :stdout \"...\" :stderr \"...\"}") ""))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shell-run!")
(td :class "px-3 py-2 font-mono text-xs text-stone-600" "(command) -> dict")
(td :class "px-3 py-2 text-stone-700" "Execute shell command, throw on non-zero exit"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "docker-build")
(td :class "px-3 py-2 font-mono text-xs text-stone-600" "(&key file tag context) -> nil")
(td :class "px-3 py-2 text-stone-700" "Build Docker image"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "docker-push")
(td :class "px-3 py-2 font-mono text-xs text-stone-600" "(tag) -> nil")
(td :class "px-3 py-2 text-stone-700" "Push image to registry"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "docker-restart")
(td :class "px-3 py-2 font-mono text-xs text-stone-600" "(service) -> nil")
(td :class "px-3 py-2 text-stone-700" "Restart Swarm service"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "git-diff-files")
(td :class "px-3 py-2 font-mono text-xs text-stone-600" "(base head) -> list")
(td :class "px-3 py-2 text-stone-700" "List changed files between commits"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "git-branch")
(td :class "px-3 py-2 font-mono text-xs text-stone-600" "() -> string")
(td :class "px-3 py-2 text-stone-700" "Current branch name"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "log-step")
(td :class "px-3 py-2 font-mono text-xs text-stone-600" "(message) -> nil")
(td :class "px-3 py-2 text-stone-700" "Formatted pipeline output"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "fail!")
(td :class "px-3 py-2 font-mono text-xs text-stone-600" "(message) -> nil")
(td :class "px-3 py-2 text-stone-700" "Abort pipeline with error")))))
(p :class "text-stone-600"
"The boundary system ensures these primitives are " (em "only") " available in the CI context. Web components cannot call " (code "shell-run!") " — the evaluator will refuse to resolve the symbol, just as it refuses to resolve any other unregistered IO primitive. The sandbox is structural, not a convention."))
(~doc-section :title "Reusable Steps" :id "steps"
(p :class "text-stone-600"
"Pipeline steps are components — same " (code "defcomp") " as UI components, same " (code "&key") " params, same composition by nesting:")
(~doc-code :code (highlight "(defcomp ~detect-changed (&key base)\n (let ((files (git-diff-files (or base \"HEAD~1\") \"HEAD\")))\n (if (some (fn (f) (starts-with? f \"shared/\")) files)\n services\n (filter (fn (svc)\n (some (fn (f) (starts-with? f (str (get svc \"dir\") \"/\"))) files))\n services))))\n\n(defcomp ~build-service (&key service)\n (let ((name (get service \"name\"))\n (tag (str registry \"/\" name \":latest\")))\n (log-step (str \"Building \" name))\n (docker-build :file (str (get service \"dir\") \"/Dockerfile\") :tag tag :context \".\")\n (docker-push tag)))\n\n(defcomp ~bootstrap-check ()\n (log-step \"Checking bootstrapped files are up to date\")\n (shell-run! \"python shared/sx/ref/bootstrap_js.py\")\n (shell-run! \"python shared/sx/ref/bootstrap_py.py\")\n (let ((diff (shell-run \"git diff --name-only shared/static/scripts/sx-ref.js shared/sx/ref/sx_ref.py\")))\n (when (not (= (get diff \"stdout\") \"\"))\n (fail! \"Bootstrapped files are stale — rebootstrap and commit\"))))" "lisp"))
(p :class "text-stone-600"
"Compare this to GitHub Actions YAML, where \"reuse\" means composite actions with " (code "uses:") " references, input/output mappings, shell script blocks inside YAML strings, and a " (a :href "https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions" :class "text-violet-600 hover:underline" "100-page syntax reference") ". SX pipeline reuse is function composition. That is all it has ever been."))
(~doc-section :title "Pipelines" :id "pipelines"
(p :class "text-stone-600"
"Two primary pipelines, each a single " (code ".sx") " file:")
(div :class "space-y-4"
(div :class "rounded border border-stone-200 p-4"
(h4 :class "font-semibold text-stone-700 mb-2" "pipeline/test.sx")
(p :class "text-sm text-stone-600" "Unit tests, SX spec tests (Python + Node), bootstrap staleness check, Tailwind CSS check. Run locally or in CI.")
(p :class "text-sm font-mono text-violet-700 mt-1" "python -m shared.sx.ci pipeline/test.sx"))
(div :class "rounded border border-stone-200 p-4"
(h4 :class "font-semibold text-stone-700 mb-2" "pipeline/deploy.sx")
(p :class "text-sm text-stone-600" "Auto-detect changed services (or accept explicit args), run tests, build Docker images, push to registry, restart Swarm services.")
(p :class "text-sm font-mono text-violet-700 mt-1" "python -m shared.sx.ci pipeline/deploy.sx blog market"))))
(~doc-section :title "Why this matters" :id "why"
(p :class "text-stone-600"
"CI pipelines are the strongest test case for \"one representation for everything.\" GitHub Actions, GitLab CI, CircleCI — all use YAML. YAML is not a programming language. So every CI system reinvents conditionals (" (code "if:") " expressions evaluated as strings), iteration (" (code "matrix:") " strategies), composition (" (code "uses:") " references with input/output schemas), and error handling (" (code "continue-on-error:") " booleans) — all in a data format that was never designed for any of it.")
(p :class "text-stone-600"
"The result is a domain-specific language trapped inside YAML, with worse syntax than any language designed to be one. Every CI pipeline of sufficient complexity becomes a programming task performed in a notation that actively resists programming.")
(p :class "text-stone-600"
"SX pipelines use real conditionals, real functions, real composition, and real error handling — because SX is a real language. The pipeline definition and the application code are the same thing. An AI that can generate SX components can generate SX pipelines. A developer who reads SX pages can read SX deploys. The representation is universal."))
(~doc-section :title "Files" :id "files"
(div :class "overflow-x-auto rounded border border-stone-200"
(table :class "w-full text-left text-sm"
(thead (tr :class "border-b border-stone-200 bg-stone-100"
(th :class "px-3 py-2 font-medium text-stone-600" "File")
(th :class "px-3 py-2 font-medium text-stone-600" "Purpose")))
(tbody
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ci.py")
(td :class "px-3 py-2 text-stone-700" "Pipeline runner CLI (~150 lines)"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ci_primitives.py")
(td :class "px-3 py-2 text-stone-700" "CI IO primitive implementations"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "pipeline/services.sx")
(td :class "px-3 py-2 text-stone-700" "Service registry (data)"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "pipeline/steps.sx")
(td :class "px-3 py-2 text-stone-700" "Reusable step components"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "pipeline/test.sx")
(td :class "px-3 py-2 text-stone-700" "Test pipeline"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "pipeline/deploy.sx")
(td :class "px-3 py-2 text-stone-700" "Deploy pipeline"))
(tr :class "border-b border-stone-100"
(td :class "px-3 py-2 font-mono text-sm text-violet-700" "shared/sx/ref/boundary.sx")
(td :class "px-3 py-2 text-stone-700" "Add CI primitive declarations"))))))))

View File

@@ -512,6 +512,7 @@
"fragment-protocol" (~plan-fragment-protocol-content)
"glue-decoupling" (~plan-glue-decoupling-content)
"social-sharing" (~plan-social-sharing-content)
"sx-ci" (~plan-sx-ci-content)
:else (~plans-index-content)))
;; ---------------------------------------------------------------------------