Plan: SX CI Pipeline — build/test/deploy in s-expressions
Pipeline definitions as .sx files evaluated by a minimal Python runner. CI primitives (shell-run, docker-build, git-diff-files) are boundary-declared IO, only available to the runner. Steps are defcomp components composable by nesting. Fixes pre-existing unclosed parens in isomorphic roadmap section. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
278
.claude/plans/sx-ci-pipeline.md
Normal file
278
.claude/plans/sx-ci-pipeline.md
Normal file
@@ -0,0 +1,278 @@
|
||||
# SX CI Pipeline — Build/Test/Deploy in S-Expressions
|
||||
|
||||
## Context
|
||||
|
||||
Rose Ash currently uses shell scripts for CI:
|
||||
- `deploy.sh` — auto-detect changed apps via git diff, build Docker images, push to registry, restart services
|
||||
- `dev.sh` — start/stop dev environment, run tests
|
||||
- `test/Dockerfile.unit` — headless test runner in Docker
|
||||
- Tailwind CSS build via CLI command with v3 config
|
||||
- SX bootstrapping via `python bootstrap_js.py`, `python bootstrap_py.py`, `python bootstrap_test.py`
|
||||
|
||||
These work, but they are opaque shell scripts with imperative logic, no reuse, and no relationship to the language the application is written in. The CI pipeline is the one remaining piece of Rose Ash infrastructure that is not expressed in s-expressions.
|
||||
|
||||
## Goal
|
||||
|
||||
Replace the shell-based CI with pipeline definitions written in SX. The pipeline runner is a minimal Python CLI that evaluates `.sx` files using the SX spec. Pipeline steps are s-expressions. Conditionals, composition, and reuse are the same language constructs used everywhere else in the codebase.
|
||||
|
||||
This is not a generic CI framework — it is a project-specific pipeline that happens to be written in SX, proving the "one representation for everything" claim from the essays.
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Same language** — Pipeline definitions use the same SX syntax, parser, and evaluator as the application
|
||||
2. **Boundary-enforced** — CI primitives (shell, docker, git) are IO primitives declared in boundary.sx, sandboxed like everything else
|
||||
3. **Composable** — Pipeline steps are components; complex pipelines compose them by nesting
|
||||
4. **Self-testing** — The pipeline can run the SX spec tests as a pipeline step, using the same spec that defines the pipeline language
|
||||
5. **Incremental** — Each phase is independently useful; shell scripts remain as fallback
|
||||
|
||||
## Implementation
|
||||
|
||||
### Phase 1: CI Spec + Runner
|
||||
|
||||
#### `shared/sx/ref/ci.sx` — CI primitives spec
|
||||
|
||||
Declare CI-specific IO primitives in the boundary:
|
||||
|
||||
```lisp
|
||||
;; Shell execution
|
||||
(define shell-run ...) ;; (shell-run "pytest shared/ -v") → {:exit 0 :stdout "..." :stderr "..."}
|
||||
(define shell-run! ...) ;; Like shell-run but throws on non-zero exit
|
||||
|
||||
;; Docker
|
||||
(define docker-build ...) ;; (docker-build :file "sx/Dockerfile" :tag "registry/sx:latest" :context ".")
|
||||
(define docker-push ...) ;; (docker-push "registry/sx:latest")
|
||||
(define docker-restart ...) ;; (docker-restart "coop_sx_docs")
|
||||
|
||||
;; Git
|
||||
(define git-diff-files ...) ;; (git-diff-files "HEAD~1" "HEAD") → ("shared/sx/parser.py" "sx/sx/essays.sx")
|
||||
(define git-branch ...) ;; (git-branch) → "macros"
|
||||
|
||||
;; Filesystem
|
||||
(define file-exists? ...) ;; (file-exists? "sx/Dockerfile") → true
|
||||
(define read-file ...) ;; (read-file "version.txt") → "1.2.3"
|
||||
|
||||
;; Pipeline control
|
||||
(define log-step ...) ;; (log-step "Building sx_docs") — formatted output
|
||||
(define fail! ...) ;; (fail! "Unit tests failed") — abort pipeline
|
||||
```
|
||||
|
||||
#### `sx-ci` — CLI runner
|
||||
|
||||
Minimal Python script (~100 lines):
|
||||
1. Loads SX evaluator (sx_ref.py)
|
||||
2. Registers CI IO primitives (subprocess, docker SDK, git)
|
||||
3. Evaluates the pipeline `.sx` file
|
||||
4. Exit code = pipeline result
|
||||
|
||||
```bash
|
||||
python -m shared.sx.ci pipeline/deploy.sx
|
||||
python -m shared.sx.ci pipeline/test.sx
|
||||
```
|
||||
|
||||
### Phase 2: Pipeline Definitions
|
||||
|
||||
#### `pipeline/services.sx` — Service registry (data)
|
||||
|
||||
```lisp
|
||||
(define services
|
||||
(list
|
||||
{:name "blog" :dir "blog" :compose "blog" :port 8001}
|
||||
{:name "market" :dir "market" :compose "market" :port 8002}
|
||||
{:name "cart" :dir "cart" :compose "cart" :port 8003}
|
||||
{:name "events" :dir "events" :compose "events" :port 8004}
|
||||
{:name "federation" :dir "federation" :compose "federation" :port 8005}
|
||||
{:name "account" :dir "account" :compose "account" :port 8006}
|
||||
{:name "relations" :dir "relations" :compose "relations" :port 8008}
|
||||
{:name "likes" :dir "likes" :compose "likes" :port 8009}
|
||||
{:name "orders" :dir "orders" :compose "orders" :port 8010}
|
||||
{:name "sx_docs" :dir "sx" :compose "sx_docs" :port 8011}))
|
||||
|
||||
(define registry "registry.rose-ash.com:5000")
|
||||
```
|
||||
|
||||
#### `pipeline/steps.sx` — Reusable step components
|
||||
|
||||
```lisp
|
||||
(defcomp ~detect-changed (&key base)
|
||||
;; Returns list of services whose source dirs have changes
|
||||
(let ((files (git-diff-files (or base "HEAD~1") "HEAD")))
|
||||
(if (some (fn (f) (starts-with? f "shared/")) files)
|
||||
services ;; shared changed → rebuild all
|
||||
(filter (fn (svc)
|
||||
(some (fn (f) (starts-with? f (str (get svc "dir") "/"))) files))
|
||||
services))))
|
||||
|
||||
(defcomp ~unit-tests ()
|
||||
(log-step "Running unit tests")
|
||||
(shell-run! "docker build -f test/Dockerfile.unit -t rose-ash-test-unit:latest . -q")
|
||||
(shell-run! "docker run --rm rose-ash-test-unit:latest"))
|
||||
|
||||
(defcomp ~sx-spec-tests ()
|
||||
(log-step "Running SX spec tests")
|
||||
(shell-run! "cd shared/sx/ref && python bootstrap_test.py")
|
||||
(shell-run! "node shared/sx/ref/test_sx_ref.js"))
|
||||
|
||||
(defcomp ~bootstrap-check ()
|
||||
(log-step "Checking bootstrapped files are up to date")
|
||||
;; Rebootstrap and check for diff
|
||||
(shell-run! "python shared/sx/ref/bootstrap_js.py")
|
||||
(shell-run! "python shared/sx/ref/bootstrap_py.py")
|
||||
(let ((diff (shell-run "git diff --name-only shared/static/scripts/sx-ref.js shared/sx/ref/sx_ref.py")))
|
||||
(when (not (= (get diff "stdout") ""))
|
||||
(fail! "Bootstrapped files are stale — rebootstrap and commit"))))
|
||||
|
||||
(defcomp ~tailwind-check ()
|
||||
(log-step "Checking tw.css is up to date")
|
||||
(shell-run! "cat <<'CSS' | npx tailwindcss -i /dev/stdin -o /tmp/tw-check.css --minify -c shared/static/styles/tailwind.config.js\n@tailwind base;\n@tailwind components;\n@tailwind utilities;\nCSS")
|
||||
(let ((diff (shell-run "diff shared/static/styles/tw.css /tmp/tw-check.css")))
|
||||
(when (not (= (get diff "exit") 0))
|
||||
(log-step "WARNING: tw.css may be stale"))))
|
||||
|
||||
(defcomp ~build-service (&key service)
|
||||
(let ((name (get service "name"))
|
||||
(dir (get service "dir"))
|
||||
(tag (str registry "/" name ":latest")))
|
||||
(log-step (str "Building " name))
|
||||
(docker-build :file (str dir "/Dockerfile") :tag tag :context ".")
|
||||
(docker-push tag)))
|
||||
|
||||
(defcomp ~restart-service (&key service)
|
||||
(let ((name (get service "compose")))
|
||||
(log-step (str "Restarting coop_" name))
|
||||
(docker-restart (str "coop_" name))))
|
||||
```
|
||||
|
||||
#### `pipeline/test.sx` — Test pipeline
|
||||
|
||||
```lisp
|
||||
(load "pipeline/steps.sx")
|
||||
|
||||
(do
|
||||
(~unit-tests)
|
||||
(~sx-spec-tests)
|
||||
(~bootstrap-check)
|
||||
(~tailwind-check)
|
||||
(log-step "All checks passed"))
|
||||
```
|
||||
|
||||
#### `pipeline/deploy.sx` — Deploy pipeline
|
||||
|
||||
```lisp
|
||||
(load "pipeline/services.sx")
|
||||
(load "pipeline/steps.sx")
|
||||
|
||||
(let ((targets (if (= (length ARGS) 0)
|
||||
(~detect-changed :base "HEAD~1")
|
||||
(filter (fn (svc) (some (fn (a) (= a (get svc "name"))) ARGS)) services))))
|
||||
(when (= (length targets) 0)
|
||||
(log-step "No changes detected")
|
||||
(exit 0))
|
||||
|
||||
(log-step (str "Deploying: " (join " " (map (fn (s) (get s "name")) targets))))
|
||||
|
||||
;; Tests first
|
||||
(~unit-tests)
|
||||
(~sx-spec-tests)
|
||||
|
||||
;; Build and push
|
||||
(for-each (fn (svc) (~build-service :service svc)) targets)
|
||||
|
||||
;; Restart
|
||||
(for-each (fn (svc) (~restart-service :service svc)) targets)
|
||||
|
||||
(log-step "Deploy complete"))
|
||||
```
|
||||
|
||||
### Phase 3: Boundary Integration
|
||||
|
||||
Add CI primitives to `boundary.sx`:
|
||||
|
||||
```lisp
|
||||
;; CI primitives (pipeline runner only — not available in web context)
|
||||
(io-primitive shell-run (command) -> dict)
|
||||
(io-primitive shell-run! (command) -> dict)
|
||||
(io-primitive docker-build (&key file tag context) -> nil)
|
||||
(io-primitive docker-push (tag) -> nil)
|
||||
(io-primitive docker-restart (service) -> nil)
|
||||
(io-primitive git-diff-files (base head) -> list)
|
||||
(io-primitive git-branch () -> string)
|
||||
(io-primitive file-exists? (path) -> boolean)
|
||||
(io-primitive read-file (path) -> string)
|
||||
(io-primitive log-step (message) -> nil)
|
||||
(io-primitive fail! (message) -> nil)
|
||||
```
|
||||
|
||||
These are only registered by the CI runner, never by the web app. The boundary enforces that web components cannot call `shell-run!`.
|
||||
|
||||
### Phase 4: Bootstrap + Runner Implementation
|
||||
|
||||
#### `shared/sx/ci.py` — Runner module
|
||||
|
||||
```python
|
||||
"""SX CI pipeline runner.
|
||||
|
||||
Usage: python -m shared.sx.ci pipeline/deploy.sx [args...]
|
||||
"""
|
||||
import sys, subprocess, os
|
||||
from .ref.sx_ref import evaluate, parse, create_env
|
||||
from .ref.boundary_parser import parse_boundary
|
||||
|
||||
def register_ci_primitives(env):
|
||||
"""Register CI IO primitives into the evaluation environment."""
|
||||
# shell-run, docker-build, git-diff-files, etc.
|
||||
# Each is a thin Python wrapper around subprocess/docker SDK
|
||||
...
|
||||
|
||||
def main():
|
||||
pipeline_file = sys.argv[1]
|
||||
args = sys.argv[2:]
|
||||
|
||||
env = create_env()
|
||||
register_ci_primitives(env)
|
||||
env_set(env, "ARGS", args)
|
||||
|
||||
with open(pipeline_file) as f:
|
||||
source = f.read()
|
||||
|
||||
result = evaluate(parse(source), env)
|
||||
sys.exit(0 if result else 1)
|
||||
```
|
||||
|
||||
### Phase 5: Documentation + Essay Section
|
||||
|
||||
- Add a section to the "No Alternative" essay about CI pipelines as proof of universality
|
||||
- Add a plan page at `/plans/sx-ci` documenting the pipeline architecture
|
||||
- The pipeline definitions themselves serve as examples of SX beyond web rendering
|
||||
|
||||
## Files
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `shared/sx/ref/ci.sx` | **NEW** — CI primitive declarations |
|
||||
| `shared/sx/ci.py` | **NEW** — Pipeline runner (~150 lines) |
|
||||
| `shared/sx/ci_primitives.py` | **NEW** — CI IO primitive implementations |
|
||||
| `pipeline/services.sx` | **NEW** — Service registry data |
|
||||
| `pipeline/steps.sx` | **NEW** — Reusable pipeline step components |
|
||||
| `pipeline/test.sx` | **NEW** — Test pipeline |
|
||||
| `pipeline/deploy.sx` | **NEW** — Deploy pipeline |
|
||||
| `shared/sx/ref/boundary.sx` | Add CI primitive declarations |
|
||||
| `sx/sx/plans.sx` | Add plan page |
|
||||
| `sx/sx/essays.sx` | Add CI section to "No Alternative" |
|
||||
|
||||
## Verification
|
||||
|
||||
1. `python -m shared.sx.ci pipeline/test.sx` — runs all checks, same results as manual
|
||||
2. `python -m shared.sx.ci pipeline/deploy.sx blog` — builds and deploys blog only
|
||||
3. `python -m shared.sx.ci pipeline/deploy.sx` — auto-detects changes, deploys affected services
|
||||
4. Pipeline output is readable: step names, pass/fail, timing
|
||||
5. Shell scripts remain as fallback — nothing is deleted
|
||||
|
||||
## Order of Implementation
|
||||
|
||||
1. Phase 1 first — get the runner evaluating simple pipeline files
|
||||
2. Phase 2 — define the actual pipeline steps
|
||||
3. Phase 3 — formal boundary declarations
|
||||
4. Phase 4 — full runner with all CI primitives
|
||||
5. Phase 5 — documentation and essay content
|
||||
|
||||
Each phase is independently committable and testable.
|
||||
Reference in New Issue
Block a user