|
|
|
|
@@ -0,0 +1,205 @@
|
|
|
|
|
;; SX Tools — Structural reading and editing tools for s-expression files.
|
|
|
|
|
;; Lives under the Applications section: /(applications.(sx-tools))
|
|
|
|
|
|
|
|
|
|
(defcomp ~sx-tools/overview-content ()
|
|
|
|
|
(~docs/page :title "SX Tools"
|
|
|
|
|
(p :class "text-stone-500 text-sm italic mb-8"
|
|
|
|
|
"A structural tree editor for s-expression files — because the thing that reads and edits code should understand the code as a tree, not as a sequence of characters.")
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "The problem" :id "problem"
|
|
|
|
|
(p "On 25 March 2026, the SX documentation site went blank. The home page stepper widget — a 310-line " (code "defisland") " — failed to render. The cause was a single extra closing parenthesis on line 222 of " (code "home-stepper.sx") ".")
|
|
|
|
|
(p "That parenthesis closed the " (code "letrec") " bindings list one level too early. Two function definitions — " (code "rebuild-preview") " and " (code "do-back") " — silently became body expressions instead of bindings. They were evaluated and discarded rather than bound in scope. Every subsequent reference to " (code "rebuild-preview") " raised " (code "Undefined symbol") ". The island rendered nothing. The page went white.")
|
|
|
|
|
(p "Finding this took an hour of systematic debugging: adding trace output to the OCaml " (code "env_has") " function, dumping the scope chain at the point of failure, counting keys in the letrec environment ("
|
|
|
|
|
(em "12 where there should have been 14") "), and finally writing a paren-depth tracer that walked the file character by character to find where the nesting diverged from expectation.")
|
|
|
|
|
(p "The fix was removing one character.")
|
|
|
|
|
(p "This is a class of bug, not an incident. S-expressions encode tree structure in linear text using matched delimiters. When those delimiters are wrong, the meaning of every subsequent expression changes. The error is silent — the parser succeeds, the evaluator runs, the wrong thing happens. The gap between the intended tree and the actual tree is invisible in the source."))
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "Why raw text fails" :id "why-text-fails"
|
|
|
|
|
(p "Claude Code reads " (code ".sx") " files as raw text and mentally reconstructs the tree structure by tracking bracket nesting. It does this imperfectly — especially in deep or wide trees where closing parentheses pile up and their correspondence to openers is lost. Consider the end of a complex island:")
|
|
|
|
|
(~docs/code :code "(set-cookie \"sx-home-stepper\" (freeze-to-sx \"home-stepper\"))))))))")
|
|
|
|
|
(p "Eight closing parentheses. Which one closes " (code "set-cookie") "? Which closes the " (code "fn") "? Which closes the binding pair? Which closes the letrec bindings list? Answering this requires counting backward through hundreds of lines. Counting is not what language models do well.")
|
|
|
|
|
(p "The same problem compounds when writing. Claude generates plausible-looking s-expression fragments that are structurally wrong — a paren added, a paren dropped, a level of nesting off. The " (code "str_replace") " tool makes this worse: replacing a string inside a deeply nested form can silently unbalance the surrounding structure in ways that are not visible until the file fails to parse — or worse, parses into a different tree."))
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "The fix: structural tools" :id "structural-tools"
|
|
|
|
|
(p "If Claude sees trees when the underlying thing is a tree, both the reading and writing problems disappear. Instead of raw text, Claude gets an annotated tree view with explicit paths:")
|
|
|
|
|
(~docs/code :code
|
|
|
|
|
(str "[0] (defisland ~home/stepper\n"
|
|
|
|
|
" [0,0] ~home/stepper\n"
|
|
|
|
|
" [0,1] ()\n"
|
|
|
|
|
" [0,2] (let\n"
|
|
|
|
|
" [0,2,0] let\n"
|
|
|
|
|
" [0,2,1] ((source ...) ... (code-tokens ...))\n"
|
|
|
|
|
" [0,2,2] (letrec\n"
|
|
|
|
|
" [0,2,2,0] letrec\n"
|
|
|
|
|
" [0,2,2,1] ((split-tag ...) ... (do-back ...))\n"
|
|
|
|
|
" [0,2,2,2] (freeze-scope ...)\n"
|
|
|
|
|
" [0,2,2,3] (let ((saved ...)) ...)\n"
|
|
|
|
|
" [0,2,2,4] (let ((parsed ...)) ...)\n"
|
|
|
|
|
" [0,2,2,5] (let ((_eff ...)) (div ...)))))"))
|
|
|
|
|
(p "The structural correspondence that is invisible in raw text is explicit here. Every node has a path. If " (code "rebuild-preview") " appears at " (code "[0,2,2,2]") " instead of " (code "[0,2,2,1,12]") ", it is immediately obvious that it is a body expression, not a letrec binding. The bug that took an hour to find would be visible on inspection.")
|
|
|
|
|
(p "For editing, Claude specifies tree operations rather than text replacements:")
|
|
|
|
|
(~docs/code :code
|
|
|
|
|
(str ";; Replace a node by path — the fragment is parsed before\n"
|
|
|
|
|
";; the file is touched. Bracket errors are impossible.\n"
|
|
|
|
|
"(replace-node \"home-stepper.sx\" [0,2,2,1,12]\n"
|
|
|
|
|
" \"(rebuild-preview (fn (target) ...))\")\n"
|
|
|
|
|
"\n"
|
|
|
|
|
";; Insert a new child at a specific position\n"
|
|
|
|
|
"(insert-child \"home-stepper.sx\" [0,2,2,1] 12\n"
|
|
|
|
|
" \"(new-function (fn () nil))\")\n"
|
|
|
|
|
"\n"
|
|
|
|
|
";; Delete a node — siblings adjust automatically\n"
|
|
|
|
|
"(delete-node \"home-stepper.sx\" [0,2,2,3])"))
|
|
|
|
|
(p "Every write operation parses the new fragment as a complete s-expression " (em "before") " navigating to the target path. If the fragment is malformed, the operation returns an error with the line and column of the parse failure. The source file is never left in a partially-edited state. Bracket mismatches become impossible by construction."))
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "Architecture" :id "architecture"
|
|
|
|
|
(p "SX Tools is an SX application. The comprehension and editing logic is written in SX, runs on the OCaml evaluator, and is exposed through two interfaces: an MCP server for Claude Code, and an interactive web application for the developer.")
|
|
|
|
|
(~docs/code :code
|
|
|
|
|
(str " ┌─────────────────┐\n"
|
|
|
|
|
" Claude Code ──▶ │ MCP Server │\n"
|
|
|
|
|
" │ (OCaml stdio) │\n"
|
|
|
|
|
" └────────┬─────────┘\n"
|
|
|
|
|
" │\n"
|
|
|
|
|
" ┌────────▼─────────┐\n"
|
|
|
|
|
" │ SX Tree Logic │\n"
|
|
|
|
|
" │ (comprehend.sx) │\n"
|
|
|
|
|
" │ (edit.sx) │\n"
|
|
|
|
|
" └────────┬─────────┘\n"
|
|
|
|
|
" │\n"
|
|
|
|
|
" ┌───────────────┼───────────────┐\n"
|
|
|
|
|
" ▼ ▼ ▼\n"
|
|
|
|
|
" .sx files Web tree editor Validation\n"
|
|
|
|
|
" (defisland) reports"))
|
|
|
|
|
(p "The " (strong "parser") " is " (code "sx-parse") " — the same parser that evaluates SX source. No new parser needed. Round-trip fidelity is inherited from the existing serializer.")
|
|
|
|
|
(p "The " (strong "tree logic") " lives in " (code "web/lib/tree-tools.sx") ". Pure functions: take a parsed tree, return annotated output or a modified tree. No IO, no side effects.")
|
|
|
|
|
(p "The " (strong "MCP server") " is a thin OCaml binary (" (code "hosts/ocaml/bin/mcp_tree.ml") ") that reads files, calls the SX tree functions via the OCaml bridge, and writes results. Stdio transport, JSON-RPC, single-threaded.")
|
|
|
|
|
(p "The " (strong "web editor") " is a " (code "defisland") " at " (code "/sx/(applications.(sx-tools))") " — the page you are reading. Interactive tree visualization with click-to-navigate, path display, and structural editing. Islands and signals make it reactive. It is both a tool and a demonstration of the SX platform."))
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "Comprehension tools" :id "comprehension"
|
|
|
|
|
(p "These are the tools Claude uses to " (em "understand") " structure before touching anything. They are read-only and have no side effects. They are not a convenience layer — they are as important as the editing tools.")
|
|
|
|
|
|
|
|
|
|
(h4 :class "font-semibold text-stone-700 mt-6 mb-2" "Annotated tree view")
|
|
|
|
|
(p "The primary comprehension tool. Every node gets its path label inline, making tree structure explicit. The structural correspondence that is invisible in raw text is readable by inspection, with no need to count brackets:")
|
|
|
|
|
(~docs/code :code
|
|
|
|
|
(str "[0] (defcomp ~card\n"
|
|
|
|
|
" [0,1] (&key title subtitle &rest children)\n"
|
|
|
|
|
" [0,2] (div :class \"card\"\n"
|
|
|
|
|
" [0,2,1] :class\n"
|
|
|
|
|
" [0,2,2] \"card\"\n"
|
|
|
|
|
" [0,2,3] (h2 title)\n"
|
|
|
|
|
" [0,2,4] (when subtitle (p subtitle))\n"
|
|
|
|
|
" [0,2,5] children))"))
|
|
|
|
|
|
|
|
|
|
(h4 :class "font-semibold text-stone-700 mt-6 mb-2" "Structural summary")
|
|
|
|
|
(p "A folded view for large files, showing shape without detail. Claude orients itself in a 300-line island, identifies the region it needs, then calls " (code "read-subtree") " on just that part:")
|
|
|
|
|
(~docs/code :code
|
|
|
|
|
(str "(defisland ~home/stepper [0]\n"
|
|
|
|
|
" (let [0,2]\n"
|
|
|
|
|
" ((source ...) (code-tokens ...)) [0,2,1]\n"
|
|
|
|
|
" (letrec [0,2,2]\n"
|
|
|
|
|
" ((split-tag ...) ... [0,2,2,1]\n"
|
|
|
|
|
" (rebuild-preview ...) [0,2,2,1,12]\n"
|
|
|
|
|
" (do-back ...)) [0,2,2,1,13]\n"
|
|
|
|
|
" (freeze-scope ...) [0,2,2,2]\n"
|
|
|
|
|
" (let ((_eff ...)) (div ...)))) [0,2,2,5]"))
|
|
|
|
|
|
|
|
|
|
(h4 :class "font-semibold text-stone-700 mt-6 mb-2" "Context view")
|
|
|
|
|
(p "Given a deep path, shows the enclosing chain back to the root. Essential for working deep in a tree without loading the entire file:")
|
|
|
|
|
(~docs/code :code
|
|
|
|
|
(str "Context for [0,2,2,1,12]:\n"
|
|
|
|
|
" [0] defisland ~home/stepper\n"
|
|
|
|
|
" [0,2] let ((source ...) ... (code-tokens ...))\n"
|
|
|
|
|
" [0,2,2] letrec ((split-tag ...) ...)\n"
|
|
|
|
|
" [0,2,2,1] bindings list (14 pairs)\n"
|
|
|
|
|
" → [0,2,2,1,12] (rebuild-preview (fn (target) ...))"))
|
|
|
|
|
|
|
|
|
|
(h4 :class "font-semibold text-stone-700 mt-6 mb-2" "Bracket-paired view")
|
|
|
|
|
(p "The raw source annotated with matched pair labels, for cases where Claude needs to see the actual syntax and verify bracket correspondence:")
|
|
|
|
|
(~docs/code :code
|
|
|
|
|
(str "(₁defcomp ~card (₂&key title subtitle &rest children)₂\n"
|
|
|
|
|
" (₃div :class \"card\"\n"
|
|
|
|
|
" (₄h2 title)₄\n"
|
|
|
|
|
" (₅when subtitle (₆p subtitle)₆)₅\n"
|
|
|
|
|
" children)₃)₁")))
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "Edit operations" :id "editing"
|
|
|
|
|
(p "All operations take a tree, perform the operation, and return either a new tree or a structured error. Nothing is mutated in place. File writing is a separate step that only happens after the edit succeeds.")
|
|
|
|
|
|
|
|
|
|
(table :class "min-w-full text-sm mb-6"
|
|
|
|
|
(thead
|
|
|
|
|
(tr
|
|
|
|
|
(th :class "text-left pr-4 pb-2 font-semibold text-stone-700" "Operation")
|
|
|
|
|
(th :class "text-left pb-2 font-semibold text-stone-700" "Description")))
|
|
|
|
|
(tbody :class "text-stone-600"
|
|
|
|
|
(tr (td :class "pr-4 py-1 font-mono text-xs" "replace-node") (td :class "py-1" "Replace the node at a path with new parsed source"))
|
|
|
|
|
(tr (td :class "pr-4 py-1 font-mono text-xs" "insert-child") (td :class "py-1" "Insert a new child at a specific index within a list"))
|
|
|
|
|
(tr (td :class "pr-4 py-1 font-mono text-xs" "delete-node") (td :class "py-1" "Remove a node — siblings shift to fill the gap"))
|
|
|
|
|
(tr (td :class "pr-4 py-1 font-mono text-xs" "wrap-node") (td :class "py-1" "Wrap a node in a new list — e.g. wrap expression in " (code "(when cond ...)")))
|
|
|
|
|
(tr (td :class "pr-4 py-1 font-mono text-xs" "validate") (td :class "py-1" "Check structural integrity — balanced parens, valid paths"))))
|
|
|
|
|
|
|
|
|
|
(p (strong "Fragment-first validation.") " Every write operation parses the new source fragment as a complete s-expression before navigating to the target path. If the fragment is malformed, the operation returns an error with the line and column of the parse failure. The source file is never touched in the failure path.")
|
|
|
|
|
(p (strong "Named paths.") " Index paths break when sibling nodes are inserted or deleted. Named paths — " (code "[Head \"letrec\", Head \"rebuild-preview\"]") " — survive structural edits and are more natural for Claude to reason about. Claude should prefer named paths; index paths are for mechanical follow-up after " (code "find-node") " has located a target."))
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "The protocol" :id "protocol"
|
|
|
|
|
(p "The tools only work if they are actually used. The MCP server must be accompanied by a protocol — enforced via " (code "CLAUDE.md") " — that prevents fallback to raw text editing.")
|
|
|
|
|
(~docs/code :code
|
|
|
|
|
(str ";; Before doing anything in an .sx file:\n"
|
|
|
|
|
";; 1. summarise → structural overview of the whole file\n"
|
|
|
|
|
";; 2. read-subtree → expand the region you intend to work in\n"
|
|
|
|
|
";; 3. get-context → understand the position of specific nodes\n"
|
|
|
|
|
";; 4. find-all → locate definitions or patterns by name\n"
|
|
|
|
|
"\n"
|
|
|
|
|
";; For every edit:\n"
|
|
|
|
|
";; 1. read-subtree → confirm the correct path\n"
|
|
|
|
|
";; 2. replace-node / insert-child / delete-node / wrap-node\n"
|
|
|
|
|
";; 3. validate → confirm structural integrity\n"
|
|
|
|
|
";; 4. read-subtree → verify the result\n"
|
|
|
|
|
"\n"
|
|
|
|
|
";; Never use str_replace on .sx files.\n"
|
|
|
|
|
";; Never proceed to an edit without first establishing\n"
|
|
|
|
|
";; where you are in the tree using the comprehension tools."))
|
|
|
|
|
(p "The comprehension-first discipline is the key insight. Claude cannot edit reliably what it does not understand reliably. The same parsed tree representation serves both needs — reading and writing are two sides of the same structural problem."))
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "Why SX, not OCaml" :id "why-sx"
|
|
|
|
|
(p "The original plan called for a pure OCaml implementation with " (code "angstrom") " parser combinators and a Wadler-Lindig pretty-printer. This is unnecessary. The SX ecosystem already has everything:")
|
|
|
|
|
(ul :class "space-y-2 text-stone-600"
|
|
|
|
|
(li (strong "Parser: ") (code "sx-parse") " already parses s-expressions correctly — it is the same parser the evaluator uses. No second parser to maintain, no divergence risk.")
|
|
|
|
|
(li (strong "Serializer: ") (code "sx-serialize") " already handles round-tripping. The existing serializer preserves structure.")
|
|
|
|
|
(li (strong "Tree operations: ") "Recursive list processing is what SX does best. Annotating a tree, folding to a summary, navigating by path — these are all natural " (code "map") "/" (code "reduce") "/" (code "filter") " operations on nested lists.")
|
|
|
|
|
(li (strong "Web UI: ") "The interactive tree editor is a " (code "defisland") " — signals for selection state, reactive DOM for the tree view, lakes for server-morphable content. The home stepper widget is proof this works.")
|
|
|
|
|
(li (strong "OCaml host: ") "The SX functions run on the OCaml evaluator. The MCP server is a thin OCaml wrapper around SX function calls. Native performance for the server, WASM for the browser — same codebase."))
|
|
|
|
|
(p "Writing the tree tools in SX means they can run in the browser (via the WASM evaluator) and on the server (via the OCaml kernel). The web editor and the MCP server share identical logic. There is one implementation, not two."))
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "Build plan" :id "build-plan"
|
|
|
|
|
(h4 :class "font-semibold text-stone-700 mt-6 mb-2" "Phase 1 — Tree comprehension functions")
|
|
|
|
|
(p "Implement " (code "annotate-tree") ", " (code "summarise") ", " (code "read-subtree") ", " (code "get-context") ", " (code "find-all") ", " (code "bracket-pairs") " as pure SX functions in " (code "web/lib/tree-tools.sx") ". Test against real project " (code ".sx") " files. Iterate on output formats until the output is genuinely easy for a language model to read — the format is load-bearing.")
|
|
|
|
|
|
|
|
|
|
(h4 :class "font-semibold text-stone-700 mt-6 mb-2" "Phase 2 — Edit operations")
|
|
|
|
|
(p "Implement " (code "replace-node") ", " (code "insert-child") ", " (code "delete-node") ", " (code "wrap-node") ", " (code "validate") " as pure SX functions. Fragment-first validation on all write operations. Test error paths exhaustively — error messages are part of the interface.")
|
|
|
|
|
|
|
|
|
|
(h4 :class "font-semibold text-stone-700 mt-6 mb-2" "Phase 3 — MCP server")
|
|
|
|
|
(p "Thin OCaml binary: stdio JSON-RPC, calls SX functions via the bridge, reads/writes files. Wire all comprehension and edit tools to MCP handlers. Manual testing with raw JSON.")
|
|
|
|
|
|
|
|
|
|
(h4 :class "font-semibold text-stone-700 mt-6 mb-2" "Phase 4 — Web editor")
|
|
|
|
|
(p (code "defisland ~sx-tools/tree-editor") " — interactive tree visualization on this page. Click a node to see its path, context, and siblings. Edit nodes through a structural interface. Islands and signals for reactivity. A tool and a demonstration.")
|
|
|
|
|
|
|
|
|
|
(h4 :class "font-semibold text-stone-700 mt-6 mb-2" "Phase 5 — Integration and iteration")
|
|
|
|
|
(p "Write the " (code "CLAUDE.md") " protocol. Run real tasks with Claude Code — both reading and editing. Observe which comprehension tools Claude actually reaches for. Observe where it still makes structural errors. Iterate on output formats and add any missing tools. The output formats deserve careful design based on observed behaviour, not just on what seems reasonable in advance."))
|
|
|
|
|
|
|
|
|
|
;; -----------------------------------------------------------------
|
|
|
|
|
(~docs/section :title "What changes" :id "what-changes"
|
|
|
|
|
(p "With SX Tools, the debugging session that found the home-stepper bug would not have happened. The workflow would have been:")
|
|
|
|
|
(ul :class "space-y-2 text-stone-600"
|
|
|
|
|
(li (code "summarise home-stepper.sx") " → see that " (code "rebuild-preview") " is at " (code "[0,4]") " (body expression) instead of " (code "[0,2,2,1,12]") " (letrec binding). The bug is visible on the summary.")
|
|
|
|
|
(li "Even without noticing the summary, " (code "validate home-stepper.sx") " would report the structural anomaly: 14 names in the letrec bindings list but only 12 binding pairs, with 2 bare expressions following.")
|
|
|
|
|
(li "The fix — " (code "delete-node") " to remove the extra paren, or " (code "wrap-node") " to restructure — would be a tree operation. No character-counting, no mental stack simulation, no risk of introducing a second paren error while fixing the first."))
|
|
|
|
|
(p "The gap between intended tree and actual tree stops being invisible. Claude sees trees, edits trees, and the brackets take care of themselves."))))
|
