rose-ash/sx/sx/plans/sx-pub.sx

;; ---------------------------------------------------------------------------
;; sx-pub: Federated SX Publishing Protocol
;; ---------------------------------------------------------------------------

(defcomp ~sx-pub/overview-content ()
  (~docs/page :title "sx-pub"

    (~docs/section :title "What It Is" :id "what"
      (p "A federated content publishing protocol built entirely in SX. No JSON. Content pinned to IPFS, referenced by CID. Blockchain-anchored provenance.")
      (p "One actor — " (code "sx") " — publishes SX specs, platform libraries, components, and documentation as content-addressed packages. Other sx-pub servers can follow, mirror, and refer to content by CID. Human-readable paths map to immutable IPFS addresses.")
      (div :class "rounded border border-violet-200 bg-violet-50 p-4 mt-4"
        (p :class "text-violet-900 font-medium" "The key insight")
        (p :class "text-violet-800" "IPFS is the database. SX is the wire format. defhandlers are the endpoints. The wire format is the programming language is the component system is the package manager.")))

    ;; -----------------------------------------------------------------------
    ;; Why ActivityPub Failed — and Why LISP Fixes It
    ;; -----------------------------------------------------------------------

    (~docs/section :title "Why ActivityPub Failed" :id "why"
      (p "ActivityPub's problems aren't bugs — they're consequences of a fundamental design error: " (strong "the wire format is dead data") ". JSON-LD activities describe what happened but can't " (em "do") " anything. Every consumer must independently interpret, render, and extend. The fediverse is N incompatible applications showing the same inert data differently.")

      (~docs/subsection :title "1. The Rendering Gap"
        (p "An ActivityPub Note looks like this:")
        (~docs/code :code "{\"type\": \"Note\",\n \"content\": \"<p>Hello <strong>world</strong></p>\",\n \"attributedTo\": \"https://example.com/users/alice\"}")
        (p "The content is an HTML string embedded in a JSON string. Two formats nested, neither evaluable. Every client — Mastodon, Pleroma, Misskey, Lemmy — parses this independently and wraps it in their own UI. A poll renders differently everywhere. An event looks different everywhere. There is no shared component model because the protocol has no concept of components.")
        (p "In sx-pub, the content " (em "is") " the component:")
        (~docs/code :code "(Note\n  :attributed-to \"https://pub.sx-web.org/pub/actor\"\n  :content (p \"Hello \" (strong \"world\")))")
        (p "No HTML-in-JSON. No parsing step. The receiver evaluates the s-expression and gets rendered output. The content carries its own rendering semantics."))

      (~docs/subsection :title "2. Inert Data vs. Evaluable Programs"
        (p "A JSON-LD activity is a description. It says \"Alice created a Note with this HTML.\" The receiving server must then decide: how do I display a Note? What if it has attachments? What about custom emoji? What about a poll? Every new feature requires every implementation to add rendering code.")
        (p "An SX activity is a " (em "program") ". It evaluates to its own UI. A poll isn't a JSON blob that each client renders differently — it's a " (code "(defcomp ~poll ...)") " that evaluates identically everywhere because the CEK machine has deterministic semantics. " (strong "The wire format is the renderer.")))

      (~docs/subsection :title "3. Extensions Are Impossible"
        (p "ActivityPub extensions (FEPs) add new JSON fields that most implementations ignore. Want to federate a recipe, a calendar event, a code snippet with syntax highlighting? Write a spec, wait years for implementations to support it, accept that most never will.")
        (p "In sx-pub, extensions are just components. Publish " (code "(defcomp ~recipe ...)") " to IPFS. Anyone who follows you gets the component. Their server can evaluate it immediately — " (strong "no implementation changes needed") ". The extension mechanism is the same as the content mechanism: pin an s-expression, reference it by CID."))

      (~docs/subsection :title "4. No Content Integrity"
        (p "ActivityPub has no content addressing. A server can silently edit a federated post. HTTP signatures verify the " (em "transport") " but not the " (em "content") ". There's no way to prove that what you see is what was originally published.")
        (p "sx-pub content is IPFS-pinned and referenced by CID (content hash). The CID " (em "is") " the identity. Mutate a single character and the CID changes. Merkle trees of CIDs get anchored to Bitcoin via OpenTimestamps — cryptographic proof of what was published and when."))

      (~docs/subsection :title "5. The N-Client Problem"
        (p "Because AP content is inert, every client must build its own renderer, interaction model, and extension system. The result: Mastodon is one application, Pleroma is another, Misskey is another. They can exchange data, but the " (em "experience") " is entirely determined by the client. Rich content degrades to plain text when the client doesn't understand it.")
        (p "sx-pub eliminates this entirely. Content carries its rendering logic. A server that receives " (code "(defcomp ~interactive-chart ...)") " can render it correctly without any prior knowledge of charts — because the component is self-contained, sandboxed (boundary enforcement prevents IO in pure components), and deterministic. " (strong "Every server renders the same content the same way.")))

      (~docs/subsection :title "6. No Dependency System"
        (p "AP has no concept of shared code. If ten ActivityPub projects all need a Markdown parser, ten teams write ten Markdown parsers. There's no mechanism to publish reusable logic that travels between servers.")
        (p "sx-pub's " (code ":requires") " field on published documents declares CID dependencies. A component that uses a chart library says " (code ":requires (list \"bafy...chart-lib\")") ". Pin the dependency tree and everything works offline. " (strong "This is a package manager built into the federation protocol.") " No npm, no crates.io, no central registry — just content-addressed s-expressions on IPFS."))

      (~docs/subsection :title "7. The @context Illusion"
        (p "JSON-LD @context was supposed to provide semantic interoperability. In practice it's a constant source of bugs. Implementations disagree on which contexts to support, which extensions to recognise, how to resolve conflicts. The W3C ActivityStreams context document is a single point of failure that the entire fediverse depends on.")
        (p "SX needs no context resolution. " (code "(Publish :actor \"...\" :object (SxDocument :cid \"...\"))") " is self-describing. The s-expression " (em "is") " the meaning. Symbols are symbols. Keywords are keywords. No indirection, no remote JSON documents, no semantic web machinery."))

      (div :class "rounded border border-rose-200 bg-rose-50 p-4 mt-6"
        (p :class "text-rose-900 font-medium" "The fundamental difference")
        (p :class "text-rose-800" "ActivityPub federates " (em "descriptions") " of content. Consumers must independently figure out how to present them. sx-pub federates " (em "programs") " — self-contained, evaluable, deterministic, content-addressed. The gap between \"data\" and \"application\" disappears because in LISP, they were never separate to begin with.")))

    ;; -----------------------------------------------------------------------
    ;; Wire Format
    ;; -----------------------------------------------------------------------

    (~docs/section :title "Wire Format" :id "wire-format"
      (p "Everything is s-expressions. No JSON-LD, no @context resolution, no nested HTML-in-JSON strings.")

      (~docs/subsection :title "Actor"
        (~docs/code :code "(SxActor\n  :id \"https://pub.sx-web.org/pub/actor\"\n  :type \"SxPublisher\"\n  :name \"sx\"\n  :summary \"SX language — specs, platforms, components\"\n  :public-key-pem \"-----BEGIN PUBLIC KEY-----\\n...\"\n  :inbox \"/pub/inbox\"\n  :outbox \"/pub/outbox\"\n  :followers \"/pub/followers\"\n  :following \"/pub/following\"\n  :collections (list\n    (SxCollection :name \"core-specs\" :href \"/pub/core-specs\")\n    (SxCollection :name \"platforms\" :href \"/pub/platforms\")\n    (SxCollection :name \"components\" :href \"/pub/components\")))"))

      (~docs/subsection :title "Publish Activity"
        (p "When content is published, it's pinned to IPFS and announced to followers:")
        (~docs/code :code "(Publish\n  :actor \"https://pub.sx-web.org/pub/actor\"\n  :published \"2026-03-24T12:00:00Z\"\n  :object (SxDocument\n    :name \"evaluator\"\n    :path \"/pub/core-specs/evaluator\"\n    :cid \"bafy...eval123\"\n    :content-type \"text/sx\"\n    :size 48000\n    :hash \"sha3-256:abc123...\"\n    :requires (list \"bafy...parser\" \"bafy...primitives\")\n    :summary \"CEK machine evaluator — frames, utils, step function\"))")
        (p "The " (code ":requires") " field declares CID dependencies — a component that imports the parser declares that dependency. Receivers can pin the full dependency graph."))

      (~docs/subsection :title "Follow — Subscribe to a Remote Server"
        (~docs/code :code "(Follow\n  :actor \"https://pub.sx-web.org/pub/actor\"\n  :object \"https://other.example.com/pub/actor\")")
        (p "When accepted, the remote server delivers " (code "(Publish ...)") " activities to our inbox. We pin their CIDs locally."))

      (~docs/subsection :title "Announce — Mirror Remote Content"
        (~docs/code :code "(Announce\n  :actor \"https://pub.sx-web.org/pub/actor\"\n  :object \"bafy...remote-cid\")")
        (p "Re-publish content from a followed server to our own followers. The CID is already pinned — we're just signalling that we endorse it."))

      (~docs/subsection :title "Anchor — Blockchain Provenance Record"
        (~docs/code :code "(Anchor\n  :tree-cid \"bafy...merkle-tree\"\n  :leaf-index 42\n  :ots-cid \"bafy...ots-proof\"\n  :btc-txid \"abc123...def\"\n  :btc-block 890123\n  :anchored-at \"2026-03-24T12:00:00Z\")")
        (p "Embedded in every " (code "(Publish ...)") " activity. Receivers can verify: the Merkle tree was anchored in that Bitcoin block, and the CID is a leaf in that tree. Inclusion proof is verifiable offline.")))

    ;; -----------------------------------------------------------------------
    ;; Endpoints
    ;; -----------------------------------------------------------------------

    (~docs/section :title "Endpoints" :id "endpoints"
      (p "All endpoints are " (code "defhandler") " definitions in " (code ".sx") " files. Python provides only IO primitives behind the boundary.")

      (table :class "w-full text-sm"
        (thead
          (tr :class "border-b border-stone-200 text-left"
            (th :class "py-2 pr-4" "Method")
            (th :class "py-2 pr-4" "Route")
            (th :class "py-2" "Purpose")))
        (tbody :class "text-stone-600"
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "GET") (td :class "py-2 pr-4 font-mono text-xs" "/pub/actor") (td :class "py-2" "Actor profile"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "GET") (td :class "py-2 pr-4 font-mono text-xs" "/pub/outbox") (td :class "py-2" "Published activity feed"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "POST") (td :class "py-2 pr-4 font-mono text-xs" "/pub/inbox") (td :class "py-2" "Receive activities from remote servers"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "GET") (td :class "py-2 pr-4 font-mono text-xs" "/pub/⟨collection⟩") (td :class "py-2" "List items in a collection"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "GET") (td :class "py-2 pr-4 font-mono text-xs" "/pub/⟨collection⟩/⟨slug⟩") (td :class "py-2" "Document by path — resolves CID, fetches from IPFS"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "GET") (td :class "py-2 pr-4 font-mono text-xs" "/pub/cid/⟨cid⟩") (td :class "py-2" "Direct CID fetch (immutable, cache forever)"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "POST") (td :class "py-2 pr-4 font-mono text-xs" "/pub/publish") (td :class "py-2" "Pin to IPFS + create Publish activity + announce"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "POST") (td :class "py-2 pr-4 font-mono text-xs" "/pub/follow") (td :class "py-2" "Follow a remote sx-pub server"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "GET") (td :class "py-2 pr-4 font-mono text-xs" "/pub/followers") (td :class "py-2" "Who follows us"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "GET") (td :class "py-2 pr-4 font-mono text-xs" "/pub/following") (td :class "py-2" "Who we follow"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4 font-mono text-xs" "GET") (td :class "py-2 pr-4 font-mono text-xs" "/.well-known/webfinger") (td :class "py-2" "Discovery"))
          (tr
            (td :class "py-2 pr-4 font-mono text-xs" "GET") (td :class "py-2 pr-4 font-mono text-xs" "/pub/anchor/⟨tree-cid⟩") (td :class "py-2" "Merkle tree verification")))))

    ;; -----------------------------------------------------------------------
    ;; Storage
    ;; -----------------------------------------------------------------------

    (~docs/section :title "Storage" :id "storage"
      (p "IPFS is the canonical content store. PostgreSQL is the local index.")

      (table :class "w-full text-sm"
        (thead
          (tr :class "border-b border-stone-200 text-left"
            (th :class "py-2 pr-4" "What")
            (th :class "py-2" "Where")))
        (tbody :class "text-stone-600"
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4" "Content (SX source)") (td :class "py-2" "IPFS — pinned, referenced by CID"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4" "Path → CID index") (td :class "py-2" "PostgreSQL"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4" "Outbox activities") (td :class "py-2" "PostgreSQL"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4" "Followers / following") (td :class "py-2" "PostgreSQL"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4" "Anchor records") (td :class "py-2" "PostgreSQL"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4" "Remote mirrored content") (td :class "py-2" "PostgreSQL + IPFS (pinned locally)"))
          (tr :class "border-b border-stone-100"
            (td :class "py-2 pr-4" "Content cache") (td :class "py-2" "Redis"))
          (tr
            (td :class "py-2 pr-4" "Actor keypair") (td :class "py-2" "PostgreSQL")))))

    ;; -----------------------------------------------------------------------
    ;; SX / Python Split
    ;; -----------------------------------------------------------------------

    (~docs/section :title "SX / Python Split" :id "split"
      (p "SX handles all composition, logic, and rendering. Python provides IO primitives behind the boundary.")

      (div :class "grid grid-cols-2 gap-6"
        (div
          (h4 :class "font-semibold text-stone-700 mb-2" "Pure SX (defhandler + defcomp)")
          (ul :class "text-sm text-stone-600 space-y-1 list-disc pl-5"
            (li "All HTTP routing and request dispatch")
            (li "Content negotiation (Accept header)")
            (li "Response construction (SX actor docs, collections)")
            (li "UI components for browsing")
            (li "Activity serialization")
            (li "Nav data for /pub/ sections")))
        (div
          (h4 :class "font-semibold text-stone-700 mb-2" "Python Helpers (IO boundary)")
          (ul :class "text-sm text-stone-600 space-y-1 list-disc pl-5"
            (li (code "pub-ipfs-add") " / " (code "pub-ipfs-get") " / " (code "pub-ipfs-pin") " — IPFS ops")
            (li (code "pub-db-query") " / " (code "pub-db-insert") " — database")
            (li (code "pub-sign") " / " (code "pub-verify-signature") " — HTTP signatures")
            (li (code "pub-hash") " — SHA3-256")
            (li (code "pub-anchor-batch") " — OpenTimestamps")
            (li (code "pub-deliver") " — POST to follower inboxes")))))

    ;; -----------------------------------------------------------------------
    ;; Flows
    ;; -----------------------------------------------------------------------

    (~docs/section :title "Flows" :id "flows"

      (~docs/subsection :title "Publish"
        (p :class "text-sm text-stone-600 mb-2" "Publishing is " (strong "anchor-then-announce") " — content is provenance-sealed before anyone sees it. This prevents a malicious follower from seeing content, anchoring it first, and claiming provenance.")
        (ol :class "text-sm text-stone-600 space-y-2 list-decimal pl-5"
          (li (code "POST /pub/publish") " with collection, slug, SX content")
          (li "Pin to IPFS → get CID (content is now addressable but not announced)")
          (li "Store path→CID in DB (status: " (code "anchoring") ")")
          (li "Build Merkle tree, submit to OpenTimestamps, wait for Bitcoin confirmation")
          (li "On confirmation: update status to " (code "anchored") ", record proof")
          (li "Construct " (code "(Publish ...)") " with anchor proof embedded")
          (li "Deliver to follower inboxes — " (strong "only now") " is it public"))
        (p :class "text-sm text-stone-500 mt-2 italic" "This means publishing isn't instant — there's a delay while the anchor confirms. That's a feature: it forces a deliberate pace and makes provenance airtight.")
        (~docs/code :code "(Publish\n  :actor \"https://pub.sx-web.org/pub/actor\"\n  :published \"2026-03-24T12:00:00Z\"\n  :object (SxDocument\n    :name \"evaluator\"\n    :cid \"bafy...eval123\"\n    :anchor (Anchor\n      :tree-cid \"bafy...merkle\"\n      :ots-cid \"bafy...proof\"\n      :btc-txid \"abc123...\"\n      :btc-block 890123)))")
        (p :class "text-sm text-stone-600" "Receivers verify: the CID was anchored in that Bitcoin block " (em "before") " the Publish was sent. No way to backdate."))

      (~docs/subsection :title "Follow"
        (ol :class "text-sm text-stone-600 space-y-2 list-decimal pl-5"
          (li (code "POST /pub/follow") " with remote actor URL")
          (li "Fetch remote actor document (" (code "text/sx") ")")
          (li "Send " (code "(Follow ...)") " to their inbox, signed with our key")
          (li "They verify signature, add us to followers, send " (code "(Accept ...)"))
          (li "We store them in our following list")
          (li "They start delivering " (code "(Publish ...)") " activities to our inbox")))

      (~docs/subsection :title "Receive (inbox)"
        (ol :class "text-sm text-stone-600 space-y-2 list-decimal pl-5"
          (li "Remote server POSTs " (code "(Publish ...)") " to our inbox")
          (li "Verify HTTP signature against their actor's public key")
          (li "Pin the referenced CID to our local IPFS node")
          (li "Index the remote content in our DB (marked as mirrored)")
          (li "Now we can resolve that CID locally — " (code "/pub/cid/bafy...") " just works")))

      (~docs/subsection :title "Anchoring"
        (p :class "text-sm text-stone-600 mb-2" "Anchoring is integrated into the publish flow — not a separate background job. Content waits for provenance before going public.")
        (ol :class "text-sm text-stone-600 space-y-2 list-decimal pl-5"
          (li "Batch pending CIDs into a Merkle tree (multiple publishes can share one tree)")
          (li "Pin Merkle tree to IPFS")
          (li "Submit Merkle root to OpenTimestamps calendar servers")
          (li "Poll for Bitcoin confirmation (typically ~1 hour)")
          (li "On confirmation: store proof, transition content to " (code "anchored") " status")
          (li "Trigger delivery of Publish activities for all items in the batch"))
        (p :class "text-sm text-stone-500 mt-2 italic" "Batching amortises the anchor cost — one Bitcoin attestation can seal hundreds of CIDs via the Merkle tree. Each CID gets an inclusion proof linking it to the anchored root.")))

    ;; -----------------------------------------------------------------------
    ;; CID ↔ Path
    ;; -----------------------------------------------------------------------

    (~docs/section :title "CID ↔ Path Translation" :id "cid-path"
      (p "Published SX refers to dependencies by CID — the immutable, universal identifier. But humans browse by path.")

      (~docs/code :code ";; Canonical SX (stored in IPFS) references by CID\n(defcomp ~my-app/dashboard (&key data)\n  ;; This component depends on a chart library published at another CID\n  (let ((chart-lib (require \"bafy...chart-lib-cid\")))\n    (div :class \"dashboard\"\n      (chart-lib/bar-chart :data data))))\n\n;; But on pub.sx-web.org you browse by path:\n;;   /pub/components/dashboard\n;;   /pub/core-specs/evaluator\n;;   /pub/platforms/javascript\n;;\n;; The DB maps: path → CID → IPFS content\n;; Following a remote server maps: their-path → CID → our local pin")

      (p "When you follow another sx-pub server and they publish a component, you get the " (code "(Publish ...)") " activity with the CID. Your server pins it. Now " (code "(require \"bafy...\")") " resolves locally — no network round-trip, no central registry, no package manager."))

    ;; -----------------------------------------------------------------------
    ;; Phases
    ;; -----------------------------------------------------------------------

    (~docs/section :title "Implementation Phases" :id "phases"
      (div :class "space-y-4"
        (div :class "rounded border border-emerald-200 bg-emerald-50 p-4"
          (h4 :class "font-semibold text-emerald-800" "Phase 1: Foundation")
          (p :class "text-emerald-700 text-sm" "DB schema, async IPFS client, actor endpoint, webfinger, " (code "/pub/actor") " returns SX actor document."))
        (div :class "rounded border border-sky-200 bg-sky-50 p-4"
          (h4 :class "font-semibold text-sky-800" "Phase 2: Publishing")
          (p :class "text-sky-700 text-sm" "Pin to IPFS, path→CID index, collection browsing. Publish the actual SX spec files as the first content."))
        (div :class "rounded border border-violet-200 bg-violet-50 p-4"
          (h4 :class "font-semibold text-violet-800" "Phase 3: Federation")
          (p :class "text-violet-700 text-sm" "Inbox/outbox, follow/accept, HTTP signature verification, activity delivery, content mirroring."))
        (div :class "rounded border border-amber-200 bg-amber-50 p-4"
          (h4 :class "font-semibold text-amber-800" "Phase 4: Anchoring")
          (p :class "text-amber-700 text-sm" "Merkle trees, OpenTimestamps, Bitcoin proof, provenance UI."))))))