datalog: dl-retract! preserves EDB in mixed relations

A "mixed" relation has both user-asserted facts AND rules with the
same head. Previously dl-retract! wiped every rule-head relation
wholesale before re-saturating — the saturator only re-derives the
IDB portion, so explicit EDB facts vanished even for a no-op retract
of a non-existent tuple. Repro:

  (let ((db (dl-program "p(a). p(b). p(X) :- q(X). q(c).")))
    (dl-retract! db (quote (p z)))
    (dl-query db (quote (p X))))

went from {a, b, c} to just {c}.

Fix: track :edb-keys provenance in the db.

  - dl-make-db now allocates an :edb-keys dict.
  - dl-add-fact! (public) marks (rel-key, tuple-key) in :edb-keys.
  - New internal dl-add-derived! does the append without marking.
  - Saturator (semi-naive + naive driver) now calls dl-add-derived!.
  - dl-retract! strips only the IDB-derived portion of rule-head
    relations (anything not in :edb-keys) and preserves the EDB
    portion through the re-saturate pass.

2 new regression tests; conformance 262/262.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

lib/datalog/api.sx

@@ -91,14 +91,18 @@
       (dl-saturate! db)
       db)))
 
-;; Remove a fact: drop matching tuples from EDB AND wipe all derived
-;; tuples (any IDB tuple may have transitively depended on the removed
-;; fact). Then re-saturate to repopulate IDB. EDB facts that were
-;; asserted via dl-add-fact! are preserved unless they match `lit`.
+;; Remove a fact and re-saturate. Mixed relations (which have BOTH
+;; user-asserted facts AND rules) are supported via :edb-keys provenance
+;; — explicit facts are marked at dl-add-fact! time, the saturator uses
+;; dl-add-derived! which doesn't mark them, so the retract pass can
+;; safely wipe IDB-derived tuples while preserving the user's EDB.
 ;;
-;; To distinguish EDB from IDB, we treat any fact for a relation that
-;; has rules as IDB; otherwise EDB. (Phase 9 simplification — Phase 10
-;; may track provenance.)
+;; Effect:
+;;   - remove tuples matching `lit` from :facts and :edb-keys
+;;   - for every relation that has a rule (i.e. potentially IDB or
+;;     mixed), drop the IDB-derived portion (anything not in :edb-keys)
+;;     so the saturator can re-derive cleanly
+;;   - re-saturate
 (define
   dl-retract!
   (fn
@@ -106,14 +110,20 @@
     (let
       ((rel-key (dl-rel-name lit)))
       (do
-        ;; Drop the matching tuple from its relation list (if EDB-only).
+        ;; Drop the matching tuple from its relation list, its facts-keys,
+        ;; its first-arg index, AND from :edb-keys (if present).
         (when
           (has-key? (get db :facts) rel-key)
           (let
             ((existing (get (get db :facts) rel-key))
               (kept (list))
               (kept-keys {})
-              (kept-index {}))
+              (kept-index {})
+              (edb-rel (cond
+                         ((has-key? (get db :edb-keys) rel-key)
+                           (get (get db :edb-keys) rel-key))
+                         (else nil)))
+              (kept-edb {}))
             (do
               (for-each
                 (fn
@@ -122,7 +132,13 @@
                     (not (dl-tuple-equal? t lit))
                     (do
                       (append! kept t)
-                      (dict-set! kept-keys (dl-tuple-key t) true)
+                      (let ((tk (dl-tuple-key t)))
+                        (do
+                          (dict-set! kept-keys tk true)
+                          (when
+                            (and (not (nil? edb-rel))
+                                 (has-key? edb-rel tk))
+                            (dict-set! kept-edb tk true))))
                       (when
                         (>= (len t) 2)
                         (let ((k (dl-arg-key (nth t 1))))
@@ -134,17 +150,51 @@
                 existing)
               (dict-set! (get db :facts) rel-key kept)
               (dict-set! (get db :facts-keys) rel-key kept-keys)
-              (dict-set! (get db :facts-index) rel-key kept-index))))
-        ;; Wipe all relations that have a rule (these are IDB) so the
-        ;; saturator regenerates them from the surviving EDB.
+              (dict-set! (get db :facts-index) rel-key kept-index)
+              (when
+                (not (nil? edb-rel))
+                (dict-set! (get db :edb-keys) rel-key kept-edb)))))
+        ;; For each rule-head relation, strip the IDB-derived tuples
+        ;; (anything not marked in :edb-keys) so the saturator can
+        ;; cleanly re-derive without leaving stale tuples that depended
+        ;; on the now-removed fact.
         (let ((rule-heads (dl-rule-head-rels db)))
           (for-each
             (fn
               (k)
-              (do
-                (dict-set! (get db :facts) k (list))
-                (dict-set! (get db :facts-keys) k {})
-                (dict-set! (get db :facts-index) k {})))
+              (when
+                (has-key? (get db :facts) k)
+                (let
+                  ((existing (get (get db :facts) k))
+                    (kept (list))
+                    (kept-keys {})
+                    (kept-index {})
+                    (edb-rel (cond
+                               ((has-key? (get db :edb-keys) k)
+                                 (get (get db :edb-keys) k))
+                               (else {}))))
+                  (do
+                    (for-each
+                      (fn
+                        (t)
+                        (let ((tk (dl-tuple-key t)))
+                          (when
+                            (has-key? edb-rel tk)
+                            (do
+                              (append! kept t)
+                              (dict-set! kept-keys tk true)
+                              (when
+                                (>= (len t) 2)
+                                (let ((kk (dl-arg-key (nth t 1))))
+                                  (do
+                                    (when
+                                      (not (has-key? kept-index kk))
+                                      (dict-set! kept-index kk (list)))
+                                    (append! (get kept-index kk) t))))))))
+                      existing)
+                    (dict-set! (get db :facts) k kept)
+                    (dict-set! (get db :facts-keys) k kept-keys)
+                    (dict-set! (get db :facts-index) k kept-index)))))
             rule-heads))
         (dl-saturate! db)
         db))))

lib/datalog/db.sx

@@ -14,7 +14,39 @@
 
 (define
   dl-make-db
-  (fn () {:facts {} :facts-keys {} :facts-index {} :rules (list) :strategy :semi-naive}))
+  (fn ()
+    {:facts {}
+     :facts-keys {}
+     :facts-index {}
+     :edb-keys {}
+     :rules (list)
+     :strategy :semi-naive}))
+
+;; Record (rel-key, tuple-key) as user-asserted EDB. dl-add-fact! calls
+;; this when an explicit fact is added; the saturator (which uses
+;; dl-add-derived!) does NOT, so derived tuples never appear here.
+;; dl-retract! consults :edb-keys to know which tuples must survive
+;; the wipe-and-resaturate round-trip.
+(define
+  dl-mark-edb!
+  (fn
+    (db rel-key tk)
+    (let
+      ((edb (get db :edb-keys)))
+      (do
+        (when
+          (not (has-key? edb rel-key))
+          (dict-set! edb rel-key {}))
+        (dict-set! (get edb rel-key) tk true)))))
+
+(define
+  dl-edb-fact?
+  (fn
+    (db rel-key tk)
+    (let
+      ((edb (get db :edb-keys)))
+      (and (has-key? edb rel-key)
+           (has-key? (get edb rel-key) tk)))))
 
 ;; Evaluation strategy. Default :semi-naive (the only strategy
 ;; currently implemented). :magic is reserved for goal-directed
@@ -196,6 +228,29 @@
   (fn
     (name) (dl-member-string? name dl-reserved-rel-names)))
 
+;; Internal: append a derived tuple to :facts without the public
+;; validation pass and without marking :edb-keys. Used by the saturator
+;; (eval.sx) and magic-sets (magic.sx). Returns true if the tuple was
+;; new, false if already present.
+(define
+  dl-add-derived!
+  (fn
+    (db lit)
+    (let
+      ((rel-key (dl-rel-name lit)))
+      (let
+        ((tuples (dl-ensure-rel! db rel-key))
+          (key-dict (get (get db :facts-keys) rel-key))
+          (tk (dl-tuple-key lit)))
+        (cond
+          ((has-key? key-dict tk) false)
+          (else
+            (do
+              (dict-set! key-dict tk true)
+              (append! tuples lit)
+              (dl-index-add! db rel-key lit)
+              true)))))))
+
 (define
   dl-add-fact!
   (fn
@@ -210,19 +265,13 @@
         (error (str "dl-add-fact!: expected ground literal, got " lit)))
       (else
         (let
-          ((rel-key (dl-rel-name lit)))
-          (let
-            ((tuples (dl-ensure-rel! db rel-key))
-              (key-dict (get (get db :facts-keys) rel-key))
-              (tk (dl-tuple-key lit)))
-            (cond
-              ((has-key? key-dict tk) false)
-              (else
-                (do
-                  (dict-set! key-dict tk true)
-                  (append! tuples lit)
-                  (dl-index-add! db rel-key lit)
-                  true)))))))))
+          ((rel-key (dl-rel-name lit)) (tk (dl-tuple-key lit)))
+          (do
+            ;; Always mark EDB origin — even if the tuple key was already
+            ;; present (e.g. previously derived), so an explicit assert
+            ;; promotes it to EDB and protects it from the IDB wipe.
+            (dl-mark-edb! db rel-key tk)
+            (dl-add-derived! db lit)))))))
 
 ;; The full safety check lives in builtins.sx (it has to know which
 ;; predicates are built-ins). dl-add-rule! calls it via forward

lib/datalog/eval.sx

@@ -130,7 +130,7 @@
             (s)
             (let
               ((derived (dl-apply-subst head s)))
-              (when (dl-add-fact! db derived) (set! new? true))))
+              (when (dl-add-derived! db derived) (set! new? true))))
           (dl-find-bindings body db (dl-empty-subst)))
         new?))))
 
@@ -326,7 +326,7 @@
       (fn
         (lit)
         (when
-          (dl-add-fact! db lit)
+          (dl-add-derived! db lit)
           (let
             ((rel (dl-rel-name lit)))
             (do

lib/datalog/scoreboard.json

@@ -1,8 +1,8 @@
 {
   "lang": "datalog",
-  "total_passed": 260,
+  "total_passed": 262,
   "total_failed": 0,
-  "total": 260,
+  "total": 262,
   "suites": [
     {"name":"tokenize","passed":30,"failed":0,"total":30},
     {"name":"parse","passed":22,"failed":0,"total":22},
@@ -12,9 +12,9 @@
     {"name":"semi_naive","passed":8,"failed":0,"total":8},
     {"name":"negation","passed":10,"failed":0,"total":10},
     {"name":"aggregates","passed":22,"failed":0,"total":22},
-    {"name":"api","passed":20,"failed":0,"total":20},
+    {"name":"api","passed":22,"failed":0,"total":22},
     {"name":"magic","passed":36,"failed":0,"total":36},
     {"name":"demo","passed":21,"failed":0,"total":21}
   ],
-  "generated": "2026-05-11T07:40:56+00:00"
+  "generated": "2026-05-11T07:50:41+00:00"
 }

lib/datalog/scoreboard.md

@@ -1,6 +1,6 @@
 # datalog scoreboard
 
-**260 / 260 passing** (0 failure(s)).
+**262 / 262 passing** (0 failure(s)).
 
 | Suite | Passed | Total | Status |
 |-------|--------|-------|--------|
@@ -12,6 +12,6 @@
 | semi_naive | 8 | 8 | ok |
 | negation | 10 | 10 | ok |
 | aggregates | 22 | 22 | ok |
-| api | 20 | 20 | ok |
+| api | 22 | 22 | ok |
 | magic | 36 | 36 | ok |
 | demo | 21 | 21 | ok |

lib/datalog/tests/api.sx

@@ -159,6 +159,35 @@
             (dl-query db (quote (ancestor tom X)))))
         (list {:X (quote bob)}))
 
+      ;; dl-retract! on a relation with BOTH explicit facts AND a rule
+      ;; (a "mixed" relation) used to wipe the EDB portion when the IDB
+      ;; was re-derived, even when the retract didn't match anything.
+      ;; :edb-keys provenance now preserves user-asserted facts.
+      (dl-api-test-set! "dl-retract! preserves EDB in mixed relation"
+        (let
+          ((db (dl-program-data
+                 (quote ((p a) (p b) (q c)))
+                 (quote ((p X <- (q X)))))))
+          (do
+            (dl-saturate! db)
+            ;; Retract a non-existent tuple — should be a no-op.
+            (dl-retract! db (quote (p z)))
+            (dl-query db (quote (p X)))))
+        (list {:X (quote a)} {:X (quote b)} {:X (quote c)}))
+
+      ;; And retracting an actual EDB fact in a mixed relation drops
+      ;; only that fact; the derived portion stays.
+      (dl-api-test-set! "dl-retract! mixed: drop EDB, keep IDB"
+        (let
+          ((db (dl-program-data
+                 (quote ((p a) (p b) (q c)))
+                 (quote ((p X <- (q X)))))))
+          (do
+            (dl-saturate! db)
+            (dl-retract! db (quote (p a)))
+            (dl-query db (quote (p X)))))
+        (list {:X (quote b)} {:X (quote c)}))
+
       ;; dl-program-data + dl-query with constants in head.
       (dl-api-test-set! "constant-in-head data"
         (dl-query

plans/datalog-on-sx.md

@@ -15,7 +15,7 @@ for rose-ash data (e.g. federation graph, content relationships).
 
 ## Status (rolling)
 
-`bash lib/datalog/conformance.sh` → **260/260 across 11 suites**
+`bash lib/datalog/conformance.sh` → **262/262 across 11 suites**
 (tokenize, parse, unify, eval, builtins, semi_naive, negation, aggregates,
 api, magic, demo). Source is ~3100 LOC, tests ~2900 LOC, public API
 documented in `lib/datalog/datalog.sx`.
@@ -320,6 +320,22 @@ large graphs.
 
 _Newest first._
 
+- 2026-05-11 — `dl-retract!` was silently destroying EDB facts in
+  "mixed" relations (those with BOTH user-asserted facts AND a rule
+  defining the same head). The retract pass wiped every rule-head
+  relation wholesale and then re-saturated — but the saturator only
+  re-derives the IDB portion, so explicit EDB facts vanished even
+  for a no-op retract of a non-existent tuple. Probe:
+  `(let ((db (dl-program "p(a). p(b). p(X) :- q(X). q(c).")))
+     (dl-retract! db (quote (p z))) (dl-query db (quote (p X))))`
+  went from `{a,b,c}` to just `{c}`.
+  Fix: tracked `:edb-keys` provenance in the db. `dl-add-fact!` (public
+  API) marks the tuple as EDB; saturator calls new internal
+  `dl-add-derived!` which doesn't mark it. `dl-retract!` now strips
+  only the IDB-derived portion of rule-head relations and preserves
+  EDB-marked tuples through the re-saturate pass. 2 new regression
+  tests; 262/262.
+
 - 2026-05-11 — Eval-semantics bug-hunt: nested `not(not(P))` was
   silently misinterpreted. Outer-level `not(...)` is parsed as
   negation, but the inner `not(banned(X))` was parsed as a regular