A "mixed" relation has both user-asserted facts AND rules with the
same head. Previously dl-retract! wiped every rule-head relation
wholesale before re-saturating — the saturator only re-derives the
IDB portion, so explicit EDB facts vanished even for a no-op retract
of a non-existent tuple. Repro:
(let ((db (dl-program "p(a). p(b). p(X) :- q(X). q(c).")))
(dl-retract! db (quote (p z)))
(dl-query db (quote (p X))))
went from {a, b, c} to just {c}.
Fix: track :edb-keys provenance in the db.
- dl-make-db now allocates an :edb-keys dict.
- dl-add-fact! (public) marks (rel-key, tuple-key) in :edb-keys.
- New internal dl-add-derived! does the append without marking.
- Saturator (semi-naive + naive driver) now calls dl-add-derived!.
- dl-retract! strips only the IDB-derived portion of rule-head
relations (anything not in :edb-keys) and preserves the EDB
portion through the re-saturate pass.
2 new regression tests; conformance 262/262.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two malformed-rule paths used to slip through:
- Empty head list `{:head () :body ()}` was accepted; the rule
would never fire but the relation-name lookup later returned
nil with confusing downstream errors.
- Non-list body (`{:head (...) :body 42}`) crashed in `rest`
during safety check with a cryptic "rest: 1 list arg".
dl-add-rule! now checks head shape (non-empty list with symbol
head) and body type (list) before any safety walk. Errors are
descriptive and surface at add time rather than during the next
saturation.
2 new eval tests.
Real bugs surfaced by parser/safety bug-hunt round:
- `not(X) :- p(X).` parsed as a regular literal with relation
"not". The user could accidentally define a `not` relation,
silently shadowing the negation construct.
- `count(N, X, p(X)) :- ...` defined a `count` relation that
would conflict with the aggregate operator.
- `<(X, 5) :- p(X).` defined a `<` relation.
- `is(N, +(1, 2)) :- p(N).` defined an `is` relation.
- `+.` (operator alone) parsed as a 0-ary fact.
Fix: dl-add-fact! and dl-add-rule! now reject any literal whose
head's relation name is in dl-reserved-rel-names — built-in
operators (< <= > >= = != + - * /), aggregate operators
(count sum min max findall), `is`, `not`, and the arrows
(:-, ?-).
4 new eval tests cover the rejection cases.
Note: an initial "no compound args in facts" check was overly
strict — it would reject findall's list output (which derives a
fact like (all_p (a b c))). Reverted that branch; treating
findall results as opaque list values rather than function
symbols.
Adds a user-facing strategy hook: dl-set-strategy! db strategy and
dl-get-strategy db. Default :semi-naive; :magic is accepted but
the actual transformation is deferred — the saturator currently
falls back to semi-naive regardless. Lets us tick the Phase 6
"Optional pass — guarded behind dl-set-strategy!" checkbox while
keeping the equivalence/perf tests pending future work.
3 new eval tests.
Returns {<rel-name>: tuple-count} for relations with any tuples
or that are rule-headed (so empty IDB shows as :rel 0 rather than
disappearing). Skips placeholder entries from internal
dl-ensure-rel! calls. 4 tests cover basic, empty IDB, mixed
EDB+IDB, and empty-db cases.
db gains :facts-index {<rel>: {<first-arg-key>: tuples}} mirroring
the membership :facts-keys index. dl-add-fact! populates the index;
dl-match-positive walks the body literal's first arg under the
current subst — when it's bound to a non-var, look up by (str arg)
instead of scanning the full relation.
For chain-style recursive rules (parent X Y), (ancestor Y Z) the
inner Y has at most one parent, so the inner lookup returns 0–1
tuples instead of N. chain-25 saturation drops from ~33s to ~18s
real (~2x). chain-50 still long but tractable; next bottleneck is
subst dict copies during unification.
dl-retract! refreshed to keep the new index consistent: kept-index
rebuilt during EDB filter, IDB wipes clear all three slots.
Differential semi-naive test bumped to chain-12, semi-only count
test to chain-25.
(p X _), (p _ Y) — the two _ are now different variables, matching
standard Datalog semantics. Previously both _ symbols were the same
SX symbol, so unification across them gave wrong answers.
Fix in db.sx: dl-rename-anon-term + dl-rename-anon-lit walk a term
or literal and replace each '_' symbol with a fresh _anon<N>.
dl-make-anon-renamer returns a counter-based name generator scoped
per call. dl-rename-anon-rule applies it to head and body of a
rule. dl-add-rule! invokes the renamer before safety check.
eval.sx: dl-query renames anon vars in the goal before search and
filters '_' out of the projection so user-facing results aren't
polluted with internal _anon<N> bindings.
The previous "underscore in head ok" test now correctly rejects
(p X _) :- q(X) as unsafe (the head's fresh anon var has no body
binder). New "underscore in body only" test confirms the safe
case. Two regression tests for rule-level and goal-level
independence.
db gains a parallel :facts-keys {<rel>: {<tuple-string>: true}}
index alongside :facts. dl-tuple-key derives a stable string via
(str lit) — (p 30) and (p 30.0) collide correctly because SX
prints them identically. dl-add-fact! membership is now O(1)
instead of O(n) list scan; insert sequences for relations sized
N drop from O(N²) to O(N).
Wall clock on chain-7 saturation halves (~12s → ~6s); chain-15
roughly halves (~50s → ~25s) under shared CPU. Larger chains
still slow due to body-join overhead in dl-find-bindings —
Blocker entry refreshed with proposed follow-ups.
dl-retract! keeps both indices consistent: kept-keys is rebuilt
during the EDB filter, IDB wipes clear both lists and key dicts.
dl-saturate! is now semi-naive: tracks a per-relation delta dict,
and on each iteration walks every positive body-literal position,
substituting the delta of its relation while joining the rest
against the previous-iteration DB. Candidates are collected before
mutating the DB so the "full" sides see a consistent snapshot.
Rules with no positive body literal (e.g. (p X) :- (= X 5).)
fall back to a one-shot naive pass via dl-collect-rule-candidates.
dl-saturate-naive! retained as the reference implementation; 8
differential tests compare per-relation tuple counts on every
recursive program. Switched dl-tuple-member? to indexed iteration
instead of recursive rest (eliminates per-step list copy). Larger
chains under bundled conformance trip O(n) membership × CPU
sharing — added a Blocker to swap relations to hash-set membership.
New lib/datalog/builtins.sx: (< <= > >= = !=) and (is X expr) with
+ - * /. dl-eval-arith recursively evaluates nested compounds.
Safety analysis now walks body left-to-right tracking the bound
set: comparisons require all args bound, is RHS vars must be bound
(LHS becomes bound), = special-cases the var/non-var combos.
db.sx keeps the simple safety check as a forward-reference
fallback; builtins.sx redefines dl-rule-check-safety to the
comprehensive version. eval.sx dispatches built-ins through
dl-eval-builtin instead of erroring. 19 new tests.
