dl-magic-query could silently diverge from dl-query when an
aggregate's inner-goal relation was IDB. The rewriter passes
aggregate body lits through unchanged (no magic propagation
generated for them), so the inner relation was empty in the magic
db and the aggregate returned 0. Repro:
(dl-eval-magic
"u(a). u(b). u(c). u(d). banned(b). banned(d).
active(X) :- u(X), not(banned(X)).
n(N) :- count(N, X, active(X))."
"?- n(N).")
=> ({:N 0}) ; should be ({:N 2})
dl-magic-query now pre-saturates the source db before copying facts
into the magic db. This guarantees equivalence with dl-query for
every stratified program; the magic benefit still comes from
goal-directed re-derivation of the query relation under the seed
(which matters for large recursive joins). The existing test cases
happened to dodge this because their aggregate inner-goals were all
EDB.
1 new regression test; conformance 274/274.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The canonical Datalog idiom for "no X has any Y":
orphan(X) :- person(X), not(parent(X, _)).
was rejected by the safety check with "negation refers to unbound
variable(s) (\"_anon1\")". The parser renames each anonymous `_`
to a fresh `_anon*` symbol so multiple `_` occurrences don't unify
with each other, and the negation safety walk then demanded all
free vars in the negated lit be bound by an earlier positive body
lit — including the renamed anonymous vars.
Anonymous vars in a negation are existentially quantified within
the negation, not requirements from outside. Added dl-non-anon-vars
to strip `_anon*` names from the `needed` set before the binding
check in dl-process-neg!. Real vars (like `X` in the orphan idiom)
still must be bound by an earlier positive body lit, just as before.
2 new regression tests (orphan idiom + multi-anon "solo" pattern);
conformance 273/273.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Datalog has no function symbols in argument positions, but the
existing dl-add-fact! / dl-add-rule! validators only checked that
literals were ground (no free variables). A compound like `+(1, 2)`
contains no variables, so:
p(+(1, 2)).
=> stored as the unreduced tuple `(p (+ 1 2))`
double(*(X, 2)) :- n(X). n(3).
=> saturates `double((* 3 2))` instead of `double(6)`
Added dl-simple-term? (number / string / symbol) and an
args-simple? walker, used by:
- dl-add-fact!: all args must be simple terms
- dl-add-rule!: rule head args must be simple terms (variables
are symbols, so they pass)
Compounds remain legal in body literals where they encode `is` /
arithmetic / aggregate sub-goals. Error messages name the offending
literal and point the user at the body-only mechanism.
2 new regression tests; conformance 271/271.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Quoted atoms with uppercase- or underscore-leading names were
misclassified as variables. `p('Hello World').` flowed through the
tokenizer's "atom" branch and through the parser's string->symbol,
producing a symbol named "Hello World". dl-var? inspects the first
character — "H" is uppercase, so the fact was rejected as non-ground
("expected ground literal").
Tokenizer now emits "string" for any '...' quoted form. Quoted atoms
become opaque string constants — matching how Datalog idiomatically
treats them, and avoiding a per-symbol "quoted" marker that would
have rippled through unification and dl-var?. The trade-off is that
'a' and a are no longer the same value (string vs symbol); for
Datalog this is the safer default.
Updated the existing "quoted atom" tokenize test, added a regression
case for an uppercase-named quoted atom, and a parse-level test that
verifies the AST. Conformance 269/269.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Type-mixed comparisons were silently inconsistent:
<("hello", 5) => no result, no error (silent false)
<(a, 5) => raises "Expected number, got symbol"
Both should fail loudly with a comprehensible message. Added
dl-compare-typeok?: <, <=, >, >= now require both operands to share
a primitive type (both numbers or both strings) and raise a clear
"comparison <op> requires same-type operands" error otherwise.
`!=` is exempted because it's the polymorphic inequality test
built on dl-tuple-equal? — cross-type pairs are legitimately unequal
and the existing semantics for that case match user intuition.
2 new regression tests; conformance 267/267.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A dict in a rule body that isn't `{:neg <positive-lit>}` (the only
recognised dict shape) used to silently fall through every dispatch
clause in dl-rule-check-safety, contributing zero bound variables.
The user would then see a confusing "head variable(s) X do not
appear in any positive body literal" pointing at the head — not at
the actual bug in the body. Typos like `{:negs ...}` are the typical
trigger.
dl-process-lit! now flags both:
- a dict that lacks :neg
- a bare number / string / symbol used as a body lit
with a clear error naming the offending literal.
1 new regression test; conformance 265/265.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`is(R, /(X, 0))` was silently producing IEEE infinity:
(dl-eval "p(10). q(R) :- p(X), is(R, /(X, 0))." "?- q(R).")
=> ({:R inf})
That value then flowed through comparisons (anything < inf, anything
> inf) and aggregations (sum of inf, max of inf) producing nonsense
results downstream. `dl-eval-arith` now checks the divisor before
the host `/` and raises "division by zero in <expr>" — surfacing
the bug at its source rather than letting infinity propagate.
1 new test; conformance 264/264.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`count(N, Y, p(X))` silently returned `N = 1` because `Y` was never
bound by the goal — every match contributed the same unbound symbol
which dl-val-member? deduped to a single entry. Similarly:
sum(S, Y, p(X)) => raises "expected number, got symbol"
findall(L, Y, p(X)) => L = (Y) (a list containing the unbound symbol)
count(N, Y, p(X)) => N = 1 (silent garbage)
Added a third validator in dl-eval-aggregate: the agg-var must
syntactically appear among the goal's variables. Error names the
variable and the goal and explains why the result would be
meaningless.
1 new test; conformance 263/263.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A "mixed" relation has both user-asserted facts AND rules with the
same head. Previously dl-retract! wiped every rule-head relation
wholesale before re-saturating — the saturator only re-derives the
IDB portion, so explicit EDB facts vanished even for a no-op retract
of a non-existent tuple. Repro:
(let ((db (dl-program "p(a). p(b). p(X) :- q(X). q(c).")))
(dl-retract! db (quote (p z)))
(dl-query db (quote (p X))))
went from {a, b, c} to just {c}.
Fix: track :edb-keys provenance in the db.
- dl-make-db now allocates an :edb-keys dict.
- dl-add-fact! (public) marks (rel-key, tuple-key) in :edb-keys.
- New internal dl-add-derived! does the append without marking.
- Saturator (semi-naive + naive driver) now calls dl-add-derived!.
- dl-retract! strips only the IDB-derived portion of rule-head
relations (anything not in :edb-keys) and preserves the EDB
portion through the re-saturate pass.
2 new regression tests; conformance 262/262.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nested `not(not(P))` silently misparsed: outer `not(...)` is
recognised as negation, but the inner `not(banned(X))` was parsed
as a positive call to a relation called `not`. With no `not`
relation present, the inner match was empty, the outer negation
succeeded vacuously, and `vip(X) :- u(X), not(not(banned(X))).`
collapsed to `vip(X) :- u(X).` — a silent double-negation = identity
fallacy.
Fix in `dl-rule-check-safety`: the positive-literal branch and
`dl-process-neg!` both reject any body literal whose relation
name is in `dl-reserved-rel-names`. Error message names the
relation and points the user at stratified negation through an
intermediate relation.
1 regression test; conformance 260/260.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug: dl-eval-aggregate accepted non-variable agg-vars and non-
literal goals silently, producing weird/incorrect counts:
- `count(N, 5, p(X))` would compute count over the single
constant 5 (always 1), ignoring p entirely.
- `count(N, X, 42)` would crash with "unknown body-literal
shape" at saturation time rather than at rule-add time.
Fix: dl-eval-aggregate now validates up front that the second
arg is a variable (the value to aggregate) and the third arg is
a positive literal (the goal). Errors are descriptive and
include the offending argument.
2 new aggregate tests.
Bug: dl-walk would infinite-loop on a circular substitution
(e.g. A→B and B→A simultaneously). The walk endlessly chased
the cycle. This couldn't be produced through dl-unify (which has
cycle-safe behavior via existing bindings), but raw dl-bind calls
or external manipulation of the subst dict could create it.
Fix: dl-walk now threads a visited-names list through the
recursion. If a variable name is already in the list, the walk
stops and returns the current term unchanged. Normal chained
walks are unaffected (A→B→C→42 still resolves to 42).
1 new unify test verifies circular substitutions don't hang.
Bug: characters not recognised by any branch of `scan!` (`?`,
`!`, `#`, `@`, `&`, `|`, `\\`, `^`, etc.) were silently consumed
via `(else (advance! 1) (scan!))`. Programs with typos would
parse to a stripped version of themselves with no warning —
`?(X).` became `(X).` and produced confusing downstream errors.
Fix: the else branch now raises a clear "unexpected character"
error with the offending char and its position.
1 new tokenize test.
Bug: dl-magic-query crashed with cryptic "rest: 1 list arg" when
the goal argument was a string, number, or arbitrary dict. The
first thing the function does is dl-rel-name + dl-adorn-goal,
both of which assume a positive-literal list shape.
Fix: explicit shape check up front. A goal must be a non-empty
list whose first element is a symbol. Otherwise raise with a
clear diagnostic. Built-in / aggregate / negation dispatch (the
fall-back to dl-query) is unchanged.
2 new magic tests cover string and bare-dict goal rejection.
Two malformed-rule paths used to slip through:
- Empty head list `{:head () :body ()}` was accepted; the rule
would never fire but the relation-name lookup later returned
nil with confusing downstream errors.
- Non-list body (`{:head (...) :body 42}`) crashed in `rest`
during safety check with a cryptic "rest: 1 list arg".
dl-add-rule! now checks head shape (non-empty list with symbol
head) and body type (list) before any safety walk. Errors are
descriptive and surface at add time rather than during the next
saturation.
2 new eval tests.
Bug: read-quoted ran to EOF silently when the closing quote was
missing. The token's value was whatever ran-to-end string had been
accumulated; the parser later saw an unexpected EOF, but the error
message blamed the wrong location ("expected `)` got eof") and
hid the real problem.
Fix: read-quoted now raises with a message that distinguishes
strings from quoted atoms, including the position where the
opening quote was lost. The escape-sequence handling and proper
closing are unaffected.
2 new tokenize tests.
Bug: `/* unclosed` was silently consumed to EOF, swallowing any
Datalog code that followed inside the (never-closing) comment.
Programs would produce empty parses with no error.
Fix: skip-block-comment! now raises when it hits EOF without
finding `*/`. Error message includes the position where the
problem was first detected. Line comments (`%`) and properly
closed block comments (`/* ... */`) are unaffected.
1 new tokenize test verifies the error path.
Bug: `n(-1).` failed to parse — the tokenizer produced op `-`
followed by number `1`, and dl-pp-parse-arg expected a term after
seeing `-` as an op (and a `(` for a compound) but found a bare
number. Users had to write `(- 0 1)` or compute via `is`.
Fix: dl-pp-parse-arg detects op `-` directly followed by a number
token (no intervening `(`) and consumes both as a single negative
number literal. Subtraction (`is(Y, -(X, 2))`) and compound
arithmetic via the operator form are unaffected — they use the
`-(` lookahead path.
2 new parser tests: negative integer literal and subtraction
compound preserved.
Real bugs surfaced by parser/safety bug-hunt round:
- `not(X) :- p(X).` parsed as a regular literal with relation
"not". The user could accidentally define a `not` relation,
silently shadowing the negation construct.
- `count(N, X, p(X)) :- ...` defined a `count` relation that
would conflict with the aggregate operator.
- `<(X, 5) :- p(X).` defined a `<` relation.
- `is(N, +(1, 2)) :- p(N).` defined an `is` relation.
- `+.` (operator alone) parsed as a 0-ary fact.
Fix: dl-add-fact! and dl-add-rule! now reject any literal whose
head's relation name is in dl-reserved-rel-names — built-in
operators (< <= > >= = != + - * /), aggregate operators
(count sum min max findall), `is`, `not`, and the arrows
(:-, ?-).
4 new eval tests cover the rejection cases.
Note: an initial "no compound args in facts" check was overly
strict — it would reject findall's list output (which derives a
fact like (all_p (a b c))). Reverted that branch; treating
findall results as opaque list values rather than function
symbols.
Bug-hunt round probed magic-sets against many edge cases. No new
bugs surfaced. Added regression tests for two patterns that
exercise the worklist post-fix:
- 3-stratum program (a → c via not-b → d via not-banned).
Distinct rule heads at three strata; magic must rewrite each.
- Aggregate-derived chain (count(src) → cnt → active threshold).
Magic correctly handles multi-step aggregate dependencies.
Magic-sets is robust against: 3-stratum negation, aggregate
chains, mutual recursion, all-bound goals, multi-arity rules,
diagonal queries, EDB-only goals, and rules whose body has
identical positive lits.
Real bug: the worklist used (set! queue (rest queue)) to pop the
head, which left queue bound to a fresh empty list as soon as the
last item was popped. Subsequent (append! queue ...) was a no-op
on the empty list — so when the head's rewrite generated new
(rel, adn) pairs to enqueue, they vanished. Multi-relation
programs (e.g. shortest -> path -> edge, or chained derived
relations) only had their head's rules rewritten; downstream
rules silently dropped.
Fix: use an index-based loop (idx 0 → len queue), with append!
adding to the same list. Items added after the current pointer
are picked up in subsequent iterations.
2 new regression tests:
- 4-level chain (a → r1 → r2 → r3 → r4) under magic returns 2
- shortest-path demo via magic equals dl-query (1 result)
Bug: dl-magic-query was skipping EDB facts for relations that had
rules ("rule-headed"). When a single relation has both EDB facts
and rules deriving more (mixed EDB+IDB), the rewritten run would
miss the EDB portion entirely, producing too few or zero results.
Fix: copy ALL existing facts to the internal mdb regardless of
whether the relation has rules. EDB-only relations bring their
tuples; mixed relations bring both EDB and any pre-saturated IDB
(which the rewritten rules would re-derive anyway).
1 new test: link relation seeded with 3 EDB tuples plus a
recursive rule via via/2. dl-magic-query rooted at `a` returns
2 results (a→b direct, a→c via via(a,e), link(e,c)).
Bug: dl-magic-query was always trying to seed a magic_<rel>^<adn>
fact for the query goal. For aggregate goals like (count N X (p X))
this produced a non-ground "fact" (magic_count^... N X (p X)) and
dl-add-fact! correctly rejected it, surfacing as an error.
Fix: dl-magic-query now detects built-in / aggregate / negation
goals up front and dispatches to plain dl-query for those cases —
magic-sets only applies to positive non-builtin literals against
rule-defined relations. Other shapes don't benefit from the
rewrite anyway.
1 new test confirms (count N X (p X)) returns the expected
{:N 3} via dl-magic-query.
Adds dl-demo-org-rules: (subordinate Mgr Emp) over a (manager
EMP MGR) graph, and (headcount Mgr N) using count aggregation
grouped by manager. Demonstrates real-world hierarchy queries
(e.g. "everyone reporting up to the CEO") + per-manager rollup.
3 new demo tests: transitive subordinates of CEO (5 entries),
CEO headcount, and direct manager headcount.
(dl-rules-of db rel-name) → list of rules with head matching
the given relation name. Useful for tooling and debugging
("show me how this relation is derived") without exposing the
internal :rules list directly.
2 new api tests cover hit and miss cases.
Returns true iff one more saturation step would derive no new
tuples. Walks every rule under the current bindings and short-
circuits as soon as one derivation would add a fresh tuple.
Useful in tests that want to assert "no work left" after a call,
or for tooling that wants to know whether `dl-saturate!` would
do anything.
3 new eval tests cover the after-saturation, before-saturation,
and after-assert states.
Demonstrates the practical effect of goal-directed evaluation:
chain of 12 nodes, semi-naive derives the full ancestor closure
(78 = 12·13/2 tuples), while a magic-rooted query at node 0
returns only its 12 descendants. Concrete check that magic
limits derivation to the query's transitive cone.
