Datalog has no function symbols in argument positions, but the
existing dl-add-fact! / dl-add-rule! validators only checked that
literals were ground (no free variables). A compound like `+(1, 2)`
contains no variables, so:
p(+(1, 2)).
=> stored as the unreduced tuple `(p (+ 1 2))`
double(*(X, 2)) :- n(X). n(3).
=> saturates `double((* 3 2))` instead of `double(6)`
Added dl-simple-term? (number / string / symbol) and an
args-simple? walker, used by:
- dl-add-fact!: all args must be simple terms
- dl-add-rule!: rule head args must be simple terms (variables
are symbols, so they pass)
Compounds remain legal in body literals where they encode `is` /
arithmetic / aggregate sub-goals. Error messages name the offending
literal and point the user at the body-only mechanism.
2 new regression tests; conformance 271/271.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Quoted atoms with uppercase- or underscore-leading names were
misclassified as variables. `p('Hello World').` flowed through the
tokenizer's "atom" branch and through the parser's string->symbol,
producing a symbol named "Hello World". dl-var? inspects the first
character — "H" is uppercase, so the fact was rejected as non-ground
("expected ground literal").
Tokenizer now emits "string" for any '...' quoted form. Quoted atoms
become opaque string constants — matching how Datalog idiomatically
treats them, and avoiding a per-symbol "quoted" marker that would
have rippled through unification and dl-var?. The trade-off is that
'a' and a are no longer the same value (string vs symbol); for
Datalog this is the safer default.
Updated the existing "quoted atom" tokenize test, added a regression
case for an uppercase-named quoted atom, and a parse-level test that
verifies the AST. Conformance 269/269.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Type-mixed comparisons were silently inconsistent:
<("hello", 5) => no result, no error (silent false)
<(a, 5) => raises "Expected number, got symbol"
Both should fail loudly with a comprehensible message. Added
dl-compare-typeok?: <, <=, >, >= now require both operands to share
a primitive type (both numbers or both strings) and raise a clear
"comparison <op> requires same-type operands" error otherwise.
`!=` is exempted because it's the polymorphic inequality test
built on dl-tuple-equal? — cross-type pairs are legitimately unequal
and the existing semantics for that case match user intuition.
2 new regression tests; conformance 267/267.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A dict in a rule body that isn't `{:neg <positive-lit>}` (the only
recognised dict shape) used to silently fall through every dispatch
clause in dl-rule-check-safety, contributing zero bound variables.
The user would then see a confusing "head variable(s) X do not
appear in any positive body literal" pointing at the head — not at
the actual bug in the body. Typos like `{:negs ...}` are the typical
trigger.
dl-process-lit! now flags both:
- a dict that lacks :neg
- a bare number / string / symbol used as a body lit
with a clear error naming the offending literal.
1 new regression test; conformance 265/265.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`is(R, /(X, 0))` was silently producing IEEE infinity:
(dl-eval "p(10). q(R) :- p(X), is(R, /(X, 0))." "?- q(R).")
=> ({:R inf})
That value then flowed through comparisons (anything < inf, anything
> inf) and aggregations (sum of inf, max of inf) producing nonsense
results downstream. `dl-eval-arith` now checks the divisor before
the host `/` and raises "division by zero in <expr>" — surfacing
the bug at its source rather than letting infinity propagate.
1 new test; conformance 264/264.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`count(N, Y, p(X))` silently returned `N = 1` because `Y` was never
bound by the goal — every match contributed the same unbound symbol
which dl-val-member? deduped to a single entry. Similarly:
sum(S, Y, p(X)) => raises "expected number, got symbol"
findall(L, Y, p(X)) => L = (Y) (a list containing the unbound symbol)
count(N, Y, p(X)) => N = 1 (silent garbage)
Added a third validator in dl-eval-aggregate: the agg-var must
syntactically appear among the goal's variables. Error names the
variable and the goal and explains why the result would be
meaningless.
1 new test; conformance 263/263.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A "mixed" relation has both user-asserted facts AND rules with the
same head. Previously dl-retract! wiped every rule-head relation
wholesale before re-saturating — the saturator only re-derives the
IDB portion, so explicit EDB facts vanished even for a no-op retract
of a non-existent tuple. Repro:
(let ((db (dl-program "p(a). p(b). p(X) :- q(X). q(c).")))
(dl-retract! db (quote (p z)))
(dl-query db (quote (p X))))
went from {a, b, c} to just {c}.
Fix: track :edb-keys provenance in the db.
- dl-make-db now allocates an :edb-keys dict.
- dl-add-fact! (public) marks (rel-key, tuple-key) in :edb-keys.
- New internal dl-add-derived! does the append without marking.
- Saturator (semi-naive + naive driver) now calls dl-add-derived!.
- dl-retract! strips only the IDB-derived portion of rule-head
relations (anything not in :edb-keys) and preserves the EDB
portion through the re-saturate pass.
2 new regression tests; conformance 262/262.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nested `not(not(P))` silently misparsed: outer `not(...)` is
recognised as negation, but the inner `not(banned(X))` was parsed
as a positive call to a relation called `not`. With no `not`
relation present, the inner match was empty, the outer negation
succeeded vacuously, and `vip(X) :- u(X), not(not(banned(X))).`
collapsed to `vip(X) :- u(X).` — a silent double-negation = identity
fallacy.
Fix in `dl-rule-check-safety`: the positive-literal branch and
`dl-process-neg!` both reject any body literal whose relation
name is in `dl-reserved-rel-names`. Error message names the
relation and points the user at stratified negation through an
intermediate relation.
1 regression test; conformance 260/260.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug: dl-eval-aggregate accepted non-variable agg-vars and non-
literal goals silently, producing weird/incorrect counts:
- `count(N, 5, p(X))` would compute count over the single
constant 5 (always 1), ignoring p entirely.
- `count(N, X, 42)` would crash with "unknown body-literal
shape" at saturation time rather than at rule-add time.
Fix: dl-eval-aggregate now validates up front that the second
arg is a variable (the value to aggregate) and the third arg is
a positive literal (the goal). Errors are descriptive and
include the offending argument.
2 new aggregate tests.
Bug: dl-walk would infinite-loop on a circular substitution
(e.g. A→B and B→A simultaneously). The walk endlessly chased
the cycle. This couldn't be produced through dl-unify (which has
cycle-safe behavior via existing bindings), but raw dl-bind calls
or external manipulation of the subst dict could create it.
Fix: dl-walk now threads a visited-names list through the
recursion. If a variable name is already in the list, the walk
stops and returns the current term unchanged. Normal chained
walks are unaffected (A→B→C→42 still resolves to 42).
1 new unify test verifies circular substitutions don't hang.
Bug: characters not recognised by any branch of `scan!` (`?`,
`!`, `#`, `@`, `&`, `|`, `\\`, `^`, etc.) were silently consumed
via `(else (advance! 1) (scan!))`. Programs with typos would
parse to a stripped version of themselves with no warning —
`?(X).` became `(X).` and produced confusing downstream errors.
Fix: the else branch now raises a clear "unexpected character"
error with the offending char and its position.
1 new tokenize test.
Bug: dl-magic-query crashed with cryptic "rest: 1 list arg" when
the goal argument was a string, number, or arbitrary dict. The
first thing the function does is dl-rel-name + dl-adorn-goal,
both of which assume a positive-literal list shape.
Fix: explicit shape check up front. A goal must be a non-empty
list whose first element is a symbol. Otherwise raise with a
clear diagnostic. Built-in / aggregate / negation dispatch (the
fall-back to dl-query) is unchanged.
2 new magic tests cover string and bare-dict goal rejection.
Two malformed-rule paths used to slip through:
- Empty head list `{:head () :body ()}` was accepted; the rule
would never fire but the relation-name lookup later returned
nil with confusing downstream errors.
- Non-list body (`{:head (...) :body 42}`) crashed in `rest`
during safety check with a cryptic "rest: 1 list arg".
dl-add-rule! now checks head shape (non-empty list with symbol
head) and body type (list) before any safety walk. Errors are
descriptive and surface at add time rather than during the next
saturation.
2 new eval tests.
Bug: read-quoted ran to EOF silently when the closing quote was
missing. The token's value was whatever ran-to-end string had been
accumulated; the parser later saw an unexpected EOF, but the error
message blamed the wrong location ("expected `)` got eof") and
hid the real problem.
Fix: read-quoted now raises with a message that distinguishes
strings from quoted atoms, including the position where the
opening quote was lost. The escape-sequence handling and proper
closing are unaffected.
2 new tokenize tests.
Bug: `/* unclosed` was silently consumed to EOF, swallowing any
Datalog code that followed inside the (never-closing) comment.
Programs would produce empty parses with no error.
Fix: skip-block-comment! now raises when it hits EOF without
finding `*/`. Error message includes the position where the
problem was first detected. Line comments (`%`) and properly
closed block comments (`/* ... */`) are unaffected.
1 new tokenize test verifies the error path.
Bug: `n(-1).` failed to parse — the tokenizer produced op `-`
followed by number `1`, and dl-pp-parse-arg expected a term after
seeing `-` as an op (and a `(` for a compound) but found a bare
number. Users had to write `(- 0 1)` or compute via `is`.
Fix: dl-pp-parse-arg detects op `-` directly followed by a number
token (no intervening `(`) and consumes both as a single negative
number literal. Subtraction (`is(Y, -(X, 2))`) and compound
arithmetic via the operator form are unaffected — they use the
`-(` lookahead path.
2 new parser tests: negative integer literal and subtraction
compound preserved.
Real bugs surfaced by parser/safety bug-hunt round:
- `not(X) :- p(X).` parsed as a regular literal with relation
"not". The user could accidentally define a `not` relation,
silently shadowing the negation construct.
- `count(N, X, p(X)) :- ...` defined a `count` relation that
would conflict with the aggregate operator.
- `<(X, 5) :- p(X).` defined a `<` relation.
- `is(N, +(1, 2)) :- p(N).` defined an `is` relation.
- `+.` (operator alone) parsed as a 0-ary fact.
Fix: dl-add-fact! and dl-add-rule! now reject any literal whose
head's relation name is in dl-reserved-rel-names — built-in
operators (< <= > >= = != + - * /), aggregate operators
(count sum min max findall), `is`, `not`, and the arrows
(:-, ?-).
4 new eval tests cover the rejection cases.
Note: an initial "no compound args in facts" check was overly
strict — it would reject findall's list output (which derives a
fact like (all_p (a b c))). Reverted that branch; treating
findall results as opaque list values rather than function
symbols.
Bug-hunt round probed magic-sets against many edge cases. No new
bugs surfaced. Added regression tests for two patterns that
exercise the worklist post-fix:
- 3-stratum program (a → c via not-b → d via not-banned).
Distinct rule heads at three strata; magic must rewrite each.
- Aggregate-derived chain (count(src) → cnt → active threshold).
Magic correctly handles multi-step aggregate dependencies.
Magic-sets is robust against: 3-stratum negation, aggregate
chains, mutual recursion, all-bound goals, multi-arity rules,
diagonal queries, EDB-only goals, and rules whose body has
identical positive lits.
Real bug: the worklist used (set! queue (rest queue)) to pop the
head, which left queue bound to a fresh empty list as soon as the
last item was popped. Subsequent (append! queue ...) was a no-op
on the empty list — so when the head's rewrite generated new
(rel, adn) pairs to enqueue, they vanished. Multi-relation
programs (e.g. shortest -> path -> edge, or chained derived
relations) only had their head's rules rewritten; downstream
rules silently dropped.
Fix: use an index-based loop (idx 0 → len queue), with append!
adding to the same list. Items added after the current pointer
are picked up in subsequent iterations.
2 new regression tests:
- 4-level chain (a → r1 → r2 → r3 → r4) under magic returns 2
- shortest-path demo via magic equals dl-query (1 result)
Bug: dl-magic-query was skipping EDB facts for relations that had
rules ("rule-headed"). When a single relation has both EDB facts
and rules deriving more (mixed EDB+IDB), the rewritten run would
miss the EDB portion entirely, producing too few or zero results.
Fix: copy ALL existing facts to the internal mdb regardless of
whether the relation has rules. EDB-only relations bring their
tuples; mixed relations bring both EDB and any pre-saturated IDB
(which the rewritten rules would re-derive anyway).
1 new test: link relation seeded with 3 EDB tuples plus a
recursive rule via via/2. dl-magic-query rooted at `a` returns
2 results (a→b direct, a→c via via(a,e), link(e,c)).
Bug: dl-magic-query was always trying to seed a magic_<rel>^<adn>
fact for the query goal. For aggregate goals like (count N X (p X))
this produced a non-ground "fact" (magic_count^... N X (p X)) and
dl-add-fact! correctly rejected it, surfacing as an error.
Fix: dl-magic-query now detects built-in / aggregate / negation
goals up front and dispatches to plain dl-query for those cases —
magic-sets only applies to positive non-builtin literals against
rule-defined relations. Other shapes don't benefit from the
rewrite anyway.
1 new test confirms (count N X (p X)) returns the expected
{:N 3} via dl-magic-query.
Adds dl-demo-org-rules: (subordinate Mgr Emp) over a (manager
EMP MGR) graph, and (headcount Mgr N) using count aggregation
grouped by manager. Demonstrates real-world hierarchy queries
(e.g. "everyone reporting up to the CEO") + per-manager rollup.
3 new demo tests: transitive subordinates of CEO (5 entries),
CEO headcount, and direct manager headcount.
(dl-rules-of db rel-name) → list of rules with head matching
the given relation name. Useful for tooling and debugging
("show me how this relation is derived") without exposing the
internal :rules list directly.
2 new api tests cover hit and miss cases.
Returns true iff one more saturation step would derive no new
tuples. Walks every rule under the current bindings and short-
circuits as soon as one derivation would add a fresh tuple.
Useful in tests that want to assert "no work left" after a call,
or for tooling that wants to know whether `dl-saturate!` would
do anything.
3 new eval tests cover the after-saturation, before-saturation,
and after-assert states.
Demonstrates the practical effect of goal-directed evaluation:
chain of 12 nodes, semi-naive derives the full ancestor closure
(78 = 12·13/2 tuples), while a magic-rooted query at node 0
returns only its 12 descendants. Concrete check that magic
limits derivation to the query's transitive cone.
Wipes every rule-headed relation (the IDB) — leaves EDB facts and
rule definitions intact. Useful for inspecting the EDB-only
baseline or for forcing a clean re-saturation.
(dl-saturate! db)
(dl-clear-idb! db) ; ancestor relation now empty
(dl-saturate! db) ; re-derives ancestor from parents
2 new api tests verify IDB-wipe and EDB-preservation.
Symmetric to dl-eval but routes single-positive-literal queries
through dl-magic-query for goal-directed evaluation. Multi-literal
query bodies fall back to standard dl-query (magic-sets is wired
for single goals only).
(dl-eval-magic source-string "?- ancestor(a, X).")
1 new api test.
Two disjoint chains, query rooted in cluster 1. Semi-naive
derives the full closure over both clusters (6 ancestor tuples).
Magic-sets only seeds magic_ancestor^bf for cluster 1, so only
2 query-relevant tuples are returned (a→b, a→c). The test
asserts both numbers, demonstrating the actual perf-shape
benefit of goal-directed evaluation.
End-to-end magic-sets entry point. Given (db, query-goal):
- copies the caller's EDB facts (relations not headed by any
rule) into a fresh internal db
- adds the magic seed fact
- adds the rewritten rules
- saturates and runs the query
- returns the substitution list
Caller's db is untouched. Equivalent to dl-query for any
fully-stratifiable program; intended as a perf alternative on
goal-shaped queries against large recursive relations.
2 new tests: equivalence to dl-query on chain-3 ancestor, and
non-mutation of the caller's db (rules count unchanged).
dl-magic-rewrite rules query-rel adn args returns:
{:rules <rewritten-rules> :seed <magic-seed-fact>}
Worklist over (rel, adn) pairs starts from the query and stops
when no new pairs appear. For each rule with head matching a
worklist pair:
- Adorned rule: head :- magic_<rel>^<adn>(bound), body...
- Propagation rules: for each positive non-builtin body lit
at position i:
magic_<lit-rel>^<lit-adn>(bound-of-lit) :-
magic_<rel>^<adn>(bound-of-head),
body[0..i-1]
- Add (lit-rel, lit-adn) to the worklist.
Built-ins, negation, and aggregates pass through without
generating propagation rules. EDB facts are unchanged.
3 new tests cover seed structure, equivalence on chain-3 (full
closure, 6 ancestor tuples — magic helps only when the EDB has
nodes outside the seed's transitive cone), and same-query-answers
under the rewritten program. Total 202/202.
Wiring up a `dl-saturate-magic!` driver and large-graph perf
benchmarks is left for a future iteration.
Adds the primitives a future magic-sets rewriter will compose:
dl-magic-rel-name rel adornment → "magic_<rel>^<adornment>"
dl-magic-lit rel adn bound-args → magic literal as SX list
dl-bound-args lit adornment → bound-position arg values
Rewriter algorithm (worklist over (rel, adornment) pairs,
generating seed, propagation, and adorned-rule outputs) is still
TODO — these helpers are inspection-only for now.
4 new magic tests cover naming, lit construction, and bound-args
extraction (mixed/free).
New lib/datalog/magic.sx — first piece of magic-sets:
dl-adorn-arg arg bound → "b" or "f"
dl-adorn-args args bound → adornment string
dl-adorn-goal goal → adornment under empty bound set
dl-adorn-lit lit bound → adornment of any literal
dl-vars-bound-by-lit lit bound → free vars this lit will bind
dl-init-head-bound head adn → bound set seeded from head adornment
dl-rule-sips rule head-adn → ({:lit :adornment} ...) per body lit
SIPS walks left-to-right tracking the bound set; recognises `is` and
aggregate result-vars as new binders, lets comparisons and negation
pass through with computed adornments.
Inspection-only — saturator doesn't yet consume these. Lays
groundwork for a future magic-sets transformation.
10 new tests cover pure adornment, SIPS over a chain rule,
head-fully-bound rules, comparisons, and `is`. Total 194/194.
Bug: dl-match-lit (the naive matcher used by dl-find-bindings)
was missing dl-aggregate? dispatch — it was only present in
dl-fbs-aux (semi-naive). Symptom:
(dl-query db '(count N X (p X)))
silently returned ().
Two fixes:
- Add aggregate branch to dl-match-lit before the positive case.
- dl-query-user-vars now projects only the result var (first arg)
of an aggregate goal — the aggregated var and inner-goal vars
are existentials and should not leak into substitutions.
2 new aggregate tests cover count and findall as direct query goals.
