dl-magic-query could silently diverge from dl-query when an
aggregate's inner-goal relation was IDB. The rewriter passes
aggregate body lits through unchanged (no magic propagation
generated for them), so the inner relation was empty in the magic
db and the aggregate returned 0. Repro:
(dl-eval-magic
"u(a). u(b). u(c). u(d). banned(b). banned(d).
active(X) :- u(X), not(banned(X)).
n(N) :- count(N, X, active(X))."
"?- n(N).")
=> ({:N 0}) ; should be ({:N 2})
dl-magic-query now pre-saturates the source db before copying facts
into the magic db. This guarantees equivalence with dl-query for
every stratified program; the magic benefit still comes from
goal-directed re-derivation of the query relation under the seed
(which matters for large recursive joins). The existing test cases
happened to dodge this because their aggregate inner-goals were all
EDB.
1 new regression test; conformance 274/274.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug: dl-magic-query crashed with cryptic "rest: 1 list arg" when
the goal argument was a string, number, or arbitrary dict. The
first thing the function does is dl-rel-name + dl-adorn-goal,
both of which assume a positive-literal list shape.
Fix: explicit shape check up front. A goal must be a non-empty
list whose first element is a symbol. Otherwise raise with a
clear diagnostic. Built-in / aggregate / negation dispatch (the
fall-back to dl-query) is unchanged.
2 new magic tests cover string and bare-dict goal rejection.
Bug-hunt round probed magic-sets against many edge cases. No new
bugs surfaced. Added regression tests for two patterns that
exercise the worklist post-fix:
- 3-stratum program (a → c via not-b → d via not-banned).
Distinct rule heads at three strata; magic must rewrite each.
- Aggregate-derived chain (count(src) → cnt → active threshold).
Magic correctly handles multi-step aggregate dependencies.
Magic-sets is robust against: 3-stratum negation, aggregate
chains, mutual recursion, all-bound goals, multi-arity rules,
diagonal queries, EDB-only goals, and rules whose body has
identical positive lits.
Real bug: the worklist used (set! queue (rest queue)) to pop the
head, which left queue bound to a fresh empty list as soon as the
last item was popped. Subsequent (append! queue ...) was a no-op
on the empty list — so when the head's rewrite generated new
(rel, adn) pairs to enqueue, they vanished. Multi-relation
programs (e.g. shortest -> path -> edge, or chained derived
relations) only had their head's rules rewritten; downstream
rules silently dropped.
Fix: use an index-based loop (idx 0 → len queue), with append!
adding to the same list. Items added after the current pointer
are picked up in subsequent iterations.
2 new regression tests:
- 4-level chain (a → r1 → r2 → r3 → r4) under magic returns 2
- shortest-path demo via magic equals dl-query (1 result)
Bug: dl-magic-query was skipping EDB facts for relations that had
rules ("rule-headed"). When a single relation has both EDB facts
and rules deriving more (mixed EDB+IDB), the rewritten run would
miss the EDB portion entirely, producing too few or zero results.
Fix: copy ALL existing facts to the internal mdb regardless of
whether the relation has rules. EDB-only relations bring their
tuples; mixed relations bring both EDB and any pre-saturated IDB
(which the rewritten rules would re-derive anyway).
1 new test: link relation seeded with 3 EDB tuples plus a
recursive rule via via/2. dl-magic-query rooted at `a` returns
2 results (a→b direct, a→c via via(a,e), link(e,c)).
Bug: dl-magic-query was always trying to seed a magic_<rel>^<adn>
fact for the query goal. For aggregate goals like (count N X (p X))
this produced a non-ground "fact" (magic_count^... N X (p X)) and
dl-add-fact! correctly rejected it, surfacing as an error.
Fix: dl-magic-query now detects built-in / aggregate / negation
goals up front and dispatches to plain dl-query for those cases —
magic-sets only applies to positive non-builtin literals against
rule-defined relations. Other shapes don't benefit from the
rewrite anyway.
1 new test confirms (count N X (p X)) returns the expected
{:N 3} via dl-magic-query.
Demonstrates the practical effect of goal-directed evaluation:
chain of 12 nodes, semi-naive derives the full ancestor closure
(78 = 12·13/2 tuples), while a magic-rooted query at node 0
returns only its 12 descendants. Concrete check that magic
limits derivation to the query's transitive cone.
Two disjoint chains, query rooted in cluster 1. Semi-naive
derives the full closure over both clusters (6 ancestor tuples).
Magic-sets only seeds magic_ancestor^bf for cluster 1, so only
2 query-relevant tuples are returned (a→b, a→c). The test
asserts both numbers, demonstrating the actual perf-shape
benefit of goal-directed evaluation.
End-to-end magic-sets entry point. Given (db, query-goal):
- copies the caller's EDB facts (relations not headed by any
rule) into a fresh internal db
- adds the magic seed fact
- adds the rewritten rules
- saturates and runs the query
- returns the substitution list
Caller's db is untouched. Equivalent to dl-query for any
fully-stratifiable program; intended as a perf alternative on
goal-shaped queries against large recursive relations.
2 new tests: equivalence to dl-query on chain-3 ancestor, and
non-mutation of the caller's db (rules count unchanged).
dl-magic-rewrite rules query-rel adn args returns:
{:rules <rewritten-rules> :seed <magic-seed-fact>}
Worklist over (rel, adn) pairs starts from the query and stops
when no new pairs appear. For each rule with head matching a
worklist pair:
- Adorned rule: head :- magic_<rel>^<adn>(bound), body...
- Propagation rules: for each positive non-builtin body lit
at position i:
magic_<lit-rel>^<lit-adn>(bound-of-lit) :-
magic_<rel>^<adn>(bound-of-head),
body[0..i-1]
- Add (lit-rel, lit-adn) to the worklist.
Built-ins, negation, and aggregates pass through without
generating propagation rules. EDB facts are unchanged.
3 new tests cover seed structure, equivalence on chain-3 (full
closure, 6 ancestor tuples — magic helps only when the EDB has
nodes outside the seed's transitive cone), and same-query-answers
under the rewritten program. Total 202/202.
Wiring up a `dl-saturate-magic!` driver and large-graph perf
benchmarks is left for a future iteration.
Adds the primitives a future magic-sets rewriter will compose:
dl-magic-rel-name rel adornment → "magic_<rel>^<adornment>"
dl-magic-lit rel adn bound-args → magic literal as SX list
dl-bound-args lit adornment → bound-position arg values
Rewriter algorithm (worklist over (rel, adornment) pairs,
generating seed, propagation, and adorned-rule outputs) is still
TODO — these helpers are inspection-only for now.
4 new magic tests cover naming, lit construction, and bound-args
extraction (mixed/free).
New lib/datalog/magic.sx — first piece of magic-sets:
dl-adorn-arg arg bound → "b" or "f"
dl-adorn-args args bound → adornment string
dl-adorn-goal goal → adornment under empty bound set
dl-adorn-lit lit bound → adornment of any literal
dl-vars-bound-by-lit lit bound → free vars this lit will bind
dl-init-head-bound head adn → bound set seeded from head adornment
dl-rule-sips rule head-adn → ({:lit :adornment} ...) per body lit
SIPS walks left-to-right tracking the bound set; recognises `is` and
aggregate result-vars as new binders, lets comparisons and negation
pass through with computed adornments.
Inspection-only — saturator doesn't yet consume these. Lays
groundwork for a future magic-sets transformation.
10 new tests cover pure adornment, SIPS over a chain rule,
head-fully-bound rules, comparisons, and `is`. Total 194/194.
