Bug: dl-magic-query was skipping EDB facts for relations that had
rules ("rule-headed"). When a single relation has both EDB facts
and rules deriving more (mixed EDB+IDB), the rewritten run would
miss the EDB portion entirely, producing too few or zero results.
Fix: copy ALL existing facts to the internal mdb regardless of
whether the relation has rules. EDB-only relations bring their
tuples; mixed relations bring both EDB and any pre-saturated IDB
(which the rewritten rules would re-derive anyway).
1 new test: link relation seeded with 3 EDB tuples plus a
recursive rule via via/2. dl-magic-query rooted at `a` returns
2 results (a→b direct, a→c via via(a,e), link(e,c)).
Bug: dl-magic-query was always trying to seed a magic_<rel>^<adn>
fact for the query goal. For aggregate goals like (count N X (p X))
this produced a non-ground "fact" (magic_count^... N X (p X)) and
dl-add-fact! correctly rejected it, surfacing as an error.
Fix: dl-magic-query now detects built-in / aggregate / negation
goals up front and dispatches to plain dl-query for those cases —
magic-sets only applies to positive non-builtin literals against
rule-defined relations. Other shapes don't benefit from the
rewrite anyway.
1 new test confirms (count N X (p X)) returns the expected
{:N 3} via dl-magic-query.
Adds dl-demo-org-rules: (subordinate Mgr Emp) over a (manager
EMP MGR) graph, and (headcount Mgr N) using count aggregation
grouped by manager. Demonstrates real-world hierarchy queries
(e.g. "everyone reporting up to the CEO") + per-manager rollup.
3 new demo tests: transitive subordinates of CEO (5 entries),
CEO headcount, and direct manager headcount.
(dl-rules-of db rel-name) → list of rules with head matching
the given relation name. Useful for tooling and debugging
("show me how this relation is derived") without exposing the
internal :rules list directly.
2 new api tests cover hit and miss cases.
Returns true iff one more saturation step would derive no new
tuples. Walks every rule under the current bindings and short-
circuits as soon as one derivation would add a fresh tuple.
Useful in tests that want to assert "no work left" after a call,
or for tooling that wants to know whether `dl-saturate!` would
do anything.
3 new eval tests cover the after-saturation, before-saturation,
and after-assert states.
Demonstrates the practical effect of goal-directed evaluation:
chain of 12 nodes, semi-naive derives the full ancestor closure
(78 = 12·13/2 tuples), while a magic-rooted query at node 0
returns only its 12 descendants. Concrete check that magic
limits derivation to the query's transitive cone.
Wipes every rule-headed relation (the IDB) — leaves EDB facts and
rule definitions intact. Useful for inspecting the EDB-only
baseline or for forcing a clean re-saturation.
(dl-saturate! db)
(dl-clear-idb! db) ; ancestor relation now empty
(dl-saturate! db) ; re-derives ancestor from parents
2 new api tests verify IDB-wipe and EDB-preservation.
Symmetric to dl-eval but routes single-positive-literal queries
through dl-magic-query for goal-directed evaluation. Multi-literal
query bodies fall back to standard dl-query (magic-sets is wired
for single goals only).
(dl-eval-magic source-string "?- ancestor(a, X).")
1 new api test.
Two disjoint chains, query rooted in cluster 1. Semi-naive
derives the full closure over both clusters (6 ancestor tuples).
Magic-sets only seeds magic_ancestor^bf for cluster 1, so only
2 query-relevant tuples are returned (a→b, a→c). The test
asserts both numbers, demonstrating the actual perf-shape
benefit of goal-directed evaluation.
End-to-end magic-sets entry point. Given (db, query-goal):
- copies the caller's EDB facts (relations not headed by any
rule) into a fresh internal db
- adds the magic seed fact
- adds the rewritten rules
- saturates and runs the query
- returns the substitution list
Caller's db is untouched. Equivalent to dl-query for any
fully-stratifiable program; intended as a perf alternative on
goal-shaped queries against large recursive relations.
2 new tests: equivalence to dl-query on chain-3 ancestor, and
non-mutation of the caller's db (rules count unchanged).
dl-magic-rewrite rules query-rel adn args returns:
{:rules <rewritten-rules> :seed <magic-seed-fact>}
Worklist over (rel, adn) pairs starts from the query and stops
when no new pairs appear. For each rule with head matching a
worklist pair:
- Adorned rule: head :- magic_<rel>^<adn>(bound), body...
- Propagation rules: for each positive non-builtin body lit
at position i:
magic_<lit-rel>^<lit-adn>(bound-of-lit) :-
magic_<rel>^<adn>(bound-of-head),
body[0..i-1]
- Add (lit-rel, lit-adn) to the worklist.
Built-ins, negation, and aggregates pass through without
generating propagation rules. EDB facts are unchanged.
3 new tests cover seed structure, equivalence on chain-3 (full
closure, 6 ancestor tuples — magic helps only when the EDB has
nodes outside the seed's transitive cone), and same-query-answers
under the rewritten program. Total 202/202.
Wiring up a `dl-saturate-magic!` driver and large-graph perf
benchmarks is left for a future iteration.
Adds the primitives a future magic-sets rewriter will compose:
dl-magic-rel-name rel adornment → "magic_<rel>^<adornment>"
dl-magic-lit rel adn bound-args → magic literal as SX list
dl-bound-args lit adornment → bound-position arg values
Rewriter algorithm (worklist over (rel, adornment) pairs,
generating seed, propagation, and adorned-rule outputs) is still
TODO — these helpers are inspection-only for now.
4 new magic tests cover naming, lit construction, and bound-args
extraction (mixed/free).
New lib/datalog/magic.sx — first piece of magic-sets:
dl-adorn-arg arg bound → "b" or "f"
dl-adorn-args args bound → adornment string
dl-adorn-goal goal → adornment under empty bound set
dl-adorn-lit lit bound → adornment of any literal
dl-vars-bound-by-lit lit bound → free vars this lit will bind
dl-init-head-bound head adn → bound set seeded from head adornment
dl-rule-sips rule head-adn → ({:lit :adornment} ...) per body lit
SIPS walks left-to-right tracking the bound set; recognises `is` and
aggregate result-vars as new binders, lets comparisons and negation
pass through with computed adornments.
Inspection-only — saturator doesn't yet consume these. Lays
groundwork for a future magic-sets transformation.
10 new tests cover pure adornment, SIPS over a chain rule,
head-fully-bound rules, comparisons, and `is`. Total 194/194.
Bug: dl-match-lit (the naive matcher used by dl-find-bindings)
was missing dl-aggregate? dispatch — it was only present in
dl-fbs-aux (semi-naive). Symptom:
(dl-query db '(count N X (p X)))
silently returned ().
Two fixes:
- Add aggregate branch to dl-match-lit before the positive case.
- dl-query-user-vars now projects only the result var (first arg)
of an aggregate goal — the aggregated var and inner-goal vars
are existentials and should not leak into substitutions.
2 new aggregate tests cover count and findall as direct query goals.
Single-call entry: dl-eval source-string query-string parses
both, builds a db via dl-program, saturates implicitly, runs
the query (extracted from the parsed `?- ...` clause), and
returns the substitution list.
Most user-friendly path:
(dl-eval "parent(a, b). ..." "?- ancestor(a, X).")
2 new api tests cover ancestor and multi-goal usage.
Adds a user-facing strategy hook: dl-set-strategy! db strategy and
dl-get-strategy db. Default :semi-naive; :magic is accepted but
the actual transformation is deferred — the saturator currently
falls back to semi-naive regardless. Lets us tick the Phase 6
"Optional pass — guarded behind dl-set-strategy!" checkbox while
keeping the equivalence/perf tests pending future work.
3 new eval tests.
dl-demo-shortest-path-rules: path enumerates X→Z with cost
W = sum of edge weights via is/+; shortest filters to the
minimum cost path per (X, Y) pair via min aggregation.
3 demo tests cover direct/multi-hop choice, multi-hop wins on
cheaper route, and unreachable-empty.
Note: cycles produce infinite distance values without a depth
filter; the rule docstring flags this and suggests adding
(<, D, MAX) for graphs that may cycle.
Returns {<rel-name>: tuple-count} for relations with any tuples
or that are rule-headed (so empty IDB shows as :rel 0 rather than
disappearing). Skips placeholder entries from internal
dl-ensure-rel! calls. 4 tests cover basic, empty IDB, mixed
EDB+IDB, and empty-db cases.
db gains :facts-index {<rel>: {<first-arg-key>: tuples}} mirroring
the membership :facts-keys index. dl-add-fact! populates the index;
dl-match-positive walks the body literal's first arg under the
current subst — when it's bound to a non-var, look up by (str arg)
instead of scanning the full relation.
For chain-style recursive rules (parent X Y), (ancestor Y Z) the
inner Y has at most one parent, so the inner lookup returns 0–1
tuples instead of N. chain-25 saturation drops from ~33s to ~18s
real (~2x). chain-50 still long but tractable; next bottleneck is
subst dict copies during unification.
dl-retract! refreshed to keep the new index consistent: kept-index
rebuilt during EDB filter, IDB wipes clear all three slots.
Differential semi-naive test bumped to chain-12, semi-only count
test to chain-25.
Adds (cotagged P T1 T2) — P has both T1 and T2 with T1 != T2 — and
(tag-pair-count T1 T2 N) which counts posts cotagged with each
distinct (T1, T2) pair. Demonstrates count aggregation against a
recursive-then-aggregated stream of derived tuples.
2 new demo tests: cooking + vegetarian co-occurrence on a small
data set, and a count-of-co-occurrences query.
dl-query now auto-dispatches on the first element's shape:
- positive literal (head is a symbol) or {:neg ...} dict → wrap
- list of literals → conjunctive query
dl-query-coerce normalizes; dl-query-user-vars collects the union
of user-named vars (deduped, '_' filtered) for projection. Old
single-literal callers unchanged.
(dl-query db '(p X)) ; single
(dl-query db '((p X) (q X))) ; conjunction
(dl-query db (list '(n X) '(> X 2))) ; with comparison
2 new api tests cover multi-goal AND and conjunction with comparison.
Bug: dl-check-stratifiable iterated body literals looking only for
explicit :neg literals, missing aggregate cycles. Now also walks
aggregates via dl-aggregate-dep-edge — q(N) :- count(N, X, q(X))
correctly errors out at saturation time.
3 new tests cover:
- recursion-through-aggregation rejected
- negation + aggregation coexist when in different strata
- min over empty derived relation produces no result
Adds the canonical Phase 10 example from the plan: "Posts about
cooking by people I follow (transitively)." dl-demo-cooking-rules
defines reach over the follow graph (recursive transitive closure)
and cooking-post-by-network joining reach + authored + (tagged P
cooking). 3 new demo tests cover transitive network, direct-only
follow, and empty-network cases.
(findall L V Goal) — bind L to the distinct V values for which Goal
holds, or the empty list when none. One-line addition to
dl-do-aggregate that returns the unreduced list. Tests cover EDB,
derived relation, and empty cases.
Useful for "give me all the X such that ..." queries without
scalar reduction.
(p X _), (p _ Y) — the two _ are now different variables, matching
standard Datalog semantics. Previously both _ symbols were the same
SX symbol, so unification across them gave wrong answers.
Fix in db.sx: dl-rename-anon-term + dl-rename-anon-lit walk a term
or literal and replace each '_' symbol with a fresh _anon<N>.
dl-make-anon-renamer returns a counter-based name generator scoped
per call. dl-rename-anon-rule applies it to head and body of a
rule. dl-add-rule! invokes the renamer before safety check.
eval.sx: dl-query renames anon vars in the goal before search and
filters '_' out of the projection so user-facing results aren't
polluted with internal _anon<N> bindings.
The previous "underscore in head ok" test now correctly rejects
(p X _) :- q(X) as unsafe (the head's fresh anon var has no body
binder). New "underscore in body only" test confirms the safe
case. Two regression tests for rule-level and goal-level
independence.
dl-find-bindings now uses dl-fb-aux lits db subst i n (indexed
iteration via nth) instead of recursive (rest lits). Eliminates
O(N²) list-copy per body of length N. chain-15 saturation 25s
→ 16s; chain-25 finishes in 33s real (vs. timeout previously).
Bumped semi_naive tests to chain-10 differential + chain-15
semi-only count (was chain-5/chain-5). Blocker entry refreshed.
New lib/datalog/demo.sx with three Datalog-as-query-language demos
over synthetic rose-ash data:
Federation: (mutual A B), (reachable A B), (foaf A C) over a
follows graph.
Content: (post-likes P N) via count aggregation, (popular P)
for likes >= 3, (interesting Me P) joining follows
+ authored + popular.
Permissions: (in-group A G) over transitive subgroup chains,
(can-access A R).
10 tests run each program against in-memory EDB tuples loaded via
dl-program-data.
Wiring to PostgreSQL and exposing as a service endpoint (/internal
/datalog) is out of scope for this loop — both would require
edits outside lib/datalog/. Programs above document the EDB shape
a real loader would populate.
db gains a parallel :facts-keys {<rel>: {<tuple-string>: true}}
index alongside :facts. dl-tuple-key derives a stable string via
(str lit) — (p 30) and (p 30.0) collide correctly because SX
prints them identically. dl-add-fact! membership is now O(1)
instead of O(n) list scan; insert sequences for relations sized
N drop from O(N²) to O(N).
Wall clock on chain-7 saturation halves (~12s → ~6s); chain-15
roughly halves (~50s → ~25s) under shared CPU. Larger chains
still slow due to body-join overhead in dl-find-bindings —
Blocker entry refreshed with proposed follow-ups.
dl-retract! keeps both indices consistent: kept-keys is rebuilt
during the EDB filter, IDB wipes clear both lists and key dicts.
New lib/datalog/api.sx: dl-program-data facts rules takes SX data
lists. Rules accept either dict form or list form using <- as the
rule arrow (since SX parses :- as a keyword). dl-rule constructor
for the dict shape. dl-assert! adds a fact and re-saturates;
dl-retract! drops EDB matches, wipes all rule-headed IDB
relations, and re-saturates from scratch — simplest correct
semantics until provenance tracking arrives.
9 API tests cover ancestor closure via data, dict-rule form,
dl-rule constructor, incremental assert/retract, cyclic-graph
reach, assert into empty, fact-style rule (no arrow), dict
passthrough.
New lib/datalog/aggregates.sx: (count R V Goal), (sum R V Goal),
(min R V Goal), (max R V Goal). dl-eval-aggregate runs
dl-find-bindings on the goal under the outer subst, collects
distinct values of V, applies the operator, binds R. Empty input:
count/sum return 0; min/max produce no binding (rule fails).
Group-by emerges naturally from outer-subst substitution into the
goal — `popular(P) :- post(P), count(N, U, liked(U, P)), >=(N, 3).`
counts per-post.
Stratifier extended: dl-aggregate-dep-edge contributes a
negation-like edge so the aggregate's goal relation is fully
derived before the aggregate fires (non-monotonicity respected).
Safety relaxed for aggregates: goal-internal vars are existentials,
only the result var becomes bound.
New lib/datalog/strata.sx: dl-build-dep-graph (relation -> deps with
:neg flag), Floyd-Warshall reachability, SCC-via-mutual-reach for
non-stratifiability detection, iterative dl-compute-strata, and
dl-group-rules-by-stratum.
eval.sx refactor:
- dl-saturate-rules! db rules — semi-naive worker over a rule subset
- dl-saturate! db — stratified driver. Rejects non-stratifiable
programs at saturation time, then iterates strata in order
- dl-match-negation — succeeds iff inner positive match is empty
Order-aware safety in dl-rule-check-safety (Phase 4) already
required negation vars to be bound by a prior positive literal.
Stratum dict keys are strings (SX dicts don't accept ints).
Phase 6 magic sets deferred — opt-in path, semi-naive default
suffices for current workloads.
dl-saturate! is now semi-naive: tracks a per-relation delta dict,
and on each iteration walks every positive body-literal position,
substituting the delta of its relation while joining the rest
against the previous-iteration DB. Candidates are collected before
mutating the DB so the "full" sides see a consistent snapshot.
Rules with no positive body literal (e.g. (p X) :- (= X 5).)
fall back to a one-shot naive pass via dl-collect-rule-candidates.
dl-saturate-naive! retained as the reference implementation; 8
differential tests compare per-relation tuple counts on every
recursive program. Switched dl-tuple-member? to indexed iteration
instead of recursive rest (eliminates per-step list copy). Larger
chains under bundled conformance trip O(n) membership × CPU
sharing — added a Blocker to swap relations to hash-set membership.
New lib/datalog/builtins.sx: (< <= > >= = !=) and (is X expr) with
+ - * /. dl-eval-arith recursively evaluates nested compounds.
Safety analysis now walks body left-to-right tracking the bound
set: comparisons require all args bound, is RHS vars must be bound
(LHS becomes bound), = special-cases the var/non-var combos.
db.sx keeps the simple safety check as a forward-reference
fallback; builtins.sx redefines dl-rule-check-safety to the
comprehensive version. eval.sx dispatches built-ins through
dl-eval-builtin instead of erroring. 19 new tests.
