sx-gitea: linear closure walk + working-tree importer (78/78 wire)

The closure walk rebuilt its seen-set with assoc — which on this kernel
copies the entire hashtable per call — and stacked pending cids with
concat; pack-cids then insertion-sorted the result. All three are
quadratic, which surfaced the moment a real repo (4.5k files) went over
the wire: a single push spent an hour in the walk. The seen-set is now a
private dict mutated in place (dict-set!, the acl engine's own pattern),
pending cids are cons-stacked, and packs are unsorted (order is
irrelevant to the receiver). Wire suite stays 78/78; every clone/fetch/
push on repo-scale histories now walks each object once.

lib/gitea/import.sx: working-tree importer — file-read + http-request
adapt the Phase 3 wire client to a live server (gitea/http-app);
staging (deterministic commits, so an interrupted import replays to
identical CIDs and resumes without re-pushing) is separate from the
single delta push; pack lines that exceed the pkt limit are skipped and
reported.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-03 17:34:31 +00:00
parent 9c009b07db
commit e48dbd42b4
2 changed files with 113 additions and 9 deletions

View File

@@ -16,6 +16,13 @@
; clone! / fetch! / push! / push-delete! — two in-memory forges can sync
; with no sockets anywhere.
;
; Scale notes: the closure walk mutates a PRIVATE seen-dict in place
; (dict-set!) and stacks pending cids with cons — `assoc` copies the
; whole hashtable per call and list concat copies its head, either of
; which makes a 10k-object walk quadratic. Pack enumeration is unsorted
; for the same reason (artdag/sort-strings is an insertion sort; pack
; order is irrelevant to the receiver).
;
; Limits: one object per pkt line => objects over ~64KB need side-band
; chunking (future extension); gitea/pkt-fits? reports this.
;
@@ -117,7 +124,8 @@
((git/tag? obj) (list (git/tag-target obj)))
(else (list)))))
; walk from pending cids; returns {:seen {cid true} :missing (cids)}
; walk from pending cids; returns {:seen {cid true} :missing (cids)}.
; `seen` must be a PRIVATE dict — it is mutated in place.
(define
gitea/closure-walk
(fn
@@ -135,11 +143,16 @@
(if
(nil? obj)
(gitea/closure-walk grepo more seen (cons cid missing))
(gitea/closure-walk
grepo
(concat (gitea/obj-refs obj) more)
(assoc seen cid true)
missing))))))))
(begin
(dict-set! seen cid true)
(gitea/closure-walk
grepo
(reduce
(fn (acc r) (cons r acc))
more
(gitea/obj-refs obj))
seen
missing)))))))))
(define
gitea/closure
@@ -158,7 +171,7 @@
(empty?
(get (gitea/closure-walk grepo cids {} (list)) :missing))))
; objects needed to bring someone with `haves` up to `wants`
; objects needed to bring someone with `haves` up to `wants` (unsorted)
(define
gitea/pack-cids
(fn
@@ -167,7 +180,7 @@
((have-set (gitea/closure grepo haves)))
(filter
(fn (c) (not (get have-set c)))
(gitea/closure-list grepo wants)))))
(keys (gitea/closure grepo wants))))))
; ── wire object encoding ─────────────────────────────────────────────
@@ -385,7 +398,7 @@
; ── client ───────────────────────────────────────────────────────────
; A remote is any dream app fn plus repo coordinates and a token — the
; same code drives an in-memory forge or (later) a real HTTP transport.
; same code drives an in-memory forge or a real HTTP transport.
(define gitea/remote (fn (app owner name token) {:name name :token token :owner owner :app app}))