sx-gitea: linear closure walk + working-tree importer (78/78 wire)
The closure walk rebuilt its seen-set with assoc — which on this kernel copies the entire hashtable per call — and stacked pending cids with concat; pack-cids then insertion-sorted the result. All three are quadratic, which surfaced the moment a real repo (4.5k files) went over the wire: a single push spent an hour in the walk. The seen-set is now a private dict mutated in place (dict-set!, the acl engine's own pattern), pending cids are cons-stacked, and packs are unsorted (order is irrelevant to the receiver). Wire suite stays 78/78; every clone/fetch/ push on repo-scale histories now walks each object once. lib/gitea/import.sx: working-tree importer — file-read + http-request adapt the Phase 3 wire client to a live server (gitea/http-app); staging (deterministic commits, so an interrupted import replays to identical CIDs and resumes without re-pushing) is separate from the single delta push; pack lines that exceed the pkt limit are skipped and reported. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -16,6 +16,13 @@
|
||||
; clone! / fetch! / push! / push-delete! — two in-memory forges can sync
|
||||
; with no sockets anywhere.
|
||||
;
|
||||
; Scale notes: the closure walk mutates a PRIVATE seen-dict in place
|
||||
; (dict-set!) and stacks pending cids with cons — `assoc` copies the
|
||||
; whole hashtable per call and list concat copies its head, either of
|
||||
; which makes a 10k-object walk quadratic. Pack enumeration is unsorted
|
||||
; for the same reason (artdag/sort-strings is an insertion sort; pack
|
||||
; order is irrelevant to the receiver).
|
||||
;
|
||||
; Limits: one object per pkt line => objects over ~64KB need side-band
|
||||
; chunking (future extension); gitea/pkt-fits? reports this.
|
||||
;
|
||||
@@ -117,7 +124,8 @@
|
||||
((git/tag? obj) (list (git/tag-target obj)))
|
||||
(else (list)))))
|
||||
|
||||
; walk from pending cids; returns {:seen {cid true} :missing (cids)}
|
||||
; walk from pending cids; returns {:seen {cid true} :missing (cids)}.
|
||||
; `seen` must be a PRIVATE dict — it is mutated in place.
|
||||
(define
|
||||
gitea/closure-walk
|
||||
(fn
|
||||
@@ -135,11 +143,16 @@
|
||||
(if
|
||||
(nil? obj)
|
||||
(gitea/closure-walk grepo more seen (cons cid missing))
|
||||
(gitea/closure-walk
|
||||
grepo
|
||||
(concat (gitea/obj-refs obj) more)
|
||||
(assoc seen cid true)
|
||||
missing))))))))
|
||||
(begin
|
||||
(dict-set! seen cid true)
|
||||
(gitea/closure-walk
|
||||
grepo
|
||||
(reduce
|
||||
(fn (acc r) (cons r acc))
|
||||
more
|
||||
(gitea/obj-refs obj))
|
||||
seen
|
||||
missing)))))))))
|
||||
|
||||
(define
|
||||
gitea/closure
|
||||
@@ -158,7 +171,7 @@
|
||||
(empty?
|
||||
(get (gitea/closure-walk grepo cids {} (list)) :missing))))
|
||||
|
||||
; objects needed to bring someone with `haves` up to `wants`
|
||||
; objects needed to bring someone with `haves` up to `wants` (unsorted)
|
||||
(define
|
||||
gitea/pack-cids
|
||||
(fn
|
||||
@@ -167,7 +180,7 @@
|
||||
((have-set (gitea/closure grepo haves)))
|
||||
(filter
|
||||
(fn (c) (not (get have-set c)))
|
||||
(gitea/closure-list grepo wants)))))
|
||||
(keys (gitea/closure grepo wants))))))
|
||||
|
||||
; ── wire object encoding ─────────────────────────────────────────────
|
||||
|
||||
@@ -385,7 +398,7 @@
|
||||
|
||||
; ── client ───────────────────────────────────────────────────────────
|
||||
; A remote is any dream app fn plus repo coordinates and a token — the
|
||||
; same code drives an in-memory forge or (later) a real HTTP transport.
|
||||
; same code drives an in-memory forge or a real HTTP transport.
|
||||
|
||||
(define gitea/remote (fn (app owner name token) {:name name :token token :owner owner :app app}))
|
||||
|
||||
|
||||
Reference in New Issue
Block a user