Commit Graph
6 Commits
Author SHA1 Message Date
alex b35d3da375 find_dupes: memory-safe pairwise Hamming at 30k+ scale
The 64-pass int64 shift-add materialized ~8GB temporaries per pass
(~27GB peak at n=31,937) — OOM on a 31GB box. Replace with
np.bitwise_count when available (numpy>=2.3) and a per-byte popcount
LUT fallback; uint8 results suffice (max distance 64). Verified on a
smoke corpus (exact-duplicate groups found, move semantics intact)
before the Beyblade X 31,937-still run.
2026-10-29 07:29:00 -04:00
alex de177c8f59 find_dupes: build aHash matrix as uint64 before xor (int64 array ctor overflows) 2026-10-18 18:53:58 -04:00
alex 107f48d118 find_dupes: triple-gate closeness (dHash<=6, aHash<=8, color-hist L1<=0.10) + strict linkage
Measured on verified same/different pairs: dHash alone has a false-
positive floor for same-style anime frames (distinct shots at distance
<=10 and even <=6 with different color content). Same-shot broadcast
replays are pixel-twins: near-zero color-hist distance and low aHash
distance too. Requires ALL three gates, no transitive chaining.
2026-10-18 18:51:45 -04:00
alex 7e128c4c32 find_dupes: replace union-find with strict complete-linkage clustering
Verified failure with 10/64 threshold + transitive chaining: a 90-image
'group' contained six different scenes (blonde outdoors, storm school,
night bedroom, manic grin, cat street, dining room) — chains of
similar-but-distinct frames glue unrelated shots together. Strict
linkage requires every member within threshold of every other.
Default quarantine dir renamed duplicates/.
2026-10-18 18:47:08 -04:00
alex 4816b007c0 find_dupes: fix uint64 hamming accumulation (numpy casts to float64) 2026-10-18 18:42:51 -04:00
alex a8cf8957d5 Add find_dupes.py: dHash near-duplicate clusterer for recurring scenes
Recurring OP/ED animation shots and eyecatch cards reappear in every
episode, so scene-gated extraction saves one copy per broadcast.
Reports groups with a keepsake suggestion; --move quarantines
redundant members to dupes/ (reversible). CPU-only perceptual hashing,
paranoid default threshold 10/64 bits.
2026-10-18 18:40:43 -04:00