The single-matrix fix still oom-killed at n=31,937: Hermes background
jobs run in a systemd scope capped near 4GB, and the n^2 uint64 xor
matrix alone is 8GB. Now row chunks of 1024 stream through both hash
gates (peak a few hundred MB temps), the only resident n^2 structure
is the bool close-matrix (1GB), and the color-histogram L1 is computed
only for rows surviving both hashes (replaces the 4GB float32 matrix).
Verified on a 2,502-file corpus incl planted cross-chunk duplicate
pair and an orphan copy (correctly untouched).
The 64-pass int64 shift-add materialized ~8GB temporaries per pass
(~27GB peak at n=31,937) — OOM on a 31GB box. Replace with
np.bitwise_count when available (numpy>=2.3) and a per-byte popcount
LUT fallback; uint8 results suffice (max distance 64). Verified on a
smoke corpus (exact-duplicate groups found, move semantics intact)
before the Beyblade X 31,937-still run.
Measured on verified same/different pairs: dHash alone has a false-
positive floor for same-style anime frames (distinct shots at distance
<=10 and even <=6 with different color content). Same-shot broadcast
replays are pixel-twins: near-zero color-hist distance and low aHash
distance too. Requires ALL three gates, no transitive chaining.
Verified failure with 10/64 threshold + transitive chaining: a 90-image
'group' contained six different scenes (blonde outdoors, storm school,
night bedroom, manic grin, cat street, dining room) — chains of
similar-but-distinct frames glue unrelated shots together. Strict
linkage requires every member within threshold of every other.
Default quarantine dir renamed duplicates/.
Recurring OP/ED animation shots and eyecatch cards reappear in every
episode, so scene-gated extraction saves one copy per broadcast.
Reports groups with a keepsake suggestion; --move quarantines
redundant members to dupes/ (reversible). CPU-only perceptual hashing,
paranoid default threshold 10/64 bits.
alex