find_dupes: streaming row-chunked compare (fits ~4GB cgroup caps)

The single-matrix fix still oom-killed at n=31,937: Hermes background jobs run in a systemd scope capped near 4GB, and the n^2 uint64 xor matrix alone is 8GB. Now row chunks of 1024 stream through both hash gates (peak a few hundred MB temps), the only resident n^2 structure is the bool close-matrix (1GB), and the color-histogram L1 is computed only for rows surviving both hashes (replaces the 4GB float32 matrix). Verified on a 2,502-file corpus incl planted cross-chunk duplicate pair and an orphan copy (correctly untouched).
find_dupes: memory-safe pairwise Hamming at 30k+ scale
2026-10-29 08:04:39 -04:00 · 2026-10-29 07:29:00 -04:00 · 2026-10-28 16:33:03 -04:00
2 changed files with 60 additions and 28 deletions
@@ -63,10 +63,31 @@ context leaks onto the wrong card:
 | `--min-aspect` / `--max-aspect` | 0.45 / 1.6 | face box W/H ratio window — effect blobs rarely keep face proportions |
 | `--edge-margin` | 0 | require faces this far inside frame edges; `0` accepts edge-touching faces (original hard-rejected them) |
 | `--scene-threshold` | 30.0 | PySceneDetect ContentDetector threshold (same as original) |
+| `--probe-stride` | 1 | within a scene still needing a face, run inference every Nth frame instead of every frame (1 = original behavior). Decode stays frame-perfect; only inference is thinned |
+| `--probe-max-per-scene` | 4 | with `--probe-stride > 1`, probes per scene before giving up and waiting for the next scene start |
 | `--imgsz` | 1280 | inference size; model was trained at 1280, `640` is faster at some recall cost |
 | `--all-files` | off | also scan OP/ED files. Off by default: music videos are full of stylized graphics (feather storms, transformation emblems) that false-positive as face-shaped; episodes carry the real character close-ups |
 | `--preview-dir` | off | also write annotated copies (green boxes) of every saved frame for eyeball QC |

+### Probe scheduling (thinned inference)
+
+The original loop predicts on *every* frame until a scene yields a face —
+free on a 3090 (~0.2 s/predict), hopeless on an iGPU (~5 s/img). With
+`--probe-stride N` the decode stays frame-perfect but inference runs only
+at scene starts and then every Nth frame, giving up after
+`--probe-max-per-scene` probes and waiting for the next scene. Ported from
+the `feat/igpu-ov` branch to main (Oct 28) so both platforms share one
+schedule.
+
+| mode (same 43k-frame special, 3090 GPU1) | wall time | stills |
+|-------------------------------------------|-----------|--------|
+| default (every frame)                     | 6m11s     | 475    |
+| `--probe-stride 25 --probe-max-per-scene 4` | 2m34s   | 459 (96.6%) |
+
+Use `25`/`4` for real runs; stride 1 keeps byte-identical behavior, so
+older extracted corpora are unaffected. Accepted trade-off: a face visible
+for less than ~stride frames between probes can slip through.
+
 ## Notes

 - Runs every frame of every scene until a face qualifies, then waits
@@ -111,42 +111,53 @@ def main():
            print(f"  {i+1}/{len(files)}", flush=True)

    names = list(data)
-    # strict (complete-linkage) clustering — NO transitive chaining:
-    # seed with the ungrouped image having most neighbours, take its
-    # within-threshold neighbourhood, then keep only members pairwise
-    # within threshold. Union-find chaining merges unrelated frames
-    # after enough hops (verified failure: 90-frame 'group' of six
-    # different scenes).
    n = len(names)
-    print(f"Pairwise compare {n}x{n} (numpy hamming) ...")
+    print(f"Pairwise compare {n}x{n} (streaming hamming, chunked) ...")
+
+    # Memory-safe at 30k+ scale under ~4 GB cgroup caps (verified: the
+    # full n^2 uint64 xor matrix — 8 GB at n=32k — got a background run
+    # oom-killed in a hermes-worker systemd scope on 2026-10-29). Row
+    # chunks of 1024 keep peak temps at a few hundred MB; only the final
+    # bool close-matrix (n^2 bytes) stays resident. uint8 distances
+    # suffice (max 64) via np.bitwise_count (numpy>=2.3) or a per-byte
+    # popcount LUT.
+    def hamming_chunk(mat, rows):
+        x = mat[rows, None] ^ mat[None, :]
+        if hasattr(np, "bitwise_count"):
+            ham = np.bitwise_count(x)
+        else:
+            bc = np.array([bin(i).count("1") for i in range(256)], dtype=np.uint8)
+            x8 = x.view(np.uint8).reshape(x.shape + (8,))
+            ham = np.zeros(x.shape, dtype=np.uint8)
+            for b in range(8):
+                ham += bc[x8[..., b]]
+            del x8
+        del x
+        return ham
+
    dmat = np.array([data[f][0] for f in names], dtype=np.uint64)
-    # hamming distances via xor + popcount (work in int64; numpy has no
-    # sane uint64 shift-add promotion)
-    x = (dmat[:, None] ^ dmat[None, :]).astype(np.int64)
-    ham = np.zeros(x.shape, dtype=np.int16)
-    for b in range(64):
-        ham += ((x >> b) & 1).astype(np.int16)
+    amat = np.array([data[f][1] for f in names], dtype=np.uint64)
+    max_a = args.max_adist if args.max_adist is not None else 8
+    CH = 1024
+    close = np.zeros((n, n), dtype=bool)
+    for s in range(0, n, CH):
+        rows = slice(s, min(s + CH, n))
+        close[rows] = ((hamming_chunk(dmat, rows) <= args.max_dist)
+                       & (hamming_chunk(amat, rows) <= max_a))
+        if (s // CH) % 8 == 0:
+            print(f"  hashes {min(s + CH, n)}/{n}", flush=True)
+    del dmat, amat

    # combined close matrix: dHash AND aHash AND color-histogram gates.
    # dHash alone has a false-positive floor (verified: distinct frames
    # at distance <=10); a second, differently-computed hash plus a
    # color signature is what separates 'same frame replayed' from
-    # 'same show, different shot'.
-    close = ham <= args.max_dist
-
-    amat = np.array([data[f][1] for f in names], dtype=np.uint64)
-    xa = (amat[:, None] ^ amat[None, :]).astype(np.int64)
-    aham = np.zeros(xa.shape, dtype=np.int16)
-    for b in range(64):
-        aham += ((xa >> b) & 1).astype(np.int16)
-    close &= aham <= (args.max_adist
-                      if args.max_adist is not None else 8)
-
+    # 'same show, different shot'. Color L1 is computed per row only for
+    # rows that survived both hashes — few rows at real corpus scale,
+    # and it replaces the old n^2 float32 matrix (4 GB at n=32k).
    hmat = np.stack([data[f][3] for f in names])  # N x 192 hist rows
-    cdist = np.zeros((n, n), dtype=np.float32)
-    for i in range(n):  # row-wise keeps it ~seconds and RAM-light
-        cdist[i] = np.abs(hmat - hmat[i]).sum(axis=1)
-    close &= cdist <= args.max_cdist
+    for i in np.flatnonzero(close.any(axis=1)):
+        close[i] &= np.abs(hmat - hmat[i]).sum(axis=1) <= args.max_cdist
    np.fill_diagonal(close, False)

    # strict (complete-linkage) clustering — NO transitive chaining: