find_dupes: streaming row-chunked compare (fits ~4GB cgroup caps)

The single-matrix fix still oom-killed at n=31,937: Hermes background
jobs run in a systemd scope capped near 4GB, and the n^2 uint64 xor
matrix alone is 8GB. Now row chunks of 1024 stream through both hash
gates (peak a few hundred MB temps), the only resident n^2 structure
is the bool close-matrix (1GB), and the color-histogram L1 is computed
only for rows surviving both hashes (replaces the 4GB float32 matrix).
Verified on a 2,502-file corpus incl planted cross-chunk duplicate
pair and an orphan copy (correctly untouched).

README.md

@@ -63,10 +63,31 @@ context leaks onto the wrong card:
 | `--min-aspect` / `--max-aspect` | 0.45 / 1.6 | face box W/H ratio window — effect blobs rarely keep face proportions |
 | `--edge-margin` | 0 | require faces this far inside frame edges; `0` accepts edge-touching faces (original hard-rejected them) |
 | `--scene-threshold` | 30.0 | PySceneDetect ContentDetector threshold (same as original) |
+| `--probe-stride` | 1 | within a scene still needing a face, run inference every Nth frame instead of every frame (1 = original behavior). Decode stays frame-perfect; only inference is thinned |
+| `--probe-max-per-scene` | 4 | with `--probe-stride > 1`, probes per scene before giving up and waiting for the next scene start |
 | `--imgsz` | 1280 | inference size; model was trained at 1280, `640` is faster at some recall cost |
 | `--all-files` | off | also scan OP/ED files. Off by default: music videos are full of stylized graphics (feather storms, transformation emblems) that false-positive as face-shaped; episodes carry the real character close-ups |
 | `--preview-dir` | off | also write annotated copies (green boxes) of every saved frame for eyeball QC |
 
+### Probe scheduling (thinned inference)
+
+The original loop predicts on *every* frame until a scene yields a face —
+free on a 3090 (~0.2 s/predict), hopeless on an iGPU (~5 s/img). With
+`--probe-stride N` the decode stays frame-perfect but inference runs only
+at scene starts and then every Nth frame, giving up after
+`--probe-max-per-scene` probes and waiting for the next scene. Ported from
+the `feat/igpu-ov` branch to main (Oct 28) so both platforms share one
+schedule.
+
+| mode (same 43k-frame special, 3090 GPU1) | wall time | stills |
+|-------------------------------------------|-----------|--------|
+| default (every frame)                     | 6m11s     | 475    |
+| `--probe-stride 25 --probe-max-per-scene 4` | 2m34s   | 459 (96.6%) |
+
+Use `25`/`4` for real runs; stride 1 keeps byte-identical behavior, so
+older extracted corpora are unaffected. Accepted trade-off: a face visible
+for less than ~stride frames between probes can slip through.
+
 ## Notes
 
 - Runs every frame of every scene until a face qualifies, then waits

anime_facesaver.py

@@ -119,6 +119,22 @@ def parse_arguments():
              "edges. 0 (default) allows cropped faces; set >0 to reject "
              "faces touching the crop lines.",
     )
+    parser.add_argument(
+        "--probe-stride",
+        type=int,
+        default=1,
+        help="Within a scene still needing a face, run detection every Nth "
+             "frame instead of every frame (1 = original behavior, free on "
+             "CUDA). For slow devices (iGPU ~5s/img) use e.g. 25: decode "
+             "stays frame-perfect, only the expensive inference is thinned.",
+    )
+    parser.add_argument(
+        "--probe-max-per-scene",
+        type=int,
+        default=4,
+        help="With --probe-stride>1, probes before giving up on a scene and "
+             "waiting for the next scene start (default 4).",
+    )
     parser.add_argument(
         "--scene-threshold",
         type=float,
@@ -272,6 +288,17 @@ def process_video(video_path, output_dir, args, model, device, draw,
     face_in_scene = False
     frame_idx = 0
 
+    # Probe schedule. probe_stride=1 reproduces the original every-frame
+    # loop (right on CUDA, where predict costs ~0.2 s). On slow devices
+    # (iGPU ~5 s/img) predicting every frame is hopeless: decode is ~300 fps,
+    # inference 0.2/s. With stride>1 we decode through frames for free and
+    # predict only at scene starts, retrying every stride frames up to
+    # probe_max attempts per scene, then waiting for the next scene start.
+    probe_stride = getattr(args, "probe_stride", 1)
+    probe_max = getattr(args, "probe_max_per_scene", 4)
+    next_probe = 0
+    probes_used = 0
+
     while True:
         ret, frame = cap.read()
         if not ret:
@@ -279,10 +306,20 @@ def process_video(video_path, output_dir, args, model, device, draw,
 
         if frame_idx in scene_starts:
             face_in_scene = False
-        # membership test on a set; pop matched starts to keep it cheap
+            next_probe = frame_idx
-        # (scenes strictly increase, so set stays small either way)
+            probes_used = 0
+
+        if face_in_scene:
+            frame_idx += 1
+            continue
+        if probe_stride > 1 and (frame_idx < next_probe
+                                 or probes_used >= probe_max):
+            frame_idx += 1
+            continue
 
         if not face_in_scene:
+            probes_used += 1
+            next_probe = frame_idx + probe_stride
             try:
                 results = model.predict(frame, conf=args.conf,
                                         iou=args.iou, imgsz=args.imgsz,

find_dupes.py

@@ -111,42 +111,53 @@ def main():
             print(f"  {i+1}/{len(files)}", flush=True)
 
     names = list(data)
-    # strict (complete-linkage) clustering — NO transitive chaining:
-    # seed with the ungrouped image having most neighbours, take its
-    # within-threshold neighbourhood, then keep only members pairwise
-    # within threshold. Union-find chaining merges unrelated frames
-    # after enough hops (verified failure: 90-frame 'group' of six
-    # different scenes).
     n = len(names)
-    print(f"Pairwise compare {n}x{n} (numpy hamming) ...")
+    print(f"Pairwise compare {n}x{n} (streaming hamming, chunked) ...")
+
+    # Memory-safe at 30k+ scale under ~4 GB cgroup caps (verified: the
+    # full n^2 uint64 xor matrix — 8 GB at n=32k — got a background run
+    # oom-killed in a hermes-worker systemd scope on 2026-10-29). Row
+    # chunks of 1024 keep peak temps at a few hundred MB; only the final
+    # bool close-matrix (n^2 bytes) stays resident. uint8 distances
+    # suffice (max 64) via np.bitwise_count (numpy>=2.3) or a per-byte
+    # popcount LUT.
+    def hamming_chunk(mat, rows):
+        x = mat[rows, None] ^ mat[None, :]
+        if hasattr(np, "bitwise_count"):
+            ham = np.bitwise_count(x)
+        else:
+            bc = np.array([bin(i).count("1") for i in range(256)], dtype=np.uint8)
+            x8 = x.view(np.uint8).reshape(x.shape + (8,))
+            ham = np.zeros(x.shape, dtype=np.uint8)
+            for b in range(8):
+                ham += bc[x8[..., b]]
+            del x8
+        del x
+        return ham
+
     dmat = np.array([data[f][0] for f in names], dtype=np.uint64)
-    # hamming distances via xor + popcount (work in int64; numpy has no
+    amat = np.array([data[f][1] for f in names], dtype=np.uint64)
-    # sane uint64 shift-add promotion)
+    max_a = args.max_adist if args.max_adist is not None else 8
-    x = (dmat[:, None] ^ dmat[None, :]).astype(np.int64)
+    CH = 1024
-    ham = np.zeros(x.shape, dtype=np.int16)
+    close = np.zeros((n, n), dtype=bool)
-    for b in range(64):
+    for s in range(0, n, CH):
-        ham += ((x >> b) & 1).astype(np.int16)
+        rows = slice(s, min(s + CH, n))
+        close[rows] = ((hamming_chunk(dmat, rows) <= args.max_dist)
+                       & (hamming_chunk(amat, rows) <= max_a))
+        if (s // CH) % 8 == 0:
+            print(f"  hashes {min(s + CH, n)}/{n}", flush=True)
+    del dmat, amat
 
     # combined close matrix: dHash AND aHash AND color-histogram gates.
     # dHash alone has a false-positive floor (verified: distinct frames
     # at distance <=10); a second, differently-computed hash plus a
     # color signature is what separates 'same frame replayed' from
-    # 'same show, different shot'.
+    # 'same show, different shot'. Color L1 is computed per row only for
-    close = ham <= args.max_dist
+    # rows that survived both hashes — few rows at real corpus scale,
-
+    # and it replaces the old n^2 float32 matrix (4 GB at n=32k).
-    amat = np.array([data[f][1] for f in names], dtype=np.uint64)
-    xa = (amat[:, None] ^ amat[None, :]).astype(np.int64)
-    aham = np.zeros(xa.shape, dtype=np.int16)
-    for b in range(64):
-        aham += ((xa >> b) & 1).astype(np.int16)
-    close &= aham <= (args.max_adist
-                      if args.max_adist is not None else 8)
-
     hmat = np.stack([data[f][3] for f in names])  # N x 192 hist rows
-    cdist = np.zeros((n, n), dtype=np.float32)
+    for i in np.flatnonzero(close.any(axis=1)):
-    for i in range(n):  # row-wise keeps it ~seconds and RAM-light
+        close[i] &= np.abs(hmat - hmat[i]).sum(axis=1) <= args.max_cdist
-        cdist[i] = np.abs(hmat - hmat[i]).sum(axis=1)
-    close &= cdist <= args.max_cdist
     np.fill_diagonal(close, False)
 
     # strict (complete-linkage) clustering — NO transitive chaining: