No part of this has been measured yet. There is no benchmark, no result, and no customer. What exists is the protocol that will decide it — written down first, in public, with the failure conditions attached — so that the numbers, when they arrive, cannot be the numbers that happened to look good.
Read this before anything else on the page. It is the part most sites leave out, and it is the only part that is fully verifiable today.
It is really two claims. They fail independently, so they are measured independently — a result that supports one and kills the other is a likely outcome, not an embarrassment.
Frame corpora sampled from video are dominated by near-duplicates — static shots, slow pans, repeated intros. Culling them at a stated similarity threshold shrinks the corpus without measurably moving retrieval quality on the surviving frames.
Falsified ifCulling at any threshold that shrinks the corpus meaningfully also drops recall@10 outside the stated tolerance.
On the culled corpus, reaching a fixed recall@10 takes fewer GPU-hours — across build and query together — than an IVF-PQ index built by FAISS-GPU on the same hardware and the same embeddings.
Falsified ifFAISS-GPU matches or beats total GPU-hours at equal recall@10, on the culled corpus, at any corpus size in the sweep.
Definitions first, because every benchmark argument in this field is really an argument about definitions. Both measured columns are empty and will stay empty until a run fills them.
| Metric | FAISS-GPU | corpusframe | Pre-reg. target |
|---|---|---|---|
| recall@10 Fraction of the true 10 nearest neighbours, by exact cosine similarity against a brute-force flat index over the same embeddings, that appear in the returned top 10. Averaged over the held-out query set. | — | — | not stated |
| query throughput (q/s)Queries per second at steady state, single node, warm index, measured at the batch size that maximises throughput for each system independently. Reported at fixed recall@10, never at a system’s own best recall. | — | — | not stated |
| index build time (s)Wall-clock from embeddings on local NVMe to a queryable index, including training, clustering, and any culling pass. Excludes embedding inference, which is identical for every system under test. | — | — | not stated |
| GPU-hours / 1e6 queries (GPU-h)Build cost amortised over a stated query volume, plus query cost. The only metric that maps onto a budget line, and the only one a buyer checks twice. | — | — | not stated |
| resident index size (GiB)Peak device memory held by the index while serving, excluding the raw embedding matrix when the index does not need it resident. | — | — | not stated |
| corpus reduction (%)Fraction of frames removed by the culling pass at the stated similarity threshold. Reported per dataset — it is a property of the footage, not of the system, and it will vary wildly between WebVid and HowTo100M. | — | — | not stated |
Open defect6 of 6 metrics have no pre-registered target. Targets written down after the numbers arrive prove nothing, so these have to be committed — publicly, with a timestamp — before run 001 executes. Until then this protocol is incomplete, and saying so here is cheaper than being caught later.
FAISS-GPU is the baseline because it is what the pipelines we are asking people to change are already running. Beating something else would not answer the question.
GpuIndexIVFPQ, nlist swept over {2^14, 2^16, 2^18}, PQ m ∈ {32, 64}, 8 bits per sub-quantiser, nprobe swept to trace the recall/throughput frontier.
The baseline that matters. This is what the pipelines we are asking people to change are already running.
GpuIndexIVFFlat, same nlist sweep, no product quantisation.
Higher recall ceiling, far worse memory footprint. Included so the memory axis is honest.
GpuIndexFlatIP, exact search, sharded across devices where required.
Not a competitor — the ground truth generator. Every recall@10 figure on this site is measured against it.
M = 32, efConstruction = 200, efSearch swept.
CPU-only reference point, because a fair number of teams are not GPU-bound at all and should know it.
Culling pass at a stated cosine threshold, then the index over surviving frames. Parameters published with run 001; the index internals are described conceptually on /how-it-works.
Named in advance so the numbers cannot be quietly reported from a different box. Anything not yet pinned is marked as unpinned rather than described vaguely.
| Parameter | Value |
|---|---|
| Node | Single node, no distributed servingMulti-node changes the economics and is out of scope for run 001. |
| GPU | 1× NVIDIA A100 80 GB SXM4PROPOSED — replace with the actual device. |
| GPU clocks | pin at run timeLocked with nvidia-smi -lgc; report the locked value. Unlocked clocks make throughput unreproducible. |
| CPU | pin at run timeModel and core count; pin at run time. |
| Host RAM | pin at run time |
| Storage | Local NVMeEmbeddings read from local disk, never from object storage, so build time is not measuring the network. |
| Driver | pin at run timeExact driver version. |
| CUDA | pin at run timeExact toolkit version. |
| FAISS | pin at run timeExact release or commit SHA, and how it was built — a conda FAISS and a source FAISS do not benchmark the same. |
| Power / thermal | pin at run timePower cap and sustained temperature. A throttling A100 has produced more bogus benchmarks than any bad algorithm. |
Short stock-footage clips with captions. Heavily redundant within clips — the easy case for culling, and worth naming as such.
Instructional video with long static shots and talking heads. Different redundancy structure to WebVid, which is exactly why both are in the sweep.
Embeddings, query workload, repetition counts, and the full procedure are in the protocol document — protocol v0.1.
Stated now, while it is still free to state. Each of these gets checked and reported in the run write-up whether or not it trips.
One email when run 001 publishes — including if it publishes a result that contradicts this page.