Why this is published first
Nothing here has been run. That is the point.
A benchmark released together with its own results is unfalsifiable after the fact: nobody outside can tell which parameters were chosen because they were principled and which were chosen because they flattered the outcome. Writing the protocol down first — in a git repository, with a timestamp and a public diff — gives up that freedom deliberately. Every later change to this document is visible, and every run report has to cite the protocol version it ran under.
The model is ClickHouse’s ClickBench: public, reproducible, includes the competition, and lets outsiders submit their own results. That is the standard this document is trying to meet, and it is a long way from met yet.
Contest this. If a parameter below is wrong, unfair, or rigged in our favour, the useful time to say so is now, while there is no result to defend. Mail [email protected] and the objection gets answered in the changelog.
Scope
In scope for run 001:
- Single-node, single-GPU indexing and serving of image-embedding vectors derived from video frames.
- Recall@10 against exact search, query throughput, index build time, resident index size, and GPU-hours per million queries.
- Corpus sizes from 1e7 to 1e9 frames, reported at each step rather than at the most flattering one.
- The effect of a near-duplicate culling pass on all of the above.
Explicitly out of scope, and therefore not claimed anywhere:
- Distributed or multi-node serving. It changes the cost model and deserves its own protocol.
- Embedding inference cost. Identical across every system under test, so including it would only dilute the comparison.
- Text-to-frame retrieval quality. This measures nearest-neighbour retrieval over a fixed embedding space; it says nothing about whether that embedding space is good.
- Filtered, hybrid, or metadata-constrained search.
- Insert and delete throughput. Real pipelines need it; run 001 does not measure it.
Systems under test
| System | Role | Configuration |
|---|---|---|
| FAISS-GPU · IVF-PQ | baseline | GpuIndexIVFPQ, nlist swept over {2^14, 2^16, 2^18}, PQ m ∈ {32, 64}, 8 bits per sub-quantiser, nprobe swept to trace the recall/throughput frontier. |
| FAISS-GPU · IVF-Flat | baseline | GpuIndexIVFFlat, same nlist sweep, no product quantisation. |
| FAISS-GPU · Flat | reference | GpuIndexFlatIP, exact search, sharded across devices where required. |
| HNSW (CPU) | reference | M = 32, efConstruction = 200, efSearch swept. |
| corpusframe | subject | Culling pass at a stated cosine threshold, then the index over surviving frames. Parameters published with run 001; the index internals are described conceptually on /how-it-works. |
Every FAISS configuration is swept, not tuned once. nprobe (or efSearch for
HNSW) is varied across its useful range to trace the whole recall/throughput
frontier, and the frontier is what gets published — including the regions where
corpusframe is behind. A frontier that wins everywhere is a sign of a broken
harness, not a good index.
Embeddings
| Parameter | Setting |
|---|---|
| Primary model | CLIP ViT-L/14 @ 224 px, 768-d image embeddings |
| Secondary sweep | OpenCLIP ViT-B/32, 512-dConfirms the result is not an artefact of one embedding geometry. |
| Normalisation | L2-normalised; similarity is inner product = cosine |
| Precision | fp16 on device, fp32 for ground truth |
| Inference cost | Excluded from all reported figuresIdentical across systems under test, so including it would only dilute the comparison. |
Hardware
Proposed. Every pin at run time row below must be replaced with an exact value
before run 001 publishes; a protocol that says “recent CUDA” is not a protocol.
| Parameter | Value |
|---|---|
| Node | Single node, no distributed servingMulti-node changes the economics and is out of scope for run 001. |
| GPU | 1× NVIDIA A100 80 GB SXM4PROPOSED — replace with the actual device. |
| GPU clocks | pin at run timeLocked with nvidia-smi -lgc; report the locked value. Unlocked clocks make throughput unreproducible. |
| CPU | pin at run timeModel and core count; pin at run time. |
| Host RAM | pin at run time |
| Storage | Local NVMeEmbeddings read from local disk, never from object storage, so build time is not measuring the network. |
| Driver | pin at run timeExact driver version. |
| CUDA | pin at run timeExact toolkit version. |
| FAISS | pin at run timeExact release or commit SHA, and how it was built — a conda FAISS and a source FAISS do not benchmark the same. |
| Power / thermal | pin at run timePower cap and sustained temperature. A throttling A100 has produced more bogus benchmarks than any bad algorithm. |
Clock locking and the power cap are listed because they are the two most common sources of benchmark numbers that nobody can reproduce. A thermally throttled A100 has ruined more comparisons than any bad algorithm.
Query workload
| Parameter | Setting |
|---|---|
| Corpus sizes | 1e7 → 1e9 frames, half-decade stepsReported per step. A single scalar at one corpus size is not a result. |
| Query set | 100k frames held out before indexingHeld out before culling too, so culling cannot quietly delete the hard queries. |
| k | 10, with k = 100 reported alongside |
| Ground truth | Exact flat search, fp32, over the pre-cull corpus |
| Warm-up | 10k queries discarded before timing |
| Repetitions | 5 runs; report median and full min/max spreadNot mean ± σ. Spread is what tells you whether to believe the median. |
| Concurrency | Single-tenant. Nothing else on the device. |
Two details worth arguing about, stated plainly:
- Queries are held out before culling, not after. If they were drawn after the culling pass, culling could quietly delete exactly the frames that are hard to retrieve and improve its own recall for free.
- Ground truth is computed against the pre-cull corpus. This is the strict reading and it is deliberately unkind to us: a culled frame that was a true neighbour counts as a miss.
Metrics
| Metric | FAISS-GPU | corpusframe | Pre-reg. target |
|---|---|---|---|
| recall@10 Fraction of the true 10 nearest neighbours, by exact cosine similarity against a brute-force flat index over the same embeddings, that appear in the returned top 10. Averaged over the held-out query set. | — | — | not stated |
| query throughput (q/s)Queries per second at steady state, single node, warm index, measured at the batch size that maximises throughput for each system independently. Reported at fixed recall@10, never at a system’s own best recall. | — | — | not stated |
| index build time (s)Wall-clock from embeddings on local NVMe to a queryable index, including training, clustering, and any culling pass. Excludes embedding inference, which is identical for every system under test. | — | — | not stated |
| GPU-hours / 1e6 queries (GPU-h)Build cost amortised over a stated query volume, plus query cost. The only metric that maps onto a budget line, and the only one a buyer checks twice. | — | — | not stated |
| resident index size (GiB)Peak device memory held by the index while serving, excluding the raw embedding matrix when the index does not need it resident. | — | — | not stated |
| corpus reduction (%)Fraction of frames removed by the culling pass at the stated similarity threshold. Reported per dataset — it is a property of the footage, not of the system, and it will vary wildly between WebVid and HowTo100M. | — | — | not stated |
Reported as median of five runs with the full min/max spread, never as mean ± σ. The spread is what tells a reader whether to believe the median.
Procedure
Decode and embed
Sample frames at 1 fps from each dataset, embed with the primary model, write fp16 embeddings plus a stable frame id to local NVMe. Publish the decode and embed scripts, and the SHA-256 of the resulting shards.
CaveatFrame counts depend on the decode step, not on the published clip counts. The actual counts get reported; no estimate appears anywhere before then.
Hold out queries
Draw 100k query frames uniformly at random from the pre-cull corpus, with a fixed seed, and remove them from every index. Publish the seed.
Build ground truth
Exact fp32 flat search for the true top-100 of every query frame against the pre-cull corpus. Expensive, done once per corpus size, cached and published as an artefact so third parties can score their own systems against it.
CaveatGround truth against the *pre-cull* corpus is the strict reading: culling cannot be credited for making its own recall easier.
Sweep the baselines
Build each FAISS configuration, sweep nprobe (or efSearch), and record (recall@10, throughput, build time, memory) at every point. Keep the whole frontier, not the best point.
Sweep corpusframe
Same corpus, same embeddings, same query set, same hardware, same day. Sweep the culling threshold and the index parameters independently so their contributions are separable.
Publish everything
Raw result CSVs, the scripts, the exact environment, and the frontier plots — including the regions where corpusframe loses. Run reports are versioned in the site repository next to this protocol, so the diff is public.
CaveatIf a run contradicts the claim, the run gets published and the claim on this page changes. That commitment is the only thing making the pre-registration worth anything.
Kill conditions
Any one of these, if it happens, gets published as prominently as a positive result would have been.
- FAISS-GPU wins on total GPU-hours at equal recall@10 anywhere in the corpus-size sweep.
- Culling at the threshold that produces the headline reduction drops recall@10 by more than 0.01 absolute.
- The advantage exists at 1e9 frames but not at 1e7 — in which case the claim is about scale, and the page will say so instead of generalising.
- Results do not reproduce within 5% on a second, differently-configured node.
- The recall/throughput frontier crosses, and FAISS-GPU is ahead in the region most pipelines actually operate in.
Reproduction
Not yet possible — there is no public repository. When run 001 publishes, this section will carry the exact commands, the environment lockfile, and the SHA-256 of every published artefact, and the harness will be released under the same repository as this document so the diff between protocol and implementation is inspectable.
The intended shape, so it can be criticised early:
# 1. decode + embed (writes fp16 shards + manifest with per-shard SHA-256)
corpusframe-bench embed --dataset webvid --fps 1 --model clip-vit-l14 --out ./shards
# 2. hold out the query set with a published seed
corpusframe-bench holdout --shards ./shards --n 100000 --seed 20260817
# 3. exact ground truth, k=100, fp32 (cached + published as an artefact)
corpusframe-bench truth --shards ./shards --queries ./queries --k 100
# 4. sweep every system over the same corpus, same day, same box
corpusframe-bench sweep --systems faiss-ivfpq,faiss-ivfflat,hnsw,corpusframe \
--sizes 1e7,3e7,1e8,3e8,1e9 --out ./results
# 5. emit the CSVs and the frontier plots the site renders from
corpusframe-bench report --results ./results --out ./reports/run-001Third parties should be able to run steps 3–5 against their own system using the published ground-truth artefact, and submit results. That is the part that makes a benchmark trustworthy, and it is the part that is easiest to quietly not build.
Changelog
| Version | Date | Change |
|---|---|---|
| v0.1 | 2026-08-17 | First publication. No results exist. Targets not yet stated. |