You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/benchmarks.md
+84-84Lines changed: 84 additions & 84 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,23 +3,23 @@
3
3
## Table of contents
4
4
*[Overview](#overview)
5
5
*[Run benchmarks](#run-benchmarks)
6
-
*[Available sets](#available-sets)
7
-
-[BM_VecSimBasics](#bm_vecsimbasics)
8
-
-[BM_BatchIterator](#bm_batchiterator)
9
-
-[BM_VecSimUpdatedIndex](#bm_vecsimupdatedindex)
6
+
*[Available sets](#available-sets)
7
+
-[BM_VecSimBasics](#bm_vecsimbasics)
8
+
-[BM_BatchIterator](#bm_batchiterator)
9
+
-[BM_VecSimUpdatedIndex](#bm_vecsimupdatedindex)
10
10
*[ann-benchmarks](#ann-benchmark)
11
11
12
12
# Overview
13
-
We use the [Google benchmark](https://github.com/google/benchmark) tool to run micro-benchmarks for the vector indexes functionality.
14
-
Google benchmark is a popular tool for benchmark code snippets, similar to unit tests. It allows a quick way to estimate the runtime of each use case based on several (identical) runs, and prints the results as output. For some tests, the output includes an additional "Recall" metric, which indicates the accuracy in the case of approximate search.
15
-
**The recall** is calculated as the size of the intersection set between the number of the ground truth results (calculated by the flat algorithm) and the returned results from the approximate algorithm (HNSW in this case), divided by the number of ground truth results:
13
+
We use the [Google benchmark](https://github.com/google/benchmark) tool to run micro-benchmarks for the vector indexes functionality.
14
+
Google benchmark is a popular tool for benchmark code snippets, similar to unit tests. It allows a quick way to estimate the runtime of each use case based on several (identical) runs, and prints the results as output. For some tests, the output includes an additional "Recall" metric, which indicates the accuracy in the case of approximate search.
15
+
**The recall** is calculated as the size of the intersection set between the number of the ground truth results (calculated by the flat algorithm) and the returned results from the approximate algorithm (HNSW in this case), divided by the number of ground truth results:
@@ -29,7 +29,7 @@ To run all test sets, call the following commands from the project root dir:
29
29
make benchmark
30
30
```
31
31
### Running a Subset of Benchmarks
32
-
To run only a subset of benchmarks that match a specified `<regex>`, set `BENCHMARK_FILTER=<regex>` environment variable. For example:
32
+
To run only a subset of benchmarks that match a specified `<regex>`, set `BENCHMARK_FILTER=<regex>` environment variable. For example:
33
33
```sh
34
34
make benchmark BENCHMARK_FILTER=fp32*
35
35
```
@@ -38,108 +38,108 @@ make benchmark BENCHMARK_FILTER=fp32*
38
38
There are currently 3 sets of benchmarks available: `BM_VecSimBasics`, `BM_BatchIterator`, and `BM_VecSimUpdatedIndex`. Each is templated according to the index data type. We run 10 iterations of each test case, unless otherwise specified.
39
39
## BM_VecSimBasics
40
40
For each combination of data type (fp32/fp64), index type (single/multi), and indexing algorithm (flat, HNSW and tiered-HNSW) the following test cases are included:
41
-
1. Measure index total `memory` (runtime and iterations number are irrelevant for this use case, just the memory metric)
42
-
2.`AddLabel` - runs for `DEFAULT_BLOCK_SIZE (= 1024)` iterations, in each we add one new label to the index from the `queries` list. Note that for a single value index each label contains one vector, meaning that the number of new labels equals the number of new vectors.
43
-
**results:** average time per label, average memory addition per vector, vectors per label.
41
+
1. Measure index total `memory` (runtime and iterations number are irrelevant for this use case, just the memory metric)
42
+
2.`AddLabel` - runs for `DEFAULT_BLOCK_SIZE (= 1024)` iterations, in each we add one new label to the index from the `queries` list. Note that for a single value index each label contains one vector, meaning that the number of new labels equals the number of new vectors.
43
+
**results:** average time per label, average memory addition per vector, vectors per label.
44
44
*At the end of the benchmark, we delete the added labels*
45
-
3.`DeleteLabel` - runs for `DEFAULT_BLOCK_SIZE (= 1024)` iterations, in each we delete one label from the index. Note that for a single value index each label contains one vector, meaning that the number of deleted labels equals the number of deleted vectors.
46
-
**results:** average time per label, average memory addition per vector (a negative value means that the memory has decreased).
45
+
3.`DeleteLabel` - runs for `DEFAULT_BLOCK_SIZE (= 1024)` iterations, in each we delete one label from the index. Note that for a single value index each label contains one vector, meaning that the number of deleted labels equals the number of deleted vectors.
46
+
**results:** average time per label, average memory addition per vector (a negative value means that the memory has decreased).
47
47
*At the end of the benchmark, we restore the deleted vectors under the same labels*
48
48
49
49
For tiered-HNSW index, we also run these additional tests:
50
50
51
51
4.`AddLabel_Async` - which is the same as `AddLabel` test, but here we also take into account the time that it takes to the background threads to ingest vectors into HNSW asynchronously (while in `AddLabel` test for the tiered index we only measure the time until vectors are stored in the flat buffer).
52
52
5.`DeleteLabel_Async` - which is the same as `DeleteLabel` test, but here we also take into account the time that it takes to the background threads to repair the HNSW graph due to the deletion (while in `DeleteLabel` test for the tiered index we only measure the time until vectors are marked as deleted). Note that the garbage collector of the tiered index is triggered when at least `swapJobsThreshold` vectors are ready to be evicted (this happens when all of their affected neighbors in the graph are repaired). We run this test for `swapJobsThreshold` in `{1, 100, 1024(DEFAULT)}`, and we collect two additional metrics: `num_zombies`, which is the number of vectors that are left to be evicted *after* the test has finished, and `cleanup_time`, which is the number of milliseconds that it took to clean these zombies.
53
53
54
-
In both tests, we should only consider the `real_time` metric (rather than the `cpu_time`), since `cpu_time` only accounts for the time that the main thread is running.
54
+
In both tests, we should only consider the `real_time` metric (rather than the `cpu_time`), since `cpu_time` only accounts for the time that the main thread is running.
55
55
#### **TopK benchmarks**
56
-
Search for the `k` nearest neighbors of the query.
57
-
6. Run `Top_K` query over the flat index (using brute-force search), for each `k=10`, `k=100` and `k=500`
58
-
**results:** average time per iteration
59
-
7. Run `Top_K` query over the HNSW index, for each pair of `{ef_runtime, k}` from the following:
60
-
`{ef_runtime=10, k=10}`
61
-
`{ef_runtime=200, k=10}`
62
-
`{ef_runtime=100, k=100}`
63
-
`{ef_runtime=200, k=100}`
64
-
`{ef_runtime=500, k=500}`
56
+
Search for the `k` nearest neighbors of the query.
57
+
6. Run `Top_K` query over the flat index (using brute-force search), for each `k=10`, `k=100` and `k=500`
58
+
**results:** average time per iteration
59
+
7. Run `Top_K` query over the HNSW index, for each pair of `{ef_runtime, k}` from the following:
60
+
`{ef_runtime=10, k=10}`
61
+
`{ef_runtime=200, k=10}`
62
+
`{ef_runtime=100, k=100}`
63
+
`{ef_runtime=200, k=100}`
64
+
`{ef_runtime=500, k=500}`
65
65
**results:** average time per iteration, recall
66
66
8. Run `Top_K` query over the tiered-HNSW index (in parallel by several threads), for each pair of `{ef_runtime, k}` as in the previous test (assuming all vector are indexed into the HNSW graph). We run for `50` iterations to get a better sense of the parallelism that can be achieved. Here as well, we should only consider the `real_time` measurement rather than the `cpu_time`.
67
67
#### **Range query benchmarks**
68
-
In range query, we search for all the vectors in the index whose distance to the query vector is lower than the range.
69
-
9. Run `Range` query over the flat index (using brute-force search), for each `radius=0.2`, `radius=0.35` and `radius=0.5`
70
-
**results:** average time per iteration, average results number per iteration
71
-
10. Run `Range` query over the HNSW index, for each pair of `{radius, epsilon}` from the following:
72
-
`{radius=0.2, epsilon=0.001}`
73
-
`{radius=0.2, epsilon=0.01}`
74
-
`{radius=0.2, epsilon=0.1}`
75
-
`{radius=0.35, epsilon=0.001}`
76
-
`{radius=0.35, epsilon=0.01}`
77
-
`{radius=0.35, epsilon=0.1}`
78
-
`{radius=0.5, epsilon=0.001}`
79
-
`{radius=0.5, epsilon=0.01}`
80
-
`{radius=0.5, epsilon=0.1}`
81
-
**results:** average time per iteration, average results number per iteration, recall
68
+
In range query, we search for all the vectors in the index whose distance to the query vector is lower than the range.
69
+
9. Run `Range` query over the flat index (using brute-force search), for each `radius=0.2`, `radius=0.35` and `radius=0.5`
70
+
**results:** average time per iteration, average results number per iteration
71
+
10. Run `Range` query over the HNSW index, for each pair of `{radius, epsilon}` from the following:
72
+
`{radius=0.2, epsilon=0.001}`
73
+
`{radius=0.2, epsilon=0.01}`
74
+
`{radius=0.2, epsilon=0.1}`
75
+
`{radius=0.35, epsilon=0.001}`
76
+
`{radius=0.35, epsilon=0.01}`
77
+
`{radius=0.35, epsilon=0.1}`
78
+
`{radius=0.5, epsilon=0.001}`
79
+
`{radius=0.5, epsilon=0.01}`
80
+
`{radius=0.5, epsilon=0.1}`
81
+
**results:** average time per iteration, average results number per iteration, recall
82
82
83
83
## BM_BatchIterator
84
-
The purpose of these tests is to benchmark batch iterator functionality. The batch iterator is a handle that enables running `Top_K` query in batches, by asking for the next best `n` results repeatedly, until there are no more results to return. We use for this test cases the same indices that were built for the "basic benchmark" for this test case as well.
84
+
The purpose of these tests is to benchmark batch iterator functionality. The batch iterator is a handle that enables running `Top_K` query in batches, by asking for the next best `n` results repeatedly, until there are no more results to return. We use for this test cases the same indices that were built for the "basic benchmark" for this test case as well.
85
85
The test cases are:
86
-
1. Fixed batch size - Run `Top_K` query for each pair of `{batch size, number of batches}` from the following:
87
-
`{batch size=10, number of batches=1}`
88
-
`{batch size=10, number of batches=3}`
89
-
`{batch size=10, number of batches=5}`
90
-
`{batch size=100, number of batches=1}`
91
-
`{batch size=100, number of batches=3}`
92
-
`{batch size=100, number of batches=5}`
93
-
`{batch size=1000, number of batches=1}`
94
-
`{batch size=1000, number of batches=3}`
95
-
`{batch size=1000, number of batches=5}`
86
+
1. Fixed batch size - Run `Top_K` query for each pair of `{batch size, number of batches}` from the following:
87
+
`{batch size=10, number of batches=1}`
88
+
`{batch size=10, number of batches=3}`
89
+
`{batch size=10, number of batches=5}`
90
+
`{batch size=100, number of batches=1}`
91
+
`{batch size=100, number of batches=3}`
92
+
`{batch size=100, number of batches=5}`
93
+
`{batch size=1000, number of batches=1}`
94
+
`{batch size=1000, number of batches=3}`
95
+
`{batch size=1000, number of batches=5}`
96
96
**Flat index results:** Time per iteration, memory delta per iteration
97
97
**HNSW index results:** Time per iteration, Recall, memory delta per iteration
98
-
2. Variable batch size - Run `Top_K` query where in each iteration the batch size is increased by a factor of 2, for each pair of `{batch initial size, number of batches}` from the following:
99
-
`{batch initial size=10, number of batches=2}`
100
-
`{batch initial size=10, number of batches=4}`
101
-
`{batch initial size=100, number of batches=2}`
102
-
`{batch initial size=100, number of batches=4}`
103
-
`{batch initial size=1000, number of batches=2}`
104
-
`{batch initial size=1000, number of batches=4}`
98
+
2. Variable batch size - Run `Top_K` query where in each iteration the batch size is increased by a factor of 2, for each pair of `{batch initial size, number of batches}` from the following:
99
+
`{batch initial size=10, number of batches=2}`
100
+
`{batch initial size=10, number of batches=4}`
101
+
`{batch initial size=100, number of batches=2}`
102
+
`{batch initial size=100, number of batches=4}`
103
+
`{batch initial size=1000, number of batches=2}`
104
+
`{batch initial size=1000, number of batches=4}`
105
105
**Flat index results:** Time per iteration
106
106
**HNSW index results:** Time per iteration, Recall, memory delta per iteration
107
-
3. Batches to Adhoc BF - In each iteration we run `Top_K` with an increasing `batch size` (initial size=10, increase factor=2) for `number of batches` and then switch to ad-hoc BF. We define `step` as the ratio between the index size to the number of vectors to go over in ad-hoc BF. The tests run for each pair of `{step, number of batches}` from the following:
108
-
`{step=5, number of batches=0}`
109
-
`{step=5, number of batches=2}`
110
-
`{step=5, number of batches=5}`
111
-
`{step=10, number of batches=0}`
112
-
`{step=10, number of batches=2}`
113
-
`{step=10, number of batches=5}`
114
-
`{step=20, number of batches=0}`
115
-
`{step=20, number of batches=2}`
116
-
`{step=20, number of batches=5}`
107
+
3. Batches to Ad-hoc BF - In each iteration we run `Top_K` with an increasing `batch size` (initial size=10, increase factor=2) for `number of batches` and then switch to ad-hoc BF. We define `step` as the ratio between the index size to the number of vectors to go over in ad-hoc BF. The tests run for each pair of `{step, number of batches}` from the following:
108
+
`{step=5, number of batches=0}`
109
+
`{step=5, number of batches=2}`
110
+
`{step=5, number of batches=5}`
111
+
`{step=10, number of batches=0}`
112
+
`{step=10, number of batches=2}`
113
+
`{step=10, number of batches=5}`
114
+
`{step=20, number of batches=0}`
115
+
`{step=20, number of batches=2}`
116
+
`{step=20, number of batches=5}`
117
117
**Flat index results:** Time per iteration
118
118
**HNSW index results:** Time per iteration, memory delta per iteration
119
119
120
120
## BM_VecSimUpdatedIndex
121
-
For this use case, we create two indices for each algorithm (flat and HNSW). The first index contains 500K vectors added to an empty index. The other index also contains 500K vectors, but this time they were added by overriding the 500K vectors of the aforementioned indices. Currently, we only test this for FP32 single-value index.
121
+
For this use case, we create two indices for each algorithm (flat and HNSW). The first index contains 500K vectors added to an empty index. The other index also contains 500K vectors, but this time they were added by overriding the 500K vectors of the aforementioned indices. Currently, we only test this for FP32 single-value index.
122
122
The test cases are:
123
123
1. Index `total memory`**before** updating
124
124
2. Index `total memory`**after** updating
125
-
3. Run `Top_K` query over the flat index **before** updating (using brute-force search), for each `k=10`, `k=100` and `k=500`
125
+
3. Run `Top_K` query over the flat index **before** updating (using brute-force search), for each `k=10`, `k=100` and `k=500`
126
126
**results:** average time per iteration
127
-
4. Run `Top_K` query over the flat index **after** updating (using brute-force search), for each `k=10`, `k=100` and `k=500`
127
+
4. Run `Top_K` query over the flat index **after** updating (using brute-force search), for each `k=10`, `k=100` and `k=500`
128
128
**results:** average time per iteration
129
-
5. Run **100** iterations of `Top_K` query over the HNSW index **before** updating, for each pair of `{ef_runtime, k}` from the following:
130
-
`{ef_runtime=10, k=10}`
131
-
`{ef_runtime=200, k=10}`
132
-
`{ef_runtime=100, k=100}`
133
-
`{ef_runtime=200, k=100}`
134
-
`{ef_runtime=500, k=500}`
129
+
5. Run **100** iterations of `Top_K` query over the HNSW index **before** updating, for each pair of `{ef_runtime, k}` from the following:
130
+
`{ef_runtime=10, k=10}`
131
+
`{ef_runtime=200, k=10}`
132
+
`{ef_runtime=100, k=100}`
133
+
`{ef_runtime=200, k=100}`
134
+
`{ef_runtime=500, k=500}`
135
+
**results:** average time per iteration, recall
136
+
6. Run **100** iterations of `Top_K` query over the HNSW index **after** updating, for each pair of `{ef_runtime, k}` from the following:
137
+
`{ef_runtime=10, k=10}`
138
+
`{ef_runtime=200, k=10}`
139
+
`{ef_runtime=100, k=100}`
140
+
`{ef_runtime=200, k=100}`
141
+
`{ef_runtime=500, k=500}`
135
142
**results:** average time per iteration, recall
136
-
6. Run **100** iterations of `Top_K` query over the HNSW index **after** updating, for each pair of `{ef_runtime, k}` from the following:
137
-
`{ef_runtime=10, k=10}`
138
-
`{ef_runtime=200, k=10}`
139
-
`{ef_runtime=100, k=100}`
140
-
`{ef_runtime=200, k=100}`
141
-
`{ef_runtime=500, k=500}`
142
-
**results:** average time per iteration, recall
143
143
144
144
# ANN-Benchmark
145
145
@@ -153,7 +153,7 @@ The `bm_dataset.py` script uses some of ANN-Benchmark datasets to measure this l
153
153
5. mnist-784-euclidean
154
154
6. sift-128-euclidean
155
155
156
-
For each dataset, the script will build an HNSW index with pre-defined build parameters and persist it to a local file in `./data` directory that will be generated (index file name for example: `glove-25-angular-M=16-ef=100.hnsw`). Note that if the file already exists in this path, the entire index will be loaded instead of rebuilding it.
156
+
For each dataset, the script will build an HNSW index with pre-defined build parameters and persist it to a local file in `./data` directory that will be generated (index file name for example: `glove-25-angular-M=16-ef=100.hnsw`). Note that if the file already exists in this path, the entire index will be loaded instead of rebuilding it.
157
157
To download the serialized indices run from the project's root directory:
@@ -166,4 +166,4 @@ Then, for 3 different pre-defined `ef_runtime` values, 1000 `Top_K` queries will
166
166
To reproduce this benchmark, first install the project's python bindings, and then invoke the script. From the project's root directory, you should run:
0 commit comments