Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 45 additions & 45 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ To add more relevance and practicality, we provide cost-effectiveness reports pa

Closely mimicking real-world production environments, we've set up diverse testing scenarios including insertion, searching, and filtered searching. To provide you with credible and reliable data, we've included public datasets from actual production scenarios, such as [SIFT](http://corpus-texmex.irisa.fr/), [GIST](http://corpus-texmex.irisa.fr/), [Cohere](https://huggingface.co/datasets/Cohere/wikipedia-22-12/tree/main/en), and a dataset generated by OpenAI from an opensource [raw dataset](https://huggingface.co/datasets/allenai/c4). It's fascinating to discover how a relatively unknown open-source database might excel in certain circumstances!

Prepare to delve into the world of VDBBench, and let it guide you in uncovering your perfect vector database match.
Prepare to delve into the world of VDBBench, and let it guide you in uncovering your perfect vector database match.

VDBBench is sponsored by Zilliz,the leading opensource vectorDB company behind Milvus. Choose smarter with VDBBench - start your free test on [zilliz cloud](https://zilliz.com/) today!

Expand All @@ -30,35 +30,35 @@ pip install vectordb-bench
**Install all database clients**

``` shell
pip install vectordb-bench[all]
pip install 'vectordb-bench[all]'
```
**Install the specific database client**

```shell
pip install vectordb-bench[pinecone]
pip install 'vectordb-bench[pinecone]'
```
All the database client supported

| Optional database client | install command |
|--------------------------|---------------------------------------------|
| pymilvus, zilliz_cloud (*default*) | `pip install vectordb-bench` |
| all (*clients requirements might be conflict with each other*) | `pip install vectordb-bench[all]` |
| qdrant | `pip install vectordb-bench[qdrant]` |
| pinecone | `pip install vectordb-bench[pinecone]` |
| weaviate | `pip install vectordb-bench[weaviate]` |
| elastic, aliyun_elasticsearch| `pip install vectordb-bench[elastic]` |
| pgvector, pgvectorscale, pgdiskann, alloydb | `pip install vectordb-bench[pgvector]` |
| pgvecto.rs | `pip install vectordb-bench[pgvecto_rs]` |
| redis | `pip install vectordb-bench[redis]` |
| memorydb | `pip install vectordb-bench[memorydb]` |
| chromadb | `pip install vectordb-bench[chromadb]` |
| awsopensearch | `pip install vectordb-bench[opensearch]` |
| aliyun_opensearch | `pip install vectordb-bench[aliyun_opensearch]` |
| mongodb | `pip install vectordb-bench[mongodb]` |
| tidb | `pip install vectordb-bench[tidb]` |
| vespa | `pip install vectordb-bench[vespa]` |
| oceanbase | `pip install vectordb-bench[oceanbase]` |
| hologres | `pip install vectordb-bench[hologres]` |
| all (*clients requirements might be conflict with each other*) | `pip install 'vectordb-bench[all]'` |
| qdrant | `pip install 'vectordb-bench[qdrant]'` |
| pinecone | `pip install 'vectordb-bench[pinecone]'` |
| weaviate | `pip install 'vectordb-bench[weaviate]'` |
| elastic, aliyun_elasticsearch| `pip install 'vectordb-bench[elastic]'` |
| pgvector, pgvectorscale, pgdiskann, alloydb | `pip install 'vectordb-bench[pgvector]'` |
| pgvecto.rs | `pip install 'vectordb-bench[pgvecto_rs]'` |
| redis | `pip install 'vectordb-bench[redis]'` |
| memorydb | `pip install 'vectordb-bench[memorydb]'` |
| chromadb | `pip install 'vectordb-bench[chromadb]'` |
| awsopensearch | `pip install 'vectordb-bench[opensearch]'` |
| aliyun_opensearch | `pip install 'vectordb-bench[aliyun_opensearch]'` |
| mongodb | `pip install 'vectordb-bench[mongodb]'` |
| tidb | `pip install 'vectordb-bench[tidb]'` |
| vespa | `pip install 'vectordb-bench[vespa]'` |
| oceanbase | `pip install 'vectordb-bench[oceanbase]'` |
| hologres | `pip install 'vectordb-bench[hologres]'` |

### Run

Expand Down Expand Up @@ -190,7 +190,7 @@ Options:
--number-of-shards INTEGER Number of primary shards for the index
--number-of-replicas INTEGER Number of replica copies for each primary
shard
# Indexing Performance
# Indexing Performance
--index-thread-qty INTEGER Thread count for native engine indexing
--index-thread-qty-during-force-merge INTEGER
Thread count during force merge operations
Expand All @@ -206,7 +206,7 @@ Options:
--engine TEXT type of engine to use valid values [faiss, lucene, s3vector]
# Memory Management
--cb-threshold TEXT k-NN Memory circuit breaker threshold

# Quantization Type
--quantization-type TEXT which type of quantization to use valid values [fp32, fp16]
--help Show this message and exit.
Expand Down Expand Up @@ -282,7 +282,7 @@ Options:

It is recommended to use the following code for installation.
```shell
pip install vectordb-bench[hologres] "psycopg[binary]" pgvector
pip install 'vectordb-bench[hologres]' 'psycopg[binary]' pgvector
```

Execute tests for the index types: HGraph.
Expand Down Expand Up @@ -319,8 +319,8 @@ Options:

The vectordbbench command can optionally read some or all the options from a yaml formatted configuration file.

By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting
the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.
By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting
the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.

The required format is:
```yaml
Expand Down Expand Up @@ -349,16 +349,16 @@ milvushnsw:
drop_old: False
load: False
```
> Notes:
> Notes:
> - Options passed on the command line will override the configuration file*
> - Parameter names use an _ not -

#### Using a batch configuration file.

The vectordbbench command can read a batch configuration file to run all the test cases in the yaml formatted configuration file.

By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting
the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.
By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting
the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.

The required format is:
```yaml
Expand Down Expand Up @@ -387,7 +387,7 @@ milvushnsw:
drop_old: False
load: False
```
> Notes:
> Notes:
> - Options can only be passed through configuration files
> - Parameter names use an _ not -

Expand All @@ -402,11 +402,11 @@ To facilitate the presentation of test results and provide a comprehensive perfo

### Scoring Rules

1. For each case, select a base value and score each system based on relative values.
- For QPS and QP$, we use the highest value as the reference, denoted as `base_QPS` or `base_QP$`, and the score of each system is `(QPS/base_QPS) * 100` or `(QP$/base_QP$) * 100`.
- For Latency, we use the lowest value as the reference, that is, `base_Latency`, and the score of each system is `(base_Latency + 10ms)/(Latency + 10ms) * 100`.
1. For each case, select a base value and score each system based on relative values.
- For QPS and QP$, we use the highest value as the reference, denoted as `base_QPS` or `base_QP$`, and the score of each system is `(QPS/base_QPS) * 100` or `(QP$/base_QP$) * 100`.
- For Latency, we use the lowest value as the reference, that is, `base_Latency`, and the score of each system is `(base_Latency + 10ms)/(Latency + 10ms) * 100`.

We want to give equal weight to different cases, and not let a case with high absolute result values become the sole reason for the overall scoring. Therefore, when scoring different systems in each case, we need to use relative values.
We want to give equal weight to different cases, and not let a case with high absolute result values become the sole reason for the overall scoring. Therefore, when scoring different systems in each case, we need to use relative values.

Also, for Latency, we add 10ms to the numerator and denominator to ensure that if every system performs particularly well in a case, its advantage will not be infinitely magnified when latency tends to 0.

Expand Down Expand Up @@ -467,7 +467,7 @@ All standard benchmark results are generated by a client running on an 8 core, 3
1. Initially, you select the systems to be tested - multiple selections are allowed. Once selected, corresponding forms will pop up to gather necessary information for using the chosen databases. The db_label is used to differentiate different instances of the same system. We recommend filling in the host size or instance type here (as we do in our standard results).
2. The next step is to select the test cases you want to perform. You can select multiple cases at once, and a form to collect corresponding parameters will appear.
3. Finally, you'll need to provide a task label to distinguish different test results. Using the same label for different tests will result in the previous results being overwritten.
Now we can only run one task at the same time.
Now we can only run one task at the same time.
![image](vectordb_bench/fig/run_test_select_db.png)
![image](vectordb_bench/fig/run_test_select_case.png)
![image](vectordb_bench/fig/run_test_submit.png)
Expand Down Expand Up @@ -508,11 +508,11 @@ We have strict requirements for the data set format, please follow them.
- Vectors data files: The file must be named `train.parquet` and should have two columns: `id` as an incrementing `int` and `emb` as an array of `float32`.
- Query test vectors: The file must be named `test.parquet` and should have two columns: `id` as an incrementing `int` and `emb` as an array of `float32`.
- We recommend limiting the number of test query vectors, like 1,000.
When conducting concurrent query tests, Vdbbench creates a large number of processes.
To minimize additional communication overhead during testing,
When conducting concurrent query tests, Vdbbench creates a large number of processes.
To minimize additional communication overhead during testing,
we prepare a complete set of test queries for each process, allowing them to run independently.
However, this means that as the number of concurrent processes increases,
the number of copied query vectors also increases significantly,
However, this means that as the number of concurrent processes increases,
the number of copied query vectors also increases significantly,
which can place substantial pressure on memory resources.
- Ground truth file: The file must be named `neighbors.parquet` and should have two columns: `id` corresponding to query vectors and `neighbors_id` as an array of `int`.

Expand Down Expand Up @@ -542,10 +542,10 @@ VDBBench aims to provide a more comprehensive, multi-faceted testing environment

**Step 2: Implement new_client.py and config.py**

1. Open new_client.py and define the NewClient class, which should inherit from the clients/api.py file's VectorDB abstract class. The VectorDB class serves as the API for benchmarking, and all DB clients must implement this abstract class.
1. Open new_client.py and define the NewClient class, which should inherit from the clients/api.py file's VectorDB abstract class. The VectorDB class serves as the API for benchmarking, and all DB clients must implement this abstract class.
Example implementation in new_client.py:
new_client.py
```python
```python
from ..api import VectorDB
class NewClient(VectorDB):
# Implement the abstract methods defined in the VectorDB class
Expand Down Expand Up @@ -574,7 +574,7 @@ class NewDBCaseConfig(DBCaseConfig):

In this final step, you will import your DB client into clients/__init__.py and update the initialization process.
1. Open clients/__init__.py and import your NewClient from new_client.py.
2. Add your NewClient to the DB enum.
2. Add your NewClient to the DB enum.
3. Update the db2client dictionary by adding an entry for your NewClient.
Example implementation in clients/__init__.py:

Expand Down Expand Up @@ -672,14 +672,14 @@ def ZillizAutoIndex(**parameters: Unpack[ZillizTypedDict]):
)
```
3. Update cli by adding:
1. Add database specific options as an Annotated TypedDict, see ZillizTypedDict above.
1. Add database specific options as an Annotated TypedDict, see ZillizTypedDict above.
2. Add index configuration specific options as an Annotated TypedDict. (example: vectordb_bench/backend/clients/pgvector/cli.py)
1. May not be needed if there is only one index config.
2. Repeat for each index configuration, nesting them if possible.
2. Repeat for each index configuration, nesting them if possible.
2. Add a index config specific function for each index type, see Zilliz above. The function name, in lowercase, will be the command name passed to the vectordbbench command.
3. Update db_config and db_case_config to match client requirements
4. Continue to add new functions for each index config.
5. Import the client cli module and command to vectordb_bench/cli/vectordbbench.py (for databases with multiple commands (index configs), this only needs to be done for one command)
5. Import the client cli module and command to vectordb_bench/cli/vectordbbench.py (for databases with multiple commands (index configs), this only needs to be done for one command)
6. Import the `get_custom_case_config` function from `vectordb_bench/cli/cli.py` and use it to add a new key `custom_case` to the `parameters` variable within the command.


Expand All @@ -697,7 +697,7 @@ For the system under test, we use the default server-side configuration to maint
For the Client, we welcome any parameter tuning to obtain better results.
### Incomplete Results
Many databases may not be able to complete all test cases due to issues such as Out of Memory (OOM), crashes, or timeouts. In these scenarios, we will clearly state these occurrences in the test results.
### Mistake Or Misrepresentation
### Mistake Or Misrepresentation
We strive for accuracy in learning and supporting various vector databases, yet there might be oversights or misapplications. For any such occurrences, feel free to [raise an issue](https://github.com/zilliztech/VectorDBBench/issues/new) or make amendments on our GitHub page.
## Timeout
In our pursuit to ensure that our benchmark reflects the reality of a production environment while guaranteeing the practicality of the system, we have implemented a timeout plan based on our experiences for various tests.
Expand Down