Add Vertex AI Vector Search Load Testing Framework (#1462)

* Adding Load Testing Framework to Professional Service Tools folder

* readme updates

* Productionize the code

* Adding pre-existing directories failing lint test to exclusion_list

* added minor adjustments

* Removing Formatting Issues

* Removal of shellcheck error

* Add pushd and popd

* Formatting issue with locust.py file

---------

Co-authored-by: Jesse Carah <[email protected]>
Co-authored-by: John DeMartino <[email protected]>
Co-authored-by: Andrew Gold <[email protected]>
Co-authored-by: jdemartino78 <[email protected]>

Author: 4 people

README.md

@@ -576,6 +576,11 @@ Platform usage.
 *   [STS Job Manager](tools/sts-job-manager/) - A petabyte-scale bucket
     migration tool utilizing
     [Storage Transfer Service](https://cloud.google.com/storage-transfer-service)
+*   [Vector Search Load Testing Framework](tools/vector-search-load-testing-framework) - This 
+    framework provides a streamlined solution for distributed load testing of Vertex AI
+    Vector Search endpoints on Google Kubernetes Engine (GKE) using
+    [Locust](https://locust.io/). It enables you to simulate production-like workloads to effectively benchmark performance, analyze scalability, and validate deployment
+    configurations.
 *   [Vertex AI Endpoint Tester] (tools/vertex-ai-endpoint-load-tester) - This 
     utility helps to methodically test variety of Vertex AI Endpoints by their
     sizes so that one can decide the right size to deploy an ML Model on Vertex

helpers/exclusion_list.txt

@@ -2,25 +2,30 @@
 ./tools/apachebeam-throttling
 ./tools/asset-inventory
 ./tools/bigquery-query-plan-exporter
+./tools/bigquery-s3tobq
 ./tools/bigquery-zos-mainframe-connector
+./tools/bqms-config-generator
 ./tools/bqpipeline
 ./tools/slo-definition-converter
 ./tools/bq-visualizer
 ./tools/cloudconnect
 ./tools/cloudera-parcel-gcsconnector
+./tools/cloud-composer-stress-testing
 ./tools/cloud-vision-utils
 ./tools/cuds-prioritized-attribution
 ./tools/dataproc-edge-node
 ./tools/dns-sync
 ./tools/gce-google-keys-to-cmek
 ./tools/gce-quota-sync
+./tools/gcpviz
 ./tools/gcp-arch-viz
 ./tools/gcs-bucket-mover
 ./tools/gcs-usage-recommender
 ./tools/gsuite-exporter
 ./tools/kunskap
 ./tools/labelmaker
 ./tools/maven-archetype-dataflow
+./tools/memorystore-cluster-ops-framework
 ./tools/ml-dataprep
 ./tools/ml-auto-eda
 ./tools/netblock-monitor
@@ -30,7 +35,12 @@
 ./examples/bigquery-audit-log
 ./examples/bigquery-billing-dashboard
 ./examples/bigquery-cross-project-slot-monitoring
+./examples/bigquery-ddl-validator
+./examples/bigquery-generic-ddl-migration-utility
+./examples/bigquery-long-running-optimizer
 ./examples/bigquery-row-access-groups
+./examples/bigquery-snowflake-tables-migration-utility
+./examples/bigquery-translation-validator-utility
 ./examples/bigquery-ml-claudeintegrations
 ./examples/bq_benchmarks
 ./examples/cloud-composer-examples
@@ -48,21 +58,28 @@
 ./examples/dataflow-bigquery-transpose
 ./examples/dataflow-data-generator
 ./examples/dataflow-elasticsearch-indexer
+./examples/dataflow-production-ready
 ./examples/dataflow-scala-kafka2avro
 ./examples/dataflow-streaming-benchmark
+./examples/dataproc-gcs-connector
+./examples/dataproc-idle-shutdown
 ./examples/dataproc-persistent-history-server
 ./examples/dlp
 ./examples/e2e-home-appliance-status-monitoring
 ./examples/gce-to-adminsdk
 ./examples/gcf-pubsub-vm-delete-event-handler
+./examples/gcs-client-encrypt
 ./examples/iot-nirvana
 ./examples/kubeflow-fairing-example
 ./examples/kubeflow-pipelines-sentiment-analysis
+./examples/left-shift-validation-pre-commit-hook
 ./examples/ml-audio-content-profiling
 ./examples/pubsub-publish-avro-example
 ./examples/python-cicd-with-cloudbuilder
 ./examples/qaoa
 ./examples/redis-cluster-gke
+./examples/redis-benchmarks
 ./examples/spinnaker
 ./examples/tensorflow-profiling-examples
 ./examples/ccai-agentassist-five9-grpc
+./examples/vertex_mlops_enterprise

tools/vector-search-load-testing-framework/Dockerfile

@@ -0,0 +1,21 @@
+FROM locustio/locust
+
+# Set working directory
+WORKDIR /tasks
+
+# Copy Locust task file
+COPY locust_tests/locust.py ./
+
+# Install dependencies
+RUN pip install -U \
+    google-auth \
+    google-cloud-storage \
+    google-cloud-logging \
+    python-dotenv \
+    google-cloud-aiplatform \
+    grpcio \
+    grpc_interceptor \
+    grpcio-status
+
+# No need to copy the config file here since it's mounted as a ConfigMap
+# The command to run will be provided by the Kubernetes deployment