POC: bosh-azure-storage-cli based blobstore client

[johha]

TL;DR

This POC adds a new blobstore client which is based on the bosh-azure-storage-cli.
We consider this as a potential replacement for deprecated fog libraries like azure fog.
The POC proves that cf push of new and existing apps is working and that the client can be used as drop in replacement.

Findings

✅ Push of new apps is working
✅ Push of existing apps is working
✅ Can be used a drop in replacement to fog
✅ app_bits cache upload/download is working correctly
✅ buildpack cache upload/download is working correctly
🚧 Findings need to be discussed with bosh team (maybe also consider renaming)
🚧 Copying files directly on the blobstore is missing in bosh-azure-storage-cli
🚧 Functionality not directly related to cf push is not yet implemented (like delete_all)
🚧 Further performance tests needed

How To Use

- Checkout POC branch in ccng

- Modify spec files and add azure-storage-cli package in capi-release

diff --git a/config/blobs.yml b/config/blobs.yml
index f350a3f8..3bb8764d 100644
--- a/config/blobs.yml
+++ b/config/blobs.yml
@@ -1,3 +1,6 @@
+azure-storage-cli/azure-storage-cli-linux-amd64:
+  size: 10183290
+  sha: sha256:4ef14f333431fc5a21705ddeb8bd349ed8011d7a6f40681c9640a38edec871e8
 expat/expat-2.5.0.tar.bz2:
   size: 569205
   object_id: 970ccd16-75ac-4c55-5280-c00c4aa8f6cc
diff --git a/jobs/cloud_controller_clock/spec b/jobs/cloud_controller_clock/spec
index 990c7f70..7d804265 100644
--- a/jobs/cloud_controller_clock/spec
+++ b/jobs/cloud_controller_clock/spec
@@ -29,6 +29,7 @@ templates:
 packages:
   - capi_utils
   - cloud_controller_ng
+  - azure-storage-cli
   - nginx
   - nginx_newrelic_plugin
   - libpq
diff --git a/jobs/cloud_controller_ng/spec b/jobs/cloud_controller_ng/spec
index 8b659760..b8c27b36 100644
--- a/jobs/cloud_controller_ng/spec
+++ b/jobs/cloud_controller_ng/spec
@@ -68,6 +68,7 @@ templates:
 packages:
   - capi_utils
   - cloud_controller_ng
+  - azure-storage-cli
   - nginx
   - nginx_newrelic_plugin
   - libpq
diff --git a/jobs/cloud_controller_worker/spec b/jobs/cloud_controller_worker/spec
index 2e405ae6..188fab68 100644
--- a/jobs/cloud_controller_worker/spec
+++ b/jobs/cloud_controller_worker/spec
@@ -39,6 +39,7 @@ templates:
 packages:
   - capi_utils
   - cloud_controller_ng
+  - azure-storage-cli
   - nginx
   - nginx_newrelic_plugin
   - libpq
diff --git a/packages/azure-storage-cli/packaging b/packages/azure-storage-cli/packaging
new file mode 100644
index 00000000..d96505a0
--- /dev/null
+++ b/packages/azure-storage-cli/packaging
@@ -0,0 +1,5 @@
+set -e
+
+mkdir -p ${BOSH_INSTALL_TARGET}/bin
+mv azure-storage-cli/azure-storage-cli-linux-amd64 ${BOSH_INSTALL_TARGET}/bin/azure-storage-cli
+chmod +x ${BOSH_INSTALL_TARGET}/bin/azure-storage-cli
\ No newline at end of file
diff --git a/packages/azure-storage-cli/spec b/packages/azure-storage-cli/spec
new file mode 100644
index 00000000..6066ea9a
--- /dev/null
+++ b/packages/azure-storage-cli/spec
@@ -0,0 +1,3 @@
+name: azure-storage-cli
+files:

- Add azure-storage-cli-linux-amd64 to capi-release

bosh add-blob ~/azure-storage-cli-0.0.166-linux-amd64 azure-storage-cli/azure-storage-cli-linux-amd64

- Build CAPI Dev Release

- Set blobstore_type: cli in cf manifest file

- Deploy

[sethboyles]

Any idea if the bosh blobstore CLIs have consistent interfaces? Do you think that in the future we could have something like one client that takes the provider CLI as an injection?

[johha]

The CLIs have in general consistent interfaces:

<cli> -c config.json put <path/to/file> <remote-blob> 
<cli> -c config.json get <remote-blob> <path/to/file>
<cli> -c config.json delete <remote-blob>
<cli> -c config.json exists <remote-blob>
<cli> -c config.json sign <remote-blob> <get|put> <seconds-to-expiration>

There are few extra options for sign for the GCS CLI but not sure yet if we need them.
In bosh they use dedicated clients which implement a common interface: https://github.com/cloudfoundry/bosh/tree/main/src/bosh-director/lib/bosh/director/blobstore
Something similar should work also work well in ccng.
The main difference between the different CLIs is within the config files as they look quite different, especially for s3.

[johha]

The bosh storage CLIs covers:

• S3: https://github.com/cloudfoundry/bosh-s3cli
• GCS: https://github.com/cloudfoundry/bosh-gcscli
• Azure: https://github.com/cloudfoundry/bosh-azure-storage-cli
• Ali: https://github.com/cloudfoundry/bosh-ali-storage-cli
• WebDav: https://github.com/cloudfoundry/bosh-davcli

❗ There is no CLI for openstack. Would this be a blocker or could we e.g. handle this with the s3 CLI?

[johha]

Results of the blobstore benchmark running with azure-storage-cli and fog-azure. The table shows the average over 3 runs:

Benchmark	Azure Storage CLI	Azure Fog
resource match timing	1.18ms	1.06ms
package upload timing	43.12ms	202.30ms
package download timing	35.85ms	66.64ms
buildpack download timing	28884.37ms	40692.25ms
droplet upload timing	36.13ms	33.34ms
droplet download timing	36.61ms	33.34ms
big droplet upload timing	1594.93ms	1858.73ms
big droplet download timing	3654.25ms	4528.07ms

Overall the performance looks promising - especially the buildpack download is much faster.
Note: This was just a basic test, for a final decision more performance tests are needed.