Add script for CLI tests with local dandi-archive

[asmacdo]

I found setting up CLI integration tests to be pretty fiddly, and so ultimately just created this script to do it for me. If others think this would be useful, it might make a nice dev tool.

Is scripts/ the right place for this?

[yarikoptic]

given what is there under scripts ATM:

❯ git ls-files scripts/
scripts/delete_from_versioned_bucket.py
scripts/papertrail.py

might as well have this one IMHO. But I think that especially given that there is no tests for it:

• would it be possible to shrink .github/workflows/cli-integration.yml to use this script instead?
• why .github/workflows/cli-integration.yml is so much more concise than this script, or in other words: what script does in addition to what that CI workflow does ATM?
• if we add it - we should shellecheckit and make pass shellcheck (does it ATM? we do not have any other shell script here so cannot even offer shellcheck workflow ATM)

[asmacdo]

I considered updating cli-integration.yaml to use this, seems like a decent idea, but I wanted to see if there was a desire for this script before spending the time to do that.

This is more involved than cli-integration.yaml because it is designed to be flexible for developer use,

Extra features for development not needed by ci:

cleanup already running containers:
https://github.com/dandi/dandi-archive/pull/2395/files#diff-bc9c2cb06738b8461f05e1e7524d53619e8bb66d01d04e25087f95ab4dafd96cR97-R109

option to skip rebuild
https://github.com/dandi/dandi-archive/pull/2395/files#diff-bc9c2cb06738b8461f05e1e7524d53619e8bb66d01d04e25087f95ab4dafd96cR115-R126

test a specific test
https://github.com/dandi/dandi-archive/pull/2395/files#diff-bc9c2cb06738b8461f05e1e7524d53619e8bb66d01d04e25087f95ab4dafd96cR153-R167

skip cleanup for manual debugging
https://github.com/dandi/dandi-archive/pull/2395/files#diff-bc9c2cb06738b8461f05e1e7524d53619e8bb66d01d04e25087f95ab4dafd96cR136-R142

[asmacdo]

There are also IMO some decent reasons not to integrate this with the workflow.

PRO:

• slightly less duplication
  • Docker build https://github.com/dandi/dandi-archive/blob/master/.github/workflows/cli-integration.yml#L20-L34
  • env var setting
• tests also cover the script

CON:

• CI has more "magic" and is less explicit
• the workflow is very explicit about what its doing for install during the matrix https://github.com/dandi/dandi-archive/blob/master/.github/workflows/cli-integration.yml#L77-L90, integration with the dev script would make it less clear I think
• CI currently uploads and uses the tarred docker image (do you know why?)
  • This might be awkward to integrate with the dev script
  • https://github.com/dandi/dandi-archive/blob/master/.github/workflows/cli-integration.yml#L35-L43

I'd still vote to keep them separate so the CI stays very clean and explicit.