[TASK] Run e2e tests in parallel

[sdischer-sap]

Context
To improve performance of e2e testing we can try out to run them in parallel. In theory there should be no naming conflicts in the subaccounts, since due to templating we give unique name to each of them.

Technical Steps

•  run e2e tests in parallel and make sure they don't interfere with each other

Additional context
In order for that to be useful, we should make sure logs are related to their test cases, which should be handled as part of #239 , more specifically to be ready for parallel testing in #270

Additional Question/Concern:

• Do we run into API Rate Limits?

[sdischer-sap]

Has a dependency to #239, so needs some rethinking as well

[enrico-kaack-comp]

I experimented in #355 with parallelisation.
The good news: I was successful in having one successful run completing in 20 min here

Subsequent runs failed.

What was necessary:

• the AddToScheme(s) function in package apis was not thread-safe since it adds schemes to a map. I created a AddToSchemeConcurrent(s) function that uses a mutex to give exclusive access to the map. By editing the template that generates zz_register.go` we can make a safe to use thread-safe version of it I think.
• increase parallel count in the test flag
• use xp-testing v1.7.0 for some concurrent map fixes

The entitlements are failing a bit unreliable now. Most likely it is due to the timeouts being too low (since now we are throttled by the k8s api server --> maybe we can increase the limits in the api server here too). Or we increase the timeouts for the entitlements.

That this is useful if it fails though, we need to solve #239 to enable parallel logging.
We can also decide to run each Test as a separate job in the pipeline (maybe use matrix testing). Disadvantage: we couple our pipeline configuration to our test execution. Advantage: No other modification is necessary since we just run one test per kind cluster and the logs will be sequential anyway.

[enrico-kaack-comp]

I tried to run each Test as a separate job in the pipeline using matrix testing: #358

That works beautifully with a total duration of 23 minutes for short tests: https://github.com/SAP/crossplane-provider-btp/actions/runs/18168130269 and long running e2e tests in 40min https://github.com/SAP/crossplane-provider-btp/actions/runs/18186798249

Advantages:

• no code modification necessary, the test run fully isolated with just a single test per instance
• clear log separation, as every job (=test) has its own job log in github
• easy to spot which test has failed
• easy to restart single failed tests to get everything green

Disadvantage:

• every Test needs to be written into the workflow file --> couples testing logic to pipeline. We might forget to add new tests
• only the main branch worflow file is executed, that means when adding new tests we have to trigger the test manually for this branch to get the changes from the workflow file in the feature branch (kind of ugly, but we are not adding new tests that often)
• higher resource usage since we are building a docker image, starting a kind in every test

[sdischer-sap]

As I understood this was already presented in a dev sync right? Just in case I misunderstood would be great to have that one ;)