Bump mysql container image to one with an aarch64 image

[alexcreasy]

Description

• Cannot deploy to k8s on AArch64 nodes using manifests in repo #266

Bumps the version of the MySQL container image in the overlays/db kustomization to 8.0.39 which is multi-arch (amd64 / aarch64)

How Has This Been Tested?

Deployed to a local kind cluster on an apple silicon MacBook, using official instructions to deploy using kind on macOS.

*https://www.kubeflow.org/docs/components/pipelines/legacy-v1/installation/localcluster-deployment/#kind
*https://www.kubeflow.org/docs/components/model-registry/installation/#standalone-installation

The only variation of above was that I deployed to kind using my branch of the manifest:
kubectl apply -k "https://github.com/alexcreasy/model-registry/manifests/kustomize/overlays/db?ref=kind"

I've verified I'm able to deploy as above, create a port forward to the service, and query the rest endpoint as indicated in the example above and receive a 200 response.

Merge criteria:

• The commits and have meaningful messages; the author will squash them after approval or will ask to merge with squash.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[alexcreasy]

Ahhh... yay the old "it worked on my laptop" scenario 🤦

I guess the new image isn't being loaded into kind on the CI jobs.

[rareddy]

isn't MR using MariaDB not mySQL that too a version that is matched with KFP?

[tarilabs]

I believe we had to stick with the currently used version, as that's the one tested to be working with MLMD.
Further, this won't solve the issue you linked, for the reasons I've summarized in #266 (comment)

Let me know your thoughts :)

[alexcreasy]

@tarilabs thanks for taking a look, I believe it will solve my issues, as I'm trying to deploy model registry without kubeflow (full explanation of what I'm trying to do on the ticket) - granted I've only run a few curl requests against the MR API on my local kind deployment, so there could be some hidden issue I'm not aware of, but so far they work as expected using the modified manifests on my fork.

Let me know if I've misunderstood the core problem you described :)

[rareddy]

I believe we had to stick with the currently used version, as that's the one tested to be working with MLMD.

I was under impression that KFP team use MariaDB for writing some windowing functions over it which were only supported in MariaDB and working fine for them. May be this only used for Red Hat's distribution.

[tarilabs]

I was under impression that KFP team use MariaDB for writing some windowing functions over it which were only supported in MariaDB and working fine for them. May be this only used for Red Hat's distribution.

I believe the latter (and I believe the "window function" should have parity between mysql and mariadb).

this is as far as I can see KF 1.8:

https://github.com/kubeflow/manifests/blob/a925b2968e68f3085092adc04e9af39d4f5045f3/apps/pipeline/upstream/base/metadata/overlays/db/metadata-db-deployment.yaml#L24

KF 1.9:

https://github.com/kubeflow/manifests/blob/a38c2be88fbafb0844c0231f0062e4b3719d4737/apps/pipeline/upstream/base/metadata/overlays/db/metadata-db-deployment.yaml#L24

and default branch "master":

https://github.com/kubeflow/manifests/blob/bf3dd8878af2a52480e60ee90d650a098eacd113/apps/pipeline/upstream/base/metadata/overlays/db/metadata-db-deployment.yaml#L24

Always mysql:8.0.3.

My recommendation is as follows:

1. send a PR first to KFP/manifest proposing this change, so we can gauge if pipeline team is in favour of aligning on this version (otherwise, we would be diverging from the common configuration)
2. for DX, which is the goal of this PR, this could be handled by a different overlay for "dev"
3. assess why tests are failing (condition on DB is not met)

[tarilabs]

I want to augment the comment in #267 (review)
with the information that the default vanilla installation uses a custom image of mysql from google container registry:

https://github.com/kubeflow/manifests/blob/a38c2be88fbafb0844c0231f0062e4b3719d4737/apps/pipeline/upstream/third-party/mysql/base/mysql-deployment.yaml#L51

👉 gcr.io/ml-pipeline/mysql:8.0.26

[tarilabs]

I still wonder why it fails at

kubectl wait --for=condition=available -n kubeflow deployment/model-registry-db --timeout=5m

[alexcreasy]

@tarilabs I'm not sure, I haven't looked into it yet - it's strange, I can deploy this to kind locally. I'll dig into this next week and see what I can find.

[alexcreasy]

I wonder if it's some issue with pulling the image? At a guess (and this is a complete guess) I wonder if the db image needs to be manually loaded into kind for the deployment to be able to use it.

[tarilabs]

I wonder if it's some issue with pulling the image? At a guess (and this is a complete guess) I wonder if the db image needs to be manually loaded into kind for the deployment to be able to use it.

Maybe, but this looks not being a problem for mysql 8.0.3 🤔

[tarilabs]

let's see if 7a8bded is of any help in digging further into the actual error 🧐

[tarilabs]

let's see if 7a8bded is of any help in digging further into the actual error 🧐

with the log extraction, locate the following:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

which brings up:

• MySQL Pod fails to become ready on KinD cluster manifests#2507
• Update Katib tests for MySQL issue with GH Runner manifests#2508
• https://kind.sigs.k8s.io/docs/user/known-issues/#apparmor
• MySQL fails to run on Github Actions using Kind actions/runner-images#181 (comment)

so the solution appears to be removing the AppArmor profile for mysql in the GHA runner...

[tarilabs]

All green now, awesome 🚀

Now, if only we had some answer on...

• Indication for support of MySQL/MariaDB pipelines#11224

[tarilabs]

btw, I assume this resolves #266

[tarilabs]

btw, this also should be of great advantage to #327 (comment)

[tarilabs]

@alexcreasy with the latest rebase event from #267 (comment) I assume we lost commits:

• 7ad916e
• bd3563d
• 7c4f26c

that were discussed with #267 (comment)

I've rebased again on main and pushed here.
Kindly account as you will use your local branch with this PR open.

Hope this helps!

[alexcreasy]

@tarilabs ahh thanks, I'll test this locally now, do you need me to do anything else?

[tarilabs]

as discussed in today's KF MR biweekly meeting, proceeding to merge.

[tarilabs]

as discussed in today's KF MR biweekly meeting, proceeding to merge.

/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tarilabs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tarilabs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment