Skip to content

[DO NOT MERGE] [Only for the long running PW job] Pathways gcsfuse intergation #401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 11 commits into
base: develop
Choose a base branch
from
Draft

[DO NOT MERGE] [Only for the long running PW job] Pathways gcsfuse intergation #401

wants to merge 11 commits into from

Conversation

shauryagup
Copy link
Collaborator

Fixes / Features

  • This enables memory mounted gcsfuse buckets for the large scale cluster
  • Changes the startup policy of the Pathways pods
  • Removes podFailurePolicy from pathways containers

Testing / Documentation

Verified on a large scale run

The startupPolicy and podFailurePolicy related changes can be merged into main

The gcsfuse related changes are not ready to be merged into main and needs further work:

  1. We need to integrate with the xpk storage API to properly plumb gcsfuse related volumes/volume mounts/annotations into PW container definitions.
  2. The changes rely on PV and PVC resources to be already present on the cluster. The storage API integration would also ensure that those resources are created by XPK automatically

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Obliviour Obliviour Awaiting requested review from Obliviour Obliviour will be requested when the pull request is marked ready for review Obliviour is a code owner

@44past4 44past4 Awaiting requested review from 44past4 44past4 will be requested when the pull request is marked ready for review 44past4 is a code owner

@sharabiani sharabiani Awaiting requested review from sharabiani sharabiani will be requested when the pull request is marked ready for review sharabiani is a code owner

@pawloch00 pawloch00 Awaiting requested review from pawloch00 pawloch00 will be requested when the pull request is marked ready for review pawloch00 is a code owner

@BluValor BluValor Awaiting requested review from BluValor BluValor will be requested when the pull request is marked ready for review BluValor is a code owner

@gcie gcie Awaiting requested review from gcie gcie will be requested when the pull request is marked ready for review gcie is a code owner

@RoshaniN RoshaniN Awaiting requested review from RoshaniN RoshaniN will be requested when the pull request is marked ready for review RoshaniN is a code owner

@scaliby scaliby Awaiting requested review from scaliby scaliby will be requested when the pull request is marked ready for review scaliby is a code owner

@jamOne- jamOne- Awaiting requested review from jamOne- jamOne- will be requested when the pull request is marked ready for review jamOne- is a code owner

@SikaGrr SikaGrr Awaiting requested review from SikaGrr SikaGrr will be requested when the pull request is marked ready for review SikaGrr is a code owner

@FIoannides FIoannides Awaiting requested review from FIoannides FIoannides will be requested when the pull request is marked ready for review FIoannides is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@shauryagup @SujeethJinesh