failed to execute component: unable to create directory for output artifact : mkdir /minio: permission denied

[sandeepbandarupalli]

Hi. I am trying to create a Kubeflow pipeline component that takes Checkpoint (type: str) as an input and outputs the model and image processor. But I am unable to do so. This is being run on Canonical Charmed Kubeflow on a local PC with Ubuntu 24.04 (followed the installation steps provided here: https://charmed-kubeflow.io/docs/get-started-with-charmed-kubeflow#configure-dashboard-access)
When I run the pipeline, it gives me the following error:

time="2024-10-02T11:30:03.410Z" level=info msg="capturing logs" argo=true
time="2024-10-02T11:30:03.437Z" level=info msg="capturing logs" argo=true
I1002 11:30:03.451215 80 launcher_v2.go:90] input ComponentSpec:{
"inputDefinitions": {
"parameters": {
"checkpoint": {
"parameterType": "STRING"
}
}
},
"outputDefinitions": {
"artifacts": {
"model_path": {
"artifactType": {
"schemaTitle": "system.Model",
"schemaVersion": "0.0.1"
}
},
"processor_path": {
"artifactType": {
"schemaTitle": "system.Artifact",
"schemaVersion": "0.0.1"
}
}
}
},
"executorLabel": "exec-load-model-processor"
}
I1002 11:30:03.451760 80 cache.go:139] Cannot detect ml-pipeline in the same namespace, default to ml-pipeline.kubeflow:8887 as KFP endpoint.
I1002 11:30:03.451782 80 cache.go:116] Connecting to cache endpoint ml-pipeline.kubeflow:8887
I1002 11:30:03.480470 80 launcher_v2.go:151] publish success.
F1002 11:30:03.480500 80 main.go:49] failed to execute component: unable to create directory "/minio/mlpipeline/v2/artifacts/model-processor-pipeline/1f61e1fb-6ea9-438d-9eea-2804d3a3046a/load-model-processor" for output artifact "model_path": mkdir /minio: permission denied
time="2024-10-02T11:30:04.438Z" level=info msg="sub-process exited" argo=true error=""
Error: exit status 1
time="2024-10-02T11:30:05.413Z" level=info msg="sub-process exited" argo=true error=""
Error: exit status 1

Can anyone tell me where I am making a mistake?

Here is the code:

from kfp import dsl
from kfp.dsl import Output, Input,Model, Artifact  
from kfp import compiler

# Define the component directly using the @dsl.component decorator
@dsl.component(base_image="sandeepb1997/kfp-custom-image:v6.1.0")
def load_model_processor(checkpoint: str, model_path: Output[Model], processor_path: Output[Artifact]):
    from transformers import RTDetrImageProcessor, RTDetrForObjectDetection
    import torch

    # Define constants
    CHECKPOINT = checkpoint
    IMAGE_SIZE = 480
    DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    # Load model
    model = RTDetrForObjectDetection.from_pretrained(CHECKPOINT).to(DEVICE)

    # Load and configure processor
    processor = RTDetrImageProcessor.from_pretrained(
        CHECKPOINT,
        do_resize=True,
        size={"width": IMAGE_SIZE, "height": IMAGE_SIZE}
    )
    
    # Save model to the model_path directory
    model.save_pretrained(model_path.path)
    
    # Save processor to the processor_path directory
    processor.save_pretrained(processor_path.path)
    
    # Log the paths for troubleshooting
    print(f"Model saved to: {model_path.path}")
    print(f"Processor saved to: {processor_path.path}")


# Define the pipeline
@dsl.pipeline(name='model-processor-pipeline', description="Pipeline to load a model and processor.")
def model_pipeline(checkpoint: str):
    # Call the component
    load_model_task = load_model_processor(checkpoint=checkpoint)

# Compile the pipeline to a YAML file
pipeline_file_path = 'model_processor_pipeline.yaml'
compiler.Compiler().compile(pipeline_func=model_pipeline, package_path=pipeline_file_path)

if __name__ == '__main__':
    # Instantiate the KFP client
    client = kfp.Client()

    # Define pipeline parameters
    checkpoint = "PekingU/rtdetr_r50vd"

    # Set up the MinIO storage path for model artifacts
    run = client.create_run_from_pipeline_func(
        model_pipeline,
        arguments={
            'checkpoint': checkpoint,
        },
        experiment_name='model-experiment',
        run_name='model-processor-run'
    )

[HumairAK]

mkdir /minio: permission denied

Couple of things come to mind:

• this likely before the change that mounts emptydir for the artifact output paths in the launcher container
• also maybe before the changes to the umask that was used for creating these directories

eitherway, this should have been resolved with the newer kfp versions, what version are you on currently?

[HumairAK]

A possible workaround is to just create these directories in your runtime image and give them the appropriate permissions

[sandeepbandarupalli]

I am on kubeflow 1.9 and kfp version is 2.9

[HumairAK]

KF 1.9 uses KFP 2.2, I think we fixed this in 2.3.0: #10857
In which case my suggestion would be to ensure the runtime image has these paths already created.