Skip to content

Rough Draft Implementation of Pipeline, PipelineJob, PipelineRun, and PipelineRunner #83

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 40 commits into
base: dev
Choose a base branch
from

Conversation

Synicix
Copy link
Contributor

@Synicix Synicix commented May 19, 2025

Added:

  • Pipeline struct (Still missing hashing and to yaml)
  • PipelienJob (Still missing hashing, plus additional fields that we might want to add
  • PipelineRun
  • DockerPipelineRunner: An async task based runner of pipelines

TODO:
General:

  • Add to yaml and hashing to the above
  • Create the storage function needed to store results
  • Formalize the yaml to match with @guzman-raphael design
  • Tag is completely missing

Pipeline:

  • Add functionailty to actually create the pod job and send it to the orchestrator (Needs discussion to figure out, the details of what to have the pod_job be)
  • Need to add a more proper pipeline unlike the toy example included in the draft at the moment
  • How to deal with multiple streams

DockerPipelineRunner:

  • Add functionailty to save pod results
  • Need to add test case to test mutiple parents
  • Need to add test case to test mutiple children
  • Discuss what functions we want to have in detail, do we just copy docker orchestrator somewhat?

Known Bugs:

  • Rust 1.87.0? update seem to screw with clippy and rust analyser, not sure how to fix it but current it is throwing incorrect warnings for test and one part of the code.

Other pending question:

  • Should we add an option for the pipeline executor accept pod jobs and run and wait on them to save the pod result like a server?

@Synicix Synicix marked this pull request as draft May 19, 2025 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant