[Feat] Support more running mode in workforce #3157

Ol1ver0413 · 2025-09-18T16:11:01Z

Description

Add pipeline running mode in workforce. #1663

Checklist

Go over all the following points, and put an x in all the boxes that apply.

I have read the CONTRIBUTION guide (required)
I have linked this PR to an issue using the Development section on the right sidebar or by adding Fixes #issue-number in the PR description (required)
I have checked if any dependencies need to be added or updated in pyproject.toml and uv lock
I have updated the tests accordingly (required for a bug fix or a new feature)
I have updated the documentation if needed:
I have added examples if this is a new feature

If you are unsure about any of these, don't hesitate to ask. We are here to help!

coderabbitai · 2025-09-18T16:11:10Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch workforce_pipeline

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Ol1ver0413 · 2025-09-21T14:55:38Z

graph TD
    A[Task: Generate 5 AI/ML Papers] --> B[Literature Researcher]
    B --> C{Fork: 5 Parallel Tasks}
    
    C --> D1[Summary Specialist 1<br/>Summarize Paper 1]
    C --> D2[Summary Specialist 2<br/>Summarize Paper 2]
    C --> D3[Summary Specialist 3<br/>Summarize Paper 3]
    C --> D4[Summary Specialist 4<br/>Summarize Paper 4]
    C --> D5[Summary Specialist 5<br/>Summarize Paper 5]
    
    D1 --> E{Join: Collect Summaries}
    D2 --> E
    D3 --> E
    D4 --> E
    D5 --> E
    
    E --> F[Research Synthesizer<br/>Analyze AI/ML Trends]
    F --> G[Final Result]
    
    style A fill:#e1f5fe
    style B fill:#fff3e0
    style D1 fill:#f3e5f5
    style D2 fill:#f3e5f5
    style D3 fill:#f3e5f5
    style D4 fill:#f3e5f5
    style D5 fill:#f3e5f5
    style F fill:#e8f5e8
    style G fill:#fff9c4

Ol1ver0413 · 2025-09-21T14:56:11Z

graph TD
    A[Task: Generate 4 RESTful API Types] --> B[API Researcher]
    B --> C{Fork: 4 Parallel Tasks}
    
    C --> D1[API Analyst 1<br/>Analyze API 1]
    C --> D2[API Analyst 2<br/>Analyze API 2]
    C --> D3[API Analyst 3<br/>Analyze API 3]
    C --> D4[API Analyst 4<br/>Analyze API 4]
    
    D1 --> E{Join: Collect Analyses}
    D2 --> E
    D3 --> E
    D4 --> E
    
    E --> F[Documentation Writer<br/>Generate API Usage Guide]
    F --> G[Final Result]
    
    style A fill:#e1f5fe
    style B fill:#fff3e0
    style D1 fill:#f3e5f5
    style D2 fill:#f3e5f5
    style D3 fill:#f3e5f5
    style D4 fill:#f3e5f5
    style F fill:#e8f5e8
    style G fill:#fff9c4

Ol1ver0413 · 2025-09-21T14:56:42Z

graph TD
    A[Task: Code Review Analysis] --> B[Code Scanner]
    B --> C{Fork: 3 Parallel Tasks}
    
    C --> D1[Code Reviewer 1<br/>Review File 1]
    C --> D2[Code Reviewer 2<br/>Review File 2]
    C --> D3[Code Reviewer 3<br/>Review File 3]
    
    D1 --> E{Join: Collect Reviews}
    D2 --> E
    D3 --> E
    
    E --> F[Review Summarizer<br/>Generate Comprehensive Report]
    F --> G[Final Result]
    
    style A fill:#e1f5fe
    style B fill:#fff3e0
    style D1 fill:#f3e5f5
    style D2 fill:#f3e5f5
    style D3 fill:#f3e5f5
    style F fill:#e8f5e8
    style G fill:#fff9c4

Ol1ver0413 · 2025-09-21T15:24:03Z

Hey @Wendong-Fan @fengju0213 ! I've already add pipeline mode in workforce, and maybe we can try more cases to test this mode.

Wendong-Fan · 2025-09-23T03:39:27Z

Hey @Wendong-Fan @fengju0213 ! I've already add pipeline mode in workforce, and maybe we can try more cases to test this mode.

thanks @Ol1ver0413 !

hesamsheikh

Thanks for the comprehensive PR @Ol1ver0413. I added a comment. Also I think the method namings could be a bit more consistent and remembrable.

  # before
  workforce.add_pipeline_task()
  workforce.fork_pipeline()
  workforce.add_parallel_pipeline_tasks()
  builder.fork()  
  builder.add_parallel_tasks() 

  # after
  workforce.pipeline_add()
  workforce.pipeline_fork()
  workforce.pipeline_join()
  workforce.pipeline_build()
  builder.add()
  builder.fork()
  builder.join()
  builder.build()

examples/workforce/pipeline_workflow_example.py

hesamsheikh · 2025-09-29T16:05:19Z

camel/societies/workforce/workforce.py

+        expected_task_ids = {task.id for task in self._pending_tasks}
+        expected_task_ids.update(task.id for task in self._completed_tasks)
+
+        completed_successful_ids = {
+            task.id for task in self._completed_tasks 
+            if task.state == TaskState.DONE
+        }
+


doesn't this create a situation if which if any of the branched tasks fails, the whole pipeline is marked as failed and the join task never executes?

doesn't this create a situation if which if any of the branched tasks fails, the whole pipeline is marked as failed and the join task never executes?

Thanks @hesamsheikh ! Maybe I need to add some branch task failure handling mechanisms into the pipeline.

Hi, @hesamsheikh. Since the original design follows a pipeline pattern, I didn’t perform task restructuring or decomposition as in the decompose mode. Therefore, I decided that if a task fails, it would still be considered successful, and the error message would be passed to the join stage. We can discuss whether there might be a better approach.

Well this method seems a bit confusing and hard to understand anyway. Could you maybe simplify it, or explain it a bit in the comments? I know how it works and what it does, but not the intent of it.

doesn't this create a situation if which if any of the branched tasks fails, the whole pipeline is marked as failed and the join task never executes?

Yeah, I'll add more comments.

a7m-1st · 2025-10-02T18:05:31Z

camel/societies/workforce/workforce.py

+
+        # Clear existing tasks and dependencies
+        self._pending_tasks.clear()
+        self._task_dependencies.clear()
+        self._assignees.clear()
+


@Ol1ver0413 , thanks for the PR, but as of now, Is the pipeline designed that it could interrupt running tasks or is it only with predefined tasks only? i.e. before workforce.start()

@Ol1ver0413 , thanks for the PR, but as of now, Is the pipeline designed that it could interrupt running tasks or is it only with predefined tasks only? i.e. before workforce.start()

Thanks @a7m-1st. The pipeline mode was designed for sequential predefined tasks, which cannot be interrupted by human. But maybe we can have a further discussion for it. Thanks again!

Wendong-Fan · 2025-10-19T13:11:13Z

hi @Ol1ver0413

Hope you're doing well!

Just wanted to check in on this PR. It looks like there's some feedback that needs to be addressed, along with some failing in pre-commit checks.

Please let us know if you have any questions or need a hand with anything. We're looking forward to getting this merged!

Cheers

Ol1ver0413 · 2025-10-19T14:42:00Z

hi @Ol1ver0413

Hope you're doing well!

Just wanted to check in on this PR. It looks like there's some feedback that needs to be addressed, along with some failing in pre-commit checks.

Please let us know if you have any questions or need a hand with anything. We're looking forward to getting this merged!

Cheers

Hi, @Wendong-Fan!
I’ve been handling some work transitions lately, which may have caused me to overlook enhancing the running mode in the workforce. I’ll make sure to complete it soon.
Thanks again.

review-notebook-app · 2025-10-20T15:18:16Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

…into workforce_pipeline

hesamsheikh

Thanks for the new changes @Ol1ver0413. Aside from the comments I previously mentioned, which don't seem resolved yet, and a lack of test cases, I have nothing more to add.

Ol1ver0413 · 2025-11-03T09:56:06Z

Thanks for the new changes @Ol1ver0413. Aside from the comments I previously mentioned, which don't seem resolved yet, and a lack of test cases, I have nothing more to add.

Thanks for the feedback! I’ll review your previous comments again and make sure to address them. I’ll also add the missing test cases soon.

…into workforce_pipeline

hesamsheikh

Thanks for the changes @Ol1ver0413
I added a minor comment, also make sure to fix pre-commit issues.
everything else seems fine to me.

hesamsheikh · 2025-11-10T13:33:33Z

camel/societies/workforce/workforce.py

+
+        # Pipeline building state
+        self._pipeline_builder: Optional[PipelineTaskBuilder] = None
+        self._pipeline_tasks_need_assignment: bool = False


unused variable

a7m-1st

Looks like all of my concerns has been solved. Thanks a lot @Ol1ver0413.
LGTM 👍

Ol1ver0413 · 2025-11-11T01:05:48Z

Thanks @a7m-1st @hesamsheikh a lot for such a long time reviewing! You guys' feedback helped me improve the PR a lot. Love U guys ❤️.

hesamsheikh · 2025-11-11T10:03:52Z

You're welcome @Ol1ver0413
Please fix the pre-commit issues so the checks are passed and this is ready for merge.

hesamsheikh · 2025-11-14T14:02:55Z

Hey @Ol1ver0413
Is this ready for the final review?

fengju0213

@Ol1ver0413 thanks for this amazing pr!

wk_pipline_0.1

2d33864

Ol1ver0413 self-assigned this Sep 18, 2025

wk_pipeline_0.2

9ce04ee

Ol1ver0413 marked this pull request as ready for review September 21, 2025 15:01

Ol1ver0413 requested review from Wendong-Fan and fengju0213 September 21, 2025 15:02

Wendong-Fan requested review from a7m-1st, hesamsheikh and nitpicker55555 September 29, 2025 07:04

Wendong-Fan added the Review Required PR need to be reviewed label Sep 29, 2025

Wendong-Fan added this to Project Camel Sep 29, 2025

Wendong-Fan added this to the Sprint 38 milestone Sep 29, 2025

Wendong-Fan linked an issue Sep 29, 2025 that may be closed by this pull request

[Feature Request] Support more running mode in workforce #1663

Closed

2 tasks

hesamsheikh reviewed Sep 29, 2025

View reviewed changes

a7m-1st reviewed Oct 2, 2025

View reviewed changes

Wendong-Fan added Waiting for Update PR has been reviewed, need to be updated based on review comment and removed Review Required PR need to be reviewed labels Oct 19, 2025

merge and resolve conflict

56527f6

Ol1ver0413 added 3 commits October 20, 2025 21:33

resolve conflict

061aa18

Merge branch 'workforce_pipeline' of https://github.com/camel-ai/camel …

17bfac8

…into workforce_pipeline

wk_pipeline_0.3

9648550

Ol1ver0413 force-pushed the workforce_pipeline branch from a3657ec to 50caa6e Compare November 2, 2025 15:46

test wk_0.6

4c9a65b

Ol1ver0413 requested review from a7m-1st and hesamsheikh November 2, 2025 16:02

del unnecessary content

6945834

hesamsheikh requested changes Nov 3, 2025

View reviewed changes

yuzhe.lin and others added 3 commits November 7, 2025 09:06

Merge branch 'workforce_pipeline' of https://github.com/camel-ai/camel …

c1f58e0

…into workforce_pipeline

wk_pipeline_0.6

9de8537

Merge remote-tracking branch 'origin/master' into workforce_pipeline

56543c4

Wendong-Fan requested a review from hesamsheikh November 10, 2025 05:47

hesamsheikh approved these changes Nov 10, 2025

View reviewed changes

a7m-1st approved these changes Nov 10, 2025

View reviewed changes

del unused variable

64b8e83

Ol1ver0413 added 3 commits November 11, 2025 18:27

fix pre-commit

549cce6

fix pre-commit_0.2

23b0be3

fix pre-commit_0.3

2a499f8

Ol1ver0413 added 2 commits November 15, 2025 11:29

final fix pre-commit issue

3fd9713

exceed the length issue

4305627

This comment was marked as abuse.

Sign in to view

fengju0213 and others added 4 commits November 20, 2025 15:13

Merge branch 'master' into workforce_pipeline

58346bb

Update workforce.py

66a07e2

Update workforce.py

d05e6c3

Update pipeline_workflow_example.py

a571012

fengju0213 approved these changes Nov 20, 2025

View reviewed changes

fengju0213 merged commit 6cbeee0 into master Nov 20, 2025
12 of 13 checks passed

fengju0213 deleted the workforce_pipeline branch November 20, 2025 08:49

[Feat] Support more running mode in workforce #3157

[Feat] Support more running mode in workforce #3157

Uh oh!

Conversation

Ol1ver0413 commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

coderabbitai bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Ol1ver0413 commented Sep 21, 2025

Uh oh!

Ol1ver0413 commented Sep 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ol1ver0413 commented Sep 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ol1ver0413 commented Sep 21, 2025

Uh oh!

Wendong-Fan commented Sep 23, 2025

Uh oh!

hesamsheikh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hesamsheikh Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

Ol1ver0413 Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Ol1ver0413 Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

hesamsheikh Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

Ol1ver0413 Oct 26, 2025

Choose a reason for hiding this comment

Uh oh!

a7m-1st Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ol1ver0413 Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Wendong-Fan commented Oct 19, 2025

Uh oh!

Ol1ver0413 commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app bot commented Oct 20, 2025

Uh oh!

hesamsheikh left a comment

Choose a reason for hiding this comment

Uh oh!

Ol1ver0413 commented Nov 3, 2025

Uh oh!

hesamsheikh left a comment

Choose a reason for hiding this comment

Uh oh!

hesamsheikh Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

a7m-1st left a comment

Choose a reason for hiding this comment

Uh oh!

Ol1ver0413 commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hesamsheikh commented Nov 11, 2025

Uh oh!

hesamsheikh commented Nov 14, 2025

Uh oh!

This comment was marked as abuse.

Ol1ver0413 commented Sep 18, 2025 •

edited

Loading

coderabbitai bot commented Sep 18, 2025 •

edited

Loading

Ol1ver0413 commented Sep 21, 2025 •

edited

Loading

Ol1ver0413 commented Sep 21, 2025 •

edited

Loading

a7m-1st Oct 2, 2025 •

edited

Loading

Ol1ver0413 commented Oct 19, 2025 •

edited

Loading

Ol1ver0413 commented Nov 11, 2025 •

edited

Loading