ci: Add support for max_inflight_responses parameter to prevent unbounded memory growth in ensemble models

[pskiran1]

What does the PR do?

This PR adds testing for a new parameter max_ensemble_inflight_responses.
Added ensemble_backpressure_test.py with custom decoupled producer and slow consumer models to validate the new feature.

Related PR: triton-inference-server/core#455, triton-inference-server/common#141

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

Where should the reviewer start?

Test plan:

• CI Pipeline ID:

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

[Copilot]

Pull Request Overview

This PR adds support for a new max_ensemble_inflight_responses parameter to ensemble models for preventing unbounded memory growth in scenarios with decoupled models and slow consumers.

• Implements backpressure mechanism to limit concurrent responses in ensemble pipelines
• Adds comprehensive test coverage including valid/invalid parameter validation
• Creates new test models for decoupled producer and slow consumer scenarios

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
qa/L0_simple_ensemble/test.sh	Adds backpressure testing logic and invalid parameter validation
qa/L0_simple_ensemble/models/slow_consumer/config.pbtxt	Configures Python backend model with intentional processing delay
qa/L0_simple_ensemble/models/slow_consumer/1/model.py	Implements model that adds 200ms delay per request to simulate slow processing
qa/L0_simple_ensemble/models/ensemble_enabled_max_inflight_responses/config.pbtxt	Ensemble configuration with backpressure parameter set to 4
qa/L0_simple_ensemble/models/ensemble_disabled_max_inflight_responses/config.pbtxt	Baseline ensemble configuration without backpressure parameter
qa/L0_simple_ensemble/models/decoupled_producer/config.pbtxt	Configures decoupled Python model for multiple response generation
qa/L0_simple_ensemble/models/decoupled_producer/1/model.py	Implements decoupled model that produces N responses based on input value
qa/L0_simple_ensemble/ensemble_backpressure_test.py	Comprehensive test suite for backpressure functionality

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[yinggeh]

Is it only happening in decoupled model, or any models with big processing speed difference?

[pskiran1]

Based on my understanding, since the normal model step will have only one response, a slow processing step will automatically block the request. In this case, memory usage will increase at a normal rate (always only one response per step accumulates in memory, unlike in a decoupled model) and may not need any additional backpressure at the step level. To effectively manage overall memory usage, a rate limiter could be sufficient.

[yinggeh]

Same here