Skip to content

Feature Enhancement: Batch Inference Support in candle-binding #32

@OneZero-Y

Description

@OneZero-Y

Is your feature request related to a problem? Please describe.
At present, the candle binding layer only supports single text inference through a single model call. When processing multiple texts through the batch classification API (/API/v1/classification/batch ), the system must loop through each text and make separate inference calls to the underlying model. This method does not fully utilize the batch processing capability of the model.

Describe the solution you'd like
Enable batch inference with candle binding layer support, allowing multiple texts to be processed in a single model forward pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions