You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
At present, the candle binding layer only supports single text inference through a single model call. When processing multiple texts through the batch classification API (/API/v1/classification/batch ), the system must loop through each text and make separate inference calls to the underlying model. This method does not fully utilize the batch processing capability of the model.
Describe the solution you'd like
Enable batch inference with candle binding layer support, allowing multiple texts to be processed in a single model forward pass.