Skip to content

[Feature] Use Batch Completion for LiteLLM #7998

Open
@junkyul

Description

@junkyul

What feature would you like to see?

I am using DSPY version 2.6.5.
I set up LM with the "other providers" approach, as shown in the getting started material.

One issue that I encounter is that there is a limit of the number of requests that can make per minute, like 1000 requests per minute. When I run optimizers or evalaution with multiple threads or even 1 thread, it easily goes over that limit and the whole program run stops due to the connection error.

Is there any way to resolve this issue?
For example, by sending a batch completion call or handling the number of requests explicitly?

If there's a way to resolve this issue already, it would be great to know.

Would you like to contribute?

  • Yes, I'd like to help implement this.
  • No, I just want to request it.

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions