[Feature] Use Batch Completion for LiteLLM

[junkyul]

What feature would you like to see?

I am using DSPY version 2.6.5.
I set up LM with the "other providers" approach, as shown in the getting started material.

One issue that I encounter is that there is a limit of the number of requests that can make per minute, like 1000 requests per minute. When I run optimizers or evalaution with multiple threads or even 1 thread, it easily goes over that limit and the whole program run stops due to the connection error.

Is there any way to resolve this issue?
For example, by sending a batch completion call or handling the number of requests explicitly?

If there's a way to resolve this issue already, it would be great to know.

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

No response

[christopher-hoernle]

Batch completion would also be great to reduce costs.

[okhat]

Don’t batch APIs promise a turnaround time of up to 24 hours? How could that make sense for any multi-stage program or optimizer?

I think a better way to handle rate limits is indeed important. But please note you can start by reducing the number of threads you set.

[christopher-hoernle]

@okhat For optimizers I agree! For multi-stage programs that operate periodically + async it's very useful. Imagine classification tasks that require multiple steps that need to be carried out daily (e.g. in healthcare).

[ryanh-ai]

If we use LiteLLM router configurations, you can use the rate limiting feature it offers at the LiteLLM level. I had started working on this, but there was seperate PR for adding Router to LiteLLM usage in DSPy - looks like that caused other issues and was rolled back.