Proposal: HTTP Client Trait for async-openai

[timvw]

Problem Statement

Currently, async-openai is tightly coupled to reqwest::Client, which prevents users from:

• Adding middleware for tracing, logging, or retry logic
• Using alternative HTTP clients
• Mocking HTTP calls for testing
• Implementing custom request/response handling

Proposed Solution

Introduce an HttpClient trait that abstracts HTTP operations, allowing any compatible client to be used.

The Trait

#[async_trait]
pub trait HttpClient: Send + Sync {
    async fn request(
        &self,
        method: Method,
        url: Url,
        headers: HashMap<String, String>,
        body: Option<Bytes>,
    ) -> Result<HttpResponse, HttpError>;
}

Implementation Changes

1. Add trait definition in a new module (e.g., http_client.rs)
2. Implement for reqwest::Client (default behavior)
3. **Implement for ClientWithMiddleware** (middleware support)
4. Update Client struct to use Arc<dyn HttpClient>
5. Add factory methods:
   • Client::new() - uses default reqwest::Client
   • Client::with_http_client() - accepts any HttpClient

Benefits

1. Middleware Support

Users can add middleware for:

• OpenTelemetry tracing - Automatic distributed tracing
• Logging - Request/response debugging
• Retry logic - Automatic retry with backoff
• Rate limiting - Respect API limits
• Metrics - Track API usage

Example:

let http_client = ClientBuilder::new(reqwest::Client::new())
    .with(TracingMiddleware::default())
    .with(RetryMiddleware::default())
    .build();

let client = Client::with_http_client(http_client, config);
// All API calls are now automatically traced and retried!

2. Testing

Enable proper unit testing without network calls:

let mock_client = MockHttpClient::new(vec![mock_response]);
let client = Client::with_http_client(mock_client, config);
// Test with deterministic responses

3. Alternative HTTP Clients

Support for:

• surf
• isahc
• ureq
• Custom implementations

4. Backward Compatibility

• Existing code continues to work unchanged
• Default behavior remains the same
• New features are opt-in

Migration Path

1. Phase 1: Add trait as optional feature

   [features]
   http-client-trait = []

2. Phase 2: Make it default but keep old API

   impl Client {
       #[deprecated]
       pub fn with_http_client(client: reqwest::Client) -> Self { ... }
       
       pub fn with_http_client_trait(client: impl HttpClient) -> Self { ... }
   }

3. Phase 3: Full migration in next major version

Similar Approaches in Other Libraries

• AWS SDK: Uses SharedHttpClient trait
• Google Cloud SDK: Uses HttpClient trait
• Azure SDK: Uses Pipeline with policies (similar to middleware)

Conclusion

This change would make async-openai more flexible, testable, and production-ready while maintaining backward compatibility. It follows established patterns in other major SDK libraries and enables important use cases like observability and testing.

Next Steps

1. Gather feedback on this proposal
2. Create a proof-of-concept PR
3. Add tests and documentation
4. Release as optional feature first

[timvw]

POC PR: #422

[timvw]

Having thought a bit more about it, having a HttpClient (and middleware) is probably not the right level in the application stack to implement tracing (even though I have a working prototype here: https://github.com/timvw/reqwest-openai-tracing)..