Skip to content

Costs tracking all calls #42

@sonic182

Description

@sonic182

Would be nice to have a costs calculation for all calls, for each provider

To return an struct that holds costs information, whenever we set the track_costs: true flag.

This could be setup like this example

# Cost tracking enabled with automatic calculation
@settings %LlmComposer.Settings{
  providers: [
    {LlmComposer.Providers.OpenAI, [
      model: "gpt-4o-mini",
      track_costs: true,  # <- here and next two settings
      input_price_per_million: "0.150",
      output_price_per_million: "0.600"
    ]}
  ]
}

For the providers that doesn't allow retrieving the cost information via API, only OpenRouter has an api that says the prices. OpenAi and Google have the pricing in their "pricing" pages.

When receiving the response, we may include in the LlmResponse struct, a new key with CostInfo proposed struct:

defmodule LlmComposer.CostInfo do
  @moduledoc """
  Struct for tracking costs and token usage of LLM API calls.

  This module provides a standardized way to track both token usage and financial costs
  across different LLM providers. It supports both streaming and non-streaming responses
  and can automatically calculate costs when provided with pricing information.

  ## Fields

  * `:input_tokens` - Number of tokens in the input/prompt
  * `:output_tokens` - Number of tokens in the output/completion
  * `:total_tokens` - Total tokens used (input + output)
  * `:input_cost` - Cost for input tokens (using Decimal for precision)
  * `:output_cost` - Cost for output tokens (using Decimal for precision)  
  * `:total_cost` - Total cost for the request (using Decimal for precision)
  * `:currency` - Currency code (e.g., "USD", "EUR")
  * `:provider_model` - The actual model used by the provider
  * `:provider_name` - The provider that served the request
  * `:input_price_per_million` - Price per million input tokens
  * `:output_price_per_million` - Price per million output tokens
  * `:metadata` - Additional provider-specific cost information

  ## Examples

      # Basic token tracking
      %LlmComposer.CostInfo{
        input_tokens: 150,
        output_tokens: 75,
        total_tokens: 225
      }

      # With pricing information - costs calculated automatically
      %LlmComposer.CostInfo{
        input_tokens: 150_000,
        output_tokens: 75_000,
        input_price_per_million: Decimal.new("1.0"),
        output_price_per_million: Decimal.new("3.0"),
        currency: "USD",
        provider_model: "gpt-4o-mini"
      }
      # Will automatically calculate:
      # input_cost: 0.15, output_cost: 0.225, total_cost: 0.375

      # Direct cost specification (bypasses automatic calculation)
      %LlmComposer.CostInfo{
        input_tokens: 150,
        output_tokens: 75,
        input_cost: Decimal.new("0.0015"),
        output_cost: Decimal.new("0.0030"),
        total_cost: Decimal.new("0.0045"),
        currency: "USD"
      }
  """
end

Streaming responses case

https://github.com/doofinder/llm_composer?tab=readme-ov-file#streaming-responses

For this case, depending of the provider, we may have the tokens info at last chunk or not (openai case). Maybe we need to expose the construct of the CostInfo struct, so implementors does use that after having the input/output tokens information and the related provider, model and prices

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions