Skip to content

Margin Of Error (MOE) stopping #244

@AlonKellner-RedHat

Description

@AlonKellner-RedHat

Given that #238 will land and that #243 will turn into a PR and land as well, it would be incredible to have target metric MOE stopping.
Basically this feature is 2 new options:

  • --target-metric - the metric to use as the target for MOE stopping, of the form ["all", "ttft", "ttft-p90", "itl-median"....] (default: "all")
  • --target-moe - the allowed error in the metric measurement (in percentages) as reflected in the MOE of that metric.
    In a benchmark run, when the MOE value of the target metric reaches the target MOE - the benchmark will end with the termination reason "target_moe_reached".

This feature will include a simple ETA value.
Essentially, calculate the amount of samples required to reach the MOE (assuming the average sample weight consists), then use the RPS to extrapolate time.

A feature like this would make guidellm much faster and reliable for most use-cases.

Metadata

Metadata

Assignees

No one assigned

    Labels

    internalfiled by core contributor or associate

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions