Skip to content

Conversation

yangw-dev
Copy link
Contributor

@yangw-dev yangw-dev commented Sep 6, 2025

Stack from ghstack (oldest at bottom):

create a new query for compiler time series fetching api.
two major things here:

  1. we do not use output from extra_Info but use the primary keys dtype, arch and device. the mapping might be different, but this is what we plan to do any way/
  2. we fetch list of commits based on timestamp, instead of fetch data from timestamp, this avoid partial data fetch situation that can creating false alert for regression.

New arch values in table

list below is nductor distinct device and arch for Pass 3 month from clickhouse oss_ci_benchmark_torchinductor table

SELECT DISTINCT device,arch
FROM oss_ci_benchmark_torchinductor
WHERE timestamp >= toUnixTimestamp(now() - INTERVAL 3 MONTH)
device arch mapping
cpu x86_64 cpu(x86_64)
cpu aarch64 cpu(aarch64)
cuda NVIDIA A10G ?
cuda NVIDIA A100-SXM4-40GB cuda(a100)
cuda NVIDIA B200 cuda(b200)
cuda NVIDIA H100 80GB HBM3 cuda(h100)
cpu arm ?
mps arm mps
rocm AMD Instinct Mi325X VF rocm(mi300x) [new instance to mark as mi300x)
cuda NVIDIA H100 cuda(h100)
cuda x86_64 ?
rocm AMD Instinct MI300X rocm(mi300x)

@huydhn I listed question mark for unknown mapping to arch values from output, like those:
image

[ghstack-poisoned]
Copy link

vercel bot commented Sep 6, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Preview Updated (UTC)
torchci Ignored Ignored Preview Sep 9, 2025 3:31am

yangw-dev added a commit that referenced this pull request Sep 6, 2025
ghstack-source-id: f27d487
Pull-Request: #7112
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 6, 2025
This was referenced Sep 6, 2025
@yangw-dev yangw-dev requested a review from huydhn September 6, 2025 04:24
[ghstack-poisoned]
yangw-dev added a commit that referenced this pull request Sep 6, 2025
ghstack-source-id: 76b63e1
Pull-Request: #7112
[ghstack-poisoned]
yangw-dev added a commit that referenced this pull request Sep 6, 2025
ghstack-source-id: 86b8261
Pull-Request: #7112
yangw-dev added a commit that referenced this pull request Sep 6, 2025
ghstack-source-id: 86b8261
Pull-Request: #7112
[ghstack-poisoned]
yangw-dev added a commit that referenced this pull request Sep 8, 2025
ghstack-source-id: d555092
Pull-Request: #7112
[ghstack-poisoned]
yangw-dev added a commit that referenced this pull request Sep 8, 2025
ghstack-source-id: 4192ac0
Pull-Request: #7112
yangw-dev added a commit that referenced this pull request Sep 9, 2025
…7092)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at
bottom):
* #7112
* #7096
* #7095
* #7094
* __->__ #7092

setup base for notification pipeline.
add config to provide guidance for each pipepline how to process data
and generate benchmark regression report
[ghstack-poisoned]
yangw-dev added a commit that referenced this pull request Sep 9, 2025
ghstack-source-id: 3e17cf6
Pull-Request: #7112
yangw-dev added a commit that referenced this pull request Sep 9, 2025
Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at
bottom):
* #7112
* #7096
* #7095
* __->__ #7094

main class to detect regression
1. use baseline data to generate baseline for regression detection,
currently support (max,min, latest, earliest) within the baseline data
window
2. decide wether the target data list under the threshold for regression
detection based on the baseline value.
3. if detected more than 2 continuous dp from current timestamp, mark as
regression
yangw-dev added a commit that referenced this pull request Sep 9, 2025
ghstack-source-id: 3e17cf6
Pull-Request: #7112
yangw-dev added a commit that referenced this pull request Sep 9, 2025
ghstack-source-id: 3e17cf6
Pull-Request: #7112
[ghstack-poisoned]
yangw-dev added a commit that referenced this pull request Sep 9, 2025
ghstack-source-id: b158378
Pull-Request: #7112
yangw-dev added a commit that referenced this pull request Sep 10, 2025
Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at
bottom):
* #7112
* #7096
* __->__ #7095

add logics to write the report to db table
yangw-dev added a commit that referenced this pull request Sep 10, 2025
Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at
bottom):
* #7112
* __->__ #7096
* #7095


# Add github notification settings
currently we only create github comment if and only a regression status
is detected.
later we can add if we want to add suspicious too,

## Prerequest
currently user must:
1. create a github issue first, and put it in the Policy section to make
this work
2. Each issue should be associated with a butterfly rule to link to
internal workplace/ oncall emails


## later improvement
as you see, currently if a regression does not resolved, it will send
notification to github everyday.
since we have those report in db, we can later do exponential
notification based on previous report status
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant