Open
Description
Description
We want to add a CI test, that collects metrics to understand if performance changes between releases.
First step:
- Add regression tests for session based transformer model https://github.com/NVIDIA-Merlin/models/blob/main/examples/usecases/transformers-next-item-prediction.ipynb
@radekosmulski shared ideas on it here: https://nvidia.slack.com/archives/CVBDJUPEZ/p1679617586380369
We will collect:
- Runtime NVTabular
- Runtime Training Model with TensorFlow
- NDCG Performance of Session-based model
I close other tickets related to it:
- Figure out a consistent way of capturing results from the new examples in ASV Merlin#223 : I think that is solved based on sample PR)
- [Task] Add criteo examples in GSW runtime to collect different metrices Merlin#233 : We changed priority to session-based models