Replacing features in a dataset with offline predictions (and how to use the existing transforms)

[5-Jeremy]

I have a rather strange need here, but please bear with me.

I have a separate (not neural network-based) prediction algorithm that I am using to generate predictions for one of the continuous variables that is being fed into the main forecasting model (A TFT). I have a training dataset consisting of many time series, and I am trying to replace the ground-truth values for that continuous variable with the predicted values which are computed offline. I think that it is necessary to do this after the TimeSeriesDataset has been created since the time series of predicted values varies based on the timestep at which prediction begins. This is because in practice the other prediction algorithm will be updating its predictions in real time, so for example the prediction that I have for time index 5 when the current time index is 1 will differ from the prediction that I have when the current time index is 2. Because of all this, my current approach is to modify the _collate_fn of the TimeSeriesDataset so I can replace the appropriate column of the x_cont tensor for each element in the batch with the predicted values. The problem I am encountering is that since the transforms have already been applied, it is hard to tell what column of x_cont corresponds to what variable. How can I reverse the transformation in order to tell which is which? I will also need to apply the appropriate transformation onto the predicted values before I insert them.

Note that I am not allowed to provide the exact code I am using, but I can try to produce a very general reproduction of it if that would help.

[5-Jeremy]

I figured out how to get the transformation and apply it to the data.

[jobs-git]

care to explain how for future readers

[5-Jeremy]

All I needed to do was make the _scalers variable of the TimeSeriesDataset available within the _collate_fn that the dataset uses. Then I could apply the transformation by extracting the appropriate scalar for a feature using _scalers[<feature_name>] and calling scaler.transform(data). This allowed me to get transformed values that I could replace the original values with.