Kernel Parameter Constraints in BoTorch

[jduerholt]

Hi all,

I saw by chance that when you use constraints on kernel parameters in botorch, you always set transform=None, and overwrite the default from gpytorch. What is the reason for this?

Here is an example:

https://github.com/pytorch/botorch/blob/72580174697380ba24ed3156761bfe024b421473/botorch/models/utils/gpytorch_modules.py#L127

Edit 1:

It seems that it is not everywhere like this, in the mixed GP in the categorical kernel it is different, there the default transform is used:

https://github.com/pytorch/botorch/blob/72580174697380ba24ed3156761bfe024b421473/botorch/models/gp_regression_mixed.py#L117

Is this on purpose?

Best,

Johannes

[Balandat]

Hi @jduerholt, the main reason for setting transform=None is that we typically use L-BFGS-B to optimize the MLL of the model, and we can explicitly include those constraints in the optimization rather than relying on the default transformations to the real line used in gpytorch. We have found that to work better in most cases. The parsing logic for this lives here: https://github.com/pytorch/botorch/blob/main/botorch/optim/utils/model_utils.py#L99-L101

One downside is that if you simply used an unconstrained optimizer (such as Adam) to optimize BoTorch model's hyperparameters you'd have to make sure to handle the constraint manually. There is some related discussion here: #2542

It seems that it is not everywhere like this, in the mixed GP in the categorical kernel it is different, there the default transform is used. Is this on purpose?

Hmm good catch - I am not sure, but I don't think this is on purpose. @saitcakmak, @dme65 do you recall if there was a reason to use the transform or was this an oversight?

[jduerholt]

Thanks, this fully reasonates with my expectations! Let me know regarding the categorical kernel, I then file a PR for this ;)

[Balandat]

Yeah that would be neat - but before landing that change I'd be curious to understand what the difference in the optimization quality / model fits is. I guess we could augment the code from the unit test with some more interesting data set to evaluate this?

[jduerholt]

So, you recommend to run some quick tests to see what is the effect of the transform on the fit? I try to do this next week.

[Balandat]

Yes, exactly!

[jduerholt]

I try to investiage this in the next week.