D0CRT: `clf_x` is a Lasso model. How to replace it with a sklearn compatible estimator?

[lionelkusch]

Why clf_x necessarily be a Lasso model, given that estimator can be any sklearn compatible estimator ?

Originally posted by @bthirion in #220 (comment)

[lionelkusch]

The usage of more abstract models can create difficulty for the computation of sigma2:

hidimstat/src/hidimstat/distilled_conditional_randomization_test.py

Line 583 in 3ecce71

sigma2 = np.linalg.norm(X_residual) ** 2 / n_samples + alpha * np.linalg.norm(

The computation of sigma2 is based on alpha and the coefficient of lasso. This is very restrictive for compatibility with the Scikitlearn Model.

[bthirion]

Well, if the model does not overfit too much, the naive variance estimator could be sufficient.
Otherwise,a safe solution is to perform cross-fitting: train the model on 80% of the data and compute the variance on the remaining samples. This is valid without restriction, but more costly.