Robust loss in Levenberg Marquardt?

[Linusnie]

The current LM implementation works with loss functions on the form sum_i f_i(x)^2, are there any plans on also handling robust residuals sum_i rho(f_i(x)^2)? As is usually required in 3D reconstruction type problems

As an example this is available in the ceres nonlinear least squares solver http://ceres-solver.org/nnls_modeling.html#_CPPv4N5ceres12LossFunctionE via the Triggs method. Though I can't say I'm familiar with the details

[vroulet]

Hello @Linusnie,
I would be interested in making a regularized generalized Gauss-Newton solver (I used it in some projects) that would handle problems of the form min_x h(f(x)) for h twice differentiable convex and f differentiable nonlinear. See e.g. https://pdfs.semanticscholar.org/3815/ec3e4b46f5909aee97112d111dbf794ba7da.pdf for more context about generalized Gauss-Newton.
Would that be enough for your purposes?

[Linusnie]

Hi @vroulet, for robust least squares rho is typically nonconvex (eg cauchy loss or truncated L2) so I don't think that method would apply unfortunately

[vroulet]

The convexity of the outer function h is simply required to ensure having descent directions for any regularizations. Truly, the algorithm can work as long as one can ensure having access to descent directions by increasing the regularization if necessary. Triggs trick appears not to be applicable if one has an indefinite positive matrix (that is we cannot ensure a descent direction). I think a solver that could handle any composite optimization problem of the form min_x h(f(x)) could cover more cases than min_x rho(f(x)**2) which is a subcase and could be treated by tweaking the choice of quadratic bound on h (which seems to be the main part of the variants for robust nonlinear least-squares (https://openaccess.thecvf.com/content_ECCV_2018/papers/Christopher_Zach_Descending_lifting_or_ECCV_2018_paper.pdf)).
Anyway, there's a long way to go for that, probably not in the next monts, but I'll keep you posted if that changes.

[Linusnie]

Ah I see, sounds like it could work then, I'll keep an eye out :) Either way thanks for the references, very interesting