linalg.norm has mismatched convention for complex numbers #666
Description
Hi all, thanks for such a still useful project.
The convention adopted in autograd for complex numbers is generally that parameters can be updated in the direction of conj(g) to minimize a function. linalg.norm follows the opposite convention, preventing it from being used in code that combines it with other functions.
This was mentioned in #393 alongside some potential fixes.
Minimal reproducer:
import numpy as np
import autograd.numpy as anp
from autograd import value_and_grad
rng = np.random.default_rng(42)
x = rng.normal(size=10) + 1j * rng.normal(10)
y = rng.normal(size=10) + 1j * rng.normal(10)
def foo(x, y):
# simple loss function
return anp.linalg.norm(x - y)
def foo2(x, y):
# alt impl
d = x - y
return anp.sum(anp.abs(d)**2)**0.5
# use foo2 for correct behavior
foo_vag = value_and_grad(foo)
for _ in range(10):
v, g = foo_vag(x, y)
# or remove conj here to make foo work
x -= 0.05 * np.conj(g)
# value should initially decrease
print(v)
# 5.761837589211455
# 5.785341294273737
# 5.8095148223718915
# 5.834344369254469
# 5.8598161062729055
# 5.885916205539433
# 5.912630863722038
# 5.939946324450408
# 5.967848899318491
# 5.996324987479975autograd v1.7.0.