Skip to content

IntervalNonlinearProblem fails with Zygote #581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mchitre opened this issue Apr 21, 2025 · 8 comments · May be fixed by SciML/SciMLBase.jl#998
Open

IntervalNonlinearProblem fails with Zygote #581

mchitre opened this issue Apr 21, 2025 · 8 comments · May be fixed by SciML/SciMLBase.jl#998
Assignees
Labels
bug Something isn't working

Comments

@mchitre
Copy link

mchitre commented Apr 21, 2025

Description of the bug 🐞

IntervalNonlinearProblem fails with Zygote as shown in the MWE below.

Presumably, it is hitting the AbstractNonlinearProblem and looking for the u0. @ChrisRackauckas thinks we specialized IntervalNonlinearProblem with forward mode by maybe missed something around reverse mode optimizations there.

Minimal Reproducible Example 👇

julia> using NonlinearSolve, Zygote

julia> f(x, p) = x^2 - p
julia> g(p) = solve(IntervalNonlinearProblem(f, (1.0, 3.0), p[1])).u

julia> g([4.1])
2.024845673131659

julia> Zygote.gradient(g, [4.1])
ERROR: type IntervalNonlinearProblem has no field u0
Stacktrace:
 [1] getproperty
   @ ~/.julia/packages/SciMLBase/c6Noy/src/problems/problem_interface.jl:5 [inlined]
 [2] (::SciMLBaseZygoteExt.var"#solu_adjoint#126"{SciMLBase.NonlinearSolution{}})(Δ::Float64)
   @ SciMLBaseZygoteExt ~/.julia/packages/SciMLBase/c6Noy/ext/SciMLBaseZygoteExt.jl:241
 [3] (::SciMLBaseZygoteExt.var"#239#back#127"{SciMLBaseZygoteExt.var"#solu_adjoint#126"{}})(Δ::Float64)
   @ SciMLBaseZygoteExt ~/.julia/packages/ZygoteRules/CkVIK/src/adjoint.jl:72
 [4] g
   @ ./REPL[11]:1 [inlined]
 [5] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
   @ Zygote ~/.julia/packages/Zygote/HdT4O/src/compiler/interface2.jl:0
 [6] (::Zygote.var"#88#89"{Zygote.Pullback{Tuple{}, Tuple{}}})(Δ::Float64)
   @ Zygote ~/.julia/packages/Zygote/HdT4O/src/compiler/interface.jl:97
 [7] gradient(f::Function, args::Vector{Float64})
   @ Zygote ~/.julia/packages/Zygote/HdT4O/src/compiler/interface.jl:154
 [8] top-level scope
   @ REPL[14]:1

Environment (please complete the following information):

  • Output of using Pkg; Pkg.status()
[8913a72c] NonlinearSolve v4.6.0
[e88e6eb3] Zygote v0.7.6
  • Output of using Pkg; Pkg.status(; mode = PKGMODE_MANIFEST)
[47edcb42] ADTypes v1.14.0
[621f4979] AbstractFFTs v1.5.0
[7d9f7c33] Accessors v0.1.42
[79e6a3ab] Adapt v4.3.0
[4fba245c] ArrayInterface v7.18.0
[4c555306] ArrayLayouts v1.11.1
[62783981] BitTwiddlingConvenienceFunctions v0.1.6
[70df07ce] BracketingNonlinearSolve v1.1.3
[2a0fbf3d] CPUSummary v0.2.6
[082447d4] ChainRules v1.72.3
[d360d2e6] ChainRulesCore v1.25.1
[fb6a15b2] CloseOpenIntervals v0.1.13
[38540f10] CommonSolve v0.2.4
[bbf7d656] CommonSubexpressions v0.3.1
[f70d9fcc] CommonWorldInvalidations v1.0.0
[34da2185] Compat v4.16.0
[a33af91c] CompositionsBase v0.1.2
[2569d6c7] ConcreteStructs v0.2.3
[187b0558] ConstructionBase v1.5.8
[adafc99b] CpuId v0.3.1
[a8cc5b0e] Crayons v4.1.1
[9a962f9c] DataAPI v1.16.0
[864edb3b] DataStructures v0.18.22
[e2d170a0] DataValueInterfaces v1.0.0
[2b5f629d] DiffEqBase v6.167.2
[163ba53b] DiffResults v1.1.0
[b552c78f] DiffRules v1.15.1
[a0c0ee7d] DifferentiationInterface v0.6.52
[ffbed154] DocStringExtensions v0.9.4
[4e289a0a] EnumX v1.0.5
[f151be2c] EnzymeCore v0.8.8
[e2ba6199] ExprTools v0.1.10
[55351af7] ExproniconLite v0.10.14
[7034ab61] FastBroadcast v0.3.5
[9aa1b823] FastClosures v0.3.2
[a4df4552] FastPower v1.1.2
[1a297f60] FillArrays v1.13.0
[6a86dc24] FiniteDiff v2.27.0
[f6369f11] ForwardDiff v1.0.1
[069b7b12] FunctionWrappers v1.1.3
[77dc65aa] FunctionWrappersWrappers v0.1.3
[46192b85] GPUArraysCore v0.2.0
[7869d1d1] IRTools v0.4.14
[615f187c] IfElse v0.1.1
[3587e190] InverseFunctions v0.1.17
[92d709cd] IrrationalConstants v0.2.4
[82899510] IteratorInterfaceExtensions v1.0.0
[692b3bcd] JLLWrappers v1.7.0
[ae98c720] Jieko v0.2.1
[ba0b0d4f] Krylov v0.10.0
[b964fa9f] LaTeXStrings v1.4.0
[10f19ff3] LayoutPointers v0.1.17
[5078a376] LazyArrays v2.6.1
[87fe0de2] LineSearch v0.1.4
[7ed4a6bd] LinearSolve v3.8.0
[2ab3a3ac] LogExpFunctions v0.3.29
[1914dd2f] MacroTools v0.5.16
[d125e4d3] ManualMemory v0.1.8
[bb5d69b7] MaybeInplace v0.1.4
[2e0e35c7] Moshi v0.3.5
[46d2c3a1] MuladdMacro v0.2.4
[77ba4419] NaNMath v1.1.3
[8913a72c] NonlinearSolve v4.6.0
[be0214bd] NonlinearSolveBase v1.5.3
[5959db7a] NonlinearSolveFirstOrder v1.3.1
[9a2c21bd] NonlinearSolveQuasiNewton v1.2.1
[26075421] NonlinearSolveSpectralMethods v1.1.1
[bac558e1] OrderedCollections v1.8.0
[d96e819e] Parameters v0.12.3
[f517fe37] Polyester v0.7.16
[1d0040c9] PolyesterWeave v0.2.2
⌅ [aea7be01] PrecompileTools v1.2.1
[21216c6a] Preferences v1.4.3
[08abe8d2] PrettyTables v2.4.0
[c1ae055f] RealDot v0.1.0
[3cdcf5f2] RecipesBase v1.3.4
[731186ca] RecursiveArrayTools v3.31.2
[189a3867] Reexport v1.2.2
[ae029012] Requires v1.3.1
[7e49a35a] RuntimeGeneratedFunctions v0.5.13
[94e857df] SIMDTypes v0.1.0
[0bca4576] SciMLBase v2.84.0
[19f34311] SciMLJacobianOperators v0.1.2
[c0aeaf25] SciMLOperators v0.3.13
[53ae85a6] SciMLStructures v1.7.0
[efcf1570] Setfield v1.1.2
[727e6d20] SimpleNonlinearSolve v2.2.2
[dc90abb0] SparseInverseSubset v0.1.2
[0a514795] SparseMatrixColorings v0.4.18
[276daf66] SpecialFunctions v2.5.1
[aedffcd0] Static v1.2.0
[0d7ed370] StaticArrayInterface v1.8.0
[1e83bf80] StaticArraysCore v1.4.3
[10745b16] Statistics v1.11.1
[7792a7ef] StrideArraysCore v0.5.7
[892a3eda] StringManipulation v0.4.1
[09ab397b] StructArrays v0.7.1
[2efcf032] SymbolicIndexingInterface v0.3.39
[3783bdb8] TableTraits v1.0.1
[bd369af6] Tables v1.12.0
[8290d209] ThreadingUtilities v0.5.3
[a759f4b9] TimerOutputs v0.5.28
[781d530d] TruncatedStacktraces v1.4.0
[3a884ed6] UnPack v1.0.2
[e88e6eb3] Zygote v0.7.6
[700de1a5] ZygoteRules v0.2.7
[1d5cc7b8] IntelOpenMP_jll v2025.0.4+0
[856f044c] MKL_jll v2025.0.1+1
[efe28fd5] OpenSpecFun_jll v0.5.6+0
[1317d2d5] oneTBB_jll v2022.0.0+0
[0dad84c5] ArgTools v1.1.2
[56f22d72] Artifacts v1.11.0
[2a0f44e3] Base64 v1.11.0
[ade2ca70] Dates v1.11.0
[8ba89e20] Distributed v1.11.0
[f43a241f] Downloads v1.6.0
[7b1f6079] FileWatching v1.11.0
[9fa8497b] Future v1.11.0
[b77e0a4c] InteractiveUtils v1.11.0
[4af54fe1] LazyArtifacts v1.11.0
[b27032c2] LibCURL v0.6.4
[76f85450] LibGit2 v1.11.0
[8f399da3] Libdl v1.11.0
[37e2e46d] LinearAlgebra v1.11.0
[56ddb016] Logging v1.11.0
[d6f4376e] Markdown v1.11.0
[ca575930] NetworkOptions v1.2.0
[44cfe95a] Pkg v1.11.0
[de0858da] Printf v1.11.0
[9a3f8284] Random v1.11.0
[ea8e919c] SHA v0.7.0
[9e88b42a] Serialization v1.11.0
[6462fe0b] Sockets v1.11.0
[2f01184e] SparseArrays v1.11.0
[4607b0f0] SuiteSparse
[fa267f1f] TOML v1.0.3
[a4e569a6] Tar v1.10.0
[cf7118a7] UUIDs v1.11.0
[4ec0a83e] Unicode v1.11.0
[e66e0078] CompilerSupportLibraries_jll v1.1.1+0
[deac9b47] LibCURL_jll v8.6.0+0
[e37daf67] LibGit2_jll v1.7.2+0
[29816b5a] LibSSH2_jll v1.11.0+1
[c8ffd9c3] MbedTLS_jll v2.28.6+0
[14a3606d] MozillaCACerts_jll v2023.12.12
[4536629a] OpenBLAS_jll v0.3.27+1
[05823500] OpenLibm_jll v0.8.5+0
[bea87d4a] SuiteSparse_jll v7.7.0+0
[83775a58] Zlib_jll v1.2.13+1
[8e850b90] libblastrampoline_jll v5.11.0+0
[8e850ede] nghttp2_jll v1.59.0+0
[3f19e933] p7zip_jll v17.4.0+2

  • Output of versioninfo()
Julia Version 1.11.5
Commit 760b2e5b739 (2025-04-14 06:53 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m1)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
@mchitre mchitre added the bug Something isn't working label Apr 21, 2025
@jClugstor jClugstor linked a pull request Apr 25, 2025 that will close this issue
5 tasks
@jClugstor
Copy link
Member

jClugstor commented May 13, 2025

I now get a completely different error when trying to run this.

ERROR: UndefVarError: `j` not defined in `Zygote`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
  [1] macro expansion
    @ ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:0 [inlined]
  [2] (::Zygote.Pullback{Tuple{Core.IntrinsicFunction, UInt64, UInt64}, Tuple{Core.IntrinsicFunction}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:110
  [3] <<
    @ ./int.jl:529 [inlined]
  [4] (::Zygote.Pullback{Tuple{typeof(<<), UInt64, UInt64}, Tuple{Zygote.Pullback{Tuple{}, Tuple{}}}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:0
  [5] >>
    @ ./int.jl:534 [inlined]
  [6] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:0
  [7] exponent
    @ ./math.jl:950 [inlined]
  [8] (::Zygote.Pullback{Tuple{typeof(exponent), Float64}, Any})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:0
  [9] #solve#7
    @ ~/.julia/packages/BracketingNonlinearSolve/iO4RJ/src/itp.jl:99 [inlined]
 [10] (::Zygote.Pullback{…})(Δ::@NamedTuple{})
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:0
 [11] #295
    @ ~/.julia/packages/Zygote/1GK3J/src/lib/lib.jl:205 [inlined]
 [12] #2169#back
    @ ~/.julia/packages/ZygoteRules/CkVIK/src/adjoint.jl:72 [inlined]
 [13] solve
    @ ~/.julia/packages/BracketingNonlinearSolve/iO4RJ/src/itp.jl:59 [inlined]
 [14] (::Zygote.Pullback{…})(Δ::@NamedTuple{})
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:0
 [15] #solve#10
    @ ~/.julia/packages/BracketingNonlinearSolve/iO4RJ/src/BracketingNonlinearSolve.jl:25 [inlined]
 [16] (::Zygote.Pullback{…})(Δ::@NamedTuple{})
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:0
 [17] solve
    @ ~/.julia/packages/BracketingNonlinearSolve/iO4RJ/src/BracketingNonlinearSolve.jl:24 [inlined]
 [18] (::Zygote.Pullback{…})(Δ::@NamedTuple{})
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:0
 [19] g
    @ ~/Documents/Work/dev/ODETesting/Adjoints/OtherAdjoints/MWE.jl:3 [inlined]
 [20] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface2.jl:0
 [21] (::Zygote.var"#78#79"{Zygote.Pullback{Tuple{}, Tuple{}}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface.jl:91
 [22] gradient(f::Function, args::Vector{Float64})
    @ Zygote ~/.julia/packages/Zygote/1GK3J/src/compiler/interface.jl:148
 [23] top-level scope
    @ ~/Documents/Work/dev/ODETesting/Adjoints/OtherAdjoints/MWE.jl:5
Some type information was truncated. Use `show(err)` to see complete types.
(OtherAdjoints) pkg> st
Status `~/Documents/Work/dev/ODETesting/Adjoints/OtherAdjoints/Project.toml`
  [8913a72c] NonlinearSolve v4.8.0
  [e88e6eb3] Zygote v0.7.7

@DhairyaLGandhi did something fix the constructor adjoint issues already? Also, any idea on what this Zygote error could be from?

@jClugstor
Copy link
Member

Looks like this is still failing because of this Zygote issue: FluxML/Zygote.jl#1573

@ChrisRackauckas
Copy link
Member

No, that means it's not hitting an adjoint overload. That's not a Zygote issue.

@jClugstor
Copy link
Member

So it shouldn't be trying to take the adjoint of exponent in the first place, right? I guess we're missing an adjoint rule for solve(prob::IntervalNonlinearProblem, ... or something like that?

@oscardssmith
Copy link
Member

I guess we're missing an adjoint rule for solve(prob::IntervalNonlinearProblem, ...

correct

@ChrisRackauckas
Copy link
Member

That is inside of ITP. Seeing that error means you're differentiating the algorithm not using a custom adjoint.

For other nonlinear problems, it catches here https://github.com/SciML/DiffEqBase.jl/blob/master/src/solve.jl#L1173 by https://github.com/SciML/DiffEqBase.jl/blob/master/ext/DiffEqBaseChainRulesCoreExt.jl#L22, because of the weird quirk that

julia> NonlinearProblem <: SciMLBase.AbstractDEProblem
true

But the bracketing methods directly catch solve https://github.com/SciML/NonlinearSolve.jl/blob/master/lib/BracketingNonlinearSolve/src/itp.jl#L59 so that isn't hit.

So BracketingNonlinearSolve needs to refactor to have a lowering stage to make p an explicit arg like https://github.com/SciML/DiffEqBase.jl/blob/master/src/solve.jl#L1136-L1175 before calling the main algorithm, then you need a chain rule to catch that. Then you replace its derivative with the implicit nonlinear solve rule.

For BacketingNonlinearSolve,f(u,p)=0 means df/du du/dp + df/dp = 0, so du/dp = - df/du \ df/dp. This is exactly SteadyStateAdjoint. So you want to do the same thing as the DiffEqBase thing, which is punt to SteadyStateAdjoint, and then make sure https://github.com/SciML/SciMLSensitivity.jl/blob/master/src/steadystate_adjoint.jl#L93-L102 works with scalar returns from f.

@jClugstor
Copy link
Member

Ok cool, I think I'm beginning to understand.

Essentially I need to make it so that there's a solve(prob::IntervalNonlinearProblem, ...) that has the p lowering stage, basically make it call solve_up. And I need to make it so that the solves in BracketingNonlinearSolve are all something like __solve and are called by solve_up.

I think once it's using solve_up instead there's actually not much more to change. IntervalNonlinearProblem <: SciMLBase.AbstractDEProblem so I believe that will be caught by the existing chain rule.

And the punting to SteadyStateAdjoint you talked about happens here https://github.com/SciML/SciMLSensitivity.jl/blob/14fb6c0fec50adcfb80a230785734d61133432ce/src/concrete_solve.jl#L284-L308 right? So that should be taken care of since IntervalNonlinearProblem <: AbstractNonlinearProblem

Nevermind, InvervalNonlinearProblem doesn't have a u0 so that messes things up. Anyways I think I know what to do now, thanks.

@ChrisRackauckas
Copy link
Member

You cannot differentiate w.r.t. u0 anyways, since the derivative of the solution of newton's method is independent of the starting guess (in general, if you're in the convergence region), and thus it's always zero. So you can use the same code path, just drop that u0 derivative by, instead of making it zero, just don't return it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants