-
Notifications
You must be signed in to change notification settings - Fork 149
Add SmallTag
type for more compact Dual
types
#748
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
SmallTag
type for more compute Dual
typesSmallTag
type for more compact Dual
types
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #748 +/- ##
==========================================
- Coverage 89.57% 86.92% -2.66%
==========================================
Files 11 10 -1
Lines 969 1025 +56
==========================================
+ Hits 868 891 +23
- Misses 101 134 +33 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
We could also revive #572, closed it was thought that SciML printing problems were solved in Base. Can you comment on the comparison? |
Base has type-folding now that helps many stack traces, but you still sometimes need a
The main difference is that debugging perturbation issues without the actual function / array types can be very confusing, so it seems useful to keep the original functionality around. Other than that, the implementation should be mostly equivalent (the hash is based in many cases on the Also it's worth mentioning that |
This is an alternative to `Tag` that provides largely the same functionality, but carries around only the hash of the function / array types instead of the full types themselves. This can make these types much less bulky to print and easier to visually scan for.
This provides a convenient interface to ask for a SmallTag.
Latest patch release seems to have tightened this up.
This old version of Julia doesn't have call-site `@inline` / `@noinline` so version-guard against this trick for now.
98b57bb
to
d88bf58
Compare
Chatted on this with @oscardssmith and @topolarity . This is better than using the objectid because it is consistent with respect to precompilation, i.e. in a new session the objectid can change depending on the order that you evaluate functions, while this is consistent. That plus on v1.11 this is const eval-able, and thus using this form of a short tag is type-stable and makes it so you will precompile the autodiff calls if you autodiff w.r.t. the same function. With those two properties though, I would be highly in favor of defaulting to this on v1.11+, since it won't hurt performance or precompilation, and I cannot see a scenario where a normal user would favor the tags based on function types. It would be odd to have huge multi-screen types on v1.10 and then good stack traces on v1.11, but IMO that's not a reason to make it wait longer. I'd like to hear @KristofferC's opinion here. But either way, I think we'd want to set this up as well in ADTypes and DifferentiationInterface @gdalle |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
end | ||
|
||
@generated function tagcount(::Type{SmallTag{H}}) where {H} | ||
:($(Threads.atomic_add!(TAGCOUNT, UInt(1)))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems related to #724, maybe someone can review that PR first?
if kind === :default | ||
return Tag(f, X) | ||
elseif kind === :small | ||
return SmallTag(f, X) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is type-unstable, can we do something about it? Perhaps pass the symbol as a Val
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One of my objections to this PR is that I don't much like adding keyword tag::Union{Symbol,Nothing} = :default
to everything. (1) it's messy, and I'm not sure how often anyone needs to pass this, (2) accepting a symbol from a special list of 2 seems like an odd interface. I'm not sure Val(:default)
is much better. (3) it's called tag
but isn't the tag, doesn't accept aTag
.
Maybe a better interface would be to use Preferences.jl, as for NaN-safe mode?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is called with @inline
to avoid that instability: https://github.com/JuliaDiff/ForwardDiff.jl/pull/748/files/d88bf58768e127e9a97a642deadbcd45163662e1#diff-aeb036d3acc3ca0b52368fdcc3dbc8bca465b98bc269a1a5b4668b1dcb2e2f7eR145
I'll move it to the definition so that there's no ambiguity
# SmallTag is similar to a Tag, but carries just a small UInt64 hash, instead | ||
# of the full type, which makes stacktraces / types easier to read while still | ||
# providing good resilience to perturbation confusion. | ||
struct SmallTag{H} | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be an actual docstring
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And it should be marked public or exported, but so should other symbols (#744)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we sure this needs a new type at all? Could we just extend Dual
's first type parameter from Union{Nothing, Tag}
to perhaps Union{Nothing, Tag, Integer}
?
julia> ForwardDiff.Dual(pi/2, 1)
Dual{Nothing}(1.5707963267948966,1.0)
julia> ForwardDiff.derivative(x -> @show(x), pi/2)
x = Dual{ForwardDiff.Tag{var"#37#38", Float64}}(1.5707963267948966,1.0)
1.0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it does need a new type, the obvious name is HashTag
.
I'm not really convinced that #724 is a proper fix for the issue it claims to solve - it is still possible for the counter to overlap, e.g., between multiple precompiled packages. I am happy to update the type exports while we're at it though |
I don't feel confident enough to evaluate that one, but I thought it looked related to what you did with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a full review but some comments:
-
If we want to keep two methods of tagging around, is this the right way to access them? It seems quite an obscure internal detail, which should never change any results. So perhaps need not be a new keyword on every user-facing function.
-
I wonder if the implementation can be much simpler, like just storing the integer as the tag, instead of adding more structs.
if kind === :default | ||
return Tag(f, X) | ||
elseif kind === :small | ||
return SmallTag(f, X) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One of my objections to this PR is that I don't much like adding keyword tag::Union{Symbol,Nothing} = :default
to everything. (1) it's messy, and I'm not sure how often anyone needs to pass this, (2) accepting a symbol from a special list of 2 seems like an odd interface. I'm not sure Val(:default)
is much better. (3) it's called tag
but isn't the tag, doesn't accept aTag
.
Maybe a better interface would be to use Preferences.jl, as for NaN-safe mode?
# SmallTag is similar to a Tag, but carries just a small UInt64 hash, instead | ||
# of the full type, which makes stacktraces / types easier to read while still | ||
# providing good resilience to perturbation confusion. | ||
struct SmallTag{H} | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we sure this needs a new type at all? Could we just extend Dual
's first type parameter from Union{Nothing, Tag}
to perhaps Union{Nothing, Tag, Integer}
?
julia> ForwardDiff.Dual(pi/2, 1)
Dual{Nothing}(1.5707963267948966,1.0)
julia> ForwardDiff.derivative(x -> @show(x), pi/2)
x = Dual{ForwardDiff.Tag{var"#37#38", Float64}}(1.5707963267948966,1.0)
1.0
@inline function ≺(::Type{SmallTag{H1}}, ::Type{SmallTag{H2}}) where {H1,H2} | ||
tagcount(SmallTag{H1}) < tagcount(SmallTag{H2}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be obvious, but can you explain a bit why this is designed this way? Why calculate hash H
and store that in the type, but call the generated tagcount(SmallTag{H1})
to get a different integer whenever you use it. Why not store the integer from TAGCOUNT
as a type parameter directly?
Also, am I correct that this doesn't make perturbation confusion impossible, only vanishingly unlikely? |
@gdalle that is correct. With a 64 bit hash, we have a 50% collision chance once you get to 4 billion different tags (that are used in the same operation), but as long as your number of tags is below millions (which should always be the case), the chance of confusion is incredibly unlikely |
This is largely to improve printing and reduce visual noise for packages with large function types (e.g. SciML)
Compare
tag = :small
:to
tag = :default
: