@functor is a bad name and nobody knows to use it

[ToucheSir]

Sensational title, but I've been seeing many recent examples of code overloading trainable instead. Given that there's no fundamental reason to call the macro which makes this struct a trainable layer "functor", We could at least alias it to something more intuitive. RFC on ideas?

[AriMKatz]

layer, module or parameterise?

[johnnychen94]

• module is a built-in name for Julia, so @module would cause more confusion...
• @parameterise/@parameterize 😆 would cause a lot of typos...

Is something like this more readable?

@parameter Dense trainable=(:w, :b)
# or...
build!(Dense, trainable=(:w, :b))

[mcabbott]

Agree that "functor" is a pretty obscure name. Once upon a time it was Flux.@treelike which is more suggestive.

Also agree that something which does trainable at the same time would be neater, it's a bit confusing that two similar things have separate quite different interfaces (and the simpler one is the one you rarely need). We could make some unified convenience macro like Flux.@walk Dense parameters=(w, b) trainable=(w) without touching Optimisers.jl/Functors.jl.

The happiest situation would be not to need such a thing at all, for most layers. It could be something like:

1. arrays of non-integer numbers are parameters (except perhaps for a few like Transpose)
2. any struct containing such must have the default constructor
3. a struct not containing parameters is leaflike, will never be re-constructed
4. for non-trainable parameters you should opt out, not in, so that it's clearer this is the exception. Could just be e.g. Flux.notrain(d::Dense) = (b = d.b,)

I bet almost all models would just work with 1,2,3. It's a constraint that you have to satisfy 2, but not a difficult one -- and we aren't aiming to handle everything in the world, only things written to be used with Flux.

[ToucheSir]

2. doesn't even need to be a constraint either if we ask users to fulfill an interface like the one in https://github.com/JuliaObjects/ConstructionBase.jl. This has the added benefit of possibly catching types which were not designed to be compatible with Flux but do have custom construction routines to work with different packages.

In the short term, a convenience macro seems the path of least resistance. Longer term, we'll want to look at discussions like JuliaObjects/ConstructionBase.jl#54 to figure out what underlying infrastructure would be required for the "happiest situation".

[CarloLucibello]

related to FluxML/Functors.jl#49 . It would be nice to assume every type "functorized" by default

[ToucheSir]

That would be great, but the tricky part about functorizing by default is that it turns the opt-in mechanism into an opt-out one. e.g. If someone defines a custom layer type that contains a ValueHistories or DataStructures type, gpu would erroneously try to convert any inner arrays in that type.

Some of these cases could be avoided by allowing user/FluxML-owned parent types to exclude certain fields, but it's not hard to come up with a counterexample there either. Opting out for certain types via piracy is another option, but a suboptimal one at best. If we can come up with a satisfactory solution for this problem, then I don't see much else in the way of auto-functorization.