discussion: separate the module file detection logic ?

[aviatesk]

Hi @timholy, thanks for your great work on Revise :)

tl;dr; In this issue I would like to propose the idea of separating the code in Revise.jl, which finds files in a loaded module, into a separate package.

background

Currently there is a strong need for a robust implementation of a logic to find included files in a module (let me call this "module file finding logic"), in Juno:

features that will use "module file finding logic":

1. automatic file's module detection
2. features based on static code analysis
   • like outline, symbol jump (and in the future, rename, and so on)
   • all those static code analysis should be done per module

Within those features, it's really important to be able to find all the files in a specific module (including files in a non-precompiled package, or files in a submodule), as accurately as possible.

In order to meet the requirement, I and @pfitzseb now have the 2-stage logic.

"2-stage logic":

1. first take a "Revise-like" logic (specifically, we're using modulefiles(mod::Module))
2. when approach 1. fails (e.g. for non-precompiled files), fallback to CSTParse-based "static module traverse"

You can find our current implementation of 1. here (and just fyi: that of 2. here)

(subnote: currently the "automatic file's module revealing" uses CodeTools, but we're planning to use "2-stage logic" for this feature as well)

proposal

It may be obvious our implementation of "Revise-like" approach is not so ideal -- I just copied all the code that was related to modulefiles from Revise.
It works now, but the problem is that its maintainability: since we really don't want to depend on Revise, we have to watch on changes in Revise and make an equivalent patch on Atom.jl like this PR.

So in conclusion, we would like to ask you to separate code in Revise that includes its logic to find files in a (loaded) module into a separate package. Then it's enough to make changes just in one place.
What do you think of this idea ?

another discussion

Also I would like to know if you have any idea to improve our "module finding logic". Is there any utility in Revise or CodeTracking that we can reuse for the "2-stage logic" ?

/cc @pfitzseb

[timholy]

This is a good idea. My main recommendation would be to use a queue_includes!-like mechanism for non-precompiled files. I think that works pretty well and doesn't require CSTParser.

I suspect those methods need better names and a clearer division of purpose. As evidence, you guys settled on modulefiles whereas Revise barely uses that method internally, only as a last-ditch approach to find code.

Where would you like the resulting "mini-package" to live? JuliaDebug?

[pfitzseb]

One important point is that we want a method that works even if a package/module isn't loaded yet, as far as I understand queue_includes! only works after having usinged something.

Where would you like the resulting "mini-package" to live? JuliaDebug?

We could create a JuliaCodeTools org and additionally put Tokenize, CSTParser, and maybe CodeTracking in there. Might be overkill for now though.

[timholy]

Would you expect that the package that has Revise's current detection tools would also import CSTParser? That's not currently a dependency of Revise's.

[pfitzseb]

No, not necessarily (or at least not if you don't want Revise to take on that dependency).

[aviatesk]

One important point is that we want a method that works even if a package/module isn't loaded yet, as far as I understand queue_includes! only works after having usinged something.

The whole point of Revise is to update function definitions in a running session, so we didn't even expect Revise has such methods, did we ? For those unloaded package/module, we should use some form of static parsing (and this is why we have "2-stage logic"), but I guess Revise will never need that.

Rather, I think, the more important aspect of our requirement is that we have to find files per-module basis rather than per-package basis. So modulefiles seems better choice than queue_includes! for us, AFAIU.

Well, what code unit do you think is best to be extracted into a separate package ?
Just for our purpose, it might be enough to extract pkgs.jl and recipes.jl into a new package, but I'm really not sure this is a reasonable unit for a general usage. Anyway it would be ideal the package stays to be lightweight though.

[timholy]

So modulefiles seems better choice than queue_includes! for us, AFAIU.

They have very different purposes; modulefiles only works for precompiled packages, and queue_includes only for non-precompiled packages, and as @pfitzseb points out only retrospectively. This could all use some cleanup and clarification!

[timholy]

extract pkgs.jl and recipes.jl into a new package

That seems reasonable. I'm not opposed to having packages that have only 2 consumers; the alternative is code-copying, which as you point out is not ideal.

[aviatesk]

sorry for my late response.

That seems reasonable. I'm not opposed to having packages that have only 2 consumers; the alternative is code-copying, which as you point out is not ideal.

Okay, cool. Then I will try to extract them into a separate package probably with some refactoring so that the Revise-specific functionalities would stay in Revise.

Do you have any suggestion for the name of the package ? I'm thinking of ModuleTracking after CodeTracking for now.

[timholy]

Sorry I didn't respond to your question. Yes, ModuleTracking would be a fine name.

[aviatesk]

hehe, I have just forgot this at all so thanks for your reminding, @timholy .
I will tackle this sometime in the next coming months once my general life stuff gets settled.