How to only install a subset of executors

[SteVwonder]

Currently, the local, rp, and flux executors are all installed and are hardcoded in psi/j/executors/__init__.py. If you try to delete any of them from the installation directory, import errors occur. This prevents running/debugging psi.j on a mac since Flux does not support Mac (no way currently to compile it or install the python bindings on it).

Based on previous ☎️ discussions, it sounds like @hategan and/or @andre-merzky have ideas on how to handle this. So going forward what do we want to do to support installing only a subset of executors? Or maybe alternatively, handling installing executors that cannot run/import.

EDIT: correct the path to the referenced __init__.py file

[andre-merzky]

ideas on how to handle this.

The proposal was to us separate repositories / python modules for each connector, and thus to install and deinstall them independently from psi/j. There is a proof-of-concept repo at here for the local connector, so we know it can be done (that repo is likely incompatible with the current code base though).

[hategan]

proof-of-concept repo at here for the local connector,

Although I think the local connector should maybe not be separated.

[hategan]

and are hardcoded in psi/j/__init__.py

I am not sure what you mean there. Executors and launchers are dynamically discovered at run-time. There are no explicit references to any particular executor in psi/j/init.py. Are we sure we are talking about the same file? (https://github.com/ExaWorks/psi-j-python/blob/main/src/psi/j/__init__.py)

[hategan]

They are referenced by psi/j/executors/__init__.py, which might be causing problems.

[SteVwonder]

Ah. Sorry. I forgot the executor part of the path. That is indeed the file I meant to reference. Thanks!

[hategan]

So I don't think that file is necessary. It makes direct imports of specific executors look a bit nicer, but that is not our primary usage scenario anyway.

That said, I do favor separating executors that have nonstandard dependencies.

[hategan]

This has been addressed by dynamic executor imports and should not be an issue any more.