All Poetry dependencies missing

[andrewvaughan]

I've been using Megalinter for a while, but it's been a minute since I've touched a Python project with it, and...

I don't recall ever have an issue using Poetry and Megalinter before - my understanding was that my .venv folder would load into the Docker container. That said, using the python flavor, it seems none of my Python linters can find my loaded dependencies:

[MegaLinter/MegaLinter]   ❓  ::group::❌ Linted [PYTHON] files with [pyright]: Found 5 error(s) - (9.22s) (expand for details)
| - Using [pyright v1.1.339] https://megalinter.io/7.7.0/descriptors/python_pyright
| - MegaLinter key: [PYTHON_PYRIGHT]
| - Rules config: identified by [pyright]
| - Number of files analyzed: [8]
| --Error detail:
| 
| added 1 package, and audited 2 packages in 4s
| 
| found 0 vulnerabilities
| /Users/andrew/Repositories/Personal/tuna-piano/TunaPiano/Audio/DeviceManager.py
|   /Users/andrew/Repositories/Personal/tuna-piano/TunaPiano/Audio/DeviceManager.py:24:8 - error: Import "sounddevice" could not be resolved (reportMissingImports)
| /Users/andrew/Repositories/Personal/tuna-piano/TunaPiano/__main__.py
|   /Users/andrew/Repositories/Personal/tuna-piano/TunaPiano/__main__.py:50:8 - error: Import "click" could not be resolved (reportMissingImports)
|   /Users/andrew/Repositories/Personal/tuna-piano/TunaPiano/__main__.py:51:8 - error: Import "inquirer" could not be resolved (reportMissingImports)
|   /Users/andrew/Repositories/Personal/tuna-piano/TunaPiano/__main__.py:60:71 - error: Cannot access member "getInputDevices" for type "object"
|     Member "getInputDevices" is unknown (reportGeneralTypeIssues)
|   /Users/andrew/Repositories/Personal/tuna-piano/TunaPiano/__main__.py:62:45 - error: Cannot access member "getDefaultInputDevice" for type "object"
|     Member "getDefaultInputDevice" is unknown (reportGeneralTypeIssues)
| 5 errors, 0 warnings, 0 informations 
| WARNING: there is a new pyright version available (v1.1.339 -> v1.1.347).
| Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`

//...

[MegaLinter/MegaLinter]   ❓  ::group::❌ Linted [PYTHON] files with [pylint]: Found 3 error(s) - (1.85s) (expand for details)
| - Using [pylint v3.0.2] https://megalinter.io/7.7.0/descriptors/python_pylint
| - MegaLinter key: [PYTHON_PYLINT]
| - Rules config: [.pylintrc]
| - Number of files analyzed: [8]
| --Error detail:
| pylint: Command line or configuration file:1: UserWarning: 'Exception' is not a proper value for the 'overgeneral-exceptions' option. Use fully qualified name (maybe 'builtins.Exception' ?) instead. This will cease to be checked at runtime in 3.1.0.
| ************* Module TunaPiano.Audio.DeviceManager
| TunaPiano/Audio/DeviceManager.py:24:0: E0401: Unable to import 'sounddevice' (import-error)
| ************* Module TunaPiano.__main__
| TunaPiano/__main__.py:50:0: E0401: Unable to import 'click' (import-error)
| TunaPiano/__main__.py:51:0: E0401: Unable to import 'inquirer' (import-error)

I've tried adding a pre-command to do a simple Python install (although this seems duplicative):

PRE_COMMANDS:
  - command: poetry install --no-root
    cwd: workspace

I've tried installing dependencies into each of the linters virtual environments (which hurt my soul):

PRE_COMMANDS:
  - command: poetry install --no-root
    venv: pyright
  - command: poetry install --no-root
    venv: pylint

I've even tried calling act locally within Poetry to see if it made a difference:

poetry run \
  act \
    --rm \
    --container-architecture "linux/amd64" \
    pull_request

I don't recall ever having to do any of these things, which is making me think I'm missing something simple. Can anyone please point me to what that is? Thanks!

[andrewvaughan]

Poking through the flavor Dockerfile, I see that megalinter creates a venv for each and every Python linter. Does this mean I have to both install my package manager and all dependencies (or something similar) to every linter that needs visibility into my dependencies?

PRE_COMMAND:
  - command: python3 -m pip install poetry
    venv: pylint
    continue_if_failed: false

  - command: python3 -m poetry install --no-root
    venv: pylint
    continue_if_failed: false

This is wildly inefficient, if so - taking up easily 10x the time and disk space required for linting - so I'm hoping someone can point me toward a more sane solution wherein I can point linters toward a shared virtual environment. I'm not sure I understand the rationale for splitting out so many virtual environments in the first place.

(Note, this still fails, because the install command can't find my pyproject.toml file... so this isn't working, but seems like it's the intended way Megalinter wants you to manage project dependencies for each of these linters?).

Most comments I see online are "just disable the import checking" which... really defeats the purpose of having a linter.

[andrewvaughan]

I figured out an interim solution and, I'll be honest, I hate it. It does the following:

1. Installs poetry once and exports a requirements.txt file from the Poetry configuration to the /venvs directory
2. Manually installs all project dependencies into each linter's virtual environment that needs it - yep, duplicating installs for each one... oof.

PRE_COMMANDS:
  # Export the Poetry configuration to a `requirements.txt` file that each of the virtual environments can use
  - command: python3 -m pip install poetry
    cwd: root
    continue_if_failed: false

  - command: poetry export --without-hashes -f requirements.txt -o /venvs/requirements.txt
    cwd: workspace
    continue_if_failed: false

  # Install dependencies for `pylint`
  - command: python3 -m pip install --no-cache-dir -r /venvs/requirements.txt
    venv: pylint
    continue_if_failed: false

  # Install dependencies for `pyright`
  - command: python3 -m pip install --no-cache-dir -r /venvs/requirements.txt
    venv: pyright
    continue_if_failed: false

I'm hoping @nvuillam or crew can impart some wisdom on me as to how this might be better managed. I really hate duplicating dependency installations on each and every test because there are dozens of virtual environments created...

[nvuillam]

@andrewvaughan there was dependencies issues when trying to install all the python based linters in the same venv, so we did not have another choice than insalling them in there own venv :/

For example, pylint was dependent to 'requests<3' and pyright was dependent to 'requests>4' , it was simply impossible to manage in the same venv, and we needed to investigate for hours to find which dependency has been upgraded, and which linter had to be downgraded by force until a new versions was released, etc....

[andrewvaughan]

It is what it is! The joy of running a project that includes tons of non-standard dependencies.

I was able to make it work with the code above, and by limiting my import checks to just one tool, I only had to do PRE_COMMANDS for pyright (disabling pylint). That's my recommendation for anyone else who comes across this thread:

1. Export your poetry dependencies to a requirements.txt file
2. Select a single tool to do import checking (in my case, pyright)
3. Disable import checking in all other tools via config
4. Install the requirements.txt via pip in the PRE_COMMANDS for the selected virtual environment

That way, you're only installing your dependencies one extra time.

Thanks again.

[nvuillam]

Congrats for finding the workaround :)

[technic960183]

Thanks @andrewvaughan for providing the solution. I also encounter this problem and find this discussion.

I'm using GitHub Action.

Here are some new update.

Start from 2.0, poetry export is no longer installed by default with poetry. So the solution might fail.
I solved it by install poetry==1.8.5.

Just for anyone might needed, I find that my dependence is easy and I can copy requirements.txt from the root of my repo by:

  - command: cp requirements.txt /venvs/requirements.txt
    cwd: workspace
    continue_if_failed: false

Still, I don't know why pyright venv is missing from the /venv/. So I remove the pyright part. I'm sure that pyright had run, it had produced log.

  [Pre] run: [ls -la /venvs] in cwd [/github/workspace]
  [Pre] result:
  total 92
  drwxr-xr-x   23 root     root          4096 Apr 27 18:14 .
  drwxr-xr-x    1 root     root          4096 May  2 18:13 ..
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 ansible-lint
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 bandit
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 black
  drwxr-xr-x    5 root     root          4096 Apr 27 18:13 checkov
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 cpplint
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 djlint
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 flake8
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 isort
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 mypy
  drwxr-xr-x    4 root     root          4096 Apr 27 18:14 proselint
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 pylint
  drwxr-xr-x    4 root     root          4096 Apr 27 18:14 rst-lint
  drwxr-xr-x    4 root     root          4096 Apr 27 18:14 rstcheck
  drwxr-xr-x    4 root     root          4096 Apr 27 18:14 rstfmt
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 ruff
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 ruff-format
  drwxr-xr-x    4 root     root          4096 Apr 27 18:14 semgrep
  drwxr-xr-x    4 root     root          4096 Apr 27 18:14 snakefmt
  drwxr-xr-x    4 root     root          4096 Apr 27 18:14 sqlfluff
  drwxr-xr-x    4 root     root          4096 Apr 27 18:13 stylelint
  drwxr-xr-x    4 root     root          4096 Apr 27 18:14 yamllint

[echoix]

Pyright, as a tool written in JavaScript or Typescript, is installed with npm in Megalinter, and not as a Python package from pypi as it was when that discussion originally was written. As such, there is no venv for pyright as you observed.

There were some issues with installation at some point, which lead to the change of the installation method.