Issue requiring modules from a non-parent directory.

[mancusi]

I have a repository which has some interesting quirks including the fact that our main node_modules folder is not in the typical Node module resolution path. Our node_modules is in a directory ./js and our source code is in ./src and a few other directories.

Normally, one could pass a NODE_PATH envvar (see here) which is a list of absolute paths that node_modules could be found at outside of the typical "parent directory" resolution algorithm to run anything in src with node.

I've created a simple example repo which replicates the structure of our repository here. Running bazel build //src:run fails with

node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module 'acorn'
Require stack:
- /private/var/tmp/_bazel_jmancusi/9291d80434b6d660950274f29d1a5907/sandbox/darwin-sandbox/24/execroot/node_modules_not_in_parent_directory/bazel-out/darwin-opt-exec-2B5CBBC6/bin/src/bin.sh.runfiles/node_modules_not_in_parent_directory/src/require_acorn.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/private/var/tmp/_bazel_jmancusi/9291d80434b6d660950274f29d1a5907/sandbox/darwin-sandbox/24/execroot/node_modules_not_in_parent_directory/bazel-out/darwin-opt-exec-2B5CBBC6/bin/src/bin.sh.runfiles/node_modules_not_in_parent_directory/src/require_acorn.js:4:16)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/private/var/tmp/_bazel_jmancusi/9291d80434b6d660950274f29d1a5907/sandbox/darwin-sandbox/24/execroot/node_modules_not_in_parent_directory/bazel-out/darwin-opt-exec-2B5CBBC6/bin/src/bin.sh.runfiles/node_modules_not_in_parent_directory/src/require_acorn.js'
  ]
}

As you can see I've been unable to successfully get this working with rules_js. Is there a suggested way of handling repositories of this structure or will I need to move the repository such that the node_modules are linked in a parent directory?

[jbedard]

I think this is difficult because NODE_PATH must be absolute and until runtime we really have no idea where that will be (in the sandbox etc).

You can set it programmatically at runtime using things such as process.env.RUNFILES but it requires a bit of a hack: https://stackoverflow.com/a/33976627

I think ideally you wouldn't depend on this. I assume it is primarily just to hide the node_modules directory? If it's all hidden within bazel is that still necessary?

[gregmagolan]

Yeah. There is no easy way to use NODE_PATH since the directory the build takes place in is a dynamic sandbox under Bazel as @jbedard said. You could in theory hard-code it based on the execroot path and build outside of the sandbox but that is not very portable between machines and probably more trouble than its worth.

In theory you could patch npm_translate_lock to take the pnpm-lock.yaml file from the js folder but still link to the root of the repository under Bazel. That would probably be the easiest way to make your configuration work under rules_js.

It should be just one line of code to change that you can carry as a patch.

rules_js/npm/private/npm_translate_lock.bzl

Line 486 in edd0e66

root_package = rctx.attr.pnpm_lock.package

        # root package is the directory of the pnpm_lock file
        root_package = rctx.attr.pnpm_lock.package

to

        # root package is hard-coded to the root package
        root_package = ""

[aaliddell]

How is this meant to work with cross module dependencies? I've been stuck on this for a while so far.

For example, module X has a set of npm deps that get bundled as deps on a js_library that users of module X can include as a dependency of their js_binary (e.g. data = ["@X//:some_library"] which has a dep on @X//:node_modules/something). At runtime, the deps loaded in module X are not available, as the node_modules produced by npm_translate_lock and npm_link_all_packages in module X is not under _main in the runfiles tree (but instead under X+) and therefore not on NODE_PATH or any of its parents.

Can this be achieved similar to how it is in rules_python, whereby the import roots are collected transitively and then added to PYTHONPATH (NODE_PATH equiv) in the bootstrapping process: https://github.com/bazel-contrib/rules_python/blob/107a8781cdd207c9079ecd733c0028d2706a49f2/python/private/python_bootstrap_template.txt#L516-L535

Or is the answer that there shouldn't be deps coming from multiple node_modules roots on a single target binary? e.g due to version skew