dune-project auto generation and it's effect on libraries

[vzaliva]

I am not sure how to exactly formulate the problem, so I will start by describing a scenario:

I have a project with the following structure:

I have the following directory structure:

ml/
   dune
   test.ml
   extracted/
                 dune
                 _(lot of .ml files)_

The ml/dune:

(executables_
 (names test)_
 (libraries extracted))

ml/extracted/dune:

(library
 (name extracted)
 (wrapped false)
 (synopsis "ML code extracted from Coq"))

Doing dune build test.ext under ml complains:

Error: Library “extracted” not found.

I've spent a lot of time to figure out why this is happening. It turns out the problem was in presence of extracted/dune-project file! If this file present, build fails with a cryptic error message. Removing this file fixes the problem.

Some thoughts:

1. It is unclear why the presence of this file changes the behavior of the build. It looks harmless as contains only the version number.
2. The error message is not helpful.
3. I did not create this file. It was auto-created when I tried to run dune in extracted directory.

It looks like the file auto-creation is a dangerous operation as it could break your build. Also, error reporting could be improved.

[rgrinberg]

A couple points to help you catch this in the future:

• I've mentioned in the discuss thread that the presence of a dune-project might affect things. I think this could have saved you some time debugging.

• You should always commit the dune-project file(s) into your repository. Your other builds aren't reproducible without this file. The dune-project file should live at the root your repository in your case.

• Private libraries aren't accessible outside of a "scope". A scope is defined by a dune-project or an `.opam file (for compat with jbuilder).

Some things that we'd like to do on our end to make the experience better:

• Have a $ dune describe command that will describe projects/packages/libraries/executables/tests and their relationships in friendly output

• Improve the error message for missing libraries. We could have reported that this private library was simply unavailable in your current scope.

[gasche]

Note: this usability bug would be fixed by #4108.