Module info patch

The problem of whitebox testing in a modular project

For whitebox testing, it is necessary to use --patch-module, --add-modules, --add-reads, --add-exports and --add-opens compiler options. A medium sized project can require a lot of these options. Writing them inside the Maven <compilerArgs> XML element is not practicable: it is tedious, redundant (the name of the module to patch is repeated in every occurrence of some options), error prone, and must be repeated in every plugins that depends on the tests (Surefire, Javadoc for test documentation, etc.).

Preferred approach

Options such as --add-reads can be put in a text file. That text file can be read by commands such as java @file where file is the file containing the options. This approach is described in Christian Stein's blog Testing In The Modular World, which proposes to write the options in a file named module-info.test. The approach described in this page is fundamentally the same, except that module-info.test is "compiled" from another file – module-info-patch.maven – for reasons explained below (in short: for readability, avoiding redundancy, adding TEST-MODULE-PATH in complement to ALL-MODULE-PATH, and because there is potentially two distinct compilation outputs: one for javac and one for java). This module-info-patch.maven file format has some similarities with another Christian's article.

Deprecated approach

Prior the approach described in this page, the Maven approach for making whitebox testing a little bit easier was to define a module-info.java file in the test source code which replace the main module-info.java file. However, this approach has some problems:

• It forces the developer to repeat all the content of the main module-info.java into the test module-info.java, then add test-specific requires, exports or opens statements. This is risky: if the copied part become different (e.g. because a developer made a change in main and forgot to copy the change in test), then the tests may not be running in the intended environment. This is also tedious if the main module-info.java is large.
• Java does not allow to overwrite module-info.java easily, maybe for security reasons. The "module-info in test" approach forces build tools to trick Java. Maven 3 was declaring test as the main code and main as the patch over test, which is a risk of weird behavior. Maven 4 does not put the project upside-down, but instead temporarily deletes the main module-info.class file so that Java pickups the test one.

Patching a module info with Java options

Instead of defining a module-info.java file in test, Maven projects can define a module-info-patch.maven. The content of module-info-patch.maven uses the same syntax as Java, C/C++, JavaScript, Groovy, etc. (/* … */ or // for comments, blocks between { … } brackets, statements ending with ;) but is not Java, hence the need for a different extension. The general principles are:

• Everything that a developer would like to change in a module-info.java file for testing purposes is declared in module-info-patch.maven.
• Everything that is not in module-info.java is not in module-info-patch.maven neither. In particular, everything that specify paths to JAR files or paths to source code stay in the pom.xml file.
• All keywords inside the patch-module block of that file map directly to Java compiler or Java launcher options.

Why a new format

The reason for a writing the Java options in a new file format which will be "compiled" to a text file usable by javac @file, instead of letting developers writing directly that text file are:

• Readability in the following ways:
  • by allowing comments, and
  • not being forced to enumerate all modules as a comma-separated list on the same line without space.
• Reduce redundancy, because the module name is declared only once per module.
  • By contrast, the "compiled" options repeat the module name in every --add-reads, --add-exports and --add-opens options.
• Allow the addition of keywords unknown to java and javac.
  • In particular, javac knows ALL-MODULE-PATH but has no way to know TEST-MODULE-PATH since the later is specific to (for example) Maven's concept of dependency scope.
• Make possible to compile two versions of @file from the same set of module-info-patch:
  • One file of options for compilation:
    • TEST-MODULE-PATH is converted to the list of modules having test and test-only Maven's scope.
    • These options do not include --add-opens because this options does not exist (makes no sense) in javac.
  • One file of options for test execution:
    • TEST-MODULE-PATH is converted to the list of modules having test and test-runtime Maven's scope.
    • These options include the --add-opens option.

Syntax

The syntax is:

• The same styles of comment as Java (/* … */ and //) are accepted.
• The first tokens, after comments, shall be patch-module followed by the name of the module to patch.
• All keywords inside patch-module are Java compiler or Java launcher options without the leading -- characters.
• Each option value ends at the ; character, which is mandatory.

The accepted keywords are add-modules, limit-modules, add-reads, add-exports and add-opens. Note that they are options where the values are package or module names, not paths to source or binary files. Options with path values (--module-path, --module-source-path, --patch-module, etc.) continue to be derived from the dependencies declared in the POM.

Options applying to all modules

All options declared in a module-info-patch.maven file apply only to the module declared after the patch-module token, except the --add-modules and --limit-modules options. These two options apply to all modules in a multi-modules project, because these options given to java or javac expect no module name. Therefore, it is not necessary to repeat add-modules TEST-MODULE-PATH in all modules: declaring that particular option in only one module of a multi-modules project is sufficient. If the --add-modules or --limit-modules options are declared in many module-info-patch.maven files of a multi-modules project, then the effective value is the union of the values declared in each file.

Special option values

The following option values have special meanings:

• SUBPROJECT-MODULES: all other modules in the current Maven (sub)project.
  • This is Maven-specific, not a standard value recognized by Java tools.
  • Allowed in: add-exports.
• TEST-MODULE-PATH: all dependencies having a test scope in the build tools.
  • This is specific to this format, not a standard value recognized by Java tools.
  • Allowed in: add-modules, add-reads and add-exports options.
• ALL-MODULE-PATH: everything on the module path, regardless if test or main.
  • This is a standard value accepted by the Java compiler.
  • Allowed in: add-modules option.
• ALL-UNNAMED: all non-modular dependencies.
  • This is a standard value accepted by the Java compiler.
  • Allowed in: add-exports option.

Example

Below is an example of a module-info-patch.maven file content for modifying the module-info of a module named my.product.foo:

/*
 * The same comments as in Java are allowed.
 */
patch-module my.product.foo {             // Put here the name of the module to patch.
    add-modules TEST-MODULE-PATH;         // Recommended value in the majority of cases.

    add-reads org.junit.jupiter.api,      // Frequently used dependency for tests.
              my.product.test.fixture;    // Put here any other dependency needed for tests.

    add-exports my.product.foo.internal   // Name of a package which is normally not exported.
             to org.junit.jupiter.api,    // Any module that need access to above package for testing.
                my.product.test.fixture;  // Can export to many modules, as a coma-separated list.

    add-exports my.product.foo.mock       // Another package to export. It may be a package defined in the tests.
             to my.product.biz;           // Another module of this project which may want to reuse test classes.
}

How module info patches are compiled

module-info-patch.maven are compiled into a file of options in the following ways:

• add-modules foo, bar; is translated to --add-modules foo,bar.
  • Note: spaces between foo and bar are removed for interpreting the option value as a single argument.
• limit-modules foo, bar; is translated to --limit-modules foo,bar.
  • Note: idem regarding spaces removal.
• add-reads foo, bar; is translated to --add-reads patched=foo,bar where patched is the module declared at the beginning of the module-info-patch.
  • Note: this is an example of above argument about reducing redundancy.
• add-exports biz to foo, bar; is translated to --add-exports patched/biz=foo,bar where patched is as above.
• add-opens biz to foo, bar; is translated to --add-opens patched/biz=foo,bar ( patched as above).
  • Note: this option is included only for runtime execution (not for compilation).

There is a separated module-info-patch.maven file for each module, and the Maven compiler plugin merges them in a single set of options for java and javac. While this format does not require the use of module source hierarchy, a goal was to fit nicely in that hierarchy. For example, the following module-info-patch.maven files:

patch-module foo {
    add-modules TEST-MODULE-PATH;
    add-reads org.junit.jupiter;
}

and

patch-module bar {
    add-modules TEST-MODULE-PATH;
    add-reads org.junit.jupiter;
}

are "compiled" into a single text file as below (note the replacement of TEST-MODULE-PATH by modules having the test scope in Maven's dependencies):

--add-modules org.junit.jupiter,etc
--add-reads foo=org.junit.jupiter
--add-reads bar=org.junit.jupiter

The "compiled" text file is located at target/test-classes/META-INF/maven/module-info-patch.args. This is the version for execution. The version for compilation is not generated, because the plugin forwards them directly to javax.tools.JavaCompiler's API.

Proof of concept implementation

This proposal has been implemented in a clone of the Maven compiler plugin and tested with a real project. Steps for testing:

• Download and install Maven 4.0.0-rc-3. That version needs to be used for the rest of this page.
• Clone https://github.com/Geomatys/maven-compiler-plugin and checkout the module-info-patch branch.
• Run mvn install for installing locally a snapshot of the compiler plugin.
• Clone https://github.com/Geomatys/sis and checkout the maven4 branch.
• Run mvn test-compile (Maven plugins other than compiler have not yet been updated).

Examples from the real application:

• Example of a module-info file.
• Example of the corresponding module-info-patch file.
• After a local build, see endorsed/target/test-classes/META-INF/maven/module-info-patch.args for the javac options built from the content of all module-info-patch files. It illustrates why maintaining this list of options directly is uneasy.

Ease of implementation

Parsing the module-info-patch file is actually relatively easy when using java.io.StreamTokenizer. The Maven plugin parser can be used as a source of inspiration. It is only one class with dependencies to other Maven classes that can easily be removed or replaced.