8364202: CDS without G1 gives build error in slowdebug, asserts in fastdebug

[tstuefe]

This patch fixes two problems:

• when building without INCLUDE_CDS_JAVA_HEAP (disabling G1 at configure, or building 32-bit), we get a linker error in slowdebug.
• when running with CDS, but without INCLUDE_CDS_JAVA_HEAP, we initialize CompressedKlassPointers via the "lenient" route where we allow optimized encoding base choice. This can lead to zero-based encoding, in which case we should not attempt to setup a protection zone at the start of the encoding range.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8364202: CDS without G1 gives build error in slowdebug, asserts in fastdebug (Enhancement - P4)

Reviewers

• Calvin Cheung (@calvinccheung - Reviewer)
• Ioi Lam (@iklam - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/26523/head:pull/26523
$ git checkout pull/26523

Update a local copy of the PR:
$ git checkout pull/26523
$ git pull https://git.openjdk.org/jdk.git pull/26523/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 26523

View PR using the GUI difftool:
$ git pr show -t 26523

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/26523.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back stuefe! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@tstuefe This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8364202: CDS without G1 gives build error in slowdebug, asserts in fastdebug

Reviewed-by: ccheung, iklam

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 10 new commits pushed to the master branch:

• 330ee87: 8360163: Replace hard-coded checks with AOTRuntimeSetup and AOTSafeClassInitializer
• d5d311f: 8361873: [GCC static analyzer] exec_md.c forkedChildProcess potential double 'close' of file descriptor '3'
• ea75431: 8363928: Specifying AOTCacheOutput with a blank path causes the JVM to crash
• ... and 7 more: https://git.openjdk.org/jdk/compare/4669005123420d8dbe86740dd9bcbee04735bc4f...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[openjdk]

@tstuefe The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[tstuefe]

Reviewer Notes, changes in CDS explained:

• Line 1626: fix bug by only establishing a protection zone if the encoding base points to the start of the archive after setup (aka we don't end up running zero-based encoding)
• Lines 1619, 1623: We only have two cases: We either need to dictate the encoding (if CDS heap area exists), in which case the encoding base must be the start of the mappings. Or, we let the JVM choose freely, in which case we end up with base=zero if possible, base=mapping start otherwise. Assert both cases.
• Line 1606: When initializing CompressedKlassPointers, we should always feed in the mapping start for klass range start. This has no practical consequences, but it is cleaner and more consistent with the CDS=off path. I also added a comment to compressedKlass.hpp in that regard.

[tstuefe]

NMT gtest failure on windows unrelated.

[tstuefe]

@tschatzl , @iklam , could you please review?

[mlbridge]

Webrevs

• 00: Full (de2f3bfe)

[calvinccheung]

Looks good.
I did some sanity testing (tiers 1 - 3, and zero build). No failures.

[iklam]

LGTM