Incompatible with C++20 modules in MSVC

[andylizf]

Description

Importing a module file using the lib will cause a compilation error.

Reproduction steps

Codes below.

Expected vs. actual results

To compile successfully.

Minimal code example

network.ixx:

module;
#include <nlohmann/json.hpp>
#include <string>
export module network;
using nlohmann::json;
export struct message {
  message(std::string str) { json doc = json::parse(str); }
};

main.cpp:

import network;
int main() { message msg(R"({"hello": "world"})"); }

Error messages

nlohmann\json.hpp(117): error C2039: 'json_sax_dom_callback_parser': is not a member of 'nlohmann::json_abi_v3_11_2::detail'
nlohmann\detail\output\serializer.hpp(35): note: see declaration of 'nlohmann::json_abi_v3_11_2::detail'
main.cpp(3): note: see reference to class template instantiation 'nlohmann::json_abi_v3_11_2::basic_json<std::map,std::vector,std::basic_string<char,std::char_traits<char>,std::allocator<char>>,bool,int64_t,uint64_t,double,std::allocator,nlohmann::json_abi_v3_11_2::adl_serializer,std::vector<uint8_t,std::allocator<uint8_t>>>' being compiled
nlohmann\byte_container_with_subtype.hpp(101): warning C4820: 'nlohmann::json_abi_v3_11_2::byte_container_with_subtype<BinaryType>': '7' bytes padding added after data member 'nlohmann::json_abi_v3_11_2::byte_container_with_subtype<BinaryType>::m_has_subtype'
        with
        [
            BinaryType=std::vector<uint8_t,std::allocator<uint8_t>>
        ]
        [            BinaryType=std::vector<uint8_t,std::allocator<uint8_t>>
        ]
nlohmann\json.hpp(4187): warning C4820: 'nlohmann::json_abi_v3_11_2::basic_json<std::map,std::vector,std::basic_string<char,std::char_traits<char>,std::allocator<char>>,bool,int64_t,uint64_t,double,std::allocator,nlohmann::json_abi_v3_11_2::adl_serializer,std::vector<uint8_t,std::allocator<uint8_t>>>': '7' bytes padding added after data member 'nlohmann::json_abi_v3_11_2::basic_json<std::map,std::vector,std::basic_string<char,std::char_traits<char>,std::allocator<char>>,bool,int64_t,uint64_t,double,std::allocator,nlohmann::json_abi_v3_11_2::adl_serializer,std::vector<uint8_t,std::allocator<uint8_t>>>::m_type

Compiler and operating system

Visual Studio 17 2022 MSVC 19.36.32323; Windows-10.0.22624

Library version

3.11.2

Validation

• The bug also occurs if the latest version from the develop branch is used.
• I can successfully compile and run the unit tests.

[nlohmann]

Maybe related to #3974 and fixed by #3975.

[andylizf]

Is it fixed now?
BTW, how to use the new version in xmake?

[gviot]

it's not fixed, I got this compilation error as well with single include and modules.
It looks kinda weird, not sure why this line in particular errors but not the one above which is almost identical.

[kiraYuukiAsuna]

same issue in version 3.11.3

[rnikander]

I have the issue, 3.11.3, Visual Studio, using C++20 modules.

[rioki]

The issue is not an issue with nlohmann/json, but rather MSVC compiler. I get the same issue with c42f/tinyformat. The compiler does not like something in the header and I just have not yet figured out what it is.

You can work around it if you add a "compilation firewall" by defining an interface in a header and shoving the include in a regular cpp. Too bad both libraries rely heavily on template to make the client code nice and small.

[JamieSharpe]

I'm having a similar issue when trying to include nlohmann/json.hpp within a C++ module file.

Dev Attribube	Value
Visual Studio Version:	17.10.0 Preview 7.0
Windows SDK Version:	10.0
Platform Toolset:	Visual Studio 2022 (v143)
C++ Language Standard:	Preview - Features from the Latest C++ Working Draft (/std:c++latest)
nlohmann-json	Version: 3.11.3

The following section of my code:

nlohmann::json config = {
	{"Version", "x.x.x.d"}
};

Provides the error:

>Main.cpp
> *\vcpkg_installed\x64 - windows\x64 - windows\include\nlohmann\json.hpp(120, 38) : error C2039 : 'json_sax_dom_callback_parser' : is not a member of 'nlohmann::json_abi_v3_11_3::detail'

I'm in a greenfield project and would love to be able to use this library to parse and export json data.

@rioki, are you able to expand on your temporary fix until either this library or MSVC have a solution?

Many thanks.

[rnikander]

For my project I moved to jsoncpp, which has been working fine with a module wrapper that I made.

[rioki]

@JamieSharpe I dropped modules altogether for the time being. I ran into 1001 issues like this. I will revisit modules in a year or two. Looks promising, not there yet.

[duerrbaby]

The issue is not an issue with nlohmann/json, but rather MSVC compiler. I get the same issue with c42f/tinyformat. The compiler does not like something in the header and I just have not yet figured out what it is.

You can work around it if you add a "compilation firewall" by defining an interface in a header and shoving the include in a regular cpp. Too bad both libraries rely heavily on template to make the client code nice and small.

Exactly how did you implement the compilation firewall? Perhaps you can show some code. I'm getting compilation errors when I try to do it. :/

[Blechwolf]

Exactly how did you implement the compilation firewall? Perhaps you can show some code. I'm getting compilation errors when I try to do it. :/

For me the following works:

JSONTools.ixx

module;

#include <filesystem>

export module JSONTools;

import DataDefs;  // defines A

namespace myns
{
    export class JSONTools final
    {
        public:
            static A readA(const std::filesystem::path& filePath);
            static void writeA(const std::filesystem::path& filePath, const A& a);
        private:
            JSONTools() = delete;
            JSONTools(const JSONTools&) = delete;
            JSONTools(JSONTools&&) = delete;
    };    

} // namespace myns

DataDefs.cpp

module;

#include <fstream>
#include <nlohmann/json.hpp>

module JSONTools;

import DataDefs;

namespace myns
{

    A JSONTools::readA(const std::filesystem::path& filePath);
    {
        std::string fileContent = readTxt(filePath);   // readTxt defined somewhere else
        nlohmann::json jsonContent = nlohmann::json::parse(fileContent);

        A result{};

        A.setVersion(jsonContent["version"].get<int>());

        // Fill A with the data read from the json

        return result;
    }

    void JSONTools::writeA(const std::filesystem::path& filePath, const A& a);
    {
        auto oFile = std::make_unique<std::ofstream>(filePath, std::ios::ate);

        nlohmann::json jsonOutput;

        jsonOutput["version"] = 1;
        // add some more content to json file

        (*oFile) << jsonOutput; 
    }

} // myns

CMakeLists.txt

project(JSONTools CXX)

find_package(nlohmann_json CONFIG REQUIRED)

set(INTERFACE_LIST
   JSONTools.ixx
)

set(SOURCE_LIST
   DataDefs.cpp
)

add_library(${PROJECT_NAME})

target_sources(${PROJECT_NAME}
   PRIVATE
      ${SOURCE_LIST}
   PUBLIC
   FILE_SET CXX_MODULES
   FILES ${INTERFACE_LIST}
)

target_link_libraries(${PROJECT_NAME}
PRIVATE
   DataDefs
   nlohmann_json::nlohmann_json
)

For every other module I am just importing JSONTools in the target_link_libraries section of the corresponding CMakeLists.txt

[Optimierungswerfer]

We are also running into this issue. Has anyone raised an issue about this with the MSVC team yet? @rioki
It appears that a Microsoft engineer already ran into this exact issue while investigating another issue report. Unfortunately, this report seems to have been stale for years.

This issue is the last roadblock for us to finally migrate parts of our codebase to C++ modules.

Dev Attribube	Value
Visual Studio Version:	17.10.7
Windows SDK Version:	10.0.18362.0
Platform Toolset:	Visual Studio 2022 (v143)
C++ Language Standard:	Preview - Features from the Latest C++ Working Draft (/std:c++latest)
nlohmann-json	Version: 3.11.3

[saekiamae]

To fix this, replace in your json.hpp file (around line 20120):

template<typename BasicJsonType, typename InputAdapterType> friend class ::nlohmann::detail::json_sax_dom_callback_parser;
with
template<typename BasicJsonType, typename InputAdapterType> friend class nlohmann::detail::json_sax_dom_callback_parser;
(remove :: from between friend class and nlohmann namespace)

[nlohmann]

Can anyone confirm this? If so, I'm happy to see a PR for this :)

[saekiamae]

All I can say is that i was getting compilation error from this issue and issue mentioned here:
#4529

and removing these '::' did fix my issue. Not sure if it is universal solution or only solution for my case but seems working to me.

[Ju1He1]

Hey guys,
we are using C++20 modules with this library. Most of the stuff works fine
(We are using latest cmake, latest msvc and C++23 for import std).

Basically what we did is putting this inside our nlohman_json module

// Global module fragment where #includes can happen
module;

// included headers will not be exported to the client
#include <nlohmann/json.hpp>

// first thing after the Global module fragment must be a module command
export module my_nlohmann_json;

// build up namespace from macros
#define NLOHMANN_NS NLOHMANN_JSON_NAMESPACE_CONCAT( \
  NLOHMANN_JSON_ABI_TAGS, \
  NLOHMANN_JSON_NAMESPACE_VERSION)

export
{
  namespace nlohmann::NLOHMANN_NS::detail
  {
    using ::nlohmann::NLOHMANN_NS::detail::json_sax_dom_callback_parser;
    using ::nlohmann::NLOHMANN_NS::detail::json_sax_dom_parser;
  }

  namespace nlohmann::NLOHMANN_NS::literals
  {
    using ::nlohmann::NLOHMANN_NS::literals::operator""_json;
    using ::nlohmann::NLOHMANN_NS::literals::operator""_json_pointer;
  }



  namespace nlohmann
  {
    using ::nlohmann::basic_json;
    using ::nlohmann::json;
    using ::nlohmann::json_sax;
    using ::nlohmann::ordered_json;
    using ::nlohmann::detail::json_sax_dom_parser;
    using ::nlohmann::detail::json_sax_dom_callback_parser;
    using ::nlohmann::to_json;
    using ::nlohmann::from_json;
  }
}

module : private;
// possible to include headers here, but they will not be exported to the client

This works fine for most of our use cases.
However the makros (like NLOHMANN_DEFINE_TYPE_INTRUSIVE cause a major issue).
Since makros are not beeing exported by modules we have to include #include <nlohmann/json.hpp> in our headers which cause stl types from classic includes and modules to collide.

Best case is the avoid name collisions (i.e. don't mix import std.compat and #include <string> for good.
If you do need to mix stl includes and imports you have to ensure for every translation unit that all includes come before the first import.

@nlohmann do you think it ts possible to extract all makors to a seperate file which ony uses either import std.compat or include <xxx>

If that is possible the approach suggestion above should work :)

[gregmarr]

@Ju1He1 Have you tried including https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/macro_scope.hpp? I don't know if it will work stand-alone, but it's a possibility. It should at least give some feedback on what would be needed in a standalone header.

[Ju1He1]

Hey @gregmarr
Thx I allready checked that but Macro Scope includes <utility> which breaks import std sadley
( i cannot guranrantee for every translation unit that my includes are placed before the imports)

[Optimierungswerfer]

@Ju1He1 that is an msvc limitation, which apparently is known by @StephanTLavavej and it is being worked on a "long term solution" for that issue (standard library includes coming after import std), although nothing has been done in that regard for almost two years now. It can be worked around by including all standard library headers through a pre-compiled header, thereby ensuring that all their include guards are set (which seems to be the issue with the redefinition errors that would otherwise occur). Not the prettiest workaround, but until Microsoft decides this is worth looking into, it is the best we can do to migrate the C++ ecosystem to modules. Because we as consumers cannot just fork all libraries we depend on and migrate them too, before we can use import std.

[Ju1He1]

@Optimierungswerfer
Did you try it with latest clang + cmake?
Currently we are only using Modules with MSVC. Clang and GCC still use classic includes.
I'm not sure if this is a MSVC only issue or if other compilers have similar issues with the type collisons

[StephanTLavavej]

nothing has been done in that regard for almost two years now.

Sorry - we've been drowning in other work. (I am currently the only maintainer able to spend time on MSVC's STL.) This is still on my priority list of things to do, but it will mostly be compiler work, and the compiler team has also been busy with other tasks.

[Optimierungswerfer]

@Ju1He1 I did not try this with the other compilers, since we don't use them in our C++23 projects yet. My last knowledge is that msvc had the best module support so far. I can't tell whether the others caught up. This specific issue has been confirmed to be an msvc limitation though at some hard to find places (e.g. https://www.youtube.com/watch?v=Dk_C_E8AtRs&t=1990s or also a small side note in an old MSVC STL release note on GitHub) and unfortunately even the Microsoft support is confused about this as they told various people (myself included) multiple times that this was intended behavior, which led to even greater confusion.

@StephanTLavavej I genuinely appreciate the clarification on this, thank you. Really looking forward to a solution there, because this has been holding my team back for years now. We optimistically tried to implement C++20 modules in our solution, since MSVC has been listed as compliant with modules for years but they never seemed to work as advertised, even in 2023 and until this day. Seeing the msvc compiler team working on C++23 and already asking for priorities for C++26, while a major feature of C++20 isn't fully done yet is a bit frustrating.