Memory leak in parser?

[ArcherSmythe]

I am loading a JSON file that is 150 MB long. I'm using this code for learning how to load the JSON, but there might be a memory leak?

#include "elite.hpp"
#include <fstream>
#include <stdlib.h>
#include <string>

int main( int argc, char * argv[] )
{

    system("ps aux | grep -E \"RSS|starfinder\"");

    elite::StarSystemVec example;
    
    if ( argc == 2 )
    {
        std::string json_str;
        json_str.resize(152651435);  // Magic number is the size of the file.
        
        FILE * infile = fopen( "../../samples-json.txt", "r" );
        if ( nullptr == infile )
        {
            printf( "Failed to open infile.\n" );
            return 1;
        };
        
        json_str[json_str.size() - 1] = 0;
        fread( json_str.data(), 1, json_str.size(), infile );
        fclose( infile );

        if ( argv[1][0] == 'y' )
        {
            example = nlohmann::json::parse( json_str );
        }
        else
        {
            // This does generate a warning!
            nlohmann::json::parse( json_str );
        }
    }
    
    else
    {
        std::string json_str;
        std::ifstream infile( "../../samples-json.txt" );
        example.resize( 300 );  // I have 300 elements in the file.
        std::getline( infile, json_str ); // consume container start
        for ( size_t idx = 0; idx < example.size(); ++idx )
        {
            std::getline( infile, json_str );
            size_t len = json_str.length();
            if ( json_str[len-1] == ',' )
            {
                json_str[len-1] = ' ';
            }

            example[idx] = nlohmann::json::parse( json_str );
        }
    }

    system("ps aux | grep -E \"RSS|starfinder\"");

    return 0;
}

Before reading the file, the application uses 6 MB of RAM, displayed at the first system call.

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
archer    200594  0.0  0.0   6400  3412 pts/2    S+   19:08   0:00 ./starfinder

If I pass y as the argument to the program, the program loads the entire file into one string and passes it to the JSON parser. If I print the memory usage after that, I see the program uses 1020 MB of RAM.

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
archer    200586 98.0  6.6 1022788 1019712 pts/2 S+   19:08   0:01 ./starfinder y

If I pass n as the argument to the program, the program loads the entire file into one string and passes it to the JSON parser. However, the resulting vector of structs returned by the parser doesn't get stored in example, so it just gets deleted. When I do this, the program uses 980 MB of RAM.

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
archer    200719 97.1  6.3 984636 981528 pts/2   S+   19:14   0:01 ./starfinder n

Finally, if I don't pass any arguments to the program, the loop in the else block reads one element (line) at a time. There are 300 elements (lines) in the file. When I print the RAM usage after that loop, I see the program uses 85 MB.

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
archer    200709 96.1  0.5  87744 84880 pts/2    S+   19:13   0:01 ./starfinder

So, if I'm interpretting this correctly, the parser seems to retain something (up to 1000 MB of "something") after the call to json::parse returns. If I manually pass significantly shorter strings that state retained by the parser is an order of magnitude smaller. Is there some way to tell json to clear any state after parsing?

Thanks,
Archer

[nlohmann]

I don't believe there is a memory leak: we execute the whole test suite with Valgrind and ASAN, see https://json.nlohmann.me/community/quality_assurance/#dynamic-analysis.

[ArcherSmythe]

Then how do I tell the parser to free its memory? Just because it clears its memory on destruction doesn't mean there's no leak: if it doesn't destroy a temporary variable before the parse function returns, that's still 1 GB of RAM that has to get swapped to disk if the application allocates 8 GB for actual processing. (Where 8 GB is the amount of available RAM.)

Edit:

Examples of two types of leak. First is the obvious one.

{
    //  Allocate 1 GB.
    char * p_char = (char *) malloc( 1 << 30 );

    //  Overwrite the pointer, so that we don't have access to the allocated memory any more.
    p_char = NULL;

    //  When this program exits, valgrind will report a 1 GB leak.
}

And then there's leaking via a variable that should be destroyed before actual application processing but isn't.

{
    //  Store 1 GB of JSON in a string.
    std::string json_str = read_file( "file_with_1GB_of_json.txt" );

    //  Parse that string.
    MyJsonStruct example = nlohmann::json::parse( json_str );

    //  Application processes the data.
    process_data( example );

    //  When this program exits, valgrind will report no leaks, since json_str will free its memory
    //  when it is destroyed.  However, I don't need json_str after parsing it and storing it in
    //  MyJsonStruct example. Because I don't destory json_str before calling process_data,
    //  process_data can't reuse that 1 GB of RAM for its processing.
}

To fix the second leak, I need to do something like this:

{
    // Uninitialized JSON struct, to be populated later.
    MyJsonStruct example;

    {
        //  Store 1 GB of JSON in a string.
        std::string json_str = read_file( "file_with_1GB_of_json.txt" );

        //  Parse that string.
        example = nlohmann::json::parse( json_str );

        //  Force json_str to go out of scope, destroying it and freeing its memory.
    }

    //  Now, process_data can use the RAM that had been occupied by json_str.

    //  Application processes the data.
    process_data( example );
}