IO

Structured IO

The objective of this tutorial is to illustrate a user's interaction with the IO utility of the AllScale API. This tutorial shows how to output simulation data to a file. The full source code can be found in the Structured IO tutorial sources directory.

Overview

• Storing - Unstructured storing of simulation data
• Restructure - Load and structure the simulation data

Storing

The values of the simulation are stored in a grid. To keep this tutorial simple the values stored in the grid are integers.

// -- size of the grid --
const int size = 10;

// -- grid --
using Grid = data::StaticGrid<int, size, size>;

Each number in the grid is unique after the initialization:

// -- initialize the grid --
algorithm::pfor(Grid::coordinate_type{}, grid.size(), [&](const auto& p){
    grid[p] = p[0] * size + p[1];
});

The instance of the FileIOManager is needed to write to files or read from them. Using this instance we can create new entries.

const std::string binary_filename = "binary";

FileIOManager& manager = FileIOManager::getInstance();

Entry binary = manager.createEntry(binary_filename, Mode::Binary);

The above code gets the FileIOManager instance and creates a new file entry binary to which the data is stored in binary mode.

auto output_stream = manager.openOutputStream(binary);

// write data to file in parallel
algorithm::pfor(Grid::coordinate_type{}, grid.size(), [&](const auto& p){
    output_stream.atomic([&](auto& out) {
        out.write(p);
        out.write(grid[p]);
    });
});

The output_stream allows to write to the binary file. Additionally to the data we store the position of the data in the grid. This information would be lost otherwise because there are no guarantees on the order of the data output in the binary file. The atomic only ensures that there are no simultaneous writes to the file. When simulating there may be other values that are important to write to the file to reconstruct the simulation values later. One of them could be the simulation time.

After all the data was written to the file one should close the output_stream.

manager.close(output_stream);

Restructure

The restructuring of the file takes place after the simulation finishes. Therefore the data of the binary file that was created during the simulation gets restructured and written in a new file.

// -- read data --
Grid grid;

FileIOManager& manager = FileIOManager::getInstance();

Entry binary = manager.createEntry(binary_filename, Mode::Binary);

// -- read data --
auto in = manager.openInputStream(binary);
for(int i = 0; i < size * size; i++) {
    auto coord = in.read<Grid::coordinate_type>();
    auto count = in.read<int>();
    grid[coord] = count;
}

For each entry in the file we first read the coordinate and then the value. The value gets inserted at the given coordinates of the grid.

The final output file is in text mode to make it human readable. Further there is no parallel writing because the file has to contain the results in the right order. The output file is a CSV file containing the values inserted in the previous section.

const std::string text_filename = "text";

Entry text = manager.createEntry(text_filename, Mode::Text);

auto output_stream_text = manager.openOutputStream(text);

for(int i = 0; i < size; i++) {
    for(int j = 0; j < size; j++) {
        output_stream_text << grid[{i, j}];
        if(j != size - 1)
            output_stream_text << ", ";
    }
    output_stream_text << "\n";
}