Implement multifile VRT parallel raster output #241
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Greatly improve raster output performance for large runs on parallel filesystems by writing one TIF file per rank along with a VRT file. Writing to /path/to/foldername/filename.vrt creates the directory foldername if it does not exist and writes data to files /path/to/foldername/filename.%d.tif No changes are needed to read VRT files. The VRT TIF files are written in tiled format as there is no performance penalty when writing but a dramatic performance increase when reading small areas from very wide rasters.
Enables GDAL virtual file systems, in particular can write VRT to cloud storage if CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE=YES is set.
Use GDAL EQUAL() macro for case-insensitive string comparison. Old code would change extension to lower case in actual filename so passing FOO.TIF would create file named FOO.tif instead.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Greatly improve raster output performance for large runs on
parallel filesystems by writing one TIF file per rank along
with a VRT file. Writing to /path/to/foldername/filename.vrt
creates the directory foldername if it does not exist and
writes data to files /path/to/foldername/filename.%d.tif
No changes are needed to read VRT files.
The VRT TIF files are written in tiled format as there is no
performance penalty when writing but a dramatic performance
increase when reading small areas from very wide rasters.