fix: create multiple dataframes from same CSVReader #44
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for fixing this! Reading the file as a stream was not "accidental" but on purpose to reduce the memory footprint. We should check the best trade-off to solve the issue of multiple readings while keeping the possibility of processing iteratively the CSV (e.g., large CSV where I need to access only one column). Maybe we can have a CSVStreamReader and CSVReader or something similar? |
|
@AuPath I pushed an attempt to introduce CSVStreamReader and refactor duplicated operations in CSVReaderAbstract. Please review the latest commit |
marioscrock
approved these changes
Apr 9, 2025
|
Fix also setOnlyDistinct behaviour in CSVReader and other readers (cf. #45) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The behaviour is now to read the whole CSV file (or String) into a
List<NamedCsvRecord>and not in a stream as was previously (accidentaly) the case. While this does work and means that #38 is corrected this also means that the whole file is always kept in memory.This does not necessarily make sense as i might have a huge CSV file where i am interested in a single column that i can obtain with the
getDataframe(String... columns)method. While the created dataframe is small i still have the whole file in memory, even if no other dataframe with more than that column is ever created.We should discuss possible optimizations for this scenario.