Add memmap to gambit #5
Conversation
| self.logger.debug(f"Reading distance matrix index from {index_filename}") | ||
| with open(index_filename, 'r') as f: | ||
| dist_matrix_index_labels = [line.strip() for line in f] | ||
| pairwise_distances_index = pandas.Index(dist_matrix_index_labels) |
There was a problem hiding this comment.
note to self: will need to ensure index is being appropriately assigned
|
|
||
| # Memory-map the pairwise distances file instead of loading it | ||
| self.logger.debug(f"Memory-mapping distance matrix from {self.pairwise_distances_filename}") | ||
| pairwise_distances_matrix = np.memmap(self.pairwise_distances_filename, dtype='float32', mode='r') |
There was a problem hiding this comment.
note for future: if RAM/disk is still an issue, we could convert the matrix to whatever integer corresponds to the significant figures we need and change the matrix to "int16" or whatever. for example, if we only need two significant figures between 0-1, we could multiply the matrix by 100 and use int8, which would reduce memory by 75%.
| species['ngenomes'] = species['ngenomes'].astype(int) | ||
| # Fill any potential NaN values with 0 before casting to integer. | ||
| # This makes the script more robust against unexpected missing values. | ||
| species['ngenomes'] = species['ngenomes'].fillna(0).astype(int) |
There was a problem hiding this comment.
are the NaN's a) intentionally introduced/permitted upstream, or b) are they an artifact of potential errors?
- a) no change requested
- b) it would be ideal to address those upstream and raise an error here if detected
| min_inter[i, j] = min_inter[j, i] = mi | ||
| #Add Species data | ||
|
|
||
| # If the assembly accessions list is not empty, calculate diameter and min_inter |
There was a problem hiding this comment.
note to self: review further following receipt of validation dataset
|
Dockerfile failing to build. Tests are corrupted |
|
gambitdb-curate argparse script description needs to be updated with new required inputs |
Add numpy memmap as an option to process data coming from npy file instead of a csv for the pairwise distance matrix.