Use a generator for instance segmentation masks #1604
Conversation
| predictions = [] | ||
| for pred, mask in zip(batch_predictions, batch_masks): | ||
| if class_filter and not self.class_names[int(pred[6])] in class_filter: | ||
| # TODO: logger.debug |
There was a problem hiding this comment.
Would a debug log be necessary? This PR deletes this TODO since it didn't seem especially helpful, but I can also add logs if needed to the generator.
| responses = [ | ||
| InstanceSegmentationInferenceResponse( | ||
| predictions=[ | ||
| predictions=( |
There was a problem hiding this comment.
Isn't it converted to list before rendering? Not sure if it help with memory usage.
There was a problem hiding this comment.
Currently, there are a handful of enterprise users who import InferencePipeline and run it without actually using any of the HTTP responses.
I'm closing this PR because it indeed does not solve the problem - the memory usage is much higher on instance segmentation execution, and reducing the memory used by these masks is a very small fraction of it (would require a much larger change to actually reduce the memory pressure on the Jetson).
Description
I am currently testing this changeset, but I notice a significant slowdown when loading large instance segmentation masks in memory - currently, it is not possible to access the class or confidence without also loading the entire mask as well. By using a generator, this code reduces memory usage by enabling access to the labels and classes of predictions without requiring the entire mask to be loaded also.
Type of change
Please delete options that are not relevant.
How has this change been tested, please provide a testcase or example of how you tested the change?
Currently profiling the change to see if it improves performance on edge devices (Jetson Orin 16GB)
Any specific deployment considerations
This does change the type from a list to a generator, but should not change any APIs or dependencies.
Docs
N/A