Skip to content

Optimize dependency tree holding on to modifier objects #245

@djhoese

Description

@djhoese

Problem description

We have a couple cases in SatPy where modifiers (which act like compositors) cache intermediate data to increase performance. A lot of these cases cache the data in a class attribute, hopefully with a WeakValueDict. The issue with this is that the class attribute always exists and will hold on to these arrays as long as python doesn't take back the memory (remove the item from the dict).

The solution would be to create the cache in the __init__ of the modifier class and make sure when the modifier is created that it is reused every time. I can't remember right now how modifiers are stored/created in the dependency tree search process.

The other needed optimization to make this completely work is to clear Nodes from the dependency tree when all parents have been created and the children are no longer needed. Care would need to be taken with the implementation of this. If we go the route of actually removing the entire child node then we need a way to know that the parent nodes are not actually "leaves" of the tree or else the Scene will think it should be loading these from a Reader. We could add an attribute or piece of Node.data to each Node to tell whether or not it is loaded which would make the clearing out of nodes pretty fast. The Node class right now is fairly generic, this may require making them a little more specific to our needs.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions