-
Notifications
You must be signed in to change notification settings - Fork 1
Cached objects
This page contains information about using the cached objects framework
This framework is designed to move the storage of large numpy objects to external files, using the facilities provided by numpy. It is in general, designed to serialize objects from a program into files. This will allow those objects to re-appear instantly, saving large calculation times. It should also allow such objects to persist across threading environments.
The Caching module appears to be the culprit behind some memory leaks, and seems to suffer from some efficiency issues. As such, it should not be used in its current state, and should be disabled if possible.
The caching module should transparently retrieve cached objects from files when they are needed by the program. It should NOT store such objects in memory, unless the loading process is prohibitively slow or the cache is especially small. It should also have the ability to request that objects be regenerated, and call the regeneration function as necessary. I'm not sure how the regeneration system should work.
As it is currently implemented, each cache understands how it should be generated. This means that caches call their generating functions. Unfortunately, this reduces some transparency in the system, and seems to make debugging harder.
Finally, the cache manager doesn't have a good sense of the status of base states.
The module needs a re-write with an eye for the appropriate API.