Added cloud storage utility to toolkit #19
Conversation
leela
changed the title
leela
force-pushed
the
18-add-cloud-storage-utility
branch
3 times, most recently
from
47d571a to
22fd201
Compare
anandology
left a comment
anandology
left a comment
There was a problem hiding this comment.
There are some features like copying a directory, downloading a whole directory etc. present in the storage.py of algoshelf/maverix that are not available here. It would be good to add them here.
Also, can you please test cases?
toolkit/storage.py
Outdated
| collection = bucket.objects.filter(Prefix=self.path) | ||
| return [obj.key for obj in collection.all()] | ||
|
|
||
| def read_dataframe(self): |
There was a problem hiding this comment.
I guess it is better have separate read_csv and read_paquet methods.
toolkit/storage.py
Outdated
| return False | ||
| raise | ||
|
|
||
| def delete(self, del_dir: bool=False): |
There was a problem hiding this comment.
The del_dir doesn't sound right. See what is used in pathlib for this kind of operation.
| """Read the contents of a path | ||
| """ | ||
| obj = self._object | ||
| return obj.get()['Body'].read() |
There was a problem hiding this comment.
If there is a read_text, then there should be a read_bytes as well. What about write? Shouldn't we have write_text and write_bytes methods?
There was a problem hiding this comment.
Also. i guess, [Body].read() returns bytes, not text. Please check.
leela
force-pushed
the
18-add-cloud-storage-utility
branch
from
22fd201 to
1397967
Compare
|
@anandology , It is still needs a proper testing, test cases and exception handling, I am on it currently. Meanwhile you can have a look at it and please pass your comments. |
leela
force-pushed
the
18-add-cloud-storage-utility
branch
9 times, most recently
from
8c917db to
0f6e99f
Compare
leela
changed the title
leela
force-pushed
the
18-add-cloud-storage-utility
branch
7 times, most recently
from
1e59f1e to
c02c02d
Compare
leela
force-pushed
the
18-add-cloud-storage-utility
branch
21 times, most recently
from
78692a6 to
85b0858
Compare
leela
force-pushed
the
18-add-cloud-storage-utility
branch
from
85b0858 to
d60c137
Compare
closes #18