HeatMap creation takes too long

[Husnain08]

@sven-mayer the Script number 4 which is named (_PY04-HeatmapCreation) is taking a very long time to run. I have started running that script on a single file one day ago and it has not completed yet. You has run that code on a computer that has 94 cores but mine has only 6 cores. Can you please provide the heatmap files so that we could move forward with the next files?

Thanks

[sven-mayer]

This script will generate around 5.7GB of files. There is not an easy way to just share these files. However, you might look a the scripts to optimize the processing further.

In any case, what do you mean by a single file? Did it generate any output files?

[Husnain08]

@sven-mayer by single file, I mean, a single pickle file. And no, this is still in processing even though I am trying to generate the output for only one activity ("Write"). I could look into this, but still, the computing power (only 6 cores) would not be enough to generate files for all the files and tasks.

I think the file size ( 5.7GB of files) is shareable. We see datasets in the range of TBs at kaggle and similar platforms. Even if I buy GPUs on AWS or GCP or IBM cloud, this could take me a while and the best way would be if someone share it to me.

Forgive my ignorance, but does uploading 5.7GB of files at Google Drive ( you could share the link and everyone would be able to download), requires some monetary investments? If it does, please let me know and I will be happy to cover the costs. I really appreciate your help @sven-mayer

[sven-mayer]

It should easily run with 6 Cores. We ran it before in less than 10. There is a multi threading it the process which will speed it up by processing them in paralle but that's all. 6 Cores should work. But by whatever you did I think it broak. Because it generates hundreds of 15 MB sized files.

But by your explanation there is not a single file yet?

[Husnain08]

@sven-mayer yes, exactly. Not a single file.

[sven-mayer]

As you modified the scripts, this might have caused a problem. You may want to test the script without any modifications.

[sven-mayer]

I just tested it with 6 cores by changing line 127 in file py/heatmap_methods.py to be: cpu_count = 6
One file takes about 7 sec

CELL OUTPUT:

Importing /P3_N6_walking.pkl
Done! (0.4337410000000004 sec.)
Creating heatmaps...
Using 6 CPU cores
Finger Thumb, Task WRITE, device_size [0.083 0.1593 0.0101]
Finger Index Finger, Task WRITE, device_size [0.083 0.1593 0.0101]
Finger Middle Finger, Task WRITE, device_size [0.083 0.1593 0.0101]
Finger Ring Finger, Task WRITE, device_size [0.083 0.1593 0.0101]
Finger Little Finger, Task WRITE, device_size [0.083 0.1593 0.0101]
Done! (5.576124 sec.)
Heatmaps saved to /heatmaps/walking/WRITE_highres/P3_n6_heatmaps.npy
-----
Importing /P19_S4_walking.pkl
Done! (0.3212679999999999 sec.)
Creating heatmaps...
Using 6 CPU cores
Finger Thumb, Task WRITE, device_size [0.07 0.137 0.0079]
Finger Index Finger, Task WRITE, device_size [0.07 0.137 0.0079]
Finger Middle Finger, Task WRITE, device_size [0.07 0.137 0.0079]
Finger Ring Finger, Task WRITE, device_size [0.07 0.137 0.0079]
Finger Little Finger, Task WRITE, device_size [0.07 0.137 0.0079]
Done! (5.214938000000002 sec.)
Heatmaps saved to /heatmaps/walking/WRITE_highres/P19_s4_heatmaps.npy
...
...
...
...
...

[Husnain08]

Hi @sven-mayer , just downloaded this repo freshly and tried again. The error is
"No module named heatmap_methods"... Can you point out what is causing this error? I did not touch a line

[sven-mayer]

I'm not quite sure how you can run a jupyter notebook in your console. But anyway, the model which is missing is in folder py/heatmap_methods.py which is also in this git repo.

HINT:
You may want to run the notebooks the propper way by executing jupyter notebook and then starting the notebook, for more information please see https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html

[Husnain08]

@sven-mayer everything works okay but now I am back to my main problem. Now the errors says

ImportError: DLL load failed: The paging file is too small for this operation to complete.

[Husnain08]

@sven-mayer can you please provide the files? I am still not getting the output.

[sven-mayer]

You will run into the same problem with the next scripts again. So it makes no sense to upload the files.

Please provide the full error log.