Skip to content

ValueError: mmap length is greater than file size #198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
maciejmajek opened this issue Aug 30, 2024 · 5 comments · May be fixed by #199
Open

ValueError: mmap length is greater than file size #198

maciejmajek opened this issue Aug 30, 2024 · 5 comments · May be fixed by #199

Comments

@maciejmajek
Copy link

When running the automatic_model_training.ipynb locally, I've come across a problem with the training script --train_model.
I was able to solve this by adding + to the mmap_mode.

self.data = {label: np.load(fl, mmap_mode='r+') for label, fl in data_files.items()}

env:
python3.10

Stack trace

File "openwakeword/openwakeword/train.py", line 845, in
batch_generator = mmap_batch_generator(
File "/home/mmajek/projects/internal/rai/openwakeword/openwakeword/data.py", line 788, in init
self.data = {label: np.load(fl, mmap_mode='r') for label, fl in data_files.items()}
File "/home/mmajek/projects/internal/rai/openwakeword/openwakeword/data.py", line 788, in
self.data = {label: np.load(fl, mmap_mode='r') for label, fl in data_files.items()}
File "/home/mmajek/.cache/pypoetry/virtualenvs/rai-qH6Oly3A-py3.10/lib/python3.10/site-packages/numpy/lib/npyio.py", line 453, in load
return format.open_memmap(file, mode=mmap_mode,
File "/home/mmajek/.cache/pypoetry/virtualenvs/rai-qH6Oly3A-py3.10/lib/python3.10/site-packages/numpy/lib/format.py", line 945, in open_memmap
marray = numpy.memmap(filename, dtype=dtype, shape=shape, order=order,
File "/home/mmajek/.cache/pypoetry/virtualenvs/rai-qH6Oly3A-py3.10/lib/python3.10/site-packages/numpy/core/memmap.py", line 268, in new
mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
ValueError: mmap length is greater than file size
@maciejmajek maciejmajek linked a pull request Aug 30, 2024 that will close this issue
Open
@adgud
Copy link

adgud commented Apr 13, 2025

Hey, this fixed this error for me too, however I encounter another one right after that:

/mnt/openWakeWord/venv/lib/python3.10/site-packages/torch_audiomentations/utils/io.py:27: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
  torchaudio.set_audio_backend("soundfile")
Traceback (most recent call last):
  File "/mnt/openWakeWord/notebooks/openwakeword/openwakeword/train.py", line 867, in <module>
    X_val_fp = np.load(config["false_positive_validation_data_path"])
  File "/mnt/openWakeWord/venv/lib/python3.10/site-packages/numpy/lib/npyio.py", line 438, in load
    raise ValueError("Cannot load file containing pickled data "
ValueError: Cannot load file containing pickled data when allow_pickle=False

Did you manage to work around after that one?

@maciejmajek
Copy link
Author

Looks like numpy changed the default behavior of np.load in 1.16 version. In 1.15 it's still allow_pickle==True by default. I'd suggest either downgrading np version (which might create conflicts) or directly adding allow_pickle=True in np.load(config["false_positive_validation_data_path"])

@adgud
Copy link

adgud commented Apr 14, 2025

Sadly, downgrading numpy is not possible as other packages (like acoustics==0.2.6) requires numpy >= 1.18.

Adding allow_pickle=True is something I tried already, but it leads to a different error:

/mnt/openWakeWord/venv/lib/python3.10/site-packages/torch_audiomentations/utils/io.py:27: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
  torchaudio.set_audio_backend("soundfile")
Traceback (most recent call last):
  File "/mnt/openWakeWord/venv/lib/python3.10/site-packages/numpy/lib/npyio.py", line 441, in load
    return pickle.load(fid, **pickle_kwargs)
EOFError: Ran out of input

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/openWakeWord/notebooks/openwakeword/openwakeword/train.py", line 868, in <module>
    X_val_fp = np.load(config["false_positive_validation_data_path"],allow_pickle=True)
  File "/mnt/openWakeWord/venv/lib/python3.10/site-packages/numpy/lib/npyio.py", line 443, in load
    raise pickle.UnpicklingError(
_pickle.UnpicklingError: Failed to interpret file 'validation_set_features.npy' as a pickle

@maciejmajek
Copy link
Author

Looks like you are in a pickle 🤷‍♂ :D. No idea, seems like the project is getting stale.

@adgud
Copy link

adgud commented Apr 17, 2025

Yeah shame it appears to be no longer maintained. Did you have any luck generating own wake words in any different way? The Google Colab notebook does not work for me either (fails at onnx to tflite conversion).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: allow numpy to extend filesize for larger memmaps. maciejmajek/openWakeWord
2 participants
@adgud @maciejmajek