zsdt: high memory usage despite WithDecoderMaxMemory setting

[abarisani]

Hello,

I am trying to understand the reasons for high memory usage when decompressing within u-root.

I see that the decompression of a 15MB Linux kernel (resulting in a 66MB decompressed output) allocates 467 MB of Go memory.

The detected WindowSize by is 128MB and even enforcing the following results in the same figures:

zstdReader, err := zstd.NewReader(r, zstd.WithDecoderMaxMemory(0x8000000), zstd.WithDecoderMaxWindow(0x8000000))

I am not sure if this is a leak, bug or intended memory allocation.

Thanks!

[klauspost]

If there is no content-size there is no way it can know that it will not need the full window will not be needed. It allocates 1MB more so it doesn't have to continually move the window. I don't know where the rest is going.

Setting the max window will not really do anything, since it only is a check for the max window size. Similar for max memory only restricts the output size.

My main recommendation would be to just stick to the max 8MB window size when encoding. The bigger window really doesn't gain you much.

Probably WithDecoderConcurrency will also reduce memory a bit.

[abarisani]

How are you getting that number?

Through runtime.ReadMemStats (Alloc and/or Sys values) as we are debugging kernel extraction under:

https://github.com/usbarmory/go-boot

A recent kernel update caused an out of memory panic in runtime malloc and that prompted tracking down the cause.

The decompression of the Linux kernel image by u-root through zstd decompression has been identified as responsible for the large memory allocation.

[klauspost]

If you buffer or otherwise keep the uncompressed/compressed input in memory, that also adds to that.

Go overallocates 100% - so there is some of it.

You are welcome to do a standalone reproducer. You can check (runtime.Memstats).InUse or grab a memory profile.

[abarisani]

No, the 467 MB are purely zstd allocation, I already computed out everything else in the figure.

I will create a reproducer.

[abarisani]

Ah, I was wrong! Using an io.Discard writer for the Copy operation results in a memory allocation figure close to the prospected 128MB (132MB).

However using a bytes.Buffer for the receiving writer results in 380MB of allocation, despite the output being 66MB in actual size.

Therefore indeed this is not zstd fault, I am sorry for the noise.

[klauspost]

No worries. Glad you found the issue!