Lost writes using AsyncCallbackResponse during low memory

[jonny5532]

Platform

ESP32

IDE / Tooling

Arduino (IDE/CLI)

What happened?

When sending a 16kb response in three parts, using AsyncCallbackResponse, the middle one sometimes goes missing (so the received data is ~5k shorter than expected - the page finishes but with ~5k fewer bytes than the Content-Length indicated).

This seems to be due to this:

ESPAsyncWebServer/src/WebResponses.cpp

Line 476 in 80af245

_writtenLength += request->client()->write((const char *)buf, outLen);

where the code assumes that the whole buffer got written (it frees it after), even if it was only partially sent. Ideally it could signal to the AsyncCallbackResponse that it needs to rewind, but being in the Abstract superclass it can't access _filledLength to decrement it.

AsyncTCP's write(...) allocates memory by default, and this system is very memory constrained, so that is probably why it is not consuming the whole buffer (probably not any of it). It looks like the code above does check space() though, but that doesn't take into account the free memory that the write() might need to allocate.

Maybe this is a situation you don't want to handle - as there's no guarantee there'll ever be enough free RAM? In practice it works if I rewind (eg. by adding a rewind method which the AsyncCallbackResponse can implement to roll back _filledLength).

Thanks!

Stack Trace

N/A

Minimal Reproductible Example (MRE)

This is the essence of it, but could build a full MRE if needed?

String *largeResponseString = new String("16k chars...");

AsyncWebServerResponse *response = request->beginResponse(
  "text/html",
  largeResponseString->length(),
  [](uint8_t *buffer, size_t maxLen, size_t alreadySent) -> size_t {
    // Calculate how much we can send in this chunk
    size_t remaining = largeResponseString->length() - alreadySent;
    size_t toSend = (remaining < maxLen) ? remaining : maxLen;

    // Copy the data from the content string to the buffer
    memcpy(buffer, largeResponseString->c_str() + alreadySent, toSend);
    // Return the number of bytes sent
    return toSend;
  },
  (AwsTemplateProcessor)nullptr  // No template processor needed
);
request->send(response);

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[mathieucarbou]

Good point.
The entire library (also at other places) is made in a way to degrading itself to avoid crashing the ESP as much as possible
So this is one of the thing that can happen yes.
Under memory pressure, there will be other places were espasync will degrade itself, like during request parsing for example.

This behavior was chosen by the original library author @me-no-dev.

For that specific point, yes if you could build a MRE / Example and put it in the example folder and send a PR. That will be a good use case to start a discussion also.

CC @willmmiles

[willmmiles]

CC @willmmiles

@mathieucarbou You pegged me all right: I did indeed solve this one in the WLED fork. Graceful handling of low memory conditions was the overarching goal behind the changes in that project. You can build a test case by consuming internal DRAM such that client()->space() > heap_caps_get_largest_free_block(MALLOC_CAP_INTERNAL | MALLOC_CAP_DEFAULT) - essentially arranging that there isn't enough heap available to allocate the full 2*MSS that the TCP connection could permit.

The solution there ended up being fairly complex:

• First, keep the output buffer around if it doesn't send completely; track how much of it is left
• Second, if there's leftover content in the output buffer at the start of an _ack call, send it first before considering anything else
• Third, double-check that there's enough space for the LwIP buffers when allocating the output buffer -- because there's no point in AsyncAbstractResponse allocating a large output buffer if the TCP stack can't then allocate a pbuf to send.

I did not opt to shrink the buffer to remove the successfully written data, though this could have been implemented as another solution for reducing ongoing memory load.

Unfortunately I'm a bit oversubscribed at the moment and I won't be in a position to PR these changes upstream anytime in the near future (sept-oct time frame maybe). If anyone else wants to take a stab at it, one key thing to be aware of there is the use of class DynamicBuffer, which is essentially a shallow replacement for std::vector<char,default_init_allocator> that permits allocation failure without exceptions. (Our project requires graceful handling of memory allocation failures on ESP8266, which doesn't support C++ exceptions, so std containers cannot be used. default_init_allocator is a construct that allows a vector to be resize()d without zero-initializing the contents, so it's slightly faster when used for transient buffers.)

[willmmiles]

Also: I believe there's a parallel issue with incomplete writes in the websockets layer should you find yourself in a condition where client()->space() > heap_caps_get_largest_free_block(MALLOC_CAP_INTERNAL | MALLOC_CAP_DEFAULT), so a message can be only partially sent. I left a TODO there but never implemented a solution.

[mathieucarbou]

Thanks @willmmiles !
I was pretty sure you got it handled ;-)

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity.