feat(buffer_memory): allow buffer spillover #510
Conversation
alecmerdler
force-pushed
the
memory-buffer-spillover
branch
from
e4ec353 to
16fe1d0
Compare
alecmerdler
changed the title
internal/impl/pure/buffer_memory.go
Outdated
|
|
||
| for (m.bytes + extraBytes) > m.cap { | ||
| if m.spilloverEnabled { | ||
| return component.ErrMessageTooLarge |
There was a problem hiding this comment.
We can introduce a new error type for spillover, if that makes sense.
There was a problem hiding this comment.
Yeah I think it would be better to add a different error message - will be easier to debug too.
There was a problem hiding this comment.
Done in e36405d and rebased.
|
@gregfurman @jem-davies Any thoughts on this approach? I'm happy to make any changes. |
Adds the 'spillover' boolean field to the 'memory' buffer component (default false). When true, any incoming message batch that will put the buffer above its configured 'limit' will be dropped. The motivation for this is to support input sources that do not gracefully handle backpressure and where the desired delivery guarantees are best effort.
alecmerdler
force-pushed
the
memory-buffer-spillover
branch
from
16fe1d0 to
2dafcb5
Compare
alecmerdler
force-pushed
the
memory-buffer-spillover
branch
from
2dafcb5 to
e36405d
Compare
|
Fixed the tests |
|
@alecmerdler 👋 Thanks for the contribution! While I'll give this proper review a bit later today, this PR got thinking whether the buffer:
memory:
limit: 1000
drop_on:
- check: this.errored()
- check: this == ""
# etc ...Could be over-engineering but am curious to hear your thoughts 🤔 |
|
Adding conditional buffering does seem like overkill and starts to mix concerns. What's missing for my use case is having any layer between inputs and buffers where we could drop on backpressure. Either way, since the |
3 participants
Adds the
spilloverboolean field to thememorybuffer component (default false). When true, any incoming message batch that will put the buffer above its configuredlimitwill be dropped.The motivation for this is to support input sources that do not gracefully handle backpressure and where the desired delivery guarantees are best effort.
Resolves #509