nsqd: deferred messages not deferred with --mem-queue-size=0

[hunterhug]

I use nsq 1.2.1 and this go-nsq sdk , but defer message has problem, anyone try it?

[hunterhug]

my docker deploy

version: '3'
services:
  mynsqlookupd:
    container_name: "mynsqlookupd"
    image: nsqio/nsq:v1.2.1
    command: /nsqlookupd --http-address=0.0.0.0:5161 --tcp-address=0.0.0.0:5160
    restart: always
    ports:
      - "5160:5160"
      - "5161:5161"
  mynsqd:
    container_name: "mynsqd"
    image: nsqio/nsq:v1.2.1
    command: /nsqd --broadcast-address=10.0.5.2 --lookupd-tcp-address=10.0.5.2:5160 --tcp-address=0.0.0.0:5150 --http-address=0.0.0.0:5151 --data-path=/data --mem-queue-size=0
    volumes:
      - /var/lib/docker/ebo/nsq/data:/data
    depends_on:
      - mynsqlookupd
    ports:
      - "5150:5150"
      - "5151:5151"
    restart: always
  mynsqadmin:
    container_name: "mynsqadmin"
    image: nsqio/nsq:v1.2.1
    command: /nsqadmin --lookupd-http-address=10.0.5.2:5161 --http-address=0.0.0.0:5171
    depends_on:
      - mynsqlookupd
    ports:
      - "5171:5171"
    restart: always

and I use this:

func (w *Producer) DeferredPublish(topic string, delay time.Duration, body []byte) error {
	return w.sendCommand(DeferredPublish(topic, delay, body))
}

the message which can not defer.

then I chaneg nsqio/nsq:v1.2.1 to nsqio/nsq:v1.2.0, it works!

nsq has bug? or this sdk has bug?

[ploxiln]

Deferred messages forget their deferred state/timing when stored to the disk queue, this is a limitation of the current disk format. So using --mem-queue-size=0 is a problem. I think that previously deferred messages were always kept in memory until nsqd shuts down, I don't exactly recall if/how/when this changed...

[ploxiln]

I'm pretty sure this is nsqd related so I'll move this to the other repo

[ploxiln]

I think this change in behavior was caused by #1159

Previously, the "memoryMsgChan" would exist, but be unbuffered, but this allowed messages to instantly pass through the topic to each channel's deferred pqueue. But now, that chan does not exist when --mem-queue-size=0 so all messages of a topic must go through the "backendChan" (disk queue) before going to any channels, and that loses the deferred metadata.

It seems like, if any channels exist, the topic "memoryMsgChan" should exist (and should be buffered to avoid random interleaving of messages going to backendChan and through memoryMsgChan). Hmm. Anyway, --mem-queue-size=0 never worked particularly great for nsqd, to work-around this I recommend using --mem-queue-size=20 or more.

[hunterhug]

I think this change in behavior was caused by #1159

Previously, the "memoryMsgChan" would exist, but be unbuffered, but this allowed messages to instantly pass through the topic to each channel's deferred pqueue. But now, that chan does not exist when --mem-queue-size=0 so all messages of a topic must go through the "backendChan" (disk queue) before going to any channels, and that loses the deferred metadata.

It seems like, if any channels exist, the topic "memoryMsgChan" should exist (and should be buffered to avoid random interleaving of messages going to backendChan and through memoryMsgChan). Hmm. Anyway, --mem-queue-size=0 never worked particularly great for nsqd, to work-around this I recommend using --mem-queue-size=20 or more.

Thanks your suggestion!

Now I use --mem-queue-size=100 for my production env., but if my messages over 100, is it will store in disk too? And if nsq shutdown and restart again, the message store in disk recover to memory, the defer message meta loss too?

So odd...

[ploxiln]

correct ... it's a known limitation, addressing it would require something like #1170 ... I'm thinking about it :)

[ploxiln]

if my messages over 100, is it will store in disk too?

Not deferred messages, I think: both the topic and the channels have a mem-queue, but the topic mem-queue is not actually used, except when there are zero channels, and also just for a millisecond or two when distributing copies of messages to channels (with metadata). Then, in the channels, deferred messages do not go to the disk-queue, they go to a separate in-memory deferred queue, I think (then when their time is up they go to the normal mem-queue or disk-queue).

if nsq shutdown and restart again, the message store in disk recover to memory, the defer message meta loss too

correct, this is the bug we already knew about, which #1170 addresses

[hunterhug]

if my messages over 100, is it will store in disk too?

Not deferred messages, I think: both the topic and the channels have a mem-queue, but the topic mem-queue is not actually used, except when there are zero channels, and also just for a millisecond or two when distributing copies of messages to channels (with metadata). Then, in the channels, deferred messages do not go to the disk-queue, they go to a separate in-memory deferred queue, I think (then when their time is up they go to the normal mem-queue or disk-queue).

if nsq shutdown and restart again, the message store in disk recover to memory, the defer message meta loss too

correct, this is the bug we already knew about, which #1170 addresses

Avoid restart nsq will cause defer message send as common message.

I has below temporary way to patch my business requirements of high availability defer message feature :

1. set --mem-queue-size=n ，which n>0 but not --mem-queue-size=0.
2. I must add some logic in my own program to deal those message, put them into a topic with defer message again.

for others' reference.

and hope nsq will soon fix this bug 💪