cat9kv : Unable to connect to qemu monitor (port 4000)

[htam-net]

Hello

Today, with upgrading to v0.68.0 (from 0.67.0), I can't launch more than one Catalyst 9kv VM/container in containerlab.
And also, minor update of the underlying Debian 12

When more than one, is launched, I get Unable to connect to qemu monitor (port 4000), retrying in a second. Until 60 retries hit and then stops the container/VM.

Here is the cmd executed (for one of them) ... and all are using the 4000 (and 5000) tcp ports for monitor and serial :
vrnetlab DEBUG qemu cmd: qemu-system-x86_64 -enable-kvm -display none -machine pc -monitor tcp:0.0.0.0:4000,server,nowait -serial telnet:0.0.0.0:5000,server,nowait -m 18432 -cpu host -smp cores=4,threads=1,sockets=1 -drive if=ide,file=/cat9kv-universalk9_16G_vga.17.15.01-overlay.qcow2 -overcommit mem-lock=off -boot order=cd -cdrom /config.img -device pci-bridge,chassis_nr=1,id=pci.1 -device virtio-net-pci,netdev=p00,mac=0C:00:c4:49:45:00 -netdev tap,id=p00,ifname=tap0,script=/etc/tc-tap-mgmt-ifup,downscript=no -device virtio-net-pci,netdev=p01,bus=pci.1,addr=0x2 -netdev socket,id=p01,listen=:10001 -device virtio-net-pci,netdev=p02,mac=0C:00:33:d2:56:02,bus=pci.1,addr=0x3 -netdev tap,id=p02,ifname=tap2,script=/etc/tc-tap-ifup,downscript=no -device virtio-net-pci,netdev=dummy1,id=dummy1,mac=0C:00:81:10:5e:00,bus=pci.1,addr=0x3 -netdev tap,ifname=dummy1,id=dummy1,script=no,downscript=no -device virtio-net-pci,netdev=dummy2,id=dummy2,mac=0C:00:15:bd:24:01,bus=pci.1,addr=0x4 -netdev tap,ifname=dummy2,id=dummy2,script=no,downscript=no -device virtio-net-pci,netdev=dummy3,id=dummy3,mac=0C:00:7b:a2:5a:02,bus=pci.1,addr=0x5 -netdev tap,ifname=dummy3,id=dummy3,script=no,downscript=no -device virtio-net-pci,netdev=dummy4,id=dummy4,mac=0C:00:d1:07:23:03,bus=pci.1,addr=0x6 -netdev tap,ifname=dummy4,id=dummy4,script=no,downscript=no -device virtio-net-pci,netdev=dummy5,id=dummy5,mac=0C:00:d0:f7:6e:04,bus=pci.1,addr=0x7 -netdev tap,ifname=dummy5,id=dummy5,script=no,downscript=no -device virtio-net-pci,netdev=dummy6,id=dummy6,mac=0C:00:7f:18:5c:05,bus=pci.1,addr=0x8 -netdev tap,ifname=dummy6,id=dummy6,script=no,downscript=no -device virtio-net-pci,netdev=dummy7,id=dummy7,mac=0C:00:86:7f:54:06,bus=pci.1,addr=0x9 -netdev tap,ifname=dummy7,id=dummy7,script=no,downscript=no

and the docker logs :

2025-06-03 07:35:54,357: base_driver    INFO opening connection to '127.0.0.1' on port '4000'
2025-06-03 07:35:54,358: base_socket    WARNING connection refused trying to open socket to 127.0.0.1 on port 4000 for address family AF_INET
2025-06-03 07:35:54,358: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 1)
2025-06-03 07:35:55,358: base_driver    INFO opening connection to '127.0.0.1' on port '4000'
2025-06-03 07:35:55,359: base_socket    WARNING connection refused trying to open socket to 127.0.0.1 on port 4000 for address family AF_INET
2025-06-03 07:35:55,359: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 2)
[..]
2025-06-03 07:36:53,385: base_driver    INFO opening connection to '127.0.0.1' on port '4000'
2025-06-03 07:36:53,386: base_socket    WARNING connection refused trying to open socket to 127.0.0.1 on port 4000 for address family AF_INET
2025-06-03 07:36:53,386: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 60)
Traceback (most recent call last):
  File "/launch.py", line 228, in <module>
    vr.start()
  File "/vrnetlab.py", line 1039, in start
    vm.work()
  File "/vrnetlab.py", line 913, in work
    self.check_qemu()
  File "/vrnetlab.py", line 927, in check_qemu
    self.start()
  File "/vrnetlab.py", line 393, in start
    raise QemuBroken(
vrnetlab.QemuBroken: Unable to connect to qemu monitor on port 4000

I have tried to add port 4000 on the EXPOSE of the DOCKERFILE for cat9kv and rebuilt the image, with the same result.

If I compare with c8000v, the "unable to connect" stopped after a dozen of retries.

Can you provide guidance to overcome this situation, please ?

NB : I have looked at the code f"tcp:0.0.0.0:40{self.num:02d},server,nowait" but it seems that self.num is always 0 in the case of cat9kv (and also c8000v).

Sincerely yours, Mathieu.

[jeremycdw]

IF you look higher in logs, can you see if you are seeing this error:

INFO STDERR: qemu-system-x86_64: -device virtio-net-pci,netdev=dummy3,id=dummy3,mac=0C:00:70:06:a6:00,bus=pci.1,addr=0x5: PCI: slot 5 function 0 not available for virtio-net-pci, in
use by virtio-net-pci,id=(null)

I hit this today after booting 2 nodes, the 3rd node threw this PCI error

[jaakoob]

I have observed the same behaviour. Depending on the time I start the labs, different amounts of devices actually start up.

The issue seems to be in the generation of random mac addresses where a bug leads to multiple vms getting the same mac addresses. I tried to get new random seeds by delaying the startup of each vm by a random time (using "startup-delay" in the clab.yml file). Though this does not seem to help.

[jaakoob]

It seems my initial analysis was not completely correct. Though I had devices with the same mac addresses, this is not the root of the problem. The root seems to be that with only one or two interfaces connected to a cat9kv node, there are duplicate device ids within the same pci bus. This leads to the qemu command failing and the telnet check failing.

[jaakoob]

The fix seems to be to add an offset of 1 to self.num_provisioned_nicsin line 645 in vrnetlab.py and rebuilding the containers.

I have not filed a pr since this seems to be in relation with PR #166 where it is assumed that the management interface is part of self.num_provisioned_nics which would make the offset irrelevant.

[hellt]

The #166 is not moving, as there are more inherent challenges with it.
If you draft a PR with the change, we can get someone else to try it out to confirm the fix is working.
Thanks!

[ericcgu]

I attempted to deploy one Nexus 9K and hit the same error. Should I open a new issue or keep the discussion here?

name: nexus

topology:
  nodes:
    nexus1:
      kind: vr-n9kv
      image: vrnetlab/cisco_n9kv:9300-10.4.3

2025-07-06 21:13:08,311: vrnetlab ERROR Unable to connect to qemu monitor (port 4000) Traceback (most recent call last): File "/launch.py", line 236, in <module> vr.start() 

[jaakoob]

Hi Eric, can you please try the above mentioned fix? Without the full error message, it is hard to determine if you are also running into the pci id problem or something else.

[ericcgu]

@jaakoob

I tested the fix. It is unrelated to the pci id problem.

I looked into the issue and I believe what is happening for me is that my VM ran out of memory. Cisco Documentation says that this needs at least 10G alone. I will try the nexus9k lite image and allocate 6 G of memory.

qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory

yaml
name: nexus

topology:
  nodes:
    nexus1:
      kind: vr-n9kv
      image: vrnetlab/cisco_n9kv:9300-10.4.3
      env:
        QEMU_MEMORY: "6144" 

[htam-net]

Hello everyone,
I have applied this fixed #351 (comment)

The fix seems to be to add an offset of 1 to self.num_provisioned_nicsin line 645 in vrnetlab.py and rebuilding the containers.

I also added different startup_delay (15s) to each cat9kv and now I have some strange results:

The "first" 2 switches cat9kv (in alphabetical order) crash still with the same error as before, but not the 3rd and 4th switches that boot happily... (Didn't tryed more switches)

Based on my design, I thought it could be to related to "port".
So I changed and connected the interface Gi1/0/1 from the first two switches in dummy mode (in containerlab : https://containerlab.dev/manual/topo-def-file/#dummy).
And now it works (I keep the startup_delay).

My only conclusion is : do not use Gi1/0/1 to connect stuff on the first two cat9kv ... Sorry for such poor explanation
By the way, and I use Gi1/0/1 on switch #3 and #4 without any crash ...

[jaakoob]

Hi @htam-net,

could you please check if you are able to use interface Gi1/0/1 regularly if you use #359 / https://github.com/jaakoob/vrnetlab/tree/fix-dummy-nic-pci-devid ?