About address conflict, yarp clean etc

[randaz81]

This QA explains what the message address conflict means and how to fix related issues.

[randaz81]

Yarp legends and superstitions:

• killing a module. It may happen that you need to kill your own yarp module because it is not more responding. Obviously, if you kill it with a -9 signal (e.g. killall -9 mymodule), the module will close abruptly without closing the yarp ports that it previously opened. Thus, if you execute the command yarp name list you will be still able to see these port listened by the yarp server. Is this bad? yes and no. Of course, it will make the list of yarp ports more difficult to be read, and you may think that a process owing that ports is still running. However, this is not related to the infamous address conflict issue and it is not preventing any application to run. If you restart the module that you just killed, it will simply reopen those ports without any problem.
  PS: Do not forget to close any opened port in the destructor of your module class. It's a bad programming habit to terminate the process without cleaning up allocated memory and closing opened ports!

• yarp clean. If you want to cleanup the list of opened ported displayed by the command yarp name list, you may execute the command yarp clean <timeout_in_seconds>. Is it useful? yes, this command will ask the server to verify if a port is still kept opened by a process, and it will remove it from the list if it does not answer within the specified timeout. Can this command fix an address conflict issue? No, this is a completely different issue. It will be not fixed issuing a yarp clean command. Is this command dangerous? It might be dangerous. If you specify a very small timeout, the port may be not able to respond in time and it will removed by the list! If you are working on slow network this may be extremely dangerous: never use a timeout smaller than 0.1s and never use this command on wireless network that may have temporary disconnections. If yarp clean unregisters core ports, such as those opened by yarprun --server, you might be not able anymore to stop a process using yarpmanager.

• stopping and restarting the yarp name server This is will not solve any address conflict issue and might generate new problems. For example, if you forget to stop a running yarp module: 1) you will not be able to stop anymore using the yarpmanager gui. 2) the running application will generate additional address conflicts (see explanation below). Ok, but I need to restart the yarp nameserver for a different reason. What should I do? First of all terminate all modules which are running on yarpmanager and on all the system terminal (check carefully on all the machines connected to yarp network.) Then, stop yarpmanager and all the yarprun --server services. They might be launched manually or through the cluster.py gui. Finally stop the yarp nameserver.

What an address conflict means.

• It means that another process is currently owning the socket that yarp server is trying to assign to the port you want to open. yarp ports are just a human-readable shortcut to an ip address (e.g. 10.0.0.10) and a socket (e.g. 10122). When two ports are connected, the machine internally establish a connection between the two ip address and the two sockets. How yarp server assigns a socket to a newly created yarp port? It's very simple. It has a base address (by default 10000), and a counter that is progressively incremented. When a new port is opened/closed, the corresponding socket is marked by the nameserver as occupied/free. However the name server assigns a new socket to a yarp port, it is not able to detect if a socket is already occupied by: 1) a socket of a non-yarp application 2) a socket corresponding to a port of yarp application which has been not registered on the nameserver (this happens if you restarted the nameserver, but some yarp application is still running!)

How to solve an address conflict.

1. never stop the nameserver.
2. check the socket of the problematic port printed in the address conflict message. (e.g. 10107)
3. on linux use the command lsof | grep <socket_number>. On windows use the command netstat -a | findstr <socket_number>. The command will display the name of the process which is currently owning the socket. The command has to be executed on the machine on which the address conflict message is displayed. If you execute the command on a different machine, the problematic socket may be shown as free, because each machine has it own sockets.
4. You now know the process responsible of the issue and you have to terminate it.
   1. If this process is a yarp process, this means that someone restarted the nameserver without terminating all running yarp processes, so there are now occupied socketes not registered by the nameserver. Another possibility is that a port of a running yarp process was unregistered by a yarp clean command executed with a very small timeout or during a temporary disconnection of the physical network. To solve the issue you need to kill the process and check there are not additional "orphan" yarp modules. On linux you can use command like: ps aux | grep -i yarp to check if some yarprun, yarpdev & C is currently running in background.
   2. if the process is not a yarp process. This is pretty uncommon since the base default address for yarp port is 10000+, an area of sockets that is typically not used by system processes. However, if a third party application has been written to use a socket in that range, let's say 11000), you might experience problems when the nameserver tries to register a new port on that socket (btw, this means that there are already 1000 ports opened on your yarp network!). In this case you may consider changing the yarp default base address to different number using the yarp conf command on all the machines of network.

Last resort:

• The script icub-main\admin\scripts\cluster-clean.sh might be helpful to terminate all yarp processes running on a cluster.
• If you restarted the nameserver and you forgot to stop all yarp process on a large cluster of machines... you might consider rebooting all the machine belonging to the cluster. If the cluster is huge, this drastic solution might be faster than hunting the problematic running processes.

[randaz81]

Quick recap on how Yarp ports work:

• When in your application you create a yarp port with port::open(), the operating system creates a socket to manage the network communication at low level. This socket is a number, e.g. 10123.

• The yarp name server associates the socket number to a string (the yarp port name) and stores it into an internal database.

• The command yarp name list just prints the contents of this database.

• When a process terminates gracefully, the operating system releases the socket(*). The association with the yarp mnemonic string (the port name) is removed from the yarp name server database only if the port::close() method is called during the closure of the process.

• If a process is not terminated gracefully (i.e. segfault, killed, etc) in any case the operating system releases the socket. It cannot exist an open socket without a process that owns it. However, if port::close() is not called, the yarp name server does not de-register the entry from the database. This means that you can still see the port with the command yarp name list. This is not a problem, because a newly opened port will ask the OS for a new socket, and a new entry will be recorded in the database, eventually overriding the previous one.

• yarp clean can be useful to clear the results of yarp name list from the aesthetical point of view, i.e. to reduce the size of the printed output by removing ports that do not have anymore an associated socket.

• In no cases yarp clean can clear/close an open socket.

• Again, sockets are managed by the operating system, so can they can be closed only by terminating the process which owns them.

---

(*) One notable exception is python: sockets are not released immediately after the termination of the script, but when the python garbage collector is triggered (sometime after ~60 seconds from the script termination)