Grafana dashboard does not detect firewall issues

[josecelano]

While I was working on this issue:

#72

I experienced an issue with the network configuration, but the Grafana dashboard was ok.

UDP packets were not being sent back to the client, but all was working fine in the Grafana dashboard.

The problem was that the track was receiving the packages and sending them back to the client. However, the counter for responses sent only counts when it successfully writes the response into the socket. That does not mean that the OS sends the packet.

We could add a new Grafana dashboard to replicate the DigitalOcean graph with additional information related to the UDP sockets we are using.

We need to research ways to do it. For example:

https://github.com/prometheus/node_exporter

But there could be an independent solution for that, easy to install, that we don't have to maintain.

Subtasks

• Research for monitoring tools.
• Setup OS monitoring for basic stuff: CPU, Memory, .. (so we don't depend on Digital Ocean graphs)
• Setup UDP socket monitoring: packets received, sent, bytes, etc
• Add smoke tests that we can run from outside the server to test basic features:
  • We can run them after a deployment, a change in the server configuration ,etc.
  • We can add a workflow for deployments and execute them after the deployment.
  • It can be very simple, like just testing some requests like this.