Monitor any server basic data (disk, cpu, ...)

• create folder to store all our monitoring binaries /usr/local/lib/kiwix_monitoring (mandatory to isolate them from other binaries especially the ones automatically installed on WMF cloud which do have their own prometheus and node-exporter) mkdir -p /usr/local/lib/kiwix_monitoring

This recipe will install:

• prometheus, with its web UI running on localhost:9091 querying node_exporter below on localhost:9101 and pushing data to our Grafana Cloud
• node_exporter, exposing metrics from node on localhost:9101 (activated on-demand by the nice systemd socket)

Prometheus

• Download Prometheus and move promtool and prometheus binaries to /usr/local/lib/kiwix_monitoring
• Add username and Group useradd -M -U kiwix_prometheus -s /usr/sbin/nologin
• Change ownership of binaries chown kiwix_prometheus:kiwix_prometheus /usr/local/lib/kiwix_monitoring/{promtool,prometheus}
• Make sure its executable chmod +x /usr/local/lib/kiwix_monitoring/{promtool,prometheus}
• Create data folder mkdir -p /var/lib/kiwix_prometheus/data
• Change data folder perms chown -R kiwix_prometheus:kiwix_prometheus /var/lib/kiwix_prometheus/data
• Add /etc/kiwix_prometheus.yml

global:
  scrape_interval: 60s

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9101']
        labels:
          instance: <your_instance_label>

remote_write:
  - url: 'https://prometheus-prod-24-prod-eu-west-2.grafana.net/api/prom/push'
    basic_auth:
      username: '1172000'
      password: 'glc_xxxxxxxxxxxxxxxxxxxxxxx'

<your_instance_label> must be replaced by a meaningful label for the machine at hand, by default the machine DNS name.

The password here must be obtained from Grafana Cloud Add new Connection wizard. You should select Hosted Prometheus Metrics. In the Set the configuration section, choose Create a new token and one will be created (in Cloud Access Policies) and showed to you with the proper encoding (starts with glc_. Just copy that part.

• Add /etc/systemd/system/kiwix_prometheus.service

[Unit]
Description=Kiwix Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=kiwix_prometheus
Group=kiwix_prometheus
Restart=on-failure
ExecStart=/usr/local/lib/kiwix_monitoring/prometheus \
  --config.file=/etc/kiwix_prometheus.yml \
  --storage.tsdb.path=/var/lib/kiwix_prometheus/data \
  --storage.tsdb.retention.time=30d \
  --web.listen-address="localhost:9091"

[Install]
WantedBy=multi-user.target

• Reload systemd systemctl daemon-reload
• Enable and Start it systemctl enable --now kiwix_prometheus.service

node_exporter

• Download node_exporter and move binary to /usr/local/lib/kiwix_monitoring/
• Add username and Group useradd -M -U kiwix_node_exporter -s /usr/sbin/nologin
• Change ownership of binary chown kiwix_node_exporter:kiwix_node_exporter /usr/local/lib/kiwix_monitoring/node_exporter
• Make sure its executable chmod +x /usr/local/lib/kiwix_monitoring/node_exporter
• Create data folder mkdir -p /var/lib/kiwix_node_exporter/textfile_collector
• Change data folder perms chown -R kiwix_node_exporter:kiwix_node_exporter /var/lib/kiwix_node_exporter/textfile_collector
• Add systemd service /etc/systemd/system/kiwix_node_exporter.service

[Unit]
Description=Kiwix Node Exporter
Requires=kiwix_node_exporter.socket

[Service]
User=kiwix_node_exporter
ExecStart=/usr/local/lib/kiwix_monitoring/node_exporter --web.systemd-socket --collector.textfile.directory=/var/lib/kiwix_node_exporter/textfile_collector --no-collector.netdev --no-collector.netclass --no-collector.softnet

[Install]
WantedBy=multi-user.target

Note the --no-collector.* flags ; these are used to reduce the amount of metrics exposed following https://github.com/kiwix/operations/issues/427 incident.

• Add systemd socket /etc/systemd/system/kiwix_node_exporter.socket

[Unit]
Description=Node Exporter

[Socket]
ListenStream=127.0.0.1:9101

[Install]
WantedBy=sockets.target

• Reload systemd systemctl daemon-reload
• Enable and Start it systemctl enable --now kiwix_node_exporter.socket && systemctl enable --now kiwix_node_exporter.service