NOTE: This article assumes you are somewhat familiar with Prometheus and the node_exporter, and already have a working setup. Visit this tutorial if you don’t.

I’ve used Prometheus + Grafana to keep an eye on my servers and devices (a lot of things support node_exporter, like OpenWRT!) for quite a while now, but they’ve always been on my local network, or at least connected to my VPN subnet.

Recently, I’ve wanted to do the same for my VPS instances, but I didn’t like the idea of having Prometheus metrics on a public facing server for anyone to see. So it was with great pleasure that I found some documentation for node_exporter recently to add TLS and Basic Authentication.

However, doing this on multiple hosts manually can be tedious, as we need to install node_exporter, copy certificates to the host, and configure the systemd unit to use a config file for the Basic Authentication. Once we’ve set up one host, we’ll run through the Ansible playbook.

First let’s figure out how to get it working manually. Right at the bottom of this readme, we find a link to how we can configure the file for our node_exporter. The minimum options we need are:

basic_auth_users:
  prom: $2b$12$MzUQjmLxPRM9WW6OI4ZzwuZHB7ubHiiSnngJxIufgZms27nw.5ZAq
tls_server_config:
  cert_file: node_exporter.crt
  key_file: node_exporter.key

But how do we generate our password, key and certificate? Prometheus has an article on the password here, and Linode has a great article generating self-signed certificates and keys.

openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 \
  -nodes -out node_exporter.crt -keyout node_exporter.key

Let’s put all of these files in the default node_exporter configuration directory - /etc/prometheus-node-exporter.

Finally, to get node_exporter to read this configuration file, we need to tell systemd to add a command line argument when it starts node_exporter. We can do this in /etc/default/prometheus-node-exporter, by making sure that the line that begins with ARGS= is set to (depending on node_exporter version):

# node_exporter version < 1.5.0:
ARGS="--web.config /etc/prometheus-node-exporter/web-config.yml"

# node_exporter version >= 1.5.0:
ARGS="--web.config.file /etc/prometheus-node-exporter/web-config.yml"

Now, if we restart our systemd unit for prometheus-node-exporter (systemctl restart prometheus-node-exporter) and visit https://ip-address:9100/metrics, we should get the self-signed warning from our browser, which we can bypass, and then a Basic Auth prompt. I’ve used the username prom and the password node_exporter in this example.

To configure our Prometheus server, we need to add a new scrape_config job. This is how I define mine:

- job_name: 'prometheus_auth'
    scheme: https
    tls_config:
      insecure_skip_verify: true
    static_configs:
      - targets: ['1.2.3.4:9100']
      - targets: ['example.com:9100']
    basic_auth:
      username: prom
      password: node_exporter

As promised, here’s the Ansible playbook, along with a README with some instructions to get you started: