Add certification to Prometheus Node Exporter

Posted May 27, 20207 min read

This article is mainly to celebrate the Node Exporter finally ushered in the v1.0.0 version.

Prometheus is the earliest open source monitoring and alarm solution by SoundCloud. And has grown into the second project after graduation from CNCF after Kubernetes. With the popularity of cloud-native concepts and the development of technologies such as Kubernetes, Prometheus has also made considerable progress in the field of monitoring.

Its main components include Prometheus, Alertmanager, Node Exporter, Blackbox Exporter and Pushgateway.

This article is mainly to celebrate Node Exporter finally ushered in the v1.0.0 version, so the focus is mainly on the security related that has been criticized, specifically using TLS and Basic Auth to improve its security.

background

Node Exporter is officially released by Prometheus, used to collect node system information, such as CPU, memory, disk and network information.
Usually, if we are using Prometheus as a monitoring solution, then Node Exporter will basically use it.

In Promethues' monitoring system, there has always been a view in the community that Metrics does not contain too private information. So you can see that most of the /metrics interface is directly exposed and there are no special security measures.

But with the large number of applications of Prometheus in production, security issues become more important.

The first solution that everyone thought of was to enable TLS for the connection between Prometheus and the monitoring target. However, because various exporters do not natively support TLS connections, we usually choose cooperate with reverse proxy to complete.

This way can meet the demand, but it is more complicated. Recently, Prometheus has modified its security model **. From the Node Exporter to other subsequent components, it will support TLS and basic auth, and also lists the latest security benchmarks(by default, TLS v1 is supported. 2 and above)

Use TLS

Here we directly practice to see how to enable TLS.

Prepare Certificate

(MoeLove) ~ mkdir -p prometheus-tls
(MoeLove) ~ cd prometheus-tls
(MoeLove) prometheus-tls openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout node_exporter.key -out node_exporter.crt -subj "/C = CN/ST = Beijing/L = Beijing/O = Moelove.info/CN = localhost "
Generating a RSA private key
..................................... +++++
.. +++++
writing new private key to 'node_exporter.key'
-----
(MoeLove) prometheus-tls ls
node_exporter.crt node_exporter.key

Through the above steps, we got two files node_exporter.crt and node_exporter.key.

Node Exporter uses TLS

Download Node Exporter v1.0.0 , and decompress it

(MoeLove) /tmp tar -zxvf node_exporter-1.0.0.linux-amd64.tar.gz
node_exporter-1.0.0.linux-amd64 /
node_exporter-1.0.0.linux-amd64/node_exporter
node_exporter-1.0.0.linux-amd64/NOTICE
node_exporter-1.0.0.linux-amd64/LICENSE
(MoeLove) /tmp cd node_exporter-1.0.0.linux-amd64
(MoeLove) node_exporter-1.0.0.linux-amd64 ls
LICENSE node_exporter NOTICE

Copy the two files node_exporter.crt and node_exporter.key generated previously to the current directory.

(MoeLove) node_exporter-1.0.0.linux-amd64 cp ~/prometheus-tls/node_exporter. *.
(MoeLove) node_exporter-1.0.0.linux-amd64 ls
LICENSE node_exporter node_exporter.crt node_exporter.key NOTICE

Write a configuration file and save it as config.yaml(freely named):

tls_server_config:
  cert_file:node_exporter.crt
  key_file:node_exporter.key

Next, use --web.config to pass the configuration file to the Node Exporter

(MoeLove) node_exporter-1.0.0.linux-amd64 ./node_exporter --web.config = config.yaml
level = info ts = 2020-05-26T17:50:12.123Z caller = node_exporter.go:177 msg = "Starting node_exporter" version = "(version = 1.0.0, branch = HEAD, revision = b9c96706a7425383902b6143d097cf6d7cfd1960)"
level = info ts = 2020-05-26T17:50:12.124Z caller = node_exporter.go:178 msg = "Build context" build_context = "(go = go1.14.3, user = root @ 3e55cc20ccc0, date = 20200526-06:01:48) "
level = info ts = 2020-05-26T17:50:12.130Z caller = node_exporter.go:105 msg = "Enabled collectors"
...
level = info ts = 2020-05-26T17:50:12.135Z caller = tls_config.go:200 msg = "TLS is enabled and it cannot be disabled on the fly." http2 = true

When you see the last log, it means that your Node Exporter has enabled TLS connection.

Of course, we can also choose to manually verify:

# Direct curl request

(MoeLove) prometheus-tls curl localhost:9100/metrics
Client sent an HTTP request to an HTTPS server.

(MoeLove) prometheus-tls curl https://localhost:9100/metrics
curl:(60) SSL certificate problem:self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It can be seen that the request cannot be directly made by curl. We can pass the certificate to curl to verify that the configuration has just been correct.

(MoeLove) prometheus-tls curl -s --cacert node_exporter.crt https://localhost:9100/metricsgrep node_exporter_build_info
# HELP node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built.
# TYPE node_exporter_build_info gauge
node_exporter_build_info {branch = "HEAD", goversion = "go1.14.3", revision = "b9c96706a7425383902b6143d097cf6d7cfd1960", version = "1.0.0"} 1

Of course, in addition to passing the certificate to curl through the --cacert parameter, you can also ignore the certificate check through the -k parameter.

(MoeLove) prometheus-tls curl -s -k https://localhost:9100/metrics | grep node_exporter_build_info
# HELP node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built.
# TYPE node_exporter_build_info gauge
node_exporter_build_info {branch = "HEAD", goversion = "go1.14.3", revision = "b9c96706a7425383902b6143d097cf6d7cfd1960", version = "1.0.0"} 1

Configure Prometheus to use TLS

Next, we will configure Prometheus to obtain metrics from the Node Exporter via HTTPS. The installation process is simple, whether you download the latest binary version directly or use the Docker image directly.

Note that I have copied the certificate issued above to the current directory.

(MoeLove) prometheus-2.18.1.linux-amd64 cp ~/prometheus-tls/node_exporter.crt.
(MoeLove) prometheus-2.18.1.linux-amd64 ls
console_libraries consoles LICENSE node_exporter.crt NOTICE prometheus prometheus.yml promtool tsdb

Next, you need to modify the configuration file so that Prometheus can capture the metrics exposed by the Node Exporter.

global:
  scrape_interval:15s
  evaluation_interval:15s

scrape_configs:
  -job_name:'prometheus'
    static_configs:
    -targets:['localhost:9090']

  -job_name:'node_exporter'
    scheme:https
    tls_config:
      ca_file:node_exporter.crt
    static_configs:
    -targets:['localhost:9100']

There is an additional scheme:https to establish a connection via HTTPS, and the certificate file used is specified in tls_config. For the complete configuration, please refer to the description of tls \ _config in the official documentation .

Finally, start Prometheus and visit /targets in the browser, if you see https://localhost:9100/metrics in the endpoint, So congratulations, Prometheus and Node Exporter are already connected via TLS.

Prometheus TLS-https://moelove.info

Add Basic Auth

In the above, I have introduced how to use TLS connection between Prometheus and Node Exporter. Next, I will introduce how to add Basic Auth to you.

It should be noted here that Basic Auth and TLS are not strongly dependent. You can use Basic Auth without enabling TLS, but I personally recommend that you do it thoroughly and enable it at the same time.

Configure password for Node Exporter

We can directly use htpasswd to generate the bcrypt password hash(this tool must be familiar to everyone).

(MoeLove) prometheus-tls htpasswd -nBC 12 '' | tr -d ':\ n'
New password:
Re-type new password:
$2y $12 $WLw2sYa.NYZoBVoCOE84qe3xNm7kbSoKVIBXP.PvqNDna60vnZhEW

Here I only used it to generate the password hash, without passing the user name.

Next, modify the configuration file used by the Node Exporter mentioned above, as follows:

tls_server_config:
  cert_file:node_exporter.crt
  key_file:node_exporter.key
basic_auth_users:
  # The currently set user name is prometheus, and you can set multiple
  prometheus:$2y $12 $WLw2sYa.NYZoBVoCOE84qe3xNm7kbSoKVIBXP.PvqNDna60vnZhEW

Start Node Exporter again, and use curl to request the metrics interface, you can see the current return of 401.

(MoeLove) prometheus-tls curl -Ik https://127.0.0.1:9100/metrics
HTTP/1.1 401 Unauthorized
Content-Type:text/plain; charset = utf-8
Www-Authenticate:Basic
X-Content-Type-Options:nosniff
Date:Wed, 27 May 2020 11:45:16 GMT
Content-Length:13

Open the Targets page of Prometheus, you will also see the current prompt 401, unable to capture metrics.

Prometheus Basic Auth-https://moelove.info

Configure Prometheus to use Basic Auth

Next, just modify the Prometheus configuration file and add basic_auth to it.

global:
  scrape_interval:15s
  evaluation_interval:15s

scrape_configs:
  -job_name:'prometheus'
    static_configs:
    -targets:['localhost:9090']

  -job_name:'node_exporter'
    scheme:https
    tls_config:
      ca_file:node_exporter.crt
    basic_auth:
      username:prometheus
      password:moelove.info
    static_configs:
    -targets:['localhost:9100']

After modifying the configuration file, just let Prometheus reload the configuration file:

(MoeLove) killall -HUP prometheus

Now refresh the Prometheus Targets page, you can see that the metrics have been crawled normally.

to sum up

This article describes how to enable the TLS connection between Prometheus and Node Exporter, and enable Basic Auth authentication for Node Exporter. Prior to this, there may be a small partner who completed it by adding counter generations, such as: Add Basic certification to Node Exporter

When used in production, it is recommended to more standardized operations, such as CA selection, password management, etc. Basic Auth of Node Exporter actually supports the configuration of multiple user name and password.

Next, the basic components provided by Prometheus will gradually promote the support of the security features mentioned in this article, including Prometheus, Alertmanager, Pushgateway and official exporter. Into.

Finally, congratulations to Node Exporter for welcoming v1.0.0.


Welcome to subscribe to my article public number [MoeLove]

TheMoeLove