Prometheus and Grafana Setup

Setup Prometheus and Grafana to view app metrics

Grainite provides a counters/metrics endpoint that includes various counters and gauges to provide insight into applications, as well as the server itself. This page goes over setting up Prometheus to access these metrics in Grafana.

Prerequisites

• Prometheus

• Grafana

Prometheus Setup

Secured Cluster/Server

Note: The following instructions apply to Grainite clusters or servers with TLS enabled.

1. Generate a client certificate.

2. Copy the new client certificates to the config directory. Example: cp ~/.gxscerts/<cluster-name>/client/<user-name>/* /etc/config/gxs-client-secret where <cluster-name> is the name of the current cluster.

3. Grainite exposes the counters at <server_ip>:5064/export-dashboard. To have Prometheus use this, set up your Prometheus configuration file (prometheus.yml) with scheme and tls_config attributes like in the following example:

prometheus.yml

...
scrape_configs:
  - job_name: 'grainite'
    scrape_interval: 5s
    metrics_path: '/export-dashboard'
    params:
        aggregate: [1]
    scheme: https
    tls_config:
      ca_file: /etc/config/gxs-client-secret/grainite_ca.crt
      key_file: /etc/config/gxs-client-secret/grainite_client.<user-name>.key
      cert_file: /etc/config/gxs-client-secret/grainite_client.<user-name>.crt
      insecure_skip_verify: true
    static_configs:
    - targets: [ '<server-IP>:5064' ]
...

Where:

• <server-IP> is the IP address of the cluster or server. You can get the IP address by following the instructions linked here.

• <user-name> corresponds to the user name specified when generating the client certificate.

4. After adding the above to the Prometheus configuration file, restart Prometheus for it to pick up the updated targets. Once Prometheus is restarted, confirm that "grainite" is shown as a target for Prometheus:

Prometheus Targets Page

Unsecured (w/o TLS)

Note: The following instructions apply to Grainite clusters or servers without TLS enabled.

Grainite exposes the counters at <server_ip>:5064/export-dashboard. To have Prometheus use this, add the following to your Prometheus configuration file (prometheus.yml):

prometheus.yml

...
scrape_configs:
  - job_name: 'grainite'
    scrape_interval: 5s
    metrics_path: '/export-dashboard'
    params:
        aggregate: [1]
    static_configs:
    - targets: ['<server-IP>:5064']
...

Where:

• <server-IP> is the IP address of the server.

After adding the above to the Prometheus configuration file, restart Prometheus for it to pick up the updated targets. Once Prometheus is restarted, confirm that "grainite" is shown as a target for Prometheus:

Prometheus Targets Page

Grafana Setup

Add Prometheus as a data source in Grafana by going to Settings -> Data sources -> Add data source and selecting Prometheus, filling in the URL, and clicking on Save & test.

Now, app counters can be accessed by using Prometheus as a data source and searching for counters/gauges beginning with gxsapp. The format for app metrics is the following:

1. Counters: gxsapp_<counter name>_total

2. Gauges: gxsapp_<gauge name>_current