# KUDO Spark Operator Monitoring

KUDO Spark Operator has metrics reporting support, which can be enabled during the installation of the operator.
By default, it supports integration with the Prometheus operator (opens new window).

Prometheus Operator relies on ServiceMonitor kind which describes the set of targets to be monitored. KUDO Spark Operator configures ServiceMonitors for both the Operator and submitted Spark Applications automatically when monitoring is enabled.

# Exporting Spark Operator and Spark Application metrics to Prometheus

# Prerequisites

KUDO v0.15.0 or later
The prometheus-operator. If you don't already have the prometheus-operator installed on your Kubernetes cluster, you can do so by following the quick start guide (opens new window).

# Metrics configuration

Metrics reporting is disabled by default and can be enabled by passing the following parameter to kudo install:

$ kubectl kudo install spark --instance=spark -p enableMetrics=true

Service endpoints and ServiceMonitor resources are configured to work with Prometheus Operator out of the box.

Full list of metrics configuration parameters and defaults is available in KUDO Spark params.yaml (opens new window).

# Running Spark Application with metrics enabled

Composing your Spark Application yaml:
- use the following Spark image which includes the JMXPrometheus exporter jar: mesosphere/spark:spark-3.0.0-hadoop-2.9-k8s
- enable Driver and Executors metrics reporting by adding the following configuration into SparkApplication spec section:
```
  monitoring:
    exposeDriverMetrics: true
    exposeExecutorMetrics: true
    prometheus:
      jmxExporterJar: "/prometheus/jmx_prometheus_javaagent-0.11.0.jar"
      port: 8090
```
  spec.momitoring.prometheus.port value should be the same for all submitted Spark Applications in order for metrics to be scraped.
- if it's necessary to expose the metrics endpoint on a port other than 8090, do the following:
  - specify desired port when installing the kudo-spark-operator:
```
kubectl kudo install spark -p appMetricsPort=<desired_port>
```
  - change the port value in the SparkApplication yaml definition (spec.monitoring.prometheus.port)
- Mark driver and/or executor with the label metrics-exposed: "true" -
```
spec:
  driver:
    labels:
       metrics-exposed: "true"
  executor:
    labels:
      metrics-exposed: "true"
```
- Install the SparkApplication:
```
kubectl apply -f <path_to_the_application_yaml>   
```
Full application configuration example is available in spark-application-with-metrics.yaml
Now, go to the prometheus dashboard (e.g. <kubernetes_endpoint_url>/ops/portal/prometheus/graph) and search for metrics starting with 'spark'. The Prometheus URI might be different depending on how you configured and installed the prometheus-operator.

# Dashboards

Dashboard installation :

Open the Grafana site (e.g. <kubernetes_endpoint_url>/ops/portal/grafana).
Press + button and pick Import item from the menu.
Copy content of the dashboard json file and paste it to the textarea on importing form.

For more information visit Grafana documentation: Importing a dashboard guide (opens new window).

← Limitations Advanced configuration options →