Monitoring services quickstart

This document describes how to use the Kubernetes Administration Dashboard and Grafana. Kubernetes Dashboard enables the cluster administrator to get overall health of the cluster and get details of each node, pod and service that is part of the cluster.

Refer to https://kubernetes.io/docs for other administration commands.

Grafana provides a UI to monitor cluster resource usage (CPU, RAM, etc), giving admins a quick way to understand cluster utilization over time.

Enabling the kubernetes admin dashboard and grafana monitoring UI

The kubernetes admin dashboard can be enabled by configuring a port for it to run on. This port can be configured the same way as any ODAS service port, by using the OKERA_PORT_CONFIGURATAION environment variable.

To enable the dashboard, set kubernetes_dashboard:admin_ui to be greater than zero. To enable grafana, set grafana:ui to be greater than zero.

Example: to start the k8s dashboard on port 10001 and the grafana UI on port 10000:

export OKERA_PORT_CONFIGURATION="$OKERA_PORT_CONFIGURATION,kubernetes_dashboard:admin_ui:10001"
export OKERA_PORT_CONFIGURATION="$OKERA_PORT_CONFIGURATION,grafana:ui:10000"

The services can be enabled independently. By default, neither service is enabled.

When enabled, both services are available on any cluster nodes at the configured port. Note that the k8s dashboard runs with SSL enabled (https) and grafana does not (http).

Getting the admin endpoints and credentials

To get the endpoints and auto-generated credentials, run admin-info on the cluster from ocadm.

$ ocadm clusters admin-info [CLUSTER ID]
Example:
$ ocadm clusters admin-info 3

The output looks like below.

{
    "kubernetes-master": "10.1.10.178",
    "grafana-ui": "http://10.1.10.178:10000",
    "kubernetes-dashboard": "https://10.1.10.178:10001",
    "dashboard-admin-token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWNybXcxIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJlYjAyYWZlNC00NzBkLTExZTgtYjY2My0wMjY2OTBkNWEyY2MiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.N_8CqLphJwawm0oazVepaSD7PqN1HBzAsy4lbdHXTpdwAPyrRHJV_lM4_VloZMq9Wbpe2npqtG7aSTgyK1ESryRt0ReT3H_JnnUSQ4t-UU1OugfTDQ5Fh6w-at7EAjxAM8_HTdpB71Hz_LkACslHKWE2tRtyyKVcZ_Lrdxg8lYzcdG2Si4-qctbc69R-mjks08yzghLK1mWJh8PQhI_peBG7qELq7BxlNqsvSLL46POErTFN8hlKCyV7wN7RsLZSTSVOyikPFbSKojjCcyo2GEuAI5n2ZidACnwdekJrZOKjXs2_ZwifLIVmxSDEw6UjTqUHnaumgoym6K7v7Nc6Gw"
}

Logging into the dashboard

Using your favorite browser go to the dashboard URL. It will prompt you for login credentials.

Example: https://10.1.10.178:10001

Select “Token” and enter the token string from above to login to the dashboard.

The dashboard requires privileges to login as it allows the logged in user to restart services, looks at configs and look at logs. Aside from being able to change the state of the running cluster, the configs and logs may have credentials. When the k8s dashboard is installed, ODAS automatically creates an dashboard admin user and admin-info returns the token for that user. It is not recommended to share this token to non-admin users of the system.

The grafana UI does not require login credentials as the UI is read-only and we do not consider any of it to be privileged.

Basic dashboard navigation

The navigation panel on the left connects to the default namespace. ODAS services are launched in this namespace. kube-system namespace will show the Kubernetes system pods and services.

Kubernetes deploys and schedules containers in groups called pods. A pod typically has one or more containers that provides an ODAS service.

You may use the dashboard ui to list the various ODAS services and confirm that the desired number of pods is running for each service. Typically, you would run one cerebro-planner pod and several cerebro-worker pods depending on your workload needs.

For example, a 10-node ODAS cluster would have the following pods:

  • 2 canary pod
  • 1 cerebro-catalog pod
  • 2 cdas-rest-server pod
  • 1 zookeeper pod
  • 1 cerebro-planner pod
  • 10 cerebro-worker pods

Let’s try a few of sample dashboard operations:

  1. Keeping the namespace as default, click on Deployments on the left navigation panel. Makes sure that all ODAS services are running. This should include cerebro-catalog, cerebro-web, and cerebro-planner.
  2. Click on Pods on the left navigation panel Make sure that the list is similar to the example above, including the desired number of pods.
  3. Click on Pods. Scroll to the right of the cerebro-planner pod and click on the icon with small horizontal bars. This will show the logs in a separate tab/window on your browser. The logs are a quick way to detect any issues with the pod.

Additional Information

Refer to Kubernetes documentation and tutorials for much more.