Skip to content

Tutorial: Managing System Tokens

This self-service tutorial guides you through a more advanced task: Managing the ODAS system token, which is mandatory for cluster internal communications.

Difficulty: Intermediate
Time needed: 1 hour

Introduction

After setting up an ODAS cluster and configuring it, a common challenge around operating ODAS clusters is how to manage the system token that is required for internal communication between services. This short tutorial introduces you to the more intricate details of tokens, the resources provided by Okera to manage said tokens, and then presents a specific approach on how to combine these for the aforementioned purpose.

JSON Web Token Information

An ODAS cluster uses a JSON Web Token (JWT) internally to authenticate remote procedure calls (RPCs) between the distinct services, which are shown in the Architecture Overview document.

JWTs are a common technology to exchange user credentials - also referred to as claims or assertions - with a PKI-based signature: a private key is used to sign the content of the token, while a matching public key is used to verify it later on. Each token, in its simplest form, comprises three parts:

  1. The header, defining how the token is constructed. Its format is in JSON.
  2. The payload, defining the claims and other token data. The format is also JSON.
  3. The signature, which is computed based on the content of the payload and signed with a private key.

Each of the parts is separately base64 encoded and concatenated using dots ".".

This can be seen in the following screenshot, taken from the https://jwt.io website's JWT Debugger. The different parts are color coded and their content shown on the right side.

JWT.io Debugger

Note

Since we did not provide a public key to verify token, the debugger reports an "Invalid Signature". Also, there are more advanced features for JWTs, for example the ability to encrypt the payload. As this does not apply to ODAS, we are not going to address those now.

Assuming we use the token from the screenshot, you can extract the payload using simple Linux shell tools.

Example: Extracting the payload from a JWT token

$ TOKEN=eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJva2VyYV9zeXN0ZW1fdXNlciIsImlzcyI6 \
IjEwLjEwMC4xLjk5IiwiZ3JvdXBzIjpbInJvb3QiLCJhZG1pbiJdLCJleHAiOjE2MTI1NjQxNDJ9.htfFWYvehUb2 \
atRMtkDnbDO1a1dgT-e4on74FjKPuzKWnCI375vh52TcfNfmArhBTFdcRgL42yPdRayC_Xf3kz_PRq-HPHpDNFp1P \
-5tFkhs4rAcekWevHrzRABtPwymQ0pUJ4ZMaUXOJSeMGfMaKedYg2HNyIhuzvlxKDmsIzx8lt7_0jprlsQQuAoW-e \
Bz3HxiG_4XOwnOvUav2uKEC4AZgL7Ynkr9_x-oHaBsJkivsuAOU8XHewzCQqO9BD7DkZqKu5vF3sE8RW4R6o1bjek \
McQc4hth3qpTF1iqYxJd5f44rYIMiBNjzPXhfPHVhYbCnu65DIf2pYtUVx-tGUHfXi_DnY9iP4KzRE-3_IjVV2OpH \
xUbCDq58LK9CIQ6VLEvyD35-K11WiJxzqXs5di1HWoqjwwGyEM7ICdhAjd27xpQeixB7_ZOLzMYh8UgaXRy5Q6Jih \
c1N91o58yKT-rOLhnklUXCXhNN9UlNFOyWmDKgdJUxvTKDhI-A7lSrXGqKclsW8yxAYv-2XwEYNYFLG0PBdME8qPE \
26tSr2Tl3nU7UzVII0q3oD3GnGtT_CJ6Gyw2Y3uNy-EKsrp08gWQjz0XPy51obfFPZRhxNm5aovM5T67fF60PMD9u \
AvzGLG6He_Anc20cXTWwGbiOq6nl1xGTiAZC-_Gt_WzDReq8

$ echo $TOKEN | awk -F. '{print $2}'
eyJzdWIiOiJva2VyYV9zeXN0ZW1fdXNlciIsImlzcyI6IjEwLjEwMC4xLjk5IiwiZ3JvdXBzIjpbInJvb3QiLCJhZ \
G1pbiJdLCJleHAiOjE2MTI1NjQxNDJ9

We can also decode the payload using command-line tools.

Example: Decoding the payload

$ echo $TOKEN | awk -F. '{print $2}' | base64 -d
{"sub":"okera_system_user","iss":"10.100.1.99","groups":["root","admin"],"exp":1612564142}

Finally, we can convert the epoch-based timestamp, provided by the exp field, that indicates when the token is going to expire.

Example: Converting the timestamp

$ date -d @1612564142
Fri Feb  5 22:29:02 UTC 2021

System Token Information

The ODAS system token is a JSON Web Token (JWT) that is created when the cluster is provisioned and stored in a Kubernetes secret.

If you are following the default installation, as explained in the Deploying ODAS section, and using the provided okctl command-line tool, the system token is created and made available to the various ODAS services in this way:

  1. Optional key pair and token creation

    The okctl update command, dependent on the configuration provided by the --config <yaml_file> parameter, may create a new public/private key pair, and may also create a new system token. You can specify your own existing key pair, or your own existing system token explicitly. If the system token is created for you, then, by default, it is stored in a directory called .auth right where the okctl update command was invoked. The provided or created private key is used to sign that system token, and the matching public key is used at runtime to verify it.

    See the JWT Authentication documentation for more details.

  2. Upload of public key and token

    The created and/or provided public key and system token are uploaded into a Kubernetes (K8s) secret. The default name is secrets, and, given you have the proper permissions, you can retrieve its content like shown in the following example.

Example: Retrieve the secrets content

$ kubectl get secret secrets -oyaml
apiVersion: v1
data:
  JWT_PUBLIC_KEY_0: LS0tLS1CRUdJTiBQVUJMSUMgS0VZL...
  SSL_CERTIFICATE_FILE_0: LS0tLS1CRUdJZJQ0FURS0tL...
  SSL_KEY_FILE_0: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0...
  SYSTEM_TOKEN_0: ZXlKMGVYQWlPaUpLVjFRaUxDSmhiR2N...
  ...
kind: Secret
metadata:
  creationTimestamp: "2020-10-28T05:02:54Z"
  managedFields:
  ...

Before we look deeper into the system token itself, we need to connect the dots and see how the ODAS containers are accessing the token (and the public key to verify it). In the above example you can see how okctl has stored the system token in the secret under the name SYSTEM_TOKEN_0. The public key is stored under the name JWT_PUBLIC_KEY_0.

Note

The postfixed number can be used to store more than one public key. For example, if there is another public key that should be tried to verify a given token, it would be stored under JWT_PUBLIC_KEY_1 (and so on).

The supplied manifest files for each ODAS service (which produce K8s deployment or daemonset objects) mounts the above secret named secrets as a volume into the derived pods.

Example: The secret is mounted as a volume into the pods

$ kubectl describe ds cerebro-worker
    ...
    Environment Variables from:
      default-odas-config  ConfigMap  Optional: false
      odas-config          ConfigMap  Optional: false
    ...
    Mounts:
      /etc/secrets from secrets (ro)
      /var/lib/okera from okera-lib (rw)
      /var/log/okera from okera-log (rw)
  Volumes:
   secrets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  secrets
    Optional:    false
  ...

The final step is that the ODAS supplied K8s configmaps, shown in the above example and named odas-config etc., are configured to tell each service where they can find the necessary data within the mounted volume. The configmaps are provided to the pods and create environment variables in the ODAS containers, which then are read by the processes inside the container accessing the data from those (virtual) files.

Example: The ODAS configmap pointing to a secret-backed file

$ kubectl get cm odas-config -oyaml | grep SYSTEM
  SYSTEM_TOKEN: /etc/secrets/SYSTEM_TOKEN_0
  ...

Finally, given the permission again to do so, you can extract the data on the commmand-line from the secret and decode it for further use.

Example: Extract and decode the system token from the secret

$ kubectl get secret secrets -oyaml | grep "  SYSTEM_TOKEN_0" | awk '{print $2}' | base64 -d
eyJ0eXAiOiJKV1QiLCJhbGciO...XwJEnF8hGHPZs-MB55FJShWfvaS5TGeIqIex1Dr1NPOGNqsI

After that you can use the above CLI tools to get the payload and inspect the details of it.

Token Expiry

As with many JWTs applications, the ODAS system token has an expiry time, usually specified as an epoch-based time (also called Unix time), which is the number of seconds elapsed since 00:00:00 UTC on 1/1/1970. The earlier example did show an expired token, dating back to Feb 5th 2021.

Example: An expiry time from the past

$ echo $TOKEN | awk -F. '{print $2}' | base64 -d
{"sub":"okera_system_user","iss":"10.100.1.99","groups":["root","admin"],"exp":1612564142}

$ date -d @1612564142
Fri Feb  5 22:29:02 UTC 2021

The default time for the okctl update generated system token is 365 days, or one year.

Note

Version before ODAS 2.5.0 used a hardcoded duration value of 100 days.

Eventually, the token will expire and a new one needs to be created.

Important

When the system token expires, all cluster internal communication fails. Users will not be able to log into the WebUI anymore nor run queries through any of the available endpoints. Administrators will see the ODAS pods start failing and being restarted constantly. This is due to the Kubernetes health checks failing, triggering the restarts.

There are, among other choices, two common ways to approach this: using okctl or custom JWT code. Each will be discussed in the following subsections.

Using okctl to create a new system token

The first option is suitable if you are already using a bastion or gateway host that admins have configured with an instance of okctl that can be used to update the ODAS cluster in question.

First, we can look at the okctl tokens create subcommand and its help:

$ ./okctl tokens create --help
usage: okctl tokens create [<flags>] <username> [<groups>...]

Create a new JWT token for a specific username

Flags:
      --help                   Show context-sensitive help (also try --help-long and --help-man).
      ...
      --auth-dir=".auth"       Location for auth related files
  -f, --force                  Overwrite the token if it already exists
  -d, --duration=8760h         How long the token should be valid for

Args:
  <username>  Username to use for this token
  [<groups>]  Groups to put in this token

Note

The above output is from ODAS 2.5.0, which allows to set a custom duration for a token. Versions before 2.5.0 use the hardcoded 100 days as mentioned above.

This means we can create a new system token using the proper username, set with the sub field to okera_system_user, and with the necessary groups claims, set to the user groups containing the system administrators. See Administration Basics for details on setting the CATALOG_ADMINS configuration value.

Example: Creating a new system token using okctl

$ ./okctl tokens create okera_system_user admin
2021/04/02 18:11:06 Token saved to: .auth/okera_system_user.token
2021/04/02 18:11:06 Token: eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJva2VyYV9zeXN0ZW1fdXNlciIsImlzcyI6IjEwLjEwMC4xLjk5IiwiZ3JvdXBzIjpbImFkbWluIl0sImV4cCI6MTYyNjAyNzA2NX0.iBJwj1JD_fsS9NxuM99D81jyVkq9RhA7FTQY1nWCvkv0pWTPqETCTICKi654Q2Lr9HHPecK2f3LcNm0OwKUZ6_eHZ-czj9Js5EVZg0v9Ct7HN-OHpfO08-cWEVGxprcTHqyAN-yZXWi5mIvy566P_tAXc8fJV_5qWh-aywwjVfsT-M_K8FnpiLFyFYhZqZe4AbDwif86ToqxNOU-JAUZsZfY6JuQY9XsjtCc76W9DrGnhhsnG-A-7eSL6W1eh-5oKRWop4JPk1RYqpNXb7p3oYlDjxZnOeSRmdmIWLLzkKwkm1zBNKZFEGgYJx8kImXAgEQVG6Imjdf-KnPikhEEeXXOMLsQs7eKGWbMHG03BV6dkv2y4Bji53ae2tj8Np-bPrA18KQAKdTfeEWA-U_v9hMNfUMODWlsJLxH2lHgBkh5Ab4fZvq7LgnizTL2LoRgyzwE6nBeSoW57uA84NX-t7ht_oqr28yM57NM5yNXMUH3Pcr3nzIDjrECtt45wAHFJI_l3gulxcTfOLW3rsZhDzzzEOkRRXPmS-g1p2ITPa6FbMhaThGiWYK1zcs5Asvfrza6dYn_HEvLZZcSXam_evXyKORMl8MyLQ-9kBHl3Etx679a64SwDpyBoZnlDoCDe3_fupBAp9DtMvAlWnO6XTKXx6SHtSIuMUtQJCbGpa0

Using custom code to generate a system token

Another option to generate a system token is using one of the available JWT libraries. The JWT.io website has many listed, which can be filtered by the programming language of your choice.

The crucial parts for creating a JWT outside of any of the Okera provided tooling are:

  1. Access to private key

    You need to have access to a private key that is used to sign the JWT, matching the public key configured on the ODAS cluster to verify if. Note that you can have more than one public key configured, which means you could have separate key pairs that are used for specific applications.

    Note

    As with all security, protecting sensitive assets, such as the private key of a PKI key pair, is of utmost importance. Okera recommends using key management services that are provided natively by your infrastructure. For instance, in AWS you could use the AWS Secrets Manager to manage your keys, and all applications and scripts source the key securely directly from that service.

  2. Proper use of system token values

    This one is not difficult, but has to be done properly for the token to be used within the cluster. The following table shows the various fields and they values.

    Field Type Value Mandatory Notes
    sub String okera_system_user Yes The subject, aka username. Fixed value.
    iss String <some_issuer> No The issuer of the token. Usually the cluster name.
    exp Number <epoch_time> Yes The expiry time of the token. Unix date, epoch-based.
    groups Array <list_of_groups> Yes The user groups, should match CATALOG_ADMINS

The following Python based code shows how a JSON object is constructed to represent the JWT payload. Using the JWT library, the payload is converted into a token and signed by a given private key.

Example: Creating a system token using Python code

import json
from datetime import datetime, timedelta, timezone
from jwt import (JWT, jwk_from_dict, jwk_from_pem)
from jwt.utils import get_int_from_datetime

instance = JWT()

# Create a JSON object representing the JWT payload
message = {
    'iss': 'https://okera.example.com/',
    'sub': 'okera_system_user',
    'groups': [ 'grp1', 'grp2' ],
    'exp': get_int_from_datetime(
        datetime.now(timezone.utc) + timedelta(days=30)),
}

# Load a RSA key from a PEM file.
with open('odas_private.key', 'rb') as fh:
    signing_key = jwk_from_pem(fh.read())

# Generate the JWT and print it to the console
jws = instance.encode(message, signing_key, alg='RS512')
print(jws)

Note how the use of the timedelta function allows to compute the expiry time using simple date units, here 30 days. As mentioned in the ODAS JWT documentation, you also need to ensure that the chosen JWT signing algorithm, here RS512 must match with what is configured on the ODAS cluster.

Running the above code emits the generated JWT and could be redirect (or directly written) into a file. Another option is to run a similar code in a serverless functions service, like AWS Lambda, that has access to the private key and can write the new JWT directly into the cluster's Kubernetes secret. More on this in the next section.

Updating the system token

No matter how you generated the JWT, the next step is provisioning it into an ODAS cluster. Like before, there are two choices: using the Okera provided okctl command-line tool, or use some other means. Each are explained in the following subsections.

Using okctl to update the system token

Assuming the cluster configuration is the default, okctl generated setup, the original system token is located in the following place:

config:
  ...
  SYSTEM_TOKEN: .auth/system.token
  ...

If this is the case, we need to copy the new token file over to the configured file and update the cluster.

Example: Updating the cluster using okctl, assuming a default installation

$ mv .auth/system.token .auth/system.token.old
$ mv .auth/okera_system_user.token .auth/system.token
$ ./okctl update --config cluster_config.yaml

Note

This assumes you are in the proper directory where okctl is located, and that you have a YAML-based configuration file named cluster_config.yaml (the name though does not matter). It also assumes that the okctl tool is able to call kubectl with the proper cluster credentials.

In case you have used the custom code to generate a system token approach, the same steps apply as just shown, ensuring you copy the newly generated token into the right place. The example below assumes you ran the custom code above and redirected the output into a file named system.token in the current location (where okctl is located).

Example: Updating the cluster using okctl, assuming a custom system token was used

$ mv .auth/system.token .auth/system.token.old
$ mv system.token .auth/system.token
$ ./okctl update --config cluster_config.yaml

The okctl update --config <config_yaml> call is going to do many things, among them is loading the content of the configured system token file and storing it in the cluster's K8s secret. After that (and assuming the token has indeed changed) it is also going to trigger a rolling restart of all ODAS containers (using K8s delete pod commands). In the end, when all pods have been restarted, the new token should be in use and the cluster operational. See Verify Cluster Token for the final step in the overall process of replacing a system token.

Using Kubernetes tools to update the system token

If you do not have a default installation, or in general prefer using native Kubernetes tools to update running clusters, you can use the following process:

  1. Directly update the system token data in the proper ODAS secret
  2. Restart all pods to ensure the new token is used

With Kubernetes, there are multiple ways to update objects. You could simply delete and apply the new values, or you could patch them directly in place, among other choices.

As shown earlier, the actual token data is stored as a base64 encoded string in a field of a secret. Assuming again you have proper K8s permissions, you can use the above created token (no matter if you used okctl or custom code or tools) to patch the existing secret, and then initiate a pod restart.

Example: Patching the system token data and restarting the pods

$ kubectl patch secrets secrets --type json --patch "[{\"op\": \"replace\", \"path\": \"/data/SYSTEM_TOKEN_0\", \"value\": \"$(cat system.token | base64 -w 0)\"}]"
$ kubectl delete pods --all --force --grace-period=0

Of interest here is that you need to use base64 -w 0 to ensure the encoded token data is a single line string.

Verify Cluster Token

Once the token is updated and the ODAS pods restarted, you should verify that the cluster is operational.

Note

As an administrator, first check if all pods are up and running, using your Kubernetes monitoring, or executing kubectl get pods on the command-line.

For instance, perform the following actions:

  • Log into the WebUI and browse the datasets in the Data tab
  • Use PyOkera or the REST API to send a query plan request to the cluster, or request a list of databases
  • Use a configured client application to run a sample query

All of these require a working cluster and communication of various cluster internal services. Should the token update have failed, the cluster is likely to not be operational at all.