2. Okera Install - Setting Up the Deployment Manager

These steps configure the DeploymentManager which can manage all of your Okera clusters. This set of steps is typically required once, to configure Okera to your environment.

Acquire and Extract the DM Tarball

Follow these steps to get the Deployment Manager tarball on the DM host.

  1. Download and extract the Deployment Manager tarball.
    • The recommended download location is /opt/okera, but it can be downloaded to anywhere.
    sudo mkdir -p /opt/okera && cd /opt/okera
    
  2. Update ownership of the destination directory.

    echo `whoami` | xargs -I '{}' sudo chown -R '{}' /opt/okera
    
  3. Get the tarball from S3.

    curl -O https://s3.amazonaws.com/okera-release-useast/1.2.3/deployment-manager-1.2.3.tar.gz
    
  4. Extract the bits.

    tar xzf deployment-manager-1.2.3.tar.gz && rm deployment-manager-1.2.3.tar.gz && ln -sfn deployment-manager-1.2.3 deployment-manager
    

Acquire the Okera CLI

Perform this for any machine on which you want to run the Okera CLI tool. Download the shell binary for either Linux or OS X (depending on which OS is running the CLI).

  • Linux
 curl -O https://s3.amazonaws.com/okera-release-useast/1.2.3/cli/linux/ocadm && chmod +x ocadm
  • OS X
 curl -O https://s3.amazonaws.com/okera-release-useast/1.2.3/cli/darwin/ocadm && chmod +x ocadm

Establish DM Logging and Install Paths

Configure the logging and local install directories. These directories should be paths on the local file system. The install directory should be restored if this machine is moved. By default, the directories are /var/log/okera and /etc/okera. The directories can be changed by setting DEPLOYMENT_MANAGER_LOG_DIR and DEPLOYMENT_MANAGER_INSTALL_DIR in the environment.

Example: Creating the Logging and Install Directories

sudo mkdir -p /var/log/okera sudo mkdir -p /etc/okera
sudo chmod 700 /etc/okera

The Deployment Manager user needs exclusive access to those directories. If the directories are created with a different user than the Deployment Manager user, run:

echo `whoami` | xargs -I '{}' sudo chown -R '{}' /var/log/okera
echo `whoami` | xargs -I '{}' sudo chown -R '{}' /etc/okera

Configure the Deployment Manager

You must configure the Deployment Manager before it can be run. The configuration is done by using environment variables and before starting up the server. It is recommended that you copy the template configuration and update it. The steps below assume that the standard install paths were used.

  1. Copy the Deployment Manager environment.

    cp /opt/okera/deployment-manager/conf/env-template.sh /etc/okera/env.sh
    
  2. Open and edit env.sh, modifying it as necessary.

The config script from /etc/okera/env.sh is automatically loaded when starting the Deployment Manager. If this is the path used, it is not necessary to source the script. If any other path is used, env.sh must be sourced before starting the Deployment Manager.

  1. Start (or restart) the deployment manager by running:
  /opt/okera/deployment-manager/bin/deployment-manager

DM Configuration Options

Below are common available Okera configuration options. They can be set as environment variables or in /etc/okera/env.sh, as explained above. They are loaded when the Deployment Manager starts.

For additional Deployment Manager configuration options, see Advanced Installation: DM Configs.

OKERA_S3_STAGING_DIR

This directory is the OKERA_S3_STAGING_DIR for logs and install files. It can exist anywhere in S3 that makes sense for your organization. Okera creates in this directory a number of required subdirectories.

Example: Setting the S3 staging directory

export OKERA_S3_STAGING_DIR=s3://<your_okera_dir>

Relational Database Credentials

The settings OKERA_DB_URL, OKERA_DB_USERNAME, and OKERA_DB_PASSWORD are the end point and credentials for the DM’s RDB backend. The OKERA_DB_URL should be host:port (typically port 3306) of the running MySQL/Aurora instance.

Example: Setting the database location and credentials

export OKERA_DB_URL=okera.xyzzzz.rds.amazonaws.com:3306
export OKERA_DB_USERNAME=okera
export OKERA_DB_PASSWORD=<password>

OKERA_CATALOG_ADMINS

ODAS clusters by default start up with one user which has admin on the system. The admin users can create and manage roles, grant roles to other users and read all datasets. This default user is the system user - the user that ODAS services use to authenticate itself and depends on how authentication was configured.

With Kerberos, the admin user is the first part of the kerberos principal.

When using JSON Web Tokens (JST), the admin user is the subject in the OKERA_SYSTEM_TOKEN.

With no authentication, the admin user is ‘root’. To specify other admin users, edit the comma-separated list of admin users and groups.

Example: Setting the list of admin users and groups.

export OKERA_CATALOG_ADMINS=admin,username1,admin-group

Admins users can grant permissions to other users/groups including the ability to grant to other users. However, only admin users can create new roles.

OKERA_PORT_CONFIGURATION

This configuration specifies the ports on which cluster services run. These ports must be available for all users connecting to Okera for metadata and data. The ports are contained in a comma-separated list of service:portname:port number. If the ports are not specified, Okera randomly selects available ports on which to expose these services.

Below are the ports that are required for clients.

  • cdas_rest_server:api
  • cerebro_catalog:hms
  • cerebro_catalog:sentry
  • cerebro_planner:planner
  • cerebro_web:webui
  • cerebro_worker:worker
  • kubernetes_dashboard:admin_ui
  • grafana:ui

An example configuration:

OKERA_PORT_CONFIGURATION="okera_worker:worker:7185,cerebro_catalog:sentry:7182"
OKERA_PORT_CONFIGURATION="$OKERA_PORT_CONFIGURATION,cdas_rest_server:api:7184"
OKERA_PORT_CONFIGURATION="$OKERA_PORT_CONFIGURATION,okera_planner:planner:7183"
OKERA_PORT_CONFIGURATION="$OKERA_PORT_CONFIGURATION,okera_catalog:hms:7180"
OKERA_PORT_CONFIGURATION="$OKERA_PORT_CONFIGURATION,okera_web:webui:7186"
OKERA_PORT_CONFIGURATION="$OKERA_PORT_CONFIGURATION,grafana:ui:10000"
export OKERA_PORT_CONFIGURATION="$OKERA_PORT_CONFIGURATION,kubernetes_dashboard:admin_ui:10001"
Kerberos Configuration

To enable Kerberos, specify these configurations:

  • OKERA_KERBEROS_PRINCIPAL

    Principal for non-REST Okera services.

  • OKERA_KERBEROS_HTTP_PRINCIPAL

    Principal for REST Okera services. By convention, this configuration should start with HTTP/ and is highly recommend for maintaining compatibility with existing tools.

  • OKERA_KERBEROS_KEYTAB_FILE

    Path to the keytab file containing both principals. This path should be accessible from the Deployment Manager, either on the local file system or in S3.

  • OKERA_KERBEROS_ENABLED_REALMS

    List of comma-separated cross realms from which to accept connections. This configuration does not need to be specified if only connections from the OKERA_KERBEROS_PRINCIPAL realm are allowed to connect.

Example: Setting up Kerberos principal variables

export OKERA_KERBEROS_PRINCIPAL="okera/cname.example.com@REALM.com"
export OKERA_KERBEROS_HTTP_PRINCIPAL="HTTP/cname.example.com@REALM.com"
export OKERA_KERBEROS_KEYTAB_FILE="/etc/okera.keytab"

For more information on how to set up a Kerberized cluster, see Kerberos Cluster Setup.

JSON Web Tokens (JWT) Configuration

To enable authentication using JSON Web Tokens (JWT), specify these configurations:

  • OKERA_JWT_PUBLIC_KEY

    Path to the public key to decrypt tokens. It must be accessible on the Deployment Manager machine, but can be on the local machine or in S3.

  • OKERA_JWT_ALGORITHM

    Algorithm used to decrypt tokens. Must be set to RSA256 or RSA512.

  • OKERA_JWT_AUTHENTICATION_SERVER_URL

    URL used to authenticate JWT tokens. A POST call is issued to this URL, specifying the token to authenticate. This cannot be set if the OKERA_JWT_PUBLIC_KEY is set.

  • OKERA_SYSTEM_TOKEN

    Path to a file that contains the token (on a single line) that Okera services uses to authenticate itself. The subject for this token acts as the Okera system user.

Example: Setting the JWT token variables with a local path

export OKERA_JWT_PUBLIC_KEY="/etc/jwt.512.pub"
export OKERA_JWT_ALGORITHM="RSA512"

Example: Setting the JWT token variables for a REST call and a token to pass

export OKERA_JWT_AUTHENTICATION_SERVER_URL="http://sso/verify-token"
export OKERA_SYSTEM_TOKEN="/etc/okera.token"

For more about authentication with JWTs, see ODAS Authentication.

HTTPS Configuration

To enable HTTPS, specify the SSL certificate and private key that Okera uses. Setting the certificate and key enables HTTPS on the REST server and Web UI.

  • OKERA_SSL_CERTIFICATE_FILE

    Path to the certificate file.

  • OKERA_SSL_KEY_FILE

    Path to the file with the private key.

  • OKERA_SSL_FQDN (optional)

    Fully qualified domain name for the cluster. If set, this must be a valid “Subject Alternate Name” in the certificate. This can be the fully qualified domain name for any node in this cluster (typically the CNAME for the REST service). This is required for some clients, which often do not allow navigating IP addresses with SSL is enabled, such as newer versions of Chrome.

Example: Setting the HTTPS environment variables

export OKERA_SSL_CERTIFICATE_FILE=/etc/okera.cert
export OKERA_SSL_KEY_FILE=/etc/okera.key
export OKERA_SSL_FQDN=rest-server.okera.com
LDAP Configuration

For information on how to set LDAP Basic Auth related environment variables, see LDAP Basic Auth.

For more details on interactions with authenticated Okera, see:

OAuth Configuration

Okera can be deployed with OAuth enabled for easier Web UI login. For more information, see

The Okera Web Portal can be configured to use shared cookies for authentication. For this to work, Okera needs to be configured with the shared cookie name and the shared cookie domain. The following environment variables need to be set either on the Deployment Manager or on the cerebro_web:webui container.

  • OKERA_SHARED_COOKIE_NAME

    The name of the cookie that contains the login token.

  • OKERA_SHARED_COOKIE_DOMAIN

    The full name of the (sub)domain on which OKERA_SHARED_COOKIE_NAME is shared. This value will be passed verbatim to the browser’s cookie domain.

Example: Setting the Shared Cookie values

export OKERA_SHARED_COOKIE_NAME=my_company_token
export OKERA_SHARED_COOKIE_DOMAIN=subdomain1.example.com

Given the above example, the Okera Web Portal will attempt to read the cookie named my_company_token for a value assumed to be the token. The Okera Web Portal must be hosted on subdomain1.example.com or a subdomain of it in order to read that value. Additionally, if the Okera Web Portal generates a new token, it will write it to the my_company_token cookie, sharing it across subdomain1.example.com.

Start the Deployment Manager

After setting the environment variables and sourcing them, start the Deployment Manager.

Example: Deployment Manager startup

/opt/okera/deployment-manager/bin/deployment-manager

This should output System up and running and the server hostport.

Note: This step is not expected to take more than 30 seconds. It usually indicates a configuration issue if the Deployment Manager does not come up in a reasonable amount of time.

The Deployment Manager runs as a daemon Its logs can be viewed by entering:

less /var/log/okera/deployment-manager.log

Any configuration issues should be visible at the end of the log.

Deployment Manager Config Validator

It is possible to run the Deployment Manager config validation without restarting the Deployment Manager process. This is convenient to test new configs or to quickly diagnose problematic configs.

To run the Deployment Manager config validation without restarting the Deployment Manager, run the deployment-manager script with the verify-configs-only argument.

Example: Validating a DM config without starting the DM

/opt/okera/deployment-manager/bin/deployment-manager verify-configs-only
...
*************************************************************
  Config validation successful.
*************************************************************

Configure ocadm

Configure ocadm to connect with a running Deployment Manager.

Run the following:

/opt/ocadm configure --server <DM_host:port>
/opt/ocadm status

Running the status command should return the result: Service available. Status: Running.

Next

Now you’re ready to create a cluster.