3. Okera Install - Setting up a ODAS Cluster

To create and launch an ODAS cluster, the Deployment Manager must be running.

Configure Node Settings

Cluster launch is broken up into two parts: actions required on the Deployment Manager and actions required on each launched EC2 instance.

Actions required on the Deployment Manager are referred to as the launch script or cluster launch script. One of those two scripts are required, and when called, should provision either a new EC2 instance or a new autoscaling group, respectively. The configured script is run from the DM master and should specify all of the required configurations described in the Cluster machine section in the prerequisites (for example, VPC, security groups, IAM Roles, etc). This script is the best place to tag machines or do other setup required by your organization. Details about launch scripts and cluster launch scripts can be found in Cluster Launch Plugin API.

Actions required on each launched EC2 machine are referred to as the init scripts. The init scripts are an optional set of scripts that run when the instance is launched on the instance machine. For example, one of these scripts could install any monitoring software that you use or configure the machine to use custom package repo locations.

Okera has provided a template launch script in the /opt/okera/deployment-manager/bin/start-ec2-machine-example.sh directory. Copy the script and adapt it to your requirements. The user-configurable values are at the top of the file, marked USER in a comment. At a minimum, a subnet ID and security group ID must be added to the script.

Example: Migrating the launch script for editing

cp /opt/okera/deployment-manager/bin/start-ec2-machine-example.sh /etc/okera/launch-ec2.sh

To verify the launch script, Okera recommends running the script with the --dryrun optional. If the configurations are valid, the script should exit with a status of 0.

Example: Launch script validation

/etc/okera/launch-ec2.sh --dryrun
Dry run succeeded
echo $?
0

When a launch script is invoked without the --dryrun flag, it is expected to return a string of the form:

<instance_ID>,<public_IP>,<private_IP>

Supplied cluster launch scripts do not need to support the –dryrun flag, as the associated AWS cli commands do not support that functionality.

Cluster launch scripts are expected to return the same output as the launch scripts with the exception that cluster launch scripts can return n lines, where each is formatted:

<instance_ID>,<public_IP>,<private_IP>

and each represent a host that was launched within the specified autoscaling group (ASG).

Note: The public IP value is optional. If your setup does not use it, its spot will appear blank.

Prep the Cluster

We’re almost ready to start a cluster from the CLI.

First create an environment. An environment captures all the configurations required to launch clusters.

Examples of environments might be for development or production, with the basic template:

ocadm environments create --name=<environment_name> --provider=AWS --launchScript=<DM_path_to_launch_script> --initScripts=<comma-separated_list_of_launch_scripts>

To run your cluster on an AWS autoscaling group (ASG), specify a –clusterLaunchScript instead of a –launchScript (those two flags are mutually exclusive)

ocadm environments create --name=<environment_name> --provider=AWS --clusterLaunchScript=<DM_path_to_cluster_launch_script> --initScripts=<comma-separated_list_of_launch_scripts>

Example: Environment definition

ocadm environments create --name=DevEnv --provider=AWS --launchScript=/etc/okera/launch-ec2.sh

Example: Environment definition that uses AWS ASGs

ocadm environments create --name=DevEnv --provider=AWS --clusterLaunchScript=/etc/okera/launch-asg.sh

Create a Cluster

Next we create at least one cluster in the environment. Multiple clusters can be created with the same environment.

Note: See Cluster Types for the available option when using the --type parameter.

Cluster creation follows this template:

ocadm clusters create --name=<cluster_name> --numNodes=<node_count> --type=STANDALONE_CLUSTER --environmentid=<environment_ID>

Where the <environment_ID> is the numeric ID assigned after environment creation in the previous section.

Example: Creating a new cluster for environment 1

ocadm clusters create --name=fintech_prod --numNodes=1 --type=STANDALONE_CLUSTER --environmentid=1

List the running clusters.

ocadm clusters list

The newly created cluster should be returned and shown as “Provisioning”. You can see more details, including how long it has been in this state using the Okera CLI.

ocadm clusters status --detail <cluster_ID>

It might be convenient to watch the cluster until the state transitions to “READY”. It takes a few minutes to provision the machines, install ODAS, and start up all the services.

watch ocadm clusters status <cluster_ID>

At this point all of Okera is up and running!

To see the externally configured endpoints, run the clusters endpoints command.

ocadm clusters endpoints 1

Note: The host and port of the Web UI are listed with the component named okera_web:webui. The Web UI will respond at http://<host>:<port>, or https://<host>:<port> if using SSL.

Restarting a Failed Cluster after Addressing Issues

During an Okera environment creation, the launch scripts, init scripts, and config files are copied to the Okera install directory. Copying the config files is done to ensure that the clusters using that environment are unaffected by any new environments or clusters with different configurations.

If the script or config files in the environment contain errors, then the errors need to be corrected and the affected clusters restarted. This includes issues with the launch-ec2.sh script and any specified init scripts.

The Okera install scripts and config files are located at the $DEPLOYMENT_MANAGER_INSTALL_DIR/env/<environment_id> directory. The DEPLOYMENT_MANAGER_INSTALL_DIR environment variable defaults to /etc/okera.

Starting Multiple Okera Catalogs Sharing the Same Metadata

Okera catalogs can be configured to share underlying metadata. This feature is supported for “active-passive” configurations. When creating the multiple clusters, supply the catalogDbNamePrefix option using the same argument value in both cases. Catalogs that share the same name also share the same metadata.

Example: Creating two clusters that share metadata:

ocadm clusters create --name=prod --numNodes=1 --type=STANDALONE_CLUSTER --environmentid=1 --catalogDbNamePrefix=metadata
ocadm clusters create --name=prod-backup --numNodes=1 --type=STANDALONE_CLUSTER --environmentid=1 --catalogDbNamePrefix=metadata

Note: If the --catalogDbNamePrefix was not explicitly specified during cluster creation, then it is the name of the cluster.