Deploying ODAS on EC2

This document will guide you through installing ODAS on EC2 machines. You will walk through the following steps:

  1. Provision an EC2 instance for ODAS.
  2. Deploy a single-node base unconfigured cluster.
  3. Updating the configuration of your cluster.
  4. Joining more instances to your cluster manually.
  5. Creating a Launch Template and Auto-Scaling Group to scale your cluster.

Prerequisites

  1. Required: Security groups
  2. Required: IAM role
  3. Optional: S3 location to store logs (required for multi-node clusters)
  4. Optional: RDS instance

Provisioning an EC2 Instance

In this step we will provision an EC2 instance to install our ODAS cluster on. Our instance will have the following configuration:

  1. Amazon Linux 2 64-bit AMI.
  2. t3.2xlarge instance type.
  3. Use the security group configured in the Prerequisites section.
  4. Have the IAM role configured in the Prerequisites section attached.
  5. Have 120GB EBS volume attached.

To create the instance, navigate to the EC2 Launch Instance wizard:

  1. Choose the Amazon Linux 2 AMI (HVM), SSD Volume Type AMI and press "Select".
  2. Choose the t3.2xlarge instance type and press "Configure Instance Details".
  3. Choose your desired VPC and Subnet (and any other normal EC2 configuration you use).
  4. Select the IAM role created for the ODAS cluster as per the prerequisites section and press "Add Storage".
  5. Configure the volume to be 120GB in size of type gp2 and press "Add Tags".
  6. Add any tags you need for this instance and press "Configure Security Group".
  7. Select the Security Group created for the ODAS cluster as per the prerequisites section and press "Review and Launch".
  8. Ensure all the settings are correct and press "Launch".

Once the instance is created, SSH into the instance. The below instructions are all done within this SSH session. All commands can be run as the default ec2-user unless they explicitly use sudo.

Deploy a Single-Node Unconfigured Cluster

When you are SSHed onto the machine, to download and run the ODAS installer, run the following command:

$ curl https://okera-release-uswest.s3.amazonaws.com/1.5.0/gravity/install.sh | sh -
Created /home/ec2-user/okera-1.5.0
Preparing for ODAS v1.5.0 Installation...
Downloading https://okera-release-uswest.s3.amazonaws.com/1.5.0/gravity/odas.tar to /home/ec2-user/okera-1.5.0/odas.tar...
Untarring /home/ec2-user/okera-1.5.0/odas.tar...
Preparing for cluster bootstrap...
2019/08/27 05:04:52 [validate:diskspace] passed
2019/08/27 05:04:52 [validate:mysql] skipped
2019/08/27 05:04:52 [validate:ldap] skipped
2019/08/27 05:04:52 [validate:s3-read/AUTOTAGGER_CONFIGURATION/json] skipped
2019/08/27 05:04:52 [validate:s3-read/JWT_PUBLIC_KEY/pubkey] skipped
2019/08/27 05:04:52 [validate:s3-read/SYSTEM_TOKEN/token] skipped
2019/08/27 05:04:52 [validate:adls-gen1-read/AUTOTAGGER_CONFIGURATION/json] skipped
2019/08/27 05:04:52 [validate:adls-gen1-read/JWT_PUBLIC_KEY/pubkey] skipped
2019/08/27 05:04:52 [validate:adls-gen1-read/SYSTEM_TOKEN/token] skipped
2019/08/27 05:04:52 [validate:s3-write/WATCHER_S3_REGION/WATCHER_LOG_DST_DIR] skipped
2019/08/27 05:04:52 [validate:s3-write/WATCHER_S3_REGION/WATCHER_AUDIT_LOG_DST_DIR] skipped
2019/08/27 05:04:52 [validate:adls-gen1-write/WATCHER_LOG_DST_DIR] skipped
2019/08/27 05:04:52 [validate:adls-gen1-write/WATCHER_AUDIT_LOG_DST_DIR] skipped
2019/08/27 05:04:52 [validate:local-dns] passed
2019/08/27 05:04:52 Created 'cluster-config.yaml'
2019/08/27 05:04:52 Created 'create.cmd'
2019/08/27 05:04:52 Created 'join.cmd'
2019/08/27 05:04:52 Created 'join.asg.cmd'
2019/08/27 05:04:52 If you are creating a new cluster, you can execute:
2019/08/27 05:04:52
2019/08/27 05:04:52   cd /home/ec2-user/okera-1.5.0 && sudo ./gravity install --token=WDEPlggIpP --advertise-addr=10.1.10.182 --cloud-provider=generic --config=cluster-config.yaml --pod-network-cidr="172.23.0.0/16" --service-cidr="172.34.0.0/16"
2019/08/27 05:04:52
2019/08/27 05:04:52 If you later want to join a new node to this cluster, you can execute the following on that node:
2019/08/27 05:04:52
2019/08/27 05:04:52   sudo ./gravity join --token=WDEPlggIpP --role=worker 10.1.10.182 --cloud-provider=generic
2019/08/27 05:04:52
2019/08/27 05:04:52 You can reference 'create.cmd', 'join.cmd' and 'join.asg.cmd' for future use

Note

The install.sh script simply downloads odas.tar from the same location, unpacks it and runs the installation commands. You can look at the contents of the script and run them manually if preferred.

This has prepared this node to have the cluster installed on it, as well outputted instructions for adding more nodes to this cluster (which we will use in later sections).

As instructed by the above output, we can now install the cluster:

$ cd /home/ec2-user/okera-1.5.0 && sudo ./gravity install --token=WDEPlggIpP --advertise-addr=10.1.10.182 --cloud-provider=generic --config=cluster-config.yaml --pod-network-cidr="172.23.0.0/16" --service-cidr="172.34.0.0/16"
Tue Aug 27 05:07:28 UTC Starting installer
Tue Aug 27 05:07:28 UTC Preparing for installation...
Tue Aug 27 05:07:35 UTC Installing application odas:1.0.0-1.5.0
Tue Aug 27 05:07:35 UTC Starting non-interactive install
Tue Aug 27 05:07:35 UTC Auto-loaded kernel module: br_netfilter
Tue Aug 27 05:07:35 UTC Auto-loaded kernel module: iptable_nat
Tue Aug 27 05:07:35 UTC Auto-loaded kernel module: iptable_filter
Tue Aug 27 05:07:35 UTC Auto-loaded kernel module: ebtables
Tue Aug 27 05:07:35 UTC Auto-loaded kernel module: overlay
Tue Aug 27 05:07:35 UTC Auto-set kernel parameter: net.ipv4.ip_forward=1
Tue Aug 27 05:07:35 UTC Auto-set kernel parameter: net.bridge.bridge-nf-call-iptables=1
Tue Aug 27 05:07:36 UTC Successfully added "master" node on 10.1.10.182
Tue Aug 27 05:07:36 UTC All agents have connected!
Tue Aug 27 05:07:36 UTC Starting the installation
Tue Aug 27 05:07:37 UTC Operation has been created
Tue Aug 27 05:07:38 UTC Execute preflight checks
Tue Aug 27 05:07:40 UTC Configure packages for all nodes
Tue Aug 27 05:07:43 UTC Bootstrap master node ip-10-1-10-182.us-west-2.compute.internal
Tue Aug 27 05:07:47 UTC Pull packages on master node ip-10-1-10-182.us-west-2.compute.internal
Tue Aug 27 05:08:22 UTC Install system software on master nodes
Tue Aug 27 05:08:23 UTC Install system package teleport:3.0.5 on master node ip-10-1-10-182.us-west-2.compute.internal
Tue Aug 27 05:08:25 UTC Install system package odas-planet:5.5.22-1.5.0-planet on master node ip-10-1-10-182.us-west-2.compute.internal
Tue Aug 27 05:08:40 UTC Wait for kubernetes to become available
Tue Aug 27 05:08:57 UTC Bootstrap Kubernetes roles and PSPs
Tue Aug 27 05:09:00 UTC Configure CoreDNS
Tue Aug 27 05:09:01 UTC Create user-supplied Kubernetes resources
Tue Aug 27 05:09:02 UTC Populate Docker registry on master node ip-10-1-10-182.us-west-2.compute.internal
Tue Aug 27 05:10:11 UTC Wait for cluster to pass health checks
Tue Aug 27 05:10:12 UTC Install system application dns-app:0.3.0
Tue Aug 27 05:10:24 UTC Install system application logging-app:5.0.2
Tue Aug 27 05:10:29 UTC Install system application monitoring-app:5.5.0
Tue Aug 27 05:10:44 UTC Install system application tiller-app:5.5.2
Tue Aug 27 05:10:57 UTC Install system application site:5.5.15
Tue Aug 27 05:12:27 UTC Install system application kubernetes:5.5.15
Tue Aug 27 05:12:28 UTC Install application odas:1.0.0-1.5.0
Tue Aug 27 05:12:44 UTC Connect to installer
Tue Aug 27 05:12:45 UTC Enable cluster leader elections
Created user "admin"
Tue Aug 27 05:12:52 UTC Operation has completed
Tue Aug 27 05:12:52 UTC Installation succeeded in 5m17.005696994s

Cluster endpoints:
    * Authentication gateway:
        - 10.1.10.182:32009
    * Cluster management URL:
        - https://10.1.10.182:32009

Application endpoints:
    * odas:1.0.0-1.5.0:
        - ODAS Planner:
            - tcp://10.1.10.182:12050
        - ODAS Worker:
            - tcp://10.1.10.182:13050
        - ODAS REST:
            - http://10.1.10.182:8089
            - http://10.1.10.182:8083
        - ODAS Presto:
            - https://10.1.10.182:14050

After 5-10 minutes, the command will complete. When it does, your cluster is doing it's final set up steps and should be ready within 2-3 minutes. You can see when your cluster is ready by running:

$ ./okctl status
2019/08/27 05:18:13 ready

Once the cluster is ready, you can use the okctl endpoints command to find the URL for the webui:

$ ./okctl endpoints
2019/08/27 05:30:55 cdas-rest-server:api (type: NodePort)
2019/08/27 05:30:55   10.1.10.182:8089
2019/08/27 05:30:55 cdas-rest-server:webui (type: NodePort)
2019/08/27 05:30:55   10.1.10.182:8083
2019/08/27 05:30:55 cerebro-planner:planner (type: NodePort)
2019/08/27 05:30:55   10.1.10.182:12050
2019/08/27 05:30:55 cerebro-worker:worker (type: NodePort)
2019/08/27 05:30:55   10.1.10.182:13050
2019/08/27 05:30:55 presto-coordinator:api (type: NodePort)
2019/08/27 05:30:55   10.1.10.182:14050

Open your browser at http://10.1.10.182:8083. Since this is an unconfigured cluster, there is no authentication and you can type in root to log in to the web UI as an administrator.

Configuring Your ODAS Cluster

ODAS clusters use a YAML configuration file, which you can learn more about here.

You can update the configuration of your cluster using okctl update.

To deploy the Quickstart configuration, which will add authentication and SSL:

$ ./okctl update --config configs/config-quickstart.yaml

You can copy this file (or the other example configuration files in the configs/ directory) and modify them for your deployment, and then apply them using okctl update.

Growing your ODAS Cluster Manually

You can join more nodes to your cluster by provisioning additional EC2 instances (follow the same EC2 instructions). Once the instance is created, run the following command to download the minimal installer:

$ curl -O https://okera-release-uswest.s3.amazonaws.com/1.5.0/gravity/gravity

Once downloaded, we can use the command outputted during the original cluster step to join our node to the cluster:

$ sudo ./gravity join --token=WDEPlggIpP --role=worker 10.1.10.182 --cloud-provider=generic
2019/08/27 05:30:22 UTC Connecting to cluster
2019/08/27 05:30:23 UTC Connected to existing cluster at 10.1.10.68
2019/08/27 05:30:24 UTC Operation has been created
2019/08/27 05:30:25 UTC Configure packages for the joining node
2019/08/27 05:30:26 UTC Bootstrap the joining node
2019/08/27 05:30:27 UTC Pull packages on the joining node
2019/08/27 05:31:28 UTC Install system package teleport:3.0.5
2019/08/27 05:31:29 UTC Install system package odas-planet:1.4.0-gravity
2019/08/27 05:31:47 UTC Wait for the planet to start
2019/08/27 05:32:27 UTC Operation has completed
2019/08/27 05:32:27 UTC Joined cluster in 2m4.103961218s

Note

If you did not save the original command, you can always retrieve it by going to the installation directory and finding it in the join.cmd file.

Growing your ODAS Cluster using Auto-Scaling Groups

Besides manually scaling your cluster, you can also configure an Auto-Scaling Group to automatically scale your cluster up and down.

We first create a Launch Configuration by going to the Launch Configuration Wizard:

  1. Choose the Amazon Linux 2 AMI (HVM), SSD Volume Type AMI and press "Select".
  2. Choose the t3.2xlarge instance type and press "Configure Details".
  3. Give your Launch Configuration a name (e.g. odas-launch-config).
  4. Expand "Advanced Details", and in the "User Data" section, put the contents of the join.asg.cmd file from your cluster install directory.
  5. Select the IAM role created for the ODAS cluster as per the prerequisites section and press "Add Storage".
  6. Configure the volume to be 120GB in size of type gp2 and press "Configure Security Group".
  7. Select the Security Group created for the ODAS cluster as per the prerequisites section and press "Review".
  8. Ensure all the settings are correct and press "Create Launch Configuration".

Once your Launch Configuration is created, we can create an Auto-Scaling Group that leverages it, navigating to the Auto-Scaling Group Wizard:

  1. Select your previously created Launch Configuration (e.g. odas-launch-config).
  2. Give your ASG a name, e.g. odas-asg and your initial desired number of instances (1 is fine to start with).
  3. Choose which VPC and subnet to create this ASG in. This should be the same as your original cluster node. Press "Configure Scaling Policies".
  4. Leave the default option of "Keep this group at its initial size" selected and press "Configure Notifications".
  5. Don't create any notifications, and press "Configure Tags".
  6. Add any tags you need for this instance and press "Review".
  7. Ensure all the settings are correct and press "Create Auto Scaling Group".

The number of instanced you selected as your initial desired number will be created and automatically joined to your ODAS cluster. You can grow and shrink the cluster as your scaling needs change. You may also put in scaling policies (e.g. based on CPU utilization) to automatically scale your cluster up or down.