Skip to content

nScale Enforcement Fleet Workers

Okera supports running Okera Enforcement Fleet worker nodes collocated with EMR or Dataproc worker nodes (Okera nScale). The steps described below for starting up and configuring nScale workers need to be followed in addition to the steps above for standard Okera-integrated EMR or Dataproc environments.

We can break the steps down as follows:

  1. Review encryption settings for nScale.
  2. Review deferred URI signing settings for nScale
  3. Make bootstrap changes for nScale. The updated bootstrap will include the Presto configuration for EMR with nScale Okera workers.
  4. Make Hive configuration changes for nScale.
  5. Make Spark configuration changes for nScale.

Encryption Settings for nScale

You can encrypt and decrypt tasks using Advanced Encryption Standard (AES). To specify the key, set the encryption_key_path option for RS_ARGS in Okera's configuration file to the path of the file that contains the key. There are no constraints on the size of this file (except that it cannot be empty). Okera uses SHA256 on the contents to get a 32-byte key (256-bit).

Another option for specifying the key uses two environment variables:

  • ENABLE_TASK_ENCRYPTION can be set to true or false.
  • TASK_ENCRYPTION_KEY specifies the path of the file that contains the key.

If ENABLE_TASK_ENCRYPTION is set to true but TASK_ENCRYPTION_KEY is not set, Okera attempts to use the JWT_PRIVATE_KEY as the encryption key if it is present. If it is not present, an error occurs.

Finally, when integrating with AWS EMR (but not Dataproc), you can add the --local-worker-encryption-key argument to odas-emr-bootstrap.sh to allow someone to set the path to the encryption key for use on EMR. This does not apply to Dataproc environments.

Deferred URI Signing Settings for nScale

An optional performance-tuning nScale setting called recordservice.task.plan.defer-signing-urls indicates whether presigning URIs in all tasks should be deferred for nScale requests initiated from Spark and Hive. Valid values are true (defer presigning URIs in requests) and false (continue presigning URIs in requests). The default is false.

The following server-side RS_ARGS options can be used to configure deferred URI signing for nScale:

  • Use defer_signed_urls_initial_num_tasks to specify the default number of nScale tasks for which deferred signing should be performed. The default is 64.

  • Use defer_signed_urls_initial_percent_tasks to specify the percentage of nScale tasks for which deferred signing should be performed. The default is 25%.

Okera defers signing for either the number of tasks specified by defer_signed_urls_initial_num_tasks or for the percentage of tasks specified by defer_signed_urls_initial_percent_tasks, whichever is lower.

All requests to refresh tasks containing presigned URIs are logged in the audit log.

Bootstrap Changes for nScale

To enable nScale, add some extra flags to your EMR Node Bootstrap script or your Dataproc metadata.

The following options are available.

Option Supported in Description
--external-objects-to-container EMR Reference objects stored in S3 as Okera configuration parameters. Okera pulls the objects referenced in the configuration parameters and mounts them in the nScale container, making the S3 paths available to Okera for processing. For example, this is helpful when configuring the SSL certificate and key required to start the OkeraEnsemble EMR access proxy in TLS/SSL mode:

--external-objects-to-container SSL_CERTIFICATE_FILE=s3://bucket/certificate-object, SSL_KEY_FILE=s3://bucket/key-object
--init <initScript> EMR The AWS S3 path init script to be run as part of bootstrap. This is useful to set environment variables such as HTTP_PROXY and OKERA_BITS_REGION.
--local-worker-args <space-separated arguments> EMR Use this to pass RS_ARGS to the nScale worker.
--local-worker-audit-dir <location> EMR
Dataproc
The S3 URI location or local storage location (for Dataproc) to which container's audit logs should be uploaded. With that, one way to call the script now becomes:

2.14.0 --planner-hostports <hostport> --local-worker-webui-port <webui-port> --local-worker-port <worker-port> hive presto spark-2.x
--local-worker-encryption-key EMR Specifies the worker encryption key. See Encryption Settings for nScale.
--local-worker-env-vars EMR Environment variables for the nScale worker. These are supplied in the following format (space-separated with a -e before each key-value pair): -e envKey1=value -e envKey2=value.
--local-worker-log-dir <location> EMR
Dataproc
The S3 URI location or local storage location (for Dataproc) to which container's logs should be uploaded.
--local-worker--port <port> EMR
Dataproc
This option should be the last option set, right before the <list of components>.
--local-worker-version <version> EMR
Dataproc
By default, nScale workers start up with 2.14.0 workers. This option allows you to bootstrap with a custom version of nScale Okera workers.
--local-worker-webui-port <port> EMR The port on which the nScale workers debug UI is exposed. By default, the UI is not exposed.
--planner-hostports <hostports> EMR
Dataproc
Link to the cerebro_planner:planner endpoint.

Hive Configuration Changes for nScale

Note: Hive configuration is only supported for nScale in EMR environments. It is not supported for nScale in Dataproc environments.

To interact with the nScale worker, Hive requires that the recordservice.workers.local-port configuration setting be specified. In hive-site, specify the recordservice.workers.local-port key, with the local worker port (<local-worker-port>) as its value. With this change, the hive-site.xml configuration settings look like this:

[
  {
    "Classification": "hive-site",
    "Properties": {
      "hive.fetch.task.conversion": "minimal",
      "hive.metastore.rawstore.impl": "com.cerebro.hive.metastore.CerebroObjectStore",
      "recordservice.planner.hostports": "<planner-host>:<planner-port>",
      "recordservice.workers.local-port" "<local-worker-port>"
    }
  }
]

Spark Configuration Changes for nScale

Note: Spark configuration changes are only supported for nScale in EMR environments. They are not necessary for nScale in Dataproc environments.

To interact with the nScale worker, Spark requires that two configuration settings be specified:

  • In spark-defaults, specify the spark.recordservice.workers.local-port key, with the local worker port number (<local-worker-port>) as its value.
  • In spark-hive-site, specify the recordservice.workers.local-port key, with the local worker port number (<local-worker-port>) as its value.

With these updates, the configuration settings become:

[
  {
    "Classification":"spark-defaults",
    "Properties": {
       "spark.recordservice.planner.hostports":"odas-planner-1.internal.net:12050",
       "spark.recordservice.workers.local-port":"<local-worker-port>"
     }
  },
  {
    "Classification":"spark-hive-site",
    "Properties":{
      "recordservice.planner.hostports":"odas-planner-1.internal.net:12050",
      "recordservice.workers.local-port":"<local-worker-port>"
    }
  }
]