Skip to content

Okera Version 2.12 Release Notes

This topic provides Release Notes for all 2.12 versions of Okera.

2.12.1 (10/29/2022)

Support for Referencing Amazon S3 Objects in Okera Configuration Parameters in nScale Amazon EMR Deployments

With this release, you can now reference objects stored in Amazon S3 as Okera configuration parameters for odas-emr-bootstrap. Okera pulls the objects referenced in the configuration parameters and mounts them in the nScale container, making the Amazon S3 paths available to Okera for processing. For example, this is helpful when configuring the SSL certificate and key required to start the OkeraEnsemble Amazon EMR access proxy in TLS/SSL mode:

--external-objects-to-container SSL_CERTIFICATE_FILE=s3://bucket/certificate-object, SSL_KEY_FILE=s3://bucket/key-object

If SSL_CERTIFICATE_FILE specifies the path to the SSL certificates file in Amazon S3 and SSL_KEY_FILE specifies the path to the SSL key file in Amazon S3, these paths can be used by the OkeraEnsemble access proxy for any necessary TLS/SSL processing.

Security Vulnerabilities (CVEs) Addressed

Okera uses Snyk and GitHub Advanced Security for security vulnerability scanning.

Bug Fixes

The following bugs were fixed in this release:

  • Updated the Okera Spark3 connector to address an incompatibility resulting from a Databricks library change in Okera-supported Databricks runtime versions 9.1 LTS and later.

  • Fixed an issue where some sensitive environment variables were not always redacted in logs.

2.12.0 (9/30/2022)

UI Login Page

The Okera UI login page has slightly changed its appearance. The options available to you on the login page remain unchanged.

New Call-to-Action UI Home Page

In this release, a new Call-to-Action Home page is introduced. Instead of account detail information, this new Home page is designed to help new users get started using Okera to protect their data. Account detail information that appeared on the page prior to this change has been moved to a user account details dialog. See UI User Account Details.

The following images show the differences between the original and new Home page.

Images

Original Okera v2.11 and earlier Home page

Call-to-Action Home page

The menu on the left side of the Home page remains unchanged, but the rest of the Call-to-Action Home page is divided into sections that are intended to be performed from top to bottom.

Note: Depending on the roles and permissions assigned your Okera username, you will see either the Call-to-Action Home page or the original User Profile Home page. In addition, the options available to you vary, based on your roles and permissions. The original format of the Home page is still used for users who do not have enough permissions to use the Call-to-Action Home page. This original format of the Home page is called the User Profile Home page. For information about how roles and permissions affect your Home page view, see Manage Access to UI Features and the Home Page Access Reference.

For complete information about the new Call-to-Action Home page, see The Okera Call-to-Action Home Page. For information about the User Profile Home page, see The Okera User Profile Home Page.

UI User Account Details

With this release, you can now obtain account details for the user logged into the UI by selecting the Account details option from the menu that appears when you select the username in the lower left corner of the UI screen. Account details are no longer available on the Home page. The following sample dialog shows account details for the admin user:

User Account Details dialog

See View User Account Details.

Connection Plaintext Credential Support

You can now specify that a connection uses plaintext credentials for the username and password, instead of paths to the secrets credential files. A new option has been added to the UI dialog that appears when you create a connection.

Connection Credential Options

Select Plaintext credentials to specify plaintext credentials; select Secret credentials file to specify a path to the secrets credential files for the connection username and password.

In addition, a new connection property okera.plaintext.credentials has been added that can be used if you create your connections using the Okera DDL and want to use plaintext credentials. Set the value of this property to true in the connection_properties section of the DDL if you want to use plaintext credentials. Set it to false if you want to use secure credentials (false is the default).

When plaintext credentials are specified, Okera generates a secret username and password for the plaintext credentials and provides the secret strings for these to your AWS, Azure, or GCP secrets manager, or Kubernetes path. To enable the use of plaintext credentials, you must specify the following new configuration parameters in the YAML file (the default file is values.yaml) for your Okera cluster:

  • Set the new configuration parameter OKERA_SECRET_PREFIX to any prefix string you want that will help you identify that a secret is generated and used by Okera. The prefix is appended in the string for the secrets file name. There is no default.

  • Set the new configuration parameter OKERA_SECRET_STORE to the type of secrets manager (AWS, Azure, or GCP) or Kubernetes path used to store the generated secret files. Valid values are AWSSM (AWS Secrets Manager), AWSPS (AWS Parameter Store), AZUREKV (Azure Key Vault), FILE (the Kubernetes path), or GCPSM (GCP Secret Manager).

For information about creating connections, see Create a Connection Using the UI and Create a Connection Using the DDL. For information about configuration parameters and updating the Okera YAML file, see Okera Configuration Parameter Reference and The Okera Configuration File.

UI Crawler Updates

The following updates have been made for crawlers in this release.

  • Users who have access granted on a crawler can now see the Registration page.

  • Crawler action buttons, including the Create Crawler button, are now disabled when a user has no permissions to use them.

  • Crawlers for different connections can now run concurrently. You can control how many can run concurrently using the new MAINTENANCE_TASKS_THREAD_LIMIT configuration parameter. This parameter specifies the maximum number of Okera background tasks (which includes crawlers) that can run concurrently. Valid values are positive integers. The default is 1. If you need to set this value higher than 8, please contact Okera technical support.

  • After creating a crawler on the Connections page in the UI, you are now redirected to the crawler details on the Registration page.

Dataset Original Column Types

This release introduces a new set of dataset (table) properties (TBLPROPERTIES) for JDBC-backed tables that store the original types for dataset columns. The prefix for these properties is jdbc.cols.original-type. You can see these properties on the Dataset Details tab in the Okera UI.

Snowflake Connection Policy Synchronization Changes

The following changes were made to Okera's policy synchronization enforcement in this release:

  • The policy synchronization enforcement mechanism used for Snowflake connections is now no longer enabled by default. You must enable it manually using the POLICY_SYNC_SCHEDULER_ENABLED configuration parameter or the okera.policy_sync.enabled advanced parameter in your Snowflake connection.

  • You can now create an Okera role hierarchy in Snowflake under the okera_role you specified when you configured Snowflake for Okera policy synchronization. During policy synchronization, Okera generates Snowflake roles dedicated to Okera policy synchronization for every Snowflake user. Depending on how many Snowflake users you have, the number of roles granted to the okera_role could be significant. To minimize Okera's role footprint in your Snowflake environment, enable the Okera role hierarchy. When enabled, all the Okera-generated Snowflake roles will be stored in a hierarchy under the primary Okera service role (okera_role) that you created when you configured your Snowflake environment for Okera. (By default, the primary Okera service role is called SERVICE_OKERA_ROLE.)

    To enable this feature, a new Snowflake connection property has been added: okera.service_role_grant.enabled. Set this parameter to true to enable the Okera role hierarchy; set it to false to disable the Okera role hierarchy. The default is false. Add this property in the Snowflake connection definition, under Advanced Properties. See Advanced Snowflake Connection Properties.

  • Specifying an Okera role in a Snowflake connection definition is now required for Okera's policy synchronization enforcement. Specify the Snowflake role created for Okera to use for the connection (by default SERVICE_OKERA_ROLE). See Step 3. Tailor the Script.

  • You can now override the settings of the EXTERNAL_OKERA_SECURE_POLICY_DB and EXTERNAL_OKERA_SECURE_POLICY_SCHEMA configuration parameters for an individual Snowflake connection using two new advanced connection parameters:

    Parameter Description
    udfDb Overrides the setting of the Okera configuration parameter EXTERNAL_OKERA_SECURE_POLICY_DB for this connection. The only valid value is the name of a Snowflake database dedicated for Okera use. The default is OKERA_UDFS.
    udfSchema Overrides the setting of the Okera configuration parameter EXTERNAL_OKERA_SECURE_POLICY_SCHEMA for this connection. The only valid value is the name of a default schema in the Snowflake database dedicated for Okera use. The default is PUBLIC.

  • Snowflake synchronization performance for tagged users has now been improved, but it may take up to two hours for Okera to identify the tagged users. This is due to the rate at which Snowflake updates its account_usage data.

Tagging Updates

Several updates were made to tagging in this release.

  • If a user has any permission for tags, including DROP and CREATE permissions, they can now see tags on the Tags page. In past releases, users were only able to see tags for which they had ALL, ADD_ATTRIBUTE, or REMOVE_ATTRIBUTE permissions and they could not see tags for which they had DROP or CREATE permissions.

  • Users without access to the datasets that have been tagged will no longer get an authorization error message, but will get a message that says "No datasets tagged."

Blocking Access to the Okera UI for Tablets

This release introduces the ability to block access to the Okera UI on tablets. The configuration parameter, BLOCK_WEB_UI_FOR_MOBILE_CLIENTS, controls this behavior. Valid values for this parameter are true (the UI is blocked on mobile devices and tablets) and false (the UI is not blocked on mobile devices and tablets). The default is false.

OkeraEnsemble nScale Amazon S3 Bucket Access Enhancements

This release enhances OkeraEnsemble nScale mode deployment in Amazon EMR Spark environments by supporting access control of both Amazon S3 buckets defined for Amazon's assume secondary role feature, and optionally those buckets to which the Okera cluster is granted access using its IAM permission.

Since nScale deploys the OkeraEnsemble access proxy with least-privilege access to Amazon EMR, it has no IAM permissions naturally and retrieves its credentials to sign Amazon S3 requests from the Okera Policy Engine (Planner). Consequently, when you deploy OkeraEnsemble in nScale mode, you must provide access to the Amazon S3 buckets using either of two methods:

  1. Using Amazon S3's assume secondary role feature. For Amazon S3 buckets that use assume secondary roles (bucket role map), the OkeraEnsemble access proxy retrieves the AWS Security Token Service (STS) credentials associated with the Amazon Resource Name (ARN) for the Amazon S3 bucket.

  2. By setting a new OKERA_SYSTEM_IAM_ROLE_ARN configuration parameter in the Okera configuration file to the IAM Amazon Resource Name (ARN) associated with the Okera cluster. When this is activated, Okera can grant OkeraEnsemble nScale users access to buckets to which the Okera cluster has access by permission through its IAM role.

For more information about OkeraEnsemble nScale mode deployment in Amazon EMR environments, see OkeraEnsemble nScale Mode Deployment in Amazon EMR Environments.

OkeraEnsemble nScale System Token Duration Controls

You can now specify the duration, in minutes, of the JWT system token for OkeraEnsemble nScale processing. A new environment variable, SYSTEM_TOKEN_DURATION_MIN, can be set on the nScale container using the Okera Amazon EMR odas-emr-bootstrap script to configure the duration of the Okera system token. For example, passing the following arguments with the odas-emr-bootstrap.sh script will configure the system token duration time to 300 minutes. Valid values are positive integers. The default value is equivalent to one day (1440 minutes).

--local-worker-env-vars "-e SYSTEM_TOKEN_DURATION_MIN=300"

This configuration setting only works when the nScale proxy is configured using JWT_PRIVATE_KEY and not with SYSTEM_TOKEN. When configured using JWT_PRIVATE_KEY, the nScale access proxy generates its own token and the SYSTEM_TOKEN_DURATION_MIN setting determines how long that token is good for. When configured with SYSTEM_TOKEN, the SYSTEM_TOKEN_DURATION_MIN setting has no effect because the JWT token identified by the SYSTEM_TOKEN path includes an embedded expiration time that cannot be governed by SYSTEM_TOKEN_DURATION_MIN setting. If both JWT_PRIVATE_KEY and SYSTEM_TOKEN are specified, the JWT_PRIVATE_KEY is used and the SYSTEM_TOKEN is ignored.

GCP Dataproc Support ( Preview Feature)

This release introduces support for Google Cloud Platform Dataproc. For more information, see Google Cloud Platform Dataproc Integration.

AWS IMDSv2 Support

This version introduces support for AWS Instance Metadata Service version 2 (IMDSv2). To enable this support, set the new AWS_USE_IMDS_V2 configuration parameter in your Okera configuration file to true. The default is false. If IMDSv2 is enabled, the EC2 instance metadata option HttpPutResponseHopLimit must be greater than 1.

Updated Input to LDAP Filtering

In this release, we have updated Okera's input for LDAP filtering. Two new configuration parameters have been defined to allow you to specify separate base DNs (distinguished names) for users and groups for LDAP server searches during its authentication processing.

  • Use GROUP_RESOLVER_LDAP_USER_BASE_DN to specify the base DN for users.
  • Use GROUP_RESOLVER_LDAP_GROUP_BASE_DN to specify the base DN for groups.

See Okera Configuration Parameter Reference for a complete list of the configuration parameters available to you for Okera configurations. For information about LDAP group resolution, see Configuring LDAP Group Resolution.

Trino Support

With this release, Okera supports Trino.

API Updates

  • A new endpoint, GET /api/v2/info has been added that allows admins to list the current set of configuration parameters in use by your Okera system.

  • The DELETE /api/v2/connection endpoint now deletes all roles and other objects created by Okera for a Snowflake connection.

  • A new endpoint, GET /api/v2/datasets has been introduced that returns a list of datasets to which the requesting user has access. This endpoint has the following parameters: limit, pageToken, databases, connection, privilege, tagMatchLevel, tags, and namePattern.

  • A new endpoint GET /api/v2/tags has been introduced that returns a paginated list of tags or returns a specific tag when a name is provided to the endpoint. If no specific name is specified, the endpoint has the following parameters: limit, pageToken, namespacePattern, namePattern, and privileges.

For information about any Okera API endpoint, see the Okera API documentation, available after you log into the Web UI by appending /api/v2-docs/api/ after the web UI port number (8083). For example: https://my.okera.installation:8083/api/v2-docs/api/.

Notable Changes

  • Okera is deprecating the use of okctl and is replacing it with the use of Helm charts. The Okera documentation has references to okctl. The okctl utility is still available, but is not recommended for production environments.

  • The Okera documentation has removed all documentation for Amazon EC2 installs (Gravity installs). Such installations are no longer supported.

Security Vulnerabilities (CVEs) Addressed

Okera uses Snyk and GitHub Advanced Security for security vulnerability scanning.

Bug Fixes

The following bugs were fixed in this release:

  • The fix in Okera version 2.11.0 that returned string format instead of tabular format in the output of show create table has been reverted.
  • With this release, Okera uses the Snowflake original column types for policy function generation.

  • The Go Snowflake library used by Okera was upgraded from version v0.13.0 to version v0.14.0.

  • The JDBC driver used by Okera Snowflake processing was upgraded from version 3.12.17 to version 3.13.20.

  • Fixed an issue in which crawlers on JDBC connections continued to run after being deleted.
  • Upgraded Okera's version of Jackson to 2.13.3.
  • Tooltips on the System page now operate and appear properly.
  • BigQuery connection advanced properties now appear correctly on the BigQuery connection modal.
  • Upgraded Okera's base Alpine version to 3.15.6.
  • Upgraded Okera's version of OpenJDK to 8.345.01-r0.
  • Any tags or tag namespaces created using uppercase letters are now converted to lowercase letters automatically. Okera uses lowercase for tag and tag namespace names.
  • Upgraded the Apache Shiro version used with Okera.

  • Fixed an error that occurred when querying Athena tables using JDBC pushdown processing. The error received was:

     An error has been thrown from the AWS Athena client. 1 validation error detected: Value '' at 'queryExecutionContext.catalog' failed to satisfy constraint: Member must have length greater than or equal to 1 [Execution ID not available]
  • Fixed theresponse to large to return error for Okera queries on large BigQuery tables in Workspace.
  • Okera's base Ubuntu image has been upgraded to bionic-20220801.
  • Fixed a problem in which Snowflake policy sync failed while trying to resolve columns for a row filter.
  • Audit log entries are now added for all CREATE_AS_OWNER implied operations.
  • Upgraded the base Alpine image used by Okera to 3.15.5.