Auditing¶

A vital part of data governance is the ability to quickly and coherently respond to security threats. This is also critical for regulatory purposes and compliancy requests. Enterprises are often required to use company-wide security information and event management (SIEM) software that allows the correlation of audit log events generated by IT systems. For any data provided by the Okera platform, there is a single audit log location with text files that contain JSON-formatted records for each monitored action. This includes all access to data and metadata, such as the Schema Registry to retrieve dataset information.

Every access to the Okera platform services is authenticated by the respective endpoint, and the access operation is written to the audit logs in great detail, containing not only the user making the request, but also the API that is used, if the request was successful, and so on. All requests eventually flow through the Okera Policy Engine (formerly called the Planner) and the Okera Enforcement Fleet workers, irrespective of the API used. The Okera Policy Engine is used, among other things, to retrieve dataset definitions and plan the distribution of the query across all available cluster nodes. These requests are authenticated and authorized by the Policy Engine, which then appends an audit event record to its local log. The workers are responsible to serve the dataset rows as needed by the clients, applying any filtering or masking on the fly. Before any data is returned, the client request is again authenticated, creating an audit event entry to be appended to the worker's local log. After serving data and concluding the client request, the workers also log the amount of data that was transmitted to and from the client. You can use this information to determine the system use for individuals and groups, and develop, for instance, charge-back models.

Audit Logs¶

The Okera Policy Engine, workers, gateway servers, and other components are configured to upload their local logs to a central storage location, usually the configured installation directory. For AWS, this is a directory in an Amazon S3 bucket to which the servers have write access. The location is configured using the OKERA_STAGING_DIR configuration parameter set in the Okera configuration file. If you want to change this configuration parameter, update the configuration file in the configuration YAML file and use the Okera Helm chart to update the cluster.

Note: Changing this setting only applies to the clusters that are started thereafter. For an existing cluster, a restart is required to set the new log location.

For example, here is what the content of a staging directory might look like:

$ aws s3 ls s3://acme-ops/prod-7/
        PRE audit/
        PRE canary/
        PRE cdas_rest_server/
        PRE cerebro_catalog/
        PRE cerebro_planner/
        PRE cerebro_web/
        PRE cerebro_worker/
        PRE clusters/
...

The first directory shown, named audit, is the one containing the uploaded audit logs from all cluster nodes. For performance reasons the data is partitioned by day inside the audit directory:

$ aws s3 ls s3://acme-ops/prod-7/audit/
        PRE ymd=2018-09-05/
        PRE ymd=2018-09-06/
        PRE ymd=2018-09-07/
        PRE ymd=2018-09-08/
        PRE ymd=2018-09-09/
        PRE ymd=2018-09-10/

Looking into a particular partition directory reveals the already uploaded logs, here one from a Policy Engine (planner) and another from a worker instance:

$ aws s3 ls s3://acme-ops/prod-7/audit/ymd=2018-09-05/
2018-09-06 08:18:06     814762 \
  planner-10.1.10.181-0-731c5cf0-4917-4947-b199-7fb257efcfba_audit.log
2018-09-10 15:25:00     143733 \
  worker-10.1.10.181-0-3e80936e-30d0-4080-a519-f1676d6544bf_audit.log

The file name is comprised of the service name, the internal IP address of the node hosting it, a sequence number, and a UUID, making the log unique for a specific instance of a service.

Note: There is usually no need to know about these files and where they come from. Using the provided audit table and views (see below) will handle the files implicitly when executing queries on them.

You can use the Insights Page in the Okera UI to easily explore and analyze these audit logs. Here is an example of the raw JSON record that is appended to the audit log for every client access:

{ "request_time":"2018-09-05 16:19:01.628368000",
  "request_id":"97451917614274a8:bd6c2b3347c651a6",
  "start_unix_time":1536164341628,
  "end_unix_time":1536164342121,
  "auth_failure":false,
  "status":"ok",
  "user":"root",
  "connected_user":"root",
  "client_network_address":"::1:52774",
  "client_application":"okera-execute-ddl",
  "num_results_returned":0,
  "statement_type":"DDL",
  "default_db":"default",
  "statement":"CREATE ROLE IF NOT EXISTS okera_public_role",
  "ae_database":"","ae_function":"",
  "ae_role":"okera_public_role",
  "ae_table":"","ae_view":""}

Other Logs¶

The Okera Policy Engine and workers generate additional log files that contain all of the log events these processes generate, including the audit logs events, and can be used by administrators for debugging purposes. For example:

$ aws s3 ls s3://acme-ops/prod-7/
2018-09-05 18:19:05     108632 2018-09-05-16-19-04-planner- \
  10.1.10.181-0-07a6e2e5-2014-4f1d-8abc-14a04618b341.log
2018-09-05 18:19:21      48361 2018-09-05-16-19-20-planner- \
  10.1.10.181-1-7df31dc6-43bd-4265-aa54-f11f5d7b22a8.log
2018-09-05 18:19:37        591 2018-09-05-16-19-36-planner- \
  10.1.10.181-2-62e16fed-b2ba-4ce6-936a-c0911c37636a.log                                                          130 ↵
2018-09-05 18:20:21        405 2018-09-05-16-20-19-web- \
  10.1.10.181-0-57950e12-360e-4629-bc89-763b567ad123.log
2018-09-05 18:27:33         76 2018-09-05-16-27-32-web- \
  10.1.10.181-1-acd846c6-6b32-4af7-99d0-880b9b0e3a46.log
2018-09-05 18:52:39         76 2018-09-05-16-52-38-web- \
  10.1.10.181-2-d41ad909-e46f-4b4d-b4a3-95a3ede57aa5.log
2018-09-05 18:20:05      13763 2018-09-05-16-20-04-cdas-rest-server- \
  10.1.10.181-0-ec031b3f-e971-4c38-adb4-01a6d9a28c42.log
2018-09-05 18:20:21        532 2018-09-05-16-20-20-cdas-rest-server- \
  10.1.10.181-1-1102aab4-8571-4d53-bffe-f51fe3084765.log
2018-09-05 18:20:37        364 2018-09-05-16-20-36-cdas-rest-server- \
  10.1.10.181-2-f512e3e5-80b9-42b5-97d0-26e9d8c8259f.log

The audit log records are prefixed in the mixed service logs with Audit.log, on a single line. For example, a service log with technical and audit log entries might look like this:

stderr: I0905 16:19:02.174656   228 impala-server.cc:1162] \
  RegisterQuery(): query_id=bb4986e8cab0d521:823bc5dbc662b3b4
stderr: I0905 16:19:02.266245   228 impala-server.cc:1212] \
  UnregisterQuery(): query_id=bb4986e8cab0d521:823bc5dbc662b3b4
stderr: I0905 16:19:02.267323   228 impala-server.cc:679] Audit.log: \
  {"request_time":"2018-09-05 16:19:02.173889000","request_id": \
  "ec4264da5511c330:a38cf5108c8b9b85","start_unix_time":1536164342173, \
  "end_unix_time":1536164342267,"auth_failure":false,"status":"ok", \
  "user":"root","connected_user":"root","client_network_address": \
  "::1:52776","client_application":"okera-execute-ddl", \
  "num_results_returned":0,"statement_type":"DDL","default_db":"default", \
  "statement":"GRANT ROLE okera_public_role TO GROUP __okera_public_group", \
  "ae_database":"","ae_function":"","ae_role":"","ae_table":"","ae_view":""}
stdout: Connecting to planner localhost:12050

Audit Log Format¶

Each audit log is a JSON record with the schema shown below. By default, the full audit log data is made available within the special okera_system database, as a table named audit_logs.

Note: The fields in the audit log schema are stable and will not be removed in future Okera releases. More fields may be added in the future.

Field	Type	Description
ymd	STRING	The table partitioning string, derived from the request_time in the form of `YYYY-MM-DD`.
request_time	STRING	Human-formatted GMT time when the request started.
request_id	STRING	Unique ID for this request.
start_unix_time	BIGINT	Unix epoch time at which the request completed (or failed).
end_unix_time	BIGINT	Unix epoch time at which the audit entry was generated.
auth_failure	BOOLEAN	Set to true if the request failed due to authorization or authentication failure.
status	STRING	Status message of the request. “ok” if the request succeeded; error details otherwise.
user	STRING	The effective user running the request. This is the user whose access permissions are used to validate the request.
connected_user	STRING	The user that is authenticated at the network connection level. This can be different from 'user' if the 'connected_user' is running on behalf of another user.
client_application	STRING	The client application, including client version, that originated the request. Examples: execute-ddl, presto (1.0.0)
client_network_address	STRING	The client host and port.
statement_type	STRING	The type of the request.
statement	STRING	Log of the request. Depending on the statement type, this will vary in format but captures the request being made.
num_results_returned	BIGINT	The number of records returned by the query.
ae_database	STRING	A list of the databases accessed during the processing of the query.
ae_table	STRING	A list of the tables accessed during the processing of the query.
ae_view	STRING	A list of the views accessed during the processing of the query.
ae_function	STRING	A list of the functions executed during the processing of the query.
ae_role	STRING	A list of the roles modified during the processing of the query.

Note: Only stable fields are documented here. Others found in the logs are not considered stable and are subject to deprecation or unannounced removal.

Enable Spark Query Logging for Databricks¶

Spark query logging for Databricks is enabled by default in Okera versions 2.8 and later. This section describes how to enable it if it is disabled in your environment.

To enable Spark query logging, complete the following steps:

In Databricks, verify that the following Databricks environment variable (in the Environment Variables section under Clusters -> Advanced Options -> Spark) is set to true:
```
AUTO_POPULATE_REQUEST_NAME=true
```
Setting this to false disables adding client_request_id to the audit logs. By default, it is set to true.
In the Okera configuration file, verify that the --audit_request_query option for RS_ARGS is set to true. See RS_ARGS Options.
```
RS_ARGS: --audit_request_query=true
```
By default, this option is set to false. When set to false, the client_request_id is not added to the audit logs.

Audit Log Views¶

For convenience, the okera_system database provides special views of the raw audit log table. These views provide implicit filtering based on the partitioning information.

Daily Audit Log View¶

The first view, named daily_audit_logs, restricts the data to the data of the last partition. Since the audit table is partitioned by day, this means the view returns the data of the current day.

Note: The audit log views use the Okera-specific LAST PARTITION feature.

For instance, the daily audit log view is defined in this manner:

CREATE VIEW IF NOT EXISTS okera_system.daily_audit_logs
COMMENT 'Audited logs from the current day.'
AS SELECT * from okera_system.steward_audit_logs(LAST PARTITION);

Weekly Audit Log View¶

The weekly_audit_logs view is similar to the daily log view, but sets the number of partitions to return to seven, that is, one week. Using this view gives access to the last seven days, including the current day (which most likely is still accumulating data).

Monthly Audit Log View¶

The monthly_audit_logs view is similar to the daily log view, but sets the number of partitions to return to thirty, that is, one month. Using this view gives access to the last thirty days, including the current day (which most likely is still accumulating data).

Steward Audit Log View¶

A more tailored audit log view is available as steward_audit_logs. These return only audit log rows that refer to the objects (database, table, or view name) to which the current user has administrative access. For example, assume there is a dataset named salesdb.transactions and a user bob who is part of the sales_steward role. A catalog or database administrator must grant the ALL privilege to the sales_steward role on the salesdb.transactions object so that user bob can see the audit log rows that pertain to that dataset.

Reporting Audit Log View¶

The Okera UI and its Insights Page use the reporting_audit_logs view. This view builds on the steward audit log view to provide access to the most important columns of the audit logs, while applying the same filtering. In other words, the user only sees audit log records for objects to which they have full access. The visual also filters out unnecessary audit records that would otherwise cause the analytics to display inaccurate aggregates. For example, the view filters out Okera service internal communication, as well as one record for events that have both a start and end entry in the logs.