This document describes how Okera handles data types and values. We differentiate between the two in that data types are used when specifying schemas (for example, during a ‘create table’ call) and values are the data that exists in a given row within a table.
Currently Supported Data Types
See the NOTES section at the bottom of this page for more information on types.
Okera must convert both values as well as data types in some situations, based on the storage format and the compute engine being used. Some platforms do not not support the full range of types that ODAS does.
Parquet and Spark DataFrames
These are the conversions that occur when working with Parquet data or Spark DataFrames values.
|Datatype||Parquet type||Spark Data frame type||Avro type|
- The string and binary data types are stored as a binary blob and not interpreted in any way.
- REAL type is now supported in ODAS. Since, Hive does not support REAL data type, odb may be used to create a field with REAL datatype. DOUBLE type can be used as an alias for REAL.
- For complex datatypes, refer to complex types
- Decimal type is returned as a string in the json resultset when the client connects to odas rest server The rest server client may choose to convert it back to decimal type as needed. Note that most compute engines/applications connect to ODAS planner directly and support and retrieve decimal type directly.