fixed

Mappings - Spark Templates were unexpectedly proposed in some situations in Mappings even when Spark was not involved

added

Addition of ability to specify deploy mode (cluster or client)

added

Addition of ability to work with resource files through a new dedicated node in Metadata and a new parameter on the Spark Submit TOOL

added

Addition of ability to handle spark session configuration when multiple targets are loaded with Spark

fixed

Template LOAD Hdfs File to Spark - The error message has been updated to be more clearer (instead of NullPointerException) when an error occurs while reading a file

added

Template - LOAD Hdfs File to Spark - new parameter "In File List"

added

Template - LOAD Hdfs File to Spark - support compressed files when using fileDriver Read Method

added

Template - Load Hdfs Json to Spark - new Template to load JSON files stored in HDFS into Spark

added

TOOL - Spark Execution Unit Launcher- add number of partitions to debug prints

fixed

Templates - use createOrReplaceTempView instead of registerTempTable which is deprecated

added

Spark - add support for HTTPS and Kerberized Livy connections

added

Spark - enhance datatype conversion when reading through JDBC

fixed

Spark - when a Spark session executed by Livy fails, the error cause is unexpectedly not returned

added

Templates updated - New Parameters 'Unlock Cdc Table' and 'Lock Cdc Table' to configure the behaviour of CDC tables locking

added

Templates updated - support having CDC sources on Templates which were not supporting it (such as staging templates)

fixed

Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates when querying the source data