Welcome Guest! Log in


This article describes the major changes of Spark Component.

If you need further information, please consult the full changelog.

Component download section can be found at this page.

Note:

Stambia DI is a flexible and agile solution. It can be quickly adapted to your needs.

If you have any question, any feature request or any issue, do not hesitate to contact us.

 

Component.Spark.2.0.2

Change Data Capture (CDC)

Multiple improvements have been performed to homogenize the usage of Change Data Capture (CDC) in the various Components.

Parameters have been homogenized, so that all Templates should now have the same CDC Parameters, with the same support of features.

Multiple fixes have also been performed to correct CDC issues. Refer to the changelog for the exact list of changes.

 

Complete changelog

The complete changelog with the list of improvements and fixed issues can be found at the following location.

 

Component.Spark.2.0.1

New Templates to load data from and into Elasticsearch

Two new dedicated Templates have been added to load data from Elasticsearch into Spark, and to load data from Spark into Elasticsearch.

 

New Templates to load data from and into Parquet HDFS files

Two new dedicated Templates have been added to load data from Parquet HDFS Files into Spark, and to load data from Spark into HDFS Parquet Files.

 

Improve datatype conversion

Datatype conversion between various systems when working with Spark has been improved to better handle the different datatypes.

 

Fix kerberos command under Windows environment

An issue about kerberos command launched under Windows environment has been fixed.

The "kinit" command launched for initializing kerberos security was not formed properly for Windows environments.

 

Fix issue with partition truncation when loading data to Hive using SCD mode

When loading data from Spark into Hive through SCD mode, there was an issue when the changes on the data lead to a partition truncation.

In this situation Template execution would fail when trying to get the partitions to truncate.

This issue has been fixed.

 

 

You have no rights to post comments

Articles

Suggest a new Article!