Welcome Guest! Log in
Stambia versions 2.x, 3.x, S17, S18, S19 and S20 are reaching End of Support January, 15th, 2024. Please consider upgrading to the supported Semarchy xDI versions. See Global Policy Support and the Semarchy Documentation.

The Stambia User Community is moving to Semarchy! All the applicable resources have already been moved or are currently being moved to their new location. Read more…


This article describes the major changes of Spark Templates.

Note:

Stambia DI is a flexible and agile solution. It can be quickly adapted to your needs.

If you have any question, any feature request or any issue, do not hesitate to contact us.

 

templates.spark.2020-06-10

New Templates to load data from and into Elasticsearch

Two new dedicated Templates have been added to load data from Elasticsearch into Spark, and to load data from Spark into Elasticsearch.

 

New Templates to load data from and into Parquet HDFS files

Two new dedicated Templates have been added to load data from Parquet HDFS Files into Spark, and to load data from Spark into HDFS Parquet Files.

 

Improve datatype conversion

Datatype conversion between various systems when working with Spark has been improved to better handle the different datatypes.

 

Fix kerberos command under Windows environment

An issue about kerberos command launched under Windows environment has been fixed.

The "kinit" command launched for initializing kerberos security was not formed properly for Windows environments.

 

Fix issue with partition truncation when loading data to Hive using SCD mode

When loading data from Spark into Hive through SCD mode, there was an issue when the changes on the data lead to a partition truncation.

In this situation Template execution would fail when trying to get the partitions to truncate.

This issue has been fixed.

 

 

Articles

Suggest a new Article!