Google BigQuery Addon

This article describes the principal changes of Google BigQuery Addon.

Note:

Stambia DI is a flexible and agile solution. It can be quickly adapted to your needs.

If you have any question, any feature request or any issue, do not hesitate to contact us.

This article is dedicated to Stambia DI S17, S18 and S19 addon.

If you are using Stambia DI 2020 (S20) or higher, please refer to this article.

addons.google.bigquery_1.1.0_20200309_102230

Prerequisites:

Google Cloud Platform Addon 1.1.0 or higher

Templates BigQuery 2020-03-09 or higher

Credentials and Project ID

A new attribute to drag and drop a Google Cloud Platform Credentials Metadata node on the BigQuery Metadata has been added, to replace the usage of 'credentialsFile' and 'projectId' properties which are now deprecated.

Note that for backward compatibility, when properties are defined they have priority on the new credentials link.

To use the new mechanism, you must remove the properties and then define the link.

Refer to this article for more information.

Temporary Bucket

A new attribute to drag and drop a Google Cloud Storage Bucket Metadata node has been added, to replace the default storage bucket attribute which is now deprecated.

Note that for backward compatibility, when the old attribute is defined, it has priority on the new storage link.

Note that old attribute has been moved under deprecated tab.

Refer to this article for more information.

Fix issue with Bigint

This version fixes an issue which was causing an SQLFeatureNotSupportedException in mapping using BigInt as source.

Fix issue with credentials error message

This version fixes an issue which was causing unclear and misleading error message when the credentials were the defined or incorrectly.

addons.google.bigquery_1.0.4.201911081030

Prerequisites:

Google Cloud Platform Addon 1.0.4 or higher

Update of Connector

Google BigQuery Connector has been updated to work with the new Google Cloud Platform Connector.

All Google APIs libraries have been updated to latest ones to benefit of the last improvements, fixes and optimizations.

They are now all shipped in the main Google Cloud Platform Connector which centralize all the dependencies.

Google BigQuery Connector now therefore requires the Google Cloud Platform Connector to be installed.

Fix encoding issue

Encoding issues have been fixed in this version.

All read and write operations are now properly performed using UTF-8 encoding.

addons.google.bigquery_1.0.3.201907191042

More Statistics are now retrieved when performing operations on Google BigQuery

Number of processed rows are now computed and retrieved when inserting data into BigQuery through the Load and Integration Templates when using the direct method.

Number of processed rows are now computed and retrieved on the step which is streaming data into Google Storage, and on the one which is extracting source data into a local file.

Performance improvement on source data export

Performances have been improved on steps extracting source data into a local file before sending it to Google Storage and finally load it into Google BigQuery.

Extraction of source data should now be significantly faster when working large sets of data.

Fixed issue with timestamp datatype

This version fixes an issue which was causing unexpected errors when loading value from or to double datatype.

Fixed issue with source smallint datatype

This version fixes an issue which was causing unexpected errors when loading value from smallint datatypes to Google BigQuery.

addons.google.bigquery_1.0.2.201906141610

Fixed issue with null values

This version fixes an issue which was causing unexpected errors when loading "null" values.

Fixed issue with double datatype