This article describes the principal changes of Google BigQuery Component.
If you need further information, please consult the full changelog.
Component download section can be found at this page.
Note:
Stambia DI is a flexible and agile solution. It can be quickly adapted to your needs.
If you have any question, any feature request or any issue, do not hesitate to contact us.
This article is dedicated to Stambia DI 2020 (S20) or higher.
If you are using Stambia DI S17, S18 and S19 please refer to this article for Google BigQuery Addon or to this article for Google BigQuery Templates.
Component.Google.BigQuery.2.2.8
Fixed issues
This version contains a list of fixed issues which can be found in the complete changelog.
Component.Google.BigQuery.2.2.7
Minor improvements and fixed issues
This version contains some minor improvements and fixed issues, which can be found in the complete changelog.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.2.6
Integration BigQuery
When loading data in append mode from BigQuery to BigQuery, if at the same time we activate deleteStrategy = key, the delete generated by the Template can have a complex subquery.
To improve performance, we always enable pre-integration systematically when deleteStrategy is set to key.
Minor improvements and fixed issues
This version contains some minor improvements and fixed issues, which can be found in the complete changelog.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.2.5
Minor improvements and fixed issues
This version contains some minor improvements and fixed issues, which can be found in the complete changelog.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.2.4
Minor improvements and fixed issues
This version contains some minor improvements and fixed issues, which can be found in the complete changelog.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.2.3
Minor improvements and fixed issues
This version contains some minor improvements and fixed issues, which can be found in the complete changelog.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.2.2
Using SQL syntax for Create table statement instead of scripting
Addition of possibility to use the Create Table standard sql syntax so we can replace the script that was used before in the tool.
This is the new sql code:
Create table %x{md:physicalPath($bigquery)}x%
(
%x{md:patternList($bigquery/ref:columns(),'[columnName] [targetCreationType] [null]',',\n\t')}x%
)
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.2.1
Minor improvements and fixed issues
This version contains some minor improvements and fixed issues, which can be found in the complete changelog.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.2.0
Data Types
Addition of two new data types, BIGNUMERIC and BIGDECIMAL.
These 2 types correspond to the BIG NUMERIC type in BigQuery, BIGDECIMAL being an alias of BIGNUMERIC.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.1.2
Minor improvements and fixed issues
This version contains some minor improvements and fixed issues, which can be found in the complete changelog.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.1.1
Fixed issues
This version contains some minor fixed issues, which can be found in the complete changelog.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.1.0
Support of EMF Compare
Several improvements have been made to support EMF Compare.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.0.9
Minor improvements and fixed issues
This version contains some minor improvements and fixed issues, which can be found in the complete changelog.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.0.8
Support using Metadata Variable
It is possible to use a Metadata Variable with Bigquery with this version.
Refer to this article for more information.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found on the full changelog.
Component.Google.BigQuery.2.0.7
Support defining and using partitions for query optimizations
When working with BigQuery tables having partitioned data, you can now define the partitions under the tables in Metadata:
After having defined partition information in Metadata, this can be used when reading data from this table to optimize the read queries.
When loading data from this BigQuery table to another BigQuery table, you can enable the "Enable Partition Pruning" parameter on the INTEGRATION BigQuery Template.
This will indicate to the Template to optimize the requests used to query the source table based on the partitions defined on it.
Note that optimizations are performed only when having on the Mapping a filter on the partitioned source table.
When there is no filter, the whole source table is read so there are no possible partition optimizations.
Component.Google.BigQuery.2.0.6
Support authentication through "ApplicationDefault" mode
Google Cloud Platform Component has evolved, allowing to authenticate through ApplicationDefault mode.
BigQuery Component has been updated to support this new mode.
Refer to Google Cloud Platform Component release notes for further information about how to use it.
Constant in GROUP BY clauses
Support using constant values in GROUP BY clauses by automatically using aliases.
Change Data Capture (CDC)
Multiple improvements have been performed to homogenize the usage of Change Data Capture (CDC) in the various Components.
Parameters have been homogenized, so that all Templates should now have the same CDC Parameters, with the same support of features.
Multiple fixes have also been performed to correct CDC issues. Refer to the changelog for the exact list of changes.
Complete changelog
The complete changelog with the list of improvements and fixed issues can be found at the following location.
Component.Google.BigQuery.2.0.5
JDBC Drivers classes and URLs
Prerequisites
- Stambia DI Designer 2020 (S20.2.0) or higher
Stambia DI Designer 20.2.0 introduces an improvement which allows Components and Modules definition to provide themselves the proper JDBC drivers classes and URLs, for the connection wizard to automatically adapt depending on the Module selected.
This Component has therefore been updated to take into account this improvement and provide the necessary JDBC drivers classes and URLs information.
Component.Google.BigQuery.2.0.4
Prerequisites
- Stambia DI Designer 2020 (S20.1.0) or higher
When you are working with Google Cloud BigQuery Metadata, you need to define the Google Cloud Platform Credentials to be able to perform operations.
A new step in the BigQuery wizard has been added to propose selecting the corresponding credentials, which will be displayed when creating a new BigQuery Metadata or when launching the wizard from an existing Metadata.
This wizard has been added to ease the configuration and usability of Google BigQuery Metadata.
Note that the credentials proposed in this wizard are retrieved from all Google Cloud Platform Metadata of current workspace, in which credentials are defined.
All existing credentials are displayed automatically.
Just select one and click on Next to continue to the next step.
Component.Google.BigQuery.2.0.3
Fixed issue with join
When perfoming joins between BigQuery tables or stages in a Mapping, generated join query was not correct in some situations, causing errors at execution.
Component.Google.BigQuery.2.0.2
Prerequisites:
- Google Cloud Platform Component 2.0.2 or higher
REJECT BigQuery
All constraints verifications are now performed in a single instruction, to limit the number of requests.
We modified this to avoid reaching the query limits of Google BigQuery APIs.
Load RDBMS to BigQuery
When the work schema is different from the target schema, the load jobs did not load the table in the correct location.
Load Google Cloud Storage Json to BigQuery
New option SuccessIfNoFile
Adding a new option : SuccessIfNoFile.
When set to 'true' the LOAD template will not fail if source file(s) is(are) do not exist and will create an empty table based on the expected structure.
This allows to continue processing even when there is no source data, for instance.
Fix work schema usage
When the work schema is different from the target schema, the load jobs did not load the table in the correct location.
Component.Google.BigQuery.2.0.1
Prerequisites:
- Google Cloud Platform Component 2.0.1 or higher
Credentials and Project ID
A new attribute to drag and drop a Google Cloud Platform Credentials Metadata node on the BigQuery Metadata has been added, to replace the usage of 'credentialsFile' and 'projectId' properties which are now deprecated.
Note that for backward compatibility, when properties are defined they have priority on the new credentials link.
To use the new mechanism, you must remove the properties and then define the link.
Refer to this article for more information.
Temporary Bucket
A new attribute to drag and drop a Google Cloud Storage Bucket Metadata node has been added, to replace the default storage bucket attribute which is now deprecated.
Note that for backward compatibility, when the old attribute is defined, it has priority on the new storage link.
Note that old attribute has been moved under deprecated tab.
Refer to this article for more information.
Fix issue with Bigint
This version fixes an issue which was causing an SQLFeatureNotSupportedException in mapping using BigInt as source.
Fix issue with credentials error message
This version fixes an issue which was causing unclear and misleading error message when the credentials were the defined or incorrectly.