This article describes the principal changes of Google BigQuery Templates.
Stambia DI is a flexible and agile solution. It can be quickly adapted to your needs.
If you have any question, any feature request or any issue, do not hesitate to contact us.
Google BigQuery technology updates
Create and drop table xpath expressions now generating Standard SQL queries.
Create table and drop table xpath expressions which can used on a Google BigQuery Metadata are now generating standard SQL queries instead of custom scripting.
This offers more comprehensive and visual representation of generated queries, and a better genericity.
Numeric datatype was missing from datatype definition and some other datatypes such as INTEGER and FLOAT were having issues with their creation masks which were incorrect.
Insert statistics when using direct mode
Statistics are now well computed as SQL_STAT_INSERT when using direct mode
This requires Stambia's Google BigQuery Connector 1.0.3 or higher
Error management improvements
Mechanism used to retrieve the errors of Google BigQuery jobs has been improved to return more accurate errors in some particular cases where the real error wasand a generic error returned instead.
Recycling of previous rejects fixed
When using the option to recycle the rejects of previous execution an extra step is executed to add those previous rejects in the integration flow.
Possible duplicates while retrieving those rejects are now filtered using DISTINCT keyword.
Two new Templates have been added:
- LOAD Rdbms to BigQuery
- INTEGRATION BigQuery
They offer the possibility to perform a load step before integrating data into BigQuery, and to perform direct BigQuery to BigQuery integrations.
INTEGRATION Rdbms to BigQuery Template is still included and preferred when wanting to integrate data directly into BigQuery without creating temporary tables.
The two new Templates simply allows to load data first into a temporary table if wanting to perform transformations, staging, ...
Other Templates have been updated to:
- Support using Standard SQL mode which is used by default from Google BigQuery Connector version 1.0.1
- Automatically add delimiters around the dataset and project name, which may contain special characters
- Automatically add delimiters around columns when their name correspond to a reserved keyword
This new Template version requires to use Stambia's Google BigQuery Connector 1.0.1 or higher