Welcome Guest! Log in
Stambia versions 2.x, 3.x, S17, S18, S19 and S20 are reaching End of Support January, 15th, 2024. Please consider upgrading to the supported Semarchy xDI versions. See Global Policy Support and the Semarchy Documentation.

The Stambia User Community is moving to Semarchy! All the applicable resources have already been moved or are currently being moved to their new location. Read more…


As explained in the presentation article, Stambia can work with Google Cloud Storage to perform operations such as creating Buckets, sending retrieving, copying, moving or deleting Blobs, defining metadata on Blobs, ...

This article demonstrates how to work with Google Cloud Storage in Stambia.

Prerequisites:

You must have previously installed Google Cloud Platform Component to be able to work with Google Cloud Storage.

Please refer to the following article that will guide you to accomplish this.

 

This article is dedicated to Stambia DI 2020 (S20.0.0) and higher. If you are on a prior version please refer to this article instead.

 

Metadata

When Stambia DI Google Cloud Platform Component is installed and configured, you can start creating your first Metadata.

This will then allow to define Google Cloud Storage information such as defining the Buckets, ...

 

Metadata creation

Create the Metadata, as usual, by selecting the technology in the Metadata Creation Wizard:

md creation

 

Click next, choose a name and click on finish.

 

Metadata Configuration

Google Cloud Credentials definition

 

When the Metadata is created, you can start defining common information about Google Cloud Storage access

The first step is to define the Google Cloud Credentials that must be used to perform operations on Google Cloud Storage.

This is mandatory as this will be used to connect to the correct Google Cloud Project and with given credentials.

 

The Storage Wizard will launch with the list of all existing credentials existing in the current workspace.

Credentials are defined in Google Cloud Project Metadata which must be created before, they contain login information about Google Cloud Platform Project to perform operations on.

Select one in the list and click on Finish.

 

Note that you can also close this wizard and open it back later if you have not created your Google Cloud Platform Metadata yet, but do not forget to define it as you'll not be able to perform operations without.

 

credentials

 

Buckets definition

 

Credentials being configured, you can now start defining Buckets nodes in your Metadata.

This is just a Metadata definition, the Buckets does not necessarily need to exist physically on Google Cloud Storage.

The connector allows to create or delete Buckets easily.

 

To add a new Bucket node, right click on the root node and choose "New > Bucket"

md new bucket

 

You can then define a logical name for this Bucket, and the real Bucket Name that will be used on Google Cloud Storage.

There is several additional storage information that can be defined in the standard and advanced tab, which allows to define more precisely the Bucket's properties, such as its location, its storage class, its retention period, ...

Note that those additional information will only be used when trying to create the corresponding Bucket with the tool allowing the create Bucket, and that when they are left empty, the default value from Google API is used.

 

md bucket definition 0

 

On the same idea, you can define Bucket labels and Bucket Life Cycle rules which should be used to configure the Bucket at Bucket creation.

 

md bucket definition 1 labels and life cycles

 

Finally, you can define 'Directories' which will be used to store and retrieve Blobs from Buckets.

This is optional and corresponds to the prefix that can be added on Blobs to organize them in virtual 'Directories'.

This will be used to filter the list of Blobs to retrieve and to create Blobs within a specific path.

 

md bucket definition 2 folders

Note that you can also perform get or put operations directly on Bucket's root, this is only for organizational purposes.

 

Examples

Below are some examples of operations that can be performed on Google Cloud Storage.

Performing operations is quite simple, from the Process Palette, create the tool corresponding to the operation you want to perform in your Process, drag and drop the corresponding Metadata on it, and execute the Process.

A bunch of options are also available to customize more precisely the operations.

 

Here is a little example to create a Process for creating a Bucket:

process example create bucket

 

Example of Process for creating a Bucket

example create bucket

 

Example of Process for sending a bunch of files into a Bucket

example send files to bucket

 

Example of Process for retrieving a bunch of files from a Bucket

example retrieve files

 

Example of Process for retrieving a filtered list of files from a Bucket, filtered using Blob Metadata Properties

example retrieve filtered list of files

 

Example of Process for copying Blobs from one folder to another folder

example copy blobs

 

Demonstration Project

Demonstration projects presenting common and advanced usage of the Google Cloud Platform and Google BigQuery in Stambia DI are available.

You can import them it in your workspace and then have a look at the Mappings and Processes examples.

It is a good start to familiarize with its usage and to see how the Metadata are configured, for instance.

 

From Stambia DI 2020 and higher, the sample projects are shipped withing the Components, and can be imported directly from the Designer as explained in Components getting started guide.

On Stambia DI S17, S18 and S19, you must download the sample projects separately from the download section.

 

 

Articles

Suggest a new Article!