Designer S20.0.x

You can find below the major changes and improvements of Stambia DI Designer 2020 (S20.0.x) version.

This article only highlights the most important changes.

If you need further information, please consult the full changelog and/or reference documentation.

 

Prerequisites

Java version

Java required version has changed in this major version.

Stambia DI 2020 now requires Java 8 or Java 11.

 

Migration

Stambia DI 2020 (S20) major version changes some important part of the software which require your attention.

There is a dedicated migration article that you must read if you want to upgrade from previous versions.

You can find this article here: migration guide

 

Analytics

If you are using Stambia DI Production Analytics, note that Stambia DI Production Analytics 3.1.1 or higher is required to work with Stambia DI Designer 2020 (S20.0.x) and Stambia DI Runtime 2020 (S20.0.x).

Designer 2020 (S20.0.x) generated packages can be imported only from this Analytics version or higher.

Runtime 2020 (S20.0.x) can be accessed only from this Analytics version or higher.

 

What's new on Stambia DI Designer 2020 (S20.0.3)

Refreshing Runtimes in background

When you refresh all the Runtimes from Runtimes root node in the Navigator, the Designer now offers to perform the operation in background.

This can be useful when there are a lot of Runtimes, which some may be unavailable, and therefore take some time to refresh everything.

You can now simply put this task in background so that you can continue working while the Runtimes refresh operation is finishing behind the scenes.

 

Bug fixes

This version fixes a list of issues which can be found on the full changelog.

Do not hesitate to take a look at it to have an idea of what have changed.

 

 

What's new on Stambia DI Designer 2020 (S20.0.2)

Fix ODBC execution issues

This version fixes an issue about ODBC execution.

When executing any Mapping / Process using ODBC, exceptions were thrown such as "java.lang.UnsupportedOperationException" or "java.sql.SQLException: No suitable driver".

Note that this was a regression introduced in previous Runtime version which was shipped within Designer S20.0.1.

 

This new Designer version ships an updated Runtime version which fixes the issue.

Complete changelog can be found at the following location.

 

What's new on Stambia DI Designer 2020 (S20.0.1)

Google Sheet Component updated

Google Sheet Component included inside Designer's installation has been updated.

We would like to take your attention about this as there are some structural changes in this Component update.

We worked hard recently to produce an updated Component for Google Sheet, which will replace the old one because Google is removing the "v3 API" we were using to communicate and operate with Google Sheet.

 

You can find more information about the removal of this Google API at the following Google article, which explains when it will be removed.

We therefore worked on a new Component version which will use the new currently supported "v4 API".

This new version will require the migration of your developments, as explained in the dedicated migration article.

 

How will you be impacted?

Google "v3 API" used by the old Component being removed gradually this year, the old driver will stop working.

You must migrate your developments to continue working, as explained in migration article.

 

Web services invocation and reverse: support NTLM authentication

Prerequisites:

• Java 11

 

Web services Metadata now supports defining a new "NTLM" authentication node for reversing and invoking Web Services secured with this type of security.

To use it, simply add the new node, define the necessary attributes on it, and choose this node as security when reversing on when configuring your Metadata.

 

Adding the new node

 

Defining the attributes

 

 

Example - Using the defined node in wsdl wizard

 

 

Example - defining manually the node on a web service

 

 

Runtime Metadata

When clicking on Runtime Editor while the default "localhost" Runtime is selected, it will now open a new Windows to create a Runtime Metadata.

This Runtime Metadata, which exists since quite a long time now, is the privileged method to define a list of Runtimes in the Designer.

Runtimes defined in Runtime Metadata will appear in the Runtime View, in the Navigator, and can be used also in Mappings and Processes to centralize all Runtime information when needed.

As a reminder, before this change, clicking on the Runtime Editor while the default "localhost" Runtime is selected was opening the "global/conf.egc" file on a default installation.

This file is historically the place where Runtimes were defined, but Runtime Metadata, added in S19.0.0, is now the preferred and advised method to define Runtimes.

You can still define Runtimes in the conf.egc file as before, for this simply open the file directly instead of clicking on Runtime Editor.

 

You can find below the new behavior of Runtime Editor button:

• When default "localhost" Runtime is selected in the Runtime list, clicking on Runtime Editor launches a wizard to help creating a new Runtime Metadata

• When a Runtime coming from a Runtime Metadata is selected in the Runtime list, clicking on Runtime Editor opens the related Runtime Metadata

• When a Runtime coming from conf.egc file is selected in the Runtime list, clicking on Runtime Editor will open the conf.egc file.

 

Generic Metadata

Generic Metadata has been slightly updated to be more convenient.

The "Type" which is specified on an element is now displayed as a prefix in the editor to visually see it in the list of elements:

 

SQL Explorer: ability to configure the number of decimals displayed

In SQL Explorer Preferences, you now have the ability to define the maximum number of decimals to be displayed when executing queries from SQL Explorer.

Note that this is a default value which will be applied on applicable JDBC Drivers, as some drivers already supply this information automatically with the request results.

This is a default value to be used when the JDBC Driver does not return any information about this with queries results.

 

SQL Explorer: ability to trim additional zeroes on displayed numbers

In SQL Explorer Preferences, you have now the ability to configure the behavior about additional zeroes in numbers.

When this new preference is enabled, all "0" after the last significant decimal will not be displayed when performing queries from SQL Explorer.

For instance, "1.23000" would be displayed as "1.23"

 

Minor improvements and bug fixes

This version contains some other minor improvements and fixes a list of issues which can be found on the full changelog.

Do not hesitate to take a look at it to have an idea of what have changed.

 

What's new on Stambia DI Designer 2020 (S20.0.0)

Packaging / Platform improvements

Eclipse Platform update

 

As you may know, Stambia DI Designer is based on the Eclipse platform for its graphical interface.

The Eclipse version from which Stambia DI Designer is based has been updated to a recent version to benefit of all the performances, improvements, and fixes which has been done since the previous one which we were using.

This offers the ability to work with Java versions higher than Java 8 such as Java 11 and therefore also changes the minimum version required to use Stambia DI Designer, which is now Java 8.

This also offers performances and usability improvements when using Stambia DI Designer on Mac OS which had some graphical glitches and issues on previous versions.

 

Components packaging

 

We revised the packaging of Components (previously called Connectors.)

They now include everything related to the corresponding technology.

Installing a Component will now install automatically everything required to work with this technology:

• Technology and Product files
• Templates
• Sample Projects
• Modules
• Documentation

 

You want to install Elasticsearch Component to use Elasticsearch in Stambia DI?

Through a few clicks, you will install everything required from one archive file, and you will be ready to go.

On previous versions you had to install all those items separately: they are now installed all at once through the new install procedure.

For instance, Templates are now installed automatically in Stambia DI Designer's internal resources, you don't need to import them anymore!

 

Installation of a component in Stambia DI 2020 (S20.0.0), from "Install New Software" menu

 

Then simply select Component's archive and install it

 

Everything is now installed, including all the previous indicated items, such as Templates and sample projects to start playing with examples right away!

You can find more information on the migration guide if you are coming from a previous version.

You can also find more information about this in the new Component installation procedure.

Do not hesitate to take a look at those articles, there are important differences about installation procedure which we performed to simplify it!

 

Additional notes about Components

The main idea behind this new feature is to facilitate installation and update as you will now install everything at once for a given Component.

One of the major points to have in mind is that you don't have to import anymore any Templates in your workspace.

They are included in Components, including standard and generic Templates.

You can still of course modify them if needed by importing them from Stambia DI Designer's internal resources as explained in getting started article.

 

Welcome page / samples projects

 

Stambia DI Designer's Welcome page has been revised!

The new welcome page now displays and gives access to:

• Overview: access to documentation
• Tutorials: access to tutorials
• Samples: access to available sample projects
• What's New: access to an overview of the new features

 

One of the most interesting point of this new welcome page is the access to sample projects.

This is a brand-new feature which allows to import sample projects containing examples about various subjects.

Those sample projects are provided both from default installation and Components installation.

You have a default list of sample projects which is filled automatically with additional ones when installing additional Components.

 

Example of sample projects list:

To open the Welcome Page again after it has been closed, go to Help > Welcome menu

 

Modules

 

We introduced a new feature and mechanism in this version: Modules.

What is a Module?

When you need to communicate with a technology such as SQL Databases, NoSQL Databases, Cloud Platforms, and more... you will now need to use what we call a Module.

What is a Module? A Module is a set of all what is necessary to communicate with a technology.

Which means in the Java world all the libraries files which are required for the communication with the given technology, such as JDBC drivers, API libraries, libraries dependencies, ...

You will manage all of those through Modules.

Modules are uniquely identified through their name and can be reused in multiple Metadata.

 

When should I use a Module?

Using a Module to communicate with a technology is mandatory for most of them, as most require some libraries for communication.

The Module to be used must be defined most of the time directly in your Metadata.

For instance, in your PostgreSQL database Metadata, you’ll need to select a Module for communicating with PostgreSQL.

The Designer will help you to select and create Modules when necessary through dedicated wizards and helpers.

 

Here is a quick overview of the Module Wizard

 

Additional notes about Modules

The main idea behind this new feature is to:

• Facilitate installation as you will now be guided to install additional libraries such as JDBC Drivers
• Allow to compartmentalize libraries to be able to have different version of libraries (such as different version of a JDBC Driver in the same installation)
• No more confusion about what to put in Stambia DI Runtime lib folder, now everything is managed through Modules.

 

Metadata improvements

Metadata creation filter

 

When you create a Metadata, there is a new option to only see available technology in current installation.

This new option is automatically enabled when creating a new Metadata, and you can uncheck it to see other technologies that are not currently installed.

As a reminder, this usually means that you have not installed the corresponding Component in you Stambia DI Designer.

Some Components are shipped on default installation and others must be installed afterwards.

 

 

Metadata editor filter

 

On Metadata editor, you can now filter what type of nodes should be displayed.

For this simply click on the filter icon which appears on the left of the text filter box.

 

Everything is displayed when opening a Metadata

 

When unchecking a node, it will be filtered from the editor

This helps to display only specific nodes when you are searching for a given type of node for instance.

 

Metadata search

 

New utility to search for Metadata in current Workspace.

You can search by Metadata category and by Metadata name.

You can, with this new search tool, find in a few clicks all Elasticsearch Metadata of your workspace, for instance!

 

Accessing the search tool

 

Searching by Metadata category

 

Searching by Metadata name

 

You can then double click on a found Metadata to open it, or click on the link to find it back in Project Explorer

Metadata Link evaluation

 

Metadata Link evaluation has been improved to support evaluating from different manner.

It is now possible to retrieve values from Metadata Link.

It is supported only on Metadata Link from a Generic Element Metadata attribute for now.

 

In this example, we create an attribute to define the path of an XML File

 

Configuration attributes overview

 

Metadata Configuration user interface has been improved to provide an overview of the value that will be overriden, and the ability to remove the overriden value in one click.

Now, when you are in Metadata editor and you are modifying a Configuration, the attributes are greyed out with an hyperlink to override them:

 

Mapping improvements

Mapping stages can be created from a menu

 

In a Mapping, when you drag and drop a set of columns from a datastore, you have now a menu which allows you use those selected columns to create a filter, or to create a stage.

This is a new alternative way to create stages on the fly.

You have the possibility to define your favorite staging area Metadata nodes in preferences, which will be displayed in this menu.

 

Creating a Mapping stage through context menu

 

If you have defined some preferred staging area nodes in your preferences, they will appear in the menu

 

Under Window / Preferences / Stambia / Mapping menu, you can define your preferred staging area nodes that will then appear in the context menu in Mapping.

 

Mapping update key provider

 

On a Mapping, you can now choose from where the updated key of a datastore should be computed.

On previous versions, the update key of a datastore was initialized when adding the datastore in Mapping and then handled directly in the Mapping by the user if needed to perform some changes.

The updated key was therefore a manual definition in Mapping.

 

Changes in this version

You can now choose which provider to use for the update key.

You have the choice between "Manual Definition" which is the default value, and which corresponds to the actual behavior of using what is defined in Mapping.

And you have now also the choice to specify that the update key should be provided from a defined Metadata Key.

 

 

When using a Metadata key provider, such as a primary key, the update key of the datastore will reflect the primary key of the Metadata.

If the given key is updated in Metadata, the Mapping will reflect the changes automatically.

When a datastore is using a specific update key provider, the key is colored in purple, as you can see in previous screenshot, and the user cannot change it manually.

 

Mapping Extra Nodes

 

FILE, JSON, and XML Metadata have the ability to define additional fields to retrieve for instance while reading files the file name, path, the node iteration, and more...

Those fields used to be defined directly in Metadata on previous versions.

When you needed those extra fields, it was required to add them manually in your Metadata before being able to use them in Mapping.

From now, you'll be able to add them dynamically in your Mapping without having to define them in Metadata.

 

Example on File

 

Example on XML

 

Depending on the node chosen, Stambia directly offers you the list of available fields applicable to this type of node.

Note that you can of course still define those nodes in your Metadata if you prefer, it can be more efficient and practical depending on if you are using them a lot for instance, in such case it can be more practical to define them directly in Metadata.

Limitation: with JSON and XML, the Extra Nodes added dynamically in Mapping can be used only on expressions executed on "Source" execution location. Extra fields are not yet supported on staging area execution location. If you want to use them on such expressions, you can define them in Metadata as before.

 

Mapping Diagrams included in model

 

Mappings were originally physically separated in two files.

The Mapping file with extension ".map" which contains all the Mapping rules and definition.

The Mapping diagram file extension '.map_diagram" which includes the layout of the Mapping, such as how the user has positioned the different boxes graphically and the user notes.

 

Changes in this version

From now everything is stored inside the Mapping file itself.

There is now only one file with extension ".map" which contains everything.

This change has been done to ease the management of versioning systems to have only one file to handle for a given Mapping, so that the Mapping and its layout are always synchronized and that there is only one file to commit and manage.

 

 

You have nothing particular to do about this point.

On existing Mappings coming from previous versions which still have a separated diagram, the diagram will be included inside the Mapping automatically the next time the Mapping is saved.

The folder "indy.diagram/map" will therefore gradually be emptied as you re-save your Mappings on the new version.

 

Process improvements

Process Palette Tools

 

All applicable Templates and Tools such as Replicators are now displayed directly in Process Palette, under the 'Tools' section.

When installing new Components in Stambia DI Designer, the corresponding tools, if any, will appear in this list automatically.

Moreover, when using them, you will now have Metadata Link Helpers to know when they need Metadata!

 

As an example, you can now use the Replicator in a few clicks

 

Process Diagrams included in model

 

Process were originally physically separated in two files.

The Process file with extension ".proc" which contains all the Process rules and operations.

The Process diagram file extension '.proc_diagram" which includes the layout of the Process, such as how the user has positioned the different boxes graphically and the user notes.

 

Changes in this version

From now everything is stored inside the Process file itself.

There is now only one file with extension ".proc" which contains everything.

This change has been done to ease the management of versioning systems to have only one file to handle for a given Process, so that the Process and its layout are always synchronized and that there is only one file to commit and manage.

You have nothing particular to do about this point.

On existing Processes coming from previous versions which still have a separated diagram, the diagram will be included inside the Process automatically the next time the Process is saved.

The folder "indy.diagram/proc" will therefore gradually be emptied as you re-save your Processes on the new version.

 

Miscellaneous improvements

Syntax coloring and helpers for Metadata Links and Eclipse String Substitution

 

When you are in Metadata, Mapping, or Process editor, Metadata Link and Eclipse String Substitution syntaxes are now colored:

 

You want to know from where this Metadata Link is coming in your Project Explorer?

You can now also CTRL + click on it to find it back.

 

Example in a Metadata

 

Example in Mapping

 

Example in Process

 

String substitution

 

Support Eclipse and local workspace environment variables in Metadata.

You can for instance dynamically use the path of the current Project, or the path of the current Stambia DI Designer installation folder.

This can be useful most notably to prepare autonomous Projects containing example files, for instance.

 

Example using the current project location

For further information and to learn how to define your own varables, refer to reference documentation

 

Designer menus revised

 

We revised some of the Designer main top menus to be clearer and more convenient.

The "Help > About Stambia" and "Help > About Stambia Designer" menus have been merged to have only one menu where to find version information.

 

 

License Manager is now in a new "Utilities" menu which will be a menu containing Stambia additional utilities.

 

 

Local Runtime Java Virtual Machine

 

When starting the local Designer's Runtime directly from the Designer, the started Runtime will now use the same Java Virtual Machine (JVM) as the Designer.

This allows to make sure the execution environment is the same when you are using Designer's Local Runtime.

Note that this applies only when starting the Runtime from the Designer from the Runtime View / Environment button.

If you start the Runtime aside of the Designer using the startup scripts, it will use the standard Runtime's startup behavior (Use of STAMBIA_JAVA_HOME environment variable if defined, etc...).

 

New Process Action "Execute FTP Command"

 

The existing "Command With FTP" Process action has been moved to deprecated category in Process Palette.

This Process Action has been replaced with a new "Execute FTP Command" Process Action which supports commands aliases.

• This new action supports both RAW FTP Commands and most commonly used aliases such as "ls", "dir", and more...
• This new action publishes new variables such as "STORED_FILES_NUMBER" to inform about the operations performed.

 

You can find the complete list of supported aliases and published variables in the reference documentation of this Process Action.

Refer to this article for further information.

 

RMIS Configuration

Designer does not anymore include the embedded certificate used for communicating with Runtimes through RMIS.

This certificate has been removed as from Stambia DI Runtime 2020 you can customize it to use your own.

Therefore, when you want to connect to Runtimes through RMIS, you may now have to configure your Designer to specify the public certificate to be used.

 

Refer to this article to learn how to configure the clients such as Designer, Analytics, or command line scripts to communicate with a Runtime which endpoints are secured with TLS.

It also proposes as a separate download the previously embedded public certificate which must be configured in the Designer if you want to connect from the Designer through RMIS on previous Runtimes which were using this certificate (Runtime S17.x.x).

 

Refer to this article to learn how to secure the various endpoints with TLS on Stambia DI Runtime 2020.

 

Minor improvements and bug fixes

This version contains some other minor improvements and fixes a list of issues which can be found on the full changelog.

Do not hesitate to take a look at it to have an idea of what have changed.