When you need to communicate with a technology such as SQL Databases, NoSQL Databases, Cloud Platforms, and more... you will now need to use Modules.
A Module is a set of all what is necessary to communicate with a technology.
You'll learn the basics about Modules in this article.
Prerequisites:
- Stambia DI Designer 2020 (S20.0.0) or higher
- Stambia DI Runtime 2020 (S20.0.0) or higher
If you are coming from a Stambia DI version prior to 2020 (S20.0.0), we suggest to also follow the migration guide which explains what Modules change and how to migrate your previous developments to use Modules mechanism.
You can find the complete Modules reference documentation at the following location.
We advise to familiarize with reference documentation which is more complete if you need to understand more precisely Modules concepts and how the different Wizards are working. The current article is a getting started article which summarize what is a Module and gives some Examples of Modules creation and usage.
What is a module?
When you need to communicate with a technology such as SQL Databases, NoSQL Databases, Cloud Platforms, and more... you will need to use what we call a Module.
What is a Module? A Module is a set of all what is necessary to communicate with a technology.
Which means in the Java world all the libraries files which are required for the communication with the given technology, such as JDBC drivers, API libraries, libraries dependencies, ...
In Stambia you will manage all of those through Modules.
Modules are uniquely identified through their name and can be reused in multiple Metadata.
Additional notes:
Modules are an autonomous and compartmentalized set of libraries.
Modules are compartmentalized, they cannot see each other and do not collide.
You can have multiple modules with different set of same libraries of different versions in the same installation.
For instance, you can have one Module with version 1.0 of a give JDBC driver, and one Module with a version 1.1 of the same driver.
When should I use a Module?
Using a Module to communicate with a technology is mandatory for most of them, as most require some libraries for communication.
The Module to be used must be defined most of the time directly in your Metadata.
For instance, in your PostgreSQL database Metadata, you’ll need to select a Module for communicating with PostgreSQL.
The Designer will help you to select and create Modules when necessary:
| Action | Comment |
| New Metadata | When creating a new Metadata you will have a step where you will choose which Module should be used for the communication. |
| Existing Metadata | On existing Metadata you can edit the Module used through the «Module» attribute which is on the root node of your Metadata. |
| Scripting Process Actions | On a Scripting Process Action you can also define a Module if you want to use additional libraries in your scripts. |
| Modules Managers |
There are two Modules Managers which allow you to manage your Modules.
You can find those two managers from Designer's top 'Utilities' menu. |
Some Modules key points
- Modules allows you to manage Java libraries which will be used by the Components to communicate with a given technology.
- Modules allows you to organize the physical architecture of your Java libraries. They are stored in a single directory and each module has its own sub-directory.
- Modules allows easier installation of the new Components. When possible, Stambia provides all third party libraries which are needed for a Module.
- The graphical wizards of Stambia DI Designer will guide you to manage your Modules.
How to use a Module?
Module Wizard will guide you to manage, create, and use Modules.
When possible, we provided all the necessary libraries for a given Module to communicate with a given technology.
When this is the case, the libraries are shipped within the Modules itself or through the selection of a third party pack.
When we cannot redistribute the libraries, for licenses purposes for instance, or when you want to use your own ones, you can also add them in Modules as explained further.
You can find below how to use the Module Wizard, what are the different Module statuses, and more.
The Module Wizard is composed of three panes:
- Select a category on left pane
- Select an existing Module or create a new one on the middle pane
- Configure, if necessary, the selected Module on the right pane
- Then click on Finish
Here is a quick overview of the Module Wizard:
The wizard is fully explained in reference documentation, do not hesitate to have a look at it
Example - Module creation and selection
Module selection and creation when creating a Metadata
We'll try to create an Elasticsearch Metadata in this example.
As usual, click on the create Metadata button and choose Elasticsearch in the list.
Click on Next to choose a name for the Metadata, and then on Next again to reach Module selection.
The Module selection part is filtered to only show the applicable category of Modules.
Choose the category you want to use and then select an existing Module or create a new one if none exist yet.
In the middle pane you select or create a new Module.
In this situation, no Module exists yet for Elasticsearch 7.x so we clicked on the "new" entry to create a new one.
The right pane allows to configure the Module if needed.
The colored status, which is yellow here, gives some tips about the Module status.
Here it indicates that the Module is not fully complete yet, it needs some additional libraries to work.
It is mandatory to configure these additional libraries to be able to create the module.
When possible, Stambia provides Third Party Packs which corresponds to the Third Party libraries required to communicate with a given technology.
A Third Party Pack has been selected here, which is considered as complete so the status of the Module is now changed to green, everything is ready.
The different statuses are fully explained in reference documentation, do not hesitate to have a look at it
Elasticsearch Metadata is now created, and it will use the Module we just created.
You can update the Module used by the Metadata here.
This Module now being created, you can also reuse it in your other Elasticsearch Metadata.
The idea of Modules is to ease the installation and configuration of libraries, once you have created your Module, you can reuse it in all applicable Metadata.
Important: Following the creation, modification, deletion of a module, you must restart the Runtime.
Example - using Modules in Scripting Process Action
When you need to use additional libraries in your scripts, for instance when you need to use specific classes which are not shipped by default, you can specify a Module containing the related libraries on the Scripting Process Action.
Note that in Module Manager you can create Modules of "Generic" category which are not linked to any particular technology, they are particularly useful when you want to use specific libraries in scripting.
To use a Module on Scripting Process Action, simply specify its name in the corresponding parameter, you can use any available Module.
Modules can be used natively with the following scripting languages: Beanshell, Rhino, Nashorn, Javascript.
Modules are not fully supported yet on following scripting languages: Jython, Groovy.
Refer to the following article for further details about Module usage in scripting.
Module Managers
Module Manager
The Module wizard presented in this article is proposed at various locations when you want to create or select a Module during Metadata creation and edition.
Additionally, you can also manage your Modules from the main Module Manager.
This allows to create, delete, or edit Modules.
You can access it from the following menu:
Utilities > Module Manager
From this manager you can:
- Create a new Module by clicking on the plus button
- Delete the selected Module by clicking on the delete button
- Edit an existing Module by double clicking on its name
The Module Manager looks like this:
Important: Following the creation, modification, deletion of a module, you must restart the Runtime.
Metadata Module Manager
You want to control the Modules used in all your Metadata?
You want to assign a given Module in mass on all corresponding Metadata?
You can use for this the Metadata Module Manager.
You can access it from the following menu:
Utilities > Metadata Module Manager
Start the wizard, then select a category of Module on the left.
You will then have the list of Metadata you have in your workspace which are applicable for the given category.
You can see for each Metadata the Module it is using.
If you want to assign a given Module, select the Module, check all the Metadata on which you want to apply this Module, and click on Perform Update.
Note that when a Module already exists for a given category, it will be preselected in "Select a Module to apply" box.
You can of course select another one using the select button at the right of the box.
Note that the categories on the left are filtered based on the Metadata you have in your workspace, it will only show categories for which you have Metadata in your Workspace. For instance, if you do not have PostgreSQL Metadata files on your workspace, you will not have PostgreSQL category on the left. This wizard is used only to assign Modules in mass in your existing Metadata. If you want to prepare Modules for technologies you haven’t created Metadata yet, prefer using the main Module Manager presented above.
Additional notes about Modules
Runtime log database and Modules
In the file stambiaRuntime \ Properties \ engineParameters.xml, a log database is defined for the logs.
Runtime's default behavior is to store its logs inside an internal H2 database. But other databases are possible.
Example of file (default configuration)
When you are using another log database than default H2 database, you have to define the corresponding Module.
Example for a MySQL Log Database, using a previously created Module named "MySQL":
Runtime Scheduler database and Modules
As for Runtime log database (see previous section), if you are using the Runtime's internal scheduler, and you configured it to use another database than default h2 database, you must define the corresponding module.
As a reminder, Runtime's internal scheduler is configured through engineScheduler.properties.
Example for a MySQL Scheduler Database, using a previously created Module named "MySQL":
Externalizing Modules folder location
On a default installation, Modules are installed and managed inside the directory stambiaRuntime/modules.
It is possible to configure another path to share it on several Designers and Runtimes.
Stambia DI Designer configuration
In Designer's installation folder, modify the "stambia.ini" file related to the launcher you are using.
If you are using stambia.exe, use stambia.ini, if you are using stambia32.exe, use stambia32.ini, and more...
Then update the property according to the folder in which you want to manage the Modules.
Example of file (default configuration)
Default value is relative to Designer's installation folder.
You can also use an absolute path.
Stambia DI Runtime configuration
In Runtime's installation folder the file initvariables.bat (windows) or initvariables.sh (Linux) contains a property to define Modules folder for the Runtime.
Simply update this file to define the folder you want to use.
Default value is relative to Runtime's installation folder.
You can also use an absolute path.
If the Runtime you are configuring is the local Runtime shipped within a Designer installation, do not forget to modify the following file to avoid the Designer to rewrite the file at startup, and therefore lose your modification
<Stambia Designer Installation Folder>/stambiaRuntime/properties/protection.rules
Example of modified protection.rules
Troubleshooting
When you try to create a Module, you don't see the corresponding category on the left
When you try to select or create a Module, you don't see the category corresponding to the technology you want to create a Module for.
For instance, when you try to create a Module for Elasticsearch, you don't see Elasticsearch in Modules categories.
If you are in this case, please double check the following items:
- Make sure the corresponding Component is installed in Stambia DI Designer. For instance, if you want to use Elasticsearch, make sure Elasticsearch Component is installed.
- Make sure you are trying to select / create a Module on the proper Metadata. The Module Wizard is filtered based on the technology from which you want to select / create a Module for. For instance, if you are on an Elasticsearch Metadata, you will not see the Modules for PostgreSQL database, this is normal. As a consequence, make sure you are on the correct Metadata.
- If you have already checked the two above points and you still have issues, do not hesitate to contact us so that we can help, and fix issues if needed.
Your Module seems OK in Stambia DI Designer but when you try to execute a Mapping / Process using it the Runtime returns that the Module is not found / does not exist
You configured your Modules in Stambia DI Designer, everything looks fine, but when you try to execute you have an error returned by the Runtime which says that the Module is not found / does not exist.
If you are in this case, please double check the following items:
- Stambia DI Runtime is taking into account its Modules at startup, so if you are working with a Module which has been created after the Runtime was started, you have to restart your Runtime.
- Make sure the Runtime on which you are executing your Mapping / Process contains the Module you are trying to use. On a default installation, Modules are located under stambiaRuntime/modules. Therefore, double check that the Runtime you are executing your Mappings / Processes on have the corresponding Modules you want to use. Most notably if you are using another Runtime than the local shipped within the Designer, or if you have changed the default location of Modules.
When executing a Mapping or a Process, you have "java.lang.ClassNotFoundException" or "java.lang.Exception: Unable to find class" exceptions
When you are executing a Mapping or a Process, you have exceptions similar to:
-
java.lang.ClassNotFoundException
-
java.lang.Exception: Unable to find class
When you have such exceptions, this means that when trying to execute your Mapping or Process, the Runtime needed to communicate with a given technology or to perform any operation, but the corresponding Java classes for this could not be found in the actual installation.
If you are in this case, please double check the following items:
- Have a look at the corresponding Metadata and check if a Module if specified in this Metadata
- If a Module is already specified, make sure that this Module contains all the additional libraries required (such as JDBC Driver, ...)
-
If no Module is specified, you have two different solutions
- Define a Module containing everything required in this Metadata
- If you prefer not defining Modules and using the fallback "default" Module instead, make sure that the "default" Module contains all the additional libraries required (such as JDBC Driver, ...)