Administration

Stambia CI/CD with Jenkins and Docker

Following the devOps principles, more and more organizations are building CI/CD chains to shorten the systems development life cycle and provide continuous delivery with high software quality.

The delivery of a new version of development made in Stambia can be seen as a four-steps process :

We will see in this article how Stambia can be integrated in a CI/CD chain using Jenkins, by deploying a sample application.

In this example, we will automate the build from sources to deployment on existing runtimes, using Jenkins as automation server and Docker for the deployment.

Prerequisites

GitHub account : You need a GitHub account to fork/clone the sample repository

Jenkins server : To automate the tasks

Nexus repository : To store the Stambia Designer and Stambia Runtime for building

Docker registry : To store the built images

Sample application overview

The sample application launchDimCustomer is made of a simple process calling loadDimCustomer mapping, which loads data from customer table of Hotel database to Datamart database.

A lauchDimCustomer script is included in the project, to build the launchDimCustomer.pck package file.

Some configuration files have also been prepared for the DEV, TST, and PRD environements

The files needed to build the runtime container have also been added to the project : Dockerfile, engineParameters.xml etc.

You can clone this project ‘dimCustomer’ from github repository : https://github.com/stambia/samples

Environment setup

We will first upload the stambia designer and stambia runtime zip files – the ones you can download from Stambia.org- to a nexus raw repository :

The resulting download URL will be http://nexus:8081/service/rest/v1/search/assets/download?name=designer/stambiaDesigner_S20.1.0.zip

Build the pipeline

Now that everything is in place, we can create the Jenkins pipeline to automate the deployment tasks.

Create a pipeline project ‘dimCustomer’ :

Once created, we can create the pipeline script skeleton and set the environment variables :

Add the variables :

pipeline {
    agent any

    environment {
        NEXUS_USER = 'nexus_user'
        NEXUS_PASSWORD = 'nexus_password'
        STB_PROJECT = 'dimCustomer'
	PKG_NAME = 'dimcustomer'
	PKG_FILE = 'launchDimCustomer.pck'
    }    

    stages {
        stage('Clean') {
            steps {
                cleanWs()
            }
        }
    }
}

We can now add the necessary steps to build our pipeline, from the sources retrieval to the deployment on the three target environments

Checkout stage

Get the sources from the github repository :

git https://github.com/stambia/samples

Download the designer and runtime zip files :

sh 'curl -X GET -L --user "${NEXUS_USER}:${NEXUS_PASSWORD}" http://nexus:8081/service/rest/v1/search/assets/download?name=designer/stambiaDesigner_S20.1.0.zip -o designer.zip'
sh 'curl -X GET -L --user "${NEXUS_USER}:${NEXUS_PASSWORD}" http://nexus:8081/service/rest/v1/search/assets/download?name=runtime/stambiaRuntime_S20.1.0.zip -o runtime.zip'

Build package stage

Unzip the designer :

sh 'unzip ${WORKSPACE}/designer.zip -d ${WORKSPACE}'

Build package using script file :

sh 'java -Djava.util.Arrays.useLegacyMergeSort=true -jar ${WORKSPACE}/stambia/plugins/org.eclipse.equinox.launcher_1.5.100.v20180827-1352.jar -application com.indy.shell.application -data "." -script "${WORKSPACE}/${STB_PROJECT}/Build/Scripts/launchDimCustomer" -console -noSplash -importFrom "${WORKSPACE}/${STB_PROJECT}" -cleanProjects -rebuildCache'

Deploy to development stage

Prepare folder and files to build the container

sh 'mv *.pck ${WORKSPACE}/stambia/stambiaRuntime/build/packages'
sh 'mkdir ${WORKSPACE}/deliveries'
sh 'mkdir ${WORKSPACE}/buildRuntime'
sh 'mkdir ${WORKSPACE}/buildRuntime/modules'
sh 'mkdir ${WORKSPACE}/buildRuntime/deliveries'

Build the delivery file using Developement configuration file

dir("${WORKSPACE}/${STB_PROJECT}/stambia/stambiaRuntime/") {
    sh './buildDelivery.sh launchDimCustomer -conffile ${WORKSPACE}/${STB_PROJECT}/Build/Configurations/launchDimCustomer.DEV.conf -deliveryfolder ${WORKSPACE}/deliveries/'
}

Build container and push to the container registry

sh 'mv ${WORKSPACE}/runtime.zip ${WORKSPACE}/buildRuntime/stambiaRuntime.zip'
sh 'mv ${WORKSPACE}/${STB_PROJECT}/Run/engineParameters.xml ${WORKSPACE}/buildRuntime'
sh 'mv ${WORKSPACE}/${STB_PROJECT}/Run/wait-for ${WORKSPACE}/buildRuntime'
sh 'mv ${WORKSPACE}/${STB_PROJECT}/Build/Modules/* ${WORKSPACE}/buildRuntime/modules/'
sh 'mv ${WORKSPACE}/${STB_PROJECT}/Build/Docker/Dockerfile-runtime ${WORKSPACE}/buildRuntime/Dockerfile'
sh 'mv ${WORKSPACE}/deliveries/* ${WORKSPACE}/buildRuntime/deliveries'
sh 'docker build ${WORKSPACE}/buildRuntime -t stambiadev/runtime:DEV'
sh 'docker push stambiadev/runtime:DEV'

Test stage

We will run the delivery file from the newly built container to test our developement

sh 'docker run --rm --env-file ./demoProject/Run/runtime.env stambiadev/runtime:DEV /opt/stambia/stambiaRuntime/startdelivery.sh -name launchDimCustomer'

Deploy to test and production stages

Then we can deploy to test and production environments using the same steps as for development, changing the configuration file for each environment :

dir("${WORKSPACE}/${STB_PROJECT}/stambia/stambiaRuntime/") {
    sh './buildDelivery.sh launchDimCustomer -conffile ${WORKSPACE}/${STB_PROJECT}/Build/Configurations/launchDimCustomer.TST.conf -deliveryfolder ${WORKSPACE}/deliveries/'
}
sh 'mv ${WORKSPACE}/deliveries/* ${WORKSPACE}/buildRuntime/deliveries'
sh 'docker build ${WORKSPACE}/buildRuntime -t stambiadev/runtime:TST'
sh 'docker push stambiadev/runtime:TST'

dir("${WORKSPACE}/${STB_PROJECT}/stambia/stambiaRuntime/") {
    sh './buildDelivery.sh launchDimCustomer -conffile ${WORKSPACE}/${STB_PROJECT}/Build/Configurations/launchDimCustomer.PRD.conf -deliveryfolder ${WORKSPACE}/deliveries/'
}
sh 'mv ${WORKSPACE}/deliveries/* ${WORKSPACE}/buildRuntime/deliveries'
sh 'docker build ${WORKSPACE}/buildRuntime -t stambiadev/runtime:PRD'
sh 'docker push stambiadev/runtime:PRD'

Test the pipeline

That’s it, we can now test the entire chain in Jenkins

We can also check the execution logs of the test step using for example Production Analytics

The launchDimCustomer app is now available in Production ! In this example, the Docker containers have been pushed to a Docker registry, and can be started on demand, or deployed and started as services

You can find the full pipeline script here : document Pipeline script (3 KB)

Stambia CI/CD with Jenkins

Following the devOps principles, more and more organizations are building CI/CD chains to shorten the systems development life cycle and provide continuous delivery with high software quality.

The delivery of a new version of development made in Stambia can be seen as a four-steps process :

We will see in this article how Stambia can be integrated in a CI/CD chain using Jenkins, by deploying a sample application.

In this example, we will automate the build from sources to deployment on existing runtimes, using Jenkins as automation server and Stambia Production Analytics for the configuration management and deployment

Prerequisites

GitHub account : You need a GitHub account to fork/clone the sample repository

Jenkins server : To automate the tasks

Nexus repository : To store the Stambia Designer for building

Stambia environment : Stambia Production Analytics and 3 runtimes for Development, Test and Production, plus Hotel and Datamart demo databases.

Sample application overview

The sample application launchDimCustomer is made of a simple process calling loadDimCustomer mapping, which loads data from customer table of Hotel database to Datamart database.

A lauchDimCustomer script is included in the project, to build the launchDimCustomer.pck package file.

You can clone this project ‘dimCustomer’ from github repository : https://github.com/stambia/samples

Environment setup

We will first upload the stambia designer zip file – the one you can download from Stambia.org- to a nexus raw repository :

The resulting download URL will be http://nexus:8081/service/rest/v1/search/assets/download?name=designer/stambiaDesigner_S20.1.0.zip

We will create a delivery project named ‘samples’ in analytics as a placeholder for the package to deploy on environments ENVDEV, ENVTST, ENVPRD :

Other useful information used later in this example :

Analytics API URL : http://analytics:8080/analytics/services/api/2/

Analytics environment names for deployment : ENVDEV, ENVTST, ENVPRD

Build the pipeline

Now that everything is in place, we can create the Jenkins pipeline to automate the deployment tasks.

Create a pipeline project ‘dimCustomer’ :

Once created, we can create the pipeline script skeleton and set the environment variables :

Add the variables

pipeline {
    agent any

    environment {
        NEXUS_USER = 'nexus_user'
        NEXUS_PASSWORD = 'nexus_password'
        STB_PROJECT = 'dimCustomer'
	PKG_NAME = 'dimcustomer'
	PKG_FILE = 'launchDimCustomer.pck'
	PKG_VERSION = '1.0'
	ANALYTICS_BASE_URL = 'http://analytics:8080/analytics/services/api/2'
	ANALYTICS_USER = 'analytics_user'
	ANALYTICS_PWD = 'analytics_password'
	ANALYTICS_AUTH = 'base64_auth_chain'
	DELIV_PROJECT = 'samples'
    }    

    stages {
        stage('Clean') {
            steps {
                cleanWs()
            }
        }
    }
}

We can now add the necessary steps to build our pipeline, from the sources retrieval to the deployment on the three target environments

Checkout stage

Get the sources from the github repository :

git https://github.com/stambia/samples

Download the designer zip file :

sh 'curl -X GET -L --user "${NEXUS_USER}:${NEXUS_PASSWORD}" http://nexus:8081/service/rest/v1/search/assets/download?name=designer/stambiaDesigner_S20.1.0.zip -o designer.zip'

Build package stage

Unzip the designer :

sh 'unzip ${WORKSPACE}/designer.zip -d ${WORKSPACE}'

Build package using script file :

sh 'java -Djava.util.Arrays.useLegacyMergeSort=true -jar ${WORKSPACE}/stambia/plugins/org.eclipse.equinox.launcher_1.5.100.v20180827-1352.jar -application com.indy.shell.application -data "." -script "${WORKSPACE}/${STB_PROJECT}/Build/Scripts/launchDimCustomer" -console -noSplash -importFrom "${WORKSPACE}/${STB_PROJECT}" -cleanProjects -rebuildCache'

Deploy to development stage

Analytics API will be used here to :

Create the package

sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}"'

Create the version ‘1.0’

sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/version/1.0" -H "accept: application/json"'

Upload the package file

sh 'curl --location --request PATCH "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/version/1.0" --header "Authorization: Basic ${ANALYTICS_AUTH}" --form "fileUpload=@${WORKSPACE}/${PKG_FILE}"'

Build and deploy the launchDimCustomer app

sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/environment/ENVDEV/version/1.0/working"'
sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/environment/ENVDEV/version/1.0/build"'
sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/environment/ENVDEV/version/1.0/deploy"'

Test stage

We are using the startCommand utility from the designer’s embedded runtime to execute delivery on the development runtime runtimeDEV on RMI port 42000 :

dir("${WORKSPACE}/stambia/stambiaRuntime/") {
sh './startcommand.sh "connect to runtimeDEV port 42000;execute delivery ${DELIV_PROJECT}/${PKG_NAME}/launchDimCustomer"'

Deploy to test and production stages

Then we can deploy to test and production environments using the same steps as for development, changing the environment name :

sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/environment/ENVTST/version/1.0/working"'
sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/environment/ENVTST/version/1.0/build"'
sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/environment/ENVTST/version/1.0/deploy"' 
sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/environment/ENVPRD/version/1.0/working"' 
sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/environment/ENVPRD/version/1.0/build"' 
sh 'curl -X POST --user "${ANALYTICS_USER}:${ANALYTICS_PWD}" "${ANALYTICS_BASE_URL}/project/${DELIV_PROJECT}/package/${PKG_NAME}/environment/ENVPRD/version/1.0/deploy"'

Test the pipeline

That’s it, we can now test the entire chain in Jenkins

We can also check the execution logs of the test step using for example Production Analytics...

...and the package V1.0 creation and deployment :

The version 1.0 of launchDimCustomer app is now available in Production !

You can find the full pipeline script here : document Pipeline script (4 KB)

Stambia CI/CD with Jenkins and Docker

Prerequisites

Sample application overview

Environment setup

Build the pipeline

Checkout stage

Build package stage

Deploy to development stage

Test stage

Deploy to test and production stages

Test the pipeline

Stambia CI/CD with Jenkins

Prerequisites

Sample application overview

Environment setup

Build the pipeline

Checkout stage

Build package stage

Deploy to development stage

Test stage

Deploy to test and production stages

Test the pipeline

Subcategories

Articles