WSO2 Puppet Deployment

WSO2 Products are accompanied with puppet modules which make your life easier when setting up and configuring a product as per the requirement/the deployment architecture. I am going to provide an introduction and guide on how to use these puppet modules for development or deployment purposes.

So if you are a developer and want to customize a WSO2 puppet module (to facilitate further flexibility or add more parameterized configurations), this post would be a good starter. 

And also if you are a user who directly want to setup/configure a puppet environment to deploy a certain enterprise deployment, you may read this.

WSO2 Puppet architecture was changed completely within last year and so now the puppet modules of each wso2 product are in separate git repositories as opposed to the old structure where all-were-in-one. The old WSO2 puppet-modules repository can be found here if you just want to have a lookup. That is now been deprecated and all the latest product related puppet scripts are written/been written under the new architecture, which I am going to describe here.

What is done by puppet...?

 

Before reading further let's clarify, what puppet does with respect to WSO2 products ? We have to understand this.

For a beginner into puppet, and for whom being ready to tackle with WSO2 puppet modules I may introduce puppet as below and this is very simple and premature introduction on what puppet does. (this concept could be common for any other puppet module too )

Following diagram (Figure 1) illustrate what is occurred simply when we use WSO2 APIM 2.1.0 puppet module to deploy and configure the product in production environment.

 

Figure 1

I guess you didn't understand this 100%. Don't worry. :-D .I am describing.

In the repository-which we call as WSO2 puppet module (for instance take wso2-apim-2.1.0 puppet module), there are configuration files acting as templates for each file that needs to be edited/configured in a product deployment. eg. axis2.xml, carbon.xml, master-datasources.xml.

The difference between an actual vanila product pack's config file and a related puppet template file is that the latter has been replaced with variables/parameters in order to change their values at runtime.

And a puppet module basically includes files (hiera files) with lists of values to be passed to each parameter/variable in each those config files. These values, which we call as hiera data are defined separately for each deployment pattern (or profile if available).

So when we "run" puppet, 3 basic steps are executed by puppet, as mentioned in the above figure.

1 - Apply the configuration data (of the required pattern), into the puppet template files.

2 - Replace the vanilla wso2am-2.1.0 product's configuration files with the modified template files of step 1.

3 - Copy the modified product pack, in step 2, into the production environment and start the product server.

Ok, now you know what we do with puppet, so shall we move in deeper. First we may clarify, the parts and particles of WSO2 puppet modules.

Organization of WSO2 Puppet repositories

If you are going to work with a certain WSO2 product (for a puppet deployment), you may have to deal with 3 functional components, which are found as git repositories.

1. The certain WSO2 product related repository
2. puppet-base repository
3. puppet-common repository

Both 2 and 3 are required for a puppet deployment of a WSO2 product.

1. The certain WSO2 product related repository

Each WSO2 product has a puppet-module repository. (i.e. puppet-apim, puppet-is, puppet-das, puppet-esb, puppet-iot, puppet-ei). Most of these has been released for latest product release ( as per the status by June 2017 ) and please find the puppet module repository list in here. These are consisted of puppet scripts that support multiple patterns of deployment and multiple profiles if available.

Let's take WSO2 API Manager puppet modules for instance. It consists of 3 puppet modules which are related to WSO2 APIM product. They are as below and the specif product related to each module is mentioned infront.

1. wso2am_runtime - WSO2 API Manager 2.1.0
2. wso2am_analytics - WSO2 APIM Analytics Server 2.1.0
3. wso2is_prepacked - Pre-packaged WSO2 Identity Server 5.3.0 (for IS  as  Key Manager APIM deployment)

And this wso2am_runtime module includes puppet scripts which facilitate deployment of APIM in 7 deployment patterns, with 5 APIM profiles.

2. puppet-base repository

WSO2 base puppet repository can be found in here. Puppet-base is also another "puppet module" according to the puppet perspective. This provides features for installing and configuring WSO2 products. On high level it does the following:

1. Install Java Runtime
2. Clean CARBON_HOME directory
3. Download and extract WSO2 product distribution
4. Apply Carbon Kernel and WSO2 product patches
5. Apply configuration data
6. Start WSO2 server as a service or in foreground

3. puppet-common repository

WSO2 Puppet Common repository provides files required for setting up a Puppet environment to deploy WSO2 products.

• manifests/site.pp: Puppet site manifest
• scripts/base.sh: Base bash script file which provides utility bash methods.
• setup.sh: The setup bash script for setting up a PUPPET_HOME environment for development work.
• vagrant A vagrant script for testing Puppet modules using VirtualBox.

Setting up a puppet environment

There are basically 2 approaches to setup a puppet environment.

1. Using vagrant and Orcle VirtualBox
2. Master agent environment
   

It is recommended to select the appropriate approach considering the requirement.

1. Using vagrant and Orcle VirtualBox

 

Vagrant can be used to setup the puppet development environment to easily test a WSO2 product's Puppet module.

In this approach, Vagrant is used to automate creation of a VirtualBox VM (Ubuntu 14.04) and deploy/install the WSO2 product using the WSO2 puppet modules.

This approach is very easier than Master-agent approach considering the convenience of setup. But this is less convenient in the case of debugging for errors as vagrant takes much time to up a WSO2 product with puppet as the process includes creating a Virtual Machine in Virtual Box too. If you are developing a WSO2 puppet module, from beginning, this is not the recommended approach. But if you are not a newbie to puppet, and so have a good expertise on how puppet modules works with WSO2 products, then you may use this approach (as u will make less errors).

And you cannot use this puppet environment to deploy and install a certain WSO2 product into an actual production environment. Because, this install the product into a VirtualBox Virtual Machine which is created automatically on the go.

For the steps to follow to use this approach follow the official WSO2 documentation Wikis in github here.

2. Master agent environment

 

  Master-agent environment can be used to deploy/install WSO2 products in actual production environments. And also if you are developing a puppet module from the beginning or doing major customizations to the existing puppet modules and your development task would take multiple days/weeks, it is better to follow this approach. Because this is convenient in the case of debugging, testing time for each run, re-running after customizations, etc. But this is bit cumbersome, to setup this master-agent environment as it takes much time and and also need multiple OS instances/computers.

To setup a master-agent puppet environment with WSO2 puppet modules, follow the steps in official WSO2 Documentation Wikis in github.


References:

https://github.com/wso2/puppet-base/blob/master/README.md
https://github.com/wso2/puppet-common/blob/master/README.md