WSO2 Puppet Deployment

WSO2 Products are accompanied with puppet modules which make your life easier when setting up and configuring a product as per the requirement/the deployment architecture. I am going to provide an introduction and guide on how to use these puppet modules for development or deployment purposes.

So if you are a developer and want to customize a WSO2 puppet module (to facilitate further flexibility or add more parameterized configurations), this post would be a good starter. 

And also if you are a user who directly want to setup/configure a puppet environment to deploy a certain enterprise deployment, you may read this.

WSO2 Puppet architecture was changed completely within last year and so now the puppet modules of each wso2 product are in separate git repositories as opposed to the old structure where all-were-in-one. The old WSO2 puppet-modules repository can be found here if you just want to have a lookup. That is now been deprecated and all the latest product related puppet scripts are written/been written under the new architecture, which I am going to describe here.

What is done by puppet...?

 

Before reading further let's clarify, what puppet does with respect to WSO2 products ? We have to understand this.

For a beginner into puppet, and for whom being ready to tackle with WSO2 puppet modules I may introduce puppet as below and this is very simple and premature introduction on what puppet does. (this concept could be common for any other puppet module too )

Following diagram (Figure 1) illustrate what is occurred simply when we use WSO2 APIM 2.1.0 puppet module to deploy and configure the product in production environment.

 

Figure 1

I guess you didn't understand this 100%. Don't worry. :-D .I am describing.

In the repository-which we call as WSO2 puppet module (for instance take wso2-apim-2.1.0 puppet module), there are configuration files acting as templates for each file that needs to be edited/configured in a product deployment. eg. axis2.xml, carbon.xml, master-datasources.xml.

The difference between an actual vanila product pack's config file and a related puppet template file is that the latter has been replaced with variables/parameters in order to change their values at runtime.

And a puppet module basically includes files (hiera files) with lists of values to be passed to each parameter/variable in each those config files. These values, which we call as hiera data are defined separately for each deployment pattern (or profile if available).

So when we "run" puppet, 3 basic steps are executed by puppet, as mentioned in the above figure.

1 - Apply the configuration data (of the required pattern), into the puppet template files.

2 - Replace the vanilla wso2am-2.1.0 product's configuration files with the modified template files of step 1.

3 - Copy the modified product pack, in step 2, into the production environment and start the product server.

Ok, now you know what we do with puppet, so shall we move in deeper. First we may clarify, the parts and particles of WSO2 puppet modules.

Organization of WSO2 Puppet repositories

If you are going to work with a certain WSO2 product (for a puppet deployment), you may have to deal with 3 functional components, which are found as git repositories.

1. The certain WSO2 product related repository
2. puppet-base repository
3. puppet-common repository

Both 2 and 3 are required for a puppet deployment of a WSO2 product.

1. The certain WSO2 product related repository

Each WSO2 product has a puppet-module repository. (i.e. puppet-apim, puppet-is, puppet-das, puppet-esb, puppet-iot, puppet-ei). Most of these has been released for latest product release ( as per the status by June 2017 ) and please find the puppet module repository list in here. These are consisted of puppet scripts that support multiple patterns of deployment and multiple profiles if available.

Let's take WSO2 API Manager puppet modules for instance. It consists of 3 puppet modules which are related to WSO2 APIM product. They are as below and the specif product related to each module is mentioned infront.

1. wso2am_runtime - WSO2 API Manager 2.1.0
2. wso2am_analytics - WSO2 APIM Analytics Server 2.1.0
3. wso2is_prepacked - Pre-packaged WSO2 Identity Server 5.3.0 (for IS  as  Key Manager APIM deployment)

And this wso2am_runtime module includes puppet scripts which facilitate deployment of APIM in 7 deployment patterns, with 5 APIM profiles.

2. puppet-base repository

WSO2 base puppet repository can be found in here. Puppet-base is also another "puppet module" according to the puppet perspective. This provides features for installing and configuring WSO2 products. On high level it does the following:

1. Install Java Runtime
2. Clean CARBON_HOME directory
3. Download and extract WSO2 product distribution
4. Apply Carbon Kernel and WSO2 product patches
5. Apply configuration data
6. Start WSO2 server as a service or in foreground

3. puppet-common repository

WSO2 Puppet Common repository provides files required for setting up a Puppet environment to deploy WSO2 products.

• manifests/site.pp: Puppet site manifest
• scripts/base.sh: Base bash script file which provides utility bash methods.
• setup.sh: The setup bash script for setting up a PUPPET_HOME environment for development work.
• vagrant A vagrant script for testing Puppet modules using VirtualBox.

Setting up a puppet environment

There are basically 2 approaches to setup a puppet environment.

1. Using vagrant and Orcle VirtualBox
2. Master agent environment
   

It is recommended to select the appropriate approach considering the requirement.

1. Using vagrant and Orcle VirtualBox

 

Vagrant can be used to setup the puppet development environment to easily test a WSO2 product's Puppet module.

In this approach, Vagrant is used to automate creation of a VirtualBox VM (Ubuntu 14.04) and deploy/install the WSO2 product using the WSO2 puppet modules.

This approach is very easier than Master-agent approach considering the convenience of setup. But this is less convenient in the case of debugging for errors as vagrant takes much time to up a WSO2 product with puppet as the process includes creating a Virtual Machine in Virtual Box too. If you are developing a WSO2 puppet module, from beginning, this is not the recommended approach. But if you are not a newbie to puppet, and so have a good expertise on how puppet modules works with WSO2 products, then you may use this approach (as u will make less errors).

And you cannot use this puppet environment to deploy and install a certain WSO2 product into an actual production environment. Because, this install the product into a VirtualBox Virtual Machine which is created automatically on the go.

For the steps to follow to use this approach follow the official WSO2 documentation Wikis in github here.

2. Master agent environment

 

  Master-agent environment can be used to deploy/install WSO2 products in actual production environments. And also if you are developing a puppet module from the beginning or doing major customizations to the existing puppet modules and your development task would take multiple days/weeks, it is better to follow this approach. Because this is convenient in the case of debugging, testing time for each run, re-running after customizations, etc. But this is bit cumbersome, to setup this master-agent environment as it takes much time and and also need multiple OS instances/computers.

To setup a master-agent puppet environment with WSO2 puppet modules, follow the steps in official WSO2 Documentation Wikis in github.


References:

https://github.com/wso2/puppet-base/blob/master/README.md
https://github.com/wso2/puppet-common/blob/master/README.md

Troubleshooting some Common Errors in Running Puppet Agent

Here I am going to guide you on how to troubleshoot some common errors in running puppet agent(client).

1. SSL Certificate Error

Puppet uses self signed certificates to communicate between Master(server) and Agent(client). When there is mismatch or verification failure, following error logs may display on the puppet agent.

Error log in Agent:

 

Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations

  (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: puppetmaster.openstacklocal]

Info: Loading facts

Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: puppetmaster.openstacklocal]

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: puppetmaster.openstacklocal]

Error log may be displayed as following too.

Error: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: puppetmaster.openstacklocal]

Solution:   

Following is the simplest solution. (recommended only if you are using a single Agent node).
Enter the following commands with root permissions,

1) on agent>> 

• rm -rf /var/lib/puppet/ssl/

2) on master>> 

• puppet cert clean --all
• service puppetmaster restart 

Then try to run agent again and the error should have been resolved.

A more elegant solution:

Usually when you encounter this kind of ssl issue, what you can do is first delete the ssl directory in the Agent.
   
     rm -rf /var/lib/puppet/ssl/

Then try to run Agent again and then the puppet will show you exactly what to do; something similar to below..

On the master:
  puppet cert clean node2-apim-publisher.openstacklocal
On the agent:
  1a. On most platforms: find /home/ubuntu/.puppet/ssl -name node2-apim-publisher.openstacklocal.pem -delete
  1b. On Windows: del "/home/ubuntu/.puppet/ssl/node2-apim-publisher.openstacklocal.pem" /f
  2. puppet agent -t

Do what puppet says as above and start puppet agent again.

I recommend to follow this solution as so here you are not deleting all the certificates related to each puppet agent. You are deleting only the relevant agent's certificate only.

2. "<unknown>" Error due to hira data file syntax error

Error log in Agent:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: (<unknown>):

Solution:

This error log with message “<unknown>” is mostly occurred due to a syntax error in a related hiera data .yaml file. So go through your hiera data files again. May be you can use some .yaml hiera data file validation online tools to validate your .yaml files. (eg. http://www.yamllint.com/)

3.  Agent node not defined on Master

Error log in Agent:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find default node or by name with 'node2-apim-publisher.openstacklocal, node2-apim-publisher' on node node2-apim-publisher.openstacklocal
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

("'node2-apim-publisher" is the hostname of my agent)

Solution:

This error occurs when you have not defined your Agent, in your master's related agent-node-defining  .pp file. This file exists usually in /etc/puppet/manifests/ of the Master and it's name can be site.pp or node.pp. You have to define the agent nodes using their hostnames in this file.

Sample node definition is as follows.

node "host-name-of-agent" {
 
}