Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device [1]. Performance of a software application/product/service can be measured/tested via load tests.
Performance of WSO2 products application systems too are widely tested via these load tests. Other than the performance tests, the JVM heap, CPU performance too can be monitored to determine the causes for certain performance issues of a system/application/product.
In this post I will discuss important facts related to these, basically on the below 2 aspects.
JMeter is a very convenient tool to generate a load to perform load tests. We can write JMeter scripts to run long running performance tests. In the case of WSO2 APIM, what we basically do is writing test scripts to call APIs published in the API Store.
Following is a simple JMeter test script, composed to test an API in the Store of WSO2 API Manager.
APIMSimpleTest.jmx
You can simply download this file and rename it to APIMSimpleTest.jmx and open it with JMeter if you want to play around with it.
Following are the basic items in this test script.
The HTTP request made to call the API is defined by this. (HTTP Request Method & Path, Web Server Protocol, Server, Port number )
This sets the 2 headers to the API call.
"View Results Tree" and "Summary Report" items under Thread group are added to view and analyze the test results. These are called "listeners" and they can be added to a thread group by Right Click on thread group> Add> Listner>
"View Results Tree" item facilitates viewing all the http requests made during the test and their responses. If you provide an empty file (with .jtl extension) to the "Write results to file/ Read from File>Filename" field, all the basic information on the http/https request will be written and saved into that file during the test.
"Summary Report" listener displays a summary of the test including Samples count, min/max/average response times, Error %, throughput, etc.
Note that you can use more listeners to analyze JMeter test results using .jtl file generated as mentioned above.
It is not required to have these listeners in the test script run time to make analyzing reports. You can just add any listener later after the test and provide the .jtl file and generate an analysis graph, table, etc.
JMeter ships with very few listeners and if you want to add more listeners you can add them to the JMeter as plugins.
Many important plugins can be downloaded from here.
Links for some useful plugins are listed below.
After adding these plugins, you may see them under the Add> Listner list of listeners.
Quick Analysis report Generation can be done using a .jtl file. This will generate a complete analysis report (with graphs/tables) as an .html web page file. This is a very important and convenient feature in JMeter.
To generate a report, just run following single jmeter command.
./jmeter -g <.jtl file> -o <output_dir_to_create_reports>
This will display many graphs/tables including the below.
When we analyze the performance of a WSO2 product, it is important to analyze on the CPU functionality, Java Heap Memory, Threads, etc. Oracle Java Mission Control (JMC) is an ideal tool for this, which is shipped with Oracle Java.
Oracle Java Mission Control is a tool suite for managing, monitoring, profiling, and troubleshooting your Java applications. Oracle Java Mission Control has been included in standard Java SDK since version 7u40. JMC consists of the JMX Console and the Java Flight Recorder. [2]
If you start JMC in machine that a WSO2 product runs, you can find tremendous amount of information on its performance and functionality.
Java Mission Control (JMC) uses Java Management Extensions (JMX) to communicate with remote Java processes and the JMX Console is a tool in JMC for monitoring and managing a running JVM instance.
This tool presents live data about memory and CPU usage, garbage collections, thread activity, and more.
To use this to monitor WSO2 product's JVM start the JMC on the computer in which the product is running. Under the JVM browser, you have to select the related JVM, "[1.x.x_xx]org.wso2.carbon.bootstrap.Bootstrap(xxxxx)".
Then right click on it and select "Start JMX console"
Now you can see the graphs, dashboards on Java heap memory, JVM CPU usage, etc. under the overview section.
The JMX console also consists of a live MBean browser, by which you can monitor and manage the MBeans related to the respective JVM.
In the case of WSO2 APIM, org.apache.synapse MBean will be useful to monitor the statistics related to API endpoints. Under the MBean tree org.apache.synapse>PassThroughLatencyView>nio_https_https, you can view average backend latency, average latency and many other feature attributes.
Java Flight Recorder is a profiling and event collection framework built into the Oracle JDK. This can be used to collect recordings and save into a file for later analysis.
Run JFR for a WSO2 product instance via JMC
To run JFR for a WSO2 product instance via JMC, for a fixed time period, follow the below steps.
To run JFR for a WSO2 product instance via command line, execute the following commands in the computer where the instance is running.
>jcmd carbon VM.unlock_commercial_features
This will unlock commercial features for the WSO2 carbon JVM, that will enable running JFR. Note that this name "carbon" here can be replaced by any word part separated by period in the related jvm-representation name "org.wso2.carbon.bootstrap.Bootstrap"
>jcmd carbon JFR.start settings=profile duration=3600s name=FullPerfTest filename=recording-1.jfr
This command will start the JFR for the WSO2 product JVM for a duration of 3600 seconds, and the recording file will be dumped to the <WSO2_PRODUCT_HOME> with the name recording-1.jfr at the end of the provided time duration.
You can refer this[3] blog post to learn on JFR in detail.
References
[1] - http://searchsoftwarequality.techtarget.com/definition/performance-testing
[2] - https://www.prosysopc.com/blog/using-java-mission-control-for-performance-monitoring/
[3] - https://medium.com/@chrishantha/using-java-flight-recorder-2367c01deacf
Performance of WSO2 products application systems too are widely tested via these load tests. Other than the performance tests, the JVM heap, CPU performance too can be monitored to determine the causes for certain performance issues of a system/application/product.
In this post I will discuss important facts related to these, basically on the below 2 aspects.
- Load Tests and result analysis.
- Using Oracle JMC for performance monitoring.
Load Tests and Result Analysis
I will take WSO2 API Manager for examples to describe this topic. In the case of WSO2 API Manager, the performance can be elaborated using the factors such as below.
- Transactions Per Second (TPS)
- Response time (minimum/average/maximum)
- Error rate
Following is a simple JMeter test script, composed to test an API in the Store of WSO2 API Manager.
APIMSimpleTest.jmx
You can simply download this file and rename it to APIMSimpleTest.jmx and open it with JMeter if you want to play around with it.
Following are the basic items in this test script.
- Thread Group - "API Calling Thread Group"
Following items exist within this test group.- HTTP Request - "Get Menu"
- View Results Tree
- Summary Report
- HTTP Header Manager
Thread Group - "API Calling Thread Group"
- Number of Threads (Users) : 2000
- Ramp-up Period(in seconds) : 100
- Scheduler configuration - Duration : 3600
HTTP Request - "Get Menu"
The HTTP request made to call the API is defined by this. (HTTP Request Method & Path, Web Server Protocol, Server, Port number )
HTTP Header Manager
This sets the 2 headers to the API call.
Analyzing JMeter test results
"View Results Tree" and "Summary Report" items under Thread group are added to view and analyze the test results. These are called "listeners" and they can be added to a thread group by Right Click on thread group> Add> Listner>
"View Results Tree" item facilitates viewing all the http requests made during the test and their responses. If you provide an empty file (with .jtl extension) to the "Write results to file/ Read from File>Filename" field, all the basic information on the http/https request will be written and saved into that file during the test.
"Summary Report" listener displays a summary of the test including Samples count, min/max/average response times, Error %, throughput, etc.
Note that you can use more listeners to analyze JMeter test results using .jtl file generated as mentioned above.
It is not required to have these listeners in the test script run time to make analyzing reports. You can just add any listener later after the test and provide the .jtl file and generate an analysis graph, table, etc.
JMeter ships with very few listeners and if you want to add more listeners you can add them to the JMeter as plugins.
Many important plugins can be downloaded from here.
Links for some useful plugins are listed below.
After adding these plugins, you may see them under the Add> Listner list of listeners.
Quick Analysis report Generation can be done using a .jtl file. This will generate a complete analysis report (with graphs/tables) as an .html web page file. This is a very important and convenient feature in JMeter.
To generate a report, just run following single jmeter command.
./jmeter -g <.jtl file> -o <output_dir_to_create_reports>
This will display many graphs/tables including the below.
- Test and Report informations
- APDEX (Application Performance Index)
- Requests Summary
- Statistics per each thread - (average,min,max response times/throughput, error rates)
- Details descriptions on the errors occured
- Over Time based Charts
- Throughput related graphs
- Response times related graphs
Using Oracle Java Mission Control for performance monitoring
Oracle Java Mission Control is a tool suite for managing, monitoring, profiling, and troubleshooting your Java applications. Oracle Java Mission Control has been included in standard Java SDK since version 7u40. JMC consists of the JMX Console and the Java Flight Recorder. [2]
If you start JMC in machine that a WSO2 product runs, you can find tremendous amount of information on its performance and functionality.
1. Using JMX Console
Java Mission Control (JMC) uses Java Management Extensions (JMX) to communicate with remote Java processes and the JMX Console is a tool in JMC for monitoring and managing a running JVM instance.
This tool presents live data about memory and CPU usage, garbage collections, thread activity, and more.
To use this to monitor WSO2 product's JVM start the JMC on the computer in which the product is running. Under the JVM browser, you have to select the related JVM, "[1.x.x_xx]org.wso2.carbon.bootstrap.Bootstrap(xxxxx)".
Then right click on it and select "Start JMX console"
Now you can see the graphs, dashboards on Java heap memory, JVM CPU usage, etc. under the overview section.
The JMX console also consists of a live MBean browser, by which you can monitor and manage the MBeans related to the respective JVM.
In the case of WSO2 APIM, org.apache.synapse MBean will be useful to monitor the statistics related to API endpoints. Under the MBean tree org.apache.synapse>PassThroughLatencyView>nio_https_https, you can view average backend latency, average latency and many other feature attributes.
2. Using Java Flight Recorder
Java Flight Recorder is a profiling and event collection framework built into the Oracle JDK. This can be used to collect recordings and save into a file for later analysis.
Run JFR for a WSO2 product instance via JMC
To run JFR for a WSO2 product instance via JMC, for a fixed time period, follow the below steps.
- Select the related JVM, "[1.x.x_xx]org.wso2.carbon.bootstrap.Bootstrap(xxxxx)".
- Then right click on it and select "Start Flight Recording". You will be prompted whether to enable Java commercial features and click "Yes" for it.
- Then provide a file name and location to dump the recording file.
- Select "Time Fixed Recording" and provide the recording time you want and click "Finish".
- After the provided time, the recording .jfr file will be saved in the given location. You can open it in JMC anytime to analyze the recordings.
Running JFR from Command Line
To run JFR for a WSO2 product instance via command line, execute the following commands in the computer where the instance is running.
>jcmd carbon VM.unlock_commercial_features
This will unlock commercial features for the WSO2 carbon JVM, that will enable running JFR. Note that this name "carbon" here can be replaced by any word part separated by period in the related jvm-representation name "org.wso2.carbon.bootstrap.Bootstrap"
>jcmd carbon JFR.start settings=profile duration=3600s name=FullPerfTest filename=recording-1.jfr
This command will start the JFR for the WSO2 product JVM for a duration of 3600 seconds, and the recording file will be dumped to the <WSO2_PRODUCT_HOME> with the name recording-1.jfr at the end of the provided time duration.
You can refer this[3] blog post to learn on JFR in detail.
References
[1] - http://searchsoftwarequality.techtarget.com/definition/performance-testing
[2] - https://www.prosysopc.com/blog/using-java-mission-control-for-performance-monitoring/
[3] - https://medium.com/@chrishantha/using-java-flight-recorder-2367c01deacf
