Posts Tagged ‘wso2’

New Relic is a popular performance monitoring system which provided realtime analytics such as performance, memory usage and cpu usage, threads, web page response time etc. You can even profile application remotely using New Relic dashboard.

This article explains how to integrate New Relic performance monitoring Java agent with WSO2 Carbon products.

Tested platform: Java 8, WSO2 ESB 5.0.0, Mac OS Sierra 10.12.3

1) Signup in New Relic website.
You will get a license key once subscribe

2) Download and extract New Relic agent jar zip files as below. It contains
i) New relic Agent Jar file
ii) newrelic.yml configuration yaml file

wget -N https://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip
unzip -q newrelic-java.zip

3) Copy newrelic.jar and newrelic.yml into

mkdir $CARBON_HOME/newrelicAgent
cp newrelic.yml $CARBON_HOME/newrelicAgent
cp newrelic.yml $CARBON_HOME/newrelicAgent

4) Set New Relic licence key in newrelic.yml
Locate to this section in license_key: ‘<%= license_key %>’ and replace it with the licence key you received at Step 1.

license_key: 'e5620kj287aee4ou7613c2ku7d56k12387bd5jyb'

5) Set java agent into $CARBON_HOME/bin/wso2server.sh as below

-javaagent:$CARBON_HOME/newrelicAgent/newrelic.jar \

Sample section looks like this

while [ "$status" = "$START_EXIT_STATUS" ]
do
    $JAVACMD \
    -Xbootclasspath/a:"$CARBON_XBOOTCLASSPATH" \
    $JVM_MEM_OPTS \
    -XX:+HeapDumpOnOutOfMemoryError \
    -XX:HeapDumpPath="$CARBON_HOME/repository/logs/heap-dump.hprof" \
    $JAVA_OPTS \
    -javaagent:$CARBON_HOME/newrelicAgent/newrelic.jar \

4) sh $ESB_HOME/bin/wso2server.sh

At startup you will see below logs in carbon log file

Mar 26, 2017 13:08:58 +0800 [12884 1] com.newrelic INFO: New Relic Agent: Loading configuration file "/Users/udara/projects/testings/relic/wso2esb-5.0.0-BETA2/newrelicAgent/./newrelic.yml"
Mar 26, 2017 13:08:59 +0800 [12884 1] com.newrelic INFO: New Relic Agent: Writing to log file: /Users/udara/projects/testings/relic/wso2esb-5.0.0-BETA2/newrelic/logs/newrelic_agent.log

5) Do some operations such as accessing management console, accessing apis etc. Then Login to New Relic dashboard where you will find statistics about your carbon product.

Screen Shot 2017-03-26 at 12.59.05 PM.jpg

Screen Shot 2017-03-26 at 1.42.35 PM

Screen Shot 2017-03-26 at 1.25.43 PM.jpg

Screen Shot 2017-03-26 at 1.47.07 PM

Beware of below error

When I tried the same with WSO2 API Manager 2.1.0 I encountered the below error at server startup. Post [2] has suggested that it is due to an issue with temp directory.  The root cause for this is WSO2 startup scripts deletes TMP_DIR at startup script which causes New Relic not able to write to the temp directory. The fix is to delete the content of TMP_DIR instead of deleting the whole directory. So you will have to change CARBON_HOME/bin/wso2server.sh as below. Just comment TMP_DIR folder deletion and modify it to remove only the folder content.

TMP_DIR="$CARBON_HOME"/tmp
#if [ -d "$TMP_DIR" ]; then
#rm -rf "$TMP_DIR"
#fi

if [-d "$TMP_DIR"]; then
rm -rf "$TMP_DIR/*"
fi
Error bootstrapping New Relic agent: java.lang.RuntimeException: java.io.IOException: No such file or directory
java.lang.RuntimeException: java.io.IOException: No such file or directory
    at com.newrelic.bootstrap.BootstrapLoader.load(BootstrapLoader.java:122)
    at com.newrelic.bootstrap.BootstrapAgent.startAgent(BootstrapAgent.java:110)
    at com.newrelic.bootstrap.BootstrapAgent.premain(BootstrapAgent.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.io.IOException: No such file or directory

References

[1] http://lasanthatechlog.blogspot.com/2015/06/integrating-wso2-products-with-new-relic.html

[2] https://discuss.newrelic.com/t/error-bootstrapping-new-relic-agent-in-hadoop-mapreduce-job/23763

Advertisements

In my earlier post, I wrote how to filter Json payload using Groovy scripts in WSO2 ESB script mediator. This post is its XML counterpart.

If you did not read my earlier post, the Script Mediator of WSO2 ESB used to invoke the functions of a variety of scripting languages such as JavaScript, Groovy, or Ruby.

In this example, payload consists of xml payload with the details of  set of employees. We are going to filter out
old employees (age >30) from this list. However using Groovy I found it easier to remove young employees and keep the old employees
in payload.

Prerequisites:
Download Groovy all dependency jar (I used groovy-all-2.2.0-beta-1.jar) into $ESB_HOME/repository/lib and start WSO2 ESB

Here is the payload before the script mediator.

<employees>
<employee>
<age>25</age>
<firstName>John</firstName>
<lastName>Doe</lastName>
</employee>
<employee>
<age>45</age>
<firstName>Anna</firstName>
<lastName>Smith</lastName>
</employee>
<employee>
<age>35</age>
<firstName>Peter</firstName>
<lastName>Jones</lastName>
</employee>
</employees>

 

Now lets write the script mediator which filter out employees younger than 30 years.

<property name="messageType"; value="application/json" scope="axis2" />
<property name="payload" expression="json-eval($.)" />
<script language="groovy">
import groovy.util.XmlSlurper;
import groovy.xml.MarkupBuilder;
import groovy.xml.StreamingMarkupBuilder;

def payload = mc.getPayloadXML();
def rootNode = new XmlSlurper().parseText(payload);
rootNode.children().findAll{it.age.text().toInteger() &lt; 30 }.replaceNode {};

mc.setPayloadXML(groovy.xml.XmlUtil.serialize(rootNode));
</script>

 

First I fetches payload using getPayloadXML provided by Synapse. Then I parse the payload as XML using parseText() of XmlSlurper class.
Then I findAll employees who’s age is less than 30 by finding and remove them. Finally serialize the object and set to synapse message context as new payload.
So the new payload consists of old employees as below

<employees>
<employee>
<age>45</age>
<firstName>Anna</firstName>
<lastName>Smith</lastName>
</employee>
<employee>
<age>35</age>
<firstName>Peter</firstName>
<lastName>Jones</lastName>
</employee>
</employees>

The Script Mediator is used to invoke the functions of a variety of scripting languages such as JavaScript, Groovy, or Ruby.
This port consists of a sample in Groovy scripting language using which which you can perform Collection operation easily.

Prerequisites:
Download Groovy all dependency jar (I used groovy-all-2.2.0-beta-1.jar) into $ESB_HOME/repository/lib and start WSO2 ESB

Let’s say that your current payload consists of set of employees represented as below.

{
  "employees": [
    {
      "firstName": "John";,
      "lastName": "Doe",
      "age":25
    },
    {
      "firstName": "Anna",
      "lastName": "Smith",
      "age":45
    },
    {
      "firstName": "Peter",
      "lastName":"Jones",
      "age":35
    }
  ]
}

Now you want to filter out the set of old(age>30) employees to apply a new insurance policy.
Let’s see how you can achieve this task using WSO2 ESB script mediator using groovy script.

<property name="messageType"; value="application/json" scope="axis2" />
<property name="payload" expression="json-eval($.)" />

<script language="groovy">
 import groovy.json.*;
 def payload = mc.getProperty("payload");
 def empList = new JsonSlurper().parseText(payload.toString());
 empList.employees = empList.employees.findAll{it.age gt; 30}
 mc.setPayloadJSON(JsonOutput.toJson(empList));
</script>

First I set property “payload” to store message payload before script mediator.
Then withing script mediator I fetches its content using mc.getProperty(). Then parse the paylod
to Json which converts Json payload string to Groovy object, List type in this case. There after I can
use Groovy funtion findAll() to filter employees using Closure age>30. Finally converts Grooby object
back to Json String in toJson() funtions and set the filtered employees as payload.

So payload will be changed as below, to consist only old employees after going through the script mediator.

{
  "employees": [
    {
      "firstName": "Anna",
      "lastName": "Smith",
      "age": 45
    },
    {
     "firstName": "Peter",
      "lastName": "Jones",
      "age": 35
    }
  ]
}

This article demonstrate how to build a sample REST service using WSO2 Micro Service Server.

Step 1: Build the product

git clone https://github.com/wso2/msf4j.git
mvn clean install

Step 2: Create sample micro service project

mvn archetype:generate \
-DarchetypeGroupId=org.wso2.msf4j \
-DarchetypeArtifactId=msf4j-microservice \
-DarchetypeVersion=1.0.0-SNAPSHOT \
-DgroupId=org.example -DartifactId=Customer-Service \
-Dversion=0.1-SNAPSHOT \
-Dpackage=org.example.service \
-DserviceClass=CustomerService

Once the project is created, it will generate following source code structure. CustomerService.java is service file generated for you.

Sample service sources  generated

Sample service sources generated

Step 3: Create sample Json service

Open CustomerService.java from your IDE and replaces generated sample methods with following method. Method “getCustomer” exposes a GET service which return a simple Customer object.

@Path("/customer")
public class CustomerService {

@GET
@Path("/")
@Produces({"application/json", "text/xml"})
public Response getCustomer() {
&nbsp;return Response.status(Response.Status.OK).entity(new Customer("udara", "wso2")).build();
}

private class Customer {
String name;
String company;

public Customer(String name, String company) {
this.name = name;
this.company = company;
}
}
}

Step 4: Run Application.java using your IDE

2016-02-10 12:17:41 INFO  MicroservicesRegistry:76 – Added microservice: org.example.service.HelloService@6aa8ceb6
2016-02-10 12:17:41 INFO  NettyListener:56 – Starting Netty Http Transport Listener
2016-02-10 12:17:42 INFO  NettyListener:80 – Netty Listener starting on port 8080
2016-02-10 12:17:42 INFO  MicroservicesRunner:122 – Microservices server started in 436ms

Step 5 : Invoke the micro service we just implemented

$ curl -X GET http://localhost:8080/customer/ | python -m json.tool
{
“company”: “wso2”,
“name”: “sampath”
}

Please note that “customer” is the path given to CustomerService class.

References

http://blog.afkham.org/2016/02/writing-your-first-java-microservices.html

This is a python code snippet I wrote to automate API creation in WSO2 API Manager. WSO2 API manager has exposed a API, Publisher API  using which we can perform APIM related task.

This python client first login to APIM and check weather there is already an API with the same name, if not it will create an API. This API has two resources each has unlimited throttling tier

  1. PUT /cart
  2. POST /checkout

In addition API has fail over endpoint, one production endpoint and one fail over endpoint. So once the API is invoked, API first try to reach the production endpoint, if production endpoint is not available it will try to reach fail over endpoint.

Once the API is created, this code will publishes the newly created API so users can subscribe to the API.

	log = LogFactory().get_log(__name__)
        log.info("Starting api creation plugin...")

        apim_domain = "localhost"
        apim_username = "admin"
        apim_password = "admin"

        endpoint = "http://api.openweathermap.org/data/2.5/weather?q=London"

        url =  'https://%s:9443/publisher/site/blocks/' %  apim_domain

        loging_url = urlparse.urljoin(url, 'user/login/ajax/login.jag')
        payload = {'action':'login', 'username':apim_username, 'password':apim_password }

        log.info("Login into APIManager %s " % loging_url)
        resp = requests.post(loging_url, data=payload, verify=False)
        log.info("APIM Logging response %s" % resp)
        cookie = resp.cookies

        swagger = {'paths': {'/cart': {'put': {'x-auth-type': 'None',
                                               'x-throttling-tier': 'Unlimited',
                                               'responses': {'200': {}}}}, '/checkout': {'post': {
            'parameters': [{
                               'schema': {'type': 'object'},
                               'description': 'Request Body',
                               'name': 'Payload',
                               'required': 'false',
                               'in': 'body',
                               }],
            'responses': {'200': {}},
            'x-auth-type': 'None',
            'x-throttling-tier': 'Unlimited',
            }}}}

        swager_json = json.dumps(swagger)

        api_url = urlparse.urljoin(url,'item-add/ajax/add.jag')

        endpoint_conf = \
            {'production_endpoints': {'url': 'http://ws.cdyne.com/phoneverify/phoneverify.asmx',
                                      'config': 'null'},
             'production_failovers': [{'url': 'http://failover_domain:30000/StorefrontDemo/api/customer'
                                          , 'config': 'null'}], 'endpoint_type': 'failover'}

        endpoint_conf['production_endpoints']['url']= endpoint
        endpoint_json = json.dumps(endpoint_conf)

        payload = {
            'action': 'addAPI',
            'name': 'Storefront',
            'context': 'storefront',
            'version': 'v1',
            'visibility': 'public',
            'endpointType': 'nonsecured',
            'tiersCollection': 'Unlimited',
            'http_checked': 'http',
            'https_checked': 'https',
            'resourceCount': 0,
            'resourceMethod-0': 'PUT',
            'resourceMethodAuthType-0': 'None',
            'resourceMethodThrottlingTier-0': 'Unlimited',
            'uriTemplate-0': 'cart',
            'resourceMethod-0': 'POST',
            'resourceMethodAuthType-0': 'None',
            'resourceMethodThrottlingTier-0': 'Unlimited',
            'uriTemplate-0': 'checkout',
            }

        payload['endpoint_config']=endpoint_json
        payload['swagger'] = swager_json

        exist_payload = {
        'action':'isAPINameExist',
        'apiName':'Storefront'
        }

        #check if API with the same name already exist
        resp = requests.post(api_url, data = exist_payload, verify = False, cookies = cookie)
        api_exist =  ('true' == json.loads(resp.text)['exist'])
        logging.info("API already exist %s " % api_exist)

        if not api_exist:            
            log.info("Creating API WebbAppAPI %s " % api_url)
            resp = requests.post(api_url, data=payload, verify=False, cookies=cookie)
            log.info("APIM api creation response %s" % resp)

            publish_url = urlparse.urljoin(url, 'life-cycles/ajax/life-cycles.jag')
            payload = {
            'action':'updateStatus',
            'name':'Storefront',
            'version':'v1',
            'provider':'admin',
            'status':'PUBLISHED',
            'publishToGateway':'true',
            'requireResubscription':'false'
            }
            log.info("Publishing API WebbAppAPI %s " % publish_url)
            resp = requests.post(publish_url, data=payload, verify=False, cookies=cookie)
            log.info("APIM api publishing response %s" % resp)

        log.info("*****************API creation plugin completed *****************")
	

Below is the created API

Storefront API

Storefront API

When using WSO2 ESB as a JMS consumer you can use ConcurrentConsumers and MaxConcurrentConsumers properties to control the number of threads used to consume messages in the JMS queue or topic.

ConcurrentConsumers is the minimum number of threads for message consuming. If there are more messages to be consumed while those running threads are busy, then additional threads are started until total number of threads reaches MaxConcurrentConsumers. Simply saying, initially “ConcurrentConsumers” of threads are started in order to consume messages. Then if there are more message in the queue/topic yet to be consumed while running threads are busy, additional threads are started in order to consume the remaining messages. Like wise more threads are started untill total number of threads reaches “MaxConcurrentConsumers” number.

Below is a sample JMSListener configuration in axis2.xml. It tells use 5 threads intially and increase number of threads upto 20 according to the load.

&lt;transportReceiver name="jms" class="org.apache.axis2.transport.jms.JMSListener"&gt; 

        &lt;parameter name="myTopicConnectionFactory" locked="false"&gt; 
                &lt;parameter locked="false" name="transport.jms.ConcurrentConsumers"&gt;5&lt;/parameter&gt; 
                &lt;parameter locked="false" name="transport.jms.MaxConcurrentConsumers"&gt;20&lt;/parameter&gt; 
                &lt;parameter name="java.naming.factory.initial" locked="false"&gt;org.apache.activemq.jndi.ActiveMQInitialContextFactory&lt;/parameter&gt; 
                &lt;parameter name="java.naming.provider.url" locked="false"&gt;failover:tcp://localhost:61616&amp;lt;/parameter&gt; 
                &lt;parameter name="transport.jms.ConnectionFactoryJNDIName" locked="false"&gt;TopicConnectionFactory&lt;/parameter&gt; 
                    &lt;parameter name="transport.jms.ConnectionFactoryType" locked="false"&gt;topic&lt;/parameter&gt; 
        &lt;/parameter&gt; 

 

Additionally you can override the values in axis.xml in your proxy too.

In order to understand and test APIs created, WSO2 API manager provides interactive documentation. WSO2AM incoparates swagger[https://developers.helloreverb.com/swagger] for this puprpose.In Swagger, we can define a API using a static JSON file. In APIM, when we create an API, it automatically generates the JSON representation of the API which is loaded by the Swagger. In this tutorial, let’s see how we can edit the JSON representation in order to add some custom header to when calling the API. Below are the steps that are going to do in this tutorial.

  • create an API
  • Use Swagger to invoke the API
  • Describe Swagger documantation
  • Edit Swagger documentation to add a custom header named “username”
  • Invoke the API using Swagger with the newly added header

 

Step1 : Create an API

Design API

 

APi Design

APi Design

Implement API

api-implement

Manage API

api manager

 

Swagger doc.

To view the swagger documentation
Locate to APIM store  ->select the API -> Click on “API Console” tab.

swagger doc

Here you see a header called “Authorization” and a query parameter named “Query Parameter”. However if you want to add another header or query parameter, you have to edit the Swagger documentation of the API.

 

Step2:

Edit Swagger doc

To edit the Swagger documentation
Locate to APIMPublisher -> Select API -> click Docs tab -> Edit Content link

edit

 

swagger doc edit

 

Below is the existing JSON representation of the API we created.

{
    &amp;quot;apiVersion&amp;quot;: &amp;quot;1.0.0&amp;quot;,
    &amp;quot;swaggerVersion&amp;quot;: &amp;quot;1.1&amp;quot;,
    &amp;quot;basePath&amp;quot;: &amp;quot;http://192.168.122.1:8280&amp;quot;,
    &amp;quot;resourcePath&amp;quot;: &amp;quot;/swagger&amp;quot;,
    &amp;quot;apis&amp;quot;: [
        {
            &amp;quot;path&amp;quot;: &amp;quot;/swagger/1.0.0/users&amp;quot;,
            &amp;quot;description&amp;quot;: &amp;quot;&amp;quot;,
            &amp;quot;operations&amp;quot;: [
                {
                    &amp;quot;httpMethod&amp;quot;: &amp;quot;GET&amp;quot;,
                    &amp;quot;summary&amp;quot;: &amp;quot;&amp;quot;,
                    &amp;quot;nickname&amp;quot;: &amp;quot;&amp;quot;,
                    &amp;quot;parameters&amp;quot;: [
                        {
                            &amp;quot;name&amp;quot;: &amp;quot;Query Parameters&amp;quot;,
                            &amp;quot;description&amp;quot;: &amp;quot;Request Query Parameters&amp;quot;,
                            &amp;quot;paramType&amp;quot;: &amp;quot;body&amp;quot;,
                            &amp;quot;required&amp;quot;: false,
                            &amp;quot;allowMultiple&amp;quot;: false,
                            &amp;quot;dataType&amp;quot;: &amp;quot;String&amp;quot;
                        },
                        {
                            &amp;quot;name&amp;quot;: &amp;quot;Authorization&amp;quot;,
                            &amp;quot;description&amp;quot;: &amp;quot;OAuth2 Authorization Header&amp;quot;,
                            &amp;quot;paramType&amp;quot;: &amp;quot;header&amp;quot;,
                            &amp;quot;required&amp;quot;: false,
                            &amp;quot;allowMultiple&amp;quot;: false,
                            &amp;quot;dataType&amp;quot;: &amp;quot;String&amp;quot;
                        }

                    ]
                }
            ]
        }
    ]
}

Under parameters section you can see the already defined header “Authorization”. Let’s add another header “username” by adding the below parameter definition under the “parameters” section.

		{
                            &amp;quot;name&amp;quot;: &amp;quot;username&amp;quot;,
                            &amp;quot;description&amp;quot;: &amp;quot;username of the user&amp;quot;,
                            &amp;quot;paramType&amp;quot;: &amp;quot;header&amp;quot;,
                            &amp;quot;required&amp;quot;: false,
                            &amp;quot;allowMultiple&amp;quot;: false,
                            &amp;quot;dataType&amp;quot;: &amp;quot;String&amp;quot;
                        }

After adding the header representation, whole Swagger documentation of the API is

{
    &amp;quot;apiVersion&amp;quot;: &amp;quot;1.0.0&amp;quot;,
    &amp;quot;swaggerVersion&amp;quot;: &amp;quot;1.1&amp;quot;,
    &amp;quot;basePath&amp;quot;: &amp;quot;http://192.168.122.1:8280&amp;quot;,
    &amp;quot;resourcePath&amp;quot;: &amp;quot;/swagger&amp;quot;,
    &amp;quot;apis&amp;quot;: [
        {
            &amp;quot;path&amp;quot;: &amp;quot;/swagger/1.0.0/users&amp;quot;,
            &amp;quot;description&amp;quot;: &amp;quot;&amp;quot;,
            &amp;quot;operations&amp;quot;: [
                {
                    &amp;quot;httpMethod&amp;quot;: &amp;quot;GET&amp;quot;,
                    &amp;quot;summary&amp;quot;: &amp;quot;&amp;quot;,
                    &amp;quot;nickname&amp;quot;: &amp;quot;&amp;quot;,
                    &amp;quot;parameters&amp;quot;: [
                        {
                            &amp;quot;name&amp;quot;: &amp;quot;Query Parameters&amp;quot;,
                            &amp;quot;description&amp;quot;: &amp;quot;Request Query Parameters&amp;quot;,
                            &amp;quot;paramType&amp;quot;: &amp;quot;body&amp;quot;,
                            &amp;quot;required&amp;quot;: false,
                            &amp;quot;allowMultiple&amp;quot;: false,
                            &amp;quot;dataType&amp;quot;: &amp;quot;String&amp;quot;
                        },
                        {
                            &amp;quot;name&amp;quot;: &amp;quot;Authorization&amp;quot;,
                            &amp;quot;description&amp;quot;: &amp;quot;OAuth2 Authorization Header&amp;quot;,
                            &amp;quot;paramType&amp;quot;: &amp;quot;header&amp;quot;,
                            &amp;quot;required&amp;quot;: false,
                            &amp;quot;allowMultiple&amp;quot;: false,
                            &amp;quot;dataType&amp;quot;: &amp;quot;String&amp;quot;
                        },
{
                            &amp;quot;name&amp;quot;: &amp;quot;username&amp;quot;,
                            &amp;quot;description&amp;quot;: &amp;quot;username of the user&amp;quot;,
                            &amp;quot;paramType&amp;quot;: &amp;quot;header&amp;quot;,
                            &amp;quot;required&amp;quot;: false,
                            &amp;quot;allowMultiple&amp;quot;: false,
                            &amp;quot;dataType&amp;quot;: &amp;quot;String&amp;quot;
                        }

                    ]
                }
            ]
        }
    ]
}

New Swagger Doc

new swagger

 

Now add the new header name to Access-Control-Allow-Headers section of repository/conf/api-manager.xml as below

&lt;Access-Control-Allow-Headers&gt;authorization,Access-Control-Allow-Origin,Content-Type, CustomHeader&lt;/Access-Control-Allow-Headers&gt;

 

HTTP request before and after

Request to the API before

udara@udara$ nc -l 7777
GET http://localhost:7777/users HTTP/1.1
Accept: */*
Host: localhost:7777
Connection: Keep-Alive
User-Agent: Synapse-PT-HttpComponents-NIO

Request to the API after adding the header to the Swagger documentation

udara@udara$ nc -l 7777
GET http://192.168.122.1:7777/users HTTP/1.1
username: udara
Accept: */*
Host: 192.168.122.1:7777
Connection: Keep-Alive
User-Agent: Synapse-PT-HttpComponents-NIO