[WSO2 ESB] Using Secure vault to store passwords securely in your synapse configuration

When you are using WSO2 ESB to do some mediation there come the times that you have to send user credentials to the backend services for authentication and authorization purposes. In that case, you have to store the usernames and passwords in your synapse configuration. One problem of that approach is that other people get to see the credentials which is not very secured.

To overcome that issue you can use the secure vault feature in ESB. What happens there is, first you create a registry entry for your password with an alias for the password. In the registry the password is NOT stored in plain text, but encrypted using the certificate in wso2carbon.jks file. This entry is stored as a property under the location /_system/config/repository/components/secure-vault in the registry. Then in your synapse configuration you use the alias instead of the password, and use a custom xpath expression to lookup for the password from the registry and decode it before sending it to the endpoint.

First of all you need to configure the ciphering part. For that you need configure the 'repository/conf/security/secret-conf.properties' file and add the following.

keystore.identity.location=<ESB_HOME_Absolute_path>/repository/resources/security/wso2carbon.jks
keystore.identity.type=JKS
keystore.identity.store.password=identity.store.password
keystore.identity.store.secretProvider=org.wso2.carbon.securevault.DefaultSecretCallbackHandler
secretRepositories.file.provider=org.wso2.securevault.secret.repository.FileBaseSecretRepositoryProvider
secretRepositories.file.location=repository/conf/security/cipher-text.properties
secretRepositories=file
keystore.identity.key.password=identity.key.password
carbon.secretProvider=org.wso2.securevault.secret.handler.SecretManagerSecretCallbackHandler
keystore.identity.key.secretProvider=org.wso2.carbon.securevault.DefaultSecretCallbackHandler
keystore.identity.alias=wso2carbon

Restart the ESB server if you have already started it. Log in, and go to the "Manage Passwords" link under the "Secure Vault Tool" entry in the left hand menu. In that page you can create a new password (or username) entry in the registry by providing a key(alias) and the actual password. Then your password will be stored in the registry encrypted with the key in wso2carbon.jks .

Now setting up the alias is done, let's see how to retrieve it in your synapse configuration. You can use the custom xpath expression called "wso2:vault-lookup('alias')" to retrieve the decrypted password mapping to the alias. See the example given below.

<log>
<property name="Sec_Password" expression="wso2:vault-lookup('pass')"/>
</log>

[WSO2 ESB] How to retrieve details of the API at runtime from synapse configuration

I have come across that many users of WSO2 ESB are using it for creating and deploying APIs and they have this requirement to retrieve the details of the API which is being called currently at runtime, from somewhere out side the API (in example from a different sequence outside the API).

Following are some synapse level properties that you can use to get some details of the REST API that is being called at runtime, and how to show those details using the log mediator.

<property name="apiname" expression="$ctx:SYNAPSE_REST_API"></property>
<property name="contxt" expression="$ctx:REST_API_CONTEXT"></property>
<property name="urlpre" expression="$ctx:REST_URL_PREFIX"></property>
<property name="restmethod" expression="$ctx:REST_METHOD"></property>

                    
<log>
   <property name="*******************URL_PREFIX***********" expression="get-property('urlpre')"></property>
</log>
<log>
   <property name="*******************API_NAME***********" expression="get-property('apiname')"></property>
</log>
<log>
   <property name="*******************API_CONTEXT***********" expression="fn:substring-after(get-property('contxt'),'/')"></property>
</log>
<log>
   <property name="*******************REST_METHOD***********" expression="get-property('restmethod')"></property>
</log>

[WSO2 API manager] Converting API GET requests to send a POST request to the backend

Some users of WSO2 API manager has this requirement of changing the HTTP method of the incoming request and send it to the backend with a different HTTP method. As an example some users want to change a GET request coming to the API extract some values from the query parameters and send it as a POST request after generating a payload which is required to the backend. Let's see how we can achieve this with WSO2 API Manager, for this blog post I'm using the latest API manager 1.9.1 version but this would work for earlier versions as well.

First of all let's get the requirement clarified first. 

I have a JAXRS service hosted in WSO2 Appserver which I want to expose through API Manager. In my service I have to send a POST request to the URL,

http://localhost:9763/jaxrs_basic/services/customers/customerservice

with a xml payload similar to below to create a new customer in the system.

<Customer>
   <name>Jack</name>
</Customer>

My real requirement is to expose this POST operation through an API such that, when the client sends a GET request to the API with name as a query parameter, it will generate the necessary payload and send it as a POST request to the back end URL above.

So Let's see how to do it.

First of all you need to create an API in APIManager publisher app and publish it. Make sure you add a resource similar to /sample/{name} with a uri-template in the API allowing GET method. And after that go to API Store log in and subscribe to the API you created. If you need help in this initial steps refer the API manager documentation [1] for further instructions.

Now, we have done the initial part, we have an API created with a GET resource. Let's see what are the modifications that we need to do for this API to convert this GET request to a POST request to be sent to the back end.

Find the API configuration xml file for your newly created API in <APIM_HOME>/repository/deployment/server/synapse-configs/default/api/ directory, and open it using a text editor.

Find the resource definition similar to below.

<resource methods="GET" uri-template="/sample/{name}" faultSequence="fault">

Now let's change few things inside the <inSequence> of this resource definition, and let me explain why we are changing them in that manner.

<property name="HTTP_METHOD" value="POST" scope="axis2" type="STRING"/>

Add the above to change the HTTP method of the outbound request to POST method.

<property name="messageType" value="text/xml" scope="axis2"/>

This will set the message type of the outbound payload to text/xml content type.

    <payloadFactory media-type="xml">
                  <format>
                      <Customer xmlns="">
                        <name>$1</name>
                      </Customer>
                  </format>
                  <args>
                    <arg expression="get-property('uri.var.name')"/>
                  </args>
    </payloadFactory>

Then you need to create the xml payload you need with the provided customer name input, that was passed as a query parameter in the GET request. You can use the payloadFactory mediator similar to above configuration to build the xml payload.

<property name="REST_URL_POSTFIX" value="" scope="axis2"/>

Finally you need to add this property in order to prevent the resource path that you send to the API GET resource being appended to the service endpoint URL. Note the empty value set in this property.

Ok that's it!!! Now it would change your API request like below to send to the backend service.

Request to API:

GET : http://hostname:port/apicontext/1.0.0/sample/john 


Request sent to the backend service :

POST : http://localhost:9763/jaxrs_basic/services/customers/customerservice

<Customer>
   <name>john</name>
</Customer>

The full synapse configuration for the GET resource is below.

<resource methods="GET" uri-template="/sample/{name}" faultSequence="fault">
      <inSequence>
         <filter source="$ctx:AM_KEY_TYPE" regex="PRODUCTION">
            <then>
               <property name="api.ut.backendRequestTime"
                         expression="get-property('SYSTEM_TIME')"/>
               <property name="HTTP_METHOD" value="POST" scope="axis2" type="STRING"/>
               <property name="messageType" value="text/xml" scope="axis2"/>
               <payloadFactory media-type="xml">
                  <format>
                      <Customer xmlns="">
                        <name>$1</name>
                      </Customer>
                  </format>
                  <args>
                    <arg expression="get-property('uri.var.name')"/>
                  </args>
               </payloadFactory>
               <property name="REST_URL_POSTFIX" value="" scope="axis2"/>
               <send>
                  <endpoint name="admin--JAXRSbasic_APIproductionEndpoint_0">
                     <http uri-template="http://localhost:9763/jaxrs_basic/services/customers/customerservice"/>
                  </endpoint>
               </send>
            </then>
            <else>
               <sequence key="_sandbox_key_error_"/>
            </else>
         </filter>
      </inSequence>
      <outSequence>
         <class name="org.wso2.carbon.apimgt.usage.publisher.APIMgtResponseHandler"/>
         <send/>
      </outSequence>
   </resource>

References

[1] https://docs.wso2.com/display/AM191/Quick+Start+Guide#QuickStartGuide-CreatinganAPIfromscratch

Integrating WSO2 BAM with WSO2 API Manager for API stat collection

WSO2 API Manager is the only open source and production ready API management middleware you can find at present. Just after a month of the release of AM 1.0.0 (in Aug 2012) it was deployed and launched in one of the biggest companies in US, as their API management platform. You can get more details about WSO2 AM from here.

In API management we basically create APIs and publish them to a store so the registered users can subscribe to those APIs and use them directly or inside their Applications to get some functionality done. Other than that in business and publisher's perspective it is really important to know statistics such as how many subscribers are using API 'X', what number of invocations coming for API 'Y', in which time period a peak load is coming to the API 'Z' etc.

 Yes, WSO2 AM supports that sort of stat collection with the integration of WSO2 Business Activity Monitor (BAM). I'm going to explain here what are the few configuration changes you have to do in order to get WSO2 BAM fitted in this picture, specially when having port offsets.

Pre-requisites

1. Download WSO2 AM binary zip from here and unzip it. Lets call the unzipped directory <AM_HOME>.
2. Download WSO2 BAM binary zip from here and unzip it. Lets call the unzipped directory <BAM_HOME>.
(Note that I am using WSO2 AM 1.3.0 and WSO2 BAM 2.0.1 for this but hopefully the following steps will work for later versions as well)

API Manager configuration

OK!! Lets start configuring the API Manager first. Assuming we are running both AM and BAM servers in the same machine (or VM), first we need to set port offset values otherwise when both servers are starting with default ports in the same machine you will see port conflict errors. (By setting an offset value all the default port allocations will be increased by the given value. I will be giving value 1 for AM and 2 for BAM as offsets)

1. Go to <AM_HOME>/repository/conf/carbon.xml and change the offset value to 1.
2. Go to <AM_HOME>/repository/conf/api-manager.xml and change the configuration as following.

<APIUsageTracking>
    <Enabled>false</Enabled>
    <PublisherClass>org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher</PublisherClass>
    <ThriftPort>7613</ThriftPort>
    <BAMServerURL>tcp://localhost:7613/</BAMServerURL>
    <BAMUsername>admin</BAMUsername>
    <BAMPassword>admin</BAMPassword>
    <DataSourceName>jdbc/WSO2AM_STATS_DB</DataSourceName>
    <GoogleAnalyticsTracking>
       <Enabled>false</Enabled>
       <TrackingID>UA-XXXXXXXX-X</TrackingID>
    </GoogleAnalyticsTracking>
</APIUsageTracking>

Note that AM uses Thrift messaging to communicate usage data to BAM. Since we are going to set the offset in BAM to 2 we need to increase ThriftPort value by 2 (7612 -> 7614). Also we need to uncomment the<DataSourceName> property.

3. Now go to  <AM_HOME>/repository/conf/datasources/master-datasources.xml and edit the 'WSO2AM_STATS_DB' datasource as following. Replace <BAM_HOME> with your absolute path to the BAM installation directory.

<datasource>
            <name>WSO2AM_STATS_DB</name>
            <description>The datasource used for getting statistics to API Manager</description>
            <jndiConfig>
                <name>jdbc/WSO2AM_STATS_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:h2:<BAM_HOME>/repository/database/APIMGTSTATS_DB;AUTO_SERVER=TRUE</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>org.h2.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
         </datasource>

This 'APIMGTSTATS_DB' in BAM is used to store the summerized stat data of AM. AM will fetch data from this database for presentation.


 BAM configuration

1. Go to <BAM_HOME>/repository/conf/carbon.xml and change the offset value to 2.
2. Now go to  <BAM_HOME>/repository/conf/datasources/master-datasources.xml and add the following config for 'WSO2AM_STATS_DB' datasource same as we did in AM. Replace <BAM_HOME> with your absolute path to the BAM installation directory.

<datasource>
            <name>WSO2AM_STATS_DB</name>
            <description>The datasource used for getting statistics to API Manager</description>
            <jndiConfig>
                <name>jdbc/WSO2AM_STATS_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:h2:<BAM_HOME>/repository/database/APIMGTSTATS_DB;AUTO_SERVER=TRUE</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>org.h2.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
         </datasource>

3. Now configuration changes are done. But still BAM will not be collecting data from AM. For that we need to run an Analytics script inside BAM. You have to copy <AM_HOME>/statistics/API_Manager_Analytics.tbox into <BAM_HOME>/repository/deployment/server/bam-toolbox directory.

Configurations are DONE!!!  Now start AM and BAM servers separately.
 

Oh Wait!!! Did you happen to see this error in BAM after server started.

ERROR {org.apache.hadoop.hive.cassandra.CassandraProxyClient} -  Error while trying to connect to cassandra host:127.0.0.1
org.apache.hadoop.hive.cassandra.CassandraException: unable to connect to server

Let's solve that as well. As I said earlier we need an analytics script to be run in BAM to collect data from AM. That script is written with the  assumption that BAM will be run with the port offset 1. Therefore in the script it has mentioned the cassandra port as 9161 (cassandra default port 9160 + 1). Since we increased the BAM offset to 2 we need to change the cassandra port to 9162. For that you need to go to BAM management console (https://localhost:9445/carbon/) and login with admin credentials. And then from the left side menu go to Main -> Manage -> Analytics -> List page. Now go to Edit page of am_stats_analyzer_451 script. Do a search for all 9161 and replace it with 9162 and save it. You won't see that error anymore.

Now you will be able to see the API Usage statistics through the API Publisher in WSO2 AM. See http://docs.wso2.org/wiki/display/AM120/Viewing+API+Statistics for more details on viewing statistics.
 

[Solved] ORA-12519, TNS:no appropriate service handler found

You might have come across this Oracle error while working with you database app connected to a Oracle 10g XE database.

ORA-12519, TNS:no appropriate service handler found

java.sql.SQLException: Listener refused the connection with the following error:

ORA-12519, TNS:no appropriate service handler found

at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:113)

at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)

at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:389)

at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:454)

at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)

at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)

at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:802)

 

 

This is how to get rid of it. Open the sqlplus command line and set the processes and scope values as below. (and restart Oracle)

 

ALTER SYSTEM SET PROCESSES=150 SCOPE=SPFILE;

That's it!!!!

How to run Jmeter tests in non-GUI mode, automated with a shell script

Apache Jmeter is a great versatile tool for performance testing of web servers, DBMS servers etc. It's a pure Java application where you can use even your own Java code to customize and randomize your tests. My intention of this post is not to tell you about all of those features, but one such cool feature of Jmeter which is running Jmeter tests in non-GUI mode.

If you have worked with Jmeter you surely should have experienced that sometimes it get stuck or crashes while you are doing high concurrency (> 300) load testing. One solution for this is to create your test using the GUI and save it as a '.jmx' file and later run that in non-GUI mode and save the test results to a '.jtl' file. In that way you will be able to carry out your load test without crashing or maybe with more concurrency.

OKEY..... This is how to do it.

1. First prepare your test plan with Jmeter and save it, let's say the file name is 'mytest.jmx'. (It gets saved as a xml configuration file with the .jmx extention).

2. Then go to Apache JMETER_HOME/bin folder using command shell. And then run Jmeter with the following parameters, to execute the test.

./jmeter -n -t \absolute_path\mytest.jmx -l \absolute_path\mytest_results.jtl

3. While the test is running the results will be saved in the 'mytest_results.jtl'. When your test is complete you can open this jtl file in Jmeter and summarize the results or create graphs using it.
To do that open Jmeter and add a Summary Report to the test plan (Add --> Listener --> Summary Report). And then go to the Summary Report UI and 'Browse' and open your mytest_results.jtl file.


OK.....next comes the real headache :-). Let's say you have to repeat the above test several times with small changes to the test. I actually had a requirement at my work to repeat the same Jmeter test with different concurrency levels and number of loops.
The hard way to do this is create separate test scripts for different concurrency levels using the GUI and run them separately as above.
A less hard way to do this is open the test file (.jmx) using a text editor and change the values every time you repeat the test, in the ThreadGroup element in the test configuration(as below), and run the same file.

<ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <stringProp name="LoopController.loops">1000</stringProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">300</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <longProp name="ThreadGroup.start_time">1339414812000</longProp>
        <longProp name="ThreadGroup.end_time">1339414812000</longProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
      </ThreadGroup>

OK.... But why not automate the above 'less hard way' and make your life easy..
If you are a shell scripting master this is a "few lines" work, but since I'm not, this is how I did it.

#!/bin/bash

i=0

conc[0]=50
conc[1]=100
conc[2]=250
conc[3]=500
conc[4]=750
conc[5]=1000
conc[6]=1250

loop[0]=5000
loop[1]=2500
loop[2]=1000
loop[3]=500
loop[4]=334
loop[5]=250
loop[6]=200

cd ./jmx
while test $i != 7
do
sed 's/_loop_/'${loop[$i]}'/g' mytest.jmx > ./jmtest.jmx
sed 's/_con_/'${conc[$i]}'/g' jmtest.jmx > jmtest_${conc[$i]}.jmx
rm jmtest.jmx
cd ../apache-jmeter-2.6/bin/
./jmeter -n -t ../../jmx/jmtest_${conc[$i]}.jmx -l ../../jtl/results_${conc[$i]}.jtl
i=`expr $i + 1`
cd ../../jmx
done

For newbies like me, I will explain the script briefly.

For this script to work first you should do a small change to your Jmeter test configuration. You have to give a variable string for num_threads and loop instead of actual values, as below.

<ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <stringProp name="LoopController.loops">_loop_</stringProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">_con_</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <longProp name="ThreadGroup.start_time">1339414812000</longProp>
        <longProp name="ThreadGroup.end_time">1339414812000</longProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
      </ThreadGroup>

In the shell script, first I put the concurrency values and loop values in to 2 different arrays.
And then using a while loop I edit the concurrency and loop values in Jmeter test script for each time I have to repeat the test. I have used sed command for that. It searches for the _loop_ and _con_ strings in the Jmeter script and will replace it with the corresponding values in above two arrays.
Then inside the loop, changed Jmeter script is executed (for the number of times the while loop is run) in non GUI mode and results are saved in different jtl files.

How to install Oracle XE 10g on Ubuntu 10.04 64bit

Once I had a hard time installing Oracle XE -10g on Ubuntu 64bit machine. Since there is no 64bit version of Oracle-XE 10g we have to add some extra lib's for Oracle to work. Following are the steps to install.

1) Download oracle-xe-universal_10.2.0.1-1.1_i386.deb from here.

2) Download libaio_0.3.104-1_i386.deb lib package from here.

3) In command line terminal go to the directory where above 2 files are downloaded.

4) Type : sudo apt-get install bc

5) Then force to install the library : dpkg -i --force-architecture libaio_0.3.104-1_i386.deb 

6) Force to install Oracle XE : dpkg -i --force-architecture oracle-xe-universal_10.2.0.1-1.1_i386.deb

7) After the installation is complete you need to configure the setup :  sudo /etc/init.d/oracle-xe configure

    -- While configuring you will be asked to assign the port numbers (default port numbers are prompted you can give the same).

    -- You will be asked to set a password for SYS and SYSTEM. Never ever give the password as "admin" you will not be able to log in after oracle is started.

       (It was the lesson I learned after struggling couple of hours, interesting that oracle doesn't support the password 'admin' :-) ). 

       Remember to give a password with more than 8 characters.

8) After configuration is complete you have to set environment variables for Oracle.

    -- Open your bashrc : sudo gedit /etc/bash.bashrc

    -- And add the following variables and save the file :

        ORACLE_HOME=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server

    PATH=$PATH:$ORACLE_HOME/bin
    export ORACLE_HOME
    export ORACLE_SID=XE    export PATH

9) It's done!! Now, to start the server : sudo /etc/init.d/oracle-xe start  (you can use start, restart, stop arguments also)

10) To start with 'sqlplus' enter the following in command line: sqlplus sys as sysdba

     Or you can go to the admin console from the  browser: http://127.0.0.1:8080/apex

--------

11) If you want to completely remove oracle : sudo dpkg -P oracle-xe-universal