Integrating WSO2 BAM with WSO2 API Manager for API stat collection

WSO2 API Manager is the only open source and production ready API management middleware you can find at present. Just after a month of the release of AM 1.0.0 (in Aug 2012) it was deployed and launched in one of the biggest companies in US, as their API management platform. You can get more details about WSO2 AM from here.

In API management we basically create APIs and publish them to a store so the registered users can subscribe to those APIs and use them directly or inside their Applications to get some functionality done. Other than that in business and publisher's perspective it is really important to know statistics such as how many subscribers are using API 'X', what number of invocations coming for API 'Y', in which time period a peak load is coming to the API 'Z' etc.

 Yes, WSO2 AM supports that sort of stat collection with the integration of WSO2 Business Activity Monitor (BAM). I'm going to explain here what are the few configuration changes you have to do in order to get WSO2 BAM fitted in this picture, specially when having port offsets.

Pre-requisites

1. Download WSO2 AM binary zip from here and unzip it. Lets call the unzipped directory <AM_HOME>.
2. Download WSO2 BAM binary zip from here and unzip it. Lets call the unzipped directory <BAM_HOME>.
(Note that I am using WSO2 AM 1.3.0 and WSO2 BAM 2.0.1 for this but hopefully the following steps will work for later versions as well)

API Manager configuration

OK!! Lets start configuring the API Manager first. Assuming we are running both AM and BAM servers in the same machine (or VM), first we need to set port offset values otherwise when both servers are starting with default ports in the same machine you will see port conflict errors. (By setting an offset value all the default port allocations will be increased by the given value. I will be giving value 1 for AM and 2 for BAM as offsets)

1. Go to <AM_HOME>/repository/conf/carbon.xml and change the offset value to 1.
2. Go to <AM_HOME>/repository/conf/api-manager.xml and change the configuration as following.

<APIUsageTracking>
    <Enabled>false</Enabled>
    <PublisherClass>org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher</PublisherClass>
    <ThriftPort>7613</ThriftPort>
    <BAMServerURL>tcp://localhost:7613/</BAMServerURL>
    <BAMUsername>admin</BAMUsername>
    <BAMPassword>admin</BAMPassword>
    <DataSourceName>jdbc/WSO2AM_STATS_DB</DataSourceName>
    <GoogleAnalyticsTracking>
       <Enabled>false</Enabled>
       <TrackingID>UA-XXXXXXXX-X</TrackingID>
    </GoogleAnalyticsTracking>
</APIUsageTracking>

Note that AM uses Thrift messaging to communicate usage data to BAM. Since we are going to set the offset in BAM to 2 we need to increase ThriftPort value by 2 (7612 -> 7614). Also we need to uncomment the<DataSourceName> property.

3. Now go to  <AM_HOME>/repository/conf/datasources/master-datasources.xml and edit the 'WSO2AM_STATS_DB' datasource as following. Replace <BAM_HOME> with your absolute path to the BAM installation directory.

<datasource>
            <name>WSO2AM_STATS_DB</name>
            <description>The datasource used for getting statistics to API Manager</description>
            <jndiConfig>
                <name>jdbc/WSO2AM_STATS_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:h2:<BAM_HOME>/repository/database/APIMGTSTATS_DB;AUTO_SERVER=TRUE</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>org.h2.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
         </datasource>

This 'APIMGTSTATS_DB' in BAM is used to store the summerized stat data of AM. AM will fetch data from this database for presentation.


 BAM configuration

1. Go to <BAM_HOME>/repository/conf/carbon.xml and change the offset value to 2.
2. Now go to  <BAM_HOME>/repository/conf/datasources/master-datasources.xml and add the following config for 'WSO2AM_STATS_DB' datasource same as we did in AM. Replace <BAM_HOME> with your absolute path to the BAM installation directory.

<datasource>
            <name>WSO2AM_STATS_DB</name>
            <description>The datasource used for getting statistics to API Manager</description>
            <jndiConfig>
                <name>jdbc/WSO2AM_STATS_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:h2:<BAM_HOME>/repository/database/APIMGTSTATS_DB;AUTO_SERVER=TRUE</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>org.h2.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
         </datasource>

3. Now configuration changes are done. But still BAM will not be collecting data from AM. For that we need to run an Analytics script inside BAM. You have to copy <AM_HOME>/statistics/API_Manager_Analytics.tbox into <BAM_HOME>/repository/deployment/server/bam-toolbox directory.

Configurations are DONE!!!  Now start AM and BAM servers separately.
 

Oh Wait!!! Did you happen to see this error in BAM after server started.

ERROR {org.apache.hadoop.hive.cassandra.CassandraProxyClient} -  Error while trying to connect to cassandra host:127.0.0.1
org.apache.hadoop.hive.cassandra.CassandraException: unable to connect to server

Let's solve that as well. As I said earlier we need an analytics script to be run in BAM to collect data from AM. That script is written with the  assumption that BAM will be run with the port offset 1. Therefore in the script it has mentioned the cassandra port as 9161 (cassandra default port 9160 + 1). Since we increased the BAM offset to 2 we need to change the cassandra port to 9162. For that you need to go to BAM management console (https://localhost:9445/carbon/) and login with admin credentials. And then from the left side menu go to Main -> Manage -> Analytics -> List page. Now go to Edit page of am_stats_analyzer_451 script. Do a search for all 9161 and replace it with 9162 and save it. You won't see that error anymore.

Now you will be able to see the API Usage statistics through the API Publisher in WSO2 AM. See http://docs.wso2.org/wiki/display/AM120/Viewing+API+Statistics for more details on viewing statistics.