The Vision Behind WSO2 User Engagement Server

banner2

“A picture worth a thousand words” is a known fact which doesn’t need proof. At present we discuss about the value and the power of data. We use various tools to analyze them, we do business intelligence capturing trends or anomalies. However in the end you need to express the meaning in a sensible manner for the key decision makers. Would you go write thousand words or rather draw something, summarize the findings and let the stakeholder explore for more.

WSO2 Middleware stack has all the pieces you need to harvest, store & analyze (both real-time and periodical) vast amount of data sets, but when it comes to summarizing and presenting, the stack had somewhat a week tool set. We envisioned the idea of “SOA Last Mile” since latter part of 2009 with products such as WSO2 Gadget Server, WSO2 Mashup Server and frameworks such as viskit. We experimented, wrote code and delivered enterprise grade products to achieve this vision.  For instance WSO2 Gadget Server was one of the most downloaded products during 2010 – 2011 time span, and even now people are using it for various data presentation purposes.

Continue reading The Vision Behind WSO2 User Engagement Server

Reading an XML file into WSO2 ESB; Transform it and expose it as an API

Recently was working on a project where I had a to read an XML file from an FTP location, transform it and expose it as an API. Used WSO2 ESB 4.6.0 for this usecase; and I thought of documenting it for later reference. So here it goes

First the proxy that read the file from FTP and dump it to a defined location, (VFSProxy.xml)

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse" name="VFSProxy" transports="vfs" startOnLoad="true" trace="disable">
<target>
<inSequence>
<log level="custom">
<property name="STATUS" value="File received"/>
</log>
<property name="OUT_ONLY" value="true"/>
<send>
<endpoint>
<address uri="vfs:file:///home/nuwanbando/temp/files/out"/>
</endpoint>
</send>
</inSequence>
</target>
<parameter name="transport.PollInterval">10</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.FileURI">vfs:ftp://<ftpserver_url>/home/nuwanbando/temp/files/in?vfs.passive=true</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///home/nuwanbando/temp/files/processed</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///home/nuwanbando/temp/files/failed</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.xml</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
</proxy>

view raw
VFSProxy.xml
hosted with ❤ by GitHub

This proxy will dump the file to "home/nuwanbando/temp/files/out" location.

This file need to be read on-demand, once requested. ESB by default does not have a mediator to read XML files so that the mediation flow can manipulate the content, Continue reading Reading an XML file into WSO2 ESB; Transform it and expose it as an API

Discovering Admin Services in a WSO2 (Carbon) Server

This should be a quick how-to type guide; This method will work in any Carbon Based WSO2 Server (AS, ESB, BPS etc). I am taking WSO2 ESB as the example.

      1. Download the ESB and extract it. (lets say I extracted to {ESB_HOME})
      2. You will want to see the service contracts once you discover them so navigate to
        {ESB_HOME}/repository/conf

        and open carbon.xml

      3. make, HideAdminServiceWSDLs to false
        <HideAdminServiceWSDLs>false</HideAdminServiceWSDLs>

        and save the document

      4. Start the ESB with OSGI Console
        sh {ESB_HOME}/bin/wso2server.sh -DosgiConsole
      5. Once the server is started, push few return keys (hit enter) to get the OSGI shell in the console
      6. In the OSGI shell type
        osgi> listAdminServices
      7. That will give a list of services as shown below,
      1. All right now you want to see the service contract, lets select “Authentication Admin” Service which is listed as
AuthenticationAdmin, AuthenticationAdmin, https://192.168.219.1:8243/services/AuthenticationAdmin
    1. Take the url from the list and paste it in the browser with ?wsdl at the end, that will give the entire contract of the service

Happy Hacking 🙂

 

{jaggery.js}

Is something I’ve been working on at wso2 for sometime, if you are following me I have tweeted the updates, releases etc. Jaggery went live with its fifth milestone, few weeks ago at jaggeryjs.org, its still young and has alot to improve.

Explaining a little bit about jaggery; Its NOT an alternative to node, its simply a toolkit to easily compose webapps and apis. Its a completely javascript way of writing all parts of the web application, and communicate throughout in json. Jaggery has out of the box features to create RESTfull applications, and its one of it’s value propositions. Jaggery can be easily extended via modules which can be written in javascript or java for that matter.

A very simple page that prints out some request properties in jaggery looks like,

<html>
   <body>
      <p>
      <%
          print("Method : " + request.getMethod() + "<br/>");
          print("Protocol : " + request.getProtocol() + "<br/>");
          print("User-Agent : " + request.getHeader("User-Agent"));
      %>
      </p>
   </body>
</html>

Jaggery docs site also provide a tool for you to try out some code and play around. So you can simply copy the above code and paste in jaggery try-it

I recently did a webinar on jaggery that explains the simple case, over looking its API and samples. Below is the recording hope it’ll help.

“Jaggery” – A web{app/service} completely in Javascript

At WSO2 we are busy with something new for the new year 🙂 and we have just pushed the milestone-1 of it, not so complete but you can sure taste it. Everybody knows that WSO2 is big on Web Services, but over the years we have seen that the barrier between Web Services and Web Applications hasn’t been lowered. “Jaggery” is an effort to diminish this barrier and to merge the development process of services and applications.

By definition it is a framework to write webapps and HTTP-focused web services in pure Javascript for all aspects of the application: front-end, communication, Server-side logic and persistence, and yes as I mentioned one of the intents of the framework is to reduce the gap between writing web apps and web services.

In general web engineers are fond of javascript (and JSON of cause), so why not write the server piece, persistence logic and the communication in-between also in JS itself ?

“Jaggery” helps you to do just that! Right now, the Javascript you compose will be fetched by Rhino engine and execute at the server, At Milestone-1 we have pushed features such as,

  • Compose server side web pages (HTML templates) : Server logic written in javascript (.JSSP)
  • Compose server side scripts purely in javascript (.JSS)
  • Includes support for,
    • HTTP request/response and session
    • JSON send/receive
    • Server side XMLHttpRequest
    • Web Service invocation
    • File I/O
    • Email
    • Atom
    • Read/write feed
    • Relational database
    • WSO2 Registry/Repository
    • Open Social gadgets

A hello world server-side script looks as follows,
https://gist.github.com/1506790.js?file=HelloWorld.jss

and a HTML mixed, server page looks like,
https://gist.github.com/1514681.js?file=gistfile1.txt

For more information and samples you can visit Jaggery M1 Homepage, and stay tuned for more updates soon 🙂

{HowTo} WSO2 Carbon Server logs to be stored in a database

Thought of documenting this for my own reference, hope this will help. I am also planing to make a knowledge base article out of it to be posted in WSO2 Oxygen Tank.

As you may know, if your are familiar with Log4j, that the logs created with it, can be stored in a database instead of a file. In order to do that you need to add few configuration values to the log4j.properties file.

In-terms of a WSO2 Carbon based server this properties file resides in CARBON_HOME/lib directory. before adding these configurations there are few steps to visit.

[1] Create database {LOG_DB}
[2] Create the log table  {LOGGING} with the following fields

create table LOGGING (
             id decimal NOT NULL,
             prio varchar(15),
             cat varchar(255),
             thread varchar(30),
             msg varchar(255),
             layout_msg varchar(255),
             throwable varchar(2000),
             the_timestamp timestamp);

[3] Create a user and assign to the DB

           {user: logger / pwd: logger}

[4] Copy jdbcappender.jar to {CARBON_HOME}/lib and {CARBON_HOME}/repository/components/lib along with the database driver

           {jdbcappender.jar can be found at http://www.dankomannhaupt.de/projects/jdbcappender/lib/jdbcappender.jar}

Thats it. All preconditions are set, Now open up the log4j.properties file and add the following appender settings to the end.

#log to db appender
log4j.appender.DB_APPENDER=org.apache.log4j.jdbcplus.JDBCAppender
log4j.appender.DB_APPENDER.url=jdbc:mysql://localhost:3306/LOG_DB
log4j.appender.DB_APPENDER.dbclass=com.mysql.jdbc.Driver
log4j.appender.DB_APPENDER.username=logger
log4j.appender.DB_APPENDER.password=logger
log4j.appender.DB_APPENDER.sql=INSERT INTO LOGGING (id, prio, cat, thread, msg, layout_msg, throwable, 
the_timestamp) VALUES (@INC@, '@PRIO@', '@CAT@', '@THREAD@', '@MSG@', '@LAYOUT:1@', '@THROWABLE@', '@TIMESTAMP@')
log4j.appender.DB_APPENDER.layout=org.apache.log4j.PatternLayout

Scroll to log4j.rootLogger and add the appender to the list,

log4j.rootLogger=ERROR, CARBON_CONSOLE, CARBON_LOGFILE, CARBON_MEMORY, CARBON_SYS_LOG, DB_APPENDER

Sharing HTTPS, HTTP sessions in tomcat hosted web-apps

The requirement is to only serve the login page securely and once the user is authenticated (s)he should be redirected to non-secure http mode. I was struggling to do this quite some time back, and just thought of documenting about it.

The requirement

The idea I had was; “It should be quite simple”, Facebook does that, Google does that and why is it still not well documented ?, However the almost all Google search results for my queries were about simply redirecting HTTP traffic to HTTPS for certain URLs, some were using URL rewriting (mod_rewrite), and some have used server configuration via Tomcat’s server.xml.

What I really wanted to achieve is to preserve the state between the protocol switch. After some considerable amount of searching I found out this is not achievable (in a very clean manner) with tomcat or rather it is a conflict between security and state management in the servlet spec itself, hence there only exist a dirty hack (not sure if this works) to get it done, but even that hack couldn’t be applied to my scenario.

So after some thinking I came up with my own hack (I think its even dirtier 😉 ) to solve the issue; Its quite simple, and involves cookie manipulation. My approach was simply read the HTTPS cookie and set it as the HTTP cookie, what I need was one jsp which is served with HTTPS and few lines of Java code.

The solution
The solution

True enough it certainly looks like a hack, but security wise its as same as the Tomcat user group has suggested. so until the new servlet specification answers this question we have to live with this. the code of converting the cookies are as follows.

   
    Cookie[] cookies = request.getCookies();
    String sessionId;
    if (cookies != null) {
        for (Cookie c : cookies) {
            if (c.getName().equals("JSESSIONID")) {
                sessionId = c.getValue();
            }
        }
    }

    Cookie k = new Cookie("JSESSIONID", sessionId);
    k.setPath(request.getContextPath());
    response.addCookie(k);

Basically what the code does is, reading the secure cookies while inside the middle.jsp and setting them without security (k.setSecure() is not mentioned hence by default its false), and that’s about it, once this is done you can simply redirect to the HTTP page.

response.sendRedirect("http://foo.com:8080/index.jsp");  

and now the cookie which originally set via HTTPS is accessible to the HTTP requests, hence the session is shared.

Web Scraping & Parsing HTML to XML in Javascript

Today I was working on a customer POC and happened to create few Google gadgets to visualize selected data sets from *.gov.uk sites. The scenario which is implemented was, mixed with inter-gadget communication and content search over data.gov.uk sites. I created three simple gadgets which communicates with each other, and one acted as the controlling gadget which pushed the search parameters to other two gadgets. The two content gadgets showed UK (1) primary school information and (2) electoral information. The pushed parameter was the postal code of different parts of UK. The direct.gov.uk has a form based implementation of this.

The Requirements for the POC was, simple and we already had working samples of such a scenario at WSO2 library.

  1. Show how one gadget can pass the context to other gadgets
  2. How gadgets can harvest data in various formats (in my previous post I explained on how to get data from RDF endpoints, which are also available in *.gov.uk sites)

The building blocks for the implementation was the search url, which was quite straight forward. for all the requests based on postal codes the direct.gov site served in the same manner (because of this important fact, the automation process became trivial). for an instance the url for primary school information retrial was,

http://local.direct.gov.uk/LDGRedirect/LocationSearch.do?LGSL=13&searchtype=1&LGIL=8&Style=&formsub=t&text=SE1+7DU

Where the param “text” changed according to the postal code. So far everything seemed straight forward, however at implementation, while using Gadgets API for content retrial, I faced problems in parsing text with javascript. Hence the gadgets.io.makeRequest supported HTML as text and the API method returned the retrieved HTML document as string making it quite impossible to process.

With some thinking and advise, I brought the Mashup Server in to the picture and used it to retrieve the data from the gov site and returned the result in XML format. Using the Mashup Server web scraping seems to be a piece of cake, We created a simple mashup using the scraper host-object and captured the result set in the search result page. The mashup code as follows,

function search(searchUrl) {
	var scraper = new Scraper(
		
		    {searchUrl}
			
			    
				
				   
				
			     
			
		
	);
	return new XMLList(scraper.response);
}

And finally the two gadgets were making service calls to the mashup service and retrieved the data as an XML object, making the data processing painless. The final version at the Gadget Server looked quite appealing.

WSO2 Gadget Server with UK gov data
Gadget Server look - in the end

Special thanks goes to Ruchira for helping me out with the mashup service 🙂 You can download the Gadget code and the Mashup service and try the scenario yourself.