Reading an XML file into WSO2 ESB; Transform it and expose it as an API

Recently was working on a project where I had a to read an XML file from an FTP location, transform it and expose it as an API. Used WSO2 ESB 4.6.0 for this usecase; and I thought of documenting it for later reference. So here it goes

First the proxy that read the file from FTP and dump it to a defined location, (VFSProxy.xml)

This proxy will dump the file to "home/nuwanbando/temp/files/out" location.

This file need to be read on-demand, once requested. ESB by default does not have a mediator to read XML files so that the mediation flow can manipulate the content, Continue reading Reading an XML file into WSO2 ESB; Transform it and expose it as an API

“Jaggery” – A web{app/service} completely in Javascript

At WSO2 we are busy with something new for the new year 🙂 and we have just pushed the milestone-1 of it, not so complete but you can sure taste it. Everybody knows that WSO2 is big on Web Services, but over the years we have seen that the barrier between Web Services and Web Applications hasn’t been lowered. “Jaggery” is an effort to diminish this barrier and to merge the development process of services and applications.

By definition it is a framework to write webapps and HTTP-focused web services in pure Javascript for all aspects of the application: front-end, communication, Server-side logic and persistence, and yes as I mentioned one of the intents of the framework is to reduce the gap between writing web apps and web services.

In general web engineers are fond of javascript (and JSON of cause), so why not write the server piece, persistence logic and the communication in-between also in JS itself ?

“Jaggery” helps you to do just that! Right now, the Javascript you compose will be fetched by Rhino engine and execute at the server, At Milestone-1 we have pushed features such as,

  • Compose server side web pages (HTML templates) : Server logic written in javascript (.JSSP)
  • Compose server side scripts purely in javascript (.JSS)
  • Includes support for,
    • HTTP request/response and session
    • JSON send/receive
    • Server side XMLHttpRequest
    • Web Service invocation
    • File I/O
    • Email
    • Atom
    • Read/write feed
    • Relational database
    • WSO2 Registry/Repository
    • Open Social gadgets

A hello world server-side script looks as follows,
https://gist.github.com/1506790.js?file=HelloWorld.jss

and a HTML mixed, server page looks like,
https://gist.github.com/1514681.js?file=gistfile1.txt

For more information and samples you can visit Jaggery M1 Homepage, and stay tuned for more updates soon 🙂

Sharing HTTPS, HTTP sessions in tomcat hosted web-apps

The requirement is to only serve the login page securely and once the user is authenticated (s)he should be redirected to non-secure http mode. I was struggling to do this quite some time back, and just thought of documenting about it.

The requirement

The idea I had was; “It should be quite simple”, Facebook does that, Google does that and why is it still not well documented ?, However the almost all Google search results for my queries were about simply redirecting HTTP traffic to HTTPS for certain URLs, some were using URL rewriting (mod_rewrite), and some have used server configuration via Tomcat’s server.xml.

What I really wanted to achieve is to preserve the state between the protocol switch. After some considerable amount of searching I found out this is not achievable (in a very clean manner) with tomcat or rather it is a conflict between security and state management in the servlet spec itself, hence there only exist a dirty hack (not sure if this works) to get it done, but even that hack couldn’t be applied to my scenario.

So after some thinking I came up with my own hack (I think its even dirtier 😉 ) to solve the issue; Its quite simple, and involves cookie manipulation. My approach was simply read the HTTPS cookie and set it as the HTTP cookie, what I need was one jsp which is served with HTTPS and few lines of Java code.

The solution
The solution

True enough it certainly looks like a hack, but security wise its as same as the Tomcat user group has suggested. so until the new servlet specification answers this question we have to live with this. the code of converting the cookies are as follows.

   
    Cookie[] cookies = request.getCookies();
    String sessionId;
    if (cookies != null) {
        for (Cookie c : cookies) {
            if (c.getName().equals("JSESSIONID")) {
                sessionId = c.getValue();
            }
        }
    }

    Cookie k = new Cookie("JSESSIONID", sessionId);
    k.setPath(request.getContextPath());
    response.addCookie(k);

Basically what the code does is, reading the secure cookies while inside the middle.jsp and setting them without security (k.setSecure() is not mentioned hence by default its false), and that’s about it, once this is done you can simply redirect to the HTTP page.

response.sendRedirect("http://foo.com:8080/index.jsp");  

and now the cookie which originally set via HTTPS is accessible to the HTTP requests, hence the session is shared.

Mashing up RDF data with WSO2 Mashup Server

Okey so this is the fun part that I promised to write about :D. I managed to cook up a use-case to demonstrate RDF querying and making use of the semantic data. The data that I am using for querying, is the rdf data sources available in the UK data.gov site. With some analysis I figured out that this task can be fundamentally archived using the combination of Mashup and Gadget Technologies. My choice of tools were WSO2 Mashup Server and WSO2 Gadget Server for their great flexibility and of cause for other obvious reasons :D. However the Mashup Server does not natively support RDF data retrieval, hence I had to do some work to get such functionality integrated. The great fact about the mashup server is its extensibility, the concept of host objects and the ability to write custom host objects and its pluggable nature comes handy in such cases. The high level architecture of what I am trying to achieve is as follows.

RDF data retrival with WSO2 Mashup server / WSO2 Gadget Server

To implement the above architecture with the tools at hand I created a custom host object that can be plugged to the Mashup Server. When dealing with semantic web related tasks and RDF data handling HP’s Jena java library comes in handy. With the use of Jena-ARQ (for SPARQL) api I managed to get the host object working with few lines of code.

.....
            Dataset dataSet = DatasetFactory.create(sparqlObject.rdfDataSource);
            // Create a new query form a given user query
            String queryString = sparqlObject.spaqrlQuery;
            Query query = QueryFactory.create(queryString);
            QueryExecution qe = QueryExecutionFactory.create(query, dataSet);
            ResultSet results = qe.execSelect();
.....
           resultString = ResultSetFormatter.asXMLString(results);
..... OR.....
           ByteArrayOutputStream bos = new ByteArrayOutputStream();
           ResultSetFormatter.outputAsJSON(bos, results);

With the host object in place, the next task was to create a Mashup in-order to query the rdf data with a given source (EndPoint or data source). The javascript service (Mashup) is created to serve this purpose, where the consumer can specify the RDF endpoint or the data source with the SPARQL query and retrieve the dataset in XML or JSON.

.....
function RdfDocQueryService(rdfDataSource, rdfQuery, resultType) {
   var sparqlObj = new SparqlHostObject();
   sparqlObj.rdfDataSource = rdfDataSource;
   sparqlObj.spaqrlQuery = rdfQuery;
   sparqlObj.resultType = resultType;
   return new XML(sparqlObj.getDataFromRdfSource());
}

Finally to bind everything together, lets try querying some data. My example usecase is to use the query at N2 blog to retrieve traffic monitoring points in UK roads. The query to retrieve the data set as follows,

#List the uri, latitude and longitude for road traffic monitoring points on the M5
PREFIX road:
PREFIX rdf:
PREFIX geo:
PREFIX wgs84:
PREFIX xsd:
SELECT ?point ?lat ?long WHERE {
  ?x a road:Road.
  ?x road:number "A4"^^xsd:NCName.
  ?x geo:point ?point.
  ?point wgs84:lat ?lat.
  ?point wgs84:long ?long.
}

To visualize these points I have created a gadget with the aid of Google Maps api. This gadget can be hosted in the Gadget Server, where it can dynamically retrieve traffic monitoring points for each road in the UK and display them in the map as follows.

Traffic points in A4 road, UK

Convert from HTML to XML with HTML Tidy

For few days I was involved with WSO2 Mashup Server 2.0 release documentation, giving a hand to the mashup team. Documentation is a painful task, but when comes to open source what matters mostly is documentation :D.
Last night I had to convert a bunch of html files (some Java Api Docs) to xml in-order to port into maven site. Formatting 30+ html files to xml !@#$%^&*@% :D. So I was googleing for a tool to automate the task. With few clicks here and there I found a nice article in Big Blue‘s developer works site, a tool called “Tidy“. When I tried to download and use I figure out that you can straight away apt-get the package and use. So,

sudo apt-get install tidy

and your box is now equiped with the tool, and can be accessed via the shell.

tidy -asxhtml -numeric  index.xml

but who wants to convert file by file when you have such a nice tool, so I spent few minutes in writing a tiny shell script to get the job done, the snippet is,

#!/bin/bash
for file in $(find $1 -type f -iname '*.html'); do
	myf=`echo $file | sed 's/html/xml/g'`
	tidy -asxhtml -numeric  $myf
done

All looked good, worked fine. However in my Api Docs I had, had few special tags, custom to our Mashup Apis (<imconfig>, <yahoo>, <mail:config>). Tidy gave error for these files since the tags are not recognized.

In such a case you can train Tidy for new tags, by adding few lines to the tidy configuration file. (/etc/tidy.config – You can also give your own config file at the prompt)

new-pre-tags: imconfig, yahoo, msn, aim, icq, jabber, username, password

There are whole bunch of tweeks you can do with tidy, [1], [2] and [3] are some useful links that you can read up when using the tool.

[1] : http://www.ibm.com/developerworks/library/x-tiptidy.html
[2] : http://tidy.sourceforge.net/
[3] : http://tidy.sourceforge.net/docs/tidy_man.html

Writing custom queries to retrieve data from WSO2 Governance Registry

WSO2 Governance Registry is a big part of wso2 governance product stack. Even though it is primarily aimed at managing, versioning, rating, and commenting on SOA artifacts it can also be used as a simple data store. with the 3.0 version the G-Reg gave support to custom query execution from the client side. This feature helps immensely when you use the registry for non-standard tasks. For me I had to do some pagination work for the comments that belongs to a particular resource, hence my approach was to write few custom quires to get the job done. The code is as follows.

	/**
	 * Returns a chunk of comments
	 * 
	 * @param resPath	Path to the comment
	 * @param start		The beginning index
	 * @param size		Size of the chunk
	 * @return			an array of comments
	 */
 public Comment[] getCommentSet(String resPath, int start, int size) {
		Registry registry = null;
		try {
			registry = ; // get an instance of the registry 

			Resource comQuery = registry.newResource();

                        // The Sql Statement
			String sql = "SELECT REG_COMMENT_ID FROM REG_RESOURCE_COMMENT RC, REG_RESOURCE R, REG_PATH P WHERE "
					+ "RC.REG_VERSION=R.REG_VERSION AND "
					+ "R.REG_NAME=? AND "
					+ "P.REG_PATH_VALUE=? AND "
					+ "P.REG_PATH_ID=R.REG_PATH_ID LIMIT ?, ?";

			// Set SQL statement as the resource content
                        comQuery.setContent(sql);

                       // Setting the media type and properties
			comQuery.setMediaType(RegistryConstants.SQL_QUERY_MEDIA_TYPE);
			comQuery.addProperty(RegistryConstants.RESULT_TYPE_PROPERTY_NAME, RegistryConstants.COMMENTS_RESULT_TYPE);

			registry.put("system/myQueries/query", comQuery);
                 String resourceName = "testResource";
                 String pathToResource = "/system/myResources"

			Map params = new HashMap();

                        //Setting the parameters
			params.put("1", resourceName);
			params.put("2", pathToResource);
			params.put("3", start);
			params.put("4", size);

                       // Executing the SQL statement
			Collection qResults = registry.executeQuery("system/myQueries/query", params);

			String[] qPaths = (String[]) qResults.getContent();

			Comment[] comments = new Comment[qPaths.length];
                        // Loading the comment data to comment object array 
			for (int i = 0; i < qPaths.length; i++) {
				if (registry.resourceExists(qPaths[i])) {
					comments[i] = (Comment) registry.get(qPaths[i]);
				}
			}

			return comments;

		} catch (Exception e) {
			String errorMsg = "Backend server error - could not get comment set";
			log.error(new MyTestException(errorMsg, e));
			return null;
		}

	}

Yeah simple as that you get your resources set without much effort. A big thank goes to Dimuthu

YUI file upload with jsp backend

For last two weeks I was working on some user interface logic and happened to use Yahoo UI library (YUI). The task was to upload an image using Ajax. Since I was new to YUI, I was looking here and there over the net for some references. There were some good ones but thats for PHP back-ends, but mine was a jsp back-end and i didn’t know how to read the object thrown out from the YUI side.

with some more digging I came across nice file handling library in Apache commons (Commons File Upload) and took use of it to do the task. the code is as follows.



YUI File UploadS
http://[PATH_TO_YUI]/yahoo-dom-event/yahoo-dom-event.js
http://[PATH_TO_YUI]/connection/connection-min.js

function init(){
  var onUploadButtonClick = function(e){
    //the second argument of setForm is crucial,
    //which tells Connection Manager this is a file upload form
    YAHOO.util.Connect.setForm('testForm', true);

    var uploadHandler = {
      upload: function(o) {
        alert(o.responseText);
      }
    };
  YAHOO.util.Connect.asyncRequest('POST', 'upload.php', uploadHandler);
  };
  YAHOO.util.Event.on('uploadButton', 'click', onUploadButtonClick);
}

YAHOO.util.Event.on(window, 'load', init);














I took the above code segment directly from a YUI file upload tutorial hence the credit goes to the author. The jsp back-end using apache commons file upload is as follows.
Continue reading YUI file upload with jsp backend