Week 5 & 6

I'm now at my 6th week at Docbyte for my internship. My next task was to check for performance errors and potential bottlenecks in EPA. To avoid premature optimization I've used Visual VM. It's a monitoring tool for Java applications. You don't have to add anything to your code to use it. Visual VM automatically checks all running Java application on your device and also has the possibility to monitor Java-apps on remote devices.
After running a version of EPA locally, I've started to monitor the CPU usage at between several points in the application. The thing I noticed was that most of the CPU time was spent on MySQL. Something made it run terribly slow. I've heard Michiel say that it was running a lot faster using sqlserver instead of mysql. After switching my local version to sqlserver and monitoring the same processes again, it was clear that this was a lot better.

Afterwards, most of the CPU usage happened in the Documentum classes, the file repository of EPA. When I switched to the null-repository I've made earlier, I noticed that it the performance increased a lot, which is actually 'stating the obvious' because there are no files to be loaded into EPA. Last but not least, I've reduced the CPU time on the garbage collector a bit by making sure that no new objects were created in loops.

Since the new XDS-repository was still being created by one of the other interns, I've couldn't continue to make noticeable performance changes in EPA.
New tasks of the DocShifter project were assigned to me: 'Automate building and packing in Jenkins' & 'Optimize Installer'. Let me first tell you about Jenkins. Jenkins is an open source continuous integration tool written in Java. It automatically builds your project. In our case it builds the docShifter projects on Stash. (Stash is the on-premises Git repository management solution for enterprise teams. It allows everyone in your organisation to easily collaborate on your Git repositories.) If Jenkins is asked to build one of the docShifter projects (receiver, beans, sender, ...), it will first check if there are changes in the version on the Stash. If the stash contains changes, Jenkins will clone the project first and afterwards start the building using Ant. Once this is done, it saves the files that were built and triggers the installer to be built. The installer uses the saved files of the other DocShifter projects to create an up-to-date installer. If the build of the installer is done, the installers (32 & 64bit) are stored in Jenkins and are ready to be used.

Once this was done, it was quite easy to create an automatic updater for DocShifter. But there was one problem. The installer and updater are created with IzPack to make sure we have a wizard-like installer. The problem was that IzPack has a couple of boudries. In this case, it can only run batch files at one point during the installation. To update docShifter, I had to stop the services from running, install the new files and start the services again. I had to make a choice between stopping them first OR starting them after the update. For now, I chose to stop the services first. Perhaps I can still change this and hopefully make it work properly. I still have to add the print service (that one of the other interns fixed) to the installer with Jenkins, but it should be just the same way as adding the receiver or sender so this shouldn't cause any problems

Switching from DocShifter to EPA

In the last two weeks, I've switched from working on DocShifter to EPA. I've finished the webservice for DocShifter almost two weeks ago. I've added some new options for this as well.

DocShifter (part two)

The client side sends a request object containing the source file, the name of the file, the requested transformation and the option of returning the converted file instantly or the location of the converted file on the server (for async transformation). It also contains a hashmap with optional preferences. For example if you are converting a picture of some type to a .png, you can add the parameters MaxWidth and MaxHeight. If you choose not to do this, the server side uses its default values.

Since DocShifter runs with 3 modules (input, transformation, output), I had to create some sort of All-transformation-module instead of an already assigned PNG, PDF, ... -module. This module creates another transformation module at runtime according to the user's choice. If the user added extra parameters, the module will use these, if not, it will use the parameters stored in the configuration of this 'All-transformation module'.

After a new transformation task is added to the queue, the server can either start to poll for the converted file by itself and return this, or it can return the location of this file on the server. This depends on what the client side told it to do. If the location is returned, another webservice starts to poll for it on the client side. This way it can be returned without having to keep the server occupied.

The next step in DocShifter is adding a Drupal plugin for so it can run on the web.

EPA

EPA is one of the main projects at Docbyte. It's an abreviation of Electronic Patient Archive and is a sort of file manager/consulting tool for hospitals. It's currently running on its third version. The front-end is build in Ext JS, which is a pure JavaScript application framework for building web applications. The back-end of the application is written in Java.

At this point, it uses Documentum as a repository for files and user information. Documentum is some sort of entreprise content managment platform. My task was to get rid of this and create some sort of null repository. This null repository does, as it name suggests, nothing. The point of doing this is to implement a different way of authenticating the user. This was my second task, I had to implement LDAP user validation into the project. This was done pretty fast. The main problem was implementing the null repository. A lot of  the code was written, using the Documentum classes. This was all hard-coded. So I had to make a lot of stuff generic so that it could run with or without Documentum. After a quite some time, it was running without any problems and I was off to my next task: Improving the general performance of EPA.

First two weeks

I've just started my third week at Docbyte, so it's definitely time to write an update on this blog. Just like I wrote in the previous introduction post, I'm currently developing a web service for DocShifter. In the first few days I had to get the print service for DocShifter working but I stumbled upon some problems with this.

Docshifter

First things first, what is DocShifter? It's a Java-written product that allows users to convert a file of a certain type into a file of another type. For example, it can change a text file (.txt) into a Portable Document Format, better known as PDF. To do this, it uses the Java Message Service (JMS) API to send conversion tasks from one component to another. Let me illustrate this for you.

As you can see on the image above, there are three main components in the project. First of all, we have a sender that places new tasks on a queue. Next, there is a receiver component that collects these tasks and runs the needed conversion. These two parts are configured by the DocShifter beans.

The advantage of dividing the project into these three pieces is that the whole process can work asynchronously. This way, the sender can always accept new tasks and put them on the queue and the user doesn't have to wait for one file to get processed before he can request another conversion. This is very important when processing rather large files.

The sender describes the different input modules of DocShifter while the receiver manages the output modules. An example of such an input is simple file system module. The user configures what directory it has to poll from and the sender will create a new task for every file that it finds in this directory. This task is placed on the JMS queue and will be fetched by the receiver. In this case we could create an output module that puts the converted file into another directory. Once we've done this, we only have to initialize a new process with these input and output modules. It's also possible to use the file system input with a completely different output module. For example, one that places the converted file in a Dropbox folder.

Print service

My first assignment was to get the print service working. This would actually make it possible for the user to print from any program to DocShifter. A lot of the code was already written for this. After diving into the source code for two days, I couldn't really understand how it was supposed to work. The whole application should run like the Line Printer Daemon/Line Printer Remote protocol (LPD or LPR). This is a network protocol for submitting print jobs to a remote printer, in this case, DocShifter.

Because I've got stuck on this, I was assigned another task. Create web service integration for DocShifter.

Web service for DocShifter

Docbyte wants to provide DocShifter as an online service, therefore, I had to create a web service that places new tasks on the queue. It has been quite some time since I've developed a web service, so the first steps went rather slow. Eventually I've chosen to use JAX-WS (Java API for XML Web Services). 

After trying out some things, I've got a simple web service running. Next step was to get it implemented into DocShifter. This was easier said than done. After hours of trying to solve errors, I found out that an unused imported library was blocking my web service. Once I had removed that, everything was working fine.

Because the project converts the files asynchronously, it's not that easy to return a converted file to the user. First of all, I'd let the sender check every 0,5 seconds if the output file had been created. To make sure that it doesn't end up in an infinite loop, the sender would stop polling after 5 seconds. 

In some way this can be approach. It's a way to make it work synchronously and send the file back to the client side of the web service as quick as possible. But in another way, it's keeping your sender from putting new tasks on the queue, especially if the system has large files to process.

So now, the client side will decide if the converted file should be send back immediately or not. If the file(s) are to big for this, the server returns the location of the output files. The files can be collected by another web service that polls this location.

Welcome

Hi,
My name is Frederik Dumazy. I started this blog because I'm doing an internship at Docbyte in Ghent. During this internship I'm going to post my experiences on this blog.  At this time I'm working on developing a web service for one of their products called DocShifter. More about this in my next blogpost.