2 months at Docbyte

It has been 2 months now since I started my internship. It's been a while since I left another blog post so here is the next one.

In my last post I was saying I had to implement the print service of DocShifter into the auto-build and install of Jenkins. That went pretty easy (with some minor difficulties). Eventually the installer was created and could be automatically build by Jenkins. My next assignment was making sure that EPA could work with any CMIS repository to store the documents. This would be a replacement for Documentum.

DocShifter web service continues...

It took me a while before I could get started on this, though. First I had to make changes in DocShifter because the web service I'd created still contained some hard coded information. All the default configurations of the transformation modules were still hard coded into the project. Oops...

I still wanted the client to have the choice of adding parameters for the configuration or using the default ones. To do this, I've made sure that the receiver, who processes the files, could check all configurations of all transformations in the database. Next it would look at the configuration of AnyTranformation module. This contains the configuration of any other transformation module. Let me make this a bit easier to understand.

Suppose we have a PDF transformation and a PNG transformation. The PDF transformation has 3 sets of configurations: pdfconf1, pdfconf2, pdfconf3. The PNG also has a configuration: basicpngconf.

The AnyTransformation module has this configuration.

1. Pdf
1. name: pdf
2. value: pdfconf2
3. description: the name of the configuration for a PDF transformation.
2. Png
1. name: png
2. value: basicpngconf
3. description: the name of the configuration for a PNG transformation.

So once a task is received by the receiver, it first checks if AnyTransformation allows the requested transformation by checking if it is available in his configuration (e.g. PDF is requested. There is a parameter with the name pdf. So we can allow a PDF transformation.)

Next it starts looking for the default configuration (e.g. pdfconf2). If this is found, the receiver creates a transformation with that configuration.

Now that we have our requested transformation module, the receiver checks if the client added his own parameters to the request. If so, it replaces the parameters of the transformation module by the ones that the user requested. This way, the default parameters can be configured by a user and a client can still choose to use the default parameters or add his own (1 or more). 

Solving errors in DocShifter.

Once the web services were fixed I had to solve a problem one of the clients of Docbyte had using DocShifter. There seemed to be an issue that caused some quality loss when converting TIFF files to PDF. After a long search and a lot of testing using different version of Java, older library version, etc. I still could not reproduce this quality loss. Eventually I created a new process on this client's server using a VPN connection. I still was not able to reproduce this error. Apparently the files were merged by another program afterwards so the error was not in DocShifter. I've wasted quite some time on this...

Another error in DocShifter was that TIFF files could not be deleted once they were transformed. After a long search I've managed to find a stream that was not closed :-/

CMIS

I finally could start to work on implementing CMIS in EPA. Well, almost. I ran into a lot of difficulties. Let me put them all in a list:

• EPA could not launch because of a ClassNotFoundException
• I could not get EPA because the Stash was down
• I could not log in because the active directory where my test user was, got relocated without being aware of this
• There were changes in the organizational units on the active directory which resulted in not being able to log in again.
• The Alfresco CMIS repository I installed to test apparently ran on the same port as EPA
• ...

So after a couple of days having trouble, I finally got started. There were a couple of features for the CMIS repository that were already written in EPA but eventually I had to change them because they didn't work properly. At this time it's possible to add files, remove them, view them and get thumbnails of image files. CMIS does not support thumbnails for other files. One of the other things that CMIS does not support is the audit. Documentum could do this but CMIS can't. There is a possibility of using a change log in CMIS but it does not show how viewed a certain file...

Week 5 & 6

I'm now at my 6th week at Docbyte for my internship. My next task was to check for performance errors and potential bottlenecks in EPA. To avoid premature optimization I've used Visual VM. It's a monitoring tool for Java applications. You don't have to add anything to your code to use it. Visual VM automatically checks all running Java application on your device and also has the possibility to monitor Java-apps on remote devices.
After running a version of EPA locally, I've started to monitor the CPU usage at between several points in the application. The thing I noticed was that most of the CPU time was spent on MySQL. Something made it run terribly slow. I've heard Michiel say that it was running a lot faster using sqlserver instead of mysql. After switching my local version to sqlserver and monitoring the same processes again, it was clear that this was a lot better.

Afterwards, most of the CPU usage happened in the Documentum classes, the file repository of EPA. When I switched to the null-repository I've made earlier, I noticed that it the performance increased a lot, which is actually 'stating the obvious' because there are no files to be loaded into EPA. Last but not least, I've reduced the CPU time on the garbage collector a bit by making sure that no new objects were created in loops.

Since the new XDS-repository was still being created by one of the other interns, I've couldn't continue to make noticeable performance changes in EPA.
New tasks of the DocShifter project were assigned to me: 'Automate building and packing in Jenkins' & 'Optimize Installer'. Let me first tell you about Jenkins. Jenkins is an open source continuous integration tool written in Java. It automatically builds your project. In our case it builds the docShifter projects on Stash. (Stash is the on-premises Git repository management solution for enterprise teams. It allows everyone in your organisation to easily collaborate on your Git repositories.) If Jenkins is asked to build one of the docShifter projects (receiver, beans, sender, ...), it will first check if there are changes in the version on the Stash. If the stash contains changes, Jenkins will clone the project first and afterwards start the building using Ant. Once this is done, it saves the files that were built and triggers the installer to be built. The installer uses the saved files of the other DocShifter projects to create an up-to-date installer. If the build of the installer is done, the installers (32 & 64bit) are stored in Jenkins and are ready to be used.

Once this was done, it was quite easy to create an automatic updater for DocShifter. But there was one problem. The installer and updater are created with IzPack to make sure we have a wizard-like installer. The problem was that IzPack has a couple of boudries. In this case, it can only run batch files at one point during the installation. To update docShifter, I had to stop the services from running, install the new files and start the services again. I had to make a choice between stopping them first OR starting them after the update. For now, I chose to stop the services first. Perhaps I can still change this and hopefully make it work properly. I still have to add the print service (that one of the other interns fixed) to the installer with Jenkins, but it should be just the same way as adding the receiver or sender so this shouldn't cause any problems

Switching from DocShifter to EPA

In the last two weeks, I've switched from working on DocShifter to EPA. I've finished the webservice for DocShifter almost two weeks ago. I've added some new options for this as well.

DocShifter (part two)

The client side sends a request object containing the source file, the name of the file, the requested transformation and the option of returning the converted file instantly or the location of the converted file on the server (for async transformation). It also contains a hashmap with optional preferences. For example if you are converting a picture of some type to a .png, you can add the parameters MaxWidth and MaxHeight. If you choose not to do this, the server side uses its default values.

Since DocShifter runs with 3 modules (input, transformation, output), I had to create some sort of All-transformation-module instead of an already assigned PNG, PDF, ... -module. This module creates another transformation module at runtime according to the user's choice. If the user added extra parameters, the module will use these, if not, it will use the parameters stored in the configuration of this 'All-transformation module'.

After a new transformation task is added to the queue, the server can either start to poll for the converted file by itself and return this, or it can return the location of this file on the server. This depends on what the client side told it to do. If the location is returned, another webservice starts to poll for it on the client side. This way it can be returned without having to keep the server occupied.

The next step in DocShifter is adding a Drupal plugin for so it can run on the web.

EPA

EPA is one of the main projects at Docbyte. It's an abreviation of Electronic Patient Archive and is a sort of file manager/consulting tool for hospitals. It's currently running on its third version. The front-end is build in Ext JS, which is a pure JavaScript application framework for building web applications. The back-end of the application is written in Java.

At this point, it uses Documentum as a repository for files and user information. Documentum is some sort of entreprise content managment platform. My task was to get rid of this and create some sort of null repository. This null repository does, as it name suggests, nothing. The point of doing this is to implement a different way of authenticating the user. This was my second task, I had to implement LDAP user validation into the project. This was done pretty fast. The main problem was implementing the null repository. A lot of  the code was written, using the Documentum classes. This was all hard-coded. So I had to make a lot of stuff generic so that it could run with or without Documentum. After a quite some time, it was running without any problems and I was off to my next task: Improving the general performance of EPA.

First two weeks

I've just started my third week at Docbyte, so it's definitely time to write an update on this blog. Just like I wrote in the previous introduction post, I'm currently developing a web service for DocShifter. In the first few days I had to get the print service for DocShifter working but I stumbled upon some problems with this.

Docshifter

First things first, what is DocShifter? It's a Java-written product that allows users to convert a file of a certain type into a file of another type. For example, it can change a text file (.txt) into a Portable Document Format, better known as PDF. To do this, it uses the Java Message Service (JMS) API to send conversion tasks from one component to another. Let me illustrate this for you.

As you can see on the image above, there are three main components in the project. First of all, we have a sender that places new tasks on a queue. Next, there is a receiver component that collects these tasks and runs the needed conversion. These two parts are configured by the DocShifter beans.

The advantage of dividing the project into these three pieces is that the whole process can work asynchronously. This way, the sender can always accept new tasks and put them on the queue and the user doesn't have to wait for one file to get processed before he can request another conversion. This is very important when processing rather large files.

The sender describes the different input modules of DocShifter while the receiver manages the output modules. An example of such an input is simple file system module. The user configures what directory it has to poll from and the sender will create a new task for every file that it finds in this directory. This task is placed on the JMS queue and will be fetched by the receiver. In this case we could create an output module that puts the converted file into another directory. Once we've done this, we only have to initialize a new process with these input and output modules. It's also possible to use the file system input with a completely different output module. For example, one that places the converted file in a Dropbox folder.

Print service

My first assignment was to get the print service working. This would actually make it possible for the user to print from any program to DocShifter. A lot of the code was already written for this. After diving into the source code for two days, I couldn't really understand how it was supposed to work. The whole application should run like the Line Printer Daemon/Line Printer Remote protocol (LPD or LPR). This is a network protocol for submitting print jobs to a remote printer, in this case, DocShifter.

Because I've got stuck on this, I was assigned another task. Create web service integration for DocShifter.

Web service for DocShifter

Docbyte wants to provide DocShifter as an online service, therefore, I had to create a web service that places new tasks on the queue. It has been quite some time since I've developed a web service, so the first steps went rather slow. Eventually I've chosen to use JAX-WS (Java API for XML Web Services). 

After trying out some things, I've got a simple web service running. Next step was to get it implemented into DocShifter. This was easier said than done. After hours of trying to solve errors, I found out that an unused imported library was blocking my web service. Once I had removed that, everything was working fine.

Because the project converts the files asynchronously, it's not that easy to return a converted file to the user. First of all, I'd let the sender check every 0,5 seconds if the output file had been created. To make sure that it doesn't end up in an infinite loop, the sender would stop polling after 5 seconds. 

In some way this can be approach. It's a way to make it work synchronously and send the file back to the client side of the web service as quick as possible. But in another way, it's keeping your sender from putting new tasks on the queue, especially if the system has large files to process.

So now, the client side will decide if the converted file should be send back immediately or not. If the file(s) are to big for this, the server returns the location of the output files. The files can be collected by another web service that polls this location.

Welcome

Hi,
My name is Frederik Dumazy. I started this blog because I'm doing an internship at Docbyte in Ghent. During this internship I'm going to post my experiences on this blog.  At this time I'm working on developing a web service for one of their products called DocShifter. More about this in my next blogpost.