Extending CTP

This article describes how to add new pipeline stages and database interfaces into CTP. It is intended for programmers, and it assumes familiarity with Java, CVS, and Ant.

1 Obtaining the Source Code

CTP is designed to be extended with pipeline stages of new types. Stages implement one or more Java interfaces. It is useful to obtain the source code and build it in order to obtain the Javadocs, even though in principle you don't need to modify the code itself.

The software for CTP is open source. All the software written by the RSNA for the project is released under the RSNA Public License. It is maintained on a CVS server at RSNA headquarters. To obtain the source code, configure a CVS client as follows:

Protocol:           Password server (:pserver)
Server:             mirc.rsna.org
Port:               2401
Repository folder:  /RSNA
Username:           cvs-reader
Password:           cvs-reader
Module:             ClinicalTrialProcessor

Together, this results in the following CVSROOT (which is constructed automatically if you use something like Tortoise-CVS on a Windows system):

:pserver:cvs-reader@mirc.rsna.org:2401/RSNA

This account has read privileges, but it cannot write into the repository, so it can check out but not commit. If you wish to be able to commit software to the CVS library, contact the MIRC project manager.

2 Building the Software

When you check out the ClinicalTrialProcessor module from CVS, you obtain a directory tree full of the sources and libraries for building the application. The top of the directory tree is ClinicalTrialProcessor. It contains several subdirectories. The source code is in the source directory, which has three subdirectories, one each for the Java sources, the files required by the application, and resources which are included in the application's jar.

Building CTP requires the Java 1.5 JDK and Ant. Running CTP requires the JRE and the JAI ImageIO Tools.

The Ant build file for CTP is in the ClinicalTrialProcessor directory and is called build.xml. To build the software on a Windows system, launch a command window, navigate to the ClinicalTrialProcessor directory, and enter ant all.

The build file contains several targets. The all target does a clean build of everything, including the Javadocs, which are put into the documentation directory. The Javadocs can be accessed with a browser by opening the file:

ClinicalTrialProcessor/documentation/index.html

The ctp-installer target just builds the application and places the installer in the products directory.

3 The Object Classes

CTP provides four classes to encapsulate files of various types. The classes are located in the org.rsna.ctp.objects package:

DicomObject - a DICOM dataset
XmlObject - an XML file containing identifiers relating the data to the trial and the trial subject
ZipObject - a zip file containing a manifest.xml file providing identifiers relating the zip file's contents to the trial and the trial subject
FileObject - a generic file of unknown contents and format

Each class provides methods allowing pipeline stages and database adapters to access the internals of an object without having to know how to parse it. See the Javadocs for a list of all the methods provided by these classes.

4 Implementing a Pipeline Stage

To be recognized as a pipeline stage, a class must implement the org.rsna.ctp.pipeline.PipelineStage interface. An abstract class, org.rsna.ctp.pipeline.AbstractPipelineStage, is provided to supply some of the basic methods required by the PipelineStage interface. All the standard stages extend this class.

Each stage type must also implement its own interface. The interfaces are:

org.rsna.ctp.pipeline.ImportService
org.rsna.ctp.pipeline.Processor
org.rsna.ctp.pipeline.StorageService
org.rsna.ctp.pipeline.ExportService

The Javadocs explain the methods which must be implemented in each stage type.

Each stage class must have a constructor that takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from the element. While it is not required that all configuration information be placed in attributes of the element, the getConfigHTML method provided by AbstractPipelineStage expects it, and if you choose to encode configuration information in another way, you must override the getConfigHTML method to make that information available to the configuration servlet.

5 Implementing a DatabaseAdapter

The DatabaseExportService pipeline stage provides a queuing mechanism for submitting files to a database interface, relieving the interface from having to manage the queue. It calls the overloaded process method of the interface with one of the four object types. Each of the objects includes methods providing access to the internals of its file, allowing the interface to interrogate objects to obtain some or all of their data to insert into an external system.

The DatabaseExportService dynamically loads the database interface class, obtaining the name of the class from the configuration element's adapterClass attribute.

5.1 The DatabaseAdapter Class

The DatabaseAdapter class, org.rsna.ctp.stdstages.database.DatabaseAdapter, is a base class for building an interface between the DatabaseExportService and an external database. To be recognized and loaded by the DatabaseExportService, an external database interface class must be an extension of DatabaseAdapter.

The DatabaseAdapter class provides a set of methods allowing the DatabaseExportService to perform various functions, all of which are explained in the Javadocs. The basic interaction model is:

When the DatabaseExportService detects that files are in its queue, it determines whether the database interface class is loaded and loads it if necessary.
It then calls the database interface’s connect() method.
For each file in the queue, it instantiates an object matching the file’s contents and calls the database interface’s process() method. There are four overloaded process methods, one for each object class.
When the queue is empty, it calls the database interface’s disconnect() method.

All the methods of the DatabaseAdapter class return a static instance of the org.rsna.ctp.pipeline.Status class to indicate the result. The values are:

Status.OK means that the operation succeeded completely.
Status.FAIL means that the operation failed and trying again will also fail. This status value indicates a problem with the object being processed.
Status.RETRY means that the operation failed but trying again later may succeed. This status value indicates a temporary problem accessing the external database.

All the methods of the DatabaseAdapter base class return the value Status.OK.

5.2 Extending the DatabaseAdapter Class

To implement a useful interface to an external database, you must extend the DatabaseAdapter class.

Since the DatabaseAdapter class implements dummy methods returning Status.OK, your class that extends DatabaseAdapter only has to override the methods that apply to your application. If, for example, you only care about XML objects, you can just override the process(XmlObject xmlObject) method and let DatabaseAdapter supply the other process() methods, thus ignoring objects of other types.

Although the DatabaseAdapter class includes a reset() method, it is not called by the DatabaseExportService because restarts are not done in CTP.

The DatabaseAdapter also includes a shutdown() method that is called when CTP is exiting. If multiple DatabaseAdapters are configured (poolSize > 1), the method is only called on the first adapter in the pool. During shutdown, all adapters in the pool are allowed to finish the last process method call before the DatabaseExportService reports that the stage is down, but only the first adapter gets the shutdown call.

Since a complete shutdown of CTP can take over 10 seconds, it is best to ensure that the data is protected in the event of, for example, a power failure. Further, since one connect() call is made for possibly multiple process() method calls, it is possible that a failure could result in no disconnect() call. Thus, depending on the design of the external system, it may be wise to commit changes in each process() call.

6 Implementing a Plugin

To be recognized as a Plugin, a class must implement the org.rsna.ctp.plugin.Plugin interface. An abstract class, org.rsna.ctp.plugin.AbstractPlugin, is provided to supply some of the basic methods required by the Plugin interface. All the standard plugins extend this class.

The Javadocs explain the methods which must be implemented in a Plugin.

Each Plugin class must have a constructor that takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from the element. While it is not required that all configuration information be placed in attributes of the element, the getConfigHTML method provided by AbstractPlugin expects it, and if you choose to encode configuration information in another way, you must override the getConfigHTML method to make that information available to the configuration servlet.

7 Connecting Your Extension Class(es) to CTP

There are two strategies for connecting extension classes into CTP.

7.1 Building an Extension Class as Part of CTP

To build extension classes into the CTP.jar file itself:

Create one or more packages for the classes under the source/java tree within the CTP sources.
Place any required JAR files in the libraries directory.
Edit the build.xml file and add the JAR files to the <path id="classpath"> element.
Build the entire application.

This approach includes the classes in the CTP.jar file and includes the additional JARs in the installer. This will cause everything to be installed when the installer is run.

The disadvantage of this approach is that it places your changes to the build file at risk when CTP changes. It also makes it impossible to distribute your extension separately from CTP.

7.2 Building an Extension JAR

Starting with versions with dates after 2009.05.28, CTP automatically recognizes JAR files placed in its libraries directory. No entries are required on a classpath. This makes it convenient to distribute extensions as separate JARs which are installed simply by dropping them into the libraries directory.

This section will walk through this process in detail. The example will be based on an SftpExportService built by Brian O'Brien at the University of Calgary.

7.2.1 Create a development directory tree

For this project, we start with a top-level directory called SftpExportService, with two child directories, libraries and source.

In the libraries directory, we place all the libraries we will require, plus the CTP.jar file which we can get from our CTP installation. For some applications, it may also be desirable to include the util.jar file, which contains the server and several helper classes.

In the source directory, we place any sources we want. They can be organized into package directories or all placed in the same directory.

7.2.2 Create the Ant build file

For this project, we place the following build.xml file in the top-level directory:

<project name="ClinicalTrialProcessor" default="all" basedir=".">

	<property name="name" value="SFTP"/>

	<property name="build" value="${basedir}/build"/>
	<property name="source" value="${basedir}/source"/>
	<property name="libraries" value="${basedir}/libraries"/>
	<property name="products" value="${basedir}/products"/>
	<property name="documentation" value="${basedir}/documentation"/>

	<property name="jarclasspath" value=""/>

	<path id="classpath">
		<pathelement location="${libraries}/CTP.jar"/>
		<pathelement location="${libraries}/log4j.jar"/>
        	<pathelement location="${libraries}/j2ssh-ant-0.2.9.jar"/>
        	<pathelement location="${libraries}/j2ssh-common-0.2.9.jar"/>
        	<pathelement location="${libraries}/j2ssh-core-0.2.9.jar"/>
        	<pathelement location="${libraries}/j2ssh-daemon-0.2.9.jar"/>
        	<pathelement location="${libraries}/jai_codec.jar"/>
        	<pathelement location="${libraries}/jai_core.jar"/>
        	<pathelement location="${libraries}/commons-logging.jar"/>
	</path>

	<target name="clean">
		<delete dir="${build}" failonerror="false"/>
		<delete dir="${documentation}" failonerror="false"/>
	</target>

	<target name="init">
		<mkdir dir="${build}"/>
		<tstamp>
			<format property="today" pattern="dd-MMMM-yyyy"/>
			<format property="now" pattern="HH:mm:ss"/>
		</tstamp>
		<echo message="Time now ${now}"/>
		<echo message="ant.java.version = ${ant.java.version}" />
		<mkdir dir="${build}"/>
		<mkdir dir="${products}"/>
	</target>

	<target name="compile" depends="init">
		<javac destdir="${build}" optimize="on"
				classpathref="classpath"
				debug="true" debuglevel="lines,vars,source">
			<src path="${source}"/>
			<!--<compilerarg value="-Xlint:unchecked"/>-->
		</javac>
	</target>

	<target name="jar" depends="compile">
		<jar jarfile="${products}/${name}.jar">
			<manifest>
	            		<attribute name="Date" value="${today} at ${now}"/>
	           		<attribute name="Java-Version" value="${ant.java.version}"/>
				<attribute name="Class-Path" value="${jarclasspath}"/>
			</manifest>
			<fileset dir="${build}" includes="**"/>
		</jar>
	</target>

	<target name="javadocs">
		<mkdir dir="${documentation}"/>
		<javadoc destdir="${documentation}" sourcefiles="${source}/**" classpathref="classpath"/>
	</target>

	<target name="all" depends="clean, jar, javadocs"/>

</project>

This build file should work for any extension project, with two changes:

Change the <property name="name" value="SFTP"/> property value to the name you want for your JAR file.
Change the <path id="classpath"> to include the JARs you reference.

7.2.3 Build the JAR

On a Windows system, open a command window in the top-level directory and enter the command ant

The build will place the JAR in the products directory. It will also build Javadocs for the extensions and place them in the documentation directory.

7.2.4 Deploy

To deploy the extension, you must place the JAR, along with any other JARs it references, in the CTP/libraries directory. Any upgrades to CTP using its installer will not overwrite your extension.