Extending CTP

This article describes how to add new pipeline stages and database interfaces into CTP. It is intended for programmers, and it assumes familiarity with Java and Ant.

1 The Source Code

CTP is designed to be extended with new plugins, pipeline stages, and database adapters. These modules implement one or more Java interfaces. It is useful to obtain the source code and build it in order to obtain the Javadocs, even though in principle you don't need to modify the code itself.

See Setting Up a MIRC Development Environment for details on getting the source code, deploying it in a directory structure, and building it.

2 The Object Classes

CTP provides four classes to encapsulate files of various types. The classes are located in the org.rsna.ctp.objects package:

• DicomObject - a DICOM dataset
• XmlObject - an XML file containing identifiers relating the data to the trial and the trial subject
• ZipObject - a zip file containing a manifest.xml file providing identifiers relating the zip file's contents to the trial and the trial subject
• FileObject - a generic file of unknown contents and format

Each class provides methods allowing pipeline stages and database adapters to access the internals of an object without having to know how to parse it. See the Javadocs for a list of all the methods provided by these classes.

3 Implementing a Pipeline Stage

To be recognized as a pipeline stage, a class must implement the org.rsna.ctp.pipeline.PipelineStage interface. An abstract class, org.rsna.ctp.pipeline.AbstractPipelineStage, is provided to supply some of the basic methods required by the PipelineStage interface. All the standard stages extend this class.

Each stage type must also implement its own interface. The interfaces are:

• org.rsna.ctp.pipeline.ImportService
• org.rsna.ctp.pipeline.Processor
• org.rsna.ctp.pipeline.StorageService
• org.rsna.ctp.pipeline.ExportService

The Javadocs explain the methods which must be implemented in each stage type.

Each stage class must have a constructor that takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from that element or its children. While it is not required that all configuration information be placed in attributes of the element, the getConfigHTML method provided by AbstractPipelineStage expects it, and if you choose to encode configuration information in another way, you must override the getConfigHTML method to make that information available to the configuration servlet.

4 Implementing a DatabaseAdapter

The DatabaseExportService pipeline stage provides a queuing mechanism for submitting files to a database interface, relieving the interface from having to manage the queue. It calls the overloaded process method of the interface with one of the four object types. Each of the objects includes methods providing access to the internals of its file, allowing the interface to interrogate objects to obtain some or all of their data to insert into an external system.

The DatabaseExportService dynamically loads the database interface class, obtaining the name of the class from the configuration element's adapterClass attribute.

4.1 The DatabaseAdapter Class

The DatabaseAdapter class, org.rsna.ctp.stdstages.database.DatabaseAdapter, is a base class for building an interface between the DatabaseExportService and an external database. To be recognized and loaded by the DatabaseExportService, an external database interface class must be an extension of DatabaseAdapter.

The DatabaseAdapter class has two constructors. The DatabaseExportService calls the constructor that accepts its configuration file element as an argument, making it possible to pass information from the configuration to the DatabaseAdapter. For backward compatibility, there is also a constructor that takes no arguments. When implementing an extension of the DatabaseAdapter class, the recommended approach is to implement the consructor that takes the configuration file element.

The DatabaseAdapter class provides a set of methods allowing the DatabaseExportService to perform various functions, all of which are explained in the Javadocs. The basic interaction model is:

• When the DatabaseExportService detects that files are in its queue, it determines whether the database interface class is loaded and loads it if necessary.
• It then calls the database interface’s connect() method.
• For each file in the queue, it instantiates an object matching the file’s contents and calls the database interface’s process() method. There are four overloaded process methods, one for each object class.
• When the queue is empty, it calls the database interface’s disconnect() method.

All the methods of the DatabaseAdapter class return a static instance of the org.rsna.ctp.pipeline.Status class to indicate the result. The values are:

• Status.OK means that the operation succeeded completely.
• Status.FAIL means that the operation failed and trying again will also fail. This status value indicates a problem with the object being processed.
• Status.RETRY means that the operation failed but trying again later may succeed. This status value indicates a temporary problem accessing the external database.

All the methods of the DatabaseAdapter base class return the value Status.OK.

4.2 Extending the DatabaseAdapter Class

To implement a useful interface to an external database, you must extend the DatabaseAdapter class.

Since the DatabaseAdapter class implements dummy methods returning Status.OK, your class that extends DatabaseAdapter only has to override the methods that apply to your application. If, for example, you only care about XML objects, you can just override the process(XmlObject xmlObject) method and let DatabaseAdapter supply the other process() methods, thus ignoring objects of other types.

Although the DatabaseAdapter class includes a reset() method, it is not called by the DatabaseExportService because restarts are not done in CTP.

The DatabaseAdapter also includes a shutdown() method that is called when CTP is exiting. If multiple DatabaseAdapters are configured (poolSize > 1), the method is only called on the first adapter in the pool. During shutdown, all adapters in the pool are allowed to finish the last process method call before the DatabaseExportService reports that the stage is down, but only the first adapter gets the shutdown call.

Since a complete shutdown of CTP can take over 10 seconds, it is best to ensure that the data is protected in the event of, for example, a power failure. Further, since one connect() call is made for possibly multiple process() method calls, it is possible that a failure could result in no disconnect() call. Thus, depending on the design of the external system, it may be wise to commit changes in each process() call.

5 Implementing a Plugin

To be recognized as a Plugin, a class must implement the org.rsna.ctp.plugin.Plugin interface. An abstract class, org.rsna.ctp.plugin.AbstractPlugin, is provided to supply some of the basic methods required by the Plugin interface. All the standard plugins extend this class.

The Javadocs explain the methods which must be implemented in a Plugin.

Each Plugin class must have a constructor that takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from the element. While it is not required that all configuration information be placed in attributes of the element, the getConfigHTML method provided by AbstractPlugin expects it, and if you choose to encode configuration information in another way, you must override the getConfigHTML method to make that information available to the configuration servlet.

5.1 Implementing an AnonymizerExtension

An AnonymizerExtension is a Plugin that adds functionality to the DicomAnonymizer. To be recognized as an AnonymizerExtension, a class must implement both the org.rsna.ctp.plugin.Plugin and org.rsna.ctp.stdstages.anonymizer.dicom.AnonymizerExtension interfaces. See Developing DICOM Anonymizer Extensions for more information.

6 Connecting Your Extension Class(es) to CTP

There are two strategies for connecting extension classes into CTP.

6.1 Building an Extension Class as Part of CTP

To build extension classes into the CTP.jar file itself:

1. Create one or more packages for the classes under the source/java tree within the CTP sources.
2. Place any required JAR files in the libraries directory.
3. Edit the build.xml file and add the JAR files to the <path id="classpath"> element.
4. Build the entire application.

This approach includes the classes in the CTP.jar file and includes the additional JARs in the installer. This will cause everything to be installed when the installer is run.

The disadvantage of this approach is that it places your changes to the build file at risk when CTP changes. It also makes it impossible to distribute your extension separately from CTP.

6.2 Building an Extension JAR

Starting with versions with dates after 2009.05.28, CTP automatically recognizes JAR files placed in the CTP/libraries directory or any its subdirectories. No entries are required on a classpath. This makes it convenient to distribute extensions as separate JARs which are installed simply by dropping them into the libraries directory.

7 Example Pipeline Stage

This section will walk through the process of creating a pipeline stage in detail. It is based on an SftpExportService built by Brian O'Brien at the University of Calgary.

7.1 Create a development directory tree

For this project, we start with a top-level directory called SftpExportService, with three child directories, libraries, source, and resources.

In the libraries directory, we place all the libraries we will require, plus the CTP.jar and util.jar files which we can get from our CTP installation.

In the source directory, we place the source modules. The Java sources can be organized into package directories or all placed in the same directory.

In the resources directory, we place any required files, at a minimum the ConfigurationTemplates.xml file that connects the extension to the configuration editor in the CTP Launcher.jar program.

7.2 Create the Source Module(s)

Here is the source code for the extension:

package org.rsna.ctp.stdstages;

import java.io.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.log4j.Logger;
import org.rsna.ctp.objects.DicomObject;
import org.rsna.ctp.objects.FileObject;
import org.rsna.ctp.pipeline.AbstractExportService;
import org.rsna.ctp.pipeline.Status;
import org.rsna.util.StringUtil;
import org.w3c.dom.Element;

import com.sshtools.j2ssh.SshClient;
import com.sshtools.j2ssh.SftpClient;
import com.sshtools.j2ssh.authentication.PublicKeyAuthenticationClient;
import com.sshtools.j2ssh.transport.publickey.SshPrivateKey;
import com.sshtools.j2ssh.transport.publickey.SshPrivateKeyFile;
import com.sshtools.j2ssh.transport.publickey.SshtoolsPrivateKeyFormat;
import com.sshtools.j2ssh.transport.publickey.SshPrivateKey;
import com.sshtools.j2ssh.transport.TransportProtocolState;
import com.sshtools.j2ssh.authentication.AuthenticationProtocolState;

/**
 * An ExportService that exports files via the Ftp protocol.
 */
public class SftpExportService extends AbstractExportService {

	static final Logger logger = Logger.getLogger(SftpExportService.class);

	String username;					
	String hostname;
	String password;
	String keystore;
	String dirStructure;
	String sftpRoot;

	/**
	 * Class constructor; creates a new instance of the ExportService.
	 * @param element the configuration element.
	 */
	public SftpExportService(Element element) throws Exception {
		super(element);
		username = element.getAttribute("username");
		hostname = element.getAttribute("hostname");
		password = element.getAttribute("password");
		keystore = element.getAttribute("keyfile");
		sftpRoot = element.getAttribute("sftpRoot");
		dirStructure = element.getAttribute("dirStructure");
	}

	/**
	 * Export a file.
	 * @param fileToExport the file to export.
	 * @return the status of the attempt to export the file.
	 */
    @Override
	public Status export(File fileToExport) {
		try {
			FileObject fileObject = FileObject.getInstance(fileToExport);
			this.send(fileObject);
			makeAuditLogEntry(fileObject, Status.OK, getName(), sftpRoot);
			return Status.OK;
		}
		catch (Exception ex) {
			logger.warn("Unable to export "+fileToExport);
			return Status.RETRY;
		}
	}

	private void send(FileObject fileObject) throws Exception {

		SshClient sshclient = null;
		try { sshclient = new SshClient(); }
		catch (Exception ex) {
			logger.warn("Unable to get the client",ex);
			throw ex;
		}

		//Establish a connection if we aren't connected
		//int state = sshclient.getConnectionState();
		try { sshclient.connect(hostname); }
		catch (Exception ex) {
			sshclient = null;
			logger.warn("Unable to connect to the server " + hostname,ex);
			throw ex;
		}

		try {
			//Authenticate using a public key
			PublicKeyAuthenticationClient pk = new PublicKeyAuthenticationClient();
			pk.setUsername(username);

			// Open up the private key file
			SshPrivateKeyFile keyfile = SshPrivateKeyFile.parse(new File(keystore));

			// Get the key
			SshPrivateKey key = keyfile.toPrivateKey(password);

			// Set the key and authenticate
			pk.setKey(key);
			int result = sshclient.authenticate(pk);			
			if(result != AuthenticationProtocolState.COMPLETE) {
				Exception ex = new Exception("Login to " + hostname + " failed result=" + result);
				throw ex;
			}
		}
		catch (Exception ex) {
			sshclient.disconnect();
			logger.warn("Unable to authenticate with " + hostname);
			throw ex;
		}

		//Construct the destination directory from the object elements	
		String dirName = replaceElementNames(dirStructure, fileObject);			
		if (dirName.equals("")) dirName = "bullpen";

		try {
			//Open the SFTP channel
			SftpClient ftpclient = sshclient.openSftpClient();

			// make the initial directory.
			ftpclient.mkdirs(sftpRoot + "/" + dirName);

			// change directory
			ftpclient.cd(sftpRoot + "/" + dirName);

			//Send the file to filename.
			//Make a name for the file on the server.
			//The "use unique name" function doesn't seem
			//to work on all servers, so make a name using
			//the makeNameFromDate method, and append the
			//supplied extension.
			String filename = StringUtil.makeNameFromDate() + fileObject.getStandardExtension();

			//logger.warn("file.getAbsolutePath() = " + file.getAbsolutePath());
			ftpclient.put(fileObject.getFile().getAbsolutePath(), filename);

			//disconnect
			ftpclient.quit();

			//Disconnect. This might not be a good idea for performance,
			//but it's probably the safest thing to do since we don't know
			//when the next file will be uploaded and the server might
			//time out on its own. As a test, this call can be removed;
			//the rest of the code should re-establish the connection
			//when necessary.
			sshclient.disconnect();
			//logger.warn("disconnect from " + hostName);
		}
		catch (Exception ex) {
			logger.warn("Unable to upload the file",ex);
			throw ex;
		}
	}
	
	private static String replaceElementNames(String string, FileObject fob) {
		if (fob instanceof DicomObject) {
			DicomObject dob = (DicomObject)fob;
			try {
				Pattern pattern = Pattern.compile("\\$\\{\\w+\\}");
				Matcher matcher = pattern.matcher(string);
				StringBuffer sb = new StringBuffer();
				while (matcher.find()) {
					String group = matcher.group();
					String dicomKeyword = group.substring(2, group.length()-1).trim();
					String repl = dob.getElementValue(dicomKeyword, null);
					if (repl == null) repl = matcher.quoteReplacement(group);
					matcher.appendReplacement(sb, repl);
				}
				matcher.appendTail(sb);
				string = sb.toString();
			}
			catch (Exception quit) { }
		}
		return string;
	}
}

7.3 Create the ConfigurationTemplates.xml File

The ConfigurationTemplates.xml file connects the extension to the configuration editor in the Launcher.jar program. This file must be placed in the base directory of the extension's JAR file. In this project, we put it in the resources directory and reference it in the Ant build.xml file.

<TemplateDefinitions>

	<Components>

		<ExportService>
			<attr name="name" required="yes" default="SftpExportService"/>
			<attr name="class" required="yes" default="org.rsna.ctp.stdstages.SftpExportService" editable="no"/>
			<attr name="root" required="yes" default="roots/SftpExportService"/>
			<attr name="enableExport" required="no" default="yes" options="yes|no"/>
			<attr name="hostname" required="yes" default="">
				<helptext>URL of the destination SFTP site (sftp://ip:port/path)</helptext>
			</attr>
			<attr name="keyfile" required="yes" default="">
				<helptext>The path to the containning the security key</helptext>
			</attr>
			<attr name="sftpRoot" required="yes" default="">
				<helptext>The root directory of the storage tree on the SFTP site</helptext>
			</attr>
			<attr name="dirStructure" required="yes" default="">
				<helptext>The structure of the storage tree under sftpRoot on the SFTP site</helptext>
			</attr>
			<attr name="username" required="yes" default="username"/>
			<attr name="password" required="yes" default="password"/>
			<attr name="acceptDicomObjects" required="no" default="yes" options="yes|no"/>
			<attr name="acceptXmlObjects" required="no" default="yes" options="yes|no"/>
			<attr name="acceptZipObjects" required="no" default="yes" options="yes|no"/>
			<attr name="acceptFileObjects" required="no" default="yes" options="yes|no"/>
			<attr name="dicomScript" required="no" default=""/>
			<attr name="xmlScript" required="no" default=""/>
			<attr name="zipScript" required="no" default=""/>
			<attr name="auditLogID" required="no" default=""/>
			<attr name="auditLogTags" required="no" default=""/>
			<attr name="throttle" required="no" default="0"/>
			<attr name="interval" required="no" default="5000"/>
			<attr name="quarantine" required="yes" default="quarantines/FtpExportService"/>
			<attr name="quarantineTimeDepth" required="no" default="0"/>
		</ExportService>
		
	</Components>

</TemplateDefinitions>

7.4 Create the Ant build file

For this project, we place the following build.xml file in the top-level directory:

<project name="SFTP" default="all" basedir=".">

	<property name="name" value="SFTP"/>

	<property name="build" value="${basedir}/build"/>
	<property name="source" value="${basedir}/source"/>
	<property name="resources" value="${basedir}/resources"/>
	<property name="libraries" value="${basedir}/libraries"/>
	<property name="products" value="${basedir}/products"/>
	<property name="documentation" value="${basedir}/documentation"/>

	<path id="classpath">
		<pathelement location="${libraries}/CTP.jar"/>
		<pathelement location="${libraries}/util.jar"/>
		<pathelement location="${libraries}/log4j.jar"/>
        	<pathelement location="${libraries}/j2ssh-ant-0.2.9.jar"/>
        	<pathelement location="${libraries}/j2ssh-common-0.2.9.jar"/>
        	<pathelement location="${libraries}/j2ssh-core-0.2.9.jar"/>
        	<pathelement location="${libraries}/j2ssh-daemon-0.2.9.jar"/>
        	<pathelement location="${libraries}/jai_codec.jar"/>
        	<pathelement location="${libraries}/jai_core.jar"/>
        	<pathelement location="${libraries}/commons-logging.jar"/>
	</path>

	<target name="clean">
		<delete dir="${build}" failonerror="false"/>
		<delete dir="${documentation}" failonerror="false"/>
	</target>

	<target name="init">
		<tstamp>
			<format property="today" pattern="dd-MMMM-yyyy"/>
			<format property="now" pattern="HH:mm:ss"/>
		</tstamp>
		<echo message="Time now ${now}"/>
		<echo message="ant.java.version = ${ant.java.version}" />
		<mkdir dir="${build}"/>
		<mkdir dir="${products}"/>
	</target>

	<target name="compile" depends="init">
		<javac destdir="${build}" optimize="on"
				includeantruntime="false"
				classpathref="classpath"
				debug="true" debuglevel="lines,vars,source">
			<src path="${source}"/>
			<!--<compilerarg value="-Xlint:unchecked"/>-->
		</javac>
	</target>

	<target name="jar" depends="compile">
		<copy overwrite="true" todir="${build}">
			<fileset dir="${resources}"/>
		</copy>
		<jar jarfile="${products}/${name}.jar">
			<manifest>
	            		<attribute name="Date" value="${today} at ${now}"/>
	           		<attribute name="Java-Version" value="${ant.java.version}"/>
			</manifest>
			<fileset dir="${build}" includes="**"/>
		</jar>
	</target>

	<target name="javadocs">
		<mkdir dir="${documentation}"/>
		<javadoc destdir="${documentation}" sourcefiles="${source}/**" classpathref="classpath"/>
	</target>

	<target name="all" depends="clean, jar, javadocs"/>

</project>

This build file should work for any extension project, with two changes:

1. Change the <property name="name" value="SFTP"/> property value to the name you want for your JAR file.
2. Change the <path id="classpath"> to include the JARs you reference.

7.5 Build the JAR

On a Windows system, open a command window in the top-level directory and enter the command ant

The build will place the JAR in the products directory. It will also build Javadocs for the extensions and place them in the documentation directory.

7.6 Deploy

To deploy the extension on an installed CTP instance, you must place the JAR, along with any other JARs it references, in the CTP/libraries directory. Any upgrades to CTP using its installer will not overwrite your extension.