Difference between revisions of "Extending CTP"

From MircWiki
Jump to navigation Jump to search
 
(69 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
This article describes how to add new pipeline stages and database interfaces into CTP. It is intended for programmers, and it assumes familiarity with Java and Ant.  
 
This article describes how to add new pipeline stages and database interfaces into CTP. It is intended for programmers, and it assumes familiarity with Java and Ant.  
  
==Obtaining the Source Code==
+
==The Source Code==
CTP is designed to be extended with pipeline stages of new types. Stages implement one or more Java interfaces. It is useful to obtain the source code and build it in order to obtain the Javadocs, even though in principle you don't need to modify the code itself.
+
CTP is designed to be extended with new plugins, pipeline stages, and database adapters. These modules implement one or more Java interfaces. It is useful to obtain the source code and build it in order to obtain the Javadocs, even though in principle you don't need to modify the code itself.
  
The software for CTP is open source. All the software written by the RSNA for the project is released under the [http://mirc.rsna.org/rsnapubliclicense RSNA Public License]. It is maintained on a CVS server at RSNA headquarters. To obtain the source code, configure a CVS client as follows:
+
See [[Setting Up a MIRC Development Environment]] for details on getting the source code, deploying it in a directory structure, and building it.
<pre>
 
Protocol:          Password server (:pserver)
 
Server:            mirc.rsna.org
 
Port:              2401
 
Repository folder:  /RSNA
 
Username:          cvs-reader
 
Password:          cvs-reader
 
Module:            ClinicalTrialProcessor
 
</pre>
 
 
 
Together, this results in the following CVSROOT (which is constructed automatically if you use something like Tortoise-CVS on a Windows system):
 
 
 
:<tt>:pserver:cvs-reader@mirc.rsna.org:2401/RSNA</tt>
 
 
 
This account has read privileges, but it cannot write into the repository, so it can check out but not commit. If you wish to be able to commit software to the CVS library, contact the MIRC project manager.
 
 
 
==Building the Software==
 
When you check out the <b>ClinicalTrialProcessor</b> module from CVS, you obtain a directory tree full of the sources and libraries for building the application. The top of the directory tree is <tt><b>ClinicalTrialProcessor</b></tt>. It contains several subdirectories. The source code is in the <tt><b>source</b></tt> directory, which has two subdirectories, one each for the Java sources and the files required by the application.
 
 
 
Building CTP requires the Java 1.6 JDK and Ant. Running CTP requires the JDK or JRE and the JAI ImageIO Tools.
 
 
 
The Ant build file for CTP is in the <tt><b>ClinicalTrialProcessor</b></tt> directory and is called <tt><b>build.xml</b></tt>. To build the software on a Windows system, launch a command window, navigate to the <tt><b>ClinicalTrialProcessor</b></tt> directory, and enter <tt><b>ant all</b></tt>.
 
 
 
The build file contains several targets. The <tt><b>all</b></tt> target does a clean build of everything, including the Javadocs, which are put into the <tt><b>documentation</b></tt> directory. The Javadocs can be accessed with a browser by opening the file:
 
 
 
:<tt>ClinicalTrialProcessor/documentation/index.html</tt>
 
 
 
The <tt><b>ctp-installer</b></tt> target just builds the application and places the installer in the <tt><b>products</b></tt> directory.
 
  
 
==The Object Classes==
 
==The Object Classes==
Line 40: Line 12:
 
* [[The CTP ZipObject|<b>ZipObject</b>]] - a zip file containing a <b>manifest.xml</b> file providing identifiers relating the zip file's contents to the trial and the trial subject
 
* [[The CTP ZipObject|<b>ZipObject</b>]] - a zip file containing a <b>manifest.xml</b> file providing identifiers relating the zip file's contents to the trial and the trial subject
 
* <b>FileObject</b> - a generic file of unknown contents and format
 
* <b>FileObject</b> - a generic file of unknown contents and format
Each class provides methods allowing pipeline stages or database adapters to access the internals of an object without having to know how to parse it. See the Javadocs for a list of all the methods provided by these classes.
+
Each class provides methods allowing pipeline stages and database adapters to access the internals of an object without having to know how to parse it. See the Javadocs for a list of all the methods provided by these classes.
  
 
==Implementing a Pipeline Stage==
 
==Implementing a Pipeline Stage==
Line 53: Line 25:
 
The Javadocs explain the methods which must be implemented in each stage type.
 
The Javadocs explain the methods which must be implemented in each stage type.
  
Each stage class must have a constructor which takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from the element. While it is not required that all configuration information be placed in attributes of the element, the <b>getConfigHTML</b> method provided by <b>AbstractPipelineStage</b> expects it, and if you choose to encode configuration information in another way, you must override the <b>getConfigHTML</b> method to make that information available to the configuration servlet.
+
Each stage class must have a constructor that takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from that element or its children. While it is not required that all configuration information be placed in attributes of the element, the <b>getConfigHTML</b> method provided by <b>AbstractPipelineStage</b> expects it, and if you choose to encode configuration information in another way, you must override the <b>getConfigHTML</b> method to make that information available to the configuration servlet.
  
 
==Implementing a DatabaseAdapter==
 
==Implementing a DatabaseAdapter==
Line 62: Line 34:
 
===The DatabaseAdapter Class===
 
===The DatabaseAdapter Class===
 
The DatabaseAdapter class, <b>org.rsna.ctp.stdstages.database.DatabaseAdapter</b>, is a base class for building an interface between the DatabaseExportService and an external database. To be recognized and loaded by the DatabaseExportService, an external database interface class must be an extension of DatabaseAdapter.
 
The DatabaseAdapter class, <b>org.rsna.ctp.stdstages.database.DatabaseAdapter</b>, is a base class for building an interface between the DatabaseExportService and an external database. To be recognized and loaded by the DatabaseExportService, an external database interface class must be an extension of DatabaseAdapter.
 +
 +
The DatabaseAdapter class has two constructors. The DatabaseExportService calls the constructor that accepts its configuration file element as an argument, making it possible to pass information from the configuration to the DatabaseAdapter. For backward compatibility, there is also a constructor that takes no arguments. When implementing an extension of the DatabaseAdapter class, the recommended approach is to implement the consructor that takes the configuration file element.
  
 
The DatabaseAdapter class provides a set of methods allowing the DatabaseExportService to perform various functions, all of which are explained in the Javadocs. The basic interaction model is:
 
The DatabaseAdapter class provides a set of methods allowing the DatabaseExportService to perform various functions, all of which are explained in the Javadocs. The basic interaction model is:
Line 80: Line 54:
 
Since the DatabaseAdapter class implements dummy methods returning Status.OK, your class that extends DatabaseAdapter only has to override the methods that apply to your application. If, for example, you only care about XML objects, you can just override the <b>process(XmlObject xmlObject)</b> method and let DatabaseAdapter supply the other <b>process()</b> methods, thus ignoring objects of other types.
 
Since the DatabaseAdapter class implements dummy methods returning Status.OK, your class that extends DatabaseAdapter only has to override the methods that apply to your application. If, for example, you only care about XML objects, you can just override the <b>process(XmlObject xmlObject)</b> method and let DatabaseAdapter supply the other <b>process()</b> methods, thus ignoring objects of other types.
  
Although the DatabaseAdapter class includes <b>reset()</b> and <b>shutdown()</b> methods, they are not called by the DatabaseExportService because restarts are not done in CTP and there is no notice of an impending shutdown. You should therefore ensure that the data is protected in the event of, for example, a power failure. Similarly, since one <b>connect()</b> call is made for possibly multiple <b>process()</b> method calls, it is possible that a failure could result in no <b>disconnect()</b> call. Thus, depending on the design of the external system, it may be wise to commit changes in each <b>process()</b> call.
+
Although the DatabaseAdapter class includes a <b>reset()</b> method, it is not called by the DatabaseExportService because restarts are not done in CTP.
 +
 
 +
The DatabaseAdapter also includes a <b>shutdown()</b> method that is called when CTP is exiting. If multiple DatabaseAdapters are configured (poolSize &gt; 1), the method is only called on the first adapter in the pool. During shutdown, all adapters in the pool are allowed to finish the last <b>process</b> method call before the DatabaseExportService reports that the stage is down, but only the first adapter gets the <b>shutdown</b> call.  
 +
 
 +
Since a complete shutdown of CTP can take over 10 seconds, it is best to ensure that the data is protected in the event of, for example, a power failure. Further, since one <b>connect()</b> call is made for possibly multiple <b>process()</b> method calls, it is possible that a failure could result in no <b>disconnect()</b> call. Thus, depending on the design of the external system, it may be wise to commit changes in each <b>process()</b> call.
 +
 
 +
==Implementing a Plugin==
 +
To be recognized as a Plugin, a class must implement the <b>org.rsna.ctp.plugin.Plugin</b> interface. An abstract class, <b>org.rsna.ctp.plugin.AbstractPlugin</b>, is provided to supply some of the basic methods required by the Plugin interface. All the standard plugins extend this class.
 +
 
 +
The Javadocs explain the methods which must be implemented in a Plugin.
 +
 
 +
Each Plugin class must have a constructor that takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from the element. While it is not required that all configuration information be placed in attributes of the element, the <b>getConfigHTML</b> method provided by <b>AbstractPlugin</b> expects it, and if you choose to encode configuration information in another way, you must override the <b>getConfigHTML</b> method to make that information available to the configuration servlet.
 +
 
 +
===Implementing an AnonymizerExtension===
 +
An AnonymizerExtension is a Plugin that adds functionality to the DicomAnonymizer. To be recognized as an AnonymizerExtension, a class must implement both the <b>org.rsna.ctp.plugin.Plugin</b> and <b>org.rsna.ctp.stdstages.anonymizer.dicom.AnonymizerExtension</b> interfaces. See [[Developing DICOM Anonymizer Extensions]] for more information.
  
 
==Connecting Your Extension Class(es) to CTP==
 
==Connecting Your Extension Class(es) to CTP==
Line 92: Line 80:
 
#Build the entire application.  
 
#Build the entire application.  
  
This approach includes the classes in CTP.jar file and includes the additional JARs in the installer. This will cause everything to be installed when the installer is run.
+
This approach includes the classes in the CTP.jar file and includes the additional JARs in the installer. This will cause everything to be installed when the installer is run.
  
The disadvantage of this approach is that it places your changes to the buld file at risk when CTP changes. It also makes it impossible to distribute your extension separately from CTP.
+
The disadvantage of this approach is that it places your changes to the build file at risk when CTP changes. It also makes it impossible to distribute your extension separately from CTP.
  
 
===Building an Extension JAR===
 
===Building an Extension JAR===
Starting with versions with dates after 2009.05.28, CTP automatically recognizes JAR files placed in its <b>libraries</b> directory. No entries are required on a classpath. This makes it convenient to distribute extensions as separate JARs which are installed simply by dropping them into the <b>libraries</b> directory.
+
Starting with versions with dates after 2009.05.28, CTP automatically recognizes JAR files placed in the <b>CTP/libraries</b> directory or any its subdirectories. No entries are required on a classpath. This makes it convenient to distribute extensions as separate JARs which are installed simply by dropping them into the <b>libraries</b> directory.
  
This section will walk through this process in detail. The example will be based on an SftpExportService built by Brian O'Brien of the University of Calgary.
+
==Example Pipeline Stage==
 +
This section will walk through the process of creating a pipeline stage in detail. It is based on an SftpExportService built by Brian O'Brien at the University of Calgary.
  
====Create a development directory tree====
+
===Create a development directory tree===
For this project, we start with a top-level directory called <b>SftpExportService</b>, with two child directories, <b>libraries</b> and <b>source</b>.
+
For this project, we start with a top-level directory called <b>SftpExportService</b>, with three child directories, <b>libraries</b>, <b>source</b>, and <b>resources</b>.
  
In the <b>libraries</b> directory, we place all the libraries we will require, plus the <b>CTP.jar</b> file.
+
In the <b>libraries</b> directory, we place all the libraries we will require, plus the <b>CTP.jar</b> and <b>util.jar</b> files which we can get from our CTP installation.
  
In the <b>source</b> directory, we place any sources we want. They can be organized into package directories or all placed in the same directory.
+
In the <b>source</b> directory, we place the source modules. The Java sources can be organized into package directories or all placed in the same directory.  
  
====Create the Ant build file====
+
In the <b>resources</b> directory, we place any required files, at a minimum the <tt>ConfigurationTemplates.xml</tt> file that connects the extension to the configuration editor in the CTP Launcher.jar program.
 +
 
 +
===Create the Source Module(s)===
 +
Here is the source code for the extension:
 +
<pre>
 +
package org.rsna.ctp.stdstages;
 +
 
 +
import java.io.*;
 +
import java.util.regex.Matcher;
 +
import java.util.regex.Pattern;
 +
import org.apache.log4j.Logger;
 +
import org.rsna.ctp.objects.DicomObject;
 +
import org.rsna.ctp.objects.FileObject;
 +
import org.rsna.ctp.pipeline.AbstractExportService;
 +
import org.rsna.ctp.pipeline.Status;
 +
import org.rsna.util.StringUtil;
 +
import org.w3c.dom.Element;
 +
 
 +
import com.sshtools.j2ssh.SshClient;
 +
import com.sshtools.j2ssh.SftpClient;
 +
import com.sshtools.j2ssh.authentication.PublicKeyAuthenticationClient;
 +
import com.sshtools.j2ssh.transport.publickey.SshPrivateKey;
 +
import com.sshtools.j2ssh.transport.publickey.SshPrivateKeyFile;
 +
import com.sshtools.j2ssh.transport.publickey.SshtoolsPrivateKeyFormat;
 +
import com.sshtools.j2ssh.transport.publickey.SshPrivateKey;
 +
import com.sshtools.j2ssh.transport.TransportProtocolState;
 +
import com.sshtools.j2ssh.authentication.AuthenticationProtocolState;
 +
 
 +
/**
 +
* An ExportService that exports files via the Ftp protocol.
 +
*/
 +
public class SftpExportService extends AbstractExportService {
 +
 
 +
static final Logger logger = Logger.getLogger(SftpExportService.class);
 +
 
 +
String username;
 +
String hostname;
 +
String password;
 +
String keystore;
 +
String dirStructure;
 +
String sftpRoot;
 +
 
 +
/**
 +
* Class constructor; creates a new instance of the ExportService.
 +
* @param element the configuration element.
 +
*/
 +
public SftpExportService(Element element) throws Exception {
 +
super(element);
 +
username = element.getAttribute("username");
 +
hostname = element.getAttribute("hostname");
 +
password = element.getAttribute("password");
 +
keystore = element.getAttribute("keyfile");
 +
sftpRoot = element.getAttribute("sftpRoot");
 +
dirStructure = element.getAttribute("dirStructure");
 +
}
 +
 
 +
/**
 +
* Export a file.
 +
* @param fileToExport the file to export.
 +
* @return the status of the attempt to export the file.
 +
*/
 +
    @Override
 +
public Status export(File fileToExport) {
 +
try {
 +
FileObject fileObject = FileObject.getInstance(fileToExport);
 +
this.send(fileObject);
 +
makeAuditLogEntry(fileObject, Status.OK, getName(), sftpRoot);
 +
return Status.OK;
 +
}
 +
catch (Exception ex) {
 +
logger.warn("Unable to export "+fileToExport);
 +
return Status.RETRY;
 +
}
 +
}
 +
 
 +
private void send(FileObject fileObject) throws Exception {
 +
 
 +
SshClient sshclient = null;
 +
try { sshclient = new SshClient(); }
 +
catch (Exception ex) {
 +
logger.warn("Unable to get the client",ex);
 +
throw ex;
 +
}
 +
 
 +
//Establish a connection if we aren't connected
 +
//int state = sshclient.getConnectionState();
 +
try { sshclient.connect(hostname); }
 +
catch (Exception ex) {
 +
sshclient = null;
 +
logger.warn("Unable to connect to the server " + hostname,ex);
 +
throw ex;
 +
}
 +
 
 +
try {
 +
//Authenticate using a public key
 +
PublicKeyAuthenticationClient pk = new PublicKeyAuthenticationClient();
 +
pk.setUsername(username);
 +
 
 +
// Open up the private key file
 +
SshPrivateKeyFile keyfile = SshPrivateKeyFile.parse(new File(keystore));
 +
 
 +
// Get the key
 +
SshPrivateKey key = keyfile.toPrivateKey(password);
 +
 
 +
// Set the key and authenticate
 +
pk.setKey(key);
 +
int result = sshclient.authenticate(pk);
 +
if(result != AuthenticationProtocolState.COMPLETE) {
 +
Exception ex = new Exception("Login to " + hostname + " failed result=" + result);
 +
throw ex;
 +
}
 +
}
 +
catch (Exception ex) {
 +
sshclient.disconnect();
 +
logger.warn("Unable to authenticate with " + hostname);
 +
throw ex;
 +
}
 +
 
 +
//Construct the destination directory from the object elements
 +
String dirName = replaceElementNames(dirStructure, fileObject);
 +
if (dirName.equals("")) dirName = "bullpen";
 +
 
 +
try {
 +
//Open the SFTP channel
 +
SftpClient ftpclient = sshclient.openSftpClient();
 +
 
 +
// make the initial directory.
 +
ftpclient.mkdirs(sftpRoot + "/" + dirName);
 +
 
 +
// change directory
 +
ftpclient.cd(sftpRoot + "/" + dirName);
 +
 
 +
//Send the file to filename.
 +
//Make a name for the file on the server.
 +
//The "use unique name" function doesn't seem
 +
//to work on all servers, so make a name using
 +
//the makeNameFromDate method, and append the
 +
//supplied extension.
 +
String filename = StringUtil.makeNameFromDate() + fileObject.getStandardExtension();
 +
 
 +
//logger.warn("file.getAbsolutePath() = " + file.getAbsolutePath());
 +
ftpclient.put(fileObject.getFile().getAbsolutePath(), filename);
 +
 
 +
//disconnect
 +
ftpclient.quit();
 +
 
 +
//Disconnect. This might not be a good idea for performance,
 +
//but it's probably the safest thing to do since we don't know
 +
//when the next file will be uploaded and the server might
 +
//time out on its own. As a test, this call can be removed;
 +
//the rest of the code should re-establish the connection
 +
//when necessary.
 +
sshclient.disconnect();
 +
//logger.warn("disconnect from " + hostName);
 +
}
 +
catch (Exception ex) {
 +
logger.warn("Unable to upload the file",ex);
 +
throw ex;
 +
}
 +
}
 +
 +
private static String replaceElementNames(String string, FileObject fob) {
 +
if (fob instanceof DicomObject) {
 +
DicomObject dob = (DicomObject)fob;
 +
try {
 +
Pattern pattern = Pattern.compile("\\$\\{\\w+\\}");
 +
Matcher matcher = pattern.matcher(string);
 +
StringBuffer sb = new StringBuffer();
 +
while (matcher.find()) {
 +
String group = matcher.group();
 +
String dicomKeyword = group.substring(2, group.length()-1).trim();
 +
String repl = dob.getElementValue(dicomKeyword, null);
 +
if (repl == null) repl = matcher.quoteReplacement(group);
 +
matcher.appendReplacement(sb, repl);
 +
}
 +
matcher.appendTail(sb);
 +
string = sb.toString();
 +
}
 +
catch (Exception quit) { }
 +
}
 +
return string;
 +
}
 +
}
 +
 
 +
</pre>
 +
 
 +
===Create the ConfigurationTemplates.xml File===
 +
The <tt><b>ConfigurationTemplates.xml</b></tt> file connects the extension to the configuration editor in the Launcher.jar program. This file must be placed in the base directory of the extension's JAR file. In this project, we put it in the <b>resources</b> directory and reference it in the Ant <tt><b>build.xml</b></tt> file.
 +
<pre>
 +
<TemplateDefinitions>
 +
 
 +
<Components>
 +
 
 +
<ExportService>
 +
<attr name="name" required="yes" default="SftpExportService"/>
 +
<attr name="class" required="yes" default="org.rsna.ctp.stdstages.SftpExportService" editable="no"/>
 +
<attr name="root" required="yes" default="roots/SftpExportService"/>
 +
<attr name="enableExport" required="no" default="yes" options="yes|no"/>
 +
<attr name="hostname" required="yes" default="">
 +
<helptext>URL of the destination SFTP site (sftp://ip:port/path)</helptext>
 +
</attr>
 +
<attr name="keyfile" required="yes" default="">
 +
<helptext>The path to the containning the security key</helptext>
 +
</attr>
 +
<attr name="sftpRoot" required="yes" default="">
 +
<helptext>The root directory of the storage tree on the SFTP site</helptext>
 +
</attr>
 +
<attr name="dirStructure" required="yes" default="">
 +
<helptext>The structure of the storage tree under sftpRoot on the SFTP site</helptext>
 +
</attr>
 +
<attr name="username" required="yes" default="username"/>
 +
<attr name="password" required="yes" default="password"/>
 +
<attr name="acceptDicomObjects" required="no" default="yes" options="yes|no"/>
 +
<attr name="acceptXmlObjects" required="no" default="yes" options="yes|no"/>
 +
<attr name="acceptZipObjects" required="no" default="yes" options="yes|no"/>
 +
<attr name="acceptFileObjects" required="no" default="yes" options="yes|no"/>
 +
<attr name="dicomScript" required="no" default=""/>
 +
<attr name="xmlScript" required="no" default=""/>
 +
<attr name="zipScript" required="no" default=""/>
 +
<attr name="auditLogID" required="no" default=""/>
 +
<attr name="auditLogTags" required="no" default=""/>
 +
<attr name="throttle" required="no" default="0"/>
 +
<attr name="interval" required="no" default="5000"/>
 +
<attr name="quarantine" required="yes" default="quarantines/FtpExportService"/>
 +
<attr name="quarantineTimeDepth" required="no" default="0"/>
 +
</ExportService>
 +
 +
</Components>
 +
 
 +
</TemplateDefinitions>
 +
</pre>
 +
 
 +
===Create the Ant build file===
 
For this project, we place the following <b><tt>build.xml</tt></b> file in the top-level directory:
 
For this project, we place the following <b><tt>build.xml</tt></b> file in the top-level directory:
  
 
<pre>
 
<pre>
<project name="ClinicalTrialProcessor" default="all" basedir=".">
+
<project name="SFTP" default="all" basedir=".">
  
 
<property name="name" value="SFTP"/>
 
<property name="name" value="SFTP"/>
Line 118: Line 339:
 
<property name="build" value="${basedir}/build"/>
 
<property name="build" value="${basedir}/build"/>
 
<property name="source" value="${basedir}/source"/>
 
<property name="source" value="${basedir}/source"/>
 +
<property name="resources" value="${basedir}/resources"/>
 
<property name="libraries" value="${basedir}/libraries"/>
 
<property name="libraries" value="${basedir}/libraries"/>
 
<property name="products" value="${basedir}/products"/>
 
<property name="products" value="${basedir}/products"/>
 
<property name="documentation" value="${basedir}/documentation"/>
 
<property name="documentation" value="${basedir}/documentation"/>
 
<property name="jarclasspath" value=""/>
 
  
 
<path id="classpath">
 
<path id="classpath">
 
<pathelement location="${libraries}/CTP.jar"/>
 
<pathelement location="${libraries}/CTP.jar"/>
<pathelement location="${libraries}/dcm4che.jar"/>
+
<pathelement location="${libraries}/util.jar"/>
<pathelement location="${libraries}/getopt.jar"/>
 
 
<pathelement location="${libraries}/log4j.jar"/>
 
<pathelement location="${libraries}/log4j.jar"/>
<pathelement location="${libraries}/edtftpj.jar"/>
 
<pathelement location="${libraries}/jdbm.jar"/>
 
 
         <pathelement location="${libraries}/j2ssh-ant-0.2.9.jar"/>
 
         <pathelement location="${libraries}/j2ssh-ant-0.2.9.jar"/>
 
         <pathelement location="${libraries}/j2ssh-common-0.2.9.jar"/>
 
         <pathelement location="${libraries}/j2ssh-common-0.2.9.jar"/>
Line 146: Line 363:
  
 
<target name="init">
 
<target name="init">
<mkdir dir="${build}"/>
 
 
<tstamp>
 
<tstamp>
 
<format property="today" pattern="dd-MMMM-yyyy"/>
 
<format property="today" pattern="dd-MMMM-yyyy"/>
Line 159: Line 375:
 
<target name="compile" depends="init">
 
<target name="compile" depends="init">
 
<javac destdir="${build}" optimize="on"
 
<javac destdir="${build}" optimize="on"
 +
includeantruntime="false"
 
classpathref="classpath"
 
classpathref="classpath"
 
debug="true" debuglevel="lines,vars,source">
 
debug="true" debuglevel="lines,vars,source">
Line 167: Line 384:
  
 
<target name="jar" depends="compile">
 
<target name="jar" depends="compile">
 +
<copy overwrite="true" todir="${build}">
 +
<fileset dir="${resources}"/>
 +
</copy>
 
<jar jarfile="${products}/${name}.jar">
 
<jar jarfile="${products}/${name}.jar">
 
<manifest>
 
<manifest>
 
            <attribute name="Date" value="${today} at ${now}"/>
 
            <attribute name="Date" value="${today} at ${now}"/>
 
          <attribute name="Java-Version" value="${ant.java.version}"/>
 
          <attribute name="Java-Version" value="${ant.java.version}"/>
<attribute name="Class-Path" value="${jarclasspath}"/>
 
 
</manifest>
 
</manifest>
 
<fileset dir="${build}" includes="**"/>
 
<fileset dir="${build}" includes="**"/>
Line 187: Line 406:
 
</pre>
 
</pre>
  
This build file should work for any extension project with two changes:
+
This build file should work for any extension project, with two changes:
 
#Change the <b><tt>&lt;property name="name" value="SFTP"/&gt;</tt></b> property value to the name you want for your JAR file.
 
#Change the <b><tt>&lt;property name="name" value="SFTP"/&gt;</tt></b> property value to the name you want for your JAR file.
#Change the <b><tt>&lt;path id="classpath"&gt;</b></tt> to include the JARs you reference.
+
#Change the <b><tt>&lt;path id="classpath"&gt;</tt></b> to include the JARs you reference.
  
====Build the JAR====
+
===Build the JAR===
 
On a Windows system, open a command window in the top-level directory and enter the command <b><tt>ant</tt></b>
 
On a Windows system, open a command window in the top-level directory and enter the command <b><tt>ant</tt></b>
  
 
The build will place the JAR in the <b>products</b> directory. It will also build Javadocs for the extensions and place them in the <b>documentation</b> directory.
 
The build will place the JAR in the <b>products</b> directory. It will also build Javadocs for the extensions and place them in the <b>documentation</b> directory.
  
====Deploy====
+
===Deploy===
To deploy the extension, you must place the JAR, along with any other JARs it references, in the <b>CTP/libraries</b> directory. Any upgrades to CTP using its installer will not overwrite your extension.
+
To deploy the extension on an installed CTP instance, you must place the JAR, along with any other JARs it references, in the <b>CTP/libraries</b> directory. Any upgrades to CTP using its installer will not overwrite your extension.

Latest revision as of 18:04, 18 June 2015

This article describes how to add new pipeline stages and database interfaces into CTP. It is intended for programmers, and it assumes familiarity with Java and Ant.

1 The Source Code

CTP is designed to be extended with new plugins, pipeline stages, and database adapters. These modules implement one or more Java interfaces. It is useful to obtain the source code and build it in order to obtain the Javadocs, even though in principle you don't need to modify the code itself.

See Setting Up a MIRC Development Environment for details on getting the source code, deploying it in a directory structure, and building it.

2 The Object Classes

CTP provides four classes to encapsulate files of various types. The classes are located in the org.rsna.ctp.objects package:

  • DicomObject - a DICOM dataset
  • XmlObject - an XML file containing identifiers relating the data to the trial and the trial subject
  • ZipObject - a zip file containing a manifest.xml file providing identifiers relating the zip file's contents to the trial and the trial subject
  • FileObject - a generic file of unknown contents and format

Each class provides methods allowing pipeline stages and database adapters to access the internals of an object without having to know how to parse it. See the Javadocs for a list of all the methods provided by these classes.

3 Implementing a Pipeline Stage

To be recognized as a pipeline stage, a class must implement the org.rsna.ctp.pipeline.PipelineStage interface. An abstract class, org.rsna.ctp.pipeline.AbstractPipelineStage, is provided to supply some of the basic methods required by the PipelineStage interface. All the standard stages extend this class.

Each stage type must also implement its own interface. The interfaces are:

  • org.rsna.ctp.pipeline.ImportService
  • org.rsna.ctp.pipeline.Processor
  • org.rsna.ctp.pipeline.StorageService
  • org.rsna.ctp.pipeline.ExportService

The Javadocs explain the methods which must be implemented in each stage type.

Each stage class must have a constructor that takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from that element or its children. While it is not required that all configuration information be placed in attributes of the element, the getConfigHTML method provided by AbstractPipelineStage expects it, and if you choose to encode configuration information in another way, you must override the getConfigHTML method to make that information available to the configuration servlet.

4 Implementing a DatabaseAdapter

The DatabaseExportService pipeline stage provides a queuing mechanism for submitting files to a database interface, relieving the interface from having to manage the queue. It calls the overloaded process method of the interface with one of the four object types. Each of the objects includes methods providing access to the internals of its file, allowing the interface to interrogate objects to obtain some or all of their data to insert into an external system.

The DatabaseExportService dynamically loads the database interface class, obtaining the name of the class from the configuration element's adapterClass attribute.

4.1 The DatabaseAdapter Class

The DatabaseAdapter class, org.rsna.ctp.stdstages.database.DatabaseAdapter, is a base class for building an interface between the DatabaseExportService and an external database. To be recognized and loaded by the DatabaseExportService, an external database interface class must be an extension of DatabaseAdapter.

The DatabaseAdapter class has two constructors. The DatabaseExportService calls the constructor that accepts its configuration file element as an argument, making it possible to pass information from the configuration to the DatabaseAdapter. For backward compatibility, there is also a constructor that takes no arguments. When implementing an extension of the DatabaseAdapter class, the recommended approach is to implement the consructor that takes the configuration file element.

The DatabaseAdapter class provides a set of methods allowing the DatabaseExportService to perform various functions, all of which are explained in the Javadocs. The basic interaction model is:

  • When the DatabaseExportService detects that files are in its queue, it determines whether the database interface class is loaded and loads it if necessary.
  • It then calls the database interface’s connect() method.
  • For each file in the queue, it instantiates an object matching the file’s contents and calls the database interface’s process() method. There are four overloaded process methods, one for each object class.
  • When the queue is empty, it calls the database interface’s disconnect() method.

All the methods of the DatabaseAdapter class return a static instance of the org.rsna.ctp.pipeline.Status class to indicate the result. The values are:

  • Status.OK means that the operation succeeded completely.
  • Status.FAIL means that the operation failed and trying again will also fail. This status value indicates a problem with the object being processed.
  • Status.RETRY means that the operation failed but trying again later may succeed. This status value indicates a temporary problem accessing the external database.

All the methods of the DatabaseAdapter base class return the value Status.OK.

4.2 Extending the DatabaseAdapter Class

To implement a useful interface to an external database, you must extend the DatabaseAdapter class.

Since the DatabaseAdapter class implements dummy methods returning Status.OK, your class that extends DatabaseAdapter only has to override the methods that apply to your application. If, for example, you only care about XML objects, you can just override the process(XmlObject xmlObject) method and let DatabaseAdapter supply the other process() methods, thus ignoring objects of other types.

Although the DatabaseAdapter class includes a reset() method, it is not called by the DatabaseExportService because restarts are not done in CTP.

The DatabaseAdapter also includes a shutdown() method that is called when CTP is exiting. If multiple DatabaseAdapters are configured (poolSize > 1), the method is only called on the first adapter in the pool. During shutdown, all adapters in the pool are allowed to finish the last process method call before the DatabaseExportService reports that the stage is down, but only the first adapter gets the shutdown call.

Since a complete shutdown of CTP can take over 10 seconds, it is best to ensure that the data is protected in the event of, for example, a power failure. Further, since one connect() call is made for possibly multiple process() method calls, it is possible that a failure could result in no disconnect() call. Thus, depending on the design of the external system, it may be wise to commit changes in each process() call.

5 Implementing a Plugin

To be recognized as a Plugin, a class must implement the org.rsna.ctp.plugin.Plugin interface. An abstract class, org.rsna.ctp.plugin.AbstractPlugin, is provided to supply some of the basic methods required by the Plugin interface. All the standard plugins extend this class.

The Javadocs explain the methods which must be implemented in a Plugin.

Each Plugin class must have a constructor that takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from the element. While it is not required that all configuration information be placed in attributes of the element, the getConfigHTML method provided by AbstractPlugin expects it, and if you choose to encode configuration information in another way, you must override the getConfigHTML method to make that information available to the configuration servlet.

5.1 Implementing an AnonymizerExtension

An AnonymizerExtension is a Plugin that adds functionality to the DicomAnonymizer. To be recognized as an AnonymizerExtension, a class must implement both the org.rsna.ctp.plugin.Plugin and org.rsna.ctp.stdstages.anonymizer.dicom.AnonymizerExtension interfaces. See Developing DICOM Anonymizer Extensions for more information.

6 Connecting Your Extension Class(es) to CTP

There are two strategies for connecting extension classes into CTP.

6.1 Building an Extension Class as Part of CTP

To build extension classes into the CTP.jar file itself:

  1. Create one or more packages for the classes under the source/java tree within the CTP sources.
  2. Place any required JAR files in the libraries directory.
  3. Edit the build.xml file and add the JAR files to the <path id="classpath"> element.
  4. Build the entire application.

This approach includes the classes in the CTP.jar file and includes the additional JARs in the installer. This will cause everything to be installed when the installer is run.

The disadvantage of this approach is that it places your changes to the build file at risk when CTP changes. It also makes it impossible to distribute your extension separately from CTP.

6.2 Building an Extension JAR

Starting with versions with dates after 2009.05.28, CTP automatically recognizes JAR files placed in the CTP/libraries directory or any its subdirectories. No entries are required on a classpath. This makes it convenient to distribute extensions as separate JARs which are installed simply by dropping them into the libraries directory.

7 Example Pipeline Stage

This section will walk through the process of creating a pipeline stage in detail. It is based on an SftpExportService built by Brian O'Brien at the University of Calgary.

7.1 Create a development directory tree

For this project, we start with a top-level directory called SftpExportService, with three child directories, libraries, source, and resources.

In the libraries directory, we place all the libraries we will require, plus the CTP.jar and util.jar files which we can get from our CTP installation.

In the source directory, we place the source modules. The Java sources can be organized into package directories or all placed in the same directory.

In the resources directory, we place any required files, at a minimum the ConfigurationTemplates.xml file that connects the extension to the configuration editor in the CTP Launcher.jar program.

7.2 Create the Source Module(s)

Here is the source code for the extension:

package org.rsna.ctp.stdstages;

import java.io.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.log4j.Logger;
import org.rsna.ctp.objects.DicomObject;
import org.rsna.ctp.objects.FileObject;
import org.rsna.ctp.pipeline.AbstractExportService;
import org.rsna.ctp.pipeline.Status;
import org.rsna.util.StringUtil;
import org.w3c.dom.Element;

import com.sshtools.j2ssh.SshClient;
import com.sshtools.j2ssh.SftpClient;
import com.sshtools.j2ssh.authentication.PublicKeyAuthenticationClient;
import com.sshtools.j2ssh.transport.publickey.SshPrivateKey;
import com.sshtools.j2ssh.transport.publickey.SshPrivateKeyFile;
import com.sshtools.j2ssh.transport.publickey.SshtoolsPrivateKeyFormat;
import com.sshtools.j2ssh.transport.publickey.SshPrivateKey;
import com.sshtools.j2ssh.transport.TransportProtocolState;
import com.sshtools.j2ssh.authentication.AuthenticationProtocolState;

/**
 * An ExportService that exports files via the Ftp protocol.
 */
public class SftpExportService extends AbstractExportService {

	static final Logger logger = Logger.getLogger(SftpExportService.class);

	String username;					
	String hostname;
	String password;
	String keystore;
	String dirStructure;
	String sftpRoot;

	/**
	 * Class constructor; creates a new instance of the ExportService.
	 * @param element the configuration element.
	 */
	public SftpExportService(Element element) throws Exception {
		super(element);
		username = element.getAttribute("username");
		hostname = element.getAttribute("hostname");
		password = element.getAttribute("password");
		keystore = element.getAttribute("keyfile");
		sftpRoot = element.getAttribute("sftpRoot");
		dirStructure = element.getAttribute("dirStructure");
	}

	/**
	 * Export a file.
	 * @param fileToExport the file to export.
	 * @return the status of the attempt to export the file.
	 */
    @Override
	public Status export(File fileToExport) {
		try {
			FileObject fileObject = FileObject.getInstance(fileToExport);
			this.send(fileObject);
			makeAuditLogEntry(fileObject, Status.OK, getName(), sftpRoot);
			return Status.OK;
		}
		catch (Exception ex) {
			logger.warn("Unable to export "+fileToExport);
			return Status.RETRY;
		}
	}

	private void send(FileObject fileObject) throws Exception {

		SshClient sshclient = null;
		try { sshclient = new SshClient(); }
		catch (Exception ex) {
			logger.warn("Unable to get the client",ex);
			throw ex;
		}

		//Establish a connection if we aren't connected
		//int state = sshclient.getConnectionState();
		try { sshclient.connect(hostname); }
		catch (Exception ex) {
			sshclient = null;
			logger.warn("Unable to connect to the server " + hostname,ex);
			throw ex;
		}

		try {
			//Authenticate using a public key
			PublicKeyAuthenticationClient pk = new PublicKeyAuthenticationClient();
			pk.setUsername(username);

			// Open up the private key file
			SshPrivateKeyFile keyfile = SshPrivateKeyFile.parse(new File(keystore));

			// Get the key
			SshPrivateKey key = keyfile.toPrivateKey(password);

			// Set the key and authenticate
			pk.setKey(key);
			int result = sshclient.authenticate(pk);			
			if(result != AuthenticationProtocolState.COMPLETE) {
				Exception ex = new Exception("Login to " + hostname + " failed result=" + result);
				throw ex;
			}
		}
		catch (Exception ex) {
			sshclient.disconnect();
			logger.warn("Unable to authenticate with " + hostname);
			throw ex;
		}

		//Construct the destination directory from the object elements	
		String dirName = replaceElementNames(dirStructure, fileObject);			
		if (dirName.equals("")) dirName = "bullpen";

		try {
			//Open the SFTP channel
			SftpClient ftpclient = sshclient.openSftpClient();

			// make the initial directory.
			ftpclient.mkdirs(sftpRoot + "/" + dirName);

			// change directory
			ftpclient.cd(sftpRoot + "/" + dirName);

			//Send the file to filename.
			//Make a name for the file on the server.
			//The "use unique name" function doesn't seem
			//to work on all servers, so make a name using
			//the makeNameFromDate method, and append the
			//supplied extension.
			String filename = StringUtil.makeNameFromDate() + fileObject.getStandardExtension();

			//logger.warn("file.getAbsolutePath() = " + file.getAbsolutePath());
			ftpclient.put(fileObject.getFile().getAbsolutePath(), filename);

			//disconnect
			ftpclient.quit();

			//Disconnect. This might not be a good idea for performance,
			//but it's probably the safest thing to do since we don't know
			//when the next file will be uploaded and the server might
			//time out on its own. As a test, this call can be removed;
			//the rest of the code should re-establish the connection
			//when necessary.
			sshclient.disconnect();
			//logger.warn("disconnect from " + hostName);
		}
		catch (Exception ex) {
			logger.warn("Unable to upload the file",ex);
			throw ex;
		}
	}
	
	private static String replaceElementNames(String string, FileObject fob) {
		if (fob instanceof DicomObject) {
			DicomObject dob = (DicomObject)fob;
			try {
				Pattern pattern = Pattern.compile("\\$\\{\\w+\\}");
				Matcher matcher = pattern.matcher(string);
				StringBuffer sb = new StringBuffer();
				while (matcher.find()) {
					String group = matcher.group();
					String dicomKeyword = group.substring(2, group.length()-1).trim();
					String repl = dob.getElementValue(dicomKeyword, null);
					if (repl == null) repl = matcher.quoteReplacement(group);
					matcher.appendReplacement(sb, repl);
				}
				matcher.appendTail(sb);
				string = sb.toString();
			}
			catch (Exception quit) { }
		}
		return string;
	}
}

7.3 Create the ConfigurationTemplates.xml File

The ConfigurationTemplates.xml file connects the extension to the configuration editor in the Launcher.jar program. This file must be placed in the base directory of the extension's JAR file. In this project, we put it in the resources directory and reference it in the Ant build.xml file.

<TemplateDefinitions>

	<Components>

		<ExportService>
			<attr name="name" required="yes" default="SftpExportService"/>
			<attr name="class" required="yes" default="org.rsna.ctp.stdstages.SftpExportService" editable="no"/>
			<attr name="root" required="yes" default="roots/SftpExportService"/>
			<attr name="enableExport" required="no" default="yes" options="yes|no"/>
			<attr name="hostname" required="yes" default="">
				<helptext>URL of the destination SFTP site (sftp://ip:port/path)</helptext>
			</attr>
			<attr name="keyfile" required="yes" default="">
				<helptext>The path to the containning the security key</helptext>
			</attr>
			<attr name="sftpRoot" required="yes" default="">
				<helptext>The root directory of the storage tree on the SFTP site</helptext>
			</attr>
			<attr name="dirStructure" required="yes" default="">
				<helptext>The structure of the storage tree under sftpRoot on the SFTP site</helptext>
			</attr>
			<attr name="username" required="yes" default="username"/>
			<attr name="password" required="yes" default="password"/>
			<attr name="acceptDicomObjects" required="no" default="yes" options="yes|no"/>
			<attr name="acceptXmlObjects" required="no" default="yes" options="yes|no"/>
			<attr name="acceptZipObjects" required="no" default="yes" options="yes|no"/>
			<attr name="acceptFileObjects" required="no" default="yes" options="yes|no"/>
			<attr name="dicomScript" required="no" default=""/>
			<attr name="xmlScript" required="no" default=""/>
			<attr name="zipScript" required="no" default=""/>
			<attr name="auditLogID" required="no" default=""/>
			<attr name="auditLogTags" required="no" default=""/>
			<attr name="throttle" required="no" default="0"/>
			<attr name="interval" required="no" default="5000"/>
			<attr name="quarantine" required="yes" default="quarantines/FtpExportService"/>
			<attr name="quarantineTimeDepth" required="no" default="0"/>
		</ExportService>
		
	</Components>

</TemplateDefinitions>

7.4 Create the Ant build file

For this project, we place the following build.xml file in the top-level directory:

<project name="SFTP" default="all" basedir=".">

	<property name="name" value="SFTP"/>

	<property name="build" value="${basedir}/build"/>
	<property name="source" value="${basedir}/source"/>
	<property name="resources" value="${basedir}/resources"/>
	<property name="libraries" value="${basedir}/libraries"/>
	<property name="products" value="${basedir}/products"/>
	<property name="documentation" value="${basedir}/documentation"/>

	<path id="classpath">
		<pathelement location="${libraries}/CTP.jar"/>
		<pathelement location="${libraries}/util.jar"/>
		<pathelement location="${libraries}/log4j.jar"/>
        	<pathelement location="${libraries}/j2ssh-ant-0.2.9.jar"/>
        	<pathelement location="${libraries}/j2ssh-common-0.2.9.jar"/>
        	<pathelement location="${libraries}/j2ssh-core-0.2.9.jar"/>
        	<pathelement location="${libraries}/j2ssh-daemon-0.2.9.jar"/>
        	<pathelement location="${libraries}/jai_codec.jar"/>
        	<pathelement location="${libraries}/jai_core.jar"/>
        	<pathelement location="${libraries}/commons-logging.jar"/>
	</path>

	<target name="clean">
		<delete dir="${build}" failonerror="false"/>
		<delete dir="${documentation}" failonerror="false"/>
	</target>

	<target name="init">
		<tstamp>
			<format property="today" pattern="dd-MMMM-yyyy"/>
			<format property="now" pattern="HH:mm:ss"/>
		</tstamp>
		<echo message="Time now ${now}"/>
		<echo message="ant.java.version = ${ant.java.version}" />
		<mkdir dir="${build}"/>
		<mkdir dir="${products}"/>
	</target>

	<target name="compile" depends="init">
		<javac destdir="${build}" optimize="on"
				includeantruntime="false"
				classpathref="classpath"
				debug="true" debuglevel="lines,vars,source">
			<src path="${source}"/>
			<!--<compilerarg value="-Xlint:unchecked"/>-->
		</javac>
	</target>

	<target name="jar" depends="compile">
		<copy overwrite="true" todir="${build}">
			<fileset dir="${resources}"/>
		</copy>
		<jar jarfile="${products}/${name}.jar">
			<manifest>
	            		<attribute name="Date" value="${today} at ${now}"/>
	           		<attribute name="Java-Version" value="${ant.java.version}"/>
			</manifest>
			<fileset dir="${build}" includes="**"/>
		</jar>
	</target>

	<target name="javadocs">
		<mkdir dir="${documentation}"/>
		<javadoc destdir="${documentation}" sourcefiles="${source}/**" classpathref="classpath"/>
	</target>

	<target name="all" depends="clean, jar, javadocs"/>

</project>

This build file should work for any extension project, with two changes:

  1. Change the <property name="name" value="SFTP"/> property value to the name you want for your JAR file.
  2. Change the <path id="classpath"> to include the JARs you reference.

7.5 Build the JAR

On a Windows system, open a command window in the top-level directory and enter the command ant

The build will place the JAR in the products directory. It will also build Javadocs for the extensions and place them in the documentation directory.

7.6 Deploy

To deploy the extension on an installed CTP instance, you must place the JAR, along with any other JARs it references, in the CTP/libraries directory. Any upgrades to CTP using its installer will not overwrite your extension.