MIRC CTP
This article describes the stand-alone processing application for clinical trials data using MIRC components and the MIRC internet transport mechanism.
1 Background
MIRC supports clinical trials through two applications, one for data acquisition at an imaging center (FieldCenter) and one for management of the data at a principal investigator's site (MIRC).
The FieldCenter application acquires images via the DICOM protocol, anonymizes them, and transfers them (typically using HTTP, although DICOM is also supported) to a principal investigator's MIRC site. It also supports other types of data files and includes an anonymizer for XML files as well. FieldCenter also contains a client for the Update Service of a MIRC site, allowing the application to save data on, and obtain software updates from, the principal investigator's site.
The MIRC site software contains a partially configurable processing pipeline for clinical trials data, consisting of:
- HttpImportService
- A receiver for HTTP connections from FieldCenter applications transferring data files into the processing pipeline.
- DicomImportService
- A receiver form DICOM datasets for iinsertion into the processing pipeline.
- Preprocessor
- A user-defined component for processing data received by the HttpImportService before it is further processed by other components.
- Anonymizer
- A component for anonymizing DICOM objects or XML objects.
- DatabaseExportService
- A component providing queue management and submission of data objects to a user-defined interface to an external database management system.
- HttpExportService
- A component in the DicomImportService pipeline providing queue management and transmission of data objects to one or more external systems using the HTTP protocol.
- DicomExportService
- A component in the HttpImportService pipeline providing queue management and transmission of data objects to one or more external systems using the DICOM protocol.
The processing pipelines for the HttpImportService and DicomImportService are different. They are not symmetrical. For example, the HttpImportService does not have access to the anonymizer except as part of the DatabaseExportService. Another limitation is that objects received via one protocol can only be exported via the other. While these limitations are consistent with the requirements of most trials, it became clear that a completely symmetrical design would provide better support for more sophisticated trials while still satisfying the requirements of simple ones.
2 ClinicalTrialProcessor
ClinicalTrialProcessor is a stand-alone program that provides all the features of a MIRC site for clinical trials in a highly configurable and extensible application. It connects to FieldCenter applications and can also connect to MIRC sites when necessary. ClinicalTrialProcessor has the following key features:
- Single-click installation.
- Support for multiple pipelines.
- Processing pipelines supporting multiple configurable stages.
- Support for multiple quarantines for data objects which are rejected during processing.
- Pre-defined implementations for key components:
- HTTP Import
- DICOM Import
- DICOM Anonymizer
- XML Anonymizer
- File Storage
- Database Export
- HTTP Export
- DICOM Export
- Web-based monitoring of the application's status, including:
- configuration
- logs
- quarantines
- status
- Support for the FieldCenter Update Service client.
2.1 Installation
The installer for ClinicalTrialProcessor is available on the RSNA MIRC site. To run the installer, the Java 1.5 (or better) JRE must be present on the system.
To run the installer, double-click the ClinicalTrialProcessor-installer.jar file and choose a directory in which to install the program. The installer can also be run in command window using the command:
- java -jar ClinicalTrialProcessor-installer.jar
To run the ClinicalTrialProcessor program, the Java Advanced Imaging ImageIO Tools must be present on the system. Java and all its components are available on the java.sun.org website. When obtaining the ImageIO Tools, pay close attention to the fact that the Java Advanced Imaging component is not the same as the Java Advanced Imaging ImageIO Tools. Only the latter are required.
The ClinicalTrialProcessor has no user interface. To run the program, open a command window, navigate to the directory in which the program was installed, and enter the command:
- java -jar ClinicalTrialProcessor.jar
When the program starts, it runs without intervention. Status and other information can be obtained through the program's integrated webserver. Accessing the server with no path information displays a page presenting buttons for each of the servlets.
The program uses two configurable files: config.xml, which is located in the same directory as the program itself, and index.html, which is located in the server's ROOT directory. Both files are intended to be configured for the specific application. Therefore, the installer does not overwrite these files when it runs. Instead, it copies two example files: example-config.xml and example-index.html. When ClinicalTrialProcessor starts, it looks to see if the non-example files are missing, and if so, it copies the example files into the non-example ones. This process allows upgrades to be done without losing any configuration work. After installing the program the first time, it should be run once in order to make the copies, and then the copies can be configured. Configuration is done by hand with any text editor (e.g., TextPad or NotePad). Care should be taken, especially with config.xml, to keep it well-formed. Opening it with a program like InternetExplorer will check it for errors.
2.2 Pipelines
A pipeline is a manager that moves data objects through a sequence of processing stages. Each stage in the pipeline performs a specific function on one or more of the four basic object types supported by MIRC:
- FileObject
- DicomObject
- XmlObject
- ZipObject
Each pipeline must contain one ImportService as its first stage. Each pipeline stage is provided access to a quarantine directory, which may be unique to the stage or shared with other stages, into which the pipeline places objects that are rejected by a stage, thus aborting further processing. At the end of the pipeline, the manager calls the ImportService to remove the processed object from its queue.
There are four types of pipeline stages. Each is briefly described in subsections below.
2.2.1 ImportService
An ImportService receives objects via a protocol and enqueues them for processing by subsequent stages.
2.2.2 StorageService
A StorageService stores an object in a file system. It is not queued, and it therefore must complete before subsequent stages can proceed. A StorageService may return the current object or the stored object in response to a request for the output object, depending on its implementation.
2.2.3 Processor
A Processor performs some kind of processing on an object. Processors are not intended to be queued. In the context of the current MIRC implementation, a Preprocessor is a Processor, as is an Anonymizer. The result of a processing stage is an object that is passed to the next stage in the pipeline.
2.2.4 ExportService
An ExportService provides queued transmission to an external system via a defined protocol. Objects in the queue are full copies of the objects submitted; therefore, subsequent processing is not impeded if a queue is paused, and modifications made subsequently do not affect the queue entry, even if they occur before transmission. (Note: This behavior is different from that of the current MIRC implementation.) After entering an object in its queue, an ExportService returns immediately.
2.3 Configuration
The ClinicalTrialProcessor configuration is specified by an XML file called config.xml located in the same directory as the program. There can be one Server element specifying the port on which the HTTP server is to operate, and multiple Pipeline elements, each specifying the stages which comprise it. The name of the element defining a stage is irrelevant and can be chosen for readability; each stage in a pipeline is actually defined by its Java class, specified in the class attribute. Stages are loaded automatically when the program starts, and the loader tests the stage's class to determine what kind of stage it represents. It is possible to extend the application beyond the pre-defined stages available in the implementation as described in #Extending ClinicalTrialProcessor.
The following is an example of a simple configuration with one pipeline which receives objects via the HTTP protocol, stores them, and exports them to a DICOM destination:
<Configuration>
<Server port="80" />
<Pipeline name="Main Pipeline">
<ImportService
name="HTTP Import"
class="org.rsna.ctp.stdstages.HttpImportService"
root="roots/http-import"
port="7777" />
<StorageService
name="Storage"
class="org.rsna.ctp.stdstages.FileStorageService"
root="D:/storage"
return-stored-file="no"
quarantine="quarantines/StorageServiceQuarantine" />
<ExportService
name="PACS Export"
class="org.rsna.ctp.stdstages.DicomExportService"
root="roots/pacs-export"
dest-url="dicom://DestinationAET:ThisAET@ipaddress:port" />
</Pipeline>
</Configuration>
Note that in the example above, non-DICOM objects are stored in the StorageService, but they are not exported by the DicomExportService. Each pipeline stage is responsible for testing the class of the object which it receives and processing (or ignoring) the object accordingly.
The following is an example of a more complex configuration. This configuration receives objects, passes them to a trial-specific Processor stage to test whether they are appropriate for the trial, anonymizes objects which make it through the preprocessor, exports them to a database, anonymizes them again to remove information which is not intended for storage, and finally stores them.
<Configuration>
<Server port="80" />
<Pipeline name="Main Pipeline">
<ImportService
name="HTTP Import"
class="org.rsna.ctp.stdstages.HttpImportService"
root="roots/http-import"
port="7777" />
<Processor
name="The Preprocessor"
class="org.myorg.MyPreprocessor"
quarantine="quarantines/PreprocessorQuarantine" />
<Processor
name="Main Anonymizer"
class="org.rsna.trials.Anonymizer"
dicom-script="dicom-anonymizer-1.properties"
xml-script="xml-anonymizer-1.script"
zip-script="zip-anonymizer-1.script"
quarantine="quarantines/MainAnonymizerQuarantine" />
<ExportService
name="Database Export"
class="org.rsna.trials.DatabaseExportService"
adapter-class="org.myorg.MyDatabaseAdapter"
root="roots/database-export" />
<Processor
name="Provenance Remover"
class="org.rsna.ctp.stdstages.Anonymizer"
dicom-script="dicom-anonymizer-2.properties"
xml-script="xml-anonymizer-2.script"
zip-script="zip-anonymizer-2.script"
quarantine="quarantines/ProvenanceRemoverQuarantine" />
<StorageService
name="Storage"
class="org.rsna.ctp.stdstages.FileStorageService"
root="D:/storage"
return-stored-file="no" />
<ExportService
name="PACS Export"
class="org.rsna.ctp.stdstages.DicomExportService"
root="roots/pacs-export"
dest-url="dicom://DestinationAET:ThisAET@ipaddress:port" />
</Pipeline>
</Configuration>
Multiple Pipeline elements may be included, but each must have its own ImportService element, and their ports must not conflict.
Each pipeline stage class has a constructor that is called with its configuration element, making it possible for special processor implementations to be passed additional parameters from the configuration. See #Implementing a Pipeline Stage for details.
2.4 Server
To provide access to the status of the components, the application includes an HTTP server which serves files and provides servlet-like functionality. Files are served from a directory tree whose root is named ROOT. The ROOT directory contains a file, index.html, which provides buttons which link to several servlets providing information about the operation of the program. This file is intended to be configured with logos, additional links, etc., and upgrades do not overwrite it. The standard servlets are:
- ConfigurationServlet displays the contents of the configuration file.
- StatusServlet displays the status of all pipeline stages.
- LogServlet provides web access to all log files in the logs directory.
- QuarantineServlet provides web access to all quarantine directories and their contents.
2.5 The Standard Stages
The application includes several built-in, standard stages which allow most trials to be operated without writing any software. The sections below show all the configuration attributes recognized by the standard stages.
Attributes which specify directories can contain either absolute paths (e.g., D:/TrialStorage) or relative paths (e.g., quarantines/http-import-quarantine). Relative paths are relative to the directory in which the ClinicalTrialProcessor is located.
Most standard stages have attributes which determine which object types are to be accepted by the stage. These attributes are:
- acceptDicomObjects
- acceptXmlObjects
- acceptZipObjects
- acceptFileObjects
The allowed values are "yes" and "no. The default value for all these attributes is "yes".
The DicomImportService and DicomExportService, which are both restricted to DicomObjects, ignore the values of these attributes.
If a standard ImportService receives an object which it is not configured to accept, it quarantines the object, or if no quarantine has been defined for the stage, it discards the object.
If a Processor, StorageService, or ExportService receives an object that it is not configured to accept, it either ignores the object or passes it unmodified to the next pipeline stage. Thus, an anonymizer which is not configured to anonymize XmlObjects receives an XmlObject, it passes it on without anonymization.
2.5.1 HttpImportService
The HttpImportService listens on a defined port for HTTP connections from FieldCenter applications and receives files transmitted using the HTTP protocol with Content-Type equal to application/x-mirc. The configuration element for the HttpImportService is:
<ImportService
name="stage name"
class="org.rsna.ctp.stdstages.HttpImportService"
root="base-directory"
port="7777"
acceptDicomObjects="yes"
acceptXmlObjects="yes"
acceptZipObjects="yes"
acceptFileObjects="yes"
quarantine="quarantine-directory" />
where:
- name is any text to be used as a label on configuration and status pages.
- root is a directory for use by the ImportService for internal storage and queuing.
- port is the port on which the ImportService listens for connections.
- acceptDicomObjects determines whether DicomObjects are to be enqueued when received.
- acceptXmlObjects determines whether XmlObjects are to be enqueued when received.
- acceptZipObjects determines whether ZipObjects are to be enqueued when received.
- acceptFileObjects determines whether FileObjects are to be enqueued when received.
- quarantine is a directory in which the ImportService is to quarantine objects that it receives but is configured not to accept.
2.5.2 DicomImportService
The DicomImportService listens on a defined port for HTTP connections from FieldCenter applications and receives files transmitted using the DICOM protocol. The configuration element for the DicomImportService is:
<ImportService
name="stage name"
class="org.rsna.ctp.stdstages.DicomImportService"
root="base-directory"
ae-title="AE Title"
port="port number" />
where:
- name is any text to be used as a label on configuration and status pages.
- root is a directory for use by the ImportService for internal storage and queuing.
- ae-title is the Application Entity Title of the ImportService's DICOM Storage SCP.
- port is the port on which the ImportService listens for connections.
2.5.3 Anonymizer
The Anonymizer is a processor stage that includes anonymizers for each of the object types which contain defined data. When the anonymizer stage is called to process an object, it calls the anonymizer which is appropriate to the object type. Each anonymizer is configured with a script file. If a script file is either not configured or absent for an object type, objects of that type are returned unmodified. The configuration element for the Anonymizer is:
<Anonymizer
name="stage name"
class="org.rsna.ctp.stdstages.Anonymizer"
acceptDicomObjects="yes"
acceptXmlObjects="yes"
acceptZipObjects="yes"
dicom-script="dicom-anonymizer.properties"
xml-script="xml-anonymizer.script"
zip-script="zip-anonymizer.script"
quarantine="quarantine-directory" />
where:
- name is any text to be used as a label on configuration and status pages.
- acceptDicomObjects determines whether DicomObjects are to be anonymized.
- acceptXmlObjects determines whether XmlObjects are to be anonymized.
- acceptZipObjects determines whether ZipObjects are to be anonymized.
- dicom-script specifies the path to the script for the DICOM anonymizer.
- xml-script specifies the path to the script for the DICOM anonymizer.
- zip-script specifies the path to the script for the Zip anonymizer (which anonymizes the manifest in a ZipObject).
- quarantine is a directory in which the Anonymizer is to quarantine objects that generate quarantine calls during processing.
Notes:
- Any object which is accepted but for which no script has been defined is quarantined. If no quarantine has been defined for the stage, the object is passed on unmodified.
- Since FileObjects do not contain formatted information, the anonymizer does not modify such objects.
2.5.4 FileStorageService
The FileStorageService stores objects in a file system. It automatically defines subdirectories (based on dates) beneath its root directory and populates them accordingly. The configuration element for the StorageService is:
<StorageService
name="stage name"
class="org.rsna.ctp.stdstages.FileStorageService"
root="D:/storage"
acceptDicomObjects="yes"
acceptXmlObjects="yes"
acceptZipObjects="yes"
acceptFileObjects="yes"
return-stored-file="yes"
quarantine="quarantine-directory" />
where:
- name is any text to be used as a label on configuration and status pages.
- root is the base directory of the storage tree.
- acceptDicomObjects determines whether DicomObjects are to be stored.
- acceptXmlObjects determines whether XmlObjects are to be stored.
- acceptZipObjects determines whether ZipObjects are to be stored.
- acceptFileObjects determines whether FileObjects are to be stored.
- return-stored-file specifies whether the original object or a new object pointing to the file in the storage system is to be returned for processing by subsequent stages. Values are "yes" and "no". The default is "yes".
- quarantine is a directory in which the StorageService is to quarantine objects that cannot be stored.
2.5.5 HttpExportService
The HttpExportService queues objects and transmits them via HTTP using the MIRC-defined Content-Type for each object type. The configuration element for the HttpExportServiceis:
<ExportService
name="stage name"
class="org.rsna.ctp.stdstages.HttpExportService"
root="base-directory"
dest-url="http://ipaddress:port/path"
acceptDicomObjects="yes"
acceptXmlObjects="yes"
acceptZipObjects="yes"
acceptFileObjects="yes"
interval="10000" />
where:
- name is any text to be used as a label on configuration and status pages.
- root is the base directory of the queuing storage for the ExportService.
- dest-url specifies the destination system's URL.
- acceptDicomObjects determines whether DicomObjects are to be exported.
- acceptXmlObjects determines whether XmlObjects are to be exported.
- acceptZipObjects determines whether ZipObjects are to be exported.
- acceptFileObjects determines whether FileObjects are to be exported.
- interval is the sleep time (in milliseconds) between polls of the export queue.
Note: The default interval is 10 seconds. The minimum allowed value is one second. The maximum allowed value is 20 seconds.
2.5.6 DicomExportService
The DicomExportService queues objects and transmits them to a DICOM Storage SCP. The configuration element for the DicomExportService is:
<ExportService
name="stage name"
class="org.rsna.ctp.stdstages.DicomExportService"
root="base-directory"
dest-url="dicom://DestinationAET:ThisAET@ipaddress:port"
interval="10000" />
where:
- name is any text to be used as a label on configuration and status pages.
- root is the base directory of the queuing storage for the ExportService.
- dest-url specifies the destination DICOM Storage SCP's URL.
- interval is the sleep time (in milliseconds) between polls of the export queue.
Note: The default interval is 10 seconds. The minimum allowed value is one second. The maximum allowed value is 20 seconds.
2.5.7 DatabaseExportService
The DatabaseExportService queues objects and submits them to a DatabaseAdapter class, which must be written specially for the database in question. The configuration element for the DatabaseExportService is:
<ExportService
name="stage name"
class="org.rsna.trials.DatabaseExportService"
adapter-class="org.myorg.MyDatabaseAdapter"
root="base-directory"
acceptDicomObjects="yes"
acceptXmlObjects="yes"
acceptZipObjects="yes"
acceptFileObjects="yes"
interval-"10000" />
where:
- name is any text to be used as a label on configuration and status pages.
- adapter-class is the class name of the database's adapter class. See #Implementing a DatabaseAdapter for more information.
- root is the base directory of the queuing storage for the ExportService.
- acceptDicomObjects determines whether DicomObjects are to be exported.
- acceptXmlObjects determines whether XmlObjects are to be exported.
- acceptZipObjects determines whether ZipObjects are to be exported.
- acceptFileObjects determines whether FileObjects are to be exported.
- interval is the sleep time (in milliseconds) between polls of the export queue.
Note: The default interval is 10 seconds. The minimum allowed value is one second. The maximum allowed value is 20 seconds.
3 Extending ClinicalTrialProcessor
ClinicalTrialProcessor is designed to be extended with pipeline stages of new types. Stages implement one or more Java interfaces, so it is necessary to get the source code for ClinicalTrialProcessor in order to extend it, even though in principle you don't need to modify the code itself.
3.1 Obtaining the Source Code
The software for ClinicalTrialProcessor is open source. All the software written by the RSNA for the project is released under the RSNA Public License. It is maintained on a CVS server at RSNA headquarters. To obtain the source code, configure a CVS client as follows:
Protocol: Password server (:pserver) Server: mirc.rsna.org Port: 2401 Repository folder: /RSNA User name: cvs-reader Password: cvs-reader Module: ClinicalTrialProcessor
Together, this results in the following CVSROOT (which is constructed automatically if you use something like Tortoise-CVS on a Windows system):
- :pserver:cvs-reader@mirc.rsna.org:2401/RSNA
This account has read privileges, but it cannot write into the repository, so it can check out but not commit. If you wish to be able to commit software to the CVS library, contact the MIRC project manager.
3.2 Building the Software
When you check out the ClinicalTrialProcessor module from CVS, you obtain a directory tree full of the sources and libraries for building the application. The top of the directory tree is ClinicalTrialProcessor. It contains several subdirectories. The source code is in the source directory, which has two subdirectories, one each for the Java sources and the files required by the application.
Running ClinicalTrialProcessor requires Java 1.5 JDK and the JAI ImageIO Tools.
The Ant build file for ClinicalTrialProcessor is in the ClinicalTrialProcessor directory and is called build.xml. To build the software on a Windows system, launch a command window, navigate to the ClinicalTrialProcessor directory, and enter ant all.
The build file contains several targets. The all target does a clean build of everything, including the Javadocs, which are put into the documentation directory. The Javadocs can be accessed with a browser by opening the file:
- ClinicalTrialProcessor/documentation/index.html
The default target, ctp-installer, just builds the application and places the installer in the products directory.
3.3 The Object Classes
ClinicalTrialProcessor provides four classes to encapsulate files of various types. The classes are located in the org.rsna.ctp.objects package:
- DicomObject - a DICOM dataset
- XmlObject - an XML file containing identifiers relating the data to the trial and the trial subject
- ZipObject - a zip file containing a manifest.xml file providing identifiers relating the zip file's contents to the trial and the trial subject
- FileObject - a generic, unformatted file of unknown contents
Each class provides methods allowing pipeline stages or database adapters to access the internals of the object without having to know how to parse it. See the Javadocs for a list of all the methods provided by these classes.
3.4 Implementing a Pipeline Stage
To be recognized as a pipeline stage, a class must implement the org.rsna.ctp.pipeline.PipelineStage interface. An abstract class, org.rsna.ctp.pipeline.AbstractStage, is provided to supply some of the basic methods required by the PipelineStage interface. All the standard stages extend this class.
Each stage type must also implement its own interface. The interfaces are:
- org.rsna.ctp.pipeline.ImportService
- org.rsna.ctp.pipeline.Processor
- org.rsna.ctp.pipeline.StorageService
- org.rsna.ctp.pipeline.ExportService
The Javadocs explain the methods which must be implemented in each stage type.
Each stage class must have a constructor which takes its configuration file XML Element as its argument. The constructor must obtain any configuration information it requires from the element. While it is not required that all configuration information be placed in attributes of the element, the getConfigHTML method provided by AbstractStage expects it, and if you choose to encode configuration information in another way, you must override the getConfigHTML method.
3.5 Implementing a DatabaseAdapter
Information on implementing a DatabaseAdapter can be found in Implementing an External Database Interface for MIRC Clinical Trials. It is important to note, however, that since the object types for ClinicalTrialProcessor are not in the same packages as those for MIRC, the DatabaseAdapter must be compiled as part of the ClinicalTrialProcessor build.
4 Open Issues
- The Update Service
- The Central Remapping Service