The CTP Module

UNDER CONSTRUCTION

The CTP module (CTP.jar) contains the code of the CTP application, including all the standard stages and plugins. This article describes the theory of operation of the packages contained in the CTP module. The CTP module is used in CTP, TFS, CTPClient, ISN, and other MIRC software. The intended audience for this article is software engineers extending or maintaining any of that software.

See Setting Up a MIRC Development Environment for information on obtaining and building the VTP module.

The CTP module contains five packages:

• org.rsna.ctp contains the CTP main class and the Configuration singleton class.
• org.rsna.ctp.objects contains classes that implement the four types of objects that CTP can process.
• org.rsna.ctp.pipeline contains the Pipeline class, interfaces that define the types of pipeline stages, and a set of abstract classes that provide starting points for developing each pipeline stage type.
• org.rsna.ctp.plugin contains the interface that defines a plugin and an abstract class that provides a starting point for developing a plugin.
• org.rsna.ctp.servlets contains the standard servlets used in CTP.
• org.rsna.ctp.stdstages contains the standard pipeline stages included in CTP.

Each of these packages will be described in the sections below, with example code where necessary to illustrate the theory of operation. It will be helpful to get the source code and build it so you can reference the Javadocs as you go along.

In addition to the packages listed above, the source code tree also contains packages that are part of helper applications used in CTP:

• org.rsna.installer contains the the installer program for CTP, TFS, and ISN.
• org.rsna.launcher contains the program used to manually launch CTP TFS, and ISN.
• org.rsna.runner contains the program that can be used in a batch job to start or stop CTP, TFS, and ISN. This program is typically used to implement a CTP Linux service.

1 Overview

CTP is a framework for pipelines and plugins.

• A pipeline is an ordered sequence of processing stages. Data objects enter a pipeline at the head end and encounter the stages in sequence.
• A pipeline stage performs some task on a data object when it arrives at the stage as it flows down the pipe.
• A plugin is a module that adds capability to the program outside the scope of a pipeline.

CTP includes a servlet container to provide a user interface for configuring certain features and for accessing data.

CTP is installed using the CTP-installer.jar program that is created when CTP is built. See Setting Up a MIRC Development Environment for information on building CTP. The installer creates a folder called CTP and puts all the necesssary files and subdirectories in it.

CTP is configured by a single XML file. CTP imposes no limits on the number of pipelines or plugins that may be configured or on the number of stages that may be configured in a single pipe. The structure of the configuration file is described in detail in CTP-The RSNA Clinical Trial Processor.

CTP can be run manually through the Launcher.jar program, or it can be run as a Windows or Linux service. See Running CTP as a Windows Service or Running CTP as a Linux Service for details.

The Launcher.jar program is located in the CTP directory. All the modules required for running CTP are located in the CTP/libraries directory or one of its subdirectories. The manifest in the CTP.jar module includes a classpath listing only the util.jar module. When CTP starts, it calls ClasspathUtil.addJars to add all the jar files in the CTP/libraries directory tree to the classpath. By constructing the classpath in this way, developers can add pipeline stages and plugins into CTP without having to modify the CTP build itself.

After setting up the classpath, CTP then loads the configuration. The configuration is encapsulated in the singleton org.rsna.ctp.Configuration class. The Configuration class parses the configuration file and loads the plugins and pipelines. Pipelines load their stages.

Pipelines, stages, and plugins have constructors that take their configuration file XML Element as an argument, and they obtain all their configuration information from it. When these objects are instantiated, the Configuration object is not yet constructed; therefore, the constructors are just for instantiation of the objects themselves, and they may not reference the Configuration object or any other stages or plugins.

After loading the configuration and instantiating the plugins, pipelines, and stages, CTP instantiates the servlet container (org.rsna.server.HttpServer in the util module). This must be done after the configuration is loaded because the configuration determines whether to run the server on HTTP or HTTPS. Because the server is not available when the plugins and stages are instantiated, those classes that add servlets into the server must not do so in their constructors.

Once the configuration has been loaded and the server has been instantiated, CTP calls the start method of the Configuration class. That, in turn, results in calling the start methods of the plugins and stages. The start methods can reference anything in the Configuration, including other stages and plugins. This two-phase startup is necessary because certain stages are designed to interact with other stages and plugins, so everything has to be constructed before any such interaction can be allowed to occur. For the same reason, plugins and stages that add servlets to the server must do so in their start methods.

CTP calls the start methods of all the plugins first, followed by the start methods of all the pipelines. The start methods of pipelines call the start methods of their stages. This sequence is important because a stage may depend on a plugin, and therefore a plugin must be completely running before it is referenced.

Pipelines are implemented as asynchronous threads. A pipeline object maintains an ordered list of all its stages. There are four types of stages:

• ImportService
• Processor
• StorageService
• ExportService

These stage types are described in CTP-The RSNA Clinical Trial Processor and Extending CTP. Each stage implements its specific interface. Data objects enter a pipe through an ImportService stage. A pipeline object maintains a separate list of its ImportService stages. When a pipeline is ready to process a new data object, it polls its ImportServices in turn until it finds one that has a data object available. As data objects flow down the pipe, they skip any ImportServices; thus, although ImportServices are configured into the pipeline, they can be thought of as being separate from it, and the order in which they appear - and even where they appear - in the pipeline is not important. (Nevertheless, for human readability, it is best to put the ImportServices first in the pipe.)

When running CTP as a Windows service, the Windows Service Manager calls the startService method in org.rsna.ctp.ClinicalTrialProcessor to start the service and the stopService method to stop it. The stopService method makes an HTTP or HTTPS connection (as defined in the configuration) to the org.rsna.ctp.servlets.ShutdownServlet servlet to actually shut down. The ShutdownServlet first stops the pipelines and then the plugins in order to ensure that the stages have access to the plugins throughout the shutdown process.

2 org.rsna.ctp

This package contains the CTP main class (org.rsna.ctp.ClinicalTrialProcessor) and the Configuration class (org.rsna.ctp.Configuration) .

3 org.rsna.ctp.objects

This package defines the four object types that CTP recognizes.

Note: To be useful in a clinical trial, a data object must contain a set of identifiers that allow the object to be related to other data objects in the trial.

• FileObject represents a file of unknown format. It is the parent class of the other object types. It supplies generic file handling methods. Since the FileObject cannot know anything about the contents of the file, it cannot provide the IDs necessary for a clinical trial; therefore, a FileObject by by itself is not useful. Its utility is only as the parent class of the other object types.
• DicomObject represents a DICOM dataset. It contains a great many methods that provide access to the individual elements in the dataset, including the IDs and UIDs defined in the DICOM standard that are important in a clinical trial. It also provides methods for creating browser-viewable images from the pixels stored in the dataset.
• XmlObject represents an XML file. It provides generic methods for accessing the important identifiers in a clinical trial. Since the schemas of XML data objects used in clinical trials vary, these methods are designed to look in a sequence of common places to find the IDs. The sequences are described in the Javadocs.
• ZipObject represents a zip file. In addition to whatever other files it may contain, a ZipObject must contain a manifest.xml file that carries the necessary clinical trial IDs. CTP treats ZipObjects as atomic data objects, although the ZipObject class provides methods for obtaining the individual files it contains.

The DicomObject, XmlObject, and ZipObject classes all contain matches methods that compute a boolean result from a script that interrogates the object. These methods are used in many pipeline stages to determine whether an object is to be processed. See The CTP DICOM Filter and The CTP XML and Zip Filters articles for information on the script languages.

Because DicomObjects are so critical to imaging trials, and because various manufacturers implement DICOM in various ways, the DicomFilter pipeline stage provides a way to list the operand values it sees when computing its boolean result. To pass those values to the DicomFilter, the matches method of the DicomObject returns a MatchResult object that encapsulates the operand values and the boolean result. (The matches methods of the other two object types just return the boolean result.)

The MatchResult class is also contained in this package, along with a SopClass class that is used by DicomObject to detect certain types of SOP Classes.

4 org.rsna.ctp.pipeline

This package contains:

• the Pipeline class
• the interface definitions for the four stage types
• abstract classes to implement stages
• the QueueManager class used by all the stages that maintain queues of data objects for import or export
• the typesafe enum Status class for returning results from export operations
• the Quarantine class for managing files that are removed from a pipe by a stage

The Status servlet displays a web page summarizing the status of all the plugins, pipelines, and stages. It obtains status information by calling the getStatusHTML methods of all the objects in the configuration. Similarly, the Configuration servlet displays the configuration parameters, and obtains that infomation by calling the getConfigHTML methods. The AbstractPipelineStage provides implementations of these methods to simplify the implementation of stages.

5 org.rsna.ctp.plugin

This package contains:

• the interface definition for a plugin
• an abstract class to implement plugins

As in the case of the AbstractPipelineStage, the AbstractPlugin class provides implementations of the getStatusHTML and getConfigHTML methods to simplify the implementation of plugins.

6 org.rsna.ctp.servlets

This package contains servlets that are used either for the administration of CTP or for access to data maintained by specific plugins or stages. These are the servlets that are loaded by CTP; some of them are in the CTP module, the rest are in the util module:

Servlet	Context	Description
ApplicationServer	webstart	Automatically generates jnlp content from query parameters and launches webstart applications.
AuditLogServlet	{id attribute}	Searches the AuditLog database, providing access to individual entries.
ConfigurationServlet	configuration	Displays the configuration on a web page.
DBVerifierServlet	databaseverifier	Provides access to the database maintained by the DatabaseVerifier stage. This stage is used to track whether objects submitted to sites running the NBIA software (for example, The Cancer Imaging Archive) have been successfully received and added to the collection.
DecipherServlet	decipher	Provides a restful web service to decipher encrypted elements in DICOM objects.
DicomAnonymizerServlet	daconfig	Provides a web-based editor for DicomAnonymizer scripts.
IDMapServlet	idmap	Searches the IDMap database, providing translations between PHI and anonymized values for certain IDs and UIDs.
LoginServlet	login	Provides a UI for logging into the server.
LoggerLevelServlet	level	Provides a UI for controlling the log4j logger levels while CTP is running.
LogServlet	logs	Provides a UI for viewing the CTP logs (in CTP/logs).
LookupServlet	lookup	Provides a web-based editor for DicomAnonymizer lookup tables.
ObjectTrackerServlet	objecttracker	Searches the ObjectTracker database, providing access to IDs and UIDs organized by patient, study date, study, and series.
QuarantineServlet	quarantines	Supports management of the quarantines.
ScriptServlet	script	Provides a web-based editor for filter scripts.
ShutdownServlet	shutdown	Gracefully shuts down CTP.
StatusServlet	status	Displays the current status of all the plugins, pipelines, and stages.
SysPropsServlet	system	Provides a UI for viewing the Java system properties.
UserManager	users	Provides a web-based editor for creating or modifying user accounts on the server.
UserServlet	user	Provides a restful web service for obtaining information about a user.

Almost all the servlets require that the user be authenticated and have the admin role. The exceptions are:

• ConfigurationServlet and StatusServlet, which are accessable for unauthenticated users.
• QuarantineServlet, which requires that the user have the qadmin role.

To honor a shutdown request, the ShutdownServlet imposes these conditions:

• If the request comes from the service manager on the same computer that is running CTP (as in a Windows service shutdown), it is honored.
• If the request comes from an authenticated user who has the shutdown role, it is honored.
• If the request comes from an authenticated user on the same computer that is running CTP, it is honored.

Parenthetically, the UserManagerServlet (in the util module) does not allow a user to be granted the shutdown role unless the user granting the shutdown role possesses the shutdown role.

7 org.rsna.ctp.stdplugins

There are two standard plugins:

• AuditLog provides a logging mechanism for use in 21CFR11-compliant clinical trials.
• Redirector runs an HTTP service that redirects an HTTP connection to an HTTPS port.

The AuditLog class maintains a JDBM database of entries that are searchable through the AuditLogServlet. To allow for multiple AuditLog plugins to appear in a single configuration, the AuditLog installs the servlet on a context that is the same as the id attribute in its configuration element