View source for The CTP Module

{| align="right"
  | __TOC__
  |}
<b><font size=5>UNDER CONSTRUCTION</font></b>

The CTP module (<b><tt>CTP.jar</tt></b>) contains the code of the CTP application, including all the standard stages and plugins. This article describes the theory of operation of the packages contained in the CTP module. The CTP module is used in CTP, TFS, CTPClient, ISN, and other MIRC software. The intended audience for this article is software engineers extending or maintaining any of that software.

See [[Setting Up a MIRC Development Environment]] for information on obtaining and building the VTP module.

The CTP module contains five packages:
* <b><tt>org.rsna.ctp</tt></b> contains the CTP main class and the Configuration singleton class.
* <b><tt>org.rsna.ctp.objects</tt></b> contains classes that implement the four types of objects that CTP can process.
* <b><tt>org.rsna.ctp.pipeline</tt></b> contains the Pipeline class, interfaces that define the types of pipeline stages, and a set of abstract classes that provide starting points for developing each pipeline stage type.
* <b><tt>org.rsna.ctp.plugin</tt></b> contains the interface that defines a plugin and an abstract class that provides a starting point for developing a plugin.
* <b><tt>org.rsna.ctp.servlets</tt></b> contains the standard servlets used in CTP.
* <b><tt>org.rsna.ctp.stdstages</tt></b> contains the standard pipeline stages included in CTP.

Each of these packages will be described in the sections below, with example code where necessary to illustrate the theory of operation. It will be helpful to get the source code and build it so you can reference the Javadocs as you go along.

In addition to the packages listed above, the source code tree also contains packages that are part of helper applications used in CTP:
* <b><tt>org.rsna.installer</tt></b> contains the the installer program for CTP, TFS, and ISN.
* <b><tt>org.rsna.launcher</tt></b> contains the program used to manually launch CTP TFS, and ISN.
* <b><tt>org.rsna.runner</tt></b> contains the program that can be used in a batch job to start or stop CTP, TFS, and ISN. This program is typically used to implement a CTP Linux service.

==Overview==
CTP is a framework for pipelines and plugins. 
* A pipeline is an ordered sequence of processing stages. Data objects enter a pipeline at the head end and encounter the stages in sequence. 
* A pipeline stage performs some task on a data object when it arrives at the stage as it flows down the pipe. 
* A plugin is a module that adds capability to the program outside the scope of a pipeline. 

CTP includes a servlet container to provide a user interface for configuring certain features and for accessing data.

CTP is installed using the <b><tt>CTP-installer.jar</tt></b> program that is created when CTP is built. See [[Setting Up a MIRC Development Environment]] for information on building CTP. The installer creates a folder called <b><tt>CTP</tt></b> and puts all the necesssary files and subdirectories in it.

CTP is configured by a single XML file. CTP imposes no limits on the number of pipelines or plugins that may be configured or on the number of stages that may be configured in a single pipe. The structure of the configuration file is described in detail in [[CTP-The RSNA Clinical Trial Processor]].

CTP can be run manually through the <b><tt>Launcher.jar</tt></b> program, or it can be run as a Windows or Linux service. See [[Running CTP as a Windows Service]] or [[Running CTP as a Linux Service]] for details.

The <b><tt>Launcher.jar</tt></b> program is located in the <b><tt>CTP</tt></b> directory. All the modules required for running CTP are located in the <b><tt>CTP/libraries</tt></b> directory or one of its subdirectories. The manifest in the <b><tt>CTP.jar</tt></b> module includes a classpath listing only the <b><tt>util.jar</tt></b> module. When CTP starts, it calls <b><tt>ClasspathUtil.addJars</tt></b> to add all the jar files in the <b><tt>CTP/libraries</tt></b> directory tree to the classpath. By constructing the classpath in this way, developers can add pipeline stages and plugins into CTP without having to modify the CTP build itself.

After setting up the classpath, CTP then loads the configuration. The configuration is encapsulated in the singleton <b><tt>org.rsna.ctp.Configuration</tt></b> class. The Configuration class parses the configuration file and loads the plugins and pipelines. Pipelines load their stages. 

Pipelines, stages, and plugins have constructors that take their configuration file XML Element as an argument, and they obtain all their configuration information from it. When these objects are instantiated, the Configuration object is not yet constructed; therefore, the constructors are just for instantiation of the objects themselves, and they may not reference the Configuration object or any other stages or plugins.

After loading the configuration and instantiating the plugins, pipelines, and stages, CTP instantiates the servlet container (<b><tt>org.rsna.server.HttpServer</tt></b> in the util module). This must be done after the configuration is loaded because the configuration determines whether to run the server on HTTP or HTTPS. Because the server is not available when the plugins and stages are instantiated, those classes that add servlets into the server must not do so in their constructors.

Once the configuration has been loaded and the server has been instantiated, CTP calls the <b><tt>start</tt></b> method of the <b><tt>Configuration</tt></b> class. That, in turn, results in calling the <b><tt>start</tt></b> methods of the plugins and stages. The <b><tt>start</tt></b> methods can reference anything in the Configuration, including other stages and plugins. This two-phase startup is necessary because certain stages are designed to interact with other stages and plugins, so everything has to be constructed before any such interaction can be allowed to occur. For the same reason, plugins and stages that add servlets to the server must do so in their <b><tt>start</tt></b> methods.

CTP calls the <b><tt>start</tt></b> methods of all the plugins first, followed by the <b><tt>start</tt></b> methods of all the pipelines. The <b><tt>start</tt></b> methods of pipelines call the <b><tt>start</tt></b> methods of their stages. This sequence is important because a stage may depend on a plugin, and therefore a plugin must be completely running before it is referenced.

Pipelines are implemented as asynchronous threads. A pipeline object maintains an ordered list of all its stages. There are four types of stages:
* ImportService
* Processor
* StorageService
* ExportService

These stage types are described in [[CTP-The RSNA Clinical Trial Processor]] and [[Extending CTP]]. Each stage implements its specific interface. Data objects enter a pipe through an ImportService stage. A pipeline object maintains a separate list of its ImportService stages. When a pipeline is ready to process a new data object, it polls its ImportServices in turn until it finds one that has a data object available. As data objects flow down the pipe, they skip any ImportServices; thus, although ImportServices are configured into the pipeline, they can be thought of as being separate from it, and the order in which they appear - and even where they appear - in the pipeline is not important. (Nevertheless, for human readability, it is best to put the ImportServices first in the pipe.)

When running CTP as a Windows service, the Windows Service Manager calls the <b><tt>startService</tt></b> method in <b><tt>org.rsna.ctp.ClinicalTrialProcessor</tt></b> to start the service and the <b><tt>stopService</tt></b> method to stop it. The <b><tt>stopService</tt></b> method makes an HTTP or HTTPS connection (as defined in the configuration) to the <b><tt>org.rsna.ctp.servlets.ShutdownServlet</tt></b> servlet to actually shut down. The <b><tt>ShutdownServlet</tt></b> first stops the pipelines and then the plugins in order to ensure that the stages have access to the plugins throughout the shutdown process.

==<b><tt>org.rsna.ctp</tt></b>==
This package contains the CTP main class (<b><tt>org.rsna.ctp.ClinicalTrialProcessor</tt></b>) and the Configuration class (<b><tt>org.rsna.ctp.Configuration</tt></b>) .

==<b><tt>org.rsna.ctp.objects</tt></b>==
This package defines the four object types that CTP recognizes.

:<em>Note: To be useful in a clinical trial, a data object must contain a set of identifiers that allow the object to be related to other data objects in the trial.</em>

* FileObject represents a file of unknown format. It is the parent class of the other object types. It supplies generic file handling methods. Since the FileObject cannot know anything about the contents of the file, it cannot provide the IDs necessary for a clinical trial; therefore, a FileObject by by itself is not useful. Its utility is only as the parent class of the other object types.
* DicomObject represents a DICOM dataset. It contains a great many methods that provide access to the individual elements in the dataset, including the IDs and UIDs defined in the DICOM standard that are important in a clinical trial. It also provides methods for creating browser-viewable images from the pixels stored in the dataset. 
* XmlObject represents an XML file. It provides generic methods for accessing the important identifiers in a clinical trial. Since the schemas of XML data objects used in clinical trials vary, these methods are designed to look in a sequence of common places to find the IDs. The sequences are described in the Javadocs.
* ZipObject represents a zip file. In addition to whatever other files it may contain, a ZipObject must contain a <b><tt>manifest.xml</tt></b> file that carries the necessary clinical trial IDs. CTP treats ZipObjects as atomic data objects, although the ZipObject class provides methods for obtaining the individual files it contains.

The DicomObject, XmlObject, and ZipObject classes all contain <b><tt>matches</tt></b> methods that compute a boolean result from a script that interrogates the object. These methods are used in many pipeline stages to determine whether an object is to be processed. See [[The CTP DICOM Filter]] and [[The CTP XML and Zip Filters]] articles for information on the script languages.

Because DicomObjects are so critical to imaging trials, and because various manufacturers implement DICOM in various ways, the DicomFilter pipeline stage provides a way to list the operand values it sees when computing its boolean result. To pass those values to the DicomFilter, the <b><tt>matches</tt></b> method of the DicomObject returns a <b><tt>MatchResult</tt></b> object that encapsulates the operand values and the boolean result. (The <b><tt>matches</tt></b> methods of the other two object types just return the boolean result.) 

The <b><tt>MatchResult</tt></b> class is also contained in this package, along with a <b><tt>SopClass</tt></b> class that is used by DicomObject to detect certain types of SOP Classes.

==<b><tt>org.rsna.ctp.pipeline</tt></b>==
This package contains:
* the <b><tt>Pipeline</tt></b> class
* the interface definitions for the four stage types
* abstract classes to implement stages
* the <b><tt>QueueManager</tt></b> class used by all the stages that maintain queues of data objects for import or export
* the typesafe enum <b><tt>Status</tt></b> class for returning results from export operations
* the <b><tt>Quarantine</tt></b> class for managing files that are removed from a pipe by a stage

The Status servlet displays a web page summarizing the status of all the plugins, pipelines, and stages. It obtains status information by calling the <b><tt>getStatusHTML</tt></b> methods of all the objects in the configuration. Similarly, the Configuration servlet displays the configuration parameters, and obtains that infomation by calling the <b><tt>getConfigHTML</tt></b> methods. The <b><tt>AbstractPipelineStage</tt></b> provides implementations of these methods to simplify the implementation of stages.

==<b><tt>org.rsna.ctp.plugin</tt></b>==
This package contains:
* the interface definition for a plugin
* an abstract class to implement plugins

As in the case of the <b><tt>AbstractPipelineStage</tt></b>, the <b><tt>AbstractPlugin</tt></b> class provides implementations of the <b><tt>getStatusHTML</tt></b> and <b><tt>getConfigHTML</tt></b> methods to simplify the implementation of plugins.

==<b><tt>org.rsna.ctp.servlets</tt></b>==
This package contains servlets that are used either for the administration of CTP or for access to data maintained by specific plugins or stages. These are the servlets that are loaded by CTP, along with their contexts:

{| class="wikitable"
|-
!Servlet
!Context
!Module
!Description

|-
| ApplicationServer
| webstart
| util
| Automatically generates <b><tt>jnlp</tt></b> content from query parameters and launches webstart applications.
|-
| AuditLogServlet
| {id attribute}
| CTP
| Searches the AuditLog database, providing access to individual entries.
|-
| ConfigurationServlet
| configuration
| CTP
| Displays the configuration on a web page.
|-
| DBVerifierServlet
| databaseverifier
| CTP
| Provides access to the database maintained by the DatabaseVerifier stage. This stage is used to track whether objects submitted to sites running the NBIA software (for example, The Cancer Imaging Archive) have been successfully received and added to the collection.
|-
| DecipherServlet
| decipher
| CTP
| Provides a restful web service to decipher encrypted elements in DICOM objects.
|-
| DicomAnonymizerServlet
| daconfig
| CTP
| Provides a web-based editor for DicomAnonymizer scripts.
|-
| IDMapServlet
| idmap
| CTP
| Searches the IDMap database, providing translations between PHI and anonymized values for certain IDs and UIDs.
|-
| LoggerLevelServlet
| level
| util
| Provides a UI for controlling the <b><tt>log4j</tt></b> logger levels while CTP is running.
|-
| LoginServlet
| login
| util
| Provides a UI for logging into the server.
|-
| LogServlet
| logs
| util
| Provides a UI for viewing the CTP logs (in <b><tt>CTP/logs</tt></b>).
|-
| LookupServlet
| lookup
| CTP
| Provides a web-based editor for DicomAnonymizer lookup tables.
|-
| ObjectTrackerServlet
| objecttracker
| CTP
| Searches the ObjectTracker database, providing access to IDs and UIDs organized by patient, study date, study, and series.
|-
| QuarantineServlet
| quarantines
| CTP
| Supports management of the quarantines.
|-
| ScriptServlet
| script
| CTP
| Provides a web-based editor for filter scripts.
|-
| ShutdownServlet
| shutdown 
| CTP
| Gracefully shuts down CTP.
|-
| StatusServlet
| status
| CTP
| Displays the current status of all the plugins, pipelines, and stages.
|-
| SysPropsServlet
| system
| util
| Provides a UI for viewing the Java system properties.
|-
| UserManagerServlet
| users
| util
| Provides a web-based editor for creating or modifying user accounts on the server.
|-
| UserServlet
| user
| util
| Provides a restful web service for obtaining information about a user.
|}

Each servlet is responsible for enforcing whatever user requirements are necessary for the function it performs. Thus, some servlets are accessable by unauthenticated users while others require that the user be authenticated and possess specific roles.

To honor a shutdown request, the ShutdownServlet imposes these conditions:
* If the request comes from the service manager on the same computer that is running CTP (as in a Windows service shutdown), it is honored.
* If the request comes from an authenticated user who has the <b><tt>shutdown</tt></b> role, it is honored.
* If the request comes from an authenticated user on the same computer that is running CTP, it is honored.

Parenthetically, the UserManagerServlet (in the util module) does not allow a user to be granted the <b><tt>shutdown</tt></b> role unless the user granting the <b><tt>shutdown</tt></b> role possesses the <b><tt>shutdown</tt></b> role.

==<b><tt>org.rsna.ctp.stdplugins</tt></b>==
There are two standard plugins:
* AuditLog provides a logging mechanism for use in 21CFR11-compliant clinical trials.
* Redirector runs an HTTP service that redirects an HTTP connection to an HTTPS port.

The <b><tt>AuditLog</tt></b> class maintains a JDBM database of entries that are searchable through the AuditLogServlet. To allow for multiple AuditLog plugins to appear in a single configuration, the AuditLog installs the servlet on a context that is the same as the <b><tt>id</tt></b> attribute in its configuration element

==<b><tt>org.rsna.ctp.stdstages</tt></b>==

==<b><tt>config</tt></b>==
The <b><tt>source/config</tt></b> directory in the CTP development tree contains the default configuration file (<b><tt>config.xml</tt></b>). This file is only copied to the disk by the CTP-installer.jar program during a new CTP installation. The file is just an example; it isn't designed to do anything particularly useful.

==<b><tt>files</tt></b>==
The <b><tt>source/files</tt></b> directory in the CTP development tree contains the files that are copied to the disk by the CTP-installer.jar program when CTP is either installed or upgraded. These files are therefore overwritten during an upgrade. 

===<b><tt>examples</tt></b>===
Certain pipeline stages are driven by scripts. When such a stage is first instantiated, it looks for the script file defined in its configuration element. If the file is not present on the disk, it copies the appropriate file from the <b><tt>examples</tt></b> directory as the starting point.

This directory also contains an <b><tt>example-config.xml</tt></b> file that CTP uses as the starting point if it cannot find the <b><tt>config.xml</tt></b> file when the program starts.

===<b><tt>pages</tt></b>===
The <b><tt>pages</tt></b> directory contains XSL files that are used by the StorageServlet to create web pages for viewing studies stored by the FileStorageService stage.

===<b><tt>profiles</tt></b>===
The <b><tt>profiles</tt></b> directory contains the DicomAnonymizer script files that implement the DICOM Supplement 142 De-identification These are contained in the <b><tt>dicom</tt></b> subdirectory. These profiles are overwritten by the CTP-installer.

In a running system, the admin user can create and store de-identification profiles through the DicomAnonymizerServlet. Such profiles are saved in the <b><tt>saved</tt></b> subdirectory, which does not exist in the development tree and which is created by the DicomAnonymizerServlet when necessary. Files in this directory are not overwritten by the CTP-installer. 

The DicomAnonymizerServlet does not allow the user to overwrite a Supplement 142 profile.

===<b><tt>scripts</tt></b>===
The <b><tt>profiles</tt></b> directory contains standard scripts used in various applications. Although these scripts can be edited by the admin user, these are overwritten by the CTP-installer, so it is generally best for the user to create a new script file if it is to bed preserved.

===<b><tt>ROOT</tt></b>===
This is the root of the servlet container. It is the default root of the contexts of all servlets. (Because a servlet may move its root in its constructor, however, be careful to look for this possibility when reading a servlet's code. Almost all the TFS servlets move their roots so they lie under the directory defined in the <b><tt>root</tt></b> attribute of the <b><tt>MIRC</tt></b> configuration element.)

Note that the <b><tt>Servlet</tt></b> base class is responsible for file serving, so a servlet serves files from whatever root it is configured to use. See [[The_Util_Module#org.rsna.servlets|org.rsna.servlets] description in the Util wiki article for more details.

The only file included in the <b><tt>ROOT</tt></b> directory is <b><tt>example-index.html</tt></b>. When the server receives a request for <b><tt>index.html</tt></b> and it cannot find the file, it copies the <b><tt>example-index.html</tt></b> file into place and serves it. The reason for this mechanism is to allow the CTP-installer to overwrite the <b><tt>example-index.html</tt></b> file while preserving any changes the site admin user may have made in the <b><tt>index.html</tt></b>. In practice, nobody in the field seems to use this feature.

===<b><tt>linux</tt></b>===
This directory contains the <b><tt>ctpService.sh</tt></b> file that is used to configure Linux to run CTP as a service.  See [[Running CTP as a Linux Service]] for details.

===<b><tt>windows</tt></b>===
This directory contains the files used to install and uninstall CTP as a Windows service:

* <b><tt>install.bat</tt></b> is the install batch job. This file is constructed by the CTP-installer program using the specific parameters of the installation (location on disk, etc.). This file is overwritten by the installer.

* <b><tt>uninstall.bat</tt></b> is the uninstall batch job. This file is overwritten by the installer.

* <b><tt>CTP.exe</tt></b> is the Windows service runner for CTP. This file is overwritten by the installer.

* <b><tt>CTPw.exe</tt></b> is the Windows service monitor for CTP. It allows the Windows administrator to adjust the memory allocation and many other parameters of the service. This file is overwritten by the installer.

For more information, see [[Running CTP as a Windows Service]].

==<b><tt>resources</tt></b>==
Files in this directory are included in the <b><tt>CTP.jar</tt></b> file. They are loaded from the classpath when they are required. They are XSL, CSS, and Javascript files used by various servlets.