Difference between revisions of "MIRC CTP"

From MircWiki
Jump to: navigation, search
(ClinicalTrialProcessor)
Line 47: Line 47:
 
#Support for the FieldCenter Update Service client.
 
#Support for the FieldCenter Update Service client.
  
==Implementation==
+
===Pipelines===
The core of the proposed implementation is a manager that orchestrates one or more pipelines.
+
A pipeline is a manager that moves data objects through a sequence of processing stages. Each stage in the pipeline performs a specific function on one or more of the four basic object types supported by MIRC:
 
+
===Pipeline===
+
A Pipeline is a manager that moves data objects through a sequence of processing stages. Each stage in the pipeline performs a specific function on one or more of the four basic object types supported by MIRC:
+
 
:*FileObject
 
:*FileObject
 
:*DicomObject
 
:*DicomObject
 
:*XmlObject
 
:*XmlObject
 
:*ZipObject
 
:*ZipObject
Each stage is an implementation of a specific interface. All stages expose certain basic methods that provide status information as well as access to the stage's output object. Each Pipeline contains one ImportService as its first component. Each pipeline stage is provided access to a Quarantine directory, which may be unique to the stage, into which the Pipeline will place objects that are rejected by a stage, thus aborting further processing. At the end of the pipeline, the manager calls the ImportService to remove the processed object from its queue.
+
Each pipeline must contain one ImportService as its first stage. Each pipeline stage is provided access to a quarantine directory, which may be unique to the stage or shared with other stages, into which the pipeline places objects that are rejected by a stage, thus aborting further processing. At the end of the pipeline, the manager calls the ImportService to remove the processed object from its queue.
  
 
====ImportService====
 
====ImportService====
Line 65: Line 62:
  
 
====Processor====
 
====Processor====
A Processor is a generic class to perform some kind of processing on an object. It is not queued. A processor exposes methods with calling signatures that are unique to the object type. In the context of the current MIRC implementation, a Preprocessor is a Processor, as is an Anonymizer. The result of a processing stage is an object that is passed to the next stage in the pipeline.
+
A Processor performs some kind of processing on an object. It is not queued. A processor exposes methods with calling signatures that are unique to the object type. In the context of the current MIRC implementation, a Preprocessor is a Processor, as is an Anonymizer. The result of a processing stage is an object that is passed to the next stage in the pipeline.
  
 
====ExportService====
 
====ExportService====
Line 71: Line 68:
  
 
===Configuration===
 
===Configuration===
The configuration is specified by an XML file as in this example:
+
The ClinicalTrialProcessor configuration is specified by an XML file. There can be one <b>Server</b> element specifying the port on which the HTTP server is to operate, and multiple <b>Pipeline</b> elements, each specifying the stages which comprise it. The name of the element defining a stage is irrelevant and can be chosen for readability; each stage in a pipeline is actually defined by its Java class, specified in the <b>class</b> attribute. Stages are loaded automatically when the program starts, and the loader tests the stage's class to see what kind of stage it represents. It is possible to extend the application beyond the pre-defined stages available in the implementation as described in the section on Extending ClinicalTrialProcessor.
 +
 
 +
The following is an example of a simple configuration with one pipeline which receives objects via the HTTP protocol, stores them, and exports them to a DICOM destination:
 
<pre>
 
<pre>
 
<Configuration>
 
<Configuration>
Line 78: Line 77:
 
         <ImportService  
 
         <ImportService  
 
             name="HTTP Import"
 
             name="HTTP Import"
             class="org.rsna.trials.HttpImportService"
+
             class="org.rsna.ctp.stdstages.HttpImportService"
             root="D:\abc\http-import"  
+
             root="roots/http-import"  
 
             port="7777"  
 
             port="7777"  
             quarantine="HttpImportQuarantine" />
+
             quarantine="quarantines/HttpImportQuarantine" />
 +
        <StorageService
 +
            name="Storage"
 +
            class="org.rsna.ctp.stdstages.StorageService"
 +
            root="D:/storage"
 +
            return-stored-file="no"
 +
            quarantine="quarantines/StorageQuarantine" />
 +
        <ExportService
 +
            name="PACS Export"
 +
            class="org.rsna.ctp.stdstages.DicomExportService"
 +
            root="roots/pacs-export"
 +
            dest-url="dicom://DestinationAET:ThisAET@ipaddress:port"
 +
            quarantine="quarantines/PacsExportQuarantine" />
 +
  </Pipeline>
 +
</Configuration>
 +
</pre>
 +
Note that in the example above, non-DICOM objects are stored in the StorageService, but they are not exported by the DicomExportService. Each pipeline stage is responsible for testing the class of the object which it receives and processing (or ignoring) the object accordingly.
 +
 
 +
The following is an example of a more complex configuration. This configuration receives objects, passes them to a trial-specific Processor stage to test whether the object is appropriate for the trial, anonymize objects which make it through the preprocessor, export them to a database, and then anonymize them again to remove information which is not intended for storage, and finally store them.
 +
<pre>
 +
<Configuration>
 +
  <Server port="80" />
 +
  <Pipeline name="Main Pipeline">
 +
        <ImportService
 +
            name="HTTP Import"
 +
            class="org.rsna.ctp.stdstages.HttpImportService"
 +
            root="roots/http-import"
 +
            port="7777"
 +
            quarantine="quarantines/HttpImportQuarantine" />
 
         <Processor
 
         <Processor
 
             name="The Preprocessor"
 
             name="The Preprocessor"
 
             class="org.myorg.MyPreprocessor"
 
             class="org.myorg.MyPreprocessor"
             quarantine="PreprocessorQuarantine" />
+
             quarantine="quarantines/PreprocessorQuarantine" />
 
         <Processor
 
         <Processor
 
             name="Main Anonymizer"
 
             name="Main Anonymizer"
Line 92: Line 119:
 
             xml-script="xml-anonymizer-1.script"  
 
             xml-script="xml-anonymizer-1.script"  
 
             zip-script="zip-anonymizer-1.script"
 
             zip-script="zip-anonymizer-1.script"
             quarantine="MainAnonymizerQuarantine" />
+
             quarantine="quarantines/MainAnonymizerQuarantine" />
 
         <ExportService
 
         <ExportService
 
             name="Database Export"
 
             name="Database Export"
 
             class="org.rsna.trials.DatabaseExportService"
 
             class="org.rsna.trials.DatabaseExportService"
 
             adapter-class="org.myorg.MyDatabaseAdapter"
 
             adapter-class="org.myorg.MyDatabaseAdapter"
             root="D:\abc\database-export"
+
             root="roots/database-export"
             quarantine="DatabaseExportQuarantine" />
+
             quarantine="quarantines/DatabaseExportQuarantine" />
 
         <Processor
 
         <Processor
 
             name="Provenance Remover"
 
             name="Provenance Remover"
             class="org.rsna.trials.Anonymizer"
+
             class="org.rsna.ctp.stdstages.Anonymizer"
 
             dicom-script="dicom-anonymizer-2.properties"
 
             dicom-script="dicom-anonymizer-2.properties"
 
             xml-script="xml-anonymizer-2.script"  
 
             xml-script="xml-anonymizer-2.script"  
 
             zip-script="zip-anonymizer-2.script"
 
             zip-script="zip-anonymizer-2.script"
             quarantine="ProvenanceRemoverQuarantine" />
+
             quarantine="quarantines/ProvenanceRemoverQuarantine" />
 
         <StorageService
 
         <StorageService
 
             name="Storage"
 
             name="Storage"
             class="org.rsna.trials.StorageService"
+
             class="org.rsna.ctp.stdstages.StorageService"
             root="D:\abc\storage"  
+
             root="D:/storage"  
 
             return-stored-file="no"
 
             return-stored-file="no"
             quarantine="StorageQuarantine" />
+
             quarantine="quarantines/StorageQuarantine" />
        <ExportService
+
        <ExportService
 
             name="PACS Export"
 
             name="PACS Export"
             class="org.rsna.trials.DicomExportService"
+
             class="org.rsna.ctp.stdstages.DicomExportService"
             root="D:\abc\pacs-export"  
+
             root="roots/pacs-export"  
 
             dest-url="dicom://DestinationAET:ThisAET@ipaddress:port"
 
             dest-url="dicom://DestinationAET:ThisAET@ipaddress:port"
             quarantine="PacsExportQuarantine" />
+
             quarantine="quarantines/PacsExportQuarantine" />
        <ExportService
+
            name="Other Export"
+
            class="org.rsna.trials.HttpExportService"
+
            root="D:\abc\other-export"
+
            dest-url="http://ipaddress:port"
+
            quarantine="OtherExportQuarantine" />
+
 
     </Pipeline>
 
     </Pipeline>
 
</Configuration>
 
</Configuration>
Line 132: Line 153:
  
 
===Server===
 
===Server===
To provide access to the status of the components, the application includes an HTTP connector which serves web pages and supports servlet-like functionality. The web pages are served from a directory tree whose root is named <b>ROOT</b>.
+
To provide access to the status of the components, the application includes an HTTP server which serves files and provides servlet-like functionality. The web pages are served from a directory tree whose root is named <b>ROOT</b>.
  
 
====ConfigurationServlet====
 
====ConfigurationServlet====
The ConfigurationServlet displays the contents of the configuration file in a web page.
+
The ConfigurationServlet displays the contents of the configuration file.
  
 
====StatusServlet====
 
====StatusServlet====
The StatusServlet displays the status of all the pipeline stages in a web page.
+
The StatusServlet displays the status of all the pipeline stages.
  
 
====LogServlet====
 
====LogServlet====
Line 146: Line 167:
 
The UpdateService supports the Update Service clients in FieldCenter applications, serving software updates and saving the remapping tables in trials configured to use it.
 
The UpdateService supports the Update Service clients in FieldCenter applications, serving software updates and saving the remapping tables in trials configured to use it.
  
==Open Questions==
+
==The Standard Stages==
#Is it necessary to support more than one Pipeline in the application?
+
#Is it necessary or desirable to serve files from Storage objects via HTTP or HTTPS?
+

Revision as of 15:29, 13 August 2007

This article describes the stand-alone processing application for clinical trials data using MIRC components and the MIRC internet transport mechanism.

1 Background

MIRC supports clinical trials through two applications, one for data acquisition at an imaging center (FieldCenter) and one for management of the data at a principal investigator's site (MIRC).

The FieldCenter application acquires images via the DICOM protocol, anonymizes them, and transfers them (typically using HTTP, although DICOM is also supported) to a principal investigator's MIRC site. It also supports other types of data files and includes an anonymizer for XML files as well. FieldCenter also contains a client for the Update Service of a MIRC site, allowing the application to save data on, and obtain software updates from, the principal investigator's site.

The MIRC site software contains a partially configurable processing pipeline for clinical trials data, consisting of:

HttpImportService
A receiver for HTTP connections from FieldCenter applications transferring data files into the processing pipeline.
DicomImportService
A receiver form DICOM datasets for iinsertion into the processing pipeline.
Preprocessor
A user-defined component for processing data received by the HttpImportService before it is further processed by other components.
Anonymizer
A component for anonymizing DICOM objects or XML objects.
DatabaseExportService
A component providing queue management and submission of data objects to a user-defined interface to an external database management system.
HttpExportService
A component in the DicomImportService pipeline providing queue management and transmission of data objects to one or more external systems using the HTTP protocol.
DicomExportService
A component in the HttpImportService pipeline providing queue management and transmission of data objects to one or more external systems using the DICOM protocol.

The processing pipelines for the HttpImportService and DicomImportService are different. They are not symmetrical. For example, the HttpImportService does not have access to the anonymizer except as part of the DatabaseExportService. Another limitation is that objects received via one protocol can only be exported via the other. While these limitations are consistent with the requirements of most trials, it became clear that a completely symmetrical design would provide better support for more sophisticated trials while still satisfying the requirements of simple ones.

2 ClinicalTrialProcessor

ClinicalTrialProcessor is a stand-alone application that provides all the features of a MIRC site for clinical trials in a highly configurable and extensible way. It connects to FieldCenter applications and can also connect to MIRC sites when necessary. ClinicalTrialProcessor has the following key features:

  1. Single-click installation.
  2. Support for multiple pipelines.
  3. Processing pipelines supporting multiple configurable stages.
  4. Support for multiple quarantines for data objects which are rejected during processing.
  5. Pre-defined implementations for key components:
    • HTTP Import
    • DICOM Import
    • DICOM Anonymizer
    • XML Anonymizer
    • File Storage
    • Database Export
    • HTTP Export
    • DICOM Export
  6. Web-based monitoring of the application's status, including:
    • configuration
    • logs
    • quarantines
    • status
  7. Support for the FieldCenter Update Service client.

2.1 Pipelines

A pipeline is a manager that moves data objects through a sequence of processing stages. Each stage in the pipeline performs a specific function on one or more of the four basic object types supported by MIRC:

  • FileObject
  • DicomObject
  • XmlObject
  • ZipObject

Each pipeline must contain one ImportService as its first stage. Each pipeline stage is provided access to a quarantine directory, which may be unique to the stage or shared with other stages, into which the pipeline places objects that are rejected by a stage, thus aborting further processing. At the end of the pipeline, the manager calls the ImportService to remove the processed object from its queue.

2.1.1 ImportService

An ImportService receives objects via a protocol and enqueues them for processing by subsequent stages.

2.1.2 StorageService

A StorageService stores an object in a file system. It is not queued, and it therefore must complete before subsequent stages can proceed. A StorageService may return the current object or the stored object in response to a request for the output object, depending on its implementation.

2.1.3 Processor

A Processor performs some kind of processing on an object. It is not queued. A processor exposes methods with calling signatures that are unique to the object type. In the context of the current MIRC implementation, a Preprocessor is a Processor, as is an Anonymizer. The result of a processing stage is an object that is passed to the next stage in the pipeline.

2.1.4 ExportService

An ExportService provides queued transmission to an external system via a defined protocol. Objects in the queue are full copies of the objects submitted; therefore, subsequent processing is not impeded if a queue is paused, and modifications made subsequently do not affect the queue entry, even if they occur before transmission. (Note: This is different from the current MIRC implementation.)

2.2 Configuration

The ClinicalTrialProcessor configuration is specified by an XML file. There can be one Server element specifying the port on which the HTTP server is to operate, and multiple Pipeline elements, each specifying the stages which comprise it. The name of the element defining a stage is irrelevant and can be chosen for readability; each stage in a pipeline is actually defined by its Java class, specified in the class attribute. Stages are loaded automatically when the program starts, and the loader tests the stage's class to see what kind of stage it represents. It is possible to extend the application beyond the pre-defined stages available in the implementation as described in the section on Extending ClinicalTrialProcessor.

The following is an example of a simple configuration with one pipeline which receives objects via the HTTP protocol, stores them, and exports them to a DICOM destination:

<Configuration>
   <Server port="80" />
   <Pipeline name="Main Pipeline">
        <ImportService 
            name="HTTP Import"
            class="org.rsna.ctp.stdstages.HttpImportService"
            root="roots/http-import" 
            port="7777" 
            quarantine="quarantines/HttpImportQuarantine" />
        <StorageService
            name="Storage"
            class="org.rsna.ctp.stdstages.StorageService"
            root="D:/storage" 
            return-stored-file="no"
            quarantine="quarantines/StorageQuarantine" />
        <ExportService
            name="PACS Export"
            class="org.rsna.ctp.stdstages.DicomExportService"
            root="roots/pacs-export" 
            dest-url="dicom://DestinationAET:ThisAET@ipaddress:port"
            quarantine="quarantines/PacsExportQuarantine" />
   </Pipeline>
</Configuration>

Note that in the example above, non-DICOM objects are stored in the StorageService, but they are not exported by the DicomExportService. Each pipeline stage is responsible for testing the class of the object which it receives and processing (or ignoring) the object accordingly.

The following is an example of a more complex configuration. This configuration receives objects, passes them to a trial-specific Processor stage to test whether the object is appropriate for the trial, anonymize objects which make it through the preprocessor, export them to a database, and then anonymize them again to remove information which is not intended for storage, and finally store them.

<Configuration>
   <Server port="80" />
   <Pipeline name="Main Pipeline">
        <ImportService 
            name="HTTP Import"
            class="org.rsna.ctp.stdstages.HttpImportService"
            root="roots/http-import" 
            port="7777" 
            quarantine="quarantines/HttpImportQuarantine" />
        <Processor
            name="The Preprocessor"
            class="org.myorg.MyPreprocessor"
            quarantine="quarantines/PreprocessorQuarantine" />
        <Processor
            name="Main Anonymizer"
            class="org.rsna.trials.Anonymizer"
            dicom-script="dicom-anonymizer-1.properties"
            xml-script="xml-anonymizer-1.script" 
            zip-script="zip-anonymizer-1.script"
            quarantine="quarantines/MainAnonymizerQuarantine" />
        <ExportService
            name="Database Export"
            class="org.rsna.trials.DatabaseExportService"
            adapter-class="org.myorg.MyDatabaseAdapter"
            root="roots/database-export"
            quarantine="quarantines/DatabaseExportQuarantine" />
        <Processor
            name="Provenance Remover"
            class="org.rsna.ctp.stdstages.Anonymizer"
            dicom-script="dicom-anonymizer-2.properties"
            xml-script="xml-anonymizer-2.script" 
            zip-script="zip-anonymizer-2.script"
            quarantine="quarantines/ProvenanceRemoverQuarantine" />
        <StorageService
            name="Storage"
            class="org.rsna.ctp.stdstages.StorageService"
            root="D:/storage" 
            return-stored-file="no"
            quarantine="quarantines/StorageQuarantine" />
         <ExportService
            name="PACS Export"
            class="org.rsna.ctp.stdstages.DicomExportService"
            root="roots/pacs-export" 
            dest-url="dicom://DestinationAET:ThisAET@ipaddress:port"
            quarantine="quarantines/PacsExportQuarantine" />
    </Pipeline>
</Configuration>

Multiple Pipeline elements may be included, but each must have its own ImportService element, and their ports must not conflict.

Each pipeline stage class has a constructor that is called with its configuration element, making it possible for special processor implementations to be passed additional parameters from the configuration.

2.3 Server

To provide access to the status of the components, the application includes an HTTP server which serves files and provides servlet-like functionality. The web pages are served from a directory tree whose root is named ROOT.

2.3.1 ConfigurationServlet

The ConfigurationServlet displays the contents of the configuration file.

2.3.2 StatusServlet

The StatusServlet displays the status of all the pipeline stages.

2.3.3 LogServlet

The LogServlet provides web access to all the log files in the logs directory.

2.3.4 UpdateService

The UpdateService supports the Update Service clients in FieldCenter applications, serving software updates and saving the remapping tables in trials configured to use it.

3 The Standard Stages