CTP Configuration for NBIA

This article describes how the CTP application can be configured to work with the National Biomedical Image Archive (NBIA). The intended audience for this article is system administrators at sites running instances of NBIA. There are several articles on CTP on the RSNA MIRC wiki. All would be useful references when reading this article.

1 A Simple Configuration File

The configuration of a CTP instance is specified by an XML file called config.xml located in the same directory as the program. The example configuration shown below assumes that anonymized data objects are transmitted to CTP by an external application using the HTTPS protocol. This configuration might be useful for testing the NCIADatabase DatabaseAdapter.

<Configuration>

   <Server port="80" />

   <Pipeline name="Main Pipeline">

        <ImportService 
            name="HTTP Import"
            class="org.rsna.ctp.stdstages.HttpImportService"
            root="roots/http-import" 
            port="7777"
            ssl="yes"
            acceptDicomObjects="yes"
            acceptXmlObjects="yes"
            acceptZipObjects="yes"
            acceptFileObjects="no"
            quarantine="quarantines/http-import" />

        <StorageService
            name="Storage"
            id="storage"
            class="org.rsna.ctp.stdstages.BasicFileStorageService"
            index="D:/storage" 
            root="D:/storage/root" 
            port="80"
            nLevels="4"
            maxSize="300"
            returnStoredFile="yes"
            quarantine="quarantines/storage" />

        <ExportService
            name="Database Export"
            class="org.rsna.ctp.stdstages.DatabaseExportService"
            adapterClass="org.rsna.ctp.stdstages.database.DatabaseAdapter"
            fileStorageServiceID="storage"
            root="roots/database-export"
            quarantine="quarantines/database-export" />

    </Pipeline>
</Configuration>

1.1 Commentary

This section draws attention to certain attributes in the configuration and explains the reasons for them.

1.1.1 Server

The configuration file above places the CTP admin web server on port 80.

1.1.2 HttpImportService

The configuration file puts the HttpImportService on port 7777 and enables secure sockets layer.

The HttpImportService is configured to accept all object types which contain UIDs. Since the BasicFileStorageService does not accept FileObjects (which do not contain UIDs), the HttpImportService is configured to reject any which are received.

If an object is received which is not accepted, it is quarantined. If it is not desired to keep copies of such objects, the quarantine attribute can be omitted.

1.1.3 Storage Service

The id attribute is critical in this configuration. It provides a way for the DatabaseExportService access the FileStorageService to obtain the URL of an object being exported. The NBIA database requires the URL to provide access to the object as well as a thumbnail image of the object (if the object is an image).

The returnStoredFile attribute is not critical in this configuration. It is set to "yes" just to keep it consistent with the more complex configuration shown below. The reason it is not critical here is that the DatabaseExportService obtains the path to the stored object based on its UID, which is the same in the queued file as it is in the stored file.

The index attribute points to any location where the BasicFileStorageService can store its index.

The root attribute points to any location where the FileStorageService can build its directory structure. It is shown in the configuration above as an absolute path to some device, and the device could be on another drive or another computer from the one where the CTP application is stored. The root could also be placed under the CTP directory tree, for example, by specifying it to be roots/storage/root. In the configuration shown above, it is assumed that the files will be stored outside the CTP tree.

The index directory must not be contained within the root directory tree. As shown above, it is acceptable to have the root within the index.

1.1.4 DatabaseExportService

The adapterClass attribute specifies the fully qualified name of the Java class which implements the NCIADatabase DatabaseAdapter.

The fileStorageServiceID attribute is critical. It tells the DatabaseExportService which BasicFileStorageService to query for the file path of the object being exported.

Note that when an object is received by an ExportService for export, a copy of the object, not a reference to it, is entered into the ExportService's queue. Thus, the object in the queue is independent of the one which was received, and subsequent stages, if any, can modify the object being passed down the pipeline without affecting the object being exported. This is important when removing provenance information as explained below.

2 A Production Configuration File

The example configuration shown below assumes that the anonymized data objects received by the HttpImportService contain provenance information which must be removed from the stored objects after the information has been passed to the NBIA database.

<Configuration>

   <Server port="80" />

   <Pipeline name="Main Pipeline">

        <HttpImportService
            name="HTTP Import"
            class="org.rsna.ctp.stdstages.HttpImportService"
            root="roots/http-import" 
            port="7777"
            ssl="yes"
            acceptDicomObjects="yes"
            acceptXmlObjects="yes"
            acceptZipObjects="yes"
            acceptFileObjects="no"
            quarantine="quarantines/http-import" />

        <BasicFileStorageService
            name="Storage"
            id="storage"
            class="org.rsna.ctp.stdstages.BasicFileStorageService"
            index="D:/storage" 
            root="D:/storage/root" 
            port="80"
            nLevels="4"
            maxSize="300"
            returnStoredFile="yes"
            quarantine="quarantines/storage" />

        <DatabaseExportService
            name="Database Export"
            class="org.rsna.ctp.stdstages.DatabaseExportService"
            adapterClass="org.rsna.ctp.stdstages.database.DatabaseAdapter"
            fileStorageServiceID="storage"
            root="roots/database-export"
            quarantine="quarantines/database-export" />

        <DicomAnonymizer
            name="DICOM Provenance Remover"
            class="org.rsna.ctp.stdstages.DicomAnonymizer"
            root="roots/provenance-remover" 
            script="roots/provenance-remover/dicom-anonymizer.script
            quarantine="quarantines/provenance-remover" />

        <XmlAnonymizer
            name="XML Provenance Remover"
            class="org.rsna.ctp.stdstages.XmlAnonymizer"
            root="roots/provenance-remover" 
            script="roots/provenance-remover/xml-anonymizer.script
            quarantine="quarantines/provenance-remover" />

    </Pipeline>
</Configuration>

2.1 Commentary

The commentary in the previous section applies to this configuration as well, but it is important to note several points about the Anonymizer stages.

2.2 FileStorageService

It is critical that the FileStorageService's returnStoredFile attribute be set to yes in order that the object received by the Anonymizer stages points to the stored file. This allows the Anonymizer stages to anonymize the object in place in the BasicFileStorageService. If the BasicFileStorageService did not return the stored file, then an Anonymizer would anonymize the object in the HttpImportService queue, leaving the non-anonymized version in the BasicFileStorageService.

2.2.1 Anonymizer

CTP actually contains three types of anonymizers (DICOM, XML, and Zip), each configured as a separate Processor stage with its own script file. A DicomAnonymizer and an XmlAnonymizer are shown; a ZipAnonymizer could be added if ZipObjects containing provenance information are to be received.

Note that the Anonymizer appears after the DatabaseExportService. This ensures that the database gets a copy of the object containing the provenance information.

When CTP starts, it looks for the script files in the specified locations, and if they are not present, example files are copied into place. The CTP admin web server includes a special servlet for editing DICOM anonymizer scripts. It also includes a servlet for editing all other script types, including those for the ZML and Zip anonymizers as well as the DICOM, XML, and Zip filters). See The CTP XML Anonymizer for information on configuring the anonymizers for XmlObjects and the manifests in ZipObjects. The DICOM anonymizer is described in The CTP DICOM Anonymizer.

3 Advanced Configuration

This section briefly notes two other points which may be of interest in special cases.

3.1 Importing from a File Tree

CTP has several standard ImportServices, and it is designed to be extended. One standard ImportService which might be useful in capturing the contents of a large archive for the NBIA database, is the ArchiveImportService, which walks a directory tree and supplies objects to its pipeline. See the wiki article and the CTP Javadocs for org.rsna.ctp.pipeline.ArchiveImportService for more information.

3.2 Preprocessing

There are three other pipeline stages which might be of use in filtering out data objects which are inappropriate for inclusion in the database. They are:

• org.rsna.ctp.stdstages.DicomFilter
• org.rsna.ctp.stdstages.XmlFilter
• org.rsna.ctp.stdstages.ZipFilter

These stages are described in the main article on CTP.

If preprocessing is desired which goes beyond the capabilities of the filters' script languages, it is possible to create a special-purpose preprocessing stage. See the CTP Javadocs for org.rsna.ctp.pipeline.Processor and org.rsna.ctp.pipeline.AbstractPipelineStage for more information.

3.3 Exporting to an XNAT Data Service

As of 2012.09.12, CTP includes a special DatabaseAdapter, TciaXnatDatabaseAdapter, that exports to an XNAT data service. The adapter gets its configuration information from the DatabaseExportService's configuration file element. This is an example of such an element:

        <DatabaseExportService
            name="DatabaseExportService"
            class="org.rsna.ctp.stdstages.DatabaseExportService"
            adapterClass="org.rsna.ctp.stdstages.database.TciaXnatDatabaseAdapter"
            poolSize="1"
            fileStorageServiceID="bfss"
            root="test/roots/DatabaseExportService"
            quarantine="test/quarantines/DatabaseExportService"
            interval="10000"
            username="admin"
            password="password"
            projectTag="(0013,1011)"
            projectName="TheProject"
            sessionTag="(0013,1010)"
            sessionName="TheSession"
            url="http://127.0.0.1:5300" />

These attributes are used by the TciaXnatDatabaseAdapter:

• username specifies the username to be provided to the XNAT data service in an Authorization header using Basic authentication. If this attribute is missing or blank, no Authorization header is sent.
• password specifies the password to be included in the Authorization header.
• projectTag specifies the element in a DicomObject to be used to provide the value of the project attribute in the XML structure sent to the XNAT data service.
• projectName specifies the default value of the project attribute, to be used if the projectTag is not specified or if the value in the DicomObject is either missing or blank.
• sessionTag specifies the element in a DicomObject to be used to provide the value of the session attribute in the XML structure sent to the XNAT data service.
• sessionName specifies the default value of the session attribute, to be used if the sessionTag is not specified or if the value in the DicomObject is either missing or blank.
• url specifies the URL of the XNAT data service. If SSL is to be used, the protocol must be specified as https.

Shown below is an example of an XML structure produced by the TciaXnatDatabaseAdapter for export:

<DICOMResource
    classUID="1.2.840.10008.5.1.4.1.1.2"
    instanceNumber="102"
    instanceUID="1.3.6.1.4.1.9328.50.3.73"
    xsi:noNamespaceSchemaLocation="dicom-resource.xsd"
    subject="1.3.6.1.4.1.9328.50.3.0022"
    project="TheProject"
    seriesNumber="2"
    seriesUID="1.3.6.1.4.1.9328.50.3.69"
    session="TheSession"
    studyDate="20000101"
    studyUID="1.3.6.1.4.1.9328.50.3.68"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <URI>
        file:/D:/JavaPrograms/CTP-XNAT/CTP/test/roots/BasicFileStorageService/store/0000000000/000/004.dcm
    </URI>
</DICOMResource>

Notes:

• The subject attribute is populated from the PatientID element in the DicomObject.
• The studyDate attribute is populated from the StudyDate element in the DicomObject.
• The studyUID attribute is populated from the StudyInstanceUID element in the DicomObject.
• The seriesUID attribute is populated from the SeriesInstanceUID element in the DicomObject.
• The seriesNumber attribute is populated from the SeriesNumber element in the DicomObject.
• The instanceUID attribute is populated from the SOPInstanceUID element in the DicomObject.
• The instanceNumber attribute is populated from the InstanceNumber element in the DicomObject.
• The classUID attribute is populated from the SOPClassUID element in the DicomObject.
• After constructing a value for the project attribute, the text is filtered to replace characters not allowed in an XNAT project identifier with the underscore character.
• After constructing a value for the session attribute, the text is filtered to replace characters not allowed in an XNAT session identifier with the underscore character.

The configuration shown below was used for testing the TciaXnatDatabaseAdapter:

<Configuration>

   <Server port="9080" />

   <Pipeline name="Import Pipeline">

        <DicomImportService
            name="DicomImportService"
            class="org.rsna.ctp.stdstages.DicomImportService"
            root="test/roots/DicomImportService"
            port="1104"
            quarantine="test/quarantines/DicomImportService" />

        <BasicFileStorageService
            name="BasicFileStorageService"
            id="bfss"
            class="org.rsna.ctp.stdstages.BasicFileStorageService"
            index="test/roots/BasicFileStorageService"
            root="test/roots/BasicFileStorageService/store"
            nLevels="3"
            maxSize="200"
            returnStoredFile="yes"
            quarantine="test/quarantines/BasicFileStorageService" >
          <jpeg wmax="10000" wmin="96" q="-1" />
        </BasicFileStorageService>

        <DatabaseExportService
            name="DatabaseExportService"
            class="org.rsna.ctp.stdstages.DatabaseExportService"
            adapterClass="org.rsna.ctp.stdstages.database.TciaXnatDatabaseAdapter"
            poolSize="1"
            fileStorageServiceID="bfss"
            root="test/roots/DatabaseExportService"
            quarantine="test/quarantines/DatabaseExportService"
            interval="10000"
            username="admin"
            password="password"
            projectTag="(0013,1011)"
            projectName="TheProject"
            sessionTag="(0013,1010)"
            sessionName="TheSession"
            url="http://127.0.0.1:5300" />

   </Pipeline>

   <Pipeline name="XNAT Simulator Pipeline">

        <HttpImportService
            name="HttpImportService"
            class="org.rsna.ctp.stdstages.HttpImportService"
            root="test/roots/HttpImportService"
            ssl="no"
            port="5300"
            requireAuthentication="yes"
            quarantine="test/quarantines/HttpImportService" />

        <FileStorageService
            name="FileStorageService"
            class="org.rsna.ctp.stdstages.FileStorageService"
            root="test/roots/FileStorageService"
            port="1181"
            returnStoredFile="no"
            quarantine="test/quarantines/FileStorageService" />

   </Pipeline>

</Configuration>