Implementing an External Database Interface for MIRC Clinical Trials
The article is intended for software developers interfacing their MIRC sites to specialized databases for clinical trials. The reader is expected to be familiar with Java and with the RSNA MIRC Source Code.
1 The Database Export Service
The Database Export Service is a component of a MIRC Storage Service. It provides a queuing mechanism for submitting files to a database interface, relieving the interface from having to manage the queue. It attempts to parse each file to determine its type and calls the overloaded process method of the interface with one of four objects, encapsulating an XML file, a zip file, a DICOM file, or a file of unparsable contents. Each of the objects includes methods providing access to the internals of the file, allowing the interface to interrogate the files to obtain some or all of their data to insert into an external system.
The Database Export Service dynamically loads the database interface class, obtaining the name of the class from the clinical trial service’s configuration file.
2 The Object Classes
As noted above, MIRC provides four classes to encapsulate files of various types. The classes are all in the org.rsna.mircsite.util package.
2.1 FileObject
The FileObject class encapsulates a file of unknown contents. It is the parent class of the other three classes and provides a method for obtaining a java.io.File object pointing to the file, as well as methods for moving, renaming, and copying the file. It also provides dummy methods for obtaining key information about the file, all of which return empty strings. These methods are overridden by subclasses that can parse their corresponding file types to obtain meaningful values.
2.2 XmlObject
The XmlObject class encapsulates an XML file, parsing the XML in the constructor and throwing an Exception if the file does not parse. In addition to the methods of the FileObject, it provides methods for accessing the XML DOM object and reading its elements and attributes. For many purposes, the getValue(String path) method makes it easy to obtain the necessary data to insert into a database, but if necessary, you can use the entire XML DOM.
The Javadocs explain how specific types of data (the uid, the study-uid, the file description) are obtained from the XML DOM object. When designing XML metadata files for a clinical trial, if you put the data in those places, MIRC will be able to manage the files and pass them to your database interface more efficiently.
2.3 ZipObject
The ZipObject class encapsulates a zip file, parsing the file in the constuctor and throwing an Exception if the file does not parse. In addition to the methods of the FileObject, it provides methods for accessing the individual files within the zip file. It also provides methods for accessing the contents of a special file, manifest.xml, that contains data necessary to identify the zip file and relate it to other clinical trial data.
The ZipObject allows a clinical trial to collect a group of related objects, for example multiple output files from an analytical program for a single analysis, and to manage them as a single object.
The Javadocs explain how identifying data are obtained from the manifest.xml file. The ZipObject is much less flexible than the XmlObject in the placement of identifying data. When designing zip metadata files for a clinical trial, you should try to construct manifests exactly by the rules described in the Javadocs.
2.4 DicomObject
The DicomObject class encapsulates a DICOM dataset, parsing the file in the constructor and throwing an Exception if the file does not parse. In addition to the methods of the FileObject, it provides access to all the DICOM attributes, as well as image conversion methods returning JPEG images of any size from the DICOM image.
3 The DatabaseAdapter Class
The DatabaseAdapter class, org.rsna.mircsite.util.DatabaseAdapter, is a base class for building an interface between the Database Export Service and an external database. To be recognized and loaded by the Database Export Service, an external database interface class must be an extension of DatabaseAdapter.
The DatabaseAdapter class provides a set of methods allowing the Database Export Service to perform various functions, all of which are explained in the Javadocs. The basic interaction model is:
- When the Database Export Service detects that files are in its queue, it verifies that the database interface class is loaded and loads it if it is not.
- It then calls the database interface’s connect() method.
- For each file in the queue, it instantiates an object matching the file’s contents and calls the database interface’s process() method. There are four overloaded process methods, one for each object class. The process() methods have two arguments: the metadata object and the URL of the MIRCdocument which references the file.
- When the queue is empty, it calls the database interface’s disconnect() method.
- If the Database Export Service is restarted after the database interface has been instantiated, it calls the interface’s reset() method.
- When the Database Export Service is completely finished with an instance of the database interface and is about to discard the instance before either exiting or instantiating a new instance of the interface, it calls the interface’s shutdown() method.
All the methods of the DatabaseAdapter class return an integer status value. Static values for the statuses are defined in the class. There are three values:
- STATUS_OK means that the operation succeeded completely.
- STATUS_FAIL means that the operation failed and trying again will also fail. This status value indicates a problem with the object being processed.
- STATUS_WAIT means that the operation failed but trying again later may succeed. This status value indicates a temporary problem accessing the external database.
All the methods of the DatabaseAdapter class return the value STATUS_OK.
4 Extending the DatabaseAdapter Class
To implement a useful interface to an external database, you must extend the DatabaseAdapter class. The build.xml file provides an example of such an implementation. It is in the testdatabase target, and it builds a jar file containing org.rsna.database.TestDatabase. This implementation only logs method calls.
Since the DatabaseAdapter class implements dummy methods returning STATUS_OK, your class that extends DatabaseAdapter only has to override the methods that apply to your application. If, for example, you only care about XML objects, you can just override the process(XmlObject xmlObject) method and let DatabaseAdapter supply the other process() methods. This will cause your interface to ignore objects of other types.
The database interface should not rely on the reset() and shutdown() methods. They are called when possible, but there are certain situations in which they cannot be called (a power failure, for example), and you should make sure that the data is protected in those situations. Similarly, since one connect() call is made for possibly multiple process() method calls, it is possible that a failure could result in no disconnect() call. Thus, depending on the design of the external system, it may be wise to commit changes in each process() call.
If you want to log information that will be visible through the “Show Log” button on the admin page, you can use the static methods in the org.rsna.mircsite.log.Log class. Remember, however, that the Log class only maintains a circular buffer of the past 200 entries, so if you make many calls, you may be removing log entries made by other objects.
5 Connecting Your Database Class to MIRC
After building your database interface and producing a jar file for it, you have to install it and configure it. The Database Export Service is part of a Storage Service, and each storage service can have its own (possibly different) external database. Therefore, the interface is installed and configured at the level of the Storage Service.
5.1 Installation
The database interface jar file must be installed in the:
directory, where [storageservice] is the name of the storage service for the clinical trial. Do not place the jar file in the Tomcat/shared/lib directory. Doing so will cause the class not to load.
The MIRCsite-installer program always deletes the
directory during an upgrade to prevent old libraries from contaminating a new release. This has the unfortunate effect of deleting the database interface jar file. You must therefore be careful to restore the database interface jar file after an upgrade.
5.2 Configuring MIRC for Your Database Class
The Admin Service provides a web interface for setting the configuration parameters. The interface is accessed by clicking the “Update Configuration” button in the “DICOM Service” column on the admin page. In the Database Export Service section of the resulting page, there are four parameters:
- Mode determines whether objects that are received are automatically forwarded to the Database Export Service. In “auto” mode, they are. In “QC” mode, they are not. In QC mode, the assumption is made that the administrator will review the MIRCdocument that contains the received objects and verify that they are appropriate for sending to the database. If they are, he can click the “Export to database” button in the “Images” column at the bottom of the displayed document’s “Document” tab. Doing so queues all the objects in the document for export. An important subtlety is that the only objects that make it into MIRCdocuments are ones that contain a Study Instance UID. Thus, a FileObject will never be sent to the Database Export Service (since a FileObject’s contents cannot be parsed by MIRC). In auto mode, however, all received objects are exported, so if you plan to receive unparseable files as part of your clinical trial, you must operate the Database Export Service in auto mode.
- Anonymizer enabled determines whether the objects are anonymized after being sent to the database. This operation is a little counter-intuitive at first sight. The operational model is that objects are anonymized before they are transmitted to the MIRC site. Typically, this would be done by the MIRC FieldCenter application which transmits them to the MIRC site. Actual anonymization takes place at that time, but in some cases, it is desirable to have the anonymizer include provenance information that is intended for the database but not for viewers of the MIRCdocument or its referenced objects. In that situation, the FieldCenter’s anonymizer is configured to add the provenance information, the object is sent to and received by the MIRC site, which stores the file in the directory for the Study Instance UID of the file and references it in the associated MIRCdocument. The file is then passed to the Database Export Service with the provenance information intact. If the anonymizer is enabled, then after the process() method of the database interface class returns, the Storage Service’s anonymizer is called to remove the provenance information.
- Sleep interval determines the frequency at which the Database Export Service checks its queue. The interval is specified in milliseconds, and a typical interval is 10,000, or 10 seconds. The Database Export Service enforces a reasonability criterion on the value, so if it is very short or very long, it uses a 10 second default.
- Database class name specifies the fully qualified name of the class, for instance, org.rsna.database.TestDatabase. If this field is empty, the Database Export Service will operate as if it were disabled.
If you want to have references to received metadata files placed in the MIRCdocuments that are automatically created for clinical trial studies, you should add a <metadata-refs/> element to the Tomcat/webapps/[storageservice]/trial/template.xml file. This element is explained in “The MIRCdocument Schema” documentation, which is available on the RSNA MIRC site (mirc.rsna.org).
5.3 Operational Note
When a file is received, the Object Processor and the Database Export Service attempts to figure out the type of file by parsing it. It first tries to parse it based on the extension of the filename. If that fails, it tries all the untried classes in the order:
- DicomObject
- ZipObject
- XmlObject
If none of the objects can be instantiated, a FileObject is created for the file. If the file is a binary object that is neither a DICOM file nor a zip file, the attempt to instantiate the XmlObject will cause the Xerces parser to encounter a fatal, but not serious, error, which it logs to the console stream where it can be seen in the log viewer or by accessing the Tomcat/logs/stdout.log file. The error is not serious because although the Xerces parser fails, it is reinstantiated whenever it is needed, so there is no effect on the operation of the system. This error will never occur in trials which produce only DICOM, zip, or XML files.