Projects:QIN:3D Slicer Annotation Image Markup:Design and Implementation

Back to Projects:QIN:3D Slicer Annotation Image Markup

Slicer Reporting Module

Terminology

• Annotation: textual structured description of the finding. How do we differentiate those from Slicer Annotations?
• Markup: visual cues placed over the image to assign annotations to image features

Implementation considerations

• We will use existing annotation hierarchies to organize the AM. AM will be organized into findings that will be located in the hierarchy under a particular volume. Each volume can have separate findings. Each finding can be assigned to only one volume.
• We are focused on the support of markup in 2D slices; 3D view annotations are secondary.
• The functionality needed to support connection between the data and AM will be added to Slicer core. User-level interactions will be available in a separate module.
• Markup is defined relative to the active slice of the active volume. AIM/DICOM SR do not have a notion of 3D coordinate space (needs verification). Slicer annotations will be restricted to be in-slice for the use cases of this project.
• Annotations text can be stored in the "description" field of the Slicer Annotations in the XML format, as determined by the annotation template used.
• Annotations text can be formatted and visualized using HTML as a report in the Slicer Annotation description area.

Input from Justin Kirby:

• allow loading reports, editing, and saving again
• allow multiple reports per volume
• allow multiple observers per volume
• include points (use case: renal application in preparation)

Markups to be supported

These will be decided based on the existing requirements from the use cases provided by the QIN community, and based on the existing capabilities of the AIM-enabled tools (OsiriX and ClearCanvas TCGA workstation). The tentative list in the order of priorities (not finalized):

• Point
• Ruler
• 3d segmentation

Example AIM templates

AIM template schema: File:AIM Template v1rv18.xsd.zip

• RANO File:RANO4 AIM template.xml.zip
• BRCA File:BRCA AIM template.xml.zip
• VASARI File:VASARI AIM template.xml.zip

GUI

Ron's suggestions

• will need to revisit the tree -- it is not user-friendly
• use radio boxes instead of dropdowns to minimize the number of clicks
• consider using webkit for form visualization?

User-level (implementation level) Workflow

Creation:

• 1 choose volume being annotated ( implementation --> create corresponding node in the hierarchy, initialize slice viewers, rotate to slice plane)
• 2 select annotation template (default: finding? option to load template?) ( --> populate the GUI with the annotation template contents)
• 3 populate template ( --> NA)
• 3 scroll to the slice
• 5 add new finding into the hierarchy ( --> need to have a button to add new finding? all subsequent markups will go under that tree element)
• 6 place markups ( --> support markup type selection from slicer toolbar? add extra markups compared to Annotation module, and remove them when user leaves the module? for 2d/3d ROIs -- invoke Editor?)
• 7 assign text (annotation of annotation in AIM?) -- all go under the active annotation hierarchy node
• 8 done: save/export results ( --> Slicer save dialog? save annotation as AIM XML or as "AIM-light"? include name of the observer, creation date)

Loading:

• may distribute the template used in annotation together with the scene or not?
• if we allow editing of the annotation/markup, when saving need to update this (think about data provenance/revision history later)

Comments:

• User can switch to multiple volumes over the annotation period. This is ok, separate hierarchy element for each volume.

Progress

Source code on github: https://github.com/fedorov/Reporting

Active task list

• Need to support MRML scene round-trip for Reporting.
• Andrey:
  • add extra handling for slice nodes: detect acquisition plane and rotate to volume slice
  • handling of markups placed not on the volume
  • start looking into AIM export: analyze functionality of QtXml module
• Nicole:
  • hierarchies
    • consistent user experience (no duplication of hierarchy/volume nodes, correct parenting)
  • markup
    • add handling of "eye" visibility icon for markup elements
  • make sure assumption of single volume in the report does not cause us difficulties later
  • trace labelmaps that are associated with the volume being annotated (listen to AddNode events, and add all label maps for volume node in the report to the hierarchy) (Nicole: almost done)
  • scene restore with ruler annotation is crashing
• AIM import: need to support lookup from image UID to slice ID

Completed tasks

• add parameter node to store selected report&volume
• disable GUI elements to make workflow more intuitive
• testing for RANO node
• hide markups that do not belong to the active report
• show only active report in the widget
• RANO MRML node events/serialization completed
• Add functionality to track the UID of the DICOM volume based on the current volume and IJK coordinates
• Add links between AM (hierarchy? finding?) and parent volume
• Start work on the module (loadable module, python GUI scripting)
• decided: we will not be adding any new widgets as part of this project

To be resolved

• add "observer" name as a Slicer setting, or query from the system settings
• add new layout? (2 over 2, no 3d viewer)
• serialization
• generic qMRMLReportingAnnotationWidget and AIM template support

Items being researched

XML-related background information

The basic need is given the AIM/AIM template schema, how to correctly parse XML documents that satisfy this schema, and how to create schema-compliant documents.

AIM API tools (Northwestern) use C++ classes corresponding the schema element types generated using Altova XMLSpy. CodeSynthesis provides another solution for this task. Both are not acceptable as they are commercial (perhaps we could reuse code generated for AIM API, but that one would need to be ported to become cross-platform).

Qt has QtXml and QtXmlPatterns modules to support XML interactions, but no C++ binding tools lice XMLSpy/CodeSynthesis (see this thread, Jan 2011).

Python seems to have some tools to generate Python classes/modules and parsers for a given schema, e.g., PyXB, PyXSD, generateDS (see the summary of available tools here).

Possible implementation approaches:

• 1. (short-term) create manually the elements and initialize attributes for a subset of AIM that we will need to support the annotations and markup we have right now.
• 2. re-use the C++ classes in NW AIM API.
• 3. use python-based approach to generate classes, validate and support serialization.

Additional considerations for python XML tools:

• both PyXB and generateDS appear to be active projects
• generateDS has limited support for namespaces, see this thread (need to clarify if we need that for AIM schema)
• PyXB generated bindings .py output includes pyxb, introducing a new package dependency; generateDS depends only on the system modules (see comparison here)
• generateDS does not seem to provide schema validation mechanisms (but we can do that using Qt?)
• PyXB: need some special handling of namespaces (?)
• generateDS needs lxml (not included in Slicer)