Multi Label Segmentation

Using multiple segmentations with the Segmentation bundle

Following a proposal of Sebastian Ordas to support multi-label segmentations, this page will be used to sketch future changes to the Segmentation bundle and the related framework.

Shortcomings of the current implementation

• 8-bit images are used for segmentations by default, wasting 7 of 8 bits, occupying far more memory than technically necessary

Wishlist

• what is not possible or cumbersome and should be changed?
  • multiple labels on the same segmentation image: have 8-bit images with 0 value as background and the remaining 255 values as labeled objects.
    • this would imply non-overlapping structures. This might be a problem, unless users use another segmentation in those cases (they would have done so anyway in the current approach)
    • we should find a way to segment overlapping structures and assign two memberships to a voxel.
• if we'd use only a single bit we would still reduce memory consumption to 1/8. This could be taken as an intermediate step in the re-engineering process.
  • Should try itk::SingleBitBinaryImage: see Insight Journal paper Alternative Memory Models for ITK Images - http://www.insight-journal.org/browse/publication/646.
  • Will need an I/O mechanism see: http://old.nabble.com/Re%3A-SingleBitBinaryImage-td30804691.html
  • itk::SingleBitBinaryImage is yet not multi-threaded. see http://www.itk.org/pipermail/insight-users/2010-November/038761.html
• what enhancements would help in which situations?
  • Primary importance enhancements: memory reduction will greatly improve usability when working with several segmentations and big datasets
  • Secondary importance enhancements:
    • have label "lock"/"editable" boolean property
    • "stick" label boundaries to neighboring labels: e.g. if the label is smoothed or dilated, so will be its neighbors. If the label is eroded, the "sticked" neighbors will grow into the space left by the eroded label.
    • "background" label should be created by default, and should also be possible to lock it.
    • "label splitter" tool: given a label (segmentation), construct a new label from each connected subpart (3D method).
    • The post http://bugs.mitk.org/show_bug.cgi?id=6591 by Andreas Fetzer looks very interesting. TODO: ask which use cases he had in mind to implement this tool (marching squares?)
    • "label assigner": given a label and a clicked position on it, build a new label from the connected subpart around that position (3D method). The same mechanism can be used to delete the connected region or add it to another existing label.
    • "label assigner" would differ from "label splitter" so that in the former, the connected region around the clicked position is either assigned to a new label, to an existing label (indicated by the user), or deleted. In the "label splitter", all connected regions of a given label will be assigned a new (and different) label automatically.
    • have a workable "paintbrush". The current implementation is just not usable. Based on a post found in http://bugs.mitk.org/show_bug.cgi?id=6589 I could improve its use quite a lot (TODO: see how to send patch). Does it make sense to have a 3D paint brush?
• have a 2D/3D region growing tool based on ITK classes [seba: add comment regarding MITK's 3D grow tool]
  • useful resources:
    • vtkEdge paintbrush classes e.g. http://www.vtkedge.org/doc/nightly/html/classvtkKWEPaintbrushMergeSketches.html#_details
    • itkSnap http://www.itksnap.org/pmwiki/pmwiki.php to support multi-label images
    • ITK's label object classes (http://www.insight-journal.org/browse/publication/176) could be useful
    • Segmentation module in Amira (as it used to be in v3.1) http://www.amira.com/images/stories/pdf/31/Amira31-manual.pdf (see Sec. 2.5)
• how does I/O work?
  • same as currently: .nrrd images. In case of single-bit images, there's yet no IO support. See http://old.nabble.com/Re%3A-SingleBitBinaryImage-td30804691.html as alternative.
• do we need a new data type for MITK (derived from BaseData?)
  • no. i think we can use 8-bit images and assign a color to each label via a lookup table
  • in the case we switch to 1-bit images, we will need to add it to MITK (derived from BaseData)
• how could these be mapped types for rendering?
  • if we continue with 8-bit images, we can use the same mappers we have so far
  • if we switch to 1-bit images, maybe not [seba TODO, check this]

Implications on the implementation

• what parts of MITK would need to be changed?
  • mitk::Tool, mitk::ToolManager, mitk::ContourUtils, ...
  • in QmitkNodeDescriptorManager add a QmitkNodeDescriptor "Labeled Image" for images with pixel type "unsigned char" (see if we want to keep purely binary images)
  • searching the code for "binary" occurrences:
    • + In mitk::ImageMapperGL2D::Paint, volume calculation should slightly change: itkimage->GetCountOfMaxValuedVoxelsNoRecompute() will need to have a "label" parameter to count the voxels with that given value. + In mitk::ImageMapperGL2D::SetDefaultProperties(), the heuristic for a labeled image would be: "uchar" == image->GetPixelType().GetItkTypeAsString(), PixelType::GetBpe() == 8 or alike + change names of "outline binary" to e.g. "outline label"
• what needs to be changed in the Segmentation view?
  • new controls:
    • + label manager widget ( e.g. have a QmitkDataStorageTable): should easily allow to (multiple) select labels for visualization or editing/locking toggling, change their representation (filled or border line), enable volume render feedback, etc.
• Use the old datastorage combobox (IMHO: the current working/reference node selection approach via the workbench selection service is not user friendly since work with the segmentation bundle requires a lot of data (multiple) selection and deselection.)

Agreed working plan

1. Agree on how to handle overlapping structures. If we work on this part of the application, we should get it right and produce a UI concept that will work both with overlapping and mutually-exclusive structures.
   1. Implement a proof of concept for using the ITK label object classes
      • bundle should be able to create a new object and prepare it for rendering (LUT or similar)
      • the contour tool should work with this object, remaining tools don't count yet
      • interpolation can be ignored at this step
2. make interpolation work again
   1. enhance the new application
      • implement other tools, mostly ignoring old code and replacing it with ITK/VTK implementations
      • implement locking, sticking and other useful features

Internal Discussion

1. Use multi-label images instead of 1-bit images, because:

• we can use existing code for rendering and file handling
• storing multiple segmentations in the same file leads to less cluttered directories

2. We need a cast filters to convert a 1-bit channel to a standard image for processing (and back again - in this case allow to integrate new label into existing image). A label map should be able to use any integer type (i.e. 8, 16, 32 bit) - default could be 8 for now, but leave possibility for extension open.

3. Overlap/Non-overlap of different segmentation channels should be handled by image flag:

• write operations on pixel level check if flag is set and clear all other bits if required

4. Replace mitk::Image::IsSegmentation() by flag (i.e. not trying to analyze the image). This flag is set when creating a segmentation or loading an image. For this, menu requires additional entry "Load segmentation".

5. Develop widget to select active channel and allow proper naming and coloring. Integrate this into tool framework.

6. Open questions:

• How to store colors and names for each segmentation channel (in application and on disk)?