Multi Label Segmentation

Using multiple segmentations with the Segmentation bundle

Following a proposal of Sebastian Ordas to support multi-label segmentations, this page will be used to sketch future changes to the Segmentation bundle and the related framework.

Shortcomings of the current implementation

• 8-bit images are used for segmentations by default, wasting 7 of 8 bits, occupying far more memory than technically necessary

Wishlist

• what is not possible or cumbersome and should be changed?
  • multiple labels on the same segmentation image: have 8-bit images with 0 value as background and the remaining 255 values as labeled objects.
  • this would imply non-overlapping structures. This might be a problem, unless users use another segmentation in those cases (they would have done so anyway in the current approach)
  • we will definitely need a way to segment overlapping structures and assign a voxel two memberships.
• if we'd use only a single bit we would still reduce memory consumption to 1/8. This could be taken as an intermediate step in the re-engineering process
• what enhancements would help in which situations?
  • Primary importance enhancements: memory reduction will greatly improve usability when working with several segmentations and big datasets
  • Secondary importance enhancements
    • have label "lock"/"editable" boolean property
    • "stick" label boundaries to neighboring labels: e.g. if the label is smoothed or dilated, so will be its neighbors. If the label is eroded, the "sticked" neighbors will grow into the space left by the eroded label.
    • "background" label should be created by default, and should also be possible to lock it.
    • "label splitter" tool: given a label(segmentation), construct a new label from each connected subpart (3D method).
    • The post http://bugs.mitk.org/show_bug.cgi?id=6591 by Andreas Fetzer looks very interesting. TODO: ask which use cases he had in mind to implement this tool (marching squares?)
    • "label assigner": given a label and a clicked position on it, build a new label from the connected subpart around that position (3D method). The same mechanism can be used to delete the connected region or add it to another existing label.
    • "label assigner" would differ from "label splitter" so that in the former, the connected region around the clicked position is either assigned to a new label, to an existing lablel, or deleted. In the "label splitter", all connected regions of a given label will be assigned a new (and different) label.
    • have a workable "paintbrush". The current implementation is just not usable. Based on a post found in: [seba complete] I could improve its use quite a lot. Does it make sense to have a 3D paint brush?
• have a 2D/3D region growing tool based on ITK classes [seba: add comment regarding MITK's 3D grow tool]
  • useful resources:
    • vtkEdge paintbrush classes e.g. http://www.vtkedge.org/doc/nightly/html/classvtkKWEPaintbrushMergeSketches.html#_details
    • itkSnap http://www.itksnap.org/pmwiki/pmwiki.php to support multi-label images
    • ITK's label object classes (http://www.insight-journal.org/browse/publication/176) could be useful
    • Segmentation module in Amira (as it used to be in v3.1) http://www.amira.com/images/stories/pdf/31/Amira31-manual.pdf (see Sec. 2.5)
• how does I/O work?
  • same as currently: .nrrd images
• do we need a new data type for MITK (derived from BaseData?)
  • no. i think we can use 8-bit images and assign a color to each label via a lookup table
  • in the case we switch to 1-bit images, we will need to add it to MITK (derived from BaseData)
• how could these be mapped types for rendering?
  • if we continue with 8-bit images, we can use the same mappers we have so far
  • if we switch to 1-bit images, maybe not [seba TODO, check this]
• how does I/O work?
  • same as currently: .nrrd images
• do we need a new data type for MITK (derived from BaseData?)
  • no. i think we can use 8-bit images and assign a color to each label via a lookup table
  • in the case we switch to 1-bit images, we will need to add it to MITK (derived from BaseData)
• how could these be mapped types for rendering?
  • we can use the same mappers we have so far

Implications on the implementation

• what parts of MITK would need to be changed?
  • mitk::Tool, mitk::ToolManager, mitk::ContourUtils, ...
  • in QmitkNodeDescriptorManager add a QmitkNodeDescriptor "Labeled Image" for images with pixel type "unsigned char" (see if we want to keep purely binary images)
  • searching the code for "binary" occurrences:

               + In mitk::ImageMapperGL2D::Paint, volume calculation should slightly change: itkimage->GetCountOfMaxValuedVoxelsNoRecompute() will need to have a "label" parameter to count the voxels with that given value.
               + In mitk::ImageMapperGL2D::SetDefaultProperties(), the heuristic for a labeled image would be: "uchar" == image->GetPixelType().GetItkTypeAsString(), PixelType::GetBpe() == 8 or alike
               + change names of "outline binary" to e.g. "outline label" 

• what needs to be changed in the Segmentation view?
  • new controls:

               + label manager widget ( e.g. have a QmitkDataStorageTable): should easily allow to (multiple) select labels for visualization or editing/locking toggling, change their representation (filled or border line), enable volume render feedback, etc. 

• Use the old datastorage combobox (IMHO: the current working/reference node selection approach via the workbench selection service is not user friendly since work with the segmentation bundle requires a lot of data (multiple) selection and deselection.)

Agreed working plan

1. Agree on how to handle overlapping structures. If we work on this part of the application, we should get it right and produce a UI concept that will work both with overlapping and mutually-exclusive structures.
   1. Implement a proof of concept for using the ITK label object classes
      • bundle should be able to create a new object and prepare it for rendering (LUT or similar)
      • the contour tool should work with this object, remaining tools don't count yet
      • interpolation can be ignored at this step
2. make interpolation work again
   1. enhance the new application
      • implement other tools, mostly ignoring old code and replacing it with ITK/VTK implementations
      • implement locking, sticking and other useful features