Multi Label Segmentation

Following a proposal of Sebastian Ordas to support multi-label segmentations, this page will be used to sketch future changes to the Segmentation bundle and the related framework.

Shortcomings of the current implementation

• 8-bit images are used for segmentations by default, wasting 7 of 8 bits, occupying far more memory than technically necessary

Wishlist

• what is not possible or cumbersome and should be changed?
  • multiple labels on the same segmentation image: have 8-bit images with 0 value as background and the remaining 255 values as labeled objects.
    • this would imply non-overlapping structures. This might be a problem, unless users use another segmentation in those cases (they would have done so anyway in the current approach)
    • we should find a way to segment overlapping structures and assign two memberships to a voxel.
• if we'd use only a single bit we would still reduce memory consumption to 1/8. This could be taken as an intermediate step in the re-engineering process.
  • Should try itk::SingleBitBinaryImage: see Insight Journal paper Alternative Memory Models for ITK Images - http://www.insight-journal.org/browse/publication/646.
  • Will need an I/O mechanism see: http://old.nabble.com/Re%3A-SingleBitBinaryImage-td30804691.html
  • itk::SingleBitBinaryImage is yet not multi-threaded. see http://www.itk.org/pipermail/insight-users/2010-November/038761.html
• what enhancements would help in which situations?
  • Primary importance enhancements: memory reduction will greatly improve usability when working with several segmentations and big datasets
  • Secondary importance enhancements:
    • have label "lock"/"editable" boolean property
    • "stick" label boundaries to neighboring labels: e.g. if the label is smoothed or dilated, so will be its neighbors. If the label is eroded, the "sticked" neighbors will grow into the space left by the eroded label.
    • "background" label should be created by default, and should also be possible to lock it.
    • "label splitter" tool: given a label (segmentation), construct a new label from each connected subpart (3D method).
    • The post http://bugs.mitk.org/show_bug.cgi?id=6591 by Andreas Fetzer looks very interesting. TODO: ask which use cases he had in mind to implement this tool (marching squares?)
    • "label assigner": given a label and a clicked position on it, build a new label from the connected subpart around that position (3D method). The same mechanism can be used to delete the connected region or add it to another existing label.
    • "label assigner" would differ from "label splitter" so that in the former, the connected region around the clicked position is either assigned to a new label, to an existing label (indicated by the user), or deleted. In the "label splitter", all connected regions of a given label will be assigned a new (and different) label automatically.
    • have a workable "paintbrush". The current implementation is just not usable. Based on a post found in http://bugs.mitk.org/show_bug.cgi?id=6589 I could improve its use quite a lot (TODO: see how to send patch). Does it make sense to have a 3D paint brush?
• have a 2D/3D region growing tool based on ITK classes [seba: add comment regarding MITK's 3D grow tool]
  • useful resources:
    • vtkEdge paintbrush classes e.g. http://www.vtkedge.org/doc/nightly/html/classvtkKWEPaintbrushMergeSketches.html#_details
    • itkSnap http://www.itksnap.org/pmwiki/pmwiki.php to support multi-label images
    • ITK's label object classes (http://www.insight-journal.org/browse/publication/176) could be useful
    • Segmentation module in Amira (as it used to be in v3.1) http://www.amira.com/images/stories/pdf/31/Amira31-manual.pdf (see Sec. 2.5)
• how does I/O work?
  • same as currently: .nrrd images. In case of single-bit images, there's yet no IO support. See http://old.nabble.com/Re%3A-SingleBitBinaryImage-td30804691.html as alternative.
• do we need a new data type for MITK (derived from BaseData?)
  • no. i think we can use 8-bit images and assign a color to each label via a lookup table
  • in the case we switch to 1-bit images, we will need to add it to MITK (derived from BaseData)
• how could these be mapped types for rendering?
  • if we continue with 8-bit images, we can use the same mappers we have so far
  • if we switch to 1-bit images, maybe not [seba TODO, check this]

Implications on the implementation

• what parts of MITK would need to be changed?
  • mitk::Tool, mitk::ToolManager, mitk::ContourUtils, ...
  • in QmitkNodeDescriptorManager add a QmitkNodeDescriptor "Labeled Image" for images with pixel type "unsigned char" (see if we want to keep purely binary images)
  • searching the code for "binary" occurrences:
    • + In mitk::ImageMapperGL2D::Paint, volume calculation should slightly change: itkimage->GetCountOfMaxValuedVoxelsNoRecompute() will need to have a "label" parameter to count the voxels with that given value. + In mitk::ImageMapperGL2D::SetDefaultProperties(), the heuristic for a labeled image would be: "uchar" == image->GetPixelType().GetItkTypeAsString(), PixelType::GetBpe() == 8 or alike + change names of "outline binary" to e.g. "outline label"
• what needs to be changed in the Segmentation view?
  • new controls:
    • + label manager widget ( e.g. have a QmitkDataStorageTable): should easily allow to (multiple) select labels for visualization or editing/locking toggling, change their representation (filled or border line), enable volume render feedback, etc.
• Use the old datastorage combobox (IMHO: the current working/reference node selection approach via the workbench selection service is not user friendly since work with the segmentation bundle requires a lot of data (multiple) selection and deselection.)

Working plan

1. Use normal 3D+t images (single channel, simple gray values, unsigned integer types) to represent labels
   • each bit of a byte represents a single laben, e.g. "Liver"
   • this allows to work withoverlapping structure
   • using color lookup tables allows to use existing code for rendering and file handling
   • storing multiple segmentations can be done using existing file I/O, putting multiple labels into a single file (leads to less cluttered directories)
   • A label map should be able to use any integer type (i.e. 8, 16, 32 bit) - default could be 8 for now, but leave possibility for extension open.
2. We need a couple of filters to handle multi-label images:
   • extract a single channel from a multi-label image
     • input: multi-label image
     • parameter: which channel/bit should be selected? (e.g. 6th bit)
     • parameter: foreground and background values in the resulting image
     • output: "classical" binary image, 8 bit, specific foreground and background values
3. integrate a binary image into a multi-label image
   • input: multi-label image
   • parameter: "classical" binary image, 8 bit, specific foreground and background values
   • parameter: target channel/bit
   • parameter: overlap mode - Allow overlapping segmentations: yes/no. If overlap is allowed, target channel is set to new values without checking or modifying any other channels. If overlap is not allowed, only 1 channel can be set at any given position. By default, a segmented pixel in the new image deletes the label in all other channels. However, other channels can be locked to avoid modification (see next parameter), in this case nothing is modified at that position.
   • parameter: locked channels - if overlap is not allowed, this bitmask specifies which channels have priority in a conflict, i.e. if the target channel or the conflicting channel should be set
   • output: multi-label image
4. Replace mitk::Image::IsSegmentation() by flag (i.e. not trying to analyze the image). This flag is set when creating a segmentation or loading an image. For this, menu requires additional entry "Load segmentation".
5. Develop widget to
   • select active channel
   • allow proper naming and coloring
   • set locking mask

Integrate this into tool framework.

1. Open questions:
   • How to store colors and names for each segmentation channel (in application and on disk)?
2. make old functionality (all tools, interpolation) work with the new classes
3. enhance with new functions, e.g. interactive 3D region-grower