Multi Label Segmentation

Using multiple segmentations with the Segmentation bundle

Following a proposal of Sebastian Ordas to support multi-label segmentations, this page will be used to sketch future changes to the Segmentation bundle and the related framework.

Shortcomings of the current implementation

• 8-bit images are used for segmentations by default, wasting 7 of 8 bits, occupying far more memory than technically necessary

Wishlist

• what is not possible or cumbersome and should be changed?
  • multiple labels on the same segmentation image: have 8-bit images with 0 value as background and the remaining 255 values as labeled objects.
  • this would imply non-overlapping structures. This might be a problem, unless users use another segmentation in those cases (they would have done so anyway in the current approach)
  • we will definitely need a way to segment overlapping structures and assign a voxel two memberships.
• if we'd use only a single bit we would still reduce memory consumption to 1/8
  • this is a good idea and could be taken as an intermediate state
• what enhancements would help in which situations?

Primary importance enhancements

• memory reduction will greatly improve usability when working with several segmentations and big datasets

Secondary importance enhancements

         o have label "lock"/"editable" boolean property

         o "stick" label boundaries to neighboring labels: e.g. if the label is smoothed or dilated, so will be its neighbors. If the label is eroded, the "sticked" neighbors will grow into the space left by the eroded label.

         o "background" label should be created by default, and should also be possible to lock it.

         o "label splitter" tool: given a label(segmentation), construct a new label from each connected subpart (3D method).

         o The post http://bugs.mitk.org/show_bug.cgi?id=6591 by Andreas Fetzer looks very interesting. TODO: ask which use cases he had in mind to implement this tool (marching squares?)

         o "label assigner": given a label and a clicked position on it, build a new label from the connected subpart around that position (3D method). The same mechanism can be used to delete the connected region or add it to another existing label. 

         o "label assigner" would differ from "label splitter" so that in the former, the connected region around the clicked position is either assigned to a new label, to an existing lablel, or deleted. In the "label splitter", all connected regions of a given label will be assigned a new (and different) label.

         o have a workable "paintbrush". The current implementation is just not usable. Based on a post found in: [seba complete] I could improve its use quite a lot. Does it make sense to have a 3D paint brush?

• have a 2D/3D region growing tool based on ITK classes [seba: add comment regarding MITK's 3D grow tool]
• useful resources:

         o vtkEdge paintbrush classes e.g. http://www.vtkedge.org/doc/nightly/html/classvtkKWEPaintbrushMergeSketches.html#_details
         o itkSnap http://www.itksnap.org/pmwiki/pmwiki.php to support multi-label images
         o ITK's label object classes (http://www.insight-journal.org/browse/publication/176) could be useful
         o Segmentation module in Amira (as it used to be in v3.1) http://www.amira.com/images/stories/pdf/31/Amira31-manual.pdf (see Sec. 2.5) 

• how does I/O work?

         o same as currently: .nrrd images 

• do we need a new data type for MITK (derived from BaseData?)

         o no. i think we can use 8-bit images and assign a color to each label via a lookup table
         o in the case we switch to 1-bit images, we will need to add it to MITK (derived from BaseData)

• how could these be mapped types for rendering?

         o if we continue with 8-bit images, we can use the same mappers we have so far 
         o if we switch to 1-bit images, maybe not [seba TODO, check this]

• how does I/O work?
  • same as currently: .nrrd images
• do we need a new data type for MITK (derived from BaseData?)
  • no. i think we can use 8-bit images and assign a color to each label as usual

#!wiki tip
 * I don't know what you "usually" do, I rarely work with multi-label images
 * are you thinking of assigning a lookup table to the image? that would work

• how could these be mapped types for rendering?
  • we can use the same mappers we have so far

Implications on the implementation

• what parts of MITK would need to be changed?
  • mitk::Tool, mitk::ToolManager, mitk::ContourUtils, ...
  • in QmitkNodeDescriptorManager add a QmitkNodeDescriptor "Labeled Image" for images with pixel type "unsigned char" (see if we want to keep purely binary images)
  • searching the code for "binary" occurrences:
    • In mitk::ImageMapperGL2D::Paint, volume calculation should slightly change: itkimage->GetCountOfMaxValuedVoxelsNoRecompute() will need to have a "label" parameter to count the voxels with that given value.
    • In mitk::ImageMapperGL2D::SetDefaultProperties(), the heuristic for a labeled image would be: "uchar" == image->GetPixelType().GetItkTypeAsString(), PixelType::GetBpe() == 8 or alike
    • change names of "outline binary" to e.g. "outline label"
• what needs to be changed in the Segmentation view?
  • new controls:
    • label manager widget ( e.g. have a QmitkDataStorageTable): should easily allow to (multiple) select labels for visualization or editing/locking toggling, change their representation (filled or border line), enable volume render feedback, etc.
• Use the old datastorage combobox (IMHO: the current working/reference node selection approach via the workbench selection service is not user friendly since work with the segmentation bundle requires a lot of data (multiple) selection and deselection.)

Proposed working plan (Sebastian)

• Stage 1: add lock/edit feature to the current approach
  • Add "binaryimage.locked" bool propery to data node (e.g. in mitk::ImageMapperGL2D::SetDefaultProperties() ). Should be set to TRUE by default.
  • Have in Tool Manager the following methods:
    • GetEditableData();
    • GetLockedData();
• For every mitk::Tool, upon finish editing (e.g. mouse release), correct the feedback image for every intersection with other editable/locked segmentations
  • The Segmentation Interpolation should also respect other locked and editable labels
• Stage 2: do not use binary images any more
• Stage 3:

#!wiki tip
I'd suggest a differnt approach, see below. The reason behind that is: I assume changing the internal representation of a segmentation is a major change and will somehow affect most other features. Esp. the suggested locking feature would work differently for the current segmentations and multi-label segmentations.

Proposed working plan (Daniel)

1. Agree on how to handle overlapping structures. If we work on this part of the application, we should get it right and produce a UI concept that will work both with overlapping and mutually-exclusive structures.
2. Implement a proof of concept for using the ITK label object classes
   • bundle should be able to create a new object and prepare it for rendering (LUT or similar)
   • the contour tool should work with this object, remaining tools don't count yet
   • interpolation can be ignored at this step
3. make interpolation work again
4. enhance the new application
   • implement other tools, mostly ignoring old code and replacing it with ITK/VTK implementations
   • implement locking, sticking and other useful features