MITK-ToF

One of the main challenges in computer-assisted interventions is the intra-operative imaging of tissue shape, motion and morphology. This information is a prerequisite for the registration of multi-modal patient specific data that can be used for surgical planning, diagnosis and intra-operative guidance. In the context of minimally invasive surgery, an increasingly attractive approach involves 3D reconstruction of the soft-tissue surfaces using Time-of-Flight (ToF) cameras or structured light camneras, which provide depth images and corresponding intensity or color images with high update rates. Using a depth camera as intraoperative imaging modality requires a seamless integration in the clinical workflow. In recent years, the Medical Imaging Interaction Toolkit (MITK) became one of the most popular open-source software frameworks for developing medical image processing and image-guided therapy applications. Similar to MITK-IGT, MITK-ToF is an extension of MITK for image-guided surgery applications. It integrates depth camera devices into the toolkit and allows the development of software for fast and flexible acquisition and processing of depth data.

1. Structure

MITK-ToF was designed to be flexible, extendable, robust, performant, as well as portable and was developed according to the software process of MITK. It provides three basic modules (cf. Fig. 1): ToFHardware, ToFProcessing and ToFUI.

ToFHardware

This module provides means for acquisition of images from a depth device in real-time as well as for recording and playing depth data. A Controller class (cf. Fig. 2) serves as the direct interface to the hardware. The Device class performs a threaded update on the controller and ensures a loss-less data acquisition with an internal buffer structure. Access to the images is realized in the Grabber class which is implemented as the source of an MITK filter pipeline. ToFHardware contains several sub-modules such as Kinect or PMD for respective camera implementations. The aim is to group these devices by external SDK, hence PMD devices are in one module and the Kinect for Xbox 360 (module Kinect) which is accessed via OpenNI is in a different module than the Kinect for Xbox One (module KinectV2). A complete overview of currently supported devices can be found here: http://mitk.org/wiki/Mitk-ToF-Camera_Support_Levels

A clue of the software framework is, that if you want to implement a new device you do not have to change the code of MITK-ToF itself. You can just generate your own sub-module for the new hardware and it will be automatically integrated via the AUTOLOAD_WITH feature. An example how to implement a new device can be found in the documentation: http://docs.mitk.org/nightly/GeneratingDeviceModulesPage.html

ToFProcessing

This module provides filter classes for processing the acquired ToF data. It seamlessly integrates into the pipeline architecture of MITK. Existing image filters of MITK and ITK as well as functions of OpenCV can be applied to the acquired ToF data in a straightforward manner. Currently, filters for surface generation, visualization, and basic preprocessing are implemented in the toolkit.

ToFUI

Several basic GUI elements have been designed facilitate application development based on MITK-ToF. Implemented as widgets (with the QT toolkit for cross-platform application and GUI development) they can be integrated into Qt-based MITK applications. The basic widgets comprise a ConnectionWidget for configuring the ToF device with the standard parameters and establishing a connection to the device, a RecorderWidget for playing and recording ToF data, a VisualizationWidget for changing visualization parameters such as color look-up tables for the gray-scale ToF images, a SurfaceGenerationWidget to configure the 3D reconstruction and a FilterConfigurationWidget for controlling the current configuration of the so-called ToFCompositeFilter (see below). A sample application based on these widgets is shown in Fig. 3 and a sample plugin (org.mitk.gui.qt.tofutil) is available for build with the MitkWorkbench application.

2. Current Features

The basic version currently supports the cameras CamCube 3.0, CamCube 2.0, CamBoard and O3 (PMDTechnologies, Siegen, Germany). New range cameras can be integrated in a straightforward manner by implementing corresponding controller and device classes.

The currently implemented ToF specific filter classes are:

• ToFCompositeFilter: Combines multiple common preprocessing steps to achieve a faster processing result. In the current version, a threshold filter for basic image segmentation as well as a bilateral filter, a spatial median filter and a temporal median filter for image smoothing can be applied.
• ToFImageRecorderFilter: Allows for recording images which have optionally already been processed by a filter pipeline.
• ToFDistanceImageToSurfaceFilter: Converts the index based distance image to a coordinate-based 3D surface representation. Texture information can be optionally added to the surface, e.g. from the corresponding intensity image.

New processing filters can be integrated by deriving them from the base class of MITK for image filters.

3. Publications and Contact

The following publications explain the MITK-ToF structure in more detail:

[1] K. Yung et al.: MITK-ToF: Time-of-Flight Kamera-Integration in das Medical Imaging Interaction Toolkit, Bildverarbeitung für die Medizin 2011, pages 204-208 (German)

[2] A. Seitel et al.: MITK-ToF - range data within MITK, Int. J. CARS, published online, May 2011

[3] A. Seitel et al.: MITK-ToF - range data within MITK, Int. J. CARS, Suppl. 1, page 126, Berlin (Germany), June 2011

For support and bug-reports please use the MITK Mailinglist

4. Getting Started

• Information about how to get started using MITK-ToF for ToF based applications can be found on a seperate page (MITK-ToF - Getting started)
• Using Kinect with MITK-ToF