Automated segmentation of regions of interest
Segmentation involves breaking up an image into regions of interest (ROIs) based on pixel intensity values, texture, shape, etc. This can be done manually by annotating regions, but automated segmentation is often preferred.
Segmentation can be broadly divided into thresholding-based methods and machine-learning-based methods. The former are based on pixel intensity values, while the latter can learn from a large number of features to distinguish between objects and background.
Thresholding-based segmentation
Traditional or classical image segmentation typically utilizes thresholding to separate objects from the background based pixel intensity values—a form of semantic segmentation. This process results in a binary mask that can be refined using morphological operations (dilation, erosion, filling of holes, etc.). Then, to segment the individual objects (instance segmentation), additional transformations are performed as part of connected components analysis to label groups of pixels that are connected to each other while separating touching objects, e.g. watershed algorithm. Note that this is a flexible, generalist approach that can work well for many types of images, but can be limited if signal to noise ratio is poor or objects are not very distinct.
Tools for thresholding-based segmentation
Machine learning segmentation
Machine learning approaches have proven very powerful for segmenting images, especially when the objects of interest are not well separated, there is a high amount of noise, poor overall contrast, etc. Rather than just using intensity values, as in thresholding, machine learning models can learn from a large number of features to distinguish between objects and background. This section will be broadly divided into classification-based approaches and deep learning approaches.
Segmentation using classification-based approaches
Classification-based approaches take computed features and use them to classify each pixel or object as belonging to a particular class (e.g. object or background). This can be done using a wide range of classifiers, such as random forests, support vector machines, etc. Typically they do not require a large amount of training data, so “painting” some labels can be sufficient. However, they can be limited in their ability to generalize to new data. Importantly, both training and inference can be very fast. Finally, classification approaches can be used for both pixel-wise and object-wise segmentation, meaning that these approaches can be applied to both segment objects, as well as classify those objects into different classes, such a cell or tissue type.
Tools for classification-based segmentation
Segmentation using deep learning
Deep learning (DL) approaches have made automated segmentation tractable for complex images, such as those with many objects, complex shapes, or low contrast. Deep learning models can learn from a large number of features, but are less interpretable and require a large amount of training data. Further, training can be slow and require a lot of computational resources, particularly graphical processing units (GPU). However, frequently DL approaches can generalize well to new data and pre-trained models are available. Typically, deep learning approaches can be used for both pixel-wise and object-wise segmentation, meaning that these approaches can be applied to both segment objects, as well as classify those objects into different classes, such a cell or tissue type. This is a rapidly evolving field with very many algorithms available, most implemented in Python using torch or tensorflow machine learning frameworks. Here we will focus on robust, most commonly used tools that have GUIs for applying and/or training the models.