Counting made clear

A guide to modern colony counting solutions

Introduction

Automated colony counting processes in the modern day rely to a large extent on image processing/computer vision based solutions, and often involve mechanical aids such as a very specific lighting and camera setup, or a high-quality, high-throughput imaging device which ensures that the input image/data is of sufficient quality to get a result that can be scientifically validated and verified by experienced biologists. 

The very first strides into the idea of automated colony counting generally utilised such aids. Devices created in the 1960s to assist with colony counting would often consist of a biological plate holder and a light source. Although technology has progressed in a great many ways since then, this is still the go to for many labs as a proven and efficacious method to assist biologists with colony counting.

Computer vision

The history of software based/algorithmic colony counting starts with the early history of image processing/computer vision based techniques. The foundations of computer vision were laid in the 1950-60s; increased knowledge into how the brain sees/processes images, and the advent of the age of computing were the major driving forces behind the movement. Early pioneers such as Hubel and Wiesel made the first conceptual leaps into this domain, mapping biological neural networks in cats to vision capabilities (Hubel and Wiesel, 1959). Some of the first work in this space was conducted by Larry Roberts, representing the entree of one of the first major image processing techniques still in widespread use today, the convolutional mask/filter. The Roberts Cross Edge Detector masks can still be seen in some image processing applications (Roberts, 1963).

Computer Vision as a field was formalised in the 1970s, and saw the addition of new techniques which derived further abstractions from the raw image data, such as shape information, edge/contour information, bulk colour information, and it improved on techniques from the 1960s, such as the Hough transform to obtain further data (Duda and Hart, 1972). Subsequently the 1980s saw even greater progression with the Canny edge detector (still widely used today in image generation, see below), scale invariant feature transforms (SIFT), and active contours (Kass, Witkin and Terzopoulos, 1988). 

Research from the 1990s onwards has generally focused on even higher-levels of abstraction, utilising statistical methods and machine learning derived techniques that allow for the determination of higher-level features such as faces, hand-written and typed digits/numbers, and crucially, phenotyping and disease characterisation in the field of medical imaging. The orchestration of robotic movements and interactions also began to receive significant interest at this time.

The latest advancements in the field of computer vision have brought it back to its inception, with artificial neural networks becoming prominent and achieving the best-in-class/state-of-the-art for techniques in fields such as object detection, image classification, segmentation, generating images, and image upscaling.

Image processing

Image processing can largely be divided into four classes of techniques. Low-level/early image processing techniques, high-level image processing techniques, machine learning/statistical based techniques, and artificial neural network/AI based techniques, with some inevitable crossover between the categories. I will cover each of these separately, in order to provide some description of what is happening in each of these types, and what is expected/achieved in each case. 

Low-level image processing techniques

Low-level image processing techniques focus on extracting basic features from images. They usually operate directly on pixel data, and examples of such techniques are image thresholding, edge detection, smoothing/blurring, and morphological transformations (i.e. stretch/shrink). More complex techniques still regarded as low-level and acting directly on pixel data are the Hough transform, Fourier transform, convolution/filtering, and histogram equalisation. They are generally early image processing techniques, and are still widely in use today, often in conjunction with other methods.

High-level image processing techniques

High-level techniques are generally more sophisticated, and used in tasks that require understanding the structure and content of an image. Such techniques include object detection/recognition, image segmentation, feature matching and point correspondence, pattern recognition, texture analysis, and shape analysis (including active contours). A variety of techniques are employed to aid these higher-level processing tasks, and many different implementations can be seen across the image processing world.

Statistical image processing techniques

Statistical techniques tend to fall mid-way between high-level image processing and low-level techniques, and are often utilised in high-level image-processing. These tend to fall under the umbrella of machine learning, and provide useful abstractions from the raw image data. Such techniques include Hidden Markov models, Markov Random fields, Markov Chain Monte Carlo sampling, Bayesian image processing, Principal component analysis, Kalman Filtering, Gaussian process regression, and statistical shape models

Neural network based image processing techniques

A variety of neural network architectures have been employed in aid of various image processing/computer vision tasks. These can be sub-classed in various ways, but the two main backbone techniques used in image processing today are the Convolutional Neural Network (CNN), and transformer-based models such as ViT. “Convolutional Neural Networks gained significant popularity after AlexNet, a deep CNN architecture, achieved a breakthrough performance in the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) (Krizhevsky, Sutskever and Hinton, 2012)”. The Vision-Transformer based model is even more recent, and again has pushed the state of the art forward compared to traditional CNN models, but CNN models are still very widely used in a variety of applications, given the greater familiarity with the model and its provenance.

Recap of the evolution of automated colony counting

As previously mentioned, medical imaging has benefited greatly from advances in image processing. Thousands of papers relating to software-based or algorithmic techniques for interpreting medical image data are available on the wider internet, and automated colony counting methods feature strongly among the represented papers. Early attempts at automating colony counting emerged in the first half of the 20th century. Such devices included simple mechanical counters that required users to point manually to each colony, basic image projection systems that enlarged the plate image, and mechanical devices to aid colony counting, such as overlaying a grid on the plate and providing darkfield illumination.

In the 60s-70s, early computer-based solutions making use of low-level image processing techniques soon emerged; these utilised newly available digital images, and techniques such as thresholding to try and obtain colony counts. They were limited in accuracy and flexibility, and generally struggled with overlapping colonies and variable shapes/sizes. The period of the 80s-90s, with improvements in underlying technologies and the advent of widescale computer access saw further progress within the field. Higher-level techniques began to be more frequently used, allowing for more robust and efficacious implementations of automated colony counting methods, including separating overlapping colonies, and better distinguishing from background noise. Charge coupled device (CCD) cameras were another key innovation during this period, which provided higher resolution imaging, and allowed systems to become more accurate and reliable. 

The digital revolution of the 2000s brought yet more sophistication, and allowed for fast high-throughput systems to be developed, with later advances in the 21st century bringing artificial neural network/AI based image processing into the mix. Interfaces became more user-friendly, and automation-ready capabilities such as barcode-scanning and API interfaces for integration with other products such as robot arms, have now made automated colony counting a highly scaleable and robust method in active use at many research-based, medical, and industrial facilities worldwide. Despite the advent of AI, a lot of solutions still rely on traditional machine-learning/statistical methods of Image processing, coupled with a high-level architecture that allows for successful resolution/classification of colonies present. There are however still some problems that can and do result from poor input data, following the old “garbage-in garbage-out” axiom.

Choosing the right solution for your needs

Several issues still occur within modern setups that can cause problems which interfere with automated colony counting methods and other associated techniques, such as phenotyping according to colour and morphology, and ensuring effective separation of colonies. Some of the most prominent amongst these are poor resolution, poor contrast, inconsistent lighting (flatness-of-light effects on one area of the plate), atmospheric effects, specular reflections (when using lidded plates), and poor colour fidelity. 
Although modern software and algorithmic solutions can mitigate these to an extent, where reliability, robustness, repeatability and academic rigour are required, the use of mechanical devices to aid the process of obtaining a scientifically valid count remains true. In today’s market, high-quality, high-resolution imaging devices can eliminate a lot of the aforementioned imaging side-effects, which tend to contribute negatively to any colony counting algorithmic solution, no matter how robust the algorithmic solution is. 

To provide an example from real life, an algorithm may rely on an adaptive thresholding mechanism to eliminate unnecessary noise from the image (background subtraction), with the adaptive part of the thresholding mechanism designed to compensate for flatness of light effects. However, a specular light effect from atmospheric conditions, could still cause some of the less contrasted colonies on the plate to be regarded as part of the background, which then renders that pixel data unavailable to the general colony counting mechanism. 

Newer techniques relying more on artificial neural network oriented solutions may not require so much pre-processing. The traditional convolutional neural network uses a variety of convolution masks/filters to extract feature data from the image. To aid it in this endeavour, it will want the maximum amount of pixel data available in the image, which it will convert into a latent space representation to aid it in its final task, in this case instance segmentation and colony counting. Even in this case, the prospective user will still find better performance, if they can ensure the input data is of high quality with high availability of pixel data, and atmospheric effects such as specular lighting are reduced. Ideally, images should be presented in a similar condition to the data the artificial neural network/AI was trained on. Although convolutional neural network filtering for feature extraction will do a reasonably good job of reducing things such as atmospheric effects in the process of its feature extraction mechanisms, especially if using data augmentation during the training process, for reasons of repeatability and academic rigour, it is still best to ensure that plates are presented to the algorithm using the same conditions for lighting, camera, resolution, and colour fidelity. This can be achieved with a very specific laboratory setup of camera and lighting, or alternatively, a specific high-quality imaging device, which is able to capture these details and present them to the algorithm. 

Phenotyping according to colour and morphology is another more recent advance when it comes to automated colony counting techniques. Artificial neural networks are again pushing the state-of-the-art on these types of classifications, and several object detection methods have been developed in recent years that can make such an endeavor a very fast and scalable process. This is however, perhaps more than any other technique, subject to differences that might occur from the use of different cameras and lighting setups, image resolution, and colour fidelity. 

If, for example, operating in a high-throughput facility such as a water treatment plant, and checking for the presence of certain microorganisms, the associated laboratory will want to be able to consistently identify particular microorganisms present in their samples, and to do so in a reliable and efficient manner. A task like this would certainly require a consistently high-definition and high quality image, free of specular reflections with a true-to-life colour fidelity, which would assist not only any prospective algorithmic solutions, but any human biologists carrying out manual counting/classification tasks, or undergoing verification checks on the algorithm’s presented result. In such a case, the go-to for most would be an imaging device designed to satisfy those requirements, highlighting the importance of using technological aids when considering an automated colony counting solution.