If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Institute of Physics of the Czech Academy of Sciences, v.v.i., Cukrovarnická 10, 162 00 Praha 6, Czech RepublicRegional Centre of Advanced Technologies and Materials, Palacký University, Šlechtitelů 27, 78371 Olomouc, Czech Republic
Institute of Physics of the Czech Academy of Sciences, v.v.i., Cukrovarnická 10, 162 00 Praha 6, Czech RepublicRegional Centre of Advanced Technologies and Materials, Palacký University, Šlechtitelů 27, 78371 Olomouc, Czech Republic
Institute of Physics of the Czech Academy of Sciences, v.v.i., Cukrovarnická 10, 162 00 Praha 6, Czech RepublicRegional Centre of Advanced Technologies and Materials, Palacký University, Šlechtitelů 27, 78371 Olomouc, Czech Republic
Feature extraction and sorting specifically tailored to scanning tunneling microscopy images.
•
Modular construction includes single function call defaults and examples for tailoring to image data.
•
Discussion of computer vision approach in manuscript to inform further development.
Abstract
Scanning tunneling and atomic force microscopies (STM/nc-AFM) are rapidly progressing to offer unprecedented spatial resolution of a diverse array of chemical species. In particular, they are employed to characterize on-surface chemical reactions by directly examining precursors and products. Chiral effects and self-assembled structures can also be investigated. This open source, modular, python based scheme automates the categorization of a variety of molecules present in medium sized (10 × 10 to 100 × 100 nm) scanned probe images.
Scanned probe techniques such as scanning tunneling microscopy (STM) and non-contact atomic force microscopy (nc-AFM) have now made direct investigation of on-surface reactions a routine experimental technique [
]. As the diverse ecosystem of organic molecules accessible by these techniques continues to grow so does the need for more sophisticated tools to extract quantitative information from larger and more complicated datasets of molecule imaging.
Extracting statistics from STM images is often done by hand [
]. This necessarily limits the size and complexity of statistical problems that can be tackled using these datasets. To address this limitation in extracting and categorizing molecules in STM images, we developed an automated scheme utilizing existing image processing libraries written using the Python programming language.
The existing widely used programs for the analysis of scanned probe data (WSxM [
] is a powerful and widely used piece of software in the biology community, but lacks compatibility and relevance for the datasets this package is designed to address. Another hindrance is that these image processing tools for biology applications are often written using commercial, closed source MATLAB code. Digital Surf has feature categorization tools but is also commercial and closed source [
There has been recent progress in developing machine-learning based tools for the automated assignment of atomic structure and defect information from scanned probe datasets, [
]. However these solutions address comparatively homogeneous datasets of more simple structures compared the present case, or require the computational resources and data (real or artificial) to train a convolutional neural network [
]. We wanted to avoid this approach and instead develop a lightweight tool to quickly identify statistical trends in a large number of STM images with a diverse manifold of initially unknown molecular species; the functions offered in this package were used to prepare Fig. 3 in Hellerstedt et al. [
Fig. 1Architecture and usage of main functions. Yellow boxes indicate optional user input parameters, blue boxes are visual outputs of the program. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Our approach addresses some specific obstacles. There are practical limitations to the data resolution versus the number of species sampled. The sorting method needs to account for different adsorption configurations of the same species (e.g., rotations and chirality). While explicit pairwise template matching has advantages in robustness, it suffers from the computational demands scaling in quadrature with the number of molecules to be sorted.
Our solution to these problems relies on the Zernike polynomial basis set to provide a “fingerprint” of coefficients representing every molecule [
]. These coefficients are robust in response to rotations and noise in the data. In conjunction with other physically motivated coefficients, they provide the input to clustering algorithms for sorting the molecules. There are many different algorithms that can be applied to this type of data for the purposes of grouping like kinds of molecules together. We discuss some of the approaches we applied and their relative efficacy.
2. Software description
This package consists of a set of functions written using the Python libraries Numpy [
]. In practice, it is a series of functions called sequentially as shown in Fig. 1. We developed the current implementation to be useful ‘out-of-the-box’ with only minimum skill prerequisites required (e.g., installing a python distribution and necessary packages). The modular nature of the data processing flow allows for a great deal of flexibility to address different datasets. The open source, repository based distribution of this package allows for further customization, and the possibility for community based development and improvement on these core functionalities.
2.1 Software architecture
We first read in the image data and apply filters to make the image suitable for molecule template extraction using an adaptive threshold. After identifying closed contours corresponding to the perimeters of each molecule, we use the interior data of each contour as a representative template image. We then extract numerical features from each molecule, consisting of the calculated Zernike moments [
] to categorize these features. Further functionality is provided to visualize the sorted categories, and make manual corrections.
Fig. 1 shows the typical use of the provided functions. We rely on the Nanonispy library to read in data from one of the prevalent formats (Nanonis SXM files). This generates an array of pixel data and the rescaling factors to convert to real-space distances.
2.2 Software functionalities
The default filtering function provided performs a Gaussian filter and plane fit subtraction of the image data. We calculate the global Otsu threshold [
] and use that to scale the offset value for feature extraction using a local thresholding method. We include diagnostic functions that can plot the filtered image data with the extracted contours, labeled by number, as well as a grid view of all the extracted molecule templates.
Zernike moments are the coefficients for representing an image decomposed into the orthogonal Zernike polynomial basis set [
]. This method has attracted attention and effort in the computer vision community for being rotationally invariant. This property is particularly useful in the context of our application because we wish to match molecules regardless of how they are absorbed on the sample surface. We utilize the Mahotas library for calculating the Zernike moments for each molecule template [
]. We set the median template diagonal length as the default value of the Zernike radius input, an assumption that could fail depending on the homogeneity of the templates.
In addition to rotational invariance, these moments are insensitive to translation, mirroring, and rescaling. To account for differences in the real space footprint of each molecule, we additionally calculate the length, as well as the maximum pixel value within each contour.
With these characteristic moments and physical length scales, we perform a clustering analysis using the algorithms available in the Scikit-learn library [
] with a threshold factor between 0.1 to 0.4 to be most effective at sorting images with no a priori knowledge of the number of categories. When the number of sorting categories is known, hierarchical clustering was found to be more accurate. The most effective sorting was accomplished using affinity propagation, where the cluster center preferences were defined using a hand-selected set of exemplar molecules.
The invariance of the Zernike moments to mirror symmetry in particular, makes them insensitive to differentiating between chiral molecules. Absolute quantifications of chirality are surprisingly difficult to define [
]. We developed a function to do a pairwise comparison of molecules within each sorted category. By comparing each molecule and its mirror image, they can be sorted into right- and left-handed categories.
All these image, contour, correlations, and categorization data can be exported/ saved for later use. Also included is a function using the interactive features of Matplotlib to manually categorize all the molecules.
3. Illustrative examples
Representative outputs for the three example datasets from hand-selected exemplar molecules are shown in Fig. 2, Fig. 3, Fig. 4. The work on the 9-azidophenanthrene system which was the source of the data in Fig. 2, Fig. 3 provided the original impetus to develop this package [
Fig. 4Helicene data sorted using hand-selected exemplars. The dimers (categories 1 and 4) were subsequently sorted by chirality. 80×80 nm, 1024×1024 pixels. Compare to Fig. 4 of Stetsovych et al. [5].
This is the first effort to our knowledge to create an open source scheme to automate the task of counting and sorting hundreds of molecules, specifically tailored for scanning probe microscopy datasets. We expect this package (available on github [
]), to be immediately useful for anyone tasked with extracting population statistics from images comparable to those included as examples. This tool has already proven useful by making it easy to quickly extract and visualize population statistics in image datasets [
]. In principle this could be applied to video on a frame-by-frame basis but the intended use is for single, static images. By scaling the statistics and quantitative information that can be extracted from these types of datasets, we anticipate that this will facilitate the ability to answer novel and more nuanced questions about how chemical processes unfold on a surface. We hope to reveal and address limitations by having this set of functions applied to the increasingly diverse ecosystem of on-surface chemistry being explored by the greater scanning probe community.
5. Conclusions
Here we have presented our efforts to automate the counting and sorting of molecules in topographic images acquired with a scanning tunneling microscope. The example datasets, images with several hundred molecules, can be sorted on a personal computer in seconds. The python based, open source, modular design of the components strives to be immediately accessible to a non-expert user, and allow for significant modification/ customization to suit individual needs. We hope that use and feedback from the wider community will allow this tool to continue to be developed and provide utility for advancing the understanding of on-surface chemistry experiments.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
S. Edalatmanesh and A. Caparrós provided useful discussion in developing the solution to the chirality sorting problem. We thank Dr. Karthäuser for critical reading of this manuscript. J.H. acknowledges the support of the CIES/ Czech Fulbright commission during the development of this package.
On-surface self-assembly and characterization of a macromolecular charge transfer complex by scanning tunneling microscopy and spectroscopy.
(Ph.D. thesis)University of British Columbia,
2016https://doi.org/10.14288/1.0305019 (URL https://open.library.ubc.ca/cIRcle/collections/ubctheses/24/items/1.0305019)
00041-0&id=gr1.jpg){kind=link}
00041-0&id=gr2.jpg){kind=link}
00041-0&id=gr3.jpg){kind=link}
00041-0&id=gr4.jpg){kind=link}