Advertisement

Graph neural networks in TensorFlow-Keras with RaggedTensor representation (kgcnn)

  • Patrick Reiser
    Correspondence
    Corresponding author at: Institute of Nanotechnology, Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany.
    Affiliations
    Institute of Nanotechnology, Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany

    Institute of Theoretical Informatics, Karlsruhe Institute of Technology (KIT), Am Fasanengarten 5, 76131 Karlsruhe, Germany
    Search for articles by this author
  • André Eberhard
    Affiliations
    Institute of Theoretical Informatics, Karlsruhe Institute of Technology (KIT), Am Fasanengarten 5, 76131 Karlsruhe, Germany
    Search for articles by this author
  • Pascal Friederich
    Correspondence
    Corresponding author at: Institute of Theoretical Informatics, Karlsruhe Institute of Technology (KIT), Am Fasanengarten 5, 76131 Karlsruhe, Germany.
    Affiliations
    Institute of Nanotechnology, Karlsruhe Institute of Technology (KIT), Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany

    Institute of Theoretical Informatics, Karlsruhe Institute of Technology (KIT), Am Fasanengarten 5, 76131 Karlsruhe, Germany
    Search for articles by this author

      Highlights

      • Python package for graph convolutional neural networks in Tensorflow-Keras using RaggedTensors.
      • Flexible integration and modular layers for setting up custom graph learning models.
      • Transparent tensor representation allows readable coding style and makes code easy to debug.
      • Popular graph neural networks are implemented as an example.
      • Implemented models expose hyperparameters for easy customization, tuning and experimenting.

      Abstract

      Graph neural networks are a versatile machine learning architecture that received a lot of attention recently due to its wide range of applications. In this technical report, we present an implementation of graph convolution and graph pooling layers for TensorFlow-Keras models, which allows a seamless and flexible integration into standard Keras layers to set up graph models in a functional way. We developed the Keras Graph Convolutional Neural Network Python package kgcnn based on TensorFlow-Keras which focus on a transparent tensor structure passed between layers and an ease-of-use mindset.

      Keywords

      Code metadata
      Tabled 1
      Current code versionv1.0.0
      Permanent link to code/repository used for this code versionhttps://github.com/SoftwareImpacts/SIMPAC-2021-57
      Permanent link to reproducible capsulehttps://codeocean.com/capsule/8499626/tree/v1
      Legal code licenceMIT Licence
      Code versioning system usedGit
      Software code languages, tools, and services usedPython 3
      Compilation requirements, operating environments & dependenciesTensorFlow 2.4
      If available link to developer documentation/manualhttps://kgcnn.readthedocs.io/en/latest/index.html
      Support email for questions[email protected]
      Software metadata
      Tabled 1
      Current software versionv1.0.0
      Permanent link to executables of this versionhttps://pypi.org/project/kgcnn/
      Permanent link to reproducible capsulehttps://codeocean.com/capsule/8499626/tree/v1
      Legal software licenceMIT Licence
      Computing platforms/Operating systemsLinux, Microsoft Windows, Mac OS
      Installation requirements & dependenciesTensorFlow 2.4
      If available, link to user manual - if formally published include a reference to the publication in the reference list
      Support email for questions[email protected]

      1. Introduction

      Graph neural networks (GNNs) are a natural extension of common neural network architectures such as convolutional neural networks (CNN) [
      • LeCun Y.
      • Boser B.
      • Denker J.S.
      • Henderson D.
      • Howard R.E.
      • Hubbard W.
      • Jackel L.D.
      Backpropagation applied to handwritten zip code recognition.
      ,
      • LeCun Y.
      • Kavukcuoglu K.
      • Farabet C.
      Convolutional networks and applications in vision.
      ,
      • Krizhevsky A.
      • Sutskever I.
      • Hinton G.E.
      Imagenet classification with deep convolutional neural networks.
      ] for image classification to graph structured data [
      • Kipf T.N.
      • Welling M.
      Semi-supervised classification with graph convolutional networks.
      ]. For example, recurrent [
      • Scarselli F.
      • Gori M.
      • Tsoi A.C.
      • Hagenbuchner M.
      • Monfardini G.
      The graph neural network model.
      ,
      • Dai H.
      • Kozareva Z.
      • Dai B.
      • Smola A.
      • Song L.
      Learning steady-states of iterative algorithms over graphs.
      ], convolutional [
      • Kipf T.N.
      • Welling M.
      Semi-supervised classification with graph convolutional networks.
      ,
      • Schütt K.T.
      • Sauceda H.E.
      • Kindermans P.-J.
      • Tkatchenko A.
      • Müller K.-R.
      Schnet – a deep learning architecture for molecules and materials.
      ,
      • Niepert M.
      • Ahmed M.
      • Kutzkov K.
      Learning convolutional neural networks for graphs.
      ,
      • Battaglia P.W.
      • Pascanu R.
      • Lai M.
      • Rezende D.
      • Kavukcuoglu K.
      Interaction networks for learning about objects, relations and physics.
      ] and spatial–temporal [
      • Yu B.
      • Yin H.
      • Zhu Z.
      Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting.
      ] graph neural networks as well as graph autoencoders [
      • Pan S.
      • Hu R.
      • Long G.
      • Jiang J.
      • Yao L.
      • Zhang C.
      Adversarially regularized graph autoencoder for graph embedding.
      ,
      • Kipf T.N.
      • Welling M.
      Variational graph auto-encoders.
      ] and graph transformer models [
      • Yao S.
      • Wang T.
      • Wan X.
      Heterogeneous graph transformer for graph-to-sequence learning.
      ,
      • Chen B.
      • Barzilay R.
      • Jaakkola T.
      Path-augmented graph transformer network.
      ] have been reported in literature. A graph G=(V,E) is defined as a set of vertices or nodes viV and edges eij=(vi,vj)E connecting two nodes. There are already comprehensive and extensive review articles for graph neural networks, which summarize and categorize relevant literature on graph learning [
      • Wu Z.
      • Pan S.
      • Chen F.
      • Long G.
      • Zhang C.
      • Yu P.S.
      A comprehensive survey on graph neural networks.
      ,
      • Zhou J.
      • Cui G.
      • Hu S.
      • Zhang Z.
      • Yang C.
      • Liu Z.
      • Wang L.
      • Li C.
      • Sun M.
      Graph neural networks: A review of methods and applications.
      ]. The most frequent applications of GNNs include node classification or graph embedding tasks. While node classification is a common task for very large graphs such as citation networks [
      • Kipf T.N.
      • Welling M.
      Variational graph auto-encoders.
      ] or social graphs [
      • Benchettara N.
      • Kanawati R.
      • Rouveirol C.
      Supervised machine learning applied to link prediction in bipartite social networks.
      ], graph embeddings learn a representation of smaller graphs such as molecules [
      • Schütt K.T.
      • Sauceda H.E.
      • Kindermans P.-J.
      • Tkatchenko A.
      • Müller K.-R.
      Schnet – a deep learning architecture for molecules and materials.
      ,
      • Merkwirth C.
      • Lengauer T.
      Automatic generation of complementary descriptors with molecular graph networks.
      ] or text classifications [
      • Angelova R.
      • Weikum G.
      Graph-based text classification: learn from your neighbors.
      ,
      • Rousseau F.
      • Kiagias E.
      • Vazirgiannis M.
      Text categorization as a graph classification problem.
      ]. Early work on applying neural networks to graphs [
      • Sperduti A.
      • Starita A.
      Supervised neural networks for the classification of structures.
      ] shaped the notion of graph neural networks and was further elaborated on by propagating information iteratively through the graph [
      • Scarselli F.
      • Gori M.
      • Tsoi A.C.
      • Hagenbuchner M.
      • Monfardini G.
      The graph neural network model.
      ,
      • Merkwirth C.
      • Lengauer T.
      Automatic generation of complementary descriptors with molecular graph networks.
      ,
      • Gori M.
      • Monfardini G.
      • Scarselli F.
      A new model for learning in graph domains.
      ,
      • Gallicchio C.
      • Micheli A.
      Graph echo state networks.
      ]. Graph networks may operate on the graph’s spectrum [
      • Bruna J.
      • Zaremba W.
      • Szlam A.
      • LeCun Y.
      Spectral networks and locally connected networks on graphs.
      ,
      • Defferrard M.
      • Bresson X.
      • Vandergheynst P.
      Convolutional neural networks on graphs with fast localized spectral filtering.
      ] or directly on its structure [
      • Micheli A.
      Neural network for graphs: A contextual constructive approach.
      ]. One of the most prominent graph convolutional neural network (GCN) introduced by Kipf et al. [
      • Kipf T.N.
      • Welling M.
      Semi-supervised classification with graph convolutional networks.
      ] stacks multiple convolutional and pooling layers for deep learning to generate a high-level node representation from which both a local node and global graph classification can be obtained. Most GCNs can be considered as message passing networks [
      • Gilmer J.
      • Schoenholz S.S.
      • Riley P.F.
      • Vinyals O.
      • Dahl G.E.
      Neural message passing for quantum chemistry.
      ,
      • Flam-Shepherd D.
      • Wu T.
      • Friederich P.
      • Aspuru-Guzik A.
      Neural message passing on high order paths.
      ], a class of graph networks in which information is propagated along edges between neighbouring nodes. In each update step t, the central node’s hidden representation hv is convolved with its neighbourhood given by:
      hvt+1=Ut(hvt,mvt+1),
      (1)


      where mvt denotes the aggregated message and Ut the update function. The message to update is usually acquired from summing message functions Mt from the neighbourhood N(v)={uV|(u,v)E} of node v:
      mvt+1=wN(v)Mt(hvt,hwt,evw).
      (2)


      More complex aggregation schemes are of course possible [
      • Hamilton W.L.
      • Ying R.
      • Leskovec J.
      Inductive representation learning on large graphs.
      ]. There is a large variety in convolution operators, which can be spectral-based [
      • Levie R.
      • Monti F.
      • Bresson X.
      • Bronstein M.M.
      Cayleynets: Graph convolutional neural networks with complex rational spectral filters.
      ] or spatial-based involving direct neighbours or a path of connected nodes to walk and collect information from [
      • Chen B.
      • Barzilay R.
      • Jaakkola T.
      Path-augmented graph transformer network.
      ,
      • Flam-Shepherd D.
      • Wu T.
      • Friederich P.
      • Aspuru-Guzik A.
      Neural message passing on high order paths.
      ]. Moreover, the message and update functions can be built from recurrent networks [
      • Yan T.
      • Zhang H.
      • Li Z.
      • Xia Y.
      Stochastic graph recurrent neural network.
      ], multi-layer perceptrons (MLP) [
      • Chen C.
      • Ye W.
      • Zuo Y.
      • Zheng C.
      • Ong S.P.
      Graph networks as a universal machine learning framework for molecules and crystals.
      ] or attention heads [
      • Veličković P.
      • Cucurull G.
      • Casanova A.
      • Romero A.
      • Liò P.
      • Bengio Y.
      Graph attention networks.
      ] which are complemented by a set of possible aggregation or pooling operations. Aggregation is usually done by a simple average of node representations or by a more refined set2set encoder part [
      • Vinyals O.
      • Bengio S.
      • Kudlur M.
      Order matters: Sequence to sequence for sets.
      ] as proposed by Gilmer et al. [
      • Gilmer J.
      • Schoenholz S.S.
      • Riley P.F.
      • Vinyals O.
      • Dahl G.E.
      Neural message passing for quantum chemistry.
      ]. A reduction of nodes in the graph is achieved by pooling similar to CNNs but which is much more challenging on arbitrarily structured data. Examples of possibly differentiable and learnable pooling filters introduced in literature are DiffPool [
      • Ying R.
      • You J.
      • Morris C.
      • Ren X.
      • Hamilton W.L.
      • Leskovec J.
      Hierarchical graph representation learning with differentiable pooling.
      ], EdgePool [
      • Diehl F.
      Edge contraction pooling for graph neural networks.
      ], gPool [
      • Gao H.
      • Ji S.
      Graph U-nets.
      ] HGP-SL,[
      • Ying R.
      • You J.
      • Morris C.
      • Ren X.
      • Hamilton W.L.
      • Leskovec J.
      Hierarchical graph representation learning with differentiable pooling.
      ], SAGPool [
      • Lee J.
      • Lee I.
      • Kang J.
      Self-attention graph pooling.
      ], iPool [
      • Gao X.
      • Xiong H.
      • Frossard P.
      Ipool – information-based pooling in hierarchical graph neural networks.
      ], EigenPool [
      • Ma Y.
      • Wang S.
      • Aggarwal C.C.
      • Tang J.
      Graph convolutional networks with eigenpooling.
      ] and graph based clustering methods such as the Graclus algorithm [
      • Defferrard M.
      • Bresson X.
      • Vandergheynst P.
      Convolutional neural networks on graphs with fast localized spectral filtering.
      ,
      • Rhee S.
      • Seo S.
      • Kim S.
      Hybrid approach of relation network and localized graph convolutional filtering for breast cancer subtype classification.
      ,
      • Dhillon I.S.
      • Guan Y.
      • Kulis B.
      Weighted graph cuts without eigenvectors a multilevel approach.
      ,
      • Simonovsky M.
      • Komodakis N.
      Dynamic edge-conditioned filters in convolutional neural networks on graphs.
      ].
      In order to utilize the full scope of different graph operations for setting up a custom GNN model, a modular framework of convolution and pooling layers is necessary. We briefly summarize and discuss existing graph libraries and their code coverage. Then, a short overview of representing graphs in tensor form is given. Finally, we introduce our graph package kgcnn for TensorFlow 2’s Keras API [
      • Abadi M.
      • Barham P.
      • Chen J.
      • Chen Z.
      • Davis A.
      • Dean J.
      • Devin M.
      • Ghemawat S.
      • Irving G.
      • Isard M.
      • Kudlur M.
      • Levenberg J.
      • Monga R.
      • Moore S.
      • Murray D.G.
      • Steiner B.
      • Tucker P.
      • Vasudevan V.
      • Warden P.
      • Wicke M.
      • Yu Y.
      • Zheng X.
      Tensorflow: A system for large-scale machine learning.
      ,
      • Abadi M.
      • Agarwal A.
      • Barham P.
      • Brevdo E.
      • Chen Z.
      • Citro C.
      • Corrado G.S.
      • Davis A.
      • Dean J.
      • Devin M.
      • Ghemawat S.
      • Goodfellow I.
      • Harp A.
      • Irving G.
      • Isard M.
      • Jia Y.
      • Jozefowicz R.
      • Kaiser L.
      • Kudlur M.
      • Levenberg J.
      • Mane D.
      • Monga R.
      • Moore S.
      • Murray D.
      • Olah C.
      • Schuster M.
      • Shlens J.
      • Steiner B.
      • Sutskever I.
      • Talwar K.
      • Tucker P.
      • Vanhoucke V.
      • Vasudevan V.
      • Viegas F.
      • Vinyals O.
      • Warden P.
      • Wattenberg M.
      • Wicke M.
      • Yu Y.
      • Zheng X.
      Tensorflow: Large-scale machine learning on heterogeneous distributed systems.
      ,
      • van Merriënboer B.
      • Bahdanau D.
      • Dumoulin V.
      • Serdyuk D.
      • Warde-Farley D.
      • Chorowski J.
      • Bengio Y.
      Blocks and fuel: Frameworks for deep learning.
      ], which seamlessly integrates graph layers into the Keras [
      • Chollet F.
      Keras.
      ] environment.

      1.1 Graph libraries

      Since graph neural networks require modified convolution and pooling operators, many Python packages for deep learning have emerged for either TensorFlow [
      • Abadi M.
      • Barham P.
      • Chen J.
      • Chen Z.
      • Davis A.
      • Dean J.
      • Devin M.
      • Ghemawat S.
      • Irving G.
      • Isard M.
      • Kudlur M.
      • Levenberg J.
      • Monga R.
      • Moore S.
      • Murray D.G.
      • Steiner B.
      • Tucker P.
      • Vasudevan V.
      • Warden P.
      • Wicke M.
      • Yu Y.
      • Zheng X.
      Tensorflow: A system for large-scale machine learning.
      ,
      • Abadi M.
      • Agarwal A.
      • Barham P.
      • Brevdo E.
      • Chen Z.
      • Citro C.
      • Corrado G.S.
      • Davis A.
      • Dean J.
      • Devin M.
      • Ghemawat S.
      • Goodfellow I.
      • Harp A.
      • Irving G.
      • Isard M.
      • Jia Y.
      • Jozefowicz R.
      • Kaiser L.
      • Kudlur M.
      • Levenberg J.
      • Mane D.
      • Monga R.
      • Moore S.
      • Murray D.
      • Olah C.
      • Schuster M.
      • Shlens J.
      • Steiner B.
      • Sutskever I.
      • Talwar K.
      • Tucker P.
      • Vanhoucke V.
      • Vasudevan V.
      • Viegas F.
      • Vinyals O.
      • Warden P.
      • Wattenberg M.
      • Wicke M.
      • Yu Y.
      • Zheng X.
      Tensorflow: Large-scale machine learning on heterogeneous distributed systems.
      ] or PyTorch [
      • Paszke A.
      • Gross S.
      • Massa F.
      • Lerer A.
      • Bradbury J.
      • Chanan G.
      • Killeen T.
      • Lin Z.
      • Gimelshein N.
      • Antiga L.
      • Desmaison A.
      • Köpf A.
      • Yang E.
      • DeVito Z.
      • Raison M.
      • Tejani A.
      • Chilamkurthy S.
      • Steiner B.
      • Fang L.
      • Bai J.
      • Chintala S.
      Pytorch: An imperative style, high-performance deep learning library.
      ] to work with graphs. We try to summarize the most notable ones without any claim that this list is complete.
      • PyTorch Geometric [
        • Fey M.
        • Lenssen J.E.
        Fast graph representation learning with pytorch geometric.
        ]. A PyTorch based graph library which is probably the largest and most used graph learning Python package up to date. It implements a huge variety of different graph models and uses a disjoint graph representation to deal with batched graphs (graph representations are discussed in the Supplementary Information).
      • Deep Graph Library (DGL) [
        • Wang M.
        • Zheng D.
        • Ye Z.
        • Gan Q.
        • Li M.
        • Song X.
        • Zhou J.
        • Ma C.
        • Yu L.
        • Gai Y.
        • Xiao T.
        • He T.
        • Karypis G.
        • Li J.
        • Zhang Z.
        Deep graph library: A graph-centric, highly-performant package for graph neural networks.
        ]. A graph model library with a flexible backend and a performance optimized implementation. It has its own graph data class with many loading options. Moreover, variants such as generative graph models [
        • Li Y.
        • Vinyals O.
        • Dyer C.
        • Pascanu R.
        • Battaglia P.
        Learning deep generative models of graphs.
        ], Capsule [
        • Sabour S.
        • Frosst N.
        • Hinton G.E.
        Dynamic routing between capsules.
        ] and transformers [
        • Vaswani A.
        • Shazeer N.
        • Parmar N.
        • Uszkoreit J.
        • Jones L.
        • Gomez A.N.
        • Kaiser L.
        • Polosukhin I.
        Attention is all you need.
        ] are included.
      • Spektral [
        • Grattarola D.
        • Alippi C.
        Graph neural networks in tensorflow and keras with spektral.
        ]. A Keras [
        • Chollet F.
        Keras.
        ] implementation of graph convolutional networks. Originally restricted to spectral graph filters [
        • Levie R.
        • Monti F.
        • Bresson X.
        • Bronstein M.M.
        Cayleynets: Graph convolutional neural networks with complex rational spectral filters.
        ], it now includes spatial convolution and pooling operations. The graph representation is made flexible by different graph modes detected by each layer.
      • StellarGraph [
        • Data61 C.
        Stellargraph machine learning library.
        ]. A Keras [
        • Chollet F.
        Keras.
        ] implementation that implements a set of convolution layers and a few pooling layers plus a custom graph data format.
      With PyTorch Geometric and DGL there are already large graph libraries with a lot of contributors from both academics and industry. The focus of the graph package presented here is on a neat integration of graphs into the TensorFlow-Keras framework in the most straightforward way. Thereby, we hope to provide Keras graph layers which can be quickly rearranged, changed and extended to build custom graph models with little effort. This implementation is focused on the new TensorFlow’s RaggedTensor class which is most suited for flexible data structures such as graphs and natural language. The main field of applications targeted with this package is graph embedding tasks of e.g. molecules, materials and contextual or knowledge graph learning.

      2. Description

      A flexible and simple integration of graph operations into the TensorFlow-Keras framework can be achieved via ragged tensors. As mentioned above, ragged tensors are capable of efficiently representing graphs and have inherent access to various methods within TensorFlow (see Supplementary Information). For more sophisticated pooling algorithms which cannot be operated on batches, a parallelization of individual graphs within the batch could be achieved with the TensorFlow map functionality, although this is less efficient than vectorized operations and depends on implementation details.
      Figure thumbnail fx1001
      Table 1Mean absolute validation error for single training on QM9 dataset for targets such as molecular orbital energies (HOMO, LUMO) and the energy gap EG in eV using popular GNN architectures implemented in kgcnn. No hyperparameter optimization or feature engineering was performed.
      ModelHOMO [eV]HOMO [eV]EG [eV]
      MPNN 
      • Gilmer J.
      • Schoenholz S.S.
      • Riley P.F.
      • Vinyals O.
      • Dahl G.E.
      Neural message passing for quantum chemistry.
      0.0610.0470.083
      Schnet 
      • Schütt K.T.
      • Sauceda H.E.
      • Kindermans P.-J.
      • Tkatchenko A.
      • Müller K.-R.
      Schnet – a deep learning architecture for molecules and materials.
      0.0440.0380.067
      MegNet 
      • Chen C.
      • Ye W.
      • Zuo Y.
      • Zheng C.
      • Ong S.P.
      Graph networks as a universal machine learning framework for molecules and crystals.
      0.0450.0370.066
      The Python package implements the following architectures as examples: GCN [
      • Kipf T.N.
      • Welling M.
      Semi-supervised classification with graph convolutional networks.
      ], Interaction network [
      • Battaglia P.W.
      • Pascanu R.
      • Lai M.
      • Rezende D.
      • Kavukcuoglu K.
      Interaction networks for learning about objects, relations and physics.
      ], message passing [
      • Gilmer J.
      • Schoenholz S.S.
      • Riley P.F.
      • Vinyals O.
      • Dahl G.E.
      Neural message passing for quantum chemistry.
      ], Schnet [
      • Schütt K.T.
      • Sauceda H.E.
      • Kindermans P.-J.
      • Tkatchenko A.
      • Müller K.-R.
      Schnet – a deep learning architecture for molecules and materials.
      ], MegNet [
      • Chen C.
      • Ye W.
      • Zuo Y.
      • Zheng C.
      • Ong S.P.
      Graph networks as a universal machine learning framework for molecules and crystals.
      ], Unet [
      • Gao H.
      • Ji S.
      Graph U-nets.
      ], GNN Explainer [
      • Ying R.
      • Bourgeois D.
      • You J.
      • Zitnik M.
      • Leskovec J.
      Gnnexplainer: Generating explanations for graph neural networks.
      ], GraphSAGE [
      • Hamilton W.L.
      • Ying R.
      • Leskovec J.
      Inductive representation learning on large graphs.
      ], GAT [
      • Veličković P.
      • Cucurull G.
      • Casanova A.
      • Romero A.
      • Liò P.
      • Bengio Y.
      Graph attention networks.
      ] and DimeNet++ [
      • Klicpera J.
      • Giri S.
      • Margraf J.T.
      • Günnemann S.
      Fast and uncertainty-aware directional message passing for non-equilibrium molecules.
      ]. The focus is set on graph embedding tasks, but also node and link classification tasks can be implemented using kgcnn. The models were tested with common bench-mark datasets such as Cora [
      • Sen P.
      • Namata G.
      • Bilgic M.
      • Getoor L.
      • Galligher B.
      • Eliassi-Rad T.
      Collective classification in network data.
      ], MUTAG [
      • Debnath A.K.
      • Lopez de Compadre R.L.
      • Debnath G.
      • Shusterman A.J.
      • Hansch C.
      Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity.
      ] and QM9[
      • Ramakrishnan R.
      • Dral P.O.
      • Rupp M.
      • von Lilienfeld O.A.
      Quantum chemistry structures and properties of 134 kilo molecules.
      ]. Typical benchmark accuracies such as chemical accuracy on the QM9 dataset are achieved with the corresponding models implemented in kgcnn (see Table 1).

      3. Software impact

      Learning dedicated graph embeddings is of high interest in for example classification tasks and molecular property [
      • Schütt K.T.
      • Gastegger M.
      • Tkatchenko A.
      • Müller K.-R.
      • Maurer R.J.
      Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions.
      ,
      • von Lilienfeld O.A.
      Quantum machine learning in chemical compound space.
      ,
      • Gómez-Bombarelli R.
      • Aguilera-Iparraguirre J.
      • Hirzel T.D.
      • Duvenaud D.
      • Maclaurin D.
      • Blood-Forsythe M.A.
      • Chae H.S.
      • Einzinger M.
      • Ha D.-G.
      • Wu T.
      • Markopoulos G.
      • Jeon S.
      • Kang H.
      • Miyazaki H.
      • Numata M.
      • Kim S.
      • Huang W.
      • Hong S.I.
      • Baldo M.
      • Adams R.P.
      • Aspuru-Guzik A.
      Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach.
      ,
      • Rupp M.
      Machine learning for quantum mechanics in a nutshell.
      ] and reaction predictions [
      • Do K.
      • Tran T.
      • Venkatesh S.
      Graph transformation policy network for chemical reaction prediction.
      ,
      • Gastegger M.
      • Marquetand P.
      High-dimensional neural network potentials for organic reactions and an improved training algorithm.
      ,
      • W. Coley C.
      • Jin W.
      • Rogers L.
      • F. Jamison T.
      • S. Jaakkola T.
      • H. Green W.
      • Barzilay R.
      • F. Jensen K.
      A graph-convolutional neural network model for the prediction of chemical reactivity.
      ,
      • Somnath V.R.
      • Bunne C.
      • Coley C.W.
      • Krause A.
      • Barzilay R.
      Learning graph models for template-free retrosynthesis.
      ]. A differentiable and continuous graph convolution model could in principle replace empirical constructed force fields for molecular dynamics simulation (MD) when trained on quantum mechanical calculations [
      • Schütt K.T.
      • Sauceda H.E.
      • Kindermans P.-J.
      • Tkatchenko A.
      • Müller K.-R.
      Schnet – a deep learning architecture for molecules and materials.
      ,
      • Behler J.
      Representing potential energy surfaces by high-dimensional neural network potentials.
      ,
      • Westermayr J.
      • Gastegger M.
      • Marquetand P.
      Combining schnet and SHARC: The schnarc machine learning approach for excited-state dynamics.
      ,
      • Behler J.
      Constructing high-dimensional neural network potentials: A tutorial review.
      ]. This includes work of the authors [
      • Flam-Shepherd D.
      • Wu T.
      • Friederich P.
      • Aspuru-Guzik A.
      Neural message passing on high order paths.
      ,
      • Reiser P.
      • Konrad M.
      • Fediai A.
      • Léon S.
      • Wenzel W.
      • Friederich P.
      Analyzing dynamical disorder for charge transport in organic semiconductors via machine learning.
      ,
      • Li J.
      • Reiser P.
      • Boswell B.R.
      • Eberhard A.
      • Burns N.Z.
      • Friederich P.
      • Lopez S.A.
      Automatic discovery of photoisomerization mechanisms with nanosecond machine learning photodynamics simulations.
      ]. In general, representation learning allows us to encode graph structured knowledge about interacting entities (i.e. nodes) and store it into a low-dimensional vector which can be further processed by machine learning (ML) models [
      • Hamilton W.L.
      • Ying R.
      • Leskovec J.
      Representation learning on graphs: Methods and applications.
      ]. The graph models re-implemented in kgcnn are widely applicable and are well established in literature [
      • Schütt K.T.
      • Sauceda H.E.
      • Kindermans P.-J.
      • Tkatchenko A.
      • Müller K.-R.
      Schnet – a deep learning architecture for molecules and materials.
      ,
      • Gilmer J.
      • Schoenholz S.S.
      • Riley P.F.
      • Vinyals O.
      • Dahl G.E.
      Neural message passing for quantum chemistry.
      ]. We believe that kgcnn can be used by researchers in other scientific fields to accommodate different graph learning tasks without detailed knowledge about graph representations or implementation details and by working with the well-known TensorFlow-Keras deep learning environment.

      4. Limitations and future development

      The current version of kgcnn is targeted for graph representation learning of small graphs such as molecules. Training on a single graph instance, e.g. a citation network which does not fit into memory is currently not explored. Although distribution strategies for training are already integrated into TensorFlow’s high-level Keras API, they have not yet been tested with the models provided by kgcnn. We plan to continue to extend the kgcnn library to incorporate new models, in particular pooling methods [
      • Ying R.
      • You J.
      • Morris C.
      • Ren X.
      • Hamilton W.L.
      • Leskovec J.
      Hierarchical graph representation learning with differentiable pooling.
      ] and to improve functionality. A future goal is to provide extremely large graph neural networks for distributed training on an exhaustive dataset for applications in chemistry and materials science.

      5. Conclusion

      In summary, we discussed a way to integrate graph convolution models into the TensorFlow-Keras deep learning framework. Main focus of our kgcnn package is the transparency of the tensor representation and the seamless integration with other Keras models.

      Declaration of Competing Interest

      The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

      Acknowledgement

      P.F. acknowledges funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 795206 .

      Appendix A. Supplementary data

      The following is the Supplementary material related to this article.

      References

        • LeCun Y.
        • Boser B.
        • Denker J.S.
        • Henderson D.
        • Howard R.E.
        • Hubbard W.
        • Jackel L.D.
        Backpropagation applied to handwritten zip code recognition.
        Neural Comput. 1989; 1 (arXiv:https://direct.mit.edu/neco/article-pdf/1/4/541/811941/neco.1989.1.4.541.pdf): 541-551https://doi.org/10.1162/neco.1989.1.4.541
        • LeCun Y.
        • Kavukcuoglu K.
        • Farabet C.
        Convolutional networks and applications in vision.
        in: Proceedings of 2010 IEEE International Symposium on Circuits and Systems. 2010: 253-256https://doi.org/10.1109/ISCAS.2010.5537907
        • Krizhevsky A.
        • Sutskever I.
        • Hinton G.E.
        Imagenet classification with deep convolutional neural networks.
        Commun. ACM. 2017; 60: 84-90https://doi.org/10.1145/3065386
        • Kipf T.N.
        • Welling M.
        Semi-supervised classification with graph convolutional networks.
        2017 (ArXiv:1609.02907 [Cs, Stat], http://arxiv.org/abs/1609.02907, URL arXiv:1609.02907)
        • Scarselli F.
        • Gori M.
        • Tsoi A.C.
        • Hagenbuchner M.
        • Monfardini G.
        The graph neural network model.
        IEEE Trans. Neural Netw. 2009; 20 (Conference Name: IEEE Transactions on Neural Networks): 61-80https://doi.org/10.1109/TNN.2008.2005605
        • Dai H.
        • Kozareva Z.
        • Dai B.
        • Smola A.
        • Song L.
        Learning steady-states of iterative algorithms over graphs.
        in: International Conference on Machine Learning. PMLR, 2018: 1106-1114 (URL http://proceedings.mlr.press/v80/dai18a.html)
        • Schütt K.T.
        • Sauceda H.E.
        • Kindermans P.-J.
        • Tkatchenko A.
        • Müller K.-R.
        Schnet – a deep learning architecture for molecules and materials.
        J. Chem. Phys. 2018; 148 (URL https://aip.scitation.org/doi/full/10.1063/1.5019779, Publisher: American Institute of Physics)241722https://doi.org/10.1063/1.5019779
        • Niepert M.
        • Ahmed M.
        • Kutzkov K.
        Learning convolutional neural networks for graphs.
        2016 (ArXiv:1605.05273 [Cs, Stat], http://arxiv.org/abs/1605.05273, URL arXiv:1605.05273)
        • Battaglia P.W.
        • Pascanu R.
        • Lai M.
        • Rezende D.
        • Kavukcuoglu K.
        Interaction networks for learning about objects, relations and physics.
        2016 (ArXiv:1612.00222 [Cs], http://arxiv.org/abs/1612.00222, URL arXiv:1612.00222)
        • Yu B.
        • Yin H.
        • Zhu Z.
        Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting.
        in: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. 2018: 3634-3640https://doi.org/10.24963/ijcai.2018/505 (http://arxiv.org/abs/1709.04875, URL arXiv:1709.04875)
        • Pan S.
        • Hu R.
        • Long G.
        • Jiang J.
        • Yao L.
        • Zhang C.
        Adversarially regularized graph autoencoder for graph embedding.
        2019 (ArXiv:1802.04407 [Cs, Stat], http://arxiv.org/abs/1802.04407, URL arXiv:1802.04407)
        • Kipf T.N.
        • Welling M.
        Variational graph auto-encoders.
        2016 (ArXiv:1611.07308 [Cs, Stat], http://arxiv.org/abs/1611.07308, URL arXiv:1611.07308)
        • Yao S.
        • Wang T.
        • Wan X.
        Heterogeneous graph transformer for graph-to-sequence learning.
        in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online2020: 7145-7154https://doi.org/10.18653/v1/2020.acl-main.640 (URL https://www.aclweb.org/anthology/2020.acl-main.640)
        • Chen B.
        • Barzilay R.
        • Jaakkola T.
        Path-augmented graph transformer network.
        2019 (ArXiv:1905.12712 [Cs, Stat], http://arxiv.org/abs/1905.12712, URL arXiv:1905.12712)
        • Wu Z.
        • Pan S.
        • Chen F.
        • Long G.
        • Zhang C.
        • Yu P.S.
        A comprehensive survey on graph neural networks.
        IEEE Trans. Neural Netw. Learn. Syst. 2020; (http://arxiv.org/abs/1901.00596, URL arXiv:1901.00596): 1-21https://doi.org/10.1109/TNNLS.2020.2978386
        • Zhou J.
        • Cui G.
        • Hu S.
        • Zhang Z.
        • Yang C.
        • Liu Z.
        • Wang L.
        • Li C.
        • Sun M.
        Graph neural networks: A review of methods and applications.
        2021 (ArXiv:1812.08434 [Cs, Stat], http://arxiv.org/abs/1812.08434, arXiv:1812.08434)
        • Benchettara N.
        • Kanawati R.
        • Rouveirol C.
        Supervised machine learning applied to link prediction in bipartite social networks.
        in: 2010 International Conference on Advances in Social Networks Analysis and Mining. 2010: 326-330https://doi.org/10.1109/ASONAM.2010.87
        • Merkwirth C.
        • Lengauer T.
        Automatic generation of complementary descriptors with molecular graph networks.
        J. Chem. Inform. Model. 2005; 45 (Publisher: American Chemical Society): 1159-1168https://doi.org/10.1021/ci049613b
        • Angelova R.
        • Weikum G.
        Graph-based text classification: learn from your neighbors.
        in: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ’06. Association for Computing Machinery, New York, NY, USA2006: 485-492https://doi.org/10.1145/1148170.1148254
        • Rousseau F.
        • Kiagias E.
        • Vazirgiannis M.
        Text categorization as a graph classification problem.
        in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China2015: 1702-1712https://doi.org/10.3115/v1/P15-1164 (URL https://www.aclweb.org/anthology/P15-1164)
        • Sperduti A.
        • Starita A.
        Supervised neural networks for the classification of structures.
        IEEE Trans. Neural Netw. 1997; 8 (Conference Name: IEEE Transactions on Neural Networks): 714-735https://doi.org/10.1109/72.572108
        • Gori M.
        • Monfardini G.
        • Scarselli F.
        A new model for learning in graph domains.
        in: Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., Vol. 2. 2005: 729-734 vol. 2https://doi.org/10.1109/IJCNN.2005.1555942
        • Gallicchio C.
        • Micheli A.
        Graph echo state networks.
        in: The 2010 International Joint Conference on Neural Networks (IJCNN). 2010: 1-8https://doi.org/10.1109/IJCNN.2010.5596796
        • Bruna J.
        • Zaremba W.
        • Szlam A.
        • LeCun Y.
        Spectral networks and locally connected networks on graphs.
        2014 (ArXiv:1312.6203 [Cs], http://arxiv.org/abs/1312.6203, URL arXiv:1312.6203)
        • Defferrard M.
        • Bresson X.
        • Vandergheynst P.
        Convolutional neural networks on graphs with fast localized spectral filtering.
        2017 (ArXiv:1606.09375 [Cs, Stat], http://arxiv.org/abs/1606.09375, URL arXiv:1606.09375)
        • Micheli A.
        Neural network for graphs: A contextual constructive approach.
        IEEE Trans. Neural Netw. 2009; 20 (Conference Name: IEEE Transactions on Neural Networks): 498-511https://doi.org/10.1109/TNN.2008.2010350
        • Gilmer J.
        • Schoenholz S.S.
        • Riley P.F.
        • Vinyals O.
        • Dahl G.E.
        Neural message passing for quantum chemistry.
        2017 (ArXiv:1704.01212 [Cs], http://arxiv.org/abs/1704.01212, URL arXiv:1704.01212)
        • Flam-Shepherd D.
        • Wu T.
        • Friederich P.
        • Aspuru-Guzik A.
        Neural message passing on high order paths.
        2020 (ArXiv:2002.10413 [Cs, Stat], http://arxiv.org/abs/2002.10413, URL arXiv:2002.10413)
        • Hamilton W.L.
        • Ying R.
        • Leskovec J.
        Inductive representation learning on large graphs.
        2018 (ArXiv:1706.02216 [Cs, Stat], http://arxiv.org/abs/1706.02216, URL arXiv:1706.02216)
        • Levie R.
        • Monti F.
        • Bresson X.
        • Bronstein M.M.
        Cayleynets: Graph convolutional neural networks with complex rational spectral filters.
        IEEE Trans. Signal Process. 2019; 67 (Conference Name: IEEE Transactions on Signal Processing): 97-109https://doi.org/10.1109/TSP.2018.2879624
        • Yan T.
        • Zhang H.
        • Li Z.
        • Xia Y.
        Stochastic graph recurrent neural network.
        2020 (ArXiv:2009.00538 [Cs, Stat], http://arxiv.org/abs/2009.00538, URL arXiv:2009.00538)
        • Chen C.
        • Ye W.
        • Zuo Y.
        • Zheng C.
        • Ong S.P.
        Graph networks as a universal machine learning framework for molecules and crystals.
        Chem. Mater. 2019; 31 (Publisher: American Chemical Society): 3564-3572https://doi.org/10.1021/acs.chemmater.9b01294
        • Veličković P.
        • Cucurull G.
        • Casanova A.
        • Romero A.
        • Liò P.
        • Bengio Y.
        Graph attention networks.
        2018 (ArXiv:1710.10903 [Cs, Stat], http://arxiv.org/abs/1710.10903, URL arXiv:1710.10903)
        • Vinyals O.
        • Bengio S.
        • Kudlur M.
        Order matters: Sequence to sequence for sets.
        2016 (ArXiv:1511.06391 [Cs, Stat], http://arxiv.org/abs/1511.06391, URL arXiv:1511.06391)
        • Ying R.
        • You J.
        • Morris C.
        • Ren X.
        • Hamilton W.L.
        • Leskovec J.
        Hierarchical graph representation learning with differentiable pooling.
        2019 (ArXiv:1806.08804 [Cs, Stat], http://arxiv.org/abs/1806.08804, URL arXiv:1806.08804)
        • Diehl F.
        Edge contraction pooling for graph neural networks.
        2019 (ArXiv:1905.10990 [Cs, Stat], http://arxiv.org/abs/1905.10990, URL arXiv:1905.10990)
        • Gao H.
        • Ji S.
        Graph U-nets.
        2019 (ArXiv:1905.05178 [Cs, Stat], http://arxiv.org/abs/1905.05178, URL arXiv:1905.05178)
        • Lee J.
        • Lee I.
        • Kang J.
        Self-attention graph pooling.
        2019 (ArXiv:1904.08082 [Cs, Stat], http://arxiv.org/abs/1904.08082, URL arXiv:1904.08082)
        • Gao X.
        • Xiong H.
        • Frossard P.
        Ipool – information-based pooling in hierarchical graph neural networks.
        2019 (ArXiv:1907.00832 [Cs, Stat], http://arxiv.org/abs/1907.00832, URL arXiv:1907.00832)
        • Ma Y.
        • Wang S.
        • Aggarwal C.C.
        • Tang J.
        Graph convolutional networks with eigenpooling.
        2019 (ArXiv:1904.13107 [Cs, Stat], http://arxiv.org/abs/1904.13107, URL arXiv:1904.13107)
        • Rhee S.
        • Seo S.
        • Kim S.
        Hybrid approach of relation network and localized graph convolutional filtering for breast cancer subtype classification.
        2018 (ArXiv:1711.05859 [Cs], http://arxiv.org/abs/1711.05859, URL arXiv:1711.05859)
        • Dhillon I.S.
        • Guan Y.
        • Kulis B.
        Weighted graph cuts without eigenvectors a multilevel approach.
        IEEE Trans. Pattern Anal. Mach. Intell. 2007; 29: 1944-1957https://doi.org/10.1109/TPAMI.2007.1115
        • Simonovsky M.
        • Komodakis N.
        Dynamic edge-conditioned filters in convolutional neural networks on graphs.
        2017 (ArXiv:1704.02901 [Cs], http://arxiv.org/abs/1704.02901, URL arXiv:1704.02901)
        • Abadi M.
        • Barham P.
        • Chen J.
        • Chen Z.
        • Davis A.
        • Dean J.
        • Devin M.
        • Ghemawat S.
        • Irving G.
        • Isard M.
        • Kudlur M.
        • Levenberg J.
        • Monga R.
        • Moore S.
        • Murray D.G.
        • Steiner B.
        • Tucker P.
        • Vasudevan V.
        • Warden P.
        • Wicke M.
        • Yu Y.
        • Zheng X.
        Tensorflow: A system for large-scale machine learning.
        2016: 265-283 (https://www.usenix.org/conference/osdi16/technical-sessions/presentation/abadi)
        • Abadi M.
        • Agarwal A.
        • Barham P.
        • Brevdo E.
        • Chen Z.
        • Citro C.
        • Corrado G.S.
        • Davis A.
        • Dean J.
        • Devin M.
        • Ghemawat S.
        • Goodfellow I.
        • Harp A.
        • Irving G.
        • Isard M.
        • Jia Y.
        • Jozefowicz R.
        • Kaiser L.
        • Kudlur M.
        • Levenberg J.
        • Mane D.
        • Monga R.
        • Moore S.
        • Murray D.
        • Olah C.
        • Schuster M.
        • Shlens J.
        • Steiner B.
        • Sutskever I.
        • Talwar K.
        • Tucker P.
        • Vanhoucke V.
        • Vasudevan V.
        • Viegas F.
        • Vinyals O.
        • Warden P.
        • Wattenberg M.
        • Wicke M.
        • Yu Y.
        • Zheng X.
        Tensorflow: Large-scale machine learning on heterogeneous distributed systems.
        2016 (ArXiv:1603.04467 [Cs], http://arxiv.org/abs/1603.04467, URL arXiv:1603.04467)
        • van Merriënboer B.
        • Bahdanau D.
        • Dumoulin V.
        • Serdyuk D.
        • Warde-Farley D.
        • Chorowski J.
        • Bengio Y.
        Blocks and fuel: Frameworks for deep learning.
        2015 (ArXiv:1506.00619 [Cs, Stat], http://arxiv.org/abs/1506.00619, URL arXiv:1506.00619 version: 1)
        • Chollet F.
        Keras.
        2015 (https://github.com/fchollet/keras, Publication Title: GitHub repository)
        • Paszke A.
        • Gross S.
        • Massa F.
        • Lerer A.
        • Bradbury J.
        • Chanan G.
        • Killeen T.
        • Lin Z.
        • Gimelshein N.
        • Antiga L.
        • Desmaison A.
        • Köpf A.
        • Yang E.
        • DeVito Z.
        • Raison M.
        • Tejani A.
        • Chilamkurthy S.
        • Steiner B.
        • Fang L.
        • Bai J.
        • Chintala S.
        Pytorch: An imperative style, high-performance deep learning library.
        2019 (ArXiv:1912.01703 [Cs, Stat], http://arxiv.org/abs/1912.01703, URL arXiv:1912.01703)
        • Fey M.
        • Lenssen J.E.
        Fast graph representation learning with pytorch geometric.
        2019 (ArXiv:1903.02428 [Cs, Stat], http://arxiv.org/abs/1903.02428, URL arXiv:1903.02428)
        • Wang M.
        • Zheng D.
        • Ye Z.
        • Gan Q.
        • Li M.
        • Song X.
        • Zhou J.
        • Ma C.
        • Yu L.
        • Gai Y.
        • Xiao T.
        • He T.
        • Karypis G.
        • Li J.
        • Zhang Z.
        Deep graph library: A graph-centric, highly-performant package for graph neural networks.
        2020 (ArXiv:1909.01315 [Cs, Stat], http://arxiv.org/abs/1909.01315, URL arXiv:1909.01315)
        • Li Y.
        • Vinyals O.
        • Dyer C.
        • Pascanu R.
        • Battaglia P.
        Learning deep generative models of graphs.
        2018 (ArXiv:1803.03324 [Cs, Stat], http://arxiv.org/abs/1803.03324, URL arXiv:1803.03324)
        • Sabour S.
        • Frosst N.
        • Hinton G.E.
        Dynamic routing between capsules.
        2017 (ArXiv:1710.09829 [Cs], http://arxiv.org/abs/1710.09829, URL arXiv:1710.09829)
        • Vaswani A.
        • Shazeer N.
        • Parmar N.
        • Uszkoreit J.
        • Jones L.
        • Gomez A.N.
        • Kaiser L.
        • Polosukhin I.
        Attention is all you need.
        2017 (ArXiv:1706.03762 [Cs], http://arxiv.org/abs/1706.03762, URL arXiv:1706.03762)
        • Grattarola D.
        • Alippi C.
        Graph neural networks in tensorflow and keras with spektral.
        2020 (ArXiv:2006.12138 [Cs, Stat], http://arxiv.org/abs/2006.12138, URL arXiv:2006.12138)
        • Data61 C.
        Stellargraph machine learning library.
        2018
        • Ying R.
        • Bourgeois D.
        • You J.
        • Zitnik M.
        • Leskovec J.
        Gnnexplainer: Generating explanations for graph neural networks.
        2019 (ArXiv:1903.03894 [Cs, Stat], http://arxiv.org/abs/1903.03894, URL arXiv:1903.03894)
        • Klicpera J.
        • Giri S.
        • Margraf J.T.
        • Günnemann S.
        Fast and uncertainty-aware directional message passing for non-equilibrium molecules.
        2020 (ArXiv:2011.14115 [Physics], http://arxiv.org/abs/2011.14115, URL arXiv:2011.14115)
        • Sen P.
        • Namata G.
        • Bilgic M.
        • Getoor L.
        • Galligher B.
        • Eliassi-Rad T.
        Collective classification in network data.
        AI Mag. 2008; 29 (https://ojs.aaai.org/index.php/aimagazine/article/view/2157, Number: 3): 93https://doi.org/10.1609/aimag.v29i3.2157
        • Debnath A.K.
        • Lopez de Compadre R.L.
        • Debnath G.
        • Shusterman A.J.
        • Hansch C.
        Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity.
        J. Med. Chem. 1991; 34 (Publisher: American Chemical Society): 786-797https://doi.org/10.1021/jm00106a046
        • Ramakrishnan R.
        • Dral P.O.
        • Rupp M.
        • von Lilienfeld O.A.
        Quantum chemistry structures and properties of 134 kilo molecules.
        Sci, Data. 2014; 1 (https://www.nature.com/articles/sdata201422, Number: 1 Publisher: Nature Publishing Group)140022https://doi.org/10.1038/sdata.2014.22
        • Schütt K.T.
        • Gastegger M.
        • Tkatchenko A.
        • Müller K.-R.
        • Maurer R.J.
        Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions.
        Nature Commun. 2019; 10 (URL https://www.nature.com/articles/s41467-019-12875-2, Number: 1 Publisher: Nature Publishing Group): 5024https://doi.org/10.1038/s41467-019-12875-2
        • von Lilienfeld O.A.
        Quantum machine learning in chemical compound space.
        Ange. Chem. Int. Ed. 2018; 57 (URL https://onlinelibrary.wiley.com/doi/abs/10.1002/anie.201709686): 4164-4169https://doi.org/10.1002/anie.201709686
        • Gómez-Bombarelli R.
        • Aguilera-Iparraguirre J.
        • Hirzel T.D.
        • Duvenaud D.
        • Maclaurin D.
        • Blood-Forsythe M.A.
        • Chae H.S.
        • Einzinger M.
        • Ha D.-G.
        • Wu T.
        • Markopoulos G.
        • Jeon S.
        • Kang H.
        • Miyazaki H.
        • Numata M.
        • Kim S.
        • Huang W.
        • Hong S.I.
        • Baldo M.
        • Adams R.P.
        • Aspuru-Guzik A.
        Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach.
        Nature Mater. 2016; 15 (URL https://www.nature.com/articles/nmat4717, Number: 10 Publisher: Nature Publishing Group): 1120-1127https://doi.org/10.1038/nmat4717
        • Rupp M.
        Machine learning for quantum mechanics in a nutshell.
        Int. J. Quantum Chem. 2015; 115 (URL https://onlinelibrary.wiley.com/doi/abs/10.1002/qua.24954): 1058-1073https://doi.org/10.1002/qua.24954
        • Do K.
        • Tran T.
        • Venkatesh S.
        Graph transformation policy network for chemical reaction prediction.
        in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD ’19. Association for Computing Machinery, New York, NY, USA2019: 750-760https://doi.org/10.1145/3292500.3330958
        • Gastegger M.
        • Marquetand P.
        High-dimensional neural network potentials for organic reactions and an improved training algorithm.
        J. Chem. Theory Comput. 2015; 11 (Publisher: American Chemical Society): 2187-2198https://doi.org/10.1021/acs.jctc.5b00211
        • W. Coley C.
        • Jin W.
        • Rogers L.
        • F. Jamison T.
        • S. Jaakkola T.
        • H. Green W.
        • Barzilay R.
        • F. Jensen K.
        A graph-convolutional neural network model for the prediction of chemical reactivity.
        Chem. Sci. 2019; 10 (https://pubs.rsc.org/en/content/articlelanding/2019/sc/c8sc04228d, Publisher: Royal Society of Chemistry): 370-377https://doi.org/10.1039/C8SC04228D
        • Somnath V.R.
        • Bunne C.
        • Coley C.W.
        • Krause A.
        • Barzilay R.
        Learning graph models for template-free retrosynthesis.
        2020 (ArXiv:2006.07038 [Cs, Stat], http://arxiv.org/abs/2006.07038, URL arXiv:2006.07038)
        • Behler J.
        Representing potential energy surfaces by high-dimensional neural network potentials.
        J. Phys.: Condens. Matter. 2014; 26 (Publisher: IOP Publishing)183001https://doi.org/10.1088/0953-8984/26/18/183001
        • Westermayr J.
        • Gastegger M.
        • Marquetand P.
        Combining schnet and SHARC: The schnarc machine learning approach for excited-state dynamics.
        J. Phys. Chem. Lett. 2020; 11 (Publisher: American Chemical Society): 3828-3834https://doi.org/10.1021/acs.jpclett.0c00527
        • Behler J.
        Constructing high-dimensional neural network potentials: A tutorial review.
        Int. J. Quantum Chem. 2015; 115 (https://onlinelibrary.wiley.com/doi/abs/10.1002/qua.24890): 1032-1050https://doi.org/10.1002/qua.24890
        • Reiser P.
        • Konrad M.
        • Fediai A.
        • Léon S.
        • Wenzel W.
        • Friederich P.
        Analyzing dynamical disorder for charge transport in organic semiconductors via machine learning.
        J. Chem. Theory Comput. 2021; (Publisher: American Chemical Society)https://doi.org/10.1021/acs.jctc.1c00191
        • Li J.
        • Reiser P.
        • Boswell B.R.
        • Eberhard A.
        • Burns N.Z.
        • Friederich P.
        • Lopez S.A.
        Automatic discovery of photoisomerization mechanisms with nanosecond machine learning photodynamics simulations.
        Chem. Sci. 2021; 12 (https://pubs.rsc.org/en/content/articlelanding/2021/sc/d0sc05610c, Publisher: The Royal Society of Chemistry): 5302-5314https://doi.org/10.1039/D0SC05610C
        • Hamilton W.L.
        • Ying R.
        • Leskovec J.
        Representation learning on graphs: Methods and applications.
        2018 (ArXiv:1709.05584 [Cs], http://arxiv.org/abs/1709.05584, URL arXiv:1709.05584)