생물정보학 앙상블 딥러닝

2020.12.07

Ensemble deep learning in bioinformatics

Abstract

The remarkable flexibility and adaptability of ensemble methods and deep learning modelshave led to the proliferation of their application in bioinformatics research.

• Traditionally, these two machine learning techniques have largely been treated as independent methodologies in bioinformatics applications.

However, the recent emergence of ensemble deep learning—wherein the two machine learning techniques are combined to achieve synergistic improvements in model accuracy, stability and reproducibility—has prompted a new wave of research and application.

• Here, we share recent key developments in ensemble deep learning and look at how their contribution has benefited a wide range of bioinformatics research from basic sequence analysis to systems biology.
• While the application of ensemble deep learning in bioinformatics is diverse and multifaceted, we identify and discuss the common challenges and opportunities in the context of bioinformatics research.

We hope this Review Article will bring together the broader community of machine learning researchers, bioinformaticians and biologists to foster future research and development in ensemble deep learning, and inspire novel bioinformatics applications that are unattainable by traditional methods.

Introduction

Bioinformatics, an interdisciplinary field of research, is at the centre of modern molecular biology, where computational methods are developed and utilized to transform biological data into knowledge and translate them for biomedical applications.

• Among the various computational methods utilized in bioinformatics research, machine learning, a branch of artificial intelligence characterized by data-driven model building, has been the key enabling computational technology1 .
• At the forefront of machine learning, ensemble learning and deep learning have independently made a substantial impact on the field of bioinformatics through their widespread applications, from basic nucleotide and protein sequence analysis to systems biology2,3 .

Until recently, ensemble and deep learning models have largely been treated as independent methodologies in bioinformatics applications.

• The fast-growing synergy between these two popular techniques, however, has attracted a new wave of development and application of next-generation machine learning methods referred to as ensemble deep learning (Fig. 1a).
• The root of ensemble deep learning can be traced back two decades, when ensembles of neural networks were found to reduce generalization error4 .
• However, the recent resurgence of ensemble deep learning models has brought about new ideas, algorithms, frameworks and architectures that substantially enrich the old paradigm.
• Through its novel application to a wide range of biological and biomedical research, ensemble deep learning is unleashing its power in dealing with key challenges, including small sample size, high-dimensionality, imbalanced class distribution, and noisy and heterogeneous data generated from diverse cellular and biological systems using an array of high-throughput omics technologies. These computational, methodological and technological undertakings and breakthroughs together are leading a phenomenal transformation of bioinformatics.

Both ensemble learning and deep learning methods have been extensively studied and reviewed in the context of bioinformatics applications5,6 .

• However, the emergence of ensemble deep learning and its application in bioinformatics has yet to be documented.
• With the aim of providing a reference point to foster research in the increasingly popular field of ensemble deep learning and its application to various challenges in bioinformatics, in this Review Article we revisit the foundation of ensemble and deep learning, and summarize and categorize the latest developments in ensemble deep learning.
• This is followed by a survey of ensemble deep learning applications in bioinformatics.
• We then discuss the remaining challenges and opportunities that we hope will inspire future research and development across multiple disciplines.

Basics of ensemble and deep learning

Ensemble learning refers to a class of strategies where instead of building a single model, multiple ‘base’ models are combined to perform tasks such as supervised and unsupervised learning7 .

• Classic ensemble methods for supervised learning fall into three categories: bagging-, boosting- and stacking-based methods.
  • In bagging8 , individual base models are trained on subsets of data sampled randomly with replacement (Fig. 1b).
  • In boosting9 , models are trained sequentially (Fig. 1c), where subsequent models focus on previous misclassified samples.
  • In stacking, a meta-learner is trained to optimally combine the predictions made by base models10 .
• Like supervised ensemble learning, conventional unsupervised ensemble learning, such as ensemble clustering11 , also relies on the generation and integration of base models (Fig. 1d).
• While their variants, including more advanced methods reviewed in the next section, have also been used in ensemble learning, a guiding principle in designing ensemble methods has been ‘many heads are better than one’12 .

Deep learning, a branch of machine learning, is rooted in artifi-cial neural networks13.

• The most fundamental architecture of deep learning models is the densely connected neural network (DNN), consisting of a series of layers of neurons; each of these is connected to all neurons in the previous layer14 .
• More sophisticated models expand on the basic architectures.
• In convolutional neural networks (CNNs) 15, each layer comprises a series of filters that ‘slide over’ the output of the previous layer to extract local features across different parts of the input.
• In recurrent neural networks (RNNs) 16 , circuits are created to feed the output of a layer back into the same layer along with new input, allowing the model to act on dependencies between upstream and downstream values in a sequence.
• Variants of RNNs have been proposed to enable more effective learning in long-term dependency tasks, with the two most common ones being long short-term memory (LSTM) 17 and gated recurrent unit (GRU) 18 .
• In residual neural networks (ResNet) 19 , shortcuts between upstream and downstream layers are introduced to improve the effectiveness of backpropagation in networks with many hidden layers.
• In autoencoders 20 , networks are constructed with an encoder and a decoder that together learn a more efficient latent space representation of the original higher-dimensional data.
• Although the difference between traditional neural networks and deep learning may seem elusive, the latter is increasingly defined by its unique architectures and ability to learn complex data representations that are beyond the capacity of classic models 21 .

Ensemble deep learning

• Deep learning is well known for its power to approximate almost any function and increasingly demonstrates predictive accuracy that surpasses human experts. However, deep learning models are not without shortcomings: they often exhibit high variance and may fall into local loss minima during training. Indeed, empirical results of ensemble methods that combine the output of multiple deep learning models have been shown to achieve better generalizability than a single model22 . In addition to simple ensemble approaches such as averaging output from individual models, combining heterogeneous models enables multifaceted abstraction of data, and may lead to better learning outcomes23 . In this section, we categorize and summarize the most representative ensemble deep learning strategies for both supervised and unsupervised tasks.

Supervised ensemble deep learning.

Ensemble across multiple models.

• The aggregation of multiple and often independent deep learning models is the most straightforward application of ensemble deep learning to classify (Fig. 2a).
  • As diversity of individual networks is an essential characteristic of a good ensemble model24 , a variety of strategies exist to promote diversity of base networks.
    • One approach is to encourage negative correlation in the classification error of base models25 .
      • The key motivation behind promoting negative correlation among base models is to encourage complementary learning of the training data to achieve better generalizability of the ensemble.
    • An alternative approach to increase base model diversity is through multiple choice learning in which each network is ‘specialized’ on a particular subset of data during the training step26 .
      • An issue associated with training and storing multiple models is the computational and storage demand involved.
      • To address this, methods that perform knowledge distillation have become increasingly popular27 .
      • One such implementation is based on the concept of a teacher–student network framework, where the teacher networks are selected from a pool of pre-trained networks and the student network distils knowledge of multiple teachers into a single and often simpler network28,29 .
      • The testing phase is storage and computationally efficient, as the samples only need to pass through a single student network.

Ensemble within a single model.

Ensemble with model branching.

Unsupervised ensemble deep learning.

Ensemble across multiple models.

Ensemble within a single model.

Theoretical advances for ensemble deep learning.

Bioinformatics applications of ensemble deep learning

Sequence analysis.

Genome analysis.

Gene expression.

Structural bioinformatics.

Proteomics.

Systems biology.

Multi-omics.

Bioimage informatics.

Challenges and opportunities

Small sample size.

High-dimensionality and class imbalance.

Data noise and heterogeneity.

Model interpretability.

Choice of network architecture.

Computational expense.

Future outlook

While the ensemble of neural networks has existed long before the deep learning era, the recent development of ensemble deep learning has substantially enriched the field with novel architectures and ensemble strategies that greatly improve model accuracy, reliability and efficiency. These innovations, together with properties such as robustness to small sample size, high-dimensionality and data noise, have transformed ensemble deep learning into a new force, leading to remarkable and widespread breakthroughs across different fields of bioinformatics applications. Nonetheless, many of the advanced ensemble techniques that harness the power of recent deep learning architectures remain under-explored in their application to bioinformatics. In addition, the development and application of models that enable interpretation of biological systems are still in their infancy. We hope this Review Article has sparked thoughts on ensemble deep learning across multiple disciplines, and will inspire future research and applications that embraces the myriad of ensemble deep learning strategies to revolutionize biological and biomedical research.

ref

5. Yang, P., Hwa Yang, Y., Zhou, B. B. & Zomaya, A. Y. A review of ensemble methods in bioinformatics. Curr. Bioinform. 5, 296–308 (2010).

6. Min, S., Lee, B. & Yoon, S. Deep learning in bioinformatics. Briefings Bioinform. 18, 851–869 (2017).