Multilevel Ensemble Approach in Classification Problems

Abstract

The article discusses an approach to solving classification problems using multi-level heterogeneous ensembles. The approach allows reducing forecast errors by gradually reducing bias and variance using multi-level heterogeneous ensembles of forecast models. The main features and prerequisites for creating ensembles are considered. The total error of the machine learning algorithm is analyzed. It consists of three components: noise, bias and variance. The constituents of these components are investigated and determined. The process of creating a heterogeneous ensemble is considered in detail. The most common methods of aggregating forecast values are analyzed: bagging, boosting and staking. The rationale for choosing the type of models for creating a heterogeneous multi-level ensemble structure is presented. A two-level architecture of the classification system based on the staking and bagging methods is proposed. A new classification algorithm has been developed to build a multi-level ensemble of models based on different basic methods. An example of implementing multi-level heterogeneous ensembles for solving classification problems is considered on two datasets: Blood Transfusion Service Center, and ILPD (Indian Liver Patient Dataset). To assess the quality of classifiers, many appropriate quality indicators were used. The results of the ensembles' functioning were analyzed. The effectiveness of multi-level heterogeneous ensembles in solving classification problems was proven.