Approach to Identification of Anomalous Values in Analysis Tasks and Data Pre-processing

Abstract

The article presents an approach to the identification of anomalous values at the stage of data analysis and pre-processing when solving machine learning problems. Detection and identification of anomalous values in the data are important for the efficiency of the preprocessing of the data set. A methodology for identifying and processing emissions in data sets has been developed. The methodology was developed within the framework of a systematic approach to solving tasks of analysis and pre-processing of data. It consists of three steps. In the first step, methods are used to detect outliers in the data set. In the second step, the causes of emissions are analyzed. In the third step, methods for processing abnormal values are chosen. The following methods of detecting and processing extreme values are considered in the work: statistical tests, model tests, metric methods, iterative methods, task substitution methods, machine learning methods, algorithm ensembles. Examples of the use of all methods of identifying emissions in the data sets described in the first step of the algorithm were analyzed. The importance of a systematic approach to the detection and processing of anomalous values at the stage of data analysis and pre-processing in machine learning tasks is proven.