How to gauge accuracy of processing big data: Teaching machine learning techniques to gauge their own accuracy

Abstract

© Springer International Publishing AG 2018. When the amount of data is reasonably small, we can usually fit this data to a simple model and use the traditional statistical methods both to estimate the parameters of this model and to gauge this model’s accuracy. For big data, it is often no longer possible to fit them by a simple model. Thus, we need to use generic machine learning techniques to find the corresponding model. The current machine learning techniques estimate the values of the corresponding parameters, but they usually do not gauge the accuracy of the corresponding general non-linear model. In this paper, we show how to modify the existing machine learning methodology so that it will not only estimate the parameters, but also estimate the accuracy of the resulting model.