Overfitting and underfitting in machine learning

overfitting

and Underfitting are two essential concepts in Machine Learning and the reasons for the poor performance of the models. The main object of every machine learning model is to give a suitable output by adjusting to the given set of unknown inputs. In a nutshell, it should be able to produce accurate and reliable data after receiving training on the data sets. Thus if the model in machine learning is performing well on training data but its performance reduces on the test data it is called overfitting. On the other hand, if the model performs poorly over both the train and test set, it is known as underfitting.

These concepts in machine learning are related to bias and variance trade-offs. The article will give you a brief idea of the two concepts and also how to avoid them. Let us understand bias and variance before we explore each concept in detail!Bias: It refers to all the assumptions that a model makes so that a function can be learned easily.

• Variance: It refers to all the changes that occur in the model when the different portion of the data set which is used for training is used.
• What is Overfitting?

  Overfitting refers to a model that performs well for training the data but displays poor performance on test data. It happens when the model learns all the details and noise of the training data to such an extent that it negatively affects the performance of the test data of the model. This indicates that the noise or the random fluctuations in the training data is selected and understood as concepts by the model. However, these concepts do not apply to new data and thereby impact the performance of the models in a negative way.

  Overfitting is likely to happen with nonlinear models and nonparametrics which are more flexible while learning a target function. You may also find many nonparametric algorithms of machine learning that include techniques or parameters to constrain and limit how much detail the model is able to learn. Thus overfitting refers to high variance and low biases in the model.

• How to Detect Overfitting in Machine Learning?

• Detecting overfitting is an impossible task unless the data is tested. It can only help to address the inherent characteristics such as the inability to generalise the data sets. Therefore you can separate the data into different subsets so that training and testing become easy. Now, the data is divided into two main parts i.e Test set and a training set. The training set has the majority of data which is 80% of the available data that it is required to train the model. The remaining 20% is the test set which is used to test the accuracy of data it didn't have interaction with before. Segmentation of data will help to examine the performance of the model so that it can be stopped when overfitting occurs. This will also help to manage the training process. Using the percentage of precision observed in two data sets the performance can be measured to conclude the detection of overfitting. To sum up, if the model of machine learning performs better on the training set when compared to the test set this means that the model is overfitted.

  Overfitting can be easily identified by keeping the metrics of validation such as loss and accuracy in check. It will generally increase to a point where they become stagnant or either start declining if the model is impacted by overfitting. In such a scenario, while in an upward trend, it seems to be a good fit but when achieved it causes the trend to begin declining and become stagnant.