Validating your approach using k-fold validation

We noted earlier how our dataset is significantly smaller than the ones we previously dealt with. This leads to several complications while training and testing. Primarily, splitting the data into training and test samples, as we did, left us with only 100 validation samples. This is hardly enough for us to assuredly deploy our model, even if we wanted to. Furthermore, our test scores may change a lot depending on which segment of the data ended up in the test set. Hence, to reduce our reliance on any particular segment of our data for testing our model, we adopted a common machine learning approach known as k-fold cross validation. Essentially, we split our data into n number of smaller partitions and used the same number of neural networks to train on each of those smaller partitions of our data. Hence, a k-fold cross validation with five folds will split up our entire training data of 506 samples into five splits of 101 samples (and one with 102). Then, we use five different neural networks, each of which trains on four splits out of the five data splits and tests itself on the remaining split of data. Then, we simply average the predictions from our five models to generate a single estimation: