Regularizing the weights

Another way to make sure that your network doesn't pick up on irrelevant features is through regularizing the weights of our model. This simply allows us to put a constraint on the complexity of the network by limiting its layer weights to only take small values. All this does is make the distribution of layer weights more regular. How do we do this? By simply adding a cost to the loss function of our network. This cost actually represents a penalization for neurons that have larger weights. Conventionally, we implement this cost in three ways, namely L1, L2, and elastic net regularization:

• L1 regularization: We add a cost that is proportional to the absolute value of our weighted coefficients.
• L2 regularization: We add a cost that is proportional to the square of the value of the weighted coefficients. This is also known as weight decay, as the weights exponentially decay to zero if no other update is scheduled.
• Elastic net regularization: This regularization method allows us to capture the complexity of our model by using a combination of both L1 and L2 regularization.