For the third reading group on the Stanford University Convolutional Neural Networks class, we went through the following slides:
- Data preprocessing, weight initialization and regularization
- Gradient, sanity checks and parameter updates (first half)
The most common regularization is L2 cross-validated. A good idea is to combine this with dropout with a reasonable value of p of 0.5, but this can be tuned. When implementing activation functions from scratch, it is important to do some gradient and sanity checks to validate your implementation.