Category Archives: Reading Group

CNN Part 6: Transfer Learning

For the sixth reading group on the Stanford University Convolutional Neural Networks class, we went through the following slides:

Take-home message:

  • The middle layers and depth of a CNN is important.
  • Transfer learning is helpful with small to medium datasets.
  • Rule of thumb:
Similar dataset Different dataset
Little data Use Linear Classifier on top layer You’re in trouble…
Try linear classifier from different stages
Lots of data Finetune a few layers Finetune a larger number of layers

CNN Part 5: Understanding and visualizing CNNs

For the fifth reading group on the Stanford University Convolutional Neural Networks class, we went through the following slides:

Take-home message:

You can backproject the content of a CNN in order to visualize the filters. You can fool a CNN by training an image (not specific to CNNs). The vast majority of the parameters of the CNN are at the fully connected layer, no matter how many convolution layers you have before. The CNN code can be highly discriminant.

CNN Part 4: Parameter Update & Optimization

For the fourth reading group on the Stanford University Convolutional Neural Networks class, we went through the following slides:

Take-home message:

There are different ways to decay your learning rate, which is recommended to avoid missing your global minima. Momentum is robust to saddle points but moves very quickly. There is no clear way to find the optimal hyperparameters, but they are very important to optimize.

CNN Part 3: Preprocessing, Initialization and Regularization

For the third reading group on the Stanford University Convolutional Neural Networks class, we went through the following slides:

Take-home message:

The most common regularization is L2 cross-validated. A good idea is to combine this with dropout with a reasonable value of p of 0.5, but this can be tuned. When implementing activation functions from scratch, it is important to do some gradient and sanity checks to validate your implementation.

CNN Part 2: Optimization and Backpropagation

For the second reading group on the Stanford University Convolutional Neural Networks class, we went through the following slides:

Take-home message:

In practice, use rectified linear or maxout activation functions, which are both piecewise linear. The bigger the network, the better, but regularization might be required.

CNN Part 1: Linear Classification

For this first reading group on the Stanford University Convolutional Neural Networks class, we went through the following slides:

Take-home message:

There are two loss functions commonly used: the softmax and the SVM. In order to optimize the weights and train effectively, we will want to minimize the chosen loss function during training.

planes

Image Representation

Introduction by Zhiming:

Because we will take the CNN course from Stanford University, this reading group will only focus on the shallow image representation, not the deep learning part. The goal of this reading group is to understand the basic idea about Bag of Words, how to import the spatial information to BoW, some advanced encoding methods (VLAD, Improved Fisher Vector), and some SVM kernels.
Continue reading Image Representation

roc_pr_space

Understanding the ROC and PR spaces

Introduction by Sébastien:

For this first reading group, I propose to go to the basics. The objective is to be able to read ROC and PR plots in order to interpret them correctly. There a thousands of papers about the evaluation of classifiers. It would be impossible to have all the knowledge in a single reading group, so I decided to focus on classifiers with two classes, and putting the emphasis mostly on the signification of the points in these spaces, and also on what can arrive to the classifiers they represent if they are combined or tuned. Curves are also a little bit addressed. The distribution of the readings I propose is the following. I only selected papers published after the year 2000. Moreover, most of the selected papers have been cited nearly one thousand times.
Continue reading Understanding the ROC and PR spaces