With fewer than 500 North Atlantic right whales left in the world's oceans, knowing the health and status of each whale is integral to the efforts of researchers working to protect the species from extinction. In the NOAA Right Whale Recognition challenge, 470 players on 364 teams competed to build a model that could identify any individual, living North Atlantic right whale from its aerial photographs.
Felix Lau entered the competition with the goal of practicing new techniques in deep learning, and ended up taking second place. This blog shares his background, some of the limitations he ran into, and a high level overview of his approach. For a more technical description of his winning solution, don't miss the post in his personal blog.
The Basics
What was your background prior to entering this challenge?
Felix (aka felixlaumon) on Kaggle
I am currently working at a fashion consulting company as a computer vision research engineer. I have trained a number of deep neural networks at work. I participated some image Kaggle competitions before (Diabetic Retinopathy Detection and the 1st Data Science Bowl). Although I didn’t do well in those competitions, the experiences helped me to get started in this competition quickly.
What made you decide to enter this competition?
I was inspired by the new deep learning ideas and techniques proposed in the last few months, and I wanted to apply them to a real problem. At the end, even though some of the ideas didn’t turn out to work well for this problem, it was still a good experience because I now understand the limitations of some of the approaches.
Let's Get Technical
What preprocessing and supervised learning methods did you use?
All my approaches were based on deep convolutional neural network as it is used by most, if not all, winners of the previous image Kaggle competitions. However it turned out for this dataset, the neural network didn't manage to extract the right features from the raw images. In particular, the classifier had troubles focusing on the “whale face” on its own, as I suspect the “whale name” alone was not a strong enough training signal.
Saliency map of the neural network trained on raw image
The approach that I found worked well was to first predict the blowhead and bonnet of the whale and crop the raw images accordingly. The goal is to guide the network to focus on the callosity pattern on the back of the whale. Then I trained a classifier using the cropped images to predict the final “whale name”.
You should check out my blog post for the details. Below is a summary of my best approach.
Blowhead and Bonnet Localizer
The blowhead and bonnet localizer was based on the VGG-net. The goal is to output the x-y coordinates of the 2 key points (blowhead and bonnet) to create aligned whale face images. I trained the network with mean squared error as its error function. Real-time augmentation (e.g. rotation, translation, contrast augmentation) was applied during training. I found that test-time augmentation improved the accuracy of the coordinates quite significantly.
Classifier
The classifier took the aligned face images to output the probability of the final “whale name”. Because the images fed to this network are always aligned, neither train-time nor test-time augmentation was applied. I tried out different network architectures and found that residual network (ResNet) works quite well.
The final classifier was an ensemble of the following networks:
3 x 19-layer VGGNet
1 x 31-layer ResNet
1 x 37-layer ResNet
1 x 67-layer ResNet
Which tools did you use?
Software – I used Python as a programming language. For frameworks, I used Theano and Lasagne to build up the neural networks, nolearn for the training loop, my nolearn_utils for real-time augmentation, scikit-learn for ensembling, and scikit-image for image processing.
Hardware – I used GTX 980ti and GTX 670 for training the model locally, and AWS EC2 g2.xlarge for optimizing their hyperparameters (with Docker and Docker Machine).
How did you spend your time on this competition?
Most of my work was done in the last 3 weeks. I spent about 20% of my time building up a baseline submission, 30% building up the infrastructure and code refactoring, 50% experimenting with alternative approaches.
Words of Wisdom
What have you taken away from this competition?
It is still a surprise to me that the model trained from raw images (an end-to-end approach) did not perform well. Providing additional training signals (e.g. bonnet and blowhead coordinates, type of the callosity pattern as in the 1st place approach) proved to be a very important trick for this competition.
When the dataset size is limited, it seems augmenting the training labels is just as important as augmenting the training data (i.e. image perturbation)
Do you have any advice for those just getting started in data science?
Building models is only a small part of what a data scientist do. Understanding the problem, collecting the dataset, designing an evaluation metric, and communicating the findings effectively are just as important.
Bio
You can find Felix on Twitter and on his blog.
For more on the NOAA Right Whale Recognition competition, click the tag below!