Deep Learning Roadmap

My own deep learning mastery roadmap, inspired by Deep Learning Papers Reading Roadmap.

There are some customized differences:

• not only academic papers but also blog posts, online courses, and other references are included
• customized for my own plans - may not include RL, NLP, etc.
• updated for 2019 SOTA

Introductory Courses

• Deeplearning.ai courses by Andrew Ng
• Fast.ai
• Dive Into Deep Learning
• Stanford's CS231n Class Notes

Basic CNN Architectures

• AlexNet (2012) [paper]
  • Alex Krizhevsky et al. "ImageNet Classification with Deep Convolutional Neural Networks"
• ZFNet (2013) [paper]
  • Zeiler et al. "Visualizing and Understanding Convolutional Networks"
• VGG (2014)
  • Simonyan et al. "Very Deep Convolutional Networks for Large-Scale Image Recognition" (2014) [Google DeepMind & Oxford's Visual Geometry Group (VGG)] [paper]
  • VGG-16: Zhang et al. "Accelerating Very Deep Convolutional Networks for Classification and Detection" [paper]
• GoogLeNet, a.k.a Inception v.1 (2014) [paper]
  • Szegedy et al. "Going Deeper with Convolutions" [Google]
  • Original LeNet page from Yann LeCun's homepage.
  • Inception v.2 and v.3 (2015) Szegedy et al. "Rethinking the Inception Architecture for Computer Vision" [paper]
  • Inception v.4 and InceptionResNet (2016) Szegedy et al. "Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning" [paper]
  • "A Simple Guide to the Versions of the Inception Network" [blogpost]
• ResNet (2015) [paper]
  • He et al. "Deep Residual Learning for Image Recognition"
• Xception (2016) [paper]
  • Chollet, Francois - "Xception: Deep Learning with Depthwise Separable Convolutions"
• MobileNet (2016) [paper]
  • Howard et al. "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications"
  • A nice paper about reducing CNN parameter sizes while maintaining performance.
• DenseNet (2016) [paper]
  • Huang et al. "Densely Connected Convolutional Networks"

Generative adversarial networks

• GAN (2014.6) [paper]
  • Goodfellow et al. "Generative Adversarial Networks"
• DCGAN (2015.11) [paper]
  • Radford et al. "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks"
• Info GAN (2016.6) [paper]
  • Chen et al. "InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets"
• Improved Techinques for Training GANs (2016.6) [paper]
  • Salimans et al. "Improved Techinques for Training GANs"
  • This paper suggests multiple GAN training techinques such as feautre matching, minibatch discrimination, one sided label smoothing, virtual batch normalization.
  • It also suggests a renown generator performance metric, called the inception score.
• f-GAN (2016.6) [paper]
  • Nowozin et al. "f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization"
• Unrolled GAN (2016.7) [paper]
  • Metz et al. "Unrolled Generative Adversarial Networks"
• ACGAN (2016.10) [paper]
  • Odena et al. "Conditional Image Synthesis With Auxiliary Classifier GANs"
• LSGAN (2016.11) [paper]
  • Mao et al. "Least Squares Generative Adversarial Networks"
• Pix2Pix (2016.11) [paper]
  • Isola et al. "Image-to-Image Translation with Conditional Adversarial Networks"
• EBGAN (2016.11) [paper]
  • Zhao et al. "Energy-based Generative Adversarial Network"
• WGAN (2017.4) [paper]
  • Arjovsky et al., "Wasserstein GAN"
• WGAN_GP (2017.5) [paper]
  • Gulrajani et al., "Improved Training of Wasserstein GANs"
  • Improves the training stability by applying "gradient penalty (GP)" to the loss function
• BEGAN (2017.5) [paper]
  • Berthelot et al. "BEGAN: Boundary Equilibrium Generative Adversarial Networks"
  • Introduces a diversity ratio, or an equilibrium constant that controls the variety - quality tradeoff, and also proposes a convergence measure using it.
• CycleGAN (2017.5) [paper]
  • DiscoGAN (2017.5) [paper]
  • DiscoGAN and CycleGAN proposes the EXACT SAME learning techniques for style transfer task using GAN, developed independently at the same time.
• Frechet Inception Distance (FID) (2017.6) [paper]
  • Heusel et al. "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium"
  • The paper's main contribution is a technique called Two Time-Scale Update Rule (TTSU), but it is mostly known for the distance metric called Frechet Inception Distance that measures the distance between two distributions of activation values.
• ProGAN (2017.10) [paper]
  • Karras et al. "Progressive Growing of GANs for Improved Quality, Stability, and Variation"
• PacGAN (2017.12) [paper]
  • Higgins et al. "PacGAN: The power of two samples in generative adversarial networks"
• BigGAN (2018) [paper]
• GauGAN (2019.3) [paper]
  • Park et al. "Semantic Image Synthesis with Spatially-Adaptive Normalization"

Advanced GANs

• DRAGAN (2017.5) [paper]
  • Kodali et al. "On Convergence and Stability of GANs"
• Are GANs Created Equal? (2017.11) [paper]
  • Lucic et al. "Are GANs Created Equal? A Large-Scale Study"
• SGAN (2017.12) [paper]
  • Chavdarova et al. "SGAN: An Alternative Training of Generative Adversarial Networks"
• MaskGAN (2018.1) [paper]
  • Fedus et al. "MaskGAN: Better Text Generation via Filling in the _____"
• Spectral Normalization (2018.2) [paper]
  • Miyato et al. "Spectral Normalization for Generative Adversarial Networks"
• SAGAN (2018.5) [paper] [tensorflow]
  • Zhang et al. "Self-Attention Generative Adversarial Networks"
• Unusual Effectiveness of Averaging in GAN Training (2018) [paper]
  • "Benefitting from training on past snapshots."
  • Uses exponential moving averaging (EMA)
• Disconnected Manifold Learning (2018.6) [paper]
  • Khayatkhoei, et al. "Disconnected Manifold Learning for Generative Adversarial Networks"
• A Note on the Inception Score (2018.6) [paper]
  • Barratt et al., "A Note on the Inception Score"
• Which Training Methods for GAN do actually converge? (2018.7) [paper]
  • Mescheder et al., "Which Training Methods for GANs do actually Converge?"
• GAN Dissection (2018.11) [paper]
  • Bau et al. "GAN Dissection: Visualizing and Understanding Generative Adversarial Networks"
• Improving Generalization and Stability for GANs (2019.2) [paper]
  • Thanh-Tung et al., "Improving Generalization and Stability of Generative Adversarial Networks"
• Augustus Odena - "Open Questions about GANs" (2019.4) [distill.pub]
  • Very nice article about current state of GAN research and discusses problems yet to be solved.

Autoencoders

• Original autoencoder (1986) [paper]
  • Rumelhart, Hinton, and Williams, "Learning Internal Representations by Error Propagation"
• AutoEncoder [science]
  • Hinton et al., "Reducing the Dimensionality of Data with Neural Networks"
• Denoising Autoencoders (2008) [paper]
  • Vincent et al. "Extracting and Composing Robust Features with Denoising Autoencoders"
• Wasserstein Autoencoder (2017) [paper]
  • Tolstikhin et al. "Wasserstein Auto Encoders"

Autoregressive models

• PixelCNN (2016) [paper]
  • van den Oord et al. "Conditional image generation with PixelCNN decoders."
• WaveNet (2016) [paper]
  • van den Oord et al. "WaveNet: A Generative Model for Raw Audio"
• tacotron?

Layer Normalizations

• Batch Normalization (2015.2) [paper]
  • Ioeffe et al. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift"
• Group Norm
• Instance Normalization (2016.7) [paper]
  • Ulyanov et al. "Instance Normalization: The Missing Ingredient for Fast Stylization"
• Santurkar et al. "How does Batch Normalization help Optimization?" (2018.5) [paper]
• Switchable Normalization (2019) [paper]
  • Luo et al. "Differentiable Learning-to-Normalize via Switchable Normalization"
• Weight Standardization (2019.3) [paper]
  • Qiao et al. "Weight Standardization"

Initializations

• Xavier Initialization (2010) [paper]
  • Glorot et al., "Understanding the difficulty of training deep feedforward neural networks"
• Kaiming (He) Initialization (2015.2) [paper]
  • He et al., "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification"
• All you need is a good init (2015.11) [paper]
  • Mishkin et al., "All you need is a good init"
• All you need is beyond a good init (2017.4) [paper]
  • Xie et al. "All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation"

Dropouts

• Dropout (2014) [paper]
  • Srivastava et al. "Dropout: A Simple Way to Prevent Neural Networks from Overfitting"
• Inverted Dropouts [notes on CS231n]
  • Multiplying the inverted keep_prob value on training so that values during inference (or testing) is consistent.
• Li et al., "Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift" (2018.1) [paper]

Meta-Learning / Representation Learning (Zero-Shot learning, Few-Shot learning)

• Zero-Data Learning (2008) [paper]
  • Larochelle et al., "Zero-data Learning of New Tasks"
• Palatucci et al., "Zero-shot Learning with Semantic Output Codes" (NIPS 2009) [paper]
• Socher et al., "Zero-Shot Learning Through Cross-Modal Transfer" (2013.1) [paper]
• Lampert et al., "Attribute-Based Classification for Zero-Shot Visual Object Categorization" (2013.7) [paper]
• Dinu et al., "Improving zero-shot learning by mitigating the hubness problem" (2014.12) [paper]
• Romera-Paredes et al. - "An embarrassingly simple approach to zero-shot learning" (2015) [paper]
• Prototypical Networks (2017.3) [paper]
  • Snell et al., "Prototypical Networks for Few-shot Learning"
• Zero-shot learning - the Good, the Bad and the Ugly" (2017.3) [paper]
  • Xian et al., "Zero-Shot Learning - The Good, the Bad and the Ugly"
• In defence of the Triplet Loss (2017.3) [paper]
  • Hermans et al., "In Defense of the Triplet Loss for Person Re-Identification"
• MAML (2017.3) [paper]
  • Finn et al, "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks"
• Triplet Loss and Online Triplet Mining in Tensorflow (2018.3) [Oliver Moindrot Blog]
• Few-Shot learning Survey (2019.4) [paper]
  • Wang et al. "Few-shot Learning: A Survey"

Transfer learning

• Survey 2018 (2018) [paper]
  • Tan et al. "A Survey on Deep Transfer Learning"

Geometric learning

• Geometric Deep Learning (2016) [paper]
  • Bronstein et al. "Geometric deep learning: going beyond Euclidean data"

Variational Autoencoders (VAE)

• VQ-VAE (2017.11) [paper]
  • van den Oord et al., "Neural Discrete Representation Learning"
• Semi-Amortized Variational Autoencoders (2018.2) [paper]
  • Kim et al. "Semi-Amortized Variational Autoencoders"

Object detection

• RCNN: https://arxiv.org/abs/1311.2524
• Fast-RCNN: https://arxiv.org/abs/1504.08083
• Faster-RCNN: https://arxiv.org/abs/1506.01497
• SSD: https://arxiv.org/abs/1512.02325
• YOLO: https://arxiv.org/abs/1506.02640
• YOLO9000: https://arxiv.org/abs/1612.08242

Semantic Segmentation

• FCN: https://arxiv.org/abs/1411.4038
• SegNet: https://arxiv.org/abs/1511.00561
• UNet: https://arxiv.org/abs/1505.04597
• PSPNet: https://arxiv.org/abs/1612.01105
• DeepLab: https://arxiv.org/abs/1606.00915
• ICNet: https://arxiv.org/abs/1704.08545
• ENet: https://arxiv.org/abs/1606.02147
• Nice survey

Sequential Model

• Seq2Seq (2014) [paper]
  • Sutskever et al. "Sequence to sequence learning with neural networks."

Neural Turing Machine

• Neural Turing Machines (2014) [paper]
  • Graves et al., "Neural turing machines."
• Pointer Networks (2015) [paper]]
  • Vinyals et al., "Pointer networks."

Attention / Question-Answering

• NMT (Neural Machine Translation) (2014) [paper]
  • Bahdanau et al, "Neural Machine Translation by Jointly Learning to Align and Translate"
• Stanford Attentive Reader (2016.6) [paper]
  • Chen et al. "A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task"
• BiDAF (2016.11) [paper]
  • Seo et al. "Bidirectional Attention Flow for Machine Comprehension"
• DrQA or Stanford Attentive Reader++ (2017.3) [paper]
  • Chen et al. "Reading Wikipedia to Answer Open-Domain Questions"
• Transformer (2017.8) [paper] [google ai blog]
  • Vaswani et al. "Attention is all you need"
• [read] Lilian Weng - "Attention? Attention!" (2018) [blog_post]
  • A nice explanation of attention mechanism and its concepts.
• BERT (2018.10) [paper]
  • Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"
• GPT-2 (2019) [paper (pdf)]
  • Radford et al. "Language Models are Unsupervised Multitask Learners"

Advanced RNNs

• Unitary evolution RNNs : https://arxiv.org/abs/1511.06464
• Recurrent Batch Norm : https://arxiv.org/abs/1603.09025
• Zoneout : https://arxiv.org/abs/1606.01305
• IndRNN : https://arxiv.org/abs/1803.04831
• DilatedRNNs : https://arxiv.org/abs/1710.02224

Model Compression

• MobileNet (2016) (see above: Basic CNN Architectures)
• ShuffleNet (2017)
  • Zhang et al. "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices"

Neural Processes

• Neural Processes (2018) [paper]
  • Garnelo et al. "Neural Processes"
• Attentive Neural Processes (2019) [paper]
  • Kim et al. "Attentive Neural Processes"
• A Visual Exploration of Gaussian Processes (2019) [Distill.pub]
  • Not a neural process, but gives very nice intuition about Gaussian Processes. Good Read.

Self-supervised learning

• Denoising AE https://www.iro.umontreal.ca/~vincentp/Publications/denoising_autoencoders_tr1316.pdf
• Exemplar Nets https://arxiv.org/abs/1406.6909
• Co-occ https://arxiv.org/abs/1511.06811
• Egomotion https://arxiv.org/abs/1505.01596
• Jigsaw https://arxiv.org/abs/1603.09246
• Context Encoders https://arxiv.org/abs/1604.07379
• Split-brain autoencoders https://arxiv.org/abs/1611.09842
• multi-task self-supervised learning https://arxiv.org/abs/1708.07860
• Audio-visual scene analysis https://arxiv.org/abs/1804.03641
• a survey https://slideplayer.com/slide/13195863/
• Supervising unsupervised learning https://arxiv.org/abs/1709.05262
• Unsupervised Representation Learning by Predicting Image Rotations https://arxiv.org/abs/1803.07728
• Mahjourian et al., "Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints" (2018.2) [paper]
• Gordon et al., "Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras" (2019.4) [paper]

Data Augmentation

• Shake Shake Regularization (2017.5) [paper]
  • Gastaldi, Xavier - "Shake-Shake Regularization"

Interpretation and Theory on Generalization, Overfitting, and Learning Capacity

• MDL (Minimum Description Length)
  • Peter Grunwald - "A tutorial introduction to the minimum description length principle" (2004) [paper]
• Grunwald et al., - "Shannon Information and Kolmogorov Complexity" (2010) [paper]
• Dauphin et al. "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization" (2014.6) [paper]
• Choromanska et al. "The Loss Surfaces of Multilayer Networks" (2014.11) [paper]
  • argues that non-convexity in NNs are not a huge problem
• Knowledge Distillation (2015.3) [paper]
  • Hinton et al., "Distilling the Knowledge in a Neural Network"
• 3-Part Learning Theory by Mostafa Samir
  • part 1: Introduction
  • part 2: Generalization Bounds
  • part 3: Regularization and Variance-Bias Tradeoff
• Deconvolution and Checkerboard Artifacts - Odena (2016) [distill.pub article]
• Keskar et al. "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima" (2016.9) [paper]
• Rethinking Generalization (2016.11) [paper]
  • Zhang et al. "Understanding deep learning requires rethinking generalization"
• Information Bottleneck (2017) [paper] [original paper on information bottleneck (2000)] [youtube-talk] [article in quantamagazine]
  • Shwartz-Ziv and Tishby, "Opening the Black Box of Deep Neural Networks via Information"
• Neyshabur et al, "Exploring Generalization in Deep Learning" (2017.7) [paper]
• Sun et al., "Revisiting Unreasonable Effectiveness of Data in Deep Learning Era" (2017.7) [paper]
• Super-Convergence (2017.8) [paper]
  • Smith et al. - "Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates"
• Don't Decay the Learning Rate, Increase the Batch Size (2017.11) [paper]
  • Smith et al. "Don't Decay the Learning Rate, Increase the Batch Size"
• Hestness et al. "Deep Learning Scaling is Predictable, Empirically" (2017.12) [paper]
• Visualizing loss landscape of neural nets (2018) [paper]
• Olson et al., "Modern Neural Networks Generalize on Small Data Sets" (NeurIPS 2018) [paper]
• Lottery Ticket Hypothesis (2018.3) [paper]
  • Frankle et al., "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks"
  • Empirically showed that zeroing small weights after training, rewinding except zeroed wegiths, and then re-triaining with 'pruned' weights showed even better results.
• Intrinsic Dimension (2018.4) [paper]
  • Li et al., "Measuring the Intrinsic Dimension of Objective Landscapes"
• Geirhos et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness" (2018.11) [paper]
• Belkin et al. "Reconciling modern machine learning and the bias-variance trade-off" (2018.12) [paper]
• Graetz - "How to visualize convolution features in 40 lines of code" (2019) [medium]
• Geiger et al. "Scaling description of generalization with number of parameters in deep learning" (2019.1) [paper]
• Are all layers created equal? (2019.2) [paper]
  • Zhang et al. "Are all layers created equal?"
• Lilian Weng - "Are Deep Neural Networks Dramatically Overfitted?" (2019.4) [lil'log]
  • Excellent article about generalization and overfitting of deep neural networks

Adversarial Attacks and Defense against attacks (RobustML)

• RobustML site
• Adversarial Examples Szegedy et al. - Intreguing Properties of Neural Networks (2013.12) [paper]
  • induces missclassification by applying small perturbations
  • this paper was the first to coin the term "Adversarial Example"
• Fast Gradient Sign Attack (FGSM) (2014.12)
  • Goodfellow et al., "Explaining and Harnessing Adversarial Examples" (ICLR 2015) [paper]
  • This paper presented the famous "panda example" (as also seen in pytorch tutorial)
• Kurakin et al., "Adversarial Machine Learning at Scale" (2016.11) [paper]
• Mandry et al., "Towards Deep Learning Models Resistant to Adversarial Attacks" (2017.6) [paper]
• Carlini et al., "Audio Adversarial Examples: Targeted Attacks on Speech-to-Text" (2018.1) [paper]

Neural architecture search (NAS) and AutoML

• GREAT AutoML Website [site]
  • They maintain a blog, a list of NAS literatures, analysis page, and a web book.
• AdaNet (2016.7) [paper] [GoogleAI blog]
  • Cortes et al. "AdaNet: Adaptive Structural Learning of Artificial Neural Networks"
• NAS (2016.12) [paper]
  • Zoph et al. "Neural Architecture Search with Reinforcement Learning"
• PNAS (2017.12) [paper]
  • Liu et al. "Progressive Neural Architecture Search"
• ENAS (2018.2) [paper]
  • Pham et al. "Efficient Neural Architecture Search via Parameter Sharing"
• DARTS (2018.6) [paper]
  • Liu et al. "DARTS: Differentiable Architecture Search"
  • Uses a continuous relaxation over the discrete neural architecture space.
• RandWire (2019) [paper]
  • Xie et al. "Exploring Randomly Wired Neural Networks for Image Recognition" [Facebook AI Research]
• A Survey on Neural Architecture Search (2019) [paper]
  • Witsuba et al., "A Survey on Neural Architecture Search"

Practical Techniques

• Andrej Karpathy - "A recipe for training neural networks" (2019) [Andrej Karpathy Blog Post]

DL roadmap reference

• https://github.com/songrotek/Deep-Learning-Papers-Reading-Roadmap
• https://github.com/terryum/awesome-deep-learning-papers
• which DL algorithms should I implement to learn? https://www.reddit.com/r/MachineLearning/comments/8vmuet/d_what_deep_learning_papers_should_i_implement_to/

Theory

• The MML(Mathematics for Machine Learning) book
• Andrej Karpathy - Yes You Should Understand Backprop
• Theoretical principles for Deep Learning
• Stanford STATS 385 - Theories of Deep Learning
  • Lecture Videos
• CSC 231 notes : http://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/

Resources

• A Selective Overview of Deep Learning (2019) [paper]
  • Fan et al. "A Selective Overview of Deep Learning"
  • A nice overview paper on deep learning up to early 2019 (about 30 pages)