Deep Learning - Weidong Kuang

. .

Résumé :

Preface xv

Mathematical Notation xxi

1 Introduction to Deep Learning 1

1.1 Introduction 1

1.2 Types of Machine Learning 2

1.2.1 Supervised Learning 3

1.2.2 Unsupervised Learning 5

1.2.3 Reinforcement Learning 6

1.3 Data Representation in Machine Learning 6

1.3.1 Tensor 6

1.3.2 Datasets: Training, Validation, and Testing 7

1.3.3 Resources of Datasets 8

1.4 An Overview of Deep Learning 8

1.4.1 Perceptron 9

1.4.2 Multilayer Neural Networks and Backpropagation 10

1.4.3 Convolutional Neural Networks (CNNs) 11

1.4.4 Recurrent Neural Networks (RNNs) 12

1.4.5 Reinforcement Learning 14

1.5 Resources for Deep Learning 14

1.5.1 Frameworks 15

1.5.2 Resources for Studying Deep Learning 15

Exercises 17

References 18

2 Linear Regression 19

2.1 Linear Regression with Single Feature 19

2.1.1 Linear Regression Model 19

2.1.2 Loss Function 20

2.1.3 Analytic Solution 20

2.1.4 Gradient Descent Algorithm 22

2.2 Linear Regression with Multiple Features 25

2.3 Linear Models for Regression 28

2.3.1 Polynomial Curve Fitting 28

2.3.2 Linear Models with Basis Functions 29

2.4 Linear Regression - a Probabilistic Perspective View 31

2.4.1 Equivalence of Least Square Error and Maximum Likelihood Estimation 32

2.4.2 Loss Analysis: Bias and Variance 33

2.5 An Example: House Price Prediction 35

2.5.1 Practical Issues: Feature Scaling and Learning Rate 35

2.5.2 Linear Regression for House Price Prediction in Python 37

2.6 Summary and Further Reading 41

Exercises 42

References 44

3 Classification and Logistic Regression 45

3.1 Logistic Regression 45

3.1.1 Classification 45

3.1.2 Logistic Regression Model 46

3.1.3 Learn the Model: Find Optimal ? Based on a Dataset 49

3.2 Performance Metrics for Classification 52

3.2.1 Metrics for Two-Class Classification 52

3.2.2 Metrics for Multi-Class Classification 54

3.2.3 Receiver Operating Characteristic (ROC) Curve 55

3.3 Implementation of Logistic Regression in Python 56

3.4 Summary 61

Exercises 62

4 Basics of Neural Networks 67

4.1 A Simplest Neural Network: A Logistic Regression Unit 67

4.2 From Regression to Neural Networks 69

4.3 Neural Network Representation: Feedforward Propagation 72

4.4 Activation Functions 73

4.5 Network Training: Backward Propagation 76

4.6 Multi-class Classification: Softmax and Cross-Entropy Loss 79

4.6.1 Softmax Activation in Neural Network 79

4.6.2 Cross-Entropy Loss and Backpropagation 80

4.7 Practice in Python 82

4.7.1 A Simple Two-layer Neural Network for Binary Classification 82

4.7.2 Multi-class Classification on MNIST Dataset 91

4.8 Summary and Further Reading 100

Exercises 101

Reference 105

5 Practical Considerations in Neural Networks 107

5.1 Multiple-Layer Neural Networks 108

5.1.1 Architecture 108

5.1.2 Forward Propagation and Backward Propagation 109

5.2 Generalization and Model Selection 111

5.2.1 Generalization, Underfitting, and Overfitting 111

5.2.2 Training Set, Validation Set, and Test Set 112

5.2.3 Model Selection and K-Fold Cross-Validation 113

5.3 Regularization 115

5.3.1 Regularization for Linear Regression 115

5.3.2 Regularization for Logistic Regression 117

5.3.3 Regularization for Neural Network 117

5.3.4 Dropout for Regularization 118

5.4 Weight Initialization 119