The Complete Mathematics of Neural Networks and Deep Learning

A complete guide to the mathematics behind neural networks and backpropagation. In this lecture, I aim to explain the mathematical phenomena, a combination of linear algebra and optimization, that underlie the most important algorithm in data science today: the feed forward neural network. Through a plethora of examples, geometrical intuitions, and not-too-tedious proofs, I will guide you from understanding how backpropagation works in single neurons to entire networks, and why we need backpropagation anyways. It's a long lecture, so I encourage you to segment out your learning time - get a notebook and take some notes, and see if you can prove the theorems yourself. As for me: I'm Adam Dhalla, a high school student from Vancouver, BC. I'm interested in how we can use algorithms from computer science to gain intuition about natural systems and environments. My website: adamdhalla.com I write here a lot: adamdhalla.medium.com Contact me: adamdhalla@protonmail.com Two good sources I recommend to supplement this lecture: Terence Parr and Jeremy Howard's The Matrix Calculus You Need for Deep Learning: https://arxiv.org/abs/1802.01528 Michael Nielsen's Online Book Neural Networks and Deep Learning, specifically the chapter on backpropagation http://neuralnetworksanddeeplearning.com/chap2.html ERRATA---- I'm pretty sure the Jacobians part plays twice - skip it when you feel like stuff is repeating, and stop when you get to the part about the "Scalar Chain Rule" (00:24:00). And, here are the timestamps for each chapter mentioned in the syllabus present at the beginning of the course. PART I - Introduction -------------------------------------------------------------- 00:00:52 1.1 Prerequisites 00:02:47 1.2 Agenda 00:04:59 1.3 Notation 00:07:00 1.4 Big Picture 00:10:34 1.5 Matrix Calculus Review 00:10:34 1.5.1 Gradients 00:14:10 1.5.2 Jacobians 00:24:00 1.5.3 New Way of Seeing the Scalar Chain Rule 00:27:12 1.5.4 Jacobian Chain Rule PART II - Forward Propagation -------------------------------------------------------------- 00:37:21 2.1 The Neuron Function 00:44:36 2.2 Weight and Bias Indexing 00:50:57 2.3 A Layer of Neurons PART III - Derivatives of Neural Networks and Gradient Descent -------------------------------------------------------------- 01:10:36 3.1 Motivation & Cost Function 01:15:17 3.2 Differentiating a Neuron's Operations 01:15:20 3.2.1 Derivative of a Binary Elementwise Function 01:31:50 3.2.2 Derivative of a Hadamard Product 01:37:20 3.2.3 Derivative of a Scalar Expansion 01:47:47 3.2.4 Derivative of a Sum 01:54:44 3.3 Derivative of a Neuron's Activation 02:10:37 3.4 Derivative of the Cost for a Simple Network (w.r.t weights) 02:33:14 3.5 Understanding the Derivative of the Cost (w.r.t weights) 02:45:38 3.6 Differentiating w.r.t the Bias 02:56:54 3.7 Gradient Descent Intuition 03:08:55 3.8 Gradient Descent Algorithm and SGD 03:25:02 3.9 Finding Derivatives of an Entire Layer (and why it doesn't work well) PART IV - Backpropagation -------------------------------------------------------------- 03:32:47 4.1 The Error of a Node 03:39:09 4.2 The Four Equations of Backpropagation 03:39:12 4.2.1 Equation 1: The Error of the last Layer 03:46:41 4.2.2 Equation 2: The Error of any layer 04:03:23 4.2.3 Equation 3: The Derivative of the Cost w.r.t any bias 04:10:55 4.2.4 Equation 4: The Derivative of the Cost w.r.t any weight 04:18:25 4.2.5 Vectorizing Equation 4 04:35:24 4.3 Tying Part III and Part IV together 04:44:18 4.4 The Backpropagation Algorithm 04:58:03 4.5 Looking Forward