The Absolutely Simplest Neural Network Backpropagation Example
2024 ж. 22 Мам.
147 669 Рет қаралды
I'm (finally after all this time) thinking of new videos. If I get attention in the donate button area, I will proceed:
www.paypal.com/donate/?busine...
sorry there is a typo: @3.33 dC/dw should be 4.5w - 2.4, not 4.5w-1.5
NEW IMPROVED VERSION AVAILABLE: • 0:03 / 9:21The Absolut...
The absolutely simplest gradient descent example with only two layers and single weight. Comment below and click like!
Dude, this was just what I needed to finally understand the basics of Back Propagation
if you _Really_ liked his video, just click the first link he put on the description 👍
GREAT, it was a perfect inspiration for me to explain this critical subject in a class. Thank you!
Really nice work. Thank you so much for your help.
@8:06 this was super useful. That's a fantastic shorthand. That's exactly the kind of thing I was looking for, something quick I can iterate over all the weights and find the most significant one for each step.
My long search ends here, you simplified this a great deal. Thanks!
Best video ever about the back propagation in the internet 🛜
Very clearly explained and easy to understand. Thank you!
just perfect, simple and with this we can extrapolate easier when in each layer there are more than one neuron! thaaaaankksss!!
This was great. Removing non linearity and including basic numbers as context help drove this material home.
If you use relu there is nothing more that that
Fantastic. This is the most simple and lucid way to explain backprop. Hats off
I had to write a comment and thank you for your very precise yet simple explanation, just what I needed. Thank you sir.
To understand mathematics, I need to see an example. An this video from start to end is awesome with quality presentation. Thank you so much.
Thanks very helpful.
Unreal explanation
You made this concept very simple. Thank you
After a long frantic search, I stumbled upon this gold. Thank you so much!
I was just looking for this explanation to align derivatives with gradient descent. Now it is crystal clear. Thanks Miakel
The best short video explanation of the concept0 on KZhead till now...
Not kidding. This is the best explanation of backpropagation on the internet. The way you're able to simplify this "complex" concept is *chef's kiss* 👌
GOD BLESS YOU DUDE! SUBSCRIBED!!!!
Great video
finally, a proper explanation.
I watched almost every videos of back propagation even Stanford but never got such clear idea until I saw this one ☝️. Best and clean explanation. My first 👍🏼 which I rarely give.
a 👍is very good, but if you click on the first link on the description, it would be even better 👍
@@webgpu 🆗
Thanks for making this
dude please make more videos. this is amazing
Thanks you
Hi I have question for you, at 3:42, you have, 1.5*2(a-y) = 4.5*w-1.51, how did you get this result?
... in case someone missed it like me - it's in the description (it's a typo). y=0.8; a=i*w = 1.5*w, so 1.5*2(a-y) =3*(1.5*w - 0.8) = 4.5*w - 3*0.8 = 4.5*w - 2.4 is the correct formula.
Thanks for the video! Awesome explanation
I'm currently programming a neural network from scratch, and I am trying to understand how to train it, and your video somewhat helped (didn't fully help cuz I'm dumb)
Thats sick bro I just implemented it
Absolutly simple. Very useful illustration not only to understand Backpropagation but also to show gradient descent optimization. Thanks a lot.
best on internet.
Thank you bro! Its so easier to visualize it when its presented like that.
Very helpful tutorial. Thanks!
What a breakthrough, thanks to you. BTW, not to nitpick, but you are missing a close paren on f(g(x), which should be f(g(x)).
excellent video, simple & clear many thanks
Thanks
Good content sir keep making these i subscribe
My maaaaaaaannnnn TYYYY
It clicked after just 3 minutes. Thanks a lot!!
This makes more sense than anything I ever heard in the past! Thank you! 🥂
It beats the 1002165794 thing and 1001600474 jumping and calculating with 1000325836 and 1000564416. Much easier 😊
you are wrong: Say me what is deltaW?
this was kicking my a$$ until i watched this video. thanks
Great illustrated, thanks
Thanks for a very explanatory video.
Thank you for sharing this video!
very clear
Absolutely amazing 🏆
Great video, going to spend some time working out it looks for multiple neurons, but a demonstration on that would be awesome
I have to say it. You have done the best video about backpropagation because you chose to explain the easiest example, no one did that out there!! Congrats prof 😊
did you _really_ like his video? Then, i'd suggest you click the first link he put on the description 👍
thank you, this is exactly what I was looking for, very useful!
Exactly what i needed
Excellent , please continue we need this kind of simplicity in NN
Helped me so much!
Bro this is awesome, I was struggling to understand chain rule, now it is clear
Awesome dude. Much appreciate your effort.
This video is very well done. Just need to understand implementation when there is more than one node per layer
Have you looked at my other videos? I have a two-dimensional case in this video: kzhead.info/sun/dcirnZGahnVreKM/bejne.html
I am so happy that I can't even express myself right now
there's a way you can express your happiness AND express your gratitude: by clicking on the first link in the description 🙂
Bro i just worked it through and it makes so much sense once you do the partial derivatives and do it step by step and show all the working
best explanation i had ever seen, thanks.
Thank you so much!
Thank you for the easiest expression for bacpropagation dude
Nice and clean. Helped me a lot!
This is the best tutorial on back prop👏
man, thanks!
You made it easy to understand. Really appreciated it. You also earned my first KZhead comment.
Great video. Just one question, this is for 1 x 1 input and batch size of 1 right?. If we have, let´s say a batch size of 2, It is just to sum (b-y)^2 to the loss function ( C= (a-y)^2 + (b-y)^2) isnt it?, with b = w * j and j = the input of the second batch size. Then you just perform the backpropation with partial derivatives. Is it correct?
I don’t get it you write 1.5*2(a-y) = 4.5w -1.5 But why? It should be 4.5w -2,4 Because 2*0,8*-1,5= -2,4 Where am I rong?
4:03 Shouldn't 3(a - y) be 3(1.5*w - 0.8) = 4.5w - 2.4? Where have you got -1.5 from?
thanks a lot... a great start for me to learn NNs :)
thanks a lot for that explanation :)
Perfect
Thank you for your video. But I’m a bit confused about 1,5.2(a-y) = 4,5.w-1,5, Might you please explain that? Thank you so much!
I think this is how he got there : 1.5 * 2(a - y) = 1.5 * 2 (iw - 0.5) = 1.5 * 2 (1.5w - 0.5) = 1.5 * (3w - 1) = 4.5w - 1.5
@@user-gq7sv9tf1m dude thanks for that, I was really scratching my head over how he got there too
i am also confused this error
@@user-gq7sv9tf1m y is 0.8 not 0.5
Thank you
I see. As previously mentioned, there are a few typos. For anyone watching, please note there are a few places where 0.8 and 0.5 are swapped for each other. That being said, this explanation has opened my eyes to the fully intuitive explanation of what is going on... Put simply, we can view each weight as an "input knob" and we want to know how each one creates the overall Cost/Loss. In order to do this, we link (chain) each component's local influence together until we have created a function that describes weight to overall cost. Once we have found that, we can adjust that knob with the aim of lowering total loss a small amount based on what we call "learning rate". Put even more succinctly, we are converting each weight's "local frame of reference" to the "global loss" frame of reference and then adjusting each weight with that knowledge. We would only need to find these functions once for a network. Once we know how every knob influences the cost, we can tweak them based on the next training input using this knowledge. The only difference between each training set will just be the model's actual output, which is then used to adjust the weights and lower the total loss.
I think there is a mistake. 4.5w -1.5 is correct. On the first slide you said 0.5 is the expected output. So "a" is the computed output and "y" is the expected output. 0.5 * 1.5 * 2 = 1.5 is correct. You need to correct the "y" next to the output neuron to 0.5.
THIS IS SOO FKING GOOD!!!!
Great video! One thing to mention is that the cost function is not always convex, in fact it is never truly convex. However, as an example this is really well explained.
Brilliant. What would be awesome is to then further expand if u would and explain multiple rows of nodes...in order to try and visualise if possible multiple routes to a node and so on...i stress "if possible...".
ECE 449 UofA
@Mikael Laine even though you say that @3:33 has a typo. i cant see the typo. 1.5 is correct because y is the actual desired out put and it is 0.5. so 3.0 * 0.5 = 1.5
The video shows what is perhaps the simplest case of a feedforward network, with all the advantages and limitations that extreme simplicity can have. From here to full generalization several steps are involved. 1.- More general processing units. Any continuously differentiable function of inputs and weights will do; these inputs and weights can belong not only to Euclidean spaces but to any Hilbert spaces as well. Derivatives are linear transformations and the derivative of a unit is the direct sum of the partial derivatives with respect to the inputs and with respect to the weights. 2.- Layers with any number of units. Single unit layers can create a bottleneck that renders the whole network useless. Putting together several units in a layer is equivalent to taking their product (as functions, in the set theoretical sense). Layers are functions of the totality of inputs and weights of the various units. The derivative of a layer is then the product of the derivatives of the units. This is a product of linear transformations. 3.- Networks with any number of layers. A network is the composition (as functions, and in the set theoretical sense) of its layers. By the chain rule the derivative of the network is the composition of the derivatives of the layers. Here we have a composition of linear transformations. 4.- Quadratic error of a function. --- This comment is becoming a too long. But a general viewpoint clarifies many aspects of BPP. If you are interested in the full story and have some familiarity with Hilbert spaces please Google for papers dealing with backpropagation in Hilbert spaces. Daniel Crespin
Great video. I believe there is a typo at 1:10. y should be 0.5 and not 0.8. That might cause some confusion, especially at 3:34, when we use numerical values to calculate the slope (C) / slope (w)
Thanks for pointing that out; perhaps time to make a new video!
yes, that should say a=1.2
+Mikael Laine I would be si glad if you could make more videos explaining these kind of concepts and how they actually work in a code level.
Did you have any particular topic in mind? I'm planning to make a quick video about the mathematical basics of backpropagation: automatic differentiation. Also I can make a video about how to implement the absolutely simples neural network in Tensorflow/Python. Let me know if you have a specific question. I do have quite a bit experience in TF.
@@mikaellaine9490 How about adding that to description? Someone else asked that question.
Very helpful
Thanks a lot :)
This video is gold.
Thank you so much! I'm 14 years old and I'm now trying to build a neural network with python without using any kind of libraries, and this video made me understand everything much better.
No way me too
Brooo WW I ended up coding something which looked good to me but for some reason It didn't work so I just gave up on it. I wish you good luck man@@Banana-anim8ions
if we take directly the derivitive dC/dw from C=(a-y)^2 is the same thing right? do we really have to split individually da/dw and dC/da ???
man 4:08 i dont undestrand how you find the valor 4.5, in expression 4.5.w-1.5,
So what is the clever part of back prop? Why does it have a special name and it isn't just called "gradient estimation"? How does it save time? It looks like it just calculates all derivatives one by one
Спасибо братан, наконец-то выкупил что после последнего слоя происходит:)
Thank you. Here is pytorch implementation. import torch import torch.nn as nn class C(nn.Module): def __init__(self): super(C, self).__init__() r = torch.zeros(1) r[0] = 0.8 self.r = nn.Parameter(r) def forward(self, i): return self.r * i class L(nn.Module): def __init__(self): super(L, self).__init__() def forward(self, p, t): loss = (p-t)*(p-t) return loss class Optim(torch.optim.Optimizer): def __init__(self, params, lr): defaults = {"lr": lr} super(Optim, self).__init__(params, defaults) self.state = {} for group in self.param_groups: for par in group["params"]: # print("par: ", par) self.state[par] = {"mom": torch.zeros_like(par.data)} def step(self): for group in self.param_groups: for par in group["params"]: grad = par.grad.data # print("grad: ", grad) mom = self.state[par]["mom"] # print("mom: ", mom) mom = mom - group["lr"] * grad # print("mom update: ", mom) par.data = par.data + mom print("Weight: ", round(par.data.item(), 4)) # r = torch.ones(1) x = torch.zeros(1) x[0] = 1.5 y = torch.zeros(1) y[0] = 0.5 c = C() o = Optim(c.parameters(), lr=0.1) l = L() print("x:", x.item(), "y:", y.item()) for j in range(5): print("_____Iter ", str(j), " _______") o.zero_grad() p = c(x) loss = l(p, y).mean() print("prediction: ", round(p.item(), 4), "loss: ", round(loss.item(), 4)) loss.backward() o.step()
This is absolutely awesome. Except..... Where did that 4.5 come from???
You’ve probably figured it out by now but just in case: i = 1.5, y=0.8, a = i•w. This means the expression for dC/dw = 1.5 • 2(1.5w - 0.8). Simplify this and you get 4.5w - 2.4. This is where the 4.5 comes from. Extra note: in the description it says -1.5 was a typo and the correct number is -2.4.
Thanks! This is Awesome. I have I question, if we make the NN more complicated a little bit (adding an activation function for each layer), what will be the difference?
Ow you did not lie on the tittle.
Where and how did you get the learning rate?
Amazing
are you able to briefly describe how the calculation at 8:20 works for a network with mutliple neurons per layer?
i like this vd
hmm, if y = .8 then should dc/dw = 4.5w - 2.4. Because .8 * 3 = 2.4, not 1.5. What am I missing?
in the final eqn why it is 4.5w-1.5 instead it should be 4.5w-2.4 since y=0.8 so 3*0.8 =2.4
Yes you are right. I noticed too.