Attention in transformers, visually explained | Chapter 6, Deep Learning
Demystifying attention, the key mechanism inside transformers and LLMs.
Instead of sponsored ad reads, these lessons are funded directly by viewers: 3b1b.co/support
Special thanks to these supporters: www.3blue1brown.com/lessons/a...
An equally valuable form of support is to simply share the videos.
Demystifying self-attention, multiple heads, and cross-attention.
Instead of sponsored ad reads, these lessons are funded directly by viewers: 3b1b.co/support
The first pass for the translated subtitles here is machine-generated, and therefore notably imperfect. To contribute edits or fixes, visit translate.3blue1brown.com/
And yes, at 22:00 (and elsewhere), "breaks" is a typo.
------------------
Here are a few other relevant resources
Build a GPT from scratch, by Andrej Karpathy
• Let's build GPT: from ...
If you want a conceptual understanding of language models from the ground up, @vcubingx just started a short series of videos on the topic:
• What does it mean for ...
If you're interested in the herculean task of interpreting what these large networks might actually be doing, the Transformer Circuits posts by Anthropic are great. In particular, it was only after reading one of these that I started thinking of the combination of the value and output matrices as being a combined low-rank map from the embedding space to itself, which, at least in my mind, made things much clearer than other sources.
transformer-circuits.pub/2021...
Site with exercises related to ML programming and GPTs
www.gptandchill.ai/codingprob...
History of language models by Brit Cruise, @ArtOfTheProblem
• ChatGPT: 30 Year Histo...
An early paper on how directions in embedding spaces have meaning:
arxiv.org/pdf/1301.3781.pdf
------------------
Timestamps:
0:00 - Recap on embeddings
1:39 - Motivating examples
4:29 - The attention pattern
11:08 - Masking
12:42 - Context size
13:10 - Values
15:44 - Counting parameters
18:21 - Cross-attention
19:19 - Multiple heads
22:16 - The output matrix
23:19 - Going deeper
24:54 - Ending
------------------
These animations are largely made using a custom Python library, manim. See the FAQ comments here:
3b1b.co/faq#manim
github.com/3b1b/manim
github.com/ManimCommunity/manim/
All code for specific videos is visible here:
github.com/3b1b/videos/
The music is by Vincent Rubinetti.
www.vincentrubinetti.com
vincerubinetti.bandcamp.com/a...
open.spotify.com/album/1dVyjw...
------------------
3blue1brown is a channel about animating math, in all senses of the word animate. If you're reading the bottom of a video description, I'm guessing you're more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on KZhead or otherwise following on whichever platform below you check most regularly.
Mailing list: 3blue1brown.substack.com
Twitter: / 3blue1brown
Instagram: / 3blue1brown
Reddit: / 3blue1brown
Facebook: / 3blue1brown
Patreon: / 3blue1brown
Website: www.3blue1brown.com
A few added notes based on common comments I see. Concerning masking self-attention, several people ask about cases where it feels like later words should update the meaning of earlier words, like languages with adjectives following nouns. The model can always put the richest meaning into the last token (e.g. early nouns getting baked into later adjectives). For example, @victorlevoso8984 noted below how empirical evidence suggests the meaning of a sentence often gets baked into the embedding of the punctuation mark at its end. Keep in mind that the model doesn't have to conceptualize things the way we humans do, and in all likelihood doesn't at all, so I wouldn't over-index on the motivating example given in this video. Also, one thing I should have called out more explicitly is how I personally like to think of vectors like embeddings, keys, queries, etc. as columns, and as a convention display them this way, but other sources, including the Attention is All You Need paper, may present them organized in a row-by-row fashion. This is relevant to parsing the equation shown at 10:29, where the expression from the paper that looks like Q K^T would, by the conventions of this video, instead look like K^T Q.
I hope you make a video explain about State Space Model in Mamba, it quite confuse me with the derivative of x
Ugh youtube aparently doesnt like me posting links to papers and deleted my reponse aparently so I'll try again. Source for what I'm saying is "Linear Representations of Sentiment in Large Language Models" but I haven't read it and have only heard a talk by the papers autors so take my vague explanation of it with a grain on salt. On a quick skim they mention the model moving summary information to "commas, periods and particular nouns."
@MarcusHilarius I had a longer response that got deleted by youtube (probably cause I linked to my paper as an example) but I think this is not exactly correct in that the purpose of the qk matrices is for each attention head to approximately only "read" the information in some of the tokens. So at each position the model could have the meaning of all the previous tokens on it but it doesn't have to, and theres all kinds of interesting dynamics about wich positions attend to each position in each layer and attention head for a given input.
Row-by-row is actually in some sense the standard notation in Deep learning, since most (all?) popular frameworks use it internally. I'm not actually sure why, I guess it's just an old GPU programming convention. It's not uncommon for researchers to use column notation in their papers, but personally I think it might be confusing for people new to the field, as the code will almost surely treat the vectors as rows (after all, usually the first data dimension is the batch size). It's not a very significant thing of course.
The most common association I've heard as to why there needs to be masking, is that otherwise the model could just cheat. Every word would have the same query, "hey, what's the word after my position x?" and then the keys would all be "I'm at x+1" or similar. The value at the x+1 embedding would just be the word itself, and it would know what it needs to write. Your model would always perfectly predict the next word. This is still interesting though. What if instead of asking the model to predict the next one word, we asked it to predict the next 10? You'd implement it by sliding the triangle in the masking matrix 10 steps (masking more). Your loss would rise dramatically, since even if the model gets the "gist" of what it should write (like if you want it to talk about "Harry Potter" somewhere) it would still be off if due to sentence structure the words "Harry Potter" weren't in the exact same place as in the original text. You'd need to change your loss calculation to somehow focus more on the "semantic correctness" of those 10 words (a challenge I'm sure), but the benefit would be that the model could generate whole phrases or sentences. I'd probably do this in some staged manner, like starting with one word loss and increasing the number of words as training goes on. I'm not sure what the benefit would be though, but might be fun to try.
I'm a university lecturer with a PhD in AI, and I cannot compete with the quality of this work. Videos like this put the entire higher education system to shame. Fantastic! ❤️
so true
If you haven't already, check out the FAQ on his site about his animations. Its called Manim and is available for anyone. It seems that Grant is on a mission to create a Math and Code Animation Army. Sign me up! I teach high school CS without a full CS degree and I was able to use it in my classroom. You can install or just test it on a Jupyter notebook. Still requires an art to scaffold the instruction, but a good visualization can expedite the process.
Another role of universities I think is to hold the knowledge system and methodology with the help of social systems and social interaction inside and with another society services also via social interaction. So, now the balance has been shifted, making Unis more provide holding services and KZhead/Online/GPT - educating and transferring the existing knowledge to wide mass of people, just because huge multiplication factor (dozens of millions of viewers vs thousands of students). This new balance shouldn't be broken or rejected but taken into account to increase quality of education as a phenomenon.
@@iTXS - For sure. I, and the wider higher education sector, need to up our game.
@@iTXS they are failing hard at that too, specially because their "social systems" felt prey to some ideologies that are not conductive to critical thinking. We need to create a better system starting from what we already on the internet. " provide holding services " Why would we need that ? Open Source has solved that problem for software, Science just need to use our software development tools. You don't need the bloated organizational structure of universities at all to actually generate and curate knowledge to a high degree of success, that's what we software developers did with things like Linux. You guys are just stuck in the past.
Are you kidding me? ONE WEEK FOR 2 MASTERPIECES?! Thank you so much!
Waiting here for someone to correct your spelling...
@@user-vt4bz2vl6j🤣 Peace man
I understand the context so no use dwelling on his misteak.
@@user-vt4bz2vl6jon the contrary, i hope nobody ever corrects the mistake just to spite you >:3
@@skmgeek Humanity never learns from its mistakes.
I've got to say - "Attention Is All You Need" is an incredible title for a research paper.
Indeed
Famously so I think. I really really like these names instead of the long boring ones. But... Also people aren't clever and the creative names would be bad, so maybe we stick to long boring ones for the most part haha
I like the boring ones. That way I can easily decide which ones to read.
It’s going to be embedded into our future history lessons of when things began. =]
There are now more than 300 papers ending in "is/are All You Need"
As a graduating PhD student working in Natural Language Processing, I still found that video to be extremely beneficial. Awesome!
Good luck for your PhD defense!
Wow, a nice competitive field! Best luck to you!!
Good luck out there!
Attention existed before the 2017 paper "Attention Is All You Need". The main contribution was that attention was... all you needed for sequence processing (you didn't need recurrence). Self-attention specifically was novel though.
Yeah 3 years earlier by Bahdanau et. al
Before people debated how many self attention blocks you should stick into a model and where and if it was even worth it. The paper proved that more was better and that you're probably better off replacing linear layers and convolutional layers with more self attention blocks.
There have been many people working on this idea long before 2017. I, myself, was working on a model very similar to this in 2010, and I was inspired by papers and books going back much further. It seems people are not fond of appreciating multiple, independent instances of creation and discovery. It happens a lot in many domains.
@@waylonbarrett3456 Interesting, do you have a source for a much earlier similar model architecture?
@Haligonian the sources for their inspiration and basis are cited in the "attention is all you need" paper. I believe it's at the bottom in the acknowledgment section. Then, you go to those papers and look at their cited sources, repeat ad infinitum. This is the true, branching complexity of creative works. There's not often some lone genius having a singular "Eureka!" moment.
3b1b is the only content producer whose videos I start by first making coffee, then upvoting, then hitting the play button.
... and then be disappointed at the end because you can't upvote a second time.
Geez Grant, I spent thousands of dollars on a very good deep learning executive certification from Carnegie Mellon, and your series here is better than their math slides. This series is really turning out great.
I wish any of the online courses I did for work had this quality.
Thank you for the mention, Grant! For those who relate to the pain of wanting more practice problems for Machine Learning, I hear you. I’ve created coding problems (run against test cases in your browser!) & quizzes covering the core ML concepts. Check out the resource Grant mentioned (linked in the description) or just click on my channel!
For real, GPTandChill got some good problems to practice on. Helped me out a lot
love your videos GPTandchill very useful
Nice way to learn ML concepts from @gptandchill
I love your coding problems. They’ve actually helped me learn ML concepts better than most of my college courses.
Love your videos. They are very detailed, organized and very helpful.
The drought of 3blue1brown content has finally been overcome!
Don’t jinx it!
It takes time to learn these concepts from scratch and make these high quality videos.
As long as the videos keep coming well thought and well researched as they are, I don't care if they take their time! Congrats Grant and 3b1b team!
Shalll I drop the next hint and see how quickly new content comes out?
WE FEAST
How I wish this video was available when the "Attention is What You Need" paper just came out. It was really hard to visualize by simply reading the paper. I read it multiple times but could not figure out what it was trying to do. Then subsequently, Jay Alammar posted a blog post called The illustrated transformer. That was a huge help for me back then. But this video raises the illustration to an entirely different level. Great job! I'm sure many undergraduates or hobbyist studying machine learning would benefit greatly.
You not only put out some of the best content on youtube but also give constant shutouts to other content creators that you admire. You are a GOAT 3Blue1Brown.
As a Master's student in Data Science and AI, I never really understood how attention worked. Thank you for making this video!
A master student in DS and AI and never understood attention sounds wrong to me
@@jabirlang216 He wasn't paying attention \s
@@jabirlang216these programs are mostly non rigorous. I only hire MS statistics or MS CS
Just Wow, the educational value of this video is incredible. There are so many highly relevant and original ideas to explain abstract concepts and drastically simplify comprehension. I'm so thankful that you've made this content available to everyone for free. I absolutely love it!!
Also I want to add, making such knowledge available to mainstream is truly a gift for humanity, and a thing that you deserve to be proud of
I cannot stress enough what a tour de force this is. It's probably one of the best math classes ever done anywhere in the world in all time. You're the best in the game and an inspiration for many. So so much thank you, Grant, you're doing God's work here.
If I could write poetry about how much I appreciate and learn from your videos, I would but I'm not a poet. Thanks to everyone who worked on these videos.
oh how great the knowledge be, that grant can share so graciously. i would be lost in space and time, if you, my precious would not shine. it got a bit saccharine, but tried my best lol i do agree though, this channel is amazing.
@@captainrob4656 There is a secret tool named chat GPT. You can use it to help you with your poem creation problems 😉.
In the vast digital sea, a beacon shines bright, Three Blue One Brown, with wisdom alight. AI, ML, and DL, in colors so clear, Guiding minds through concepts, year after year. With clarity and depth, their tales unfold, Grateful hearts cherish the knowledge they've told.
In realms where algorithms dance and code reigns supreme, Where data whispers secrets and dreams in every scheme, There dwells a sage, a maestro of the digital domain, Whose words and visuals in tandem, ignite the brain. Through the ethers of the web, his teachings unfurl, Like constellations in the night, they guide and swirl, With each stroke of insight, he paints a vivid scene, Unraveling mysteries, illuminating what has been. Oh, if only words could craft the depth of gratitude, For the knowledge gained, the horizons pursued, Yet in this humble verse, let it be known and heard, The gratitude abounds, for the wisdom conferred. To the minds behind the screens, the unseen hands, Who weave the fabric of understanding in digital lands, We extend our heartfelt thanks, our deepest regard, For kindling the flame of knowledge, burning bright and hard. - By Chat GPT 3.5
Lessons truly grasped The videos open minds Thankfulness ensues
Joining the list of commenters who have worked with transformers, built and trained them, explained them to others, and still learned really basic things from this video about what they're doing that just hadn't occurred to me. Great work.
Give an example.
The volume of work, attention to detail and clarity we get from Grant is staggering. Bravo sir.
It is simply inexplicable how valuable these videos are to humanity, and I mean that literally. The way your videos convey the underlying ideas and intuitions of such complex technology opens the door for tomorrow's smart minds to learn even faster.
This video will become a part of history.
bot
@@MrZorroZorroZlmao
Finally someone explains it right. One simple thing but important that I would emphasize for the new people in AI is that all matrix/vectors used in this process are "discovered" by the model with the gradient descent mechanism (reward mechanism) for the especific task. In this cases, predict next word.
By far this is (with ch. 5) the best explanation of Transformer architecture that I've ever seen. I previously understood Attention intuitively but struggled to explain this during job interviews under stress. You just helped me land a job. Thank you!
You made the best attention video I have watched online. This should be a compulary part of all NLP courses right now. Thanks for releasing this banger!
This is pure gold!. Never seen such a good explanation of the attention mechanism before. Thank you for this.
I will absolutely pay attention to this one
This video is worth gold. Trying to understand exactly why and how self attention and cross attention works nearly brought me to tears these past years and this video explains it so well and with visualized examples.
Rewatching the video is all i need apparantly :) On a serious note i've heard so much about this "attention" mechanic and always wondered how it worked so thank you for making this video!
Ooh, you've improved the patreon preview so much in just several days. Bloody well done, sir.
Thanks, and my appreciation to you and others who helped give feedback on the earlier version.
I work with ML for computer vision and have never really understood transformers. This is by far the most clear explanation of them that I have seen!
I'm very much not a math person but I've been binging your stuff for a good 4 days now. Between the general sense of wonder, I keep being reminded of the HG Wells quote "History is a race between education and catastrophe" Idk if that ethos is something you deliberately bake into this channel but I very much see it everything you upload and I can't thank you enough. Idk if you fully appreciate the reach your efforts could have but either way, thank you for taking the time to break this stuff down while also not dumbing it down. I don't understand everything you say but you make me curious enough to figure it out and I feel genuinely empowered as a result. I'm sure I'm not just speaking for myself.
The dissemination of scientific concepts transformed into art. Immensely enjoyable and useful for the education of millions. Thank you as many times as the number of model parameters!
I'm a Computer Science student currently working with a Transformer for my master thesis and this video is absolute gold to me. I think this is the best explanation video I've ever seen. Holy shit, it is so clear and insightful. I'm so looking forward to the third video of the series!!!! The first one was absolutely amazing too. Thank you sooo much for this genius piece of work!!!!
Man I admire your patience to explain this step by step and with great visuals. I can't thank you enough for making such complex topics somewhat comprehensible for people without the corresponding background. The day you started this channel deserves to be a national holiday.
Grant, your work provides immeasurable value. This video is a striking display of the towering heights mankind is able to achieve.
I'm still absolutely fathomed we're now using O(n^3) algorithms like that because we have so much hardware in the last 10 to 20 years. That's the magic of LLMs, its not that they are "large", but that we have engineered enough hardware to be able to play with such things. But in the end its all just a lot of linear algebra, isn't it ?
Imagine 25 years ago, you have your Pentium MMX, with, if you're lucky, an 8MB Voodoo and 32MB RAM, and 8GB disk, and you're told: in 20 years you're going to have a little supercomputer in your computer, thousands of operands wide, 8GB of RAM on there or more. You might not be entirely surprised, Moore's Law sort of checks out, after all you had a 486 just previously and a simple Trident VGA, and maybe a 1MHz Commodore 64 previously before that. And yet it is impossible to grasp this power. And that we'll all be using a dark faced chatting app that runs slower than AIM/ICQ did back then while not doing a whole lot more. But that you would also be able to run mad neural network stuff on your computer with an eerily almost human like capability in language and other complex fields.
So much value in one video. You are making the world a better place.
Truly
You hold this opinion when someone unleash not aligned AI on the world.
@@MrFujinko then its the "someone" who IS the problem in the first place dah
@@Otomega1 You ain't a teamplayer. This is modern world, everyone is to blame, everyone works for the system. You and I and anyone reading is guilty.
D.Trump can lead the resistance against misaligned AI; he has developed several novel strategies for reducing rogue AI systems into a harmless atavistic state.
Just incredible of how clear everything is explained for someone who's doing a nlp course right now, thank you so much!
The way these complex concepts are visualized makes the material very approachable for newcomers like myself. I appreciate this work very much!
There are people … all over the world … like me … who really, really, really appreciate you. I cannot thank you enough for taking the time to share your knowledge and help others to understand this technology much more deeply. Seriously, kudos and sincerest thanks. ❤
To me the part that was more confusing when studying this for the first time was the seemingly arbitrary separation between Keys, Queries and Values. In particular, the Value matrix seemed sort of redundant if you already had the Keys, and the terminology coming from databases doesn't help much imo. This makes it much more clear, thank you!
I think the separation exists because otherwise you'd need some super array for all possible combinations. At least that's the intuition I have about it.
Grant is all you need. This was propably the tenth video or podcast about the subject and only now I understand the underlying motivation for each component it has.
Thank you really, I never wanted to dive deep into how transformers work because Im not using them so much and so I had never understood these concepts as clearly as this, until then I was satisfied by the fact of just understanding the equations. But you explain it in such a clear way it's so refreshing. This is the number 1 reference series of videos on transformers that I would recommend.
This is history in the making. This video will benefit humanity for years to come! Absolutely best out there with amazing visuals
The video says that masking is for preventing later tokens from influencing earlier ones. How does GPT3 handle the sentence: "The cat that left the town came back this morning"? The phrase "left the town" modifies the noun "cat." Later tokens do influence earlier ones in this case.
In that case, masked self-attention would have to do something a little different from how we humans might naturally think about it, for example, baking in all the meaning of a cat leaving the town into the embedding associated with "town".
@@3blue1brownis it right to say that all of the meaning is embedded in the last token, so "cat ""left" "town" "came" and "back" are all embedded in "morning"?
@@Scubadooper Well, remember that the point is to have, on the last layer, the embedding on a word be the prediction of the next word ? So yes the goal is exactly to have the whole meaning of the sentence be on the last token
@@3blue1brown Incorrect, the word "town" would become empty and meaningless once the cat leaves it, this example would make the GPUs explode.
@@Scubadooper You can think about it like "THE MORNING", that important morning when the cat came back after leaving the town (again that sad town that the cat left) previously.
The amount of intellectual satisfaction this series has brought me 📈📈📈 Thanks a Ton.
I have become a witness to such great genius with such a thorough understanding of these concepts. If at all there has been a list of greatest teachers, this guy needs to be up there
I’d really love a series like this on quantum computing. I don’t think any such video exists, let alone anything of 3B1B quality. Well, here’s begging. :)
Videos like this put the entire higher education system to shame. Fantastic! I can’t believe you can watch such an excellent lesson on KZhead for free!
Through my journey in the academic world, I never understood a concept so good and familiarized myself with it compared to learning about it here. Thank you so much for your time and effort.
What an absolute masterpiece thank you so much for breaking this down - I read that paper over and over but never reached the enlightment this video presents ❤
Your work for humanity is invaluable. Aside from publishing free videos about these subjects, the effort you put into making something this complex to something reachable to all should always be commended. I have tried to understand this subject multiple times, reading the papers and everything, but I couldn't create a simple model in my head about what it was doing or what it brought to the table to be such a breakthrough in this field. Thanks a lot!
Do you mean "immeasurable value"? I certainly value it
Maybe you should say "is invaluable" vs "has no value". ;)
@@nelkabosal Thanks!
@@Scubadooper Thanks!
The Simpsons Dr. Nick: “Inflammable means FLAMMABLE??!”
I currently should be studying chemistry and instead I'm here watching your awsome videos. Thanks for presenting these topics so they are interesting
As director of video content for a major educational publisher, this is some of the best educational content I’ve ever seen. Your content gives me ideas of how to shape the future of undergraduate level STEM videos. A true legend and inspiration in this space- thank you for the meticulously outstanding work that you do.
These series are so very good. Thank you for making the dense field of machine learning more accessible.
26 minutes of pure joy!
This comment was 4 minutes after the video released!!
YOU DIDN’T WATCH THE VIDEO!!!!!!
mtfk , this topic is so hard, can't understand what grant is saying after 6 minutes in the video!
There's only one other orator I like as much as 3blue1brown, just one. And that other orator who is just as pellucid, and engaging is Donald Trump. Yep, I only pay attention to 3blue1brown and Donald Trump; it's almost like they are the same person.
Great video as always! Minor quibble at 9:00, I have always heard and understood “attend to” as being from the perspective of the query (the video uses the key’s perspective) so it would be “the embedding of creature attends to fluffy and blue” instead. It doesn’t really matter since the dot product is symmetric, I just haven’t heard it used colloquially that direction (maybe due to the axis that the softmax is applied on?)
Skimming through to determine if this would explain it well to friends who are interested but yeah, I noticed all the matrices in this video are transposed relative to the notation. He's not wrong with what the calculations are doing but it's more confusing this way, especially when he shows softmax(qk^t/z)v as the notation. With this formulation, Q,K,V should have rows of projections so the attention matrix is properly formed. Softmax is then done over rows so when multiplying with V on the right, you're aggregating as weighted rows of V.
I had the same initially, but I'm not a native english speaker. I googled the meaning of attend to and it gave me: "has to do with". So if you think about it like that, fluffy and blue have to do with creature, not the other way around. Maybe that helps.
Congratulations, your transformers series is a technical education masterpiece. I’ve been building products with LLMs for the past two years and shamefully only understood them at a very superficial level. You have a gift for clarifying without oversimplifying and condescension that is rare.
It's amazing how videos like this can cut your learning curve. I have attempted to understand self-attention in the past, but i found youtube videos or articles that were too conceptuals, or papers that got bogged down in mathematical notation. This is the perfectly accessible for anyone that understands ML basics and matrices and vectors. Keep up with the amazing work!
3Blue1Brown Is All You Need
Would there be a Nobel price for educational content this here would be a strong contender, well done!
Hi , i'm not good speaking English , I'm learning and I wanna say thanks you, in spanish there aren't these movie (i'm sorry i don't know say this in youtube, I don't know if this say video like spanish) well, Thanks you I am 16 years old and this help me
Your way of explaining concepts feels so interactive to me. Follows the narrative > A question appears > 3B1D answers it immediately > On to the next subject. Thanks for the superb content!
What a beautiful illustration of attention. I want more on this topic!! Variations, details, depth, concepts, illustrations - this is just wonderful. I could watch this video on repeat and learn something new each time.
When I originally learned how matrix math worked in 9th grade, I originally asked the question many students do. “What’s the point?” If I ever end up teaching 9th grade math, I have a lot of cool examples to share with students that are similarly doubtful
By the time i was at that point at some time in the 90s, i already tried implementing a 3D engine so i knew how useful they can be. But back then, we didn't really have SIMD. This came a little later, which made it that much more relevant. I think it's similar, in that you're reformulating things to make them as regular and as mutually independent as possible specifically in order to be able to parallelise them, but it gives you some neat advantages and expressive power beyond that as well. So many mathematical concepts could be considered optional but all of them increase your expressive power.
Indeed, Tranformers are more than meets the eye
If we put a mask on Figure 01 to make it look more human... would it be a Transformer robot in disguise? 😂
Real transformers have windings. And they constantly buzz or hum faintly.
These videos should be on the Trending list. One doesn’t necessarily need to have a CS/Math background to appreciate this content and the immense effort behind creating this masterpiece! Grant, the world is indebted to your contributions in revolutionising how Math can be taught! You TEACH intuition!
I was able to follow the thought without a single pause, even not being a native English speaker. Wery well done and presented!
It is 1:00 in Sydney right now and I’m up late watching your video from my bed. I should probably get some sleep, I have morning classes, it’s just your content is to God damned interesting. Plus, I’m a teenager. I can’t be separated from my phone accept by 16th century French style beheading. POST MORE VIDEOS! If I can’t sleep you shouldn’t get the luxury!
Q: at 24:27 we learn that all the layers can have different meaning behind how they manipulate with words and each one is not realted to another I get that the final output needs to be the same dimention as the original word in order to nudge it, but why are the value down matrix sizes the same (the rank limitation, the compression from 12k to 128 dimentional space) if some layers may benefit from more dimentions while others might not need as much?
I found his explanation of Value Down/Up a bit unnecessarily complicated first, but I get why he explained it this way. As you said, the output has to match the input (because transformers are also "residual networks" where input and output always lie in the same vector space). So the value matrix has to be a huge symmetric matrix (if you don't use this factorization trick). Query/Key on the other hand don't have to be that huge - in fact if they are too huge then chances are high that everything is not related to everything (because the chance that two random vectors in a huge dimensional space point into the same direction is extremely low). The idea of multi-head attention is that you make query, key and value matrix all same size but split them into several blocks/heads. This way, each block of the value matrix is multiplied by a different attention pattern matrix. The idea is that you keep the number of key, query and value parameters the same while increasing the expressive power of the model (or to use the metaphors from 3blue1brown: you can now ask multiple questions in a single transformer block).
@@kaiduhrkop4335 I see, so some matricies can be recycled and we just have more attention matrix parameters if we want to increase it
I would not call it "recycling". In the previous video he showed that certain directions in the embedding space correspond to "gender", others to, say, "color". Now, very different words in a sentence are important to determine the gender of a word than its color. So you can think of one attention head is searching for words that are associated with gender and only effects the gender dimension in your embedding. Another attention head reacts on colors and only changing the color dimension. In extreme case you could have a single attention head per dimension, but then your attention mechanism would be one-dimensional and very lame. So instead you have one attention head that influences, say, 128 different dimensions of your embeddings that all are influenced by similar words (e.g. they all attent to adjectives) while another attention head is influencing other 128 dimensions and attents more to verbs. Multi-Head Attention is an in-between solution between "one attention mechanism per dimension" and "one attention mechanism for the complete embedding vector".
I can see how much work has gone into creating this masterpiece. Thank you for sharing this with the world.
Possible error at (10:30): *Q_i* and *K_j* should be _row vectors_ so that *{QKᵀ}_{i,j} = Q_i ⋅ K_j* is their dot product. Then, *P = softmax(QKᵀ, dim=-1)* is applied along the row dimension so that each query *Q_i* resolves to a probability row-vector *P_i = (p_{i,1}, ..., p_{i,n})* over the keys that it aligns with most. If *V_i* is also a row vector, then its corresponding output attended row vector *Y_i = ∑_j p_{i,j} V_j* is an affine combination over all row vectors in *V* .
Watching this video made me realize how convoluted the terms we, the research community of AI, have made deep learning to be. Planer / simpler terms are needed in research paper as paper like Attention is All You Need is read by all generation of researchers coming after us.
I'm afraid this applies to all the science fields. People tend to create jargon in subjects they are working on to make their job more efficient / faster for them and others like them. So papers are shorter, more compressed. Others to be able to read it need to learn those "compressions algorithms" ;) Unfortunately it makes it harder to learn anything from scratch so we need people that "decompress" it for us and teach us ho we can do it ourselves.
Researchers use tough language to keep outsider out
@@venugopal-nc3nz Nah, it's very natural process that happens for everybody. More you know more you "compress" the data to make it more effitient to process. That way one sentence can have very deep meaning for others like you and you don't need to write 20 pages to describe new concept you are working on. In fact deep learning implements exactly this: very sophisticated compression algorithms ;)
Computer science is a field so full of misnomers, that it's itself a misnomer. It's a field of engineering not science, and its subjects are not computers, those are the subject of computer engineering (electrical engineering), but software. It's also not software engineering per se, which is concerned with software architecture, though that can be considered a sub-field of computer science. It is more like information processing engineering, with heavy focus on algorithms and data structures. Unfortunately a certain amount of hindsight is needed to name things elegantly in a way that would be more leaning towards self explanatory, but there is resistance to renaming things, because it would break existing workflow.
@@SianaGearz I would argue that computer science is part of science, at least AI part we are talking about. Of course there is a lot of engineering there but without science those engineers would have nothing to do ;)
I can’t believe you can watch such an excellent lesson on KZhead for free!
I woke up this morning and was feeling excited because I knew you will drop the transformers video today! That is what your videos mean to me! Thank you!!! :)
At the end of the day it is the mathematicians who have the best understanding of any concept, be it physics or AI or just pure abstract math. Thank you Grant for your amazing contribution and taking us along this journey...
I've been spending a lot of time, trying to understand transformers, before watching this. I have built a very very small one myself, to better understand, but with my limited resources, it can't do much. Anyway, the Q,K,V was very confusing to me, but now I finally understand. The thing that is most difficult to wrap my head around are the insane dimensions sizes and parameter counts. Those numbers are so high and all of those multiplying together, to get 57 billion parameters for the attention block themselves, is insane and fascinating to me. And then adding the rest, that is not attention...
Hi Akari! If you have time, I would love to talk with you about your process of building the transfomer. I"m , as well, in the process of understanding all this world. What do you think about having a chat? We learn more when we teach ;)
I see upload => I watch
Jesus loves you ❤️ Please repent and turn to him and receive Salvation before it is too late. The end times written about in the Bible are already happening in the world. Jesus is the son of God and he died for our sins on the cross and God raised him from the dead on the third day. Jesus is waiting for you with open arms but time is running out. Please repent and turn to him before it is too late. Accept Jesus into your heart and invite him to be Lord and saviour of your life and confess and believe that Jesus is Lord, that he died for your sins on the cross and that God raised him from the dead. Confess that you are a sinner in need of God's Grace and ask God to forgive you for all your sins through Jesus. Jesus loves you. Nothing can compare to how he loves you. When he hung on that cross, he thought of you. As they tore open his back, he thought of your prayer time with him. As the thorns dug into his head, he thought of you spending time in the word of God. As the spears went into his side, he imagined embracing you in heaven.
@@L17_8 SPAM
Like a delicious coffee, like a fine wine, I sip every minute of these videos deliberately. It should not end in only 26 minute!!
Great, great, masterpiece...I read a lot about transformers and attentions mechanisms, but this makes everything so much clearer..
Love your content ❤ best learning channel
I consider you a great teacher, one of the best in the world 🎉
Jesus loves you ❤️ Please repent and turn to him and receive Salvation before it is too late. The end times written about in the Bible are already happening in the world. Jesus is the son of God and he died for our sins on the cross and God raised him from the dead on the third day. Jesus is waiting for you with open arms but time is running out. Please repent and turn to him before it is too late. Accept Jesus into your heart and invite him to be Lord and saviour of your life and confess and believe that Jesus is Lord, that he died for your sins on the cross and that God raised him from the dead. Confess that you are a sinner in need of God's Grace and ask God to forgive you for all your sins through Jesus. Jesus loves you. Nothing can compare to how he loves you. When he hung on that cross, he thought of you. As they tore open his back, he thought of your prayer time with him. As the thorns dug into his head, he thought of you spending time in the word of God. As the spears went into his side, he imagined embracing you in heaven.
@@L17_8 Cease posting your spam and liking your own comments.
Consistency 🗿
mind blown. had to spend another hour laying down processing all concepts here and what they entail a really impactful explanation
It is very helpful video series. This lecture provides the main features, their principles, and reasons for using them. The most surprising thing about this lecture is that if you watch this series hard, even ordinary high school students can easily and quickly gain relevant knowledge and submit simple reports.
Great video. Kind of saddens me that we're leaving the golden age of open sourced AI research. "OpenAI" used all the publicly shared transformative research from before (Transformers / Convolutions, Batchnorm etc.) used it, and then shut the door behind them. Well done to them for making billions. But unfortunately, this means that the next revolutionary advancement like Transformers aren't going to be published openly.
First here😮
I already knew about Attention mechanism. Jumped onto this video since it was recommended in a newsletter. Would like to thank you for explaining things in concise and easy-to-follow manner with graphics. Learned many knew things and implementation details.
Brilliant! Even when you start to understand how it works, it seems miraculous that it works as well as it does. I love the idea of adjusting the embedding vector to new positions in semantic-space depending on the learned relationships with previous words.
Those who completely understood attention after this video will be feeling like God/Goddess 👼👼
5 views after 1 minute, bro fell off 💀
44k 3hrs in
6 likes after 3 hours, bro fell off
@@chromacat248fr
@@arberstudio bro wouldn't get the joke even if he got it painted on the ceiling of his bedroom
I've read about key/query before, but now it starts to make sense. Thank you!
You made it look so effortlessly easy. Well done.
I honestly did not expect the next video this fast. Thanks a lot of this video. As usual, it didn't fail to deliver
The quality of this video is truly inspiring. Clear and concise explanations, great examples and analogies, highly visual.... I like to think of myself as a good science communicator and this is the level that I want to attain, fantastic
Thank you so much Grant! Even before starting the video I know it's going to be a masterpiece
Absolutely amazing. Recently I've become a student of data science, professionally I'm responsible for RAN network strategy for telecom operator. I am completely deeply amazed how your work helps me with both of this fields! Indescribable :)
Humanity owes you a debt of gratitude beyond measure. As for myself, I can't help but feel a tinge of disappointment. Your original artworks in this channel, in their essence, serve as poignant reminders that life is fleeting, revealing the vast expanse of what I have yet to comprehend (not to mention "to teach").
Incredible video! This is going to help so many people - the impact really can’t be overstated.
17:50 -- Love the 3b1b humblebrag here. essentially "Those paper writers make things confusing, and I am here to lead you with knowledge". Thank you Grant for bringing this to all of us!
Great video again - really getting a feel for all of this - though it's still way over my head ....- and that transition @10:22 was beautifully done!
Another one this quick?!?! You're one of my favorite channels on this platform, thank you for the high quality and informative videos