How are memories stored in neural networks? | The Hopfield Network

2024 ж. 4 Мам.
644 364 Рет қаралды

Can we measure memories in networks of neurons in bytes? Or should we think of our memory differently?
Submission to the Summer of Math Exposition 2022 (#SoME2). More information: summerofmathexposition.substa...
Time stamps:
0:00 - Where is your memory?
1:41 - Computer memory in a nutshell
2:58 - Modeling neural networks
4:42 - Memories in dynamical systems
9:54 - Learning
13:36 - Memory capacity and conclusion
Animations largely made using the manim community edition:
www.manim.community/
Original Paper on Hopfield Networks:
Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8), 2554-2558.
Neuron image by Santiago Ramón y Cajal, The pyramidal neuron of the cerebral cortex, 1904 Ink and pencil on paper, 8 5/8 x 6 7/8 in. Credit: Cajal Institute (CSIC), Madrid
Music: Aakash Gandhi - "Dreamland"

Пікірлер
  • As a neural network, I confirm that this is how we store memories

    @aegontargaryen597@aegontargaryen5978 ай бұрын
    • early

      6 ай бұрын
    • Lmao

      @ugestacoolie5998@ugestacoolie59986 ай бұрын
    • All the screws of those who are OBSESSED to think that people are nothing but cybernetic machines are rather loose…

      @claudiamanta1943@claudiamanta19434 ай бұрын
    • @@claudiamanta1943 I haven't seen anyone thinking that people are cybernetic machines lol

      @aegontargaryen597@aegontargaryen5974 ай бұрын
    • LMAO

      @merozemory@merozemory4 ай бұрын
  • is this a first video? it's surprisingly good

    @murilospineli2241@murilospineli2241 Жыл бұрын
    • My god I need to subscribe immediately

      @jasoneel76@jasoneel76 Жыл бұрын
    • Oh my god i didn't figures it out until you commented!

      @jaumeivars@jaumeivars Жыл бұрын
    • When something is made by someone who cares, you get a good result. Caring is the first ingredient in the recipe for success.

      @ryanpmcguire@ryanpmcguire Жыл бұрын
    • I thought the same!!! This guy is really good. Thanks!!

      @fadonoso@fadonoso Жыл бұрын
    • The first video uploaded to a channel does not mean the person/people editing/producing the video had no prior experience. That being said, it's a fantastic first upload and this channel is off to a tremendous start!

      @NickSklias@NickSklias Жыл бұрын
  • This guy explained such a complex topic so well, great job.

    @piggy9539@piggy9539 Жыл бұрын
    • It's going viral soooo ye

      @alexanderhandjiev2845@alexanderhandjiev2845 Жыл бұрын
    • @Yatish38 yeah and the subs for one video is crazy but deserved too

      @mihailmilev9909@mihailmilev9909 Жыл бұрын
    • @Yatish38 Really good, yes, deserves more, also yes

      @RoseOnFire@RoseOnFire Жыл бұрын
    • Yeah i had trouble with the super/subscripts in all the math. So i just made a video easily explaining multilayer back prop with only 1 subscript. It's so easy now!

      @TheRainHarvester@TheRainHarvester Жыл бұрын
    • If humans know how it works, it’s a simple subject that is explained badly. If they don’t, it’s too complicated anyway. For example, we know nothing about how the brain works because it is simple too complicated. However, we know plenty about gravity, other areas of anatomy, computers, and more.

      @pizzainc.1465@pizzainc.14652 ай бұрын
  • You nailed the 3blue1brown feeling whilst being ligther on the maths. I wish you the best of luck with future videos! 🙂

    @luke2642@luke2642 Жыл бұрын
    • He likely uses Manim, the software made and used by 3b1b.

      @ianthehunter3532@ianthehunter353211 ай бұрын
    • Are you guys also watching the 3B1B videos about GPTs?

      @poultrypants@poultrypants25 күн бұрын
    • Aww, you never did a follow up!

      @luke2642@luke26425 күн бұрын
  • I've been idly musing about the brain's memory in terms of binary and bytes for years and always thought it was such a limited way of looking at the subject but didn't know any better. I asked around in the past about alternative ways of looking at the subject but never came away with a decent answer. This video has been such an eye opener and i think i'm going to have to take a deeper dive into the subject. Thanks!

    @Sazazezer@Sazazezer Жыл бұрын
    • You should haken's book on synergetics

      @Daniel-ih4zh@Daniel-ih4zh Жыл бұрын
    • You mean, like, deep into neural nets? ;P

      @andrewhancock2451@andrewhancock2451 Жыл бұрын
    • I heard in an scifi anime (which is surprisingly more accurately scientific than it may seem despite being scifi) called Steins;Gate that said the brain can hold a little over 4 terabytes of memories, but that was just a claim, I'm not sure if that's true.

      @SpringySpring04@SpringySpring0411 ай бұрын
    • Memory doesn't just exist in the brain; there is evidence it is dispersed throughout the entire body. People have had large chunks of their brain removed and their memory was unaffected. People born with basically no brain, just a small layer around the skull and the 'cavity' where the brain should be was filled with cerebral-spinal fluid. And yet their IQ and memory was well above average. I know it is hard for purely materialistic to understand, but consciousness is more fundamental than physical matter, with the brain merely being an interface for the physical world A true ghost in the machine; consciousness does not arise from the machine being complex; it is controlling the biological machine

      @pyropulseIXXI@pyropulseIXXI11 ай бұрын
    • @@pyropulseIXXI Exactly, I posted a heap of times about this already, this video really inspired.. brain caching, permanent information, rinse and repeat with new context. Optimization is often overlooked. It will be obvious sooner or later, there is no other way to efficiently battle entropy. We get sense data and need a nervous system and brain in general, but uncertainty is the ultimate optimization. Probabilistic by design.. part of why the speed of light is so high. We see the very very distant past, and are disconnected from it.. we get data and interpret it and wonder how the heck physical matter organized itself a certain way and why the math doesn't work 0 to 1 is a logical impossibility.. the Big Bang is a misinterpretation, but a plausible back story for the universe and adds to the immersion. We process things bound by our brain capacity (see Einstein's freak brain!) and easily fall into materialist traps of our own design. Logically the universe and biology is simple, but obfuscated by so many LAYERS of background processes ! we call it subconscious.. yeah it's busy !

      @goldnutter412@goldnutter41211 ай бұрын
  • This is remarkable. How the heck does this channel only have 1 video?!

    @keithmiller3770@keithmiller3770 Жыл бұрын
    • Still waiting :( ...

      @rabiaedaylmaz1198@rabiaedaylmaz119811 ай бұрын
    • yeah i knew this video would be good. i noticed it few months back when i was beginning to study neural networks, just stored it, knew wasn't ready yet. now finely watched, as expected, it turned out good. nothing like clickbat stuff that usually out there on youtube

      @alexb3617@alexb361710 ай бұрын
    • Everyone must start somewhere :)

      @Mikee512@Mikee51218 күн бұрын
  • "Contrast this with what we believe about the brain..." I like how careful you are with how you make your statements.

    @sebastianmuller4649@sebastianmuller4649 Жыл бұрын
  • The amount of effort required to not only curate such complex information in an intuitive way, but also to animate the concepts in the way you've done, is clearly tremendous - thank you for taking the time to produce this! I'm inspired by your work, really looking forward to more!

    @mmarrotte101@mmarrotte101 Жыл бұрын
  • You cannot fathom my disappointment when I looked at your channel and discovered that this is your only video. What do you mean there aren't any others? I need more! Seriously, interesting topic, crystal clear presentation, keep up the great work!

    @gianlucazinarof8617@gianlucazinarof8617 Жыл бұрын
    • Do you know, what software able to produce and animate these animation? adobe after effects or any software else?

      @Mangputt@Mangputt Жыл бұрын
    • It is a custom toolbox written in Python, called Manim. Here it is the community edition, but Manim was originally devised and open-sourced by 3blue1brown. There is a link in the description if you want to learn more.

      @layerwiselectures@layerwiselectures Жыл бұрын
    • @@layerwiselectures thankyou... I still waiting for your next videos. Keep it going Bro 🔥

      @Mangputt@Mangputt Жыл бұрын
  • This is amazing! I finished the video and immediately went looking for the next one where you'd explain the more accurate models. Then realized this is the first and only video on this channel. Can't wait for the upcoming stuff. Congrats for the launch!

    @NajibElMokhtari@NajibElMokhtari Жыл бұрын
  • It's really amazing how something as simple as a neuron has highly emergent properties that can do very complex tasks.

    @macchiato_1881@macchiato_1881 Жыл бұрын
  • I studied Hopfield networks last year in the framework of spin glasses and all assumptions were dropped seemingly out of nowhere in a big confusing mess (as usually happens in lectures). This video really tied up all together and now makes so much sense beyond memorising proofs. Excellent video and brilliant demonstration.

    @frenk3928@frenk3928 Жыл бұрын
  • Dude made one video and then stops uploading. You have a talent.

    @Ana_Saudah@Ana_Saudah3 ай бұрын
  • This is one of the most important pieces of media I have yet consumed to better understand this world. It is incredible to realize how one's brain works in principle. Thank you so much!

    @sciencelab4225@sciencelab4225 Жыл бұрын
  • Hopfield nets were the first thing that got me interested into studying neural networks some years back, but I hadn't seen such an eloquent presentation on it since now. Will wait for the subsequent videos of this series.

    @pafloxyq@pafloxyq Жыл бұрын
  • Please continue making videos! This was not only super interesting but also very well explained and animated. I hope to see more videos on this channel soon!

    @konradd8545@konradd8545 Жыл бұрын
  • Looking forward to watching the second part! And many thanks for choosing dark text on a light background. It looks like a calm creamy ocean of fascination!

    @Wonders_of_Reality@Wonders_of_Reality Жыл бұрын
  • Wow, this video is really eye-opening. As someone who has been interested in both memory sports and math, it really intrigues me how these false memories arise. Whenever I make a mistake memorizing a deck of cards, I'm always trying to figure out what when wrong, and it's cool to see a mathematical model also make similar mistakes in your video.

    @BrandonWong55@BrandonWong55 Жыл бұрын
  • An excellent explanation. I'm starting to realize what a profound influence this type of neural network model has had in recent decades on all kinds of disciplines ranging from machine learning to the development on new theories on PTSD and how traumatic experiences alter brain function. Thank you for providing such a succinct and illuminating video on a complex and vitally important subject.

    @brandontylerburt@brandontylerburt Жыл бұрын
  • The best illustrated explanation on Memory modelling of inter-connected neuron (switch), bit and quirks of Neural Networks , formation and significance of multiple "stable states", and how they are presented as row & col vectors, update function as Matrices. Your video is both clear and excellent visual intro to the topics. I am really looking forward to your next video. Helping more ppl to visualize how transformers, recurrent net, back propagation, main idea of ML papers, etc are being modeled.

    @exinangai2216@exinangai2216 Жыл бұрын
  • This is absolutely incredible. Just immense, wow. I was initially skeptical of neural networks because they lacked the "logic" side of the problem (as I think a lot of people were at first) but I knew they were unreasonably effective, so I assumed they had their place. But now, I think they have more than a place, they are officially critical to our understanding of the world. In short, they were always cool, but now we've proven it. Thank you for making this video! It is so well made by the way!

    @TR-kn3sn@TR-kn3sn Жыл бұрын
  • Brilliant exposition for both mathematicians and nonmathematicians. It took a few decades to get me (mathematician) interested in neural networks. Now I'm hooked. Musing about connections to Markov processes and PDE...

    @martingisser273@martingisser273 Жыл бұрын
  • guys, just wanted to remind you that this video is easily in the top 1% videos ever produced by a human

    @question_mark@question_mark2 күн бұрын
  • This is… ridiculously good. Thank you very much for making this, and really looking forward to the next one!

    @eldertom@eldertom Жыл бұрын
  • Such a unique topic, so beautifully explained! Thank you so much for this piece of art!

    @YasheshBharti@YasheshBharti Жыл бұрын
  • Different "seed" states will call up different memories. I wonder if this could be analogous to how certain ideas link together, and maybe some of the underlying information of related ideas is actually a correlated section of their memory patterns.

    @HansLemurson@HansLemurson Жыл бұрын
    • This is wrong, our brain can excite and depress certain neurons with neurotransmitters, hormones, data from sensors of the environment and our own body etc...

      @MrNatsuDragneel@MrNatsuDragneel Жыл бұрын
    • This explains why I have an idea at my desk, get up to walk across the room and forget what I was thinking about. Get back to sitting at my desk seed state, and remember it, get up and forget it again.

      @hufficag@hufficag Жыл бұрын
    • @@hufficag omg we figured it our also lmfao

      @mihailmilev9909@mihailmilev9909 Жыл бұрын
    • @@hufficag just take a look on en.wikipedia.org/wiki/Encoding_specificity_principle

      @floriantesto4925@floriantesto4925 Жыл бұрын
    • the video is meant to explain how modern artificial neural networks work. biological brains don't work that way. Take for example a deer. Within an hour of being born, a deer can walk and within a few hours it can run. Clearly the baby deer didn't learn how to run in the womb. The evidence suggests some kinds of memory are genetic, while other skills are learned. Even if there was seed states, keep in mind memories are stored redundantly in biological brains. the obvious example is stroke patients that temporarily loose skills, but then recover them. Assuming there something like a seed, it would be seeds or parts of a seed stored in multiple places. then there's the fact that brains are analog, not digital. In classic computer science, associative memory is two elements linked with an address that points to both elements. In neural networks like GPT3, the association is captured by many weights. One could consider those weights as a seed, but the analogy breaks down. A common problem with neurons in a neural network is that a single neuron may participate in multiple patterns. If you change 1 weight, it can cause the network to perform better on one type of task and worse on a different task. The problem is called polysemantic neurons. There are people working on fixing polysemantic neurons, but there's no general purpose fix today.

      @woolfel@woolfel Жыл бұрын
  • You are by far the best teacher for soft soft . It's very complicated at first - overwhelming, actually - but, you make it doable for

    @rafaeladivapinho2399@rafaeladivapinho2399 Жыл бұрын
  • It’s rare to find a concise video about a learning topic. This is awesome! Hope your channel grows huge!!🎉

    @histeward@histeward Жыл бұрын
  • This is just an outstanding explanation and it was a very interesting subject. And it's even more amazing that even though I know very little about this subject I could understand almost everything Keep up with the great work :)

    @br.3250@br.3250 Жыл бұрын
  • Love it! Hopefully this video will get big, this question is wonderful and really makes you think outside the box, and I think you explained it perfeclty.

    @tf3confirmedbuthv54@tf3confirmedbuthv54 Жыл бұрын
  • I really like how you explain neural networks with necessary elements only which helps me focus on the concept itself! Nice work!

    @hcky6054@hcky6054 Жыл бұрын
  • can't believe it's your first video. it's very well explained and the background sound makes it even more compiling.

    @LiquidMasti@LiquidMasti Жыл бұрын
  • Perfect video. I didn't know you could look at neural networks from this neat mathematical vantage point. I thought about eigenvectors and how the eigenvector with the largest eigenvalue would influence the network sort of as a dominant weight that increase exponentially , this would need a slightly different f the function you operate on the resulting weighted sum this would work for storing one memory maybe with increase possible states of single neuron and the right activtion function you will be ablse to store many patterns. Amazing video once again.

    @redaabakhti768@redaabakhti768 Жыл бұрын
  • I don't know why you haven't made another video in 9 months, but I'm hoping you continue this series. Maybe go into the math a little bit more, but keep it simple so even a simpleton like me can understand it.

    @anon_y_mousse@anon_y_mousse11 ай бұрын
  • the IDEA explained in this video is something a was looking for for a looong time, thank you so much you have no idea how helpful you have been.

    @bakdiabderrahmane8009@bakdiabderrahmane8009 Жыл бұрын
  • Thanks for opening my eyes. I already knew about recurrent networks and stuff, but now I think about them with an existential crisis. Thanks for such an awesome explanation!

    @hardronix@hardronix11 ай бұрын
  • Even a reductionist model of neural network (disregarding the dynamic spatio-temporal interplay of a plethora of neurotransmitters, neuromodulators, receptors, secondary messengers and downward pathways) leaves one awe-struck by hinting toward the tremendous scale of capabilities in a biological neural system. Thank you for this wonderful presentation and thanks to the YT algorithm for suggesting it! May your channel and its viewership grow exponentially!

    @dererzherzog@dererzherzog Жыл бұрын
    • And don't forget, the _in_ capabilities. Even from this extremely simplified model, you can see why it's hard to write memories, and keep them stable against change (which of course happens even when you're just _reading_ the memory). The computer on your desk can easily store terabytes of information without a glitch just by "remembering" the information one by one, while the computer in your skull needs to keep reinforcing what you want to remember, and keeps overwriting related memories as others are read or changed. Mind, this is most likely a critical component of learning and intelligence in the first place - your memories aren't just static data stored elsewhere, they're ultimately a part of the computational network itself. Every experience has _some_ effect on your decision-making, and memories interact with each other (and with other inputs from all over the brain and the outside world).

      @LuaanTi@LuaanTi Жыл бұрын
    • @@LuaanTi apparently the brain throws out TONS of data all the time.. anyway won't repeat the other posts.. maybe read those first above.. but it's a good filter/cache for information processing.. short, long and permanent information. Only what is really significant to us, lasts.. and we come back to evolve past our problem with entropy. The unknown is enticing.. fear of the unknown is like getting stuck in loops on tangents.. or outright stuck and unable to deal with whatever patterns we're experiencing. The optimal configuration for experience is of course permanent state, temporary brain and universe. Next time, the patterns will have a completely different basis, while still understandable ! almost every random set of DNA and environment will be the same pattern. Choose, which gives result (quantum mechanics delivering data on demand), JIT type of interpretation with the brain cache, more choices, more life experience, on and on we go. Because we are self aware, we are aware of entropy.. time advances, we collect more data. More entropy unless we evolve away from "fear" or.. I guess you could call it over management of the data structure. That's the nature of information ? "Let it be" playing in the background..

      @goldnutter412@goldnutter41211 ай бұрын
  • I guess you have to make a distinction between short term memory (stored on activity patterns) and long term memory (this video? stored in the network weights). The latter is much simpler and in theory there could be a single neuron for a single memory, although in practice there are more efficient encodings.

    @cmilkau@cmilkau Жыл бұрын
  • I just learned your video was for 3Blue1Brown's contest by watching his video about the contest. Makes sense why you only have 1 video. I originally watched it weeks ago and was dumbfounded by it being your debut. PPLLLEEEAASSSEE make more! You have a knack for explaining a very complex topic in terms us commoners can understand!

    @Ryush806@Ryush806 Жыл бұрын
  • Mindblowingly good! You left me in awe, please keep uploading videos!!

    @jaumefabregaroda@jaumefabregaroda Жыл бұрын
  • Yes yes yes this is exactly the video i didnt even know i needed. incredibly good explanation and animation

    @schnitzelsemmel@schnitzelsemmel Жыл бұрын
    • same

      @Red_Fox_Miro@Red_Fox_Miro Жыл бұрын
  • Super interesting video, got me really thinking about the correlations to real neurons.. very well made as well :)

    @nulled7888@nulled7888 Жыл бұрын
  • By looking at this video, you just updated the neural network in your brain to be able to remember stuff about neural networks. Brilliant.

    @Tigrou7777@Tigrou7777 Жыл бұрын
  • Super interesting and fantastically presented! I wish there were more videos like this! Love that you're not afraid of diving into deep concepts and details. Instant sub.

    @alexharvey9721@alexharvey9721 Жыл бұрын
  • Love the presentation. I think that the mistakes you were frustrated and fascinated with are maybe a chance to generate something new from something stored. It reminds me of our ability to be creative by morphing our memories over time. It's like when you have memory A and memory be and over time they fade and become memory AB which then turn into memory C. And sometimes memory C might actually be something useful. It also reminds me of generative AI. Of course, this is all just layman talk and I have no idea if this has anything to do with how human memory actually works.

    @somedudeok1451@somedudeok145111 ай бұрын
  • This was super interesting and awesome!! I'm excited for the next video :)

    @venamaybe@venamaybe Жыл бұрын
    • looking forward to the next video too

      @user-dp1od3gd3u@user-dp1od3gd3u4 ай бұрын
  • Very rarely KZhead has video with this much deep insight. Treasure.

    @hengchunsong301@hengchunsong301 Жыл бұрын
  • Absolutely mindbogglingly well done. Can't wait to see more videos from you!

    @matthiasdenhamer3925@matthiasdenhamer392511 ай бұрын
  • What I'm getting out of this is that there is still a scalar memory parameter in the number of neurons or size of the weight matrix, and storing patterns in it is similar to file compression, allowing the same network to represent multiple patterns, but with a compression ratio that is actually disadvantageous, requiring O(n^2) weights to store O(n) patterns. Something of a Rube-Goldberg for memory systems.

    @brandonzhang5808@brandonzhang5808 Жыл бұрын
    • Eactly, this does not prove that our memory can't be measured in Gb of information.

      @dominikwitczak1655@dominikwitczak1655 Жыл бұрын
  • Awesome video, comparatively digestible explanations and really good visualisation of what you're describing.

    @alzario@alzario Жыл бұрын
  • I find this explanation is fascinating. Shines light on neural networks from an uncommon direction. Great Video 👍

    @TenderBug@TenderBug Жыл бұрын
  • Amazing video. Very insightful. I cannot imagine how much work went into the animations. Very well done!

    @JoelGreenyer@JoelGreenyer Жыл бұрын
  • Amazing video! Great job, i would love you to explain autoencoders

    @ofrigal9557@ofrigal9557 Жыл бұрын
  • This is the best SoME video I’ve seen so far. Such an intuitive explanation for a niche yet mindblowing idea. Maybe it’s my lack of intuition for high level math or my affinity to computer science, but most of the other videos had me closing out of it mid way, while this one was interesting in its entirety.

    @d0tz_@d0tz_ Жыл бұрын
  • The content you provide is greatly appreciated. We are looking forward to your new contents. ❤

    @sinaomidvar2807@sinaomidvar28072 ай бұрын
  • Instant subscription. This video was extremely clear and well-made. Looking forward to your next lectures!

    @Gnevnyj@Gnevnyj Жыл бұрын
  • If you change the procedure to use a synchronous update, and make the activation function the identity, then you can analyze the dynamics of the weights matrix W. That is, v_{n+1} = W*v_{n}. In this case, a memory which you converge to is simply a stable eigenvector of the matrix W. When you "train" the network with multiple memories, you are performing multiple rank-one updates to the matrix. This also explains why the maximum number of memories you can store in W varies linearly with the size of W, since a matrix of size NxN cannot have a rank greater than N. In other words, if you tried to train W on N+1 memories, you are guaranteed that at least one of those memories is a linear combination of the other memories. Not only that, if you add a memory with a rank-one update to W, that does not necessarily mean that memory will be an eigenvector of W. The memories can combine such that an eigenvector emerges which is not equal to any of the memories. This approach also gives another perspective on why you can converge to the "flipped" version of the memory. An eigenvector defines a 1-dimensional space which is simply a scalar times a vector, where the scalar can be positive or negative. Also, the properties of the W that emerges from Hebbian Learning are very interesting. That is, W is symmetric and has a zero-diagonal. It is a property of symmetric matrices that their eigenvalues are all real. So we would not expect the network to oscillate around some solution. Also, the trace of a symmetric matrix is equal to the sum of the eigenvalues, so that means if we have negative eigenvalues, we must also have positive eigenvalues. Again, this is just some conclusions for the case of synchronous update and identity activation function. It may not be so relevant to the case you discuss.

    @theClaytron@theClaytron Жыл бұрын
    • That's perfect, thank you. The original paper does not use this reasoning to arrive at the linear memory capacity, but the similarity of the conclusions is striking. Thank you for this addition!

      @layerwiselectures@layerwiselectures Жыл бұрын
    • @@layerwiselectures Thank you for your excellent video! I am looking forward to the sequel.

      @theClaytron@theClaytron Жыл бұрын
    • @@layerwiselectures Have you heard of adaptive resonance theory? Could the stable states in the network be harmonics? Like if each pattern is stored not just spatially but at a specific frequency of oscillation then you could maximize the number of memories a single neuron could contribute to. Kind of like the tv's that play multiple images on the same screen at once

      @isaacaraya3848@isaacaraya3848 Жыл бұрын
    • Great insight to complement video. Could you expand on why symmetric matrices can't oscillate around a solution? Might save me some googling

      @isaacaraya3848@isaacaraya3848 Жыл бұрын
  • Thanks for this great lecture! My idea for a next video topic is that you can talk about differences between artificial perceptrons and biological neurons

    @dominicbaumer6038@dominicbaumer6038 Жыл бұрын
    • Yes, thank you! Okay, goes on the list.

      @layerwiselectures@layerwiselectures Жыл бұрын
  • Wow this video is Gold, I'm impressed by the fact that this is your first! I will look at your career with great interest

    @fabiospinelli4179@fabiospinelli4179 Жыл бұрын
  • you knocked it out of the park dude! need more of the good stuff!

    @awritrojitbanerjee1055@awritrojitbanerjee1055 Жыл бұрын
  • Hm, I don’t know that this really shows that the memory capacity can’t be appropriately described in terms of “how many bits”. That there aren’t addresses for physical locations for different chunks of memory, sure. But, information theory isn’t fundamentally inapplicable to the brain’s ability to remember things? Like, if there is some random distribution of possible observations (e.g. a sequences of symbols, or other combinations of e.g. color, shape, texture, whatever) at one time, and then at a later time the person says their guess about all the information needed to describe which of the observations were randomly selected, then, ((the entropy of the distribution of observations) - (the conditional entropy of the distribution of observations given the person’s recollection of the observation)) should give a lower bound of how much information about the observation the person remembers on average. This can be expressed in bits. Of course, this isn’t describing the person’s entire memory capacity! Just how much they remember from these observations in such tests. This could, I think, be used to measure an estimate of a lower bound on how much information (measured in bits) a person can rapidly memorize for a short amount of time. Presumably how much info one could memorize would depend on the way the information is expressed. For example, I would expect a random sequence of words would be easier to memorize than a random sequence of literal bits with approximately the same entropy. And that’s part of why I say it would only be an (approximation of) a lower bound. (Approximation because there’d be measurement error due to the random variation.) We can also probably put an extremely weak upper bound on total capacity based on the amount of information theoretically needed to describe the physical state of the brain. If nothing else, there is at least the Bekenstein bound based on just the size of the brain along with the fact that it isn’t part of a black hole. This is so weak of an upper bound that it is barely worth mentioning. I don’t know how to get a better estimate of “the entire memory capacity of the brain” than these though. Well, of course, you can look at how much data people have memorized over longer periods of time. Oh, maybe some estimates could be made based on people with eidetic memory. Like, if we query facts about randomly selected dates and check how accurate what is recalled is, then, we could extrapolate that and assume it would be equally accurate for all the many dates in the same range which we didn’t ask about, then we could estimate from that how much they remember overall from that range of time. This could probably give an estimate which reveals a rather large amount of information stored. And, presumably, people with an eidetic memory don’t have just, an inherently larger memory capacity, so much as, just more stuff gets committed to memory? (I could be wrong about that.) So I think this would still give a good estimated bound on how much memory capacity a brain in general has.

    @drdca8263@drdca8263 Жыл бұрын
    • Wow, thanks for the comment. There's a lot to think about. My favorite part is the black hole information bound. No you're right, information theory isn't fundamentally inapplicable here and I didn't mean to imply it was. Personally I'm not clear on whether the "bits" in a hard disk are information theoretic bits. Are they? Maybe in the sense that, if I didn't know the state of the hard disk, and then I looked to find a certain pattern there, I would have gained that many bits of information? But then on the other hand I wouldn't expect every pattern to be equally likely to appear on a hard disk or in RAM. What I'm getting at is that, yes, there must be a way to compare the information in both memory systems (computer and neural) in bits, but no, it isn't usually being done when we talk about computer memory. My goal was to show that casually speaking of bits of memory has a tendency to obscure real differences between the two memory systems. If in the process I appeared to discount information theory ... oh my, not my intention. Thanks for the comment again!

      @layerwiselectures@layerwiselectures Жыл бұрын
    • @@layerwiselectures I think I phrased my comment a bit too critically given how much I liked the video. I quite liked the video, and subscribed (before your reply). I think computer bits can be related to bits in information theory by, if you sample from a distribution many times, the optimal way to store such a sequence on disk, would have its length be on average, the number of samples times the entropy of the distribution. And, I think if the number of samples is large, then the variance in how much storage is needed overall should go down, eventually to the point where you can regard it as nearly always the same. ...uh, assuming the variance of the number of bits each sample contributes, is finite (which it will be if the number of possible outcomes is finite). So that’s at least *a* connection. It perhaps isn’t quite as close a correspondence as one might hope? Well, the amount of information in an information theory sense that can be stored by a hard drive, is bounded by the storage capacity, and at least for some distributions of a random variable, this bound is exact. But for other random variables where the outcomes aren’t just 2^n equally likely outcomes, then I guess there can be some overhead compared to the ideal, in storing a finite number of samples. But in the limit of many samples, I think the two notions of “amount of information” become the essentially the same? In some sense anyway.

      @drdca8263@drdca8263 Жыл бұрын
    • @@layerwiselectures You're correct in saying that information theoretic bits don't always correspond exactly to hard drive bits. In order to calculate the amount of information stored, you need to also understand the encoding. For instance, ASCII text uses 7-bits of information per character to store text, but these 7 bits of information are often stored at byte boundaries, resulting in each character taking up 8 bits of space on the disk. So the leading bit is wasted. Compression can also reduce the amount of bits needed to store a certain piece of text drastically. So, as long as your data is not maximally compressed (which is hard to tell), then your hard drive bits is not the best measure of the amount of information stored. However, in practice, we typically use file size as a good proxy for information.

      @sullivan3503@sullivan3503 Жыл бұрын
  • That's really neat. I wonder what happens if you add "eyes" to the network, i.e. some subset of neurons whose activation is driven by an external signal instead of the update function.

    @Kaepsele337@Kaepsele337 Жыл бұрын
    • Good point. The original paper actually mentions an input current to the neurons. This would then be combined with the summed input of all other neurons, so it would modify the update function rather than replacing it. I left this out for simplicity. However, it's actually an important detail once you start thinking about it. It sort of biases the network to a certain state and would explain where the starting state of the network comes from, if it's not random. The story would then go like this: the "eyes" of the network (love the metaphor btw) would cue the network with a partial state and the network would then autocomplete to its memory state. Kinda-sort-of-almost-like seeing a visual cue of some sort and then being reminded of a scene from your past - if we're being a little bit fanciful. Thanks for the comment.

      @layerwiselectures@layerwiselectures Жыл бұрын
  • This is brilliant. I've never seen any video about Hopfield network as interesting as this video. All they show are just math. Man you should do more video on brain and neuron networks like this :D

    @LamNguyen-zc8xs@LamNguyen-zc8xs Жыл бұрын
  • great first video! keep it going I'll be here to support you!

    @pumkin7222@pumkin7222 Жыл бұрын
  • Watched

    @user-of9sr8bm9i@user-of9sr8bm9i Жыл бұрын
  • Too complicated for somebody with 0 knowledge in math

    @okopyl@okopyl Жыл бұрын
    • Do you have the ability to learn? I do, and I'm slow, especially on things with many steps that must be done in an exact way or the whole "thing" will be incorrect

      @BrandonMcCurry999@BrandonMcCurry9998 күн бұрын
    • Or were you not talking about you

      @BrandonMcCurry999@BrandonMcCurry9998 күн бұрын
    • Difficult doesn't mean it can't be done, many things have taken alot of work for me specifically Some things come naturally I tend to go with what comes naturally, Plato my strengths

      @BrandonMcCurry999@BrandonMcCurry9998 күн бұрын
    • Elementary school

      @dorionporath4363@dorionporath4363Күн бұрын
  • Thank you for this very high quality and interesting video! The visuals and examples really help in making the topic easier to understand. God bless you and keep up the great work!

    @arvid_music@arvid_music Жыл бұрын
  • This video REALLY helped me make sense of Hopfield Networks. Outstanding work

    @htfx11@htfx11 Жыл бұрын
  • This had amazing animations and insights I have been wondering about for a long time, and although I didn't understand much of the math I still felt grounded by insight after insight to the end

    @Neptoid@Neptoid Жыл бұрын
  • What a great quality video! Please keep releasing more like this!

    @meteorinc4259@meteorinc4259 Жыл бұрын
  • Nice video!! Really like how you broke down complicated terms into more manageable discussions.

    @donewithwork@donewithwork11 ай бұрын
  • Wow! Awesome first video! Subbed and eager for more!

    @fabioteixeira868@fabioteixeira868 Жыл бұрын
  • Amazing video, I tried my self to model a neuron network on my own but barely even knew where to start. Fascinating subject and an excellent video.

    @clementej03@clementej03 Жыл бұрын
  • just stumble into this video on my feed absolute awesome please make more videos like this !!!!

    @TheMan84466@TheMan84466 Жыл бұрын
  • had an idea to write about this, "is the brain really a RAM or ROM?", but seeing you do justice to it, I can safely procrastinate for another day. it's kinda sad too that you only have one video. please do more.

    @Prince-xp9vk@Prince-xp9vk Жыл бұрын
  • I really like your definition of memory! This way we can use fix point analysis and non-linear dynamics to model memory. Great!

    @hanswoast7@hanswoast7 Жыл бұрын
  • Congratulations on winning the honorable mention prize for the Some2 competition. Your entry is my favorite

    @chanyan2024@chanyan2024 Жыл бұрын
  • This is awsome ! Man, you really understand things deep and what is more, you really explain the essential!

    @ariondorin4480@ariondorin448011 ай бұрын
  • This was a fantastic explanation. Waiting for more from you!!

    @satadrudas3675@satadrudas36752 ай бұрын
  • Awesome video, this clarified some things I'd wondered about and taught me some things I hadn't thought about yet.

    @AdobadoFantastico@AdobadoFantastico Жыл бұрын
  • I'm so happy to watch you're first video before you made the second one.

    @depresty@depresty11 ай бұрын
  • Excelent explanation in quite concise form. Great job !

    @AK-vx4dy@AK-vx4dy4 ай бұрын
  • What a fantastic explanation. I'm looking towards more content from you!

    @user-pd4bb8tx2n@user-pd4bb8tx2n6 ай бұрын
  • Magnificent! Thank you.

    @michaeljames5936@michaeljames593611 ай бұрын
  • Astonished that this video is the first video by you. Kudos 👌

    @manofculture69@manofculture69 Жыл бұрын
  • I can't imagine, how great you put effort to this Informative Video. The way you explain, animate and Compositing evidence, awesome.. (Sorry for my bad English)

    @Mangputt@Mangputt Жыл бұрын
  • What a fantastic video! I'm eager to see what comes next!

    @fr3zer677@fr3zer677 Жыл бұрын
  • Nice way of presenting and storytelling, got me motivated to do some experiments. Thank you for the video.

    @alexanderk5835@alexanderk5835 Жыл бұрын
  • That was superb! Can’t wait for the next one!

    @bishalsantra@bishalsantra Жыл бұрын
  • Pieces of images and thoughts put back together in our brains. Makes a clearer picture.

    @jslevenson101@jslevenson101 Жыл бұрын
  • Well done. Let me add my encouragement for you to keep on keeping on.

    @2ndfloorsongs@2ndfloorsongs Жыл бұрын
  • Dude, this is the best video on the topic I've found (and I've looked). When can we expect a follow-up? Would love to watch it.

    @loganpowell@loganpowell2 ай бұрын
  • Looking forward to the modern Hopfield network video! Very clear explanation

    @revimfadli4666@revimfadli4666 Жыл бұрын
  • When a video catches your curiosity in the first two minutes that’s because it’s going to be a piece of art !

    @FelipeCampelo0@FelipeCampelo03 ай бұрын
  • didnt realize the views and subscribers until casually saw comments. keep it up, such a good video!

    @asanzramaxi@asanzramaxi Жыл бұрын
  • Well done! I'm sure I'm not the only one who would appreciate it if you did more =)

    @Personal-M.I.S.@Personal-M.I.S.3 ай бұрын
  • The conclusion really was what I was here for. I didn't get most of it but I got the point of it!

    @acyned8079@acyned8079 Жыл бұрын
  • Can't wait for the follow up video!

    @mauroabidalcarrer4083@mauroabidalcarrer40836 ай бұрын
  • Thanks for the tutorial, it's much faster than any other method I came across.

    @toni8632@toni8632 Жыл бұрын
  • This is unbelievable video. Thank you!!!

    @naiqiangqian4080@naiqiangqian408011 ай бұрын
KZhead