Reinforcement Learning: Machine Learning Meets Control Theory

2024 ж. 25 Мам.
251 456 Рет қаралды

Reinforcement learning is a powerful technique at the intersection of machine learning and control theory, and it is inspired by how biological systems learn to interact with their environment. In this video, we provide a high level overview of reinforcement learning, along with leading algorithms and impressive applications.
Citable link for this video: doi.org/10.52843/cassyni.x2t0sp
@eigensteve on Twitter
eigensteve.com
databookuw.com
This video was produced at the University of Washington
%%% CHAPTERS %%%
0:00 Introduction
3:34 Reinforcement Learning Overview
7:30 Mathematics of Reinforcement Learning
12:32 Markov Decision Process
13:33 Credit Assignment Problem
15:38 Optimization Techniques for RL
18:54 Examples of Reinforcement Learning
21:50 Q-Learning
23:53 Hindsight Replay

Пікірлер
  • Steve is a phenomenal lecturer, isn't he?

    @ronniec8805@ronniec88053 жыл бұрын
    • never seen a better one

      @cnbrksnr@cnbrksnr3 жыл бұрын
    • very much so

      @uforskammet@uforskammet3 жыл бұрын
    • He is!

      @flybekvc@flybekvc3 жыл бұрын
    • Yessss

      @mail2cmin@mail2cmin3 жыл бұрын
    • no, he is the most phenomenal one!! Respect

      @reocam8918@reocam89183 жыл бұрын
  • *"WELCOME BACK"*

    @whasuklee@whasuklee3 жыл бұрын
  • Viewing reinforcement learning as time delayed supervised learning is a really good way of looking at it.

    @ethanspastlivestreams@ethanspastlivestreams3 жыл бұрын
    • Indeed!

      @JousefM@JousefM3 жыл бұрын
  • I wish i knew this channel at the start of quarantine

    @steven-bt7ud@steven-bt7ud3 жыл бұрын
    • I found about the channel just as quarantine had started. It was quite the treat.

      @hysenndregjoni853@hysenndregjoni8533 жыл бұрын
  • Just wanted to comment about how much I love these videos. Last year while applying for PhDs I was searching for passions. In a discussion with my friend (a computer scientist), I accidentally outlined genetic programming without knowing it. My friend told me so and I went researching. Found these videos and became enthralled. Now I have a PhD studentship in soft robotics and plan to use SINDy to help with modelling and control and honestly think that giving machines brains may be my future work too. Thanks Brunton, my passion was helped by your own.

    @Headbangnuker@Headbangnuker3 жыл бұрын
    • That is amazing to hear! Helping people develop their passions is exactly why I do this!

      @Eigensteve@Eigensteve3 жыл бұрын
  • This is THE BEST explanation on reinforcement learning over all the articles, books, or youtube videos, that I've seen so far. Period.

    @s3pi0n@s3pi0n2 жыл бұрын
  • I still have no idea as to who could possibly dislike these videos

    @christiankraghjespersen994@christiankraghjespersen9943 жыл бұрын
    • u

      @phaZZi6461@phaZZi64613 жыл бұрын
    • @@phaZZi6461 I wanted to add a comment but 69 looks so good \

      @devashishbose1521@devashishbose15213 жыл бұрын
  • Would love for a full series on how can we use RL to control real world dynamical systems!

    @sankalp1391@sankalp13913 жыл бұрын
  • I just cannot express how grateful I am to prof Steve Brunton for posting these videos. Waking up at 6am to watch him explain is the most satisfying thing ever. Thank you! We all are grateful.

    @baseljamal8907@baseljamal89073 жыл бұрын
  • Is there something you dont know dude? You seem to be an expert on everything. You are such an inspiration.

    @elultimopujilense@elultimopujilense3 жыл бұрын
  • Great channel! Please record more videos on the edge of reinforcement learning and control theory. Congrats on your work.

    @sistemasecontroles@sistemasecontroles3 жыл бұрын
  • Wow! I would love to see Prof take on RL topics!

    @wizardOfRobots@wizardOfRobots3 жыл бұрын
  • This sweet spot between control theory and machine learning definitely interests me, especially applied to astrodynamical systems. Please, continue making these videos, Professor Brunton!

    @thiagocesarlousadamarsola3990@thiagocesarlousadamarsola39903 жыл бұрын
  • The lecture was very well constructed. Well done! As an electrical engineering student trying to specialize in ML I find that you really hit the mark when it comes to putting these though and convoluted topics together with examples.

    @AW_tuber@AW_tuber2 жыл бұрын
  • I am binge watching this chanel from past 3 hours

    @souravjha2146@souravjha21463 жыл бұрын
  • Every time he said "good", i felt appreciated for not giving up on a lecture whose subject is far, far away from mine and im pushing myself to try and learn the concept. thank you, steve.. much love.!

    @subhikshaniyer613@subhikshaniyer613 Жыл бұрын
  • Never clicked a video that fast 😆. Great content prof as always love it!

    @LuisGonzalez-gz1qg@LuisGonzalez-gz1qg3 жыл бұрын
  • I love how you emphasize the intersection between machine learning and control (theory). That's exactely what sparks my interest about reinforcement learning!

    @Spiegeldondi@Spiegeldondi3 жыл бұрын
    • Glad you like it! I always found this connection fascinating and a very natural way to merge the two fields.

      @Eigensteve@Eigensteve3 жыл бұрын
  • Looks like I'm not the only one working on a video early in the morning! Really cool stuff, love the doggie!!

    @fzigunov@fzigunov3 жыл бұрын
  • I am doing research on the Model-based RL for safety-critical systems; I really enjoy doing it. These are so cool. Thanks for making videos on this topic!

    @nahidmahmud8234@nahidmahmud82343 жыл бұрын
  • Mr.Brunton saves me from my final review. His lectures made crystal clear those seemingly unfathomable terms. Just watched him videos for days and I already like him!

    @spencerhong4687@spencerhong46872 жыл бұрын
    • those bipedals are too cute they deserve another cmt

      @spencerhong4687@spencerhong46872 жыл бұрын
  • That's an awesome video indeed. A great introduction to RL!

    @msauditech@msauditech4 ай бұрын
  • Simply great subject and excellent presentation thank you prof for all your efforts

    @kouider76@kouider763 жыл бұрын
  • Dear Steve, Im very, very grateful that I get to watch such extraordinary instructive videos for free!!! Thinking that elsewhere in the world people are killing others atm (as in Kabul), it gives me a lot of hope seeing how people like you just make the world a little better and allmost brings tears into my eyes. You have such great talent in teaching, thank you!

    @carlphilip4393@carlphilip43932 жыл бұрын
  • I just found out your channel and the contents you cover is a treasure to me!

    @ghostofhacker2818@ghostofhacker28183 жыл бұрын
  • I really love your content, please keep spoiling us! These were the fastest 26 minutes! I learnt a lot and I'm looking forward to the python lab implementations of these concepts! Thank you very much for your work.

    @diegoguisasola3858@diegoguisasola38583 жыл бұрын
  • Hi Professor Steve, Lovely presentation.

    @SRIMANTASANTRA@SRIMANTASANTRA3 жыл бұрын
  • Theeeere we go Steve! Waited for this :)

    @JousefM@JousefM3 жыл бұрын
  • Hi, Steve. I've been working on Fluid Mechanics 25 years or so. Always using experimental and some analytical tools to approach the subject. I had a lot colleagues migrating to CFD back in the 2000s because these methods seem to find valid results with "little" effort in comparison to expensive, frustrating and time-consuming experiments. So I always disregarded CFD as nice tool that could predict a lot of stuff that you will never know if it is correct or not. However, I have to say, that from some time, reinforced (see what I did there?) by new material that I am studying and your papers on ML for Fluid Mechanics I am looking at the subject with new eyes. Thank you very much for your material and the dedication you put in every video.

    @theclassoftorchia3856@theclassoftorchia3856 Жыл бұрын
  • Professor you're awesome. My thesis topic is, deep reinforcement learning based robotic arm torque control. I love control theory and machine learning. Thx for your support.

    @RasitEvduzen@RasitEvduzen3 жыл бұрын
  • top quality this is what they said about education on the internet that "the best teacher can teach everyone" this is that video for this topic

    @loopuleasa@loopuleasa3 жыл бұрын
  • It's really interesting to watch this video, although I have also studied and read it a few times, its boredom is hard to describe. thank you teacher

    @tai94bn@tai94bn Жыл бұрын
  • Great video! If everyone was as great on KZhead as your delivery we would have a lot more passion in the area. Keep up the good work, train on!

    @Voke@Voke3 жыл бұрын
  • Awesome lecture! Thanks Steve. I really enjoyed watching this!

    @terryliu3635@terryliu3635Ай бұрын
  • Thank you so much for this lecture. I really enjoy your videos, this is helpful as a PhD student. I also bought your book "Data-driven science and engineering" which have nice explanations for the tools I use. Keep on this awesome work! Greetings from France!

    @pierredubois8715@pierredubois87153 жыл бұрын
  • Waiting this topic from long time, your lectures are so clear. Thanks lot.

    @sachinr3823@sachinr38233 жыл бұрын
  • As a CS grad student who took RL in the last semester ... this is truly the best refresher I have seen until now. Thanks a lot for uploading.

    @teegnas@teegnas3 жыл бұрын
    • Great to hear!

      @Eigensteve@Eigensteve3 жыл бұрын
  • you have created such high quality content that i just really enjoy watching it instead of playing games :)))

    @cuongnguyentranhuu4616@cuongnguyentranhuu46163 жыл бұрын
  • Explained in an understandable way and RL nicely connected to control theory!

    @alistja4337@alistja43373 жыл бұрын
  • Very well illustrated! Thanks

    @chrisogonas@chrisogonas2 жыл бұрын
  • Your series are excellent . They have a good pace and use powerful graphics to explain difficult concepts. I've watched many of your videos on my TV which doesn't allow me to give a thumbs up. See here it is. I am not a Python programmer , but I am sure that those watching who DO use Python must have itchy fingers.

    @pellythirteen5654@pellythirteen56542 жыл бұрын
  • I’ve been following your content for a at least 4 years now! It’s the reason I am a robotics control engineer now, you pulled me through 4th year control systems with your conveniently-timed boot camp. Please keep up the great content! PS are you accepting PhD students?

    @marco_gallone@marco_gallone3 жыл бұрын
  • That's so interesting and well explained. Thank you !

    @Kong9901@Kong99013 жыл бұрын
    • Glad you liked it!

      @Eigensteve@Eigensteve3 жыл бұрын
  • Thank you, professor Steve Brunton. I am pleased to inform you that I am considering to do, after my master degree in computer engineering, a PhD related to the Data Driven Control Theory subject and the merit in part is also your.

    @francesco884@francesco8842 жыл бұрын
  • What I like is I don't pay for this knowledge. I was planning to take a data science certificate, but you know what. Let me spend 6 months learning by myself I have spent a solid 1 month only on your videos starting from SVD. it has been amazing. I love when a small thing builds up into a bigger thing. Soon I will make a sample project based on what I have learned from your video.

    @pradiptahafid@pradiptahafid Жыл бұрын
  • I was waiting for this!!!

    @melvinlara6151@melvinlara61513 жыл бұрын
  • Keep up the good work, love your videos

    @leopardus4712@leopardus47123 жыл бұрын
  • The essence of this content is profoundly influential. A book with akin messages was transformative. "Game Theory and the Pursuit of Algorithmic Fairness" by Jack Frostwell

    @JoeM370@JoeM3705 ай бұрын
  • How very mean, I was looking forward to see the trial 7 right away. Great explaining. Thanks

    @merv893@merv893 Жыл бұрын
  • phenomenal video, thank you

    @riccardodelpozzo8683@riccardodelpozzo86832 жыл бұрын
  • After watching these videos I have actually understood the concept of reinforcement learning. I might be wrong but to me it seems it generalizes the feedback loops into more abstract concepts of agent action policy environment etc. In a feedback loop we have control policy which is a PID controller that controls the behaviour of a plant it is attached to. The model of the plant is environment here and the action is taken by the output of the PID controller. The reward in feedback loop is to converge to desired output value at the steady state by ignoring its transition time values, so it is in a sense a semi supervised learning. The states in the feedback loop is derivative components of the system. In noisy systems, sometimes it is crucial to remove derivative component to avoid impulsive behaviour which corresponds to state feedback from environment to agent in RL. By thinking like this, RL is more meaningful to me as an engineer, that RL is a generalized feedback system where we try to get a desired output given some input to the system. Thank you for these video series!!

    @tytuer@tytuer2 жыл бұрын
  • Steven lectures are great help to the society ❤

    @givemeArupee@givemeArupee4 ай бұрын
  • You have a nice way of explaining the topics.

    @dr.-ing.shehzadhasan3387@dr.-ing.shehzadhasan33873 жыл бұрын
    • Thanks!

      @Eigensteve@Eigensteve3 жыл бұрын
  • it's brilliant ! . Keep working with this topic please

    @timanb2491@timanb24912 жыл бұрын
  • Steve is one of the gifted teachers. I wish you can guide postgraduate to make a good publication in control and learning by highlighting the hot topics and promising research aspects.

    @hudhuduot@hudhuduot3 жыл бұрын
    • Thanks so much!

      @Eigensteve@Eigensteve3 жыл бұрын
  • the fantastic lecture that I've ever seen...

    @sounghwanhwang5422@sounghwanhwang54222 жыл бұрын
  • Perfect video, I will watch all other at one time 😍

    @mertbozkir@mertbozkir2 жыл бұрын
  • RL can be interpreted from this perspective, amazing

    @minglee5164@minglee51642 жыл бұрын
  • I've been seriously considering starting a degree in A.I./Machine learning but with videos of this quality available for free, it is hard to justify the cost. Subscribed and liked!

    @thelazygardener9493@thelazygardener94933 жыл бұрын
    • Just incase you read this and have time to reply... Do you have any suggestions for an education path to your level of understanding? There are degrees for data science, computer science, artificial intelligence, software engineering, etc. They all seem so inter-related. I want to know them all but I'm struggling to pick a starting point. My current level of related education is highschool level advanced maths and a year of teaching myself MQL4/5 and R code mostly from free resources online. Just so you know my starting point (or state haha).

      @thelazygardener9493@thelazygardener94933 жыл бұрын
  • Great lesson.. Thank you

    @pardonchawatama941@pardonchawatama9412 жыл бұрын
  • Dude, you are the best lecturer. DONE

    @focusonlife3242@focusonlife32422 жыл бұрын
  • This is really really great teaching.

    @TheProblembaer2@TheProblembaer25 ай бұрын
  • Amazing Clarity

    @arrahul316@arrahul316 Жыл бұрын
  • Yay! Hero has decided to teach Reinforcement Learning

    @subramaniannk3364@subramaniannk33643 жыл бұрын
  • Excellent video ❤️

    @GenAIWithNandakishor@GenAIWithNandakishor3 жыл бұрын
  • Great video, thanks.

    @matejsuty5024@matejsuty50242 жыл бұрын
  • love your lectures

    @TheRestalyn@TheRestalyn Жыл бұрын
  • A very well done lecture. Bravo! I'd like to make a suggestion, if I may, to modify the Policy function as pi(s,a) = Pr(A = a, S = s); A is the place holder for an action, and a is the actions of taking; S is the place holder for the state and s is the given state.

    @lazyoneswapples2962@lazyoneswapples2962 Жыл бұрын
  • All of your lecture series are very good and very helpful. A series on convex optimization problems would be good. Any thoughts about it?

    @SiriGadipudi@SiriGadipudi Жыл бұрын
  • Kudos on the awesome lecture

    @pliniocastro1546@pliniocastro15463 жыл бұрын
  • I really like your videos. Keep up the good work! :)

    @radhen171992@radhen1719923 жыл бұрын
  • Hi Steve. I am an amateur mathematician (hoping to go pro) who is really into category theory. Have you or your team ever looked at this? Usually, when you see two subjects talking about the same thing, it's a good bet that category theory is working in the background. And I just looked at category theorist Tai-Danae Bradley and her explanation of SVD in terms of category theory. Thanks! AWESOME CHANNEL!

    @MichaelEvans-yq7xj@MichaelEvans-yq7xj3 жыл бұрын
  • I thought I was witnessing a breakthrough concept trying to link deterministic control theory with machine learning. But, when you mentioned the words probability and policy, I was disappointed. Looking forward to more conceptual lectures. Could also highlight real world applications. Thanks.

    @ramanikrishnamurthy7086@ramanikrishnamurthy70863 жыл бұрын
  • nice lecture sir! thanks a lot!

    @5621346@56213463 жыл бұрын
  • I really like your explanation

    @HD-qq3bn@HD-qq3bn3 жыл бұрын
  • Thank you, professor!

    @aiahmed608@aiahmed6082 жыл бұрын
  • thank you very much for your lesson, it is really useful to me!

    @Lucas_Lima606@Lucas_Lima6062 жыл бұрын
  • Amazing feeling to watch a video After completing a project on the same topic

    @aniruddhadatta925@aniruddhadatta9253 жыл бұрын
  • we also look forward to your explanation for GAN in the future

    @HD-qq3bn@HD-qq3bn3 жыл бұрын
  • I love u, Steve! I have been currently working on Machine Teaching and Project Bonsai. I really needed to know this.

    @tabindahayat3492@tabindahayat34922 жыл бұрын
  • amazing work as usual !! ... could you please consider doing a lecture about whole body control for robotics?

    @hazemahmed8333@hazemahmed83333 жыл бұрын
  • the best one i have ever seen

    @fahimehjabbarinia401@fahimehjabbarinia401 Жыл бұрын
  • Glad this content is on KZhead -- the past year kind of derailed me going to grad school. Question, could the reward structure of a chess game be broken into incremental steps? As in, the main reward is to win, but couldn't a game be discretized into incremental rewards defined by the value of a target and the probability that a sequence of moves would capture a high value target? Or, is that just Q-Learning in different words?

    @TomOtero1984@TomOtero19843 жыл бұрын
  • I think it's also important to mention the distinction between discrete and continuous action spaces.

    @Turcian@Turcian3 жыл бұрын
  • i have been trying to teach my guys that machine learning and control theory (fuzzy autotuning) is the same principle. This video will be used!

    @car0lm1k3@car0lm1k33 жыл бұрын
  • hey Steve, love your videos! Wondering if the videos in this playlist are in the correct order?

    @elsabassi2533@elsabassi2533 Жыл бұрын
  • I Like this video, but there is already a very common way to optimise the density problem. It’s called actor critic, where you basically get a second network to learn what reward it is expecting (q-learning) where the actor is a policy gradient network. Works fine so far and I know it’s not enough to get away from the semi-supervised but let’s be honest, the “semi” is what really defines the technique. Because the agent needs to learn what “could” be good in future by itself without a supervisor. That’s how animals and humans learn too. so a fully supervised agent wouldn’t be exploring the world on its own anymore. Greetings Firestorm (ETH-Zurich Student)

    @Firestorm-tq7fy@Firestorm-tq7fy2 жыл бұрын
  • Thanks Steve.

    @praharaj2007@praharaj20073 жыл бұрын
    • You are very welcome!

      @Eigensteve@Eigensteve3 жыл бұрын
  • I wish my teachers had seen your videos before trying to teach us these subjects :)

    @be_milad@be_milad Жыл бұрын
  • just a question, i am viewing the Control boot camp playlist, so it goes from control dynamic systems with non minimum phase to control theory and convid 19 to reinforcement learning. is this the correct way of viewing the videos? i feel there was for to talk about on the previous topics maybe i am wrong. Thanks to you/your team for all your amazing videos. i finished my controls classes 3 years ago when i found ur videos, i have been going throught all your playlists and am loving it! Is there a specific job that requires/teaches these skills? the closest job to this i found were graduate automation engineer .

    @ajj7794@ajj77942 жыл бұрын
  • Prof Brunton: You are one bad-ass teacher!!!🤓

    @carriefu458@carriefu4582 жыл бұрын
  • Prof. Steve. Please make videos on Immersed Boundary Methods and how PINNs can help to solve complex problems in this area.

    @shuvranilsanyal445@shuvranilsanyal4452 жыл бұрын
  • Legendary!!!!

    @HassanKhan-cs8ho@HassanKhan-cs8ho3 жыл бұрын
  • Love the videos

    @youtubeenjoyer199@youtubeenjoyer199 Жыл бұрын
  • Thank you!

    @Anon_life@Anon_life3 жыл бұрын
  • Thank you ❤

    @carolineseffen7458@carolineseffen74582 ай бұрын
  • Hey Steve! Loved your lecture! Could you tell me what your setup is? I love your production, setup, and content of course! Some questions: 1. Do you have a screen/script in front of you and a green screen behind? 2. Which cam and mic do you use? Is it only a lav mic? I assume it's not shotgun since you're far away from any particular point of the frame. 3. How much time does it take to create a video like this one? 4. How many dry runs do you usually do? Or for this video in particular? You're setting a new standard for production (and beyond haha), keep up the good work! I'd really appreciate your answers, thank you in advance!

    @TheAIEpiphany@TheAIEpiphany3 жыл бұрын
    • Thanks, glad you like it! No script, but I have a screen so I can see where I am relative to the presentation. I use a lav mic and a canon 4k camera. I usually do everything in one run, sometimes I redo the intro a couple times until i'm happy with it.

      @Eigensteve@Eigensteve3 жыл бұрын
    • @@Eigensteve thanks Steve!

      @TheAIEpiphany@TheAIEpiphany3 жыл бұрын
  • Wow! Thank you so much. Maybe the next lecture can be about UMAP please :D?

    @gama3181@gama31813 жыл бұрын
  • Amazing 🤩

    @usmleck7000@usmleck70003 жыл бұрын
  • It would be an honor to be supervised for a PhD by him.

    @Physicsandmathswithpraveen@Physicsandmathswithpraveen3 жыл бұрын
KZhead