KZhead

Apache Kafka Crash Course | What is Kafka?

Piyush Garg
2024 ж. 16 Мам.
345 546 Рет қаралды
Жүктеу

In this video, we delve into the fascinating world of Kafka, an open-source distributed event streaming platform. Join us as we explore the fundamental concepts and components of Kafka, including Kafka Topics, Message Partitions, and Consumer Groups.
Throughout the video, we will comprehensively understand Kafka's architecture and its significance in modern data-driven applications. Whether you're a beginner or already familiar with Kafka, there's something valuable for everyone in this discussion.
But that's not all! We believe in learning by doing, so we'll also walk you through a hands-on coding example using Node.js. By the end of this tutorial, you'll have a practical understanding of how to implement Kafka in your Node.js projects.
Quick Links
► Gist and Code: gist.github.com/piyushgarg-de...
► Docker Master Course: learn.piyushgarg.dev/learn/do...
Use Coupon "DOCKER" to get 25% OFF
Timestamps
00:00 Kafka Introduction
00:18 Problem Statement
13:15 How Kafka Solves the Problem
22:30 Architecture of Kafka
37:35 Consumer Groups in Kafka
44:42 Kafka with Node.js
Video Titles
► What is Kafka?
► Why do we need Kafka?
► Kafka Crash Course
► Mastering Kafka
► Kafka Nodejs Tutorial
Hashtags
#Kafka #RealTimeDataStreaming #nodejs #EventStreaming #DistributedSystems #BigData #DataStreaming #Nodejs #CodingTutorial #Developer #SoftwareEngineering #TechnologyExplained #DataProcessing #ScalableApplications #ConsumerGroups #MessagePartitions #DataDrivenApplications #OpenSource #Programming

Пікірлер

  • Great Work. Keep it up. Thanks. Saptadeep.

    @SaptadeepDutta_Ex-Xerox@SaptadeepDutta_Ex-Xerox5 ай бұрын

    • Thank you so much, highly appreciated 😀🙌

      @piyushgargdev@piyushgargdev5 ай бұрын

    • brooo, this is crazy donation

      @SurajSingh-lu8ei@SurajSingh-lu8ei3 ай бұрын

    • Sir send me some money also

      @salman1098@salman10983 ай бұрын

    • ​@@salman1098😂😂😂

      @mohammedaltaf4316@mohammedaltaf43163 ай бұрын

    • Bhai mne do din se khana nhi khaya hai 200 rupye krdo

      @secularguy________@secularguy________2 ай бұрын

  • To sabse pehle ham smjhenge kafka hota kya. Ham dekhenge ki kafka kya hota hai. Ye boht important hai ki ham jaane ki kafka kya hota hai. Mai aapko btaunga ki kafka hota kya hai. We'll see what kafka is. :) Boht ache!

    @jitmanewtyagi565@jitmanewtyagi5659 күн бұрын

  • Great lecture….. in one hour you can easily visualise and learn all about kafka in and out. The best part of the video is when he is coding along with visualising what he is coding.

    @uNIK908@uNIK9083 ай бұрын

  • The way you explained made me to learn this topic *Kafka* which I never heard before I scrolled this random video …. hats off to you for teaching like a friend....👏🙌👏

    @sunilmankoje3331@sunilmankoje33316 ай бұрын

  • Really, you literally took me into the world of kafka. Seriously your explanation is superb.

    @ashokkumar-vi7or@ashokkumar-vi7or6 ай бұрын

  • I tried many youtube courses for kafka but your way of explaning is so great.. please keep the good work..

    @anilmaurya55@anilmaurya556 ай бұрын

  • Thank you so much bro, you really don't know how much help you have made by explaining this concept so easily 🙏🙏

    @sudeepkulkarni7593@sudeepkulkarni75938 ай бұрын

  • I have watched the full video from starting to end and enjoyed it. Its amazing to see how clearly you understand the concept and your command on node syntax 🎉

    @aRookieProgrammer@aRookieProgrammer8 ай бұрын

  • Bro.. this guy is crazyyyyyyyyy. This was way better than other code videos i watched

    @48_subhambanerjee22@48_subhambanerjee225 ай бұрын

  • Your visual way of teaching is too good, thanks mate

    @ashwajitthukral9101@ashwajitthukral91013 ай бұрын

  • What a brilliant explanation! I wasn't familiar with Kafka before, and you explained it in layman's language superbly! Kudos to you. Keep creating videos like this!

    @NilanjanH@NilanjanH2 ай бұрын

  • Lemme try: Kafka is a distributed message queue, generally used to decouple micro services where one api produces some data, and many others consumes it. Simple. I don't know why he confused people with DB and Kafka. Kafka is really a queue, you only read data sequentially. Kafka CAN hold data for an infinite period if you want. But it's not made for 'querying' data, instead, it holds the data temporarily so that the producer api don't need to wait for the consumer api to 'read' this data, it can simply dump the data onto the queue and let consumer api read whenever it finds time for it. If you're even a little confused between Kafka and a DB, this guy is to blame for that.

    @tusharsnn@tusharsnn8 ай бұрын

    • KafkaStreams and ksqldb, built on top of kafka. But yes you are right!

      @NileshAkhade@NileshAkhade8 ай бұрын

    • He is right, he is telling the working of kafka. It's seems that you only uses the technology not understand there working.

      @bhuvneshsaini93@bhuvneshsaini937 ай бұрын

    • When he tried to give definition of throughput, I knew it's time to move to some other video. Bro just keep on repeating zomato uber example multiple times.

      @arvindersingh9588@arvindersingh95886 ай бұрын

    • I assume you would have better videos on KZhead for explaining Kafka, if you are blaming him for confusing others. If not, then appreciate his efforts for teaching for free,

      @BANANAS2011@BANANAS2011Ай бұрын

    • ​@@BANANAS2011 Appreciate efforts who give will give you alcohol when thirsty? Why are you braindead?

      @rahulkumarmaity7497@rahulkumarmaity7497Ай бұрын

  • Great job piyush on explaining the details both visually and with the code approach 💯 While getting the hang of kafka as we go further i started to get the question of how kafka does the message transfer and when you talked about queue and sna architecture approach it clicked the brain. It's the similar how amazon SNS and SQS services works and from that point things got a lot for relatable. I had a chance to work with pusher messaging service which had the similar pub/sub messaging architecture. This kafka session turned out to be really amazing. Looking forward for more of something like this in future.

    @yashsolanki069@yashsolanki0699 ай бұрын

  • I've watched the full video through to the finish and really enjoyed it. Your command of node syntax and your obvious comprehension of the subject are outstanding. 🎉

    @msrajawat298@msrajawat2985 ай бұрын

  • although I do not know node.js but the way you taught kafka its simply awesome.

    @SuvoHossain@SuvoHossain8 ай бұрын

  • What a true sentence "Half knowledge is very dangerous". Good work buddy. Sweetly swallowing the bitter medicine.

    @PranavBhatt_er@PranavBhatt_er8 ай бұрын

  • This is the best kafka course I have watched on youtube

    @user-vq1oq4sl5k@user-vq1oq4sl5k8 ай бұрын

  • Piyush, I am Usman from Pakistan and I really love your videos, The way to tell things from very basic and use drawings for visualizing things and clarity, it's literally just amazing and easy to follow. Thank you so much for such valuable content. Lots of love from Pakistan to you💖

    @huuks527@huuks5278 ай бұрын

  • Kya video banaya hai Bhai Saab..Top notch..1 hour full utilised..

    @deb846@deb8468 ай бұрын

  • i so happy that i found your Channel , I was browsing through your channel and found you have created really good content on the things which are currently trending.Thank you ❤

    @ankitaburman5406@ankitaburman54068 ай бұрын

  • The way of explaining was extremely good, idk why your Channel is so underrated 💥

    @yashbhatnagar1505@yashbhatnagar15059 ай бұрын

  • Was waiting for "Why kafka has such a high throughput" answer. Great explanation, thank you for the effort.

    @roshanchandel7929@roshanchandel7929Ай бұрын

  • Best video on kafka ever. Loved your simple explanation and demo with visualiztion.

    @mquanit@mquanit7 ай бұрын

  • Genuinely one of the best crash course on youtube I have come across in a while Thank you Can you also have further video on such topics like data indexing and sharding

    @tarunsankhla2328@tarunsankhla23284 ай бұрын

  • BEST EXPLAINATION. I'm so sad that I have discovered this channel soo late.

    @jayeshpatil8046@jayeshpatil80465 ай бұрын

  • You have explained it in very simpler way. Thanks to you man ☺

    @mayanksgajjar@mayanksgajjar9 ай бұрын

  • The Way @Piyush is teaching is amazing. I just opened video to over through but ended up watching it completely. I'm working as Sr. Software Engineer and can say that this is bater then many paid videos. Thanks Man.

    @DevMRaza@DevMRaza25 күн бұрын

  • Really awesome vedio , good understanding , Hindi m h toh attention span bhi acha rha ! Great job

    @HimanshuSingh-ek3mu@HimanshuSingh-ek3mu9 ай бұрын

  • Got this channel randomly, you’re a gem ❤

    @HasanAli-vp6ci@HasanAli-vp6ci8 ай бұрын

  • bro, you are the only Indian KZheadr explaining/teaching real world tech.

    @salman0ansari@salman0ansari8 ай бұрын

  • Thank you for the hard work. DB throughput is low because of ACID properties which is not possible in case of Bulk Insert that is without redo/undo so there is a huge risk of data loss but in the cases of Junk data ( Facebook, discord, etc) you accept the risk. In case of financial transactions you need to come up with a strategy where Kafka when full storage and failures in database stops producing until consumer errors are fixed to avoid data loss.

    @SandipBajwa@SandipBajwa8 ай бұрын

  • I always wanted to learn kafka , Thank you so much to putting out such great content. You made me understand every topic where i was struggling Much Thanks 🙏

    @ritikdwivedi5983@ritikdwivedi59834 ай бұрын

  • Amazingly simplified explanation, superb!

    @pavanjain2178@pavanjain21788 ай бұрын

  • This is the first video I discovered in this channel. Maja aa gaya bhai. Thank You!

    @rabindrajoshi9258@rabindrajoshi92587 ай бұрын

  • One of the great crash course video on Kafka. Great work Piyush 🏆

    @MaxDevil1989@MaxDevil19898 ай бұрын

  • HI Piyush, This is indeed a crystal clear video about Apache Kafka. Thanks for sharing this wonderful knowledge for the IT developer fraternity. Thanks for this wonderful video and making the concept so clear.

    @eshantkumargola@eshantkumargola4 ай бұрын

  • Great Work. You have provided the example which will be never forgotten.

    @rupesh.saw04@rupesh.saw04Ай бұрын

  • What a great explanation! Hats off to you, sir. I'm wondering why you have only 11K subscribers. You deserve millions!

    @saqibkhanz8417@saqibkhanz84179 ай бұрын

    • Not many people interested in Kafka know Hindi, and not many people who know Hindi wants to know Kafka.

      @krissh_the_dev@krissh_the_dev8 ай бұрын

    • he is already at 49k in 2months

      @DanielSmith-hd9iq@DanielSmith-hd9iq6 ай бұрын

    • @@DanielSmith-hd9iq 60k when i joined

      @pankajkatiyar2665@pankajkatiyar26655 ай бұрын

    • 68k when I joined .. nice explanation

      @sandeep_khangar@sandeep_khangar5 ай бұрын

    • ​​@@sandeep_khangar72k when I joined

      @Agile691@Agile6914 ай бұрын

  • Perfect explanation. Also Liked the way of presentation starting with real life problem and ending with actual implementation of it.❤❤

    @abhishekashware4781@abhishekashware47816 ай бұрын

  • one of the best you tube channel found on you tube. you are doing great work bro ❤❤

    @sourabhpunase873@sourabhpunase8738 ай бұрын

  • I really learn from your vedios, can you please make vedios on the topics :- 1. design pattern in js 2. React with typescript 3. Typescript standalone course 4. Next js 5. Full stack application build from scratch 6. Asynchronous processing 7. Messages queue 8. Redis 10. Jenkins 11. Advance react js

    @HimanshuSingh-ek3mu@HimanshuSingh-ek3mu9 ай бұрын

    • you don't need a course to learn typescript/react-typescript u just need to use it

      @caresbruh@caresbruh8 ай бұрын

    • For 6th and 7th, I would highly recommend Namaste Javascript playlist of Akshay Saini

      @aayushgupta6914@aayushgupta69146 ай бұрын

  • Wonderful explanation.. Loved the way you explained it minutely..Thanks for the amazing video..!!

    @morningmotivation4745@morningmotivation47456 ай бұрын

  • This is one of the best tutorial on Kafka, Thank you :)

    @vkrisshna45@vkrisshna457 ай бұрын

  • Mazza Agaya, Kaffi Samay Baad Itne Achi Technical Explanation Video Dekha. Yo have a teaching style similar to Brad Traverse.

    @vaibhavshukla9346@vaibhavshukla93468 ай бұрын

  • This is awesome explanation simple , clear and concise with humble tone.

    @AbhayGupta-oz1ix@AbhayGupta-oz1ix3 ай бұрын

  • You explained it very easily Piyush, Keep pushing such content 👍

    @rajanwalia4tech@rajanwalia4tech8 ай бұрын

  • Bro, Just watched 7 mins of your video and I have to come to the comment section to say that the language you are using is too simple and easy to understand. "Database fat Jayega" 😃. Perfect style to explain a core developer like me. Good Job.

    @prashantvishnoi2878@prashantvishnoi28788 ай бұрын

  • It's great and depth knowledge about Kafka by piyush and he is putting (hard +smart)work to spread best knowledge......

    @anuraggupta1008@anuraggupta10089 ай бұрын

    • Yygv

      @ByYouTube2@ByYouTube28 ай бұрын

    • Yygvvvvv

      @ByYouTube2@ByYouTube28 ай бұрын

  • All my Kafka concepts are finally clear today!

    @suchanachakrabarti6783@suchanachakrabarti6783Ай бұрын

  • 😮 the way you explain at 1.75x is so cool. I never knew what Kafka is now I feel like a Kafka expert.

    @nil.4@nil.45 ай бұрын

  • Very Clear and Concise explanation! The way you dumb down complex topics for your audience is remarkable! Keep up the good work :)

    @snehit0123@snehit01232 ай бұрын

  • Great appreciated to you, You are the one of my favourite you tubers. Keeps continue this type of video regularly. Thanks you so much.

    @awesomeshivamshorts@awesomeshivamshorts2 ай бұрын

  • Thank you for the crisp explanation!😊

    @gowrimanohari6682@gowrimanohari668216 күн бұрын

  • crisp explanation. Great way describing each concept

    @KamalSingh-xg6pq@KamalSingh-xg6pq8 ай бұрын

  • EXCELLANT CONTENT, THE BEST EXPLAINATION I HAVE SEEN ON KAFKA.

    @MrDeepakmp@MrDeepakmp7 ай бұрын

  • One of the greatest videos I have ever watched!!!! Thanks Piyush bhai

    @pnig12117@pnig121175 ай бұрын

  • so basic explanation with basic examples which is really easy to understand

    @user-rd9xx9ng7h@user-rd9xx9ng7h6 ай бұрын

  • this channel is a gold mine.

    @chromaxetian496@chromaxetian4969 ай бұрын

  • Crystal Clear. Thank you so much.

    @MuhammadAyaz-gp7xd@MuhammadAyaz-gp7xd29 күн бұрын

  • I have no words this video is amazing please make whole series for kaflka please...........

    @annyd3406@annyd34069 ай бұрын

  • It came randomly to my recommendation and I clicked on it…before I had only heard of Kafka but you explained it so well. I don’t know if I’ll be using this stuff anytime soon but sure as hell won’t forget how it works ❤

    @BABA-oi2cl@BABA-oi2cl9 ай бұрын

    • same

      @krushnnabaviskar2170@krushnnabaviskar21702 ай бұрын

  • Super interesting and one of the simplest and consumable explanation. 👏

    @vshlmlvliveit@vshlmlvliveit9 ай бұрын

  • Great work. Understood everybit. One of the best crash courses. Thanks

    @mohitmotwani9256@mohitmotwani92564 ай бұрын

  • Thanks for this amazing course ❤

    @iamakashkumarram@iamakashkumarram9 ай бұрын

  • Use this whole scenario with a database. Thanks in advance for this soundful knowledge....

    @officialismailshah@officialismailshah8 ай бұрын

  • Layer by layer explaination is finest 👌keep it up 👍

    @vrushalrt@vrushalrt4 ай бұрын

  • great work Piyush. Amazing explanation 👏

    @user-iy4br6hr1v@user-iy4br6hr1v8 ай бұрын

  • Thank you soo much piyush bhaiya ❤

    @pritamkumar6454@pritamkumar64549 ай бұрын

  • apka explaination acha tha bhaiya video ka har ek part samajh aya

    @abhidagreat2914@abhidagreat29143 ай бұрын

  • Thank you very much for a great content and explanations on kafka, you made it very clear, and easy in understanding the topic.🙂❤✌

    @satishb9975@satishb99758 ай бұрын

  • Ky samjhaya hai bhai really impressive! You got a new subscriber.

    @RahulSharma-wh8sq@RahulSharma-wh8sq7 ай бұрын

  • This is really a wonderful explanation. Easy to understand for non-tech person as well. Thanks😀

    @ashunautiyal1@ashunautiyal1Ай бұрын

  • Love this stuff. ❤ Hey Piyush, on Friday i need to give a session on Kafka. Would you please any PPT or documents for this topic. If you have?

    @neutralengineer3873@neutralengineer38739 ай бұрын

  • Please continue providing such valuable information ❤

    @rajukadel1007@rajukadel10076 ай бұрын

  • Random recommendation resulted into Kafka learning. Thank you❣

    @adityak5193@adityak51939 ай бұрын

  • bhai boht acha explain karte ho aap. Ek baar me samajh aa jaata haii.

    @RinkuHasija@RinkuHasija9 ай бұрын

  • Excellent, mind blowing explanation THE BEST you can say

    @user-mg8ol5gp1j@user-mg8ol5gp1j9 ай бұрын

  • Excellent delivery of the concept thanks man!

    @mohammadmaaz7474@mohammadmaaz747419 күн бұрын

  • amazing! best crash course video on kafka.

    @manishkeer8338@manishkeer83387 ай бұрын

  • Superb explanations Piyush. Thank you so much❤

    @AmeyChittar@AmeyChittar5 ай бұрын

  • please also make a video on elastic stack why we need to use ?? what kind of problems it solves?

    @lalitkishork73@lalitkishork739 ай бұрын

    • +1

      @iampiyush100@iampiyush1009 ай бұрын

    • +1

      @Infinite_Mainak@Infinite_Mainak9 ай бұрын

  • Very beautifully explained! Thank you sir for uploading these videos.

    @shishirkumar9014@shishirkumar90142 ай бұрын

  • Bro very well explained thank you so much i don’t usually comment unless it is something extraordinary, keep up the good work cheers!

    @akshaykalghatgi984@akshaykalghatgi984Ай бұрын

  • Thank you. Well explained. I really enjoyed watching it. Very Helpful.

    @bhupendrachauhan4412@bhupendrachauhan44125 ай бұрын

  • In an Iron Man movie, Kafka could have been useful in a scene where Tony Stark (Iron Man) needs to process and manage a large volume of real-time data or communications. Here's a hypothetical scenario where Kafka could play a role: Scene: Tony Stark is in his high-tech lab, and he's remotely controlling his Iron Man suit, which is deployed in a distant location to handle a crisis. He needs to receive and process real-time data from various sensors on the suit, such as vital signs, telemetry data, and external environmental data, while also receiving live video feeds. How Kafka could be useful: 1. **Real-time Data Ingestion**: Kafka could be used to ingest data from these sensors and video feeds in real-time. Each type of data (vital signs, telemetry, video) could be treated as a separate Kafka topic. 2. **Data Processing**: Tony needs to process this data for real-time decision-making. Kafka Streams, a component of Kafka, could be used to perform real-time data processing, such as analyzing vital signs for signs of distress, stabilizing the suit's functions, and identifying threats in the video feed. 3. **Reliability**: In a high-stakes situation like this, Kafka's reliability ensures that no data is lost. If there are network interruptions or delays, Kafka can buffer and replay messages, ensuring that Tony has access to all the critical data. 4. **Scalability**: If the crisis intensifies and more data needs to be processed, Kafka can scale horizontally by adding more Kafka brokers, allowing Tony to handle the increased data flow without performance issues. 5. **Monitoring**: Kafka provides extensive monitoring capabilities, which could be depicted in the movie as Tony monitoring the health of the data pipeline in real-time, ensuring that he has a clear view of the suit's status. In this scenario, Kafka would enable Tony Stark to efficiently manage and respond to real-time data, enhancing his ability to control the Iron Man suit and handle the crisis effectively. It would add a layer of realism to the technological aspects of the movie.

    @danielmarkj@danielmarkj8 ай бұрын

  • Thank you I am learning spring boot this is very helpful.

    @prashlovessamosa@prashlovessamosa9 ай бұрын

  • very easy to understand explanation, Thank you ! keep it up !

    @kryptu@kryptu7 ай бұрын

  • you are best brother , keep going and yeah I would also love to see video on elastic search

    @lanslord11@lanslord119 ай бұрын

  • Great explanation brother. Thank you a lot!!!

    @ravikanhreddymanik7767@ravikanhreddymanik77678 ай бұрын

  • Great watch just before an interview!

    @nikhil182@nikhil1829 ай бұрын

  • Bro, you have a special talent to make difficult topics easy, it was my first kafka tutorial and i understand it well enough, thanks a lot ❤❤

    @AmarjeetKumar-wt1hz@AmarjeetKumar-wt1hz9 ай бұрын

    • agree

      @mdalishanali3909@mdalishanali39099 ай бұрын

    • Totally agree ❤❤😥😥

      @sheikhfajlayrabbi6070@sheikhfajlayrabbi60706 ай бұрын

  • It was a very good explanation of Kafka, I loved it. I am goind to implement it to my blogging website that I created.

    @adityaruhela5761@adityaruhela57619 ай бұрын

  • bhai, kya explanation dete ho yaar, maza aagaya

    @coderwork8335@coderwork83357 ай бұрын

  • Thank you so much for the great explanation!

    @ahmarajeel@ahmarajeel4 ай бұрын

  • Amazing crash course!! Loved it!!

    @AYUSHSINGH-py1pp@AYUSHSINGH-py1pp4 ай бұрын

  • This was a great video! thanks for sharing.

    @bilalahmedkhan5876@bilalahmedkhan58768 ай бұрын

  • Great video bhaiya Please continue such amazing content

    @AmanTiwari-ww7td@AmanTiwari-ww7td9 ай бұрын

  • What a great Explaination, Please make more videos like this

    @codeinfotainment9085@codeinfotainment90856 ай бұрын

  • Thank you so much, very well explained.

    @krushnnabaviskar2170@krushnnabaviskar21702 ай бұрын

  • Very nicely explained !! Wonderful work mate 🙂...... you made it so easy to understand such a complex topic and that too in minutes.

    @chintanthakkar1732@chintanthakkar1732Ай бұрын
KZhead