Complete Python Pandas Data Science Tutorial! (Reading CSV/Excel files, Sorting, Filtering, Groupby)
Practice your Python Pandas data science skills with problems on StrataScratch!
stratascratch.com/?via=keith
Data & code used in this Tutorial: github.com/KeithGalli/pandas
Python Pandas Documentation: pandas.pydata.org/pandas-docs/...
Let me know if you have any questions!
In this video we walk through many of the fundamental concepts to use the Python Pandas Data Science Library. We start off by installing pandas and loading in an example csv. We then look at different ways to read the data. Read a column, rows, specific cell, etc. Also ways to read data based on conditioning. We then move into some more advanced ways to sort & filter data. We look at making conditional changes to our data. We also start doing aggregate stats using the groupby function. We finished the video talking about how you would work with a very large dataset (many gigabytes)
I realized as I upload this video there are some additional things I want to talk about in a later video. The first thing that comes to mind immediately is using the apply() function on a dataframe to alter the data using a custom or lambda function. If you have questions on this or anything else before I get around to making a part 2, feel free to write me a note in the comments.
If you enjoyed this video, be sure to throw it a like and make sure to subscribe to not miss any future videos!
Thanks for watching friends! Happy coding! :)
Join the Python Army to get access to perks!
KZhead - / @keithgalli
Patreon - / keithgalli
---------------------------------------------
Follow me on social media!
Instagram | / keithgalli
Twitter | / keithgalli
---------------------------------------------
Link to original source of data from Kaggle: www.kaggle.com/abcsds/pokemon
---------------------------------------------
Video Outline!
0:00 - Why Pandas?
1:46 - Installing Pandas
2:03 - Getting the data used in this video
3:50 - Loading the data into Pandas (CSVs, Excel, TXTs, etc.)
8:49 - Reading Data (Getting Rows, Columns, Cells, Headers, etc.)
13:10 - Iterate through each Row
14:11 - Getting rows based on a specific condition
15:47 - High Level description of your data (min, max, mean, std dev, etc.)
16:24 - Sorting Values (Alphabetically, Numerically)
18:19 - Making Changes to the DataFrame
18:56 - Adding a column
21:22 - Deleting a column
22:14 - Summing Multiple Columns to Create new Column.
24:14 - Rearranging columns
28:06 - Saving our Data (CSV, Excel, TXT, etc.)
31:47 - Filtering Data (based on multiple conditions)
35:40 - Reset Index
37:41 - Regex Filtering (filter based on textual patterns)
43:08 - Conditional Changes
47:57 - Aggregate Statistics using Groupby (Sum, Mean, Counting)
54:53 - Working with large amounts of data (setting chunksize)
-------------------------
If you are curious to learn how I make my tutorials, check out this video: • How to Make a High Qua...
*I use affiliate links on the products that I recommend. I may earn a purchase commission or a referral bonus from the usage of these links.
Hey ya'll! I created a second channel with more Python content (including additional Pandas tips & tricks). Please consider subscribing 😊 kzhead.info
You cleverly edited the code between 25:50 to 25:59 list(df.columns.values) to list(df.columns)😉👍
Error:Cannot mask with non-boolean array containing NA / NaN values - gives me error when usinf df.loc (on 40:49 in video)? df.loc[df['Our Global Company'].str.contains('Smith', regex=True)]: this is code, I imported another .xlsx table when practising.
@@Vribejs go google it... you can't expect him to do it for you. He checked the documentation just to give us a good overview of pandas.... google out your error if not you will not learn.
i have been working on a excelworkbook having 8 worksheet and i m performing operations on data nd want to place dataframe in the 6 sheet in place of its data .but everytime i do all other sheets gets vanished nd a single gets get formed with the dataframe .plzz help me in appending df into an existing excel
Hey Keith , can can please help me to download the csv.file on an android tablet. sorry for bad english.
I know this is 5 years old but I learned more about using Pandas from this one video than all the other videos ive watched on the topic combined! Just awesome! Thank you!
Glad that it is still helpful!!
This video was super helpful, thank you Keith! In case anyone gets to the end of this video, around 48:00, Keith talks about the groupby operator and starts to go over the section "Aggregate Statistic using Groupby (Sum, Mean Counting)". You might run into errors due something that changed after Pandas version 2.0.0. Instead of writing: df.groupby(["Type 1"]).mean() Try writing: df.groupby(["Type 1"]).mean(numeric_only=True) After version 2.0.0 the numeric_only value was changed to False versus True as it's default, causing errors such as "can not convert strings". Hope this is helpful, have a good one!
Thank you very much, I ran into the problem, this is really helpful! :)
Thank you
Thanks Man .
Thanks Dude
Was facing the same issue, thanks a lot.
This 1 hour video did more for me than entire semester of my Data Analysis course... Amazing
SAME DUDE omg
me too!
Same here bro
Were you paying attention?
am a self taught but this one saved me
A strugling biologist here thanks you! We are mostly dealing with big data and it can get a little overwhelming, but you made it a lot easier!
Awesome
big data in a csv file? lol
Hey, you might enjoy SAS
they dont lie when they say data is everywhere and every field needs data scientists
Best Pandas tutorial on KZhead, especially 24:25
hahahahaa
LOL
jaajajajaaj
Huge hhhhh
29:19 is where he's getting texts from actual pandas
Mannnn your one of the best Python go-tos PERIOD. Straight to the point and easy to understand. thanks for teaching us all!
Hi Keith - not sure you will read this but wanted to sincerely thank you for this tutorial. 3 years ago this was the first python video I ever watched after graduating from unrelated subject. Today I'm typing this from a business class lounge at JFK, on my way to London where I just got a job as a quant developer at a hedge fund, building pricing models and infra for trading. Worked hard for this but if not for your videos I could be at a very different place. Thank you from the bottom of my heart, your work means a lot to many people. Cheers!
Bro hire me
can you tell us more about your journey? :)
Keith You are more than a teacher. Your level of simplicity in explaining Python in details is out of the moon. Keep up the good work. Your video is always my “go to” any time. Again, thanks a lot for using your skills as a blessing to people around the world.
One of the best tutorial that I've ever seen in KZhead! Thumbs UP!
Excellent!! I like the way you organize the videos on different topics and functions of working with data. Please make more videos on how to work data science in Python. E.g. Statistical analysis (descriptive statistics, t-test, linear regression) or data processing tutorial (like what we do in SQL).
Wasted an hour watching a completely useless video on pandas, didn't understand a thing...... Then found this pure gold of a video, it really helped me a lot. Why didn't I click it earlier............
lol you had me in the first half 😂
glad it helped!
@@KeithGalli yeah, really nice job explaining it Currently watching the other pandas video (real life problems)
Great video! One of the best pandas tutorials I've seen. I have one comment though. When you run (at 40:00) df.loc[df['Name'].str.contains('Mega')]) You are actually including Meganium in this filter, even though it is not a Mega pokemon. So, one needs to include a space after Mega, such as: df.loc[df['Name'].str.contains('Mega ')]) One can see that this makes a difference because when you run len(df.loc[df['Name'].str.contains('Mega')])) and len(df.loc[df['Name'].str.contains('Mega ')])), to know the number of rows, there are two distinct outputs (respectively 49 and 48)
this is an excellent tutorial, especially the filtering/conditional changes section. I have always loved how google sheets has built in queries, and I wanted to be able to do a lot of the same things using pandas. This essentially gave me all of the power I needed! thanks!
Awesome tutorial! One advice I'd have for any python developers is to get in practice of working within virtual environments. Really helps to avoid conflicts when you're working on a project which may require some older versions of a library but your other projects may require latest ones, stuff like that.
I have bought multiple Udemy courses on pandas and this one blows them all out of the water, and it’s free! I’m deff subbing!
I haven't started this yet, but based on your previous videos I know this is going to be great. Thanks Keith, you are a great teacher.
Video Outline! 0:45 - Why Pandas? 1:46 - Installing Pandas 2:03 - Getting the data used in this video 3:50 - Loading the data into Pandas (CSVs, Excel, TXTs, etc.) 8:49 - Reading Data (Getting Rows, Columns, Cells, Headers, etc.) 13:10 - Iterate through each Row 14:11 - Getting rows based on a specific condition 15:47 - High Level description of your data (min, max, mean, std dev, etc.) 16:24 - Sorting Values (Alphabetically, Numerically) 18:19 - Making Changes to the DataFrame 18:56 - Adding a column 21:22 - Deleting a column 22:14 - Summing Multiple Columns to Create new Column. 24:14 - Rearranging columns 28:06 - Saving our Data (CSV, Excel, TXT, etc.) 31:47 - Filtering Data (based on multiple conditions) 35:40 - Reset Index 37:41 - Regex Filtering (filter based on textual patterns) 43:08 - Conditional Changes 47:57 - Aggregate Statistics using Groupby (Sum, Mean, Counting) 54:53 - Working with large amounts of data (setting chunksize) Thanks for watching friends! :) Let me know if you have any questions
YES!!! THANK YOU!
Thank you so much for posting this! I have a test in Python soon, so I've been watching this for a review. You explain everything so well and make it easy to follow. I also like how the data was from Pokémon - it makes it more relatable.
great tutorial
A reference notes to help you while you watch the video. docs.google.com/document/d/16qcfjwLp1vV-5VnIOGuDC2vxkHQ534_RzQd2Gihk7x8/edit?usp=sharing
Hey dude, love this video by the way but I have a question, can this data be used for machine learning? I have my exams coming up where I have to find a dataset to make predictions and stuff. Are these pokemon cards, do they have label and features if you understand what i'm talking about? Any help would be greatly appreaciated. Thanks in advance.
This video helped my massively! Been learning through online python courses with people trying to act and saying unnatural jokes, but your video felt super natural and easy to watch. Many thanks!
There is something to the way Keith teaches that keeps me coming back. Besides being a good teacher and utilizing techniques which help people grasp the material quickly and remember for long time, he sends forth a wave of positivism. He is such a positive, energetic person. Thanks for sharing your knowledge. May it grow and enable you to bless more people with it.
I watched the entire video in 30 minutes and learned more than I did with hours of video content. Amazing work.
I can't believe I watched this for free, thank you so much!
This was pretty good. I would also check udemy or r/learnpython for other free resources. Found a 30 hour FREE pandas course there the other day
www.udemy.com/course/the-ultimate-pandas-bootcamp-advanced-python-data-analysis/?couponCode=FF041817B54B4BC9EB6B
@@johnwiley1221 It's not free now, unfortunately :(
ki
The documentation is also free.
Dude you deserved all the subs for this video alone. You explained everything so good. keep it up :)
Started my PhD in hydrogeology and learning Python from the scratch. I love your work, keep it up!
53:30 you can use .size() to get the count of each Pokemon type instead of adding a new column. It would look like this: df.groupby(['Type 1']).size() Great tutorial!!
Coming from the R environment, I must say this is an excellent tutorial to learn about Pandas. I'm very happy to learn that the tools I use in R for data management can be implemented in a similar way in Python. Thanks for taking the time to put this together! Great job.
Same here
I agree - coming to Python from RStudio and after looking at videos all day this is definitely the most helpful and intuitive video!
sometimes the syntax may be getting confused for python and r right? if you use both
can someone tell me why is r so encouraged in the data science/analysis circle when python can do everything and more and it is so intuitive
@@manan-543 I think Python is far more general and overall can do a lot more, but in my field, packages associated with statistical models are far more abundant in R than in Python. For example, I'm not sure Python comes even close to R for the implementation of Bayesian hierarchical models, GLMMs, GAMMs, etc. Also, methods papers often publish packages in R, so it seems to remain the default for statistics. Until the statisticians start switching in large numbers I'm not sure this is gonna change anytime soon; and when it does, it probably will be Julia, not Python.
Between jobs for the first time in decades I wanted to learn data science using software other than just Excel and Access. Your video was well explained and frankly better than anything else I have seen so far involving Python and Pandas. Thank you for a job well done.
Great video to get people up and running. It took me two hours to watch, take notes, and test out some examples. I feel like this was time very well spent. Thank you for this.
I like the way he interacts with his viewers
This is an amazing tutorial! Please keep publishing like this. very well explained! I would love to see about matplotlib, numpy and if you can get inside machine learning
Very simple yet comprehensive tutorial on Pandas. You had my attention throughout. I do use Pandas for data analytics along with numpy. That said I learnt quite a few tips and tricks. Thank you for sharing your knowledge. Way to go Keith! Liked and subscribed.
I have been learning python and using pandas for about 3 months now and done innumerable searches on the internet with questions regarding use of specific statements and coding. I wished I had come across your video earlier! You are a born teacher and know how to layout and explain complex terms and concepts. How can someone that looks so young have such a strong grasp on presentation and user needs? The concepts you explain are the same things I have sought information on for 3 months but all in one place and succinctly explained. Thank you for all your work.
Thank you, Keith, for making this super helpful tutorial. You're a great teacher!
dude this is an amazing introduction to pandas. Really helpful, thanks a lot
Excellent Tutorial Keith. Very clear, at the right speed and interesting to learn from. This material is very suitable for a self learner. Keep it up.
Wow, thanks for this tutorial. I'm starting on python and took a course of udemy, but it was confusing, with your explanations many doubts are cleared up. Thanks Keith:)
Thank you so much for your time and effort. This is the best python tutorial I have watched. Straight forward and well organized. I appreciate the time stamps.
Excellent tutorial; exactly what I was looking for. Liked and subbed. Thank you for sharing your expertise.
On point Keith. 5 hrs worth training covered in an hour. Made my day.
Saved my day! I started learning Pandas, but when I missed several months during circumstances and this video about basics helped me quick comeback. Thank you!
This 1 hour course is all I need for my data analysis course. This is the best video I found on KZhead. Thanks ❤️❤️❤️
Comprehensive, perfectly paced.... Lovely tutorial!
WOW! This was just what I have been looking for! Fantastic tutorial! You explained everything very well and clear from start to finish. Best Pandas tutorial on youtube for sure! Thanks man :)
Awesome video Keith! I'm a beginner programmer but your explanation is super clear! Thanks for the videos:)
Bro I started a data science internship in the beggining of the Year, we use a lot of pandas and you are saving my life from day 1. Thanks again, you are a god send! Subbed on both channels, cheers!
Ok I've been learning Pandas for a while now, over many different sources, and this one video has shown me much more helpful little hints and tips than all of the other material I've looked at previously!!! Thannnnnk you! Please do more Pandas stuff as this has been so awesome =]
Been looking for a complete tutorial on Pandas. This is amazing! Thank you.
You break down all the details in a way that I can't believe this is for free. Very high quality stuff. I was up and running with this library in short order
This is such high effort content. I was amazed that you actually went back and sped up the video where you said you would. 11/10 great tutorial. Thanks!
Great tutorial. I'm just starting with Python, and this is a great video for picking up a lot of knowledge fast. You asked for suggestions about other videos. I'd definitely like to see more tutorials that are about cleaning data. That's the hardest part, I think. The most laborious and time consuming. I'm learning the re module (regular expressions) for Python, but it's going SLOWLY.
Dude... you should make more videos... you are a natural born teacher!!
I have my final exam in my data structures course soon, this was a great way to study and work through things my class covered months ago, thanks so much!
thanks for useful video If anybody have a problem with calculating the mean of Type 1 grouped data, use this: df= pd.read_csv('modified.csv') df.groupby(['Type 1']).mean(numeric_only=True) instand of this: df= pd.read_csv('modified.csv') df.groupby(['Type 1']).mean() That way, it won't include string-type data in the mean and sum functions.
thanks it helped a lot...can't understand the error while all the values are numreic already
so is it got updated now, since you can only perform the method on int or float columns ...
Makes me want to play the old Emerald games again, wonderful tutorial, keep them coming
Thank you Keith for this video, absolutely amazing and valuable for many! THANK YOU!
Glad you found it helpful! :)
The best pandas tutorial, best mentor having strong grip on subject
Keith, I stumbled across your video from random search on deeper understanding of pandas . I felt like you did a great job presenting the material. Well done man, it was easy to follow and understand. I did appreciate the part at the end with the chunk size and group by - would like further explanation of those concepts. Thanks.
I just went through your numpy tutorial. And that's the reason I come here. Thumb up!
Appreciate it!!
This is an extremely usefull tutorial. You explain so good bro. Thank you very much. Like and subscribed. Hugs.
I swear this is the most useful python channel on KZhead. Top stuff.
What an excellent video. I have watched so so many tutorials, but this is definitely one of the very best.
Wow man! Holy smokes that was such an amazing breakdown. I came into this knowing nothing about Pandas and now I want to get back to work with my personal data! Thank you so so so so much. I’m off to find the documentation!
Glad you enjoyed! Your comment made my day :)
27:15 It seems that the dataframe got scrambled up a bit there, most likely from having the cell running multiple times. Even when there was an error message, it appears that either the Total or the Legendary column was moved to the left of HP. Upon running the cell again (with the corrected version?) it calculated a new Total adding the previous values and generating corrupted results.
Like, seriously, The best of the best Pandas Course I have done so far! Starts off easily and basically and the explodes!
The best tutorial I have seen so far on data analytics. I now see how python/pandas helps in data analytics. Thank you very much for making and sharing this video.
Awesome tutorial Keith, I learnt a lot by following your hour long tutorial. Created a new notebook instead of using the GIT version as it doesn't show what happens before you commented the code.
I came for the tutorial, stayed for the cutesy pokemon stuff, really warmed my heart
Gold medal bro, I was searching extensively for a good data science resource and reddit just sent me to random coursera/edx courses that used to be free but don't appear to be anymore. Your content is highly organized, extremely concise, and well thought out. There is a reason that only .01% of the votes are downvotes. THANK YOU!
Well Done Keith, you truly nailed pandas and made python learners life much easier :) I have started learning Python a month ago and have no language background, and I'm glad i find you.
I've been looking for a good pandas and python video for quite sometime now. I have to say that this is really amazing. You've explained it so well that a beginner like me could easily understand. Great job and thank you. Can't wait for more videos. (if possible, matplotlib)
This tutorial helped me alot. Thank you so much!
Your iterative approach is very effective to expand concepts without overloading. Great job. On to the next one.
Great job dude.
I learned so much, thank you. Then at the end...that music tho. I lost it! LOL! Did not see it coming.
Keith, you are the best Python instructor! Very easy to follow. Thank you!
Day 1 on my journey to learn data analysis with python, this vid and kaggle's free pandas course is just what i needed to give me more motivation to keep learning.
finally.. a new video... I was waiting for a Long Time😍😋
"stop texting me! I'm making a video!" "who has the nerve" hahahahahahha you explained well, thank you.
that is funny :))
you've been really helpful mate, been struggling with pandas/numpy and finally im getting somewhere! more of this please!!!
This helped me so much Keith thank you. Work does not want me to use excel anymore for any data frame so I have to learn python. I have spent nearly a month trying to learn, even something as simple as adding a data set to python I was getting wrong. Now I am so confident with the work I have produced.
In the chunksize section, you pick a well-documented bad practice, namely calling pd.concat inside a for loop. As the loop runs repeatedly, this operation becomes more and more expensive (because new_df gets longer and longer). Per the pandas documentation, the better approach is to append each df to a list and then pd.concat the list elements just once, after the for loop.
Hello, can you please provide with a tutorial for that? Quite new and clueless here.
dataHere = [] for chunk in pd.read_csv('modified.csv', chunksize=5): dataHere.append(chunk) newnew = pd.concat(dataHere) This looks right?
Thanks for posting! As an MIT student taking a data analysis class, this video was very helpful, more useful than the other tutorials online!!
Found it very useful too!
"As an MIT student" Weird flex but ok
From the bottom of my heart, Thank you very much. May you never lack. May the elements, forces, and the entire Creation align itself for your own good.
2 years after this video was posted, I'm here watching and learning Tons of stuff. Thanks man!!!!
Me tooo today i watched it Comedy 😂😂😂
I loved the fact you used pokemon as data set it was fun learning I could also check my knowledge about pokemon hahha Love love
What the hell, I imagined this topic in afternoon and video recommended after only few hours. And the shocking fact I didn't even searched about this topic from many days.
thast AI at work
This was such a good walk through for Pandas covering so much information. Thank you so much, I hope you continue to do more videos.
No BS, To the point! Man, probably the best tutorial out there. 🔥🔥 You have a sub right here. Thank you so very much for explaining this like no one ever has!
SQL person w/ limited exposure to Python here. This was useful as hell.
This was such a great introduction to pandas and on DataFrame. This is exactly what I was looking for. Since I hadn't previously downloaded pandas onto my mac, and didn't feel like installing anaconda either, I was running into some troubles installing pandas with just "pip install pandas" so I thought I would include the instructions as to how I did it. simply do: pip install pandas --user If nose and tornado aren’t downloaded do: pip install nose --user then pip install tornado --user (nose needs to be installed first) then terminal also suggested I add it to my path, so I did: sudo nano /etc/paths add the path at the end of the file do ^X and then Y then hit enter
You've just got me 30% of my whole assignment. Thanks dude
Thanks for this content. I really apriciate when people pass their knowledge ahead. I'm starting with Data Analytics, after getting to know the basics of Python. Your video helped me a ton! Hope we get to see other videos with panda a cases of use.
39:27 i was looking for .str.contains('*') filter thanks
F for "MEGAnium" that got filtered out while being old school Poke 😂 Great course! Looking forward to learn more from you!
I fixed it by writing "Mega " in the code.
This was genuinly so helpful, thank you! I am mostly through a data science course and have been struggling to figure out actual applications for the information I have learned. This was excellent!
Glad you found it helpful!!
Thanks Keith for this video! I'm new on data science I'm using your videos to practice and learn a lot more. Really thanks!
When I start making money with these knowledge, I'll give you some share!
damn I found the right one!!
Working on my CS Bachelor final project, this video has been so useful to me as I'm using Python. Thank you very much!
Omgeeeeee!!!! Thank you so much!!! I've searched sooooo many videos trying to help with the delimiter problem I've had (i didn't know that was the problem) and you're the ONLY one I've found that even mentions it!!! 🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾