Requests vs HTTPX vs Aiohttp | Which One to Pick?
Exploring API communication in your app, considering requests vs httpx? Or maybe even aiohttp?! In this video, I’ll share with you my preferred choice and why you should consider it as well.
Git repo: git.arjan.codes/2023/httpx
👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
💻 ArjanCodes Blog: www.arjancodes.com/blog
✍🏻 Take a quiz on this topic: www.learntail.com/quiz/dxgeth
Try Learntail for FREE ➡️ www.learntail.com/
🎓 Courses:
The Software Designer Mindset: www.arjancodes.com/mindset
The Software Architect Mindset: Pre-register now! www.arjancodes.com/architect
Next Level Python: Become a Python Expert: www.arjancodes.com/next-level...
The 30-Day Design Challenge: www.arjancodes.com/30ddc
🛒 GEAR & RECOMMENDED BOOKS: kit.co/arjancodes.
👍 If you enjoyed this content, give this video a like. If you want to watch more of my upcoming videos, consider subscribing to my channel!
Social channels:
💬 Discord: discord.arjan.codes
🐦Twitter: / arjancodes
🌍LinkedIn: / arjancodes
🕵Facebook: / arjancodes
📱Instagram: / arjancodes
♪ Tiktok: / arjancodes
👀 Code reviewers:
- Yoriz
- Ryan Laursen
- Dale Hagglund
🎥 Video edited by Mark Bacskai: / bacskaimark
🔖 Chapters:
0:00 Intro
0:45 Popularity
2:05 Requests
6:43 Aiohttp
8:40 Httpx
10:26 Concurrent requests
13:33 Licenses
14:49 Outro
#arjancodes #softwaredesign #python
DISCLAIMER - The links in this description might be affiliate links. If you purchase a product or service through one of those links, I may receive a small commission. There is no additional charge to you. Thanks for supporting my channel so I can continue to provide you with free content each week!
👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
You should make a video on API rate limiting, especially in async contexts! Also, if you're going to make performance comparisons, you should use a specialised tool like Hyperfine, which runs your program multiple times and gives you stats like average and standard deviation. It's more robust than printing the elapsed time from within the script and running the command by hand multiple times. It would correctly recognise those longer times as outliers.
Rate limiting can easily be bypassed by using proxy. The proxy should be high quality tho. Otherwise making server sleep until limit lifted is same as using requests
@@ankit.chaurasia No it can't if we are talking about API with APIKEY for example. Rate limiting is not that easy.
@@IwoGda No one can bypass rate limit for given api key as they will register every request of yours. When requesting with async nature it will put every request in each thread. I think the problem is that u aren't understanding async architecture
@@ankit.chaurasia I was just responding to your "Rate limiting can easily be bypassed by using proxy", doesn't matter if it's async or not.
I agree. I thought that aiolimiter would be enough for me and it was but then I ran a larger job unknowingly and it caused database timeouts from the server I was requesting from. I still need to look into it and to slow down this big job I just used an asynchronous lock so I could run that one job slow. I think part of the issue was although I was using aiolimiter and limiting requests, I think there is an httpx connection pool that can have like 100 requests per connection at once. Need to test more though.
I realized the need for httpx when I needed to make three api call before sending a response to the client (each api cost about a sec) imagine waiting for 4 sec before sending a response using requests(totally unacceptable 😂) async does wonders in that area
License is an important part, i wish you did a video explaining what license a dev should use if they want their work to be open-source or if they want to commercialize it or be both at the same time.
It's also worth noting that HTTPX has HTTP2 support which could be quite helpful in various scenarios.
as someone who is new to web development, what are some of those aforementioned various scenarios?
HTTP/2 support in HTTPX can be particularly beneficial in scenarios where you need to handle multiple requests concurrently, as it allows for more efficient use of network resources. It's also useful in situations where reduced latency and improved page load speeds are critical, like in high-traffic web applications or services.
Also worth mentioning is nyquests, comes with automatic handling of http1.1, http2 and http3 out of the box. They also say it's the fastest implementation compared to the standard libs mentioned here.
@@anonapacheI can't find "nyquests" anywhere, not in PyPI, not in GitHub...
For small simple projects it doesn't really matter, what you use. The problems come when you are stuck in deep corporate code, where you need to take into account legacy code, licenses, rate limits and lots of other stuff. Recently, I stumbled on an interesting case regarding certificates and the inability of requests library to handle those properly. I dealt with this via another library, but it took me quite some time to diagnose that the issue is actually with the library and not with the concrete certificate or server. So, from my stand point of view, it would be interesting to see a video about dealing with rate limits, concurrency and certificates.
nice video mate, I had no idea requests had this multiple session patern. Thanks!
Heck yeah. I needed a video on interacting with APIs. I'm working with API's and figuring out how to store API keys securely and the data received from them is my next step. Thanks for covering more options than requests.
Great explanation. Many thanks!
Nice. I'm actually swapping over to HTTPX for my projects at our company -- first notable work should get a working mock this weekend. Great to see see this. Thanks. (Gotta say: I love when blocking ('normal) & non-blocking (e.g. async) are both first class citizens. Makes growing projects easier, and one doesn't know which quick solutions are going to be grown into somethign more serious.
thanks!! didn’t know about requests sessions. was already using requests in my CLI tool, but now that i just implemented session usage i got 2x performance speed from it 😬 game changer!
Great to hear!
i've learned so much from your channel! thank you !
Glad to hear that! ☺️
Great video. Whether to optimize for simplicity or performance depends on the specific use case. I tend to favor optimizing for time to market and fighting the urge to do premature optimization. Recently, I have had to use async http requests on some new projects where the ability to do concurrent requests is a core project requirement. I have not tried httpx but I like what I saw in this video.
Useful and clear video, thank you! 👍
You’re welcome!
I literally just introduced httpx in my team, to show a demo where I send some iot data to an API that then plots the charts on its frontend platform, it’s actually quite handy since we mostly used requests before and now I can do async stuff. And of course with pydantic classes it’s a very nice workflow
Thanks for another useful video, will have to get my hands dirty with sessions, I think. What drives me absolutely nuts with async in Python is that I cannot reasonably limit the concurrency. I may be spoiled from Scala with ZIO, but what I want to be able to do is say "I have 100'000 requests I need to send, send them async, but never more than 8 in parallel because that's what the server can handle", and I still don't know of a good way to do that with asyncio.
You can easily achieve this with asyncio.Semaphore
This was really cool and really useful. Thank you!
Glad it was helpful!
That's a nice video for a beginner in web requests. What i kinda expected to see there was concurrency with requests using threads. Just to show that it's still possible, but also mention that it's not always the best approach compared to asyncio 😇
I've always used AioHttp myself. I just like how it works, it's fast and the concurrent model is nice.
What a uplifting content. I felt compelled to comment so that u you receive feedback on how good your content about Python is. Congrats and happy 2024. Waiting for you to add some more content on the course to buy your python one.
I appreciate that! Happy holidays! :)
Just out of curiosity why not setup a venv with python@3.11 to make the demo work with aiohttp ? Also python@3.12.0rc2 ... video production must take a while as 3.12 was released October 2, 2023 and its now 2023-12-16. Video quality and audio are top notch and the explanations are pleasant to listen to, 🙂cheers.
raise_for_status is how I typically handle error responses in requests calls. I would love to see examples, esp. with concurrency, that handle responses not on the happy path.
Oh, awesome timing. I was thinking of what to move to from requests(thought of httpx).
Glad it was helpful!
I'm a huge fan of asdf for managing multiple versions of python (and it has many other plugins). Just mentioning it in case you have a package that requires a specific python version (3.11 vs 3.12 for example) you can easily have them both installed and switch around based on different projects.
For anything that is being put into production: performance > simplicity. If you put something into production for the first time, you quickly realize how expensive your computational resources are and also how important performance is for the experience of the user, especially if you have solution that must scale well.
But on the flip side simplicity ensures your software is easier to maintain and allows new engineers to quickly get up to speed with the codebase. Its a trade-off that you need to decide
👍excellent video, as always! Personally, I try to never favour simplicity over performance or the other way around. I try to assess the performance needs for a project, depending on its specifics: most of the time, at this stage, performance does not appear as being a critical issue, but sometimes you can anticipate that it will: I just try not to be over-pessimistic and not give in to premature optimisation. Then I go with the simplest possible implementation (the fastest, easiest to write, easiest to read, and in most cases easiest to maintain or change later, option: in that case, would be requests) in order to get the functionality up and running. Then, if tests show that things go smoothly, there is no need for perfomance improvements. On the other hand, if the test results show "poor" performance, it means that it's time to improve, and possibly to switch libs/modules. "Poor" here isn't something absolute, it really depends on the project and context(s). For some projects, a 10ms improvement makes a huge final difference, for some other ones, not really... So, yes, after writing, I realise that I do tend to favour simplicity over performance hahahaha 😅😂 but I try to do it in a way that is never detrimental to the project. I prefer simplicity (who doesn't?) until this simplicity becomes a bottleneck. In short: use the right tool for the job 😅
Thanks, that's really awesome. =)
Thank you!
4 years ago, i used a library named grequests, for requests + Gevent. It worked great for a project where i a had to fetch about 50 invoices in pdf forlat from an api. Very simple to use. I will give a try to httpx.
Great video as always.
Glad you enjoyed it!
as always, simple and informative, thank you! Seems like in your last HTTPX example, the printing takes most of the time, maybe put the time capture of just the fetching, not priting. Either way, love the example, thank you.
Thanks
That was a very nice Arjancodes-style classic video! Thanks. In any case, may we need a very basic introduction to those magic keywords: "await", "asynch", "gather"? Good Software Engineers don't use words they don't understand...
Great video as always, Arjan! I recently named a CustomGPT after you, but it isn’t quite as helpful, lol
I'm honored! :) I also made a GPT, you can try it here: chat.openai.com/g/g-h8RebagkT-arjancodes.
Thanks for this video. For years, I kinda stayed away from asyncio and aiohhtp -- but recently I've gotten back to them as, especially, asyncio got kinda simpler. Most of my async codes using RQ etc. are being rewritten to that. And just for a curiosity, I've been using this simple class / context manager to measure exc time (a decorator is great for functions but if you want to measure a block of code, I am for that): class Timer: """Context Manager measuring time needed to run and finish a piece of code. Example: with Timer() as timer: my_code_goes_here """ def __init__(self): self.start_time = None def __enter__(self): self.start_time = time.time() return self def __exit__(self, *_): elapsed_time = time.time() - self.start_time print(f"Time elapsed: {elapsed_time} seconds")
If you want to still use requests in an asynchronous manner, you can use Gevent with monkey patching the socket library. The advantage of Gevent is that you don't have to use async/await and you can still program in plain python.
Happy watching)
Watching your videos is like a therapy after i've spent a year butchering my python project. Im constantly fighting the decision "Add new feature" vs "take some time to learn better programming" vs "relax so i dont hit the burned out wall".
I personally prefer httpx, too, but for a different reason: it supports the trio async framework out of the box. IMO it is much easier to use (more structured, better API) than AsycIO from the standard lib, in all but the simplest use cases. Thanks for the video! As always, i learned a lot!
Hey Arjan, thanks for the video. This was very helpful for my current work. I was already wondering if there is a way to keep the current session going on. The second improvement with concurrent requests is interesting, but I already using own dedicated threads for each HTTP request. Otherwise I would wait a very long time to gather all the information of many many devices I'm asking for information :D
Re. the httpx option: If the API has rate limiting where should a strategy like exponential backoff be implemented? Great video BTW. Thanks.
I like network performance over simplicity for large applications.
Great comparison, as always, thanks for that sir! I was familiar with requests only by this point. Objection your honor: if you were to use larger sample, or more endpoints, the difference would be more comparable. Anyway, I have a question to our typing guru: is it allright to use Any for requests (or other libs) response? I always struggle with this one, just hate putting Any, but what do I know what would I get in return of the endpoint?
If you're a fan of requests, just use it with a concurrent thread pool; it has a little bit more overhead than async but is as fast as async without wired async await syntax.
Thumbs up as always, Arjan. The "AIO" is probably an acronym for "Asynch I/O" or something like that, so the letters would be pronounced individually. And in English, the "ai" diphthong would *usually* be pronounced as a long "a", not a long "i". Except, of course, when it's not: it's English. ;-) Something that caught my attention that's worth clarification: the "gather" function in HTTPx. If you have 5 outstanding requests, and one of them somehow goes awry and doesn't finish, are all the others hung forever? Can you, say, specify a timeout and get a return code that says "Not all are done, so you need to check them individually"?
Yes you... you can set a raise on 4xx and 5xx errors
7:00 aiohttp does work with Python 3.12 starting from version 3.9
At the end you mention Licensing, but the issue is typically not about "Commercial" usage but of "Proprietary" usage. For instance the GPL licenses allow Commercial, but not Proprietary use.
I love the keyboard in your into. Could you please provide the make and model number?
Would love a deep dive into Async programming with Python for backend web-devs.
I wished you had included Axios on this comparison.
Were you using a VPN in this video? It appears that your IP may be exposed if not. Not sure if you care about that though, but figured Id share. Awesome video. Ive been stuck using requests for so long that I havent even considered looking at alternatives!
performance vs. simplicity.. depends on the use case. If i just need to get an api call, simplicity, if i am making lots of them and i need the code to be performant... performance
Nice to see the options available, but you didn't delve into any feature comparisons for advanced things like streaming post bodies which aiohttp handles better than httpx.
I think for aiohttp + one `with` statement does not add some complexity - this is just another way of code structure. Maybe interesting that httpx works on top of anyio, so it can be runned on both: asyncio and trio
Also regarding - performance/simplicity - I would say simplicity - it is simply in Python's DNA, unlike performance :)
Popular python libraries suffer bad API design. Check out the two most popular web frameworks: Flask and FastAPI. If you look at them closely, you see absolutely inexcusable design decisions. It seems that most devs see the "look how compact the code is to run this hello world example" in the QuickStart session of the doc and then completely overlook everything else.
What if we use it to make a small internal tool at a company? Do we still need license for that?
im a noob programmer so im gonna go with simplicity > performance. But i doubt a business application is gonna opt for that
Does it make sense to use connection pooling, when I just send out one request?
I'm confused, I understand connection pooling, but what exactly is the benefit of using aiohttp or httpx over simply using threads? The code is pretty simple IMO, where you just have futures instead of the async await syntax, AND you don't rely on any external lib.
Oh! Help me out here: with's going on with `func(session: requests.Session)`? Doesn't look like either a type hint or a named parameter assignment. I'm a decent Python programmer, but my knowledge of its features is quite limited :^)
it's a typehint, since it's after a : symbol. it looks different because it was imported from the package requests. One could also import Session directly like from requests import Session and just have that as a type it. It doesn't really matter and it depends on the coding style mostly
Just a typehint telling exactly what object it is expecting
@@Michallote ah, I expected -> for each hint, but it's only used for the return type, right?
@@Kknewkles the -> is for the return type of the function. The : is for the parameter's type
@@_Akhilleus_ right. I've used it once or twice and forgot that since.
Why main func needs the async keyword?
Simplicity until you need performance! (and consider switching to Go or Rust lol)
Rust is not much simplicity (Unless you compare it to C++), and imo overkill in almost all scenarios. Go on the other hand is a great middle option - Slightly more tedious to write than Python, but a lot(!) more performance.
requests for simple scripts and pocs. aiohttp for production
Isn't it pretty simple to add concurrency to requests by just wrapping each request in a future/promise (whatever it is in python), making it essentially identical to aoihttp and httpx?
async for the win
what about niquests?
I am surprised you aren't using venvs to run multiple Python versions.
aiohttp is the best. Period. HTTPX is flaky and as you saw in this video is unpredictable. I never encounter these issues with aiohttp; it just always works.
"it just always works" - Didn't Arjan show that aiohhtp doesn't (yet) work on python 3.12😉
@@ErikS- Touché. But it’s absolutely solid on 3.11
I started with aiohttp but I much prefer httpx
Hmm maybe a video about licenses would be a good idea?
We are all using python... Simplicity first
I use requests with gevent for my scrapers and I don't need all this dirty async boilerplate code to get comparable performance as httpx, without any headache. For now, I don't see the interest of httpx and aiohttp.
Seeing Arjan use type Any. My whole world has crumbled.
If aiohttp has issues in python 3.12, httpx gets a chance to catch up in popularity.
aiosonic is now the fastest.
Wasn't requests replaced by urllib in standard python libraries?
Requests uses it under the hood
It's surprising that aiohttp have problems with 3.12 ... makes me happy that I had settled with httpx for my asynchronous HTTP needs. A friend of mine need to query several million data points to grab JPEG files. Her original non-async code takes _days_ to complete. With concurrency (I suggested to her to use httpx), her script can now finish overnight. Her code has like 600 requests on the fly at any given time, and leveraging HTTP/2 which is supported by httpx. She leverages asyncio.wait() to be able to gather only completed tasks and handle the results as they come in, generating additional tasks periodically to keep the number of in-session requests at 600.