What runs ChatGPT? Inside Microsoft's AI supercomputer | Featuring Mark Russinovich
Get an inside look at the AI supercomputer infrastructure built to run ChatGPT and other large language models, and see how to leverage it for your workloads in Azure, at any scale.
Go behind the scenes:
-For how we collaborated with NVIDIA to deliver purpose-built AI infrastructure with NVIDIA GPUs
-How Project Forge checkpointing works to restore job states if a long training job fails or needs to be migrated
-How we used LoRA fine-tuning to update a fraction of the base model for more training throughput and smaller checkpoints
-How UK-based company, Wayve, is using Azure's AI supercomputer infrastructure for self-driving cars
-And how Confidential Computing works with Azure AI to combine datasets without sharing personally identifiable information for secure multiparty collaborations.
Mark Russinovich, Azure CTO, joins Jeremy Chapman to break it down.
► QUICK LINKS:
00:00 - Introduction
01:15 - AI innovation building specialized hardware and software
04:22 - Optimizing hardware
05:40 - Improved throughput
06:17 - Project Forge
08:01 - Project Forge checkpointing demo
10:02 - LoRA fine tuning
11:29 - Use AI supercomputer infrastructure for your workloads
12:34 - How Wayve is leveraging AI supercomputer infrastructure
13:47 - How Confidential Computing works with Azure AI
15:21 - Wrap up
► Link References:
Leverage Azure AI capabilities for yourself at aka.ms/AzureAIInfrastructure
► Unfamiliar with Microsoft Mechanics?
As Microsoft's official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft.
• Subscribe to our KZhead: / microsoftmechanicsseries
• Talk with other IT Pros, join us on the Microsoft Tech Community: techcommunity.microsoft.com/t...
• Watch or listen from anywhere, subscribe to our podcast: microsoftmechanics.libsyn.com...
► Keep getting this insider knowledge, join us on social:
• Follow us on Twitter: / msftmechanics
• Share knowledge on LinkedIn: / microsoft-mechanics
• Enjoy us on Instagram: / msftmechanics
• Loosen up with us on TikTok: / msftmechanics
#Azure #OpenAI #Supercomputer #LLM
This was a great Microsoft ad.
Definitely they’ve earned this right.
Well, duh! What else did you expect?
😂
@Pathipati Sai Well, I wouldn't go that far. I'd say they are /less/ evil than they used to be, but that's really not a difficult achievement.
Great Nvidia ad as well
A fascinating look at the big picture, I've always wondered about that. This vid shows it . I'm an Apple Guy at home but I am so impressed with what MS has done in the last few years. I run MS Edge and Bing Ai on my desktop at home and at work. The brilliance of these people is just beyond belief.
Finally some actual insight into the system! Thank you so much for clarifying LoRA usage, clustering and your "AIOps" apporach 🙏 Good to see someone so knowledgeable talk about it!
microsoft is just killing it in this area. seriously impressive engineering. i wonder if they'll ever release an ai agent for pc that will fix os-level issues (eventlog issues, give human-readable advice based on eventlog errors, etc)
Microsoft Copilot?
No more low level techs.
NEVER 😂
It's called 'Clippy'
Windows Copilot is already a thing. Things are moving fast.
Love hearing about the hardware that exists behind the scenes - thanks for sharing
Holy crap. The number of iterations in the technology stack that they've done in the background is massive. To make these tools usable, scalable, distributed, and likely things I don't even comprehend since it's out of my domain, is massive. I can't imagine how much it costs to use this stack, probably an enterprise level offering.
Impressive to see how Microsoft has transformed.
The new CEO is a vision-er much better than Steve Developer, develope, developer... MS should come back with windows phone.
Much better than Meta, at least.
how?
Absolutely fascinating look into the BTS of these incredibly powerful AI tools now. I was always guessing that the easier something looks and feels for the consumer, ALOT of manpower, cost, time, resources would have gone into making that a reality. Massive respect to Microsoft for the transparency and releasing these videos on how they're building their AI capabilities. Incredible time to be alive!
A very good session showing what is under the covers with Microsoft's AI computing. Microsoft is going a good job with the confidential compute which is ultra important for business who want to use AI for both the business & their customers.
It's impressive how GPUs have become such a powerhouse. CUDA was a bright idea way back when. Wish AMD competed better here.
Don't forget Tensor Cores
george hotz is going to change that - AMD will be a powerhouse soon
@@SpeedboycentralI personally admire George but this is not a simple problems. Even he himself gave up on AMD hardware after trying to run the 7900XTX to utilize his own AI framework, Tinygrad. But in AMD has already build a translation layer for ROCm to run CUDA on AMD hardware. How much of a performance penalty? We don’t know.
after listening to these guys.... i feel that my work is useless and only a few people over the world get to do really impressive stuff.
what do you do
Kinda yeah. People benefit in all industries from a small group of capable and driven people
Comparison is the killer of joy!
Same. Chin up, your work is good too.
@@monad_tcp This is unfortunately true because everyone wants the easiest path to make the most money.
I love it when Mark is on the show, he's an absolute tech heart throb!
Just said that to my wife. The look I got! 🤣😂
@@TheB1nary rookie mistake, I quite quickly found out my wife is very judgmental about my tech-celeb crushes. For example I definitely don't talk about John Savill in front of her any more! 😂
@@TomWhi Duly noted -- I often watch the man-beast John Savill and brag about him! 🤣
@@TheB1nary it's hard not to! The brain on that guy...!
He is legend ! 😊 #FluffYouSony
Epic video, thank you :)
This was so nice. Cant wait to learn more about Azure.
So cool. Keep it up!
Russinovich Is one of my biggest inspiration in IT!
Just came to my youtube this video... it is very impressive, the infrastructure.
The legendary Mark Russinovich.
Excellent presentation
The fat tree topology still used in every data center, brilliant and simple ideas live very long
Very impressive!
Great video!
I’ve never seen comp specs as this sexy. It’s freakin insane.
Mark is the Master of the Microsoft Universe!!!! He has got more genius in his pinkie that I got in my entire body.
Mark is the Azure, fascinating amount of knowledge came from this guy in short period of time. Go systernals! 😊
Great insight into a great technology!
holy... I had no idea Mark would be quite young still - he was already god-level windows guru when friends and I started our careers in the late 90s - he must've broken through the technological ceiling at 25 or something!
I am a fitness freak and my first thought when I saw him was like...ok, this somewhat elderly guy is EFFICIENT due to his physical condition. He actually looks like another fitness freak ....which makes me wonder...I always thought efficient nerds were pale and skinny as they work too much, but this guy must be semi-retired to be in that shape ? If not, then I don´t understand how he does it !
Rumour has it that he keeps a self portrait in the loft.
@@brownianmotion6319 🤣That would be quite "gray" !
@@staffanlundberg He only works part time (sold Winternals to MS long ago) and can afford a good gym with pro trainer(s). Good for him, I say.
15:00 - Confidential GPU seems like a great idea; individual models can retain their IP while simultaneously contributing to a larger brain.
Its intersting to see how supercomputers have changed over time. Supercomputer before the Cray 1 in 1976 were the size of warehouses. Cray built small supercomputers then starting in the 90s they started getting bigger again and now they are the size of warehouses again.
The Crays were from that brief time when supercomputers were custom-engineered architecture. Every supercomputer in recent decades is made by connecting off-the-shelf hardware together. The engineering challenge is in making all that hardware operate coherently without excessive overhead and able to tolerate the inevitable hardware faults. You take a pile of hardware accelerators and stuff them in a server, a stack of those servers in a rack, a row of those in a room, wire it all together with the fastest ethernet or infiniband you can afford, then hire some crazy-good computer science specialists to make them all work together, and some pretty-good HVAC and power engineers to stop the thing from melting itsself or the local substation.
I didn't imagine Mark Russinovich like this, when I used his tiny but very powerful tools several decades ago 😁 Bginfo, filemon etc..
Same😂
@@dibu28 procexp64.exe FTW! 🥳
Awesome stuff appricaite all the effor Microsoft puts in to build the future!
It was fun and informative listening to the CTO talk about Azure infrastructure. His vocabulary is amazing and the jargons he uses is so smooth. It shows his tenure!
Very interesting insight, thank you! Microsoft is back :-)
I remember Mark Russinovich from the old days, from sysinternals tools
lol yep
What I am wondering is if they have built in industrial process to use the heat produced for some industrial process as well as thermocoupling to capture energy from the differential? Is the heat built into the model to process like dehydration, or chemical process?
I don't think people how insane MSFT execution of their Ai strategy is - I have never ever seen a company execute a strategy so cohesively as a unit - not even small business - let alone a behemoth like MSFT To get every single department of MSFT to work collaboratively to embed Ai into every product is unheard of ---
Do you build power plants by these servers?
Can't believe I spent nearly 16 minutes watching a MS Azure advert... :) So interesting
BTW, there's your "moat" right there: MS is in the perfect position in the middle of the triangle between hardware (nvidia), foundation systems (OpenAI) and the customer (Azure)!
Impressive!
Loved it
1:37 did they actually called it MEGATRON?
Yup they did
Indeed. From the blog, "the largest and the most powerful monolithic transformer language model trained to date, with 530 billion parameters" www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
LOL :D
Uh ohh we need Optimus.
@@MasayaShida...more than meets the eye
At this point Microsoft is killing it. On services: Azure, productivity suite, search engine, and also with edge browser being the best browser on the market; I do everything on edge- reading pdf, it reads text now, fast browsing. Microsoft is making a big comeback
13:42 "... and that's where we excel." Me: "Word."
Quite the work
Keep it up, thanks microsoft.
Sounds wonderfully like a talk about the Turbo Encabulator 🤣
he knows his stuff.
chat gpt와 대화할때, 대화 한줄에 얼마의 전력과 냉각수를 사용할까요? 물 50ml정도를 냉각수로 사용하나요? 궁금하네요 How much power and coolant do you use for a line of conversation when you talk to a chat gpt? Do you use about 50ml of water as cooling water? I'm wondering.
Less than a bitcoin lol
9:45 I'm glad criu is now used in mainstream
Why have I never heard of a Microsoft's Megatron Turing? It is three times bigger than gpt-3 and can do a variety of stuff, even use natural language...
So how come your linear and at the options of Wi-Fi won't align together
Brilliant
Watching the free market at work is an amazing thing 🤙🏽😎🖤🐓
I like to imagine a time traveling 1700s pilgrim watching this video and trying to make sense of anything said here.
So ... tell us ... are you using megatron-turing nlg 530b for your own enterprise decisions?
Omg amazing video
it's great to see how MS and NVIDIA are working together to build the AI infra. Though wondering how good is it compared to Amazon and GCP?
Azure's support for NVIDIA Quantum InfiniBand and NVLink is a differentiator here and it's something Azure has had a head start on vs. other vendors who currently offer it.
Does it requires a reboot?
That's good 👍🏻
So is chatGPT essentially located in one location? Where he described the server stacks and gpu’s.
There are multiple instances of GPT-4 running concurrently in multiple locations to run ChatGPT and other GPT-based services.
that's an amazing story. Now I feel jealous =). Why can't I work on the most sophisticated hardware for the most cutting edge technologies, and just providing services to our Biz customers (whoever they may be) =). (P.S. technically, I work in Microsoft though, but still - our sub-org is way far behind)
From the man who brought us Sysinternals!
What is the reliance of ChatGPT on phase detractors and magneto reluctance?
10:08 it is a bad idea to name it as LoRa since the name is used for radio communication modules. I guess the guy invented the name never "googled" it.
Thanks for Sysinternals
Very impressive. Makes me regret selling Microsoft stock last year - they are just so ahead of the game when it comes to AI. Fortunately though it is still driving my S&P 500 and NASDAQ 100 ETF gains for this year so still benefitting from the excellent innovations from Microsoft and OpenAI.
Its now the ENTRA CTO:-)
well adapt that AI for gaming generating and imagine that every person will be able to create their own personalized games or software's verry nice
Great and skillful people are pushing big companies. Microsoft understood its weakness and now self-healing step by step.
If I saw this without knowing that it’s successfully in production, I’d be think to myself “yeah it’s a lot of hype, but does it really work?” My perception of Microsoft has now changed.
What is in production?
Thats a lot of compute. 😯
I love Chat GPT!
8:59 so this is why GPU access is so expensive in all the other products, you just hog the GPU, damn, I want this.
How do you input 530 billion parameters
Are we going to see the latest gen nvidia H100s being used here? How much of a difference would it make if nvidia open sourced their drivers?
Lol. Never going to happen!
@@SlCKB0Y-sb1kg I haven't followed the news lately, but why? Last time I checked the out cry was to halt LLM research before these H100s are deployed. Why say, never gonna happen?
@@therealb888 Sorry, I should have given context. Various individuals and organisations related to Linux have been extremely vocal with trying to get Nvidia to open their drivers for literally decades. Nvidia has always been extremely hostile to open source and consider their drivers to be an integral part of their intellectual property.
@@SlCKB0Y-sb1kg I know right, even Linus flipped nvidia 😂. I hope there would be leak of all their firmware, drivers, etc A change in senior management?, even better. But now with AI the demand for open sourcing should be from deep pockets. Even microsoft sold windows source code to governments. It's an unfortunate misery that AI research relies on nvidia cuda & it's closed source stack while AMD is going out of it's way to opensource. Time has proven that in software, opensource wins.
@@therealb888 Yea, I love it! kzhead.info/sun/nL2QqrGuo5xohYk/bejne.html . Realistically though, I think the fact that the whole AI industry is so dependent on Nvidia will become really problematic in the near future.
What are the energy optimization techniques you use to save the power for all these forms information processing?🤠🤔What does your electric meter display read now-a-days? 🧐🤫🤑🤯 Can you portray energy consumption in modules/sections and represent it in a graphical workflow for the whole picture? Not trying to be sarcastic, just asking. Okay🤗👍
Chat GPT is the new Crypto :D
Is there any reason why GPT-4 can't handle calculating simple problems like power factors or the height of the meniscus in a capillary tube without making mistakes?
when combined with Wolfram to help it do the math, it can do better. Gpt itself is bad at being exact with numbers, being good with the numbers isn't one of the things it is trained in, that's not the 'game' GPT itself is playing. GPT itself is playing more of a 'if this is written so far, what do you think should be written next' game, that it is pretty good at
In my playing around with the Chat AIs I’ve found that they are both better than you expect and worse than you expect. There are some surprisingly simple things they still can’t do. But there is also an art to formulating your prompts too. So you can sometimes correct their answers with a little prodding.
what about STABILITY AI , weredo they fit into all of this ??
Super!
Sold.
Chuck Norris once opened Windows and ChatGPT was invented by the operating system to respond 😂
It would be great if it could make a customized videogame just for me at no charge.
Can somebody tell me what's the name of the VSC plugin that creates separate blocks for each python method? It looks neat
Will MS finish the Project Milo now?
3:40 why they use GPUs and without gpu is it possible ? Apple like technology used
Running exclusively on CPUs would be exponentially slower for the same amount of money spent on hardware. GPUs are purposely built for parallel and vector processing, which is what a lot of AI needs. CPUs, meanwhile, are built for general instruction processing.
Is there a slow down button for his speaking? I can't keep up
Here from MisoDope
12:20 That keyboard looks as if the video is mirrored, but the text does not look mirrored. What's going on here?
She have a left hand keyboard
If Microsoft now launches Surface devices with its own in house AI chip with Windows 12 full with AI features its going to change the personal computers... I hope
cool
And plebs believe Microsoft is just using OpenAI’s AI when GPT-4 solely exists because of Microsoft’s supercomputer.
Does everything built on azure belong to Microsoft?
where can I start learning stuff to understand more than 1% of what these fine gentlemen are talking about - I feel absolutely clueless, lol
How can I prove that I am the only one using this application, and it is a real person? But my question is, why is it a real person?
this feels like I walked into a far more advance alien civilization. I don't understand a thing. 😆😆😂😂🤣🤣
Damn so hardware is not the bottle neck for AI huh, it only needs 96 GPU 😮
The network cards seemed to have been the bottle neck, 8 GPUs later and 8k per second video streaming possible
With their checkpoint/save type mechanism it was cut down to less GPU's than that and requiring less memory. Really quite brilliant
That was for fine tuning to a specific domain. 24 for LoRA, low rank adaptive fine tuning.
Microsoft Research is being synonymously with OpenAI
crazy.
GPT 4 FT ... Holly shi....how to get access
Yea but can it run Crysis?
What does it take to run it… computers.. who would have thought.