Support AI and Games on Patreon:
/ ai_and_games
--
Back in 2019 Nvidia unveiled DLSS: an AI-powered technology for boosting the output of their graphical processing units (GPUs) for gaming PCs. But what on earth is it? How does it work? We take a look and explore its function, continued evolution and market competition.
[00:00] Introduction
[01:19] Why Do We Need DLSS?
[05:42] Texture Super Resolution
[06:40] How Does It Work?
[12:29] Competition from AMD
[14:23] Closing
--
AI and Games is a KZhead series on research and applications of Artificial Intelligence in video games. It's supported through and wouldn't be possible without the wonderful people who support it via Patreon and KZhead memberships.
/ ai_and_games
/ @aiandgames
--
You can follow AI and Games on Twitter:
/ aiandgames
Join me for live streams over on Twitch:
/ aiandgames
Join our Discord Community:
bit.ly/AIandGamesDiscord
Get yourself an AI and Games t-shirt over on Teespring!
teespring.com/stores/aiandgames
#NvidiaRTX #DLSS #DeepLearning
So if you're wondering why this episode and the previous one on texture upscaling were released in the same weekend, it's because they were written at the same time. Typically I treat each video as a complete project from start to finish, lest it's a series on the same game in multiple parts (see Division 2, Sea of Thieves, Horizon Zero Dawn etc.). But I was asked by patrons to do the episode on texture upscaling, and if you've watched that video you'll note I spend a minute explaining that DLSS is not the same thing. Because I realised it's easy to conflate the two if you're not familiar with them. So, it then made sense to dedicate another video just to DLSS. Can I just say how weird it is to spend a lot of time talking about **graphics** in these videos?
Great video, 2 of them in the same weekend! Regarding these techniques, recently Intel indicated that it would enter the market with its own solution - one that seems to be similar to DLSS, but theoretically should not be exclusive to specific hardware.
Looking forward to Part 2 when XeSS comes out.
@@rlfernandes9538 Yeah I saw Intel announced their efforts but figured I'd hold off integrating it into the episode given a lack of detail at this time. So hopefully we'll come back to it.
this aged like milk lol, you gona need DLSS 3.0 video
That catawampus reference to Mikey Neumann from FilmJoy is spot on 😁
If it ain't broke... steal it 😎
I'm glad that we have a more open and easy solution with Fidelity FX, It has been so open and easy to implement that some game modders have put FSR in various games like GTA 5 by them selves! :) I myself have only used AMD open Image Filters SDK for image denoising (which is good for ray tracing) The fun thing is that AMD Fidelity Super Resolution (FSR) even supports older Nvidia Geforce cards, so old that they don't support Nvidias own DLSS solution :) So customers owning older Nvidia cards can enjoy and prolong their cards with AMD software :D (hurray for open software) I'm even looking forward to when FSR incorporates motion vectors and a better temporal AA, which would be its next step. (Right now even though FSR and DLSS are quite similar in image quality to performance, DLSS 2.0 is a bit better and uses motion vectors) Also I'm looking forward to Intel Xe GPU cards (yes you heard me correctly, Intel is also soon entering the discreet GPU space along with Nvidia and AMD) and their Super sampling solution XeSS. XeSS seems to be promising being able to run on all hardware and also using motion vectors. As Tommy said, this is a new era of graphic computing with AI. Hopefully we will have an deep dive episode about AI Super Samplers and how they even might become in the future, specially with ray tracing and denoising AI.
While I do applaud an open source and hardware agnostic method, it doesn't change the fact that an ASIC style approach is generally by definition always going to be faster and more powerful.
@@JohnVanderbeck Regardless, I still prefer the first option. Having your hands constantly tied by the whims of Nvidia is incredibly annoying. For all we know, DLSS could work just fine on Pascal cards. It's all proprietary stuff anyway, and arbitrary lock-out of features on this type of hardware happens all the time (see for example the feature difference between consumer and professional Nvidia cards).
Tried FSR and it looks like ass on supported games, maybe I'm just sensitive to oversharpening stuff to make it less blurry when upscaled LOL. Intel XeSS will run significant worse on Nvidia and AMD hardwares, just look at the tensor performance difference. Alchemist has 8x more tensor performance than Navi22 (FP16 and INT8) and 1.5x more than GA104, but XeSS doesn't run on Nvidia tensor cores meaning it will run on CUDA, which means Nvidia and AMD hardwares will be severly handicapped with XeSS.
I played a few games with dlss having been on quality without knowing and when I was playing around with settings I turned it off to native rendering and the game looked worse 😅😅
I feel that this may need to be reiterated: AMD's FidelityFX Super Resolution technology is based on existing image processing techniques and does not use AI upscaling in any way. The tech is also entirely spatial, and has no temporal component in its current iteration, which makes it flexible and easy to implement but significantly limits its power. While both are postprocess upscalers, FidelityFX Super Resolution is in no way comparable to DLSS, and AMD at no point ever suggested it was - in fact the two technologies could even be used simultaneously, although there wouldn't be much point. The fans and media on the other hand seem to have positioned the two technologies as some sort of rivals, I guess because of the clickbait potential.
because that already happened numerous times in the past ever since the days of ATI Radeon GPUs... ATI/AMD always trailed behind Nvidia when it came into introducing a new graphical enrichment features for their GPUs... they only always came up with sort of knock off of Nvidia features after Nvidia introduced them first, the only 2 notable exception were TressFX and Mantle (there were a few others but not significant enough)...
This is something I didn't expect, BUT it make so much sense! Really glad you covered it. Hope you cover more ML videos.
Tommy learned the number one rule of doing graphical technology analysis.......using the word *bespoke*
Hey, I really love vids like this. Helps me understand what more complex settings do which helps me make more educated decisions when deciding what to do in my settings for games. I'd love more videos following similar topics like antialiasing, Rtx, and other more confusion settings. Love your vids. Thanks for your consideration.
INCREDIBLY interesting. Fantastic job, love your videos and these are a real treat!
Much love Tommy, great video! Hope you're doing well :))
Extremely well done. I'm an engineer by trade and I learned a ton from this... There was lot I didn't know here.
Thanks for the vid
Remarquable work, thanks for putting it simply ^^
I know it's all a little confusion with the dumb names, but FidelityFX is the name of a line of various software solutions from AMD, and it's FidelityFX Super Resolution (or FSR) specifically that is their image upscaling technology.
Very interesting.
Can you do a video on the A Life system from the STALKER series of games? A system which at one point could have the player miss the plot entirely by taking too long. I've always been intrigued by it and think it would be a really interesting topic to hear about from your point of view.
I presume you had at least part of this video finished a good while before you published so you may allready be aware of this but when you talk about FSR you seem to be unfamiliar with the wide range of hardware it supports. Officially FSR suppport goes all the way back to RX460 cards and one of there tech demos from june had it running on a gtx 1060 and I have made plenty of use of it on my RTX 30 series card, was just a bit jarring to hear you unsure that was possible after such a well researched video. Really enjoyed the video it was very informative,
FSR is nice, but a different approach to upscaling with it's own downsides. Looking forward to Intel's version so I can get the temporal version of upscaling on my 6700 XT as well
me watching this in 480p: hmm yes, these pixels look like pixels
I'm still confused at how it is cheaper, computationally speaking, to GUESS for a given number of pixels than to render everything at the final resolution. Or, to put it another way, what is it about DLSS that makes it more efficient to build a whole different post process pipeline on the card, than to just add more traditional rendering power?
it is in essence lest accurate than "traditional" rendering. Instead of taking time to render the pixels accurately it kind of estimates where each pixel should be based on previous data.
That's the beauty of tensor cores, the amount of samples they can pump out per millisecond is pretty crazy. You do lose some rendering time with dlss, it's just crazy efficient and naturally by decreasing your render resolution you gain some overhead in render time, this is also why there is a limitation when using dlss at high frame rates (144hz+) What's great about this is it's dedicated work aside from the traditional cores allows it to not be hampered as much, but the trade off with these new cores leads to higher power draws. Still worth imo
Well, long division takes longer than just ballpark numbers based on experience. The idea behind DLSS is that you do the computing in advance while training the AI. You render it at low rez and high rez and have it compute an algorithm that takes you from the low rez to the high rez image. It's not perfect, but it's good enough and easy to compute. The next cool thing is foveated rendering where it tracks your eyes and only renders things at high resolution where you're actually looking at, seems to work great on the PSVR2.
It wouldn't be a video game graphics video without the word "bespoke" in it :D
great video ! Nice to learn AMD is doing a cool alternative to nvidea
And it’s leagues and I mean leagues behind nvidia
Dlss is magic I swear, crazy how you can run a game at max and still have good frame rates.
Roko’s Basilisk is a thought experiment based upon the concept of a super intelligent A.I, wreaking time travel vengeance upon all those who did not contribute to its’ creation. Which implies that by even knowing about this, you then become a potential target and thus a victim of that A.I.’s future wrath.
I mean if it helps get more likes and subscribes I'm down with that.
DLSS is a good technology, though in my experience the effectiveness of it is not as good as some sources would give the impression (looking at you DF). Having said that, DLSS Quality mode produces more than acceptable image quality and does net some performance increase. Balanced and Performance, while better than normal upscaling, really start to look rough.
How something that should be scientific can cause some conflicting information? DF sings non stop praise to DLSS while being rather cold with AMD's FXSR. While other places are super impressed with SR. Weird...
@@Refreshment01 I lot of people are looking at it essentially only at 4k in quality or ultra quality modes, where FSR works best. A lot of people are also confusing the extra bit of sharpness that FSR provides with there being more detail shown in the scene. Honestly I think it largely comes from a desire to root for the underdog and/or a general dislike of nVidia. Personally in the one game I've been able to try FSR in, it looked identical to regular old resolution scaling + CAS so I wasn't at all impressed. DLSS on the other hand has been very good in most games for me, amazing in some. It also has another benefit that doesn't get talked about much - far superior AA compared to other AA methods the games provide. This is particularly important in a game like Death Stranding where the TAA has horrible ghosting (especially in cutscenes) and FXAA exhibits tons of shimmering. There have been some artifacts caused by DLSS but those have been pretty well cleaned up by newer versions.
@@Refreshment01 Opinions and experiences differ. People all have their own natural biases that are intrinsic to their perspective, and so something which is subjective such as how clear an image is or how 'good looking' the end result is will never all agree. DLSS is in my opinion an emerging technology that is very useful for running 1440p at high framerates, and it does that wonderfully ~in my experiences~ with the hardware. NVidia earns itself enemies when it alienates consumers because of price, but the underlying tech being touted does seem to work quite well.
Can anyone expand on how Shield TV does the AI upscaling? i've been searching information since the feature was added to the Shield TV. Is it a dedicated processor outside the X1?
I think so , btw check digital foundry they have a video on it too
@@klurikon3004 i've seen it. No concrete details about how the Shield is handling the AI upscaling. Just image quality comparisons
I don't think there is. I think if you're not running a game or something of the like, but just decoding video, you might as well dedicate the bulk of GPU rasterisation/shader pipeline to an upscaling algorithm. After all if it was implemented on Tensor Cores originally, you can divine from that, that it only performs matrix multiplications, and you can do those on normal CUDA cores just the same, just less power-efficiently than if you had a dedicated hardware.
@@SianaGearz Oh, so Shield TV only does it for video? Maybe i got confused because some time later Nvidia showed the functionality with some games. So they were probably from their cloud streaming service. Thanks for the answer.
Will the steamdeck have amds FSR?
FSR is open-source. There's no reason why FSR (which is a software solution) would not be able to run on the Steam Deck (which is just a portable general-purpose Linux machine). I'm not 100% on what the situation with FSR and the AMDGPU driver is on Linux (I haven't tested it yet), but I DO know that proton-ge (the community fork by Glorious Eggroll) already allows you to run Windows games on Linux through Steam Play with full FSR support on all titles. However, I can't tell you anything more than that as I have zero experience with it unfortunately, haven't had the time to test it.
I would like to see someone experiment with running both FidelityFX and DLSS in combination.
I'm no expert, but this seems redundant. The whole idea is to render a lower resolution image and upscale it using DLSS or FidelityFX. If you use one of them to upscale the image, why would you use the other one? To upscale it even more? If so, you can just use the same technology for bigger upscaling (probably leading to worse results). There's no intuitive way to combine the two, it's not like you can just turn them both on
How would you compare it with FSR? Can it come close and help consoles to maintain their relevancy in coming years
FSR is visually worse than DLSS, for obvious reasons (no machine learning behind it). Digital Foundy did a deep comparison between them on the latest version of Metro Exodus, between the newly added FSR and DLSS 2. If you only have access to FSR, it's still a good option though, but if you can choose, DLSS is a no brainer here.
how can dlss help consoles when it's an nvidia-only solution? the xbox & ps5 use amd chips.
At high resolutions they are almost equal, so both highest quality at 4k. Lower FSR starts to lose. Both implementations have their limitations, e.g. DLSS temporal artifacts with fast moving objects. FSR data only from one frame.
@@NaumRusomarov i meant FSR only
@@saiyannegi oh that. well the consoles are still new, and there are other non-ml/ai upscaling algorithms that are as good as or even better than fsr. so it's up to amd, sony & ms to implement this on the console.
if you're playing in 4k and put it in quality mode, will the game downsize to 1440p and then supersamp it or will it supersamp to a higher resolution?
If your output is set to 3840/2160 Quality renders at 1440p then upscales to 4K
There's this channel two minute papers. I'm not sure if you've heard of them, but they've gone over using AI to upgrade the quality of graphics. They used GTA V and when I say the method they demonstrated made it so photo-realistic, I am NOT JOKING.
The ray tracer I wrote for my final in computer graphics almost instantly ran a room simulation on a gtx 1080. Seems you wrote your sim poorly
Yeah... I don't doubt I could have optimised the code a little (and I exaggerated, it took like... 3 seconds?). But the real problem was I wrote my ray tracer in 2003. 😉
@@AIandGames Ahh i see, big jump since then even without the rtx cores.
Probably get this request a lot, but would love to see something on how Football Manager works! I can't find anything explaining how the algorithm/simulation works. Closest was a documentary on Football Manager but that only lightly touched on the code and more focused on the real world effects and scouting network. It must be some closely guarded secret because i know when Collyer Brothers left Eidos, they took the code and database with them.
It's not a request I've received all that often, but a lack of football games on the channel does irk me. I was actually introduced to some of the folks over at Sports Interactive a while back and tried to arrange something, but no luck thus far! Still, I'll keep trying.
I've used Fidelity FX with an NVidia card. It works fine, but not quite as good as native resolution. I can't compare it to DLSS as I don't own any games that support it.
A bit inaccurate, first step in DLSS is upscaling the image temporarily, a then it's fed to NN that fixes artifacts like ghosting and shimmering.
DLSS and fidelity FX will have a huge impact on VR due to the resolution required to have real immersive VR and you can already plug fidelity FX into any unreal or unity game with a .dll file made by Holger Frydrych who is a german modder. I've tried it and its night and day with various VR experiences and turns what can be a stuttery mess to buttery smooth. Now we just need a standalone headset with a bit more grunt, foveated rendering, eye tracking and able to use both DLSS and fidelity fx so we get the best of all tech (so probably an nvidia chip).
So can you for example use the .dll file for half life alyx? And how do you turn DLSS on if there are no settings in the menu.
@@Shiffo unfortunately half life alyx can't AFAIK, but other games that run unity or unreal. No setting in game you have a text file to dial in the settings, scale and sharpness and it injects that into the game
@@NabledMusic thanks, any VR games that you recommend using this for?
@@Shiffo skyrim vr so you can load up some nice mods and balance out your performance, I used it with beat saber for a test, it does make things a touch blurry around the edges but you can dial in the sharpness to mostly solve that
I've been enjoying FSR on my 6900xt but its cool to learn about the other side.
This honestly seems to be mostly WRONG when it comes to DLSS 2.0. Here is Nvidia explaining how it actually works: kzhead.info/sun/l5mknqyyrWJ_hH0/bejne.html DLSS 2.0 isn't using an deep learning trained algorithm to directly upscale and anti alias a rendered image like DLSS 1.0 did, but is instead a form of temporal anti aliasing and upscaling or spatial temporal super sampling that uses a deep learning trained algorithm for heuristics have sample distribution between the temporal information within the frames. Also Nvidia Shield is NOT using DLSS. It has a separate video optimized upscaling feature.
2:18 This brings back horrible memories for me of the near projectile vomiting I experienced playing Turok dinosaur hunter on the N64. That field of view was just awful.
6:25 Or it *is* development and you get the GTA Definitive Edition Remaster.
I really do wonder if they might make a dlss option that isnt anti aliased. I really like how aliasing looks and the fact that dlss gets rid of it anoyys me. Untill then i plan on just continuing to run my games at native resolutions so i get the sweet sweet sharp pixely edges.
I mean your image is getting super-sampled, that is inherently going to have an anti-aliasing effect
I'm just thinking: Nvidia: We made new tech that needs dedicated hardware to function! and its amazing you must buy it now! AMD: We made new tech oh and here's the source code do what you want!
It sounds easy, but FSR is far inferior than DLSS from technical perspective. Of course if you are a regular gamer, who cares how you get 10-20+ fps, it is worth either way. In the other hand, for example in VR, FSR can show its technological disadvantage, because DLSS reconstructs the image, because using a lots of data and VR puts the image very, very close to your eyes, you will able to differentiate the details between FSR and DLSS, because DLSS sometimes add more detail or keep it, but FSR always takes away detail, because how it works. But as I said, in a regular gaming scenario it is not that important.
Take a shot every time he says "the likes"
I honestly can't tell the difference between any of the comparisons. I know that was point for a lot of the comparisons of this video, that DLSS 4k takes less processing but looks the same, but honestly I barely even notice the difference with 4k upscaling or even when RTS is turned on. The processing is interesting but at the end of the day the point is to sell expensive hardware to nerds who like to see the number go up or down. I feel like there are more interesting applications of the catalyst tech behind DLSS because graphics are basically as good as they need to be. In general it seems like hardware is much more capable than the software we run on it.
I don't feel this episode does a great job? Like, we learn nothing of how DLSS works. We don't even learn things that we know, like the fact that DLSS 2.0 is a solution that disambiguates occlusion/contribution on motion vectors, and doesn't necessarily directly concern itself with pixel colour values. DLSS 2.0 emits contribution weights applied to the source pixels from preceding frames which are trivially resolved to final colour. Motion vectors are produced by all modern games for the purposes of multi-frame accumulation anti-aliasing (such as TXAA), and also for the purpose of motion blur. However TXAA and similar temporal anti-aliasing solutions exhibit excessive blurring, because the contribution of each pixel of source images on the final image is not trivial, because some of the source pixels end up representing partially hidden geometry which shouldn't contribute to the final image. Using depth data in conjunction with motion vectors helps, but is also error prone since motion vectors are usually 2D instead of 3D, and makes it much more computationally intensive. DLSS 2.0 appears to improve the contribution weighting while being able to run on dedicated Tensor core hardware, detracting little from the game performance. Potentially, it is akin to estimating the actual 3D contribution of the motion. It's also common to all multi frame reconstruction methods be it DLSS or TXAA that the implementation injects a certain sub-pixel magnitude per-frame motion vector into the rendering pipeline, to expose the visual information that would fall between the pixel sampling points. So fundamentally, DLSS doesn't generate any new visual information by itself, it leverages the renderer to do so over several frames. Now that FSR has been released in source form to the public, it has become apparent that it's an approximation of the classic Lanczos resampling. As a low order approximation, it probably comes out slightly blurrier than ideal, but it's followed up by CAS which hides this flaw and improves the appearance further.
Why stop at recreating an image at a higher resolution? Wouldn't it be possible to train a dataset for the game at max quality with all the settings turned up to max, and have an AI interpret that and output a maxed out game without actually rendering all that? You wouldn't even be limited by the game's graphics anymore, and could have a game look like anything at runtime. Kind of like those experiments where they take GTAV footage and do style transfer on it using IRL datasets. If something like that is possible, we might see graphics cards that are less "graphics cards" and more dedicated AI interpreters. Where you're still pushing polygons, but the final visual is based on whatever dataset you inject into it. You could even have different "flavors" for the same game by having different aftermarket datasets, thereby making the game look completely different. Honestly I can see that being the future of gaming.
Definitely possible to try to use AI to render a final image, easiest way to get started with this approach would be to start having AI replace screen space effects or deferred passes, could more easily replace SSAO and SSR. There's been papers written on Neural Network Ambient Occlusion or NNAO from 5 years ago.
Spending a top of the line GPU is kinda useless for now. I mean there's very few games supporting it.
In the distant future ai will be the one making the games lol
Believe me. You will not see it even when you live 10 thousands years more. They need to hire wizard for such thing to be a reality kid
It's almost like I can now download a better graphics card...
Are you scottish? 😁
Maybe... 😎
No I don't want to buy control, no matter how many times game 'journalists' show it to me
That's cute you call me a journalist.
yet another reason why console gaming is superior to pc. if you're spending this much time, energy, and (most importantly) money, on a top of the line gaming pc, then you're pretty much the same as those idiots who pour most of their paychecks into pimping out their cars, and as a result still live with their parents. Priorities my friend. Priorities.
The thing is that a graphics card is a great investment because you can also use it for content creation and editing/rendering video's instead of a console with is 100% dedicated to gaming.
I'm gonna be honest I genuinely cannot see the difference.
Just got into pc gaming a year ago. Personally I don't give a shit about graphics. I'm not gonna lie tho setting my graphics on high/ultra and getting 60+ on all of them..that shit Is beautiful lol especially control Jesus man played on ps4 for free and got it on pc for free. Boring game, but damn is it beautiful.
DLSS could look like crap sometimes.
Nah Looks great all the time
Your description of DLSS 1.0 works, but your description of DLSS 2.0 is quite inaccurate. You basically equate it to the same as DLSS 1.0, only with motion vectors and no more game-specific training. In fact it's a completely different and more or less unrelated technique. While DLSS 1.0 was a purely spatial technique like AMD's FSR, DLSS 2.0 is essentially a form of temporal upsampling, taking jittered samples at each frame to capture sub-pixel resolution, using the AI model to decide if it should be using the temporal data (projected from prior frames using the motion vectors) or spatial data (at lower resolution). nVidia did a deep-dive at GTC 2020 that's worth a watch: kzhead.info/sun/l5mknqyyrWJ_hH0/bejne.html
First
ray-tracing is still in its infancy, imho. most games have this weird look like everything has a thick layer of oil on it with ray tracing on, and this is on top of the flickering and the copious number of weird ray-tracing bugs and glitches. it's a cool tech, but it seems like the game studios need a few more years to figure how to properly use ray-tracing in their games. dlss 2 with the "one model to rule them all", assuming it's even one model, could actually bite everyone in the ass if nvidia releases an update that makes some games worse. only nvidia has control over it. imho, they should have open-sourced the algorithm and the sdk and made it so that each dev. studio trained their own model. whatever you train, you train for your game with your own tweaks and features. you don't even need "a supercomputer" to do that. why not even both? :-| it seems like everything that nvidia does, it's done to keep you locked-in into their platform.
It really depends on the game in my experience. There have been a few titles where it's honestly very impactful, and others where I honestly didn't give a shit. I think it's not only about using it effectively (though this is most certainly an important aspect), it's just that it's a technique that will always be more valuable for certain games over others.