KZhead
Ойын-сауық

I generated a photorealistic music video with AI (GauGAN2)

DoodleChaos
2022 ж. 1 Нау.
993 561 Рет қаралды
Жүктеу

I fell in love with AI generated landscapes recently. Although this tool is only meant for images, I decided to see if making a cohesive video was possible. Choosing which parts of the landscape would represent different parts of the beat was a lot of fun! Maintaining continuity from one frame to another is challenging, but after much trial and error, I've found that minimizing the motion of the horizon maximizes cohesion between frames.
To support these projects and include your name in the next video, here is my Patreon page: / doodlechaos
Discord: / discord (I spoil what the next video will be pretty often here)
Twitter: / doodlechaos
Official tiktok: / doodlechaos
Shorts Channel: / @doodlechaosshorts2905
Instagram Shorts: / doodlechaos_shorts
Email: contact@doodlechaos.com
Here is a link to the web version if you'd like to try it yourself!
Here is a quick explanation of my process for this project:
1. Animate segmentation maps in sync with music in After Effects.
2. Export this animation as a series of images.
3. Convert the image sequence into landscape photos via GauGan2.
I was able to automate the conversion process. My first attempt used AHK to move my mouse and click to upload the correct image, convert, download, and name. This was sketchy and actually crashed my chrome browser somehow (I think due to downloading so many images and they were all being cached?). Attempt two was to run it locally. Unfortunately, the NVIDIA Canvas downloadable software doesn't allow png segmentation maps as input (only .can files). My final solution works by sending images directly to the gaugan.org server via python. If you're interested in trying out something like this, I'll make the script I used available for download on patreon.
Resources:
www.nvidia.com/en-us/research...
blogs.nvidia.com/blog/2022/03...
Music Credit:
Track: Lost Sky - Vision [NCS Release]
Music provided by NoCopyrightSounds.
Watch: • Lost Sky - Vision | Du...
Free Download / Stream: ncs.io/VisionID
Shoutout to Winkey for giving me the idea to make this Bob Ross themed.

Пікірлер

  • That was super original. I love seeing experimental ideas like this!!!!!!

    @CreezyCreations@CreezyCreations2 жыл бұрын

    • Thanks Creezy!

      @DoodleChaos@DoodleChaos2 жыл бұрын

    • 🔥👌

      @DavidRichard69@DavidRichard692 жыл бұрын

    • Calfreezy

      @juancena8286@juancena82862 жыл бұрын

    • ​@@DoodleChaosI noticed that the link doesn't work anymore, can you get a new one (although it may be because I'm on mobile)

      @thinecyber_cat@thinecyber_cat7 ай бұрын

  • Now that was different! I loved the part at 3:20!

    @DaksDominoes@DaksDominoes2 жыл бұрын

    • Thanks! That part was the main reason for the epilepsy warning at the start.

      @DoodleChaos@DoodleChaos2 жыл бұрын

    • ok

      @Mereaux@Mereaux2 жыл бұрын

    • Me too! Best part

      @ledubois6@ledubois62 жыл бұрын

    • looked like an actual visualizer

      @calebcisco1641@calebcisco16412 жыл бұрын

    • @@Mereaux oh yeah yeah oh yeah

      @beep2997@beep29972 жыл бұрын

  • This is such an interesting medium. At first I thought it was a little weird but cool. But actually this is super cool.

    @oceanman207@oceanman2072 жыл бұрын

    • @Anna ♪ Jeez, you gotta warn us before dropping something like that. I also don't see the relevance.

      @JoveRogers97@JoveRogers972 жыл бұрын

    • @@JoveRogers97 It's just a bot. Report and move on

      @oceanman207@oceanman2072 жыл бұрын

    • @@oceanman207 I already did. My comment is not meant for Anna.

      @JoveRogers97@JoveRogers972 жыл бұрын

    • Its called acapella

      @bokunochannel84207@bokunochannel842072 жыл бұрын

    • Nice

      @masilovon7173@masilovon71732 жыл бұрын

  • He's using his actual powers to terraform the Earth now

    @not3852@not38522 жыл бұрын

    • Who Bob Ross?😂

      @jamiecurran3544@jamiecurran35449 ай бұрын

  • WOAH. That visualizer totally caught me off guard. Now that is amazing.

    @oceanman207@oceanman2072 жыл бұрын

  • watch out guys, they'll be coming for you as an instrument next

    @Comma2898@Comma28982 жыл бұрын

    • I wouldn't even mind

      @SwarumtheForum@SwarumtheForum2 жыл бұрын

    • @@SwarumtheForum in fact, that might be a good idea

      @bongosbongos@bongosbongos2 жыл бұрын

    • Yea

      @thatonecountryballanimator@thatonecountryballanimator Жыл бұрын

  • What an interesting way to synchronise music!! Each frame is truly beautiful, I see how you fell in love with it! Nice job!! :D

    @ducktoastthing3238@ducktoastthing32382 жыл бұрын

    • I'm glad you like it! Thanks :D

      @DoodleChaos@DoodleChaos2 жыл бұрын

    • yes

      @ahyes3398@ahyes33982 жыл бұрын

  • i've seen this AI but wow i've never seen it put like this :O this is so cool, especially those sound waves at 3:14

    @GroovyDominoes@GroovyDominoes2 жыл бұрын

    • Sup Groovy

      @Jevil-Gun@Jevil-Gun2 жыл бұрын

    • @Jax Hin ai is getting smarter 😂

      @aksvinss@aksvinss2 жыл бұрын

    • Npc getting more smarter than hoomans

      @JayLouis0@JayLouis02 жыл бұрын

    • @Jax Hin not npc, its ai

      @gent4626@gent46262 жыл бұрын

    • Plot twist: the npc made it

      @francodoraga@francodoraga2 жыл бұрын

  • It's really cool how the picture moves with the music the mountain moving remind me of my cat crawling under a blanket

    @Saji_0@Saji_02 жыл бұрын

  • Props to the AI for moving mountains, changing the terra and creating/drying ocean for this sick beat

    @evernarten42@evernarten422 жыл бұрын

  • This is such a creative idea. You even managed to seamlessly work your own logo in as an integral part of the video. And the little behind-the-scenes at the end!

    @EverythingTheorist@EverythingTheorist2 жыл бұрын

  • As someone who has played around with GauGAN a LOT, I commend you for your skills. It is truly incredible and hard to get consistent and looking how you want it. Well done.

    @ActualAshCam@ActualAshCam2 жыл бұрын

  • It's crazy that an AI can be used to generate such realistic looking landscapes

    @not3852@not38522 жыл бұрын

  • As someone who’s been following AI image generation for the past several years, it’s such a pleasure to be able to see it utilized to this degree. Very innovative and beautiful!

    @damonirvine8910@damonirvine89102 жыл бұрын

  • Awesome! Personally I love the part at 1:32. Every frame looks like a real landscape.

    @tsurugizaki@tsurugizaki2 жыл бұрын

  • What a clever idea! It's so cool that you could pause the video almost anywhere, and there would be a beautiful landscape. And the way the landscape flowed and changed was amazing!

    @alexlik4197@alexlik41972 жыл бұрын

  • Wow! That was trippy and cool - amazing what can be done with today's software...and even more amazing, YOUR brain for coming up with these ideas and then managing to execute them too! This channel is a deep well of happiness for me and I'm so grateful you keep making things to share with us! Thank you!

    @Beryllahawk@Beryllahawk2 жыл бұрын

    • won't be long before the AI starts coming up with these ideas as well

      @witext@witext Жыл бұрын

  • Okay, I heard the expression "rolling hills" before, but can we also use it for mountains and/or islands? Great video, as always! Very atmospheric. A little harsh on the eyes, at times, but a real treat nonetheless =)

    @Malthinae@Malthinae2 жыл бұрын

  • The beauty of nature + the beauty of music *chef's kiss

    @mateowinata182@mateowinata1822 жыл бұрын

  • Visualizer part was definitely the best, but generally the whole video just woah'd me!

    @Matiaffa@Matiaffa2 жыл бұрын

  • As someone who's already seen and used this AI, I know what it's capable of, but was not expecting something this good. Great job, can't imagine how long it took to make this masterpiece!

    @austinh1242@austinh12422 жыл бұрын

  • This is so satisfying. The video. The music. These images. This is what AI should be used for.

    @sizo02@sizo022 жыл бұрын

    • Kinda depressing

      @davidenorbedo5555@davidenorbedo55552 жыл бұрын

    • @@davidenorbedo5555 no u

      @Mylok_@Mylok_2 жыл бұрын

  • That's absolutely beautiful. Your artistry never ceases to amaze.

    @IronDino@IronDino2 жыл бұрын

  • Please do more of this, it pleased my synesthete brain SO MUCH!

    @thornsdale7443@thornsdale74432 жыл бұрын

  • Love the lil river dabs 1:36

    @someonekenobi@someonekenobi2 жыл бұрын

  • 1:58 "KoOraih" wise words

    @11teen08@11teen082 жыл бұрын

    • Just like this ratio!

      @I.-_-._.-_-._.-_-._.-_-._.-_-I@I.-_-._.-_-._.-_-._.-_-._.-_-I2 жыл бұрын

  • DoodleChaos getting into AI art is one of the things I didn't even know was on my wishlist for the world. Absolutely fabulous! Usually, I avoid supporting people on Patreon when they're already a huge success... better to help those where my contribution makes things happen that wouldn't otherwise, you know? But if you're going to get into hefty machine learning models, maybe you'll need it, haha.

    @Mnnvint@Mnnvint2 жыл бұрын

  • *Absolutely astonishing 😍😍😍*

    @manuelr.7461@manuelr.74612 жыл бұрын

  • Bro it's as if the nature is dancing wtf Like the river part Amazing!!! And then those sound waves! Damn dude!

    @omshetkar06@omshetkar062 жыл бұрын

  • Okay, that was really cool. I always enjoy your video's of you making some kind of way to sync with music, and this, out of everything you have made so far, has blown my mind. Keep up the amazing work DoodleChaos.

    @reconab12@reconab122 жыл бұрын

  • Its always a pleasure and an honour to be able to appreciate the videos you create for us to watch. That said, I would love to see some of this new tech accompanied by some classical music like in the older videos. Still, great videos :) thank you the hard work you put into them! :D

    @Wolforce@Wolforce2 жыл бұрын

  • How!, You make the most interesting and awe-inspiring videos EVER. This was different and in a very postive way. I love how the terraforming of and and sea synchronized with the electronic music to form eye-catching videos. Keep up the synchronization between upbeat music and the interesting visuals. :)

    @TryxnTX@TryxnTX2 жыл бұрын

  • The best part is that I could actually see a band creating a music video like this

    @Gaaraloverr1@Gaaraloverr12 жыл бұрын

  • the reasons ive subscribed to ur channel is all the different and creative ways u show the music. This video was a great example. Cant wait for ur next video!

    @GhamayYameen@GhamayYameen2 жыл бұрын

  • This is amazing! You're one of my favorite KZhead channels, I never know what to expect next but it's always so cool.

    @ZandoFox@ZandoFox2 жыл бұрын

  • Thanks for amazing work! I especially love the part where your logo pops up :) (oh and floral equalizer, too!)

    @sonohanabira9703@sonohanabira97032 жыл бұрын

  • Damn, patrick's rock has grown much older in this video.

    @gtdmg489@gtdmg4892 жыл бұрын

  • Another incredibly unique, extremely satisfying video to watch and listen too! Loved how you incorporated your logo 👍

    @margaretrobbins1549@margaretrobbins15492 жыл бұрын

  • Fun fact: This isn't actually AI generated. The music is DoodleChaos's gigachad vocals and instrumentals, and he simply transformed mountains, rivers, and forests to get the images.

    @irchonite1953@irchonite19532 жыл бұрын

  • It looks really good, congrats ! It's one of your best vidéo

    @yuxeo2513@yuxeo25132 жыл бұрын

  • And as always, doodle shoved us another very interesting approach to music! Very nice job man!!

    @burak1382@burak13822 жыл бұрын

  • So cool! My favourite bit was with the overview of the valleys. And when it "dipped" diagonally to the side 😄. Loved it

    @Merip1214@Merip1214 Жыл бұрын

  • Amazing stuff!😁👍

    @jamiecurran3544@jamiecurran35449 ай бұрын

  • I can't tell you how mesmerizing that was to watch. The beat of the music to the image's changes from frame to frame is strangely soothing while incredibly interesting. I can honestly watch this a thousand times in a row and not get bored of it! I would love to see another video like this from you someday, if you want to do it again. 12/10 Brilliant job!

    @novaliska8847@novaliska88472 жыл бұрын

  • good but nothing will be as beautiful and satifying as your works of absolute are you are the best at this doodle! i also didnt realise the stone henge thing whas the doodle chaos logo!.

    @natedinoffria1609@natedinoffria16092 жыл бұрын

  • Whaaat this is so interesting (like I've never seen something like this)

    @dayzm6708@dayzm67082 жыл бұрын

  • At first, i didnt notice wut it was at 2:48 but then i realized for it was your logo picture. That... was pretty awesome if i do say so myself! Props to you man! :P

    @xXMasterAzureXx@xXMasterAzureXx2 жыл бұрын

  • Great job! :) When you publish it always makes my day! You make the best music content in my opinion! Thank you for all you do for us! :) I appreciate all the time you spend on these video's and I'm sure a ton more people also do! :) Keep up the great work!

    @Enderglow@Enderglow2 жыл бұрын

    • Thanks Enderglow, I appreciate that, have a great day!

      @DoodleChaos@DoodleChaos2 жыл бұрын

    • @@DoodleChaos Thank you so much!!! :)

      @Enderglow@Enderglow2 жыл бұрын

  • Wooow! What a creative idea! And it turned out amazing! Watching you play with different media is so cool and inspiring!

    @Artsia@Artsia2 жыл бұрын

  • You keep getting better and better, love the change from what you normally do :D

    @VipersBiteGacha@VipersBiteGacha2 жыл бұрын

  • Wow, this is one of the coolest concepts I've seen in a while! Very mesmerizing to watch!

    @nintendork64@nintendork642 жыл бұрын

  • I've been watching youtube since 2012. I have seen a lot of things. I've learned a lot of things about how different programs and skills work. Generally even when I don't know entirely or are out right wrong about how something works I can still put forth a guess that I'm happy with. This is the first video I have seen in my entire life where I have absolutely no idea how it happened or how it worked. Like... you can paint a pretty realistic cloud to go over water if your patient enough... But then the grass and water reflecting trees... and the Trees!? Those trees are so detailed it must have been computer. But where how does the stuff move between it so well... Agh, at the end you revealed a bit with the colors and shapes and I guess the computer fills in those shapes with landscapes... but how are... some of those landscapes have to be from actual nature outside... but there's no way enough real world stuff exists to let the animation be smooth... I have no idea. This is amazing and I don't have to know how it happened. All hats off to you sir... or ma'am... or whatever you prefer Very well done

    @superguyrichard@superguyrichard2 жыл бұрын

  • THIS IS AWESOME!! I wanted to make something like this for so long, to see it out in the world by a youtuber I love is amazing!

    @analye6814@analye68142 жыл бұрын

  • Absolutely fantastic work and great choices

    @andythemany2826@andythemany28262 жыл бұрын

  • I CRY that this only has 1/8 million views. DoodleChaos you are someone specially gifted. I don't know if ANYONE has done anything like this before.

    @FrancisHatesStairs@FrancisHatesStairs Жыл бұрын

  • Very awesome video. 3:00 it took me a while to notice it was the doodlechaos symbol, but when I did I thought it was cool. Looking froward to more music syncs, and do like the use of NCS songs. I like a lot of NCS songs, but some I don't, so far you have used some I like!

    @thunderatsea3843@thunderatsea38432 жыл бұрын

    • (edit, ish. The moving rock I thought was odd, but it works. The clouds do work better for moving though.)

      @thunderatsea3843@thunderatsea38432 жыл бұрын

  • Thanks for continuing to make such cool stuff!

    @Vesperon@Vesperon Жыл бұрын

  • 3:20 the plants representing the frequency spectrum absolutely blew my fucking mind that was insanely cool

    @TebiByyte@TebiByyte2 жыл бұрын

  • I thought you did each fame by hand but the animation in after effects to an image sequence is a complete GENIUS idea. I don't know how you do it. This is incredible.

    @Arieljaay@Arieljaay2 жыл бұрын

  • Incredible! :D Good job adding your logo in there LOL

    @ElectronicOkapiMusic@ElectronicOkapiMusic2 жыл бұрын

  • This is super cool. It’s incredible how amazing your video are.

    @ArmiYT@ArmiYT2 жыл бұрын

  • Incredible job! I think it puts my old experiments to shame!

    @jonnyjive5@jonnyjive5 Жыл бұрын

  • Really cool, I liked the part with the doodlechaos logo changing around!

    @JollyJakeS@JollyJakeS2 жыл бұрын

    • Hey Jake!!!

      @DoodleChaos@DoodleChaos2 жыл бұрын

  • Yooo, the visualizer looks so cool!

    @knicklichtjedi@knicklichtjedi2 жыл бұрын

  • No matter what the medium, DoodleChaos shares his opinion to people through music and visuals without fail.

    @RussellScott59@RussellScott592 жыл бұрын

  • Great job man, this is mind blowing

    @galaxystudiosgaming@galaxystudiosgaming2 жыл бұрын

  • I needed that smile today... thanks for the hard work as always, Doodle 💐🐱💐

    @daQuietsho@daQuietsho2 жыл бұрын

  • Underrated and amazing. Great job.

    @mrrfyW_@mrrfyW_2 жыл бұрын

  • wow this is really taking up a notch! hope to see more like this

    @adamrowedotcom@adamrowedotcom2 жыл бұрын

  • pretty different, why not? turned out better than i expected lmao good stuff as always

    @jerecakes1@jerecakes12 жыл бұрын

  • This is so original and unique! What a cool idea. I can't stop watching.

    @chmeebs@chmeebs Жыл бұрын

  • I actually cracked up at the mountain lake whipping around the valley (1:30) 😂😂 Good job, AI

    @JoveRogers97@JoveRogers972 жыл бұрын

  • Wow! This must have taken a while!! And it looks so beautiful!

    @Kay-ql2wl@Kay-ql2wl2 жыл бұрын

  • When the song drop so hard the mountains deform

    @vxunaTM_xuandanhphan@vxunaTM_xuandanhphan2 жыл бұрын

  • I loved the plant EQ. The AI came up with some interesting interpretations for your animations. I love it.

    @speeding2fast@speeding2fast2 жыл бұрын

  • Wow! This is incredible and underrated!

    @choompymolly@choompymolly2 жыл бұрын

  • Your creativity is simply inspiring! Well done!

    @PaulaHoell@PaulaHoell2 жыл бұрын

  • This was a very interesting way to do an audio visualizer

    @superNova5837@superNova5837 Жыл бұрын

  • Never thought about using gaugan this way. Wonderful ! :D

    @Mylok_@Mylok_2 жыл бұрын

  • That! Was art! You really amaze me and I am in love with your work! Keep blowing peoples mind! You rock!

    @TheBrownFox3@TheBrownFox32 жыл бұрын

  • The way it does this is so trippy! Corridor Crew needs to see this

    @ieatorangepaint@ieatorangepaint2 жыл бұрын

  • Now this is epic (and creative too, love it)

    @user-ij1ox7yj8p@user-ij1ox7yj8p2 жыл бұрын

  • This idea of use from the release of that ai is the most original i saw Very nice job

    @comeycallate9959@comeycallate99592 жыл бұрын

  • Gorgeous and really cool!

    @chillseeker@chillseeker2 жыл бұрын

  • The AI does such a good job at making drawings into beautiful realistic landscapes 😻😻😻

    @JonnyDiggit@JonnyDiggit2 жыл бұрын

  • Damn this is so cool! I love everything about this! 💕

    @Nasrul260@Nasrul2602 жыл бұрын

  • every time I think you can't up your last video, you pull out something like this. Absolutely insane

    @bigmasterbob2060@bigmasterbob20602 жыл бұрын

  • This is super creative and I love it!

    @optical6816@optical68162 жыл бұрын

  • 3:12 The floral equalizer was ✨✨✨

    @jmag579@jmag5792 жыл бұрын

  • This is so cool. Subbed!

    @NanetteNette@NanetteNette2 жыл бұрын

  • Just amazing! Make more soon plz!

    @repairstudio4940@repairstudio4940 Жыл бұрын

  • This is amazing!!

    @tinom.@tinom.2 жыл бұрын

  • This is AMAZING!

    @DaCatmasterX@DaCatmasterX2 жыл бұрын

  • I thought "wow, this is really different content from Two Minute Papers," then I saw which account this was. Amazing stuff!

    @IanWatson@IanWatson2 жыл бұрын

  • this is awesome, im def gna try this out for myself. u just put me onto something big, thanks bro 🤘🏻

    @cocique@cocique Жыл бұрын

  • Not gonna lie, that is honestly really cool.

    @DeerJerky@DeerJerky2 жыл бұрын

  • Just amazing!

    @SpacEagle17@SpacEagle172 жыл бұрын

  • I was thinking about this same music today and see this... is just magic. Good work♡

    @Night-TW@Night-TW2 жыл бұрын

  • It’s very amazing , you are so smart and ingenious ,this video is fire 🔥🔥🔥

    @lecactusenrage9899@lecactusenrage98992 жыл бұрын

  • Broooooo this is amazing never thought you could do this, this is amazing how long have you done this!? And the best part Was in the end at the drop!

    @xyler4852@xyler48522 жыл бұрын
KZhead