Virtual Production at Home with an iPhone and TV - Unreal Engine
2024 ж. 2 Мам.
85 732 Рет қаралды
I figured out how to do a basic virtual production setup at home without expensive VR trackers, using just your iPhone
I think it'll be incredibly useful to me personally so I'm sharing it with all of you, cause currently you need to get pieces of information on this from a whole bunch of places that aren't all up to date.
Enjoy!
Follow me on Instagram:
/ notquitemax
Timestamps:
0:00 - Demo
0:30 - Intro and Foreword
1:33 - Installing Unreal
2:03 - Project Setup
3:12 - Environment Assets
3:55 - Scene Setup
6:29 - iPhone Setup
7:53 - Magic!
8:12 - One Last Thing
8:33 - Filming Setup
8:58 - Zero-G Acting
9:13 - Foreground
9:32 - Making the MCRN Cup
9:53 - Foreground Challenges
10:30 - Final Thoughts
this is incredible! That fact we can get so close to the quality of a volume at home, so freaking cool
dude - that error fix!!!! WORKS!!! finally!!!! took about 6 month to finally get her - thank you
Great vid. Your advice about prioritising the importance of the foreground 'stuff' was invaluable.
THANK YOU for this! So much more straightforward than most of these! ❤❤❤
Keep up the grind, mate. I'm glad I found this channel.
Awesome tool. Look forward to trying it.
This channel is criminally underrated. Holy jeezus. I swear it's always the best channels that have less than a 1000 subscribers.
Thank you 🙏🏻 😌
The best channels have less than 1k subscribers because they haven't tasted monetization yet. Which means they only have to output good stuff to get recognized. As soon as they start earning more and more money, enshittification ensues and they start caring more about the money, because by then, they can.
@@_dominick I’m eligible for monetization as of this morning. I’ll continue to only make videos when I have something valuable to say 🤷🏻♂️
This is insanely cool. Going to give this a try for my videos in the future.
great technical and content quality my dude
Thank you mate! It’s just awesome. I just start to research vp. Thanks!
welcome to the new era of youtube! Your sir are one of the fore runners!
Thank you so much for this! this is awsome!
the acronym too! god damn. You have such a good eye for visuals and god you are talented. Thank you so much for sharing your knowledge!
It's the most sci-fi clothing item I own 😁
Love this! Nice job!
That was cool. Nicely done sir.
Very cool, thanks for making this entertaining
I love this! I managed to use Connectify to directly wifi connect between my iPhone 14pro and my windows laptop and this works like a charm with very little lag I might add! Thank you so much for figuring this out! This is an excellent setup to teach Virtual Production to my students and very little cost!
That's awesome! Glad I could help 🙌🏻
Woh.. Subbed.. need more. Amazing job
wow! it looks amazing!!
Love this bro!
Great video, really cool stuff!
Dude this is next level!
Thanks! 🙌🏻
That was amazing
Great work!
At 10:17 you caught me off-guard and that reminded me some hazardous installation I also had in the past 😂 Brilliant video nonetheless, thank you for that!
Your c-stand has so much character 🤣
WOW, this is cool!
i did something very similar to this many years ago using johnny lee's head tracking system using a Wiimote. Very cool stuff! ty for sharing!
Wiimote was ahead of it's time
That's awesome! And your haircut is not that bad, btw.
this guy just changed my future... subscribing is not enough... how can i support you BROOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
Stay tuned 😉
Now that’s cool. Thank ya
This video is AWESOME!!!!!!!!!!!!!!!!!!!!!!!!!!!! Thank you so much!!!! Your jokes are top notch, totally lmao 🤣🤣🤣🤣🤣 I am going to try this at home. Thanks again!!!!
dude i like your STYLE BOY
If you hava a Datashow and a reverse projection screen, you can enlarge the background to a big wall. Need just space enough behind it.
Nice thanks!!
One of the best step by steps i've found been trying to learn unreal engine hopefully you do more longer tutorials on unreal film making and more also if you recommend any other videos to watch or paid courses that would be great.
There's definitely more coming, but I want to get more familiar with the engine stuff first I feel like the second video I made is as far as I can take the current "stretch the viewport" setup and I need to look into the proper ways of doing things more The best sources of info I've seen are the official Unreal Engine channel, Epic Games forums (helped me figure out the pixel streaming vs. remote session issue) and just random people on Reddit :D
Thanks for showing the process! Would a polarizer filter help you with reflections on the TV? That's the only thing that I found jarring about it
As I mentioned in the other comment - it might, but depending on the exact TV you have, it might also remove the entire image from it :D Mine is a Samsung Frame 75" and it goes black with a polarizer while retaining like 60% of the reflections 🤷🏻♂ Your mileage may vary
Those reflections weren't too bad with just the talking headshot! Would love to see another video on this, optimizing out those reflections.
A projector would basically solve it, but with a TV the lights might need to be pretty far away depending on the shot, which isn't always viable
@@MaxRovensky Hey, great video! Would a polarizer filter on your camera perhaps help with the reflections?
@@asolidmindproductions it would, but depending on the TV it could also help with removing the picture from it 🫠
@@MaxRovensky Haha true. Guess we’ll just have to test it out! Again, great video and thanks for the tutorial, its really appreciated!
this video is fucking great! I have the excact projector u showed and mainly use it for rear projection but ur method looks super interesting… definitely gonna try it out!!
I liked that hit button
$100 green screen sheet and you can do full body shots, I think it’s still better than rear-projection especially for indie creators
Yeah but I like the in-camera approach cause all I have to do is grade and deliver, I don't need to spend time cleaning up spill 🙂 Green screen will eliminate reflecting though which is an issue on consumer TVs
Wow !!!!
really cool hack!
Do you have more tutorials on this subject as I find you very clear and concise only question is how to get the background on to the tv screen is this just using casting like we use for youtube videos to tv??
I filmed one for how to do a more real-world project with this, coming soon, hopefully this weekend 🙌🏻 Re: getting it on screen, I did it via HDMI cable for this one and via AirPlay in the upcoming one. Cable has less latency so that's preferred Also I'm not doing it the "proper" way, there's more setup for that that I haven't gotten around to (nDisplay and all that), so I'm just moving the Unreal window to the TV display and collapsing all panels so the viewport is as large as it can be
@@MaxRovensky ok I understand now. I did see tuts on that nsai or whatever but they do lousy tut where you explain things so well. I look forward to the upcoming video tut. Thankyou
just saw this reply. will try this@@MaxRovensky
Came from FilmRiot, ...amazing Tutorial! So you need to correct perspective in UE5 because the iphone is a bit over the camera sensor?
I think it would only matter if you're using a longer lens At 35mm the distance between iPhone and sensor isn't noticeable enough to matter but at 50mm+ it would probably affect the shot Alternative solution is to attach the iPhone directly to the back of the camera with like, electrical tape 😁
Love your work man. recpect . do you think this might work with gyro system in Blackmagic cinema pocket too? 🙌
I don’t think there’s a way to stream gyro data from it in realtime and even then UE needs a specific flavor of data to be streamed It’s possible to run the clip through Gyroflow in post and export the motion data in its format, so there’s a chance someone will engineer a way for UE to ingest that and apply to a camera path, but I haven’t seen such a solution and it would definitely not be realtime 🤔
Would it be possible to record the virtual camera in UE without all the gizmos and details of camera on the screen? Thank you @@MaxRovensky
@@motoristube Check the followup video for removing gizmos. As for recording in UE - it's definitely possible, but I haven't looked into it yet
Excellent video, thanks for sharing it, how do you send the image to full screen without all the information on the screen? do you run ndisplay?
Not yet I just fullscreen the window and collapse all the panels, so the viewport takes up 95% of the vertical space and 100% horizontal You also need to disable sprites, see my latest video. It doesn’t show up in this environment, but likely will for whatever you’re using as your scene
@@MaxRovensky Thank you very much for your response, I will watch your latest video, thanks again
What I don't see is how to remove the rule of thirds graphic and the other icons from the Vcam actor
@@lucaia9710 8:12 🙂🙏🏻
❤❤❤
I spotted those fpv drones on your desk! Next is tracking an fpv drone in unreal... I have tried with a different approach but didn't have a big enough drone to carry a phone at the time... haha
I mean, it probably makes more sense to just make a motion track for the camera in Unreal at that point 😁 The selling point of FPV drones is they let you get angles only possible with CGI, but if you’re already doing CGI - might as well go the easy way haha
yah but you wouldn’t have the bragging rights
Fantastic! Thank you. I now need only to change my Android for an iPhone or to get a PC if I understand correctly? 😀
The OS that runs Unreal doesn't matter, but at the time of making this, the LiveLink VCAM app was only available on iOS That might have changed though, I haven't looked
Looked amazing.... except for your reflection on the TV screen. How would you do this exact process using a screen and a projector? Would it cast a shadow on the screen?
At that point you'd probably want an ambient light rejecting screen, which counters that issue, and because the projector has to be short throw for this to work, the screen would _technically_ be a light emitting surface that would counter the shadows, unless you put a light that's brighter than the screen such that it casts an obvious hard shadow onto the screen
@MaxRovensky So I'd be better off with a 75-inch TV, then...How would I get rid of my reflection on the TV then? I saw yours when you did the two passes before going off the screen...
@@ITSEGFILMS you _might_ be able to use a polarizer, but it depends on your TV As I’ve shown in my follow up video, in my case the polarizer cuts the image from the TV before it cuts reflections, so your mileage may vary The approach that worked best for me so far was to use a scene that’s less dark and place lights at an angle that points away from the TV more
Nice video :) What screen recording software do you use?
Screen Studio by Adam Pietrasiak
I like the idea of using the iPhone as a live view camera tracker. I’d like to use the same idea but using a green screen so I have no reflections. So basically shoot in real time with a film camera against a green screen with the iPhone mounted on top but displaying the unreal image via a monitor. Now that for me would be very interesting
That's what I've seen a lot of people do with Unreal - record the motion track, render out the scene offline and composite via green screen. Might explore it in the future since apparently I am an unreal engine channel now 😅
I’m actually following an account who does it in real time on a green screen. I’ll find out how’s it’s done and share
Have you tried using a polarizing filter on your camera to eliminate or minimize reflections?
@@andrewwelch5017 yes, doesn't work in my case (see my last video) but your mileage may vary
@@woodswork Hi, is there a video somewhere showing this green screen approach?
Hello there! Thanks for this wonderful tutorial. But you said at the last part that I just need to plug in the tv. So I assumed that when I do that the virtual background will just go full screen automatically on the TV but mine didn't work. Am I missing something? Thanks for the help :)
Nope, what I'm doing is just expanding the viewport as far as I can, it's still showing the engine's "shell", I'm just filming underneath it There is a "proper" way but I'm still investigating it, tutorial coming in the future
@@MaxRovensky Hi thanks for the reply. So what you are saying is I need to drag the whole UE window to the extended monitor/TV?
@@MikeyNavarrete yep, and collapse the side and bottom panels
Great video btw, even if my misophonia was raging with the swallowing spit sounds 😩
I did not realize how audible it was until it was already uploaded it 😅 It'll be better next time!
@@MaxRovensky noo! You’re totally fine, it’s us with this condition. It’s maddening. Your video was excellent and helpful
Great video. Super funny. Cut the dialogue (not tutorial) faster, less silence makes the pace easier to follow
If you're ever in Chicago, you'll have to let me buy you a frosty pint of thank you!
Hi, is it possible to set this up to use the iPhone as the actual capture camera as well? Rather than needing a second camera?
I'm 50% sure it will stop transmitting the gyro data when it goes to the background, but I haven't tested it
@@MaxRovensky okay thanks for taking the time to respond. I’ll do some testing in the next couple weeks and report back here if I find anything interesting
So you are using on a Macbook Pro then correct? I've been really weary about switching to Mac because I've heard horror stories about using Unreal with it. Can you tell me your experience please? Thank you! Great vid!!
Complex scenes start dropping frames after about 10 minutes (baseline M1 Pro) I just upgraded to M3 Max, haven’t tested it yet, but should be much better
@MaxRovensky if you could do an M3 Macbook Pro review of how it handles UE5 you'll be like the first online to do so.
brilliant ! why didnt you shoot w the iphone ? loved the story too... cheers from Rio !
I not sure if the app will stay connected to Unreal when in background, haven't tried it 🤔
Now you keed to get a "reflection free" TV, and you're set!!
is there a lag between camera motion and the projection? were you able to get rid of it?
Yeah, shutting off stuff that's out of camera helps, also not using a laptop on battery power 😅 This is purely a hardware constraint, the better your hardware - the more you'll be able to run at >24fps
oh! I thought it's due to the wireless connectivity. Thanks.@@MaxRovensky
@@YanivCreations there's slight wireless latency, it's kinda immovable because it's using Wi-Fi instead of a custom RF protocol like VR trackers would, but so long as you don't do any super fast pans - should be fine The delay you're seeing in this video is definitelt just framerate drops
Hey just subbed. Was wondering why when u stsrt talking then showing every1 your space ship, something goes wrong. Looks like your not grounded to thd floor. Would live to know how and why this is happening. Its a strange optic that breaks the illusion.
The frame rate drops a bit at that part, the movement is a little behind and choppy It's a hardware issue - I'm using a battery powered laptop to render everything, which isn't a great idea 😅
Nice 😂
where are you based in max?
Its cool to me
plug in to tv and "film stuff" seems like it's skipping some steps. is that an nDisplay set up or a second monitor ?
Nope, expand the viewport as much as you can, collapse all the side and bottom panels and disable gizmos (I mention it in the follow-up video) I'm looking into the proper nDisplay setup, but this was perfectly sufficient for what I was trying to do, only sacrificing about 3% of the top area of the TV
Which iPhone is required? Any iPhone works? Or only iPhone with Lidar scanner (pro models) Please confirm.
So long as it can run the app - yes. Any iPhone that can run at least iOS 16 should work, so anything from iPhone 8 up theoretically
@@MaxRovensky Thanks
hey, after adding Vactor i coundnt resposition my camera. it just goes in forward and backward direction no right, left, up or down.. how?
You should be able to use arrow keys and mouse, just make sure that proper camera is selected (the one inside vcam actor)
This will only go so far for a film lol. But backgrounds for a static talking theme, why not lol
The more serious you want to go, the bigger display you need - bigger TV, projector, dual projectors, all the way up to a proper LED wall setup
How would you use a projector without casting shadows? Seems like an insanely efficient low-cost solution compared to renting an LED Volume for $50,000 / day lol...
Ultra short throw, they're set up like 30cm from the wall, you'll pretty much always be in front of it
So it's only rotation right? Cause it's visible when you finished setup the iphone.
It tracks every axis, but forward/backward motion isn't as noticeable at 35mm as when you pan
@@MaxRovenskyhm... doesn't looks like moving at all at 08:09 for me... anyway... good work
@@gudeandi it does, but baaaaarely, not enough travel distance to be meaningful
Does it cross over to PC's and Android phones?
PC most likely will work fine, but the LiveLink VCAM app is iOS-only at the moment as far as I remember Though that might change or someone might make a 3rd party one, I haven’t checked
Why not just use Aero or AR tech ? Would do the same thing with a far less learning curve?
This would be even better if i didnt see your reflection in the tv in the final product
The scene is mostly glass so it’s technically not awful 😁 But yes, I mentioned it in the follow up video, using a projector solves this
The question is can you do it on an android device?
As far as I know the LiveLink Vcam app is iOS-only at the moment, though that might change
Looks great, to bad you had the reflection of you in the first seconds, that disperses the illusion a bit.
Yeah, it's hard to get around it unless you have a TV that's coated against reflections or a projector. It kinda works here I think cause those are windows and metal beams, which can be reflective/glossy 🤷🏻♂️ But yeah, something to keep in mind for other types of shots
can i use ipad instead of iphone? i dont feel invested in iphone.. feels like way too overpriced for the hardware.
The app should work, but mounting an iPad to the camera is probably gonna be annoying :D
@@MaxRovensky yeah, but android have much powerful hardware in same price.. but the app is apple exclusive for now it seems.
@@ALOKSHARMAMD I don't personally care about marginal differences in hardware I use iOS because of the ecosystem, Apple Log and software availability (such as this app for example 😅)
I've built a career on trying to do this without any of these things and now you can do it with a TV and a fkin phone! Of course, now everyone can do it, it will make the whole system obsolete and worthless and onto the next thing we'll all need to learn and lose our minds over.
The Mandalorian bad?
It’s visually great, but unfortunately virtual production can’t do anything to help the writing 🫠
10:20 Shitting Rig as best ;-)
I call it a g-stand It's like a c-stand, but ghetto
A cup In zero gravity in general is questionable.
Just make sure it's empty before you cut thrust 🤷🏻♂️
@@MaxRovenskyThe fluid inside a cup In zero gravity would most likely stay inside due to tension. You would have to assume a straw to sip it out because pouring it out is not a option.
I am about to have to explain to my accountants why I bought a projector for nearly 5K 😂 also shitty rig for the win
If you do actually end up buying the projector - hit me up 😂
Just a couple of steps lol
Why all this click-around through the entire environment? It'd be so much more streamlined if there would be a wizard (or just ONE panel) collecting all relevant settings for such a project 😐
You could definitely have multiple VCam objects set up around the scene and switch between them as needed, but you still have to navigate around the viewport first time around to place them I haven't looked deep enough into it, but if Unreal supports camera switch transitions similar to Unity's Cinemachine, you could do some very interesting things by switching a handheld operated camera to a different perspective I should look into it 🤔
please don’t start any tutorial this level about how to install Unreal 😂
Hey I'm documenting my own process 😁
Looks like an iPhone production
Still no reason for me to buy a apple product😅
You can do exactly the same with any other phone
@@ZwaetschgeRaeuber how?
Do you feel edgy now?
@@ZwaetschgeRaeuberyou can’t, as of rn iPhone is the only smartphone to have lidar. The iPhone is solving the camera real time using apples in house developed ARKit and the rear mounted lidar sensor. Google has done some limited stuff with AR and basic camera solve but there are no lidar sensors present on any android device and probably won’t ever be as it’s a very niche market
Dont think so
The unreal engine sure literally said it’s not for mobile devices smh
Unreal Engine has had plenty of mobile features for the last decade.
Bro…don’t do this to me…don’t inspire me…don’t make me buy more stuff and start new projects when I have 5 I haven’t finished yet…
dewit 👀
lol, this is so bad.. you're diving into a TV, I mean.. your reflections are everywhere wtf
Good thing that windows and smooth metallic surfaces are both reflective 😉
Sorry bro. It's looks amateur.
Cause it is 🙂 Gotta start somewhere!
This is guy is trippin. It looks awesome, mate! I'd watch a whole series like this.
@@MaxRovenskybrother only the FACT that you can do this tracking with the iPhone (while a year ago you needed special tracking equipment and camera to match the frame rates) is INSANE. I’m so glad i bumped to your video. Thumbs up big time! It just gave me so much things to work and think about
It is amateur but the concept is there. Now make it better. Look at all vfx before and now where they are. Build on the concept.
It's an iPhone and a Tv screen offcourse its going to look ameteur compared to a billion dollar film studio with million dollar lighting and million dollar props and practical effects.