So a little bit about me.
I’ve been interested in Lightfields and 3d conversion for around 2 years now.
Finally I pulled the trigger on a Lumepad 2 like a week or two ago.
I’m digging a lot of the images.
I like to enhance my own depth maps, and eventually I want to convert whole movies into 3d files for the tablet, I have this dream at least from my personal library of being able to see anything in the 21st century equivelant of “Turner Vision”.
Anyway, I’ve noticed there isn’t just a distinct photo viewer, just something where I could playback a 2x2 image.
Recently I’ve discovered an application for depth map production called “Depth Anything”.
Which far surpasses anything I’ve used before.
If possible I’d love to see this integrated someday.
I use a mix of Blender and Keentools to do a lot of my work, then I edit them together in GIMP, I’ve used software built by Ugo Capeto to visualize the depth maps.
I’d love an app like this that could visualize depth maps and images together. That way I could find flaws and edit them.
A lot of the work I do requires a desktop, and I’m cool with that, but I’d love to be able to demonstrate the raw power of depth maps.
AI depth maps generated (Specifically the automated variety from Leia, tend to have a concave appearence when compared to some of the ones I’ve made), I’d love to share them.
Specific apps within this tablet refuse to playback 2x2 images, and I don’t have enough understanding of the format to combine a custom depth map I made with a photo to get an output directly off of the display, I’d love that.
Finally, using a Depth map of the custom guided variety, for either a video, or a photo, adding new lighting FX in order to make the image pop more, or some kind of AI recognition of texture on a mesh from the depth map, would definitely add some pop to post conversions.
One issue is, I tried playing back depth maps in depth player, that were in the full color variety, yellow to purple to black, and it never read them correctly, it always read them as concave or otherwise.
So, one thing I’m curious about is how the Leia software reads a depth map, and generates a correct image.
I had to correct the depth map, by de-saturating it, in order to visualize it and see it properly. I’ve always enjoyed seeing depth maps and images as point clouds that I could rotate around.
I’m curious if the tablet’s software is reading full color depth maps, through a grayscale analyzer (Or whatever you call it) and that’s why some automatically generated images have a concave appearance.
I’m learning that with LIF files, it wouldn’t recognize 2x2 images, unless I uploaded that onto the social network component. I found it inconvenient.
WIth the LIF format that problem appears to be resolved.
This looks pretty cool. I’m gonna check out the code. That said, the LeiaPlayer can convert images/videos to 3D no problem. You can definitely copy full 2D movies and watch in 3D, in real time, no need for an off-line conversion. Just load up the content in LeiaPlayer and click the 3D button on the bottom right.
All of Leia’s displays are multi-view, but we just focus on stereo 3D content on Lume Pad 2 because we found it was the vast majority of content viewed on earlier devices (over 90% of usage was 2-view) and focusing on multi-view made stereo 3D content suffer.
You can still do multi-view on Lume Pad 2, but you must use Leia FaceTracking to deliver it to users using the Leia SDK.
A good example of this is CakePlayer on the Leia Appstore on Lume Pad 2. It’s a true multi-view light field video player! So you can experience multi-view imagery on Lume Pad 2.
I think it uses MiDaS via AI, since I’m trying to do stereo conversions on films and I’m doing tests using AI and face reconstruction from monocular vid, I’m seeking out the best.
Like, I want to play them back as point clouds to review them.
The goal is that you could pop it into a lightfield display or in the future, a Smalleygram device. But I want depth maps at 1080p with vid, but I have a long list of details I’m putting together for this.
When I know I’m not stupid, I’ll share it, although I’m still questioning whether or not I’m stupid lmao.
Well, I’m not exactly doing stereo conversion, I want more out of my work.
But I’m using monocular video and monochrome depth maps a lot of the time, and I find there are limitations to those.
I think that a depth map with the detail I’m chasing could assist in object tracking and upscaling, but again, everything I’m playing with is from a monocular source, it’s not natively shot in 3D or on a lightfield camera, so I’m rebuilding details a lot of the time, or kitbashing.
So, take a scene from LOST season 2, I’ll use a depth map of the SWAN hatch, I’ll rebuild Terry O’quinn’s face or Matthew Fox’s face, and I can get good results for still images.
Eventually I want to move into video.
I do this in Blender with the tools I have. I want to watch it on a multi view display.
But, I want to overlay different results, and enhance the final depth map with an embossing technique.
I want the depth map to be metadata, and the LIF format is the ideal format as far as still images are concerned, IDK how it compares to Apple’s HEIF, which also has depth metadata, but Apple doesn’t offer a multiview display with a lone excaption on the front of it’s headset, only for you to see the user’s eyes.
Then there is one other reason I’m doing offline conversion.
It’s also about enhancement.
With shows like LOST, which are legacy shows, I want to be able to
de age actors in flashbacks using deepfakes and I want a kind of hyper realism.
So I watched “The Irishman” and while not satisfied with the end result of their de aging tech, I looked at what ILM did, and I was very intrigued with how the sausage was made.
I then saw a deepfake overlayed on the Irishman, and concluded that deepfakes work very well, as long as the layering is done right.
Kind of like building a floor or anything.
Otoy started a project with Trek, which I was absolutely giddy over.
So I’ve been compiling information on that as well.
What I’m seeking there is to further inform the deepfake using a depth map, so I can get a more convincing illusion of watching someone like Locke age and wither through time before he makes it to the island. Or even get John Noble in Fringe to look a bit younger in Fringe.
This is where I think I’m dumb, because what I learned from that was modern cinematography as Christopher Nolan said “Has Depth” and what he meant is, with the invention of steadycam, blocking, and more chaotic cinematography, you can enhance the viewing experience on a monocular display.
Television has been doing this for sometime, with the way the refresh rate works, you can literally halfway see a stereo image as light moves to your eyes, it tricks your brain.
What intrigues me about these autostereo/MV display devices is that they can also train your eyes to better perceive real world depth, it’s a workout for your visual cortex and you’ll catch yourself doing things. While this probably wasn’t a problem prior to the 1980s, with the growing consumption of media, and using more and more screens, I think our visual cortex has been a bit stunted and we don’t perceive depth IRL the way we used to.
Please forgive me for the long winded explanations, and the wall of text.
I’ve also noticed that at least on Lumepad, native stereo images look stunning.
Whereas my conversions don’t necessarily look as good, they’re enhanced.
So I’m trying to see what I can do to make a monocular image look even more convincing after conversion.
It could be the use of depth in the image, but the images I convert still look a bit flat.
Good, but flat.
I want them to stand out more. I want them to look like they were natively shot in 3D.
Thanks for the info. Yeah, I’ve been working with 2D conversion since last year. Even generating stereo is a challenge due to disocclusion. Even with a perfect depth map, you still can’t see what is behind objects in the foreground. With very intensive algorithms you could cache temporal data (e.g. what is behind an object may have been visible in a previous frame or a future frame) though this takes a large amount of memory and likely can’t be real-time right now. So generating 4 views seems like a stretch without lots of artifacts.
I have some ideas for that, as you stimulated the discussion between me and my brother.
My interest in video started in '05 when I was 15 and learned what Bluray was going to be.
I’m building a little hardware arsenal up to do what I want but the work I want to do requires
Morphable models
Moving face reconstructon
a very fine tuned Z-pass, (Blender)
(There are things I can get from 3d informed depth passes, things I can get from manual segmentation, and things I can get from AI, I try to combine them.)
I might make 10 depth maps per image, and then make on average 15 corrections before I have what I call an epoch or foundation map for the next.
I did one of Picard, that I have on the Leiapix app right now, I think that’s based on my 6th map.
The One I just made is PDX.png, PDX2.png, and PDXWild.PNG
Each one is numbered, PDX wild incorporates RGB data from the main image and worked far better than I expected, I think you could use it to raytrace out some minor reflections in the image to make something more stunning (although revisionist).
An Emboss with no alpha channel (Look up total moving facial reconstruction)
3d models when availalble.
In Gaming there is a tool called Reshade and it gives you a depth map per view, I’m thinking you could render subviews for that. I’m a big Fallout fanboy (It’s so Shlocky and darkly satirical).
I think I’d need a stereo depth map, I also attempted to restore the color depth map, the purple to yellow, and after my enhancement, got some astonishing results.
I can’t provide all of the color data when it’s missing, but I think if the structural data is provided via pointcloud, an AI could read that structural data and perhaps reconstruct the missing data.
For Video, it’s going to be very complicated, and I may only get 5 minutes of results in a 3 month period, so my work is probably going to end up being a demo reel, I want to imagine something grand where I could take an entire library and do it, but I think I’d be delusional.
If you had a team of people working on a toolbox you might be able to optimize it, and even give training data to the GANs for even better results.
I think the best conversion choice for testing would be severence,
There is something about Shallow depth of field on a lot of things that ruins it, and Severence has this sharp anamorphic look that would probably look way better for my work, I couldn’t share it if I used it, but I think stills would be okay.
When I started, I learned I had to get extremely precise depth maps for stereo or 4V, and was like…I’m not gonna do that.
I would get sick thinking about the undertaking.
Then I started to hyper fixate on the level of precision.
I would feel even sicker.
I imagined how light would warp.
When “Clash of the Titans 3D” was released in 2011, it was considered one of the worst 3d conversions of all time. They rushed it.
I didn’t exactly understand the process.
I kept hyper fixating for years on “Why” and “How”.
I started making terrible depth maps a year ago, before finding the tools I needed and using Blender.
I learned that sick feeling was good.
I started making them, and then I took a break, to get ready for one of these displays.
I’m gonna do a lot more soon, every day I make something.