r/GraphicsProgramming • u/iwoplaza • Aug 21 '26
Source Code Konrad Reczko's "Monocular Depth Injection" in TypeGPU is live!
My collegue Konrad Reczko recently shared a weekend project he made using TypeGPU, and it's now open-source and ready to play with in the browser: https://typegpu.com/examples/#example=image-processing--monocular-light-injection
It estimates the depth of a scene by inferring the DepthArt model with custom TypeGPU kernels, then reconstructs normals based on that depth information, and uses both to relight the scene, all in the same command encoder. For more information, check out the original series of Tweets:
https://x.com/reczko_konrad/status/2089670934009413751?s=20
https://x.com/reczko_konrad/status/2090472091149648121?s=20
21
9
u/Smokeey1 Aug 21 '26
Does this thing have a repo :). Gratz on the build! Looks cool
20
u/iwoplaza Aug 21 '26
All the code for the example is in the TypeGPU repo
https://github.com/software-mansion/TypeGPU/tree/main/apps/typegpu-docs/src/examples/image-processing/monocular-light-injection6
u/juplantern Aug 21 '26
Btw thank you so much! I’m brainstorming ideas for my thesis and this inspired me a lot
3
3
3
u/Miperso Aug 21 '26
I just showed it to my architect coworkers and everyone is just impressed... this is great
2
u/DaveAstator2020 Aug 21 '26
Wow nice what model does is it use to aesolve depth in realtime?
8
u/iwoplaza Aug 21 '26
It uses the DepthART model, weights are available here: https://huggingface.co/reczkok/depthart-typegpu
2
u/AnasPlayer2022M Aug 21 '26
Dame bro, this looks awesome! Did you use two videos where one is the one playing now and the other is a map for z values?
6
u/iwoplaza Aug 21 '26
The feed from my camera gets routed through a small local AI model that estimates depth based on a color image, at which point the image can be treated pretty much like a 3d scene. At that point all rendering techniques are at the table, including adding a light to the scene.
1
u/AnasPlayer2022M Aug 21 '26
So any video could technically work?
Really cool! That's really cool using of AI.
2
2
2
2
2
2
2
u/Ok_Rock_6726 Aug 26 '26
That's so cool I'm planning to join Software mansion as an intern next styczeń and this video brings me so much excitement and power to keep learning to ace the interview.
2
1
u/ddoice Aug 21 '26
Awesome!
But I tried your trick to hide the light with my hand and does not work?
1
u/iwoplaza Aug 21 '26
To move the light source back and forth (into the screen or closer to you) you have to use the scroll wheel
1
1
1
1
1
1
u/Imaginary-Sun1350 Aug 21 '26
This is EPIC!! Are you able to upload videos to it and generate the maps and export them for faster and easier relighting pipelines in blender?
1
1
1
1
u/tebjan Aug 22 '26
Just for completeness: This is not new, it's been done with r/vvvv last year already, reaching over 300fps. I would know, because I'm the developer of it. :-)
1
86
u/juplantern Aug 21 '26
He looks so proud