Pay attention to the gaming industry! No more weird mirrors with Mirror-NeRF

NeRFs, or Neural Radiance Fields, use a combination of RNN and CNN to capture an object’s physical characteristics, such as shape, material, and texture. They can generate realistic images of objects under different lighting conditions. They have proven most useful in medicine, robotics, and entertainment due to their ability to create high-resolution images.

3D reconstruction and rendering of scenes with mirrors, which are ubiquitous in the real world, has been a long-standing challenge in computer vision. To deal with the inconsistencies in mirror reconstruction with NeRF, researchers at Zhejiang University introduce Mirror-NeRF, which correctly reproduces the reflection in the mirror in a total radiation field by submitting the reflection probability and tracing the rays according to the light transport model of Whitted Ray Tracking.

NeRF, RefNeRF and NeRFReN all three methods generated specular reflection from new viewpoints by interpolating the previously learned reflections. However, they have limitations in reliably inferring reflections not seen during training and synthesizing reflections for newly introduced objects or mirrors in the scene. The newly introduced technique Mirror-NeRF can accurately draw the reflection in the mirror and serve various scene modification applications by integrating the physical ray tracing into the neural rendering process.

Five synthetic and four real datasets were created and quantitative comparisons of new display synthesis on the metrics Peak signal-to-noise ratio (PSNR), Structural Similarity Index Measure (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) were made. Due to the uneven surface of the mirror, which greatly affects the reflection quality, several regularization terms were also introduced in the optimization process. With all regularization terms enabled, we successfully obtained the exact reflection in the mirror with the highest image quality.

The results showed that NeRF, Ref-NeRF, and NeRFReN struggled to produce the reflection of the objects whose reflection has high-frequency variations in color, such as the distorted hanging image in the meeting room mirror, a blurred curtain in the mirror of the office and lounge, and a “foggy” garment in the mirror in the clothing store. On the other hand, the new method reproduced detailed reflections of objects by tracing the reflected rays. Although there are huge advances in working with mirrors, researchers have yet to incorporate refraction into the framework.

In conclusion, this breakthrough promises new avenues in the gaming and film industry. Artists may wish to create complex visual effects and use mirror manipulation, for example by replacing reflections in the mirror with another scene. We can synthesize the photorealistic display of the new scene in the mirror with multi-view consistency.

Check out Paper. All credit for this research goes to the researchers on this project. Also, don’t forget to participate our 28k+ ML SubReddit, 40k+ Facebook Community, Discord channeland Email newsletterwhere we share the latest AI research news, cool AI projects and more.