Global Illumination Rendering Based on Generative Adversarial Model and Light Decomposition

doi:10.3778/j.issn.1002-8331.2110-0124

Abstract

Abstract: A global illumination rendering network based on generative adversarial model and light decomposition is proposed to solve the problem of fuzzy high-frequency features in global illumination image reconstruction. It learns the abstract representation of light transport from graphics attributes and encodes such representation to predict global illumination image. Firstly, it decomposes light into diffuse and specular components, and uses independent generative adversarial networks to learn and infer light subgraphs end-to-end, which avoids the mutual interference of mixed lighting and ensure the clear reproduction of high-frequency details. Secondly, the rendering network uses auto-encoder as fundamental network to generate images, and adds multi-scale feature fusion blocks to extract features from different receptive field to express complicated special effects convincingly, such as shadow, secondary reflection, and so on. Thirdly, two enhanced adversarial loss functions, rotation loss and feature loss, are used to increase the stability of network training. The results show that, compared with denoising algorithms and image generative models, this method can reconstruct more realistic global illumination image with more high-frequency details and increase PSNR by 8%~20%.

Key words: global illumination rendering, light decomposition, generative adversarial network（GAN）, auto-encoder, multi-scale fusion

摘要： 针对现有全局光照图像重建高频特征效果模糊的问题，提出一种基于生成对抗模型及光路分解的全局光照绘制网络，以各类图形辅助属性（法线、深度、粗糙度等）为主要输入，学习光照传输的抽象表示并编码，用于推理光照图像。第一，将光照解耦为漫反射和镜面反射两部分，设计独立的生成对抗网络端到端地学习和推理光照子图，避免混频光照的相互干扰，保证高频细节的清晰重现。第二，使用自编码器作为绘制网络的基本结构，添加多尺度特征融合模块用于不同感受野下的特征合成，以促进阴影、镜面反射等复杂特效的有效表达。第三，使用旋转损失和特征损失两种增强的对抗损失函数，增加网络训练的稳定性。实验结果表明，与现有降噪或图像生成模型相比，该方法能够有效地生成视觉上更逼真的全局光照图像，保留更多高频细节，PSNR指标提升8%~20%。

关键词: 全局光照绘制, 光路分解, 生成对抗网络, 自编码器, 多尺度融合

LIANG Xiao, WANG Niting, WANG Jingwen, OUYANG Jiao. Global Illumination Rendering Based on Generative Adversarial Model and Light Decomposition[J]. Computer Engineering and Applications, 2023, 59(4): 243-251.

梁晓, 王妮婷, 王静雯, 欧阳娇. 基于生成对抗模型及光路分解的全局光照绘制[J]. 计算机工程与应用, 2023, 59(4): 243-251.

References

[1] COOK R L.Stochastic sampling in computer graphics[J].ACM Transactions on Graphics，1986，5（1）：51-72.
[2] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[3] GOODFELLOW I J，POUGET A J，MIRZA M，et al.Generative adversarial networks[J].arXiv：1406.2661，2014.
[4] CHAITANYA C R A，KAPLANYAN A S，SCHIED C，et al.Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder[J].ACM Transactions on Graphics，2017，36（4）：98.
[5] BAKO S，VOGELS T，MCWILLIAMS B，et al.Kernel-predicting convolutional networks for denoising Monte Carlo renderings[J].ACM Transactions on Graphics，2017，36（4）：97.
[6] GHARBI M，LI T M，AITTALA M，et al.Sample-based Monte Carlo denoising using a kernel-splatting network[J].ACM Transactions on Graphics，2019，38（4）：125.
[7] ESLAMI S M A，REZENDE D J，BESSE F，et al.Neural scene representation and rendering[J].Science，2018，360（6394）：1204-1210.
[8] GORAL C M，TORRANCE K E，GREENBERG D P，et al.Modeling the interaction of light between diffuse surfaces[J].ACM SIGGRAPH Computer Graphics，1984，18（3）：213-222.
[9] HEY H，PURGATHOFER W.Advanced radiance estimation for photon map global illumination[J].Computer Graphics Forum，2002，21（3）：541-545.
[10] PETSCHNIGG G，SZELISKI R，AGRAWALA M，et al.Digital photography with flash and no-flash image pairs[J].ACM Transactions on Graphics，2004，23（3）：664-672.
[11] BUADES A，COLL B，MOREL J M.A review of image denoising algorithms，with a new one[J].Multiscale Modeling & Simulation，2005，4（2）：490-530.
[12] DELBRACIO M，MUSé P，BUADES A，et al.Boosting Monte Carlo rendering by ray histogram fusion[J].ACM Transactions on Graphics，2014，33（1）：8.
[13] HE K，SUN J，TANG X.Guided image filtering[C]//11th European Conference on Computer Vision.Berlin，Heidelberg，Springer，2010：1-14.
[14] BAUSZAT P，EISEMANN M，MAGNOR M.Guided image filtering for interactive high-quality global illumination[C]//Proceedings of the 22nd Eurographics Conference on Rendering，2011：1361-1368.
[15] BITTERLI B，ROUSSELLE F，MOON B，et al.Nonlinearly weighted first-order regression for denoising Monte Carlo renderings[J].Computer Graphics Forum，2016，35（4）：107-117.
[16] KALANTARI N K，BAKO S，SEN P.A machine learning approach for filtering Monte Carlo noise[J].ACM Transactions on Graphics，2015，34（4）：122.
[17] VOGELS T，ROUSSELLE F，MCWILLIAMS B，et al.Denoising with kernel prediction and asymmetric loss functions[J].ACM Transactions on Graphics，2018，37（4）：124.
[18] CHO K，VAN M B.Learning phrase representations using RNN encoder-decoder for statistical machine translation[J].arXiv：1406.1078，2014.
[19] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[20] WANG T C，LIU M Y，ZHU J Y，et al.High-resolution image synthesis and semantic manipulation with conditional GANs[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：8798-8807.
[21] 刘晓芸，姚承宗，曾晓勤.基于RBF神经网络的全局光照实时绘制[J].计算机仿真，2021，38（9）：424-428.
LIU X Y，YAO C Z，ZENG X Q.Real-time global illumination based on RBF neural network[J].Computer Simulation，2021，38（9）：424-428.
[22] GRANSKOG J，ROUSSELLE F，PAPAS M，et al.Compositional neural scene representations for shading inference[J].ACM Transactions on Graphics，2020，39（4）：135.
[23] CHEN T，ZHAI X，RITTER M，et al.Self-supervised GANs via auxiliary rotation loss[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：12154-12163.
[24] LAND E H，MCCANN J J.Lightness and Retinex theory[J].Journal of the Optical Society of America，1971，61（1）：1-11.
[25] BOUSSEAU A，PARIS S，DURAND F.User-assisted intrinsic images[C]//ACM SIGGRAPH Asia 2009 Papers，2009：1-10.
[26] 曹天池，李秀实，李丹，等.基于图像分解的光照估计算法[J].计算机工程与科学，2021，43（8）：1422-1428.
CAO T C，LI X S，LI D，et al.Illumination estimation based on image decomposition[J].Computer Engineering & Science，2021，43（8）：1422-1428.
[27] ZIMMER H，ROUSSELLE F，JAKOB W，et al.Path-space motion estimation and decomposition for robust animation filtering[J].Computer Graphics Forum，2015，34（4）：131-142.
[28] GOODFELLOW I，BENGIO Y，COURVILLE A，et al.Deep learning[M].Cambridge：MIT press，2016.
[29] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[30] LOTTER W，KREIMAN G，COX D.Unsupervised learning of visual structure using predictive generative networks[J].arXiv：1511.06380，2015.
[31] LUMBERYARD A.Amazon Lumberyard Bistro，open research content archive（ORCA）[EB/OL].（2017-07）[2021-09-24].http：//developer.nvidia.com/orca/amazon-lumberyard-bistro.
[32] GAMES E.Unreal engine sun temple，open research content archive（ORCA）[EB/OL].（2017-10）[2021-09-24].http：//developer.nvidia.com/orca/epic-games-sun-temple.
[33] WINKELMANN M.Zero-day，open research content archive（ORCA）[EB/OL].（2019-11）[2021-09-24].https：//developer.nvidia.com/orca/beeple-zero-day.
[34] KINGMA D P，BA J.Adam：a method for stochastic optimization[J].arXiv：1412.6980，2014.