Abstract:

Limited input information from single image will seriously affect the performance of rain streak removal. Compared with ordinary 2D images, Light Field Images（LFIs） can record abundant information of a 3D scene. Aiming at such problems and using light field image characteristics, a dual LSTM neural network based rain streaks removal method for LFIs is proposed. The proposed neural network includes two sub-networks. One is Rain Streaks Detecting Network（RSDNet） and the other one is Background Restoring Network （BRNet）. Proposed method contains three main steps. First, a matching-cost-optimization-based method is adopted to calculate the depth maps for sub-views within the 3D EPI. Second, RSDNet is used to extract rain streaks. Dual LSTM is proposed to transmit high frequency information of rain streaks to the BRNet. Third, the BRNet is used to restore the background LFI to obtain the rain-free sub-views. In order to train and evaluate the proposed neural network, a rainy light field dataset is constructed, in which each LFI is composed of a real-world LFI and a synthetic rain streak LFI. Extensive experimental results demonstrate that the proposed method can effectively restore rainy LFIs.

Key words: light field images, Long Short-Term Memory（LSTM）, rain removal, image restoration

摘要：

单张图像去雨由于有限的输入信息，会严重影响去雨效果。光场图像不同于普通2D图像，能够记录三维场景的丰富结构信息。针对此类问题并且利用光场图像特点，提出一种基于双LSTM神经网络的光场图像去雨算法。提出的神经网络包括雨条纹检测网络（Rain Streaks Detecting Network，RSDNet）和背景修复网络（Background Restoring Network，BRNet）。所提算法主要包含三个步骤。使用匹配成本量最优化方法计算光场图像子视点的深度图；利用RSDNet提取雨条纹，并利用LSTM结构将雨条纹高频信息传递给BRNet；借助BRNet网络修复背景图像得到无雨子视点图像。为了训练和测试所提算法，构建了一个由真实背景场景光场图像和雨图像合成的有雨图像光场数据集。充分的实验结果表明，提出的算法能够有效地修复光场图像。

关键词: 光场图像, 长短期记忆网络（LSTM）, 图像去雨, 图像恢复