Blind deconvolution and denoising in dual exposure problem

Miguel Tallón1, Javier Mateos1, S. Derin Babacan2, Rafael Molina1, Aggelos K. Katsaggelos3

1Departamento de Ciencias de la Computación e I.A., Universidad de Granada, Granada, Spain
{mtallon, jmd, rms}@decsai.ugr.es

2Beckman Institute, University of Illinois at Urbana-Champaign, IL USA
dbabacan@illinois.edu

3Electrical Engineering Computer Science Department, Northwestern University, Evanston, IL USA
aggk@eecs.northwestern.edu

Abstract

We aim at recovering a sharp image in low light conditions by combining information from images with different exposure time. In particular, we will demonstrate that a pair of images, consisting of a correctly exposed but blurred image and a sharp but very noisy image due to the short exposure time, is enough to recover a free of blurring and noise image by combining the information of the pair of images.  Since the correctly exposed image needed of a the long exposure time, it presents blur that is mainly caused by camera shake or object motion. On the other hand, the noise in the underexposed image is introduced by the gain factor applied to the sensor when the ISO is set to an high value.

The problem of recovering a sharp image from the pair of blurred and noisy images has been addressed from two points of view: First, we  considered a spatially invariant blur and, thus, an algorithm which combines observation models is proposed. The proposed method achieved better results than the competitors we compared with. Unfortunately, due to the spatially invariant structure of the blur we assumed, the proposed method did not provide satisfactory results in some real problems.

To solve the limitations of this method, we propose new method that performs a space-variant blur estimation and, then, a effective denoising/deconvolution method is applied for combining the images in the pair. The proposed algorithm start dividing the both the blurred and the noisy images into small overlapped blocks where the blur is considered to be spatially invariant and a blurring kernel is estimated for each block from the information of both images in the pair. Then, a novel deconvolution algorithm that incorporates a combination of image prior models into a spatially-varying deblurring/denoising framework and automatically estimates all the needed parameters is applied to each block. The method is able to evaluate, for each block, if the deblur procedure was sucessfully. In blocks where not enough information to estimate the blur or blocks with different blurs, where the deblur procedure cannot recover a sharp image, a denoising method is applied. Finally, the information from all the blocks is combined into the final restored image. Note that this approach applies a locally spatial invariant but globally spatial variant deconvolution process. Experiments on both synthetic and real images are provided to validate the proposed approach.

Code

Available upon request to jmd@decsai.ugr.es

Experiments

The datasets used in the experiments can be downloaded from here

Experiments with synthetic images

Original image Simulated short-exposure image
after calibration.

 

Blur 1 Blur 2
Blur 3 Blur 4

Blurred images simulating long-exposure shots. The set of PSFs used to
generate each image is shown below the corresponding image. The values
of the PSFs are mapped to the range [0,255] for visualization purposes.

Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) figures of merit for the images of the synthetic experiments
  Blur kernel set
  Blur 1 Blur 2 Blur 3 Blur 4
  PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM
Blurred 23,88 0,75 23,79 0,75 20,99 0,51 20,80 0,49
Noisy 22,80 0,85 22,80 0,85 22,71 0,85 22,77 0,85
Denoising [1] 24,41 0,89 24,40 0,89 24,07 0,89 24,25 0,89
Deconblind (spatially-variant) 22,95 0,87 21,87 0,88 22,07 0,85 22,10 0,89
Deconvblind (spatially-invariant) 22,30 0,78 22,06 0,78 18,76 0,80 18,57 0,83
Method in [2] (spatially-variant.) 22,69 0,86 20,55 0,85 22,53 0,87 23,31 0,90
Method in [2] (spatially-invariant) 22,09 0,77 21,33 0,76 22,43 0,85 23,47 0,89
Method in [3] 22,61 0,78 24,12 0,84 25,88 0,83 26,60 0,86
Method in [4] 25,88 0,85 26,38 0,87 27,45 0,89 28,25 0,92
Proposed method (λ = 0) 25,46 0,85 25,48 0,87 20,51 0,58 20,19 0,56
Proposed method (λ = 1) 27,77 0,94 28,49 0,94 27,45 0,93 28,31 0,94
Proposed method (estimated λ) 28,26 0,95 29,15 0,95 28,26 0,95 28,74 0,95
Proposed method (best λ) 28,62 0,95 29,53 0,96 29,08 0,96 30,74 0,97

Experiments with real images

Long-exposure image Calibrated short-exposure image.

Denoising [1] Method in [2]
Spatially-Invariant [3] Spatially-Variant [4]

Long-exposure image Calibrated short-exposure image.

Denoising [1] Method in [2]
Method in [5] Spatially-invariant [3]
Spatially-variant [4]

References

[1] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. on Image Processing, vol. 16, no. 8, pp. 2080–2095, 2007.

[2] S. D. Babacan, J. Wang, R. Molina, and A. K. Katsaggelos, “Bayesian blind deconvolution from differently exposed image pairs,” IEEE Trans. on Image Processing, vol. 19, no. 11, pp. 2874–2888, 2010.

[3] M. Tallón, J. Mateos, S. D. Babacan, R. Molina, and A. K. Katsaggelos, “Combining observation models in dual exposure problems using the Kullback-Leibler divergence,” in 18th European Signal Processing Conference (EUSIPCO), 2010, pp. 323–327.

[4] M. Tallón, J. Mateos, S.D. Babacan, R. Molina, and A. K. Katsaggelos, “Space-variant blur deconvolution and denoising in the dual exposure problem”, Information Fusion, 2013.

[5] M. Šorel, F. Šroubek, and J. Flusser, “Towards super-resolution in the presence of spatially varying blur,” in Super-resolution imaging, Milanfar Peyman, Ed., chapter 7, pp. 187–218.
CRC Press, 2010.

[6] M. Tallón, J. Mateos, R. Molina, and A.K. Katsaggelos, “Image prior combination in spacevariant blur deconvolution for the dual exposure problem,” in 7th International Symposium on Image and Signal Processing and Analysis (ISPA), 2011, pp. 408–413.

[7] M. Tallón, J. Mateos, S.D. Babacan, R. Molina, and A.K. Katsaggelos, “Space-variant kernel deconvolution for dual exposure problem,” in 19th European Signal Processing Conference (EUSIPCO), 2011, pp. 1678–1682.

 

Copyright © Miguel Tallón, S.Derin Babacan, Javier Mateos, Rafael Molina, Aggelos K. Katsaggelos, 2012