Compressive Light Field Reconstructions Using Deep Learning

• doi: 10.1109/CVPRW.2017.168
• title: Compressive Light Field Reconstructions Using Deep Learning
• publisher: IEEE
• isbn: 978-1-5386-0734-3
• issn: 2160-7516
• rank: 1984
• access_type: LOCKED
• content_type: Conferences
• abstract: Light field imaging is limited in its computational processing demands of high sampling for both spatial and angular dimensions. Single-shot light field cameras sacrifice spatial resolution to sample angular viewpoints, typically by multiplexing incoming rays onto a 2D sensor array. While this resolution can be recovered using compressive sensing, these iterative solutions are slow in processing a light field. We present a deep learning approach using a new, two branch network architecture, consisting jointly of an autoencoder and a 4D CNN, to recover a high resolution 4D light field from a single coded 2D image. This network decreases reconstruction time significantly while achieving average PSNR values of 26-32 dB on a variety of light fields. In particular, reconstruction time is decreased from 35 minutes to 6.7 minutes as compared to the dictionary method for equivalent visual quality. These reconstructions are performed at small sampling/compression ratios as low as 8%, allowing for cheaper coded light field cameras. We test our network reconstructions on synthetic light fields, simulated coded measurements of real light fields captured from a Lytro Illum camera, and real coded images from a custom CMOS diffractive light field camera. The combination of compressive light field capture with deep learning allows the potential for real-time light field video acquisition systems in the future.
• article_number: 8014902
• pdf_url: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8014902
• html_url: https://ieeexplore.ieee.org/document/8014902/
• abstract_url: https://ieeexplore.ieee.org/document/8014902/
• publication_title: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
• conference_location: Honolulu, HI, USA
• conference_dates: 21-26 July 2017
• publication_number: 8014302
• is_number: 8014734
• publication_year: 2017
• publication_date: 21-26 July 2017
• start_page: 1277
• end_page: 1286
• citing_paper_count: 31
• citing_patent_count: 2
• download_count: 1079
• insert_date: 20170824
• index_terms:
  • ieee_terms:
    • Image reconstruction
    • Cameras
    • Machine learning
    • Spatial resolution
    • Two dimensional displays
    • Network architecture
  • dynamic_index_terms:
    • Deep Learning
    • Light Field
    • Compressive Light Field
    • High-resolution
    • Spatial Resolution
    • Autoencoder
    • Visual Quality
    • Sensor Array
    • Network Reconstruction
    • Real Field
    • Branch Network
    • Reconstruction Time
    • Light Capture
    • Camera Array
    • Plenoptic Camera
    • Light Field Camera
    • Synthetic Field
    • Loss Function
    • Loss-of-function
    • Neural Network
    • Training Data
    • High Spatial Resolution
    • Image Reconstruction
    • Iterative Reconstruction
    • Tomographic Reconstruction
    • Compressive Measurements
    • Extract Meaningful Information
    • Random Matrix
    • Random Matrices
    • View Synthesis
    • Compression Ratio
    • Parallax
    • Random Projection
    • Iterative Solver
    • Dictionary Learning
    • Series Of Convolutional Layers
• isbn_formats:
  • format: Print on Demand(PoD) ISBN, value: 978-1-5386-0734-3, isbnType: New-2005
  • format: Electronic ISBN, value: 978-1-5386-0733-6, isbnType: New-2005
• authors:
  • Author Name: Mayank Gupta
    Affiliation: Arizona State University
    Author URL: https://ieeexplore.ieee.org/author/37086058406
    ID: 37086058406
    Order: 1
    Author Affiliations:
    • Arizona State University
  • Author Name: Arjun Jauhari
    Affiliation: Cornell University
    Author URL: https://ieeexplore.ieee.org/author/37086053113
    ID: 37086053113
    Order: 2
    Author Affiliations:
    • Cornell University
  • Author Name: Kuldeep Kulkarni
    Affiliation: Arizona State University
    Author URL: https://ieeexplore.ieee.org/author/38550413700
    ID: 38550413700
    Order: 3
    Author Affiliations:
    • Arizona State University
  • Author Name: Suren Jayasuriya
    Affiliation: Carnegie Mellon University
    Author URL: https://ieeexplore.ieee.org/author/37085521636
    ID: 37085521636
    Order: 4
    Author Affiliations:
    • Carnegie Mellon University
  • Author Name: Alyosha Molnar
    Affiliation: Cornell University
    Author URL: https://ieeexplore.ieee.org/author/37282515600
    ID: 37282515600
    Order: 5
    Author Affiliations:
    • Cornell University
  • Author Name: Pavan Turaga
    Affiliation: Arizona State University
    Author URL: https://ieeexplore.ieee.org/author/37428672600
    ID: 37428672600
    Order: 6
    Author Affiliations:
    • Arizona State University

Image Sensor

• sensor_type: CMOS
• resolution: 364x540
• dynamic_range: dB (unspecified)
• pixel_size: unknown (unspecified)
• dark_current: unknown (unspecified)

Optical Data

• focal_length: unknown (unspecified)
• aperture: unknown (unspecified)
• field_of_view: unknown (unspecified)
• distortion: unknown (unspecified)

Performance Metrics

• frame_rate: 30 fps (for Lytro Illum)
• signal_to_noise_ratio: 30-32 dB
• sensitivity: ISO (specific value not found)
• shutter_speed: unknown (unspecified)
• power_consumption: mW (specific value not found)
• noise: 0.1 and 0.2 (for Gaussian noise simulations)

Applications & Benefits

• cell_imaging: Biomedical applications using light field techniques.
• benefits: Enables high-resolution light field capture, supports various applications including view synthesis and depth mapping.

Supporting Organizations

• supported_by: Qualcomm, NSF (National Science Foundation)

Manuscript Details

• publication_date: 21-26 July 2017

Relevancy Score

• score: 9
• missing_fields:
  • fill factor
  • quantum efficiency
  • readout speed
  • noise sources (temporal noise, fixed-pattern noise)
  • analog-to-digital conversion techniques
  • microlenses
  • on-chip colour filters
  • global shutter/rolling shutter
  • backside illumination

Processed JSON Filename

request_d663472b-736a-4a00-8b83-2f7e8450b046-compressive_light_field_reconstructions_using_deep_learning.json