Efficient Spacetime Sampling With Pixelwise Coded Exposure For Highspeed Imaging

• doi: 10.1109/TPAMI.2013.129
• title: Efficient Space-Time Sampling with Pixel-Wise Coded Exposure for High-Speed Imaging
• publisher: IEEE
• isbn:
• issn: 1939-3539
• rank: 2358
• access_type: LOCKED
• content_type: Journals
• abstract: Cameras face a fundamental trade-off between spatial and temporal resolution. Digital still cameras can capture images with high spatial resolution, but most high-speed video cameras have relatively low spatial resolution. It is hard to overcome this trade-off without incurring a significant increase in hardware costs. In this paper, we propose techniques for sampling, representing, and reconstructing the space-time volume to overcome this trade-off. Our approach has two important distinctions compared to previous works: 1) We achieve sparse representation of videos by learning an overcomplete dictionary on video patches, and 2) we adhere to practical hardware constraints on sampling schemes imposed by architectures of current image sensors, which means that our sampling function can be implemented on CMOS image sensors with modified control units in the future. We evaluate components of our approach, sampling function and sparse representation, by comparing them to several existing approaches. We also implement a prototype imaging system with pixel-wise coded exposure control using a liquid crystal on silicon device. System characteristics such as field of view and modulation transfer function are evaluated for our imaging system. Both simulations and experiments on a wide range of scenes show that our method can effectively reconstruct a video from a single coded image while maintaining high spatial resolution.
• article_number: 6552198
• pdf_url: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6552198
• html_url: https://ieeexplore.ieee.org/document/6552198/
• abstract_url: https://ieeexplore.ieee.org/document/6552198/
• publication_title: IEEE Transactions on Pattern Analysis and Machine Intelligence
• conference_location:
• conference_dates:
• publication_number: 34
• is_number: 6687161
• publication_year: 2014
• publication_date: Feb. 2014
• start_page: 248
• end_page: 260
• citing_paper_count: 49
• citing_patent_count: 6
• download_count: 2414
• insert_date: 20130702
• index_terms:
  • ieee_terms:
    • Dictionaries
    • Spatial resolution
    • Cameras
    • Image reconstruction
    • Streaming media
  • author_terms:
    • Space-time sampling
    • dictionary learning
    • sparse reconstruction
    • computational camera
  • dynamic_index_terms:
    • High-resolution
    • Spatial Resolution
    • Imaging System
    • Field Of View
    • Low Resolution
    • High Spatial Resolution
    • Image Sensor
    • Camera Sensor
    • Sparse Representation
    • High-speed Video
    • Modulation Transfer Function
    • Optical Transfer Function
    • Objective Lens
    • Objective Lenses
    • Microscope Objective
    • Integration Time
    • Polarizability
    • Frame Rate
    • Periodic Motion
    • Periodic Movements
    • Oscillatory Motion
    • Movement Period
    • Reconstruction Results
    • Kinds Of Devices
    • Spatial Light Modulator
    • Discrete Cosine Transform
    • Dictionary Learning
    • Digital Micromirror Device
    • High Dynamic Range Image
    • Rolling Shutter
    • P-polarized
    • S-polarized
    • S-polarized Light
    • P-polarized Light
    • Readout Noise
    • Regeneration Buffer
    • Framebuffer
    • Display Memory
    • Input Frames
    • Video Features
    • Video Characteristics
    • Sensor Plane
    • High-speed Motion
• authors:
  • Author Name: Dengyu Liu
    Affiliation: Chester F. Carlson Center for Imaging Science, Rochester Institute of Technology, Rochester, NY
    Author URL: https://ieeexplore.ieee.org/author/38564576700
    ID: 38564576700
    Order: 1
    Author Affiliations:
    • Chester F. Carlson Center for Imaging Science, Rochester Institute of Technology, Rochester, NY
  • Author Name: Jinwei Gu
    Affiliation: Chester F. Carlson Center for Imaging Science, Rochester Institute of Technology, Rochester, NY
    Author URL: https://ieeexplore.ieee.org/author/38194854500
    ID: 38194854500
    Order: 2
    Author Affiliations:
    • Chester F. Carlson Center for Imaging Science, Rochester Institute of Technology, Rochester, NY
  • Author Name: Yasunobu Hitomi
    Affiliation: Sony Corporation, Tokyo, Japan
    Author URL: https://ieeexplore.ieee.org/author/38196155500
    ID: 38196155500
    Order: 3
    Author Affiliations:
    • Sony Corporation, Tokyo, Japan
  • Author Name: Mohit Gupta
    Affiliation: Department of Computer Science, Columbia University, New York, NY
    Author URL: https://ieeexplore.ieee.org/author/37416652700
    ID: 37416652700
    Order: 4
    Author Affiliations:
    • Department of Computer Science, Columbia University, New York, NY
  • Author Name: Tomoo Mitsunaga
    Affiliation: Sony Corporation, Tokyo, Japan
    Author URL: https://ieeexplore.ieee.org/author/37355695600
    ID: 37355695600
    Order: 5
    Author Affiliations:
    • Sony Corporation, Tokyo, Japan
  • Author Name: Shree K. Nayar
    Affiliation: Department of Computer Science, Columbia University, New York, NY
    Author URL: https://ieeexplore.ieee.org/author/37278100400
    ID: 37278100400
    Order: 6
    Author Affiliations:
    • Department of Computer Science, Columbia University, New York, NY

Image Sensor

• sensor_type: CMOS
• resolution: 1384x1036
• dynamic_range: Not specified in the text
• pixel_size: Not specified in the text
• dark_current: Not specified in the text

Optical Data

• focal_length: 16mm
• aperture: F1.4
• field_of_view: 30.75° horizontal; 23.31° vertical
• distortion: Not specified in the text

Performance Metrics

• frame_rate: Not specified in the text
• signal_to_noise_ratio: Not specified in the text
• sensitivity: Not specified in the text
• shutter_speed: 27ms (for eyelid example), 18ms (for toy plane and ball example)
• power_consumption: Not specified in the text
• noise: Photon noise and readout noise, with evaluation of noise standard deviation from 0.001 to 0.2

Applications & Benefits

• cell_imaging: Not specified in the text
• benefits: High-speed imaging while maintaining high spatial resolution.

Supporting Organizations

• supported_by: Sony Corporation, US National Science Foundation, US Office of Naval Research, Xerox.

Manuscript Details

• publication_date: Feb. 2014

Relevancy Score

• score: 10
• missing_fields:
  • fill factor
  • quantum efficiency
  • readout speed
  • noise sources
  • analog-to-digital conversion techniques
  • microlenses
  • on-chip colour filters
  • global or rolling shutters
  • backside illumination

Processed JSON Filename

request_20de13bb-bf8e-4c02-884d-ead95825b173-efficient_spacetime_sampling_with_pixelwise_coded_exposure_for_highspeed_imaging.json