Asp Vision Optically Computing The First Layer Of Convolutional Neural Networks Using Angle Sensitive Pixels

• doi: 10.1109/CVPR.2016.104
• title: ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks Using Angle Sensitive Pixels
• publisher: IEEE
• isbn: 978-1-4673-8852-8
• issn: 1063-6919
• rank: 1366
• access_type: LOCKED
• content_type: Conferences
• abstract: Deep learning using convolutional neural networks (CNNs) is quickly becoming the state-of-the-art for challenging computer vision applications. However, deep learning's power consumption and bandwidth requirements currently limit its application in embedded and mobile systems with tight energy budgets. In this paper, we explore the energy savings of optically computing the first layer of CNNs. To do so, we utilize bio-inspired Angle Sensitive Pixels (ASPs), custom CMOS diffractive image sensors which act similar to Gabor filter banks in the V1 layer of the human visual cortex. ASPs replace both image sensing and the first layer of a conventional CNN by directly performing optical edge filtering, saving sensing energy, data bandwidth, and CNN FLOPS to compute. Our experimental results (both on synthetic data and a hardware prototype) for a variety of vision tasks such as digit recognition, object recognition, and face identification demonstrate 97% reduction in image sensor power consumption and 90% reduction in data bandwidth from sensor to CPU, while achieving similar performance compared to traditional deep learning pipelines.
• article_number: 7780473
• pdf_url: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7780473
• html_url: https://ieeexplore.ieee.org/document/7780473/
• abstract_url: https://ieeexplore.ieee.org/document/7780473/
• publication_title: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• conference_location: Las Vegas, NV, USA
• conference_dates: 27-30 June 2016
• publication_number: 7776647
• is_number: 7780329
• publication_year: 2016
• publication_date: 27-30 June 2016
• start_page: 903
• end_page: 912
• citing_paper_count: 36
• citing_patent_count: 2
• download_count: 861
• insert_date: 20161212
• index_terms:
  • ieee_terms:
    • Optical imaging
    • Optical sensors
    • Bandwidth
    • Biomedical optical imaging
    • Image sensors
    • Optical device fabrication
    • Machine learning
  • dynamic_index_terms:
    • Convolutional Neural Network
    • Angle-sensitive Pixels
    • Deep Learning
    • Computer Vision
    • Energy Conservation
    • Energy-saving
    • Power Consumption
    • Visual Cortex
    • Visual Cortices
    • Object Recognition
    • Image Sensor
    • Camera Sensor
    • Vision Tasks
    • Network Bandwidth
    • Data Bandwidth
    • Modulation Bandwidth
    • Internet Bandwidth
    • Face Identity
    • Optical Character Recognition
    • Text Recognition
    • Digit Recognition
    • Text Detection
    • Optical Recognition
    • Floating-point Operations
    • Bandwidth Requirements
    • Gabor Filters
    • Edge Filter
    • Hardware Prototype
    • Conventional Convolutional Neural Networks
    • Human Visual Cortex
    • Dynamic Vision Sensor
    • Event Camera
    • Gabor Wavelet
    • Optical Processing
    • Optical Signal Processing
    • Optical Calculations
    • Optical Computing
    • Potential Savings
    • Energy Efficiency
    • Number Of Filters
    • Visual Recognition
    • Diffraction Grating
    • Diffraction Orders
    • Natural Scene Statistics
    • Natural Image Statistics
    • Light Field
• isbn_formats:
  • format: Print on Demand(PoD) ISBN, value: 978-1-4673-8852-8, isbnType: New-2005
  • format: Electronic ISBN, value: 978-1-4673-8851-1, isbnType: New-2005
• authors:
  • Author Name: Huaijin G. Chen
    Affiliation: Rice University
    Author URL: https://ieeexplore.ieee.org/author/37086163474
    ID: 37086163474
    Order: 1
    Author Affiliations:
    • Rice University
  • Author Name: Suren Jayasuriya
    Affiliation: Cornell University
    Author URL: https://ieeexplore.ieee.org/author/37085521636
    ID: 37085521636
    Order: 2
    Author Affiliations:
    • Cornell University
  • Author Name: Jiyue Yang
    Affiliation: Cornell University
    Author URL: https://ieeexplore.ieee.org/author/37089004112
    ID: 37089004112
    Order: 3
    Author Affiliations:
    • Cornell University
  • Author Name: Judy Stephen
    Affiliation: Cornell University
    Author URL: https://ieeexplore.ieee.org/author/37086163173
    ID: 37086163173
    Order: 4
    Author Affiliations:
    • Cornell University
  • Author Name: Sriram Sivaramakrishnan
    Affiliation: Cornell University
    Author URL: https://ieeexplore.ieee.org/author/38235561000
    ID: 38235561000
    Order: 5
    Author Affiliations:
    • Cornell University
  • Author Name: Ashok Veeraraghavan
    Affiliation: Rice University
    Author URL: https://ieeexplore.ieee.org/author/37282938900
    ID: 37282938900
    Order: 6
    Author Affiliations:
    • Rice University
  • Author Name: Alyosha Molnar
    Affiliation:
    Author URL: https://ieeexplore.ieee.org/author/37282515600
    ID: 37282515600
    Order: 7
    

Image Sensor

• sensor_type: CMOS
• resolution: 384 x 384 (effective ASP tile resolution)
• dynamic_range: Not specified
• pixel_size: 10 µm
• dark_current: Not specified

Optical Data

• focal_length: F1.2
• aperture: Not specified
• field_of_view: Not specified
• distortion: Not specified

Performance Metrics

• frame_rate: 30 fps
• signal_to_noise_ratio: Not explicitly stated, but discussed in context of low efficiency and high noise
• power_consumption: 1.8 mW for the entire ASP sensor; 33 pJ/frame/pixel for ASP vs 340 pJ/frame/pixel for traditional sensors
• noise: High noise present in the prototype due to readout circuits and external amplifiers

Applications & Benefits

• cell_imaging: Not specified
• benefits: 90% reduction in power consumption and bandwidth from traditional sensors, energy savings by optically computing the first layer of CNNs.

Supporting Organizations

• supported_by: NSF CCF-1527501, THECB-NHARP 13308, NSF CAREER-1150329

Manuscript Details

• publication_date: 27-30 June 2016

Relevancy Score

• score: 9

Processed JSON Filename

request_3d403c31-3366-4410-bbe8-a0e0c8414f72-asp_vision_optically_computing_the_first_layer_of_convolutional_neural_networks_using_angle_sensitive_pixels.json