A Multimode Vision Sensor With Temporal Contrast Pixel And Columnparallel Local Binary Pattern Extraction For Dynamic Depth Sensing Using Stereo Vision

• doi: 10.1109/JSSC.2023.3292051
• title: A Multimode Vision Sensor With Temporal Contrast Pixel and Column-Parallel Local Binary Pattern Extraction for Dynamic Depth Sensing Using Stereo Vision
• publisher: IEEE
• isbn:
• issn: 1558-173X
• rank: 836
• access_type: LOCKED
• content_type: Journals
• abstract: This article presents a 0.56/0.8 V multimode vision sensor for temporal and spatial information extraction with processing in sensor (PIS) technique. The proposed six transistors and one capacitor (6T1C) temporal contrast pixel (TCP) structure using the exposure compensation scheme (ECS) can realize in-pixel temporal contrast calculation and motion event reporting (ER) with global shutter exposure and seamless frame difference (FD) based on the pulsewidth modulation (PWM) operation. In addition, the column-parallel local binary pattern (LBP) extraction provides further spatial feature information without the power-consuming analog-to-digital conversion to meet the power-efficient requirement for always-on-edge devices. According to the temporal and spatial information extracted from FD and LBP modes, this work realizes the PIS operations for dynamic depth calculation of a stereo vision system, which filters out the static scene and provides the depth information of the dynamic objects only. To further alleviate the data transfer bandwidth, the region of interest (ROI) capability is also embedded in the sensor for data windowing and object locating. The prototyped vision sensor was fabricated in TSMC’s standard 0.18 $\mu \text{m}$ CMOS process and verified. The vision sensor provides five operating modes that can adapt to different tasks, which include the raw image mode (IM), FD mode, ER mode, LBP mode, and ROI mode. The measurement results show a maximum frame rate of 540/819/540 frames/s at a power consumption of 390/162.6/151.9 $\mu \text{W}$ with an achieved iFoM of 45.5/12.5/17.7 pJ/pixel $\cdot $ frame in IM/ER/LBP mode, respectively.
• article_number: 10185151
• pdf_url: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10185151
• html_url: https://ieeexplore.ieee.org/document/10185151/
• abstract_url: https://ieeexplore.ieee.org/document/10185151/
• publication_title: IEEE Journal of Solid-State Circuits
• conference_location:
• conference_dates:
• publication_number: 4
• is_number: 10262388
• publication_year: 2023
• publication_date: Oct. 2023
• start_page: 2767
• end_page: 2777
• citing_paper_count: 2
• citing_patent_count: 0
• download_count: 1544
• insert_date: 20230718
• index_terms:
  • ieee_terms:
    • Vision sensors
    • Feature extraction
    • Motion detection
    • Sensors
    • Voltage
    • Stereo vision
    • Pulse width modulation
  • author_terms:
    • Feature extraction
    • frame difference (FD)
    • global shutter
    • local binary pattern (LBP)
    • motion detection
    • region of interest (ROI)
    • saliency detection
    • stereo vision
    • vision sensor
  • dynamic_index_terms:
    • Stereopsis
    • Stereovision
    • Stereoscopic Vision
    • Vision Sensors
    • Temporal Contrast
    • Spatial Information
    • Visual System
    • Power Consumption
    • Spatial Features
    • Spatial Characteristics
    • Operation Mode
    • Pulse Width
    • Pulsewidth Modulation
    • Information Extraction
    • Frame Rate
    • Raw Images
    • Temporal Information
    • Analog-to-digital Converter
    • Analog-to-digital
    • Digital-to-analog Converter
    • Depth Information
    • Extract Spatial Information
    • Maximum Frame Rate
    • Maximal Frame
    • Motion Events
    • Spatial Feature Information
    • Raw Data
    • Dynamic Vision Sensor
    • Event Camera
    • Region Of Interest Extraction
    • Unmanned Aerial Vehicles
    • Local Binary Pattern Features
    • Motion Detection
    • Motion Sensors
    • Obstacle Avoidance
    • Obstacle Detection
    • Edge Devices
    • Pixel Pitch
    • Dark Noise
    • Event Detection
• authors:
  • Author Name: Min-Yang Chiu
    Affiliation: Novatek Microelectronics Corporation, Hsinchu, Taiwan
    Author URL: https://ieeexplore.ieee.org/author/37089015159
    ID: 37089015159
    Order: 1
    Author Affiliations:
    • Novatek Microelectronics Corporation, Hsinchu, Taiwan
  • Author Name: Guan-Cheng Chen
    Affiliation: Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
    Author URL: https://ieeexplore.ieee.org/author/37088931050
    ID: 37088931050
    Order: 2
    Author Affiliations:
    • Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
  • Author Name: Tzu-Hsiang Hsu
    Affiliation: Mediatek Inc., Hsinchu, Taiwan
    Author URL: https://ieeexplore.ieee.org/author/37086335074
    ID: 37086335074
    Order: 3
    Author Affiliations:
    • Mediatek Inc., Hsinchu, Taiwan
  • Author Name: Ren-Shuo Liu
    Affiliation: Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
    Author URL: https://ieeexplore.ieee.org/author/37086262728
    ID: 37086262728
    Order: 4
    Author Affiliations:
    • Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
  • Author Name: Chung-Chuan Lo
    Affiliation: College of Life Science, National Tsing Hua University, Hsinchu, Taiwan
    Author URL: https://ieeexplore.ieee.org/author/37728954000
    ID: 37728954000
    Order: 5
    Author Affiliations:
    • College of Life Science, National Tsing Hua University, Hsinchu, Taiwan
  • Author Name: Kea-Tiong Tang
    Affiliation: Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
    Author URL: https://ieeexplore.ieee.org/author/37290163100
    ID: 37290163100
    Order: 6
    Author Affiliations:
    • Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
  • Author Name: Meng-Fan Chang
    Affiliation: Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
    Author URL: https://ieeexplore.ieee.org/author/37420875800
    ID: 37420875800
    Order: 7
    Author Affiliations:
    • Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
  • Author Name: Chih-Cheng Hsieh
    Affiliation: Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan
    Author URL: https://ieeexplore.ieee.org/author/37365236900
    ID: 37365236900
    Order: 8
    Author Affiliations:
    • Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan

Image Sensor

• sensor_type: CMOS
• resolution: 126x126
• dynamic_range: 43.58 dB
• pixel_size: 12 µm
• dark_current: 5.94 DN

Optical Data

• focal_length: 6 mm
• aperture: Not specified
• field_of_view: Not specified
• distortion: Not specified

Performance Metrics

• frame_rate: 540/819/540 fps
• power_consumption: 390/162.6/151.9 µW
• noise: 1.6% FPN, 0.59 DN (after frame rate reduction)

Applications & Benefits

• cell_imaging: Dynamic depth calculation, motion detection, feature extraction for intelligent functions
• benefits: Low-power, low-latency imaging for edge devices and efficient data transfer with reduced bandwidth.

Supporting Organizations

• supported_by: Qualcomm, Taiwan University Research Collaboration Project, Taiwan Semiconductor Research Institute (TSRI), Novatek Microelectronics Corp., and Ministry of Science and Technology (MOST), Taiwan.

Manuscript Details

• publication_date: Oct. 2023

Relevancy Score

• score: 10

Processed JSON Filename

request_47410efa-9482-4779-92d2-2f81c540f09c-a_multimode_vision_sensor_with_temporal_contrast_pixel_and_columnparallel_local_binary_pattern_extraction_for_dynamic_depth_sensing_using_stereo_vision.json