Evsassisted Joint Deblurring Rollingshutter Correction And Video Frame Interpolation Through Sensor Inverse Modeling

• doi: 10.1109/CVPR52733.2024.02378
• title: EVS-Assisted Joint Deblurring, Rolling-Shutter Correction and Video Frame Interpolation Through Sensor Inverse Modeling
• publisher: IEEE
• isbn: 979-8-3503-5301-3
• issn: 1063-6919
• rank: 2340
• access_type: LOCKED
• content_type: Conferences
• abstract: Event-based Vision Sensors (EVS) gain popularity in en-hancing CMOS Image Sensor (CIS) video capture. Nonide-alities of EVS such as pixel or readout latency can significantly influence the quality of the enhanced images and warrant dedicated consideration in the design of fusion algorithms. A novel approach for jointly computing de-blurred, rolling-shutter artifact corrected high-speed videos with frame rates up to 10000 FPS using inherently blurry rolling shutter CIS frames of 120 FPS to 150 FPS in conjunction with EVS data from a hybrid CIS-EVS sensor is presented. EVS pixel latency, readout latency and the sensor's refractory period are explicitly incorporated into the measurement model. This inverse function problem is solved on a per-pixel manner using an optimization-based framework. The interpolated images are subsequently processed by a novel refinement network. The proposed method is evaluated using simulated and measured datasets, under natural and controlled environments. Extensive experiments show reduced shadowing effect, a 4 dB increment in PSNR, and a 12 % improvement in LPIPS score compared to state-of-the-art methods.
• article_number: 10656189
• pdf_url: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10656189
• html_url: https://ieeexplore.ieee.org/document/10656189/
• abstract_url: https://ieeexplore.ieee.org/document/10656189/
• publication_title: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• conference_location: Seattle, WA, USA
• conference_dates: 16-22 June 2024
• publication_number: 10654794
• is_number: 10654797
• publication_year: 2024
• publication_date: 16-22 June 2024
• start_page: 25172
• end_page: 25181
• citing_paper_count: 1
• citing_patent_count: 0
• download_count: 103
• insert_date: 20240916
• index_terms:
  • ieee_terms:
    • Image quality
    • Interpolation
    • Computer vision
    • Inverse problems
    • Vision sensors
    • Pattern recognition
    • Optimization
  • author_terms:
    • Event Camera
    • Deblurring
    • Rolling-Shutter Correction
    • Video Frame Interpolation
  • dynamic_index_terms:
    • Inverse Model
    • Deblurring
    • Frame Interpolation
    • Video Interpolation
    • Video Frame Interpolation
    • Frame Rate
    • Environmental Control
    • Inverse Problem
    • Forward Problem
    • Peak Signal-to-noise Ratio
    • PSNR
    • Video Capture
    • Refractory Period
    • Refinement Network
    • Rolling Shutter
    • Image Quality
    • Low-pass
    • Low-pass Filter
    • Basketball
    • Nonlinear Programming
    • Nonlinear Optimization
    • Straightforward Way
    • Photocurrent
    • Explicit Model
    • Spatial Attention
    • Optical Flow
    • Graph Optimization
    • Improve Image Quality
    • Measurement Equation
    • Hybrid Sensor
    • Delay Compensation
    • Fast Motion
    • Ground Truth Image
    • Indoor Scenarios
    • Trigger Threshold
    • Perceptual Loss
• isbn_formats:
  • format: Print on Demand(PoD) ISBN, value: 979-8-3503-5301-3, isbnType: New-2005
  • format: Electronic ISBN, value: 979-8-3503-5300-6, isbnType: New-2005
• authors:
  • Author Name: Rui Jiang
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/695981431986564
    ID: 695981431986564
    Order: 1
    Author Affiliations:
    • OMNIVISION
  • Author Name: Fangwen Tu
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/37088953186
    ID: 37088953186
    Order: 2
    Author Affiliations:
    • OMNIVISION
  • Author Name: Yixuan Long
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/37085706118
    ID: 37085706118
    Order: 3
    Author Affiliations:
    • OMNIVISION
  • Author Name: Aabhaas Vaish
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/567164797859859
    ID: 567164797859859
    Order: 4
    Author Affiliations:
    • OMNIVISION
  • Author Name: Bowen Zhou
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/37086437635
    ID: 37086437635
    Order: 5
    Author Affiliations:
    • OMNIVISION
  • Author Name: Qinyi Wang
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/37086656351
    ID: 37086656351
    Order: 6
    Author Affiliations:
    • OMNIVISION
  • Author Name: Wei Zhang
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/724537812968001
    ID: 724537812968001
    Order: 7
    Author Affiliations:
    • OMNIVISION
  • Author Name: Yuntan Fang
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/38241081400
    ID: 38241081400
    Order: 8
    Author Affiliations:
    • OMNIVISION
  • Author Name: Luis Eduardo Garcia Capel
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/858475423524793
    ID: 858475423524793
    Order: 9
    Author Affiliations:
    • OMNIVISION
  • Author Name: Bo Mu
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/37089982727
    ID: 37089982727
    Order: 10
    Author Affiliations:
    • OMNIVISION
  • Author Name: Tiejun Dai
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/37088734748
    ID: 37088734748
    Order: 11
    Author Affiliations:
    • OMNIVISION
  • Author Name: Andreas Suess
    Affiliation: OMNIVISION
    Author URL: https://ieeexplore.ieee.org/author/37088149710
    ID: 37088149710
    Order: 12
    Author Affiliations:
    • OMNIVISION

Image Sensor

• sensor_type: CIS-EVS hybrid sensor
• resolution: 15-MPixel CIS + 1-Mpixel EVS
• dynamic_range: 120 dB
• pixel_size: 0.18 µm
• dark_current: characterized in measurements

Optical Data

• focal_length: not specified
• aperture: not specified
• field_of_view: not specified
• distortion: not specified

Performance Metrics

• frame_rate: up to 10,000 FPS
• shutter_speed: 120-150 FPS for CIS
• noise: specified by factors such as readout latency and pixel latency

Applications & Benefits

• cell_imaging: enables applications in medical and imaging fields
• benefits: enhanced imaging quality and reduced artifacts through effective deblurring and frame interpolation

Supporting Organizations

• supported_by: OMNIVISION

Manuscript Details

• publication_date: 16-22 June 2024

Relevancy Score

• score: 9
• missing_fields:
  • focal_length
  • aperture
  • field_of_view
  • distortion
  • signal_to_noise_ratio
  • sensitivity
  • power_consumption

Processed JSON Filename

request_13c7e069-8045-4afd-8c77-f69a50b20122-evsassisted_joint_deblurring_rollingshutter_correction_and_video_frame_interpolation_through_sensor_inverse_modeling.json