Eventenhanced Snapshot Compressive Videography At 10K Fps

• doi: 10.1109/TPAMI.2024.3496788
• title: Event-Enhanced Snapshot Compressive Videography at 10K FPS
• publisher: IEEE
• isbn:
• issn: 1939-3539
• rank: 2418
• access_type: LOCKED
• content_type: Journals
• abstract: Video snapshot compressive imaging (SCI) encodes the target dynamic scene compactly into a snapshot and reconstructs its high-speed frame sequence afterward, greatly reducing the required data footprint and transmission bandwidth as well as enabling high-speed imaging with a low frame rate intensity camera. In implementation, high-speed dynamics are encoded via temporally varying patterns, and only frames at corresponding temporal intervals can be reconstructed, while the dynamics occurring between consecutive frames are lost. To unlock the potential of conventional snapshot compressive videography, we propose a novel hybrid “intensity$+$+ event imaging scheme by incorporating an event camera into a video SCI setup. Our proposed system consists of a dual-path optical setup to record the coded intensity measurement and intermediate event signals simultaneously, which is compact and photon-efficient by collecting the half photons discarded in conventional video SCI. Correspondingly, we developed a dual-branch Transformer utilizing the reciprocal relationship between two data modes to decode dense video frames. Extensive experiments on both simulated and real-captured data demonstrate our superiority to state-of-the-art video SCI and video frame interpolation (VFI) methods. Benefiting from the new hybrid design leveraging both intrinsic redundancy in videos and the unique feature of event cameras, we achieve high-quality videography at 0.1ms time intervals with a low-cost CMOS image sensor working at 24 FPS.
• article_number: 10750378
• pdf_url: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10750378
• html_url: https://ieeexplore.ieee.org/document/10750378/
• abstract_url: https://ieeexplore.ieee.org/document/10750378/
• publication_title: IEEE Transactions on Pattern Analysis and Machine Intelligence
• conference_location:
• conference_dates:
• publication_number: 34
• is_number: 10835210
• publication_year: 2025
• publication_date: Feb. 2025
• start_page: 1266
• end_page: 1278
• citing_paper_count: 0
• citing_patent_count: 0
• download_count: 308
• insert_date: 20241111
• index_terms:
  • ieee_terms:
    • Imaging
    • Image reconstruction
    • Cameras
    • Optical sensors
    • Optical imaging
    • High-speed optical techniques
    • Interpolation
    • Heuristic algorithms
    • Transformers
    • Ultrafast optics
  • author_terms:
    • Ultrafast imaging
    • snapshot compressive imaging
    • event camera
    • dual-path optical setup
    • dual-branch transformer
  • dynamic_index_terms:
    • Transformer
    • Intensity Measurements
    • Frame Rate
    • Video Frames
    • Optical Setup
    • Optic Apparatus
    • Imaging Strategy
    • Imaging Scheme
    • High-speed Imaging
    • Transmission Bandwidth
    • Intermediate Events
    • Dynamic Vision Sensor
    • Event Camera
    • Deep Network
    • Deep Neural Network
    • Running Time
    • Simulated Datasets
    • Dynamic Information
    • Kinetic Information
    • Reconstruction Algorithm
    • Peak Signal-to-noise Ratio
    • PSNR
    • Computer Image
    • Image Calculator
    • Reconstruction Results
    • Compression Ratio
    • Registration Parameters
    • Feature Extraction Block
    • Floating-point Operations
    • Conventional Camera
    • Reconstruction Performance
    • Middle Frame
    • Domain Gap
    • Intermediate Frames
    • High Dynamic Range
    • Fast Motion
• authors:
  • Author Name: Bo Zhang
    Affiliation: Department of Automation, Tsinghua University, Beijing, China
    Author URL: https://ieeexplore.ieee.org/author/37089079811
    ID: 37089079811
    Order: 1
    Author Affiliations:
    • Department of Automation, Tsinghua University, Beijing, China
  • Author Name: Jinli Suo
    Affiliation: Department of Automation, Tsinghua University, Beijing, China
    Author URL: https://ieeexplore.ieee.org/author/37390948800
    ID: 37390948800
    Order: 2
    Author Affiliations:
    • Department of Automation, Tsinghua University, Beijing, China
    • Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China
    • Shanghai Artificial Intelligence Laboratory, Shanghai, China
  • Author Name: Qionghai Dai
    Affiliation: Department of Automation, Tsinghua University, Beijing, China
    Author URL: https://ieeexplore.ieee.org/author/37273514100
    ID: 37273514100
    Order: 3
    Author Affiliations:
    • Department of Automation, Tsinghua University, Beijing, China
    • Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China

Image Sensor

• sensor_type: CMOS
• resolution: 2056x2464
• dynamic_range: not specified
• pixel_size: not specified
• dark_current: not specified

Optical Data

• focal_length: 50 mm
• aperture: not specified
• field_of_view: not specified
• distortion: not specified

Performance Metrics

• frame_rate: 24 fps; 10095 fps reconstructed

Applications & Benefits

• cell_imaging: not specified
• benefits: Enables high-speed imaging with low frame rate, improves image quality through advanced computational techniques.

Supporting Organizations

• supported_by: National Natural Science Foundation of China

Manuscript Details

• publication_date: Feb. 2025

Relevancy Score

• score: 9
• missing_fields:
  • dynamic_range
  • dark_current
  • pixel_size
  • signal_to_noise_ratio
  • noise
  • sensitivity
  • shutter_speed
  • power_consumption
  • aperture
  • field_of_view
  • distortion

Processed JSON Filename

request_2dbd8d26-4606-45b4-af59-0f07881c1615-eventenhanced_snapshot_compressive_videography_at_10k_fps.json