results: HyperKon 在 hyperspectral pan-sharpening 和 hyperspectral image classification 任务中表现出色，它的 Top-1 重建精度为 98%，而且在 hyperspectral image classification 任务中超过了现有的 state-of-the-art 方法，这表明 hyperspectral-native backbones 在 hyperspectral 图像分析中具有重要的作用。Abstract
The exceptional spectral resolution of hyperspectral imagery enables material insights that are not possible with RGB or multispectral images. Yet, the full potential of this data is often underutilized by deep learning techniques due to the scarcity of hyperspectral-native CNN backbones. To bridge this gap, we introduce HyperKon, a self-supervised contrastive learning network designed and trained on hyperspectral data from the EnMAP Hyperspectral Satellite\cite{kaufmann2012environmental}. HyperKon uniquely leverages the high spectral continuity, range, and resolution of hyperspectral data through a spectral attention mechanism and specialized convolutional layers. We also perform a thorough ablation study on different kinds of layers, showing their performance in understanding hyperspectral layers. It achieves an outstanding 98% Top-1 retrieval accuracy and outperforms traditional RGB-trained backbones in hyperspectral pan-sharpening tasks. Additionally, in hyperspectral image classification, HyperKon surpasses state-of-the-art methods, indicating a paradigm shift in hyperspectral image analysis and underscoring the importance of hyperspectral-native backbones.
摘要
<> hyperspectral imagery 的特殊频谱分辨率允许材质探测，其可能性不存在于RGB或多spectral图像中。然而，深度学习技术在使用这些数据时经常下Utilized，因为 hyperspectral-native CNN 背bone 的缺乏。为bridging这个差距，我们介绍 HyperKon，一种自我超VI的contrastive学习网络，设计并在EnMAP 卫星发射的 hyperspectral 数据上进行了训练。HyperKon 利用高 spectral 细化、范围和分辨率，通过 spectral 注意力机制和专门的 convolutional 层，实现了高效的材质理解。我们还进行了不同类型层的ablation研究，证明其在理解 hyperspectral 层的能力。它实现了98% 的 Top-1 检索精度，并在 hyperspectral 扫描任务中超过了传统RGB 训练的 backbone。此外，在 hyperspectral 图像分类任务中，HyperKon 超越了当前状态的方法，表明了一种新的材质图像分析 paradigm shift，并证明了 hyperspectral-native backbones 的重要性。<>
Deep Refinement-Based Joint Source Channel Coding over Time-Varying Channels
results: 与稳定通道条件下的主流DL-based JSCC方法相当，并且在时变通道条件下表现出了强大的Robustness。Abstract
In recent developments, deep learning (DL)-based joint source-channel coding (JSCC) for wireless image transmission has made significant strides in performance enhancement. Nonetheless, the majority of existing DL-based JSCC methods are tailored for scenarios featuring stable channel conditions, notably a fixed signal-to-noise ratio (SNR). This specialization poses a limitation, as their performance tends to wane in practical scenarios marked by highly dynamic channels, given that a fixed SNR inadequately represents the dynamic nature of such channels. In response to this challenge, we introduce a novel solution, namely deep refinement-based JSCC (DRJSCC). This innovative method is designed to seamlessly adapt to channels exhibiting temporal variations. By leveraging instantaneous channel state information (CSI), we dynamically optimize the encoding strategy through re-encoding the channel symbols. This dynamic adjustment ensures that the encoding strategy consistently aligns with the varying channel conditions during the transmission process. Specifically, our approach begins with the division of encoded symbols into multiple blocks, which are transmitted progressively to the receiver. In the event of changing channel conditions, we propose a mechanism to re-encode the remaining blocks, allowing them to adapt to the current channel conditions. Experimental results show that the DRJSCC scheme achieves comparable performance to the other mainstream DL-based JSCC models in stable channel conditions, and also exhibits great robustness against time-varying channels.
摘要
近期发展，深度学习（DL）基于联合源码混合（JSCC）技术在无线图像传输中取得了显著的进步。然而，大多数现有的DL基于JSCC方法都是针对具有稳定频率的通道条件设计的，即固定的信号听频率（SNR）。这种特殊化存在限制，因为它们在实际场景中具有高度动态的通道条件下表现不佳，因为固定的SNR无法准确表示动态的通道条件。为回应这个挑战，我们介绍了一种新的解决方案，即深度修正基于JSCC（DRJSCC）。这种创新的方法是为了贯穿时间变化的通道条件而设计的，通过利用实时的通道状态信息（CSI） dynamically 优化编码策略，以确保编码策略与通道条件的变化保持一致。具体来说，我们的方法是将编码符号分成多个块，然后将块传输给接收器进行进行编码。在通道条件发生变化时，我们提议使用重新编码剩下的块，以适应当前的通道条件。实验结果表明，DRJSCC 方案与其他主流的 DL 基于 JSCC 模型在稳定通道条件下具有相同的性能，并且在时间变化的通道条件下表现出了很好的鲁棒性。
Neural-Optic Co-Designed Polarization-Multiplexed Metalens for Compact Computational Spectral Imaging
results: 实验结果表明，该框架在实际应用场景中具有出色的空间-спектраль重建性能，证明了该系统在计算成像领域中的可行性和有效性。Abstract
As the realm of spectral imaging applications extends its reach into the domains of mobile technology and augmented reality, the demands for compact yet high-fidelity systems become increasingly pronounced. Conventional methodologies, exemplified by coded aperture snapshot spectral imaging systems, are significantly limited by their cumbersome physical dimensions and form factors. To address this inherent challenge, diffractive optical elements (DOEs) have been repeatedly employed as a means to mitigate issues related to the bulky nature of these systems. Nonetheless, it's essential to note that the capabilities of DOEs primarily revolve around the modulation of the phase of light. Here, we introduce an end-to-end computational spectral imaging framework based on a polarization-multiplexed metalens. A distinguishing feature of this approach lies in its capacity to simultaneously modulate orthogonal polarization channels. When harnessed in conjunction with a neural network, it facilitates the attainment of high-fidelity spectral reconstruction. Importantly, the framework is intrinsically fully differentiable, a feature that permits the joint optimization of both the metalens structure and the parameters governing the neural network. The experimental results presented herein validate the exceptional spatial-spectral reconstruction performance, underscoring the efficacy of this system in practical, real-world scenarios. This innovative approach transcends the traditional boundaries separating hardware and software in the realm of computational imaging and holds the promise of substantially propelling the miniaturization of spectral imaging systems.
摘要
随着spectral imaging应用领域扩展到移动技术和增强现实领域，需求更加强大、 yet compact high-fidelity system 的要求变得越来越明显。传统方法，例如coded aperture snapshot spectral imaging system，受限于其庞大的物理尺寸和形态因素。为了解决这一内在挑战，diffractive optical elements (DOEs) 被重复使用，以减轻spectral imaging system 的尺寸和重量。然而，需要注意的是，DOEs 的能力主要是修改光的频谱。在这里，我们提出了一种基于polarization-multiplexed metalens的 end-to-end计算 spectral imaging 框架。该approach的一个特点是同时修改两个垂直的极化通道。当与神经网络结合使用时，它可以实现高精度的频谱重建。进一步地，这种框架是完全可导的，这使得可以同时优化 metalens 结构和神经网络参数。实验结果表明，这种系统在实际应用场景中具有出色的空间-频谱重建性能，证明了该系统的可行性。这种创新的approach 突破了传统的硬件和软件之间的分界线，并且拥有极大地Minimize spectral imaging system 的大小。