paper_authors: Hu Zhang, Luziwei Leng, Kaiwei Che, Qian Liu, Jie Cheng, Qinghai Guo, Jiangxing Liao, Ran Cheng
for: 这篇论文旨在应用事件驱动神经网络（SNN）来进行事件基数据的物体探测。
methods: 这篇论文使用了SNN的传入层、遗传层和调节层，并使用了脉幅动力学来调节网络活动。
results: 这篇论文使用了SNN取得了47.7%的精确率（map50）在Gen1标准dataset上，比前一代SNN提高9.7%，并且比使用注意力机制的对照模型还要好。Abstract
Event-based sensors, with their high temporal resolution (1us) and dynamical range (120dB), have the potential to be deployed in high-speed platforms such as vehicles and drones. However, the highly sparse and fluctuating nature of events poses challenges for conventional object detection techniques based on Artificial Neural Networks (ANNs). In contrast, Spiking Neural Networks (SNNs) are well-suited for representing event-based data due to their inherent temporal dynamics. In particular, we demonstrate that the membrane potential dynamics can modulate network activity upon fluctuating events and strengthen features of sparse input. In addition, the spike-triggered adaptive threshold can stabilize training which further improves network performance. Based on this, we develop an efficient spiking feature pyramid network for event-based object detection. Our proposed SNN outperforms previous SNNs and sophisticated ANNs with attention mechanisms, achieving a mean average precision (map50) of 47.7% on the Gen1 benchmark dataset. This result significantly surpasses the previous best SNN by 9.7% and demonstrates the potential of SNNs for event-based vision. Our model has a concise architecture while maintaining high accuracy and much lower computation cost as a result of sparse computation. Our code will be publicly available.
摘要
Event-based 感测器，具有高度的时间分辨率（1us）和动态范围（120dB），有可能在高速平台 such as 车辆和无人机上进行部署。然而，事件的高度稀疏和波动性带来了传统的人工神经网络（ANNs）中的挑战。相比之下，脉冲神经网络（SNNs）因其内置的时间动力学而适用于表示事件基本数据。具体来说，我们表明了膜电位动力学可以在事件波动时调整网络活动，并强制特征的稀疏输入。此外，使用触发适应阈值可以稳定训练，从而进一步提高网络性能。基于这些，我们开发了高效的脉冲特征峰网络，用于事件基本对象检测。我们的提出的 SNN 在 Gen1 测试集上达到了47.7%的 Mean Average Precision（map50），这比前一代最佳 SNN 高出9.7%，并证明了 SNN 在事件基本视觉中的潜力。我们的模型具有简洁的架构，同时保持高度准确和较低的计算成本。我们的代码将公开。
Data-free Black-box Attack based on Diffusion Model
results: 通过使用干扰码修饰方法，可以使扩散模型生成的数据更加符合目标模型的批判标准，并且可以提高黑盒攻击的成功率和减少查询预算。Abstract
Since the training data for the target model in a data-free black-box attack is not available, most recent schemes utilize GANs to generate data for training substitute model. However, these GANs-based schemes suffer from low training efficiency as the generator needs to be retrained for each target model during the substitute training process, as well as low generation quality. To overcome these limitations, we consider utilizing the diffusion model to generate data, and propose a data-free black-box attack scheme based on diffusion model to improve the efficiency and accuracy of substitute training. Despite the data generated by the diffusion model exhibits high quality, it presents diverse domain distributions and contains many samples that do not meet the discriminative criteria of the target model. To further facilitate the diffusion model to generate data suitable for the target model, we propose a Latent Code Augmentation (LCA) method to guide the diffusion model in generating data. With the guidance of LCA, the data generated by the diffusion model not only meets the discriminative criteria of the target model but also exhibits high diversity. By utilizing this data, it is possible to train substitute model that closely resemble the target model more efficiently. Extensive experiments demonstrate that our LCA achieves higher attack success rates and requires fewer query budgets compared to GANs-based schemes for different target models.
摘要
由于目标模型的训练数据不可获取，最近的方案多利用GANs生成数据进行训练代理模型。然而，这些GANs基本方案受到低训练效率和低生成质量的限制。为了突破这些限制，我们考虑使用扩散模型来生成数据，并提出了基于扩散模型的数据自由黑盒攻击方案，以提高代理训练的效率和准确率。尽管扩散模型生成的数据具有高质量，但它们具有多样化的领域分布和含有大量不符合目标模型的检准标准的样本。为了更好地使用扩散模型生成数据，我们提出了秘密代码增强（LCA）方法，以导引扩散模型生成数据。通过LCA的引导，扩散模型生成的数据不仅符合目标模型的检准标准，而且具有高多样性。通过利用这些数据，我们可以更高效地训练代理模型，并且需要更少的查询预算。我们的LCA在不同的目标模型上实现了更高的攻击成功率和更低的查询预算。
Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry
paper_authors: Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, Youngjung Uh
for: 这 paper 旨在更好地理解Diffusion Model（DM）的潜在空间，从几何学角度进行分析。
methods: 这 paper 使用了 pullback 度量来找到 $\mathcal{X}$ 中的本地几何基底和其对应的 $\mathcal{H}$ 中的中间特征图。
results: 这 paper 发现了一种不supervised image editing能力，通过 traversal 在 $\mathbf{x}$ 空间中。此外，paper 还分析了这些结构如何随 diffusion 步骤的变化，以及如何基于文本条件进行修改。Abstract
Despite the success of diffusion models (DMs), we still lack a thorough understanding of their latent space. To understand the latent space $\mathbf{x}_t \in \mathcal{X}$, we analyze them from a geometrical perspective. Specifically, we utilize the pullback metric to find the local latent basis in $\mathcal{X}$ and their corresponding local tangent basis in $\mathcal{H}$, the intermediate feature maps of DMs. The discovered latent basis enables unsupervised image editing capability through latent space traversal. We investigate the discovered structure from two perspectives. First, we examine how geometric structure evolves over diffusion timesteps. Through analysis, we show that 1) the model focuses on low-frequency components early in the generative process and attunes to high-frequency details later; 2) At early timesteps, different samples share similar tangent spaces; and 3) The simpler datasets that DMs trained on, the more consistent the tangent space for each timestep. Second, we investigate how the geometric structure changes based on text conditioning in Stable Diffusion. The results show that 1) similar prompts yield comparable tangent spaces; and 2) the model depends less on text conditions in later timesteps. To the best of our knowledge, this paper is the first to present image editing through $\mathbf{x}$-space traversal and provide thorough analyses of the latent structure of DMs.
摘要
不withstanding the success of diffusion models (DMs), we still lack a thorough understanding of their latent space. To understand the latent space $\mathbf{x}_t \in \mathcal{X}$, we analyze them from a geometrical perspective. Specifically, we utilize the pullback metric to find the local latent basis in $\mathcal{X}$ and their corresponding local tangent basis in $\mathcal{H}$, the intermediate feature maps of DMs. The discovered latent basis enables unsupervised image editing capability through latent space traversal. We investigate the discovered structure from two perspectives. First, we examine how geometric structure evolves over diffusion timesteps. Through analysis, we show that 1) the model focuses on low-frequency components early in the generative process and attunes to high-frequency details later; 2) At early timesteps, different samples share similar tangent spaces; and 3) The simpler datasets that DMs trained on, the more consistent the tangent space for each timestep. Second, we investigate how the geometric structure changes based on text conditioning in Stable Diffusion. The results show that 1) similar prompts yield comparable tangent spaces; and 2) the model depends less on text conditions in later timesteps. To the best of our knowledge, this paper is the first to present image editing through $\mathbf{x}$-space traversal and provide thorough analyses of the latent structure of DMs.Note: The translation is in Simplified Chinese, which is the standardized form of Chinese used in mainland China and Singapore. The traditional Chinese form of the translation is slightly different due to the difference in word order and character usage.
Treatment Outcome Prediction for Intracerebral Hemorrhage via Generative Prognostic Model with Imaging and Tabular Data
For: 预测 интра脑出血(ICH) 治疗结果* Methods: 使用 imaging 和 tabular 数据建立报告模型，并使用 variational autoencoder 模型生成低维度预测分数，以Address selection bias* Results: 对实际临床数据进行了广泛的实验，并显示了substantial improvement 在治疗结果预测 compared to 现有的状态 искусственный智能方法。Here’s the breakdown of each point:* For: 预测 ICH 治疗结果 (What the paper is written for)* Methods: 使用 imaging 和 tabular 数据建立报告模型，并使用 variational autoencoder 模型生成低维度预测分数，以Address selection bias (What methods the paper uses)* Results: 对实际临床数据进行了广泛的实验，并显示了substantial improvement 在治疗结果预测 compared to 现有的状态 искусственный智能方法。 (What results the paper gets)Abstract
Intracerebral hemorrhage (ICH) is the second most common and deadliest form of stroke. Despite medical advances, predicting treat ment outcomes for ICH remains a challenge. This paper proposes a novel prognostic model that utilizes both imaging and tabular data to predict treatment outcome for ICH. Our model is trained on observational data collected from non-randomized controlled trials, providing reliable predictions of treatment success. Specifically, we propose to employ a variational autoencoder model to generate a low-dimensional prognostic score, which can effectively address the selection bias resulting from the non-randomized controlled trials. Importantly, we develop a variational distributions combination module that combines the information from imaging data, non-imaging clinical data, and treatment assignment to accurately generate the prognostic score. We conducted extensive experiments on a real-world clinical dataset of intracerebral hemorrhage. Our proposed method demonstrates a substantial improvement in treatment outcome prediction compared to existing state-of-the-art approaches. Code is available at https://github.com/med-air/TOP-GPM
摘要
中风血盖（ICH）是第二常见且最致命的 stroke 型。医学进步不withstanding，预测 IC 的治疗结果仍然是一大挑战。本文提出了一种新的 прогности 模型，该模型利用了 Both imaging 和 tabular data 预测 IC 的治疗结果。我们的模型在非随机化控制试验中收集的 observational 数据上进行训练，以提供可靠的治疗成功预测。特别是，我们提出了一种 variational autoencoder 模型，用于生成低维度的 прогности 分数，以有效地 Addressing the selection bias resulting from non-randomized controlled trials。我们还开发了一种 variational distributions combination module，用于组合 imaging 数据、非 imaging клиниче数据和治疗分配信息，以准确生成 prognostic 分数。我们对实际患有中风血盖的临床数据进行了广泛的实验，并证明了我们的提出方法可以具有显著改善的治疗结果预测效果，比现有的状态 искусственный智能方法更好。代码可以在 https://github.com/med-air/TOP-GPM 上找到。
Multiscale Video Pretraining for Long-Term Activity Forecasting
paper_authors: Reuben Tan, Matthias De Lange, Michael Iuzzolino, Bryan A. Plummer, Kate Saenko, Karl Ridgeway, Lorenzo Torresani
for: 预测人类活动长期趋势，提高机器学习模型对未看过数据的泛化能力。
methods: 提出了一种新的自我监督预训练方法—多 scales video pretraining (MVP)，通过学习预测视频片断的上下文化表示来学习Robust Representation。
results: 与现有方法进行比较，MVP在长期动作预测和视频概要预测任务上表现出了显著的性能优势，对于视频概要预测任务，MVP的表现提高了Relative Performance Gain超过20%。Abstract
Long-term activity forecasting is an especially challenging research problem because it requires understanding the temporal relationships between observed actions, as well as the variability and complexity of human activities. Despite relying on strong supervision via expensive human annotations, state-of-the-art forecasting approaches often generalize poorly to unseen data. To alleviate this issue, we propose Multiscale Video Pretraining (MVP), a novel self-supervised pretraining approach that learns robust representations for forecasting by learning to predict contextualized representations of future video clips over multiple timescales. MVP is based on our observation that actions in videos have a multiscale nature, where atomic actions typically occur at a short timescale and more complex actions may span longer timescales. We compare MVP to state-of-the-art self-supervised video learning approaches on downstream long-term forecasting tasks including long-term action anticipation and video summary prediction. Our comprehensive experiments across the Ego4D and Epic-Kitchens-55/100 datasets demonstrate that MVP out-performs state-of-the-art methods by significant margins. Notably, MVP obtains a relative performance gain of over 20% accuracy in video summary forecasting over existing methods.
摘要
长期活动预测是一个特别困难的研究问题，因为它需要理解视频中观察到的动作之间的时间关系，以及人类活动的多样性和复杂性。尽管通过昂贵的人工纠正，使用现状的预测方法可以取得良好的性能，但这些方法在未看到的数据上 generalize 很差。为了解决这个问题，我们提出了多Scale Video Pretraining（MVP），一种新的自我监督预训练方法，该方法通过学习预测视频clip的多Scale representation来学习Robust的表示。MVP基于我们观察到的动作在视频中有多Scale性质， atomic 动作通常发生在短时间尺度，而更复杂的动作可能 span 更长的时间尺度。我们与现状的自我监督视频学习方法进行比较，在下游长期预测任务中，包括长期动作预测和视频概要预测。我们在Ego4D和Epic-Kitchens-55/100 datasets上进行了广泛的实验，结果表明MVP在性能上超过了现状的方法，具体来说，MVP在视频概要预测任务中取得了20%以上的相对性能提升。
Spatiotemporal Modeling Encounters 3D Medical Image Analysis: Slice-Shift UNet with Multi-View Fusion
results: 对于Multi-Modality Abdominal Multi-Organ Segmentation (AMOS)和Multi-Atlas Labeling Beyond the Cranial Vault (BTCV)数据集，这paper的方法得到了较高的效率和相当于顶尖模型的性能。Abstract
As a fundamental part of computational healthcare, Computer Tomography (CT) and Magnetic Resonance Imaging (MRI) provide volumetric data, making the development of algorithms for 3D image analysis a necessity. Despite being computationally cheap, 2D Convolutional Neural Networks can only extract spatial information. In contrast, 3D CNNs can extract three-dimensional features, but they have higher computational costs and latency, which is a limitation for clinical practice that requires fast and efficient models. Inspired by the field of video action recognition we propose a new 2D-based model dubbed Slice SHift UNet (SSH-UNet) which encodes three-dimensional features at 2D CNN's complexity. More precisely multi-view features are collaboratively learned by performing 2D convolutions along the three orthogonal planes of a volume and imposing a weights-sharing mechanism. The third dimension, which is neglected by the 2D convolution, is reincorporated by shifting a portion of the feature maps along the slices' axis. The effectiveness of our approach is validated in Multi-Modality Abdominal Multi-Organ Segmentation (AMOS) and Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) datasets, showing that SSH-UNet is more efficient while on par in performance with state-of-the-art architectures.
摘要
为computational healthcare的基本部分，计算机断层成像（CT）和核磁共振成像（MRI）提供了体积数据，因此开发3D图像分析算法是必要的。然而，使用2D卷积神经网络（CNN）只能提取空间信息，而3D CNN则可以提取三维特征，但它们的计算成本和延迟较高，这限制了临床实践中需要快速和高效的模型。 drawing inspiration from the field of video action recognition, we propose a new 2D-based model called Slice SHift UNet (SSH-UNet) that encodes three-dimensional features at the complexity of 2D CNNs. Specifically, we perform 2D convolutions along the three orthogonal planes of a volume and share weights across different planes to collaboratively learn multi-view features. The third dimension, which is neglected by the 2D convolution, is reincorporated by shifting a portion of the feature maps along the slices' axis. Our approach is validated on the Multi-Modality Abdominal Multi-Organ Segmentation (AMOS) and Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) datasets, showing that SSH-UNet is more efficient while on par in performance with state-of-the-art architectures.
Multi-View Vertebra Localization and Identification from CT Images
results: 我们的方法可以在只使用两个2D网络的情况下，准确地位置和识别骨架在CT图像中。并且与状态艺术方法相比，我们的方法显著超越了其表现。代码可以在https://github.com/ShanghaiTech-IMPACT/Multi-View-Vertebra-Localization-and-Identification-from-CT-Images上获取。Abstract
Accurately localizing and identifying vertebrae from CT images is crucial for various clinical applications. However, most existing efforts are performed on 3D with cropping patch operation, suffering from the large computation costs and limited global information. In this paper, we propose a multi-view vertebra localization and identification from CT images, converting the 3D problem into a 2D localization and identification task on different views. Without the limitation of the 3D cropped patch, our method can learn the multi-view global information naturally. Moreover, to better capture the anatomical structure information from different view perspectives, a multi-view contrastive learning strategy is developed to pre-train the backbone. Additionally, we further propose a Sequence Loss to maintain the sequential structure embedded along the vertebrae. Evaluation results demonstrate that, with only two 2D networks, our method can localize and identify vertebrae in CT images accurately, and outperforms the state-of-the-art methods consistently. Our code is available at https://github.com/ShanghaiTech-IMPACT/Multi-View-Vertebra-Localization-and-Identification-from-CT-Images.
摘要
通过CT图像进行精准的骨vertebrae的本地化和识别是许多临床应用中的关键。然而，大多数现有的尝试都是在3D中进行，通过剪辑patch操作，它们受到了大量计算成本和局部信息的限制。在这篇论文中，我们提出了基于多视图的骨vertebrae本地化和识别方法，将3D问题转化为2D本地化和识别任务。不同于剪辑patch的限制，我们的方法可以自然地学习多视图的全局信息。此外，为了更好地捕捉不同视图角度的骨 vertebrae结构信息，我们开发了一种多视图冲击学习策略来预训练干部。此外，我们还提出了一种序列损失来保持骨vertebrae中的顺序结构嵌入。评估结果表明，只有两个2D网络，我们的方法可以在CT图像中精准地本地化和识别骨vertebrae，并在状态艺术方法上一直保持领先。我们的代码可以在https://github.com/ShanghaiTech-IMPACT/Multi-View-Vertebra-Localization-and-Identification-from-CT-Images中找到。
Learning Provably Robust Estimators for Inverse Problems via Jittering
for: 本研究 investigate whether jittering, a simple regularization technique, can be used to train efficient worst-case robust deep neural networks for inverse problems.
results: 研究发现，使用jittering可以实现最佳的 $\ell_2$-worst-case robust estimator for linear denoising,并且通过训练深度神经网络（U-net）对自然图像减 noise、deconvolution和加速Magnetic Resonance Imaging（MRI）进行了实验，结果表明，jittering可以增强最坏情况Robustness，但可能不适用于 inverse problems beyond denoising。此外，研究还发现，训练在真实数据上，通常包含一些噪声，可以提高模型的Robustness。Abstract
Deep neural networks provide excellent performance for inverse problems such as denoising. However, neural networks can be sensitive to adversarial or worst-case perturbations. This raises the question of whether such networks can be trained efficiently to be worst-case robust. In this paper, we investigate whether jittering, a simple regularization technique that adds isotropic Gaussian noise during training, is effective for learning worst-case robust estimators for inverse problems. While well studied for prediction in classification tasks, the effectiveness of jittering for inverse problems has not been systematically investigated. In this paper, we present a novel analytical characterization of the optimal $\ell_2$-worst-case robust estimator for linear denoising and show that jittering yields optimal robust denoisers. Furthermore, we examine jittering empirically via training deep neural networks (U-nets) for natural image denoising, deconvolution, and accelerated magnetic resonance imaging (MRI). The results show that jittering significantly enhances the worst-case robustness, but can be suboptimal for inverse problems beyond denoising. Moreover, our results imply that training on real data which often contains slight noise is somewhat robustness enhancing.
摘要
深度神经网络在逆问题中提供了优秀的性能，但神经网络可能会受到敌意或最坏情况的攻击。这引发了我们是否可以有效地训练神经网络以适应最坏情况的问题。在这篇论文中，我们研究了加入 ISO 型 Gaussian 噪声 durante 训练是否可以有效地学习最坏情况Robust 估计器。虽然这种技术在预测 classification 任务中已经得到了广泛的研究，但对于逆问题的研究尚未得到系统的探讨。在这篇论文中，我们提出了一种新的分析方法，用于计算最佳 $\ell_2$ 最坏情况Robust 估计器。我们发现，在线性噪声降解 задании下，加入噪声可以实现最佳的Robust 性能。此外，我们通过训练深度神经网络（U-net）进行自然图像降解、减 convolution 和加速核磁共振成像（MRI）任务，实验结果表明，加入噪声可以显著提高最坏情况Robust 性能，但在涉及到逆问题的更高级别上可能不是最佳的方法。此外，我们的结果还 imply 训练在真实数据上，常常包含一定的噪声，可能会有一定的 robustness 提升。
Exposing the Troublemakers in Described Object Detection
methods: 我们采用了Description Detection Dataset ($D^3$)，其特点是包含flexible language expressions和全面的描述对象标注，并评估了之前的SOTA方法在$D^3$上的性能。此外，我们还提出了一种基线方法，通过重新构建训练数据和引入二分类子任务来大幅提高REC方法的性能。
results: 我们发现现有REC方法在$D^3$上存在许多问题，包括信任分数、拒绝负例、多目标场景等。此外，最新的bi-functional方法也不太适用于DOD任务，因为它们在训练和测试过程中分离的程序和推理策略。我们的基线方法在$D^3$上表现出色，大幅超越现有方法。Abstract
Detecting objects based on language descriptions is a popular task that includes Open-Vocabulary object Detection (OVD) and Referring Expression Comprehension (REC). In this paper, we advance them to a more practical setting called Described Object Detection (DOD) by expanding category names to flexible language expressions for OVD and overcoming the limitation of REC to only grounding the pre-existing object. We establish the research foundation for DOD tasks by constructing a Description Detection Dataset ($D^3$), featuring flexible language expressions and annotating all described objects without omission. By evaluating previous SOTA methods on $D^3$, we find some troublemakers that fail current REC, OVD, and bi-functional methods. REC methods struggle with confidence scores, rejecting negative instances, and multi-target scenarios, while OVD methods face constraints with long and complex descriptions. Recent bi-functional methods also do not work well on DOD due to their separated training procedures and inference strategies for REC and OVD tasks. Building upon the aforementioned findings, we propose a baseline that largely improves REC methods by reconstructing the training data and introducing a binary classification sub-task, outperforming existing methods. Data and code is available at https://github.com/shikras/d-cube.
摘要
检测基于语言描述的对象是一个受欢迎的任务，包括开放词汇对象检测（OVD）和引用表达理解（REC）。在这篇论文中，我们将其推进到更实用的设定中，称为描述对象检测（DOD），通过扩展类别名称到 flexible language expression 来进一步提高 OVD 的精度。我们建立了描述检测任务的研究基础，constructing a Description Detection Dataset ($D^3$), featuring flexible language expressions and annotating all described objects without omission。通过评估先前的 SOTA 方法在 $D^3$ 上，我们发现了一些“困难者”，其中 REC 方法受到负样本、多目标场景和 confidence score 等限制，而 OVD 方法则面临长和复杂的描述句所带来的限制。最近的 bi-functional 方法也不太适合 DOD 任务，因为它们在 REC 和 OVD 任务之间具有分开的训练过程和推理策略。基于以上发现，我们提出了一个基线方案，可以大幅提高 REC 方法的性能，通过重新构建训练数据和引入 binary classification 子任务。数据和代码可以在 https://github.com/shikras/d-cube 上获取。
Compact & Capable: Harnessing Graph Neural Networks and Edge Convolution for Medical Image Classification
results: 比较 GNN 和 pre-trained DNNs，GNN 能够在 MedMNIST 数据集上显示出与 DNNs 相似的表现，并且仅需要1000个参数，相较于 DNNs 的训练时间和数据量。Abstract
Graph-based neural network models are gaining traction in the field of representation learning due to their ability to uncover latent topological relationships between entities that are otherwise challenging to identify. These models have been employed across a diverse range of domains, encompassing drug discovery, protein interactions, semantic segmentation, and fluid dynamics research. In this study, we investigate the potential of Graph Neural Networks (GNNs) for medical image classification. We introduce a novel model that combines GNNs and edge convolution, leveraging the interconnectedness of RGB channel feature values to strongly represent connections between crucial graph nodes. Our proposed model not only performs on par with state-of-the-art Deep Neural Networks (DNNs) but does so with 1000 times fewer parameters, resulting in reduced training time and data requirements. We compare our Graph Convolutional Neural Network (GCNN) to pre-trained DNNs for classifying MedMNIST dataset classes, revealing promising prospects for GNNs in medical image analysis. Our results also encourage further exploration of advanced graph-based models such as Graph Attention Networks (GAT) and Graph Auto-Encoders in the medical imaging domain. The proposed model yields more reliable, interpretable, and accurate outcomes for tasks like semantic segmentation and image classification compared to simpler GCNNs
摘要
“基于图的神经网络模型在 Representation Learning 领域受到广泛应用，因为它们可以捕捉到难以识别的实体之间的隐藏 topological 关系。这些模型在药物发现、蛋白结合、 semantic segmentation 和 fluid dynamics 等领域都有应用。在这项研究中，我们研究了医学图像分类中 Graph Neural Networks（GNNs）的潜力。我们提出了一种新的模型，它将 GNNs 和边 convolution 结合起来，利用 RGB 通道特征值之间的连接来强大地表示图像中关键节点之间的连接。我们的提出的模型不仅与 state-of-the-art Deep Neural Networks（DNNs）的性能相当，而且只需要1000倍 fewer 参数，从而减少了训练时间和数据需求。我们对 MedMNIST 数据集类型进行比较，发现 GNNs 在医学图像分类中有极好的前景。我们的结果还鼓励了在医学图像分类和 semantic segmentation 等任务中进一步探索更高级别的图基于模型，如 Graph Attention Networks（GAT）和 Graph Auto-Encoders。我们的模型在 semantic segmentation 和图像分类任务中比 simpler GNNs 更可靠、更加可解释、更高精度。”
Fast Full-frame Video Stabilization with Iterative Optimization
results: 提供了高效且视觉质量高的视频稳定方法，并通过实验证明其在计算速度和视觉质量两个方面的优势Abstract
Video stabilization refers to the problem of transforming a shaky video into a visually pleasing one. The question of how to strike a good trade-off between visual quality and computational speed has remained one of the open challenges in video stabilization. Inspired by the analogy between wobbly frames and jigsaw puzzles, we propose an iterative optimization-based learning approach using synthetic datasets for video stabilization, which consists of two interacting submodules: motion trajectory smoothing and full-frame outpainting. First, we develop a two-level (coarse-to-fine) stabilizing algorithm based on the probabilistic flow field. The confidence map associated with the estimated optical flow is exploited to guide the search for shared regions through backpropagation. Second, we take a divide-and-conquer approach and propose a novel multiframe fusion strategy to render full-frame stabilized views. An important new insight brought about by our iterative optimization approach is that the target video can be interpreted as the fixed point of nonlinear mapping for video stabilization. We formulate video stabilization as a problem of minimizing the amount of jerkiness in motion trajectories, which guarantees convergence with the help of fixed-point theory. Extensive experimental results are reported to demonstrate the superiority of the proposed approach in terms of computational speed and visual quality. The code will be available on GitHub.
摘要
视频稳定化问题是将晃动视频转化为美观的视频。计算速度和视觉质量之间的平衡问题一直是视频稳定化领域的开放问题。我们提出了基于iterative optimization-based learning方法的视频稳定化方法，该方法包括两个互动子模块：运动轨迹缓和全帧补充。首先，我们开发了一种两级（粗细到细）稳定化算法，基于概率流场。利用估算的光学流场的信息来引导搜索共享区域，我们使用回传propagation。其次，我们提出了一种新的分解和融合策略，以生成稳定视频全帧视图。我们发现，通过iterative optimization方法，目标视频可以被视为非线性映射的稳定点，我们将视频稳定化问题解释为减少运动轨迹中的抖动量，以 garantía converge。我们通过实验报告了我们的方法的计算速度和视觉质量的超越性。代码将在GitHub上发布。
paper_authors: Wolfgang Boettcher, Lukas Hoyer, Ozan Unal, Ke Li, Dengxin Dai
for: 提高移动自适应系统中的深度估计精度
methods: 动态适应LiDAR扫描模式的深度完成网络
results: 与专业LiDAR模型相比，单个模型可以在多个LiDAR扫描模式上显示更好的性能，并且可以扩展到未经训练的扫描模式Abstract
Depth estimation is one of the essential tasks to be addressed when creating mobile autonomous systems. While monocular depth estimation methods have improved in recent times, depth completion provides more accurate and reliable depth maps by additionally using sparse depth information from other sensors such as LiDAR. However, current methods are specifically trained for a single LiDAR sensor. As the scanning pattern differs between sensors, every new sensor would require re-training a specialized depth completion model, which is computationally inefficient and not flexible. Therefore, we propose to dynamically adapt the depth completion model to the used sensor type enabling LiDAR adaptive depth completion. Specifically, we propose a meta depth completion network that uses data patterns derived from the data to learn a task network to alter weights of the main depth completion network to solve a given depth completion task effectively. The method demonstrates a strong capability to work on multiple LiDAR scanning patterns and can also generalize to scanning patterns that are unseen during training. While using a single model, our method yields significantly better results than a non-adaptive baseline trained on different LiDAR patterns. It outperforms LiDAR-specific expert models for very sparse cases. These advantages allow flexible deployment of a single depth completion model on different sensors, which could also prove valuable to process the input of nascent LiDAR technology with adaptive instead of fixed scanning patterns.
摘要
深度估算是创建移动自主系统中的一项重要任务。虽然单投射深度估算方法在最近有所进步，但深度完成提供更加准确和可靠的深度地图，并使用其他感知器 such as LiDAR 上的稀疏深度信息。然而，当前的方法都是特定的 LiDAR 扫描模式进行训练。因此，我们提出了动态适应 LiDAR 类型的深度完成模型，以便在不同的 LiDAR 扫描模式上进行适应性的深度完成。具体来说，我们提出了一个元深度完成网络，通过数据模式来学习一个任务网络，以修改主深度完成网络中的权重，以解决给定的深度完成任务。该方法能够在多个 LiDAR 扫描模式上工作，并且还能够对未在训练中看到的扫描模式进行泛化。使用单一模型，我们的方法能够在不同 LiDAR 扫描模式上显著提高结果，并且在非常稀疏的情况下也能够超越特定 LiDAR 模型。这些优势使得我们的方法可以在不同感知器上进行灵活的部署，这也可以证明有用于处理新型 LiDAR 技术的适应式扫描模式。
ICF-SRSR: Invertible scale-Conditional Function for Self-Supervised Real-world Single Image Super-Resolution
results: 该论文的实验表明，ICF-SRSR可以在无监督情况下处理SISR问题，并且在实际世界中表现更好于使用synthesized paired图像进行训练的方法。此外，ICF-SRSR还可以生成更加真实和可行的LR-HR对，使现有的监督SISR网络更加可靠。Abstract
Single image super-resolution (SISR) is a challenging ill-posed problem that aims to up-sample a given low-resolution (LR) image to a high-resolution (HR) counterpart. Due to the difficulty in obtaining real LR-HR training pairs, recent approaches are trained on simulated LR images degraded by simplified down-sampling operators, e.g., bicubic. Such an approach can be problematic in practice because of the large gap between the synthesized and real-world LR images. To alleviate the issue, we propose a novel Invertible scale-Conditional Function (ICF), which can scale an input image and then restore the original input with different scale conditions. By leveraging the proposed ICF, we construct a novel self-supervised SISR framework (ICF-SRSR) to handle the real-world SR task without using any paired/unpaired training data. Furthermore, our ICF-SRSR can generate realistic and feasible LR-HR pairs, which can make existing supervised SISR networks more robust. Extensive experiments demonstrate the effectiveness of the proposed method in handling SISR in a fully self-supervised manner. Our ICF-SRSR demonstrates superior performance compared to the existing methods trained on synthetic paired images in real-world scenarios and exhibits comparable performance compared to state-of-the-art supervised/unsupervised methods on public benchmark datasets.
摘要
单图超分辨 (SISR) 是一个具有挑战性的不定性问题，旨在将给定的低分辨 (LR) 图像提升到高分辨 (HR) 对应的图像。由于实际获得LR-HR训练对的困难，现有的方法通常是使用简化下采样算法，如二次方程，进行模拟LR图像。这种方法在实践中可能存在大量的差异，这使得SR任务变得更加困难。为了解决这个问题，我们提议一种新的归一化可逆函数 (ICF)，可以将输入图像缩放，然后将原始输入图像还原，并且可以在不同的比例条件下进行缩放。通过利用我们提议的ICF，我们建立了一种新的自适应SISR框架 (ICF-SRSR)，可以在实际的SR任务中处理不需要使用任何配对/非配对训练数据。此外，我们的ICF-SRSR可以生成真实可行的LR-HR对，这可以使得现有的supervised SISR网络更加强大。我们进行了广泛的实验，并证明了我们的ICF-SRSR可以在不需要任何配对训练数据的情况下 Handle SISR任务。我们的ICF-SRSR在实际场景中表现出了supervised/非配对方法的相当性，并且在公共的benchmark datasets上达到了相当的性能。
CLIP-KD: An Empirical Study of Distilling CLIP Models
results: 研究发现最简单的特征模仿与MSE损失最佳，并且交互对比学习和关系基于精简也对性能有益。在应用于多个学生网络，CLIP精简在零shotImageNet分类和跨模态检索benchmark上具有consistent的改进。Abstract
CLIP has become a promising language-supervised visual pre-training framework and achieves excellent performance over a wide range of tasks. This paper aims to distill small CLIP models supervised by a large teacher CLIP model. We propose several distillation strategies, including relation, feature, gradient and contrastive paradigm, to examine the impact on CLIP distillation. We show that the simplest feature mimicry with MSE loss performs best. Moreover, interactive contrastive learning and relation-based distillation are also critical in performance improvement. We apply the unified method to distill several student networks trained on 15 million (image, text) pairs. Distillation improves the student CLIP models consistently over zero-shot ImageNet classification and cross-modal retrieval benchmarks. We hope our empirical study will become an important baseline for future CLIP distillation research. The code is available at \url{https://github.com/winycg/CLIP-KD}.
摘要
CLIP 已成为一个有前途的语言监督 visual 预训练框架，在各种任务上表现出色。本文目的是将大 teacher CLIP 模型监督小 CLIP 模型进行学习减少。我们提出了多种减少策略，包括关系、特征、梯度和对比方法，以确定 CLIP 减少的影响。我们发现最简单的特征模仿方法使用 MSE 损失最佳。此外，交互式对比学习和关系基于减少也对性能进行了贡献。我们应用这种简化方法到多个学生网络，每个网络都在 15 百万（图像、文本）对的训练下进行学习。减少提高了学生 CLIP 模型在零shot ImageNet 分类和 cross-modal 检索标准准则上的表现。我们希望我们的实验研究将成为未来 CLIP 减少研究的重要基准。代码可以在 \url{https://github.com/winycg/CLIP-KD} 上获取。
COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts
paper_authors: Xiaofeng Mao, Yuefeng Chen, Yao Zhu, Da Chen, Hang Su, Rong Zhang, Hui Xue for:This paper aims to provide a comprehensive assessment of the robustness of object detection models under natural distribution shifts, and to introduce a new test dataset called COCO-O to benchmark the OOD robustness of detectors.methods:The authors use a large-scale dataset called COCO-O, which contains six types of natural distribution shifts, to evaluate the OOD robustness of more than 100 modern object detection models. They also conduct experiments to study the effect of different components of the detector, such as the backbone, detection head, and augmentation techniques, on its OOD robustness.results:The authors find that most classic detectors do not exhibit strong OOD generalization, and that the backbone is the most important part for robustness. They also find that an end-to-end detection transformer design does not bring any enhancement, and may even reduce robustness. Finally, they show that large-scale foundation models have made a great leap on robust object detection. The dataset will be available at https://github.com/alibaba/easyrobust/tree/main/benchmarks/coco_o.Here is the Chinese version of the three key points:for:这篇论文的目的是为了提供一个广泛的对象检测模型在自然分布变化下的robustness的评估，并为了 benchmarking 对象检测模型的OOD robustness而引入了COCO-O测试集。methods:作者使用了COCO-O测试集，这个测试集包含6种自然分布变化，来评估现代对象检测模型的OOD robustness。他们还进行了不同组件的检测器，如背bone、检测头和增强技术的效果的研究。results:作者发现大多数经典的检测器在自然分布变化下并没有强大的OOD普适性，背bone是检测器的robustness中最重要的部分。他们还发现，检测transformer设计并不提供任何改进，可能甚至会降低robustness。最后，他们发现大规模基础模型在robust对象检测方面做出了很大的进步。Abstract
Practical object detection application can lose its effectiveness on image inputs with natural distribution shifts. This problem leads the research community to pay more attention on the robustness of detectors under Out-Of-Distribution (OOD) inputs. Existing works construct datasets to benchmark the detector's OOD robustness for a specific application scenario, e.g., Autonomous Driving. However, these datasets lack universality and are hard to benchmark general detectors built on common tasks such as COCO. To give a more comprehensive robustness assessment, we introduce COCO-O(ut-of-distribution), a test dataset based on COCO with 6 types of natural distribution shifts. COCO-O has a large distribution gap with training data and results in a significant 55.7% relative performance drop on a Faster R-CNN detector. We leverage COCO-O to conduct experiments on more than 100 modern object detectors to investigate if their improvements are credible or just over-fitting to the COCO test set. Unfortunately, most classic detectors in early years do not exhibit strong OOD generalization. We further study the robustness effect on recent breakthroughs of detector's architecture design, augmentation and pre-training techniques. Some empirical findings are revealed: 1) Compared with detection head or neck, backbone is the most important part for robustness; 2) An end-to-end detection transformer design brings no enhancement, and may even reduce robustness; 3) Large-scale foundation models have made a great leap on robust object detection. We hope our COCO-O could provide a rich testbed for robustness study of object detection. The dataset will be available at https://github.com/alibaba/easyrobust/tree/main/benchmarks/coco_o.
摘要
实际应用中的物体检测应用可能会在自然分布变化下失效。这个问题导致研究人员更加关注检测器在不同分布下的Robustness。现有的 dataset 用于测试检测器的 OOD Robustness，如 autonomous driving 应用场景。然而，这些 dataset 缺乏通用性，难以测试基于 common task 如 COCO 的检测器。为了给出更加全面的 Robustness 评估，我们引入 COCO-O（out-of-distribution）测试集，基于 COCO 的6种自然分布变化。COCO-O 与训练数据之间存在大的分布差，导致 Faster R-CNN 检测器的Relative performance drop 为 55.7%。我们使用 COCO-O 对 более than 100 种现代 object detection 算法进行实验，以查找他们是否具有可靠的 OOD 通用性。结果显示，大多数早期的检测器不具有强的 OOD 泛化能力。我们进一步研究了检测器的architecture design，增强技术和预训练技术的Robustness效果。我们发现：1）与检测头或 neck 相比，后凹是检测器 Robustness 的关键部分; 2） end-to-end 检测转换设计不具有改善效果，可能 même 减少 Robustness; 3）大规模基础模型在 Robust object detection 方面做出了很大的进步。我们希望 COCO-O 可以为 Robustness 研究提供一个丰富的测试平台。 dataset 将在 https://github.com/alibaba/easyrobust/tree/main/benchmarks/coco_o 上提供。
Persistent-Transient Duality: A Multi-mechanism Approach for Modeling Human-Object Interaction
results: 在两个丰富的数据集和多种设置下，这个模型在HOI动作预测中具有了superior的性能，证明了其适用性。Abstract
Humans are highly adaptable, swiftly switching between different modes to progressively handle different tasks, situations and contexts. In Human-object interaction (HOI) activities, these modes can be attributed to two mechanisms: (1) the large-scale consistent plan for the whole activity and (2) the small-scale children interactive actions that start and end along the timeline. While neuroscience and cognitive science have confirmed this multi-mechanism nature of human behavior, machine modeling approaches for human motion are trailing behind. While attempted to use gradually morphing structures (e.g., graph attention networks) to model the dynamic HOI patterns, they miss the expeditious and discrete mode-switching nature of the human motion. To bridge that gap, this work proposes to model two concurrent mechanisms that jointly control human motion: the Persistent process that runs continually on the global scale, and the Transient sub-processes that operate intermittently on the local context of the human while interacting with objects. These two mechanisms form an interactive Persistent-Transient Duality that synergistically governs the activity sequences. We model this conceptual duality by a parent-child neural network of Persistent and Transient channels with a dedicated neural module for dynamic mechanism switching. The framework is trialed on HOI motion forecasting. On two rich datasets and a wide variety of settings, the model consistently delivers superior performances, proving its suitability for the challenge.
摘要
人类具有高度适应性，快速 switching между不同模式以逐渐处理不同任务、情况和上下文。在人机交互（HOI）活动中，这些模式可以归结为两种机制：（1）整体活动的大规模一致计划，和（2）在时间轴上开始和结束的小规模儿童交互动作。而神经科学和认知科学已经证实了人类行为的多机制性，但机器模型人体运动的方法尚未跟上。尝试使用渐变变换结构（如图像注意力网络）来模型动态 HOI 模式，但它们缺乏人类运动快速和精细的模式转换特点。为了bridging这个差距，本工作提出了同时控制人体运动的两个机制：持续过程，该过程在全局范围内一直运行，和间歇性子进程，该子进程在人类与对象交互时在本地上运行。这两种机制组成了互动的持续-间歇性双重性，这种双重性同时控制活动序列。我们使用父母-孩子神经网络，其中持续通道和间歇性通道各自拥有特定的神经元模块，以实现动态机制转换。这种概念双重性在 HOI 动作预测中得到证明，在两个丰富的数据集和多种设置下，模型一致地表现出优秀的表现，证明其适用性。
AMAE: Adaptation of Pre-Trained Masked Autoencoder for Dual-Distribution Anomaly Detection in Chest X-Rays
results: AMAE在不同的异常比例下评估了其效果，并与其他自动标注和双分布异常检测方法进行比较。结果显示，AMAE在三个公开的胸部X-raybenchmark上设置了新的State-of-the-art。Abstract
Unsupervised anomaly detection in medical images such as chest radiographs is stepping into the spotlight as it mitigates the scarcity of the labor-intensive and costly expert annotation of anomaly data. However, nearly all existing methods are formulated as a one-class classification trained only on representations from the normal class and discard a potentially significant portion of the unlabeled data. This paper focuses on a more practical setting, dual distribution anomaly detection for chest X-rays, using the entire training data, including both normal and unlabeled images. Inspired by a modern self-supervised vision transformer model trained using partial image inputs to reconstruct missing image regions -- we propose AMAE, a two-stage algorithm for adaptation of the pre-trained masked autoencoder (MAE). Starting from MAE initialization, AMAE first creates synthetic anomalies from only normal training images and trains a lightweight classifier on frozen transformer features. Subsequently, we propose an adaptation strategy to leverage unlabeled images containing anomalies. The adaptation scheme is accomplished by assigning pseudo-labels to unlabeled images and using two separate MAE based modules to model the normative and anomalous distributions of pseudo-labeled images. The effectiveness of the proposed adaptation strategy is evaluated with different anomaly ratios in an unlabeled training set. AMAE leads to consistent performance gains over competing self-supervised and dual distribution anomaly detection methods, setting the new state-of-the-art on three public chest X-ray benchmarks: RSNA, NIH-CXR, and VinDr-CXR.
摘要
不监督异常检测在医学影像中，如胸部X射线图像，正在受到关注，因为它解决了专业人员对异常数据的繁琐和成本的批注短缺。然而，大多数现有方法都是基于一个类型的分类，只使用正常类的表示来训练。这篇论文关注了更实际的设定：对胸部X射线图像进行双分布异常检测，使用整个训练数据，包括正常和无标记图像。 Drawing inspiration from a modern self-supervised vision transformer model trained using partial image inputs to reconstruct missing image regions，我们提出了AMAE算法，一个两个阶段的算法，用于适应预先训练的封闭自动encoder（MAE）。从MAE初始化开始，AMAE首先将正常训练图像中生成假异常，并训练一个轻量级分类器。接着，我们提出了适应策略，使用无标记图像中含异常的图像进行适应。这种适应策略是通过将无标记图像 assign pseudo-labels，并使用两个基于MAE模块来模型无标记图像的正常和异常分布。我们对不同异常比例的无标记训练集进行评估，AMAE在自我超vised和双分布异常检测方法中表现出优异，创造了新的state-of-the-art在三个公共胸部X射线标准benchmark上：RSNA、NIH-CXR和VinDr-CXR。
CarPatch: A Synthetic Benchmark for Radiance Field Evaluation on Vehicle Components
results: 本研究提供了一个公共可用的Synthetic benchmark dataset，可以作为评估和比较不同技术的导向。Abstract
Neural Radiance Fields (NeRFs) have gained widespread recognition as a highly effective technique for representing 3D reconstructions of objects and scenes derived from sets of images. Despite their efficiency, NeRF models can pose challenges in certain scenarios such as vehicle inspection, where the lack of sufficient data or the presence of challenging elements (e.g. reflections) strongly impact the accuracy of the reconstruction. To this aim, we introduce CarPatch, a novel synthetic benchmark of vehicles. In addition to a set of images annotated with their intrinsic and extrinsic camera parameters, the corresponding depth maps and semantic segmentation masks have been generated for each view. Global and part-based metrics have been defined and used to evaluate, compare, and better characterize some state-of-the-art techniques. The dataset is publicly released at https://aimagelab.ing.unimore.it/go/carpatch and can be used as an evaluation guide and as a baseline for future work on this challenging topic.
摘要
neural radiance fields (NeRFs) 已经得到广泛的认可作为用于表示基于图像集的物体和场景三维重建的高效技术。尽管它们有效，但 NeRF 模型在某些情况下可能会遇到挑战，如车辆检查，因为缺乏足够的数据或存在挑战性的元素（例如反射）会强烈影响重建的准确性。为此，我们引入了 CarPatch，一个新的人工测试集。除了每个视图的图像、摄像机参数、深度地图和semantic segmentation映射之外，还包括了每个视图的全球和部分评价指标。这些指标用于评估、比较和更好地描述一些现状技术的性能。该数据集publicly released at ，可以用作评估指南和未来工作的基准。
Dense Transformer based Enhanced Coding Network for Unsupervised Metal Artifact Reduction
results: 实验和模型讨论表明，DTEC-Net 能够优于前一代方法，对 CT 影像进行自动化阴极 artifact 除除法，同时保留 CT 影像的结构资讯。Abstract
CT images corrupted by metal artifacts have serious negative effects on clinical diagnosis. Considering the difficulty of collecting paired data with ground truth in clinical settings, unsupervised methods for metal artifact reduction are of high interest. However, it is difficult for previous unsupervised methods to retain structural information from CT images while handling the non-local characteristics of metal artifacts. To address these challenges, we proposed a novel Dense Transformer based Enhanced Coding Network (DTEC-Net) for unsupervised metal artifact reduction. Specifically, we introduce a Hierarchical Disentangling Encoder, supported by the high-order dense process, and transformer to obtain densely encoded sequences with long-range correspondence. Then, we present a second-order disentanglement method to improve the dense sequence's decoding process. Extensive experiments and model discussions illustrate DTEC-Net's effectiveness, which outperforms the previous state-of-the-art methods on a benchmark dataset, and greatly reduces metal artifacts while restoring richer texture details.
摘要
Damage Vision Mining Opportunity for Imbalanced Anomaly Detection
results: 研究发现，在受损测量数据中，使用深度学习方法可以提高预测维护和应急维护的精度。此外，研究还发现了一个有用的正常检测应用，可以在受损测量数据中检测到异常情况。Abstract
In past decade, previous balanced datasets have been used to advance algorithms for classification, object detection, semantic segmentation, and anomaly detection in industrial applications. Specifically, for condition-based maintenance, automating visual inspection is crucial to ensure high quality. Deterioration prognostic attempts to optimize the fine decision process for predictive maintenance and proactive repair. In civil infrastructure and living environment, damage data mining cannot avoid the imbalanced data issue because of rare unseen events and high quality status by improved operations. For visual inspection, deteriorated class acquired from the surface of concrete and steel components are occasionally imbalanced. From numerous related surveys, we summarize that imbalanced data problems can be categorized into four types; 1) missing range of target and label valuables, 2) majority-minority class imbalance, 3) foreground-background of spatial imbalance, 4) long-tailed class of pixel-wise imbalance. Since 2015, there has been many imbalanced studies using deep learning approaches that includes regression, image classification, object detection, semantic segmentation. However, anomaly detection for imbalanced data is not yet well known. In the study, we highlight one-class anomaly detection application whether anomalous class or not, and demonstrate clear examples on imbalanced vision datasets: blood smear, lung infection, hazardous driving, wooden, concrete deterioration, river sludge, and disaster damage. Illustrated in Fig.1, we provide key results on damage vision mining advantage, hypothesizing that the more effective range of positive ratio, the higher accuracy gain of anomaly detection application. In our imbalanced studies, compared with the balanced case of positive ratio 1/1, we find that there is applicable positive ratio, where the accuracy are consistently high.
摘要
过去一个 décennial，以前的平衡数据集被用于提高分类、物体检测、semantic segmentation和异常检测在工业应用中的算法。具体来说，为condition-based maintenance，自动化视觉检查是关键以确保高质量。预测维护和抢修的决策进程优化。在公共基础设施和生活环境中，损害数据挖掘无法避免偏移数据问题，因为罕见的未看到事件和高质量状态的提高。对于视觉检查，坏化类型从混凝土和钢结构的表面获得的数据 occasionally imbalanced。从多个相关的调查中，我们总结出四种偏移数据问题：1）目标和标签值的缺失范围，2）多数少数类别偏移，3）前景背景的空间偏移，4）像素级偏移。自2015年以来，有很多关于偏移数据的研究使用深度学习方法，包括回归、图像分类、物体检测和semantic segmentation。然而，异常检测对偏移数据还未得到充分的研究。在本研究中，我们强调一类异常检测应用，无论异常类或不，并提供了清晰的示例，包括血癌、肺感染、危险驾驶、木材、混凝土衰老、河流淤泥和灾害损害。图1中，我们提供了危害视觉矿物优势，假设更高的正确率，异常检测应用的更高精度。在我们的偏移研究中，与平衡情况相比，我们发现了可采用的正确比例，其准确率一致高。
Industrial Segment Anything – a Case Study in Aircraft Manufacturing, Intralogistics, Maintenance, Repair, and Overhaul
paper_authors: Keno Moenck, Arne Wendt, Philipp Prünte, Julian Koch, Arne Sahrhage, Johann Gierecker, Ole Schmedemann, Falko Kähler, Dirk Holst, Martin Gomse, Thorsten Schüppstuhl, Daniel Schoepflin
results: 论文对飞机生产专业领域中的制造、内部物流、维护、维修和更换过程进行了应用，并考虑了领域知识的投入。Abstract
Deploying deep learning-based applications in specialized domains like the aircraft production industry typically suffers from the training data availability problem. Only a few datasets represent non-everyday objects, situations, and tasks. Recent advantages in research around Vision Foundation Models (VFM) opened a new area of tasks and models with high generalization capabilities in non-semantic and semantic predictions. As recently demonstrated by the Segment Anything Project, exploiting VFM's zero-shot capabilities is a promising direction in tackling the boundaries spanned by data, context, and sensor variety. Although, investigating its application within specific domains is subject to ongoing research. This paper contributes here by surveying applications of the SAM in aircraft production-specific use cases. We include manufacturing, intralogistics, as well as maintenance, repair, and overhaul processes, also representing a variety of other neighboring industrial domains. Besides presenting the various use cases, we further discuss the injection of domain knowledge.
摘要
通常在特殊领域 like 飞机生产 industri에서推广深度学习基于应用typically suffers from the training data availability problem。只有一些数据集表示不同的对象、情况和任务。 current Advantages in Research on Vision Foundation Models (VFM) opened a new area of tasks and models with high generalization capabilities in non-semantic and semantic predictions。 As recently demonstrated by the Segment Anything Project， exploiting VFM's zero-shot capabilities is a promising direction in tackling the boundaries spanned by data, context, and sensor variety。Although，investigating its application within specific domains is subject to ongoing research。This paper contributes here by surveying applications of the SAM in aircraft production-specific use cases。We include manufacturing, intralogistics, as well as maintenance, repair, and overhaul processes，also representing a variety of other neighboring industrial domains。Besides presenting the various use cases，we further discuss the injection of domain knowledge。
Global k-Space Interpolation for Dynamic MRI Reconstruction using Masked Image Modeling
paper_authors: Jiazhen Pan, Suprosanna Shit, Özgün Turgut, Wenqi Huang, Hongwei Bran Li, Nil Stolt-Ansó, Thomas Küstner, Kerstin Hammernik, Daniel Rueckert
for: This paper focuses on improving dynamic magnetic resonance imaging (MRI) reconstruction by interpolating undersampled k-space data before obtaining images with Fourier transform.
methods: The proposed approach uses a Transformer-based k-space Global Interpolation Network (k-GIN) to learn global dependencies among low- and high-frequency components of 2D+t k-space, and a novel k-space Iterative Refinement Module (k-IRM) to enhance high-frequency components learning.
results: The proposed method outperforms baseline methods in terms of both quantitative and qualitative measures, and achieves higher robustness and generalizability in highly-undersampled MR data.Here’s the Chinese translation of the three points:
results: 提议的方法相比基eline方法，在量化和质量上都有显著提高，并在高度受损的 MR 数据中具有更高的Robustness和普适性。Abstract
In dynamic Magnetic Resonance Imaging (MRI), k-space is typically undersampled due to limited scan time, resulting in aliasing artifacts in the image domain. Hence, dynamic MR reconstruction requires not only modeling spatial frequency components in the x and y directions of k-space but also considering temporal redundancy. Most previous works rely on image-domain regularizers (priors) to conduct MR reconstruction. In contrast, we focus on interpolating the undersampled k-space before obtaining images with Fourier transform. In this work, we connect masked image modeling with k-space interpolation and propose a novel Transformer-based k-space Global Interpolation Network, termed k-GIN. Our k-GIN learns global dependencies among low- and high-frequency components of 2D+t k-space and uses it to interpolate unsampled data. Further, we propose a novel k-space Iterative Refinement Module (k-IRM) to enhance the high-frequency components learning. We evaluate our approach on 92 in-house 2D+t cardiac MR subjects and compare it to MR reconstruction methods with image-domain regularizers. Experiments show that our proposed k-space interpolation method quantitatively and qualitatively outperforms baseline methods. Importantly, the proposed approach achieves substantially higher robustness and generalizability in cases of highly-undersampled MR data.
摘要
在动态磁共振成像（MRI）中，通常因为扫描时间有限，因此会出现嵌套artefacts在图像领域。因此，动态MR重建需要不仅考虑图像频谱中的空间频率组件，还需要考虑时间重复性。大多数前一些工作都是通过图像频谱约束（约束）来进行MR重建。在这种情况下，我们将掩码图像模型与嵌套空间 interpolate的方法相连接，并提出了一种新的Transformer基于的嵌套空间全球 interpolate网络，称为k-GIN。我们的k-GIN可以学习2D+t嵌套空间中低频和高频组件之间的全球依赖关系，并使用它来 interpolate不扫描的数据。此外，我们还提出了一种新的嵌套空间迭代优化模块（k-IRM），以提高高频组件的学习。我们对92个自有2D+t心脏MR数据进行了评估，并与图像频谱约束的MR重建方法进行比较。实验结果表明，我们的提posed方法在量化和质量上都超过了基eline方法。其中，我们的方法在高度压缩MR数据的情况下具有显著更高的Robustness和普适性。
A Theoretically Guaranteed Quaternion Weighted Schatten p-norm Minimization Method for Color Image Restoration
methods: 这篇论文提出了一种基于四元数的WNNM方法（QWNNM），该方法可以将颜色图像 Represented as a whole in the quaternion domain，并且保持了颜色通道之间的自然协同关系。此外，该论文还提出了一种基于四元数的WSNM模型（QWSNM），用于解决CIR问题。
results: 该论文通过对两种CIR任务， namely color image denoising和deblurring，进行了广泛的实验，并证明了QWSNM方法在量化和质量上都有优于许多现有的方法。此外，论文还提供了一种初步的理论收敛分析，表明QWNNM和QWSNM的解决方案都具有固定点收敛保证。Abstract
Inspired by the fact that the matrix formulated by nonlocal similar patches in a natural image is of low rank, the rank approximation issue have been extensively investigated over the past decades, among which weighted nuclear norm minimization (WNNM) and weighted Schatten $p$-norm minimization (WSNM) are two prevailing methods have shown great superiority in various image restoration (IR) problems. Due to the physical characteristic of color images, color image restoration (CIR) is often a much more difficult task than its grayscale image counterpart. However, when applied to CIR, the traditional WNNM/WSNM method only processes three color channels individually and fails to consider their cross-channel correlations. Very recently, a quaternion-based WNNM approach (QWNNM) has been developed to mitigate this issue, which is capable of representing the color image as a whole in the quaternion domain and preserving the inherent correlation among the three color channels. Despite its empirical success, unfortunately, the convergence behavior of QWNNM has not been strictly studied yet. In this paper, on the one side, we extend the WSNM into quaternion domain and correspondingly propose a novel quaternion-based WSNM model (QWSNM) for tackling the CIR problems. Extensive experiments on two representative CIR tasks, including color image denoising and deblurring, demonstrate that the proposed QWSNM method performs favorably against many state-of-the-art alternatives, in both quantitative and qualitative evaluations. On the other side, more importantly, we preliminarily provide a theoretical convergence analysis, that is, by modifying the quaternion alternating direction method of multipliers (QADMM) through a simple continuation strategy, we theoretically prove that both the solution sequences generated by the QWNNM and QWSNM have fixed-point convergence guarantees.
摘要
基于自然图像中非local相似区域矩阵的低级数据结构，过去几十年内，对矩阵近似问题进行了广泛的研究，其中包括权重核函数最小化（WNNM）和权重斜率p- norm最小化（WSNM）等两种方法，在不同的图像修复（IR）问题中显示出了优异的表现。然而，由于图像的物理特性，对于颜色图像的修复（CIR）是对灰度图像修复的一个非常更加困难的任务。然而，传统的WNNM/WSNM方法只是对每个色道进行独立处理，而忽略了它们之间的相互关系。最近，一种基于四元数的WNNM方法（QWNNM）已经开发出来，可以将颜色图像作为一个整体在四元数域中表示，并保留它们之间的自然相互关系。虽然它在实际中表现了良好，但它们的减法性还没有得到严格的研究。在这篇论文中，我们首先将WSNM扩展到四元数域，并对此提出了一种新的四元数基于WNNM模型（QWSNM），用于解决CIR问题。我们在两个代表性的CIR任务上进行了广泛的实验，包括颜色图像噪声去除和颜色图像补做。结果表明，我们提出的QWSNM方法在量化和质量上的评价中表现出色，胜过许多当前的状态艺术。此外，我们还提供了一种初步的理论减法分析，即通过修改四元数alternating direction method of multipliers（QADMM）的简单继续策略，我们 theoretically proves that both the solution sequences generated by QWNNM and QWSNM have fixed-point convergence guarantees.
PG-RCNN: Semantic Surface Point Generation for 3D Object Detection
paper_authors: Inyong Koo, Inyoung Lee, Se-Ho Kim, Hee-Seon Kim, Woo-jin Jeon, Changick Kim
for: 该 paper 是为了解决 LiDAR 数据中 объек的三维检测困难而写的。
methods: 该 paper 使用了点云补充方法，包括使用预训练网络生成 RoI 中的点云。
results: 该 paper 提出了 Point Generation R-CNN（PG-RCNN），一种新的端到端检测器，可以生成准确的前景对象的 semantic surface points。Abstract
One of the main challenges in LiDAR-based 3D object detection is that the sensors often fail to capture the complete spatial information about the objects due to long distance and occlusion. Two-stage detectors with point cloud completion approaches tackle this problem by adding more points to the regions of interest (RoIs) with a pre-trained network. However, these methods generate dense point clouds of objects for all region proposals, assuming that objects always exist in the RoIs. This leads to the indiscriminate point generation for incorrect proposals as well. Motivated by this, we propose Point Generation R-CNN (PG-RCNN), a novel end-to-end detector that generates semantic surface points of foreground objects for accurate detection. Our method uses a jointly trained RoI point generation module to process the contextual information of RoIs and estimate the complete shape and displacement of foreground objects. For every generated point, PG-RCNN assigns a semantic feature that indicates the estimated foreground probability. Extensive experiments show that the point clouds generated by our method provide geometrically and semantically rich information for refining false positive and misaligned proposals. PG-RCNN achieves competitive performance on the KITTI benchmark, with significantly fewer parameters than state-of-the-art models. The code is available at https://github.com/quotation2520/PG-RCNN.
摘要
Motivated by this, we propose Point Generation R-CNN (PG-RCNN), a novel end-to-end detector that generates semantic surface points of foreground objects for accurate detection. Our method uses a jointly trained RoI point generation module to process the contextual information of RoIs and estimate the complete shape and displacement of foreground objects. For every generated point, PG-RCNN assigns a semantic feature that indicates the estimated foreground probability.Extensive experiments show that the point clouds generated by our method provide geometrically and semantically rich information for refining false positive and misaligned proposals. PG-RCNN achieves competitive performance on the KITTI benchmark, with significantly fewer parameters than state-of-the-art models. The code is available at https://github.com/quotation2520/PG-RCNN.Translated into Simplified Chinese:一个主要挑战在LiDAR基于3D物体检测中是感知器通常无法捕捉物体的完整空间信息，这主要是因为距离较远和遮挡。两Stage检测器通过添加更多的点云来补充RoI中的点云，但这些方法生成的点云都是对所有的区域提案中的物体，假设物体总是存在于RoI中。这会导致无用的点云生成和预测错误的提案。我们提出了Point Generation R-CNN（PG-RCNN），一种新的端到端检测器，它可以生成对象的语义表面点，以提高检测的准确性。我们的方法使用一个同时训练的RoI点生成模块，以处理RoI中的上下文信息，并估计前景对象的完整形状和偏移。每个生成的点都被PG-RCNN分配一个语义特征，这个特征指示了对象的预测概率。广泛的实验表明，PG-RCNN生成的点云具有很好的准确性和语义特征，可以用于修正错误的提案和偏移。PG-RCNN在KITTI测试benchmark上达到了竞争性性能，并且 Parameters 比 state-of-the-art 模型少得多。代码可以在https://github.com/quotation2520/PG-RCNN中下载。
Automatic lobe segmentation using attentive cross entropy and end-to-end fissure generation
paper_authors: Qi Su, Na Wang, Jiawen Xie, Yinan Chen, Xiaofan Zhang
for: automatic lung lobe segmentation algorithm for the diagnosis and treatment of lung diseases
methods: task-specific loss function to pay attention to the area around the pulmonary fissure, end-to-end pulmonary fissure generation method, registration-based loss function to alleviate convergence difficulty
results: dice scores of 97.83% on private dataset STLB and 94.75% on public LUNA16 datasetAbstract
The automatic lung lobe segmentation algorithm is of great significance for the diagnosis and treatment of lung diseases, however, which has great challenges due to the incompleteness of pulmonary fissures in lung CT images and the large variability of pathological features. Therefore, we propose a new automatic lung lobe segmentation framework, in which we urge the model to pay attention to the area around the pulmonary fissure during the training process, which is realized by a task-specific loss function. In addition, we introduce an end-to-end pulmonary fissure generation method in the auxiliary pulmonary fissure segmentation task, without any additional network branch. Finally, we propose a registration-based loss function to alleviate the convergence difficulty of the Dice loss supervised pulmonary fissure segmentation task. We achieve 97.83% and 94.75% dice scores on our private dataset STLB and public LUNA16 dataset respectively.
摘要
“自动肺lobus分割算法具有诊断和治疗肺病的重要意义，但受到肺 CT 影像中肺裂的不完整性和病理特征的大幅度variability所困扰。因此，我们提出了一个新的自动肺lobus分割框架，其中我们要求模型在训练过程中对肺裂附近区域做出优化。此外，我们引入了一个独立的辅助肺裂分割任务，并在这个任务中使用了一个统一的损失函数。最后，我们提出了一个注册损失函数，以解决基于 Dice 损失函数的肺裂分割任务中的整合问题。我们在私人数据集 STLB 和公共数据集 LUNA16 上实现了97.83% 和 94.75%的 Dice 分数。”
Semi-Supervised Medical Image Segmentation with Co-Distribution Alignment
results: 根据论文的实验结果，Co-DA方法在三个公共dataset上都有较好的性能，尤其是在2D CaDIS dataset和3D LGE-MRI和ACDC dataset上，它可以仅使用24%的标注数据而 достиieving an mIoU of 0.8515和Dice score of 0.8824和0.8773，即使只使用20%的数据。Abstract
Medical image segmentation has made significant progress when a large amount of labeled data are available. However, annotating medical image segmentation datasets is expensive due to the requirement of professional skills. Additionally, classes are often unevenly distributed in medical images, which severely affects the classification performance on minority classes. To address these problems, this paper proposes Co-Distribution Alignment (Co-DA) for semi-supervised medical image segmentation. Specifically, Co-DA aligns marginal predictions on unlabeled data to marginal predictions on labeled data in a class-wise manner with two differently initialized models before using the pseudo-labels generated by one model to supervise the other. Besides, we design an over-expectation cross-entropy loss for filtering the unlabeled pixels to reduce noise in their pseudo-labels. Quantitative and qualitative experiments on three public datasets demonstrate that the proposed approach outperforms existing state-of-the-art semi-supervised medical image segmentation methods on both the 2D CaDIS dataset and the 3D LGE-MRI and ACDC datasets, achieving an mIoU of 0.8515 with only 24% labeled data on CaDIS, and a Dice score of 0.8824 and 0.8773 with only 20% data on LGE-MRI and ACDC, respectively.
摘要
医疗图像分割技术在有大量标注数据时已经做出了 significiant进步。然而，为了创建医疗图像分割数据集， annotating 医疗图像分割数据集是昂贵的，主要因为需要专业技能。此外，医疗图像中的类别经常不均匀分布，这会严重地影响少数类别的分类性能。为了解决这些问题，这篇论文提出了Co-Distribution Alignment（Co-DA）方法，用于 semi-supervised 医疗图像分割。具体来说，Co-DA 方法将未标注数据中的边缘预测与已标注数据中的边缘预测进行类别匹配，使用两个不同初始化的模型来实现。此外，我们还设计了过期cross-entropy损失函数，用于筛选未标注的像素，以降低它们的 Pseudo-labels 中的噪音。我们对三个公共数据集进行了量化和质量测试，结果显示，我们的方法在 CaDIS 数据集上的 mIoU 为 0.8515，只使用 24% 的标注数据；在 LGE-MRI 和 ACDC 数据集上，我们的方法的 Dice 分数分别为 0.8824 和 0.8773，只使用 20% 的数据。
Phase Matching for Out-of-Distribution Generalization
results: 经过实验证明，提出的方法可以在多个标准准 benchmark上达到领先的性能水平，并且在不同分布下的泛化和Out-of-distribution（OOD） robustness任务中表现出色。Abstract
The Fourier transform, serving as an explicit decomposition method for visual signals, has been employed to explain the out-of-distribution generalization behaviors of Convolutional Neural Networks (CNNs). Previous studies have indicated that the amplitude spectrum is susceptible to the disturbance caused by distribution shifts. On the other hand, the phase spectrum preserves highly-structured spatial information, which is crucial for robust visual representation learning. However, the spatial relationships of phase spectrum remain unexplored in previous researches. In this paper, we aim to clarify the relationships between Domain Generalization (DG) and the frequency components, and explore the spatial relationships of the phase spectrum. Specifically, we first introduce a Fourier-based structural causal model which interprets the phase spectrum as semi-causal factors and the amplitude spectrum as non-causal factors. Then, we propose Phase Matching (PhaMa) to address DG problems. Our method introduces perturbations on the amplitude spectrum and establishes spatial relationships to match the phase components. Through experiments on multiple benchmarks, we demonstrate that our proposed method achieves state-of-the-art performance in domain generalization and out-of-distribution robustness tasks.
摘要
《傅里叶变换在视觉信号中的明确分解方法》，已经用于解释深度学习模型在不同分布下的泛化行为。先前的研究表明，振荡спектrum容易受到分布变化的影响。然而，相对于振荡спектrum，频率спектrum preserve了高度结构化的空间信息，这是重要的视觉表示学习的基础。然而，频率спектrum中的空间关系尚未在先前的研究中得到了探讨。在这篇论文中，我们想要解释频率спектrum和频率组件之间的关系，并探讨频率спектrum中的空间关系。我们首先介绍了一种基于傅里叶变换的结构 causal模型，其中 interprets频率спектrum为半 causal因素，而振荡спектrum为非 causal因素。然后，我们提出了phasematching（PhaMa）方法，用于解决频率特征泛化问题。我们的方法在振荡spectrum中引入了干扰并建立了空间关系，以匹配频率спектrum的组件。通过多个标准benchmark experiment表明，我们提出的方法可以在频率特征泛化和out-of-distribution Robustness任务中达到状态之巅的性能。
Sparse annotation strategies for segmentation of short axis cardiac MRI
results: 研究发现，训练 sparse volumes 和 sparse annotations 可以获得高达 0.85 的 Dice 分数，并且比使用完整数据集（160 和 240 个数据集）更好。此外，中部的剖面标注对分割性能最有利，而脊梁区域的标注对分割性能最差。Abstract
Short axis cardiac MRI segmentation is a well-researched topic, with excellent results achieved by state-of-the-art models in a supervised setting. However, annotating MRI volumes is time-consuming and expensive. Many different approaches (e.g. transfer learning, data augmentation, few-shot learning, etc.) have emerged in an effort to use fewer annotated data and still achieve similar performance as a fully supervised model. Nevertheless, to the best of our knowledge, none of these works focus on which slices of MRI volumes are most important to annotate for yielding the best segmentation results. In this paper, we investigate the effects of training with sparse volumes, i.e. reducing the number of cases annotated, and sparse annotations, i.e. reducing the number of slices annotated per case. We evaluate the segmentation performance using the state-of-the-art nnU-Net model on two public datasets to identify which slices are the most important to annotate. We have shown that training on a significantly reduced dataset (48 annotated volumes) can give a Dice score greater than 0.85 and results comparable to using the full dataset (160 and 240 volumes for each dataset respectively). In general, training on more slice annotations provides more valuable information compared to training on more volumes. Further, annotating slices from the middle of volumes yields the most beneficial results in terms of segmentation performance, and the apical region the worst. When evaluating the trade-off between annotating volumes against slices, annotating as many slices as possible instead of annotating more volumes is a better strategy.
摘要
短轴心臓MRI分割是已有广泛研究的话题，现有前沿模型在监督环境下实现了出色的结果。然而，对MRI卷积的标注是时间consuming和昂贵的。多种方法（如转移学习、数据扩展、少数案例学习等）在尝试使用 fewer annotated data 并 still achieve similar performance as a fully supervised model 中出现。然而，到目前为止，这些工作没有关注在哪些MRI卷积中最重要的标注，以获得最佳分割结果。在这篇论文中，我们investigate the effects of training with sparse volumes和 sparse annotations。我们使用state-of-the-art nnU-Net模型在两个公共数据集上评估分割性能，以确定哪些卷积是最重要的标注。我们发现，使用很少的数据（48个标注卷积）可以达到 Dice 分数大于 0.85 和与全数据集（160和240卷积）的结果相当。总的来说，训练更多的卷积标注提供更多有价值的信息，而不是训练更多的卷积。此外，从中心部分标注MRI卷积可以提供最佳分割结果，而Apical区域则是最差。当评估标注卷积与卷积之间的权衡时，可以看到， annotating as many slices as possible instead of annotating more volumes 是一个更好的策略。
Attribute Regularized Soft Introspective VAE: Towards Cardiac Attribute Regularization Through MRI Domains
results: 实验表明，Attri-SIVAE模型在不同的MRI数据集上表现相当于Attributed regularized VAE，并且可以在不同的数据集上保持同等的规范水平。Abstract
Deep generative models have emerged as influential instruments for data generation and manipulation. Enhancing the controllability of these models by selectively modifying data attributes has been a recent focus. Variational Autoencoders (VAEs) have shown promise in capturing hidden attributes but often produce blurry reconstructions. Controlling these attributes through different imaging domains is difficult in medical imaging. Recently, Soft Introspective VAE leverage the benefits of both VAEs and Generative Adversarial Networks (GANs), which have demonstrated impressive image synthesis capabilities, by incorporating an adversarial loss into VAE training. In this work, we propose the Attributed Soft Introspective VAE (Attri-SIVAE) by incorporating an attribute regularized loss, into the Soft-Intro VAE framework. We evaluate experimentally the proposed method on cardiac MRI data from different domains, such as various scanner vendors and acquisition centers. The proposed method achieves similar performance in terms of reconstruction and regularization compared to the state-of-the-art Attributed regularized VAE but additionally also succeeds in keeping the same regularization level when tested on a different dataset, unlike the compared method.
摘要
深度生成模型已经成为数据生成和修改的重要工具。提高这些模型的可控性，通过选择性地修改数据特性，是最近的焦点。变量自编码器（VAEs）已经表现出捕捉隐藏特性的抑或，但它们经常生成模糊的重建。在医学成像中，控制这些特性通过不同的成像频谱是困难的。最近，软 introspective VAE 利用 VAEs 和生成对抗网络（GANs）的优点，通过在 VAE 训练中添加对抗损失来提高图像合成能力。在这项工作中，我们提议 incorporating attribute regularized loss 到 Soft-Intro VAE 框架中，并对其进行实验评估。我们发现，提议的方法在不同的 MRI 数据集上实现了相似的重建和规范性能，与 state-of-the-art attributed regularized VAE 相比，同时还能在不同的数据集上保持同等的规范水平。
ExWarp: Extrapolation and Warping-based Temporal Supersampling for High-frequency Displays
paper_authors: Akanksha Dixit, Yashashwee Chakrabarty, Smruti R. Sarangi
For: The paper aims to increase the frame rate of high-frequency displays by 4x with minimal reduction in perceived image quality.* Methods: The paper proposes using reinforcement learning (RL) to intelligently choose between slower DNN-based extrapolation and faster warping-based methods.* Results: The proposed approach, called Exwarp, achieves a 4x increase in frame rate with minimal reduction in image quality.Abstract
High-frequency displays are gaining immense popularity because of their increasing use in video games and virtual reality applications. However, the issue is that the underlying GPUs cannot continuously generate frames at this high rate -- this results in a less smooth and responsive experience. Furthermore, if the frame rate is not synchronized with the refresh rate, the user may experience screen tearing and stuttering. Previous works propose increasing the frame rate to provide a smooth experience on modern displays by predicting new frames based on past or future frames. Interpolation and extrapolation are two widely used algorithms that predict new frames. Interpolation requires waiting for the future frame to make a prediction, which adds additional latency. On the other hand, extrapolation provides a better quality of experience because it relies solely on past frames -- it does not incur any additional latency. The simplest method to extrapolate a frame is to warp the previous frame using motion vectors; however, the warped frame may contain improperly rendered visual artifacts due to dynamic objects -- this makes it very challenging to design such a scheme. Past work has used DNNs to get good accuracy, however, these approaches are slow. This paper proposes Exwarp -- an approach based on reinforcement learning (RL) to intelligently choose between the slower DNN-based extrapolation and faster warping-based methods to increase the frame rate by 4x with an almost negligible reduction in the perceived image quality.
摘要
高频显示器在游戏和虚拟现实应用中的使用越来越普遍，但是下面的 GPU 无法不断生成这高速帧，这会导致用户体验不平滑和不响应。此外，如果帧率与刷新率不同步，用户可能会经历屏扑和停顿。先前的工作建议通过预测新帧来提高现代显示器的帧率，以提供平滑的用户体验。 interpolate 和 extrapolate 是两种广泛使用的预测算法。 interpolate 需要等待未来的帧来作预测，这会添加额外的延迟。 extrapolate 提供了更好的用户体验，因为它仅基于过去的帧，无需添加额外的延迟。在 extrapolate 帧时，最简单的方法是通过运动向量来扭曲上一帧，但是扭曲后的帧可能会包含不正确渲染的视觉artefacts，这使得设计这种方案非常困难。过去的工作使用 DNN 来获得好的准确性，但这些方法比较慢。这篇论文提出了 Exwarp，一种基于 reinforcement learning（RL）的方法，可以智能地选择 slower DNN-based extrapolation 和 faster warping-based方法，以提高帧率4倍，并且几乎无法感受到图像质量的下降。
SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentation
paper_authors: Yiqing Wang, Zihan Li, Jieru Mei, Zihao Wei, Li Liu, Chen Wang, Shengtian Sang, Alan Yuille, Cihang Xie, Yuyin Zhou for:这篇论文主要目的是提高自主学习方法 для医疗影像分割。methods:论文使用的方法包括masked multi-view encoder和cross-view decoder，以及一种新的多视图学习方法。results:论文比前一个状态的自主学习方法Swin UNITR显示了明显的优势，能够更好地 интегрирова多视图信息，提高模型的准确率和数据效率。Abstract
Recent advancements in large-scale Vision Transformers have made significant strides in improving pre-trained models for medical image segmentation. However, these methods face a notable challenge in acquiring a substantial amount of pre-training data, particularly within the medical field. To address this limitation, we present Masked Multi-view with Swin Transformers (SwinMM), a novel multi-view pipeline for enabling accurate and data-efficient self-supervised medical image analysis. Our strategy harnesses the potential of multi-view information by incorporating two principal components. In the pre-training phase, we deploy a masked multi-view encoder devised to concurrently train masked multi-view observations through a range of diverse proxy tasks. These tasks span image reconstruction, rotation, contrastive learning, and a novel task that employs a mutual learning paradigm. This new task capitalizes on the consistency between predictions from various perspectives, enabling the extraction of hidden multi-view information from 3D medical data. In the fine-tuning stage, a cross-view decoder is developed to aggregate the multi-view information through a cross-attention block. Compared with the previous state-of-the-art self-supervised learning method Swin UNETR, SwinMM demonstrates a notable advantage on several medical image segmentation tasks. It allows for a smooth integration of multi-view information, significantly boosting both the accuracy and data-efficiency of the model. Code and models are available at https://github.com/UCSC-VLAA/SwinMM/.
摘要
PRIOR: Prototype Representation Joint Learning from Medical Images and Reports
results: 实验结果表明，提出的方法在五个下游任务中（包括监督分类、零扩展分类、图像到文本检索、semantic segmentation和物体检测）均表现出色，并且在不同的数据集大小设置下也具有优异性。Abstract
Contrastive learning based vision-language joint pre-training has emerged as a successful representation learning strategy. In this paper, we present a prototype representation learning framework incorporating both global and local alignment between medical images and reports. In contrast to standard global multi-modality alignment methods, we employ a local alignment module for fine-grained representation. Furthermore, a cross-modality conditional reconstruction module is designed to interchange information across modalities in the training phase by reconstructing masked images and reports. For reconstructing long reports, a sentence-wise prototype memory bank is constructed, enabling the network to focus on low-level localized visual and high-level clinical linguistic features. Additionally, a non-auto-regressive generation paradigm is proposed for reconstructing non-sequential reports. Experimental results on five downstream tasks, including supervised classification, zero-shot classification, image-to-text retrieval, semantic segmentation, and object detection, show the proposed method outperforms other state-of-the-art methods across multiple datasets and under different dataset size settings. The code is available at https://github.com/QtacierP/PRIOR.
摘要
医疗图像和报告的共同预训练基于对比学习已经成为一种成功的表示学习策略。在这篇论文中，我们提出了一种原型学习框架，其中包括医疗图像和报告之间的全局和局部对齐。与标准的全局多Modalities对齐方法不同，我们使用了局部对齐模块，以获得细化的表示。此外，我们还设计了跨Modalities的Conditional重建模块，用于在训练阶段交换modalities之间的信息，通过重建遮盖的图像和报告来进行交换。为恢复长报告，我们构建了句子级prototype记忆银行，使得网络能够关注低级的本地视觉和高级的医学语言特征。此外，我们还提出了一种非自动生成 paradigma，用于恢复非顺序报告。实验结果表明，我们的方法在五个下游任务中，包括supervised分类、零shot分类、图像到文本检索、semantic segmentation和物体检测中，都超过了其他当前state-of-the-art方法。代码可以在https://github.com/QtacierP/PRIOR上获取。
A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation
methods: 我们提出了一个在线知识distillation (KD) 框架，可以同时学习高效 yet 紧凑的 CNN 和 ViT 模型，并通过两个关键技术突破 CNN 和 ViT 的局限性。首先，我们提出了不同类征distillation (HFD)，以提高学生在低层特征空间的一致性，模仿 CNN 和 ViT 之间的不同特征。其次，我们提出了双向选择distillation (BSD)，可以动态地将选择的知识传递给对方。这包括1) Region-wise BSD 确定知识传递的方向在特征空间中，2) Pixel-wise BSD 确定在极值空间中传递哪些预测知识。
results: 我们的提出的框架在三个 benchmark 数据集上进行了广泛的实验，并与现有的在线distillation方法相比，表现出了很大的提升。此外，我们的方法还证明了在学习 ViT 和 CNN 模型之间协同学习的可能性。Abstract
In this paper, we strive to answer the question "how to collaboratively learn convolutional neural network (CNN)-based and vision transformer (ViT)-based models by selecting and exchanging the reliable knowledge between them for semantic segmentation?" Accordingly, we propose an online knowledge distillation (KD) framework that can simultaneously learn compact yet effective CNN-based and ViT-based models with two key technical breakthroughs to take full advantage of CNNs and ViT while compensating their limitations. Firstly, we propose heterogeneous feature distillation (HFD) to improve students' consistency in low-layer feature space by mimicking heterogeneous features between CNNs and ViT. Secondly, to facilitate the two students to learn reliable knowledge from each other, we propose bidirectional selective distillation (BSD) that can dynamically transfer selective knowledge. This is achieved by 1) region-wise BSD determining the directions of knowledge transferred between the corresponding regions in the feature space and 2) pixel-wise BSD discerning which of the prediction knowledge to be transferred in the logit space. Extensive experiments on three benchmark datasets demonstrate that our proposed framework outperforms the state-of-the-art online distillation methods by a large margin, and shows its efficacy in learning collaboratively between ViT-based and CNN-based models.
摘要
在本文中，我们努力回答“如何通过选择和交换可靠知识来协同学习卷积神经网络（CNN）和视觉 трансформа（ViT）模型以进行semantic segmentation？”为此，我们提出了在线知识储存（KD）框架，可同时学习高效又紧凑的CNN和ViT模型，并且具有两个关键技术突破，以全面利用CNN和ViT的优势，同时补做它们的局限性。首先，我们提出了不同类征储存（HFD），以提高学生在低层特征空间的一致性，模仿CNN和ViT之间的不同特征。其次，为了让两个学生之间学习可靠的知识，我们提出了双向选择储存（BSD），可动态传递选择的知识。这实现了1）在特征空间中确定 переда知识的方向，2）在逻辑空间中选择要传递的预测知识。我们对三个标准测试集进行了广泛的实验，结果显示，我们提出的框架在在线储存方法中升级了状态之差，并在学习协同CNN和ViT模型方面表现出了 efficacy。
MataDoc: Margin and Text Aware Document Dewarping for Arbitrary Boundary
paper_authors: Beiya Dai, Xing li, Qunyi Xie, Yulin Li, Xiameng Qin, Chengquan Zhang, Kun Yao, Junyu Han
for: DocUNet, DIR300, WarpDoc datasets
methods: margin regularization, background consistency, word position consistency
results: superior performance on documents with incomplete boundariesAbstract
Document dewarping from a distorted camera-captured image is of great value for OCR and document understanding. The document boundary plays an important role which is more evident than the inner region in document dewarping. Current learning-based methods mainly focus on complete boundary cases, leading to poor document correction performance of documents with incomplete boundaries. In contrast to these methods, this paper proposes MataDoc, the first method focusing on arbitrary boundary document dewarping with margin and text aware regularizations. Specifically, we design the margin regularization by explicitly considering background consistency to enhance boundary perception. Moreover, we introduce word position consistency to keep text lines straight in rectified document images. To produce a comprehensive evaluation of MataDoc, we propose a novel benchmark ArbDoc, mainly consisting of document images with arbitrary boundaries in four typical scenarios. Extensive experiments confirm the superiority of MataDoc with consideration for the incomplete boundary on ArbDoc and also demonstrate the effectiveness of the proposed method on DocUNet, DIR300, and WarpDoc datasets.
摘要
文档去扭曲从扭曲捕捉的图像中是很有价值的，尤其是文档边界的角色更加重要。现有的学习型方法主要关注完整边界的文档去扭曲，导致文档修正性能较差。与这些方法不同，本文提出了MataDoc，第一种专门关注任意边界文档去扭曲方法，并添加了边界追求和文本意识regularization。具体来说，我们设计了边界追求的margin regularization，通过考虑背景一致性来增强边界感知。此外，我们引入了文本位置一致的regularization，以保持文本线条在修正后的图像中 straight。为了对MataDoc进行全面的评估，我们提出了一个新的benchmark ArbDoc，主要包括四种典型的文档场景，其中文档边界具有任意形态。广泛的实验表明，MataDoc在ArbDoc上的性能卓越，同时也在DocUNet、DIR300和WarpDoc数据集上表现出色。
Interpolating between Images with Diffusion Models
results: 该论文通过在多种主体姿态、图像风格和图像内容中进行 interpolating，并通过CLIP选择最高质量图像，以证明该方法可以获得有力的 interpolating结果。Abstract
One little-explored frontier of image generation and editing is the task of interpolating between two input images, a feature missing from all currently deployed image generation pipelines. We argue that such a feature can expand the creative applications of such models, and propose a method for zero-shot interpolation using latent diffusion models. We apply interpolation in the latent space at a sequence of decreasing noise levels, then perform denoising conditioned on interpolated text embeddings derived from textual inversion and (optionally) subject poses. For greater consistency, or to specify additional criteria, we can generate several candidates and use CLIP to select the highest quality image. We obtain convincing interpolations across diverse subject poses, image styles, and image content, and show that standard quantitative metrics such as FID are insufficient to measure the quality of an interpolation. Code and data are available at https://clintonjwang.github.io/interpolation.
摘要
一个未经探索的前ier是将两个输入图像之间进行 interpolating，这是现有的图像生成管道中缺失的一个特性。我们认为这样的特性可以扩展图像生成模型的创作应用，并提出了采用潜在扩散模型进行零 shot interpolating的方法。我们在潜在空间中逐渐减少噪声水平进行 interpolating，然后使用文本倒转和（可选）主体姿态来进行杜然处理。为了更好地保持一致性，或者设置其他参数，我们可以生成多个候选图像，并使用 CLIP 选择最高质量的图像。我们在不同的主体姿态、图像风格和图像内容中获得了令人满意的 interpolating，并证明了标准的量化指标如 FID 不够Measure interpolating 的质量。代码和数据可以在 https://clintonjwang.github.io/interpolation 上获取。
results: 比前方法更加可靠和真实地生成中间帧结果，并且在实验中表明了考虑事件特征的重要性Abstract
Dynamic vision sensors or event cameras provide rich complementary information for video frame interpolation. Existing state-of-the-art methods follow the paradigm of combining both synthesis-based and warping networks. However, few of those methods fully respect the intrinsic characteristics of events streams. Given that event cameras only encode intensity changes and polarity rather than color intensities, estimating optical flow from events is arguably more difficult than from RGB information. We therefore propose to incorporate RGB information in an event-guided optical flow refinement strategy. Moreover, in light of the quasi-continuous nature of the time signals provided by event cameras, we propose a divide-and-conquer strategy in which event-based intermediate frame synthesis happens incrementally in multiple simplified stages rather than in a single, long stage. Extensive experiments on both synthetic and real-world datasets show that these modifications lead to more reliable and realistic intermediate frame results than previous video frame interpolation methods. Our findings underline that a careful consideration of event characteristics such as high temporal density and elevated noise benefits interpolation accuracy.
摘要
“动态视觉传感器或事件摄像机可提供丰富的补充信息，以帮助视频帧 interpolate。现有的state-of-the-art方法通常采用组合synthesis-based和折叠网络的方法。然而，这些方法很少充分尊重事件流的内在特征。因为事件摄像机只记录了INTENSITY变化和方向，而不是颜色强度，因此从事件中估算光流 arguably 更加困难 than from RGB信息。我们因此提议将RGB信息 integrate into event-guided optical flow refinement策略。此外，由于事件摄像机提供的时间信号具有 quasi-连续性，我们提议采用分段的 divide-and-conquer策略，在多个简化的阶段中进行事件基本中间帧synthesis，而不是在单一、长阶段中进行。广泛的实验表明，这些修改可以更加可靠和真实地 interpolate 视频帧结果，than previous video frame interpolation方法。我们的发现也 подчеркивает，对事件特征，如高时间密度和提高的噪声，的仔细考虑，可以提高插值精度。”
MFMAN-YOLO: A Method for Detecting Pole-like Obstacles in Complex Environment
results: 实验结果显示，方法的检测精度、回归率和均值精度分别为94.7%、93.1%和97.4%，检测帧率达400f/s。这种方法可以在实时和精度高的情况下探测复杂环境中的杆体物体。Abstract
In real-world traffic, there are various uncertainties and complexities in road and weather conditions. To solve the problem that the feature information of pole-like obstacles in complex environments is easily lost, resulting in low detection accuracy and low real-time performance, a multi-scale hybrid attention mechanism detection algorithm is proposed in this paper. First, the optimal transport function Monge-Kantorovich (MK) is incorporated not only to solve the problem of overlapping multiple prediction frames with optimal matching but also the MK function can be regularized to prevent model over-fitting; then, the features at different scales are up-sampled separately according to the optimized efficient multi-scale feature pyramid. Finally, the extraction of multi-scale feature space channel information is enhanced in complex environments based on the hybrid attention mechanism, which suppresses the irrelevant complex environment background information and focuses the feature information of pole-like obstacles. Meanwhile, this paper conducts real road test experiments in a variety of complex environments. The experimental results show that the detection precision, recall, and average precision of the method are 94.7%, 93.1%, and 97.4%, respectively, and the detection frame rate is 400 f/s. This research method can detect pole-like obstacles in a complex road environment in real time and accurately, which further promotes innovation and progress in the field of automatic driving.
摘要
实际交通中有各种不确定性和复杂性，以致 pole-like obstacles 的特征信息在复杂环境中易丢失，导致检测精度低下、实时性低下。为解决这个问题，本文提出了一种多尺度混合注意力机制检测算法。首先，通过 Monge-Kantorovich（MK）函数进行最佳运输函数，不仅可以解决多个预测帧的最佳匹配问题，还可以将 MK 函数进行正则化，以防止模型过拟合；然后，在不同尺度上分别更新独立的高效多尺度特征 pyramid。最后，在复杂环境中提高多尺度特征空间通道信息抽取的能力，通过混合注意力机制，抑制不相关的复杂环境背景信息，专注于检测 pole-like obstacles 的特征信息。同时，本文在实际公路测试中进行了多种复杂环境的实验，实验结果表明，该方法的检测精度、回归率和平均精度分别为 94.7%、93.1% 和 97.4%，检测帧率为 400 f/s。这种检测方法可以在复杂交通环境中实时和准确地检测 pole-like obstacles，为自动驾驶技术的进一步创新和发展做出了贡献。
Towards Generalizable Deepfake Detection by Primary Region Regularization
paper_authors: Harry Cheng, Yangyang Guo, Tianyi Wang, Liqiang Nie, Mohan Kankanhalli
for: 提高深伪检测方法的泛化能力，对于未见过的伪造和修改方法进行扩展
methods: 使用新的调整方法来提高深伪检测方法的泛化能力，包括删除主要区域图像的调整
results: 与多个基eline比较，提高了平均性能表现6%，并且与一些现有的基eline竞争Here’s the breakdown of each sentence:* “for”: This sentence indicates the purpose of the paper, which is to improve the generalization ability of deepfake detection methods.* “methods”: This sentence describes the approach used in the paper to achieve the purpose, which is to use a novel regularization perspective to enhance the generalization capability of deepfake detectors.* “results”: This sentence summarizes the performance of the proposed method compared to baseline methods, showing an average improvement of 6% and competitive performance with state-of-the-art baselines.Abstract
The existing deepfake detection methods have reached a bottleneck in generalizing to unseen forgeries and manipulation approaches. Based on the observation that the deepfake detectors exhibit a preference for overfitting the specific primary regions in input, this paper enhances the generalization capability from a novel regularization perspective. This can be simply achieved by augmenting the images through primary region removal, thereby preventing the detector from over-relying on data bias. Our method consists of two stages, namely the static localization for primary region maps, as well as the dynamic exploitation of primary region masks. The proposed method can be seamlessly integrated into different backbones without affecting their inference efficiency. We conduct extensive experiments over three widely used deepfake datasets - DFDC, DF-1.0, and Celeb-DF with five backbones. Our method demonstrates an average performance improvement of 6% across different backbones and performs competitively with several state-of-the-art baselines.
摘要
现有的深伪检测方法已达到泛化到未经见 forgery 和 manipulation 方法的瓶颈。基于观察到深伪检测器偏好特定的主要区域在输入中过拟合的观察，这篇文章提高了泛化能力从一个新的规范化视角。这可以简单地通过除去输入图像的主要区域来实现，以防止检测器过于依赖数据偏好。我们的方法包括两个阶段：首先，静态地LOCALIZATION FOR PRIMARY REGION MAPS，然后是动态利用主要区域面罩。我们的方法可以轻松地与不同的背bone结合使用，无需影响其推理效率。我们在DFDC、DF-1.0和Celeb-DF三个广泛使用的深伪数据集上进行了广泛的实验，我们的方法在不同的背bone上显示了平均提高6%的性能，并与一些状态机器人的基准值竞争。
On the Connection between Pre-training Data Diversity and Fine-tuning Robustness
paper_authors: Vivek Ramanujan, Thao Nguyen, Sewoong Oh, Ludwig Schmidt, Ali Farhadi for: 这个论文的目的是研究预训练策略对深度学习模型的泛化性的影响。methods: 作者使用了多种自然和 sintetic 数据源来生成不同的预训练分布，并通过评估这些预训练分布对下游模型的抗衰假设来研究预训练分布的影响。results: 研究发现，预训练分布中数据量的变化是主要影响下游模型的抗衰假设的因素，而其他因素对下游模型的抗衰假设具有有限的影响。例如，减少 ImageNet 预训练类别的数量，同时增加每个类别中的图像数量（即保持总数据量不变）并不会影响下游模型的抗衰假设。Abstract
Pre-training has been widely adopted in deep learning to improve model performance, especially when the training data for a target task is limited. In our work, we seek to understand the implications of this training strategy on the generalization properties of downstream models. More specifically, we ask the following question: how do properties of the pre-training distribution affect the robustness of a fine-tuned model? The properties we explore include the label space, label semantics, image diversity, data domains, and data quantity of the pre-training distribution. We find that the primary factor influencing downstream effective robustness (Taori et al., 2020) is data quantity, while other factors have limited significance. For example, reducing the number of ImageNet pre-training classes by 4x while increasing the number of images per class by 4x (that is, keeping total data quantity fixed) does not impact the robustness of fine-tuned models. We demonstrate our findings on pre-training distributions drawn from various natural and synthetic data sources, primarily using the iWildCam-WILDS distribution shift as a test for downstream robustness.
摘要
pré-entraînement a été largement adopté dans l'apprentissage profond pour améliorer les performances des modèles, especialment lorsque les données d'entraînement pour une tâche cible sont limitées. Dans notre travail, nous voulons comprendre les implications de cette stratégie d'entraînement sur les propriétés de généralisation des modèles downstream. Plus spécifiquement, nous posons la question suivante : comment les propriétés de la distribution de pré-entraînement affectent-elles la robustesse des modèles fine-tunés ? Les propriétés que nous explorons comprennent l'espace de labels, les semantiques de labels, la diversité d'images, les domaines de données et la quantité de données de la distribution de pré-entraînement. Nous trouvons que le facteur primordial influençant la robustesse effective downstream (Taori et al., 2020) est la quantité de données, tandis que les autres facteurs ont un impact limité. Par exemple, en réduisant le nombre de classes de pré-entraînement d'ImageNet de 4 fois while augmentant le nombre d'images par classe de 4 fois (c'est-à-dire en gardant la quantité totale de données fixée), n'a pas d'impact sur la robustesse des modèles fine-tunés. Nous démontrons nos résultats sur des distributions de pré-entraînement tirées de divers sources de données naturelles et synthétiques, en utilisant principalement la distribution shift iWildCam-WILDS comme un test pour la robustesse downstream.
Entropy Transformer Networks: A Learning Approach via Tangent Bundle Data Manifold
for: 该 paper ocuses on developing an accurate and efficient image transformation approach for Convolutional Neural Networks (CNNs) architectures.
methods: 该 paper 用 novel Entropy Spatial Transformer Networks (ESTN) interpolate on the data manifold distributions, 使用 random samples 和 tangent space 计算 transformer parameters. 同时， authors 还提出了一种简单 yet effective technique 来 normalize the non-zero values of convolution operation.
results: experiments 表明， ESTN 可以在多种 computer vision tasks 中提高预测精度， 包括图像重建和分类， 而减少计算成本。Abstract
This paper focuses on an accurate and fast interpolation approach for image transformation employed in the design of CNN architectures. Standard Spatial Transformer Networks (STNs) use bilinear or linear interpolation as their interpolation, with unrealistic assumptions about the underlying data distributions, which leads to poor performance under scale variations. Moreover, STNs do not preserve the norm of gradients in propagation due to their dependency on sparse neighboring pixels. To address this problem, a novel Entropy STN (ESTN) is proposed that interpolates on the data manifold distributions. In particular, random samples are generated for each pixel in association with the tangent space of the data manifold and construct a linear approximation of their intensity values with an entropy regularizer to compute the transformer parameters. A simple yet effective technique is also proposed to normalize the non-zero values of the convolution operation, to fine-tune the layers for gradients' norm-regularization during training. Experiments on challenging benchmarks show that the proposed ESTN can improve predictive accuracy over a range of computer vision tasks, including image reconstruction, and classification, while reducing the computational cost.
摘要
To address these problems, a novel Entropy STN (ESTN) is proposed that interpolates on the data manifold distributions. Specifically, random samples are generated for each pixel in association with the tangent space of the data manifold, and a linear approximation of their intensity values is computed with an entropy regularizer to compute the transformer parameters. Additionally, a simple yet effective technique is proposed to normalize the non-zero values of the convolution operation to fine-tune the layers for gradients' norm-regularization during training.Experiments on challenging benchmarks show that the proposed ESTN can improve predictive accuracy over a range of computer vision tasks, including image reconstruction and classification, while reducing the computational cost.
Cross Contrasting Feature Perturbation for Domain Generalization
paper_authors: Chenming Li, Daoan Zhang, Wenjian Huang, Jianguo Zhang for: This paper focuses on the problem of domain generalization (DG) and proposes a novel framework called Cross Contrasting Feature Perturbation (CCFP) to simulate domain shift and improve the robustness of the model.methods: The proposed CCFP framework uses an online one-stage approach and generates perturbed features in the latent space while regularizing the model prediction against domain shift. The framework includes learnable feature perturbations and semantic consistency constraints to improve the quality of the perturbed features.results: The proposed method outperforms the previous state-of-the-art in the standard DomainBed benchmark with a strict evaluation protocol. Quantitative analyses show that the method can effectively alleviate the domain shift problem in out-of-distribution (OOD) scenarios.Abstract
Domain generalization (DG) aims to learn a robust model from source domains that generalize well on unseen target domains. Recent studies focus on generating novel domain samples or features to diversify distributions complementary to source domains. Yet, these approaches can hardly deal with the restriction that the samples synthesized from various domains can cause semantic distortion. In this paper, we propose an online one-stage Cross Contrasting Feature Perturbation (CCFP) framework to simulate domain shift by generating perturbed features in the latent space while regularizing the model prediction against domain shift. Different from the previous fixed synthesizing strategy, we design modules with learnable feature perturbations and semantic consistency constraints. In contrast to prior work, our method does not use any generative-based models or domain labels. We conduct extensive experiments on a standard DomainBed benchmark with a strict evaluation protocol for a fair comparison. Comprehensive experiments show that our method outperforms the previous state-of-the-art, and quantitative analyses illustrate that our approach can alleviate the domain shift problem in out-of-distribution (OOD) scenarios.
摘要
域间泛化（DG）目标是从源域学习一个可以在未看过的目标域中进行泛化的模型。最近的研究主要关注生成新的域样本或特征以增加分布的多样性，但这些方法很难处理源域样本生成的semantic扭曲问题。在这篇论文中，我们提出了在线一阶段 Cross Contrasting Feature Perturbation（CCFP）框架，通过在幂空间生成受损特征来模拟域转移，同时对模型预测进行域转移 regularization。与前一代固定生成策略不同，我们设计了可学习的特征干扰和semantic一致约束。与先前的工作不同，我们的方法不使用任何生成型模型或域标签。我们在DomainBed标准测试床上进行了广泛的实验，并对比评估准则进行严格的评估。广泛的实验结果表明，我们的方法超过了先前的最佳实现，并且量化分析表明，我们的方法可以在OOD（out-of-distribution）场景中缓解域转移问题。
AdvDiff: Generating Unrestricted Adversarial Examples using Diffusion Models
results: 实验结果表明，AdvDiff可以高效地生成不受限制的反击示例，并且在MNIST和ImageNet datasets上的实验结果表明，AdvDiff的攻击性和生成质量都高于基于GAN的方法。Abstract
Unrestricted adversarial attacks present a serious threat to deep learning models and adversarial defense techniques. They pose severe security problems for deep learning applications because they can effectively bypass defense mechanisms. However, previous attack methods often utilize Generative Adversarial Networks (GANs), which are not theoretically provable and thus generate unrealistic examples by incorporating adversarial objectives, especially for large-scale datasets like ImageNet. In this paper, we propose a new method, called AdvDiff, to generate unrestricted adversarial examples with diffusion models. We design two novel adversarial guidance techniques to conduct adversarial sampling in the reverse generation process of diffusion models. These two techniques are effective and stable to generate high-quality, realistic adversarial examples by integrating gradients of the target classifier interpretably. Experimental results on MNIST and ImageNet datasets demonstrate that AdvDiff is effective to generate unrestricted adversarial examples, which outperforms GAN-based methods in terms of attack performance and generation quality.
摘要
深度学习模型面临无限制 adversarial 攻击的威胁，这些攻击可以快速绕过防御机制。然而，过去的攻击方法常用生成式对抗网络（GAN），这些网络不能有效地证明其可行性，特别是在大规模的数据集如 ImageNet 上。在本文中，我们提出一种新的方法，称为 AdvDiff，以使用扩散模型生成无限制 adversarial 示例。我们设计了两种新的对抗导航技术，以在扩散模型的反生成过程中进行对抗采样。这两种技术能够生成高质量、实用的 adversarial 示例，通过可见的梯度来具体化目标分类器的解释。实验结果表明，AdvDiff 在 MNIST 和 ImageNet 数据集上效果地生成了无限制 adversarial 示例，其性能和生成质量都高于基于 GAN 的方法。
Rethinking Data Distillation: Do Not Overlook Calibration
methods: 本文提出了Masked Temperature Scaling (MTS)和Masked Distillation Training (MDT)两种方法，可以在神经网络训练过程中进行数据缩写和混合，以提高神经网络的准确率和可靠性。
results: 本文的实验结果表明，使用MTS和MDT可以有效地修正由大型源数据抽取而得到的神经网络，提高其准确率和可靠性，同时保持数据缩写的效率。Abstract
Neural networks trained on distilled data often produce over-confident output and require correction by calibration methods. Existing calibration methods such as temperature scaling and mixup work well for networks trained on original large-scale data. However, we find that these methods fail to calibrate networks trained on data distilled from large source datasets. In this paper, we show that distilled data lead to networks that are not calibratable due to (i) a more concentrated distribution of the maximum logits and (ii) the loss of information that is semantically meaningful but unrelated to classification tasks. To address this problem, we propose Masked Temperature Scaling (MTS) and Masked Distillation Training (MDT) which mitigate the limitations of distilled data and achieve better calibration results while maintaining the efficiency of dataset distillation.
摘要
neural networks 经过精炼数据训练后常会产生过度自信的输出，需要使用均衡方法进行调整。现有的均衡方法，如温度升降和混合方法，对原始大规模数据训练的网络具有良好的效果。然而，我们发现这些方法对含拟合数据训练的网络无法进行均衡。在这篇论文中，我们发现了含拟合数据导致网络无法均衡的两个问题：（一）含拟合数据导致网络的最大幂值分布更加集中，（二）含拟合数据丢失了与分类任务无关 yet semantically meaningful的信息。为解决这问题，我们提出了Masked Temperature Scaling（MTS）和Masked Distillation Training（MDT）两种方法，它们可以缓解含拟合数据的限制，实现更好的均衡结果，同时保持数据精炼的效率。
Robust face anti-spoofing framework with Convolutional Vision Transformer
results: 该模型在不同数据集中的跨域设定中表现出了7.3%$p$和12.9%$p$的提升，并在九个参考模型中得到了最高的平均排名。Abstract
Owing to the advances in image processing technology and large-scale datasets, companies have implemented facial authentication processes, thereby stimulating increased focus on face anti-spoofing (FAS) against realistic presentation attacks. Recently, various attempts have been made to improve face recognition performance using both global and local learning on face images; however, to the best of our knowledge, this is the first study to investigate whether the robustness of FAS against domain shifts is improved by considering global information and local cues in face images captured using self-attention and convolutional layers. This study proposes a convolutional vision transformer-based framework that achieves robust performance for various unseen domain data. Our model resulted in 7.3%$p$ and 12.9%$p$ increases in FAS performance compared to models using only a convolutional neural network or vision transformer, respectively. It also shows the highest average rank in sub-protocols of cross-dataset setting over the other nine benchmark models for domain generalization.
摘要
Translated into Simplified Chinese:因为图像处理技术的进步和大规模数据集，公司已经实施了人脸验证过程，从而促使更多关注面对攻击（FAS）的真实演示攻击。最近，有很多尝试来提高人脸识别性能使用全球和地方学习方法，但到目前为止，这是第一个研究是否可以通过考虑全球信息和地方指示来提高人脸验证性能对域shift。这种研究提出了基于 convolutional vision transformer 框架的方法，实现了对不同数据集的 Robust 性能。我们的模型比只使用 convolutional neural network 或 vision transformer 模型使用时提高了7.3%$p$ 和 12.9%$p$ 的 FAS性能。它还在横跨数据集设定下的各个子协议中显示了最高的平均排名。
EnTri: Ensemble Learning with Tri-level Representations for Explainable Scene Recognition
results: 在MIT67、SUN397和UIUC8 datasets上实现了87.69%、75.56%和99.17%的分类精度，比 estado-of-the-art方法有竞争力。Abstract
Scene recognition based on deep-learning has made significant progress, but there are still limitations in its performance due to challenges posed by inter-class similarities and intra-class dissimilarities. Furthermore, prior research has primarily focused on improving classification accuracy, yet it has given less attention to achieving interpretable, precise scene classification. Therefore, we are motivated to propose EnTri, an ensemble scene recognition framework that employs ensemble learning using a hierarchy of visual features. EnTri represents features at three distinct levels of detail: pixel-level, semantic segmentation-level, and object class and frequency level. By incorporating distinct feature encoding schemes of differing complexity and leveraging ensemble strategies, our approach aims to improve classification accuracy while enhancing transparency and interpretability via visual and textual explanations. To achieve interpretability, we devised an extension algorithm that generates both visual and textual explanations highlighting various properties of a given scene that contribute to the final prediction of its category. This includes information about objects, statistics, spatial layout, and textural details. Through experiments on benchmark scene classification datasets, EnTri has demonstrated superiority in terms of recognition accuracy, achieving competitive performance compared to state-of-the-art approaches, with an accuracy of 87.69%, 75.56%, and 99.17% on the MIT67, SUN397, and UIUC8 datasets, respectively.
摘要
EnTri represents features at three distinct levels of detail: pixel-level, semantic segmentation-level, and object class and frequency level. By incorporating distinct feature encoding schemes of differing complexity and leveraging ensemble strategies, our approach aims to improve classification accuracy while enhancing transparency and interpretability via visual and textual explanations.To achieve interpretability, we devised an extension algorithm that generates both visual and textual explanations highlighting various properties of a given scene that contribute to the final prediction of its category. This includes information about objects, statistics, spatial layout, and textural details.Through experiments on benchmark scene classification datasets, EnTri has demonstrated superiority in terms of recognition accuracy, achieving competitive performance compared to state-of-the-art approaches, with an accuracy of 87.69%, 75.56%, and 99.17% on the MIT67, SUN397, and UIUC8 datasets, respectively.
SwIPE: Efficient and Robust Medical Image Segmentation with Implicit Patch Embeddings
results: 实验结果显示，这个方法可以与现有的离散方法进行比较，并且在两个任务（2D肿瘤分类和3D腹部器官分类）上获得了更好的结果，并且需要较少的参数。此外，这个方法也展示了较好的数据效率和数据类型的适应性。Abstract
Modern medical image segmentation methods primarily use discrete representations in the form of rasterized masks to learn features and generate predictions. Although effective, this paradigm is spatially inflexible, scales poorly to higher-resolution images, and lacks direct understanding of object shapes. To address these limitations, some recent works utilized implicit neural representations (INRs) to learn continuous representations for segmentation. However, these methods often directly adopted components designed for 3D shape reconstruction. More importantly, these formulations were also constrained to either point-based or global contexts, lacking contextual understanding or local fine-grained details, respectively--both critical for accurate segmentation. To remedy this, we propose a novel approach, SwIPE (Segmentation with Implicit Patch Embeddings), that leverages the advantages of INRs and predicts shapes at the patch level--rather than at the point level or image level--to enable both accurate local boundary delineation and global shape coherence. Extensive evaluations on two tasks (2D polyp segmentation and 3D abdominal organ segmentation) show that SwIPE significantly improves over recent implicit approaches and outperforms state-of-the-art discrete methods with over 10x fewer parameters. Our method also demonstrates superior data efficiency and improved robustness to data shifts across image resolutions and datasets. Code is available on Github.
摘要
现代医学图像分割方法主要使用精度为矩阵的批处理来学习特征和生成预测。虽然有效，但这种方法具有不可修复的局限性，包括空间不灵活、高分辨率图像扩展不良、直接没有对物体形状的理解。为了解决这些限制，一些最近的研究使用了卷积神经网络（INR）来学习连续表示，以提高分割精度。然而，这些方法通常直接采用了设计 для三维形态重建的组件，而且受限于点级或全局上下文，缺乏当地细节或形态准确性。为了改善这个问题，我们提出了一种新的方法：SwIPE（分割with Implicit Patch Embeddings），它利用INR的优点，预测形状在patch水平（而不是点级或图像级），以实现准确的本地边界定义和全局形态协调。我们对两个任务（2D菌体分割和3D腹部器官分割）进行了广泛的评估，结果表明SwIPE在最近的隐式方法中显著提高，并在精度和数据效率方面超过了状态机 discrete方法。我们的方法还在数据偏移和图像分辨率之间具有更好的数据效率和数据弹性。代码可以在Github上获取。
Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection
paper_authors: Liu Yuyang, Cong Yang, Goswami Dipam, Liu Xialei, Joost van de Weijer for: 这篇论文的目的是解决对incremental object detection（IOD）中的catastrophic forgetting问题。methods: 本文使用了一个称为Augmented Box Replay（ABR）的新方法，它将仅储存和重复过去任务中的前景物体，以避免预义遗传问题。此外，本文也提出了一种创新的注意力捕捉RoI特征的Attentive RoI Distillation损失，它使用领域特征的空间注意力来锁定现在的模型对过去模型的重要信息的注意。results: ABR有效地降低了之前任务的遗传，同时保持了目前任务的高柔养性。此外，ABR还能够储存和重复的减少储存需求，相比于标准的图像重复。实验结果表明，本文的模型在Pascal-VOC和COCO dataset上具有现场的表现。Abstract
In incremental learning, replaying stored samples from previous tasks together with current task samples is one of the most efficient approaches to address catastrophic forgetting. However, unlike incremental classification, image replay has not been successfully applied to incremental object detection (IOD). In this paper, we identify the overlooked problem of foreground shift as the main reason for this. Foreground shift only occurs when replaying images of previous tasks and refers to the fact that their background might contain foreground objects of the current task. To overcome this problem, a novel and efficient Augmented Box Replay (ABR) method is developed that only stores and replays foreground objects and thereby circumvents the foreground shift problem. In addition, we propose an innovative Attentive RoI Distillation loss that uses spatial attention from region-of-interest (RoI) features to constrain current model to focus on the most important information from old model. ABR significantly reduces forgetting of previous classes while maintaining high plasticity in current classes. Moreover, it considerably reduces the storage requirements when compared to standard image replay. Comprehensive experiments on Pascal-VOC and COCO datasets support the state-of-the-art performance of our model.
摘要
在增量学习中，重新播放之前任务的样本和当前任务的样本是解决快速卷积承忘的一种最有效的方法。然而，与增量分类不同，图像重新播放在增量物体检测（IOD）中尚未得到成功。在这篇论文中，我们认为背景变化导致的前景偏移是主要的问题。背景变化仅发生在重新播放前任务的图像时，并且指的是图像的背景中可能包含当前任务的前景对象。为解决这个问题，我们开发了一种新的和高效的增量盒子重播（ABR）方法，该方法仅存储和重播前景对象，因此可以避免前景偏移问题。此外，我们提出了一种创新的注意力捕捉的区域特征练习损失（Attentive RoI Distillation loss），该损失使当前模型通过区域特征中的注意力来约束当前模型关注到最重要的信息。ABR显著减少了之前类的忘记，同时保持当前类的高灵活性。此外，它比标准图像重新播放要减少存储需求。我们在 Pascal-VOC 和 COCO 数据集上进行了广泛的实验，并证明了我们的模型的状态级表现。
TransNet: Transparent Object Manipulation Through Category-Level Pose Estimation
methods: 本研究提出了一种两stage管道 named TransNet，包括本地化深度完成和表面法向估计两个阶段。TransNet使用了一种新的surface normal estimation方法，并且使用了一种新的depth completion方法来提高pose estimation的准确性。
results: 对于一个大规模的透明物体 dataset，TransNet achieved improved pose estimation accuracy compared to a state-of-the-art category-level pose estimation approach. In addition, TransNet was used to build an autonomous transparent object manipulation system for robotic pick-and-place and pouring tasks, which demonstrated its effectiveness in real-world applications.Abstract
Transparent objects present multiple distinct challenges to visual perception systems. First, their lack of distinguishing visual features makes transparent objects harder to detect and localize than opaque objects. Even humans find certain transparent surfaces with little specular reflection or refraction, like glass doors, difficult to perceive. A second challenge is that depth sensors typically used for opaque object perception cannot obtain accurate depth measurements on transparent surfaces due to their unique reflective properties. Stemming from these challenges, we observe that transparent object instances within the same category, such as cups, look more similar to each other than to ordinary opaque objects of that same category. Given this observation, the present paper explores the possibility of category-level transparent object pose estimation rather than instance-level pose estimation. We propose \textit{\textbf{TransNet}, a two-stage pipeline that estimates category-level transparent object pose using localized depth completion and surface normal estimation. TransNet is evaluated in terms of pose estimation accuracy on a large-scale transparent object dataset and compared to a state-of-the-art category-level pose estimation approach. Results from this comparison demonstrate that TransNet achieves improved pose estimation accuracy on transparent objects. Moreover, we use TransNet to build an autonomous transparent object manipulation system for robotic pick-and-place and pouring tasks.
摘要
trasparent objects present multiple distinct challenges to visual perception systems. First, their lack of distinguishing visual features makes transparent objects harder to detect and localize than opaque objects. Even humans find certain transparent surfaces with little specular reflection or refraction, like glass doors, difficult to perceive. A second challenge is that depth sensors typically used for opaque object perception cannot obtain accurate depth measurements on transparent surfaces due to their unique reflective properties. Stemming from these challenges, we observe that transparent object instances within the same category, such as cups, look more similar to each other than to ordinary opaque objects of that same category. Given this observation, the present paper explores the possibility of category-level transparent object pose estimation rather than instance-level pose estimation. We propose \textbf{\textit{TransNet}， a two-stage pipeline that estimates category-level transparent object pose using localized depth completion and surface normal estimation. TransNet is evaluated in terms of pose estimation accuracy on a large-scale transparent object dataset and compared to a state-of-the-art category-level pose estimation approach. Results from this comparison demonstrate that TransNet achieves improved pose estimation accuracy on transparent objects. Moreover, we use TransNet to build an autonomous transparent object manipulation system for robotic pick-and-place and pouring tasks.Note: Please note that the translation is in Simplified Chinese, which is one of the two standard versions of Chinese. If you need the translation in Traditional Chinese, please let me know.
Iterative Robust Visual Grounding with Masked Reference based Centerpoint Supervision
paper_authors: Menghao Li, Chunlei Wang, Wenquan Feng, Shuchang Lyu, Guangliang Cheng, Xiangtai Li, Binghao Liu, Qi Zhao for: 本研究旨在解决现有的视觉固定问题，即基于给定的描述生成假阳性对象。methods: 本研究提出了一种Iterative Robust Visual Grounding（IR-VG）框架，包括多层视语融合（IMVF）和掩码参考中心点监督（MRCS）等技术，以提高对描述的匹配和对图像中的细节特征的捕捉。results: 对五个常见的视觉固定数据集和两个新提出的鲁棒视觉固定数据集进行了广泛的实验，并得到了新的最佳性能记录，相比之前的最佳方法在两个新提出的鲁棒视觉固定数据集上提高了25%和10%。此外，该方法还在五个常见的视觉固定数据集上得到了证明。Abstract
Visual Grounding (VG) aims at localizing target objects from an image based on given expressions and has made significant progress with the development of detection and vision transformer. However, existing VG methods tend to generate false-alarm objects when presented with inaccurate or irrelevant descriptions, which commonly occur in practical applications. Moreover, existing methods fail to capture fine-grained features, accurate localization, and sufficient context comprehension from the whole image and textual descriptions. To address both issues, we propose an Iterative Robust Visual Grounding (IR-VG) framework with Masked Reference based Centerpoint Supervision (MRCS). The framework introduces iterative multi-level vision-language fusion (IMVF) for better alignment. We use MRCS to ahieve more accurate localization with point-wised feature supervision. Then, to improve the robustness of VG, we also present a multi-stage false-alarm sensitive decoder (MFSD) to prevent the generation of false-alarm objects when presented with inaccurate expressions. The proposed framework is evaluated on five regular VG datasets and two newly constructed robust VG datasets. Extensive experiments demonstrate that IR-VG achieves new state-of-the-art (SOTA) results, with improvements of 25\% and 10\% compared to existing SOTA approaches on the two newly proposed robust VG datasets. Moreover, the proposed framework is also verified effective on five regular VG datasets. Codes and models will be publicly at https://github.com/cv516Buaa/IR-VG.
摘要
Visual Grounding (VG) target objetcs from an image based on given expressions and has made significant progress with the development of detection and vision transformer. However, existing VG methods tend to generate false-alarm objects when presented with inaccurate or irrelevant descriptions, which commonly occur in practical applications. Moreover, existing methods fail to capture fine-grained features, accurate localization, and sufficient context comprehension from the whole image and textual descriptions. To address both issues, we propose an Iterative Robust Visual Grounding (IR-VG) framework with Masked Reference based Centerpoint Supervision (MRCS). The framework introduces iterative multi-level vision-language fusion (IMVF) for better alignment. We use MRCS to achieve more accurate localization with point-wised feature supervision. Then, to improve the robustness of VG, we also present a multi-stage false-alarm sensitive decoder (MFSD) to prevent the generation of false-alarm objects when presented with inaccurate expressions. The proposed framework is evaluated on five regular VG datasets and two newly constructed robust VG datasets. Extensive experiments demonstrate that IR-VG achieves new state-of-the-art (SOTA) results, with improvements of 25% and 10% compared to existing SOTA approaches on the two newly proposed robust VG datasets. Moreover, the proposed framework is also verified effective on five regular VG datasets. Codes and models will be publicly available at https://github.com/cv516Buaa/IR-VG.
Assessing Intra-class Diversity and Quality of Synthetically Generated Images in a Biomedical and Non-biomedical Setting
paper_authors: Muhammad Muneeb Saad, Mubashir Husain Rehmani, Ruairi O’Reilly
for: This paper aims to evaluate the effectiveness of using Generative Adversarial Networks (GANs) for data augmentation in biomedical image analysis, and to investigate the impact of different sample sizes on the diversity and quality of synthetic images.
methods: The paper uses Multi-scale Structural Similarity Index Measure, Cosine Distance, and Frechet Inception Distance to evaluate the diversity and quality of synthetic images generated by a Deep Convolutional GAN in both biomedical and non-biomedical imaging modalities.
results: The results show that the metrics scores for diversity and quality vary significantly across biomedical-to-biomedical and biomedical-to-non-biomedical imaging modalities, and that the diversity and quality of synthetic images are affected by the sample size used for training the GAN.Abstract
In biomedical image analysis, data imbalance is common across several imaging modalities. Data augmentation is one of the key solutions in addressing this limitation. Generative Adversarial Networks (GANs) are increasingly being relied upon for data augmentation tasks. Biomedical image features are sensitive to evaluating the efficacy of synthetic images. These features can have a significant impact on metric scores when evaluating synthetic images across different biomedical imaging modalities. Synthetically generated images can be evaluated by comparing the diversity and quality of real images. Multi-scale Structural Similarity Index Measure and Cosine Distance are used to evaluate intra-class diversity, while Frechet Inception Distance is used to evaluate the quality of synthetic images. Assessing these metrics for biomedical and non-biomedical imaging is important to investigate an informed strategy in evaluating the diversity and quality of synthetic images. In this work, an empirical assessment of these metrics is conducted for the Deep Convolutional GAN in a biomedical and non-biomedical setting. The diversity and quality of synthetic images are evaluated using different sample sizes. This research intends to investigate the variance in diversity and quality across biomedical and non-biomedical imaging modalities. Results demonstrate that the metrics scores for diversity and quality vary significantly across biomedical-to-biomedical and biomedical-to-non-biomedical imaging modalities.
摘要
在生物医学影像分析中，数据不均衡是广泛存在的问题，而数据扩充是解决这个问题的关键方法之一。生成敌对网络（GANs）在数据扩充任务中得到了越来越多的应用。生物医学影像特征对于评估 sintetic 图像的效果非常敏感。这些特征在评估不同生物医学成像模式下的 sintetic 图像时可以有显著的影响。 sintetic 图像可以通过比较真实图像的多样性和质量来评估。多尺度结构相似度指标和夹角距离是评估内部多样性的方法，而干扰抽象距离则是评估 sintetic 图像质量的方法。为了了解 informed 策略，需要对生物医学和非生物医学成像进行评估。本研究通过对这些指标进行实际评估，研究了不同样本大小下的多样性和质量的变化。结果表明，在生物医学到生物医学和生物医学到非生物医学的转换中，指标分数差异显著。