For: This paper proposes a new architecture for vision called X-MLP, which is designed to be independent from convolutions and self-attention operations.* Methods: The X-MLP architecture is constructed absolutely upon fully connected layers and is free from patch embedding. It decouples the features extremely and utilizes MLPs to interact the information across the dimension of width, height, and channel independently and alternately.* Results: X-MLP is tested on ten benchmark datasets and obtains better performance than other vision MLP models. It even surpasses CNNs by a clear margin on various datasets. Additionally, the paper visualizes the information communication between any couples of pixels in the feature map and observes the phenomenon of capturing long-range dependency.Here is the simplified Chinese text for the three key points:* For: 这篇论文提出了一种新的视觉建模方法，即X-MLP，它不依赖卷积和自注意力操作。* Methods: X-MLP采用全连接层构建，不需要负权重映射。它强制Feature decoupling，通过多层Perceptron交互信息，独立地处理宽高和通道维度。* Results: X-MLP在十个基准数据集上进行测试，与其他视觉MLP模型相比，表现更好。甚至在不同的数据集上，X-MLP还超过了CNN。此外，通过数学还原空间权重，可以visualize任意像素对像的信息交换，观察长距离关系。Abstract
Convolutional neural networks (CNNs) and vision transformers (ViT) have obtained great achievements in computer vision. Recently, the research of multi-layer perceptron (MLP) architectures for vision have been popular again. Vision MLPs are designed to be independent from convolutions and self-attention operations. However, existing vision MLP architectures always depend on convolution for patch embedding. Thus we propose X-MLP, an architecture constructed absolutely upon fully connected layers and free from patch embedding. It decouples the features extremely and utilizes MLPs to interact the information across the dimension of width, height and channel independently and alternately. X-MLP is tested on ten benchmark datasets, all obtaining better performance than other vision MLP models. It even surpasses CNNs by a clear margin on various dataset. Furthermore, through mathematically restoring the spatial weights, we visualize the information communication between any couples of pixels in the feature map and observe the phenomenon of capturing long-range dependency.
摘要
卷积神经网络（CNN）和视Transformer（ViT）在计算机视觉领域已经取得了很大的成就。最近，视觉多层批处理（MLP）建筑的研究又再次受到了关注。视觉MLP建筑始终依赖于混合层和自我注意力操作。然而，现有的视觉MLP建筑都是通过混合层进行质心编码。因此，我们提出X-MLP，一种完全基于全连接层的建筑，不依赖于混合层和质心编码。它将特征分解得非常细致，并使用MLP进行特征之间的信息交互，在宽度、高度和通道维度上独立地交换信息。X-MLP在十个 benchmark 数据集上进行测试，都超过了其他视觉MLP模型。它甚至在不同的数据集上超过了 CNN。此外，通过数学还原空间权重，我们可以视觉化特征图中任意对的像素对之间的信息交换，并观察捕捉长距离相互作用。
ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition
results: 我们的结果显示，使用CLIP视觉token和架构设计的涉及式注意力Transformer（ClipSitu XTF）可以在semantic role labeling（值）任务中带来14.1%的提升，使得我们的模型在imSitu数据集上达到了最高的top-1准确率。Abstract
Situation Recognition is the task of generating a structured summary of what is happening in an image using an activity verb and the semantic roles played by actors and objects. In this task, the same activity verb can describe a diverse set of situations as well as the same actor or object category can play a diverse set of semantic roles depending on the situation depicted in the image. Hence model needs to understand the context of the image and the visual-linguistic meaning of semantic roles. Therefore, we leverage the CLIP foundational model that has learned the context of images via language descriptions. We show that deeper-and-wider multi-layer perceptron (MLP) blocks obtain noteworthy results for the situation recognition task by using CLIP image and text embedding features and it even outperforms the state-of-the-art CoFormer, a Transformer-based model, thanks to the external implicit visual-linguistic knowledge encapsulated by CLIP and the expressive power of modern MLP block designs. Motivated by this, we design a cross-attention-based Transformer using CLIP visual tokens that model the relation between textual roles and visual entities. Our cross-attention-based Transformer known as ClipSitu XTF outperforms existing state-of-the-art by a large margin of 14.1% on semantic role labelling (value) for top-1 accuracy using imSitu dataset. We will make the code publicly available.
摘要
Situation recognition是将图像中的活动和角色描述为结构化的摘要。在这个任务中，同一个活动词可以描述多种不同的情况，同一个actor或object category可以扮演多种不同的semantic role，具体取决于图像中所示的情况。因此，模型需要理解图像的上下文和视觉语言意义。为此，我们利用CLIP基础模型，它通过语言描述学习了图像的上下文。我们发现，在使用CLIP图像和文本嵌入特征的更深更宽多层感知（MLP）块时，可以获得优秀的结果，并在CoFormer，一种基于Transformer的模型，之上出现胜利。这是因为CLIP中的外部隐式视觉语言知识和现代MLP块设计的表达能力。为了进一步解决这个问题，我们设计了一个基于交叉关注的Transformer，使用CLIP的视觉token来模拟文本角色和视觉实体之间的关系。我们称之为ClipSitu XTF，它在imSitu数据集上的semantic role标签（值）的top-1准确率上超越了现有的状态艺术。我们将代码公开。
A multi-task learning framework for carotid plaque segmentation and classification from ultrasound images
results: 实验结果显示，提出的方法可以significantly提高与单个任务训练的网络性能，其中分类任务的准确率为 85.82%，分割任务的 dice similarity 系数为 84.92%。在减少研究中，结果表明了 RWM 和 SWM 都对网络性能的提高有所贡献。因此，我们认为该方法可以在临床试验和实践中用于血管壁瘤分析。Abstract
Carotid plaque segmentation and classification play important roles in the treatment of atherosclerosis and assessment for risk of stroke. Although deep learning methods have been used for carotid plaque segmentation and classification, most focused on a single task and ignored the relationship between the segmentation and classification of carotid plaques. Therefore, we propose a multi-task learning framework for ultrasound carotid plaque segmentation and classification, which utilizes a region-weight module (RWM) and a sample-weight module (SWM) to exploit the correlation between these two tasks. The RWM provides a plaque regional prior knowledge to the classification task, while the SWM is designed to learn the categorical sample weight for the segmentation task. A total of 1270 2D ultrasound images of carotid plaques were collected from Zhongnan Hospital (Wuhan, China) for our experiments. The results of the experiments showed that the proposed method can significantly improve the performance compared to existing networks trained for a single task, with an accuracy of 85.82% for classification and a Dice similarity coefficient of 84.92% for segmentation. In the ablation study, the results demonstrated that both the designed RWM and SWM were beneficial in improving the network's performance. Therefore, we believe that the proposed method could be useful for carotid plaque analysis in clinical trials and practice.
摘要
<>translate("Carotid plaque segmentation and classification play important roles in the treatment of atherosclerosis and assessment for risk of stroke. Although deep learning methods have been used for carotid plaque segmentation and classification, most focused on a single task and ignored the relationship between the segmentation and classification of carotid plaques. Therefore, we propose a multi-task learning framework for ultrasound carotid plaque segmentation and classification, which utilizes a region-weight module (RWM) and a sample-weight module (SWM) to exploit the correlation between these two tasks. The RWM provides a plaque regional prior knowledge to the classification task, while the SWM is designed to learn the categorical sample weight for the segmentation task. A total of 1270 2D ultrasound images of carotid plaques were collected from Zhongnan Hospital (Wuhan, China) for our experiments. The results of the experiments showed that the proposed method can significantly improve the performance compared to existing networks trained for a single task, with an accuracy of 85.82% for classification and a Dice similarity coefficient of 84.92% for segmentation. In the ablation study, the results demonstrated that both the designed RWM and SWM were beneficial in improving the network's performance. Therefore, we believe that the proposed method could be useful for carotid plaque analysis in clinical trials and practice.")Here's the translation:脉络栓分割和分类在血栓病治疗和风险评估中发挥重要作用。虽然深度学习方法已经用于脉络栓分割和分类，但大多数方法只关注单一任务，忽略了脉络栓分割和分类之间的关系。因此，我们提出了一种多任务学习框架 для超声脉络栓分割和分类，该框架利用区域权重模块（RWM）和样本权重模块（SWM）来利用这两个任务之间的相关性。RWM提供了脉络栓区域先验知识 для分类任务，而SWM是用于学习分类任务中的 categorical 样本权重。我们对1270个2D超声脉络栓图像进行了实验，结果显示，我们的方法可以与现有的单任务网络相比，提高表现，具体数据为85.82%的准确率和84.92%的相似度。在剥离研究中，结果表明，我们设计的RWM和SWM都对网络表现有益。因此，我们认为该方法在临床试验和实践中可能是有用的。
results: 这篇论文的结果显示，使用TinySiamese可以大幅降低训练时间和汇入时间，并且在鉴定和分类任务中获得了比标准Siamese更高的精度。Abstract
Biometric recognition is the process of verifying or classifying human characteristics in images or videos. It is a complex task that requires machine learning algorithms, including convolutional neural networks (CNNs) and Siamese networks. Besides, there are several limitations to consider when using these algorithms for image verification and classification tasks. In fact, training may be computationally intensive, requiring specialized hardware and significant computational resources to train and deploy. Moreover, it necessitates a large amount of labeled data, which can be time-consuming and costly to obtain. The main advantage of the proposed TinySiamese compared to the standard Siamese is that it does not require the whole CNN for training. In fact, using a pre-trained CNN as a feature extractor and the TinySiamese to learn the extracted features gave almost the same performance and efficiency as the standard Siamese for biometric verification. In this way, the TinySiamese solves the problems of memory and computational time with a small number of layers which did not exceed 7. It can be run under low-power machines which possess a normal GPU and cannot allocate a large RAM space. Using TinySiamese with only 8 GO of memory, the matching time decreased by 76.78% on the B2F (Biometric images of Fingerprints and Faces), FVC2000, FVC2002 and FVC2004 while the training time for 10 epochs went down by approximately 93.14% on the B2F, FVC2002, THDD-part1 and CASIA-B datasets. The accuracy of the fingerprint, gait (NM-angle 180 degree) and face verification tasks was better than the accuracy of a standard Siamese by 0.87%, 20.24% and 3.85% respectively. TinySiamese achieved comparable accuracy with related works for the fingerprint and gait classification tasks.
摘要
生物特征识别是将人体特征从图像或视频中鉴别或分类的过程。这是一项复杂的任务，需要机器学习算法，包括卷积神经网络（CNN）和同一个网络（Siamese）。然而，使用这些算法进行图像鉴别和分类任务时，存在一些限制。首先，训练可能会是 computationally intensive，需要特殊的硬件和大量计算资源来训练和部署。此外，需要大量标注数据，可能需要很长时间和成本来获得。与标准Siamese相比，我们提出的TinySiamese具有一些优点。首先，它不需要整个CNN进行训练。相反，使用预训练的CNN作为特征提取器，然后使用TinySiamese来学习提取的特征。这与标准Siamese的性能和效率几乎相同。此外，TinySiamese通过减少内存和计算时间的问题，使用只有7层的小型神经网络。这使得它可以在低功耗机器上运行，并且可以避免大量的内存和计算资源。使用TinySiamese，我们可以在10个epoch的训练时间下降约93.14%，而匹配时间下降约76.78%。此外，在B2F、FVC2000、FVC2002和FVC2004等数据集上，TinySiamese的精度比标准Siamese高出0.87%、20.24%和3.85%。此外，TinySiamese在指纹、步态（NM-angle 180度）和脸部鉴别任务中达到了相当的精度。
Bidirectional Temporal Diffusion Model for Temporally Consistent Human Animation
results: 我们的方法在实验中表现出了强大的表现，与现有的单向方法相比，具有真实的时间准确性。Abstract
We introduce a method to generate temporally coherent human animation from a single image, a video, or a random noise. This problem has been formulated as modeling of an auto-regressive generation, i.e., to regress past frames to decode future frames. However, such unidirectional generation is highly prone to motion drifting over time, generating unrealistic human animation with significant artifacts such as appearance distortion. We claim that bidirectional temporal modeling enforces temporal coherence on a generative network by largely suppressing the motion ambiguity of human appearance. To prove our claim, we design a novel human animation framework using a denoising diffusion model: a neural network learns to generate the image of a person by denoising temporal Gaussian noises whose intermediate results are cross-conditioned bidirectionally between consecutive frames. In the experiments, our method demonstrates strong performance compared to existing unidirectional approaches with realistic temporal coherence
摘要
我们提出了一种将单一图像、视频或随机变数转换为有着时间对称的人体动画的方法。这问题被视为一个自动复原生成问题，即从过去的几帧图像中预测未来几帧图像。但这种单向生成很容易受到时间漂移的影响，导致生成的人体动画有着严重的扭转现象和错误。我们声称，将时间方向统一到生成网络中可以严重抑制人体出现的动作抽象。为了证明我们的声明，我们设计了一个新的人体动画框架，使用一个对应推对推的滤除扩散模型：一个神经网络将从随机 Gaussian 噪声中获得人体图像，并在过去和未来几帧图像之间进行对应推对推的滤除过程。在实验中，我们的方法与现有的单向方法相比，具有更强的时间对称性和更真实的人体动画。
A MIL Approach for Anomaly Detection in Surveillance Videos from Multiple Camera Views
results: 对多个视角PETS-2009benchmark dataset进行重新标注，并使用多视角combined损失函数和Sultani的MIL排名函数来训练一个回归网络，得到了与单视角配置相比显著的F1分数提升。Abstract
Occlusion and clutter are two scene states that make it difficult to detect anomalies in surveillance video. Furthermore, anomaly events are rare and, as a consequence, class imbalance and lack of labeled anomaly data are also key features of this task. Therefore, weakly supervised methods are heavily researched for this application. In this paper, we tackle these typical problems of anomaly detection in surveillance video by combining Multiple Instance Learning (MIL) to deal with the lack of labels and Multiple Camera Views (MC) to reduce occlusion and clutter effects. In the resulting MC-MIL algorithm we apply a multiple camera combined loss function to train a regression network with Sultani's MIL ranking function. To evaluate the MC-MIL algorithm first proposed here, the multiple camera PETS-2009 benchmark dataset was re-labeled for the anomaly detection task from multiple camera views. The result shows a significant performance improvement in F1 score compared to the single-camera configuration.
摘要
干扰和堵塞是两种场景状态，使得异常检测在监视视频中具有棘手。此外，异常事件是罕见的，因此类别不平衡和缺乏异常数据也是这个任务的关键特点。因此，弱地监视方法受到了广泛的研究。在这篇论文中，我们解决了监视视频中异常检测中的常见问题，包括缺乏标签和干扰和堵塞的影响。我们提出了 MC-MIL 算法，该算法通过将多个摄像头视图组合成一个多视图损失函数，来训练一个 regression 网络，并使用 Sulani 的 MIL 排名函数。为评估 MC-MIL 算法，我们对多Camera PETS-2009 数据集进行了重新标注，以便进行异常检测任务。结果表明，相比单摄像头配置，MC-MIL 算法在 F1 分数上显著提高了性能。
Partial-label Learning with Mixed Closed-set and Open-set Out-of-candidate Examples
results: 对比州的PLL方法，本研究的方法得到了更高的性能。Abstract
Partial-label learning (PLL) relies on a key assumption that the true label of each training example must be in the candidate label set. This restrictive assumption may be violated in complex real-world scenarios, and thus the true label of some collected examples could be unexpectedly outside the assigned candidate label set. In this paper, we term the examples whose true label is outside the candidate label set OOC (out-of-candidate) examples, and pioneer a new PLL study to learn with OOC examples. We consider two types of OOC examples in reality, i.e., the closed-set/open-set OOC examples whose true label is inside/outside the known label space. To solve this new PLL problem, we first calculate the wooden cross-entropy loss from candidate and non-candidate labels respectively, and dynamically differentiate the two types of OOC examples based on specially designed criteria. Then, for closed-set OOC examples, we conduct reversed label disambiguation in the non-candidate label set; for open-set OOC examples, we leverage them for training by utilizing an effective regularization strategy that dynamically assigns random candidate labels from the candidate label set. In this way, the two types of OOC examples can be differentiated and further leveraged for model training. Extensive experiments demonstrate that our proposed method outperforms state-of-the-art PLL methods.
摘要
ARHNet: Adaptive Region Harmonization for Lesion-aware Augmentation to Improve Segmentation Performance
paper_authors: Jiayu Huo, Yang Liu, Xi Ouyang, Alejandro Granados, Sebastien Ourselin, Rachel Sparks
for: 提高MRI扫描图像中脑损伤的识别精度，以提供patients with prognoses和 neurosurgical monitoring。
methods: 使用CNN模型进行图像分割，并采用进步的数据增强策略来提高模型的可靠性。
results: 在实验中，ARHNet在实验中提高了下游分割性能，并在真实和Synthetic图像上达到了最高的表现。Abstract
Accurately segmenting brain lesions in MRI scans is critical for providing patients with prognoses and neurological monitoring. However, the performance of CNN-based segmentation methods is constrained by the limited training set size. Advanced data augmentation is an effective strategy to improve the model's robustness. However, they often introduce intensity disparities between foreground and background areas and boundary artifacts, which weakens the effectiveness of such strategies. In this paper, we propose a foreground harmonization framework (ARHNet) to tackle intensity disparities and make synthetic images look more realistic. In particular, we propose an Adaptive Region Harmonization (ARH) module to dynamically align foreground feature maps to the background with an attention mechanism. We demonstrate the efficacy of our method in improving the segmentation performance using real and synthetic images. Experimental results on the ATLAS 2.0 dataset show that ARHNet outperforms other methods for image harmonization tasks, and boosts the down-stream segmentation performance. Our code is publicly available at https://github.com/King-HAW/ARHNet.
摘要
优敏诊断患者的脑部病变需要高精度的MRI扫描图像分割，以提供病人诊断和 neuroscience 监测。然而，基于Convolutional Neural Network（CNN）的分割方法的性能受训练集大小的限制。高级数据增强技术可以提高模型的鲁棒性，但它们经常导致Intensity Disparity（ID）和边界artefacts，这会削弱这些策略的效果。本文提出了一种前景协调框架（ARHNet），以解决ID和Synthetic Image的不真实性问题。特别是，我们提出了一种适应区域协调（ARH）模块，通过关注机制来动态将前景特征图与背景图进行对齐。我们通过实验表明，ARHNet可以提高下游分割性能，并在ATLAS 2.0数据集上超越其他图像协调任务的方法。代码可以在https://github.com/King-HAW/ARHNet上获取。
Referring Video Object Segmentation with Inter-Frame Interaction and Cross-Modal Correlation
results: 经验 результаты表明，IFIRVOS在三个标准测试集上表现出了与state-of-the-art方法相比的superiority，并且 validate了我们提出的模块的有效性。Abstract
Referring video object segmentation (RVOS) aims to segment the target object in a video sequence described by a language expression. Typical query-based methods process the video sequence in a frame-independent manner to reduce the high computational cost, which however affects the performance due to the lack of inter-frame interaction for temporal coherence modeling and spatio-temporal representation learning of the referred object. Besides, they directly adopt the raw and high-level sentence feature as the language queries to decode the visual features, where the weak correlation between visual and linguistic features also increases the difficulty of decoding the target information and limits the performance of the model. In this paper, we proposes a novel RVOS framework, dubbed IFIRVOS, to address these issues. Specifically, we design a plug-and-play inter-frame interaction module in the Transformer decoder to efficiently learn the spatio-temporal features of the referred object, so as to decode the object information in the video sequence more precisely and generate more accurate segmentation results. Moreover, we devise the vision-language interaction module before the multimodal Transformer to enhance the correlation between the visual and linguistic features, thus facilitating the process of decoding object information from visual features by language queries in Transformer decoder and improving the segmentation performance. Extensive experimental results on three benchmarks validate the superiority of our IFIRVOS over state-of-the-art methods and the effectiveness of our proposed modules.
摘要
Traditional query-based methods for Referring Video Object Segmentation (RVOS) process the video sequence in a frame-independent manner, which can reduce computational cost but also affects performance due to the lack of inter-frame interaction for temporal coherence modeling and spatio-temporal representation learning of the referred object. Moreover, they directly use raw and high-level sentence features as language queries to decode visual features, which can lead to weak correlation between visual and linguistic features, limiting the performance of the model.In this paper, we propose a novel RVOS framework, called IFIRVOS, to address these issues. Specifically, we design a plug-and-play inter-frame interaction module in the Transformer decoder to efficiently learn spatio-temporal features of the referred object, allowing for more precise decoding of object information in the video sequence and generating more accurate segmentation results. Furthermore, we propose a vision-language interaction module before the multimodal Transformer to enhance the correlation between visual and linguistic features, facilitating the process of decoding object information from visual features using language queries in the Transformer decoder and improving segmentation performance.Experimental results on three benchmarks demonstrate the superiority of our IFIRVOS over state-of-the-art methods and the effectiveness of our proposed modules.
End-to-End Out-of-distribution Detection with Self-supervised Sampling
paper_authors: Sen Pei, Jiaxi Sun, Peng Qin, Qi Chen, Xinglong Wu, Xun Wang
for: 提高 Open World 中的 Out-of-distribution (OOD) 检测精度，帮助模型在关闭集合训练后在开放世界中识别未知数据。
methods: 提出一个通用概率框架来看待多种已有方法，并提出一种无需 OOD 数据的模型，即 Self-supervised Sampling for OOD Detection (SSOD)，利用了 convolution 操作的本地性来从 ID 数据中抽取自然的 OOD 信号，并同时优化 OOD 检测和正常 ID 分类。
results: 在多个大规模 benchmark 上实现了竞争力的状态平台，比如 SUN 上的 FPR95 下的性能提高了大约 48.99% 到 35.52% 之间，比最近的方法，如 KNN，有大幅度的优势。Abstract
Out-of-distribution (OOD) detection empowers the model trained on the closed set to identify unknown data in the open world. Though many prior techniques have yielded considerable improvements, two crucial obstacles still remain. Firstly, a unified perspective has yet to be presented to view the developed arts with individual designs, which is vital for providing insights into the related directions. Secondly, most research focuses on the post-processing schemes of the pre-trained features while disregarding the superiority of end-to-end training, dramatically limiting the upper bound of OOD detection. To tackle these issues, we propose a general probabilistic framework to interpret many existing methods and an OOD-data-free model, namely Self-supervised Sampling for OOD Detection (SSOD), to unfold the potential of end-to-end learning. SSOD efficiently exploits natural OOD signals from the in-distribution (ID) data based on the local property of convolution. With these supervisions, it jointly optimizes the OOD detection and conventional ID classification. Extensive experiments reveal that SSOD establishes competitive state-of-the-art performance on many large-scale benchmarks, where it outperforms the most recent approaches, such as KNN, by a large margin, e.g., 48.99% to 35.52% on SUN at FPR95.
摘要
SUGAR: Spherical Ultrafast Graph Attention Framework for Cortical Surface Registration
paper_authors: Jianxun Ren, Ning An, Youjia Zhang, Danyang Wang, Zhenyu Sun, Cong Lin, Weigang Cui, Weiwei Wang, Ying Zhou, Wei Zhang, Qingyu Hu, Ping Zhang, Dan Hu, Danhong Wang, Hesheng Liu for:SUGAR is designed to improve cortical surface registration, specifically addressing the challenge of aligning cortical functional and anatomical features across individuals.methods:SUGAR is a unified unsupervised deep-learning framework that incorporates a U-Net-based spherical graph attention network and leverages the Euler angle representation for deformation. The framework includes a similarity loss, fold loss, and multiple distortion losses to preserve topology and minimize various types of distortions.results:SUGAR exhibits comparable or superior registration performance in accuracy, distortion, and test-retest reliability compared to conventional and learning-based methods. Additionally, SUGAR achieves remarkable sub-second processing times, offering a notable speed-up of approximately 12,000 times in registering 9,000 subjects from the UK Biobank dataset in just 32 minutes.Abstract
Cortical surface registration plays a crucial role in aligning cortical functional and anatomical features across individuals. However, conventional registration algorithms are computationally inefficient. Recently, learning-based registration algorithms have emerged as a promising solution, significantly improving processing efficiency. Nonetheless, there remains a gap in the development of a learning-based method that exceeds the state-of-the-art conventional methods simultaneously in computational efficiency, registration accuracy, and distortion control, despite the theoretically greater representational capabilities of deep learning approaches. To address the challenge, we present SUGAR, a unified unsupervised deep-learning framework for both rigid and non-rigid registration. SUGAR incorporates a U-Net-based spherical graph attention network and leverages the Euler angle representation for deformation. In addition to the similarity loss, we introduce fold and multiple distortion losses, to preserve topology and minimize various types of distortions. Furthermore, we propose a data augmentation strategy specifically tailored for spherical surface registration, enhancing the registration performance. Through extensive evaluation involving over 10,000 scans from 7 diverse datasets, we showed that our framework exhibits comparable or superior registration performance in accuracy, distortion, and test-retest reliability compared to conventional and learning-based methods. Additionally, SUGAR achieves remarkable sub-second processing times, offering a notable speed-up of approximately 12,000 times in registering 9,000 subjects from the UK Biobank dataset in just 32 minutes. This combination of high registration performance and accelerated processing time may greatly benefit large-scale neuroimaging studies.
摘要
cortical surface registration 在跨个体标准化中扮演关键角色，但传统的注册算法 computationally inefficient。 recent learning-based registration algorithms 作为一个有前途的解决方案，Significantly improving processing efficiency。然而，there remains a gap in the development of a learning-based method that exceeds the state-of-the-art conventional methods simultaneously in computational efficiency, registration accuracy, and distortion control，despite the theoretically greater representational capabilities of deep learning approaches。To address the challenge，we present SUGAR，a unified unsupervised deep-learning framework for both rigid and non-rigid registration。SUGAR incorporates a U-Net-based spherical graph attention network and leverages the Euler angle representation for deformation。In addition to the similarity loss，we introduce fold and multiple distortion losses，to preserve topology and minimize various types of distortions。Furthermore，we propose a data augmentation strategy specifically tailored for spherical surface registration，enhancing the registration performance。Through extensive evaluation involving over 10,000 scans from 7 diverse datasets，we showed that our framework exhibits comparable or superior registration performance in accuracy, distortion, and test-retest reliability compared to conventional and learning-based methods。Additionally，SUGAR achieves remarkable sub-second processing times，offering a notable speed-up of approximately 12,000 times in registering 9,000 subjects from the UK Biobank dataset in just 32 minutes。This combination of high registration performance and accelerated processing time may greatly benefit large-scale neuroimaging studies。
Data-Free Quantization via Mixed-Precision Compensation without Fine-Tuning
results: 实验表明，DF-MPC 能够在无数据和 fine-tuning 过程的情况下，提高模型的精度，比较有效率。Abstract
Neural network quantization is a very promising solution in the field of model compression, but its resulting accuracy highly depends on a training/fine-tuning process and requires the original data. This not only brings heavy computation and time costs but also is not conducive to privacy and sensitive information protection. Therefore, a few recent works are starting to focus on data-free quantization. However, data-free quantization does not perform well while dealing with ultra-low precision quantization. Although researchers utilize generative methods of synthetic data to address this problem partially, data synthesis needs to take a lot of computation and time. In this paper, we propose a data-free mixed-precision compensation (DF-MPC) method to recover the performance of an ultra-low precision quantized model without any data and fine-tuning process. By assuming the quantized error caused by a low-precision quantized layer can be restored via the reconstruction of a high-precision quantized layer, we mathematically formulate the reconstruction loss between the pre-trained full-precision model and its layer-wise mixed-precision quantized model. Based on our formulation, we theoretically deduce the closed-form solution by minimizing the reconstruction loss of the feature maps. Since DF-MPC does not require any original/synthetic data, it is a more efficient method to approximate the full-precision model. Experimentally, our DF-MPC is able to achieve higher accuracy for an ultra-low precision quantized model compared to the recent methods without any data and fine-tuning process.
摘要
neural network 量化是一种非常有前途的解决方案在模型压缩领域，但它的结果准确度具有训练/精度调整过程和原始数据的依赖关系。这不仅带来了重量计算和时间成本，还不利于隐私信息保护。因此，一些最近的研究开始关注无数据量化。然而，无数据量化在ultra-low precision量化时不太好进行。虽然研究人员利用生成方法生成的假数据来解决这个问题，但数据生成需要很多计算和时间。在这篇论文中，我们提出了一种无数据混合精度补偿方法（DF-MPC）来恢复ultra-low precision量化模型的性能。我们假设量化错误由low-precision量化层引起的可以通过高精度量化层的重建来恢复。我们使用数学形式来表述重建loss между预训练全精度模型和它的层 wise混合精度量化模型。根据我们的形式，我们可以 theoretically得出closed-form解。由于DF-MPC不需要任何原始/生成数据，它是一种更高效的方法来近似全精度模型。实验表明，我们的DF-MPC可以在ultra-low precision量化模型中达到更高的准确度，比悉些最近的方法不需要任何数据和精度调整过程。
TopicFM+: Boosting Accuracy and Efficiency of Topic-Assisted Feature Matching
results: 我们通过了广泛的实验，证明了我们的方法在难以处理的场景中具有显著的优势。具体来说，我们的方法可以减少计算成本，同时保持高效性和高匹配精度。代码将在https://github.com/TruongKhang/TopicFM 上更新。Abstract
This study tackles the challenge of image matching in difficult scenarios, such as scenes with significant variations or limited texture, with a strong emphasis on computational efficiency. Previous studies have attempted to address this challenge by encoding global scene contexts using Transformers. However, these approaches suffer from high computational costs and may not capture sufficient high-level contextual information, such as structural shapes or semantic instances. Consequently, the encoded features may lack discriminative power in challenging scenes. To overcome these limitations, we propose a novel image-matching method that leverages a topic-modeling strategy to capture high-level contexts in images. Our method represents each image as a multinomial distribution over topics, where each topic represents a latent semantic instance. By incorporating these topics, we can effectively capture comprehensive context information and obtain discriminative and high-quality features. Additionally, our method effectively matches features within corresponding semantic regions by estimating the covisible topics. To enhance the efficiency of feature matching, we have designed a network with a pooling-and-merging attention module. This module reduces computation by employing attention only on fixed-sized topics and small-sized features. Through extensive experiments, we have demonstrated the superiority of our method in challenging scenarios. Specifically, our method significantly reduces computational costs while maintaining higher image-matching accuracy compared to state-of-the-art methods. The code will be updated soon at https://github.com/TruongKhang/TopicFM
摘要
To overcome these limitations, we propose a novel image-matching method that leverages a topic-modeling strategy to capture high-level contexts in images. Our method represents each image as a multinomial distribution over topics, where each topic represents a latent semantic instance. By incorporating these topics, we can effectively capture comprehensive context information and obtain discriminative and high-quality features. Additionally, our method effectively matches features within corresponding semantic regions by estimating the covisible topics.To enhance the efficiency of feature matching, we have designed a network with a pooling-and-merging attention module. This module reduces computation by employing attention only on fixed-sized topics and small-sized features. Through extensive experiments, we have demonstrated the superiority of our method in challenging scenarios. Specifically, our method significantly reduces computational costs while maintaining higher image-matching accuracy compared to state-of-the-art methods. The code will be updated soon at .
Seeing is not Believing: An Identity Hider for Human Vision Privacy Protection
methods: 使用 StyleGAN2 manipulate 原始 face 的 latent space，生成虚拟 face，并将虚拟 face 的视觉内容传播到原始 face 上，然后替换背景。
results: 提出了一种可以具有出色隐私保护和人脸识别稳定性的身份隐藏方法，并通过实验证明其性能优秀。Abstract
Massive captured face images are stored in the database for the identification of individuals. However, the stored images can be observed intentionally or unintentionally by data managers, which is not at the will of individuals and may cause privacy violations. Existing protection works only slightly change the visual content of the face while maintaining the utility of identification, making it susceptible to the inference of the true identity by human vision. In this paper, we propose an identity hider that enables significant visual content change for human vision while preserving high identifiability for face recognizers. Firstly, the identity hider generates a virtual face with new visual content by manipulating the latent space in StyleGAN2. In particular, the virtual face has the same irrelevant attributes as the original face, e.g., pose and expression. Secondly, the visual content of the virtual face is transferred into the original face and then the background is replaced with the original one. In addition, the identity hider has strong transferability, which ensures an arbitrary face recognizer can achieve satisfactory accuracy. Adequate experiments show that the proposed identity hider achieves excellent performance on privacy protection and identifiability preservation.
摘要
巨大的捕捉到的面部图像被存储在数据库中，用于个体识别。然而，存储的图像可能会被意外或非意外地被数据管理人员见到，这会导致个人隐私被侵犯。现有的保护措施只是轻度地改变视觉内容，保留了识别功能，这使得真实身份易于推测。在这篇论文中，我们提出了一种隐身器，允许对人类视觉来源进行显著视觉内容变化，同时保留高度识别性。首先，隐身器使用StyleGAN2中的latent space进行抽象，生成一个虚拟面，该虚拟面具有与原始面相同的无关属性，例如姿势和表情。其次，虚拟面中的视觉内容被转移到原始面上，并将背景替换为原始的背景。此外，隐身器具有强大的传送性，使得任何面Recognizer都可以达到满意的准确率。经过合适的实验，我们的隐身器实现了出色的隐私保护和识别性 preserved。
Domain Transfer Through Image-to-Image Translation for Uncertainty-Aware Prostate Cancer Classification
results: 研究结果显示，这个新方法可以将AUC提高到20%以上，与先前的研究相比（98.4% vs. 76.2%）。这显示了这个方法的可行性和有效性。Abstract
Prostate Cancer (PCa) is often diagnosed using High-resolution 3.0 Tesla(T) MRI, which has been widely established in clinics. However, there are still many medical centers that use 1.5T MRI units in the actual diagnostic process of PCa. In the past few years, deep learning-based models have been proven to be efficient on the PCa classification task and can be successfully used to support radiologists during the diagnostic process. However, training such models often requires a vast amount of data, and sometimes it is unobtainable in practice. Additionally, multi-source MRIs can pose challenges due to cross-domain distribution differences. In this paper, we have presented a novel approach for unpaired image-to-image translation of prostate mp-MRI for classifying clinically significant PCa, to be applied in data-constrained settings. First, we introduce domain transfer, a novel pipeline to translate unpaired 3.0T multi-parametric prostate MRIs to 1.5T, to increase the number of training data. Second, we estimate the uncertainty of our models through an evidential deep learning approach; and leverage the dataset filtering technique during the training process. Furthermore, we introduce a simple, yet efficient Evidential Focal Loss that incorporates the focal loss with evidential uncertainty to train our model. Our experiments demonstrate that the proposed method significantly improves the Area Under ROC Curve (AUC) by over 20% compared to the previous work (98.4% vs. 76.2%). We envision that providing prediction uncertainty to radiologists may help them focus more on uncertain cases and thus expedite the diagnostic process effectively. Our code is available at https://github.com/med-i-lab/DT_UE_PCa
摘要
prostato cancer (PCa) oftener diagnosed using High-resolution 3.0 Tesla(T) MRI, which has been widely established in clinics. However, there are still many medical centers that use 1.5T MRI units in the actual diagnostic process of PCa. In the past few years, deep learning-based models have been proven to be efficient on the PCa classification task and can be successfully used to support radiologists during the diagnostic process. However, training such models often requires a vast amount of data, and sometimes it is unobtainable in practice. Additionally, multi-source MRIs can pose challenges due to cross-domain distribution differences. In this paper, we have presented a novel approach for unpaired image-to-image translation of prostate mp-MRI for classifying clinically significant PCa, to be applied in data-constrained settings. First, we introduce domain transfer, a novel pipeline to translate unpaired 3.0T multi-parametric prostate MRIs to 1.5T, to increase the number of training data. Second, we estimate the uncertainty of our models through an evidential deep learning approach; and leverage the dataset filtering technique during the training process. Furthermore, we introduce a simple, yet efficient Evidential Focal Loss that incorporates the focal loss with evidential uncertainty to train our model. Our experiments demonstrate that the proposed method significantly improves the Area Under ROC Curve (AUC) by over 20% compared to the previous work (98.4% vs. 76.2%). We envision that providing prediction uncertainty to radiologists may help them focus more on uncertain cases and thus expedite the diagnostic process effectively. Our code is available at https://github.com/med-i-lab/DT_UE_PCa.
for: This paper is written to explore and improve the efficiency of black-box patch attacks on deep neural networks (DNNs), specifically in the decision-based setting.
methods: The paper proposes a new method called DevoPatch, which uses a differential evolutionary algorithm to optimize patches for black-box patch attacks. The method models patches using paired key-points and uses targeted images as the initialization of patches, and parameter optimizations are all performed on the integer domain.
results: The paper demonstrates that DevoPatch outperforms state-of-the-art black-box patch attacks in terms of patch area and attack success rate within a given query budget on image classification and face verification. Additionally, the paper conducts the vulnerability evaluation of ViT and MLP on image classification in the decision-based patch attack setting for the first time.Abstract
Deep neural networks (DNNs) have been showed to be highly vulnerable to imperceptible adversarial perturbations. As a complementary type of adversary, patch attacks that introduce perceptible perturbations to the images have attracted the interest of researchers. Existing patch attacks rely on the architecture of the model or the probabilities of predictions and perform poorly in the decision-based setting, which can still construct a perturbation with the minimal information exposed -- the top-1 predicted label. In this work, we first explore the decision-based patch attack. To enhance the attack efficiency, we model the patches using paired key-points and use targeted images as the initialization of patches, and parameter optimizations are all performed on the integer domain. Then, we propose a differential evolutionary algorithm named DevoPatch for query-efficient decision-based patch attacks. Experiments demonstrate that DevoPatch outperforms the state-of-the-art black-box patch attacks in terms of patch area and attack success rate within a given query budget on image classification and face verification. Additionally, we conduct the vulnerability evaluation of ViT and MLP on image classification in the decision-based patch attack setting for the first time. Using DevoPatch, we can evaluate the robustness of models to black-box patch attacks. We believe this method could inspire the design and deployment of robust vision models based on various DNN architectures in the future.
摘要
Translation notes:* "Deep neural networks" is translated as "深度神经网络" (shēn dào shén zhī wǎng luò)* "adversarial perturbations" is translated as "敌对扰动" (dí zhòu zhào dòng)* "patch attacks" is translated as "质子攻击" (zhì zǐ gōng jī)* "decision-based setting" is translated as "决策基础设定" (jīe yì jī bǎsè)* "top-1 predicted label" is translated as "预测的第一个标签" (yù jí de dì yī gè biǎo)* "DevoPatch" is translated as "DevoPatch" (德朋补丁)* "query-efficient" is translated as "高效的查询" (gāo fáng de kè qiú)* "black-box patch attacks" is translated as "黑盒质子攻击" (hēi bāo zhì zǐ gōng jī)* "ViT" and "MLP" are translated as "ViT" and "MLP" (Vision Transformer and Multi-Layer Perceptron) respectively.
Weighted Anisotropic-Isotropic Total Variation for Poisson Denoising
results: numerical experiments显示，我们的算法在影像质量和计算效率方面都比其他Poisson噪声去除方法表现更好。Abstract
Poisson noise commonly occurs in images captured by photon-limited imaging systems such as in astronomy and medicine. As the distribution of Poisson noise depends on the pixel intensity value, noise levels vary from pixels to pixels. Hence, denoising a Poisson-corrupted image while preserving important details can be challenging. In this paper, we propose a Poisson denoising model by incorporating the weighted anisotropic-isotropic total variation (AITV) as a regularization. We then develop an alternating direction method of multipliers with a combination of a proximal operator for an efficient implementation. Lastly, numerical experiments demonstrate that our algorithm outperforms other Poisson denoising methods in terms of image quality and computational efficiency.
摘要
POisson 噪声通常发生在由光子限制的摄影系统中，如天文学和医学中的图像摄影。由于噪声分布取决于像素Intensity值，噪声水平各像素不同，因此去噪Poisson损带的图像保持重要细节是有挑战性的。在这篇论文中，我们提出了一种含有加重weighted anisotropic-isotropic总变量(AITV)的Poisson去噪模型。然后，我们开发了一种alternating direction method of multipliers，并使用距离算子进行高效实现。最后，数值实验表明，我们的算法在图像质量和计算效率方面比其他Poisson去噪方法更高效。Note: "Poisson denoising" is translated as "去噪Poisson" in Simplified Chinese, which is a common way to refer to the process of removing Poisson noise from an image.
One Copy Is All You Need: Resource-Efficient Streaming of Medical Imaging Data at Scale
results: 研究发现，使用MIST可以对医疗影像储存和流程实现大量数据储存和网络带宽的节省，并且可以维持深度学习应用的诊断质量。Abstract
Large-scale medical imaging datasets have accelerated development of artificial intelligence tools for clinical decision support. However, the large size of these datasets is a bottleneck for users with limited storage and bandwidth. Many users may not even require such large datasets as AI models are often trained on lower resolution images. If users could directly download at their desired resolution, storage and bandwidth requirements would significantly decrease. However, it is impossible to anticipate every users' requirements and impractical to store the data at multiple resolutions. What if we could store images at a single resolution but send them at different ones? We propose MIST, an open-source framework to operationalize progressive resolution for streaming medical images at multiple resolutions from a single high-resolution copy. We demonstrate that MIST can dramatically reduce imaging infrastructure inefficiencies for hosting and streaming medical images by >90%, while maintaining diagnostic quality for deep learning applications.
摘要
大规模医疗影像数据集已经推动人工智能工具的开发，以支持临床决策。然而，这些大规模数据集的尺寸成为用户储存和带宽限制的瓶颈。许多用户可能不需要这么大的数据集，因为人工智能模型通常在较低分辨率的图像上训练。如果用户可以直接下载他们所需的分辨率，储存和带宽要求就会减少很多。然而，预测每个用户的需求是不可能的，并且不可能将数据存储在多个分辨率下。我们提出了MIST框架，一个开源的框架，用于在多个分辨率下进行进程式分辨率的流动医疗影像传输。我们示示了MIST可以减少医疗影像基础设施的不效ientecies >90%，同时保持深度学习应用的诊断质量。
results: 对各种图像特征（如颜色、文本、噪声、大小）的分析，发现攻击袋 exhibit 强大的彩度相对性和颜色传输异常，并且对噪声有较强的抗性。基于这些发现，提出了进一步减少袋颜色的方法。Abstract
Adversarial attack patches have gained increasing attention due to their practical applicability in physical-world scenarios. However, the bright colors used in attack patches represent a significant drawback, as they can be easily identified by human observers. Moreover, even though these attacks have been highly successful in deceiving target networks, which specific features of the attack patch contribute to its success are still unknown. Our paper introduces a brightness-restricted patch (BrPatch) that uses optical characteristics to effectively reduce conspicuousness while preserving image independence. We also conducted an analysis of the impact of various image features (such as color, texture, noise, and size) on the effectiveness of an attack patch in physical-world deployment. Our experiments show that attack patches exhibit strong redundancy to brightness and are resistant to color transfer and noise. Based on our findings, we propose some additional methods to further reduce the conspicuousness of BrPatch. Our findings also explain the robustness of attack patches observed in physical-world scenarios.
摘要
侵略攻击贴图在实际场景中获得了越来越多的关注，但是侵略贴图中使用的鲜明颜色带来了一定的缺点，因为它们可以轻易被人类识别。尽管这些攻击已经在目标网络中取得了高度的成功，但是哪些特征使得攻击贴图成功还不清楚。我们的论文介绍了一种具有限制颜色的攻击贴图（BrPatch），使用光学特性来有效地减少鲜明程度，保持图像独立性。我们还进行了图像特征（如颜色、文本、噪声和大小）对攻击贴图的影响分析。我们的实验表明，攻击贴图具有强大的颜色减少和噪声抗耗特性。基于我们的发现，我们提出了进一步减少鲜明度的方法。我们的发现也解释了实际场景中观察到的攻击贴图 Robustness。
Applications of Binary Similarity and Distance Measures
paper_authors: Manoj Muniswamaiah, Tilak Agerwala, Charles C. Tappert
for: 本研究探讨了二进制相似度测量在不同领域的应用。
methods: 本研究使用了二进制距离度量法和相似度测量方法。
results: 研究发现，二进制相似度测量在生物特征识别、手写字符识别和复眼图像识别等领域具有广泛的应用前景。Here’s a breakdown of each point:
for: 本研究探讨了二进制相似度测量在不同领域的应用。 (The paper explores the application of binary similarity measures in various fields.)
methods: 本研究使用了二进制距离度量法和相似度测量方法。 (The study uses binary distance measures and similarity measurement methods.)
results: 研究发现，二进制相似度测量在生物特征识别、手写字符识别和复眼图像识别等领域具有广泛的应用前景。 (The study finds that binary similarity measures have broad applications in fields such as biometric identification, handwritten character recognition, and iris image recognition.)Abstract
In the recent past, binary similarity measures have been applied in solving biometric identification problems, including fingerprint, handwritten character detection, and in iris image recognition. The application of the relevant measurements has also resulted in more accurate data analysis. This paper surveys the applicability of binary similarity and distance measures in various fields.
摘要
Note:* "二进制" (er-jie) means "binary" in Chinese.* "相似度度量" (xiang-si-duo-liang) means "similarity measures" or "distance measures" in Chinese.* "生物метриiddle" (shēng-wù-métài) means "biometric identification" in Chinese.* "指纹" (zhǐ-yīn) means "fingerprint" in Chinese.* "手写字符" (shǒu-xī-zi-fu) means "handwritten character" in Chinese.* "远景图像" (yuán-jǐng-tú-xìng) means "iris image" in Chinese.
Improving CNN-based Person Re-identification using score Normalization
For: 本文提出了一种新的人识别方法，用于解决多视角和不同照明和背景的问题。* Methods: 本文使用了卷积神经网络（CNN）来提取特征，并使用了交叉视角Quadratic Discriminant Analysis（XQDA）来学习度量。* Results: 在四个复杂的数据集上进行测试，包括VIPer、GRID、CUHK01和PRID450S，并取得了承诺的结果，例如在GRID、CUHK01、VIPer和PRID450S数据集上，无 норма化情况下的rank-20率准确率分别为61.92%、83.90%、92.03%和96.22%，而经过分数 нормализа后，它们分别提高至64.64%、89.30%、92.78%和98.76%。Abstract
Person re-identification (PRe-ID) is a crucial task in security, surveillance, and retail analysis, which involves identifying an individual across multiple cameras and views. However, it is a challenging task due to changes in illumination, background, and viewpoint. Efficient feature extraction and metric learning algorithms are essential for a successful PRe-ID system. This paper proposes a novel approach for PRe-ID, which combines a Convolutional Neural Network (CNN) based feature extraction method with Cross-view Quadratic Discriminant Analysis (XQDA) for metric learning. Additionally, a matching algorithm that employs Mahalanobis distance and a score normalization process to address inconsistencies between camera scores is implemented. The proposed approach is tested on four challenging datasets, including VIPeR, GRID, CUHK01, and PRID450S, and promising results are obtained. For example, without normalization, the rank-20 rate accuracies of the GRID, CUHK01, VIPeR and PRID450S datasets were 61.92%, 83.90%, 92.03%, 96.22%; however, after score normalization, they have increased to 64.64%, 89.30%, 92.78%, and 98.76%, respectively. Accordingly, the promising results on four challenging datasets indicate the effectiveness of the proposed approach.
摘要
人员重识别（PRe-ID）是安全、监测和营销分析中的关键任务，它的目标是在多个摄像头和视图下识别个体。然而，由于照明、背景和视角的变化，PRe-IDTask 是一项具有挑战性的任务。efficient的特征提取和度量学习算法是一个成功的 PRe-ID 系统的关键。这篇论文提出了一种新的 PRe-ID 方法，该方法结合了卷积神经网络（CNN）基于特征提取方法和跨视图quadratice Discriminant Analysis（XQDA）度量学习算法。此外，一种使用 Mahalanobis 距离和分数 нормализа处理来解决相机分数不一致的匹配算法也被实现。提议的方法在四个复杂的数据集上进行测试，包括 VIPeR、GRID、CUHK01 和 PRID450S，并取得了俊czy的结果。例如，无Normalization 的 GRID、CUHK01、VIPeR 和 PRID450S 数据集的 rank-20 率准确率分别为 61.92%、83.90%、92.03% 和 96.22%；然而，通过分数 Normalization 后，它们的 rank-20 率准确率分别提高到 64.64%、89.30%、92.78% 和 98.76%。因此，在四个复杂的数据集上取得的俊czy的结果表明了提议的方法的有效性。
MobileViG: Graph-Based Sparse Attention for Mobile Vision Applications
paper_authors: Mustafa Munir, William Avery, Radu Marculescu
For: This paper proposes a new graph-based sparse attention mechanism, Sparse Vision Graph Attention (SVGA), and a hybrid CNN-GNN architecture, MobileViG, for vision tasks on mobile devices.* Methods: The proposed SVGA mechanism is designed to reduce the computational cost of representing images as graph structures, making it more suitable for mobile devices. The MobileViG architecture combines SVGA with a CNN backbone to achieve better performance and efficiency.* Results: The proposed MobileViG model achieves state-of-the-art performance and efficiency on image classification, object detection, and instance segmentation tasks on mobile devices. The fastest model, MobileViG-Ti, achieves 75.7% top-1 accuracy on ImageNet-1K with 0.78 ms inference latency, while the largest model, MobileViG-B, obtains 82.6% top-1 accuracy with only 2.30 ms latency.Abstract
Traditionally, convolutional neural networks (CNN) and vision transformers (ViT) have dominated computer vision. However, recently proposed vision graph neural networks (ViG) provide a new avenue for exploration. Unfortunately, for mobile applications, ViGs are computationally expensive due to the overhead of representing images as graph structures. In this work, we propose a new graph-based sparse attention mechanism, Sparse Vision Graph Attention (SVGA), that is designed for ViGs running on mobile devices. Additionally, we propose the first hybrid CNN-GNN architecture for vision tasks on mobile devices, MobileViG, which uses SVGA. Extensive experiments show that MobileViG beats existing ViG models and existing mobile CNN and ViT architectures in terms of accuracy and/or speed on image classification, object detection, and instance segmentation tasks. Our fastest model, MobileViG-Ti, achieves 75.7% top-1 accuracy on ImageNet-1K with 0.78 ms inference latency on iPhone 13 Mini NPU (compiled with CoreML), which is faster than MobileNetV2x1.4 (1.02 ms, 74.7% top-1) and MobileNetV2x1.0 (0.81 ms, 71.8% top-1). Our largest model, MobileViG-B obtains 82.6% top-1 accuracy with only 2.30 ms latency, which is faster and more accurate than the similarly sized EfficientFormer-L3 model (2.77 ms, 82.4%). Our work proves that well designed hybrid CNN-GNN architectures can be a new avenue of exploration for designing models that are extremely fast and accurate on mobile devices. Our code is publicly available at https://github.com/SLDGroup/MobileViG.
摘要
传统上，卷积神经网络（CNN）和视图转换器（ViT）在计算机视觉领域占据主导地位，但最近提出的视图图神经网络（ViG）提供了一个新的探索途径。然而，由于将图像表示为图结构的开销，ViG在移动设备上运行时 computationally expensive。在这种情况下，我们提出了一种新的图形基于的稀疏注意机制——图像稀疏视Graph注意（SVGA），用于ViG在移动设备上运行。此外，我们还提出了首个在移动设备上运行的hybrid CNN-GNN架构——MobileViG，它使用了SVGA。我们的实验表明，MobileViG可以在图像分类、物体检测和实例分割任务中比现有的ViG模型和移动设备上的CNN和ViT架构更高的准确率和/或速度。我们最快的模型MobileViG-Ti在ImageNet-1K上取得75.7%的顶部1准确率，并且在iPhone 13 Mini NPU上编译后的执行时间为0.78毫秒，比MobileNetV2x1.4（1.02毫秒，74.7%）和MobileNetV2x1.0（0.81毫秒，71.8%）更快。我们最大的模型MobileViG-B在82.6%的顶部1准确率下，只需2.30毫秒的执行时间，比同等大小的EfficientFormer-L3模型（2.77毫秒，82.4%）更快和更准确。我们的工作证明了，以Well designed hybrid CNN-GNN架构为基础的模型可以在移动设备上设计出非常快速和准确的模型。我们的代码公开在https://github.com/SLDGroup/MobileViG上。