X. Rong Li

Reza Rezaie, X. Rong Li

Conditionally Markov (CM) sequences are powerful mathematical tools for modeling random phenomena. There are several classes of CM sequences one of which is the reciprocal sequence. Reciprocal sequences have been widely used in many areas including image processing, intelligent systems, and acausal systems. To use them in application, we need not only their applicable dynamic models, but also some general approaches to designing parameters of dynamic models. Dynamic models governing two important classes of nonsingular Gaussian (NG) CM sequences (called $CM_L$ and $CM_F$ models), and a dynamic model governing the NG reciprocal sequence (called reciprocal $CM_L$ model) were presented in our previous work. In this paper, these models are studied in more detail and general approaches are presented for their parameter design. It is shown that every reciprocal $CM_L$ model can be induced by a Markov model and parameters of the reciprocal $CM_L$ model can be obtained from those of the Markov model. Also, it is shown how NG CM sequences can be represented in terms of an NG Markov sequence and an independent NG vector. This representation provides a general approach for parameter design of $CM_L$ and $CM_F$ models. In addition, it leads to a better understanding of CM sequences, including the reciprocal sequence.

Reza Rezaie, X. Rong Li

The conditionally Markov (CM) sequence contains different classes including Markov, reciprocal, and so-called $CM_L$ and $CM_F$ (two special classes of CM sequences). Each class has its own forward and backward dynamic models. The evolution of a CM sequence can be described by different models. For example, a Markov sequence can be described by a Markov model, as well as by reciprocal, $CM_L$, and $CM_F$ models. Also, sometimes a forward model is available, but it is desirable to have a backward model for the same sequence (e.g., in smoothing). Therefore, it is important to study relationships between different dynamic models of a CM sequence. This paper discusses such relationships between models of nonsingular Gaussian (NG) $CM_L$, $CM_F$, reciprocal, and Markov sequences. Two models are said to be explicitly sample-equivalent if not only they govern the same sequence, but also a one-one correspondence between their sample paths is made explicitly. A unified approach is presented, such that given a forward/backward $CM_L$/$CM_F$/reciprocal/Markov model, any explicitly equivalent model can be obtained. As a special case, a backward Markov model explicitly equivalent to a given forward Markov model can be obtained regardless of the singularity/nonsingularity of the state transition matrix of the model.

Reza Rezaie, X. Rong Li

The conditionally Markov (CM) sequence contains several classes, including the reciprocal sequence. Reciprocal sequences have been widely used in many areas of engineering, including image processing, acausal systems, intelligent systems, and intent inference. In this paper, the reciprocal sequence is studied from the CM sequence point of view, which is different from the viewpoint of the literature and leads to more insight into the reciprocal sequence. Based on this viewpoint, new results, properties, and easily applicable tools are obtained for the reciprocal sequence. The nonsingular Gaussian (NG) reciprocal sequence is modeled and characterized from the CM viewpoint. It is shown that an NG sequence is reciprocal if and only if it is both $CM_L$ and $CM_F$ (two special classes of CM sequences). New dynamic models are presented for the NG reciprocal sequence. These models (unlike the existing one, which is driven by colored noise) are driven by white noise and are easily applicable. As a special reciprocal sequence, the Markov sequence is also discussed. Finally, it can be seen how all CM sequences, including Markov and reciprocal, are unified.

Reza Rezaie, X. Rong Li

Markov processes are widely used in modeling random phenomena/problems. However, they may not be adequate in some cases where more general processes are needed. The conditionally Markov (CM) process is a generalization of the Markov process based on conditioning. There are several classes of CM processes (one of them is the class of reciprocal processes), which provide more capability (than Markov) for modeling random phenomena. Reciprocal processes have been used in many different applications (e.g., image processing, intent inference, intelligent systems). In this paper, nonsingular Gaussian (NG) CM sequences are studied, characterized, and their dynamic models are presented. The presented results provide effective tools for studying reciprocal sequences from the CM viewpoint, which is different from that of the literature. Also, the presented models and characterizations serve as a basis for application of CM sequences, e.g., in motion trajectory modeling with destination information.

Boyong Wu, Chao Yan, Chen Hu et al.

This paper presents Step-Audio 2, an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation. By integrating a latent audio encoder and reasoning-centric reinforcement learning (RL), Step-Audio 2 achieves promising performance in automatic speech recognition (ASR) and audio understanding. To facilitate genuine end-to-end speech conversation, Step-Audio 2 incorporates the generation of discrete audio tokens into language modeling, significantly enhancing its responsiveness to paralinguistic information such as speaking styles and emotions. To effectively leverage the rich textual and acoustic knowledge in real-world data, Step-Audio 2 integrates retrieval-augmented generation (RAG) and is able to call external tools such as web search to mitigate hallucination and audio search to switch timbres. Trained on millions of hours of speech and audio data, Step-Audio 2 delivers intelligence and expressiveness across diverse conversational scenarios. Evaluation results demonstrate that Step-Audio 2 achieves state-of-the-art performance on various audio understanding and conversational benchmarks compared to other open-source and commercial solutions. Please visit https://github.com/stepfun-ai/Step-Audio2 for more information.

Min Wu, Xiaofu Li, Haoze Wu et al.

Building on VeriX (Verified eXplainability, arXiv:2212.01051), a system for producing optimal verified explanations for machine learning models, we present VeriX+, which significantly improves both the size and the generation time of formal explanations. We introduce a bound propagation-based sensitivity technique to improve the size, and a binary search-based traversal with confidence ranking for improving time -- the two techniques are orthogonal and can be used independently or together. We also show how to adapt the QuickXplain algorithm to our setting to provide a trade-off between size and time. Experimental evaluations on standard benchmarks demonstrate significant improvements on both metrics, e.g., a size reduction of $38\%$ on the GTSRB dataset and a time reduction of $90\%$ on MNIST. We demonstrate that our approach is scalable to transformers and real-world scenarios such as autonomous aircraft taxiing and sentiment analysis. We conclude by showcasing several novel applications of formal explanations.

Zhiying Jiang, Xingyuan Li, Jinyuan Liu et al.

Image stitching seamlessly integrates images captured from varying perspectives into a single wide field-of-view image. Such integration not only broadens the captured scene but also augments holistic perception in computer vision applications. Given a pair of captured images, subtle perturbations and distortions which go unnoticed by the human visual system tend to attack the correspondence matching, impairing the performance of image stitching algorithms. In light of this challenge, this paper presents the first attempt to improve the robustness of image stitching against adversarial attacks. Specifically, we introduce a stitching-oriented attack~(SoA), tailored to amplify the alignment loss within overlapping regions, thereby targeting the feature matching procedure. To establish an attack resistant model, we delve into the robustness of stitching architecture and develop an adaptive adversarial training~(AAT) to balance attack resistance with stitching precision. In this way, we relieve the gap between the routine adversarial training and benign models, ensuring resilience without quality compromise. Comprehensive evaluation across real-world and synthetic datasets validate the deterioration of SoA on stitching performance. Furthermore, AAT emerges as a more robust solution against adversarial perturbations, delivering superior stitching results. Code is available at:https://github.com/Jzy2017/TRIS.

Zirui Wang, Jiayi Zhang, Tianwei Guan et al.

Image fusion is a fundamental and important task in computer vision, aiming to combine complementary information from different modalities to fuse images. In recent years, diffusion models have made significant developments in the field of image fusion. However, diffusion models often require complex computations and redundant inference time, which reduces the applicability of these methods. To address this issue, we propose RFfusion, an efficient one-step diffusion model for image fusion based on Rectified Flow. We incorporate Rectified Flow into the image fusion task to straighten the sampling path in the diffusion model, achieving one-step sampling without the need for additional training, while still maintaining high-quality fusion results. Furthermore, we propose a task-specific variational autoencoder (VAE) architecture tailored for image fusion, where the fusion operation is embedded within the latent space to further reduce computational complexity. To address the inherent discrepancy between conventional reconstruction-oriented VAE objectives and the requirements of image fusion, we introduce a two-stage training strategy. This approach facilitates the effective learning and integration of complementary information from multi-modal source images, thereby enabling the model to retain fine-grained structural details while significantly enhancing inference efficiency. Extensive experiments demonstrate that our method outperforms other state-of-the-art methods in terms of both inference speed and fusion quality. Code is available at https://github.com/zirui0625/RFfusion.

Yang Zou, Zhixin Chen, Zhipeng Zhang et al.

Image super-resolution (SR) is a classical yet still active low-level vision problem that aims to reconstruct high-resolution (HR) images from their low-resolution (LR) counterparts, serving as a key technique for image enhancement. Current approaches to address SR tasks, such as transformer-based and diffusion-based methods, are either dedicated to extracting RGB image features or assuming similar degradation patterns, neglecting the inherent modal disparities between infrared and visible images. When directly applied to infrared image SR tasks, these methods inevitably distort the infrared spectral distribution, compromising the machine perception in downstream tasks. In this work, we emphasize the infrared spectral distribution fidelity and propose a Contourlet refinement gate framework to restore infrared modal-specific features while preserving spectral distribution fidelity. Our approach captures high-pass subbands from multi-scale and multi-directional infrared spectral decomposition to recover infrared-degraded information through a gate architecture. The proposed Spectral Fidelity Loss regularizes the spectral frequency distribution during reconstruction, which ensures the preservation of both high- and low-frequency components and maintains the fidelity of infrared-specific features. We propose a two-stage prompt-learning optimization to guide the model in learning infrared HR characteristics from LR degradation. Extensive experiments demonstrate that our approach outperforms existing image SR models in both visual and perceptual tasks while notably enhancing machine perception in downstream tasks. Our code is available at https://github.com/hey-it-s-me/CoRPLE.

Songcheng Du, Yang Zou, Zixu Wang et al.

Hyperspectral and Multispectral Image Fusion (HMIF) aims to fuse low-resolution hyperspectral images (LR-HSIs) and high-resolution multispectral images (HR-MSIs) to reconstruct high spatial and high spectral resolution images. Current methods typically apply direct fusion from the two modalities without effective supervision, leading to an incomplete perception of deep modality-complementary information and a limited understanding of inter-modality correlations. To address these issues, we propose a simple yet effective solution for unsupervised HMIF, revealing that modality decoupling is key to improving fusion performance. Specifically, we propose an end-to-end self-supervised \textbf{Mo}dality-Decoupled \textbf{S}patial-\textbf{S}pectral Fusion (\textbf{MossFuse}) framework that decouples shared and complementary information across modalities and aggregates a concise representation of both LR-HSIs and HR-MSIs to reduce modality redundancy. Also, we introduce the subspace clustering loss as a clear guide to decouple modality-shared features from modality-complementary ones. Systematic experiments over multiple datasets demonstrate that our simple and effective approach consistently outperforms the existing HMIF methods while requiring considerably fewer parameters with reduced inference time. The anonymous source code is in \href{https://github.com/dusongcheng/MossFuse}{MossFuse}.

Xingyuan Li, Yang Zou, Jinyuan Liu et al.

With the rapid progression of deep learning technologies, multi-modality image fusion has become increasingly prevalent in object detection tasks. Despite its popularity, the inherent disparities in how different sources depict scene content make fusion a challenging problem. Current fusion methodologies identify shared characteristics between the two modalities and integrate them within this shared domain using either iterative optimization or deep learning architectures, which often neglect the intricate semantic relationships between modalities, resulting in a superficial understanding of inter-modal connections and, consequently, suboptimal fusion outcomes. To address this, we introduce a text-guided multi-modality image fusion method that leverages the high-level semantics from textual descriptions to integrate semantics from infrared and visible images. This method capitalizes on the complementary characteristics of diverse modalities, bolstering both the accuracy and robustness of object detection. The codebook is utilized to enhance a streamlined and concise depiction of the fused intra- and inter-domain dynamics, fine-tuned for optimal performance in detection tasks. We present a bilevel optimization strategy that establishes a nexus between the joint problem of fusion and detection, optimizing both processes concurrently. Furthermore, we introduce the first dataset of paired infrared and visible images accompanied by text prompts, paving the way for future research. Extensive experiments on several datasets demonstrate that our method not only produces visually superior fusion results but also achieves a higher detection mAP over existing methods, achieving state-of-the-art results.

Xingyuan Li, Jinyuan Liu, Long Ma et al.

Monocular 3D object detection plays a pivotal role in the field of autonomous driving and numerous deep learning-based methods have made significant breakthroughs in this area. Despite the advancements in detection accuracy and efficiency, these models tend to fail when faced with such attacks, rendering them ineffective. Therefore, bolstering the adversarial robustness of 3D detection models has become a crucial issue that demands immediate attention and innovative solutions. To mitigate this issue, we propose a depth-aware robust adversarial training method for monocular 3D object detection, dubbed DART3D. Specifically, we first design an adversarial attack that iteratively degrades the 2D and 3D perception capabilities of 3D object detection models(IDP), serves as the foundation for our subsequent defense mechanism. In response to this attack, we propose an uncertainty-based residual learning method for adversarial training. Our adversarial training approach capitalizes on the inherent uncertainty, enabling the model to significantly improve its robustness against adversarial attacks. We conducted extensive experiments on the KITTI 3D datasets, demonstrating that DART3D surpasses direct adversarial training (the most popular approach) under attacks in 3D object detection $AP_{R40}$ of car category for the Easy, Moderate, and Hard settings, with improvements of 4.415%, 4.112%, and 3.195%, respectively.

Xingyuan Li, Jinyuan Liu, Yixin Lei et al.

3D object detection plays a crucial role in numerous intelligent vision systems. Detection in the open world inevitably encounters various adverse scenes, such as dense fog, heavy rain, and low light conditions. Although existing efforts primarily focus on diversifying network architecture or training schemes, resulting in significant progress in 3D object detection, most of these learnable modules fail in adverse scenes, thereby hindering detection performance. To address this issue, this paper proposes a monocular 3D detection model designed to perceive twin depth in adverse scenes, termed MonoTDP, which effectively mitigates the degradation of detection performance in various harsh environments. Specifically, we first introduce an adaptive learning strategy to aid the model in handling uncontrollable weather conditions, significantly resisting degradation caused by various degrading factors. Then, to address the depth/content loss in adverse regions, we propose a novel twin depth perception module that simultaneously estimates scene and object depth, enabling the integration of scene-level features and object-level features. Additionally, we assemble a new adverse 3D object detection dataset encompassing a wide range of challenging scenes, including rainy, foggy, and low light weather conditions, with each type of scene containing 7,481 images. Experimental results demonstrate that our proposed method outperforms current state-of-the-art approaches by an average of 3.12% in terms of AP_R40 for car category across various adverse environments.

Reza Rezaie, X. Rong Li

Conditionally Markov (CM) sequences are powerful mathematical tools for modeling problems. One class of CM sequences is the reciprocal sequence. In application, we need not only CM dynamic models, but also know how to design model parameters. Models of two important classes of nonsingular Gaussian (NG) CM sequences, called $CM_L$ and $CM_F$ models, and a model of the NG reciprocal sequence, called reciprocal $CM_L$ model, were presented in our previous works and their applications were discussed. In this paper, these models are studied in more detail, in particular their parameter design. It is shown that every reciprocal $CM_L$ model can be induced by a Markov model. Then, parameters of each reciprocal $CM_L$ model can be obtained from those of the Markov model. Also, it is shown that a NG $CM_L$ ($CM_F$) sequence can be represented by a sum of a NG Markov sequence and an uncorrelated NG vector. This (necessary and sufficient) representation provides a basis for designing parameters of a $CM_L$ ($CM_F$) model. From the CM viewpoint, a representation is also obtained for NG reciprocal sequences. This representation is simple and reveals an important property of reciprocal sequences. As a result, the significance of studying reciprocal sequences from the CM viewpoint is demonstrated. A full spectrum of dynamic models from a $CM_L$ model to a reciprocal $CM_L$ model is also presented. Some examples are presented for illustration.