Yu Cheng

Qing Huang, Yishun Wu, Zhenchang Xing et al.

We conduct the first empirical study on using knowledge transfer to improve the generalization ability of large language models (LLMs) in software engineering tasks, which often require LLMs to generalize beyond their training data. Our proposed general knowledge transfer approach guides the LLM towards a similar and familiar API or code snippet it has encountered before, improving the model's generalization ability for unseen knowledge. We apply this approach to three software engineering tasks: API inference, code example generation, and FQN inference, and find transfer span, transfer strategy, and transfer architecture as key factors affecting the method. Our findings demonstrate the feasibility of knowledge transfer and its potential to enhance LLMs' performance in various software engineering tasks. The effectiveness of knowledge transfer varies depending on the target domain and task, with the hierarchical strategy being more effective than direct transfer, and AI-Chain outperforming CoT in prompt design. The implications of these findings extend beyond software engineering tasks and suggest that knowledge transfer can enhance LLMs' ability to handle unknowns in any natural language task.

Cheng Yang, Yang Sui, Jinqi Xiao et al.

The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. Compared to traditional LLMs, MoE LLMs outperform traditional LLMs by achieving higher performance with considerably fewer activated parameters. Despite this efficiency, their enormous parameter size still leads to high deployment costs. In this paper, we introduce a two-stage compression method tailored for MoE to reduce the model size and decrease the computational cost. First, in the inter-expert pruning stage, we analyze the importance of each layer and propose the Layer-wise Genetic Search and Block-wise KT-Reception Field with the non-uniform pruning ratio to prune the individual expert. Second, in the intra-expert decomposition stage, we apply the low-rank decomposition to further compress the parameters within the remaining experts. Extensive experiments on Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Mixtral-8$\times$7B demonstrate that our proposed methods can both reduce the model size and enhance inference efficiency while maintaining performance in various zero-shot tasks. The code will be available at \url{https://github.com/xiaochengsky/MoEI-2.git}

Tingchen Fu, Jiawei Gu, Yafu Li et al.

Instruction-following is essential for aligning large language models (LLMs) with user intent. While recent reasoning-oriented models exhibit impressive performance on complex mathematical problems, their ability to adhere to natural language instructions remains underexplored. In this work, we introduce MathIF, a dedicated benchmark for evaluating instruction-following in mathematical reasoning tasks. Our empirical analysis reveals a consistent tension between scaling up reasoning capacity and maintaining controllability, as models that reason more effectively often struggle to comply with user directives. We find that models tuned on distilled long chains-of-thought or trained with reasoning-oriented reinforcement learning often degrade in instruction adherence, especially when generation length increases. Furthermore, we show that even simple interventions can partially recover obedience, though at the cost of reasoning performance. These findings highlight a fundamental tension in current LLM training paradigms and motivate the need for more instruction-aware reasoning models. We release the code and data at https://github.com/TingchenFu/MathIF.

Devu Manikantan Shilay, Kin Gwn Lorey, Tianshu Weiz et al.

A significant challenge in energy system cyber security is the current inability to detect cyber-physical attacks targeting and originating from distributed grid-edge devices such as photovoltaics (PV) panels, smart flexible loads, and electric vehicles. We address this concern by designing and developing a distributed, multi-modal anomaly detection approach that can sense the health of the device and the electric power grid from the edge. This is realized by exploiting unsupervised machine learning algorithms on multiple sources of time-series data, fusing these multiple local observations and flagging anomalies when a deviation from the normal behavior is observed. We particularly focus on the cyber-physical threats to the distributed PVs that has the potential to cause local disturbances or grid instabilities by creating supply-demand mismatch, reverse power flow conditions etc. We use an open source power system simulation tool called GridLAB-D, loaded with real smart home and solar datasets to simulate the smart grid scenarios and to illustrate the impact of PV attacks on the power system. Various attacks targeting PV panels that create voltage fluctuations, reverse power flow etc were designed and performed. We observe that while individual unsupervised learning algorithms such as OCSVMs, Corrupt RF and PCA surpasses in identifying particular attack type, PCA with Convex Hull outperforms all algorithms in identifying all designed attacks with a true positive rate of 83.64% and an accuracy of 95.78%. Our key insight is that due to the heterogeneous nature of the distribution grid and the uncertainty in the type of the attack being launched, relying on single mode of information for defense can lead to increased false alarms and missed detection rates as one can design attacks to hide within those uncertainties and remain stealthy.

Zhiyang Xu, Chao Feng, Rulin Shao et al.

Despite vision-language models' (VLMs) remarkable capabilities as versatile visual assistants, two substantial challenges persist within the existing VLM frameworks: (1) lacking task diversity in pretraining and visual instruction tuning, and (2) annotation error and bias in GPT-4 synthesized instruction tuning data. Both challenges lead to issues such as poor generalizability, hallucination, and catastrophic forgetting. To address these challenges, we construct Vision-Flan, the most diverse publicly available visual instruction tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances sourced from academic datasets, and each task is accompanied by an expert-written instruction. In addition, we propose a two-stage instruction tuning framework, in which VLMs are firstly finetuned on Vision-Flan and further tuned on GPT-4 synthesized data. We find this two-stage tuning framework significantly outperforms the traditional single-stage visual instruction tuning framework and achieves the state-of-the-art performance across a wide range of multi-modal evaluation benchmarks. Finally, we conduct in-depth analyses to understand visual instruction tuning and our findings reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs' capabilities but rather modulates the model's responses to human-preferred formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can effectively align VLM responses with human-preference; (3) Visual instruction tuning mainly helps large-language models (LLMs) to understand visual features.

Jingyuan Qi, Zhiyang Xu, Rulin Shao et al.

Current vision-language models (VLMs) still exhibit inferior performance on knowledge-intensive tasks, primarily due to the challenge of accurately encoding all the associations between visual objects and scenes to their corresponding entities and background knowledge. While retrieval augmentation methods offer an efficient way to integrate external knowledge, extending them to vision-language domain presents unique challenges in (1) precisely retrieving relevant information from external sources due to the inherent discrepancy within the multimodal queries, and (2) being resilient to the irrelevant, extraneous and noisy information contained in the retrieved multimodal knowledge snippets. In this work, we introduce RORA-VLM, a novel and robust retrieval augmentation framework specifically tailored for VLMs, with two key innovations: (1) a 2-stage retrieval process with image-anchored textual-query expansion to synergistically combine the visual and textual information in the query and retrieve the most relevant multimodal knowledge snippets; and (2) a robust retrieval augmentation method that strengthens the resilience of VLMs against irrelevant information in the retrieved multimodal knowledge by injecting adversarial noises into the retrieval-augmented training process, and filters out extraneous visual information, such as unrelated entities presented in images, via a query-oriented visual token refinement strategy. We conduct extensive experiments to validate the effectiveness and robustness of our proposed methods on three widely adopted benchmark datasets. Our results demonstrate that with a minimal amount of training instance, RORA-VLM enables the base model to achieve significant performance improvement and constantly outperform state-of-the-art retrieval-augmented VLMs on all benchmarks while also exhibiting a novel zero-shot domain transfer capability.

Xingwu Sun, Shuaipeng Li, Ruobing Xie et al.

Low-precision training is considered an effective strategy for reducing both training and downstream inference costs. Previous scaling laws for precision mainly focus on integer quantization, which pay less attention to the constituents in floating-point (FP) quantization, and thus cannot well fit the LLM losses in this scenario. In contrast, while FP quantization training is more commonly implemented in production, it's research has been relatively superficial. In this paper, we thoroughly explore the effects of FP quantization targets, exponent bits, mantissa bits, and the calculation granularity of the scaling factor in FP quantization training performance of LLM models. In addition to an accurate FP quantization unified scaling law, we also provide valuable suggestions for the community: (1) Exponent bits contribute slightly more to the model performance than mantissa bits. We provide the optimal exponent-mantissa bit ratio for different bit numbers, which is available for future reference by hardware manufacturers; (2) We discover the formation of the critical data size in low-precision LLM training. Too much training data exceeding the critical data size will inversely bring in degradation of LLM performance; (3) The optimal FP quantization precision is directly proportional to the computational power, but within a wide computational power range. We estimate that the best cost-performance precision should lie between 4-8 bits.

Oluwaseun T. Ajayi, Yu Cheng

With the prevailing efforts to combat the coronavirus disease 2019 (COVID-19) pandemic, there are still uncertainties that are yet to be discovered about its spread, future impact, and resurgence. In this paper, we present a three-stage data-driven approach to distill the hidden information about COVID-19. The first stage employs a Bayesian network structure learning method to identify the causal relationships among COVID-19 symptoms and their intrinsic demographic variables. As a second stage, the output from the Bayesian network structure learning, serves as a useful guide to train an unsupervised machine learning (ML) algorithm that uncovers the similarities in patients' symptoms through clustering. The final stage then leverages the labels obtained from clustering to train a demographic symptom identification (DSID) model which predicts a patient's symptom class and the corresponding demographic probability distribution. We applied our method on the COVID-19 dataset obtained from the Centers for Disease Control and Prevention (CDC) in the United States. Results from the experiments show a testing accuracy of 99.99%, as against the 41.15% accuracy of a heuristic ML method. This strongly reveals the viability of our Bayesian network and ML approach in understanding the relationship between the virus symptoms, and providing insights on patients' stratification towards reducing the severity of the virus.

Wenlong Shen, Yu Cheng, Bo Yin et al.

Key establishment is one fundamental issue in wireless security. The widely used Diffie-Hellman key exchange is vulnerable to the man-in-the-middle attack. This paper presents a novel in-band solution for defending the man-in-the-middle attack during the key establishment process for wireless devices. Our solution is based on the insight that an attacker inevitably affects the link layer behavior of the wireless channel, and this behavior change introduced by the attacker can be detected by the legitimate users. Specifically, we propose a key exchange protocol and its corresponding channel access mechanism for the protocol message transmission, in which the Diffie-Hellman parameter is transmitted multiple times in a row without being interrupted by other data transmission on the same wireless channel. The proposed key exchange protocol forces the MITM attacker to cause multiple packet collisions consecutively at the receiver side, which can then be monitored by the proposed detection algorithm. The performance of the proposed solution is validated through both theoretical analysis and simulation: the proposed solution is secure against the MITM attack and can achieve an arbitrarily low false positive ratio. This proposed link layer solution works completely in-band, and can be easily implemented on off-the-shelf wireless devices without the requirement of any special hardware.

Wenlong Shen, Bo Yin, Xianghui Cao et al.

The emerging ad hoc clouds form a new cloud computing paradigm by leveraging untapped local computation and storage resources. An important application application over ad hoc clouds is outsourcing computationally intensive problems to nearby cloud agents to solve in a distributed manner.

Devu Manikantan Shila, Xianghui Cao, Yu Cheng et al.

ZigBee has been recently drawing a lot of attention as a promising solution for ubiquitous computing. The ZigBee devices are normally resource-limited, making the network susceptible to a variety of security threats. This paper presents a severe attack on ZigBee networks termed as ghost, which leverages the underlying vulnerabilities of the IEEE 802.15.4 security suites to deplete the energy of the devices. We manifest that the impact of ghost is severe as it can reduce the lifetime of devices from years to days and facilitate a variety of threats including denial of service and replay attacks. We highlight that merely deploying a standard suite of advanced security techniques does not necessarily guarantee improved security, but instead might be leveraged by adversaries to cause severe disruption in the network. We propose several recommendations on how to localize and withstand the ghost and other related attacks in ZigBee networks. Extensive simulations are provided to show the impact of the ghost and the performance of the proposed recommendations. Moreover, physical experiments also have been conducted and the observations confirm the severity of the impact by the ghost attack. We believe that the presented work will aid the researchers to improve the security of ZigBee further.