Tao Song

Yuwei Fan, Tao Song, Chenlong Feng et al.

The remarkable achievements of large models in the fields of natural language processing (NLP) and computer vision (CV) have sparked interest in their application to time series forecasting within industrial contexts. This paper explores the application of a pre-trained large time series model, Timer, which was initially trained on a wide range of time series data from multiple domains, in the prediction of Supervisory Control and Data Acquisition (SCADA) data collected from wind turbines. The model was fine-tuned on SCADA datasets sourced from two wind farms, which exhibited differing characteristics, and its accuracy was subsequently evaluated. Additionally, the impact of data volume was studied to evaluate the few-shot ability of the Timer. Finally, an application study on one-turbine fine-tuning for whole-plant prediction was implemented where both few-shot and cross-turbine generalization capacity is required. The results reveal that the pre-trained large model does not consistently outperform other baseline models in terms of prediction accuracy whenever the data is abundant or not, but demonstrates superior performance in the application study. This result underscores the distinctive advantages of the pre-trained large time series model in facilitating swift deployment.

Yun Dai, Tejas Dharamsi, Byron Hsu et al.

Training extremely large language models (LLMs) with billions of parameters is a computationally intensive task that pushes the limits of current data parallel training systems. While techniques like ZeRO++ have enabled efficient distributed training of such giant models on inexpensive low-bandwidth clusters, they can suffer from convergence issues due to potential race conditions in the hierarchical partitioning (hpZ) scheme employed to reduce cross-machine communication. In this work, we first show how these race conditions cause instability when training models with billions of parameters. We then propose a modification to the partitioning algorithm that addresses these convergence challenges while maintaining competitive training efficiency. Empirical evaluation on training the multi-billion parameters Falcon Models and Llama-2 models demonstrates the updated algorithm's ability to achieve reliable convergence on these massive models, where stock ZeRO++ hpZ fails to converge. The updated algorithm enables robust training of larger models with 98\% throughput and model training speed improvement without sacrificing the quality of convergence.

Xiangde Luo, Tao Song, Guotai Wang et al.

Lung nodule detection from 3D Computed Tomography scans plays a vital role in efficient lung cancer screening. Despite the SOTA performance obtained by recent anchor-based detectors using CNNs for this task, they require predetermined anchor parameters such as the size, number, and aspect ratio of anchors, and have limited robustness when dealing with lung nodules with a massive variety of sizes. To overcome these problems, we propose a 3D sphere representation-based center-points matching detection network that is anchor-free and automatically predicts the position, radius, and offset of nodules without the manual design of nodule/anchor parameters. The SCPM-Net consists of two novel components: sphere representation and center points matching. First, to match the nodule annotation in clinical practice, we replace the commonly used bounding box with our proposed bounding sphere to represent nodules with the centroid, radius, and local offset in 3D space. A compatible sphere-based intersection over-union loss function is introduced to train the lung nodule detection network stably and efficiently. Second, we empower the network anchor-free by designing a positive center-points selection and matching process, which naturally discards pre-determined anchor boxes. An online hard example mining and re-focal loss subsequently enable the CPM process to be more robust, resulting in more accurate point assignment and mitigation of class imbalance. In addition, to better capture spatial information and 3D context for the detection, we propose to fuse multi-level spatial coordinate maps with the feature extractor and combine them with 3D squeeze-and-excitation attention modules. Experimental results on the LUNA16 dataset showed that our proposed framework achieves superior performance compared with existing anchor-based and anchor-free methods for lung nodule detection.

Dao-Shun Wang, Tao Song, Lin Dong et al.

Visual cryptography scheme (VCS) is an encryption technique that utilizes human visual system in recovering secret image and it does not require any complex calculation. However, the contrast of the reconstructed image could be quite low. A number of reversing-based VCSs (or VCSs with reversing) (RVCS) have been proposed for binary secret images, allowing participants to perform a reversing operation on shares (or shadows). This reversing operation can be easily implemented by current copy machines. Some existing traditional VCS schemes without reversing (nRVCS) can be extended to RVCS with the same pixel expansion for binary image, and the RVCS can achieve ideal contrast, significantly higher than that of the corresponding nRVCS. In the application of greyscale VCS, the contrast is much lower than that of the binary cases. Therefore, it is more desirable to improve the contrast in the greyscale image reconstruction. However, when greyscale images are involved, one cannot take advantage of this reversing operation so easily. Many existing greyscale nRVCS cannot be directly extended to RVCS. In this paper, we first give a new greyscale nRVCS with minimum pixel expansion and propose an optimal-contrast greyscale RVCS (GRVCS) by using basis matrices of perfect black nRVCS. Also, we propose an optimal GRVCS even though the basis matrices are not perfect black. Finally, we design an optimal-contrast GRVCS with minimum number of shares held by each participant. The proposed schemes can satisfy different user requirement, previous RVCSs for binary images can be viewed as special cases in the schemes proposed here.