William C. Headley

Daniel Rosen, Illa Rochez, Caleb McIrvin et al.

Radio Frequency Reinforcement Learning (RFRL) is anticipated to be a widely applicable technology in the next generation of wireless communication systems, particularly 6G and next-gen military communications. Given this, our research is focused on developing a tool to promote the development of RFRL techniques that leverage spectrum sensing. In particular, the tool was designed to address two cognitive radio applications, specifically dynamic spectrum access and jamming. In order to train and test reinforcement learning (RL) algorithms for these applications, a simulation environment is necessary to simulate the conditions that an agent will encounter within the Radio Frequency (RF) spectrum. In this paper, such an environment has been developed, herein referred to as the RFRL Gym. Through the RFRL Gym, users can design their own scenarios to model what an RL agent may encounter within the RF spectrum as well as experiment with different spectrum sensing techniques. Additionally, the RFRL Gym is a subclass of OpenAI gym, enabling the use of third-party ML/RL Libraries. We plan to open-source this codebase to enable other researchers to utilize the RFRL Gym to test their own scenarios and RL algorithms, ultimately leading to the advancement of RL research in the wireless communications domain. This paper describes in further detail the components of the Gym, results from example scenarios, and plans for future additions. Index Terms-machine learning, reinforcement learning, wireless communications, dynamic spectrum access, OpenAI gym

Sriniketh Vangaru, Daniel Rosen, Dylan Green et al.

Technological trends show that Radio Frequency Reinforcement Learning (RFRL) will play a prominent role in the wireless communication systems of the future. Applications of RFRL range from military communications jamming to enhancing WiFi networks. Before deploying algorithms for these purposes, they must be trained in a simulation environment to ensure adequate performance. For this reason, we previously created the RFRL Gym: a standardized, accessible tool for the development and testing of reinforcement learning (RL) algorithms in the wireless communications space. This environment leveraged the OpenAI Gym framework and featured customizable simulation scenarios within the RF spectrum. However, the RFRL Gym was limited to training a single RL agent per simulation; this is not ideal, as most real-world RF scenarios will contain multiple intelligent agents in cooperative, competitive, or mixed settings, which is a natural consequence of spectrum congestion. Therefore, through integration with Ray RLlib, multi-agent reinforcement learning (MARL) functionality for training and assessment has been added to the RFRL Gym, making it even more of a robust tool for RF spectrum simulation. This paper provides an overview of the updated RFRL Gym environment. In this work, the general framework of the tool is described relative to comparable existing resources, highlighting the significant additions and refactoring we have applied to the Gym. Afterward, results from testing various RF scenarios in the MARL environment and future additions are discussed.

Richard Lau, Lihan Yao, Todd Huster et al.

This paper describes a systematic approach towards building a new family of neural networks based on a delay-loop version of a reservoir neural network. The resulting architecture, called Scaled-Time-Attention Robust Edge (STARE) network, exploits hyper dimensional space and non-multiply-and-add computation to achieve a simpler architecture, which has shallow layers, is simple to train, and is better suited for Edge applications, such as Internet of Things (IoT), over traditional deep neural networks. STARE incorporates new AI concepts such as Attention and Context, and is best suited for temporal feature extraction and classification. We demonstrate that STARE is applicable to a variety of applications with improved performance and lower implementation complexity. In particular, we showed a novel way of applying a dual-loop configuration to detection and identification of drone vs bird in a counter Unmanned Air Systems (UAS) detection application by exploiting both spatial (video frame) and temporal (trajectory) information. We also demonstrated that the STARE performance approaches that of a State-of-the-Art deep neural network in classifying RF modulations, and outperforms Long Short-term Memory (LSTM) in a special case of Mackey Glass time series prediction. To demonstrate hardware efficiency, we designed and developed an FPGA implementation of the STARE algorithm to demonstrate its low-power and high-throughput operations. In addition, we illustrate an efficient structure for integrating a massively parallel implementation of the STARE algorithm for ASIC implementation.

Matthew DelVecchio, Bryse Flowers, William C. Headley

Recent work has shown the impact of adversarial machine learning on deep neural networks (DNNs) developed for Radio Frequency Machine Learning (RFML) applications. While these attacks have been shown to be successful in disrupting the performance of an eavesdropper, they fail to fully support the primary goal of successful intended communication. To remedy this, a communications-aware attack framework was recently developed that allows for a more effective balance between the opposing goals of evasion and intended communication through the novel use of a DNN to intelligently create the adversarial communication signal. Given the near ubiquitous usage of forward error correction (FEC) coding in the majority of deployed systems to correct errors that arise, incorporating FEC in this framework is a natural extension of this prior work and will allow for improved performance in more adverse environments. This work therefore provides contributions to the framework through improved loss functions and design considerations to incorporate inherent knowledge of the usage of FEC codes within the transmitted signal. Performance analysis shows that FEC coding improves the communications aware adversarial attack even if no explicit knowledge of the coding scheme is assumed and allows for improved performance over the prior art in balancing the opposing goals of evasion and intended communications.

Yalin E. Sagduyu, Yi Shi, Tugba Erpek et al.

Wireless systems are vulnerable to various attacks such as jamming and eavesdropping due to the shared and broadcast nature of wireless medium. To support both attack and defense strategies, machine learning (ML) provides automated means to learn from and adapt to wireless communication characteristics that are hard to capture by hand-crafted features and models. This article discusses motivation, background, and scope of research efforts that bridge ML and wireless security. Motivated by research directions surveyed in the context of ML for wireless security, ML-based attack and defense solutions and emerging adversarial ML techniques in the wireless domain are identified along with a roadmap to foster research efforts in bridging ML and wireless security.

Yi Shi, Kemal Davaslioglu, Yalin E. Sagduyu et al.

Dynamic spectrum access (DSA) benefits from detection and classification of interference sources including in-network users, out-network users, and jammers that may all coexist in a wireless network. We present a deep learning based signal (modulation) classification solution in a realistic wireless network setting, where 1) signal types may change over time; 2) some signal types may be unknown for which there is no training data; 3) signals may be spoofed such as the smart jammers replaying other signal types; and 4) different signal types may be superimposed due to the interference from concurrent transmissions. For case 1, we apply continual learning and train a Convolutional Neural Network (CNN) using an Elastic Weight Consolidation (EWC) based loss. For case 2, we detect unknown signals via outlier detection applied to the outputs of convolutional layers using Minimum Covariance Determinant (MCD) and k-means clustering methods. For case 3, we extend the CNN structure to capture phase shifts due to radio hardware effects to identify the spoofing signal sources. For case 4, we apply blind source separation using Independent Component Analysis (ICA) to separate interfering signals. We utilize the signal classification results in a distributed scheduling protocol, where in-network (secondary) users employ signal classification scores to make channel access decisions and share the spectrum with each other while avoiding interference with out-network (primary) users and jammers. Compared with benchmark TDMA-based schemes, we show that distributed scheduling constructed upon signal classification results provides major improvements to in-network user throughput and out-network user success ratio.

Bryse Flowers, R. Michael Buehrer, William C. Headley

Recent advancements in radio frequency machine learning (RFML) have demonstrated the use of raw in-phase and quadrature (IQ) samples for multiple spectrum sensing tasks. Yet, deep learning techniques have been shown, in other applications, to be vulnerable to adversarial machine learning (ML) techniques, which seek to craft small perturbations that are added to the input to cause a misclassification. The current work differentiates the threats that adversarial ML poses to RFML systems based on where the attack is executed from: direct access to classifier input, synchronously transmitted over the air (OTA), or asynchronously transmitted from a separate device. Additionally, the current work develops a methodology for evaluating adversarial success in the context of wireless communications, where the primary metric of interest is bit error rate and not human perception, as is the case in image recognition. The methodology is demonstrated using the well known Fast Gradient Sign Method to evaluate the vulnerabilities of raw IQ based Automatic Modulation Classification and concludes RFML is vulnerable to adversarial examples, even in OTA attacks. However, RFML domain specific receiver effects, which would be encountered in an OTA attack, can present significant impairments to adversarial evasion.