Joshua Siegel

Adam M. Terwilliger, Joshua E. Siegel

In a world increasingly dependent on road-based transportation, it is essential to understand vehicles. We introduce the AI mechanic, an acoustic vehicle characterization deep learning system, as an integrated approach using sound captured from mobile devices to enhance transparency and understanding of vehicles and their condition for non-expert users. We develop and implement novel cascading architectures for vehicle understanding, which we define as sequential, conditional, multi-level networks that process raw audio to extract highly-granular insights. To showcase the viability of cascading architectures, we build a multi-task convolutional neural network that predicts and cascades vehicle attributes to enhance fault detection. We train and test these models on a synthesized dataset reflecting more than 40 hours of augmented audio and achieve >92% validation set accuracy on attributes (fuel type, engine configuration, cylinder count and aspiration type). Our cascading architecture additionally achieved 93.6% validation and 86.8% test set accuracy on misfire fault prediction, demonstrating margins of 16.4% / 7.8% and 4.2% / 1.5% improvement over naïve and parallel baselines. We explore experimental studies focused on acoustic features, data augmentation, feature fusion, and data reliability. Finally, we conclude with a discussion of broader implications, future directions, and application areas for this work.

Siddhartha Bhattacharya, Daniel Helo, Joshua Siegel

Federated learning (FL) enables a collaborative environment for training machine learning models without sharing training data between users. This is typically achieved by aggregating model gradients on a central server. Decentralized federated learning is a rising paradigm that enables users to collaboratively train machine learning models in a peer-to-peer manner, without the need for a central aggregation server. However, before applying decentralized FL in real-world use training environments, nodes that deviate from the FL process (Byzantine nodes) must be considered when selecting an aggregation function. Recent research has focused on Byzantine-robust aggregation for client-server or fully connected networks, but has not yet evaluated such aggregation schemes for complex topologies possible with decentralized FL. Thus, the need for empirical evidence of Byzantine robustness in differing network topologies is evident. This work investigates the effects of state-of-the-art Byzantine-robust aggregation methods in complex, large-scale network structures. We find that state-of-the-art Byzantine robust aggregation strategies are not resilient within large non-fully connected networks. As such, our findings point the field towards the development of topology-aware aggregation schemes, especially necessary within the context of large scale real-world deployment.

Beñat Froemming-Aldanondo, Tatiana Rastoskueva, Michael Evans et al.

Reliable lane-following is essential for automated and assisted driving, yet existing solutions often rely on models that require extensive computational resources, limiting their deployment in compute-constrained vehicles. We evaluate five low-resource lane-following algorithms designed for real-time operation on vehicles with limited computing resources. Performance was assessed through simulation and deployment on real drive-by-wire electric vehicles, with evaluation metrics including reliability, comfort, speed, and adaptability. The top-performing methods used unsupervised learning to detect and separate lane lines with processing time under 10 ms per frame, outperforming compute-intensive and poor generalizing deep learning approaches. These approaches demonstrated robustness across lighting conditions, road textures, and lane geometries. The findings highlight the potential for efficient lane detection approaches to enhance the accessibility and reliability of autonomous vehicle technologies. Reducing computing requirements enables lane keeping to be widely deployed in vehicles as part of lower-level automation, including active safety systems.

Joshua Siegel, Gregory Falco

Cyber-Physical systems (CPS) have complex lifecycles involving multiple stakeholders, and the transparency of both hardware and software components' supply chain is opaque at best. This raises concerns for stakeholders who may not trust that what they receive is what was requested. There is an opportunity to build a cyberphysical titling process offering universal traceability and the ability to differentiate systems based on provenance. Today, RFID tags and barcodes address some of these needs, though they are easily manipulated due to non-linkage with an object or system's intrinsic characteristics. We propose cyberphysical sequencing as a low-cost, light-weight and pervasive means of adding track-and-trace capabilities to any asset that ties a system's physical identity to a unique and invariant digital identifier. CPS sequencing offers benefits similar Digital Twins' for identifying and managing the provenance and identity of an asset throughout its life with far fewer computational and other resources.

Georgios Pappas, Joshua E. Siegel, Jacob Rutkowski et al.

The present cross-disciplinary research explores pedestrian-autonomous vehicle interactions in a safe, virtual environment. We first present contemporary tools in the field and then propose the design and development of a new application that facilitates pedestrian point of view research. We conduct a three-step user experience experiment where participants answer questions before and after using the application in various scenarios. Behavioral results in virtuality, especially when there were consequences, tend to simulate real life sufficiently well to make design choices, and we received valuable insights into human/vehicle interaction. Our tool seemed to start raising participant awareness of autonomous vehicles and their capabilities and limitations, which is an important step in overcoming public distrust of AVs. Further, studying how users respect or take advantage of AVs may help inform future operating mode indicator design as well as algorithm biases that might support socially-optimal AV operation.

Joshua E. Siegel, Umberto Coda

We survey the state-of-the-art in offboard diagnostics for vehicles, their occupants, and environments, with particular focus on vibroacoustic approaches. We identify promising application areas including data-driven management for shared mobility and automated fleets, usage-based insurance, and vehicle, occupant, and environmental state and condition monitoring. We close by exploring the particular application of vibroacoustic monitoring to vehicle diagnostics and prognostics and propose the introduction of automated vehicle- and context-specific model selection as a means of improving algorithm performance, e.g. to enable smartphone-resident diagnostics. Towards this vision, four strong-performing, interdependent classifiers are presented as a proof-of-concept for identifying vehicle configuration from acoustic signatures. The described approach may serve as the first step in developing "universal diagnostics," with applicability extending beyond the automotive domain.

Piyush Gupta, Demetris Coleman, Joshua E. Siegel

Automated vehicles' neural networks suffer from overfit, poor generalizability, and untrained edge cases due to limited data availability. Researchers synthesize randomized edge-case scenarios to assist in the training process, though simulation introduces potential for overfit to latent rules and features. Automating worst-case scenario generation could yield informative data for improving self driving. To this end, we introduce a "Physically Adversarial Intelligent Network" (PAIN), wherein self-driving vehicles interact aggressively in the CARLA simulation environment. We train two agents, a protagonist and an adversary, using dueling double deep Q networks (DDDQNs) with prioritized experience replay. The coupled networks alternately seek-to-collide and to avoid collisions such that the "defensive" avoidance algorithm increases the mean-time-to-failure and distance traveled under non-hostile operating conditions. The trained protagonist becomes more resilient to environmental uncertainty and less prone to corner case failures resulting in collisions than the agent trained without an adversary.

Gregory Falco, Joshua E. Siegel

Automotive software is increasingly complex and critical to safe vehicle operation, and related embedded systems must remain up-to-date to ensure long-term system performance. Update mechanisms and data modification tools introduce opportunities for malicious actors to compromise these cyber-physical systems, and for trusted actors to mistakenly install incompatible software versions. A distributed and stratified "black box" audit trail for automotive software and data provenance is proposed to assure users, service providers, and original equipment manufacturers (OEMs) of vehicular software integrity and reliability. The proposed black box architecture is both layered and diffuse, employing distributed hash tables (DHT), a parity system and a public blockchain to provide high resilience, assurance, scalability, and efficiency for automotive and other high-assurance systems.

Joshua E. Siegel, Georgios Pappas, Konstantinos Politopoulos et al.

We identify the need for a gamified self-driving simulator where game mechanics encourage high-quality data capture, and design and apply such a simulator to collecting lane-following training data. The resulting synthetic data enables a Convolutional Neural Network (CNN) to drive an in-game vehicle. We simultaneously develop a physical test platform based on a radio-controlled vehicle and the Robotic Operating System (ROS) and successfully transfer the simulation-trained model to the physical domain without modification. The cross-platform simulator facilitates unsupervised crowdsourcing, helping to collect diverse data emulating complex, dynamic environment data, infrequent events, and edge cases. The physical platform provides a low-cost solution for validating simulation-trained models or enabling rapid transfer learning, thereby improving the safety and resilience of self-driving algorithms.

Yongbin Sun, Yue Wang, Ziwei Liu et al.

Generating 3D point clouds is challenging yet highly desired. This work presents a novel autoregressive model, PointGrow, which can generate diverse and realistic point cloud samples from scratch or conditioned on semantic contexts. This model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points, allowing inter-point correlations to be well-exploited and 3D shape generative processes to be better interpreted. Since point cloud object shapes are typically encoded by long-range dependencies, we augment our model with dedicated self-attention modules to capture such relations. Extensive evaluations show that PointGrow achieves satisfying performance on both unconditional and conditional point cloud generation tasks, with respect to realism and diversity. Several important applications, such as unsupervised feature learning and shape arithmetic operations, are also demonstrated.