Joshua E. Siegel

Adam M. Terwilliger, Joshua E. Siegel

In a world increasingly dependent on road-based transportation, it is essential to understand vehicles. We introduce the AI mechanic, an acoustic vehicle characterization deep learning system, as an integrated approach using sound captured from mobile devices to enhance transparency and understanding of vehicles and their condition for non-expert users. We develop and implement novel cascading architectures for vehicle understanding, which we define as sequential, conditional, multi-level networks that process raw audio to extract highly-granular insights. To showcase the viability of cascading architectures, we build a multi-task convolutional neural network that predicts and cascades vehicle attributes to enhance fault detection. We train and test these models on a synthesized dataset reflecting more than 40 hours of augmented audio and achieve >92% validation set accuracy on attributes (fuel type, engine configuration, cylinder count and aspiration type). Our cascading architecture additionally achieved 93.6% validation and 86.8% test set accuracy on misfire fault prediction, demonstrating margins of 16.4% / 7.8% and 4.2% / 1.5% improvement over naïve and parallel baselines. We explore experimental studies focused on acoustic features, data augmentation, feature fusion, and data reliability. Finally, we conclude with a discussion of broader implications, future directions, and application areas for this work.

Georgios Pappas, Joshua E. Siegel, Jacob Rutkowski et al.

The present cross-disciplinary research explores pedestrian-autonomous vehicle interactions in a safe, virtual environment. We first present contemporary tools in the field and then propose the design and development of a new application that facilitates pedestrian point of view research. We conduct a three-step user experience experiment where participants answer questions before and after using the application in various scenarios. Behavioral results in virtuality, especially when there were consequences, tend to simulate real life sufficiently well to make design choices, and we received valuable insights into human/vehicle interaction. Our tool seemed to start raising participant awareness of autonomous vehicles and their capabilities and limitations, which is an important step in overcoming public distrust of AVs. Further, studying how users respect or take advantage of AVs may help inform future operating mode indicator design as well as algorithm biases that might support socially-optimal AV operation.

Joshua E. Siegel, Umberto Coda

We survey the state-of-the-art in offboard diagnostics for vehicles, their occupants, and environments, with particular focus on vibroacoustic approaches. We identify promising application areas including data-driven management for shared mobility and automated fleets, usage-based insurance, and vehicle, occupant, and environmental state and condition monitoring. We close by exploring the particular application of vibroacoustic monitoring to vehicle diagnostics and prognostics and propose the introduction of automated vehicle- and context-specific model selection as a means of improving algorithm performance, e.g. to enable smartphone-resident diagnostics. Towards this vision, four strong-performing, interdependent classifiers are presented as a proof-of-concept for identifying vehicle configuration from acoustic signatures. The described approach may serve as the first step in developing "universal diagnostics," with applicability extending beyond the automotive domain.

Piyush Gupta, Demetris Coleman, Joshua E. Siegel

Automated vehicles' neural networks suffer from overfit, poor generalizability, and untrained edge cases due to limited data availability. Researchers synthesize randomized edge-case scenarios to assist in the training process, though simulation introduces potential for overfit to latent rules and features. Automating worst-case scenario generation could yield informative data for improving self driving. To this end, we introduce a "Physically Adversarial Intelligent Network" (PAIN), wherein self-driving vehicles interact aggressively in the CARLA simulation environment. We train two agents, a protagonist and an adversary, using dueling double deep Q networks (DDDQNs) with prioritized experience replay. The coupled networks alternately seek-to-collide and to avoid collisions such that the "defensive" avoidance algorithm increases the mean-time-to-failure and distance traveled under non-hostile operating conditions. The trained protagonist becomes more resilient to environmental uncertainty and less prone to corner case failures resulting in collisions than the agent trained without an adversary.

Gregory Falco, Joshua E. Siegel

Automotive software is increasingly complex and critical to safe vehicle operation, and related embedded systems must remain up-to-date to ensure long-term system performance. Update mechanisms and data modification tools introduce opportunities for malicious actors to compromise these cyber-physical systems, and for trusted actors to mistakenly install incompatible software versions. A distributed and stratified "black box" audit trail for automotive software and data provenance is proposed to assure users, service providers, and original equipment manufacturers (OEMs) of vehicular software integrity and reliability. The proposed black box architecture is both layered and diffuse, employing distributed hash tables (DHT), a parity system and a public blockchain to provide high resilience, assurance, scalability, and efficiency for automotive and other high-assurance systems.

Joshua E. Siegel, Georgios Pappas, Konstantinos Politopoulos et al.

We identify the need for a gamified self-driving simulator where game mechanics encourage high-quality data capture, and design and apply such a simulator to collecting lane-following training data. The resulting synthetic data enables a Convolutional Neural Network (CNN) to drive an in-game vehicle. We simultaneously develop a physical test platform based on a radio-controlled vehicle and the Robotic Operating System (ROS) and successfully transfer the simulation-trained model to the physical domain without modification. The cross-platform simulator facilitates unsupervised crowdsourcing, helping to collect diverse data emulating complex, dynamic environment data, infrequent events, and edge cases. The physical platform provides a low-cost solution for validating simulation-trained models or enabling rapid transfer learning, thereby improving the safety and resilience of self-driving algorithms.

Joshua E. Siegel, Maria F. Beemer, Steven M. Shepard

Manufacturers struggle to produce low-cost, robust and complex components at manufacturing lot-size one. Additive processes like Fused Filament Fabrication (FFF) inexpensively produce complex geometries, but defects limit viability in critical applications. We present an approach to high-accuracy, high-throughput and low-cost automated non-destructive testing (NDT) for FFF interlayer delamination using Flash Thermography (FT) data processed with Thermographic Signal Reconstruction (TSR) and Artificial Intelligence (AI). A Deep Neural Network (DNN) attains 95.4% per-pixel accuracy when differentiating four delamination thicknesses 5mm subsurface in PolyLactic Acid (PLA) widgets, and 98.6% accuracy in differentiating acceptable from unacceptable condition for the same components. Automated inspection enables time- and cost-efficient 100% inspection for delamination defects, supporting FFF's use in critical and small-batch applications.

Yongbin Sun, Yue Wang, Ziwei Liu et al.

Generating 3D point clouds is challenging yet highly desired. This work presents a novel autoregressive model, PointGrow, which can generate diverse and realistic point cloud samples from scratch or conditioned on semantic contexts. This model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points, allowing inter-point correlations to be well-exploited and 3D shape generative processes to be better interpreted. Since point cloud object shapes are typically encoded by long-range dependencies, we augment our model with dedicated self-attention modules to capture such relations. Extensive evaluations show that PointGrow achieves satisfying performance on both unconditional and conditional point cloud generation tasks, with respect to realism and diversity. Several important applications, such as unsupervised feature learning and shape arithmetic operations, are also demonstrated.