Azim Eskandarian

Ce Zhang, Azim Eskandarian

Reliable object detection using cameras plays a crucial role in enabling autonomous vehicles to perceive their surroundings. However, existing camera-based object detection approaches for autonomous driving lack the ability to provide comprehensive feedback on detection performance for individual frames. To address this limitation, we propose a novel evaluation metric, named as the detection quality index (DQI), which assesses the performance of camera-based object detection algorithms and provides frame-by-frame feedback on detection quality. The DQI is generated by combining the intensity of the fine-grained saliency map with the output results of the object detection algorithm. Additionally, we have developed a superpixel-based attention network (SPA-NET) that utilizes raw image pixels and superpixels as input to predict the proposed DQI evaluation metric. To validate our approach, we conducted experiments on three open-source datasets. The results demonstrate that the proposed evaluation metric accurately assesses the detection quality of camera-based systems in autonomous driving environments. Furthermore, the proposed SPA-NET outperforms other popular image-based quality regression models. This highlights the effectiveness of the DQI in evaluating a camera's ability to perceive visual scenes. Overall, our work introduces a valuable self-evaluation tool for camera-based object detection in autonomous vehicles.

Anshul Nayak, Azim Eskandarian, Zachary Doerzaph

Past research on pedestrian trajectory forecasting mainly focused on deterministic predictions which provide only point estimates of future states. These future estimates can help an autonomous vehicle plan its trajectory and avoid collision. However, under dynamic traffic scenarios, planning based on deterministic predictions is not trustworthy. Rather, estimating the uncertainty associated with the predicted states with a certain level of confidence can lead to robust path planning. Hence, the authors propose to quantify this uncertainty during forecasting using stochastic approximation which deterministic approaches fail to capture. The current method is simple and applies Bayesian approximation during inference to standard neural network architectures for estimating uncertainty. The authors compared the predictions between the probabilistic neural network (NN) models with the standard deterministic models. The results indicate that the mean predicted path of probabilistic models was closer to the ground truth when compared with the deterministic prediction. Further, the effect of stochastic dropout of weights and long-term prediction on future state uncertainty has been studied. It was found that the probabilistic models produced better performance metrics like average displacement error (ADE) and final displacement error (FDE). Finally, the study has been extended to multiple datasets providing a comprehensive comparison for each model.

Ning Ding, Ce Zhang, Azim Eskandarian

A lack of driver's vigilance is the main cause of most vehicle crashes. Electroencephalography(EEG) has been reliable and efficient tool for drivers' drowsiness estimation. Even though previous studies have developed accurate and robust driver's vigilance detection algorithms, these methods are still facing challenges on following areas: (a) small sample size training, (b) anomaly signal detection, and (c) subject-independent classification. In this paper, we propose a generalized few-shot model, namely EEG-Fest, to improve aforementioned drawbacks. The EEG-Fest model can (a) classify the query sample's drowsiness with a few samples, (b) identify whether a query sample is anomaly signals or not, and (c) achieve subject independent classification. The proposed algorithm achieves state-of-the-art results on the SEED-VIG dataset and the SADT dataset. The accuracy of the drowsy class achieves 92% and 94% for 1-shot and 5-shot support samples in the SEED-VIG dataset, and 62% and 78% for 1-shot and 5-shot support samples in the SADT dataset.

Shathushan Sivashangaran, Vihaan Dutta, Apoorva Khairnar et al.

Road traffic accidents are a leading cause of fatalities worldwide. In the US, human error causes 94% of crashes, resulting in excess of 7,000 pedestrian fatalities and $500 billion in costs annually. Autonomous Vehicles (AVs) with emergency collision avoidance systems that operate at the limits of vehicle dynamics at a high frequency, a dual constraint of nonlinear kinodynamic accuracy and computational efficiency, further enhance safety benefits during adverse weather and cybersecurity breaches, and to evade dangerous human driving when AVs and human drivers share roads. This paper parameterizes a Deep Reinforcement Learning (DRL) collision avoidance policy Out-Of-Distribution (OOD) utilizing race car overtaking, without explicit geometric mimicry reference trajectory guidance, in simulation, with a physics-informed, simulator exploit-aware reward to encode nonlinear vehicle kinodynamics. Two policies are evaluated, a default uni-direction and a reversed heading variant that navigates in the opposite direction to other cars, which both consistently outperform a Model Predictive Control and Artificial Potential Function (MPC-APF) baseline, with zero-shot transfer to proportionally scaled hardware, across three intersection collision scenarios, at 31x fewer Floating Point Operations (FLOPS) and 64x lower inference latency. The reversed heading policy outperforms the default racing overtaking policy in head-to-head collisions by 30% and the baseline by 50%, and matches the former in side collisions, where both DRL policies evade 10% greater than numerical optimal control.

Shathushan Sivashangaran, Apoorva Khairnar, Azim Eskandarian

Autonomous Mobile Robot (AMR) navigation in dynamic environments that may be GPS denied, without a-priori maps, is an unsolved problem with potential to improve humanity's capabilities. Conventional modular methods are computationally inefficient, and require explicit feature extraction and engineering that inhibit generalization and deployment at scale. We present an Out-of-Distribution (OOD) Deep Reinforcement Learning (DRL) approach that includes functionality in unstructured terrain and dynamic obstacle avoidance capabilities. We leverage accelerated simulation training in a racetrack with a transition probability to parameterize spatial reasoning with intrinsic exploratory behavior, in a compact, computationally efficient Artificial Neural Network (ANN), which we transfer zero-shot with a reward component to mitigate differences between simulation and real world physics. Our approach enables utility without a separate high-level planner or real-time cartography and utilizes a fraction of the computation resources of modular methods, enabling execution in a range of AMRs with different embedded computer payloads.

Ning Ding, Azim Eskandarian

Object detection is a crucial component of autonomous driving, and many detection applications have been developed to address this task. These applications often rely on backbone architectures, which extract representation features from inputs to perform the object detection task. The quality of the features extracted by the backbone architecture can have a significant impact on the overall detection performance. Many researchers have focused on developing new and improved backbone architectures to enhance the efficiency and accuracy of object detection applications. While these backbone architectures have shown state-of-the-art performance on generic object detection datasets like MS-COCO and PASCAL-VOC, evaluating their performance under an autonomous driving environment has not been previously explored. To address this, our study evaluates three well-known autonomous vehicle datasets, namely KITTI, NuScenes, and BDD, to compare the performance of different backbone architectures on object detection tasks.

Ning Ding, Ce Zhang, Azim Eskandarian

Object detection (OD) is crucial to autonomous driving. On the other hand, unknown objects, which have not been seen in training sample set, are one of the reasons that hinder autonomous vehicles from driving beyond the operational domain. To addresss this issue, we propose a saliency-based OD algorithm (SalienDet) to detect unknown objects. Our SalienDet utilizes a saliency-based algorithm to enhance image features for object proposal generation. Moreover, we design a dataset relabeling approach to differentiate the unknown objects from all objects in training sample set to achieve Open-World Detection. To validate the performance of SalienDet, we evaluate SalienDet on KITTI, nuScenes, and BDD datasets, and the result indicates that it outperforms existing algorithms for unknown object detection. Notably, SalienDet can be easily adapted for incremental learning in open-world detection tasks. The project page is \url{https://github.com/dingmike001/SalienDet-Open-Detection.git}.

Shathushan Sivashangaran, Apoorva Khairnar, Sepideh Gohari et al.

Autonomous racing without prebuilt maps is a grand challenge for embedded robotics that requires kinodynamic planning from instantaneous sensor data at the acceleration and tire friction limits. Out-Of-Distribution (OOD) generalization to various racetrack configurations utilizes Machine Learning (ML) to encode the mathematical relation between sensor data and vehicle actuation for end-to-end control, with implicit localization. These comprise Behavioral Cloning (BC) that is capped to human reaction times and Deep Reinforcement Learning (DRL) which requires large-scale collisions for comprehensive training that can be infeasible without simulation but is arduous to transfer to reality, thus exhibiting greater performance than BC in simulation, but actuation instability on hardware. This paper presents a DRL method that parameterizes nonlinear vehicle dynamics from the spectral distribution of depth measurements with a non-geometric, physics-informed reward, to infer vehicle time-optimal and overtaking racing controls with an Artificial Neural Network (ANN) that utilizes less than 1% of the computation of BC and model-based DRL. Slaloming from simulation to reality transfer and variance-induced conservatism are eliminated with the combination of a physics engine exploit-aware reward and the replacement of an explicit collision penalty with an implicit truncation of the value horizon. The policy outperforms human demonstrations by 12% in OOD tracks on proportionally scaled hardware, by maximizing the friction circle with tire dynamics that resemble an empirical Pacejka tire model. System identification illuminates a functional bifurcation where the first layer compresses spatial observations to extract digitized track features with higher resolution in corner apexes, and the second encodes nonlinear dynamics.

Goodarz Mehr, Azim Eskandarian

Bird's-eye view (BEV) perception has garnered significant attention in autonomous driving in recent years, in part because BEV representation facilitates multi-modal sensor fusion. BEV representation enables a variety of perception tasks including BEV segmentation, a concise view of the environment useful for planning a vehicle's trajectory. However, this representation is not fully supported by existing datasets, and creation of new datasets for this purpose can be a time-consuming endeavor. To address this challenge, we introduce SimBEV. SimBEV is a randomized synthetic data generation tool that is extensively configurable and scalable, supports a wide array of sensors, incorporates information from multiple sources to capture accurate BEV ground truth, and enables a variety of perception tasks including BEV segmentation and 3D object detection. SimBEV is used to create the SimBEV dataset, a large collection of annotated perception data from diverse driving scenarios. SimBEV and the SimBEV dataset are open and available to the public.

Ce Zhang, Azim Eskandarian

LiDAR is one of the most crucial sensors for autonomous vehicle perception. However, current LiDAR-based point cloud perception algorithms lack comprehensive and rigorous LiDAR quality assessment methods, leading to uncertainty in detection performance. Additionally, existing point cloud quality assessment algorithms are predominantly designed for indoor environments or single-object scenarios. In this paper, we introduce a novel image-guided point cloud quality assessment algorithm for outdoor autonomous driving environments, named the Image-Guided Outdoor Point Cloud Quality Assessment (IGO-PQA) algorithm. Our proposed algorithm comprises two main components. The first component is the IGO-PQA generation algorithm, which leverages point cloud data, corresponding RGB surrounding view images, and agent objects' ground truth annotations to generate an overall quality score for a single-frame LiDAR-based point cloud. The second component is a transformer-based IGO-PQA regression algorithm for no-reference outdoor point cloud quality assessment. This regression algorithm allows for the direct prediction of IGO-PQA scores in an online manner, without requiring image data and object ground truth annotations. We evaluate our proposed algorithm using the nuScenes and Waymo open datasets. The IGO-PQA generation algorithm provides consistent and reasonable perception quality indices. Furthermore, our proposed IGO-PQA regression algorithm achieves a Pearson Linear Correlation Coefficient (PLCC) of 0.86 on the nuScenes dataset and 0.97 on the Waymo dataset.

Anshul Nayak, Azim Eskandarian

Perception and planning under occlusion is essential for safety-critical tasks. Occlusion-aware planning often requires communicating the information of the occluded object to the ego agent for safe navigation. However, communicating rich sensor information under adverse conditions during communication loss and limited bandwidth may not be always feasible. Further, in GPS denied environments and indoor navigation, localizing and sharing of occluded objects can be challenging. To overcome this, relative pose estimation between connected agents sharing a common field of view can be a computationally effective way of communicating information about surrounding objects. In this paper, we design an end-to-end network that cooperatively estimates the current states of occluded pedestrian in the reference frame of ego agent and then predicts the trajectory with safety guarantees. Experimentally, we show that the uncertainty-aware trajectory prediction of occluded pedestrian by the ego agent is almost similar to the ground truth trajectory assuming no occlusion. The current research holds promise for uncertainty-aware navigation among multiple connected agents under occlusion.

Anshul Nayak, Azim Eskandarian, Zachary Doerzaph et al.

One of the fundamental challenges in the prediction of dynamic agents is robustness. Usually, most predictions are deterministic estimates of future states which are over-confident and prone to error. Recently, few works have addressed capturing uncertainty during forecasting of future states. However, these probabilistic estimation methods fail to account for the upstream noise in perception data during tracking. Sensors always have noise and state estimation becomes even more difficult under adverse weather conditions and occlusion. Traditionally, Bayes filters have been used to fuse information from noisy sensors to update states with associated belief. But, they fail to address non-linearities and long-term predictions. Therefore, we propose an end-to-end estimator that can take noisy sensor measurements and make robust future state predictions with uncertainty bounds while simultaneously taking into consideration the upstream perceptual uncertainty. For the current research, we consider an encoder-decoder based deep ensemble network for capturing both perception and predictive uncertainty simultaneously. We compared the current model to other approximate Bayesian inference methods. Overall, deep ensembles provided more robust predictions and the consideration of upstream uncertainty further increased the estimation accuracy for the model.

Ce Zhang, Azim Eskandarian, Xuelai Du

Environment perception is crucial for autonomous vehicle (AV) safety. Most existing AV perception algorithms have not studied the surrounding environment complexity and failed to include the environment complexity parameter. This paper proposes a novel attention-based neural network model to predict the complexity level of the surrounding driving environment. The proposed model takes naturalistic driving videos and corresponding vehicle dynamics parameters as input. It consists of a Yolo-v3 object detection algorithm, a heat map generation algorithm, CNN-based feature extractors, and attention-based feature extractors for both video and time-series vehicle dynamics data inputs to extract features. The output from the proposed algorithm is a surrounding environment complexity parameter. The Berkeley DeepDrive dataset (BDD Dataset) and subjectively labeled surrounding environment complexity levels are used for model training and validation to evaluate the algorithm. The proposed attention-based network achieves 91.22% average classification accuracy to classify the surrounding environment complexity. It proves that the environment complexity level can be accurately predicted and applied for future AVs' environment perception studies.

Ce Zhang, Young-Keun Kim, Azim Eskandarian

Classification of EEG-based motor imagery (MI) is a crucial non-invasive application in brain-computer interface (BCI) research. This paper proposes a novel convolutional neural network (CNN) architecture for accurate and robust EEG-based MI classification that outperforms the state-of-the-art methods. The proposed CNN model, namely EEG-Inception, is built on the backbone of the Inception-Time network, which showed to be highly efficient and accurate for time-series classification. Also, the proposed network is an end-to-end classification, as it takes the raw EEG signals as the input and does not require complex EEG signal-preprocessing. Furthermore, this paper proposes a novel data augmentation method for EEG signals to enhance the accuracy, at least by 3%, and reduce overfitting with limited BCI datasets. The proposed model outperforms all the state-of-the-art methods by achieving the average accuracy of 88.4% and 88.6% on the 2008 BCI Competition IV 2a (four-classes) and 2b datasets (binary-classes), respectively. Furthermore, it takes less than 0.025 seconds to test a sample suitable for real-time processing. Moreover, the classification standard deviation for nine different subjects achieves the lowest value of 5.5 for the 2b dataset and 7.1 for the 2a dataset, which validates that the proposed method is highly robust. From the experiment results, it can be inferred that the EEG-Inception network exhibits a strong potential as a subject-independent classifier for EEG-based MI tasks.

Goodarz Mehr, Azim Eskandarian

This paper introduces Sentinel, an onboard system for intelligent vehicles that guides their lane changing behavior during a freeway incident with the goal of reducing traffic congestion, capacity drop, and delay. When an incident blocking the lanes ahead is detected, Sentinel calculates the probability of leaving the blocked lane(s) before reaching the incident point at each time step. It advises the vehicle to leave the blocked lane(s) when that probability drops below a certain threshold, as the vehicle nears the congestion boundary. By doing this, Sentinel reduces the number of late-stage lane changes of vehicles in the blocked lane(s) trying to move to other lanes, and distributes those maneuvers upstream of the incident point. A simulation case study is conducted in which one lane of a four-lane section of the I-66 interstate highway in the U.S. is temporarily blocked due to an incident, to understand how Sentinel impacts traffic flow and how different parameters - traffic flow, system penetration rate, and incident duration - affect Sentinel's performance. The results show that Sentinel has a positive impact on traffic flow, reducing average delay by up to 37%, particularly when it has a considerable penetration rate. Working alongside Traffic Incident Management Systems (TIMS), Sentinel can be a valuable asset for reducing traffic delay and potentially saving billions of dollars annually in costs associated with congestion caused by freeway incidents.

Ce Zhang, Azim Eskandarian

Common spatial pattern (CSP) is a popular feature extraction method for electroencephalogram (EEG) motor imagery (MI). This study modifies the conventional CSP algorithm to improve the multi-class MI classification accuracy and ensure the computation process is efficient. The EEG MI data is gathered from the Brain-Computer Interface (BCI) Competition IV. At first, a bandpass filter and a time-frequency analysis are performed for each experiment trial. Then, the optimal EEG signals for every experiment trials are selected based on the signal energy for CSP feature extraction. In the end, the extracted features are classified by three classifiers, linear discriminant analysis (LDA), naïve Bayes (NVB), and support vector machine (SVM), in parallel for classification accuracy comparison. The experiment results show the proposed algorithm average computation time is 37.22% less than the FBCSP (1st winner in the BCI Competition IV) and 4.98% longer than the conventional CSP method. For the classification rate, the proposed algorithm kappa value achieved 2nd highest compared with the top 3 winners in BCI Competition IV.

Ce Zhang, Azim Eskandarian

Drivers cognitive and physiological states affect their ability to control their vehicles. Thus, these driver states are important to the safety of automobiles. The design of advanced driver assistance systems (ADAS) or autonomous vehicles will depend on their ability to interact effectively with the driver. A deeper understanding of the driver state is, therefore, paramount. EEG is proven to be one of the most effective methods for driver state monitoring and human error detection. This paper discusses EEG-based driver state detection systems and their corresponding analysis algorithms over the last three decades. First, the commonly used EEG system setup for driver state studies is introduced. Then, the EEG signal preprocessing, feature extraction, and classification algorithms for driver state detection are reviewed. Finally, EEG-based driver state monitoring research is reviewed in-depth, and its future development is discussed. It is concluded that the current EEG-based driver state monitoring algorithms are promising for safety applications. However, many improvements are still required in EEG artifact reduction, real-time processing, and between-subject classification accuracy.

Seontake Oh, Ji-Hwan You, Azim Eskandarian et al.

A misalignment of LiDAR as low as a few degrees could cause a significant error in obstacle detection and mapping that could cause safety and quality issues. In this paper, an accurate inspection system is proposed for estimating a LiDAR alignment error after sensor attachment on a mobility system such as a vehicle or robot. The proposed method uses only a single target board at the fixed position to estimate the three orientations (roll, tilt, and yaw) and the horizontal position of the LiDAR attachment with sub-degree and millimeter level accuracy. After the proposed preprocessing steps, the feature beam points that are the closest to each target corner are extracted and used to calculate the sensor attachment pose with respect to the target board frame using a nonlinear optimization method and with a low computational cost. The performance of the proposed method is evaluated using a test bench that can control the reference yaw and horizontal translation of LiDAR within ranges of 3 degrees and 30 millimeters, respectively. The experimental results for a low-resolution 16 channel LiDAR (Velodyne VLP-16) confirmed that misalignment could be estimated with accuracy within 0.2 degrees and 4 mm. The high accuracy and simplicity of the proposed system make it practical for large-scale industrial applications such as automobile or robot manufacturing process that inspects the sensor attachment for the safety quality control.

Ji-Hwan You, Seon Taek Oh, Jae-Eun Park et al.

This paper presents a novel automatic calibration system to estimate the extrinsic parameters of LiDAR mounted on a mobile platform for sensor misalignment inspection in the large-scale production of highly automated vehicles. To obtain subdegree and subcentimeter accuracy levels of extrinsic calibration, this study proposed a new concept of a target board with embedded photodetector arrays, named the PD-target system, to find the precise position of the correspondence laser beams on the target surface. Furthermore, the proposed system requires only the simple design of the target board at the fixed pose in a close range to be readily applicable in the automobile manufacturing environment. The experimental evaluation of the proposed system on low-resolution LiDAR showed that the LiDAR offset pose can be estimated within 0.1 degree and 3 mm levels of precision. The high accuracy and simplicity of the proposed calibration system make it practical for large-scale applications for the reliability and safety of autonomous systems.

Goodarz Mehr, Azim Eskandarian

This paper proposes a model to estimate the probability of a vehicle reaching a near-term goal state using one or multiple lane changes based on parameters corresponding to traffic conditions and driving behavior. The proposed model not only has broad application in path planning and autonomous vehicle navigation, it can also be incorporated in advance warning systems to reduce traffic delay during recurrent and non-recurrent congestion. The model is first formulated for a two-lane road segment through systemic reduction of the number of parameters and transforming the problem into an abstract statistical form, for which the probability can be calculated numerically. It is then extended to cases with a higher number of lanes using the law of total probability. VISSIM simulations are used to validate the predictions of the model and study the effect of different parameters on the probability. For most cases, simulation results are within 4% of model predictions, and the effect of different parameters such as driving behavior and traffic density on the probability match our expectation. The model can be implemented with near real-time performance, with computation time increasing linearly with the number of lanes.

Yuan Lin, Azim Eskandarian

Adaptive Cruise Control (ACC) is an Advanced Driver Assistance System (ADAS) that enables vehicle following with desired inter-vehicular distances. Cooperative Adaptive Cruise Control (CACC) is upgraded ACC that utilizes additional inter-vehicular wireless communication to share vehicle states such as acceleration to enable shorter gap following. Both ACC and CACC rely on range sensors such as radar to obtain the actual inter-vehicular distance for gap-keeping control. The range sensor may lose detection of the target, the preceding vehicle, on curvy roads or steep hills due to limited angle of view. Unfavourable weather conditions, target selection failure, or hardware issue may also result in target detection loss. During target detection loss, the vehicle following system usually falls back to Cruise Control (CC) wherein the follower vehicle maintains a constant speed. In this work, we propose an alternative way to obtain the inter-vehicular distance during target detection loss to continue vehicle following. The proposed algorithm integrates inter-vehicular communication, accurate vehicle localization, and a digital map with lane center information to approximate the inter-vehicular distance. In-lab robot following experiments demonstrated that the proposed algorithm provided desirable inter-vehicular distance approximation. Although the algorithm is intended for vehicle following application, it can also be used for other scenarios that demand vehicles' relative distance approximation. The work also showcases our in-lab development effort of robotic emulation of traffic for connected and automated vehicles.