Artificial Intelligence for Cost-Aware Resource Prediction in Big Data Pipelines
This addresses cost and performance issues in cloud environments for big data applications, but it is incremental as it applies an existing method to new data.
This work tackled the problem of inefficient resource allocation in cloud computing by using a Random Forest regression model to predict resource utilization in big data pipelines, achieving high predictive accuracy with an R Square of 0.99, MAE of 0.0048, and RMSE of 0.137.
Efficient resource allocation is a key challenge in modern cloud computing. Over-provisioning leads to unnecessary costs, while under-provisioning risks performance degradation and SLA violations. This work presents an artificial intelligence approach to predict resource utilization in big data pipelines using Random Forest regression. We preprocess the Google Borg cluster traces to clean, transform, and extract relevant features (CPU, memory, usage distributions). The model achieves high predictive accuracy (R Square = 0.99, MAE = 0.0048, RMSE = 0.137), capturing non-linear relationships between workload characteristics and resource utilization. Error analysis reveals impressive performance on small-to-medium jobs, with higher variance in rare large-scale jobs. These results demonstrate the potential of AI-driven prediction for cost-aware autoscaling in cloud environments, reducing unnecessary provisioning while safeguarding service quality.