HYBRID AI ARCHITECTURE FOR DYNAMIC WORKLOAD SCHEDULING IN LARGE-SCALE DISTRIBUTED COMPUTING SYSTEMS

DOI: 10.31673/2412-4338.2025.013820

Abstract

This research presents a novel hybrid artificial intelligence architecture for optimizing task scheduling in large-scale distributed computing environments, combining distributed AI Schedulers (AIS) with a centralized Decision Tree (DT) layer to achieve superior scheduling accuracy and adaptability. Traditional scheduling approaches struggle with heterogeneous environments, leading to suboptimal resource utilization, which our two-tiered architecture addresses through cluster-level AI Schedulers that pre-select suitable nodes based on four key metrics: performance, data transfer rate, operational time, and security level. The neural network architecture employs two hidden layers with ReLU activation functions, while the Decision Tree layer uses an enhanced CART algorithm for final node selection, incorporating both primary characteristics and historical performance data. Experiments conducted across deployment scales from 5 clusters (500 nodes) to 30 clusters (15000 nodes) demonstrate the hybrid approach achieves 99-100% accuracy in node selection, significantly outperforming the standalone AI Scheduler's 94-96% accuracy, while maintaining consistent performance and transparent decision-making processes across all scales. This architecture proves particularly effective for cloud computing environments, IoT deployments, and distributed systems requiring sophisticated resource allocation, contributing a scalable, accurate, and interpretable scheduling solution that effectively combines local intelligence with centralized decision-making for broad applicability in domains requiring dynamic resource allocation in distributed environments.

Keywords: Distributed task scheduling, artificial intelligence, hybrid architecture, decision trees, neural networks, heterogeneous systems, resource optimization, cloud computing