Reinforcement learning-based optimization of distributed data pipelines for cost-efficient and adaptive workload scheduling in cloud data platforms

Abstract

Distributed data pipelines in cloud data platforms must satisfy deadline commitments while keeping infrastructure spending low, yet most production schedulers still rely on fixed provisioning or hand-tuned threshold rules that react to load only after queues have already grown. This paper presents RL-Sched, a reinforcement learning controller that jointly decides how many compute nodes to provision, whether to acquire them on the on-demand or the preemptible spot market, and which dispatch rule to apply to the waiting pipeline jobs. The scheduling problem is formulated as a Markov decision process whose reward combines infrastructure cost, penalties for missed service level agreements (SLAs), and a small queueing term, and it is solved with a double deep Q-network trained under domain randomization over load, burst, and spot-market conditions. Because evaluating a controller of this kind directly on a production platform is risky, the study used a discrete-time simulator of a cloud data platform with directed-acyclic-graph jobs, diurnal and bursty arrivals, a mean-reverting spot market with price-dependent preemption, and provisioning delays. RL-Sched was compared against two static provisioning policies and two tuned reactive autoscalers over 40 held-out episodes in each of five scenarios. In the nominal scenario, RL-Sched reduced daily cost by 31.2 percent relative to the strongest tuned heuristic (from $95.0 to $65.4) and raised average utilization from 49.8 to 68.4 percent, while keeping SLA violations at 0.33 percent. Cost reductions of 33.6 to 52.6 percent were retained under higher load, bursty arrivals, volatile spot prices, and a surge outside the training range, with violation rates between 0.37 and 0.86 percent. Ablation experiments showed that both the spot-market action and the learned dispatch rule contribute materially. The results also exposed a limitation, namely roughly 2.7 times more scaling actions than the heuristic, which motivates switching-cost regularization in future work.