A reinforcement reinforcement learning for data orchestration in spark streaming framework with tensorFlow agents

Abstract

This work introduces a novel architecture that integrates the Proximal Policy Optimiza tion (PPO) algorithm supported by TensorFlow Agents (TF-Agents) to intelligently or chestrate in-memory data for Spark Streaming applications. Apache Spark Streaming is known for its reliability, but it can suffer from performance issues. This is because the Java Virtual Machine is not efficient in managing the heap for data caching. As a re sult, out-of-memory (OOM) problems can occur, which can compromise data integrity, increase latency, and decrease throughput. Additionally, memory-borrowing operations can be strenuous, leading to OOM exceptions, high latency, and system crashes. The proposed architecture introduces a novel framework that enhances the existing policy module of SparkStreaming++ (SS++). It enables the adaptation of various algorithms implemented by TF-Agents and custom implementations, thereby offering a compelling alternative to the current heuristic-based and previous reinforcement learning-based poli cies. The PPO algorithm is implemented on this new architecture to showcase its efficacy. This unique framework operates independently of Spark’s native backpressure mecha nisms and presents an improved approach to policy implementation. The findings indi cate that PPO consistently achieved higher throughput, surpassing SAQN and Adaptive by up to 24.6% and 25.5%, respectively. However, PPO’s processing times and schedul ing delays were longer than those of SAQN and Adaptive. Although PPO was robust and efficient in high-data-intensity situations, there are still areas for improvement. PPO ex hibited excellent stability, especially in high-data-intensity scenarios, and maintained an application performance, with an average response time of 0.04 seconds. The results sug gest that PPO is best suited for high-throughput applications, but minimizing scheduling delays and processing times is