A Scalable Data Analytics Framework for Real-Time Big Data Processing in Distributed Cloud Environments

Abstract

The increasing volume and velocity of data generated from diverse sources have made
real-time analytics a critical requirement in distributed cloud environments. Conventional
batch processing approaches are insufficient for latency-sensitive applications, necessitating
scalable and efficient stream-based solutions. This paper proposes a scalable data analytics
framework for real-time big data processing that integrates distributed stream processing,
adaptive resource management, and efficient task scheduling. The framework is designed to
handle dynamic workloads by leveraging in-memory computation and distributed storage,
ensuring low latency and high throughput. A drift-aware adaptation mechanism is
incorporated to maintain analytical accuracy under evolving data distributions. Additionally,
a resource-aware scheduling strategy is employed to optimize load balancing and system
performance across distributed nodes. Experimental evaluation demonstrates that the
proposed framework achieves significant improvements in processing latency, scalability, and
resource utilization compared to existing approaches.