Real-Time Cloud-Native Data Processing Through Distributed Stream Analytics and Adaptive Workload Scheduling
DOI:
https://doi.org/10.15662/IJEETR.2026.0805008Keywords:
Cloud-native computing, distributed stream analytics, adaptive workload scheduling, real-time data processing, stream processing, dynamic resource allocation, Kubernetes, fault tolerance, resource optimization, low-latency computing.Abstract
Real-time cloud-native data processing has become essential for applications that require continuous data ingestion, low-latency analytics, and responsive decision-making across distributed computing environments. The increasing volume, velocity, and diversity of data generated by Internet of Things devices, financial transactions, online services, and enterprise applications have created significant challenges for conventional batch-processing architectures. This study examines a cloud-native framework that integrates distributed stream analytics with adaptive workload scheduling to improve processing efficiency, resource utilization, scalability, and fault tolerance. The proposed methodology combines event-driven data ingestion, distributed stream processing, dynamic resource allocation, and workload-aware scheduling to accommodate fluctuating data rates and heterogeneous computational demands. Stream-processing tasks are distributed across multiple computing nodes, while scheduling decisions are adjusted according to workload intensity, processing latency, queue length, and resource availability. The framework also considers state management, backpressure control, and fault recovery to maintain reliable processing under changing operating conditions. Its evaluation framework focuses on end-to-end latency, throughput, CPU and memory utilization, scheduling overhead, and recovery time under different workload scenarios. By coordinating stream-processing execution with adaptive scheduling decisions, the proposed approach aims to reduce resource contention and improve responsiveness during workload fluctuations. The study provides a methodological foundation for developing scalable, resilient, and cost-efficient cloud-native data-processing systems suitable for modern real-time analytical applications
References
1. Mothukuri, V., Parizi, R. M., Pouriyeh, S., Huang, Y., Dehghantanha, A., & Srivastava, G. (2021). A survey on security and privacy of federated learning. Future Generation Computer Systems, 115, 619–640. Elsevier. https://doi.org/10.1016/j.future.2020.10.007
2. Vani, M., & Dadlani, D. (2026, July). AI-Driven Root Cause Analysis for Kubernetes Clusters. In 2026 IEEE 9th International Conference on Big Data and Artificial Intelligence (BDAI) (pp. 222-227). IEEE.
3. Basavala, S. R. (2024). Zero trust API security architecture for microservices-based enterprise applications. International Journal of Computer Technology and Electronics Communication, 7(4), 9269–9280.
4. Lingala, B. (2025). Automating Enterprise Decision-Making with Online Transaction Data. Journal of Computational Analysis & Applications, 34(12).
5. Mathew, A. (2021). Artificial intelligence for offence and defense-the future of cybersecurity. Educational Research, 3(3), 159-163.
6. Chandran, A. S. (2023). A unified Spark ETL stack across Cloudera CDP, Azure HDInsight, and Databricks. International Journal of Future Innovative Science and Technology, 6(6), 11769–11779.
7. Pochincharla, S. (2025). Secure Explainable Machine Learning Architecture for Smart Enterprise Systems and Real Time Risk Governance. International Journal of Computer Technology and Electronics Communication, 8(5), 11554-11562.
8. Sudhan, S. K. H. H., & Kumar, S. S. (2016). Gallant Use of Cloud by a Novel Framework of Encrypted Biometric Authentication and Multi Level Data Protection. Indian Journal of Science and Technology, 9, 44.
9. Reddy, K. V. (2025). Automating UVM frameworks using artificial intelligence and machine learning for complex SoC verification. International Journal of Research and Applied Innovations, 8(6), 13283–13287.
10. Karrothu, A. (2025). Engineering scalable cloud-native distributed systems for real-time security telemetry ingestion and threat detection. Journal of Computational Analysis and Applications, 34(12), 1175–1188.
11. Jayaraman, S., Rajendran, S., & P, S. P. (2019). Fuzzy c-means clustering and elliptic curve cryptography using privacy preserving in cloud. International Journal of Business Intelligence and Data Mining, 15(3), 273-287.
12. Ahuja, D. (2026). A Platform Engineering Maturity Model For Cloud-Native Enterprise Transformation. Journal of International Crisis & Risk Communication Research (JICRCR), 9(4), 126.
13. Divi, V. R., Gullapalli, S., & Brahmandam, B. A. (2026, July). Containing the Cascade: A Benchmark and Reference Mediator for Failure Propagation in Tool-Using LLM Multi-Agent Systems. In 2026 5th International Conference on Electronics Representation and Algorithm (ICERA) (pp. 604-609). IEEE.
14. Padmanabham, S. (2025). Security and Compliance in Integration Architectures: A Framework for Modern Enterprises. International Journal of Computing and Engineering, 7(16), 45-55.
15. Singh, M. P. (2024). Building Cyber-Resilient Cloud Infrastructure Integrating Zero Trust, Identity Security, and Continuous Threat Monitoring. IACSE-International Journal of Cyber Security (IACSE-IJCS), 5(2), 7-23.
16. Polamarasetty, V. K. (2021). Modernizing SAP sales and distribution systems through ABAP-based enterprise solutions. International Journal of Research and Applied Innovations, 4(2), 4925-4930.
17. Jinna, K. R., & Subramani, S. K. (2022). Supply-chain-aware CI/CD for regulated enterprises: Integrating artifact signing, secret injection and promotion gates in Kubernetes-native pipelines. International Journal of Computer Technology and Electronics Communication, 5(6), 16275–16285.
18. Ponugoti, M. (2024). AI-driven microservice architectures: Enhancing compliance and decision intelligence in cloud environments. International Journal of Advanced Engineering Science and Information Technology (IJAESIT), 7(5), 14880.
19. Raja, G. V., & Mali, R. K. (2021). Federated learning frameworks for privacy-preserving artificial intelligence applications. International Journal of Research Publications in Engineering, Technology and Management (IJRPETM), 4(3), 4946-4950.
20. Shaik, N. (2025). Artificial intelligence in healthcare economics: Transforming cost efficiency, member experience, and preventative care. International Journal of Research Publications in Engineering, Technology and Management, 8(2), 11850–11854.
21. Mysula, S. (2024). Event-driven integration architecture using publication methods for real-time DealCloud business operations & workflows. International Journal of Research and Applied Innovations, 7(4), 11233–11239.
22. Sugumar, R. (2022). Explainable Deep Learning Framework for Financial Fraud Detection and Intelligent Risk Analysis. International Journal of Emerging Trends in Engineering and Management Research, 7(5), 12528.
23. Aragani, V. M., Maroju, P. K., & Mudunuri, L. N. R. (2024, December). Implementing Non-functional Production Regression Testing with Kubernetes. In International Conference on Information and Communication Technology for Competitive Strategies (pp. 231-244). Singapore: Springer Nature Singapore.
24. Boddu, Y., Majumder, R. Q., Natta, P. K., & Pahune, S. A. (2026, May). Enhancing Credit Card Risk Control Systems through Data Governance Using Metadata-Centric Machine Learning Models. In 2026 International Conference on Computing, Communication, Security and Intelligent Systems (IC3SIS) (pp. 1-6). IEEE.
25. Karuturi, Y. (2026). Digital twin architecture for enterprise SAP landscapes for predictive maintenance and capacity planning. International Journal of Future Innovative Science and Technology, 9(2), pp. 554–563.
26. Lenin, S. T., & Venkatasalam, K. (2025). Effective classification of heart disease using convolutional neural networks. Circuits, Systems, and Signal Processing, 44(2), 911-935.
27. Neela, S. (2024). Middleware-based interoperability models for hybrid and multi-cloud enterprise environments. International Journal of Emerging Trends in Computer Science and Information Technology, 5(1), 195–210. https://doi.org/10.63282/3050-9246.IJETCSIT-V5I1P119
28. Madabathula, L. (2026, May). A Resilient Multi-Petabyte Lakehouse Architecture for Regulated Enterprises: Design Patterns and Failure Recovery. In 2026 2nd International Conference on Sustainable Computing and Integrated Communication in Changing Landscape of AI (ICSCAI) (pp. 731-735). IEEE.
29. Gopinathan, V. R. (2022). Advanced Predictive Decision Intelligence Using Machine Learning for Strategic Enterprise Management Platforms. International Journal of Research and Applied Innovations, 5(6), 8162-8169.
30. Chinthamani, R. (2024). Modernizing enterprise workload identity through SPIFFE-based certificate automation. International Journal of Research and Applied Innovations, 7(4), 11240–11245.
31. Kalabhavi, V. (2023). Automated quote generation using AI for product configuration, pricing and external order integration in SAP CPQ. International Research Journal of Innovative Engineering, 7(3), 12503–12515.
32. Kari, M. (2026, March). Intelligent Deep Learning-Based System for Improved Phishing Identification Accuracy in Web Platforms. In 2026 IEEE International Conference for Convergence in Computing Technology (I3CTCON) (pp. 1-6). IEEE.
33. Srikanth, D. V. (2025). AI-powered entrepreneurial ecosystems: a computational model for sustainable startup growth in green tech sectors. Int. J. Environ. Sci, 1068-1077.
34. Kopparthi, G. S. (2023). Trustworthy artificial intelligence for high-stakes decision-making: Explainability, human oversight, and system reliability. International Research Journal of Innovative Engineering, 7(4), 12847–12860.
35. Gadige, C. D. (2023). Observability for distributed CRM architectures: Monitoring Salesforce, APIs, middleware, events, and AI agents. International Journal of Future Innovative Science and Technology, 6(5), 11388–11397.
36. Gunasekaran, R. M. N. (2025). Real-Time Risk Monitoring using Big Data Streaming and Machine Learning in Modern Banking Systems. International Journal of Research and Applied Innovations, 8(3), 13086-13096.
37. Muppidi, N. K. R. (2026, July). Secure Context Cache: Token-Efficient and Least-Privilege Shared Memory for Enterprise Developer Agents. In 2026 5th International Conference on Electronics Representation and Algorithm (ICERA) (pp. 679-684). IEEE.
38. Aaviti, P. K. (2026). AI-Ready Data Lakehouse Architecture for Banking and Financial Services. International Journal of Research and Applied Innovations, 9(3), 691-701.
39. Sunkara, R. (2023). Intelligent test engineering with large language models: A framework for automated test generation and defect triage. Journal of Advanced Research in Technology and Management Sciences, 5(1), 53–73.
40. Fernández, E. B., Monge, R., & Hashizume, K. (2014). A survey on security issues in cloud computing. International Journal of Information Security, 13, 113–170. Springer. https://doi.org/10.1007/s10207-013-0208-7
