Adaptive Multi-Cloud Lakehouses for Real-Time Analytics
Keywords:
Cloud computing; data integration; data management; data warehousing; enterprise architecture; enterprise information systems; data analytics; analytics; data governance; policy-based access control.Abstract
Enterprise analytic frameworks increasingly rely on multi-cloud architectures, yet these systems are rarely described in a systematic, evidence-based way. This paper presents and evaluates a representative lakehouse-based architecture designed to deliver near-real-time analysis of user- and system-generated data across cloud providers. The architecture addresses three critical multi-cloud processing requirements: adaptive resource scaling, cross-cloud data consistency, and support for data governance and privacy policies. We survey the supporting technologies that underpin these requirements, including dynamic resource orchestration, cross-cloud data synchronisation, integration with data governance solutions, and streaming ingestion, querying, and processing. Together, these findings inform a flexible, adaptive architecture for near-real-time analytics in multi-cloud data ecosystems.
Large-scale, multi-user information systems deployed as public cloud services continuously generate events, alerts, and notifications. Combined with user-generated inputs, this telemetry can be exploited within a multi-cloud architecture to enrich the enterprise information model, enforce privacy policies, and monitor service-level agreements. Building the cloud-native data pipelines needed to support real-time analytics across such multi-cloud ecosystems raises significant challenges, particularly in meeting strict latency constraints while upholding safety and privacy requirements. Meeting these demands imposes specific technological constraints on data processing, data lake/data warehouse integration, and the provision of cross-cloud data consistency and synchronisation.
References
1. Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. CIDR.
2. Sawadogo, P. N., & Darmont, J. (2021). On data lake architectures and metadata management. Journal of Intelligent Information Systems, 56(1), 97–120.
3. Reddy, R. (2023). Predictive Health Insights: Ai And Ml's Frontier In Disease Prevention And Patient Management. Available at SSRN.
4. Begoli, E., Goethert, I., & Knight, K. (2021). A lakehouse architecture for the management and analysis of heterogeneous data for biomedical research and mega-biobanks. In Proceedings of the 2021 IEEE International Conference on Big Data (Big Data 2021) (pp. 4643–4651).
5. Paruchuri, J. K. (2021). Lakehouse architecture: Unifying data lakes and data warehouses. International Journal of Computer Techniques, 8(1).
6. Kaur, K., Garg, S., & Kaur, K. (2021). Multi-cloud adoption challenges for the cloud-native era. International Journal of Cloud Applications and Computing, 11(2).
7. Anwarbasha, H., Sasi Kumar, S., & Dhanasekaran, D. (2021). An efficient and secure protocol for checking remote data integrity in multi-cloud environment. Scientific Reports, 11, 13755.
8. Srinivas Kalisetty. (2023). Big Data–Driven Cloud Collaboration Models for Enhancing Supplier–Retailer Synchronization in Modern Manufacturing Supply Chains. Journal of Computational Analysis and Applications (JoCAAA), 31(4), 2188–2205. Retrieved from https://eudoxuspress.com/index.php/pub/article/view/4232
9. Rau, A., & Suri, P. K. (2021). Towards securing cloud data in the multi-cloud scenario. Periodicals of Engineering and Natural Sciences, 9(2), 868–872.
10. Bice, V., & Hesse, W. (2021). Modeling metadata in data lakes—A generic model. Data & Knowledge Engineering, 136, 101931.
11. Machado, I. A., Costa, C., & Santos, M. Y. (2022). Data mesh: Concepts and principles of a paradigm shift in data architectures. Procedia Computer Science, 196, 263–271.
12. Pamisetty, V. (2023). Transforming Community Engagement with Generative AI: Harnessing Machine Learning and Neural Networks for Hunger Alleviation and Global Food Security. Journal for Re Attach Therapy and Developmental Diversities.
13. Wieder, P., & Nolte, H. (2022). Toward data lakes as central building blocks for data management and analysis. Frontiers in Big Data, 5, 945720.
14. Zichichi, M., Ferretti, S., D’Angelo, G., & Rodríguez-Doncel, V. (2022). Data governance through a multi-DLT architecture in view of the GDPR. Cluster Computing, 25(6), 4515–4542.
15. Nadal, S., Jovanovic, P., Bilalli, B., & Romero, O. (2022). Operationalizing and automating data governance. Journal of Big Data, 9, 117.
16. Armenatzoglou, N., Basu, S., Bhanoori, N., Cai, M., Chainani, N., Chinta, K., Govindaraju, V., Green, T. J., Gupta, M., Hilling, S., Hotinger, E., Leshinsky, Y., Liang, J., McCreedy, M., Nagel, F., Pandis, I., Parchas, P., Pathak, R., Polychroniou, O., Rahman, F., Saxena, G., Soundararajan, G., Subramanian, S., & Terry, D. (2022). Amazon Redshift re-invented. In Proceedings of the 2022 International Conference on Management of Data (pp. 2205–2217).
17. Behm, A., Palkar, S., Agarwal, U., Armstrong, T., Cashman, D., Dave, A., Greenstein, T., Hovsepian, S., Johnson, R., Krishnan, A. S., Leventis, P., Luszczak, A., Menon, P., Mokhtar, M., Pang, G., Paranjpye, S., Rahn, G., Samwel, B., van Bussel, T., van Hovell, H., Xue, M., Xin, R., & Zaharia, M. (2022). Photon: A fast query engine for lakehouse systems. In Proceedings of the 2022 International Conference on Management of Data (pp. 2326–2339).
18. Meda, R., & Pamisetty, A. (2023). Intelligent Infrastructure for Real-Time Inventory and Logistics in Retail Supply Chains. Educational Administration: Theory and Practice, 29 (4), 5215–5233.
19. Tosun, E., Machado, I. A., Costa, C., & Santos, M. Y. (2022). Data mesh: Concepts and principles of a paradigm shift in data architectures. Procedia Computer Science, 196, 263–271.
20. Shahane, R. (2022). Enhancing data governance in multi-cloud environments: A focused evaluation of Microsoft Azure’s capabilities and integration strategies. Journal of Computational Analysis and Applications, 30(2), 536–548.
21. Arul, K. (2022). Data engineering challenges in multi-cloud environments: Strategies for efficient big data integration and analytics. International Journal of Scientific Research and Management, 10(6).
22. Dubey, S., Diwivedi, P., Shukla, A., Kushwaha, R., Shukla, A., & Gupta, S. (2022). A review on data storage in multi-cloud environment for privacy preserving. International Journal of Food and Nutritional Sciences, 11(Special Issue 6), 458–464.
23. Pandugula, C., & Nampalli, R. C. R. Optimizing Retail Performance: Cloud-Enabled Big Data Strategies for Enhanced Consumer Insights.
24. Liu, Y., Zhang, H., & Wang, J. (2022). Observations and expectations on recent developments of data lakes. Procedia Computer Science, 214, 405–411.
25. Sun, Y., Meehan, T., Xie, W., Basmanova, M., Erling, O., Zhong, R., Wang, K., Agarwal, S., Gupta, A., Logothetis, D., & others. (2023). Presto: A decade of SQL analytics at Meta. Proceedings of the VLDB Endowment.
26. Ait Errami, S., Hajji, H., Ait El Kadi, K., & Badir, H. (2023). Spatial big data architecture: From data warehouses and data lakes to the LakeHouse. Journal of Parallel and Distributed Computing, 176, 70–79.
27. Goedegebuure, A., Kumara, I., Driessen, S., Monsieur, G., Di Nucci, D., van den Heuvel, W.-J., & Tamburri, D. A. (2023). Data mesh: A systematic gray literature review. ACM Computing Surveys.
28. Mazumdar, D., Hughes, J., & Onofre, J. B. (2023). The data lakehouse: Data warehousing and more. arXiv.
29. Thamma, S. R. (2023). Lakehouse architecture: Bridging on-prem big data and modern data platforms. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 9(4).
30. Soon, K. Y. (2023). Data lakehouse architecture for self-service data analytics. Universiti Malaysia Pahang Al-Sultan Abdullah.
