Tiered Lakehouses for Real-Time MLOps
Keywords:
Feature stores, DataOps, Continuous Integration, Continuous Delivery, Cloud-Native platforms, Medallion Architecture, Climate Simulation, City of New York, NYC Citywide Climate Networks, NYC City Sensor Data, Medium Data, Continuous Data Engineering .Abstract
As large-scale machine learning in production becomes ubiquitous across industries, both engineering and research teams are pushing for faster, continuously retrained models. Yet building and running a model — even with a well-established MLOps framework — is often only a small part of the overall task. Feature engineering and feature selection typically consume the majority of a project's data and ML effort. A Medallion Architecture purpose-built for real-time feature engineering, paired with an enterprise Feature Store, can accelerate this process by integrating the latest data insights into models and bringing both new features and models under continuous integration pipelines.
This architecture incorporates key DataOps and MLOps best practices for target labeling, feature quality, and CI/CD pipelines, while allowing features to be built in parallel — either as a dedicated MLOps modeling step or as part of standard product development. We present ML pipelines and Feature Stores for scalable, real-time feature engineering, covering the Medallion Architecture and a large-scale implementation of it, along with best practices for real-time streaming data ingestion and processing and continuous integration/continuous delivery (CI/CD) for features. The resulting architecture is designed for data scale, balancing the engineering demands of feature development with horizontal scaling through sharding strategies aligned to both the features being built and cost objectives for exploratory data analysis and feature selection.
References
1. Armbrust, M., Das, T., Sun, L., Yavuz, B., Zhu, S., Murthy, M., Torres, J., van Hovell, H., Ionescu, A., Łuszczak, A., Świtakowski, M., Szafrański, M., Li, X., Ueshin, T., Mokhtar, M., Boncz, P., Ghodsi, A., Paranjpye, S., Senster, P., Xin, R., & Zaharia, M. (2020). Delta Lake: High-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment, 13(12), 3411–3424.
2. Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. In Proceedings of the 7th Biennial Conference on Innovative Data Systems Research (CIDR 2021).
3. MANGALAMPALLI, B. M., KOLLA, S. H., APPA RAO NAGUBANDI, D. R., & SEGIREDDY, A. R. (2025). AN INTELLIGENT, REAL-TIME DIGITAL FABRIC FOR HEALTHCARE AND FINANCIAL ECOSYSTEMS USING AUTONOMOUS LEARNING AND GENERATIVE SYSTEMS. TPM–Testing, Psychometrics, Methodology in Applied Psychology, 32(S9 (2025): Posted 15 December), 3070-3086.
4. Munappy, A. R., Mattos, D. I., Bosch, J., Olsson, H. H., & Dakkak, A. (2020). From ad-hoc data analytics to DataOps. In Proceedings of the International Conference on Software and Systems Processes (ICSSP 2020), 165–174.
5. Tamburri, D. A. (2020). Sustainable MLOps—Trends and challenges. In Proceedings of the 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 2020), 17–23.
6. Karlas, B., Interlandi, M., Renggli, C., Wu, W., Zhang, C., Babu, D. M. I., Edwards, J., Lauren, C., Xu, A., & Weimer, M. (2020). Building continuous integration services for machine learning. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1163–1173.
7. Mangalampalli, B. M., Bandi, V. D. V. K., Kolla, S. K., & Kumar, M. V. K. (2025). Towards Self-Evolving Healthcare Intelligence: Integrating Advanced Learning Systems with Real-Time Clinical Data Pipelines. Cultura: International Journal of Philosophy of Culture and Axiology, 22(12s), 464-486.
8. Lwakatare, L. E., Crnković, I., Rånge, E., & Bosch, J. (2020). From a data science driven process to a continuous delivery process for machine learning systems. In I. Crnkovic, M. Chechik, & P. Grünbacher (Eds.), Product-focused software process improvement (pp. 165–181). Springer.
9. Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2020). Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering, 48(1), 1–36.
10. Renggli, C., Rimanic, L., Gürel, N. M., Karlaš, B., Wu, W., & Zhang, C. (2021). A data quality-driven view of MLOps. IEEE Data Engineering Bulletin, 44(1), 11–23.
11. Mäkinen, S. S., Skogström, H., Laaksonen, E., & Mikkonen, T. (2021). Who needs MLOps: What data scientists seek to accomplish and how can MLOps help? In 2021 IEEE/ACM 1st Workshop on AI Engineering—Software Engineering for AI (WAIN), 110–112.
12. Hai, R., Koutras, C., Quix, C., & Jarke, M. (2023). Data lakes: A survey of functions and systems. IEEE Transactions on Knowledge and Data Engineering, 35(12), 12571–12590.
13. Kreuzberger, D., Kühl, N., & Hirschl, S. (2023). Machine learning operations (MLOps): Overview, definition, and architecture. IEEE Access, 11, 31866–31879.
14. Hewage, N., & Meedeniya, D. (2022). Machine learning operations: A survey on MLOps tool support. In Proceedings of the 2022 IEEE International Conference on Industrial and Information Systems.
15. Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2022). Challenges in deploying machine learning: A survey of case studies. ACM Computing Surveys, 55(6), Article 114, 1–29.
16. Behm, A., Palkar, S., Agarwal, U., Armstrong, T., Cashman, D., Dave, A., Greenstein, T., Hovsepian, S., Johnson, R., Krishnan, A. S., Leventis, P., Luszczak, A., Menon, P., Mokhtar, M., Pang, G., Paranjpye, S., Rahn, G., Samwel, B., van Bussel, T., van Hovell, H., Xue, M., Xin, R., & Zaharia, M. (2022). Photon: A fast query engine for lakehouse systems. In Proceedings of the 2022 International Conference on Management of Data, 2326–2339.
17. Mangalampalli, B. M., Kolla, S. K., Bandi, V. D. V. K., Yandamuri, U. S., & Rani, P. S. (2025). Designing Intelligent Healthcare Ecosystems through Adaptive Data Integration and Autonomous Learning Systems. Vascular and Endovascular Review, 8(20s), 330-347.
18. Schneider, J., Gröger, C., Lutsch, A., Schwarz, H., & Mitschang, B. (2024). The lakehouse: State of the art on concepts and technologies. SN Computer Science, 5, Article 449.
19. AbouZaid, A., Barclay, P. J., Chrysoulas, C., & Pitropakis, N. (2025). Building a modern data platform based on the data lakehouse architecture and cloud-native ecosystem. Discover Applied Sciences, 7, Article 166.
20. Tagliabue, J., & Greco, C. (2024). Reproducible data science over data lakes: Replayable data pipelines with Bauplan and Nessie. In Proceedings of the 2024 International Workshop on Data Management for End-to-End Machine Learning.
21. Bollineni, S. (2022). Enhancing data lakehouse architecture with DevOps and MLOps practices. Journal of Mathematical & Computer Applications, 1(2), 1–5.
22. Bollineni, S. (2023). DevOps approaches to managing and deploying machine learning algorithms in lakehouse environments. International Journal of Business, Management and Economics Research, 7(8).
23. Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2022). Machine learning deployment: Challenges and opportunities in production environments. ACM Computing Surveys, 55(6), 1–29.
24. Renggli, C., Rimanic, L., Gürel, N. M., Karlaš, B., Wu, W., & Zhang, C. (2021). Data quality in machine learning systems: A systematic perspective on MLOps. IEEE Data Engineering Bulletin, 44(1), 11–23.
25. Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021). Understanding deep learning requires rethinking generalization. Communications of the ACM, 64(3), 107–115.
26. Xu, Y., Liu, X., Cao, X., Huang, C., Liu, E., Qian, S., & Zhang, Y. (2021). Artificial intelligence: A powerful paradigm for scientific research. The Innovation, 2(4), 100179.
27. Sculley, D., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2020). Machine learning: The high-interest credit card of technical debt. In Proceedings of the 2020 International Conference on Machine Learning and Applications.
28. Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2020). Software engineering for machine learning: A case study. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice, 291–300.
29. Tamburri, D. A., Kazman, R., & Fahimi, H. (2020). Cloud-native software architecture for machine learning operations. In Proceedings of the International Conference on Cloud Computing and Services Science.
30. Sato, D., Wider, A., & Windheuser, C. (2020). Continuous delivery for machine learning: MLOps and machine learning pipelines. IEEE Software, 37(5), 31–37.
31. Karlaš, B., Wu, W., Renggli, C., Zhang, C., & Weimer, M. (2021). Towards data-centric machine learning operations: Managing data quality and lifecycle. IEEE Data Engineering Bulletin, 44(1), 24–35.
32. Mangala, N., & Kolla, S. H. (2025). Real-Time Feature Engineering for Streaming AI Workloads Using PySpark and Azure Event Hubs. International Journal of Multidisciplinary Research in Science, Engineering, Technology & Management, 1(6), 93-105.
33. Hai, R., Quix, C., & Jarke, M. (2021). Data lakes: A survey of architectures, management, and applications. Information Systems, 100, 101799.
34. AbouZaid, A., Barclay, P. J., Chrysoulas, C., & Pitropakis, N. (2025). DataOps and cloud-native approaches for resilient lakehouse platforms. Discover Applied Sciences, 7, Article 166.
