Metadata Driven ETL Automation through Code Generation: Accelerating Cloud Modernization in Retail Systems
DOI:
https://doi.org/10.15662/IJEETR.2024.0603003Keywords:
Metadata-driven ETL automation, Code generation for data integration pipelines, Cloud data warehouse modernization in retail, Automated ETL frameworks and metadata management, Model-driven data engineering architectures, Scalable ETL pipelines for retail analytics, Legacy-to-cloud data migration and modernizationAbstract
The fast digitalization of retail businesses has led to the spread of heterogeneous sources of data, high data speed, and more sophisticated analytical demands. Conventional extract-transform-load (ETL) pipelines that are typically manually built and specific to a particular schema and platform find it difficult to match these requirements. These pipelines are also generally hard, hard to support, and risky to adjust when modernizing on cloud where schema upgrades, business logic and migrations to new platforms are frequent. These restrictions are a major obstacle to agility and slowness in the provision of data-generated insights in contemporary retail systems.
In order to overcome these difficulties, this paper discusses a metadata-based ETL automation solution which relies on systematic code generation. The proposed solution separates ETL logic and physical implementations by making metadata a first-class design artifact, and allows generating reusable and configurable data integration pipelines automatically. The paradigm minimizes the effort used in developing manuals, maintains consistency, and increases flexibility when migrating to the cloud and making modernization efforts. Code generation also quickens the generation of pipelines by converting metadata descriptions into implemented ETL workflows specific to the cloud-native execution situations.
The paper suggests a major framework which incorporates a central metadata storage, a code generating engine, and ETL orchestration elements on a cloud-based system. The framework facilitates schema abstraction, automatic pipeline re-generation and integration with the latest cloud data warehouses and analytics systems. The design of the architecture focuses on scalability, maintainability, and evolution simplicity, which make the architecture suitable to evolving dynamic retail data.
Experimental analysis shows that the suggested metadata-driven ETL framework can save a lot of time in development and is much more maintainable than the conventional hand-written ETL pipelines. Based on the results, there is more adaptability to schema modifications and increased scalability with increasing volumes of data, which confirms the efficiency of metadata-based code generations as a viable approach to speeding up the cloud modernization of retail systems.
References
1. Abayomi, A. A., Ogeawuchi, J. C., Akpe, O. E., & Agboola, O. A. (2022). Systematic Review of Scalable CRM Data Migration Frameworks in Financial Institutions Undergoing Digital Transformation. International Journal of Multidisciplinary Research and Growth Evaluation, 3(1), 1093–1098. https://doi.org/10.54660/.ijmrge.2022.3.1.1093-1098
2. AP Khandelwal. (2022). AI-Driven Mainframe Modernization: Unlocking Legacy Data for Cloud Analytics. Sarcouncil.Com. Retrieved from https://sarcouncil.com/2025/06/ai-driven-mainframe-modernization-unlocking-legacy-data-for-cloud-analytics
3. Bellini, E., Bellini, P., Cenni, D., Nesi, P., Pantaleo, G., Paoli, I., & Paolucci, M. (2021). An IOE and big multimedia data approach for urban transport system resilience management in smart cities. Sensors (Switzerland), 21(2), 1–35. https://doi.org/10.3390/s21020435
4. Biase, F. D. (2013). Legacy to Cloud Migration: Assessing the Cloud Readiness of Legacy Software Systems - legacy-to-cloud-migration-assessing-the-cloud-readiness-of-legacy-software-systems. University of Applied Scieneces Northwestern Switzerland. Retrieved from http://www.fhnw.ch/business/msc-bis/research-and-development/master-theses-library/year/2013-master-thesis/legacy-to-cloud-migration-assessing-the-cloud-readiness-of-legacy-software-systems
5. Chimakurthi, V. N. S. S. (2019). Application Portfolio Profiling and Appraisal as Part of Enterprise Adoption of Cloud Computing. Global Disclosure of Economics and Business, 8(2), 129–142. https://doi.org/10.18034/gdeb.v8i2.610
6. Curcin, V., Fairweather, E., Danger, R., & Corrigan, D. (2017). Templates as a method for implementing data provenance in decision support systems. Journal of Biomedical Informatics, 65, 1–21. https://doi.org/10.1016/j.jbi.2016.10.022
7. Dineva, K., & Atanasova, T. (2022). Cloud Data-Driven Intelligent Monitoring System for Interactive Smart Farming. Sensors, 22(17). https://doi.org/10.3390/s22176566
8. Gade, K. R. (2021). Migrations: Cloud Migration Strategies, Data Migration Challenges, and Legacy System Modernization. Journal of Computing and Information Technology, 1(1). Retrieved from https://universe-publisher.com/index.php/jcit/article/view/2
9. Grafberger, S., Groth, P., & Schelter, S. (2023). Automating and Optimizing Data-Centric What-If Analyses on Native Machine Learning Pipelines. Proceedings of the ACM on Management of Data, 1(2), 1–26. https://doi.org/10.1145/3589273
10. Guerriero, M., Tamburri, D. A., & Nitto, E. D. (2021). Stream Gen: Model-driven Development of Distributed Streaming Applications. ACM Transactions on Software Engineering and Methodology, 30(1). https://doi.org/10.1145/3408895
11. Gurcan, F., & Cagiltay, N. E. (2019). Big Data Software Engineering: Analysis of Knowledge Domains and Skill Sets Using LDA-Based Topic Modeling. IEEE Access, 7, 82541–82552. https://doi.org/10.1109/ACCESS.2019.2924075
12. Hanine, M., Lachgar, M., Elmahfoudi, S., & Boutkhoum, O. (2021). MDA Approach for Designing and Developing Data Warehouses: A Systematic Review & Proposal. International Journal of Online and Biomedical Engineering, 17(10), 99–110. https://doi.org/10.3991/ijoe.v17i10.24667
13. Huang, X., Liu, Y., Huang, L., Onstein, E., & Merschbrock, C. (2023, May 1). BIM and IoT data fusion: The data process model perspective. Automation in Construction. Elsevier B.V. https://doi.org/10.1016/j.autcon.2023.104792
14. B. K. Alti, “A Policy-Driven Architecture for Enterprise-Scale Patch and Configuration Governance Using Red Hat Satellite,” Letters in High Energy Physics, vol. 2024, Article ID 8141, Feb. 2024. DOI: https://doi.org/10.52783/lhep.2024.1606
15. Kumar, S., Thumburu, R., Analyist, S. E., Asea, A., Boveri, B., & Corresponding, S. (2021). EDI Migration and Legacy System Modernization: A Roadmap. Innovative Engineering Sciences Journal, 1(1). Retrieved from https://inscipub.com/IESJ/article/view/362
16. Kurz, S., De Gersem, H., Galetzka, A., Klaedtke, A., Liebsch, M., Loukrezis, D., … Schmidt, M. (2022). Hybrid modeling: towards the next level of scientific computing in engineering. Journal of Mathematics in Industry, 12(1). https://doi.org/10.1186/s13362-022-00123-0
17. Leonard, A., & Bradshaw, K. (2020). SQL Server Data Automation Through Frameworks: Building Metadata-Driven Frameworks with T-SQL, SSIS, and Azure Data Factory. SQL Server Data Automation Through Frameworks: Building Metadata-Driven Frameworks with T-SQL, SSIS, and Azure Data Factory (pp. 1–391). Springer International Publishing. https://doi.org/10.1007/978-1-4842-6213-9
18. B. K. Alti, “Systematic Enforcement of CIS-Aligned Security Controls for Kubernetes Worker Nodes,” The Eastasouth Journal of Information System and Computer Science, Vol. 1, No. 01, August, pp. 156 – 168, Aug. 2023, DOI: https://esj.eastasouth-institute.com/index.php/esiscs/article/view/864
19. Mardikoraem, M., Wang, Z., Pascual, N., & Woldring, D. (2023, November 1). Generative models for protein sequence modeling: recent advances and future directions. Briefings in Bioinformatics. Oxford University Press. https://doi.org/10.1093/bib/bbad358
20. Mishra, A. (2020). Legacy System Modernization: Effective Strategies and Best Practices. International Journal of Leading Research Publication (IJLRP) IJLRP20031245, 1(3).
21. B. K. Alti, “Continuous Security Validation of Linux Systems Using Configuration-as-Code,” The Eastasouth Journal of Information System and Computer Science, Vol. 1, No. 02, December, pp. 184-193 DOI: https://esj.eastasouth-institute.com/index.php/esiscs/article/view/863
22. Mirabello, C., Azinas, S., & Carroni, M. (2023). Unmasking AlphaFold: integration of experiments and predictions with a smarter template mechanism. BioRxiv, 1–14.
23. Moradi, R., Cofre-Martel, S., Lopez Droguett, E., Modarres, M., & Groth, K. M. (2022). Integration of deep learning and Bayesian networks for condition and operation risk monitoring of complex engineering systems. Reliability Engineering and System Safety, 222. https://doi.org/10.1016/j.ress.2022.108433
24. Mohagheghi, P., & Sæther, T. (2011). Software engineering challenges for migration to the Service Cloud Paradigm: Ongoing work in the REMICS project. In Proceedings - 2011 IEEE World Congress on Services, SERVICES 2011 (pp. 507–514). https://doi.org/10.1109/SERVICES.2011.26
25. Nichols, D. A., Miller, R. A., Jadrnicek, R., Chiu, H., DeSalvo, S. V., Grifin, K. S., … Kushida, C. A. (2013). Data storage and processing procedures of a sleep research data management system. Sleep, 36, A420–A421. Retrieved from http://www.embase.com/search/results?subaction=viewrecord&from=export&id=L71514044
26. Ogunwole, O., Onukwulu, E. C., Joel, M. O., Adaga, E. M., & Ibeh, A. I. (2023). Modernizing Legacy Systems: A Scalable Approach to Next-Generation Data Architectures and Seamless Integration. International Journal of Multidisciplinary Research and Growth Evaluation., 4(1), 901–909. https://doi.org/10.54660/.ijmrge.2023.4.1.901-909
27. Paskaleva, G., Mazak-Huemer, A., Wimmer, M., & Bednar, T. (2021). Leveraging integration facades for model-based tool interoperability. Automation in Construction, 128. https://doi.org/10.1016/j.autcon.2021.103689
28. Parri, J., Patara, F., Sampietro, S., & Vicario, E. (2021). A framework for Model-Driven Engineering of resilient software-controlled systems. Computing, 103(4), 589–612. https://doi.org/10.1007/s00607-020-00841-6
29. Patil, S. (2023). Optimizing Legacy Systems for Cloud Migration: Patterns and Pitfalls in AWS Transition. International Journal of Computing and Engineering, 4(4), 6–16. https://doi.org/10.47941/ijce.3161
30. Ramchand, K., Baruwal Chhetri, M., & Kowalczyk, R. (2021). Enterprise adoption of cloud computing with application portfolio profiling and application portfolio assessment. Journal of Cloud Computing, 10(1). https://doi.org/10.1186/s13677-020-00210-w
31. Reddy Gade, K. (2021). Migrations: Cloud Migration Strategies, Data Migration Challenges, and Legacy System Modernization. Journal of Computing and Information Technology (Vol. 1). Retrieved from https://universe-publisher.com/index.php/jcit/index
32. Kakarla, Roshan., & Sannareddy, Sai Bharath. (2024). AI-Driven DevOps Automation for CI/CD Pipeline Optimization. Eastasouth Journal of Information System and Computer Science (ESISCS), 2(01), 70–78. https://doi.org/10.58812/esiscs.v2i01.849
33. SABIRI, K., BENABBOU, F., HAIN, M., MOUTACHAOUIK, H., & AKODADI, K. (2016). A Survey of Cloud Migration Methods: A Comparison and Proposition. International Journal of Advanced Computer Science and Applications, 7(5). https://doi.org/10.14569/ijacsa.2016.070579
34. Sleiti, A. K., Kapat, J. S., & Vesely, L. (2022, November 1). Digital twin in energy industry: Proposed robust digital twin for power plant and other complex capital-intensive large engineering systems. Energy Reports. Elsevier Ltd. https://doi.org/10.1016/j.egyr.2022.02.305
35. Sannareddy, Sai Bharath. (2024). GenAI-Driven Observability and Incident Response Control Plane for Cloud-Native Systems.International Journal of Research and Applied Innovations (IJRAI), 7(6), 11817–11828. https://doi.org/10.15662/IJRAI.2024.0706027
36. Suleykin, A., & Panfilov, P. (2020). Metadata-Driven Industrial-Grade ETL System. In Proceedings - 2020 IEEE International Conference on Big Data, Big Data 2020 (pp. 2433–2442). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/BigData50022.2020.9378367
37. Tomingas, K., Kliimask, M., & Tammet, T. (2015). Data Integration Patterns for Data Warehouse Automation. In Advances in Intelligent Systems and Computing (Vol. 312, pp. 41–55). Springer Verlag. https://doi.org/10.1007/978-3-319-10518-5_4
38. Tyc, J., Selami, T., Hensel, D. S., & Hensel, M. (2023, June 1). A Scoping Review of Voxel-Model Applications to Enable Multi-Domain Data Integration in Architectural Design and Urban Planning. Architecture. Multidisciplinary Digital Publishing Institute (MDPI). https://doi.org/10.3390/architecture3020010
39. Xie, C., Du, S., Wang, J., Lao, J., & Song, H. (2023, May 1). Intelligent modeling with physics-informed machine learning for petroleum engineering problems. Advances in Geo-Energy Research. Yandy Scientific Press. https://doi.org/10.46690/ager.2023.05.01
40. Zacharewicz, G., Daclin, N., Doumeingts, G., & Haidar, H. (2020). Model Driven Interoperability for System Engineering. Modelling, 1(2), 94–121. https://doi.org/10.3390/modelling1020007
