Предоставление ресурсов облачных платформ для выполнения композитных приложений в парадигме «поток работ как сервис»
Ключевые слова:
облачные вычисления, система управления потоками работ, планирование, предоставление ресурсов, виртуальная машина, контейнер
Аннотация
Статья посвящена анализу существующих моделей, методов и алгоритмического обеспечения для выполнения наукоемких приложений на облачных платформах в рамках новой концепции «поток работ как сервис» (Workflow as a Service (WaaS)). Платформы WaaS, представляющие собой так называемые мультиарендные среды, позволяют реализовать эффективные механизмы для управления непрерывными потоками разнотипных работ в облачных вычислениях.
Рассмотрен ряд важнейших аспектов: наличие различных провайдеров IaaS, предоставляющих разнотипные ресурсы; территориальная распределенность центров обработки данных; разнородность потоков работ, поступающих на платформу WaaS; необходимость реализации принципа «платить за использование» для конкретного пользователя; решение проблемы размещения на физических серверах виртуальных машин и создания множества контейнеров в них, каждый из которых может быть использован задачами разных потоков. Выделены перспективные направления в развитии методов предоставления ресурсов облачных платформ на основе технологий разворачивания виртуальных машин и контейнеров для выполнения композитных приложений в рамках активно развивающейся парадигмы WaaS.
Литература
1. Workflows Community Summit [Электрон. ресурс] https://workflowsri.org/summits/community (дата обращения 24.02.2023).
2. Existing Workflow Systems [Электрон. ресурс] https://s.apache.org/existing-workflow-systems (дата обращения 24.02.2023).
3. Rodriguez M.A., Buyya R. Scheduling Dynamic Workloads in Multi-tenant Scientific Workflow as a Service Platforms // Future Generation Computer Systems. 2018. V. 79. Pp. 739—750.
4. Hilman M.H., Rodriguez M.A., Buyya R. Workflow-as-a-Service Cloud Platform and Deployment of Bioinformatics Workflow Applications [Электрон. ресурс] https://www.researchgate.net/scientific-contributions/Maria-A-Rodriguez-2114894132 (дата обращения 24.02.2023).
5. Abrishami S., Naghibzadeh M., Epema D.H. Deadline-constrained Workflow Scheduling Algorithms for Infrastructure as a Service Clouds // Future Generation Computer Systems. 2013. V. 29(1). Pp. 158—169.
6. Calheiros R.N., Buyya R. Meeting Deadlines of Scientific Workflows in Public Clouds with Tasks Replication // IEEE Trans. Parallel Distrib. Syst. 2014. V. 25(7). Pp. 1787—1796.
7. Liu S., Ren K., Deng K., Song J. A Task Backfill-based Scientific Workflow Scheduling Strategy on Cloud Platform // Proc. Intern. Conf. Information Sci. and Technol. 2016. Pp. 105—110.
8. Deldari A., Naghibzadeh M., Abrishami S. CCA: a Deadline-constrained Workflow Scheduling Algorithm for Multicore Resources on the Cloud // J. Supercomput. 2017. V. 73(2). Pp. 756—781.
9. Maechling P. e. a. SCEC CyberShake Workflows — Аutomating Probabilistic Seismic Hazard Analysis Сalculations // Workflows for E-Science. N.-Y.: Springer, 2007. Pp. 143—163.
10. Deelman E. e. a. The Cost of Doing Science on the Cloud: the Montage Example // Proc. ACM/IEEE Conf. Supercomputing. 2008. P. 50.
11. Vockler J.-S. e. a. Experiences Using Cloud Computing for a Scientific Workflow Application // Proc. Intern. Workshop Sci. Cloud Computing. 2011. Pp. 15—24.
12. Malawski M., Juve G., Deelman E., Nabrzyski J. Algorithms for Cost- and Deadline-constrained Provisioning for Scientific Workflow Ensembles in IaaS Clouds // Future Generation Computer Syst. 2015. V. 48. Pp. 1—18.
13. Pietri I. e. a. Energy-constrained Provisioning for Scientific Workflow Ensembles // Proc. International Conf. Cloud and Green Computing. 2013. Pp. 34—41.
14. Jiang Q., Lee Y.C., Zomaya A.Y. Executing Large Scale Scientific Workflow Ensembles in Public Clouds // Proc. Intern. Conf. Parallel Proc. 2015. Pp. 520—529.
15. Bryk P., Malawski M., Juve G., Deelman E. Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds // J. Grid Comput. 2016. V. 14(2). Pp. 359—378.
16. Yu Z., Shi W. A Planner-guided Scheduling Strategy for Multiple Workflow Applications // Proc. Intern. Conf. Parallel Processing. 2008. Pp. 1—8.
17. Jiang H.-J. e. a. Scheduling Concurrent Workflows in HPC Cloud through Exploiting Schedule Gaps // Proc. Intern. Conf. Algorithms and Architectures for Parallel Processing. 2011. Pp. 282—293.
18. Stavrinides G.L., Karatza H.D. A Cost-effective and QoS-aware Approach to Scheduling Real-time Workflow Applications in PaaS and SaaS Clouds // Proc. Intern. Conf. Future Internet of Things and Cloud. 2015. Pp. 231—239.
19. Xu M., Cui L., Wang H., Bi Y. A Multiple QoS-constrained Scheduling Strategy of Multiple Workflows for Cloud Computing // Proc. IEEE Intern. Symp. Parallel and Distributed Processing with Appl. 2009. Pp. 629—634.
20. Chen W., Lee Y.C., Fekete A., Zomaya A.Y. Adaptive Multiple-workflow Scheduling with Task Rearrangement // J. Supercomput. 2015. V. 71(4). Pp. 1297—1317.
21. Zhou A.C., He B., Liu C. Monetary Cost Optimizations for Hosting Workflow-as-a-Service in IaaS Clouds // IEEE Trans. Cloud Comput. 2016. V. 4(1). Pp. 34—48.
22. Mao M., Humphrey M. Auto-scaling to Minimize Cost and Meet Application Deadlines in Cloud Workflows // Proc. Intern. Conf. High Performance Computing, Networking, Storage and Analysis. 2011. P. 49.
23. Shi J., Luo J., Dong F., Zhang J. A Budget and Deadline-aware Scientific Workflow Resource Provisioning and Scheduling Mechanism for Cloud // Proc. IEEE Intern. Conf. Computer Supported Cooperative Work in Design. 2014. Pp. 672—677.
24. Wang J. e. a. Workflow as a Service in the Cloud: Architecture and Scheduling Algorithms // Proc. Comput. Sci. 2014. V. 29. Pp. 546—556.
25. Gerlach W. e. a. Skyport: Container-based Execution Environment Management for Multi-cloud Scientific Workflows // Proc. Intern. Workshop on Data-Intensive Computing in the Clouds. 2014. Pp. 25—32.
26. Filgueira R. e. a. Asterism: Pegasus and Dispel4py Hybrid Workflows for Data-intensive Science // Proc. Intern. Workshop Data-Intensive Computing in the Cloud. 2016. Pp. 1—8.
27. Esteves S., Veiga L. Waas: Workflow-as-a-service for the Cloud with Scheduling of Continuous and Data-intensive Workflows // Comput. J. 2016. V. 59(3). Pp. 371—383.
28. Hashem I.A.T. e. a. The Rise of “Big Data” on Cloud Computing: Review and Open Research Issues // Inf. Syst. 2015. V. 47. Pp. 98—115.
29. Senyo P.K., Addae E., Boateng R. Cloud Computing Research: a Review of Research Themes, Frameworks, Methods and Future Research Directions // Int. J. Inf. Manag. 2018. V. 38(1). Pp. 128—139.
30. Stergiou C., Psannis K.E., Kim B.G., Gupta B. Secure Integration of IoT and Cloud Computing // Future Gener. Comput. Syst. 2018. V. 78. Pp. 964—975.
31. Kumar M.R.V., Raghunathan S. Heterogeneity and Thermal-aware Adaptive Heuristics for Energy Efficient Consolidation of Virtual Infrastructure Clouds // J. Comput. Syst. Sci. 2016. V. 82(2). Pp. 191—212.
32. Lopez-Pires F., Baran B. Virtual Machine Placement Literature Review [Электрон. ресурс] https://arxiv.org/abs/1506.01509 (дата обращения 24.02.2023).
33. Usmani Z., Singh S. A Survey of Virtual Machine Placement Techniques in a Cloud Data Center // Proc. Comput. Sci. 2016. V. 78. Pp. 491—498.
34. Liu X.F. e. a. An Energy Efficient ant Colony System for Virtual Machine Placement in Cloud Computing // IEEE Trans. Evol. Comput. 2016. V. 22(1). Pp. 113—128.
35. Abdel-Basset M., Abdle-Fatah L., Sangaiah A.K. An Improved Lévy Based Whale Optimization Algorithm for Bandwidth-efficient Virtual Machine Placement in Cloud Computing Environment // Cluster Computing. 2018. V. 22(1). Pp. 1—16.
36. Shabeera T.P., Kumar S.M., Salam S.M., Krishnan K.M. Optimizing VM Allocation and Data Placement for Data-intensive Applications in Cloud using ACO Metaheuristic Algorithm // Eng. Sci. Technol. Int. J. 2017. V. 20(2). Pp. 616—628.
37. Abdelaziz A. e. a. Intelligent Algorithms for Optimal Selection of Virtual Machine in Cloud Environment, Towards Enhance Healthcare Services // Proc. Intern. Conf. Advanced Intelligent Systems and Informatics. 2017. Pp. 289—298.
38. Ghobaei-Arani M., Shamsi M., Rahmanian A.A. An Efficient Approach for Improving Virtual Machine Placement in Cloud Computing Environment // J. Exp. Theor. Artif. Intell. 2017. V. 29(6). Pp. 1149—171.
39. Saber T., Thorburn J., Murphy L., Ventresque A. VM Reassignment in Hybrid Clouds for Large Decentralized Companies: a Multi-objective Challenge // Future Gener. Comput. Syst. 2018. V. 79. Pp. 751—764.
40. Gao Y. e. a. A Multi-objective ant Colony System Algorithm for Virtual Machine Placement in Cloud Computing // J. Comput. Syst. Sci. 2013. V. 79(8). Pp. 1230—1242.
41. Ganesan T., Vasant P., Litvinchev I. Chaotic Simulator for Bilevel Optimization of Virtual Machine Placements in Cloud Computing // J. Operations Research Soc. China. 2022. V. 10(4). Pp. 703—723.
42. Deelman E. e. a. The Future of Scientific Workflows // Int. J. High Perform. Comput. Appl. 2018. V. 32(1). Pp. 159—175.
43. Chen W., Ferreira da Silva R., Livny M., Wenger K. Pegasus: a Workflow Management System for Science Automation // Futur. Gener. Comput. Syst. 2015. V. 46. Pp. 17—35.
44. Apache Airflow Documentation [Электрон. ресурс] https://airflow.apache.org/ (дата обращения 24.02.2023).
45. Hull D. e. a. Taverna: a Tool for Building and Running Workflows of Services // Nucleic Acids Research. 2006. V. 34. Pp. 729—732.
46. Altintas I. e. a. Kepler: an Extensible System for Design and Execution of Scientific Workflows // Proc. XVI Intern. Conf. Scientific and Statistical Database Management. 2004. Pp. 423—424.
47. Kramer M., Wurz H.M., Altenhofen C. Executing Cyclic Scientific Workflows in the Cloud // J. Cloud Computing: Advances, Systems and Appl. 2021. V. 10(25). Pp. 1—26.
48. Toporkov V., Yemelyanov D., Toporkova A. Coordinated Global and Private Job-flow Scheduling in Grid Virtual Organizations // Simulation Modelling Practice and Theory. 2021. V. 107(14). P. 102228.
49. Toporkov V., Yemelyanov D. Micro-scheduling for Dependable Resources Allocation // Performance Evaluation Models for Distributed Service Networks. Studies in Systems, Decision and Control. 2021. V. 343. Pp. 81—105.
50. Toporkov V., Yemelyanov D., Grigorenko M. Optimization of Resources Allocation in High Performance Computing under Utilization Uncertainty // Proc. Intern. Conf. Computational Sci. 2021. V. 14747. Pp. 540—553.
51. Toporkov V., Yemelyanov D., Bulkhak A. Machine Learning-Based Scheduling and Resources Allocation in Distributed Computing // Proc. Internat. Conf. Computational Sci. 2022. V. 13353. Pp. 3—16.
---
Для цитирования: Топорков В.В., Емельянов Д.М., Булхак А.Н. Предоставление ресурсов облачных платформ для выполнения композитных приложений в парадигме «поток работ как сервис» // Вестник МЭИ. 2023. № 5. С. 156—168. DOI: 10.24160/1993-6982-2023-5-156-168
---
Работа выполнена при поддержке: Российского научного фонда (грант № 22-21-00372), https://rscf.ru/project/22-21-00372/
#
1. Workflows Community Summit [Electron. Resurs] https://workflowsri.org/summits/community (Data Obrashcheniya 24.02.2023).
2. Existing Workflow Systems [Electron. Resurs] https://s.apache.org/existing-workflow-systems (Data Obrashcheniya 24.02.2023).
3. Rodriguez M.A., Buyya R. Scheduling Dynamic Workloads in Multi-tenant Scientific Workflow as a Service Platforms. Future Generation Computer Systems. 2018;79:739—750.
4. Hilman M.H., Rodriguez M.A., Buyya R. Workflow-as-a-Service Cloud Platform and Deployment of Bioinformatics Workflow Applications [Electron. Resurs] https://www.researchgate.net/scientific-contributions/Maria-A-Rodriguez-2114894132 (Data Obrashcheniya 24.02.2023).
5. Abrishami S., Naghibzadeh M., Epema D.H. Deadline-constrained Workflow Scheduling Algorithms for Infrastructure as a Service Clouds. Future Generation Computer Systems. 2013;29(1):158—169.
6. Calheiros R.N., Buyya R. Meeting Deadlines of Scientific Workflows in Public Clouds with Tasks Replication. IEEE Trans. Parallel Distrib. Syst. 2014;25(7):1787—1796.
7. Liu S., Ren K., Deng K., Song J. A Task Backfill-based Scientific Workflow Scheduling Strategy on Cloud Platform. Proc. Intern. Conf. Information Sci. and Technol. 2016:105—110.
8. Deldari A., Naghibzadeh M., Abrishami S. CCA: a Deadline-constrained Workflow Scheduling Algorithm for Multicore Resources on the Cloud. J. Supercomput. 2017;73(2):756—781.
9. Maechling P. e. a. SCEC CyberShake Workflows — Аutomating Probabilistic Seismic Hazard Analysis Сalculations. Workflows for E-Science. N.-Y.: Springer, 2007:143—163.
10. Deelman E. e. a. The Cost of Doing Science on the Cloud: the Montage Example. Proc. ACM/IEEE Conf. Supercomputing. 2008:50.
11. Vockler J.-S. e. a. Experiences Using Cloud Computing for a Scientific Workflow Application. Proc. Intern. Workshop Sci. Cloud Computing. 2011:15—24.
12. Malawski M., Juve G., Deelman E., Nabrzyski J. Algorithms for Cost- and Deadline-constrained Provisioning for Scientific Workflow Ensembles in IaaS Clouds. Future Generation Computer Syst. 2015;48:1—18.
13. Pietri I. e. a. Energy-constrained Provisioning for Scientific Workflow Ensembles. Proc. International Conf. Cloud and Green Computing. 2013:34—41.
14. Jiang Q., Lee Y.C., Zomaya A.Y. Executing Large Scale Scientific Workflow Ensembles in Public Clouds. Proc. Intern. Conf. Parallel Proc. 2015:520—529.
15. Bryk P., Malawski M., Juve G., Deelman E. Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds. J. Grid Comput. 2016;14(2):359—378.
16. Yu Z., Shi W. A Planner-guided Scheduling Strategy for Multiple Workflow Applications. Proc. Intern. Conf. Parallel Processing. 2008:1—8.
17. Jiang H.-J. e. a. Scheduling Concurrent Workflows in HPC Cloud through Exploiting Schedule Gaps. Proc. Intern. Conf. Algorithms and Architectures for Parallel Processing. 2011:282—293.
18. Stavrinides G.L., Karatza H.D. A Cost-effective and QoS-aware Approach to Scheduling Real-time Workflow Applications in PaaS and SaaS Clouds. Proc. Intern. Conf. Future Internet of Things and Cloud. 2015:231—239.
19. Xu M., Cui L., Wang H., Bi Y. A Multiple QoS-constrained Scheduling Strategy of Multiple Workflows for Cloud Computing. Proc. IEEE Intern. Symp. Parallel and Distributed Processing with Appl. 2009:629—634.
20. Chen W., Lee Y.C., Fekete A., Zomaya A.Y. Adaptive Multiple-workflow Scheduling with Task Rearrangement. J. Supercomput. 2015;71(4):1297—1317.
21. Zhou A.C., He B., Liu C. Monetary Cost Optimizations for Hosting Workflow-as-a-Service in IaaS Clouds. IEEE Trans. Cloud Comput. 2016;4(1):34—48.
22. Mao M., Humphrey M. Auto-scaling to Minimize Cost and Meet Application Deadlines in Cloud Workflows. Proc. Intern. Conf. High Performance Computing, Networking, Storage and Analysis. 2011:49.
23. Shi J., Luo J., Dong F., Zhang J. A Budget and Deadline-aware Scientific Workflow Resource Provisioning and Scheduling Mechanism for Cloud. Proc. IEEE Intern. Conf. Computer Supported Cooperative Work in Design. 2014:672—677.
24. Wang J. e. a. Workflow as a Service in the Cloud: Architecture and Scheduling Algorithms. Proc. Comput. Sci. 2014;29:546—556.
25. Gerlach W. e. a. Skyport: Container-based Execution Environment Management for Multi-cloud Scientific Workflows. Proc. Intern. Workshop on Data-Intensive Computing in the Clouds. 2014:25—32.
26. Filgueira R. e. a. Asterism: Pegasus and Dispel4py Hybrid Workflows for Data-intensive Science. Proc. Intern. Workshop Data-Intensive Computing in the Cloud. 2016:1—8.
27. Esteves S., Veiga L. Waas: Workflow-as-a-service for the Cloud with Scheduling of Continuous and Data-intensive Workflows. Comput. J. 2016;59(3):371—383.
28. Hashem I.A.T. e. a. The Rise of “Big Data” on Cloud Computing: Review and Open Research Issues. Inf. Syst. 2015;47:98—115.
29. Senyo P.K., Addae E., Boateng R. Cloud Computing Research: a Review of Research Themes, Frameworks, Methods and Future Research Directions. Int. J. Inf. Manag. 2018;38(1):128—139.
30. Stergiou C., Psannis K.E., Kim B.G., Gupta B. Secure Integration of IoT and Cloud Computing. Future Gener. Comput. Syst. 2018;78:964—975.
31. Kumar M.R.V., Raghunathan S. Heterogeneity and Thermal-aware Adaptive Heuristics for Energy Efficient Consolidation of Virtual Infrastructure Clouds. J. Comput. Syst. Sci. 2016;82(2):191—212.
32. Lopez-Pires F., Baran B. Virtual Machine Placement Literature Review [Electron. Resurs] https://arxiv.org/abs/1506.01509 (Data Obrashcheniya 24.02.2023).
33. Usmani Z., Singh S. A Survey of Virtual Machine Placement Techniques in a Cloud Data Center. Proc. Comput. Sci. 2016;78:491—498.
34. Liu X.F. e. a. An Energy Efficient ant Colony System for Virtual Machine Placement in Cloud Computing. IEEE Trans. Evol. Comput. 2016;22(1):113—128.
35. Abdel-Basset M., Abdle-Fatah L., Sangaiah A.K. An Improved Lévy Based Whale Optimization Algorithm for Bandwidth-efficient Virtual Machine Placement in Cloud Computing Environment. Cluster Computing. 2018;22(1):1—16.
36. Shabeera T.P., Kumar S.M., Salam S.M., Krishnan K.M. Optimizing VM Allocation and Data Placement for Data-intensive Applications in Cloud using ACO Metaheuristic Algorithm. Eng. Sci. Technol. Int. J. 2017;20(2):616—628.
37. Abdelaziz A. e. a. Intelligent Algorithms for Optimal Selection of Virtual Machine in Cloud Environment, Towards Enhance Healthcare Services. Proc. Intern. Conf. Advanced Intelligent Systems and Informatics. 2017:289—298.
38. Ghobaei-Arani M., Shamsi M., Rahmanian A.A. An Efficient Approach for Improving Virtual Machine Placement in Cloud Computing Environment. J. Exp. Theor. Artif. Intell. 2017;29(6):1149—171.
39. Saber T., Thorburn J., Murphy L., Ventresque A. VM Reassignment in Hybrid Clouds for Large Decentralized Companies: a Multi-objective Challenge. Future Gener. Comput. Syst. 2018;79:751—764.
40. Gao Y. e. a. A Multi-objective ant Colony System Algorithm for Virtual Machine Placement in Cloud Computing. J. Comput. Syst. Sci. 2013;79(8):1230—1242.
41. Ganesan T., Vasant P., Litvinchev I. Chaotic Simulator for Bilevel Optimization of Virtual Machine Placements in Cloud Computing. J. Operations Research Soc. China. 2022;10(4):703—723.
42. Deelman E. e. a. The Future of Scientific Workflows. Int. J. High Perform. Comput. Appl. 2018;32(1):159—175.
43. Chen W., Ferreira da Silva R., Livny M., Wenger K. Pegasus: a Workflow Management System for Science Automation. Futur. Gener. Comput. Syst. 2015;46:17—35.
44. Apache Airflow Documentation [Electron. Resurs] https://airflow.apache.org/ (Data Obrashcheniya 24.02.2023).
45. Hull D. e. a. Taverna: a Tool for Building and Running Workflows of Services. Nucleic Acids Research. 2006;34:729—732.
46. Altintas I. e. a. Kepler: an Extensible System for Design and Execution of Scientific Workflows. Proc. XVI Intern. Conf. Scientific and Statistical Database Management. 2004:423—424.
47. Kramer M., Wurz H.M., Altenhofen C. Executing Cyclic Scientific Workflows in the Cloud. J. Cloud Computing: Advances, Systems and Appl. 2021;10(25):1—26.
48. Toporkov V., Yemelyanov D., Toporkova A. Coordinated Global and Private Job-flow Scheduling in Grid Virtual Organizations. Simulation Modelling Practice and Theory. 2021;107(14):102228.
49. Toporkov V., Yemelyanov D. Micro-scheduling for Dependable Resources Allocation. Performance Evaluation Models for Distributed Service Networks. Studies in Systems, Decision and Control. 2021;343:81—105.
50. Toporkov V., Yemelyanov D., Grigorenko M. Optimization of Resources Allocation in High Performance Computing under Utilization Uncertainty. Proc. Intern. Conf. Computational Sci. 2021;14747:540—553.
51. Toporkov V., Yemelyanov D., Bulkhak A. Machine Learning-Based Scheduling and Resources Allocation in Distributed Computing. Proc. Internat. Conf. Computational Sci. 2022;13353:3—16
---
For citation: Toporkov V.V., Yemelyanov D.M., Bulkhak A.N. Providing Cloud Resources for Running Composite Applications in the Workflow-as-a-Service Paradigm. Bulletin of MPEI. 2023;5:156—168. (in Russian). DOI: 10.24160/1993-6982-2023-5-156-168
---
The work is executed at support: Russian Science Foundation (Grant No. 22-21-00372), https://rscf.ru/project/22-21-00372/
2. Existing Workflow Systems [Электрон. ресурс] https://s.apache.org/existing-workflow-systems (дата обращения 24.02.2023).
3. Rodriguez M.A., Buyya R. Scheduling Dynamic Workloads in Multi-tenant Scientific Workflow as a Service Platforms // Future Generation Computer Systems. 2018. V. 79. Pp. 739—750.
4. Hilman M.H., Rodriguez M.A., Buyya R. Workflow-as-a-Service Cloud Platform and Deployment of Bioinformatics Workflow Applications [Электрон. ресурс] https://www.researchgate.net/scientific-contributions/Maria-A-Rodriguez-2114894132 (дата обращения 24.02.2023).
5. Abrishami S., Naghibzadeh M., Epema D.H. Deadline-constrained Workflow Scheduling Algorithms for Infrastructure as a Service Clouds // Future Generation Computer Systems. 2013. V. 29(1). Pp. 158—169.
6. Calheiros R.N., Buyya R. Meeting Deadlines of Scientific Workflows in Public Clouds with Tasks Replication // IEEE Trans. Parallel Distrib. Syst. 2014. V. 25(7). Pp. 1787—1796.
7. Liu S., Ren K., Deng K., Song J. A Task Backfill-based Scientific Workflow Scheduling Strategy on Cloud Platform // Proc. Intern. Conf. Information Sci. and Technol. 2016. Pp. 105—110.
8. Deldari A., Naghibzadeh M., Abrishami S. CCA: a Deadline-constrained Workflow Scheduling Algorithm for Multicore Resources on the Cloud // J. Supercomput. 2017. V. 73(2). Pp. 756—781.
9. Maechling P. e. a. SCEC CyberShake Workflows — Аutomating Probabilistic Seismic Hazard Analysis Сalculations // Workflows for E-Science. N.-Y.: Springer, 2007. Pp. 143—163.
10. Deelman E. e. a. The Cost of Doing Science on the Cloud: the Montage Example // Proc. ACM/IEEE Conf. Supercomputing. 2008. P. 50.
11. Vockler J.-S. e. a. Experiences Using Cloud Computing for a Scientific Workflow Application // Proc. Intern. Workshop Sci. Cloud Computing. 2011. Pp. 15—24.
12. Malawski M., Juve G., Deelman E., Nabrzyski J. Algorithms for Cost- and Deadline-constrained Provisioning for Scientific Workflow Ensembles in IaaS Clouds // Future Generation Computer Syst. 2015. V. 48. Pp. 1—18.
13. Pietri I. e. a. Energy-constrained Provisioning for Scientific Workflow Ensembles // Proc. International Conf. Cloud and Green Computing. 2013. Pp. 34—41.
14. Jiang Q., Lee Y.C., Zomaya A.Y. Executing Large Scale Scientific Workflow Ensembles in Public Clouds // Proc. Intern. Conf. Parallel Proc. 2015. Pp. 520—529.
15. Bryk P., Malawski M., Juve G., Deelman E. Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds // J. Grid Comput. 2016. V. 14(2). Pp. 359—378.
16. Yu Z., Shi W. A Planner-guided Scheduling Strategy for Multiple Workflow Applications // Proc. Intern. Conf. Parallel Processing. 2008. Pp. 1—8.
17. Jiang H.-J. e. a. Scheduling Concurrent Workflows in HPC Cloud through Exploiting Schedule Gaps // Proc. Intern. Conf. Algorithms and Architectures for Parallel Processing. 2011. Pp. 282—293.
18. Stavrinides G.L., Karatza H.D. A Cost-effective and QoS-aware Approach to Scheduling Real-time Workflow Applications in PaaS and SaaS Clouds // Proc. Intern. Conf. Future Internet of Things and Cloud. 2015. Pp. 231—239.
19. Xu M., Cui L., Wang H., Bi Y. A Multiple QoS-constrained Scheduling Strategy of Multiple Workflows for Cloud Computing // Proc. IEEE Intern. Symp. Parallel and Distributed Processing with Appl. 2009. Pp. 629—634.
20. Chen W., Lee Y.C., Fekete A., Zomaya A.Y. Adaptive Multiple-workflow Scheduling with Task Rearrangement // J. Supercomput. 2015. V. 71(4). Pp. 1297—1317.
21. Zhou A.C., He B., Liu C. Monetary Cost Optimizations for Hosting Workflow-as-a-Service in IaaS Clouds // IEEE Trans. Cloud Comput. 2016. V. 4(1). Pp. 34—48.
22. Mao M., Humphrey M. Auto-scaling to Minimize Cost and Meet Application Deadlines in Cloud Workflows // Proc. Intern. Conf. High Performance Computing, Networking, Storage and Analysis. 2011. P. 49.
23. Shi J., Luo J., Dong F., Zhang J. A Budget and Deadline-aware Scientific Workflow Resource Provisioning and Scheduling Mechanism for Cloud // Proc. IEEE Intern. Conf. Computer Supported Cooperative Work in Design. 2014. Pp. 672—677.
24. Wang J. e. a. Workflow as a Service in the Cloud: Architecture and Scheduling Algorithms // Proc. Comput. Sci. 2014. V. 29. Pp. 546—556.
25. Gerlach W. e. a. Skyport: Container-based Execution Environment Management for Multi-cloud Scientific Workflows // Proc. Intern. Workshop on Data-Intensive Computing in the Clouds. 2014. Pp. 25—32.
26. Filgueira R. e. a. Asterism: Pegasus and Dispel4py Hybrid Workflows for Data-intensive Science // Proc. Intern. Workshop Data-Intensive Computing in the Cloud. 2016. Pp. 1—8.
27. Esteves S., Veiga L. Waas: Workflow-as-a-service for the Cloud with Scheduling of Continuous and Data-intensive Workflows // Comput. J. 2016. V. 59(3). Pp. 371—383.
28. Hashem I.A.T. e. a. The Rise of “Big Data” on Cloud Computing: Review and Open Research Issues // Inf. Syst. 2015. V. 47. Pp. 98—115.
29. Senyo P.K., Addae E., Boateng R. Cloud Computing Research: a Review of Research Themes, Frameworks, Methods and Future Research Directions // Int. J. Inf. Manag. 2018. V. 38(1). Pp. 128—139.
30. Stergiou C., Psannis K.E., Kim B.G., Gupta B. Secure Integration of IoT and Cloud Computing // Future Gener. Comput. Syst. 2018. V. 78. Pp. 964—975.
31. Kumar M.R.V., Raghunathan S. Heterogeneity and Thermal-aware Adaptive Heuristics for Energy Efficient Consolidation of Virtual Infrastructure Clouds // J. Comput. Syst. Sci. 2016. V. 82(2). Pp. 191—212.
32. Lopez-Pires F., Baran B. Virtual Machine Placement Literature Review [Электрон. ресурс] https://arxiv.org/abs/1506.01509 (дата обращения 24.02.2023).
33. Usmani Z., Singh S. A Survey of Virtual Machine Placement Techniques in a Cloud Data Center // Proc. Comput. Sci. 2016. V. 78. Pp. 491—498.
34. Liu X.F. e. a. An Energy Efficient ant Colony System for Virtual Machine Placement in Cloud Computing // IEEE Trans. Evol. Comput. 2016. V. 22(1). Pp. 113—128.
35. Abdel-Basset M., Abdle-Fatah L., Sangaiah A.K. An Improved Lévy Based Whale Optimization Algorithm for Bandwidth-efficient Virtual Machine Placement in Cloud Computing Environment // Cluster Computing. 2018. V. 22(1). Pp. 1—16.
36. Shabeera T.P., Kumar S.M., Salam S.M., Krishnan K.M. Optimizing VM Allocation and Data Placement for Data-intensive Applications in Cloud using ACO Metaheuristic Algorithm // Eng. Sci. Technol. Int. J. 2017. V. 20(2). Pp. 616—628.
37. Abdelaziz A. e. a. Intelligent Algorithms for Optimal Selection of Virtual Machine in Cloud Environment, Towards Enhance Healthcare Services // Proc. Intern. Conf. Advanced Intelligent Systems and Informatics. 2017. Pp. 289—298.
38. Ghobaei-Arani M., Shamsi M., Rahmanian A.A. An Efficient Approach for Improving Virtual Machine Placement in Cloud Computing Environment // J. Exp. Theor. Artif. Intell. 2017. V. 29(6). Pp. 1149—171.
39. Saber T., Thorburn J., Murphy L., Ventresque A. VM Reassignment in Hybrid Clouds for Large Decentralized Companies: a Multi-objective Challenge // Future Gener. Comput. Syst. 2018. V. 79. Pp. 751—764.
40. Gao Y. e. a. A Multi-objective ant Colony System Algorithm for Virtual Machine Placement in Cloud Computing // J. Comput. Syst. Sci. 2013. V. 79(8). Pp. 1230—1242.
41. Ganesan T., Vasant P., Litvinchev I. Chaotic Simulator for Bilevel Optimization of Virtual Machine Placements in Cloud Computing // J. Operations Research Soc. China. 2022. V. 10(4). Pp. 703—723.
42. Deelman E. e. a. The Future of Scientific Workflows // Int. J. High Perform. Comput. Appl. 2018. V. 32(1). Pp. 159—175.
43. Chen W., Ferreira da Silva R., Livny M., Wenger K. Pegasus: a Workflow Management System for Science Automation // Futur. Gener. Comput. Syst. 2015. V. 46. Pp. 17—35.
44. Apache Airflow Documentation [Электрон. ресурс] https://airflow.apache.org/ (дата обращения 24.02.2023).
45. Hull D. e. a. Taverna: a Tool for Building and Running Workflows of Services // Nucleic Acids Research. 2006. V. 34. Pp. 729—732.
46. Altintas I. e. a. Kepler: an Extensible System for Design and Execution of Scientific Workflows // Proc. XVI Intern. Conf. Scientific and Statistical Database Management. 2004. Pp. 423—424.
47. Kramer M., Wurz H.M., Altenhofen C. Executing Cyclic Scientific Workflows in the Cloud // J. Cloud Computing: Advances, Systems and Appl. 2021. V. 10(25). Pp. 1—26.
48. Toporkov V., Yemelyanov D., Toporkova A. Coordinated Global and Private Job-flow Scheduling in Grid Virtual Organizations // Simulation Modelling Practice and Theory. 2021. V. 107(14). P. 102228.
49. Toporkov V., Yemelyanov D. Micro-scheduling for Dependable Resources Allocation // Performance Evaluation Models for Distributed Service Networks. Studies in Systems, Decision and Control. 2021. V. 343. Pp. 81—105.
50. Toporkov V., Yemelyanov D., Grigorenko M. Optimization of Resources Allocation in High Performance Computing under Utilization Uncertainty // Proc. Intern. Conf. Computational Sci. 2021. V. 14747. Pp. 540—553.
51. Toporkov V., Yemelyanov D., Bulkhak A. Machine Learning-Based Scheduling and Resources Allocation in Distributed Computing // Proc. Internat. Conf. Computational Sci. 2022. V. 13353. Pp. 3—16.
---
Для цитирования: Топорков В.В., Емельянов Д.М., Булхак А.Н. Предоставление ресурсов облачных платформ для выполнения композитных приложений в парадигме «поток работ как сервис» // Вестник МЭИ. 2023. № 5. С. 156—168. DOI: 10.24160/1993-6982-2023-5-156-168
---
Работа выполнена при поддержке: Российского научного фонда (грант № 22-21-00372), https://rscf.ru/project/22-21-00372/
#
1. Workflows Community Summit [Electron. Resurs] https://workflowsri.org/summits/community (Data Obrashcheniya 24.02.2023).
2. Existing Workflow Systems [Electron. Resurs] https://s.apache.org/existing-workflow-systems (Data Obrashcheniya 24.02.2023).
3. Rodriguez M.A., Buyya R. Scheduling Dynamic Workloads in Multi-tenant Scientific Workflow as a Service Platforms. Future Generation Computer Systems. 2018;79:739—750.
4. Hilman M.H., Rodriguez M.A., Buyya R. Workflow-as-a-Service Cloud Platform and Deployment of Bioinformatics Workflow Applications [Electron. Resurs] https://www.researchgate.net/scientific-contributions/Maria-A-Rodriguez-2114894132 (Data Obrashcheniya 24.02.2023).
5. Abrishami S., Naghibzadeh M., Epema D.H. Deadline-constrained Workflow Scheduling Algorithms for Infrastructure as a Service Clouds. Future Generation Computer Systems. 2013;29(1):158—169.
6. Calheiros R.N., Buyya R. Meeting Deadlines of Scientific Workflows in Public Clouds with Tasks Replication. IEEE Trans. Parallel Distrib. Syst. 2014;25(7):1787—1796.
7. Liu S., Ren K., Deng K., Song J. A Task Backfill-based Scientific Workflow Scheduling Strategy on Cloud Platform. Proc. Intern. Conf. Information Sci. and Technol. 2016:105—110.
8. Deldari A., Naghibzadeh M., Abrishami S. CCA: a Deadline-constrained Workflow Scheduling Algorithm for Multicore Resources on the Cloud. J. Supercomput. 2017;73(2):756—781.
9. Maechling P. e. a. SCEC CyberShake Workflows — Аutomating Probabilistic Seismic Hazard Analysis Сalculations. Workflows for E-Science. N.-Y.: Springer, 2007:143—163.
10. Deelman E. e. a. The Cost of Doing Science on the Cloud: the Montage Example. Proc. ACM/IEEE Conf. Supercomputing. 2008:50.
11. Vockler J.-S. e. a. Experiences Using Cloud Computing for a Scientific Workflow Application. Proc. Intern. Workshop Sci. Cloud Computing. 2011:15—24.
12. Malawski M., Juve G., Deelman E., Nabrzyski J. Algorithms for Cost- and Deadline-constrained Provisioning for Scientific Workflow Ensembles in IaaS Clouds. Future Generation Computer Syst. 2015;48:1—18.
13. Pietri I. e. a. Energy-constrained Provisioning for Scientific Workflow Ensembles. Proc. International Conf. Cloud and Green Computing. 2013:34—41.
14. Jiang Q., Lee Y.C., Zomaya A.Y. Executing Large Scale Scientific Workflow Ensembles in Public Clouds. Proc. Intern. Conf. Parallel Proc. 2015:520—529.
15. Bryk P., Malawski M., Juve G., Deelman E. Storage-aware Algorithms for Scheduling of Workflow Ensembles in Clouds. J. Grid Comput. 2016;14(2):359—378.
16. Yu Z., Shi W. A Planner-guided Scheduling Strategy for Multiple Workflow Applications. Proc. Intern. Conf. Parallel Processing. 2008:1—8.
17. Jiang H.-J. e. a. Scheduling Concurrent Workflows in HPC Cloud through Exploiting Schedule Gaps. Proc. Intern. Conf. Algorithms and Architectures for Parallel Processing. 2011:282—293.
18. Stavrinides G.L., Karatza H.D. A Cost-effective and QoS-aware Approach to Scheduling Real-time Workflow Applications in PaaS and SaaS Clouds. Proc. Intern. Conf. Future Internet of Things and Cloud. 2015:231—239.
19. Xu M., Cui L., Wang H., Bi Y. A Multiple QoS-constrained Scheduling Strategy of Multiple Workflows for Cloud Computing. Proc. IEEE Intern. Symp. Parallel and Distributed Processing with Appl. 2009:629—634.
20. Chen W., Lee Y.C., Fekete A., Zomaya A.Y. Adaptive Multiple-workflow Scheduling with Task Rearrangement. J. Supercomput. 2015;71(4):1297—1317.
21. Zhou A.C., He B., Liu C. Monetary Cost Optimizations for Hosting Workflow-as-a-Service in IaaS Clouds. IEEE Trans. Cloud Comput. 2016;4(1):34—48.
22. Mao M., Humphrey M. Auto-scaling to Minimize Cost and Meet Application Deadlines in Cloud Workflows. Proc. Intern. Conf. High Performance Computing, Networking, Storage and Analysis. 2011:49.
23. Shi J., Luo J., Dong F., Zhang J. A Budget and Deadline-aware Scientific Workflow Resource Provisioning and Scheduling Mechanism for Cloud. Proc. IEEE Intern. Conf. Computer Supported Cooperative Work in Design. 2014:672—677.
24. Wang J. e. a. Workflow as a Service in the Cloud: Architecture and Scheduling Algorithms. Proc. Comput. Sci. 2014;29:546—556.
25. Gerlach W. e. a. Skyport: Container-based Execution Environment Management for Multi-cloud Scientific Workflows. Proc. Intern. Workshop on Data-Intensive Computing in the Clouds. 2014:25—32.
26. Filgueira R. e. a. Asterism: Pegasus and Dispel4py Hybrid Workflows for Data-intensive Science. Proc. Intern. Workshop Data-Intensive Computing in the Cloud. 2016:1—8.
27. Esteves S., Veiga L. Waas: Workflow-as-a-service for the Cloud with Scheduling of Continuous and Data-intensive Workflows. Comput. J. 2016;59(3):371—383.
28. Hashem I.A.T. e. a. The Rise of “Big Data” on Cloud Computing: Review and Open Research Issues. Inf. Syst. 2015;47:98—115.
29. Senyo P.K., Addae E., Boateng R. Cloud Computing Research: a Review of Research Themes, Frameworks, Methods and Future Research Directions. Int. J. Inf. Manag. 2018;38(1):128—139.
30. Stergiou C., Psannis K.E., Kim B.G., Gupta B. Secure Integration of IoT and Cloud Computing. Future Gener. Comput. Syst. 2018;78:964—975.
31. Kumar M.R.V., Raghunathan S. Heterogeneity and Thermal-aware Adaptive Heuristics for Energy Efficient Consolidation of Virtual Infrastructure Clouds. J. Comput. Syst. Sci. 2016;82(2):191—212.
32. Lopez-Pires F., Baran B. Virtual Machine Placement Literature Review [Electron. Resurs] https://arxiv.org/abs/1506.01509 (Data Obrashcheniya 24.02.2023).
33. Usmani Z., Singh S. A Survey of Virtual Machine Placement Techniques in a Cloud Data Center. Proc. Comput. Sci. 2016;78:491—498.
34. Liu X.F. e. a. An Energy Efficient ant Colony System for Virtual Machine Placement in Cloud Computing. IEEE Trans. Evol. Comput. 2016;22(1):113—128.
35. Abdel-Basset M., Abdle-Fatah L., Sangaiah A.K. An Improved Lévy Based Whale Optimization Algorithm for Bandwidth-efficient Virtual Machine Placement in Cloud Computing Environment. Cluster Computing. 2018;22(1):1—16.
36. Shabeera T.P., Kumar S.M., Salam S.M., Krishnan K.M. Optimizing VM Allocation and Data Placement for Data-intensive Applications in Cloud using ACO Metaheuristic Algorithm. Eng. Sci. Technol. Int. J. 2017;20(2):616—628.
37. Abdelaziz A. e. a. Intelligent Algorithms for Optimal Selection of Virtual Machine in Cloud Environment, Towards Enhance Healthcare Services. Proc. Intern. Conf. Advanced Intelligent Systems and Informatics. 2017:289—298.
38. Ghobaei-Arani M., Shamsi M., Rahmanian A.A. An Efficient Approach for Improving Virtual Machine Placement in Cloud Computing Environment. J. Exp. Theor. Artif. Intell. 2017;29(6):1149—171.
39. Saber T., Thorburn J., Murphy L., Ventresque A. VM Reassignment in Hybrid Clouds for Large Decentralized Companies: a Multi-objective Challenge. Future Gener. Comput. Syst. 2018;79:751—764.
40. Gao Y. e. a. A Multi-objective ant Colony System Algorithm for Virtual Machine Placement in Cloud Computing. J. Comput. Syst. Sci. 2013;79(8):1230—1242.
41. Ganesan T., Vasant P., Litvinchev I. Chaotic Simulator for Bilevel Optimization of Virtual Machine Placements in Cloud Computing. J. Operations Research Soc. China. 2022;10(4):703—723.
42. Deelman E. e. a. The Future of Scientific Workflows. Int. J. High Perform. Comput. Appl. 2018;32(1):159—175.
43. Chen W., Ferreira da Silva R., Livny M., Wenger K. Pegasus: a Workflow Management System for Science Automation. Futur. Gener. Comput. Syst. 2015;46:17—35.
44. Apache Airflow Documentation [Electron. Resurs] https://airflow.apache.org/ (Data Obrashcheniya 24.02.2023).
45. Hull D. e. a. Taverna: a Tool for Building and Running Workflows of Services. Nucleic Acids Research. 2006;34:729—732.
46. Altintas I. e. a. Kepler: an Extensible System for Design and Execution of Scientific Workflows. Proc. XVI Intern. Conf. Scientific and Statistical Database Management. 2004:423—424.
47. Kramer M., Wurz H.M., Altenhofen C. Executing Cyclic Scientific Workflows in the Cloud. J. Cloud Computing: Advances, Systems and Appl. 2021;10(25):1—26.
48. Toporkov V., Yemelyanov D., Toporkova A. Coordinated Global and Private Job-flow Scheduling in Grid Virtual Organizations. Simulation Modelling Practice and Theory. 2021;107(14):102228.
49. Toporkov V., Yemelyanov D. Micro-scheduling for Dependable Resources Allocation. Performance Evaluation Models for Distributed Service Networks. Studies in Systems, Decision and Control. 2021;343:81—105.
50. Toporkov V., Yemelyanov D., Grigorenko M. Optimization of Resources Allocation in High Performance Computing under Utilization Uncertainty. Proc. Intern. Conf. Computational Sci. 2021;14747:540—553.
51. Toporkov V., Yemelyanov D., Bulkhak A. Machine Learning-Based Scheduling and Resources Allocation in Distributed Computing. Proc. Internat. Conf. Computational Sci. 2022;13353:3—16
---
For citation: Toporkov V.V., Yemelyanov D.M., Bulkhak A.N. Providing Cloud Resources for Running Composite Applications in the Workflow-as-a-Service Paradigm. Bulletin of MPEI. 2023;5:156—168. (in Russian). DOI: 10.24160/1993-6982-2023-5-156-168
---
The work is executed at support: Russian Science Foundation (Grant No. 22-21-00372), https://rscf.ru/project/22-21-00372/
Опубликован
2023-06-06
Выпуск
Раздел
Вычислительные системы и их элементы (технические науки) (2.3.2.)
