Sketch Colorization Based on Generative-Adversarial Neural Networks
DOI:
https://doi.org/10.24160/1993-6982-2022-1-120-129Keywords:
sketch colorization, generative adversarial neural networks, deep learningAbstract
Generative adversarial neural networks (GAN) successfully perform automatic colorization of character sketches. Such models can be further improved in such aspects as increasing the throughput, improving the colorization quality, and reducing the model size. Steps aimed at improving the colorization quality are taken. Known solutions are reviewed, and an initial GAN model with eight blocks in the generator encoder and decoder is developed and trained based on the review results. The second GAN model is obtained on the basis of the first one by including residual blocks in the generator encoder blocks with simultaneously using the attention blocks and residual blocks in the generator decoder. The third GAN model has been developed based on the second one: one block is added to the generator encoder and decoder. All models, including the initial and modified ones, have been trained on the same data set. The models have been trained either with or without using additional information about the image color (the color palette of the reference image or color hint labels). The trained models are evaluated with respect to the quality of the images they generate (colored sketches), determined by the Frechet Inception Distance metric. All modified GAN models generate images with quality superior to that of the initial model.
References
1. Анатомический атлас человека в картинках [Электрон. ресурс] www.medikweb.ru/anatom_atlas/ (дата обращения 01.03.2021).
2. Геологические карты и атласы [Электрон. ресурс] www.jurassic.ru/maps.html (дата обращения 01.03.2021).
3. Yifan L., Zengchang Q., Zhenbo L., Hua W. Auto-painter: Cartoon Image Generation from Sketch by Using Conditional Generative Adversarial Networks [Электрон. ресурс] www.arxiv.org/pdf/1705.01908.pdf (дата обращения 01.03.2021).
4. Ronneberger O., Fischer P., Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. [Электрон. ресурс] www.arxiv.org/pdf/1505.04597.pdf (дата обращения 01.03.2021).
5. Yuanzheng C., Xinzhu M. User-guided Deep Anime Line Art Colorization with Conditional Adversarial Networks [Электрон. ресурс] www.arxiv.org/pdf/1808.03240.pdf (дата обращения 01.03.2021).
6. Xie S. e. a. Aggregated Residual Transformations for Deep Neural Networks [Электрон. ресурс] www.arxiv.org/pdf/1611.05431.pdf (дата обращения 01.03.2021).
7. Kaiming H., Xiangyu Z., Shaoqing R. Deep Residual Learning for Image Recognition. [Электрон. ресурс] www.arxiv.org/pdf/1512.03385.pdf (дата обращения 01.03.2021).
8. Gang L., Xin C., Yanzhong H. Anime Sketch Coloring with Swish-gated Residual U-net and Spectrally Normalized GAN // Eng. Letters. 2019. V. 27. Iss. 3. Pp. 1—7.
9. Ye R. e. a. Interactive Anime Sketch Colorization with Style Consistency via a Deep Residual Neural Network // Proc. Intern. Conf. Technologies and Appl. Artificial Intelligence. Kaohsiung, 2019. Pp. 1—5.
10. Lvmin Z., Yi J., Xin L. Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN [Электрон. ресурс] www.arxiv.org/abs/1706.03319 (дата обращения 01.03.2021).
11. Lvmin Z. e. a. Two-stage Sketch Colorization [Электрон. ресурс] www.ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8944253 (дата обращения 01.03.2021).
12. Ozan Oktay e. a. Attention U-Net: Learning Where to Look for the Pancreas [Электрон. ресурс] www.arxiv.org/pdf/1804.03999.pdf (дата обращения 01.03.2021).
13. Fei W. e. a. Residual Attention Network for Image Classification [Электрон. ресурс] www.arxiv.org/pdf/1704.06904.pdf (дата обращения 01.03.2021).
14. Masashi A., Yuichi S., Yasuyuki T. Do You Like Sclera? Sclera-region Detection and Colorization for Anime Character Line Drawings // Intern. J. Networked and Distributed Computing. 2019. V. 7(3). Pp. 113—120.
15. Kataoka Y., Matsubara T., Uehara K. Automatic Manga Colorization with Color Style by Generative Adversarial Nets // Proc. XVIII IEEE/ACIS Intern. Conf. Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, Kanazawa, 2017. Pp. 495—499.
16. Junsoo L., Eungyeup K., Yunsung L. Reference-based Sketch Image Colorization using Augmented-self Reference and Dense Semantic Correspondence www.arxiv.org/pdf/2005.05207 (дата обращения 01.03.2021).
17. Ren H., Li J., Gao N. Automatic Sketch Colorization with Tandem Conditional Adversarial Networks // Proc. XI Intern. Symp. Computational Intelligence and Design. Hangzhou, 2018. Pp. 11—15.
18. Taebum K. Anime Sketch Colorization Pair [Электрон. ресурс] www.kaggle.com/ktaebum/anime-sketch-colorization-pair (дата обращения 01.03.2021).
19. Yuxin W., Kaiming H. Group Normlization [Электрон. ресурс] www.arxiv.org/pdf/1803.08494.pdf (дата обращения 01.03.2021).
20. Heusel M., Ramsauer H., Unterthiner T., Nessler B., Hochreiter S. GANs Trained by a Two Time-scale Update Rule Converge to a Local Nash Equilibrium [Электрон. ресурс] www.arxiv.org/pdf/1706.08500.pdf (дата обращения 01.03.2021).
---
Для цитирования: Бартеньев О.В., Салахутдинов Э.Р. Колоризация эскизов на основе генеративно-состязательных нейронных сетей // Вестник МЭИ. 2022. № 1. С. 120—129. DOI: 10.24160/1993-6982-2022-1-120-129.
#
1. Anatomicheskiy Atlas Cheloveka v Kartinkakh [Elektron. Resurs] www.medikweb.ru/anatom_atlas/ (Data Obrashcheniya 01.03.2021). (in Russian).
2. Geologicheskie Karty i Atlasy [Elektron. Resurs] www.jurassic.ru/maps.html (Data Obrashcheniya 01.03.2021). (in Russian).
3. Yifan L., Zengchang Q., Zhenbo L., Hua W. Auto-painter: Cartoon Image Generation from Sketch by Using Conditional Generative Adversarial Networks [Elektron. Resurs] www.arxiv.org/pdf/1705.01908.pdf (Data Obrashcheniya 01.03.2021).
4. Ronneberger O., Fischer P., Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. [Elektron. Resurs] www.arxiv.org/pdf/1505.04597.pdf (Data Obrashcheniya 01.03.2021).
5. Yuanzheng C., Xinzhu M. User-guided Deep Anime Line Art Colorization with Conditional Adversarial Networks [Elektron. Resurs] www.arxiv.org/pdf/1808.03240.pdf (Data Obrashcheniya 01.03.2021).
6. Xie S. e. a. Aggregated Residual Transformations for Deep Neural Networks [Elektron. Resurs] www.arxiv.org/pdf/1611.05431.pdf (Data Obrashcheniya 01.03.2021).
7. Kaiming H., Xiangyu Z., Shaoqing R. Deep Residual Learning for Image Recognition. [Elektron. Resurs] www.arxiv.org/pdf/1512.03385.pdf (Data Obrashcheniya 01.03.2021).
8. Gang L., Xin C., Yanzhong H. Anime Sketch Coloring with Swish-gated Residual U-net and Spectrally Normalized GAN. Eng. Letters. 2019;27;3:1—7.
9. Ye R. e. a. Interactive Anime Sketch Colorization with Style Consistency via a Deep Residual Neural Network. Proc. Intern. Conf. Technologies and Appl. Artificial Intelligence. Kaohsiung, 2019:1—5.
10. Lvmin Z., Yi J., Xin L. Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN [Elektron. Resurs] www.arxiv.org/abs/1706.03319 (Data Obrashcheniya 01.03.2021).
11. Lvmin Z. e. a. Two-stage Sketch Colorization [Elektron. Resurs] www.ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8944253 (Data Obrashcheniya 01.03.2021).
12. Ozan Oktay e. a. Attention U-Net: Learning Where to Look for the Pancreas [Elektron. Resurs] www.arxiv.org/pdf/1804.03999.pdf (Data Obrashcheniya 01.03.2021).
13. Fei W. e. a. Residual Attention Network for Image Classification [Elektron. Resurs] www.arxiv.org/pdf/1704.06904.pdf (Data Obrashcheniya 01.03.2021).
14. Masashi A., Yuichi S., Yasuyuki T. Do You Like Sclera? Sclera-region Detection and Colorization for Anime Character Line Drawings. Intern. J. Networked and Distributed Computing. 2019;7(3):113—120.
15. Kataoka Y., Matsubara T., Uehara K. Automatic Manga Colorization with Color Style by Generative Adversarial Nets. Proc. XVIII IEEE/ACIS Intern. Conf. Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, Kanazawa, 2017:495—499.
16. Junsoo L., Eungyeup K., Yunsung L. Reference-based Sketch Image Colorization using Augmented-self Reference and Dense Semantic Correspondence www.arxiv.org/pdf/2005.05207 (Data Obrashcheniya 01.03.2021).
17. Ren H., Li J., Gao N. Automatic Sketch Colorization with Tandem Conditional Adversarial Networks. Proc. XI Intern. Symp. Computational Intelligence and Design. Hangzhou, 2018:11—15.
18. Taebum K. Anime Sketch Colorization Pair [Elektron. Resurs] www.kaggle.com/ktaebum/anime-sketch-colorization-pair (Data Obrashcheniya 01.03.2021).
19. Yuxin W., Kaiming H. Group Normlization [Elektron. Resurs] www.arxiv.org/pdf/1803.08494.pdf (Data Obrashcheniya 01.03.2021).
20. Heusel M., Ramsauer H., Unterthiner T., Nessler B., Hochreiter S. GANs Trained by a Two Time-scale Update Rule Converge to a Local Nash Equilibrium [Elektron. Resurs] www.arxiv.org/pdf/1706.08500.pdf (Data Obrashcheniya 01.03.2021).
---
For citation: Bartenyev O.V., Salakhutdinov E.R. Sketch Colorization Based on Generative-Adversarial Neural Networks. Bulletin of MPEI. 2022;1:120—129. (in Russian). DOI: 10.24160/1993-6982-2022-1-120-129.
2. Геологические карты и атласы [Электрон. ресурс] www.jurassic.ru/maps.html (дата обращения 01.03.2021).
3. Yifan L., Zengchang Q., Zhenbo L., Hua W. Auto-painter: Cartoon Image Generation from Sketch by Using Conditional Generative Adversarial Networks [Электрон. ресурс] www.arxiv.org/pdf/1705.01908.pdf (дата обращения 01.03.2021).
4. Ronneberger O., Fischer P., Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. [Электрон. ресурс] www.arxiv.org/pdf/1505.04597.pdf (дата обращения 01.03.2021).
5. Yuanzheng C., Xinzhu M. User-guided Deep Anime Line Art Colorization with Conditional Adversarial Networks [Электрон. ресурс] www.arxiv.org/pdf/1808.03240.pdf (дата обращения 01.03.2021).
6. Xie S. e. a. Aggregated Residual Transformations for Deep Neural Networks [Электрон. ресурс] www.arxiv.org/pdf/1611.05431.pdf (дата обращения 01.03.2021).
7. Kaiming H., Xiangyu Z., Shaoqing R. Deep Residual Learning for Image Recognition. [Электрон. ресурс] www.arxiv.org/pdf/1512.03385.pdf (дата обращения 01.03.2021).
8. Gang L., Xin C., Yanzhong H. Anime Sketch Coloring with Swish-gated Residual U-net and Spectrally Normalized GAN // Eng. Letters. 2019. V. 27. Iss. 3. Pp. 1—7.
9. Ye R. e. a. Interactive Anime Sketch Colorization with Style Consistency via a Deep Residual Neural Network // Proc. Intern. Conf. Technologies and Appl. Artificial Intelligence. Kaohsiung, 2019. Pp. 1—5.
10. Lvmin Z., Yi J., Xin L. Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN [Электрон. ресурс] www.arxiv.org/abs/1706.03319 (дата обращения 01.03.2021).
11. Lvmin Z. e. a. Two-stage Sketch Colorization [Электрон. ресурс] www.ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8944253 (дата обращения 01.03.2021).
12. Ozan Oktay e. a. Attention U-Net: Learning Where to Look for the Pancreas [Электрон. ресурс] www.arxiv.org/pdf/1804.03999.pdf (дата обращения 01.03.2021).
13. Fei W. e. a. Residual Attention Network for Image Classification [Электрон. ресурс] www.arxiv.org/pdf/1704.06904.pdf (дата обращения 01.03.2021).
14. Masashi A., Yuichi S., Yasuyuki T. Do You Like Sclera? Sclera-region Detection and Colorization for Anime Character Line Drawings // Intern. J. Networked and Distributed Computing. 2019. V. 7(3). Pp. 113—120.
15. Kataoka Y., Matsubara T., Uehara K. Automatic Manga Colorization with Color Style by Generative Adversarial Nets // Proc. XVIII IEEE/ACIS Intern. Conf. Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, Kanazawa, 2017. Pp. 495—499.
16. Junsoo L., Eungyeup K., Yunsung L. Reference-based Sketch Image Colorization using Augmented-self Reference and Dense Semantic Correspondence www.arxiv.org/pdf/2005.05207 (дата обращения 01.03.2021).
17. Ren H., Li J., Gao N. Automatic Sketch Colorization with Tandem Conditional Adversarial Networks // Proc. XI Intern. Symp. Computational Intelligence and Design. Hangzhou, 2018. Pp. 11—15.
18. Taebum K. Anime Sketch Colorization Pair [Электрон. ресурс] www.kaggle.com/ktaebum/anime-sketch-colorization-pair (дата обращения 01.03.2021).
19. Yuxin W., Kaiming H. Group Normlization [Электрон. ресурс] www.arxiv.org/pdf/1803.08494.pdf (дата обращения 01.03.2021).
20. Heusel M., Ramsauer H., Unterthiner T., Nessler B., Hochreiter S. GANs Trained by a Two Time-scale Update Rule Converge to a Local Nash Equilibrium [Электрон. ресурс] www.arxiv.org/pdf/1706.08500.pdf (дата обращения 01.03.2021).
---
Для цитирования: Бартеньев О.В., Салахутдинов Э.Р. Колоризация эскизов на основе генеративно-состязательных нейронных сетей // Вестник МЭИ. 2022. № 1. С. 120—129. DOI: 10.24160/1993-6982-2022-1-120-129.
#
1. Anatomicheskiy Atlas Cheloveka v Kartinkakh [Elektron. Resurs] www.medikweb.ru/anatom_atlas/ (Data Obrashcheniya 01.03.2021). (in Russian).
2. Geologicheskie Karty i Atlasy [Elektron. Resurs] www.jurassic.ru/maps.html (Data Obrashcheniya 01.03.2021). (in Russian).
3. Yifan L., Zengchang Q., Zhenbo L., Hua W. Auto-painter: Cartoon Image Generation from Sketch by Using Conditional Generative Adversarial Networks [Elektron. Resurs] www.arxiv.org/pdf/1705.01908.pdf (Data Obrashcheniya 01.03.2021).
4. Ronneberger O., Fischer P., Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. [Elektron. Resurs] www.arxiv.org/pdf/1505.04597.pdf (Data Obrashcheniya 01.03.2021).
5. Yuanzheng C., Xinzhu M. User-guided Deep Anime Line Art Colorization with Conditional Adversarial Networks [Elektron. Resurs] www.arxiv.org/pdf/1808.03240.pdf (Data Obrashcheniya 01.03.2021).
6. Xie S. e. a. Aggregated Residual Transformations for Deep Neural Networks [Elektron. Resurs] www.arxiv.org/pdf/1611.05431.pdf (Data Obrashcheniya 01.03.2021).
7. Kaiming H., Xiangyu Z., Shaoqing R. Deep Residual Learning for Image Recognition. [Elektron. Resurs] www.arxiv.org/pdf/1512.03385.pdf (Data Obrashcheniya 01.03.2021).
8. Gang L., Xin C., Yanzhong H. Anime Sketch Coloring with Swish-gated Residual U-net and Spectrally Normalized GAN. Eng. Letters. 2019;27;3:1—7.
9. Ye R. e. a. Interactive Anime Sketch Colorization with Style Consistency via a Deep Residual Neural Network. Proc. Intern. Conf. Technologies and Appl. Artificial Intelligence. Kaohsiung, 2019:1—5.
10. Lvmin Z., Yi J., Xin L. Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN [Elektron. Resurs] www.arxiv.org/abs/1706.03319 (Data Obrashcheniya 01.03.2021).
11. Lvmin Z. e. a. Two-stage Sketch Colorization [Elektron. Resurs] www.ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8944253 (Data Obrashcheniya 01.03.2021).
12. Ozan Oktay e. a. Attention U-Net: Learning Where to Look for the Pancreas [Elektron. Resurs] www.arxiv.org/pdf/1804.03999.pdf (Data Obrashcheniya 01.03.2021).
13. Fei W. e. a. Residual Attention Network for Image Classification [Elektron. Resurs] www.arxiv.org/pdf/1704.06904.pdf (Data Obrashcheniya 01.03.2021).
14. Masashi A., Yuichi S., Yasuyuki T. Do You Like Sclera? Sclera-region Detection and Colorization for Anime Character Line Drawings. Intern. J. Networked and Distributed Computing. 2019;7(3):113—120.
15. Kataoka Y., Matsubara T., Uehara K. Automatic Manga Colorization with Color Style by Generative Adversarial Nets. Proc. XVIII IEEE/ACIS Intern. Conf. Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, Kanazawa, 2017:495—499.
16. Junsoo L., Eungyeup K., Yunsung L. Reference-based Sketch Image Colorization using Augmented-self Reference and Dense Semantic Correspondence www.arxiv.org/pdf/2005.05207 (Data Obrashcheniya 01.03.2021).
17. Ren H., Li J., Gao N. Automatic Sketch Colorization with Tandem Conditional Adversarial Networks. Proc. XI Intern. Symp. Computational Intelligence and Design. Hangzhou, 2018:11—15.
18. Taebum K. Anime Sketch Colorization Pair [Elektron. Resurs] www.kaggle.com/ktaebum/anime-sketch-colorization-pair (Data Obrashcheniya 01.03.2021).
19. Yuxin W., Kaiming H. Group Normlization [Elektron. Resurs] www.arxiv.org/pdf/1803.08494.pdf (Data Obrashcheniya 01.03.2021).
20. Heusel M., Ramsauer H., Unterthiner T., Nessler B., Hochreiter S. GANs Trained by a Two Time-scale Update Rule Converge to a Local Nash Equilibrium [Elektron. Resurs] www.arxiv.org/pdf/1706.08500.pdf (Data Obrashcheniya 01.03.2021).
---
For citation: Bartenyev O.V., Salakhutdinov E.R. Sketch Colorization Based on Generative-Adversarial Neural Networks. Bulletin of MPEI. 2022;1:120—129. (in Russian). DOI: 10.24160/1993-6982-2022-1-120-129.
Downloads
Published
2021-05-10
Issue
Section
Mathematical and Software Support of Computing Machines, Complexes and Computer (05.13.11)
