Оценка эффективности методов токенизации текста
Ключевые слова:
токен, словарь, классификатор, нейронная сеть, набор данных
Аннотация
Рассмотрены методы токенизации текста, использующие словарь токенов и обходящиеся без него. Приведены алгоритмы токенизации и формирования словарей токенов. Оценка эффективности методов токенизации выполнена на задачах разметки текста по частям речи (РТЧ), классификации документов (КД) и расстановки знаков препинания (РЗП). Задачи РТЧ и РЗП сведены к задаче классификации n-грамм (соответственно, n = 3 и n = 5). В качестве классификатора взята нейронная сеть — многослойный перцептрон, принимающий на входе в случае КД векторные представления документов, либо, если решается задача РТЧ или РЗП, — векторные представления n-грамм. Векторные представления сформированы с использованием word2vec-модели. Проводимые эксперименты позволяют оценить эффективность семи методов токенизации. Три из них, образующие группу G1, переводят элементы текста в токены, выбирая их из предварительно созданных словарей. Словари построены с употреблением методов BPE, BPE-dropout и Kudo. Другие четыре, входящие в группу G2, выполняют токенизацию, не обращаясь к словарю токенов — это методы, формирующие токены в виде исходных словоформ; начальных форм словоформ; основ словоформ; основ и аффиксов словоформ. Словари и векторные представления токенов формируются процедурами Python по корпусу текста, объединяющему корпусы задач КД и РЗП. Используемые корпусы содержат тексты на русском языке. В решаемых задачах примененные методы токенизации демонстрируют разную эффективность: рост F1 при употреблении лучшего метода группы G2 вместо лучшего метода группы G1 составил в задаче РТЧ — 4,04%; КД — 2,29%; РЗП — 3,34%.
Литература
1. Keras [Электрон. ресурс] https://keras.io/ (дата обращения 01.02.2023).
2. Бартеньев О.В. Сравнительная оценка эффективности моделей текста в задаче классификации документов // Вестник МЭИ. 2021. № 5. С. 117—127.
3. Chi Z. e. a. XLM-E: Cross-lingual Language Model Pre-training via ELECTRA // Proceedings 60th Annual Meeting of the Association for Computational Linguistics. Dublin, 2022. V. 1. Pp. 6170—6182.
4. Arora G. iNLTK: Natural Language Toolkit for Indic Languages // Proc. Second Workshop for NLP Open Source Software (NLP-OSS). 2020. Pp. 66—71.
5. Kudo T., Richardson J. SentencePiece: a Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. // Proc. Conf. on Empirical Methods in Natural Language Processing: System Demonstrations. Brussels, 2018. Pp. 66—71.
6. Mikolov T. e. a. Distributed Representations of Words and Phrases and their Compositionality // Neural Information Processing Systems. 2013. V. 26. Pp. 1—9.
7. Le Q., Mikolov T. Distributed Representations of Sentences and Documents // Proc. XXXI Intern. Conf. Machine Learning. Beijing, 2014. Pp. 1188—1196.
8. Mikolov T. e. a. Advances in Pre-training Distributed Word Representations // Proceedings of LREC. Miyazaki, 2018. Pp. 52—55.
9. Pennington J., Socher R., Manning C. Glove: Global Vectors for Word Representation. // Proc. Conf. Empirical Methods in Natural Language Processing. 2014. Pp. 1532—1543.
10. Adesam Y., Berdicevskis A. Part-of-speech Tagging of Swedish Texts in the Neural Era // Proceedings XXIII Nordic Conf. Computational Linguistics (NoDaLiDa). Reykjavik, 2021. Pp. 200—209.
11. Haldar G., Mittal A., Gupta P. DSC-IITISM at FinCausal 2021: Combining POS Tagging with Attention-based Contextual Representations for Identifying Causal Relationships in Financial Documents // Proc. III Financial Narrative Processing Workshop. Lancaster, 2021. Pp. 49—53.
12. Alshammeri M., Atwell E., Alsalka M.A. Classifying Verses of the Quran using Doc2vec // Proc. XVIII Intern. Conf. Natural Language Processing. Silchar, 2021. Pp. 284—288.
13. Schmitt M. e. a. Joint Aspect and Polarity Classification for Aspect-based Sentiment Analysis with End-to-end Neural Networks 2018. In // Proc. Conf. Empirical Methods in Natural Language Processing. Brussels, 2018. Pp. 1109—1114.
14. Ebadulla D.M. e. a. A Comparative Study on Language Models for the Kannada Language // Proceedings IV Intern. Conf. Natural Language and Speech Processing. Trento, 2021. Pp. 280—284.
15. WordPiece Tokenization. [Электрон. ресурс] https://huggingface.co/course/chapter6/6?fw=pt (дата обращения 01.02.2023).
16. Alam T., Khan A., Alam F. Punctuation Restoration using Transformer Models for High-and Low-resource Languages // Proc. VI Workshop on Noisy User-generated Text. 2020. Pp. 132—142.
17. Pogoda M., Walkowiak T. Comprehensive Punctuation Restoration for English and Polish // Proc. Findings of the Association for Computational Linguistics: EMNLP. Punta Cana, 2021. Pp. 4610—4619.
18. Nagy A., Bial B., Ács J. Automatic Punctuation Restoration with BERT Models [Электрон. ресурс] https://arxiv.org/pdf/2101.07343v1.pdf#page=1 (дата обращения 01.02.2023).
19. Devlin J. e. a. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. - 2019. In // Proc. Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, 2019. V. 1. Pp. 4171—4186.
20. Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units 2016. In // Proc. 54th Annual Meeting of the Association for Computational Linguistics. Berlin, 2016. V. 1. Pp. 1715—1725.
21. Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates // Proc. 56th Annual Meeting of the Association for Computational Linguistics. Melbourne, 2018. V. 1. Pp. 66—75.
22. Provilkov I., Emelianenko D., Voita E. BPE-Dropout: Simple and Effective Subword Regularization // Proc. 58th Annual Meeting of the Association for Computational Linguistics. 2020. Pp. 1882—1892.
23. Papineni K. et. al. BLEU: a Method for Automatic Evaluation of Machine Translation. - 2002.In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
24. IWSLT 15 English-Vietnamese Dataset. - 2015. [Электрон. ресурс] https://metatext.io/datasets/iwslt-15-english-vietnamese (дата обращения 01.02.2023).
25. Vaswani A. et. al. Attention is all you need. - 2017. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
26. BERT in DeepPavlov. [Электронный ресурс] URL: http://docs.deeppavlov.ai/en/master/features/models/bert.html (дата обращения 01.02.2022).
27. Бартеньев О. В. Автоматическая расстановка знаков препинания с помощью нейронных сетей // Вестник МЭИ. 2022. № 6. С. 146—159. DOI: 10.24160/1993-6982-2022-6-146-159.
28. Бабайцева В. В. Русский язык. Теория. – М.: Дрофа, 1998. – 432 с.
29. Основа слова. [Электрон. ресурс] https://studopedia.ru/6_82986_osnova-slova-tipi-osnov.html (дата обращения 01.02.2023).
30. Морфологический анализатор pymorphy2. [Электрон. ресурс] https://pymorphy2.readthedocs.io/en/stable/ (дата обращения 01.02.2023).
31. Открытый корпус. [Электрон. ресурс] http://opencorpora.org/ (дата обращения 01.02.2023).
32. NLTK Documentation. nltk.stem.snowball module. [Электрон. ресурс] https://www.nltk.org/api/nltk.stem.snowball.html (дата обращения 01.02.2023).
33. Gage P. A New Algorithm for Data Compression. – 1994. [Электрон. ресурс] https://www.derczynski.com/papers/archive/BPE_Gage.pdf (дата обращения 01.02.2023).
34. BERT base model (uncased). [Электрон. ресурс] https://huggingface.co/bert-base-uncased (дата обращения 01.02.2023).
35. Expectation–maximization algorithm. [Электрон. ресурс] https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm (дата обращения 01.02.2023).
36. SentencePiece Python Wrapper. [Электрон. ресурс] https://pypi.org/project/sentencepiece/ (дата обращения 01.02.2023).
37. Набор данных для разметки текста по частям речи. [Электрон. ресурс] https://disk.yandex.ru/d/op1mwroFBMyKcw (дата обращения 01.02.2023).
38. Набор данных для классификации документов. [Электрон. ресурс] https://www.kaggle.com/datasets/olegbartenyev/doc-cls (дата обращения 01.02.2023).
39. Корпус для создания словаря токенов. [Электрон. ресурс] https://disk.yandex.ru/d/uFPs-eRW6Kaa8Q (дата обращения 01.02.2023).
40. Scikit-learn. [Электронн. ресурс] http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html\ (дата обращения 01.02.2023).
41. Бартеньев О. В. Параметры, влияющие на эффективность нейронной сети, созданной средствами Keras. [Электронн. ресурс] http://www.100byte.ru/python/factors/factors.html (дата обращения 01.02.2023).
42. BBC Dataset [Электрон. ресурс] http://www.mlg.ucd.ie/datasets/bbc.html (дата обращения 01.02.2023).
---
Для цитирования: Бартеньев О.В. Оценка эффективности методов токенизации текста // Вестник МЭИ. 2023. № 6. С. 144—156. DOI: 10.24160/1993-6982-2023-6-144-156
#
1. Keras [Elektron. Resurs] https://keras.io/ (Data Obrashcheniya 01.02.2023).
2. Barten'ev O.V. Sravnitel'naya Otsenka Effektivnosti Modeley Teksta v Zadache Klassifikatsii Dokumentov. Vestnik MEI. 2021;5:117—127. (in Russian).
3. Chi Z. e. a. XLM-E: Cross-lingual Language Model Pre-training via ELECTRA. Proceedings 60th Annual Meeting of the Association for Computational Linguistics. Dublin, 2022;1:6170—6182.
4. Arora G. iNLTK: Natural Language Toolkit for Indic Languages. Proc. Second Workshop for NLP Open Source Software (NLP-OSS). 2020:66—71.
5. Kudo T., Richardson J. SentencePiece: a Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing.. Proc. Conf. on Empirical Methods in Natural Language Processing: System Demonstrations. Brussels, 2018:66—71.
6. Mikolov T. e. a. Distributed Representations of Words and Phrases and their Compositionality. Neural Information Processing Systems. 2013;26:1—9.
7. Le Q., Mikolov T. Distributed Representations of Sentences and Documents. Proc. XXXI Intern. Conf. Machine Learning. Beijing, 2014:1188—1196.
8. Mikolov T. e. a. Advances in Pre-training Distributed Word Representations. Proceedings of LREC. Miyazaki, 2018:52—55.
9. Pennington J., Socher R., Manning C. Glove: Global Vectors for Word Representation.. Proc. Conf. Empirical Methods in Natural Language Processing. 2014:1532—1543.
10. Adesam Y., Berdicevskis A. Part-of-speech Tagging of Swedish Texts in the Neural Era. Proceedings XXIII Nordic Conf. Computational Linguistics (NoDaLiDa). Reykjavik, 2021:200—209.
11. Haldar G., Mittal A., Gupta P. DSC-IITISM at FinCausal 2021: Combining POS Tagging with Attention-based Contextual Representations for Identifying Causal Relationships in Financial Documents. Proc. III Financial Narrative Processing Workshop. Lancaster, 2021: 49—53.
12. Alshammeri M., Atwell E., Alsalka M.A. Classifying Verses of the Quran using Doc2vec. Proc. XVIII Intern. Conf. Natural Language Processing. Silchar, 2021:284—288.
13. Schmitt M. e. a. Joint Aspect and Polarity Classification for Aspect-based Sentiment Analysis with End-to-end Neural Networks 2018. In. Proc. Conf. Empirical Methods in Natural Language Processing. Brussels, 2018:1109—1114.
14. Ebadulla D.M. e. a. A Comparative Study on Language Models for the Kannada Language. Proceedings IV Intern. Conf. Natural Language and Speech Processing. Trento, 2021:280—284.
15. WordPiece Tokenization. [Elektron. Resurs] https://huggingface.co/course/chapter6/6?fw=pt (Data Obrashcheniya 01.02.2023).
16. Alam T., Khan A., Alam F. Punctuation Restoration using Transformer Models for High-and Low-resource Languages. Proc. VI Workshop on Noisy User-generated Text. 2020:132—142.
17. Pogoda M., Walkowiak T. Comprehensive Punctuation Restoration for English and Polish. Proc. Findings of the Association for Computational Linguistics: EMNLP. Punta Cana, 2021:4610—4619.
18. Nagy A., Bial B., Ács J. Automatic Punctuation Restoration with BERT Models [Elektron. Resurs] https://arxiv.org/pdf/2101.07343v1.pdf#page=1 (Data Obrashcheniya 01.02.2023).
19. Devlin J. e. a. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. - 2019. In. Proc. Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, 2019;1:4171—4186.
20. Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units 2016. In. Proc. 54th Annual Meeting of the Association for Computational Linguistics. Berlin, 2016;1:1715—1725.
21. Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. Proc. 56th Annual Meeting of the Association for Computational Linguistics. Melbourne, 2018;1:66—75.
22. Provilkov I., Emelianenko D., Voita E. BPE-Dropout: Simple and Effective Subword Regularization. Proc. 58th Annual Meeting of the Association for Computational Linguistics. 2020:1882—1892.
23. Papineni K. e. a. BLEU: a Method for Automatic Evaluation of Machine Translation. Proc 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia, 2002:311—318.
24. IWSLT 15 English-vietnamese Dataset [Elektron. Resurs] https://metatext.io/datasets/iwslt-15-english-viet-namese (Data Obrashcheniya 01.02.2023).
25. Vaswani A. e. a. Attention is All You Need. Proc. XXXI Conf. Neural Information Proc. Systems. Long Beach, 2017:1—15.
26. BERT in DeepPavlov. [Elektron. Resurs] http://docs.deeppavlov.ai/en/master/features/models/bert.html (Data Obrashcheniya 01.02.2022).
27. Barten'ev O.V. Avtomaticheskaya Rasstanovka Znakov Prepinaniya s Pomoshch'yu Neyronnykh Setey. Vestnik MEI. 2022;6:146—159. (in Russian).
28. Babaytseva V.V. Russkiy Yazyk. Teoriya. M.: Drofa, 1998. (in Russian).
29. Osnova Slova [Elektron. Resurs] https://studope-
dia.ru/6_82986_osnova-slova-tipi-osnov.html (Data Obra-
shcheniya 01.02.2023). (in Russian).
30. Morfologicheskiy Analizator pymorphy2 [Elektron. Resurs] https://pymorphy2.readthedocs.io/en/stable/ (Data Obrashcheniya 01.02.2023). (in Russian).
31. Otkrytyy Korpus [Elektron. Resurs] http://open-corpora.org/ (Data Obrashcheniya 01.02.2023). (in Russian).
32. NLTK Documentation nltk.stem.snowball Module [Elektron. Resurs] https://www.nltk.org/api/nltk.stem.snow-ball.html (Data Obrashcheniya 01.02.2023).
33. Gage P. A New Algorithm for Data Compression [Elektron. Resurs] https://www.derczynski.com/papers/archive/BPE_Gage.pdf (Data Obrashcheniya 01.02.2023).
34. BERT Base Model (Uncased). [Elektron. Resurs] https://huggingface.co/bert-base-uncased (Data Obrashche-niya 01.02.2023).
35. Expectation–maximization Algorithm [Elektron. Resurs] https://en.wikipedia.org/wiki/Expectation%E2%
80%93maximization_algorithm (Data Obrashcheniya 01.02.2023).
36. SentencePiece Python Wrapper [Elektron. Resurs] https://pypi.org/project/sentencepiece/ (Data Obrashcheniya 01.02.2023).
37. Nabor Dannykh dlya Razmetki Teksta po Chastyam Rechi [Elektron. Resurs] https://disk.yandex.ru/d/op1mwroF- BMyKcw (Data Obrashcheniya 01.02.2023). (in Russian).
38. Nabor Dannykh dlya Klassifikatsii Dokumentov [Elektron. Resurs] https://www.kaggle.com/datasets/olegbar-
tenyev/doc-cls (Data Obrashcheniya 01.02.2023). (in Russian).
39. Korpus dlya Sozdaniya Slovarya Tokenov [Elek-tron. Resurs] https://disk.yandex.ru/d/uFPs-eRW6Kaa8Q (Data Obrashcheniya 01.02.2023). (in Russian).
40. Scikit-learn [Elektron. Resurs] http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html (Data Obrashcheniya 01.02.2023).
41. Barten'ev O.V. Parametry, Vliyayushchie na Ef-fektivnost' Neyronnoy Seti, Sozdannoy Sredstvami Keras [Elektron. Resurs] http://www.100byte.ru/python/factors/factors.html (Data Obrashcheniya 01.02.2023). (in Russian).
42. BBC Dataset [Elektron. Resurs] http://www.mlg.ucd.ie/datasets/bbc.html (Data Obrashcheniya 01.02.2023)
---
For citation: Bartenyev O.V. Evaluating the Effectiveness of Text Tokenization Methods. Bulletin of MPEI. 2023;6:144—156. (in Russian). DOI: 10.24160/1993-6982-2023-6-144-156
2. Бартеньев О.В. Сравнительная оценка эффективности моделей текста в задаче классификации документов // Вестник МЭИ. 2021. № 5. С. 117—127.
3. Chi Z. e. a. XLM-E: Cross-lingual Language Model Pre-training via ELECTRA // Proceedings 60th Annual Meeting of the Association for Computational Linguistics. Dublin, 2022. V. 1. Pp. 6170—6182.
4. Arora G. iNLTK: Natural Language Toolkit for Indic Languages // Proc. Second Workshop for NLP Open Source Software (NLP-OSS). 2020. Pp. 66—71.
5. Kudo T., Richardson J. SentencePiece: a Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. // Proc. Conf. on Empirical Methods in Natural Language Processing: System Demonstrations. Brussels, 2018. Pp. 66—71.
6. Mikolov T. e. a. Distributed Representations of Words and Phrases and their Compositionality // Neural Information Processing Systems. 2013. V. 26. Pp. 1—9.
7. Le Q., Mikolov T. Distributed Representations of Sentences and Documents // Proc. XXXI Intern. Conf. Machine Learning. Beijing, 2014. Pp. 1188—1196.
8. Mikolov T. e. a. Advances in Pre-training Distributed Word Representations // Proceedings of LREC. Miyazaki, 2018. Pp. 52—55.
9. Pennington J., Socher R., Manning C. Glove: Global Vectors for Word Representation. // Proc. Conf. Empirical Methods in Natural Language Processing. 2014. Pp. 1532—1543.
10. Adesam Y., Berdicevskis A. Part-of-speech Tagging of Swedish Texts in the Neural Era // Proceedings XXIII Nordic Conf. Computational Linguistics (NoDaLiDa). Reykjavik, 2021. Pp. 200—209.
11. Haldar G., Mittal A., Gupta P. DSC-IITISM at FinCausal 2021: Combining POS Tagging with Attention-based Contextual Representations for Identifying Causal Relationships in Financial Documents // Proc. III Financial Narrative Processing Workshop. Lancaster, 2021. Pp. 49—53.
12. Alshammeri M., Atwell E., Alsalka M.A. Classifying Verses of the Quran using Doc2vec // Proc. XVIII Intern. Conf. Natural Language Processing. Silchar, 2021. Pp. 284—288.
13. Schmitt M. e. a. Joint Aspect and Polarity Classification for Aspect-based Sentiment Analysis with End-to-end Neural Networks 2018. In // Proc. Conf. Empirical Methods in Natural Language Processing. Brussels, 2018. Pp. 1109—1114.
14. Ebadulla D.M. e. a. A Comparative Study on Language Models for the Kannada Language // Proceedings IV Intern. Conf. Natural Language and Speech Processing. Trento, 2021. Pp. 280—284.
15. WordPiece Tokenization. [Электрон. ресурс] https://huggingface.co/course/chapter6/6?fw=pt (дата обращения 01.02.2023).
16. Alam T., Khan A., Alam F. Punctuation Restoration using Transformer Models for High-and Low-resource Languages // Proc. VI Workshop on Noisy User-generated Text. 2020. Pp. 132—142.
17. Pogoda M., Walkowiak T. Comprehensive Punctuation Restoration for English and Polish // Proc. Findings of the Association for Computational Linguistics: EMNLP. Punta Cana, 2021. Pp. 4610—4619.
18. Nagy A., Bial B., Ács J. Automatic Punctuation Restoration with BERT Models [Электрон. ресурс] https://arxiv.org/pdf/2101.07343v1.pdf#page=1 (дата обращения 01.02.2023).
19. Devlin J. e. a. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. - 2019. In // Proc. Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, 2019. V. 1. Pp. 4171—4186.
20. Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units 2016. In // Proc. 54th Annual Meeting of the Association for Computational Linguistics. Berlin, 2016. V. 1. Pp. 1715—1725.
21. Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates // Proc. 56th Annual Meeting of the Association for Computational Linguistics. Melbourne, 2018. V. 1. Pp. 66—75.
22. Provilkov I., Emelianenko D., Voita E. BPE-Dropout: Simple and Effective Subword Regularization // Proc. 58th Annual Meeting of the Association for Computational Linguistics. 2020. Pp. 1882—1892.
23. Papineni K. et. al. BLEU: a Method for Automatic Evaluation of Machine Translation. - 2002.In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
24. IWSLT 15 English-Vietnamese Dataset. - 2015. [Электрон. ресурс] https://metatext.io/datasets/iwslt-15-english-vietnamese (дата обращения 01.02.2023).
25. Vaswani A. et. al. Attention is all you need. - 2017. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
26. BERT in DeepPavlov. [Электронный ресурс] URL: http://docs.deeppavlov.ai/en/master/features/models/bert.html (дата обращения 01.02.2022).
27. Бартеньев О. В. Автоматическая расстановка знаков препинания с помощью нейронных сетей // Вестник МЭИ. 2022. № 6. С. 146—159. DOI: 10.24160/1993-6982-2022-6-146-159.
28. Бабайцева В. В. Русский язык. Теория. – М.: Дрофа, 1998. – 432 с.
29. Основа слова. [Электрон. ресурс] https://studopedia.ru/6_82986_osnova-slova-tipi-osnov.html (дата обращения 01.02.2023).
30. Морфологический анализатор pymorphy2. [Электрон. ресурс] https://pymorphy2.readthedocs.io/en/stable/ (дата обращения 01.02.2023).
31. Открытый корпус. [Электрон. ресурс] http://opencorpora.org/ (дата обращения 01.02.2023).
32. NLTK Documentation. nltk.stem.snowball module. [Электрон. ресурс] https://www.nltk.org/api/nltk.stem.snowball.html (дата обращения 01.02.2023).
33. Gage P. A New Algorithm for Data Compression. – 1994. [Электрон. ресурс] https://www.derczynski.com/papers/archive/BPE_Gage.pdf (дата обращения 01.02.2023).
34. BERT base model (uncased). [Электрон. ресурс] https://huggingface.co/bert-base-uncased (дата обращения 01.02.2023).
35. Expectation–maximization algorithm. [Электрон. ресурс] https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm (дата обращения 01.02.2023).
36. SentencePiece Python Wrapper. [Электрон. ресурс] https://pypi.org/project/sentencepiece/ (дата обращения 01.02.2023).
37. Набор данных для разметки текста по частям речи. [Электрон. ресурс] https://disk.yandex.ru/d/op1mwroFBMyKcw (дата обращения 01.02.2023).
38. Набор данных для классификации документов. [Электрон. ресурс] https://www.kaggle.com/datasets/olegbartenyev/doc-cls (дата обращения 01.02.2023).
39. Корпус для создания словаря токенов. [Электрон. ресурс] https://disk.yandex.ru/d/uFPs-eRW6Kaa8Q (дата обращения 01.02.2023).
40. Scikit-learn. [Электронн. ресурс] http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html\ (дата обращения 01.02.2023).
41. Бартеньев О. В. Параметры, влияющие на эффективность нейронной сети, созданной средствами Keras. [Электронн. ресурс] http://www.100byte.ru/python/factors/factors.html (дата обращения 01.02.2023).
42. BBC Dataset [Электрон. ресурс] http://www.mlg.ucd.ie/datasets/bbc.html (дата обращения 01.02.2023).
---
Для цитирования: Бартеньев О.В. Оценка эффективности методов токенизации текста // Вестник МЭИ. 2023. № 6. С. 144—156. DOI: 10.24160/1993-6982-2023-6-144-156
#
1. Keras [Elektron. Resurs] https://keras.io/ (Data Obrashcheniya 01.02.2023).
2. Barten'ev O.V. Sravnitel'naya Otsenka Effektivnosti Modeley Teksta v Zadache Klassifikatsii Dokumentov. Vestnik MEI. 2021;5:117—127. (in Russian).
3. Chi Z. e. a. XLM-E: Cross-lingual Language Model Pre-training via ELECTRA. Proceedings 60th Annual Meeting of the Association for Computational Linguistics. Dublin, 2022;1:6170—6182.
4. Arora G. iNLTK: Natural Language Toolkit for Indic Languages. Proc. Second Workshop for NLP Open Source Software (NLP-OSS). 2020:66—71.
5. Kudo T., Richardson J. SentencePiece: a Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing.. Proc. Conf. on Empirical Methods in Natural Language Processing: System Demonstrations. Brussels, 2018:66—71.
6. Mikolov T. e. a. Distributed Representations of Words and Phrases and their Compositionality. Neural Information Processing Systems. 2013;26:1—9.
7. Le Q., Mikolov T. Distributed Representations of Sentences and Documents. Proc. XXXI Intern. Conf. Machine Learning. Beijing, 2014:1188—1196.
8. Mikolov T. e. a. Advances in Pre-training Distributed Word Representations. Proceedings of LREC. Miyazaki, 2018:52—55.
9. Pennington J., Socher R., Manning C. Glove: Global Vectors for Word Representation.. Proc. Conf. Empirical Methods in Natural Language Processing. 2014:1532—1543.
10. Adesam Y., Berdicevskis A. Part-of-speech Tagging of Swedish Texts in the Neural Era. Proceedings XXIII Nordic Conf. Computational Linguistics (NoDaLiDa). Reykjavik, 2021:200—209.
11. Haldar G., Mittal A., Gupta P. DSC-IITISM at FinCausal 2021: Combining POS Tagging with Attention-based Contextual Representations for Identifying Causal Relationships in Financial Documents. Proc. III Financial Narrative Processing Workshop. Lancaster, 2021: 49—53.
12. Alshammeri M., Atwell E., Alsalka M.A. Classifying Verses of the Quran using Doc2vec. Proc. XVIII Intern. Conf. Natural Language Processing. Silchar, 2021:284—288.
13. Schmitt M. e. a. Joint Aspect and Polarity Classification for Aspect-based Sentiment Analysis with End-to-end Neural Networks 2018. In. Proc. Conf. Empirical Methods in Natural Language Processing. Brussels, 2018:1109—1114.
14. Ebadulla D.M. e. a. A Comparative Study on Language Models for the Kannada Language. Proceedings IV Intern. Conf. Natural Language and Speech Processing. Trento, 2021:280—284.
15. WordPiece Tokenization. [Elektron. Resurs] https://huggingface.co/course/chapter6/6?fw=pt (Data Obrashcheniya 01.02.2023).
16. Alam T., Khan A., Alam F. Punctuation Restoration using Transformer Models for High-and Low-resource Languages. Proc. VI Workshop on Noisy User-generated Text. 2020:132—142.
17. Pogoda M., Walkowiak T. Comprehensive Punctuation Restoration for English and Polish. Proc. Findings of the Association for Computational Linguistics: EMNLP. Punta Cana, 2021:4610—4619.
18. Nagy A., Bial B., Ács J. Automatic Punctuation Restoration with BERT Models [Elektron. Resurs] https://arxiv.org/pdf/2101.07343v1.pdf#page=1 (Data Obrashcheniya 01.02.2023).
19. Devlin J. e. a. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. - 2019. In. Proc. Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, 2019;1:4171—4186.
20. Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units 2016. In. Proc. 54th Annual Meeting of the Association for Computational Linguistics. Berlin, 2016;1:1715—1725.
21. Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. Proc. 56th Annual Meeting of the Association for Computational Linguistics. Melbourne, 2018;1:66—75.
22. Provilkov I., Emelianenko D., Voita E. BPE-Dropout: Simple and Effective Subword Regularization. Proc. 58th Annual Meeting of the Association for Computational Linguistics. 2020:1882—1892.
23. Papineni K. e. a. BLEU: a Method for Automatic Evaluation of Machine Translation. Proc 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia, 2002:311—318.
24. IWSLT 15 English-vietnamese Dataset [Elektron. Resurs] https://metatext.io/datasets/iwslt-15-english-viet-namese (Data Obrashcheniya 01.02.2023).
25. Vaswani A. e. a. Attention is All You Need. Proc. XXXI Conf. Neural Information Proc. Systems. Long Beach, 2017:1—15.
26. BERT in DeepPavlov. [Elektron. Resurs] http://docs.deeppavlov.ai/en/master/features/models/bert.html (Data Obrashcheniya 01.02.2022).
27. Barten'ev O.V. Avtomaticheskaya Rasstanovka Znakov Prepinaniya s Pomoshch'yu Neyronnykh Setey. Vestnik MEI. 2022;6:146—159. (in Russian).
28. Babaytseva V.V. Russkiy Yazyk. Teoriya. M.: Drofa, 1998. (in Russian).
29. Osnova Slova [Elektron. Resurs] https://studope-
dia.ru/6_82986_osnova-slova-tipi-osnov.html (Data Obra-
shcheniya 01.02.2023). (in Russian).
30. Morfologicheskiy Analizator pymorphy2 [Elektron. Resurs] https://pymorphy2.readthedocs.io/en/stable/ (Data Obrashcheniya 01.02.2023). (in Russian).
31. Otkrytyy Korpus [Elektron. Resurs] http://open-corpora.org/ (Data Obrashcheniya 01.02.2023). (in Russian).
32. NLTK Documentation nltk.stem.snowball Module [Elektron. Resurs] https://www.nltk.org/api/nltk.stem.snow-ball.html (Data Obrashcheniya 01.02.2023).
33. Gage P. A New Algorithm for Data Compression [Elektron. Resurs] https://www.derczynski.com/papers/archive/BPE_Gage.pdf (Data Obrashcheniya 01.02.2023).
34. BERT Base Model (Uncased). [Elektron. Resurs] https://huggingface.co/bert-base-uncased (Data Obrashche-niya 01.02.2023).
35. Expectation–maximization Algorithm [Elektron. Resurs] https://en.wikipedia.org/wiki/Expectation%E2%
80%93maximization_algorithm (Data Obrashcheniya 01.02.2023).
36. SentencePiece Python Wrapper [Elektron. Resurs] https://pypi.org/project/sentencepiece/ (Data Obrashcheniya 01.02.2023).
37. Nabor Dannykh dlya Razmetki Teksta po Chastyam Rechi [Elektron. Resurs] https://disk.yandex.ru/d/op1mwroF- BMyKcw (Data Obrashcheniya 01.02.2023). (in Russian).
38. Nabor Dannykh dlya Klassifikatsii Dokumentov [Elektron. Resurs] https://www.kaggle.com/datasets/olegbar-
tenyev/doc-cls (Data Obrashcheniya 01.02.2023). (in Russian).
39. Korpus dlya Sozdaniya Slovarya Tokenov [Elek-tron. Resurs] https://disk.yandex.ru/d/uFPs-eRW6Kaa8Q (Data Obrashcheniya 01.02.2023). (in Russian).
40. Scikit-learn [Elektron. Resurs] http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html (Data Obrashcheniya 01.02.2023).
41. Barten'ev O.V. Parametry, Vliyayushchie na Ef-fektivnost' Neyronnoy Seti, Sozdannoy Sredstvami Keras [Elektron. Resurs] http://www.100byte.ru/python/factors/factors.html (Data Obrashcheniya 01.02.2023). (in Russian).
42. BBC Dataset [Elektron. Resurs] http://www.mlg.ucd.ie/datasets/bbc.html (Data Obrashcheniya 01.02.2023)
---
For citation: Bartenyev O.V. Evaluating the Effectiveness of Text Tokenization Methods. Bulletin of MPEI. 2023;6:144—156. (in Russian). DOI: 10.24160/1993-6982-2023-6-144-156
Опубликован
2023-09-05
Выпуск
Раздел
Математическое и программное обеспечение вычислительных систем, комплексов и компьютерных сетей (технические науки) (2.3.5)
