2313-8912Research Result. Theoretical and Applied Linguistics2313-891210.18413/2313-8912-2023-9-1-0-33060TEXT COMPLEXITY PREDICTORS: METHODS AND APPROACHES FOR ASSESSMENT<strong>Terminology use in school textbooks: corpus analysis</strong><strong>Terminology use in school textbooks: corpus analysis</strong>MonakhovSergei I.MonakhovSergei I.sergomon@gmail.comTurchanenkoVladimir V.TurchanenkoVladimir V.turchanenko@mail.ruCherdakovDmitrii N.CherdakovDmitrii N.dm.cherdakov@gmail.comFriedrich Schiller University Jena, GermanySaint Petersburg University, RussiaInstitute of Russian Literature (Pushkinsky Dom) of the Russian Academy of Sciences, Russia20239100

The article presents the methods and results of the study that investigated the use of terminology in textbooks for secondary schools in Russia. The data were taken from a full-text DIY corpus of 207 textbooks for grades 5-11. The toolkit included models trained with the Word2Vec algorithms driven by the ideas of distributional semantics. The models were used to improve traditional automatic term extraction based on word frequency statistics. Numerical representation of word collocation patterns and their semantic similarity enabled the following: more effective automatic term extraction with a clear dividing line between terminology per se and high-frequency common words; comparative analysis of inventory and functioning of terms in textbooks for different school subjects and grades; analysis of the dynamics of new terms entering educational and methodological complexes and insights into terminological relations between textbooks for different grades. The study included another DIY corpus compiled of scholarly articles across the subjects taught at school. It was used to identify differences in term use in textbooks and scholarly texts as well as in non-specific and popular science contexts. The latter was facilitated by the RusVectōrēs word embedding model. The comprehensive analysis identified some patterns in term functioning relevant for particular school subjects or groups of subjects. The results were evaluated in view of the theory of text complexity, teaching methodology and didactics. The study found some contradictions between the expected and real text complexity. It also showed certain discrepancy between text complexity and basic didactic principles.

The article presents the methods and results of the study that investigated the use of terminology in textbooks for secondary schools in Russia. The data were taken from a full-text DIY corpus of 207 textbooks for grades 5-11. The toolkit included models trained with the Word2Vec algorithms driven by the ideas of distributional semantics. The models were used to improve traditional automatic term extraction based on word frequency statistics. Numerical representation of word collocation patterns and their semantic similarity enabled the following: more effective automatic term extraction with a clear dividing line between terminology per se and high-frequency common words; comparative analysis of inventory and functioning of terms in textbooks for different school subjects and grades; analysis of the dynamics of new terms entering educational and methodological complexes and insights into terminological relations between textbooks for different grades. The study included another DIY corpus compiled of scholarly articles across the subjects taught at school. It was used to identify differences in term use in textbooks and scholarly texts as well as in non-specific and popular science contexts. The latter was facilitated by the RusVectōrēs word embedding model. The comprehensive analysis identified some patterns in term functioning relevant for particular school subjects or groups of subjects. The results were evaluated in view of the theory of text complexity, teaching methodology and didactics. The study found some contradictions between the expected and real text complexity. It also showed certain discrepancy between text complexity and basic didactic principles.

TermTerminologySchool textbookText complexityWord frequencyVector representationWord2VecNeural networkTermTerminologySchool textbookText complexityWord frequencyVector representationWord2VecNeural network

The reported study was funded by the&nbsp;Russian Foundation for Basic Research, Project number 19-29-14032 mk &ldquo;Study of terminological subsystems of modern school textbooks in Russian with the help of word embedding models Word2Vec and neural networks&rdquo;.

Список литературыBrownlee,&nbsp;J. (2017). Deep Learning for Natural Language Processing: Develop Deep Learning Models for your Natural Language Problems, Machine Learning Mastery Publ., Vermont, USA. (In&nbsp;English)Cabré,&nbsp;M.&nbsp;T., Estopà,&nbsp;R. and Vivaldi,&nbsp;J. (2001). Automatic Term Detection: a Review of Current Systems, in Bourigault,&nbsp;D., Jacquemin,&nbsp;Ch. and L&rsquo;Homme,&nbsp;M.-C. (eds.), Recent Advances in Computational Terminology, John Benjamins Publ., Amsterdam, Netherlands, 53&ndash;87. DOI:&nbsp;10.1075/nlp.2.04cab (In&nbsp;English)Durda,&nbsp;K. and Buchanan,&nbsp;L. (2008). WINDSORS: Windsor Improved Norms of Distance and Similarity of Representations of Semantics, Behavior Research Methods, 40, 705&ndash;712. DOI:&nbsp;10.3758/BRM.40.3.705 (In&nbsp;English)Fisher,&nbsp;D., Frey,&nbsp;N. and Lapp,&nbsp;D. (2016). Text Complexity: Stretching Readers with Texts and Tasks, Corwin Press, Thousand Oaks, CA, USA. (In&nbsp;English)Flor,&nbsp;M., Klebanov,&nbsp;B. and Sheehan,&nbsp;K. (2013). Lexical Tightness and Text Complexity, Proceedings of the 2th Workshop of Natural Language Processing for Improving Textual Accessibility (NLP4ITA), Atlanta, USA, 29&ndash;38. (In&nbsp;English)Glazkova,&nbsp;A., Egorov,&nbsp;Yu. and Glazkov,&nbsp;M. (2021). A Comparative Study of Feature Types for Age-Based Text Classification, in van der Aalst,&nbsp;W. et al. (eds.), Analysis of Images, Social Networks and Texts. AIST 2020. Lecture Notes in Computer Science, 12602, Springer Publ., Cham, Switzerland, 120&ndash;134. (In&nbsp;English)Iomdin,&nbsp;B.&nbsp;L. and Morozov,&nbsp;D.&nbsp;A. (2021). Who Can Understand &ldquo;Dunno&rdquo;? Automatic Assessment of Text Complexity in Children&rsquo;s Literature, Russkaya Rech&rsquo;, 5, 55&ndash;68. DOI:&nbsp;10.31857/S013161170017239-1 (In&nbsp;Russian)Jones,&nbsp;M.&nbsp;N. and Mewhort,&nbsp;D.&nbsp;J.&nbsp;K. (2007). Representing Word Meaning and Order Information in a Composite Holographic Lexicon, Psychological Review, 114, 1&ndash;37. DOI:&nbsp;10.1037/0033-295X.114.1.1 (In&nbsp;English)Kilgarriff,&nbsp;A., Jakub&iacute;ček,&nbsp;M., Kov&aacute;ř,&nbsp;V. et al. (2014). Finding Terms in Corpora for Many Languages with the Sketch Engine, Proceedings of the Demonstrations at the 14th Conference the European Chapter of the Association for Computational Linguistics, Gothenburg, Sweden, 53&ndash;56. DOI:&nbsp;10.3115/v1/E14-2014 (In&nbsp;English)Korkontzelos,&nbsp;I. and Ananiadou,&nbsp;S. (2014). Term Extraction, in Mitkov,&nbsp;R. (ed.), Oxford Handbook of Computational Linguistics, Oxford University Press, Oxford, UK, 991&ndash;1012. (In&nbsp;English)Kutuzov,&nbsp;A. and Kuzmenko,&nbsp;E. (2017). WebVectors: A Toolkit for Building Web Interfaces for Vector Semantic Models, in Ignatov,&nbsp;D. et al. (ed.), Analysis of Images, Social Networks and Texts. AIST 2016. Communications in Computer and Information Science, 661, Springer Publ., Cham, Switzerland, 155&ndash;161. (In&nbsp;English)Laposhina,&nbsp;A.&nbsp;N., Lebedeva,&nbsp;M.&nbsp;U. and Berlin Khenis,&nbsp;A. (2022). Word Frequency and Text Complexity: An Eye-tracking Study of Young Russian Readers, Russian Journal of Linguistics, 26&nbsp;(2), 493&ndash;514. DOI:&nbsp;10.22363/2687-0088-30084. (In&nbsp;Russian)Laposhina,&nbsp;А.&nbsp;N., Veselovskaya,&nbsp;Т.&nbsp;S., Lebedeva,&nbsp;M.&nbsp;U. and Kupreshchenko,&nbsp;O.&nbsp;F. (2019). Lexical Analysis of the Russian Language Textbooks for Primary School: Corpus Study, Computational Linguistics and Intellectual Technologies: papers from the Annual International Conference &ldquo;Dialogue&rdquo;, Moscow, Russia, 18&nbsp;(25), 351&ndash;363. (In&nbsp;Russian)Leichik,&nbsp;V.&nbsp;M. (2007). Terminovedenie: predmet, metody, struktura [Terminology Studies: Subject, Methods, Structure], LKI Publishing House, Moscow, Russia. (In&nbsp;Russian)Levy,&nbsp;O. and Goldberg,&nbsp;Y. (2014). Linguistic Regularities in Sparse and Explicit Word Representations, Proceedings of the Eighteenth Conference on Computational Natural Language Learning, Baltimore, USA, 171&ndash;180. DOI:&nbsp;10.3115/ v1/W14-1618 (In&nbsp;English)Lukashevich,&nbsp;N.&nbsp;V. and Logachev,&nbsp;Yu.&nbsp;M. (2010). Combining Features for Automatic Term Extraction, Numerical Methods and Programming, 11&nbsp;(4), 108&ndash;116. (In&nbsp;Russian)Martynova,&nbsp;E.&nbsp;V., Solnyshkina,&nbsp;M.&nbsp;I., Merzlyakova,&nbsp;A.&nbsp;F. and Gizatulina,&nbsp;D.&nbsp;Yu. (2020). Lexical Parameters of the Academic Text (Based on the Texts of the Academic Corpus of the Russian Language), Philology and Culture, 3, 72&ndash;80. DOI:&nbsp;10.26907/2074-0239-2020-61-3-72-80 (In&nbsp;Russian)Mikk,&nbsp;Ya.&nbsp;A. (1981). Optimizatsiya slozhnosti uchebnogo teksta: V pomoshch&#39; avtoram i redaktoram [Optimizing the complexity of educational text: To help authors and editors], Prosveshchenie, Moscow, Russia. (In&nbsp;Russian)Mikolov,&nbsp;T., Sutskever,&nbsp;I., Chen,&nbsp;K. et al. (2013a). Distributed Representations of Words and Phrases and their Compositionality, Advances in Neural Information Processing Systems 26, 27th Annual Conference on Neural Information Processing Systems 2013, Lake Tahoe, USA, 3136&ndash;3144. (In&nbsp;English)Mikolov,&nbsp;T., Yih,&nbsp;W.&nbsp;T and Zweig,&nbsp;G. (2013b). Linguistic Regularities in Continuous Space Word Representations, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, USA, 746&ndash;751. (In&nbsp;English)Mitrofanova,&nbsp;O.&nbsp;A. and Zakharov,&nbsp;V.&nbsp;P. (2009). Automatic Analysis of Terminology in the Russian Text Corpus on Corpus Linguistics, Computational Linguistics and Intellectual Technologies: papers from the Annual International Conference &ldquo;Dialogue&rdquo;, Bekasosvo, Russia, 8&nbsp;(15), 321&ndash;328. (In&nbsp;Russian)Monakhov,&nbsp;S.&nbsp;I., Turchanenko,&nbsp;V.&nbsp;V. and Cherdakov,&nbsp;D.&nbsp;N. (2022). Terminology in Textbooks and Research Articles: Cluster Analysis of Corpus Data, Proceedings of 6th International Conference &ldquo;Informatization of Education and E-learning Methodology: Digital Technologies in Education&rdquo;, Krasnoyarsk, Russia, 3, 228&ndash;233. (In&nbsp;Russian)Morozov,&nbsp;D.&nbsp;A. and Iomdin,&nbsp;B.&nbsp;L. (2019). Criteria of Semantic Complexity of Words, Computational Linguistics and Intellectual Technologies: papers from the Annual International Conference &ldquo;Dialogue&rdquo;, Moscow, Russia, 18&nbsp;(25), 119&ndash;131. (In&nbsp;Russian)Nokel,&nbsp;M.&nbsp;A., Bolshakova,&nbsp;E.&nbsp;I. and Loukachevitch,&nbsp;N.&nbsp;V. (2012). Combining Multiple Features for Single-word Term Extraction, Computational Linguistics and Intellectual Technologies: papers from the Annual International Conference &ldquo;Dialogue&rdquo;, Bekasosvo, Russia, 11&nbsp;(18), 1, 490&ndash;501. (In&nbsp;English)Piotrovsky,&nbsp;R.&nbsp;G. and Yastrebova,&nbsp;S.&nbsp;V. (1969). Statistical Term Recognition, in Piotrovskij,&nbsp;R.&nbsp;G. (ed.), Statistika teksta [Text statistics], Belorusskij gosudarstvennyj universitet, Minsk, Belarus, 1, 249&ndash;259. (In&nbsp;Russian)Rohde,&nbsp;D.&nbsp;L., Gonnerman,&nbsp;L.&nbsp;M. and Plaut,&nbsp;D.&nbsp;C. (2006). An Improved Model of Semantic Similarity Based on Lexical Co-Occurrence, Communications of the ACM, 8, 627&ndash;633. (In&nbsp;English)Schwanenflugel,&nbsp;P.&nbsp;J. (1991). Why are Abstract Concepts Hard to Understand?, in Schwanenflugel,&nbsp;P.&nbsp;J. (ed.), The psychology of word meanings, Lawrence Erlbaum Associates Inc., Hillsdale, USA, 223&ndash;250. (In&nbsp;English)Sharoff,&nbsp;S. (2022). What Neural Networks Know about Linguistic Complexity, Russian Journal of Linguistics, 26&nbsp;(2), 371&ndash;390. DOI:&nbsp;10.22363/2687-0088-30178 (In&nbsp;English)Shpakovsky,&nbsp;Yu.&nbsp;F. (2007). Estimation of Perception Difficulty and Optimization of the Educational Text Complexity (on the Material of Texts in Chemistry), Abstract of Ph.D. dissertation, Linguistics, Minsk State Linguistic University, Minsk, Belarus. (In&nbsp;Russian)Solnyshkina,&nbsp;M.&nbsp;I. (2022). Measuring Text Complexity: State of the Art, Collection of Scientific Papers X Jubilee International Scientific Conference &ldquo;Teacher. Student. Textbook (in the Context of Global Challenges of Modern Times)&rdquo;, Moscow, Russia, 20&ndash;24. (In&nbsp;Russian)Solnyshkina,&nbsp;M.&nbsp;I. and Kiselnikov,&nbsp;A.&nbsp;S. (2015). Text Complexity: Study Phases in Russian Linguistics, Tomsk State University Journal of Philology, 6&nbsp;(38), 86&ndash;99. DOI:&nbsp;10.17223/19986645/38/7 (In&nbsp;Russian)Solnyshkina,&nbsp;M.&nbsp;I., McNamara,&nbsp;D. and Zamaletdinov,&nbsp;R.&nbsp;R. (2022). Natural Language Processing and Discourse Complexity Studies, Russian Journal of Linguistics, 26&nbsp;(2), 317&ndash;341. DOI:&nbsp;10.22363/2687-0088-30171 (In&nbsp;Russian)Solovyev,&nbsp;V.&nbsp;D., Ivanov,&nbsp;V.&nbsp;V. and Solnyshkina,&nbsp;M.&nbsp;I. (2018). Assessment of Reading Difficulty Levels in Russian Academic Texts: Approaches and Metrics, Journal of Intelligent &amp; Fuzzy Systems, 34&nbsp;(2), 3049&ndash;3058. DOI:&nbsp;10.3233/JIFS-169489 (In&nbsp;English)Solovyev,&nbsp;V.&nbsp;D., Solnyshkina,&nbsp;M.&nbsp;I. and McNamara,&nbsp;D. (2022). Computational Linguistics and Discourse Complexology: Paradigms and Research Methods, Russian Journal of Linguistics, 26&nbsp;(2), 275&ndash;316. DOI:&nbsp;10.22363/2687-0088-30161 (In&nbsp;English)Stepanova,&nbsp;D.&nbsp;V. (2017). Analiz metodov avtomaticheskogo vydeleniya terminov iz nauchno-tekhnicheskih tekstov [Analysis of Methods for Automatic Terms Extraction from Scientific and Technical Texts], Aktual&#39;nye problemy sovremennoj prikladnoj lingvistiki [Current problems of modern applied linguistics], Minskij gosudarstvennyj lingvisticheskij universitet, Minsk, 62&ndash;67. (In&nbsp;Russian)Tatarinov,&nbsp;V.&nbsp;A. (2006). Obshchee terminovedenie: Entsiklopedicheskij slovar&#39; [Terminology Studies: Encyclopedic Dictionary], Moskovskij Litsej, Moscow, Russia. (In&nbsp;Russian)Turney,&nbsp;P.&nbsp;D. and Pantel,&nbsp;P. (2010). From Frequency to Meaning: Vector Space Models of Semantics, Journal of Artificial Intelligence Research, 37, 141&ndash;188. DOI:&nbsp;10.1613/jair.2934 (In&nbsp;English)