AI as a literary author from the reader’s perspective: findings from a psycholinguistic experiment assessing synthetic amateur prose
Large language models (LLMs) are performing an increasing variety of tasks, including generating long texts that aspire to literary quality. The aim of this article is to compare subjective assessments of the quality of literary texts created by large language models with human-written texts in the genre of fan fiction based on the Harry Potter novels. The material for the evaluation consisted of six texts in English: three generated by AI using different approaches, and three written by human authors. The present study utilizes a psycholinguistic experimental design. Eighteen informants evaluated the texts according to eight criteria reflecting key categories of literariness: coherence (integrity), semantic and lexico-grammatical cohesion, narrative rhythm, emotionality, subjective modality of doubt, syntactic complexity, and expressiveness (idiomaticity, creativity, strong textual positions). The data processing included statistical analysis of objective parameters of linguistic complexity in the texts, subjective evaluations obtained from informants, as well as clustering of vectorized retellings of the texts provided by the informants.
The results showed that the gap between human and AI-generated texts varies across different categories. Thanks to specific generation methods, the synthetic texts managed to approach human-written ones in terms of overall coherence and syntactic complexity. However, the most significant differences emerged in the categories of rhythm, semantic cohesion, and the presence of the subjective modality of doubt, which readers rated significantly lower in the synthetic texts. Nevertheless, one of the generated texts was of a quality comparable to that of a human writer in terms of idiomaticity, creativity and the ability to take a strong position on a topic. Clustering of summaries confirmed that this text, unlike the other two generated ones, led to a more uniform understanding of the plot among readers.
It can be concluded that the main challenge for AI is not so much the construction of a coherent, hierarchical text structure, but rather the creation of a dynamic, psychologically authentic and rhythmically organised narrative. The categories of semantic cohesion, rhythm, and modality of doubt currently represent the frontier separating artificial intelligence from human intelligence in the realm of literary creativity.
Figures
Kolmogorova, A. V. (2026). AI as a literary author from the reader’s perspective: findings from a psycholinguistic experiment assessing synthetic amateur prose, Research Result. Theoretical and Applied Linguistics, 12 (3), 170–199.
While nobody left any comments to this publication.
You can be first.
Arnold, I. V. (2019). Semantics. Stylistics. Intertextuality: Collection of Scientific Papers, FLINTA, Moscow, 448 p. (In Russian)
Bazilevskiy, M. P. (2018). Research of new criteria for detecting first-order residual autocorrelation in regression models, Mathematics and Mathematical Models, 13–25. DOI: 10.24108/MATHM.0318.0000102 (In Russian)
Galperin, I. R. (2007). Text as an Object of Linguistic Research, URSS, Moscow, 137 p. (In Russian)
Girshman, M. (1974). Rhythm and Integrity of Prose Literary Work, Questions of Literature, 11, 128–150. (In Russian)
Lotman, Yu. M. (1998). Structure of the Artistic Text, in Lotman Yu. M. On Art, Iskusstvo – SPB, St. Petersburg, 383 p. (In Russian)
Matveeva, T. V. (1990). Functional Styles in Terms of Text Categories: A Synchronic-Comparative Essay, Ural University Press, Sverdlovsk, 172 p. (In Russian)
Shakhovskiy, V. I. (2010). Emotions: Pre-Linguistics, Linguistics, Linguoculturology, Librokom Moscow, 128 p. (In Russian)
Shklovskiy, V. (1925). Structure of the Short Story and Novel, in On the Theory of Prose, Krug Publishing House, Moscow, 56–69. (In Russian)
Boriskin, A. and Galimzianova, D. (2024). The LSG ChallengeWorkshop at INLG 2024: Prompting Techniques for Crafting Extended Narratives with LLMs, Proceedings of the 17th International Natural Language Generation Conference: Generation Challenges, Tokyo, Japan, Association for Computational Linguistics, 118–122. (In English)
Chakrabarty, T., Padmakumar, V. and He, H. (2022). Help Me Write a Poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), Abu Dhabi, United Arab Emirates, Association for Computational Linguistics, 6848–6863. (In English)
Charolles, M. (1978). Introduction aux problems de la cohérence des textes, Langue française, 38, 7–41. (In French)
Charaudeau, P. and Maingueneau, D. (2002). Dictionnaire d'Analyse du Discours, Seuil, Paris, 661 p. (In French)
Combettes, B. (1978). Thématisation et progression thématique dans les récits d'enfants, Langue française, 38, 74–86. (In French)
Flesch, R. (1948). A new readability yardstick, Journal of Applied Psychology, 32 (3), 221–233. (In English)
Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension, Psychological Review, 101(3), 371–395. (In English)
Guan, J. and Huang, M. (2020). UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, Association for Computational Linguistics, 9157–9166. (In English)
Iaroshenko, P.V. and Louckachevitch, N.V. (2025). Large Language Models versus Native Speakers in Emotional Assessment of Russian Words, Supercomputing Frontiers and Innovations, 12, 3, 20–30. DOI: 10.14529/jsfi250302 (In English)
Ippolito, D., Yuan, A., Coenen, A. and Burnam, S. (2022). Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers. DOI: 10.48550/arXiv.2211.05030 (In English)
Jin, Y., Kadam, V. and Wanvarie, D. (2022). Plot Writing From Pre-Trained Language Models, Proceedings of the 15th International Conference on Natural Language Generation (INLG 2022), 52–67. (In English)
Johnson-Laird, P. N. (1983). Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Cambridge, MA: Harvard University Press, 513 p. (In English)
Kolmogorova, A., Kulikova, E. and Lobanov, V. (2025). Aspect-Based Sentiment Analysis Using Large Language Models on Museum Visitor Reviews, Supercomputing Frontiers and Innovations, 12, 3, 121–140. DOI: 10.14529/jsfi250309 (In English)
Kryscinski, W., Keskar, N.S., McCann, B., Xiong, C. and Socher, R. (2019). Neural Text Summarization: A Critical Evaluation, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, Association for Computational Linguistics, 540–551. (In English)
Lee, Y., Ka, S., Son, B., Kang, P. and Kang, J. Navigating the Path of Writing: Outline-guided Text Generation with Large Language Models. arXiv:2404.13919 [cs.CL]. (In English)
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S. and Kiela, D. (2021). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401v4. DOI: 10.48550/arXiv.2005.11401 (In English)
Mann, W. C., & Thompson, S. A. (1988). Rhetorical Structure Theory: Toward a functional theory of text organization, Text, 8(3), 243–281. (In English)
Migal, A., Seredina, D., Telnina, L., Nazarov, N., Kolmogorova, A. and Mikhaylovskiy, N. (2024). Overview of Long Story Generation Challenge (LSGC) at INLG 2024, Proceedings of the 17th International Natural Language Generation Conference: Generation Challenges, Tokyo, Japan, Association for Computational Linguistics, 47–53. (In English)
Mikhaylovskiy, N. States of LLM-generated Texts and Phase Transitions between them. arXiv:2503.06330 [cs.CL]. DOI: 10.48550/arXiv.2503.06330 (In English)
Seredina, D. (2024). A Report on LSG 2024: LLM Fine-Tuning for Fictional Stories Generation, Proceedings of the 17th International Natural Language Generation Conference: Generation Challenges, Tokyo, Japan, Association for Computational Linguistics, 123–127. (In English)
Shen, C., Cheng, L., Nguyen, X., You, Y. and Bing, L. (2023). Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization. DOI: 10.48550/arXiv.2305.13091 (In English)
Singh, N., Bernal, G., Savchenko, D. and Glassman, E.L. (2023). Where to hide a stolen elephant: Leaps in creative writing with multimodal machine intelligence, ACM Transactions on Computer-Human Interaction, 30 (5), 1–57. (In English)
Takahashi, S. and Tanaka-Ishii, K. (2019). Evaluating computational language models with scaling properties of natural language, Computational Linguistics, 45 (3), 481–513. DOI: 10.48550/arXiv.1906.09379 (In English)
Vatolin, A. (2025). Structured Sentiment Analysis with Large Language Models: A Winning Solution for RuOpinionNE-2024, Proceedings of International Conference on Computational Linguistics and Intelligent Technologies Dialogue 2025. (In English)
Zhang, Y., Sun, Q., Guo, Q., Wu, J., Qiu, X. and Huang, X. (2022). Do Large Language Models Know What They Don't Know? arXiv:2210.03025. DOI: 10.48550/arXiv.2209.01515 (In English)
Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., Zhang, H., Gonzalez, J.E. and Stoica, I. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685v4. DOI: 10.48550/arXiv.2306.05685 (In English)
Zhong, W., Naradowsky, J., Takamura, H., Kobayashi, I. and Miyao, Y. (2023). Fiction-writing mode: An effective control for human-machine collaborative writing, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 1752–1765. (In English)
Zhou, D. and Sterman, S. (2024). Ai.llude: Encouraging Rewriting AI-Generated Text to Support Creative Expression. DOI: 10.48550/arXiv.2405.17843 (In English)
Corpus Materials
Olesha, Yu. (1961). Not a Day Without a Line, FTM, Moscow, 464 p. (In Russian)
Filippov, K. A. (2007). Text Linguistics: A Course of Lectures, St. Petersburg University Press, St. Petersburg, 329 p. (In Russian)
The article is an output of a research project HSE-BR-2025-032 implemented as part of the Basic Research Project at HSE University.