ellibsЭлектронные библиотекиRussian Digital Libraries Journal1562-5419Казанский (Приволжский) федеральный университет10.26907/1562-5419-2025-28-5-1120-1137ellibs-612Research ArticleСтатьиАбстрактивная суммаризация новостей внешней торговли на основе нового специализированного корпуса данныхAbstractive Summarization for Trade News Analysis Based on a New Domain-Specific DatasetЛютоваДарья АндреевнаLyutovaDaria Andreevnalyutovad@gmail.comМалыхВалентин АндреевичMalykhValentin Andreevichvalentin.malykh@phystech.eduВсероссийская академия внешней торговлиRussian Foreign Trade AcademyУниверситет ИТМОITMO University20251912202528511201137Copyright © Лютова Д.А., Малых В.А., 20252025Лютова Д.А., Малых В.А.Lyutova D.A., Malykh V.A.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ellibs.elpub.ru/jour/article/view/612

Представлен TradeNewsSum — корпус для абстрактивной генерации аннотаций к новостям внешней торговли, охватывающий русско- и англоязычные публикации из профильных источников. Все рефераты подготовлены вручную по унифицированным правилам. Проведены эксперименты с дообучением трансформерных и seq2seq-моделей и автоматическую оценку по схеме LLM-as-a-judge. Наилучшие результаты показала LLaMA 3.1 в режиме инструкционного промптинга, продемонстрировав высокие значения по метрикам, включая фактологическую полноту.

We present TradeNewsSum—a corpus for abstractive summarization of international trade news—covering Russian- and English-language publications from domain-specific sources. All summaries are manually prepared following unified guidelines. We conducted experiments with fine-tuning transformer and seq2seq models and performed automatic evaluation using the LLM-as-a-judge scheme. LLaMA 3.1 in instruction-prompting mode achieved the best results, showing high scores across metrics, including factual completeness.

абстрактивное реферированиемногоязычный корпусновости внешней торговлисанкцииторговые режимыTradeNewsSumтрансформерыбольшие языковые моделиLLM-as-a-judgeNER-оценка сущностейabstractive summarizationmultilingual corpusinternational trade newssanctionstrade regimesTradeNewsSumtransformerslarge language modelsLLM-as-a-judgeNER-based entity evaluationReferencesBahdanau D. et al. End-to-end attention-based large vocabulary speech recognition // 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2016. P. 4945–4949.Bahdanau D. et al. End-to-end attention-based large vocabulary speech recognition // 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2016. P. 4945–4949.Banerjee S., Lavie A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments // Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 2005. P. 65–72.Banerjee S., Lavie A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments // Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 2005. P. 65–72.Fabbri A. R. et al. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model // arXiv preprint arXiv:1906.01749. 2019.Fabbri A. R. et al. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model // arXiv preprint arXiv:1906.01749. 2019.Fischer T., Remus S., Biemann C. Measuring faithfulness of abstractive summaries // Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022). 2022. P. 63–73.Fischer T., Remus S., Biemann C. Measuring faithfulness of abstractive summaries // Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022). 2022. P. 63–73.Fu J. et al. Gptscore: Evaluate as you desire // arXiv preprint arXiv:2302.04166. 2023.Fu J. et al. Gptscore: Evaluate as you desire // arXiv preprint arXiv:2302.04166. 2023.Gavrilov D., Kalaidin P., Malykh V. Self-attentive model for headline generation // Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Part II 41. Springer International Publishing, 2019. P. 87–93.Gavrilov D., Kalaidin P., Malykh V. Self-attentive model for headline generation // Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Part II 41. Springer International Publishing, 2019. P. 87–93.Goyal T., Li J. J., Durrett G. News summarization and evaluation in the era of gpt-3 // arXiv preprint arXiv:2209.12356. 2022.Goyal T., Li J. J., Durrett G. News summarization and evaluation in the era of gpt-3 // arXiv preprint arXiv:2209.12356. 2022.Grusky M., Naaman M., Artzi Y. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies // arXiv preprint arXiv:1804.11283. 2018.Grusky M., Naaman M., Artzi Y. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies // arXiv preprint arXiv:1804.11283. 2018.Gusev I. Dataset for automatic summarization of Russian news // Artificial Intelligence and Natural Language: 9th Conference, AINL 2020, Helsinki, Finland, October 7–9, 2020, Proceedings 9. Springer International Publishing, 2020. P. 122–134.Gusev I. Dataset for automatic summarization of Russian news // Artificial Intelligence and Natural Language: 9th Conference, AINL 2020, Helsinki, Finland, October 7–9, 2020, Proceedings 9. Springer International Publishing, 2020. P. 122–134.Hasan T. et al. XL-sum: Large-scale multilingual abstractive summarization for 44 languages // arXiv preprint arXiv:2106.13822. 2021.Hasan T. et al. XL-sum: Large-scale multilingual abstractive summarization for 44 languages // arXiv preprint arXiv:2106.13822. 2021.Kryściński W. et al. Neural text summarization: A critical evaluation // arXiv preprint arXiv:1908.08960. 2019.Kryściński W. et al. Neural text summarization: A critical evaluation // arXiv preprint arXiv:1908.08960. 2019.Lewis M. et al. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension // arXiv preprint arXiv:1910.13461. 2019.Lewis M. et al. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension // arXiv preprint arXiv:1910.13461. 2019.Liu Y. et al. G-eval: NLG evaluation using gpt-4 with better human alignment // arXiv preprint arXiv:2303.16634. 2023.Liu Y. et al. G-eval: NLG evaluation using gpt-4 with better human alignment // arXiv preprint arXiv:2303.16634. 2023.Narayan S., Cohen S. B., Lapata M. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization // arXiv preprint arXiv:1808.08745. 2018.Narayan S., Cohen S. B., Lapata M. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization // arXiv preprint arXiv:1808.08745. 2018.Paulus R., Xiong C., Socher R. A deep reinforced model for abstractive summarization // arXiv preprint arXiv:1705.04304. 2017.Paulus R., Xiong C., Socher R. A deep reinforced model for abstractive summarization // arXiv preprint arXiv:1705.04304. 2017.Raffel C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer // Journal of machine learning research. 2020. Vol. 21, No. 140. P. 1–67.Raffel C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer // Journal of machine learning research. 2020. Vol. 21, No. 140. P. 1–67.Rush A.M., Chopra S., Weston J. A neural attention model for abstractive sentence summarization // arXiv preprint arXiv:1509.00685. 2015.Rush A.M., Chopra S., Weston J. A neural attention model for abstractive sentence summarization // arXiv preprint arXiv:1509.00685. 2015.Sandhaus E. The New York Times Annotated Corpus Overview [Electronic resource]. Philadelphia: Linguistic Data Consortium, 2008. (LDC Catalog No. LDC2008T19). https://gwern.net/doc/ai/dataset/2008-sandhaus.pdf (accessed: 21.05.2025).Sandhaus E. The New York Times Annotated Corpus Overview [Electronic resource]. Philadelphia: Linguistic Data Consortium, 2008. (LDC Catalog No. LDC2008T19). https://gwern.net/doc/ai/dataset/2008-sandhaus.pdf (accessed: 21.05.2025).Scialom T. et al. MLSUM: The multilingual summarization corpus // arXiv preprint arXiv:2004.14900. 2020.Scialom T. et al. MLSUM: The multilingual summarization corpus // arXiv preprint arXiv:2004.14900. 2020.See A., Liu P. J., Manning C.D. A Neural Attention Model for Abstractive Sentence Summarization [Electronic resource]. 2016.See A., Liu P. J., Manning C.D. A Neural Attention Model for Abstractive Sentence Summarization [Electronic resource]. 2016.https://github.com/abisee/cnn-dailymail (accessed 07.04.2025).https://github.com/abisee/cnn-dailymail (accessed 07.04.2025).See A., Liu P.J., Manning C.D. Get to the point: Summarization with pointer-generator networks // arXiv preprint arXiv:1704.04368. 2017.See A., Liu P.J., Manning C.D. Get to the point: Summarization with pointer-generator networks // arXiv preprint arXiv:1704.04368. 2017.Varab D., Schluter N. MassiveSumm: a very large-scale, very multilingual, news summarisation dataset // Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. P. 10150–10161.Varab D., Schluter N. MassiveSumm: a very large-scale, very multilingual, news summarisation dataset // Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. P. 10150–10161.Vaswani A. et al. Attention is all you need // Advances in neural information processing systems. 2017. Vol. 30.Vaswani A. et al. Attention is all you need // Advances in neural information processing systems. 2017. Vol. 30.Xin L., Liutova D., Malykh V. Cross-Language Summarization in Russian and Chinese Using the Reinforcement Learning // International Conference on Analysis of Images, Social Networks and Texts. Cham: Springer Nature Switzerland, 2024. P. 179–192.Xin L., Liutova D., Malykh V. Cross-Language Summarization in Russian and Chinese Using the Reinforcement Learning // International Conference on Analysis of Images, Social Networks and Texts. Cham: Springer Nature Switzerland, 2024. P. 179–192.Yutkin M. Lenta.Ru News Dataset [Electronic resource]. 2018. Available at: https://github.com/yutkin/Lenta.Ru-News-Dataset (accessed 04.05.2025).Yutkin M. Lenta.Ru News Dataset [Electronic resource]. 2018. Available at: https://github.com/yutkin/Lenta.Ru-News-Dataset (accessed 04.05.2025).Zhang J. et al. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization // International conference on machine learning. PMLR, 2020. P. 11328–11339.Zhang J. et al. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization // International conference on machine learning. PMLR, 2020. P. 11328–11339.Zhang T. et al. Bertscore: Evaluating text generation with bert // arXiv preprint arXiv:1904.09675. 2019.Zhang T. et al. Bertscore: Evaluating text generation with bert // arXiv preprint arXiv:1904.09675. 2019.

The authors declare that there are no conflicts of interest present.