ellibsЭлектронные библиотекиRussian Digital Libraries Journal1562-5419Казанский (Приволжский) федеральный университетellibs-17Research ArticleСтатьиРазработка системы эмоциональной оценки на основе обучения с подкреплением и нейробиологически инспирированных методовThe system of emotional appraisal based on reinforcement learning and bio-inspired methodsМайороваЕ. Ю.eugeniamaiorova@gmail.comТалановМ. О.max.talanov@gmail.comЛоуР.robert.lowe@his.seКазанский (Приволжский) федеральный университетRussian FederationUniversity of GothenburgRussian Federation201628062016193193215Copyright © Майорова Е.Ю., Таланов М.О., Лоу Р., 20162016Майорова Е.Ю., Таланов М.О., Лоу Р.Майорова Е.Ю., Таланов М.О., Лоу Р.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ellibs.elpub.ru/jour/article/view/17

Объектом проведенного исследования является эмоциональная оценка искусственного интеллекта. В качестве системы реализации эмоциональной оценки выбрана система обучения с подкреплением. В результате симуляции построенной модели получены графики, показывающие активность структур мозга, участвующих в процессе их воздействия друг на друга. В ходе настройки системы удалось добиться четырех вспышек активности на таламусе вместо ожидаемых пяти.

I research and lecture in Cognitive Science where my particular interest is in emotions – neural networks modeling and applications – and animal and human learning.

куб Лёвхеймаэмоциональная оценкаNESTNeuCogARReferencesМаксим Таланов. Эмоциональные вычисления. URL: http://postnauka. ru/video/45297.Максим Таланов. Эмоциональные вычисления. URL: http://postnauka. ru/video/45297.Lowenstein G., Lerner J.S. The role of affect in decision-making // In R. Davidson, K. Scherer, H. Goldsmith (Eds.) Handbook of Affective Science. New York: Oxford University Press, 2003. P. 619–642.Lowenstein G., Lerner J.S. The role of affect in decision-making // In R. Davidson, K. Scherer, H. Goldsmith (Eds.) Handbook of Affective Science. New York: Oxford University Press, 2003. P. 619–642.Максим Таланов. Эмоциональный искусственный интеллект. URL: http://postnauka.ru/video/45296.Максим Таланов. Эмоциональный искусственный интеллект. URL: http://postnauka.ru/video/45296.Tom Ziemke, Robert Lowe. On the Role of Emotion in Embodied Cognitive Architectures: From Organisms to Robots. Springer Science+Business Media, LLC 2009. P. 71–73.Tom Ziemke, Robert Lowe. On the Role of Emotion in Embodied Cognitive Architectures: From Organisms to Robots. Springer Science+Business Media, LLC 2009. P. 71–73.David Sander, Didier Grandjean, Klaus R. Scherer. A systems approach to appraisal mechanisms in emotion. Geneva Emotion Research Group, Department of Psychology, University of Geneva, 2005. P. 140–148.David Sander, Didier Grandjean, Klaus R. Scherer. A systems approach to appraisal mechanisms in emotion. Geneva Emotion Research Group, Department of Psychology, University of Geneva, 2005. P. 140–148.Petta P. The role of emotion in a tractable architecture for situated cognizers // In: Trappl R., Petta P., Payr S. Eds. Emotions in Humans and Artifacts. Cambridge, MA: MIT Press, 2003. P. 87–88.Petta P. The role of emotion in a tractable architecture for situated cognizers // In: Trappl R., Petta P., Payr S. Eds. Emotions in Humans and Artifacts. Cambridge, MA: MIT Press, 2003. P. 87–88.Minsky Marvin. The Emotion Machine: Commonsense Thinking, Artifiial Intelligence, and the Future of the Human Mind. Simon and Schuster, 2007. P. 256–258.Minsky Marvin. The Emotion Machine: Commonsense Thinking, Artifiial Intelligence, and the Future of the Human Mind. Simon and Schuster, 2007. P. 256–258.Wörgötter F., Porr B. Temporal Sequence Learning, Prediction, and Control – a Review of different models and their relation to biological mechanisms. Department of Psychology, University of Stirling, 2005. P. 45.Wörgötter F., Porr B. Temporal Sequence Learning, Prediction, and Control – a Review of different models and their relation to biological mechanisms. Department of Psychology, University of Stirling, 2005. P. 45.Ortony A., Norman D., Revelle W. Affect and proto-affect in effective functioning // In: Fellous J-M, Arbib M.A., Eds. Who need emotions? New York: Oxford University Press, 2005.Ortony A., Norman D., Revelle W. Affect and proto-affect in effective functioning // In: Fellous J-M, Arbib M.A., Eds. Who need emotions? New York: Oxford University Press, 2005.Damasio A.R. The feeling of what happens: body, emotion and the making of consciousness. Heinemann: London, 1999. 400 p.Damasio A.R. The feeling of what happens: body, emotion and the making of consciousness. Heinemann: London, 1999. 400 p.Rolls E. Emotion explained. Oxford: Oxford University Press, 2005.Rolls E. Emotion explained. Oxford: Oxford University Press, 2005.Phelps E. Emotion and cognition: Insights from studies of the human amygdala // Annu. Rev. Psychol. 2006. V. 57. P. 27–53.Phelps E. Emotion and cognition: Insights from studies of the human amygdala // Annu. Rev. Psychol. 2006. V. 57. P. 27–53.Scherer K.R., Ekman P. On the nature and function of emotion: a component process approach // In: Approaches to Emotion. Hillsdale, N.J.: Lawrence Erlbaum, 1984. P. 293–317.Scherer K.R., Ekman P. On the nature and function of emotion: a component process approach // In: Approaches to Emotion. Hillsdale, N.J.: Lawrence Erlbaum, 1984. P. 293–317.Paulus Martin P., Angela J.Yu. Emotion and decision-making: affect-driven belief systems in anxiety and depression // Trends in Cognitive Sciences. September 2012. V. 16, No 9. P. 476–483.Paulus Martin P., Angela J.Yu. Emotion and decision-making: affect-driven belief systems in anxiety and depression // Trends in Cognitive Sciences. September 2012. V. 16, No 9. P. 476–483.Kahneman D., Tversky A. Prospect theory: an analysis of decision under risk // Econometrica. 1979. V. 47. P. 263–291.Kahneman D., Tversky A. Prospect theory: an analysis of decision under risk // Econometrica. 1979. V. 47. P. 263–291.Mukherjee K. A dual system model of preferences under risk // Psychol. Rev. 2010. V. 117. P. 243–255.Mukherjee K. A dual system model of preferences under risk // Psychol. Rev. 2010. V. 117. P. 243–255.Hsee C.K., Rottenstreich Y. Music, pandas, and muggers: on the affective psychology of value // J. Exp. Psychol. Gen. 2004. V. 133. P. 23–30.Hsee C.K., Rottenstreich Y. Music, pandas, and muggers: on the affective psychology of value // J. Exp. Psychol. Gen. 2004. V. 133. P. 23–30.Kusev P., van Schaik P. Preferences under risk: content-dependent behavior and psychological processing //Front. Psychol. 2011. V. 2. P. 269–271.Kusev P., van Schaik P. Preferences under risk: content-dependent behavior and psychological processing //Front. Psychol. 2011. V. 2. P. 269–271.Breazeal C. Designing sociable robots. Cambridge, MA: MIT Press, 2002. 244 p.Breazeal C. Designing sociable robots. Cambridge, MA: MIT Press, 2002. 244 p.Kelley A.E. Neurochemical networks encoding emotion and motivation: An evolutionary perspective // In: Fellous J-M., Arbib M.A., Eds. Who needs emotions? The brain meets the robot. New York: Oxford University Press, 2005.Kelley A.E. Neurochemical networks encoding emotion and motivation: An evolutionary perspective // In: Fellous J-M., Arbib M.A., Eds. Who needs emotions? The brain meets the robot. New York: Oxford University Press, 2005.Max Talanov, Jordi Vallverdu, Salvatore Distefano, Manuel Mazzara, Radhakrishnan Delhibabu. neuromodulating cognitive architecture: towards biomimetic emotional AI // Advanced Information Networking and Applications (AINA), 2015 IEEE 29th International Conference. P. 587–592.Max Talanov, Jordi Vallverdu, Salvatore Distefano, Manuel Mazzara, Radhakrishnan Delhibabu. neuromodulating cognitive architecture: towards biomimetic emotional AI // Advanced Information Networking and Applications (AINA), 2015 IEEE 29th International Conference. P. 587–592.Аллахвердов В.М., Богданова С.И. и др. Психология: учеб. / отв. ред. А.А. Крылов. М.: Проспект, 2005. С. 214–217.Аллахвердов В.М., Богданова С.И. и др. Психология: учеб. / отв. ред. А.А. Крылов. М.: Проспект, 2005. С. 214–217.Vernon David. Artificial Cognitive Systems: a Primer. The MIT Press Cambridge, Massachusetts London, England, 2014. 288 p.Vernon David. Artificial Cognitive Systems: a Primer. The MIT Press Cambridge, Massachusetts London, England, 2014. 288 p.McCarthy J., Hayes P.J. Some philosophical problems from the standpoint of artificial intelligence at the Wayback Machine //In: Meltzer B., Michie D., Eds. Machine Intelligence. Edinburgh: Edinburgh University Press, 1969. No 4. P. 463–502 (archived August 25, 2013).McCarthy J., Hayes P.J. Some philosophical problems from the standpoint of artificial intelligence at the Wayback Machine //In: Meltzer B., Michie D., Eds. Machine Intelligence. Edinburgh: Edinburgh University Press, 1969. No 4. P. 463–502 (archived August 25, 2013).Таланов Максим. Марвин Минский и эмоциональные машины. URL: https://postnauka.ru/faq/58727.Таланов Максим. Марвин Минский и эмоциональные машины. URL: https://postnauka.ru/faq/58727.Lövheim H. A new three-dimensional model for emotions and monoamine neurotransmitters // Med Hypotheses. 2012. V. 8. P. 341–348.Lövheim H. A new three-dimensional model for emotions and monoamine neurotransmitters // Med Hypotheses. 2012. V. 8. P. 341–348.Tomkins S. Affect theory // In: P. Ekman, W. Friesen, P. Ellsworth, Eds. Emotions in the Human Face. Cambridge: Cambridge University Press, 1982. P. 355–395.Tomkins S. Affect theory // In: P. Ekman, W. Friesen, P. Ellsworth, Eds. Emotions in the Human Face. Cambridge: Cambridge University Press, 1982. P. 355–395.Smith Craig A., Lazarus Richard S. Emotion and Adaptation // In: L.A. Pervin, Ed. Handbook of Personality: Theory and Research. New York: Guilford, 1990. P. 609–637.Smith Craig A., Lazarus Richard S. Emotion and Adaptation // In: L.A. Pervin, Ed. Handbook of Personality: Theory and Research. New York: Guilford, 1990. P. 609–637.Lazarus Richard S. Progress on a cognitive-motivational-relational theory of emotion // American Psychologist. 1991. V. 46, No 8. P. 819–834.Lazarus Richard S. Progress on a cognitive-motivational-relational theory of emotion // American Psychologist. 1991. V. 46, No 8. P. 819–834.Niv Yael. Reinforcement learning in the brain // Psychology Department & Princeton Neuroscience Institute, Princeton University, 2009.Niv Yael. Reinforcement learning in the brain // Psychology Department & Princeton Neuroscience Institute, Princeton University, 2009.Barto A.G. Adaptive critic and the basal ganglia // In J.C. Houk, J.L. Davis, D.G. Beiser, Eds. Models of information processing in the basal ganglia. Cambridge: MIT Press, 1995. P. 215–232.Barto A.G. Adaptive critic and the basal ganglia // In J.C. Houk, J.L. Davis, D.G. Beiser, Eds. Models of information processing in the basal ganglia. Cambridge: MIT Press, 1995. P. 215–232.Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward // Science. 1997. No 275. P. 1593–1599.Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward // Science. 1997. No 275. P. 1593–1599.Wickens J.R., Kotter R. Cellular models of reinforcement // In: J.C. Houk, J.L. Davis, D.G. Beiser, Eds. Models of information processing in the basal ganglia. MIT Press, 1995. P. 187–214.Wickens J.R., Kotter R. Cellular models of reinforcement // In: J.C. Houk, J.L. Davis, D.G. Beiser, Eds. Models of information processing in the basal ganglia. MIT Press, 1995. P. 187–214.Barto A.G., Sutton R.S., Watkins C.J.C.H. Learning and sequential decision making // In: M. Gabriel, J. Moore, Eds. Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press, 1990. P. 593–602.Barto A.G., Sutton R.S., Watkins C.J.C.H. Learning and sequential decision making // In: M. Gabriel, J. Moore, Eds. Learning and computational neuroscience: Foundations of adaptive networks. Cambridge, MA: MIT Press, 1990. P. 593–602.Bertsekas D.P., Tsitsiklis J.N. Neuro-dynamic programming. Athena Sc., Scientific, 1996. 512 p.Bertsekas D.P., Tsitsiklis J.N. Neuro-dynamic programming. Athena Sc., Scientific, 1996. 512 p.Sutton R.S., Barto A.G. Reinforcement Learning. An Introduction. Bradford Books, MIT Press, Cambridge, MA, 2002 edition, 1998. 320 p.Sutton R.S., Barto A.G. Reinforcement Learning. An Introduction. Bradford Books, MIT Press, Cambridge, MA, 2002 edition, 1998. 320 p.Bellman R.E. Dynamic Programming. Princeton: Princeton University Press, 1957. 392 p.Bellman R.E. Dynamic Programming. Princeton: Princeton University Press, 1957. 392 p.Sutton R.S. Learning to predict by the methods of temporal differences // Machine Learning. August 1988. V. 3, Issue 1. P. 9–44.Sutton R.S. Learning to predict by the methods of temporal differences // Machine Learning. August 1988. V. 3, Issue 1. P. 9–44.Sutton R.S. Generalization in reinforcement learning: successful examples using sparse coarse coding // In: D.S. Touretzky, M.C. Mozer, M.E. Hasselmo, Eds. Advances in Neural Information Processing Systems: Proceedings of the 1995 Conference. Cambridge, MA, 1996. P. 1038–1044.Sutton R.S. Generalization in reinforcement learning: successful examples using sparse coarse coding // In: D.S. Touretzky, M.C. Mozer, M.E. Hasselmo, Eds. Advances in Neural Information Processing Systems: Proceedings of the 1995 Conference. Cambridge, MA, 1996. P. 1038–1044.Rummery G.A. Problem solving with reinforcement learning. PhD thesis. Cambridge University, Cambridge, 1995. 52 p.Rummery G.A. Problem solving with reinforcement learning. PhD thesis. Cambridge University, Cambridge, 1995. 52 p.Watkins C.J.C.H. Learning from delayed rewards. PhD thesis. University of Cambridge, Cambridge, England, 1989. 234 p. URL: https://www.cs.rhul.ac.uk/home/ chrisw/new_thesis.pdf.Watkins C.J.C.H. Learning from delayed rewards. PhD thesis. University of Cambridge, Cambridge, England, 1989. 234 p. URL: https://www.cs.rhul.ac.uk/home/ chrisw/new_thesis.pdf.Watkins C.J.C.H., Dayan P. Technical note: Q-Learning // Machine Learning. 1992. V. 7, Issue 8. P. 279–292. URL: http://www.gatsby.ucl.ac.uk/~dayan/papers/ cjch.pdf.Watkins C.J.C.H., Dayan P. Technical note: Q-Learning // Machine Learning. 1992. V. 7, Issue 8. P. 279–292. URL: http://www.gatsby.ucl.ac.uk/~dayan/papers/ cjch.pdf.Pavlov I.P. Conditioned reflexes. London: Oxford University Press, 1927. URL: http://s-f-walker.org.uk/pubsebooks/pdfs/Conditioned-Reflexes-Pavlov.pdf.Pavlov I.P. Conditioned reflexes. London: Oxford University Press, 1927. URL: http://s-f-walker.org.uk/pubsebooks/pdfs/Conditioned-Reflexes-Pavlov.pdf.Воронцов К.В. Обучение с подкреплением (Reinforcement Learning) URL: http://www.machinelearning.ru/wiki/images/archive/3/35/20140621071329! Voron-ML-RL-slides.pdf.Воронцов К.В. Обучение с подкреплением (Reinforcement Learning) URL: http://www.machinelearning.ru/wiki/images/archive/3/35/20140621071329! Voron-ML-RL-slides.pdf.Bellman R. A Markovian decision process // Journal of Mathematics and Mechanics. 1957. No 6. P. 716–719.Bellman R. A Markovian decision process // Journal of Mathematics and Mechanics. 1957. No 6. P. 716–719.Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement // In: A.H. Black, W.F. Prokasy, Eds. Classical conditioning II: Current research and theory. New York, NY: Appleton-Century-Crofts, 1972. P. 64–99.Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement // In: A.H. Black, W.F. Prokasy, Eds. Classical conditioning II: Current research and theory. New York, NY: Appleton-Century-Crofts, 1972. P. 64–99.Gewaltig Marc-Oliver, Diesmann Markus. NEST (NEural Simulation Tool) // Scholarpedia. 2007. V. 2, No 4. P. 1430. URL: http://www.scholarpedia.org/article/ NEST_(NEural_Simulation_Tool).Gewaltig Marc-Oliver, Diesmann Markus. NEST (NEural Simulation Tool) // Scholarpedia. 2007. V. 2, No 4. P. 1430. URL: http://www.scholarpedia.org/article/ NEST_(NEural_Simulation_Tool).Supercomputers Ready for Use as Discovery Machines for Neuroscience // Frontiers in Neuroinformatics. November 2012. V. 6. P. 1–12.Supercomputers Ready for Use as Discovery Machines for Neuroscience // Frontiers in Neuroinformatics. November 2012. V. 6. P. 1–12.Diesmann M., Gewaltig M. NEST: an environment for neural systems simulations // Forschung und wisschenschaftliches Rechnen, Beiträge zum Heinz-Billing-Preis. 2001. Bd. 58. S. 43–70.Diesmann M., Gewaltig M. NEST: an environment for neural systems simulations // Forschung und wisschenschaftliches Rechnen, Beiträge zum Heinz-Billing-Preis. 2001. Bd. 58. S. 43–70.Picard R.W. Affective Computing. MIT Press, 1997.Picard R.W. Affective Computing. MIT Press, 1997.

The authors declare that there are no conflicts of interest present.