In this article we present a novel neural network implementation of Associative Two-Process (ATP) theory based on an Actor–Critic-like architecture. Our implementation emphasizes the affective components of differential reward magnitude and reward omission expectation and thus we model Affective-Associative Two-Process theory (Aff-ATP). ATP has been used to explain the findings of differential outcomes training (DOT) procedures, which emphasize learning differentially valuated outcomes for cueing actions previously associated with those outcomes. ATP hypothesizes the existence of a ‘prospective’ memory route through which outcome expectations can bring to bear on decision making and can even substitute for decision making based on the ‘retrospective’ inputs of standard working memory. While DOT procedures are well recognized in the animal learning literature they have not previously been computationally modelled. The model presented in this article helps clarify the role of ATP computationally through the capturing of empirical data based on DOT. Our Aff-ATP model illuminates the different roles that prospective and retrospective memory can have in decision making (combining inputs to action selection functions). In specific cases, the model’s prospective route allows for adaptive switching (correct action selection prior to learning) following changes in the stimulus–response–outcome contingencies.
AtmacaS.SebanzN.KnoblichG. (2011). The joint Flanker effect: Sharing tasks with real and imagined co-actors. Experimental Brain Research, 211, 371–385.
2.
BalkeniusC.MorénJ. (2001). Emotional learning: a computational model of the amygdala.Cybernetics and Systems: An International Journal, 32, 611–636.
3.
BalkeniusC.MorénJ.WinbergS. (2009). Interactions between motivation, emotion and attention: from biology to robotics. In Proceedings of the Ninth International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems (p. 146). Lund: Lund University, Cognitive Studies.
4.
BillingE.BalkeniusC. (2014). Modeling the Interplay between conditioning and attention in a humanoid robot: habituation and attentional blocking. In The Fourth Joint IEEE Conference on Development and Learning and on Epigenetic Robotics, submitted.
5.
BoureauY.-L.DayanP. (2010). Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology Review, 1, 1–24.
6.
DamasioA. R. (1999). The Feeling of What Happens: Body and Emotion in the Making of Consciousness. Boston, MA: Houghton Mifflin Harcourt.
7.
DawN. D.KakadeS.DayanP. (2002). Opponent interactions between serotonin and dopamine. Neural Networks, 15, 603–616.
8.
de WitS.DickinsonA. (2009). Associative theories of goal-directed behaviour: A case for animal–human translational models. Psychological Research PRPF, 73(4), 463–476.
9.
DoyaK. (2008). Modulators of decision making. Nature Neuroscience, 11(4), 410–416.
10.
EstebanL.PlazaV.López-CrespoG.VivasA. B.EstévezA. F. (2014). Differential outcomes training improves face recognition memory in children and in adults with Down syndrome. Research in Developmental Disabilities, 35(6), 1384–1392.
11.
EstévezA. F. (2005). The differential outcomes effect: A useful tool to improve discriminative learning in humans. The Behavior Analyst Today, 6(4), 216.
12.
EstévezA. F.FuentesL. J. (2001). The differential outcome effect as a useful tool to improve conditional discrimination learning in children. Learning and Motivation, 32, 48–64.
13.
EstévezA. F.OvermierB.FuentesL. J. (2003). Differential outcomes effect in children: Demonstration and mechanisms. Learning and Motivation, 34, 148–167.
14.
FrémauxN.SprekelerH.GerstnerW. (2013). Reinforcement learning using a continuous time actor–critic framework with spiking neurons. PLoS Computational Biology, 9(4), 1–21.
15.
FriedrichA. M.ZentallT. R. (2011). A differential-outcome effect in pigeons using spatial hedonically nondifferential outcomes. Learning and Behavior, 39, 68–78.
16.
GrossbergS.SeidmanD. (2006). Neural dynamics of autistic behaviors: Cognitive, emotional, and timing substrates. Psychological Review, 113, 483–525.
17.
KennerlyS. W.WaltonM. E. (2011). Decision making and reward in frontal cortex: complementary evidence from neurophysiological and neuropsychological studies. Behavioral Neuroscience, 125(3), 297–317.
18.
KruseJ. M.OvermierJ. B. (1982). Anticipation of reward omission as a cue for choice behavior. Learning and Motivation, 13, 505–525.
19.
LoweR.ZiemkeT. (2013). Exploring the relationship of reward and punishment in reinforcement learning. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) (pp. 140–147).
20.
LoweR.AlmérA.BillingE.SandamirskayaY.BalkeniusC. (2016b). Affective-Associative Two-Process theory: A neurocomputational account of partial reinforcement extinction effects, (submitted).
21.
LoweR.AlmérA.LindbladG.GanderP.MichaelJ.VesperC. (2016). Minimalist social-affective value for use in joint action: A neural-computational hypothesis. Frontiers in Computational Neuroscience, 10. Available at: https://doi.org/10.3389/fncom.2016.00088
22.
LoweR.SandamirskayaY.BillingE. (2014). A neural dynamic model of associative two-process theory: The differential outcomes effect and infant development. In 4th International Conference on Development and Learning and on Epigenetic Robotics (pp. 440–447).
23.
MakiP.OvermierJ. B.DelosS.GutmannA. J. (1995). Expectancies as factors influencing conditional discrimination performance of children. The Psychological Record, 45, 45–71.
24.
MartínezL.Marí-BeffaP.Roldán-TapiaD.Ramos-LizanaJ.FuentesL.J.EstévezA.F., (2012). Training with differential outcomes enhances discriminative learning and visuospatial recognition memory in children born prematurely. Research in Developmental Disabilities, 33, 76–84.
25.
MiceliM.CastelfranchiC. (2014). Expectancy and emotion. Oxford: Oxford University Press.
26.
MorénJ. (2002). Emotion and Learning: A Computational Model of the Amygdala, PhD thesis, Lund University.
27.
MorénJ.BalkeniusC. (2000). A computational model of emotional learning in the amygdala. In MayerJ.-A.BerthozA.FloreanoD.RoitblatH. L.WilsonS. W. (eds.), From Animals to Animats 6 (pp. 383–391). Cambridge, MA: MIT Press.
28.
MowrerO. H. (1947). On the dual nature of learning: A reinterpretation of “conditioning” and “problem-solving”. Harvard Educational Review, 17, 102–148.
29.
OvermierJ. B.LawryJ.A. (1979). Pavlovian conditioning and the mediation of behavior. The Psychology of Learning and Motivation, 13, 1–55.
30.
PetersonG. B.TrapoldM. A. (1980). Effects of altering outcome expectancies on pigeons’ delayed conditional discrimination performance. Learning and Motivation, 11, 267–288.
31.
RedgraveP.PrescottT. J.GurneyK. (1999). The basal ganglia: a vertebrate solution to the selection problem?Neuroscience, 89, 1009–1023.
32.
RescorlaR. A.WagnerA. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and non- reinforcement. In BlackA. H.ProkasyW. F. (eds), Classical Conditioning II: Current Research and Theory. New York: Appleton-Century-Crofts.
33.
RollsE. (1992). Neurophysiology and functions of the primate amygdala. In AggletonJ. (ed.), The Amygdala: Neurobiological Aspects of Emotion, Memory and Mental Dysfunction (pp. 143–165). New York: Wiley-Liss.
34.
RollsE. T. (1999). The Brain and Emotion. Oxford: Oxford University Press.
35.
RollsE. T. (2013). What are emotional states, and why do we have them?Emotion Review, 5(3), 241–247.
36.
SavageL. M.RamosR. L. (2009). Reward expectation alters learning and memory: The impact of amygdala on appetitive-driven behaviors. Behavioural Brain Research, 198, 1–12.
37.
SchoenbaumG.SaddorisM. P.StalnakerT.A. (2007). Reconciling the roles of orbitofrontal cortex in reversal learning and the encoding of outcome expectancies. Annals of the New York Academy of Science, 1121, 320–335.
38.
SchultzW. (2007). Multiple dopamine functions at different time courses. Annual Review of Neuroscience, 30, 259–288.
39.
SebanzN.KnoblichG.PrinzW. (2005). How two share a task: Corepresenting stimulus–response mappings. Journal of Experimental Psychology: Human Perception and Performance, 31, 1234–1246.
40.
StaddonJ. (2014). The New Behaviorism. Hove: Psychology Press.
41.
SuriR. E. (2002). TD models of reward predictive responses in dopamine neurons. Neural Networks, 15, 523–533.
42.
SuriR.SchultzW. (1998). Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Experimental Brain Research, 121(3), 350–354.
43.
SuttonR. S.BartoA. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: The MIT Press.
44.
TrapoldM. A. (1970). Are expectancies based upon different positive reinforcing events discriminably different?Learning and Motivation, 1, 129–140.
45.
TrapoldM. A.OvermierJ. B. (1972). The second learning process in instrumental learning. In Classical Conditioning II: Current Research and Theory (pp. 427–452). New York: Appleton-Century-Crofts
46.
UrcuioliP. (1990). Some relationships between outcome expectancies and sample stimuli in pigeons’ delayed matching. Animal Learning and Behavior, 18(3), 302–314.
47.
UrcuioliP. (2005). Behavioral and associative effects of differential outcomes in discriminating learning. Learning and Behavior, 33(1), 1–21.
48.
UrcuioliP. (2013). Stimulus control and stimulus class formation. In MaddenG. J.DubeW. V.HackenbergT. D.HanleyG. P.LattalK. A. (eds), APA Handbook of Behavior Analysis (Vol. 1, pp. 361–386). Washington, DC: American Psychological Association.
49.
WassermanE. A.ZentallT. R. (eds) (2006). Comparative Cognition: Experimental Explorations of Animal Intelligence. Oxford: Oxford University Press.
50.
WatanabeM.HikosakaK.SakagamiM.ShirakawaS. (2007). Reward expectancy-related prefrontal neuronal activities: Are they neural substrates of “affective” working memory?Cortex, 43, 53–64.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
