General value functions (GVFs) in the reinforcement learning (RL) literature are long-term predictive summaries of the outcomes of agents following specific policies in the environment. Affordances as perceived action possibilities with specific valence may be cast into predicted policy-relative goodness and modeled as GVFs. A systematic explication of this connection shows that GVFs and especially their deep-learning embodiments (1) realize affordance prediction as a form of direct perception, (2) illuminate the fundamental connection between action and perception in affordance, and (3) offer a scalable way to learn affordances using RL methods. Through an extensive review of existing literature on GVF applications and representative affordance research in robotics, we demonstrate that GVFs provide the right framework for learning affordances in real-world applications. In addition, we highlight a few new avenues of research opened up by the perspective of “affordance as GVF,” including using GVFs for orchestrating complex behaviors.
AbbeelP.CoatesA.QuigleyM.NgA. Y. (2006). An application of reinforcement learning to aerobatic helicopter flight. In Proceedings of the 19th International Conference on Neural Information Processing Systems, NIPS’06 (pp. 1–8). MIT Press.
2.
AbbeelP.NgA. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In ICML ’04: Proceedings of the Twenty-First International Conference on Machine learning (p. 1). ACM.
3.
AbeyruwanS.SeekircherA.VisserU. (2014). Off-policy general value functions to represent dynamic role assignments in RoboCup 3D soccer simulation (abs/1402.4525). CoRR. http://arxiv.org/abs/1402.4525
BergerS. E.TheuringC.AdolphK. E. (2007). How and when infants learn to climb stairs. Infant Behavior and Development, 30(1), 36–49. https://doi.org/10.1016/j.infbeh.2006.11.002
6.
CesariP.FormentiF.OlivatoP. (2003). A common perceptual parameter for stair climbing for children, young and old adults. Human Movement Science, 22, 111–124. https://doi.org/10.1016/S0167-9457(03)00003-4
7.
ChemeroA. (2003). An outline of a theory of affordances. Ecological Psychology, 15(2), 181–195.
CruzF.MaggS.WeberC.WermterS. (2016). Training agents with interactive reinforcement learning and contextual affordances. IEEE Transactions on Cognitive and Developmental Systems, 8, 271–284.
12.
DalrympleA. N.RoszkoD. A.SuttonR. S.MushahwarV. K. (2020). Pavlovian control of intraspinal microstimulation to produce over-ground walking. Journal of Neural Engineering, 17(3), Article 036002.
13.
DeisenrothM. P.RasmussenC. E. (2011). Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11 (pp. 465–472). Omnipress.
EdwardsA. L.DawsonM. R.HebertJ. S.SherstanC.SuttonR. S.ChanK. M.PilarskiP. M. (2016). Application of real-time machine learning to myoelectric prosthesis control: A case series in adaptive switching. Prosthetics and Orthotics International, 40(5), 573–581.
EysenbachB.GuptaA.IbarzJ.LevineS. (2018). Diversity is all you need: Learning skills without a reward function (abs/1802.06070). CoRR. https://arxiv.org/abs/1802.06070
18.
FajenB.RileyM.TurveyM. (2008). Information, affordances, and the control of action in sport. International Journal of Sport Psychology, 40, 79–107.
GibsonJ. J. (1979). The ecological approach to visual perception. Houghton Mifflin.
25.
GravesD.JinJ.LuoJ. (2020a). Lispr: An options framework for policy reuse with reinforcement learning.
26.
GravesD.NguyenN. M.HassanzadehK.JinJ. (2020b). Learning predictive representations in autonomous driving to improve deep reinforcement learning. CoRR. https://arxiv.org/abs/2006.15110
27.
GravesD.RezaeeK.ScheidemanS. (2019, November). Perception as prediction using general value functions in autonomous driving applications [Paper presentation]. 2019 IEEE International Conference on Intelligent Robots and Systems, Macau, China.
28.
GrayR. (2002). “markov at the bat”: A model of cognitive processing in baseball batters. Psychological Science, 13, 542–547.
GüntherJ. (2018). Machine intelligence for adaptable closed loop and open loop production engineering systems [PhD thesis]. Technische Universität München.
31.
GüntherJ.AdyN. M.KearneyA.DawsonM. R.PilarskiP. M. (2020). Examining the use of temporal-difference incremental delta-bar-delta for real-world predictive knowledge architectures. Frontiers in Robotics and AI, 7, Article 34.
32.
GüntherJ.KearneyA.DawsonM. R.SherstanC.PilarskiP. M. (2018). Predictions, surprise, and predictions of surprise in general value function architectures. In AAAI 2018 Fall Symposium on Reasoning and Learning in Real-World Systems for Long-Term Autonomy (pp. 22–29). AAAI.
33.
GüntherJ.PilarskiP. M.HelfrichG.ShenH.DiepoldK. (2016). Intelligent laser welding through representation, prediction, and control learning: An architecture with deep neural networks and reinforcement learning. Mechatronics, 34, 1–11. https://doi.org/10.1016/j.mechatronics.2015.09.004
34.
HarutyunyanA.DabneyW.BorsaD.HeessN.MunosR.PrecupD. (2019). The termination critic (abs/1902.09996). CoRR. http://arxiv.org/abs/1902.09996
35.
HassaninM.KhanS.TahtaliM. (2018). Visual affordance and function understanding: A survey (abs/1807.06775). CoRR. http://arxiv.org/abs/1807.06775
36.
JamoneL.UgurE.CangelosiA.FadigaL.BernardinoA.PiaterJ.Santos-VictorJ. (2018). Affordances in psychology, neuroscience, and robotics: A survey. IEEE Transactions on Cognitive and Developmental Systems, 10(1), 4–25.
37.
JensenT. W.PedersenS. B. (2016). Affect and affordances: The role of action and emotion in social interaction. Cognitive Semiotics, 9(1), 79–103.
38.
JinJ.NguyenN. M.SakibN.GravesD.YaoH.JagersandM. (2020). Mapless navigation among dynamics with social-safety-awareness: A reinforcement learning approach from 2D laser scans. In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 6979–6985). IEEE. https://doi.org/10.1109/ICRA40945.2020.9197148
39.
KhetarpalK.AhmedZ.ComaniciG.AbelD.PrecupD. (2020, July). What can I do here? A theory of affordances in reinforcement learning [Conference session]. Proceedings of the 37th International Conference on Machine Learning, ICML2020, Vienna, Austria.
40.
KonczakJ.MeeuwsenH.CressM. E. (1992). Changing affordances in stair climbing: The perception of maximum climbability in young and older adults. Journal of Experimental Psychology. Human Perception and Performance, 18, 691–697. https://doi.org/10.1037//0096-1523.18.3.691
41.
LeeD. (1976). A theory of visual control of braking based on information about time-to-collision. Perception, 5, 437–459. https://doi.org/10.1068/p050437
ManuelliL.GaoW.FlorenceP.TedrakeR. (2019). kPAM: Keypoint affordances for category-level robotic manipulation. https://arxiv.org/abs/1903.06684
44.
MarkL. S. (1987). Eyeheight-scaled information about affordances: A study of sitting and stair climbing. Journal of Experimental Psychology. Human Perception and Performance, 13, 3361–3370.
45.
McDowellJ. (1994). The content of perceptual experience. Philosopical Quarterly, 44(175), 190–205. https://doi.org/10.2307/2219740
46.
MnihV.KavukcuogluK.SilverD.GravesA.AntonoglouI.WierstraD.RiedmillerM. (2013). Playing Atari with deep reinforcement learning (abs/1312.5602). CoRR. http://arxiv.org/abs/1312.5602
47.
ModayilJ.WhiteA.SuttonR. S. (2012). Multi-timescale nexting in a reinforcement learning robot. In ZiemkeT.BalkeniusC.HallamJ. (eds.), From animals to animats 12 (pp. 299–309). Springer.
48.
MullerS.FaddeP. (2015). The relationship between visual anticipation and baseball batting game statistics. Journal of Applied Sport Psychology, 28, 49–61. https://doi.org/10.1080/10413200.2015.1058867
49.
NgA. Y.RussellS. J. (2000). Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML ’00 (pp. 663–670). Morgan Kaufmann Publishers Inc.
50.
NöeA. (2004). Action in perception. MIT Press.
51.
NormanD. (1988). The design of everyday things. Basic Books.
52.
OpenAI AkkayaI.AndrychowiczM.ChociejM.LitwinM.McGrewB.PetronA.PainoA.PlappertM.PowellG.RibasR.SchneiderJ.TezakN.TworekJ.WelinderP.WengL.YuanQ.ZarembaW.ZhangL. (2019). Solving Rubik’s cube with a robot hand. https://arxiv.org/abs/1910.07113
53.
PalettaL.FritzG.KintzlerF.IrranJ.DorffnerG. (2007). Perception and developmental learning of affordances in autonomous robots. In HertzbergJ.BeetzM.EnglertR. (Eds.), KI 2007: Advances in artificial intelligence (pp. 235–250). Springer.
54.
PilarskiP. M.DawsonM. R.DegrisT.CareyJ. P.ChanK. M.HebertJ. S.SuttonR. S. (2013). Adaptive artificial limbs: A real-time approach to prediction and anticipation. IEEE Robotics Automation Magazine, 20(1), 53–64. https://doi.org/10.1109/MRA.2012.2229948
55.
RafieeB.GhiassianS.WhiteA.SuttonR. S. (2019). Prediction in intelligence: An empirical comparison of off-policy algorithms on robots. In ElkindE.VelosoM.AgmonN.TaylorM. E. (Eds.), Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, Montreal, QC, Canada, May 13–17, 2019 (pp. 332–334). International Foundation for Autonomous Agents and MultiAgent Systems. http://dl.acm.org/citation.cfm?id=3331711.
56.
RafolsE. J.RingM. B.SuttonR. S.TannerB. (2005). Using predictive representations to improve generalization in reinforcement learning. In Proceedings of the 19th International Joint Conference on Artificial Intelligence, IJCAI’05 (pp. 835–840). Morgan Kaufmann Publishers Inc. http://dl.acm.org/citation.cfm?id=1642293.1642427
57.
ŞahinE.ÇakmakM.DoğarM. R.UğurE.ÜçolukG. (2007). To afford or not to afford: A new formalization of affordances toward affordance-based robot control. Adaptive Behavior, 15(4), 447–472. https://doi.org/10.1177/1059712307084689
58.
SchaulT.RingM. (2013). Better generalization with forecasts. In Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, IJCAI ’13 (pp. 1656–1662). AAAI Press. http://dl.acm.org/citation.cfm?id=2540128.2540366
59.
SchlegelM.ChungW.GravesD.QianJ.WhiteM. (2019). Importance resampling for off-policy prediction (abs/1906.04328). CoRR. http://arxiv.org/abs/1906.04328
60.
SchlegelM.WhiteA.PattersonA.WhiteM. (2018). General value function networks (abs/1807.06763). CoRR. http://arxiv.org/abs/1807.06763
61.
SherstanC.PilarskiP. (2014, September). Multilayer general value functions for robotic prediction and control [Conference session]. 2014IROS Workshop on AI and Robotics, Chicago, IL, United States. https://doi.org/10.13140/2.1.1545.6006
62.
SilverD.HuangA.MaddisonC. J.GuezA.SifreL.van den DriesscheG.SchrittwieserJ.AntonoglouI.PanneershelvamV.LanctotM.DielemanS.GreweD.NhamJ.KalchbrennerN.SutskeverI.LillicrapT.LeachM.KavukcuogluK.GraepelT.HassabisD. (2016). Mastering the game of go with deep neural networks and tree search. Nature, 529, 484–503. http://www.nature.com/nature/journal/v529/n7587/full/nature16961.html
63.
SilverD.HubertT.SchrittwieserJ.AntonoglouI.LaiM.GuezA.LanctotM.SifreL.KumaranD.GraepelT.LillicrapT.SimonyanK.HassabisD. (2018). A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419), 1140–1144. https://science.sciencemag.org/content/362/6419/1140
64.
SmithB. C. (1996). On the origin of objects. MIT Press.
SuttonR. S.BartoA. G. (1998). Introduction to reinforcement learning. MIT Press.
70.
SuttonR. S.BartoA. G. (2018). Reinforcement learning: An introduction. MIT Press.
71.
SuttonR. S.ModayilJ.DelpM.DegrisT.PilarskiP. M.WhiteA.PrecupD. (2011). Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and MultiAgent Systems—Volume 2, AAMAS ’11 (pp. 761–768). International Foundation for Autonomous Agents and MultiAgent Systems. http://dl.acm.org/citation.cfm?id=2031678.2031726
72.
SuttonR. S.PrecupD.SinghS. (1999). Between MDPS and semi-MDPS: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1), 181–211. https://doi.org/10.1016/S0004-3702(99)00052-1
73.
SuttonR. S.SzepesváriC.GeramifardA.BowlingM. (2008). Dyna-style planning with linear function approximation and prioritized sweeping. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI.’08 (pp. 528–536). AUAI Press.
ThillS.CaligioreD.BorghiA.ZiemkeT.BaldassarreG. (2013). Theories and computational models of affordance and mirror systems: An integrative review. Neuroscience and Biobehavioral Reviews, 37. https://doi.org/10.1016/j.neubiorev.2013.01.012
76.
ThomasV.BengioE.FedusW.PondardJ.BeaudoinP.LarochelleH.PineauJ.PrecupD.BengioY. (2018). Disentangling the independently controllable factors of variation by interacting with the world (abs/1802.09484). CoRR. http://arxiv.org/abs/1802.09484
ToussaintM. (2004). Learning a world model and planning with a self-organizing, dynamic neural system. In ThrunS.SaulL. K.SchölkopfB. (Eds.), Advances in neural information processing systems16 (pp. 926–936). MIT Press.
UgurE.DogarM. R.CakmakM.SahinE. (2007). The learning and use of traversability affordance using range images on a mobile robot. In Proceedings of 2007 IEEE International Conference on Robotics and Automation (pp. 1721–1726). IEEE. https://doi.org/10.1109/ROBOT.2007.363571
81.
UgurE.SahinE.OztopE. (2009). Predicting future object states using learned affordances. In 2009 24th International Symposium on Computer and Information Sciences (pp. 415–419). IEEE.
82.
UrmsonC.AnhaltJ.BaeH.BagnellJ. A. D.BakerC. R.BittnerR. E.BrownT.ClarkM. N.DarmsM.DemitrishD.DolanJ. M.DugginsD.FergusonD.GalataliT.GeyerC. M.GittlemanM.HarbaughS.HebertM.HowardT.. . . ZiglarJ. (2008). Autonomous driving in urban environments: Boss and the urban challenge. Journal of Field Robotics, 25(8): 425–466.
83.
VinyalsO.BabuschkinI.CzarneckiW.MathieuM.DudzikA.ChungJ.ChoiD.PowellR.EwaldsT.GeorgievP.OhJ.HorganD.KroissM.DanihelkaI.HuangA.SifreL.CaiT.AgapiouJ.JaderbergM.SilverD. (2019). Grandmaster level in Starcraft II using multi-agent reinforcement learning. Nature, 575, 350–354. https://doi.org/10.1038/s41586-019-1724-z
WarrenW. (1984). Perceiving affordances: Visual guidance of stair climbing. Journal of Experimental Psychology: Human Perception and Performance, 10, 683–703. https://doi.org/10.1037/0096-1523.10.5.683
86.
WhiteA. (2015). Developing a predictive approach to knowledge [PhD thesis]. University of Alberta.
87.
WhiteM. (2017, August). Unifying task specification in reinforcement learning [Conference session]. Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia.
88.
WuJ.SunX.ZengA.SongS.LeeJ.RusinkiewiczS.FunkhouserT. (2020, July). Spatial action maps for mobile manipulation [Conference session]. Proceedings of Robotics: Science and Systems (RSS), Corvalis, OR, United States. https://doi.org/10.15607/RSS.2020.XVI.035
89.
YamanobeN.WanW.Ramirez-AlpizarI. G.PetitD.TsujiT.AkizukiS.HashimotoM.NagataK.HaradaK. (2017). A brief review of affordance in robotic manipulation research. Advanced Robotics, 31, 1086–1101.
90.
YangT.HaoJ.MengZ.ZhangZ.HuY.ChenY.FanC.WangW.LiuW.WangZ.PengJ. (2020). Efficient deep reinforcement learning via adaptive policy transfer. In BessiereC. (Ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 (pp. 3094–3100). International Joint Conferences on Artificial Intelligence Organization. https://doi.org/10.24963/ijcai.2020/428.
91.
YarrowK.BrownP.KrakauerJ. (2009). Inside the brain of an elite athlete: The neural processes that support high achievement in sports. Nature Reviews Neuroscience, 10, 585–596.
92.
ZechP.HallerS.LakaniS. R.RidgeB.UgurE.PiaterJ. (2017). Computational models of affordance in robotics: A taxonomy and systematic classification. Adaptive Behavior, 25(5), 235–271.
93.
ZengA.SongS.YuK. T.DonlonE.HoganF. R.BauzaM.MaD.TaylorO.LiuM.RomoE.FazeliN.AletF.DafleN. C.HolladayR.MoronaI.NairP. Q.GreenD.TaylorI.LiuW.. . . RodriguezA. (2018). Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. IEEE International Conference on Robotics and Automation (ICRA), pp. 3750–3757, doi: 10.1109/ICRA.2018.8461044.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
