Although many researchers have suggested that compositional concepts should be sensorimotor grounded, the method to accomplish this remains unclear. This article introduces a second-order neural network with parametric biases (sNNPB) that learns compositional structures based on sensorimotor time series data. The data was produced by a simulated robot that executed distinct object interactions (move-to and orient-toward). We show that various sNNPB setups can learn to compositionally imitate object interactions beyond the interactions that were specifically trained, which was not possible with previous neural network (NN) architectures, including recurrent neural networks (RNNs). We also show that these imitation capabilities are accomplished by developing a self-organized, geometrically arranged compositional concept structure in the PB values and task-oriented, Braitenberg-like sensory encodings in hidden sensory layers. Because second-order connections were necessary to accomplish the task, we hypothesize that such connections may be essential to drive the learning of both sensorimotor-grounded compositional structures and Braitenberg-like, behavior-oriented ‘‘pro-presentations.’’ From a cognitive perspective, we show how sensorimotor time series of interactions may be processed to generate the signals necessary to develop semantically compositional structures.
Andersen, R.A., Snyder, L.H., Bradley, D.C., & Xing, J. ( 1997). Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annual Review of Neuroscience, 20, 303-330.
2.
Arie, H., Endo, T., Jeong, S., Lee, M., Sugano, S., & Tani, J. ( 2010). Integrative learning between language and action: A neuro-robotics experiment. In 20th International Conference on Artificial Neural Networks, ICANN2010 (pp. 256-265).
3.
Barsalou, L.W. ( 1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577-600.
4.
Braitenberg, V. ( 1984). Vehicles: Experiments in synthetic psychology . Cambridge, MA: MIT Press.
5.
Cangelosi, A., & Riga, T. ( 2006). An embodied model for sensorimotor grounding and grounding transfer: Experiments with epigenetic robots. Cognitive Science , 30, 673-689.
Chang, S.W.C., Papadimitriou, C., & Snyder, L.H. ( 2009). Using a compound gain field to compute a reach plan. Neuron, 64, 744-755.
8.
Clausner, T. ( 2005). Image schema paradoxes: Implications for cognitive semantics . In B. Hampe (Ed.), From perception to meaning: Image schemas in cognitive linguistics (pp. 93-110). Berlin : Mouton de Gruyter.
9.
Dominey, P. ( 2006). From holophrases to abstract grammatical constructions: insights from simulation studies. In E. Clark & B. Kelly (Eds.), Construction in acquisition (pp. 137-162). Stanford CA : CSLI Publications.
10.
Fodor, J., & Lepore, E. ( 2002). The compositionality papers. Oxford : Oxford University Press.
11.
Gentner, D., & Markman, A. ( 1997). Structure mapping in analogy and similarity. American Psychologist, 52, 45-56.
12.
Graziano, M.S.A. ( 2006). The organization of behavioral repertoire in motor cortex . Annual Review of Neuroscience, 29, 105-134.
13.
Harnad, S. ( 1990). The symbol grounding problem. Physica D: Nonlinear Phenomena, 42, 335-346.
14.
Holmes, N.P., & Spence, C. ( 2004). The body schema and multisensory representation(s) of peripersonal space. Cognitive Processing, 5, 94-105.
15.
Johnson, M. ( 1987). The body in the mind: The bodily basis of meaning, imagination and reason. Chicago: University of Chicago Press.
16.
Karmiloff-Smith, A. ( 1992). Beyond modularity: A developmental perspective on cognitive science. Cambridge, MA: MIT Press .
17.
Lakoff, G. ( 1987). Women, fire, and dangerous things: What categories reveal about the mind. Chicago: University of Chicago Press.
18.
Mandler, J.M. ( 1992). How to build a baby: II. Conceptual primitives . Psychological Review, 99, 587-604.
19.
Markman, A., & Stilwell, C. ( 2001). Role-governed categories. Journal of Experimental and Theoretical Artificial Intelligence, 13, 329-358.
20.
Meltzoff, A. ( 1988a). Imitation, objects, tools, and the rudiments of language in human ontogeny. Human Evolution, 3, 45-64.
21.
Meltzoff, A. ( 1988b). Infant imitation and memory: Nine-month-olds in immediate and deferred tests. Child Development, 59, 217-225.
22.
Milner, A.D., & Goodale, M.A. ( 2008). Two visual systems re-viewed. Neuropsychologia , 46, 774-785.
23.
Nelson, K. ( 1988). Where do taxonomic categories come from?. Human Development, 31, 3-10.
24.
Nishide, S., Nakagawa, T., Ogata, T., Tani, J., Takahashi, T., & Okuno, H.G. ( 2009). Modeling tool-body assimilation using second-order recurrent neural network. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS-2009 (pp. 5376- 5381).
25.
Nishimoto, R., & Tani, J. ( 2004). Learning to generate combinatorial action sequences utilizing the initial sensitivity of deterministic dynamical systems. Neural Networks, 17, 925-933.
26.
Pastra K., Aloimonos Y. (in press). The minimalist grammar of action. Philosophical Transactions of the Royal Society B: Biological Sciences.
Riesenhuber, M., & Poggio, T. ( 2000). Models of object recognition. Nature Neuroscience, 3, 1199-1204.
29.
Rizzolatti, G., Fadiga, L., Fogassi, L., & Gallese, V. ( 1997). Enhanced: The space around us. Science , 277, 190-191.
30.
Salinas, E., & Sejnowski, T.J. (2001). Gain modulation in the central nervous system: Where behavior, neurophysiology, and computation meet . The Neuroscientist, 7, 430-440.
31.
Steels, L. ( 2008). The symbol grounding problem has been solved. So what’s next? In M. de Vega, A. M. Glenberg & A. C. Graesser (Eds.), Symbols and embodiment: Debates on meaning and cognition (pp. 223-244). New Haven: Academic Press.
32.
Sugita, Y., & Butz, M. ( 2010). Towards Emergent Strong Systematicity in a Simple Dynamical Connectionist Network. Paper presented at the Workshop on Cognitive and Neural Models for Automated Processing of Speech and Text (CONAS2010).
33.
Sugita, Y., & Butz, M.V. ( 2011). Compositionality and embodiment in harmony. AMD Newsletter, 8, 8-9.
34.
Sugita, Y., & Tani, J. ( 2005). Learning semantic combinatoriality from the interaction between linguistic and behavioral processes. Adaptive Behavior , 13, 33-52.
35.
Tani, J., Ito, M., & Sugita, Y. ( 2004). Self-organization of distributedly represented multiple behavior schemata in a mirror system: reviews of robot experiments using RNNPB . Neural Networks, 17, 1273-1289.
36.
Thrun, S., & O’Sullivan, J. (1996). Discovering structure in multiple learning tasks: The TC algorithm. In Proceedings of the Thirteenth International Conference on Machine Learning (pp. 489-497).
37.
Tuci, E., Ferrauto, T., Massera, G., & Nolfi, S. ( 2010). Co-development of linguistic and behavioural skills: Compositional semantics and behaviour generalisation. In Proceedings of the 11th International Conference on Simulation of Adaptive Behavior (SAB2010). Springer-Verlag, Berlin (pp. 523-532).
38.
Tversky, A. ( 1984). Features of Similarity. Psychological Review, 84, 327-352.
39.
Yamashita, Y., & Tani, J. ( 2008). Emergence of functional hierarchy in a multiple timescale neural network model: A humanoid robot experiment. PLoS Computational Biology, 4(11), e100022011.
